Kl

Keywords: mixture regression models, clustering, high dimension, variable selection, model selection, oracle inequality, functional data, electricity consumption, wavelets

High-dimensional mixture regression models, application to functional data.

Finite mixture regression models are useful for modeling the relationship between a response and predictors, arising from different subpopulations. In this thesis, we focus on high-dimensional predictors and a high-dimensional response. First of all, we provide an 1 -oracle inequality satisfied by the Lasso estimator. We focus on this estimator for its 1 -regularization properties rather than for the variable selection procedure. We also propose two procedures to deal with this issue. The first procedure leads to estimate the unknown conditional mixture density by a maximum likelihood estimator, restricted to the relevant variables selected by an 1 -penalized maximum likelihood estimator. The second procedure considers jointly predictor selection and rank reduction for obtaining lower-dimensional approximations of parameters matrices. For each procedure, we get an oracle inequality, which derives the penalty shape of the criterion, depending on the complexity of the random model collection. We extend these procedures to the functional case, where predictors and responses are functions. For this purpose, we use a wavelet-based approach. For each situation, we provide algorithms, apply and evaluate our methods both on simulations and real datasets. In particular, we illustrate the first procedure on an electricity load consumption dataset.

Modèles de mélange pour la régression en grande dimension, application aux données fonctionnelles.

Les modèles de mélange pour la régression sont utilisés pour modéliser la relation entre la réponse et les prédicteurs, pour des données issues de différentes sous-populations. Dans cette thèse, on étudie des prédicteurs de grande dimension et une réponse de grande dimension. Tout d'abord, on obtient une inégalité oracle 1 satisfaite par l'estimateur du Lasso. On s'intéresse à cet estimateur pour ses propriétés de régularisation 1 .

On propose aussi deux procédures pour pallier ce problème de classification en grande dimension. La première procédure utilise l'estimateur du maximum de vraisemblance pour estimer la densité conditionnelle inconnue, en se restreignant aux variables actives sélectionnées par un estimateur de type Lasso. La seconde procédure considère la sélection de variables et la réduction de rang pour diminuer la dimension. Pour chaque procédure, on obtient une inégalité oracle, qui explicite la pénalité nécessaire pour sélectionner un modèle proche de l'oracle. On étend ces procédures au cas des données fonctionnelles, où les prédicteurs et la réponse peuvent être des fonctions. Dans ce but, on utilise une approche par ondelettes. Pour chaque procédure, on fournit des algorithmes, et on applique et évalue nos méthodes sur des simulations et des données réelles. En particulier, on illustre la première méthode par des données de consommation électrique.

Mots-clés : modèles de mélange en régression, classification non supervisée, grande dimension, sélection de variables, sélection de modèles, inégalité oracle, données fonctionnelles, consommation électrique, ondelettes.

Introduction

Faire des classes pour mieux modéliser un échantillon est une méthode classique en statistiques. L'exemple pratique développé dans cette thèse est la classification de consommateurs électriques en Irlande. Habituellement, on classe des individus dans un problème d'estimation, mais ici, pour souligner l'aspect prédictif, on classe les observations qui ont le même type de comportement d'un jour sur l'autre. C'est une classification plus fine, qui agit sur des régresseurs homoscédastiques. Dans notre exemple, on classifie les consommateurs sur un modèle de régression d'un jour par rapport à leur consommation de la veille. Dans cette thèse, on a développé ce contexte d'un point de vue méthodologique, en proposant deux procédures de classification en régression par modèles de mélange qui pallient le problème de grande dimension, mais aussi d'un point de vue théorique en justifiant la partie sélection de modèles de nos procédures, et d'un point de vue pratique en travaillant sur le jeu de données réelles de consommation électrique en Irlande. Dans cette introduction, nous présentons les notions développées dans cette thèse.

Régression linéaire

Les méthodes de régression consistent à chercher le lien qui existe entre deux variables aléatoires X et Y . La variable X représente les régresseurs, les variables explicatives, alors que Y décrit la réponse. Le modèle linéaire Gaussien, le modèle le plus simple en régression, suppose que Y dépend de X de façon linéaire, à un bruit Gaussien près. Plus formellement, si X et Y sont deux vecteurs aléatoires, X ∈ R p et Y ∈ R, étudier un modèle linéaire Gaussien sur (X, Y ) consiste à trouver β ∈ R p tel que

Y = βX + (1)
où ∼ N (0, σ 2 ), σ 2 étant connue ou à estimer suivant les cas. L'étude de ce modèle est, à ce jour, assez complète. Soit (x, y) = ((x 1 , y 1 ), . . . , (x n , y n )) un échantillon. Si l'échantillon est de taille suffisante, on connaît un estimateur consistant (c'est-à-dire qui converge vers la vraie valeur), l'estimateur des moindres carrés (qui coïncide dans ce cas avec l'estimateur du maximum de vraisemblance). On notera ( β, σ2 ) cet estimateur, où β = (x t x) -1 x t y ; σ2 = ||y -x β|| 2 n -p .

11 Introduction ( β, Σ) ∈ R q×p × S ++ q , où β = (x t x) -1 x t y ;

(2) Σ = (y -x β) l 1 , (y -x β) l 2 n -p 1≤l 1 ≤q 1≤l 2 ≤q ; et S ++ q est l'ensemble des matrices symétriques définies positives de taille q. Ce modèle de régression est utilisé, par exemple, pour prédire de nouvelles valeurs. Si on connaît le modèle sous-jacent à nos données, c'est-à-dire que l'on a déterminé β et Σ à partir d'un échantillon ((x 1 , y 1 ), . . . , (x n , y n )), et que l'on observe un nouvel x n+1 , on peut calculer ŷn+1 = βx n+1 . Dans ce cas ŷn+1 est la valeur dite prédite. Si on s'intéresse au couple des variables aléatoires (X, Y ), on peut estimer la densité du couple, mais on peut aussi étudier la densité conditionnelle. C'est cette dernière quantité que l'on a décrite par un modèle linéaire dans (1). Les covariables peuvent aussi avoir différents statuts : soit elles sont fixées, déterministes, soit elles sont aléatoires, et on travaille alors conditionnellement à la loi sous-jacente. Dans cette thèse, on s'intéresse à la loi conditionnelle, pour des régresseurs fixes ou aléatoires. Cependant, cette hypothèse de modèle linéaire est très contraignante : si on observe un échantillon (x, y) = ((x 1 , y 1 ), . . . , (x n , y n )), on suppose que chaque y i dépend de la même façon de x i , à un bruit près, pour i ∈ {1, . . . , n}. Si le modèle est adapté aux données, le bruit sera petit, ce qui signifie que les coefficients de la matrice de covariance Σ seront petits. Cependant, de nombreux jeux de données ne peuvent pas être bien résumés par un modèle linéaire.

Modèles de mélange en régression

Pour affiner ce modèle, on peut choisir de construire plusieurs classes, et de faire dépendre nos estimateurs β et Σ de la classe. Plus formellement, on étudie un modèle de mélange en régression de K Gaussiennes : si on observe un échantillon (x, y) = ((x 1 , y 1 ), . . . , (x n , y n )) ∈ (R p × R q ) n , et si on suppose que l'observation i appartient à la classe k, alors il existe β k et Σ k tels que

y i = β k x i + i où i ∼ N (0, Σ k ).
Dans cette thèse, on considère un mélange avec un nombre de classes K fini. Les modèles de mélange sont traditionnellement étudiés dans des problèmes d'estimation (citons par exemple McLachlan et Basford, [START_REF] Mclachlan | Mixture Models: Inference and Applications to Clustering[END_REF], McLachlan et Peel, [START_REF] Mclachlan | Finite Mixture Models[END_REF], qui sont deux livres fondateurs pour les modèles de mélange, Fraley et Raftery, [START_REF] Fraley | Model-based clustering, discriminant analysis, and density estimation[END_REF], pour un état de l'art, ou encore Celeux et Govaert, [START_REF] Celeux | Gaussian parsimonious clustering models[END_REF], pour l'étude d'une densité de mélange dans un but de classification). L'idée principale est d'estimer la densité inconnue s * par un mélange de densités classiques (s θ k ) 1≤k≤K : on peut alors écrire s * = Plusieurs idées émergent avec les modèles de mélange, en régression ou non. Si on connaît les paramètres de notre modèle, on peut calculer pour chaque observation la probabilité a posteriori d'appartenir à une classe. D'après le principe du maximum a posteriori (noté MAP), on peut alors affecter une classe à chaque observation. Pour ce faire, on calcule la probabilité a posteriori de chaque observation d'appartenir à une classe, et on affecte les observations aux classes les plus probables. On peut ainsi caractériser chaque observation par les paramètres de la densité conditionnelle associée à la classe. Dans le cadre de la classification supervisée, on connaît l'affectation de chaque observation, et on cherche à comprendre la formation des classes pour classer une nouvelle observation ; quand on fait de la classification semi-supervisée, on connaît certaines affectations, et on cherche à comprendre le modèle (souvent, c'est très coûteux de connaître les affectations des observations) ; dans le cadre de la classification non supervisée, on ne connaît pas du tout les affectations. Nous nous plaçons dans cette thèse dans le contexte de classification non supervisée, avec une approche en régression. Cela a déjà été envisagé, citons par exemple Städler et coauteurs ( [START_REF] Städler | 1 -penalization for mixture regression models[END_REF]) ou Meynet ([Mey13]), qui travaillent avec ce modèle pour des réponses Y univariées. On observe des couples (x, y) = ((x 1 , y 1 ), . . . , (x n , y n )) ∈ (R p × R q ) n , et on veut faire des classes, pour regrouper les observations (x i , y i ) pour i ∈ {1, . . . , n} qui ont la même relation entre y i et x i . Sur certains jeux de données, cette approche semble naturelle, et on connaît le nombre de classes. Néanmoins, sur un jeu de données quelconque, on va surtout chercher à mieux comprendre la relation entre les variables aléatoires Y et X en regroupant les observations qui ont la même dépendance entre Y et X. La structure de groupes n'est pas forcément clairement induite par les données, et on ne sait pas forcément combien de classes on a intérêt à former : dans certains cas, il va falloir estimer K. Si on considère le modèle de mélange de Gaussiennes multivariées en régression à K classes, on Introduction peut décrire ce modèle à l'aide des outils statistiques classiques. Si on suppose que les vecteurs aléatoires Y sont indépendants conditionnellement à X, on considère la densité conditionnelle s K ξ , où

s K ξ : R q → R y → s K ξ (y|x) = K k=1 π k (2π) q/2 det(Σ k ) 1/2 exp - 1 2 (y -β k x) t Σ -1 k (y -β k x) ;
où les paramètres à estimer sont ξ = (π 1 , . . . , π K , β 1 , . . . , β K , Σ 1 , . . . , Σ K ) ∈ Ξ K , avec

Ξ K = Π K × R q×p K × S ++ q K ; Π K = (π 1 , . . . , π K ) ∈ [0, 1] K K k=1 π k = 1 .
A partir de cette densité conditionnelle, on peut définir l'estimateur du maximum de vraisemblance par

ξEMV K = argmin ξ∈Ξ K - 1 n l(ξ, x, y) ; (3)
où la log-vraisemblance est définie par l(ξ, x, y) = n i=1 log s K ξ (y i |x i ) .

Comme dans la plupart des modèles de mélange classiques, dans le cadre de la régression, l'estimateur du maximum de vraisemblance n'est pas explicite. Cela complique l'analyse théorique : on n'a pas accès à la loi des estimateurs, ni des affectations. D'un point de vue pratique, on a recours à l'algorithme EM pour approcher cet estimateur. Cet algorithme, introduit par Dempster et coauteurs dans [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF], permet d'estimer les paramètres d'un modèle de mélange. Il consiste à alterner deux étapes, jusqu'à convergence des paramètres ou d'une fonction des paramètres. Décrivons ce résultat.

On note Z = (Z 1 , . . . , Z n ) le vecteur aléatoire des affectations des observations, et Z l'ensemble de toutes les partitions possibles : Z i,k = 1 si l'observation i est issue de la classe k, 0 sinon. On notera aussi, pour i ∈ {1, . . . , n}, f (z i |x i , y i , ξ) la probabilité que Z i = z i sachant (x i , y i ) et connaissant le paramètre ξ. On définit la log-vraisemblance complétée par l c (ξ, (x, y, z)) = n i=1 log f (z i |x i , y i , ξ) + log s K ξ (y i |x i ) .

Alors, on peut décomposer la log-vraisemblance comme suit :

l(ξ, x, y) = Q(ξ| ξ) -H(ξ| ξ)
où Q(ξ| ξ) est l'espérance sur les variables latentes Z de la vraisemblance complétée,

Q(ξ| ξ) = z∈Z n i=1 l c (ξ, x i , y i , z i )f (z i |x i , y i , ξ) et H(ξ| ξ) = z∈Z n i=1
log f (z|x i , y i , ξ)f (z i |x i , y i , ξ) l'espérance étant prise sur les variables latentes Z. Dans l'algorithme EM, on itère le calcul jusqu'à la convergence et la maximisation de Q(ξ|ξ (ite) ), où ξ (ite) est l'estimation des paramètres à l'itération (ite) ∈ N * de l'algorithme EM. En effet, si on fait croître ξ → Q(ξ|ξ (ite) ), on fait aussi croître la vraisemblance. Dempster, dans [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF], a donc proposé l'algorithme suivant.

Algorithme 1 : Algorithme EM Data : x,y,K Result : ξEMV

Introduction

Sélection de variables et estimateur du Lasso

Nous nous sommes intéressés à un problème de classification en régression non supervisée de données en grande dimension, c'est-à-dire que les vecteurs aléatoires X ∈ R p et Y ∈ R q peuvent être des vecteurs de grande taille, éventuellement plus grande que le nombre d'observations n.

Ce problème est très étudié actuellement. En effet, avec l'amélioration de l'informatique, on a de plus en plus de données de grande dimension, et on a accès à de plus en plus de variables explicatives pour décrire un même objet. Dans le cadre du modèle linéaire, si p > n, on ne peut plus calculer β avec la formule (2) : la matrice x t x n'est plus inversible. En fait, on cherche à estimer, dans le cas du modèle linéaire, pq + q 2 paramètres, ce qui peut être plus grand que le nombre d'observations n si p et q sont grands.

Dans un premier temps, restreignons-nous au modèle linéaire. L'estimateur du Lasso, introduit par Tibshirani dans [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF], et parallèlement par Chen et coauteurs en théorie du signal dans [START_REF] Chen | Atomic decomposition by basis pursuit[END_REF], est un estimateur classique pour pallier le problème de la grande dimension. On peut aussi citer l'estimateur de Dantzig, introduit par Candès et Tao dans [START_REF] Candes | The Dantzig selector: Statistical estimation when p is much larger than n[END_REF] ; l'estimateur Ridge, qui utilise une pénalité 2 plutôt que la pénalité 1 exploitée par l'estimateur du Lasso, introduit par Hoerl et Kennard dans [START_REF] Hoerl | Ridge Regression: Biased Estimation for Nonorthogonal Problems[END_REF] ; l'Elastic net, introduit par Zou et Hastie dans [START_REF] Zou | Regularization and variable selection via the elastic net[END_REF], et qui est défini avec une double pénalisation 1 et 2 , qui fait donc un compromis entre l'estimateur Ridge et l'estimateur Lasso ; le Fused Lasso, introduit par Tibshirani dans [TSR + 05] ; le Lasso adaptatif, introduit par Zou dans [START_REF] Zou | The adaptive lasso and its oracle properties[END_REF], l'estimateur du Group-Lasso, introduit par Yuan et Lin dans [START_REF] Yuan | Model selection and estimation in regression with grouped variables[END_REF], pour avoir de la parcimonie par groupes, . . . Dans cette thèse, nous nous concentrerons sur l'estimateur du Lasso (ou du Group-Lasso), même si certains résultats théoriques restent valables pour n'importe quelle méthode de sélection de variables. L'idée introduite par [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF] est de supposer que la matrice β dans le modèle linéaire est creuse, ce qui réduit le nombre de paramètres à estimer. En effet, s'il y a peu de coefficients non nuls, en notant J ⊂ P ({1, . . . , q} × {1, . . . , p}) l'ensemble des indices des coefficients de la matrice de régression non nuls et |J| son cardinal, la taille de l'échantillon pourra être plus grande que le nombre de paramètres à estimer, qui est alors |J| + q 2 . Dans le cas du modèle linéaire (1), l'estimateur du Lasso est défini par βLasso (λ) = argmin

β∈R q×p ||Y -βX|| 2 2 + λ||β|| 1 ; (4) β β
Figure 2 -Illustration de l'estimateur du Lasso (à gauche) et de l'estimateur Ridge (à droite). β représente l'estimateur des moindres carrés, les lignes correspondant à l'erreur des moindres carrés, la boule 1 correspond au problème de l'estimateur du Lasso et la boule 2 correspond au problème de l'estimateur Ridge. Cette figure est issue de [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF].

Cet estimateur a été beaucoup étudié ces dernières années. Citons Friedman, Hastie et Tibshirani, [START_REF] Hastie | The Elements of Statistical Learning[END_REF] qui ont étudié le chemin de régularisation du Lasso ; Bickel, Ritov et Tsybakov, [START_REF] Bickel | Simultaneous analysis of Lasso and Dantzig selector[END_REF], qui ont étudié cet estimateur en comparaison avec le sélecteur Dantzig. On peut aussi citer Osborne, [START_REF] Osborne | On the lasso and its dual[END_REF], qui étudie l'estimateur du Lasso et sa forme duale. Notons que la norme 1 apparaît ici comme une relaxation convexe de la pénalité 0 (où ||β|| 0 = Card(j|β j = 0)), qui vise à estimer les coefficients par 0. Grâce à la géométrie de la boule 1 , le Lasso a tendance à annuler des coefficients, comme on peut le voir sur la figure 2. Plus λ est grand, plus on pénalise, et plus on a de coefficients de βLasso (λ) qui sont nuls. Si λ = 0, on retrouve l'estimateur du maximum de vraisemblance. Résumons les résultats principaux obtenus ces dernières années pour l'estimateur du Lasso. Cet estimateur peut facilement être approximé par l'algorithme LARS, introduit dans [START_REF] Efron | Least angle regression[END_REF]. La relaxation convexe de la pénalité 0 par la pénalité 1 permet d'approcher numériquement plus facilement cet estimateur. On sait qu'il est linéaire par morceaux, et on peut expliciter ses valeurs grâce aux conditions de Karush-Kuhn-Tucker. Citons par exemple [START_REF] Efron | Least angle regression[END_REF] ou [START_REF] Zou | On the "degrees of freedom" of the lasso[END_REF]. D'un point de vue théorique, sous des hypothèses plus ou moins fortes, il existe des inégalités oracle pour l'erreur de prédiction ou l'erreur q des coefficients estimés, avec un paramètre de régularisation de l'ordre de log(p)/n. On peut citer par exemple Bickel, Ritov et Tsybakov, [START_REF] Bickel | Simultaneous analysis of Lasso and Dantzig selector[END_REF], qui obtiennent une inégalité oracle pour le risque en prédiction dans un modèle général non paramétrique en régression, et une inégalité oracle pour la perte p en estimation (1 ≤ p ≤ 2) pour le modèle linéaire. Les hypothèses nécessaires et les résultats obtenus dans ce sens sont résumés par van de Geer et Bühlmann dans [START_REF] Van De Geer | On the conditions used to prove oracle results for the Lasso[END_REF]. Il est aussi à noter que l'estimateur du Lasso a de bonnes propriétés de sélection de variables. Citons par exemple Meinshausen et Bühlmann [START_REF] Meinshausen | High-dimensional graphs and variable selection with the lasso[END_REF], Zhang et Huang, dans [START_REF] Zhang | The sparsity and bias of the LASSO selection in highdimensional linear regression[END_REF], Zhao et Yu, dans [START_REF] Zhao | On model selection consistency of lasso[END_REF], ou Meinshausen et Yu, [START_REF] Meinshausen | Lasso-type recovery of sparse representations for high-dimensional data[END_REF], qui montrent que le Lasso est consistant en sélection de variables sous diverses hypothèses plus ou moins contraignantes. Il paraît donc cohérent d'utiliser l'estimateur du Lasso pour sélectionner les variables importantes. Dans les modèles de mélange de modèles linéaires Gaussiens, on peut étendre la définition de l'estimateur du Lasso par ξLasso (λ) = argmin

ξ∈Ξ K Introduction où l λ (ξ, x, y) = l(ξ, x, y) -λ K k=1 π k ||P k β k || 1 ; Ξ K = Π K × (R q×p ) K × S ++ q K ;
où P k est la racine de Cholesky de l'inverse de la matrice de covariance, i.e. P t k P k = Σ -1 k , pour tout k ∈ {1, . . . , K}. Cette définition a été introduite par Städler et coauteurs dans [START_REF] Städler | 1 -penalization for mixture regression models[END_REF]. La pénalité est différente de celle de l'estimateur (4). Premièrement, on ne pénalise non pas par la moyenne conditionnelle dans chaque classe, mais par une version reparamétrisée par la variance. En effet, dans les modèles de mélange, il est important d'avoir un bon estimateur de la variance, et de l'estimer en même temps que la moyenne, pour ne pas favoriser les classes à variance trop élevée. De plus, pour avoir un estimateur invariant par changement d'échelle, il faut prendre en compte la variance dans la pénalité 1 . Ensuite, l'étude de la racine de Cholesky de l'inverse de la matrice de covariance plutôt que de la matrice de covariance permet d'obtenir une optimisation convexe, ce qui facilitera la partie algorithmique. Enfin, on pénalise la vraisemblance par la somme de l'estimateur de la moyenne reparamétrisé par la variance, pondérée sur chaque classe, pour prendre en compte la différence de taille entre les différentes classes. Notons que la sélection de variables dans les modèles de mélange a déjà été envisagée dans des problèmes d'estimation. Citons par exemple Raftery et Dean, [START_REF] Raftery | Variable selection for model-based clustering[END_REF], ou Maugis et Michel dans [START_REF] Maugis | A non asymptotic penalized criterion for Gaussian mixture model selection[END_REF]. L'estimateur (5) peut s'approcher algorithmiquement à l'aide d'une généralisation de l'algorithme EM, introduite pas Städler et coauteurs dans le cas univarié, et étendue dans cette thèse. Dans le cadre des modèles de mélange, en régression, on connaît principalement deux résultats théoriques pour l'estimateur du Lasso, valables pour Y réel et à nombre de classes K fixé et connu. Pour Y ∈ R, et X ∈ R p , Städler et coauteurs, dans [START_REF] Städler | 1 -penalization for mixture regression models[END_REF], ont montré que, sous la condition de valeurs propres restreintes (notée REC, citée ci-dessous), l'estimateur du Lasso vérifie une inégalité oracle pour des covariables fixes. Rappelons le contexte de mélange de Gaussiennes en régression univariées. Si Y , conditionnellement à X, appartient à la classe k, on note Y = β k X + , avec ∼ N (0, σ 2 k ). On note de plus φ k = σ -1 k β k , et J l'ensemble des indices des coefficients non nuls de la matrice de régression.

Hypothèse REC. Il existe κ ≥ 1 tel que, pour tout φ ∈ (R p ) K vérifiant ||φ J c || ≤ 6||φ J || 1 , on a

||φ J || 2 2 ≤ κ 2 K k=1 φ t k Σx φ k où Σx = 1 n n i=1 x t i x i .
l'ensemble des paramètres ΞK = ξ ∈ Ξ K pour tout k ∈ {1, . . . , K}, max z∈{1,...,q}

sup x∈[0,1] p |[β k x] z | ≤ A β , a Σ ≤ m(Σ -1 k ) ≤ M (Σ -1 k ) ≤ A Σ , a π ≤ π k ; (6)
où m(A) et M (A) désignent respectivement la valeur absolue de la plus petite et de la plus grande valeur propre de la matrice A. Soit, pour ξ = (π, β, Σ) ∈ Ξ K ,

N [2] 1 (s K ξ ) = ||β|| 1 = K k=1 p j=1 q z=1 |[β k ] z,j |. ( 7
)
la pénalité envisagée, et KL n la divergence de Kullback-Leibler à design fixé :

KL n (s, t) = 1 n n i=1 KL(s(.|x i ), t(.|x i )) = 1 n n i=1 E s log s(.|x i ) t(.|x i ) .
Voici le théorème que nous obtenons.

Inégalité oracle 1 pour le Lasso. Soit (x, y) = (x i , y i ) 1≤i≤n les observations, issues d'une densité conditionnelle inconnue s * = s ξ 0 , où ξ 0 ∈ ΞK , cet ensemble étant défini par l'équation (6), et le nombre de classes K étant fixé. On notera a ∨ b = max(a, b). Soit N [2] 1 (s K ξ ) définie par (7). Pour λ ≥ 0, on définit l'estimateur du Lasso, noté ŝLasso (λ), par ŝLasso (λ) = argmin

s K ξ ∈S - 1 n n i=1 log(s K ξ (y i |x i )) + λN [2] 1 (s K ξ ) ; (8)
avec S = s K ξ , ξ ∈ ΞK . Si λ ≥ κ A Σ ∨ 1 a π 1 + 4(q + 1)A Σ A 2 β + log(n) a Σ K n 1 + q log(n) K log(2p + 1)
avec κ une constante positive, alors l'estimateur (8) vérifie l'inégalité suivante :

E[KL n (s * , ŝLasso (λ))] ≤(1 + κ -1 ) inf s K ξ ∈S KL n (s * , s K ξ ) + λN [2] 1 (s K ξ ) + λ + κ K n e -1 2 π q/2 a π A q/2 Σ 2q + κ K 3/2 √ n A Σ ∨ 1 a π 1 + 4(q + 1)A Σ A 2 β + log(n) a Σ × 1 + A β + q a Σ

Introduction

Ce théorème peut être vu comme une inégalité oracle 1 , mais ce n'est pas l'approche que l'on souhaite développer ici. En effet, la démonstration de ce théorème passe par un théorème de sélection de modèles, mais ce thème sera abordé dans la partie correspondante. Ici, on voit plutôt ce théorème comme une assurance que l'estimateur du Lasso fonctionne bien pour la régularisation 1 . La particularité de ce résultat est qu'il ne requiert que peu d'hypothèses : on travaille avec des prédicteurs fixés, qui sont supposés (sans restriction) être inclus dans [0, 1] p , et les paramètres de nos densités conditionnelles sont supposés bornés, au sens où ils appartiennent à ΞK . Cette hypothèse est nécessaire, pour assurer en particulier que la vraisemblance est finie.

On la retrouvera dans les autres théorèmes démontrés dans cette thèse. On remarque aussi que la borne sur le paramètre de régularisation λ n'est pas celle, optimale, classique, obtenue dans d'autres cas plus généraux, mais cela est dû au fait que nous ne faisons pas d'hypothèses sur les régresseurs. L'article de van de Geer, [vdG13], permet d'obtenir cette borne optimale sous des hypothèses plus fortes sur le design. Il est à noter que le design est supposé fixe ici. Comme l'estimateur du Lasso surestime les paramètres (citons par exemple Fan et Peng, [START_REF] Fan | Nonconcave penalized likelihood with a diverging number of parameters[END_REF], ou Zhang [START_REF] Zhang | Nearly unbiased variable selection under minimax concave penalty[END_REF]), nous proposons de l'utiliser pour la sélection de variables, et non pour l'estimation des paramètres. Ainsi, pour un paramètre de régularisation λ ≥ 0 à définir, on sélectionnera les variables importantes pour expliquer Y en fonction de X. Plus formellement, définissons la notion de variable active pour la classification.

Définition. Une variable est active pour la classification si elle est non nulle dans au moins une classe : la variable d'indice (z, j) ∈ {1, . . . , q} × {1, . . . , p} est active s'il existe k ∈ {1, . . . , K} tel que [β k ] z,j = 0.

Ainsi, pour un certain λ ≥ 0, on peut estimer ξLasso (λ) et en déduire l'ensemble J λ des variables actives pour la classification.

Ré-estimations

En se restreignant aux variables sélectionnées indicées par J λ par l'estimateur du Lasso de paramètre de régularisation λ ≥ 0, on travaille avec un modèle de dimension plus petite. En effet, plutôt que (pq + q 2 + 1)K -1 paramètres à estimer, on en a (|J λ | + q 2 + 1)K -1. Si on suppose de plus que la matrice de covariance est diagonale (ce qui implique que les variables sont non corrélées), on obtient un modèle de dimension (|J λ | + q + 1)K -1, et la dimension du modèle peut devenir plus petite que le nombre d'observations, ou au moins être raisonnable. On peut alors utiliser un autre estimateur, restreint aux variables sélectionnées, qui aura de bonnes propriétés d'estimation en dimension raisonnable (meilleures que celles de l'estimateur du Lasso). Ré-estimer les paramètres sur les variables sélectionnées n'est pas une idée nouvelle. On veut tirer parti des avantages de la sélection de variables par l'estimateur du Lasso (ou par une autre technique), mais on veut aussi diminuer le biais induit par cet estimateur. Citons par exemple Belloni et Chernozhukov, [START_REF] Belloni | 1 -penalized quantile regression in highdimensional sparse models[END_REF], qui obtiennent une inégalité oracle pour montrer que l'estimateur du maximum de vraisemblance calculé sur les variables sélectionnées par le Lasso fonctionne mieux que l'estimateur du Lasso, pour un modèle linéaire en grande dimension. On peut aussi citer Zhang et Sun, [START_REF] Sun | Scaled sparse linear regression[END_REF], qui estiment le bruit et la matrice de régression dans un modèle linéaire en grande dimension par l'estimateur des moindres carrés après sélection de modèles.

On propose dans une première procédure, appelée procédure Lasso-EMV, d'estimer les paramètres, en se restreignant aux variables actives, par l'estimateur du maximum de vraisemblance, qui a de bonnes propriétés pour un échantillon suffisamment grand.

On propose aussi, dans une seconde procédure que l'on appellera procédure Lasso-Rang, d'utiliser le maximum de vraisemblance avec une contrainte de faible rang. En effet, jusqu'ici, nous n'avons pas tenu compte de la structure matricielle de β. Comme les matrices de covariance (Σ k ) 1≤k≤K sont supposées diagonales, on aurait pu travailler avec chaque coordonnée de Y comme q problèmes distincts et indépendants. En cherchant une structure de faible rang, on suppose que peu de combinaisons linéaires de prédicteurs suffisent à expliquer la réponse. C'est aussi une seconde méthode pour diminuer la dimension, au cas où la sélection de variables par pénalisation 1 ne soit pas suffisante. Certains jeux de données sont particulièrement propices à cette réduction dimensionnelle par faible rang. On peut citer par exemple l'analyse d'image fMRI (Harrison, Penny, Frishen, [START_REF] Friston | Dynamic Causal Modelling[END_REF]), l'analyse de décodage de données EEG (Anderson, Stolz, Shamsunder, [START_REF] Anderson | Multivariate autoregressive models for classification of spontaneous electroencephalographic signals during mental tasks[END_REF]), la modélisation de réponse de neurones (Brown, Kass, Mitra, [START_REF] Brown | Multiple neural spike train data analysis: stateof-the-art and future challenges[END_REF]), ou encore l'analyse de données génomiques (Bunea, She, Wegkamp, [START_REF] Bunea | Optimal selection of reduced rank estimators of high-dimensional matrices[END_REF]). D'un point de vue plus théorique, on peut citer Izenman ([Ize75]) qui a introduit cette méthode dans le cas du modèle linéaire, Giraud ([Gir11]) ou Bunea et coauteurs ( [START_REF] Bunea | Consistent selection via the Lasso for high dimensional approximating regression models[END_REF]) qui ont complété l'étude théorique et pratique de sélection de rang. Dans cette thèse, nous employons ces méthodes dans un cadre de mélange en régression. Il est à noter qu'on aura sélectionné, par l'estimateur du Lasso, des colonnes, ou des lignes, ou les deux, mais que les lignes ou les colonnes où seuls certains coefficients sont inactifs ne seront pas sélectionnées, la structure matricielle étant requise pour estimer un paramètre par faible rang. On n'impose pas que toutes les moyennes conditionnelles aient le même rang. Notons ξEMV J et ξRang J les estimateurs associés à chaque procédure, avec J comme ensemble des variables actives :

ξEMV J = argmin ξ∈Ξ (K,J) - 1 n l(ξ [J] , x, y) ; (9) ξRang J = argmin ξ∈ Ξ(K,J,R) - 1 n l(ξ [J] , x, y) ; (10) où Ξ(K,J,R) = ξ = (π, β, Σ) ∈ Ξ (K,J) | rang(β k ) = R k pour tout k ∈ {1, .
. . , K} , où ξ [J] signifie que l'on a sélectionné les variables d'indice J, et où Ξ (K,J) est défini par

Ξ (K,J) = Π K × (R q×p ) K × (S ++ q ) K Π K = (π 1 , . . . , π K ) ∈ (0, 1) K K k=1 π k = 1
A noter que l'on ré-estime tous les paramètres de notre modèle : les moyennes conditionnelles, les variances et les poids. D'un point de vue pratique, la généralisation de l'algorithme EM (Algorithme 1 page 14) permet de calculer l'estimateur du maximum de vraisemblance, sous contrainte de faible rang ou non, dans le cas de mélange de Gaussiennes en régression.

m O = argmin m∈M [l(s * , ŝm )]. (11)
Malheureusement, on ne peut pas évaluer cette quantité, puisque l'on n'a pas accès à s * . On va utiliser l'oracle en théorie pour évaluer notre sélection de modèles : on veut que le risque associé à l'estimateur du modèle sélectionné soit le plus près possible de celui de l'oracle. Une partie des résultats théoriques en sélection de modèles sont des inégalités oracle, qui permettent d'assurer la cohérence de la sélection de modèles. Ces inégalités sont de la forme, pour m l'indice du modèle sélectionné,

E(l(s * , ŝ m)) ≤ C 1 E(l(s * , s O )) + C 2 n (12)
où (C 1 , C 2 ) sont des constantes absolues, avec C 1 la plus proche possible de 1. L'inégalité oracle est dite exacte si C 1 = 1.

Désormais, décrivons comment sélectionner un modèle. On va minimiser un critère pénalisé, pour parvenir à un compromis biais/variance. En effet, on peut décomposer le risque de la façon suivante :

l(s * , ŝm ) = l(s * , s m ) biaism + E (γ(s m ) -γ(ŝ m )) variancem ; où s m ∈ argmin t∈Sm [E(γ(t))] (c
l(s * , ŝm ) -γ n (ŝ m ) = E(γ(ŝ m )) -E(γ(s m )) vm + E(γ(s m )) -E(γ(s * )) (1) -(γ n (ŝ m ) -γ n (s m )) vm -(γ n (s m ) -γ n (s * ))
∈ R + et tout s m ∈ S m , 0 H [.] ( , S m (s m , ), d H )d ≤ φ m ( ); où S m (s m , ) = {t ∈ S m , d H (t, s m ) ≤ }. La complexité du modèle D m est alors définie par n 2 m avec m l'unique solution de 1 φ m ( ) = √ n . ( 13
)
Notons que la complexité du modèle ne dépend pas du crochet d'entropie des modèles globaux S m , mais des ensembles plus petits, localisés. C'est une hypothèse plus faible. Pour des raisons techniques, une hypothèse de séparabilité est aussi nécessaire.

Hypothèse Sep m . Il existe un ensemble dénombrable S m de S m et un ensemble Y m avec λ(R q \ Y m ) = 0, pour λ la mesure de Lebesgue, tel que pour tout t ∈ S m , il existe une suite (t l ) l≥1 d'éléments de S m telle que pour tout y ∈ Y m , log(t l (y)) tend vers log(t(y)) quand l tend vers l'infini.

On a aussi besoin d'une hypothèse de théorie de l'information sur notre collection de modèles.

Hypothèse K. La famille de nombres positifs (w m ) m∈M vérifie m∈M e -wm ≤ Ω < +∞.

Alors, on peut écrire le théorème général de sélection de modèles.

Inégalité oracle pour une famille d'EMV. Soient (X 1 , . . . , X n ) des variables aléatoires de densité inconnue s * . On en observe une réalisation (x 1 , . . . , x n ). Soit {S m } m∈M une collection de modèles au plus dénombrable, où, pour tout m ∈ M, les éléments de S m sont des densités de probabilité, et S m vérifie l'hypothèse Sep m . On considère de plus la collection d'estimateurs de maxima de vraisemblance à ρ près notés (ŝ m ) m∈M : on a, pour tout m ∈ M, -

1 n n i=1 ln(ŝ m (x i )) ≤ inf t∈Sm - 1 n n i=1 ln(t(x i )) + ρ.

Introduction

Soient {w m } m∈M une famille de nombres positifs vérifiant l'hypothèse K, et, pour tout m ∈ M, on considère φ m qui vérifie la condition H m , avec m l'unique solution positive de l'équation

φ m ( ) = √ n 2 .
On suppose de plus que, pour tout m ∈ M, l'hypothèse Sep m est vérifiée. Soit pen : M → R + et soit le critère de log-vraisemblance pénalisé

crit(m) = - 1 n n i=1 ln(ŝ m (x i ))) + pen(m).
Alors, il existe des constantes κ et C telles que, lorsque

pen(m) ≥ κ 2 m + w m n
pour tout m ∈ M, alors il existe m qui minimise le critère crit sur M, et de plus, Ces auteurs voient le problème de sélection du nombre de composantes et de sélection de variables comme un problème de sélection de modèles. Pour le rang, Giraud, dans [START_REF] Giraud | Low rank multivariate regression[END_REF], et Bunea, dans [START_REF] Bunea | Consistent selection via the Lasso for high dimensional approximating regression models[END_REF], proposent une pénalité pour choisir de façon optimale le rang. Ils obtiennent, à variance connue et inconnue, des inégalités oracle qui permettent de sélectionner le rang, où la pénalité est proportionnelle au rang. Ma et Sun, dans [START_REF] Ma | Adaptive sparse reduced-rank regression[END_REF], obtiennent pour ces modèles une borne minimax, ce qui revient à dire que la pénalité construite est optimale pour la sélection du rang. Les procédures d'estimation que l'on a décrites dans la partie précédente sont sujettes à des choix de paramètres (le nombre de classes, le paramètre de régularisation du Lasso, et le rang dans la deuxième procédure). On peut réécrire la sélection de ces paramètres comme un problème de sélection de modèles : en faisant varier ces paramètres, on obtient une collection de modèles.

E s (d 2 H (s, ŝ m)) ≤ C inf m∈M inf t∈Sm KL(s, t) + pen(m) + ρ + Ω n où d H est
Commençons par définir les collections de modèles associés à chacune de nos procédures.

Pour la procédure Lasso-EMV, pour (K, J) ∈ K × J ,

S (K,J) = y ∈ R q → s (K,J) ξ (y|x) (14) s (K,J) ξ (y|x) = K k=1 π k 2π det(Σ k ) exp - 1 2 (y -β [J] k x) t Σ -1 k (y -β [J] k x) ξ = (π 1 , . . . , π K , β
Pour la procédure Lasso-Rang, pour (K, J, R) ∈ K × J × R,

S (K,J,R) = y ∈ R q → s (K,J,R) ξ (y|x) (15) s (K,J,R) ξ (y|x) = K k=1 π k 2π det(Σ k ) exp - 1 2 (y -(β R k k ) [J] x) t Σ -1 k (y -(β R k k ) [J] x) ξ = (π 1 , . . . , π K , (β R 1 1 ) [J] , . . . , (β R K K ) [J] , Σ 1 , . . . , Σ K ) ∈ Ξ (K,J,R) Ξ(K,J,R) = Π K × Ψ (K,J,R) × (S ++ q ) K Ψ (K,J,R) = ((β R 1 1 ) [J] , . . . , (β R K K ) [J] ) ∈ (R q×p ) K pour tout k ∈ {1, . . . , K}, Rang(β k ) = R k .
où K est l'ensemble des valeurs possibles pour le nombre de composantes, J est l'ensemble des ensembles d'indices de variables actives possibles, et R est l'ensemble des valeurs possibles pour les vecteurs de rang.

Pour obtenir des résultats théoriques, on a besoin de borner nos paramètres. On considère

S B (K,J) = s (K,J) ξ ∈ S (K,J) ξ ∈ Ξ(K,J) (16)
Ξ(K,J) = Π K × ([-A β , A β ] |J| ) K × ([a Σ , A Σ ] q ) K (17) et S B (K,J,R) = s (K,J,R) ξ ∈ S (K,J,R) ξ ∈ Ξ(K,J,R) (18)
Ξ(K,J,R) = Π K × Ψ(K,J,R) × ([a Σ , A Σ ] q ) K (19) Ψ(K,J,R) = (β R 1 1 , . . . , β R K K ) ∈ Ψ (K,J,R) pour tout k ∈ {1, . . . , K}, β R k k = R k l=1 σ l u t l v l , σ l < A σ .
Comme on travaille en régression, on définit une version tensorisée de la divergence de Kullback-Leibler

KL ⊗n (s, t) = E 1 n n i=1 KL(s(.|x i ), t(.|x i )) ;
une version tensorisée de la distance d'Hellinger,

(d ⊗n H ) 2 (s, t) = E 1 n n i=1 d 2 H (s(.|x i ), t(.|x i )) .
On a aussi besoin de la divergence de Jensen-Kullback-Leibler, définie par, pour ρ ∈ (0, 1), par

JKL ρ (s, t) = 1 ρ KL(s, (1 -ρ)s + ρt);
et de sa version tensorisée

JKL ⊗n ρ (s, t) = E 1 n n i=1 JKL ρ (s(.|x i ), t(.|x i )) .
Le nombre de classes est supposé inconnu, on va l'estimer ici, contrairement à l'inégalité oracle

Introduction

Inégalité oracle Lasso-EMV. Soit (x i , y i ) i∈{1,...,n} les observations, issues d'une densité conditionnelle inconnue s * . Soit S (K,J) définie par (14). On considère J L ⊂ J la sous-collection d'ensembles d'indices construite en suivant le chemin de régularisation de l'estimateur du Lasso. Pour (K, J) ∈ K × J L , notons S B (K,J) le modèle défini par (16). On considère l'estimateur du maximum de vraisemblance

ŝ(K,J) = argmin s (K,J) ξ ∈S B (K,J) - 1 n n i=1 log(s (K,J) ξ (y i |x i )) . Notons D (K,J) la dimension du modèle S B (K,J) , D (K,J) = K(|J| + q + 1) -1. Soit s(K,J) ξ ∈ S B (K,J)
telle que

KL ⊗n (s * , s(K,J) ξ ) ≤ inf t∈S B (K,J) KL ⊗n (s * , t) + δ KL n ; et soit τ > 0 tel que s(K,J) ξ ≥ e -τ s * . Soit pen : K × J → R + , et supposons qu'il existe une constante absolue κ > 0 telle que, pour tout (K, J) ∈ K × J , pen(K, J) ≥ κ D (K,J) n B 2 (A β , A Σ , a Σ ) -log D (K,J) n B 2 (A β , A Σ , a Σ ) ∧ 1 +(1 ∨ τ ) log 4epq (D (K,J) -q 2 ) ∧ pq ;
où les constantes A β , A Σ , a Σ sont définies par (17). Si on sélectionne le modèle indicé par ( K, Ĵ), où

( K, Ĵ) = argmin (K,J)∈K×J L - n i=1 log(ŝ (K,J) (y i |x i )) + pen(K, J) , alors l'estimateur ŝ( K, Ĵ) vérifie, pour tout ρ ∈ (0, 1), E JKL ⊗n ρ (s * , ŝ m) ≤C E inf (K,J)∈K×J L inf t∈S (K,J) KL ⊗n (s * , t) + pen(K, J) + 4 n ;
pour une constante C absolue.

Ce théorème nous donne une pénalité théorique pour laquelle le modèle minimisant le critère pénalisé a de bonnes propriétés d'estimation. Les constantes, bien que non optimales, sont explicites, en fonction des bornes de l'espace des paramètres. La pénalité est presque proportionnelle (à un terme logarithmique près) à la dimension du modèle. Le terme logarithmique a été étudié dans la thèse de Meynet, [START_REF] Meynet | Sélection de variables pour la classification non supervisée en grande dimension[END_REF]. Ici, en pratique, on prend la pénalité proportionnelle à la dimension.

et telle qu'il existe τ > 0 tel que s(K,J,R) ≥ e -τ s * . ( 20
)
On considère la collection d'estimateurs {ŝ (K,J,R) } (K,J,R)∈K×J ×R de S B (K,J,R) , vérifiant

ŝ(K,J,R) = argmin s (K,J,R) ξ ∈S B (K,J,R) - 1 n n i=1 log s (K,J,R) ξ (y i |x i ) . Notons D (K,J,R) la dimension du modèle S B (K,J,R) . Soit pen : K × J × R → R + définie par, pour tout (K, J, R) ∈ K × J × R, pen(K, J, R) ≥κ D (K,J,R) n 2B 2 (A β , A Σ , a Σ , A σ ) -log D (K,J,R) n B 2 (A β , A Σ , a Σ , A σ ) ∧ 1 + log 4epq (D (K,J,R) -q 2 ) ∧ pq + K k=1 R k , avec κ > 0 une constante absolue. Alors, l'estimateur ŝ( K, Ĵ, R) , avec ( K, Ĵ, R) = argmin (K,J,R)∈K×J L ×R - 1 n n i=1 log(ŝ (K,J,R) (y i |x i )) + pen(K, J, R) ,
vérifie, pour tout ρ ∈ (0, 1),

E JKL ⊗n ρ s * , ŝ( K, Ĵ, R) (21)
≤ C E inf

(K,J,R)∈K×J L ×R inf t∈S (K,J,R) KL ⊗n (s * , t) + pen(K, J, R) + 4 n , pour une constante C > 0.
Ce théorème nous donne une pénalité théorique pour laquelle le modèle minimisant le critère pénalisé a de bonnes propriétés d'estimation.

Ces deux théorèmes ne sont pas des inégalités oracle exactes, à cause de la constante C, mais on contrôle cette constante. Ces résultats sont non-asymptotiques, ce qui nous permet de les utiliser dans notre cadre de grande dimension. Ils justifient l'utilisation de l'heuristique des pentes.

Introduction

Plus précisément, on choisit de projeter les fonctions étudiées sur des bases d'ondelettes. On décompose de façon hiérarchique les signaux dans le domaine temps-échelle. On peut alors décrire une fonction à valeurs réelles par une approximation de cette

f (t) = ∞ j=1 b j α j (t).
On peut choisir une base d'ondelettes

B = {φ, ψ l,h } l≥0 0≤h≤2 l -1 , où -ψ est une ondelette réelle, vérifiant ψ ∈ L 1 ∩ L 2 , tψ ∈ L 1 , et R ψ(t)dt = 0. -pour tout t, ψ l,h (t) = 2 l/2 ψ(2 l t -h) pour (l, h) ∈ Z 2 .
φ une fonction d'échelle associée à ψ.

-pour tout t, φ l,h (t) = 2 l/2 φ(2 l t -h) pour (l, h) ∈ Z 2 . On peut alors écrire, pour tout f , pour tout t,

f (t) = h∈Z β L,h (f )φ L,h (t) + h∈Z l≤L d l,h (f )ψ l,h (t) où β l,h (f ) =< f, φ l,h > pour tout (l, h) ∈ Z 2 , d l,h (f ) =< f, ψ l,h > pour tout (l, h) ∈ Z 2 .
Alors, plutôt que de considérer l'échantillon (f i ) 1≤i≤n , on peut travailler avec l'échantillon

(x i ) 1≤i≤n = (β L,h (f i ), (d l,h (f i )) l≥L,h∈Z ) 1≤i≤n .
Comme la base est orthonormale, supposer que les (f i ) 1≤i≤n suivent une loi Gaussienne revient à supposer que les (x i ) 1≤i≤n suivent une loi Gaussienne. D'un point de vue pratique, la décomposition d'un signal sur une base d'ondelettes est très efficace. Citons Mallat, [START_REF] Mallat | A wavelet tour of signal processing[END_REF], pour une description détaillée des ondelettes et de leur utilisation. L'intérêt majeur est la décomposition temps-échelle du signal, qui permet d'analyser les coefficients. De plus, si on choisit bien la base, on peut résumer un signal en peu de coefficients, ce qui permet de réduire la dimension du problème. D'un point de vue plus pratique, on peut citer Misiti et coauteurs, [START_REF] Misiti | Clustering signals using wavelets[END_REF] pour la mise en pratique de la décomposition d'un signal sur une base d'ondelettes. L'application principale de cette thèse est la classification des consommateurs électriques dans un but de prédiction de la consommation agrégée. Si on prévoit la consommation de chaque consommateur, et qu'on somme ces prédictions, on va sommer les erreurs de prédiction, donc on peut faire beaucoup d'erreurs. Si on prévoit la consommation totale, on risque de faire des erreurs en n'étudiant pas assez les variations de chaque consommation individuelle. Cela explique le besoin de faire de la classification, et la classification en régression est faite dans un but de prédiction. Cependant, on sait que la prédiction de la consommation électrique peut être améliorée avec des modèles beaucoup plus adaptés. L'objectif de cette procédure est de classer ensemble, dans une étape préliminaire, les consommateurs qui ont le même comportement d'un jour à l'autre, ces groupes seront alors construits dans un but de prédiction.

Plan de la thèse

Cette thèse est principalement centrée sur les modèles de mélange en régression, et les problèmes de classification avec des données de régression en grande dimension. Elle est découpée en 5 chapitres, qui peuvent tous être lus de manière indépendante. Le premier chapitre est consacré à l'étude du modèle principal. On décrit le modèle de mélange de Gaussiennes en régression, où la réponse et les régresseurs sont multivariés. On propose plusieurs approches pour estimer les paramètres de ce modèle, entre autres en grande dimension (pour la réponse et pour les régresseurs). On définit, dans ce cadre, plusieurs estimateurs pour les paramètres inconnus : une extension de l'estimateur du Lasso, l'estimateur du maximum de vraisemblance et l'estimateur du maximum de vraisemblance sous contrainte de faible rang. Dans chaque cas, on a cherché à optimiser les définitions des estimateurs dans un but algorithmique. On décrit, de manière précise, les deux procédures proposées dans cette thèse pour classifier des variables dans un cadre de régression en grande dimension, et estimer le modèle sous-jacent. C'est une partie méthodologique, qui décrit précisément le fonctionnement de nos procédures, et qui explique comment les mettre en oeuvre numériquement. Des illustrations numériques sont proposées, pour confirmer en pratique l'utilisation de nos procédures. On utilise pour cela des données simulées, où l'aspect de grande dimension, l'aspect fonctionnel, et l'aspect de classification sont surlignés, et on utilise aussi des données de référence, où la vraie densité (inconnue donc) n'appartient plus à la collection de modèles considérée. Dans un deuxième chapitre, on obtient un résultat théorique pour l'estimateur du Lasso dans les modèles de mélange de Gaussiennes en régression, en tant que régularisateur 1 . Remarquons qu'ici, la pénalité est différente de celle du chapitre 1, cet estimateur étant une extension directe de l'estimateur du Lasso introduit par Tibshirani pour le modèle linéaire. Nous établissons une inégalité oracle 1 qui compare le risque de prédiction de l'estimateur Lasso à l'oracle 1 . Le point important de cette inégalité oracle est que, contrairement aux résultats habituels sur l'estimateur du Lasso, nous n'avons pas besoin d'hypothèses sur la non colinéarité entre les variables. En contrepartie, la borne sur le paramètre de régularisation n'est pas optimale, dans le sens où des résultats d'optimalité ont été démontrés pour une borne inférieure à celle que l'on obtient, mais sous des hypothèses plus contraignantes. Notons que les constantes sont toutes explicites, même si l'optimalité de ces quantités n'est pas garantie. Dans les chapitres 3 et 4, on propose une étude théorique de nos procédures de classification. On justifie théoriquement l'étape de sélection de modèles en établissant une inégalité oracle dans chaque cas (correspondant respectivement à l'inégalité oracle pour la procédure Lasso-EMV et l'inégalité oracle pour la procédure Lasso-Rang). Dans un premier temps, on a obtenu un théorème général de sélection de modèles, qui permet de choisir un modèle parmi une sous-collection aléatoire, dans un cadre de régression. Ce résultat, démontré à l'aide d'inégalités de concentration et de contrôles par calcul d'entropie métrique, est une généralisation à une sous-collection aléatoire de modèles d'un résultat déjà existant. Cette amélioration nous permet d'obtenir une inégalité oracle pour chacune de nos procédures : en effet, nous considérons une sous-collection aléatoire de modèles, décrite par le chemin de régularisation de l'estimateur du Lasso, et cet effet Introduction aléatoire nécessite de prendre des précautions dans les inégalités de concentration. Ce résultat fournit une forme de pénalité minimale garantissant que l'estimateur du maximum de vraisemblance pénalisé est proche de l'oracle 0 . En appliquant une telle pénalité lors de nos procédures, nous sommes sûrs d'obtenir un estimateur avec un faible risque de prédiction. L'hypothèse majeure que l'on fait pour obtenir ce résultat est de borner les paramètres du modèle de mélange. Remarquons que la pénalité n'est pas proportionnelle à la dimension, il y a un terme logarithmique en plus. On peut alors s'interroger quant à la nécessité de ce terme. On illustre aussi cette étape dans chacun des chapitres sur des jeux de données simulées et des jeux de données de référence. Il est important de souligner que ces résultats, théoriques et pratiques, sont envisageables car nous avons réestimé les paramètres par l'estimateur du maximum de vraisemblance, en se restreignant aux variables actives pour ne plus avoir de problème de grande dimension. Dans le chapitre 5, on s'intéresse à un jeu de données réelles. On met en pratique la procédure Lasso-EMV de classification des données en régression en grande dimension pour comprendre comment classer les consommateurs électriques, dans le but d'améliorer la prédiction. Ce travail a été effectué en collaboration avec Yannig Goude et Jean-Michel Poggi. Le jeu de données utilisé est un jeu de données irlandaises, publiques. Il s'agit de consommations électriques individuelles, relevées sur une année. Nous avons aussi accès à des données explicatives, telles que la température, et des données personnelles pour chaque consommateur. Nous avons utilisé la procédure Lasso-EMV de trois manières différentes. Un problème simple, qui nous a permis de calibrer la méthode, est de considérer la consommation agrégée sur les individus, et de classifier les transitions de jour. Les données sont alors assez stables, et les résultats interprétables (on veut classer les transitions de jours de semaine ensemble par exemple). Le deuxième schéma envisagé est de classifier les consommateurs, sur leur consommation moyenne. Pour ne pas perdre l'aspect temporel, on a considéré les jours moyens. Finalement, pour compléter l'analyse, on a classifié les consommateurs sur leur comportement sur deux jours fixés. Le problème majeur de ce schéma est l'instabilité des données. Cependant, l'analyse des résultats, par des critères classiques en consommation électrique, ou grâce aux variables explicatives disponibles avec ce jeu de données, permet de justifier l'intérêt de notre méthode pour ce jeu de données. A travers ce manuscrit, nous illustrons donc l'utilisation des modèles de mélange de Gaussiennes en régression, d'un point de vue méthodologique, mis en oeuvre d'un point de vue pratique, et justifié d'un point de vue théorique.

Perspectives

Pour poursuivre l'exploration des résultats de nos méthodes sur des données réelles, on pourrait utiliser un modèle de prédiction dans chaque classe. L'idée serait alors de comparer la prédiction obtenue plus classiquement avec la prédiction agrégée obtenue à l'aide de notre classification. minimax. On pourrait aussi s'intéresser à des intervalles de confiance. Le théorème de van de Geer et al., dans [START_REF] Van De Geer | On asymptotically optimal confidence regions and tests for high-dimensional models[END_REF], valable pour des pertes convexes, peut être généralisé à notre cas, et on obtient ainsi assez facilement un intervalle de confiance pour la matrice de régression. Cependant, dans un but de prédiction, il pourrait être plus intéressant d'obtenir un intervalle de confiance pour la réponse, mais c'est un problème bien plus difficile. Dans un but de classification, on pourrait aussi vouloir obtenir des résultats similaires aux inégalités oracles pour un autre critère que la divergence de Kullback-Leibler, plus orienté classification.

Notations

In this thesis, we denote (unless otherwise stated) by capital letter random variables, by lower case observations, and in bold letters the observation vector. For a matrix A, we denote by [A] i,. its ith row, [A] .,j its jth column, and [A] i,j its coefficient indexed by (i, j). For a vector B, we denote by [B] j its jth component.

Usual notations

c A complement of A A t transpose of A T r(A)
trace of a square matrix A E(X) esperance of the random variable X Var(X) variance of the random variable X N Gaussian distribution N q Gaussian multivariate distribution of size q χ 2 chi-squared distribution B orthonormal basis 1 A indicator function on a set A a floor function of a a ∨ b notation for the maximum between a and b a ∧ b notation for the minimum between a and b f g f is asymptotically equivalent to g ∆ discriminant for a quadratic polynomial function I q identity matrix of size q < a, b > scalar product between a and b P({1, . . . , p}) set of parts of {1, . . . , p} S ++ q set of positive-definite matrix of size q Variables and observations X regressors: random variable of size p x i i th observation of the variable X x vector of the observations Y response: random variable of size q y i i th observation of the variable Y y vector of the observations Z random variable for the affectation: vector of size K, Z k = 1 if the variable Y , conditionally to X, belongs to the cluster k, 0 otherwise z i observation of the variable Z for the observation y i conditionally to X i = x i F functional regressor: random functional variable f i i th observation of the variable F G functional response: random functional response g i i th observation of the variable G ỹ reparametrization of observation y, matrix of size n × K × q x reparametrization of observation x, matrix of size n × K × p Gaussian variable i i th observation of the variable ŷi prediction of the value of y i from observation x i Parameters β conditional mean, of size q × p × K σ variance, in univariate models, of size K Σ covariance matrix, in multivariate models, of size q × q × K Φ reparametrized conditional mean, of size q × p × K P reparametrized covariance matrix, of size q × q × K π proportions coefficients, of size K τ A Posteriori Probability, matrix of size n × K ξ vector of all parameters before reparametrization: (π, β, Σ) θ vector of all parameters after reparametrization: (π, φ, P) λ regularization parameter for the Lasso estimator λ k,j,z Lasso regularization parameter to cancel coefficient [φ k ] z,j in mixture models Ω parameter for the assumption K w m weights for the assumption K τ i,k (θ) probability for the observation i to belong to the cluster k, according to the parameter θ κ parameter for the slope heuristic estimator of θ by the Lasso estimator, with regularization parameter λ βLasso (λ) estimator of β by the Lasso estimator, with regularization parameter λ βLasso (A)

Acronym AIC Akaike
R
estimator of β by the Lasso estimator, with regularization parameter A according to the dual formulae ξLasso

K (λ)
estimator of ξ K by the Lasso estimator, with regularization parameter λ ξEMV K estimator of ξ K by the maximum likelihood estimator ξEMV J estimator of ξ K by the maximum likelihood estimator, restricted to J for the relevant variables ξRank J estimator of ξ K by the low rank estimator, restricted to J for the relevant variables Σx Gram matrix, according to the sample x θGroup-Lasso (λ) estimator of θ by the Group-Lasso estimator with regularization parameter λ βLR (λ) estimator of β by the low-rank estimator, restricted to variables detected by βLasso (λ) P LR (λ) estimator of P by the low-rank estimator, restricted to variables detected by βLasso (λ)

Sets of densities H (K,J) set of conditional densities, with parameters θ, with K clusters, and J for relevant variable set Ȟ(K,J) set of conditional densities, with parameters θ, with K clusters, and J for relevant variable set, and with vector of ranks R S (K,J) set of conditional densities, with parameters ξ, with K clusters, and J for relevant variable set S (K,J,R) set of conditional densities, with parameters ξ, with K clusters, and J for relevant variable set, and with vector of ranks R S B (K,J) subset of S (K,J) with bounded parameters dimension of the model S (K,J) D (K,J,R) dimension of the model S (K,J,R)

S B (K,J,R) subset of S (K,J,

Sets of parameters

Θ K set of θ with K components Θ (K,J)
set of θ with K components and J for relevant variables set Θ (K,J,R) set of θ with K components and J for relevant variables set, and R for vector of ranks for the conditional mean

Ξ K set of ξ with K components Ξ (K,J)
set of ξ with K components and J for relevant variables set Ξ (K,J,R) set of ξ with K components and J for relevant variables set, and R for vector of ranks for the conditional mean Ξ(K,J,R) subset of Ξ (K,J,R) with bounded parameters upper triangular matrices, of size q J set of relevant variables J λ set of relevant variables dected by the Lasso estimator with regularization parameter λ J set of relevant variables detected by the Group-Lasso estimator Ψ (K,J,R) set of conditional means, with J for relevant columns Ψ(K,J,R) subset of Ψ (K,J,R) with bounded parameters and R for vector of ranks, in a mixture with K components F J set of conditional Gaussian density, with bounded conditional means and bounded covariance coefficients, and relevant variables set defined by J F (J,R) set of conditional Gaussian density, with relevant variables set defined by J and vector of ranks defined by R, We take advantage of regression datasets to underline reliance between the regressors and the responses, and cluster data from this approach. This idea could be interesting for prediction, because observations sharing the same reliance will be considered in the same cluster.

K
and
In addition, we focus on functional dataset, for which the projection onto a wavelet basis leads to sparse representation. We also illustrate those procedures on simulated and benchmark dataset.

Introduction

Owing to the increasing of high-dimensional datasets, regression models for multivariate response and high-dimensional predictors have become important tools. The goal of this chapter is to describe two procedures which cluster regression datasets. We focus on the model-based clustering. Each cluster is represented by a parametric conditional distribution, the entire dataset being modeled by a mixture of these distributions. It provides a rigorous statistical framework, and allows to understand the role of each variable in the clustering process. The model considered is then, for i ∈ {1, . . . , n}, if (y i , x i ) ∈ R q × R p belongs to the component k, there exists an unknown q ×p matrix of coefficients β k and an unknown covariance matrix Σ k such that

y i = β k x i + i (1.1)
where i ∼ N q (0, Σ k ). We will work with high-dimensional datasets, that is to say q × p could be larger than the sample size n, then we have to reduce the dimension. Two ways will be considered here, coefficients sparsity and ranks sparsity. We could work with a sparse model if the matrix β could be estimated by a matrix with few nonzero coefficients. The well-known Lasso estimator, introduced by Tibshirani in 1996 in [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF] for linear models, is the solution chosen here. Indeed, the Lasso estimator is used for variable selection, cite for example Meinshausen and Bühlmann in [START_REF] Meinshausen | Stability selection[END_REF] for stability selection results. We could also cite the book of Bühlmann and van de Geer [START_REF] Bühlmann | Statistics for High-Dimensional Data: Methods, Theory and Applications[END_REF] for an overview of the Lasso estimator.

If we look for rank sparsity for β, we have to assume that a lot of regressors are linearly dependent. This approach date's back to the 1950's, and was initiated by Anderson in [START_REF] Anderson | Estimating Linear Restrictions on Regression Coefficients for Multivariate Normal Distributions[END_REF] for the linear model. Izenman, in [START_REF] Izenman | Reduced-rank regression for the multivariate linear model[END_REF], introduced the term of reduced-rank regression for this class of models. A number of important works followed, cite for example Giraud in [START_REF] Giraud | Low rank multivariate regression[END_REF] and Bunea et al. in [START_REF] Bunea | Joint variable and rank selection for parsimonious estimation of high-dimensional matrices[END_REF] for recent results. Nevertheless, the linear regression model used in those methods is appropriate for modeling the relationship between response and predictors when the reliance is the same for all observations, and it is inadequate for settings in which the regression coefficients differ across subgroups of the observations, then we will consider here mixture models.

An important example of high-dimensional datasets is functional datasets (functional predictors and/or functional response). They have been studied for example in the book of Ramsay and Silverman, [START_REF] Ramsay | Functional data analysis[END_REF]. A lot of recent works have been done on regression models for functional datasets: for example, cite the article of Ciarleglio ([CO14]) which deals with scalar response To take into account the matrix structure, we will consider this approach in our mixture models. We consider finite mixture of Gaussian regression model. We propose two different procedures, considering more or less the matrix structure. Both of them have the same frame. Firstly, an 1 -penalized likelihood approach is considered to determine potential sets of relevant variables. Introduced by Tibshirani in [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF], the Lasso is used to select variables. This allows one to efficiently construct a data-driven model subcollection with reasonable complexity, even for highdimensional situations, with different sparsities, varying the regularization parameter in the 1penalized likelihood function. The second step of the procedures consists to estimate parameters in a better way than by the Lasso estimator. Then, we select a model among the collection using the slope heuristic, which is developed by Birgé and Massart in [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF]. Differences between the both procedures are the estimation of parameters in each model. The first one, later called Lasso-MLE procedure, uses the maximum likelihood estimator rather than the 1 -penalized maximum likelihood estimator. It avoids estimation problems due to the 1 -penalization shrinkage. The second one, called Lasso-Rank procedure, deals with low rank estimation. For each model in the collection, we construct a subcollection of models with conditional means estimated by various low ranks matrices. It leads to sparsity and for the coefficients, and for the rank, and consider the conditional mean with its matrix structure. The chapter is organized as follows. Section 1.2 deals with Gaussian mixture regression models. It describes the model collection that we will consider. In Section 1.3, we describe both procedures that we propose to solve the problem of clustering high-dimensional regression data. Section 1.4 presents an illustrative example, to highlight each choice involved by both procedures. Section 1.5 states the functional data case, with a description of the projection proposed to convert these functions into coefficients data. We end this section by study of simulated and benchmark data. Finally, a conclusion section ends this chapter.

Gaussian mixture regression models

We have to construct a statistical framework on the observations. Because we estimate the conditional densities by multivariate Gaussian in each cluster, the model used is a finite Gaussian mixture regression model. Städler et al in [START_REF] Städler | 1 -penalization for mixture regression models[END_REF] describe this model, when X is multidimensional, and Y is scalar. We generalize it in the multivariate response case in this section. Moreover, we will describe a model collection of Gaussian mixture regression models, with several sparsities.

Gaussian mixture regression

We observe n independent couples (x i , y i ) 1≤i≤n , realizations of random variables (X i , Y i ) 1≤i≤n , with Y i ∈ R q and X i ∈ R p for all i ∈ {1, . . . , n}, coming from a probability distribution with unknown conditional density denoted by s * . We want to perform model-based clustering, then we assume that data could be well approximated by a mixture conditional density s(y|x) = K k=1 π k s k (y|x), with K unknown. To get a Gaussian mixture regression model, we suppose that, if Y conditionally to X belongs to the cluster k,

Y = β k X + ;
where ∼ N q (0, Σ k ). We then assume that s k is a multivariate Gaussian conditional density. Thus, the random response variable Y ∈ R q depends on a set of explanatory variables, written X ∈ R p , through a regression-type model. By considering multivariate response, we could work with more general datasets. Indeed, we could for example explain a functional response by a functional regressor, as done with the electricity dataset in Section 1.5.3. Some assumptions are in order, for a mixture of K Gaussian regression models.

-the variables Y i conditionally to X i are independent, for all i ∈ {1, . . . , n} ; -we let

Y i |X i = x i ∼ s K ξ (y|x i )dy, with s K ξ (y|x) = K k=1 π k (2π) q/2 det(Σ k ) 1/2 exp - (y -β k x) t Σ -1 k (y -β k x) 2 ξ = (π 1 , . . . , π K , β 1 , . . . , β K , Σ 1 , . . . , Σ K ) ∈ Ξ K = Π K × (R q×p ) K × (S ++ q ) K Π K = (π 1 , . . . , π k ); π k > 0 for k ∈ {1, . . . , K} and K k=1 π k = 1 S ++ q
is the set of symmetric positive definite matrices on R q .

Then, we want to estimate the conditional density function s K ξ from the observations. For all k ∈ {1, . . . , K}, β k is the matrix of regression coefficients, and Σ k is the covariance matrix in the mixture component k. The π k s are the mixture proportions. Actually, for all k ∈ {1, . . . , K}, for all z ∈ {1, . . . , q},

[β k x] z = p j=1 [β k ] z,j
x j is the zth component of the conditional mean of the mixture component k for the conditional density s K ξ (.|x). In order to have a scale-invariant maximum likelihood estimator, and to have a convex optimization problem, we reparametrize the model described above by generalizing the reparametrization described in [START_REF] Städler | 1 -penalization for mixture regression models[END_REF]. For all k ∈ {1, . . . , K}, we then define Φ k = P k β k , in which t P k P k = Σ -1 k (it is the Cholesky decomposition of the positive definite matrix Σ -1 k ). Our hypotheses could now be rewritten: -the variables Y i conditionally to X i are independent, for all i ∈ {1, . . . , n} ;

GAUSSIAN MIXTURE REGRESSION MODELS

-we let Y

i |X i = x i ∼ h K θ (y|x i )dy, for i ∈ {1, . . . , n} , with h K θ (y|x) = K k=1 π k det(P k ) (2π) q/2 exp - (P k y -Φ k x) t (P k y -Φ k x) 2 θ = (π 1 , . . . , π K , Φ 1 , . . . , Φ K , P 1 , . . . , P K ) ∈ Θ K = Π K × (R p×q ) K × (T q ) K Π K = (π 1 , . . . , π K ); π k > 0 for k ∈ {1, . . . , K} and K k=1 π k = 1
T q is the set of lower triangular matrix with non-negative diagonal entries.

The log-likelihood of this model is equal to, according to the sample

(x i , y i ) 1≤i≤n , l(θ, x, y) = n i=1 log K k=1 π k det(P k ) (2π) q/2 exp - (P k y i -Φ k x i ) t (P k y i -Φ k x i ) 2 ;
and the maximum log-likelihood estimator (later denoted by MLE) is

θMLE := argmin θ∈Θ K - 1 n l(θ, x, y) .
This estimator is scale-invariant, and the optimization is convex in each cluster. Since we deal with the p × q >> n case, this estimator has to be regularized to obtain accurate estimates. As a result, we propose the 1 -norm penalized MLE θLasso (λ) := argmin

θ∈Θ K - 1 n l λ (θ, x, y) ; (1.2)
where

- 1 n l λ (θ, x, y) = - 1 n l(θ, x, y) + λ K k=1 π k ||Φ k || 1 ;
where

||Φ k || 1 = p j=1 q z=1 |[Φ k ] z,j |,
and with λ > 0 to specify. This estimator is not the usual 1 -estimator, called the Lasso estimator, introduced by Tibshirani in [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF]. It penalizes the 1 -norm of the coefficients and small variances simultaneously, which has some close relations to the Bayesian Lasso (see Park and Casella [PC08]). Moreover, the reparametrization allows us to consider non-standardized data. Notice that we restrict ourselves in this chapter to diagonal covariance matrices which are dependent of the clusters, that is to say for all k ∈ {1, . . . , K},

Σ k = Diag ([Σ k ] 1,1 , . . . , [Σ k ] q,q ).
Then, with the renormalization described above, the restriction becomes, for all k ∈ {1, . . . , K}, P k = Diag ([P k ] 1,1 , . . . , [P k ] q,q ). In that case, we assume that variables are not correlated, which is a strong assumption, but it allows to reduce easily the dimension.

Clustering with Gaussian mixture regression

Suppose we know how many clusters there are, denoted by K, and assume that we get, from the observations, an estimator θ such that h K θ well approximate the unknown conditional density s * . Then, we want to group the data into clusters between observations which seem similar. From a different point of view, we can look at this problem as a missing data problem. Indeed, the complete data are ((x 1 , y 1 , z 1 ), . . . , (x n , y n , z n )) in which the latent random variables are

Z = (Z 1 , . . . , Z n ), Z i = ([Z i ] 1 , . . . , [Z i ] K ) for i ∈ {1, . . . , n} being defined by CHAPTER 1. TWO PROCEDURES [Z i ] k = 1 if Y i arises from the k th subpopulation ; 0 otherwise.
Thanks to the estimation θ, we could use the Maximum A Posteriori principle (later denoted MAP principle) to cluster data. Specifically, for all i ∈ {1, . . . , n}, for all k ∈ {1, . . . , K}, consider

τ i,k (θ) = π k det(P k ) exp -1 2 (P k y i -Φ k x i ) t (P k y i -Φ k x i ) K r=1 π k det(P k ) exp -1 2 (P k y i -Φ k x i ) t (P k y i -Φ k x i
) the posterior probability of y i coming from the component number k, where θ = (π, φ, P). Then, the data are partitioned by the following rule:

[Z i ] k = 1 if τ i,k ( θ) > τ i,l ( θ
) for all l = k ; 0 otherwise.

EM algorithm

From an algorithmic point of view, we will use a generalization of the EM algorithm to compute the MLE and the 1 -norm penalized MLE. The EM algorithm was introduced by Dempster et al. in [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF] to approximate the maximum likelihood estimator of parameters of mixture model. It is an iterative process based on the minimization of the expectation of the empirical contrast for the complete data conditionally to the observations and the current estimation of the parameter θ (ite) at each iteration (ite) ∈ N * . Thanks to the Karush-Kuhn-Tucker conditions, we could extend the second step to compute the maximum likelihood estimators, penalized or not, under rank constraint or not, as it was done in the scalar case in [START_REF] Städler | 1 -penalization for mixture regression models[END_REF]. All those calculus are available in Appendix 1.7.1. We therefore obtain the next updating formulae for the Lasso estimator defined by (1.2). Remark that it includes maximum likelihood estimator, and the rank constraint could be easily computed according to a singular value decomposition.

π (ite+1) k = π (ite) k + t (ite) n k n -π (ite) k
;

(1.3)

[P k ] (ite+1) z,z = n k [ỹ] (ite) k,z , [Φ k ] (ite) z,. [x] (ite) k,. + √ ∆ 2n k ||[ỹ] (ite) k,z || 2 2 ;
(1.4)

[Φ k ] (ite+1) z,j =            -[S k ] (ite) j,z +nλπ (ite) k ||[x] (ite) k,j || 2 2 if [S k ] (ite) j,z > nλπ (ite) k ; - [S k ] (ite) j,z +nλπ (ite) k ||[x] (ite) k,j || 2 2 if [S k ] (ite) j,z < -nλπ (ite) k ; 0 else ;
(1.5)

GAUSSIAN MIXTURE REGRESSION MODELS

44

with, for j ∈ {1, . . . , p}, k ∈ {1, . . . , K}, z ∈ {1, . . . , q},

[S k ] (ite) j,z = - n i=1 [x i ] (ite) k,j [P k ] (ite) z,z [ỹ i ] (ite) k,z + p j 2 =1 j 2 =j [x i ] (ite) k,j [x i ] (ite) k,j 2 [Φ k ] (ite)
z,j 2 ;

(1.6)

n k = n i=1 τ (ite) i,k ; ([ỹ i ] (ite) k,z , [x i ] (ite) k,j ) = τ (ite) i,k ([y i ] z , [x i ] j ); ∆ = -n k [ỹ] (ite) k,z , [Φ k ] (ite) z,. [x] (ite) k,. 2 -4||[ỹ] (ite) k,z || 2 2 ; τ (ite) i,k = π (ite) k det P (ite) k exp -1/2 P (ite) k y i -Φ (ite) k x i t P (ite) k y i -Φ (ite) k x i K r=1 π (ite) k det P (ite) k exp -1/2 P (ite) k y i -Φ (ite) k x i t P (ite) k y i -Φ (ite) k x i ; (1.7)
and t (ite) ∈ (0, 1], the largest value in the grid {δ l , l ∈ N}, 0 < δ < 1, such that the function is not increasing.

In our case, the EM algorithm corresponds to switch between the E-step which corresponds to the calculus of (1.3), (1.4) and (1.5), and the M-step, which corresponds to the calculus of (1.7).

To avoid convergence to local maximum rather than global maximum, we need to precise the initialization and the stopping rules. We initialize the clustering with the k-means algorithm on the couples (x i , y i ) 1≤i≤n . According to this clustering, we compute the linear regression estimators in each class. Then, we run a small number of times the EM-algorithm, repeat this initialization many times, and keep the one which maximizes the log-likelihood function: how the computation will start is important. Finally, to stop the algorithm, we could wait for any convergence, but the EM algorithm is known to check the convergence hypothesis, without converging, because of local maximum. Consequently, we choose to fix a minimum number of iterations to ensure non-local maximum, and to specify a maximum number of iterations to ensure stopping. Between these two bounds, we stop if there is convergence of the log-likelihood and of the parameters (with a relative criteria), adapted from [START_REF] Städler | 1 -penalization for mixture regression models[END_REF].

The model collection

We want to deal with high-dimensional data, that is why we have to determine which variables are relevant for the Gaussian regression mixture clustering. Indeed, we observe a small sample and we have to estimate many coefficients: we have a problem of identifiability. The sample size n is smaller than K(pq + q + 1) -1, the size of parameters to estimate. A way to solve this problem is to select few variables to describe the problem. We then assume that we could estimate s * by a sparse model.

To reduce the dimension, we want to determine which variables are useful for the clustering, and which are not. It leads to the definition of an irrelevant variable.

Definition 1.2.1. A variable indexed by (z, j) ∈ {1, . . . , q} × {1, . . . , p} is irrelevant for the clustering if

[Φ 1 ] z,j = . . . = [Φ K ] z,j = 0.
A relevant variable is a variable which is not irrelevant: at least in one cluster, this variable is not equal to zero. We denote by J the relevant variables set.

We denote by Φ

[J]

k the matrix with 0 on the set c J, for all k ∈ {1, . . . , K}, and H (K,J) the model with K components and with J for relevant variables set:

CHAPTER 1. TWO PROCEDURES H (K,J) = y ∈ R q |x ∈ R p → h (K,J) θ (y|x) ;
(1.8)

where

h (K,J) θ (y|x) = K k=1 π k det(P k ) (2π) q/2 exp - (P k y -Φ [J] k x) t (P k y -Φ [J] k x) 2 ,
and

θ = (π 1 , . . . , π K , Φ [J] 1 , . . . , Φ [J] K , P 1 , . . . , P K ) ∈ Θ (K,J) = Π K × R q×p K × R q + K ;
where the notation A [J] means that J is the relevant set variable for the matrix A.

We will construct a model collection, by varying the number of components K and the relevant variables subset J.

Two procedures

The goal of our procedures is, given a sample (x, y) = ((x 1 , y 1 ), . . . , (x n , y n )) ∈ (R p × R q ) n , to discover the structure of the variable Y according to X. Thus, we have to estimate, according to the representation of H (K,J) , the number of clusters K, the relevant variables set J, and the parameters θ. To overcome this difficulty, we want to take advantage of the sparsity property of the 1 -penalization to perform automatic variable selection in clustering high-dimensional data. Then, we could compute another estimator restricted on relevant variables, which will work better because it is no longer an high-dimensional issue. Thus, we avoid shrinkage problems due to the Lasso estimator. The first procedure takes advantage of the maximum likelihood estimator, whereas the second one takes into account the matrix structure of Φ with a low rank estimation.

Lasso-MLE procedure

This procedure is decomposed into three main steps: we construct a model collection, then in each we compute the maximum likelihood estimator, and we choose the best one among the model collection.

The first step consists of constructing a model collection {H (K,J) } (K,J)∈M in which H (K,J) is defined by equation (1.8), and the model collection is indexed by M = K × J . We denote by K ⊂ N * the possible number of components. We assume we could bound K without loss of estimation. We also note J ⊂ P({1, . . . , q} × {1, . . . , p}).

To detect the relevant variables, and construct the set J ∈ J , we penalize the empirical contrast by an 1 -penalty on the mean parameters proportional to

||Φ k || 1 = p j=1 q z=1 |[Φ k ] z,j |.
In the 1 -procedures, the choice of the regularization parameters is often difficult: fixing the number of components K ∈ K, we propose to construct a data-driven grid G K of regularization parameters by using the updating formulae of the mixture parameters in the EM algorithm. We can give a formula for λ, the regularization parameter, depending on which coefficient we want to cancel, for all k ∈ {1, . . . , K}, j ∈ {1, . . . , p}, z ∈ {1, . . . , q}:

[Φ k ] z,j = 0 ⇔ λ k,j,z = |[S k ] j,z | nπ k ;
with [S k ] j,z defined by (1.6). Then, we define the data-driven grid by

G K = {λ k,j,z , k ∈ {1, . . . , K}, j ∈ {1, . . . , p}, z ∈ {1, . . . , q}} .

TWO PROCEDURES

46

We could compute it from maximum likelihood estimations. Then, for each λ ∈ G K , we could compute the Lasso estimator defined by

θLasso (λ) = argmin θ∈Θ (K,J) 1 n n i=1 log(h (K,J) θ (y i |x i )) + λ K k=1 π k ||Φ k || 1 .
For a fixed number of mixture components K ∈ K and a regularization parameter λ ∈ G K , we could use an EM algorithm, recalled in Appendix 1.7.1, to approximate this estimator. Then, for each K ∈ K, and for each λ ∈ G K , we could construct the relevant variables set J λ . We denote by J the collection of all these sets. The second step consists of approximating the MLE ĥ(K,J) = argmin

h∈H (K,J) - 1 n n i=1 log(h(y i |x i )) ;
using the EM algorithm for each model (K, J) ∈ M. The third step is devoted to model selection. Rather than select the regularization parameter, we select the refitted model. We use the slope heuristic described in [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF]. Explain briefly how it works. Firstly, models are grouping according to their dimension D, to obtain a model collection {H D } D∈D . The dimension of a model is the number of parameters estimated in the model. For each dimension D, let ĥD be the estimator maximizing the likelihood among the estimators associated to a model of dimension D. Also, the function D/n → 1/n n i=1 log( ĥD (y i |x i )) has a linear behavior for large dimensions. We estimate the slope, denoted by κ, which will be used to calibrate the penalty. The minimizer D of the penalized criterion -1/n n i=1 log( ĥD (y i |x i )) + 2κD/n is determined, and the model selected is (K D, J D). Remark that D = K(|J| + q + 1) -1. Note that the model is selected after parameters refitting, which avoids issue of regularization parameter selection. For an oracle inequality to justify the slope heuristic used here, see [START_REF] Devijver | Finite mixture regression: A sparse variable selection by model selection for clustering[END_REF].

Lasso-Rank procedure

Whereas the previous procedure does not take into account the multivariate structure, we propose a second procedure to perform this point. For each model belonging to the collection H (K,J) , a subcollection is constructed, varying the rank of Φ. Let us describe this procedure. As in the Lasso-MLE procedure, we first construct a collection of models, thanks to the 1approach. For λ ≥ 0, we obtain an estimator for θ, denoted by θLasso (λ), for each model belonging to the collection. We could deduce the set of relevant columns, denoted by J λ , and this for all K ∈ K: we deduce J the collection of relevant variables set. The second step consists to construct a subcollection of models with rank sparsity, denoted by

{ Ȟ(K,J,R) } (K,J,R)∈ M.
The model Ȟ(K,J,R) has K components, the set J for active variables, and R is the vector of the ranks of the matrix of regression coefficients in each group:

Ȟ(K,J,R) = y ∈ R q |x ∈ R p → h (K,J,R) θ (y|x) (1.9) where h (K,J,R) θ (y|x) = K k=1 π k det(P k ) (2π) q/2 exp - (P k y -(Φ R k k ) [J] x) t (P k y -(Φ R k k ) [J] x) 2 ; θ = (π 1 , . . . , π K , (Φ R 1 1 ) [J] , . . . , (Φ R K K )) [J] , P 1 , . . . , P K ) ∈ Θ (K,J,R) = Π K × Ψ (K,J,R) × R q + K ; Ψ (K,J,R) = ((Φ R 1 1 ) [J] , . . . , (Φ R K K ) [J] ) ∈ R q×p K Rank(Φ k ) = R k for all k ∈ {1, . . . , K} ;
CHAPTER 1. TWO PROCEDURES and M R = K ×J ×R. We denote by K ⊂ N * the possible number of components, J a collection of subsets of {1, . . . , p}, and R the set of vectors of size K ∈ K with ranks values for each mean matrix. We could compute the MLE under the rank constraint thanks to an EM algorithm. Indeed, we could constrain the estimation of Φ k , for the cluster k, to have a rank equal to R k , in keeping only the R k largest singular values. More details are given in Section 1.7.1. It leads to an estimator of the mean with row sparsity and low rank for each model. As described in the above section, a model is selected using the slope heuristic. This step is justified theoretically in [START_REF] Devijver | Joint rank and variable selection for parsimonious estimation in highdimension finite mixture regression model[END_REF].

Illustrative example

We illustrate our procedures on four different simulated datasets, adapted from [SBG10], belonging to the model collection. They have been both implemented in Matlab, with the help of Benjamin Auder, and the Matlab code is available. Firstly, we will present models used in these simulations. Then, we validate numerically each step, and we finally compare results of our procedures with others. Remark that we propose here some examples to illustrate our methods, but not a complete analysis. We highlight some issues which seems important. Moreover, we do not illustrate the one-component case, focusing on the clustering. If, on some dataset, we are not convinced by the clustering, we could add to the model collection models with one component, more or less sparse, using the same pattern (computing the Lasso estimator to get the relevant variables for various regularization parameters, and refit parameters with the maximum likelihood estimator, under rank constraint or not), and select a model among this collection of linear and mixture models.

The model

Let x be a sample of size n distributed according to multivariate standard Gaussian. We consider a mixture of two components. Besides, we fix the number of active variables to 4 in each cluster. More precisely, the first four variables of Y are explained respectively by the four first variables of X. Fix π = 1 2 , 1 2 and P k = I q for all k ∈ {1, 2}. The difficulty of the clustering is partially controlled by the signal-to-noise ratio. In this context, we could extend the natural idea of the SNR with the following definition, where Tr(A) denotes the trace of the matrix A.

SNR = Tr(Var(Y )) Tr(Var(Y |β k = 0 for all k ∈ {1, . . . , K}))
.

Remark that it only controls the distance between the signal with or without the noise, and not the distance between the both signals.

We compute four different models, varying n, the SNR, and the distance between the clusters. Details are available in the Take a sample of Y according to a Gaussian mixture, meaning in β k X and with covariance matrix Σ k = (P t k P k ) -1 = σI q , for the cluster k. We run our procedures with the number of components varying in K = {2, . . . , 5}. The model 1 illustrates our procedures in low dimension models. Moreover, it is chosen in the next section to illustrate well each step of the procedure (variable selection, models construction and model selection). Model 5 is considered high-dimensional, because p × K > n. The model 2 is easier than the others, because clusters are not so close to each other according to the noise. Model 3 is constructed as the models 1 and 2, but n is small and the noise is more important. We will see that it gives difficulty for the clustering. Model 4 has a larger SNR, nevertheless, the problem of clustering is difficult, because each β k is closer to the others. Our procedures are run 20 times, and we compute statistics on our results over those 20 simulations: it is a small number, but the whole procedure is time-consuming, and results are convincing enough. For the initialization, we repeat 50 times the initialization, and keep the one which maximizes the log-likelihood function after 10 iterations. Those choices are size-dependent, a numerical study not reported here concludes that it is enough in that case.

Sparsity and model selection

To illustrate the both procedures, all the analyses made in this section are done from the model 1, since the choice of each step is clear. Firstly, we compute the grid of regularization parameters. More precisely, each regularization parameter is computed from maximum likelihood estimations (using EM algorithm), and give an associated sparsity (computed by the Lasso estimator, using again the EM algorithm). In Figure (1.1) and Figure (1.2), the collection of relevant variables selected by this grid are plotted. Firstly, we could notice that the number of relevant variables selected by the Lasso decreases with the regularization parameter. We could analyze more precisely which variables are selected, that is to say if we select true relevant or false relevant variables. If the regularization parameter is not too large, the true relevant variables are selected. Even more, if the regularization parameter is well-chosen, we select only the true relevant variables. In our example, we remark that if λ = 0.09, we have selected exactly the true relevant variables. This grid construction seems to be well-chosen according to these simulations. From this variable selection, each procedure (Lasso-MLE or Lasso-Rank) leads to a model collection, varying the sparsity thanks to the regularization parameters grid, and the number of components. Among this collection, we select a model with the slope heuristic. We want to select the best model by improving a penalized criterion. This penalty is computed ) and (1.4) represent the log-likelihood in function of the dimension of the models, for model collections constructed respectively by the Lasso-MLE procedure and by the Lasso-Rank procedure. The couples are plotted by points, whereas the estimated slope is specified by a dotted line. We could observe more than a line (4 for the Lasso-MLE procedure, more for the Lasso-Rank procedure). This phenomenon could be explained by a linear behavior for each mixture, fixing the number of classes, and for each rank. Nevertheless, slopes are almost the same, and select the same model. In practice, we estimate the slope with the Capushe package [START_REF] Baudry | Slope heuristics: overview and implementation[END_REF].

Assessment

We compare our procedures to three other procedures on simulated models 1, 2, 3 and 4. Firstly, let us give some remarks about the model 1. For each procedure, we get a good clustering and a very low Kullback-Leibler divergence. Indeed, the sample size is large, and the estimations are good. That is the reason why we focus in this section on models 2, 3 and 4. To compare our procedures with others, the Kullback-Leibler divergence with the true density and the ARI (the Adjusted Rand Index, measuring the similarity between two data clusterings, knowing that the closer to 1 the ARI, the more similar the two partitions) are computed, and we note which variables are selected, and how many clusters are selected. For more details on the ARI, see [START_REF] Rand | Objective criteria for the evaluation of clustering methods[END_REF]. From the Lasso-MLE model collection, we construct two models, to compare our procedures with. We compute the oracle (the model which minimizes the Kullback-Leibler divergence with the true density), and the model which is selected by the BIC criterion instead of the slope heuristic. Thanks to the oracle, we know how good we could get from this model collection for the Kullback-Leibler divergence, and how this model, as good it is possible for the log-likelihood, performs the clustering. The third procedure we compare with is the maximum likelihood estimator, assuming that we know how many clusters there are, fixed to 2. We use this procedure to show that variable selection is needed. In each case, we apply the MAP principle, to compare clusterings. We do not plot the Kullback-Leibler divergence for the MLE procedure, because values are too high, and make the boxplots unreadable. ). Our models are not as well as constructed as previously, but it is explained by the high-dimensional context. It is explained by the high Kullback-Leibler divergence. Nevertheless, our performances in clustering are really good. Note that the Kullback-Leibler divergence is smaller for the Lasso-MLE procedure, thanks to the maximum likelihood refitting. Moreover, the true model has not any matrix structure. If we look after the MLE, where we do not use the sparsity hypothesis, we could conclude that estimations are not satisfactory, which could be explained by an high-dimensional issue. The Lasso-MLE procedure, the Lasso-Rank procedure and the BIC model work almost as well as the oracle, which mind that the models are well selected. 2), we summarize results about variable selection. For each model, for each procedure, we compute how many true relevant and false relevant variables are selected. The true model has 8 relevant variables, which are always recognized. The Lasso-MLE has less false relevant variables than the others, which means that the true structure was found. The Lasso-Rank has 24 false relevant variables, because of the matrix structure: the true rank in each component was 4, then the estimator restricted on relevant variables is a 4 × 4 matrix, and we get 12 false relevant variables in each component. Nevertheless, we do not have more variables, that is to say the model constructed is the best possible. The BIC estimator and the oracle have a large variability for the false relevant variables. For the number of components, we find that all the procedures have selected the true number 2. Thanks to the MLE, the first procedure has good estimations (better than the second one). Nevertheless, depending on the data, the second procedure could be more attractive. If there is a matrix structure, for example if most of the variation of the response Y is caught by a small number of linear combinations of the predictors, the second procedure will work better. We could conclude that the model collection is well constructed, and that the clustering is well-done.

Functional datasets

One of the main interest of our methods is to be applied to functional datasets. Indeed, in different fields of applications, considered data are functions. The functional data analysis has been popularized first by Ramsay and Silverman in their book [START_REF] Ramsay | Functional data analysis[END_REF]. It gives a description of the main tools to analyze functional datasets. Another book is the Ferraty and Vieu one's [START_REF] Ferraty | Nonparametric functional data analysis : theory and practice[END_REF]. However, the main part of the existing literature about functional regression is concentrated on the case Y scalar and X functional. For example, we can cite Zhao et al., in [START_REF] Zhao | Wavelet-Based LASSO in Functional Linear Regression[END_REF] for using wavelet basis in linear model, Yao et al. ([YFL11]) for functional mixture regression, or Ciarleglio et al. ( [START_REF] Ciarleglio | Wavelet-based scalar-on-function finite mixture regression models[END_REF]) for using wavelet basis in functional mixture regression. In this section, we concentrate on Y and X both functional. In this regression context, we could be interested in clustering: it leads to identify the individuals involved in the same reliance between Y and X. Denote that, with functional datasets, we have to denoise and smooth signals to remove the noise and capture only the important patterns in the data. Here, we explain how our procedures can be applied in this context. Note that we could apply our procedures with scalar response and functional regressor, or, on the contrary, with functional response for scalar regressor. We explain how our procedures are generalized in the more difficult case, the other cases resulting of that. Remark that we focus here on the wavelet basis, to take advantage of the time-scale decomposition, but the same analysis is available on Fourier basis or splines.

FUNCTIONAL DATASETS 54

Functional regression model

Suppose we observe a centered sample of functions (f i , g i ) 1≤i≤n , associated with the random variables (F, G), coming from a probability distribution with unknown conditional density s * . We want to estimate this model by a functional mixture model: if the variables (F, G) come from the component k, there exists a function β k such that

G(t) = Ix F (u)β k (u, t)du + (t), (1.10)
where is a residual function. This linear model is introduced in Ramsay and Silverman's book [START_REF] Ramsay | Functional data analysis[END_REF]. They propose to project onto basis and the response, and the regressors. We extend their model in mixture model, to consider several subgroups for a sample. If we assume that, for all t, for all i ∈ {1, . . . , n}, i (t) ∼ N (0, Σ k ), the model (1.10) is an integrated version of the model (1.1). Depending on the cluster k, the linear reliance of G with respect to F is described by the function β k .

Two procedures to deal with functional datasets

Projection onto a wavelet basis

To deal with functional data, we project them onto some basis, to obtain data as described in the Gaussian mixture regression models (1.1). In this chapter, we choose to deal with wavelet basis, given that they represent localized features of functions in a sparse way. If the coefficients matrix x and y are sparse, the regression matrix β has more chance to be sparse. Moreover, we could represent a signal with a few coefficients dataset, then it is a way to reduce the dimension.

For details about the wavelet theory, see the Mallat's book [START_REF] Mallat | A wavelet tour of signal processing[END_REF].

Begin by an overview of some important aspects of wavelet basis. Let ψ a real wavelet function, satisfying

ψ ∈ L 1 ∩ L 2 , tψ ∈ L 1 , and R ψ(t)dt = 0.
We denote by ψ l,h the function defined from ψ by dyadic dilation and translation:

ψ l,h (t) = 2 l/2 ψ(2 l t -h) for (l, h) ∈ Z 2 .
We could define wavelet coefficients of a signal f by

d l,h (f ) = R f (t)ψ l,h (t)dt for (l, h) ∈ Z 2 .
Let ϕ be a scaling function related to ψ, and ϕ l,h the dilatation and translation of ϕ for (l, h) ∈ Z 2 . We also define, for

(l, h) ∈ Z 2 , β l,h (f ) = R f (t)ϕ l,h (
t)dt. Note that scaling functions serve to construct approximations of the function of interest, while the wavelet functions serve to provide the details not captured by successive approximations. We denote by V l the space generated by {ϕ l,h } h∈Z , and by W l the space egenrated by {ψ l,h } h∈Z for all l ∈ Z. Remark that

V l-1 = V l ⊕ W l for all l ∈ Z L 2 = ⊕ l∈Z W l
Let L ∈ N * . For a signal f , we could define the approximation at the level L by

A L = l>L h∈Z d l,h ψ l,h ;
and f could be decomposed by the approximation at the level L and the details (d l,h ) l<L . The decomposition of the basis between scaling function and wavelet function emphasizes on the local nature of the wavelets, and that is an important aspect in our procedures, because we want to know which details allow us to cluster two observations together. Consider the sample (f i , g i ) 1≤i≤n , and introduce the wavelet expansion of f i in the basis B: for all t ∈ [0, 1],

f i (t) = h∈Z β L,h (f i )ϕ L,h (t) A L + l≤L h∈Z d l,h (f i )ψ l,h (t). The collection {β L,h (f i ), d l,h (f i )} l≤L,h∈Z is the Discrete Wavelet Transform (DWT) of f in the basis B.
Because we project onto an orthonormal basis, this leads to a n-sample (x 1 , . . . , x n ) of wavelet coefficient decomposition vectors, with

f i = W x i ;
in which x i is the vector of the discretized values of the signal, x i the matrix of coefficients in the basis B, and W a p × p matrix defined by ϕ and ψ. The DWT can be performed by a computationally fast pyramid algorithm (see Mallat,[START_REF] Mallat | A theory for multiresolution signal decomposition: the wavelet representation[END_REF]). In the same way, there exists W such that g i = W y i , with y = (y 1 , . . . , y n ) a n sample of wavelet coefficient decomposition vectors.

Because the matrices W and W are orthogonal, we keep the mixture structure, and the noise is also Gaussian. We could consider the wavelet coefficient dataset (x, y) = ((x 1 , y 1 ), . . . , (x n , y n )), which defines of n observations whose probability distribution could be modeled by the finite Gaussian mixture regression model (1.1).

Our procedures

We could apply our both procedures to this dataset, and obtain a clustering of the data. Indeed, rather than considering (f , g), we run our procedures on the sample (x, y), varying the number of clusters in K.

The notion of relevant variable is natural: the function ϕ l,h or ψ l,h is irrelevant if it appears in none of the wavelet coefficient decomposition of the functions in each cluster.

Numerical experiments

We will illustrate our procedures on functional datasets by using the Matlab wavelet toolbox (see Misiti et al. in [START_REF] Misiti | Matlab Wavelet Toolbox User's Guide. Version 3[END_REF] for details). Firstly, we simulate functional datasets, where the true model belongs to the model collection. Then, we run our procedure on an electricity dataset, to cluster successive days. We have access to time series, measured every half-hour, of a load consumption, on 70 days. We extract the signal of each day, and construct couples by each day and its eve, and we aim at clustering these couples. To finish, we test our procedures on the well-known Tecator dataset. This benchmark dataset corresponds to the spectrometric curves and fat contents of meat. These experiments illustrate different aspects of our procedures. Indeed, the simulated example proves that our procedures work in a functional context. The second example is a toy example used to validate the classification, on real data already studied, and in which we clearly understand the clusters. The last example illustrates the use of the classification to perform prediction, and the description given by our procedures to the model constructed.

Simulated functional data

Firstly, we simulate a mixture regression model. Let f be a sample of the noised cosine function, discretized on a 15 points grid. Let g be, depending on the cluster, either f , or the function -f , computed by a white-noise. We use the Daubechies-2 basis at level 2 to decompose the signal.

Our procedures are run 20 times, and the number of clusters are fixed to K = 2. Then our procedures run on the projection are compared with the oracle among the collection constructed by the Lasso-MLE procedure, and with the model selected by the BIC criterion among this collection. The MLE is also computed. This simulated dataset proves that our procedures also perform clustering functional data, considering the projection dataset.

Electricity dataset

We also study the clustering on electricity dataset. This example is studied in [START_REF] Misiti | Clustering signals using wavelets[END_REF]. We work on a sample of size 70 of couples of days, which is plotted in Figure 5.1. For each couple, we have access to the half-hour load consumption. As we said previously, we want to cluster the relationship between two successive days. In Figure 1.15, we plot a week of consumption.

The regression model taken is F for the first day, and G for the second day of each couple. Besides, discretization of each day on 48 points, every half-hour, is made available. In our opinion, a linear model is not appropriate, as the behavior from the eve to the day depends on which day we consider: there is a difference between working days and weekend days, as involved in Figure 1.15.

To apply our procedures, we project f and g onto wavelet basis. The symmlet-4 basis, at level 5, is used. We run our procedures with the number of clusters varying from 2 to 6. Our both procedures select a model with 4 components. The first one considers couples of weekdays, the second Friday-Saturday, the third component is Saturday-Sunday and the fourth considers Sunday-Monday. This result is faithful with the knowledge we have about these data. Indeed, working days have the same behavior, depending on the eve, whereas days off have not the same behavior, depending on working days, and conversely. Moreover, in the article [START_REF] Misiti | Clustering signals using wavelets[END_REF], which also studied this example, they get the same classification.

Tecator dataset

This example deals with spectrometric data. More precisely, a food sample has been considered, which contained finely chopped pure meat with different fat contents. The data consist of a 100channel spectrum of absorbances in the wavelength range 850 -1050 nm, and of the percentage of fat. We observe a sample of size 215. Those data have been studied in a lot of articles, cite for example Ferraty and Vieu's book [START_REF] Ferraty | Nonparametric functional data analysis : theory and practice[END_REF]. They work on different approaches. They test prediction, and classification, supervised (where the fat content become a class, larger or smaller than 20%), or not (ignoring the response variable). In this work, we focus on clustering data according to the reliance between the fat content and the absorbance spectrum. We could not predict the response variable, because we do not know the class of a new observation. Estimate it is a difficult problem, in which we are not involved in this chapter.

We will take advantage of our procedures to know which coefficients, in the wavelet basis decomposition of the spectrum, are useful to describe the fat content. The sample will be split into two subsamples, 165 observations for the learning set, and 50 observations for the test set. We split it to have the same marginal distribution for the response in each sample. The spectrum is a function, which we decompose into the Haar basis, at level 6. Nevertheless, our model did not take into account a constant coefficient to describe the response. Thereby, before run our procedure, we center and the y according to the learning sample, and each function x i for all observations in the whole sample. Then, we could estimate the mean of the response by the mean μ over the learning sample. We construct models on the training set by our procedure Lasso-MLE. Thanks to the estimations, we have access to relevant variables, and we could reconstruct signals keeping only relevant variables. We have also access to the a posteriori probability, which leads to know which observation is with high probability in which cluster. However, for some observations, the a posteriori probability do not ensure the clustering, being almost the same for different clusters. The procedure selects two models, which we describe here. In Figures 1.16 and 1.17, we represent clusters done on the training set for the different models. The graph on the left is a candidate for representing each cluster, constructed by the mean of spectrum over an a posteriori probability greater than 0.6. We plot the curve reconstruction, keeping only relevant variables in the wavelet decomposition. On the right side, we present the boxplot of the fat values in each class, for observations with an a posteriori probability greater than 0.6.

The first model has two classes, which could be distinguish in the absorbance spectrum by the bump on wavelength around 940 nm. The first cluster is dominating, with π1 = 0.95. The fat content is smaller in the first cluster than in the second cluster. According to the signal reconstruction, we could see that almost all variables have been selected. This model seems consistent according to the classification goal.

The second model has 3 classes, and we could remark different important wavelength. Around 940 nm, there is some difference between classes, corresponding to the bump underline in the model 1, but also around 970 nm, with higher or smaller values. The first class is dominating, with π1 = 0.89. Just a few of variables have been selected, which give to this model the understanding property of which coefficients are discriminating.

We could discuss about those models. The first one selects only two classes, but almost all variables, whereas the second model has more classes, and less variables: there is a trade-off between clusters and variable selection for the dimension reduction. Because the models are constructed on the learning sample, MAPE are lower than for the test sample. Nevertheless, results are similar, saying that models are well constructed. This is particularly the case for the model 1, which is more consistent over a new sample.

CONCLUSION 60

To conclude this study, we could highlight the advantages of our procedure on these data. It provides a clustering of data, similar to the one done with supervised clustering in [START_REF] Ferraty | Nonparametric functional data analysis : theory and practice[END_REF], but we could explain how this clustering is done. This work has been done with the Lasso-MLE procedure. Nevertheless, the same kind of results have been get with the Lasso-Rank procedure.

Conclusion

In this chapter, two procedures are proposed to cluster regression data. Detecting the relevant clustering variables, they are especially designed for high-dimensional datasets. We use an 1 -regularization procedure to select variables, and then deduce a reasonable random model collection. Thus, we recast estimations of parameters of these models into a general model selection problem. These procedures are compared with usual criteria on simulated data: the BIC criterion used to select a model, the maximum-likelihood estimator, and to the oracle when we know it. In addition, we compare our procedures to others on benchmark data. One main asset of those procedures is that it can be applied to functional datasets. We also develop this point of view.

Appendices

In those appendices, we develop calculus for EM algorithm updating formulae in Section 1.7.1, for Lasso and maximum likelihood estimators, and for low ranks estimators. In Section 1.7.2, we extend our procedures with the Group-Lasso estimator to select relevant variables, rather than use the Lasso estimator.

EM algorithms

EM algorithm for the Lasso estimator Introduced by Dempster et al. in [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF], the EM (Expectation-Maximization) algorithm is used to compute maximum likelihood estimators, penalized or not. The expected complete negative log-likelihood is denoted by

Q(θ|θ ) = - 1 n E θ (l c (θ, X, Y, Z)|X, Y) in which l c (θ, X, Y, Z) = n i=1 K k=1 [Z i ] k log det(P k ) (2π) q/2 exp - 1 2 (P k Y i -X i Φ k ) t (P k Y i -X i Φ k ) + [Z i ] k log(π k );
with [Z i ] k are independent and identically distributed unobserved multinomial variables, showing the component-membership of the i th observation in the finite mixture regression model. The expected complete penalized negative log-likelihood is

Q pen (θ|θ ) = Q(θ|θ ) + λ K k=1 π k ||Φ k || 1 .

CHAPTER 1. TWO PROCEDURES

Calculus for updating formula -E-step: compute Q(θ|θ (ite) ), or, equivalently, compute for k ∈ {1, . . . , K}, i ∈ {1, . . . , n}, τ

(ite) i,k = E θ (ite) ([Z i ] k |Y) = π (ite) k det P (ite) k exp -1 2 P (ite) k Y i -X i Φ (ite) k t P (ite) k Y i -X i Φ (ite) k K r=1 π (ite) k det P (ite) k exp -1 2 P (ite) k Y i -X i Φ (ite) k t P (ite) k Y i -X i Φ (ite) k
This formula updates the clustering, thanks to the MAP principle. -M-step: improve Q pen (θ|θ (ite) ).

For this, rewrite the Karush-Kuhn-Tucker conditions. We have

Q pen (θ|θ (ite) ) = - 1 n n i=1 K k=1 E θ (ite) [Z i ] k log det(P k ) (2π) q/2 exp - 1 2 (P k Y i -X i Φ k ) t (P k Y i -X i Φ k ) |Y - 1 n n i=1 K k=1 E θ (ite) ([Z i ] k log π k |Y) + λ K k=1 π k ||Φ k || 1 = - 1 n n i=1 K k=1 - 1 2 (P k Y i -X i Φ k ) t (P k Y i -X i Φ k )E θ (ite) ([Z i ] k |Y) - 1 n n i=1 K k=1 q z=1 log [P k ] z,z √ 2π E θ (ite) [[Z i ] k |Y] - 1 n n i=1 K k=1 E θ (ite) ([Z i ] k |Y) log π k + λ K k=1 π k ||Φ k || 1 . (1.11)
Firstly, we optimize this formula with respect to π: it is equivalent to optimize

- 1 n n i=1 K k=1 τ i,k log(π k ) + λ K k=1 π k ||Φ k || 1 .
We obtain

π (ite+1) k = π (ite) k + t (ite) n i=1 τ i,k n -π (ite) k ;
with t (ite) ∈ (0, 1], the largest value in the grid {δ l , l ∈ N}, with 0 < δ < 1, such that the function is not increasing.

To optimize (1.11) with respect to (Φ, P), we could rewrite the expression: it is similar to the optimization of

- 1 n n i=1 τ i,k q z=1 log([P k ] z,z ) - 1 2 (P k Ỹi -Xi Φ k ) t (P k Ỹi -Xi Φ k ) + λπ k ||Φ k || 1
for all k ∈ {1, . . . , K}, which is equivalent to the optimization of

- 1 n n k q z=1 log([P k ] z,z ) + 1 2n n i=1 q z=1 [P k ] z,z [ Ỹi ] k,z -[Φ k ] z,. [ Xi ] k,. 2
+ λπ k ||Φ k || 1 ; 1.7. APPENDICES

62

where n k = n i=1 τ i,k . The minimum in [P k ] z,z is the function which cancel its partial derivative with respect to [P k ] z,z :

- n k n 1 [P k ] z,z + 1 2n n i=1 2[ Ỹi ] k,z [P k ] z,z [ Ỹi ] k,z -[Φ k ] z,. [ Xi ] k,. = 0
for all k ∈ {1, . . . , K}, for all z ∈ {1, . . . , q}, which is equivalent to

-1 + 1 n k [P k ] 2 z,z n i=1 [ Ỹi ] 2 k,z - 1 n k [P k ] z,z n i=1 [ Ỹi ] k,z [Φ k ] z,. [ Xi ] k,. = 0 ⇔ -1 + [P k ] 2 z,z 1 n k ||[ Ỹ] k,z || 2 2 -[P k ] z,z 1 n k [ Ỹ] k,z , [Φ k ] z,. [ X] k,. = 0.
The discriminant is

∆ = - 1 n k [ Ỹ] k,z , [Φ k ] z,. [ X] k,. 2 - 4 n k ||[ Ỹ] k,z || 2 2 .
Then, for all k ∈ {1, . . . , K}, for all z ∈ {1, . . . , q},

[P k ] z,z = n k [ Ỹ] k,z , [Φ k ] z,. [ X] k,. + √ ∆ 2n k ||[ Ỹ] k,z || 2 2 .
We could also look at the equation (1.11) as a function of the variable Φ: according to the partial derivative with respect to [Φ k ] z,j , we obtain for all z ∈ {1, . . . , q}, for all k ∈ {1, . . . , K}, for all j ∈ {1, . . . , p},

n i=1 [ Xi ] k,j   [P k ] z,z [ Ỹi ] k,z - p j 2 =1 [ Xi ] k,j 2 [Φ k ] z,j 2   -nλπ k sgn([Φ k ] z,j ) = 0.
Then, for all k ∈ {1, . . . , K}, j ∈ {1, . . . , p}, z ∈ {1, . . . , q},

[Φ k ] z,j = n i=1 [ Xi ] k,j [P k ] z,z [ Ỹi ] k,z -p j 2 =1 j 2 =j [ Xi ] k,j [ Xi ] k,j 2 [Φ k ] z,j 2 -nλπ k sgn([Φ k ] j,z ) ||[ X] k,j || 2 2 .
To reduce notations, let, for all k ∈ {1, . . . , K}, j ∈ {1, . . . , p}, z ∈ {1, . . . , q},

[S k ] j,z = - n i=1 [ Xi ] k,j [P k ] z,z [ Ỹi ] k,z + p j 2 =1 j 2 =j [ Xi ] k,j [ Xi ] k,j 2 [Φ k ] z,j 2 .
Then

[Φ k ] z,j = -[S k ] j,z -nλπ k sgn([Φ k ] z,j ) ||[ X] k,j || 2 2 =        -[S k ] j,z +nλπ k ||[ X] k,j || 2 2 if [S k ] j,z > nλπ k - [S k ] j,z +nλπ k ||[ X] k,j || 2 2 if [S k ] j,z < -nλπ k 0 elsewhere.

CHAPTER 1. TWO PROCEDURES

From these equalities, we could write the updating formulae. For j ∈ {1, . . . , p}, k ∈ {1, . . . , K}, z ∈ {1, . . . , q},

[S k ] (ite) j,z = - n i=1 [ Xi ] k,j [P k ] (ite) z,z [ Ỹi ] k,z + p j 2 =1 j 2 =j [ Xi ] k,j [ Xi ] k,j 2 [Φ k ] (ite)
z,j 2 ;

n k = n i=1 τ i,k ; ([ Ỹi ] k,. , [ Xi ] k,. ) = √ τ i,k (Y i , X i ).
and t (ite) ∈ (0, 1], the largest value in the grid {δ l , l ∈ N}, 0 < δ < 1, such that the function is not increasing.

EM algorithm for the rank procedure

To take into account the matrix structure, we want to make a dimension reduction on the rank of the mean matrix. If we know to which cluster each sample belongs, we could compute the low rank estimator for linear model in each component. Indeed, an estimator of fixed rank r is known in the linear regression case: denoting A + the Moore-Penrose pseudo-inverse of A, and

[A] r = U D r V t in which D r is obtained from D by setting (D r ) i,i = 0 for i ≥ r + 1, with U DV t the singular decomposition of A, if Y = βX + Σ, an estimator of β with rank r is βr = [(x t x) + x t y] r .
We do not know the clustering of the sample, but the E-step in the EM algorithm computes it. We suppose in this case that Σ k and π k are known, for all k ∈ {1, . . . , K}. We use this algorithm to determine Φ k , for all k ∈ {1, . . . , K}, with ranks fixed to R = (R 1 , . . . , R K ).

-E-step: compute for k ∈ {1, . . . , K}, i ∈ {1, . . . , n},

τ i,k = E θ (ite) ([Z i ] k |Y ) = π (ite) k det P (ite) k exp -1 2 P (ite) k y i -x i Φ (ite) k t P (ite) k y i -x i Φ (ite) k K r=1 π (ite) k det P (ite) k exp -1 2 P (ite) k y i -x i Φ (ite) k t P (ite) k y i -x i Φ (ite) k
-M-step: assign each observation in its estimated cluster, by the MAP principle applied thanks to the E-step. We say that Y i comes from component number argmax

k∈{1,...,K} τ (ite) i,k .
Then, we can define βk (ite) = (x t |k x |k ) -1 x t |k y |k , in which x |k and y |k are a restriction of the sample to the cluster k, which we decompose in singular value with β(ite) k = U SV t . Using the singular value decomposition described before, we obtain the estimator.

Group-Lasso MLE and Group-Lasso Rank procedures

One way to perform those procedures is to consider the Group-Lasso estimator rather than the Lasso estimator to select relevant variables. Indeed, this estimator is more natural, according to the relevant variable definition. Nevertheless, results are very similar, because we select grouped variables in both case, selected by the Lasso or by the Group-Lasso estimator. In this section, we describe our procedures with the Group-Lasso estimator, which could be understood as an improvement of our procedures.

Context -definitions

Our both procedures take advantage of the Lasso estimator to select relevant variables, to reduce the dimension in case of high-dimensional datasets. First, recall what is a relevant variable.

Definition 1.7.1. A variable indexed by (z, j) ∈ {1, . . . , q} × {1, . . . , p} is irrelevant for the clustering if [Φ 1 ] z,j = . . . = [Φ K ] z,j = 0. A relevant variable is a variable which is not irrelevant. We denote by J the relevant variables set.

According to this definition, we could introduce the Group-Lasso estimator.

Definition 1.7.2. The Lasso estimator for mixture regression models with regularization parameter λ ≥ 0 is defined by

θLasso (λ) := argmin θ∈Θ K - 1 n l λ (θ) ;
where

- 1 n l λ (θ) = - 1 n l(θ) + λ K k=1 π k ||Φ k || 1 ;
where

||Φ k || 1 = p j=1 q z=1 |[Φ k ] z,j
|, and with λ to specify.

It is the estimator used in the both procedures described in previous parts.

Definition 1.7.3. The Group-Lasso estimator for mixture regression models with regularization parameter λ ≥ 0 is defined by θGroup-Lasso (λ) := argmin

θ∈Θ K - 1 n lλ (θ) ;
where

- 1 n lλ (θ) = - 1 n l(θ) + λ p j=1 q z=1 √ k||[Φ] z,j || 2 ;
where

||[Φ] z,j || 2 2 = K k=1 |[Φ k ] z,j | 2
, and with λ to specify.

This Group-Lasso estimator has the advantage to cancel grouped variables rather than variables one by one. It is consistent with the relevant variable definition. Nevertheless, depending on datasets, it could be interesting to look after which variables are canceled first. One way could be to extend this work with Lasso-Group-Lasso estimator, described for the example for the linear model in [START_REF] Simon | A sparsegroup lasso[END_REF].

Let describe two additional procedures, which will use the Group-Lasso estimator rather than the Lasso estimator to detect relevant variables.

Group-Lasso-MLE procedure

This procedure is decomposed into three main steps: we construct a model collection, then in each model we compute the maximum likelihood estimator, and we select the best one among all the models. The first step consists of constructing a collection of models {H (K, J) } (K, J)∈M in which H (K, J) is defined by

H (K, J) = {y ∈ R q |x ∈ R p → h θ (y|x)} ; (1.12)
where

h θ (y|x) = K k=1 π k det(P k ) (2π) q/2 exp - (P k y -Φ [ J] k x) t (P k y -Φ [ J] k x) 2
,

and θ = (π 1 , . . . , π K , Φ 1 , . . . , Φ K , ρ 1 , . . . , ρ K ) ∈ Π K × R q×p K × R q + K .
The model collection is indexed by M = K × J . Denote K ⊂ N * the possible number of components. We could bound K without loss of estimation. Denote also J a collection of subsets of {1, . . . , q} × {1, . . . , p}, constructed by the Group-Lasso estimator.

To detect the relevant variables, and construct the set J ∈ J , we will use the Group-Lasso estimator defined by (1.7.3). In the 1 -procedures, the choice of the regularization parameters is often difficult: fixing the number of components K ∈ K, we propose to construct a data-driven grid G K of regularization parameters by using the updating formulae of the mixture parameters in the EM algorithm.

Then, for each λ ∈ G K , we could compute the Group-Lasso estimator defined by θGroup-Lasso = argmin

θ∈Θ K    - 1 n n i=1 log(h θ (y i |x i )) + λ p j=1 q z=1 √ K||[Φ] z,j || 2    .
For a fixed number of mixture components K ∈ K and a regularization parameter λ, we could use a generalized EM algorithm to approximate this estimator. Then, for each K ∈ K, and for each λ ∈ G K , we have constructed the relevant variables set Jλ . We denote by J the collection of all these sets. The second step consists of approximating the MLE ĥ(K, J) = argmin

t∈H (K, J) - 1 n n i=1 log(t(y i |x i )) ;
using the EM algorithm for each model (K, J) ∈ M. The third step is devoted to model selection. We use the slope heuristic described in [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF].

Group-Lasso-Rank procedure

As when the relevant variables were selected by the Lasso estimator, whereas the previous procedure does not take into account the multivariate structure, we propose a second procedure to perform this point. For each model belonging to the collection H (K, J) , a subcollection is constructed, varying the rank of Φ. Let us describe this procedure. As in the Group-Lasso-MLE procedure, we first construct a collection of models, thanks to the 1 -approach. We obtain an estimator for θ, denoted by θGroup-Lasso , for each model belonging to the collection. We could deduce the set of relevant variables, denoted by J, and this for all K ∈ K: we deduce J the collection of set of relevant variables. The second step consists to construct a subcollection of models with rank sparsity, denoted by

{ Ȟ(K, J,R) } (K, J,R)∈ M.
The model { Ȟ(K, J,R) } has K components, the set J for active variables, and R is the vector of the ranks of the matrix of regression coefficients in each group: This chapter focuses on the Lasso estimator for its regularization properties. We consider a finite mixture of Gaussian regressions for high-dimensional heterogeneous data, where the number of covariates and the dimension of the response may be much larger than the sample size. We estimate the unknown conditional density by an 1 -penalized maximum likelihood estimator. We provide an 1 -oracle inequality satisfied by this Lasso estimator according to the Kullback-Leibler loss. This result is an extension of the 1 -oracle inequality established by Meynet in [START_REF] Meynet | An 1 -oracle inequality for the lasso in finite mixture gaussian regression models[END_REF] in the multivariate case. It is deduced from a model selection theorem, the Lasso being viewed as the solution of a penalized maximum likelihood model selection procedure over a collection of 1 -ball models.

Ȟ(K, J,R) = y ∈ R q |x ∈ R p → h (K, J,R) θ (y|x) (1.13) 1.7. APPENDICES 66 where h (K, J,R) θ (y|x) = K k=1 π k det(P k ) (2π) q/2 exp - (P k y -(Φ R k k ) [ J] x) t (P k y -(Φ R k k ) [ J] x) 2 ; θ =(π 1 , . . . , π K , Φ R 1 1 , . . . , Φ R K K , P 1 , . . . , P K ) ∈ Π K × Ψ R K × R q + K ; Ψ R K = (Φ R 1 1 , . . . , Φ R K K ) ∈ R q×p K |Rank(Φ 1 ) = R 1 , . . . , Rank(Φ K ) = R K ; and MR = K × J × R. Denote K ⊂ N * the

Introduction

Finite mixture regression models are useful for modeling the relationship between response and predictors, arising from different subpopulations. Due to recent improvements, we are faced with high-dimensional data where the number of variables can be much larger than the sample size. We have to reduce the dimension to avoid identifiability problems. Considering a mixture of linear models, an assumption widely used is to say that only a few covariates explain the response. Among various methods, we focus on the 1 -penalized least squares estimator of parameters to lead to sparse regression matrix. Indeed, it is a convex surrogate for the nonconvex 0 -penalization, and produces sparse solutions. First introduced by Tibshirani in [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF] in a linear model Y = Xβ + , where X ∈ R p , Y ∈ R, and ∼ N (0, Σ), the Lasso estimator is defined in the linear model by βLasso (λ) = argmin

β∈R p ||Y -Xβ|| 2 2 + λ||β|| 1 , λ > 0.
Many results have been proved to study the performance of this estimator. For example, cite [START_REF] Bickel | Simultaneous analysis of Lasso and Dantzig selector[END_REF] and [START_REF] Efron | Least angle regression[END_REF], for studying this estimator as a variable selection procedure in this linear model case. Note that those results need strong assumptions on the Gram matrix X t X, as the restrictive eigenvalue condition, that can be not fulfilled in practice. A summary of assumptions and results is given by Bühlmann and van de Geer in [START_REF] Van De Geer | On the conditions used to prove oracle results for the Lasso[END_REF]. One can also cite van de Geer in [START_REF] Van De Geer | The adaptive and the thresholded lasso for potentially misspecified models (and a lower bound for the lasso)[END_REF] and discussions, who precises a chaining argument to perform rate, even in a non linear case.

If we assume that (x i , y i ) 1≤i≤n arise from different subpopulations, we could work with finite mixture regression models. Indeed, the homogeneity assumption of the linear model is often inadequate and restrictive. This model was introduced by Städler et al., in [START_REF] Städler | 1 -penalization for mixture regression models[END_REF]. They assume that, for i ∈ {1, . . . , n}, the observation y i , conditionally to X i = x i , comes from a conditional density s ξ 0 (.|x i ) which is a finite mixture of K Gaussian conditional densities with proportion vector π, where

Y i |X i = x i ∼ s ξ 0 (y i |x i ) = K k=1 π 0 k √ 2πσ 0 k exp - (y i -β 0 k x i ) 2 2(σ 0 k ) 2
for some parameters ξ 0 = (π 0 k , β 0 k , σ 0 k ) 1≤k≤K . They extend the Lasso estimator by ŝLasso (λ) = argmin

s K ξ    - 1 n n i=1 log(s K ξ (y i |x i )) + λ K k=1 π k p j=1 |σ -1 k [β k ] j |    , λ > 0 (2.1)

NOTATIONS AND FRAMEWORK

70

For this estimator, they provide an 0 -oracle inequality satisfied by ŝLasso (λ), according to the restricted eigenvalue condition also, and margin conditions, which lead to link the Kullback-Leibler loss function to the 2 -norm of the parameters.

Another way to study this estimator is to look after the Lasso for its 1 -regularization properties. For example, cite [START_REF] Massart | The Lasso as an 1 -ball model selection procedure[END_REF], [START_REF] Meynet | An 1 -oracle inequality for the lasso in finite mixture gaussian regression models[END_REF], and [START_REF] Rigollet | Exponential screening and optimal rates of sparse estimation[END_REF]. Contrary to the 0 -results, some 1 -results are valid with no assumptions, neither on the Gram matrix, nor on the margin. This can be achieved due to the fact that they are looking for rate of convergence of order 1/ √ n rather than 1/n. For finite mixture Gaussian regression models, we could cite Meynet in [START_REF] Meynet | An 1 -oracle inequality for the lasso in finite mixture gaussian regression models[END_REF] who give an 1 -oracle inequality for the Lasso estimator (2.1). In this chapter, we extend this result to finite mixture of multivariate Gaussian regression models. We will work with random multivariate variables (X, Y ) ∈ R p × R q . As in [START_REF] Meynet | An 1 -oracle inequality for the lasso in finite mixture gaussian regression models[END_REF], we shall restrict to the fixed design case, that is to say non-random regressors. We observe (x i ) 1≤i≤n . Without any restriction, we could assume that the regressors x i ∈ [0, 1] p for all i ∈ {1, . . . , n}. Under only bounded parameters assumption, we provide a lower bound on the Lasso regularization parameter λ which guarantees such an oracle inequality. This result is non-asymptotic: the number of observations is fixed, and the number p of covariates can grow. Remark that the number K of clusters in the mixture is supposed to be known. Our result is deduced from a finite mixture multivariate Gaussian regression model selection theorem for 1 -penalized maximum likelihood conditional density estimation. We establish the general theorem following the one of Meynet in [START_REF] Meynet | An 1 -oracle inequality for the lasso in finite mixture gaussian regression models[END_REF], which combines Vapnik's structural risk minimization method (see Vapnik in [START_REF] Vapnik | Estimation of Dependences Based on Empirical Data: Springer Series in Statistics[END_REF]) and theory around model selection (see Le Pennec and Cohen in [START_REF] Cohen | Conditional density estimation by penalized likelihood model selection and applications[END_REF] and Massart in [START_REF] Massart | Concentration inequalities and model selection[END_REF]). As in Massart and Meynet in [START_REF] Massart | The Lasso as an 1 -ball model selection procedure[END_REF], our oracle inequality is deduced from this general theorem, the Lasso being viewed as the solution of a penalized maximum likelihood model selection procedure over a countable collection of 1 -ball models. The chapter is organized as follows. The model and the framework are introduced in Section 2.2. In Section 2.3, we state the main result of the chapter, which is an 1 -oracle inequality satisfied by the Lasso in finite mixture of multivariate Gaussian regression models. Section 2.4 is devoted to the proof of this result and of the general theorem, deriving from two easier propositions. Those propositions are proved in Section 2.5, whereas details of lemma states in Section 2.6.

Notations and framework

Finite mixture regression model

We observe n independent couples (x, y) = (x i , y i ) 1≤i≤n ∈ ([0, 1] p ×R q ) n , with y i ∈ R q a random observation, realization of variable Y i , and x i ∈ [0, 1] p fixed for all i ∈ {1, . . . , n}. We assume that, conditionally to the x i s, the Y i s are independent identically distributed with conditional density s ξ 0 (.|x i ) which is a finite mixture of K Gaussian regressions with unknown parameters ξ 0 . In this chapter, K is fixed, then we do not precise it with unknown parameters. We will estimate the unknown conditional density by a finite mixture of K Gaussian regressions. Each subpopulation is then estimated by a multivariate linear model. Detail the conditional density.

CHAPTER 2. AN 1 -ORACLE INEQUALITY FOR THE LASSO IN FINITE MIXTURE OF MULTIVARIATE GAUSSIAN REGRESSION MODELS

For all y ∈ R q , for all x ∈ [0, 1] p ,

s ξ (y|x) = K k=1 π k (2π) q/2 det(Σ k ) 1/2 exp - (y -β k x) t Σ -1 k (y -β k x) 2 (2.2) ξ = (π 1 , . . . , π K , β 1 , . . . , β K , Σ 1 , . . . , Σ K ) ∈ Ξ = Π K × (R q×p ) K × (S ++ q ) K Π K = (π 1 , . . . , π K ); π k > 0 for k ∈ {1, . . . , K} and K k=1 π k = 1 S ++ q
is the set of symmetric positive definite matrices on R q .

We want to estimate ξ 0 from the observations. For all k ∈ {1, . . . , K}, β k is the matrix of regression coefficients, and Σ k is the covariance matrix in the mixture component k, whereas the π k s are the mixture proportions. For x ∈ [0, 1] p , we define the parameter ξ(x) of the conditional density s ξ (.|x) by

ξ(x) = (π 1 , . . . , π K , β 1 x, . . . , β K x, Σ 1 , . . . , Σ K ) ∈ R K × (R q ) K × (S ++ q ) K .
For all k ∈ {1, . . . , K}, for all x ∈ [0, 1] p , for all z ∈ {1, . . . , q},

[β k x] z = p j=1 [β k ] z,j [x] j
, and then β k x ∈ R q is the mean vector of the mixture component k for the conditional density s ξ (.|x).

Boundedness assumption on the mixture and component parameters

Denote, for a matrix A, m(A) the modulus of the smallest eigenvalue of A, and M (A) the modulus of the largest eigenvalue of A. We shall restrict our study to bounded parameters vector ξ = (π, β, Σ), where π = (π 1 , . . . , π K ), β = (β 1 , . . . , β K ), Σ = (Σ 1 , . . . , Σ K ). Specifically, we assume that there exists deterministic positive constants A β , a Σ , A Σ , a π such that ξ belongs to Ξ, with

Ξ = ξ ∈ Ξ : for all k ∈ {1, . . . , K}, max z∈{1,...,q} sup x∈[0,1] p |[β k x] z | ≤ A β , a Σ ≤ m(Σ -1 k ) ≤ M (Σ -1 k ) ≤ A Σ , a π ≤ π k . (2.3)
Let S the set of conditional densities s ξ ,

S = s ξ , ξ ∈ Ξ .

Maximum likelihood estimator and penalization

In a maximum likelihood approach, we consider the Kullback-Leibler information as the loss function, which is defined for two densities s and t by

KL(s, t) =      R q log s(y) t(y) s(y)dy if sdy << tdy; + ∞ otherwise.
(2.4)

In a regression framework, we have to adapt this definition to take into account the structure of conditional densities. For the fixed covariates (x 1 , . . . , x n ), we consider the average loss function

KL n (s, t) = 1 n n i=1 KL(s(.|x i ), t(.|x i )) = 1 n n i=1 R q log s(y|x i ) t(y|x i ) s(y|x i )dy.

ORACLE INEQUALITY

72

Using the maximum likelihood approach, we want to estimate s ξ 0 by the conditional density s ξ which maximizes the likelihood conditionally to (x i ) 1≤i≤n . Nevertheless, because we work with high-dimensional data, we have to regularize the maximum likelihood estimator. We consider the 1 -regularization, and a generalization of the estimator associated, the Lasso estimator, which we define by ŝLasso (λ) := argmin

s ξ ∈S    - 1 n n i=1 log(s ξ (y i |x i )) + λ K k=1 q z=1 p j=1 |[β k ] z,j |    ;
where λ > 0 is a regularization parameter, for ξ = (π, β, Σ).

We define also, for s ξ defined as in (2.2), and with parameters ξ = (π, β, Σ),

N [2] 1 (s ξ ) = ||β|| 1 = K k=1 p j=1 q z=1 |[β k ] z,j |.
(2.5)

Oracle inequality

In this section, we provide an 1 -oracle inequality satisfied by the Lasso estimator in finite mixture multivariate Gaussian regression models, which is the main result of this chapter.

Theorem 2.3.1. We observe n couples (x, y) = ((x 1 , y 1 ), . . . , (x n , y n )) ∈ ([0, 1] p × R q ) n coming from the conditional density s ξ 0 , where ξ 0 ∈ Ξ, where Ξ = ξ ∈ Ξ : for all k ∈ {1, . . . , K}, max z∈{1,...,q}

sup x∈[0,1] p |[β k x] z | ≤ A β , a Σ ≤ m(Σ -1 k ) ≤ M (Σ -1 k ) ≤ A Σ , a π ≤ π k .
Denote by a ∨ b = max(a, b).

We define the Lasso estimator, denoted by ŝLasso (λ), for λ ≥ 0, by ŝLasso (λ) = argmin

s ξ ∈S - 1 n n i=1 log(s ξ (y i |x i )) + λN [2]
1 (s ξ ) ;

(2.6)

with S = s ξ , ξ ∈ Ξ
and where, for ξ = (π, β, Σ),

N [2] 1 (s ξ ) = ||β|| 1 = K k=1 p j=1 q z=1 |[β k ] z,j |.
Then, if

λ ≥ κ A Σ ∨ 1 a π 1 + 4(q + 1)A Σ A 2 β + log(n) a Σ K n 1 + q log(n) K log(2p + 1)
with κ an absolute positive constant, the estimator (2.6) satisfies the following 1 -oracle inequality. MIXTURE OF MULTIVARIATE GAUSSIAN REGRESSION MODELS

E[KL n (s ξ 0 , ŝLasso (λ))] ≤ (1 + κ -1 ) inf s ξ ∈S KL n (s ξ 0 , s ξ ) + λN [2] 1 (s ξ ) + λ + κ K n e -1 2 π q/2 a π A q/2 Σ 2q + κ K n A Σ ∨ 1 a π 1 + 4(q + 1)A Σ A 2 β + log(n) a Σ ×K 1 + A β + q a Σ 2
where κ is a positive constant.

This theorem provides information about the performance of the Lasso as an 1 -regularization algorithm. If the regularization parameter λ is properly chosen, the Lasso estimator, which is the solution of the 1 -penalized empirical risk minimization problem, behaves as well as the deterministic Lasso, which is the solution of the 1 -penalized true risk minimization problem, up to an error term of order λ.

Our result is non-asymptotic: the number n of observations is fixed while the number p of covariates and the size q of he response can grow with respect to n and can be much larger than n. The number K of clusters in the mixture is fixed.

There is no assumption neither on the Gram matrix, nor on the margin, which are classical assumptions for oracle inequality for the Lasso estimator. Moreover, this kind of assumptions involve unknown constants, whereas here, every constants are explicit. We could compare this result with the 0 -oracle inequality established in [START_REF] Städler | 1 -penalization for mixture regression models[END_REF], which need those assumptions, and is therefore difficult to interpret. Nevertheless, they get faster rate, the error term in the oracle inequality being of order 1/n rather than 1/ √ n. The main assumption we make to establish the Theorem 2.3.1 is the boundedness of the parameters, which is also assumed in [START_REF] Städler | 1 -penalization for mixture regression models[END_REF]. It is needed, to tackle the problem of the unboundedness of the parameter space (see [START_REF] Mclachlan | Finite Mixture Models[END_REF] for example). Moreover, we let regressors to belong to [0, 1] p . Because we work with fixed covariates, they are finite. To simplify the reading, we choose to rescale x to get ||x|| ∞ ≤ 1. Nevertheless, if we not rescale the covariates, and the regularization parameter λ bound and the error term of the oracle inequality depend linearly of ||x|| ∞ . The regularization parameter λ bound is of order (q 2 + q)/ √ n log(n) 2 log(2p + 1). For q = 1, we recognize the same order, as regards to the sample size n and the number of covariates p, to the 1 -oracle inequality in [START_REF] Meynet | An 1 -oracle inequality for the lasso in finite mixture gaussian regression models[END_REF]. Van de Geer, in [START_REF] Van De Geer | The adaptive and the thresholded lasso for potentially misspecified models (and a lower bound for the lasso)[END_REF], gives some tools to improve the bound of the regularization parameter to log(p) n . Nevertheless, we have to control eigenvalues of the Gram matrix of some functions (ψ j (x i )) 1≤j≤D 1≤i≤n , D being the number of parameters to estimate, where ψ j (x i ) satisfies

| log(s ξ (y i |x i )) -log(s ξ (y i |x i ))| ≤ D j=1 |ξ j -ξj |ψ j (x i ).
In our case of mixture of regression models, control eigenvalues of the Gram matrix of functions (ψ j (x i )) 1≤j≤D 1≤i≤n corresponds to make some assumptions, as REC, to avoid dimension reliance on n, K and p. Without this kind of assumptions, we could not guarantee that our bound is of order

log(p)
n , because we could not guarantee that eigenvalues does not depend on dimensions. In

PROOF OF THE ORACLE INEQUALITY

74

order to get a result with smaller assumptions, we do not use the chaining argument developed in [START_REF] Van De Geer | The adaptive and the thresholded lasso for potentially misspecified models (and a lower bound for the lasso)[END_REF]. Nevertheless, one can easily compute that, under restricted eigenvalue condition, we could perform the order of the regularization parameter to λ log(p) n log(n).

Proof of the oracle inequality

Main propositions used in this proof

The first result we will prove is the next theorem, which is an 1 -ball mixture multivariate regression model selection theorem for 1 -penalized maximum likelihood conditional density estimation in the Gaussian framework.

Theorem 2.4.1. We observe (x i , y i ) 1≤i≤n with unknown conditional Gaussian mixture density s ξ 0 .

For all m ∈ N * , we consider the 1 -ball S m = {s ξ ∈ S, N [2] 1 (s ξ ) ≤ m} for S = {s ξ , ξ ∈ Ξ}, and Ξ defined by

Ξ = ξ ∈ Ξ : for all k ∈ {1, . . . , K}, max z∈{1,...,q} sup x∈[0,1] p |[β k x] z | ≤ A β , a Σ ≤ m(||Σ -1 k ||) ≤ M (Σ -1 k ) ≤ A Σ , a π ≤ π k . For ξ = (π, β, Σ), let N [2] 1 (s ξ ) = ||β|| 1 = K k=1 p j=1 q z=1 |[β k ] z,j |.
Let ŝm an η m -log-likelihood minimizer in S m , for η m ≥ 0:

- 1 n n i=1 log(ŝ m (y i |x i )) ≤ inf sm∈Sm - 1 n n i=1 log(s m (y i |x i )) + η m .
Assume that for all m ∈ N * , the penalty function satisfies pen(m) = λm with

λ ≥ κ A Σ ∨ 1 a π 1 + 4(q + 1)A Σ A 2 β + log(n) a Σ K n 1 + q log(n) K log(2p + 1) for a constant κ. Then, if m is such that - 1 n n i=1 log(ŝ m(y i |x i )) + pen( m) ≤ inf m∈N * - 1 n n i=1 log(ŝ m (y i |x i )) + pen(m) + η for η ≥ 0, the estimator ŝ m satisfies E(KL n (s ξ 0 , ŝ m)) ≤(1 + κ -1 ) inf m∈N * inf sm∈Sm KL n (s ξ 0 , s m ) + pen(m) + η m + η + κ K n e -1 2 π q/2 A q/2 Σ 2qa π + κ K n K A Σ ∨ 1 a π 1 + 4(q + 1) 2 A Σ A 2 β + log(n) a σ × 1 + A β + q a Σ 2 ;
where κ is a positive constant.

It is an 1 -ball mixture regression model selection theorem for 1 -penalized maximum likelihood conditional density estimation in the Gaussian framework. Its proof could be deduced from the two following propositions, which split the result if the variable Y is large enough or not.

Proposition 2.4.2. We observe (x i , y i ) 1≤i≤n , with unknown conditional density denoted by s ξ 0 . Let M n > 0, and consider the event

T := max i∈{1,...,n} max z∈{1,...,q} |[Y i ] z | ≤ M n .
For all m ∈ N * , we consider the 1 -ball

S m = {s ξ ∈ S, N [2] 1 (s ξ ) ≤ m}
where S = {s ξ , ξ ∈ Ξ} and

N [2] 1 (s ξ ) = ||β|| 1 = K k=1 p j=1 q z=1 |[β k ] z,j | for ξ = (π, β, Σ).
Let ŝm an η m -log-likelihood minimizer in S m , for η m ≥ 0:

- 1 n n i=1 log(ŝ m (y i |x i )) ≤ inf sm∈Sm - 1 n n i=1 log(s m (y i |x i )) + η m . Let C Mn = max 1 aπ , A Σ + 1 2 (|M n | + A β ) 2 A 2 Σ , q(|Mn|+A β )A Σ 2
. Assume that for all m ∈ N * , the penalty function satisfies pen(m) = λm with

λ ≥ κ 4C Mn √ n √ K 1 + 9q log(n) K log(2p + 1)
for some absolute constant κ. Then, any estimate ŝ m with m such that

- 1 n n i=1 log(ŝ m(y i |x i )) + pen( m) ≤ inf m∈N * - 1 n n i=1 log(ŝ m (y i |x i )) + pen(m) + η for η ≥ 0, satisfies E(KL n (s ξ 0 , ŝ m)1 T ) ≤(1 + κ -1 ) inf m∈N * inf sm∈Sm KL n (s ξ 0 , s m ) + pen(m) + η m + κ K 3/2 qC Mn √ n 1 + A β + q a Σ 2 ;
where κ is an absolute positive constant.

Proposition 2.4.3. Let s ξ 0 , T and ŝ m defined as in the previous proposition. Then,

E(KL n (s ξ 0 , ŝ m)1 T c ) ≤ e -1/2 π q/2 A q/2 Σ 2Knqa π e -1/4(M 2 n -2MnA β )a Σ .

Notations

To prove those two propositions, and the theorem, begin with some notations.

For any measurable function g : R q → R , we consider the empirical norm

g n := 1 n n i=1 g 2 (y i |x i );
its conditional expectation

E(g(Y |X)) = R q
g(y|x)s ξ 0 (y|x)dy;

its empirical process

P n (g) := 1 n n i=1 g(y i |x i );
and its normalized process

ν n (g) := P n (g) -E X (P n (g)) = 1 n n i=1 g(y i |x i ) - R q g(y|x i )s ξ 0 (y|x i )dy .
For all m ∈ N * , for all model S m , we define F m by

F m = f m = -log s m s ξ 0 , s m ∈ S m .
Let δ KL > 0. For all m ∈ N * , let η m ≥ 0. There exist two functions, denoted by ŝ m and sm , belonging to S m , such that

P n (-log(ŝ m)) ≤ inf sm∈Sm P n (-log(s m )) + η m ; KL n (s ξ 0 , sm ) ≤ inf sm∈Sm KL n (s ξ 0 , s m ) + δ KL . ( 2
M (m) = m ∈ N * |P n (-log(ŝ m )) + pen(m ) ≤ P n (-log(ŝ m )) + pen(m) + η . ( 2
, A Σ + 1 2 (|M n | + A β ) 2 A 2 Σ , q(|M n | + A β )A Σ /2 . Assume that, for all m ∈ N * , pen(m) = λm, with λ ≥ κC Mn K n 1 + q log(n) K log(2p + 1) .
We derive from the two propositions that there exists κ such that, if m satisfies

- 1 n n i=1 log(ŝ m(y i |x i )) + pen( m) ≤ inf m∈N * - 1 n n i=1 log(ŝ m (y i |x i )) + pen(m) + η; then ŝ m satisfies E(KL n (s ξ 0 , ŝ m)) = E(KL n (s ξ 0 , ŝ m)1 T ) + E(KL n (s ξ 0 , ŝ m)1 T c ) ≤(1 + κ -1 ) inf m∈N * inf sm∈Sm KL n (s ξ 0 , s m ) + pen(m) + η m + κ C Mn √ n K 3/2 q 1 + (A β + q a Σ ) 2 + η + κ e -1/2 π q/2 A q/2 Σ 2Knqa π e -1/4(M 2 n -2MnA β )a Σ .
In order to optimize this equation with respect to M n , we consider M n the positive solution of the polynomial

log(n) - 1 4 (X 2 -2XA β )a Σ = 0; we obtain M n = A β + A 2 β + 4 log(n) a Σ and √ ne -1/4(M 2 n -2MnA β )a Σ = 1 √ n . On the other hand, C Mn ≤ A Σ ∨ 1 a π 1 + q + 1 2 A Σ (M n + A β ) 2 ≤ A Σ ∨ 1 a π 1 + 4(q + 1)A Σ A 2 β + log(n) a Σ .
We get

E(KL n (s ξ 0 , ŝ m)) = E(KL n (s ξ 0 , ŝ m)1 T ) + E(KL n (s ξ 0 , ŝ m)1 T c ) ≤ (1 + κ -1 ) inf m∈N * inf sm∈Sm KL n (s ξ 0 , s m ) + pen(m) + η m + η +κ K n e -1 2 π q/2 (qA Σ ) q/2 2qa π +κ K n A Σ ∨ 1 a π 1 + 4(q + 1)A Σ A 2 β + log(n) a Σ ×K 1 + A β + q a Σ 2 .

Proof of the Theorem 2.3.1

We will show that there exists η m ≥ 0, and η ≥ 0 such that ŝLasso (λ) satisfies the hypothesis of the Theorem 2.4.1, which will lead to Theorem 2.3.1. First, let show that there exists m ∈ N * and η m ≥ 0 such that the Lasso estimator is an

η m -log-likelihood minimizer in S m . For all λ ≥ 0, if m λ = N [2] 1 (ŝ(λ)) , ŝLasso (λ) = argmin s∈S N [2] 1 (s)≤m λ - 1 n n i=1 log(s(y i |x i )) .
We could take η m = 0.

Secondly, let show that there exists η ≥ 0 such that

- 1 n n i=1 log(ŝ Lasso (λ)(y i |x i )) + pen(m λ ) ≤ inf m∈N * - 1 n n i=1 log(ŝ m (y i |x i )) + pen(m) + η.
Taking pen(m λ ) = λm λ ,

- 1 n n i=1 log(ŝ Lasso (λ)(y i |x i )) + pen(m λ ) = - 1 n n i=1 log(ŝ Lasso (λ)(y i |x i )) + λm λ ≤ - 1 n n i=1 log(ŝ Lasso (λ)(y i |x i )) + λN [2] 1 (ŝ Lasso (λ)) + λ ≤ inf s ξ ∈S - 1 n n i=1 log(s ξ (y i |x i )) + λN [2] 1 (s ξ ) + λ ≤ inf m∈N * inf s ξ ∈Sm - 1 n n i=1 log(s ξ (y i |x i )) + λN [2] 1 (s ξ ) + λ ≤ inf m∈N * inf s ξ ∈Sm - 1 n n i=1 log(s ξ (y i |x i )) + λm + λ ≤ inf m∈N * - 1 n n i=1 log(ŝ m (y i |x i )) + λm + λ.
which is exactly the goal, with η = λ. Then, according to the Theorem 2.4.1, with m = m λ , and ŝ m = ŝLasso (λ), for

λ ≥ κ A Σ ∨ 1 a π 1 + 4(q + 1)A Σ A 2 β + log(n) a Σ K n 1 + q log(n) K log(2p + 1) ,
we get the oracle inequality.

2.5 Proof of the theorem according to T or T c

Proof of the Proposition 2.4.2

This proposition corresponds to the main theorem according to the event T . To prove it, we need some preliminary results.

From our notations, reminded in Section 2.4.2, we have, for all m ∈ N * for all m ∈ M (m),

P n ( fm ) + pen(m ) ≤ P n ( fm ) + pen(m) + η ≤ P n ( fm ) + pen(m) + η m + η;

E(P n ( fm )) + pen(m ) ≤ E(P n ( fm )) + pen(m) + η m + η + ν n ( fm ) -ν n ( fm ); KL n (s ξ 0 , ŝm ) + pen(m ) ≤ inf sm∈Sm KL n (s ξ 0 , s m ) + δ KL + pen(m) + η m + η + ν n ( fm ) -ν n ( fm );
(2.9) thanks to the inequality (2.7).

The goal is to bound -ν n ( fm ) = ν n (-fm ).

To control this term, we use the following lemma.

|[Y i ] z | ≤ M n . Let C Mn = max 1 aπ , A Σ + 1 2 (|M n | + A β ) 2 A 2 Σ , q(|Mn|+A β )A Σ 2
and

∆ m = m log(n) K log(2p + 1) + 6 1 + K A β + q a Σ .
Then, on the event T , for all m ∈ N * , for all t > 0, with probability greater than 1 -e -t , sup

f m ∈F m |ν n (-f m )| ≤ 4C Mn √ n 9 √ Kq∆ m + √ 2 √ t 1 + K A β + q a Σ Proof. Page 83
From (2.9), on the event T , for all m ∈ N * , for all m ∈ M (m), for all t > 0, with probability greater than 1 -e -t ,

KL n (s ξ 0 , ŝm ) + pen(m ) ≤ inf sm∈Sm KL n (s ξ 0 , s m ) + δ KL + pen(m) + ν n ( fm ) + 4C Mn √ n 9 √ Kq∆ m + √ 2 √ t 1 + K A β + q a Σ + η m + η ≤ inf sm∈Sm KL n (s ξ 0 , s m ) + pen(m) + ν n ( fm ) + 4 C Mn √ n 9 √ Kq∆ m + 1 2 √ K 1 + K A β + q a Σ 2 + √ Kt + η m + η + δ KL , the last inequality being true because 2ab ≤ 1 √ K a 2 + √ Kb 2 . Let z > 0 such that t = z + m + m .
On the event T , for all m ∈ N, for all m ∈ M (m), with probability greater than 1 -e -(z+m+m ) ,

KL n (s ξ 0 , ŝm ) + pen(m ) ≤ inf sm∈Sm KL n (s ξ 0 , s m ) + pen(m) + ν n ( fm ) + 4 C Mn √ n 9 √ Kq∆ m + 1 2 √ K 1 + K A β + q a Σ 2 + 4 C Mn √ n √ K(z + m + m ) + η m + η + δ KL . KL n (s ξ 0 , ŝm ) -ν n ( fm ) ≤ inf sm∈Sm KL n (s ξ 0 , s m ) + pen(m) + 4 C Mn √ n √ Km + 4C Mn √ n √ K(m + 9q∆ m ) -pen(m ) + 4C Mn √ n 1 2 √ K 1 + K A β + q a Σ 2 + √ Kz + η m + η + δ KL .
Let κ ≥ 1, and assume that pen(m) = λm with

λ ≥ 4C Mn √ n √ K 1 + 9q log(n) K log(2p + 1)

PROOF OF THE THEOREM ACCORDING TO T OR T C

80 Then, as

∆ m = m log(n) K log(2p + 1) + 6 1 + K A β + q a Σ , with κ -1 = 4C Mn √ n √ K 1 λ ≤ 1 1 + 9q log(n) K log(2p + 1) ,
we get that

KL n (s ξ 0 , ŝm ) -ν n ( fm ) ≤ inf sm∈Sm KL n (s ξ 0 , s m ) + (1 + κ -1 ) pen(m) + 4C Mn √ n 1 2 √ K 1 + K A β + q a Σ 2 + 4C Mn √ n 54 √ Kq 1 + K A β + q a Σ + √ Kz + η + δ KL + η m ≤ inf sm∈Sm KL n (s ξ 0 , s m ) + (1 + κ -1 ) pen(m) + 4C Mn √ n 27K 3/2 + 27 + 1/2 √ K 1 + K A β + q a Σ 2 + √ Kz + η m + η + δ KL .
Let m such that

m∈N * m ∈M (m) e -(z+m+m ) ≥ 1 -e -z (m,m )∈(N * ) 2 e -m-m ≥ 1 -e -z ,
we could sum up over all models. On the event T , for all z > 0, with probability greater than 1 -e -z ,

KL n (s ξ 0 , ŝ m) -ν n ( fm ) ≤ inf m∈N * inf sm∈Sm KL n (s ξ 0 , s m ) + (1 + κ -1 ) pen(m) + η m + 4C Mn √ n 27K 3/2 + 55q 2 √ K 1 + K A β + q a Σ 2 + √ Kz + η + δ KL .
By integrating over z > 0, and noticing that E(ν n ( fm )) = 0 and that δ KL can be chosen arbitrary

CHAPTER 2. AN 1 -ORACLE INEQUALITY FOR THE LASSO IN FINITE MIXTURE OF MULTIVARIATE GAUSSIAN REGRESSION MODELS

small, we get

E(KL n (s ξ 0 , ŝ m)1 T ) ≤ inf m∈N * inf sm∈Sm KL n (s ξ 0 , s m ) + (1 + κ -1 ) pen(m) + η m + 4C Mn √ n 27K 3/2 + q √ K 55 2 1 + K A β + q a Σ 2 + √ K + η ≤ inf m∈N * inf sm∈Sm KL n (s ξ 0 , s m ) + (1 + κ -1 ) pen(m) + η m + 332K 3/2 qC Mn √ n 1 + A β + q a Σ 2 + η.

Proof of the Proposition 2.4.3

We want an upper bound of E KL n (s ξ 0 , ŝ m)1 T c . Thanks to the Cauchy-Schwarz inequality,

E KL n (s ξ 0 , ŝ m)1 T c ≤ E(KL 2 n (s ξ 0 , ŝ m)) P (T c ).
However, for all s ξ ∈ S,

KL n (s ξ 0 , s ξ ) = 1 n n i=1 R q log s ξ 0 (y|x i ) s ξ (y|x i ) s ξ 0 (y|x i )dy = 1 n n i=1 R q log(s ξ 0 (y|x i ))s ξ 0 (y|x i )dy - R q log(s ξ (y|x i ))s ξ 0 (y|x i )dy ≤ - 1 n n i=1 R q log(s ξ (y|x i ))s ξ 0 (y|x i )dy.
Because parameters are assumed to be bounded, according to the set (2.3), we get, with (β 0 , Σ 0 , π 0 ) the parameters of s ξ 0 and (β, Σ, π) the parameters of s ξ ,

log(s ξ (y|x i ))s ξ 0 (y|x i ) = log K k=1 π k (2π) q/2 det(Σ k ) exp - (y -β k x i ) t Σ -1 k (y -β k x i ) 2 × K k=1 π 0 k (2π) q/2 det(Σ 0 k ) exp - (y -β 0 k x i ) t (Σ 0 k ) -1 (y -β 0 k x i ) 2 ≥ log   K a π det(Σ -1 1 ) (2π) q/2 exp -(y t Σ -1 1 y + x t i β t 1 Σ -1 1 β 1 x i )   × K a π det((Σ 0 1 ) -1 ) (2π) q/2 exp -(y t Σ -1 1 y + x t i β t 1 Σ -1 1 β 1 x i ) ≥ log K a π a q/2 Σ (2π) q/2 exp -(y t y + A 2 β )A Σ × K a π a q/2 Σ (2π) q/2 exp -(y t y + A 2 β )A Σ .
Indeed, for u ∈ R q , if we use the eigenvalue decomposition of Σ = P t DP ,

|u t Σu| = |u t P t DP u| ≤ ||P u|| 2 ||DP U || 2 ≤ M (D)||P u|| 2 2 ≤ M (D)||u|| 2 2 ≤ A Σ ||u|| 2 2 .
To recognize the expectation of a Gaussian standardized variables, we put u = √ 2A Σ y:

KL(s ξ 0 (.|x i ), s ξ (.|x i )) ≤ - Ka π e -A 2 β A Σ a q/2 Σ (2A Σ ) q/2 R q log Ka q/2 Σ a π (2π) q/2 -A 2 β A Σ - u t u 2 e -u t u 2 (2π) q/2 du ≤ - a q/2 Σ Ka π e -A 2 β A Σ (2A Σ ) q/2 E log Ka π a q/2 Σ (2π) q/2 -A 2 β A Σ - U 2 2 ≤ - Ka q/2 Σ a π e -A 2 β A Σ (2A Σ ) q/2 log Ka π a q/2 Σ (2π) q/2 -A 2 β A Σ - 1 2 ≤ - Ka q/2 Σ a π e -A 2 β A Σ -1/2 (2π) q/2 A q/2 Σ e 1/2 π q/2 log Ka π e -A 2 β A Σ -1/2 a q/2 Σ (2π) q/2 ≤ e -1/2 π q/2 A q/2 Σ ;
where U ∼ N q (0, 1). We have used that for all t ∈ R, t log(t) ≥ -e -1 . Then, we get, for all s ξ ∈ S,

KL n (s ξ 0 , s ξ ) ≤ 1 n n i=1 KL(s ξ 0 (.|x i ), s ξ (.|x i )) ≤ e -1/2 π q/2 A q/2 Σ .
As it is true for all s ξ ∈ S, it is true for ŝ m, then

E(KL 2 n (s ξ 0 , ŝ m)) ≤ e -1/2 π q/2 A q/2 Σ .
For the last step, we need to bound P (T c ).

P (T c ) = E(1 T c ) = E(E X (1 T c )) = E(P X (T c )) ≤ E n i=1 P X (||Y i || ∞ > M n ) . Nevertheless, let Y x ∼ K k=1 π k N q (β k x, Σ k ), then, P (||Y x || ∞ > M n ) = R q 1 {||Yx||∞≥Mn} K k=1 π k 1 (2π) q/2 det(Σ k ) e - (y-β k x) t Σ -1 k (y-β k x) 2 dy = K k=1 π k R q 1 {||Yx||∞≥Mn} 1 (2π) q/2 det(Σ k ) e - (y-β k x) t Σ -1 k (y-β k x) 2 dy = K k=1 π k P X (||Y k x || ∞ > M n ) ≤ K k=1 q z=1 π k P X (|[Y k x ] z | > M n ) with Y k x ∼ N (β k x, Σ k ) and [Y k x ] z ∼ N ([β k x] z , [Σ k ] z,z ).

CHAPTER 2. AN 1 -ORACLE INEQUALITY FOR THE LASSO IN FINITE MIXTURE OF MULTIVARIATE GAUSSIAN REGRESSION MODELS

We need to control P

X (|[Y k x ] z | > M n ), for all z ∈ {1, . . . , q}. P X (|[Y k x ] z | > M n ) = P X ([Y k x ] z > M n ) + P X ([Y x,k ] z < -M n ) = P X U > M n -[β k x] z [Σ k ] z,z + P X U < -M n -[β k x] z [Σ k ] z,z = P X U > M n -[β k x] z [Σ k ] z,z + P X U > M n + [β k x] z [Σ k ] z,z ≤ e -1 2 Mn-[β k x]z √ [Σ k ]z,z 2 + e -1 2
Mn+[β k x]z √ [Σ k ]z,z 2 ≤ 2e -1 2 Mn-|[β k x]z | √ [Σ k ]z,z 2 ≤ 2e -1 2 M 2 n -2Mn|[β k x]z |+|[β k x]z | 2 [Σ k ]z,z
.

where U ∼ N (0, 1). Then,

P (||Y x || ∞ > M n ) ≤ 2Kqe -1 2 (M 2 n -2MnA β )a Σ ,
and we get

P (T c ) ≤ E n i=1 2Kqa π e -1 2 (M 2 n -2MnA β )a Σ ≤ 2Kna π qe -1 2 (M 2 n -2MnA β )a Σ .
We have obtained the wanted bound for E(KL n (s ξ 0 , ŝ m)1 T c ).

Some details

2.6.1 Proof of the Lemma 2.5.1

First, give some tools to prove the Lemma 2.5.1.

We define ||g|| n = 1 n n i=1 g 2 (y i |x i ) for any measurable function g. Let m ∈ N * . We have

sup fm∈Fm |ν n (-f m )| = sup fm∈Fm 1 n n i=1 (f m (y i |x i ) -E(f m (Y i |x i ))) .
To control the deviation of such a quantity, we shall combine concentration with symmetrization arguments. We first use the following concentration inequality which can be found in [START_REF] Boucheron | Concentration Inequalities: A Nonasymptotic Theory of Independence[END_REF].

Lemme 2.6.1. Let (Z 1 , . . . , Z n ) be independent random variables with values in some space Z and let Γ be a class of real-valued functions on Z. Assume that there exists R n a non-random constant such that sup γ∈Γ ||γ|| n ≤ R n . Then, for all t > 0,

P sup γ∈Γ 1 n n i=1 γ(Z i ) -E(γ(Z i )) > E sup γ∈Γ 1 n n i=1 γ(Z i ) -E(γ(Z i )) + 2 √ 2R n t n ≤ e -t .
Proof. See [START_REF] Boucheron | Concentration Inequalities: A Nonasymptotic Theory of Independence[END_REF].

Then, we propose to bound E sup γ∈Γ

1 n n i=1 γ(Z i ) -E(γ(Z i ))
thanks to the following symmetrization argument. The proof of this result can be found in [START_REF] Aw Van Der Vaart | Weak Convergence and Empirical Processes: With Applications to Statistics[END_REF].

Lemme 2.6.2. Let (Z 1 , . . . , Z n ) be independent random variables with values in some space Z and let Γ be a class of real-valued functions on Z. Let ( 1 , . . . , n ) be a Rademacher sequence independent of (Z 1 , . . . , Z n ). Then,

E sup γ∈Γ 1 n n i=1 γ(Z i ) -E(γ(Z i )) ≤ 2 E sup γ∈Γ 1 n n i=1 i γ(Z i ) .
Proof. See [START_REF] Aw Van Der Vaart | Weak Convergence and Empirical Processes: With Applications to Statistics[END_REF].

Then, we have to control E(sup γ∈Γ

1 n n i=1 i γ(Z i ) ).
Lemme 2.6.3. Let (Z 1 , . . . , Z n ) be independent random variables with values in some space Z and let Γ be a class of real-valued functions on Z. Let ( 1 , . . . , n ) be a Rademacher sequence independent of (Z 1 , . . . , Z n ). Define R n a non-random constant such that

sup γ∈Γ ||γ|| n ≤ R n .
Then, for all S ∈ N * ,

E sup γ∈Γ 1 n n i=1 i γ(Z i ) ≤ R n 6 √ n S s=1 2 -s log(1 + N (2 -s R n , Γ, ||.|| n )) + 2 -S
where N (δ, Γ, ||.|| n ) stands for the δ-packing number of the set of functions Γ equipped with the metric induced by the norm ||.|| n .

Proof. See [START_REF] Massart | Concentration inequalities and model selection[END_REF].

In our case, we get the following lemma.

Lemme 2.6.4. Let m ∈ N * . Consider ( 1 , . . . , n ) a Rademacher sequence independent of (Y 1 , . . . , Y n ). Then, on the event T ,

E sup fm∈Fm n i=1 i f m (Y i |x i ) ≤ 18 √ K C Mn q √ n ∆ m ;
where

∆ m := m log(n) K log(2p + 1) + 6(1 + K(A β + q a Σ )).
Proof. Let m ∈ N * . According to Lemma 2.6.5, we get that on the event T ,

sup fm∈Fm ||f m || n ≤ R n := 2C Mn (1 + K(A β + q a Σ )).

CHAPTER 2. AN 1 -ORACLE INEQUALITY FOR THE LASSO IN FINITE MIXTURE OF MULTIVARIATE GAUSSIAN REGRESSION MODELS

Besides, on the event T , for all S ∈ N * , S s=1

2 -s log[1 + N (2 -s R n , F m , ||.|| n )] ≤ S s=1 2 -s log(2N (2 -s R n , F m , ||.|| n )) ≤ S s=1 2 -s log(2) + log(2p + 1) 2 s+1 C Mn qKm R n + S s=1 2 -s K log 1 + 2 s+3 C Mn qK R n a Σ 1 + 2 s+3 C Mn R n according to Lemma 4.15 ≤ S s=1 2 -s log(2) + log(2p + 1) 2 s+1 C Mn qKm R n + S s=1 2 -s K log 1 + 2 s+3 C Mn R n max(1, qK/a Σ ) 2 ≤ S s=1 2 -s log(2) + log(2p + 1) 2 s+1 C Mn qKm R n + 2(s + 3)K log(2)q/a Σ ≤ 2C Mn Kmq R n S log(2p + 1) + log(2) 1 + √ q a Σ √ 6K + 2 S s=1 2 -s √ s ≤ 2C Mn Kmq R n S log(2p + 1) + log(2) 1 + √ q a Σ √ 6K + √ q √ K √ 2e 2 - √ e because 2 -s √ s ≤ √ e 2 s
for all s ∈ N * . Then, thanks to the Lemma 2.6.3,

E sup fm∈Fm 1 n n i=1 i f m (Y i |x i ) ≤ R n 6 √ n S s=1 2 -s log[1 + N (2 -s R n , F m , ||.|| n )] + 2 -S ≤ R n 6 √ n 2C Mn Kmq R n S log(2p + 1) + log(2) 1 + q a Σ √ 6K + q a Σ √ K 2e 2 - √ e + 2 -S .
Taking S = log(n) log(2) to obtain the same order in the both terms depending on S, we could deduce that

E sup fm∈Fm 1 n n i=1 i f m (Y i |x i ) ≤ 12C Mn Kmq √ n log(2p + 1) log(n) log (2)
+ 2C Mn 1 + K A β + q a Σ log(2) √ n 1 + √ 6K + √ 2e 2 - √ 2e + 1 n ≤ 18C Mn Kmq √ n log(2p + 1) log(n) + 2 √ K √ n C Mn 1 + K A β + q a Σ log(2) 1 + √ 6 + √ 2e 2 - √ 2e + 1 ≤18 √ K √ n C Mn mq K log(2p + 1) log(n) + 6 1 + K A β + q a Σ .
It completes the proof.

We are now able to prove the Lemma 2.5.1.

sup fm∈Fm |ν n (-f m )| = sup fm∈Fm 1 n n i=1 (f m (y i |x i ) -E X (f m (Y i |x i ))) ≤ E sup fm∈Fm n i=1 f m (Y i |x i ) -E(f m (Y i |x i )) + 2 √ 2R n t n
with probability greater than 1 -e -t and where R n is a constant computed from the Lemma 2.6.5

≤ 2 E sup fm∈Fm n i=1 i f m (Y i |x i ) + 2 √ 2R n t n with i a Rademacher sequence, independent of Z i ≤ 2 18 √ K C Mn q √ n ∆ m + 2 √ 2R n t n ≤ 4C Mn 9 √ Kq √ n ∆ m + √ 2 t n 1 + K A β + q a Σ .
2.6.2 Lemma 2.6.5 and Lemma 4.15

Lemme 2.6.5. On the event

T = max i∈{1,...,n} max z∈{1,...,q} |[Y i ] z | ≤ M n , for all m ∈ N * , sup fm∈Fm ||f m || n ≤ 2C Mn 1 + K A β + q a Σ := R n .

CHAPTER 2. AN 1 -ORACLE INEQUALITY FOR THE LASSO IN FINITE MIXTURE OF MULTIVARIATE GAUSSIAN REGRESSION MODELS

Proof. Let m ∈ N * . Because f m ∈ F m = f m = -log sm s ξ 0 , s m ∈ S m , there exists s m ∈ S m such that f m = -log sm s ξ 0 . For all x ∈ [0, 1] p , denote ξ(x) = (π, β 1 x, . . . , β K x, Σ) the parameters of s m (.|x). For all i ∈ {1, . . . , n},

|f m (y i |x i )|1 T = | log(s m (y i |x i )) -log(s ξ 0 (y i |x i ))|1 T ≤ sup x∈[0,1] p sup ξ∈Ξ ∂ log(s ξ (y i |x)) ∂ξ ||ξ(x i ) -ξ 0 (x i )|| 1 1 T ,
thanks to the Taylor formula. Then, we need an upper bound of the partial derivate. For all x ∈ [0, 1] p , for all y ∈ R q , we could write

log(s ξ (y|x)) = log K k=1 h k (x, y)
where, for all k ∈ {1, . . . , K},

h k (x, y) = π k (2π) q/2 det Σ k × exp   - 1 2   q z 2 =1   q z 1 =1 y z 1 - p j=1 x j [β k ] z 1 ,j   [Σ k ] -1 z 1 ,z 2     y z 2 - p j=1 [β k ] z 2 ,j x j     .
Then, for all l ∈ {1, . . . , K}, for all z 1 ∈ {1, . . . , q}, for all z 2 ∈ {1, . . . , q}, for all y ∈ R q , for all

x ∈ [0, 1] p , ∂ log(s ξ (y|x)) ∂([β l x] z 1 ) = h l (x, y) K k=1 h k (x, y) - 1 2 q z 2 =1 [Σ l ] -1 z 1 ,z 2 ([β l x] z 2 -y z 2 ) ≤ q(|y| + A β )A Σ 2 ; ∂ log(s ξ (y|x)) ∂([Σ l ] z 1 ,z 2 ) = 1 K k=1 h k (x, y) × -h l Cof z 1 ,z 2 (Σ l ) det(Σ l ) - h l (x, y)(y z 1 -[β l x] z 1 )(y z 2 -[β l x] z 2 )[Σ l ] -2 z 1 ,z 2 2 ≤ -Cof z 1 ,z 2 (Σ l ) det(Σ l ) + (y z 1 -[β l x] z 1 )(y z 2 -[β l x] z 2 )[Σ l ] -2 z 1 ,z 2 2 ≤A Σ + 1 2 (|y| + A β ) 2 A 2 Σ ,
where Cof z 1 ,z 2 (Σ k ) is the (z 1 , z 2 )-cofactor of Σ k . We also have, for all l ∈ {1, . . . , K}, for all x ∈ [0, 1] p , for all y ∈ R q , ∂ log(s ξ (y, x))

∂π l = h l (x, y) π l K k=1 h k (x, y) ≤ 1 a π .
Thus, for all y ∈ R q ,

sup x∈[0,1] p sup ξ∈ Ξ ∂ log(s ξ (y|x)) ∂ξ ≤ max 1 a π , A Σ + 1 2 (|y| + A β ) 2 A 2 Σ , q(|y| + A β )A Σ 2 = C y .
We have

C y ≤ A Σ ∧ 1 aπ 1 + q+1 2 A Σ (|y| + A β ) 2 . For all m ∈ N * , |f m (y i |x i )|1 T ≤ C y i ||ξ(x i ) -ξ 0 (x i )|| 1 1 T ≤ C Mn K k=1 (||β k x i -β 0 k x i || 1 + ||Σ k -Σ 0 k || 1 + |π k -π 0 k |).
Since f m and f 0 m belong to Ξ, we obtain

|f m (y i |x i )|1 T ≤ 2C Mn (KA β + K q a Σ + 1)
and then sup

fm∈Fm ||f m || n 1 T ≤ 2C Mn (KA β + K q a Σ + 1).
For the next results, we need the following lemma, proved in [START_REF] Meynet | An 1 -oracle inequality for the lasso in finite mixture gaussian regression models[END_REF].

Lemme 2.6.6. Let δ > 0 and (A i,j ) i∈{1,...,n} j∈{1,...,p} ∈ [0, 1] n×p . There exists a family B of (2p + 1) 1/δ 2 vectors of R p such that for all µ ∈ R p in the 1 -ball, there exists µ ∈ B such that

1 n n i=1   p j=1 (µ j -µ j )A i,j   2 ≤ δ 2 .
Proof. See [START_REF] Meynet | An 1 -oracle inequality for the lasso in finite mixture gaussian regression models[END_REF].

With this lemma, we can prove the following one:

Lemme 2.6.7. Let δ > 0 and m ∈ N * . On the event T , we have the upper bound of the δ-packing number of the set of functions F m equipped with the metric induced by the norm ||.|| n :

N (δ, F m , ||.|| n ) ≤ (2p + 1) 4C 2 Mn K 2 q 2 m 2 /δ 2 1 + 8C Mn qK a Σ δ K 1 + 8C Mn δ K . Proof. Let m ∈ N * and f m ∈ F m . There exists s m ∈ S m such that f m = -log(s m /s ξ 0 ). Intro- duce s m in S and put f m = -log(s m /s ξ 0 ). Denote by (β k , Σ k , π k ) 1≤k≤K and (β k , Σ k , π k ) 1≤k≤K
the parameters of the densities s m and s m respectively. First, applying Taylor's inequality, on the event

T = max i∈{1,...,n} max z∈{1,...,q} |[Y i ] z | ≤ M n ,
we get, for all i ∈ {1, . . . , n},

|f m (y i |x i ) -f m (y i |x i )|1 T = | log(s m (y i |x i )) -log(s m (y i |x i ))|1 T ≤ sup x∈[0,1] p sup ξ∈ Ξ ∂ log(s ξ (y i |x)) ∂ξ ||ξ(x i ) -ξ (x i )|| 1 1 T ≤ C Mn K k=1 q z=1 [β k x i ] z -[β k x i ] z + ||Σ k -Σ k || 1 + |π k -π k | .

CHAPTER 2. AN 1 -ORACLE INEQUALITY FOR THE LASSO IN FINITE MIXTURE OF MULTIVARIATE GAUSSIAN REGRESSION MODELS

Thanks to the Cauchy-Schwarz inequality, we get that

(f m (y i |x i ) -f m (y i |x i )) 2 1 T ≤ 2C 2 Mn   K k=1 q z=1 β k x i -β k x i 2 + (||Σ -Σ || 1 + ||π -π ||) 2   ≤ 2C 2 Mn   Kq K k=1 q z=1   p j=1 [β k ] z,j [x i ] j - p j=1 [β k ] z,j [x i ] j   2 + (||Σ -Σ || 1 + ||π -π ||) 2   ,
and

||f m -f m || 2 n 1 T ≤2C 2 Mn   Kq K k=1 q z=1 1 n n i=1   p j=1 [β k ] z,j [x i ] j - p j=1 [β k ] z,j [x i ] j   2 + (||Σ -Σ || 1 + ||π -π ||) 2 .
Denote by

a = Kq K k=1 q z=1 1 n n i=1   p j=1 [β k ] z,j [x i ] j - p j=1 [β k ] z,j [x i ] j   2 .
Then, for all δ > 0, if

a ≤ δ 2 /(4C 2 Mn ) ||Σ -Σ || 1 ≤ δ/(4C Mn ) ||π -π || ≤ δ/(4C Mn ) then ||f m -f m || 2 n ≤ δ 2 .
To bound a, we write

a = Kqm 2 K =1 q z=1 1 n n i=1   p j=1 [β k ] z,j m [x i ] j - p j=1 [β k ] z,j m [x i ] j   2
and we apply Lemma 2.6.6 to [β k ] z,. /m for all k ∈ {1, . . . , K}, and for all z ∈ {1, . . . , q}. Since

s m ∈ S m , we have q z=1 p j=1 [β k ] z,j m
≤ 1, thus there exists a family B of (2p + 1) 4C 2 Mn q 2 K 2 m 2 /δ 2 vectors of R p such that for all k ∈ {1, . . . , K}, for all z ∈ {1, . . . , q}, for all [β k ] z,. , there exists

[β k ] z,. ∈ B such that a ≤ δ 2 /(4C 2
Mn ). Moreover, since ||Σ|| 1 ≤ qK a Σ and ||π|| 1 ≤ 1, we get that, on the event T ,

N (δ, F m , ||.|| n ) ≤ card(B)N δ 4C Mn , B K 1 ( qK A Σ ), ||.|| 1 N δ 4C Mn , B K 1 (1), ||.|| 1 ≤ (2p + 1) 4C 2 Mn q 2 K 2 m 2 /δ 2 1 + 8C Mn qK a Σ δ K 1 + 8C Mn δ K Chapter 3
An oracle inequality for the Lasso-MLE procedure data analysis. First, for some data-driven regularization parameters, we construct a relevant variables set. Then, restricted on those sets, we compute the maximum likelihood estimator.

Considering the model collection with various number of components and various sparsities, we select a model thanks to the slope heuristic. Then, we get a clustering of the data thanks to the Maximum A Posteriori principle. This procedure could be used to cluster heterogeneous multivariate regression data and understand which variables explain the clustering, in high-dimension. Considering a regression clustering could refine a clustering, and it could be more adapted for instance for prediction. In this chapter, we focus on the theoretical point of view. We define a penalized criterion which allows to select a model (defined by the number of clusters and the set of relevant variables) from a non-asymptotic point of view. Penalizing the empirical contrast is an idea emerging from the seventies. Akaike, in [START_REF] Akaike | A new look at the statistical model identification[END_REF], proposed the Akaike's Information Criterion (AIC) in 1973, and Schwarz in 1978 in [START_REF] Schwarz | Estimating the Dimension of a Model[END_REF] suggested the Bayesian Information Criterion (BIC). Those criteria are based on asymptotic heuristics. To deal with non-asymptotic observations, Birgé and Massart in [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF] and Barron et al. in [START_REF] Yang | Information-theoretic determination of minimax rates of convergence[END_REF], define a penalized data-driven criterion, which leads to oracle inequalities for model selection. The aim of our approach is to define penalized data-driven criterion which leads to an oracle inequality for our procedure. In our context of regression, Cohen and Le Pennec, in [START_REF] Cohen | Conditional density estimation by penalized likelihood model selection and applications[END_REF], proposed a general model selection theorem for maximum likelihood estimation, adapted from Massart's Theorem in [START_REF] Massart | Concentration inequalities and model selection[END_REF]. Nevertheless, we can not apply it directly, because it is stated for a deterministic model collection, whereas our data-driven model collection is random, constructed by the Lasso estimator. As Maugis and Meynet have done in [START_REF] Meynet | A sparse variable selection procedure in model-based clustering[END_REF] to generalize Massart's Theorem, we extend the theorem to cope with the randomness of our model collection. By applying this general theorem to the finite mixture regression random model collection constructed by our procedure, we derive a convenient theoretical penalty as well as an associated non-asymptotic penalized criteria and an oracle inequality fulfilled by our Lasso-MLE estimator. The advantage of this procedure is that it does not need any restrictive assumption.

To obtain the oracle inequality, we use a general theorem proposed by Massart in [START_REF] Massart | Concentration inequalities and model selection[END_REF], which gives the form of the penalty and associated oracle inequality in term of the Kullback-Leibler and Hellinger loss. In our case of regression, Cohen and Le Pennec, in [START_REF] Cohen | Conditional density estimation by penalized likelihood model selection and applications[END_REF], generalize this theorem in term of Kullback-Leibler and Jensen-Kullback-Leibler loss. Those theorems are based on the centred process control with the bracketing entropy, allowing to evaluate the size of the models. Our setting is more general, because we work with a random family denoted by M. We have to control the centred process thanks to Bernstein's inequality. The rest of this chapter is organized as follows. In the Section 3.2, we define the multivariate Gaussian mixture regression model, and we describe the main steps of the procedure we propose. We also illustrate the requirement of refitting by some simulations. We present our oracle inequality in the Section 3.3.2. In Section 3.4, we illustrate the procedure on simulated dataset and benchmark dataset. Finally, in Section 3.5, we give some tools to understand the proof of the oracle inequality, with a global theorem of model selection with a random collection in Section 3.5.1 and sketch of proofs after. All the details are given in Appendix.

The Lasso-MLE procedure

In order to cluster high-dimensional regression data, we will work with the multivariate Gaussian mixture regression model. This model is developed in [START_REF] Städler | 1 -penalization for mixture regression models[END_REF] in the scalar response case. We generalize it in Section 3.2.1. Moreover, we want to construct a model collection. We propose, in Section 3.2.2, a procedure called Lasso-MLE which constructs a model collection, with various sparsity and various number of components, of Gaussian mixture regression models.

The different sparsities solve the high-dimensional problem. We conclude this section with

CHAPTER 3. AN ORACLE INEQUALITY FOR THE LASSO-MLE PROCEDURE

simulations, which illustrate the advantage of refitting.

Gaussian mixture regression model

We observe n independent couples (x i , y i ) 1≤i≤n realizing random variables (X, Y ), where X ∈ R p , and Y ∈ R q comes from a probability distribution with unknown conditional density denoted by s * . To solve a clustering problem, we use a finite mixture model in regression. In particular, we will approximate the density of Y conditionally to X with a mixture of K multivariate Gaussian regression models. If the observation i belongs to the cluster k, we are looking for β k ∈ R q×p such that y i = β k x i + , where ∼ N (0, Σ k ). Remark that we also have to estimate the number of clusters K. Thus, the random response variable Y ∈ R q depends on a set of random explanatory variables, written X ∈ R p , through a regression-type model. Give more precisions on the assumptions on the model we use.

-The variables Y i , conditionally to X i , are independent, for all i ∈ {1, . . . , n} ; -

Y i |X i = x i ∼ s K ξ (y|x i )dy, with s K ξ (y|x) = K k=1 π k (2π) q/2 det(Σ k ) 1/2 exp - (y -β k x) t Σ -1 k (y -β k x) 2 (3.1) ξ = (π 1 , . . . , π K , β 1 , . . . , β K , Σ 1 , . . . , Σ K ) ∈ Ξ K = Π K × (R q×p ) K × (S ++ q ) K Π K = (π 1 , . . . , π K ); π k > 0 for k ∈ {1, . . . , K} and K k=1 π k = 1 S ++ q
is the set of symmetric positive definite matrices on R q .

We want to estimate the conditional density function s K ξ from the observations. For all k ∈ {1, . . . , K}, β k is the matrix of regression coefficients, and Σ k is the covariance matrix in the mixture component k. The π k s are the mixture proportions. In fact, for a regressor x, for all k ∈ {1, . . . , K}, for all z ∈ {1, . . . , q}, [β k x] z = p j=1 [β k ] z,j x j is the zth component of the mean of the mixture component k. To deal with high-dimensional data, we select variables. Definition 3.2.1. A variable (z, j) ∈ {1, . . . , q} × {1, . . . , p} is said to be irrelevant if, for all k ∈ {1, . . . , K}, [β k ] z,j = 0. A variable is relevant if it is not irrelevant. A model is said to be sparse if there are a few of relevant variables.

We denote by Φ [J] the matrix with 0 on the set c J, and S (K,J) the model with K components and with J for relevant variables set:

S (K,J) = y ∈ R q |x ∈ R p → s (K,J) ξ (y|x) (3.2)
where

s (K,J) ξ (y|x) = K k=1 π k (2π) q/2 det(Σ k ) 1/2 exp - (y -β [J] k x) t Σ -1 k (y -β [J] k x) 2 .
This is the main model used in this chapter. To construct the set of relevant variables J, we use the Lasso estimator. Rather than select a regularization parameter, we consider a collection, which leads to a model collection. Detail the procedure.

The Lasso-MLE procedure

The procedure we propose, which is particularly interesting in high-dimension, could be decomposed into three main steps. First, we construct a model collection, with models more or less sparse and with more or less components. Then, we refit estimations with the maximum likelihood estimator. Finally, we select a model thanks to the slope heuristic. It leads to a clustering according to the MAP principle on the selected model.

Model collection construction

The first step consists of constructing a collection of models {S (K,J) } (K,J)∈M in which the model S (K,J) is defined by equation (3.2), and the model collection is indexed by M = K × J . Denote by K ⊂ N * the possible number of components, and denote by J a collection of subsets of {1, . . . , q} × {1, . . . , p}.

To detect the relevant variables, and construct the set J in each model, we generalize the Lasso estimator. Indeed, we penalize the empirical contrast by an 1 -penalty on the mean parameters proportional to

||P k β k || 1 = p j=1 q z=1 |(P k β k ) z,j |, where P t k P k = Σ -1
k for all k ∈ {1, . . . , K}. Then, we will consider ξLasso

K (λ) = argmin ξ=(π,β,Σ)∈Ξ K - 1 n n i=1 log(s K ξ (y i |x i )) + λ K k=1 π k ||P k β k || 1 .
This leads to penalize simultaneously the 1 -norm of the mean coefficients and small variances.

Computing those estimators lead to construct the relevant variables set. For a fixed number of mixture components K ∈ K, denote by G K a candidate of regularization parameters. Fix a parameter λ ∈ G K , we could then use an EM algorithm to compute the Lasso estimator, and construct the set of relevant variables J (λ,K) , saying the non-zero coefficients. We denote by J the random collection of all these sets, J = K∈K λ∈G K J (λ,K) .

Refitting The second step consists of approximating the maximum likelihood estimator ŝ(K,J) = argmin

t∈S (K,J) - 1 n n i=1 log(t(y i |x i ))
using an EM algorithm for each model (K, J) ∈ K×J . Remark that we estimate all parameters, to reduce bias induced by the Lasso estimator.

Model selection

The third step is devoted to model selection. We get a model collection, and we need to select the best one. Because we do not have access to s * , we can not take the one which minimizes the risk. The Theorem 3.5.1 solves this problem: we get a penalty achieving to an oracle inequality. Then, even if we do not have access to s * , we know that we can do almost like the oracle.

Why refit the Lasso estimator?

In order to illustrate the refitting, we compute multivariate data, the restricted eigenvalue condition being not satisfied, and run our procedure. We consider an extension of the model studied in Giraud et al. article [START_REF] Baraud | Gaussian model selection with an unknown variance[END_REF] in the Section 6.3. Indeed, this model is a linear regression with a scalar response which does not satisfy the restricted eigenvalues condition. Then, we define different classes, to get a finite mixture regression model, which does not satisfied the restricted eigenvalues condition, and extend the dimension for multivariate response. We could compare

[x] 1 = (1, -1, 0, . . . , 0) t / √ 2 [x] 2 = (-1, 1.001, 0, . . . , 0) t / √ 1 + 0.001 2 [x] 3 = (1/ √ 2, 1/ √ 2, 1/n, . . . , 1/n) t / 1 + (n -2)/n 2
and for 4 ≤ j ≤ n, let [x] j be the j th vector of the canonical basis of R n . We take a sample of size n = 20, and vector of size p = q = 10. We consider two classes, each of them defined by [β 1 ] z,j = 10 and [β 2 ] z,j = -10 for j ∈ {1, . . . , 2}, z ∈ {1, . . . , 10}. Moreover, we define the covariance matrix of the noise by a diagonal matrix with 0.01 for diagonal coefficient in each class.

We run our procedure on this model, and compare it with the Lasso estimator, without refitting. We compute the model selected by the slope heuristic over the model collection constructed by the Lasso estimator. In Figure 3.1 are the boxplots of each procedure, running 20 times. The Kullback-Leibler divergence is computed over a sample of size 5000. We could see that a refitting after variable selection by the Lasso estimator leads to a better estimation, according to the Kullback-Leibler loss.

An oracle inequality for the Lasso-MLE model

Before state the main theorem of this chapter, we need to precise some definitions and notations.

Notations and framework

We assume that the observations (x i , y i ) 1≤i≤n are realizations of random variables (X, Y ) where X ∈ R p and Y ∈ R q . For (K, J) ∈ K × J , for a model S (K,J) , we denote by ŝ(K,J) the maximum likelihood estimator

ŝ(K,J) = argmin s (K,J) ξ ∈S (K,J) - n i=1 log s (K,J) ξ (y i |x i ) .
To avoid existence issue, we could work with almost minimizer of this quantity and define an η-log-likelihood minimizer:

n i=1 -log(ŝ (K,J) (y i |x i )) ≤ inf s (K,J) ξ ∈S (K,J) n i=1 -log s (K,J) ξ (y i |x i ) + η.

AN ORACLE INEQUALITY FOR THE LASSO-MLE MODEL

96

The best model in this collection is the one with the smallest risk. However, because we do not have access to the true density s * , we can not select the best model, which we call the oracle. Thereby, there is a trade-off between a bias term measuring the closeness of s * to the set S (K,J) and a variance term depending on the complexity of the set S (K,J) and on the sample size. A good set S (K,J) will be one for which this trade-off leads to a small risk bound. Because we are working with a maximum likelihood approach, the most natural quality measure is thus the Kullback-Leibler divergence denoted by KL.

KL(s, t) =      R log s(y) t(y) s(y)dy if sdy << tdy; + ∞ otherwise; (3.3)
for s and t two densities.

As we deal with conditional densities and not classical densities, the previous divergence should be adapted. We define the tensorized Kullback-Leibler divergence by

KL ⊗n (s, t) = E 1 n n i=1 KL(s(.|x i ), t(.|x i )) .
This divergence, defined in [START_REF] Cohen | Conditional density estimation by penalized likelihood model selection and applications[END_REF] appears as the natural one in this regression setting. Namely, we use the Jensen-Kullback-Leibler divergence JKL ρ with ρ ∈ (0, 1), which is defined by

JKL ρ (s, t) = 1 ρ KL(s, (1 -ρ)s + ρt);
and the tensorized one

JKL ⊗n ρ (s, t) = E 1 n n i=1 JKL ρ (s(.|x i ), t(.|x i )) .
This divergence is studied in [START_REF] Cohen | Conditional density estimation by penalized likelihood model selection and applications[END_REF]. We use this divergence rather than the Kullback-Leibler one because we need a boundedness assumption on the controlled functions that is not satisfied by the log-likelihood differences -log s (K,J) ξ /s * . When considering the Jensen-Kullback-Leibler divergence, those ratios are replaced by ratios

- 1 ρ log   (1 -ρ)s * + ρs (K,J) ξ s *  
that are close to the log-likelihood differences when s m are close to s * and always upper bounded by -log(1 -ρ)/ρ. Indeed, this bound is needed to use deviation inequalities for sums of random variables and their suprema, which is the key of the proof of oracle type inequality.

Oracle inequality

We denote by (S (K,J) ) (K,J)∈K×J L the model collection constructed by the Lasso-MLE procedure, with J L a random subcollection of P({1, . . . , q}×{1, . . . , p}) constructed by the Lasso estimator. The grid of regularization parameter considered is data-driven, then random. Because we work in high-dimension, we could not look at all subsets of P({1, . . . , q}×{1, . . . , p}). Considering the Lasso estimator through its regularization path is the solution chosen here, but it needs more control because of the random family. To get theoretical results, we need to work with restricted

CHAPTER 3. AN ORACLE INEQUALITY FOR THE LASSO-MLE PROCEDURE parameters. Assume Σ k diagonal, with Σ k = diag([Σ k ] 1,1 , . . . , [Σ k ] q,q
), for all k ∈ {1, . . . , K}. We define

S B (K,J) = s (K,J) ξ ∈ S (K,J) for all k ∈ {1, . . . , K}, [β k ] [J] ∈ [-A β , A β ] q×p , a Σ ≤ [Σ k ] z,z ≤ A Σ for all z ∈ {1, . . . , q}, for all k ∈ {1, . . . , K} . (3.4)
Moreover, we assume that the covariates X belong to an hypercube. Without any restriction, we could assume that X ∈ [0, 1] p .

Remark 3.3.1. We have to denote that in this chapter, the relevant variables set is designed by the Lasso estimator. Nevertheless, any tool could be used to construct this set, and we obtain analog results. We could work with any random subcollection of P({1, . . . , q} × {1, . . . , p}), the controlled size being required in high-dimensional case.

Theorem 3.3.2. Let (x i , y i ) 1≤i≤n the observations, with unknown conditional density s * . Let S (K,J) defined by (3.2). For (K, J) ∈ K×J L , J L being a random subcollection of P({1, . . . , q}× {1, . . . , p}) constructed by the Lasso estimator, denote S B (K,J) the model defined by (3.4). Consider the maximum likelihood estimator

ŝ(K,J) = argmin s (K,J) ξ ∈S B (K,J) - 1 n n i=1 log s (K,J) ξ (y i |x i ) .
Denote by D (K,J) the dimension of the model S B (K,J) , D (K,J) = K(|J| + q + 1) -1. Let s(K,J) ∈ S B (K,J) such that KL ⊗n (s * , s(K,J) ) ≤ inf

t∈S B (K,J) KL ⊗n (s * , t) + δ KL n ;
and let τ > 0 such that s(K,J) ≥ e -τ s * . Let pen : K × J → R + , and suppose that there exists an absolute constant κ > 0 and an absolute constant B(A β , A Σ , a Σ ) such that, for all

(K, J) ∈ K × J , pen(K, J) ≥ κ D (K,J) n B 2 (A β , A Σ , a Σ ) -log D (K,J) n B 2 (A β , A Σ , a Σ ) ∧ 1 +(1 ∨ τ ) log 4epq (D (K,J) -q 2 ) ∧ pq .
Then, the estimator ŝ( K, Ĵ) , with

( K, Ĵ) = argmin (K,J)∈K×J L - 1 n n i=1 log(ŝ (K,J) (y i |x i )) + pen(K, J) satisfies E JKL ⊗n ρ (s * , ŝ( K, Ĵ) ) ≤C 1 E inf (K,J)∈K×J L inf t∈S B (K,J)
KL ⊗n (s * , t) + pen(K, J)

+ C 2 (1 ∨ τ ) n ;
for some absolute positive constants C 1 and C 2 .

NUMERICAL EXPERIMENTS

98

This oracle inequality compares performances of our estimator with the best model in the collection. Nevertheless, as we consider mixture of Gaussian, if we take enough clusters, we could approximate well a lot of densities. This result could be compared with the oracle inequality get in [START_REF] Städler | 1 -penalization for mixture regression models[END_REF], Theorem 4. Indeed, under restricted eigenvalues condition and fixed design, they get an oracle inequality for the Lasso estimator in finite mixture regression model, with scalar response and high-dimension regressors. Note that they control the divergence with the true parameters. We get a similar result for the Lasso-MLE estimator. Moreover, our procedure work in a more general framework, the only assumption needed is to be bounded.

Numerical experiments

To illustrate this procedure, we study some simulations and real data. The main algorithm is a generalized version of the EM algorithm, which is used many times for the procedure. We first use it to compute maximum likelihood estimator, to construct the regularization parameter grid.

Then, we use it to compute the Lasso estimator for each regularization parameter belonging to the grid, and we are able to construct the relevant variables set. Finally, we could compute the maximum likelihood estimator, restricted to those relevant variables in each model. Among this model collection, we select one using the Capushe package. More details, as initialization rule, stopping rule, and more numerical experiments, are available in [START_REF] Devijver | Model-based clustering for high-dimensional data[END_REF].

Simulation illustration

We illustrate the procedure on a simulated dataset, adapted from [START_REF] Städler | 1 -penalization for mixture regression models[END_REF]. Let x be a sample of size n = 100 distributed according to multivariate standard Gaussian. We consider a mixture of two components, and we fix the dimension of the regressor and of the response variables to p = q = 10. Besides, we fix the number of relevant variables to 4 in each cluster. More precisely, the first four variables of Y are explained respectively by the four first variables of X.

Fix π = 1 2 , 1 2 , [β 1 ] [J] = 3, [β 2 ] [J]
= -2 and P k = 3I q for all k ∈ {1, 2}. The difficulty of the clustering is partially controlled by the signal-to-noise ratio. In this context, we could extend the natural idea of the SNR with the following definition, where Tr(A) denotes the trace of the matrix A.

SNR =

Tr(Var(Y ))

Tr(Var(Y |β k = 0 for all k ∈ {1, . . . , K})) = 1.88.
We take a sample of Y knowing X = x according to a Gaussian mixture, meaning in β k x and with covariance matrix Σ k = (P t k P k ) -1 = σI q , for the cluster k ∈ {1, 2}. We run our procedures with the number of components varying in K = {2, . . . , 5}. To compare our procedure with others, we compute the Kullback-Leibler divergence with the true density, the ARI (the Adjusted Rand Index measures the similarity between two data clusterings, knowing that the closer to 1 the ARI, the more similar the two partitions), and how many clusters are selected. From the Lasso-MLE model collection, we construct two models, to compare our procedures with. We compute the oracle (the model which minimizes the Kullback-Leibler divergence with the true density), and the model which is selected by the BIC criterion instead of the slope heuristic. Thanks to the oracle, we know how good we could be from this model collection for the Kullback-Leibler divergence, and how this model, as good it is possible for the contrast, performs the clustering. The third procedure we compare with is the maximum likelihood estimator, assuming that we know how many clusters there are, fixed to 2. We use this procedure to show that variable selection is necessary. Results are summarized in Figure 3.2 and in Figure 3.3. The Kullback-Leibler divergence is smaller for models selected in our model collection (else by BIC, or by slope heuristic, or the oracle) than for the model constructed by the MLE. The ARI is closer to 1 in those case, and, moreover, is better for the model selected by the slope heuristic. We could conclude that the model collection is well constructed, selecting relevant variables, and also that the model is well selected among this collection, near the oracle.

Real data

We also illustrate the procedure on the Tecator dataset, which deal with spectrometric data. We summarize here results which are described in [START_REF] Devijver | Model-based clustering for high-dimensional data[END_REF]. Those data have been studied in a lot of articles, cite for example Ferraty and Vieu's book [START_REF] Ferraty | Nonparametric functional data analysis : theory and practice[END_REF]. The data consist of a 100-channel spectrum of absorbances in the wavelength range 850 -1050 nm, and of the percentage of fat. We observe a sample of size 215. In this work, we focus on clustering data according to the reliance between the fat content and the absorbance spectrum. The sample will be split into two subsamples, 165 observations for the learning set, and 50 observations for the test set. We split it to have the same marginal distribution of the response in each sample.

The spectrum is a function, which we decompose into the Haar basis, at level 6.

The procedure selects two models, which we describe here. In Figures (3.4) and (3.5), we represent clusters done on the training set for the different models.

The graph on the left is a candidate for representing each cluster, constructed by the mean of spectrum over an a posteriori probability greater than 0.6. We plot the curve reconstruction, keeping only active variables in the wavelet decomposition. On the right side, we present the boxplot of the fat values in each class, for observations with an a posteriori probability greater than 0.6.

The first model has two clusters, which could be distinguish in the absorbance spectrum by the bump on wavelength around 940 nm. The first class is dominating, with π1 = 0.95. The fat content is smaller in the first class than in the second class. According to the signal reconstruction, we could see that almost all variables have been selected. This model seems consistent according to the classification goal.

The second model has 3 clusters, and we could remark different important wavelength. Around 940 nm, there is some differences between clusters, corresponding to the bump underline in the model 1, but also around 970 nm, with higher or smaller values. The first class is dominating,

CHAPTER 3. AN ORACLE INEQUALITY FOR THE LASSO-MLE PROCEDURE

Tools for proof

In this section, we present the tools needed to understand the proof. First, we present a general theorem for model selection in regression among a random collection. Then, in subsection 3.5.2, we present the proof of this theorem, and in the next subsection we explain how we could use the main theorem to get the oracle inequality. All details are available in Appendix.

General theory of model selection with the maximum likelihood estimator.

To

(d ⊗n H ) 2 (s, t) = E 1 n n i=1 d 2 H (s(.|x i ), t(.|x i )) .
A bracket [l, u] is a pair of functions such that for all (x, y) ∈ X × Y, l(y, x) ≤ s(y|x) ≤ u(y, x).

The bracketing entropy H [.] ( , S, d ⊗n H ) of a set S is defined as the logarithm of the minimum number of brackets [l, u] of width d ⊗n H (l, u) smaller than such that every functions of S belong to one of these brackets. It leads to the Assumption H m .

Assumption H m . There is a non-decreasing function φ m such that → 1 φ m ( ) is nonincreasing on (0, +∞) and for every ∈ R + and every s m ∈ S m , 0

H [.] ( , S m (s m , ), d ⊗n H )d ≤ φ m ( );
where S m (s m , ) = {t ∈ S m , d ⊗n H (t, s m ) ≤ }. The model complexity D m is then defined as n 2 m with m the unique root of

1 φ m ( ) = √ n . (3.5)
Remark that the model complexity depends on the bracketing entropies not of the global models S m but of the ones of smaller localized sets. This is a weaker assumption.

For technical reason, a separability assumption is also required.

Assumption Sep m . There exists a countable subset S m of S m and a set Y m with λ(Y \Y m ) = 0 such that for every t ∈ S m , there exists a sequence (t l ) l≥1 of elements of S m such that for every x and every y ∈ Y m , log(t l (y|x)) goes to log(t(y|x)) as l goes to infinity.

This assumption leads to work with a countable family, which allows to cope with the randomness of ŝm . We also need an information theory type assumption on our collection. We assume the existence of a Kraft-type inequality for the collection.

Assumption K. There is a family (w m ) m∈M of non-negative numbers such that m∈M e -wm ≤ Ω < +∞.

The difference with Cohen and Le Pennec's Theorem is that we consider a random collection of models M, included in the whole collection M. In our procedure, we deal with high-dimensional models, and we cannot look after all the models: we have to restrict ourselves to a smaller subcollection of models, which is then random. In the proof of the theorem, we have to be careful with the recentred process of -log(s m /s * ). Because we conclude by taking the expectation, if M is fixed, this term is non-interesting, but if we consider a random family, we have to use the Bernstein inequality to control this quantity, and then we have to make the assumption (3.6). Let state our main global theorem. Then, for any ρ ∈ (0, 1) and any C 1 > 1, there are two constants κ 0 and C 2 depending only on ρ and C 1 such that, as soon as for every index m ∈ M,

pen(m) ≥ κ(D m + (1 ∨ τ )w m ) (3.7)
with κ > κ 0 , and where the model complexity D m is defined in (3.5), the penalized likelihood estimate ŝ m with m ∈ M such that

- n i=1 log(ŝ m(y i |x i )) + pen( m) ≤ inf m∈ M - n i=1 log(ŝ m (y i |x i )) + pen(m) + η satisfies E(JKL ⊗n ρ (s * , ŝ m)) ≤C 1 E inf m∈ M inf t∈Sm KL ⊗n (s * , t) + 2 pen(m) n + C 2 (1 ∨ τ ) Ω 2 n + η + η n . (3.8)
Obviously, one of the models minimizes the right hand side. Unfortunately, there is no way to know which one without knowing s * . Hence, this oracle model can not be used to estimate s * . We nevertheless propose a data-driven strategy to select an estimate among the collection of estimates {ŝ m } m∈ M according to a selection rule that performs almost as well as if we had known this oracle, according to the absolute constant C 1 . Using simply the log-likelihood of the estimate in each model as a criterion is not sufficient. It is an underestimation of the true risk of the estimate and this leads to select models that are too complex. By adding an adapted penalty pen(m), one hopes to compensate for both the variance term and the bias term between

CHAPTER 3. AN ORACLE INEQUALITY FOR THE LASSO-MLE PROCEDURE

-1/n n i=1 log (ŝ m(y i |x i )/s * (y i |x i )) and inf sm∈Sm KL ⊗n (s * , s m ). For a given choice of pen(m), the best model S m is chosen as the one whose index is an almost minimizer of the penalized η-log-likelihood. Talk about the assumption (3.6). If s is bounded, with a compact support, this assumption is satisfied. It is also satisfied in other cases, more particular. Then it is not a strong assumption, but it is needed to control the random family. This theorem is available for whatever model collection constructed, whereas assumptions H m , K and Sep m are satisfied. In the following, we will use this theorem for the procedure we propose to cluster high-dimensional data. Nevertheless, this theorem is not specific for our context, and could be used whatever the problem. Remark that the constant associated to the Assumption K appears squared in the bound. It is due to the random subcollection M of M, if the model collection is fixed, we get a linear bound. Moreover, the weights w m appears linearly in the penalty bound.

Proof of the general theorem

For any model S m , we have denoted by sm a function such that

KL ⊗n (s * , sm ) ≤ inf sm∈Sm KL ⊗n (s * , s m ) + δ KL n . Fix m ∈ M such that KL ⊗n (s * , sm ) < +∞. Introduce M(m) = m ∈ M P n (-log ŝm ) + pen(m ) n ≤ P n (-log ŝm ) + pen(m) n + η n ;
where P n (g) = 1/n n i=1 g(y i |x i ). We define the functions kl(s m ), kl(ŝ m ) and jkl ρ (ŝ m ) by

kl(s m ) = -log sm s * ; kl(ŝ m ) = -log ŝm s * ; jkl ρ (ŝ m ) = - 1 ρ log (1 -ρ)s * + ρŝ m s * .
For every m ∈ M(m), by definition,

P n (kl(ŝ m )) + pen(m ) n ≤P n (kl(ŝ m )) + pen(m) + η n ≤P n (kl(s m )) + pen(m) + η + η n .
Let ν ⊗n n (g) denote the recentred process P n (g) -P ⊗n (g). By concavity of the logarithm,

kl(ŝ m ) ≥ jkl ρ (ŝ m ),
and then

P ⊗n (jkl ρ (ŝ m )) -ν ⊗n n (kl(s m )) ≤P ⊗n (kl(s m )) + pen(m) n -ν ⊗n n (jkl ρ (ŝ m )) + η + η n - pen(m ) n ,
which is equivalent to

JKL ⊗n ρ (s * , ŝm ) -ν ⊗n n (kl(s m )) ≤ KL ⊗n (s * , sm ) + pen(m) n -ν ⊗n n (jkl ρ (ŝ m )) + η + η n - pen(m ) n .
(3.9)

Mimic the proof as done in Cohen and Le Pennec [START_REF] Cohen | Conditional density estimation by penalized likelihood model selection and applications[END_REF], we could obtain that except on a set of probability less than e -w m -w , for all w, for all z m > σ m , there exist absolute constants κ 0 , κ 1 , κ 2 such that

-ν ⊗n n (jkl ρ (ŝ m )) z 2 m + κ 0 (d ⊗n H ) 2 (s * , ŝm ) ≤ κ 1 σ m z m + κ 2 w m + w nz 2 m + 18 ρ w m + w nz 2 m . (3.10)
To obtain this inequality we use the hypothesis Sep m and H m . This control is derived from maximal inequalities, described in [START_REF] Massart | Concentration inequalities and model selection[END_REF].

Our purpose is now to control ν ⊗n n (kl(s m )). This is the difference with the Theorem of Cohen and Le Pennec: we work with a random subcollection M L of M. By definition of kl and ν ⊗n n ,

ν ⊗n n (kl(s m )) = - 1 n n i=1 log sm (y i |x i ) s * (y i |x i ) + E 1 n n i=1 log sm (Y i |X i ) s * (Y i |X i ) .
We want to apply Bernstein's inequality, which is recalled in Appendix. If we denote by Z i the random variable

Z i = -1 n log sm(Yi|Xi) s * (Y i |X i ) , we get ν ⊗n n (kl(s m )) = n i=1 (Z i -E(Z i )).
We need to control the moments of Z i to apply Bernstein's inequality.

Lemme 3.5.2. Let s * and sm two conditional densities with respect to the Lebesgue measure. Assume that there exists τ > 0 such that log

s * sm ∞ ≤ τ . Then, E 1 n n i=1 R q log s * (y|x i ) sm (y|x i ) 2 s * (y|x i )dy ≤ τ 2 e -τ + τ -1 KL ⊗n (s * , sm ).
We prove this lemma in Appendix 3.6.2. Because

τ 2 e -τ +τ -1 ∼ τ →∞ τ , there exists A such that τ 2
e -τ +τ -1 ≤ 2τ for all τ ≥ A. For τ ∈ (0, A], because this function is continuous and equivalent to 2 in 0, there exists B > 0 such that

τ 2 e -τ +τ -1 ≤ B. We obtain that n i=1 E(Z 2 i ) ≤ 1 n δ(1 ∨ τ ) KL ⊗n (s * , sm ), where δ = 2 ∨ B. Moreover, for all integers K ≥ 3, n i=1 E((Z i ) K + ) ≤ n i=1 1 n K R q log s * (y|x i ) sm (y|x i ) K + s * (y|x i )dy ≤ n n K R q log s * (y|x) sm (y|x) K-2 log s * (y|x) sm (y|x) 2 1 s * ≥sm(y|x) s * (y|x)dy ≤ n n K τ K-2 δ(1 ∨ τ ) KL ⊗n (s * , sm ). PROCEDURE
Assumptions of Bernstein's inequality are then satisfied, with

v = δ(1 ∨ τ ) KL ⊗n (s * , sm ) n , c = τ n ,
thus, for all u > 0, except on a set with probability less than e -u ,

ν ⊗n n (kl(s m )) ≤ √ 2vu + cu.
Thus, for all z > 0, for all u > 0, except on a set with probability less than e -u ,

ν ⊗n n (kl(s m )) z 2 + KL ⊗n (s * , sm ) ≤ √ 2vu + cu z 2 + KL ⊗n (s * , sm ) ≤ √ vu z 2 KL ⊗n (s * , sm ) + cu z 2 . (3.11)
We apply this bound to u = w + w m + w m . We get that, except on a set with probability less than e -(w+wm+w m ) , using that a 2 + b 2 ≥ a 2 , from the inequality (3.10),

-ν ⊗n n (jkl ρ (ŝ m )) ≤ z 2 m + κ 0 (d ⊗n H ) 2 (s * , ŝm ) κ 1 + κ 2 θ + 18 θ 2 ρ ,
and, from the inequality (3.11),

ν ⊗n n (kl(s m )) ≤ (β + β 2 ) z 2 m,m + KL ⊗n (s, s m ) ,
where we have chosen

z m = θ σ 2 m + w m + w n ,
with θ > 1 to fix later, and

z m,m = β -1 v 2 KL ⊗n (s * , sm ) + c (w + w m + w m ),
with β > 0 to fix later.

Coming back to the inequality (3.9),

JKL ⊗n ρ (s * , ŝm ) ≤ KL ⊗n (s * , sm ) + pen(m) n + (z 2 m + κ 0 (d ⊗n H ) 2 (s * , ŝm )) κ 1 + κ 2 θ + 18 θ 2 ρ + η + η n - pen(m ) n + (β + β 2 )(z 2 m,m + KL ⊗n (s * , sm )).
Recall that sm is chosen such that

KL ⊗n (s * , sm ) ≤ inf sm∈Sm KL ⊗n (s * , s m ) + δ KL n .
Put κ(β) = 1 + (β + β 2 ), and let 1 > 0, we define θ 1 by κ 0

κ 1 +κ 2 θ 1 + 18 θ 2 1 ρ = C ρ 1 where C ρ is defined by C ρ (d ⊗n H ) 2 (s * , ŝm ) ≤ JKL ⊗n ρ (s * , ŝm ), and put κ 2 = Cρ 1 κ 0 . We get that (1 -1 ) JKL ⊗n ρ (s * , ŝm ) ≤κ(β) KL ⊗n (s * , s m ) + pen(m) n - pen(m ) n + κ(β) δ KL n + η + η n + z 2 m κ 2 + (β + β 2 )z 2 m,m . Since τ ≤ 1 ∨ τ , if we choose β such that (β + β 2 )(δ/2 + 1) = αθ -2 1 β -2
, and if we put κ 1 = αγ -2 (β -2 + 1), since 1 ≤ 1 ∨ τ , using the expressions of z m and z m,m , we get that

(1 -1 ) JKL ⊗n ρ (s * , ŝm ) ≤κ(β) KL ⊗n (s * , s m ) + pen(m) n - pen(m ) n + κ(β) δ KL n + η + η n + κ 2 θ 2 1 σ 2 m + w + w m n + κ 1 (1 ∨ τ ) w + w m + w m n ≤κ(β) KL ⊗n (s * , s m ) + pen(m) n + κ 1 (1 ∨ τ ) w m n + - pen(m ) n + κ 2 θ 2 1 σ 2 m + w m n + κ 1 (1 ∨ τ ) w m n + δ KL n + η + η n + (κ 2 θ 2 1 + κ 1 (1 ∨ τ )) w n .
Now, assume that κ 1 ≥ κ in inequality (3.7), we get = Ω 2 e -w from the Assumption K. We then have simultaneously for all m ∈ M, for all m ∈ M(m), except on a set with probability less than Ω 2 e -w ,

(1 -1 ) JKL ⊗n ρ (s * , ŝm ) ≤κ(β) KL ⊗n (s * , s m ) + 2 pen(m) n + δ KL n + η + η n + (κ 2 θ 2 1 + κ 1 (1 ∨ τ ))
(1 -1 ) JKL ⊗n ρ (s * , ŝm ) ≤κ(β) KL ⊗n (s * , s m ) + 2 pen(m) n + δ KL n + η + η n + κ 2 θ 2 1 + κ 1 (1 ∨ τ ) w n .
It is in particular satisfied for all m ∈ M and m ∈ M(m), and, since m ∈ M(m) for all m ∈ M, we deduce that except on a set with probability less than Ω 2 e -w ,

JKL ⊗n ρ (s * , ŝ m) ≤ 1 (1 -1 ) × inf m∈ M κ(β) KL ⊗n (s * , s m ) + 2 pen(m) n + δ KL n + η + η n + κ 2 θ 2 1 + κ 1 (1 ∨ τ ) w n .

CHAPTER 3. AN ORACLE INEQUALITY FOR THE LASSO-MLE PROCEDURE

By integrating over all w > 0, because for any non negative random variable Z and any a > 0, E(Z) = a z≥0 P (Z > az)dz, we obtain that

E JKL ⊗n ρ (s * , ŝ m) - 1 (1 -1 ) inf m∈ M κ(β) KL ⊗n (s * , s m ) + 2 pen(m) n + δ KL + η + η n κ 0 θ 2 ≤ κ 2 θ 2 1 + κ 1 (1 ∨ τ ) Ω 2 n .
As δ KL can be chosen arbitrary small, this implies that

E(JKL ⊗n (s * , ŝ m)) ≤ 1 1 -1 E inf m∈ M κ(β) KL ⊗n (s * , s m ) + pen(m) n + η + η n + (κ 2 θ 2 1 + κ 1 (1 ∨ τ )) Ω 2 n ≤C 1 E inf m∈ M inf t∈Sm KL ⊗n (s * , t) + pen(m) n + C 2 (1 ∨ τ ) Ω 2 n + η + η n with C 1 = 2 1-1 and C 2 = κ 2 θ 2 1 + κ 1 .

Sketch of the proof of the oracle inequality 3.3.2

To prove the Theorem 3.3.2, we have to apply the Theorem 3.5.1. Then, our model collection has to satisfy all the assumptions. Here, m = (K, J). The Assumption Sep m is true when we consider Gaussian densities. If s * is bounded, with compact support, the assumption (3.6) is satisfied. It is also true in other particular cases. We have to look after assumption H m and Assumption K. Here we present only the main step to prove these assumptions. All the details are in Appendix.

Assumption H m

We could take φ m ( ) = 0 H [.] ( , S m , d ⊗n H )d for all > 0. It could be better to consider more local version of the integrated square root entropy, but the global one is enough in this case to define the penalty. As done in Cohen and Le Pennec [START_REF] Cohen | Conditional density estimation by penalized likelihood model selection and applications[END_REF], we could decompose the entropy by

H [.] ( , S B (K,J) , d ⊗n H ) ≤ H [.] ( , Π K , d ⊗n H ) + KH [.] ( , F J , d ⊗n H ) where S B (K,J) =      y ∈ R q |x ∈ R p → s (K,J) ξ (y|x) = K k=1 π k ϕ(y|β [J] k x, Σ k ) ξ = π 1 , . . . , π K , β [J] 1 , . . . , β [J] K , Σ 1 , . . . , Σ K ∈ Ξ(K,J) Ξ(K,J) = Π K × ([-A β , A β ] q×p ) K × ([a Σ , A Σ ] q ) K      Π K = (π 1 , . . . , π K ) ∈ (0, 1) K ; K k=1 π k = 1 F J = ϕ(.|β [J] X, Σ); β ∈ [-A β , A β ] q×p , Σ = diag([Σ] 1,1 , . . . , [Σ] q,q ) ∈ [a Σ , A Σ ] q
where ϕ denote the Gaussian density, and A β , a Σ , A Σ are absolute constants.

TOOLS FOR PROOF 108

Calculus for the proportions We could apply a result proved by Wasserman and Genovese in [START_REF] Genovese | Rates of convergence for the Gaussian mixture sieve[END_REF] to bound the entropy for the proportions. We get that

H [.] ( , Π K , d ⊗n H ) ≤ log K(2πe) K/2 3 K-1 .
Calculus for the Gaussian The family

B (F J ) =                  l(y, x) = (1 + δ) -p 2 q-3q/4 ϕ(y|ν J x, (1 + δ) -1/4 B [1] ) u(y, x) = (1 + δ) p 2 q+3q/4 ϕ(y|ν J x, (1 + δ)B [2] ) B [a] = diag(b i(1) , . . . , b i(q)
), with i a permutation, for a ∈ {1, 2}, and

   b l = (1 + δ) 1-l/2 A Σ , l ∈ {2, . . . , N } ∀(z, j) ∈ J c , ν z,j = 0 ∀(z, j) ∈ J, ν z,j = √ cδA Σ u z,j                  (3.12)
is an -bracket covering for F J , where u z,j is a net for the mean, N is the number of parameters needed to recover all the variance set, δ = 1 √ 2(p 2 q+3/4q) , and c = 5(1-2 -1/4 ) 8 . We obtain that

|B (F J )| ≤ 2 2A β √ cA Σ |J| A Σ a Σ + 1 2 δ -1-|J| ;
and then we get

H [.] ( , F J , d ⊗n H ) ≤ log 2 2A β √ cA Σ |J| A Σ a Σ + 1 2 δ -1-|J| . Proposition 3.5.3. Put D (K,J) = K(1 + |J|).
For all ∈ (0, 1),

H [.] ( , S B (K,J) , d ⊗n H ) ≤ log(C) + D (K,J) log 1 ; with C = 2K(2πe) K/2 2A β √ cA Σ K|J| 3 K-1 A Σ a Σ + 1 2 K .
Determination of a function φ We could take

φ (K,J) ( ) = D (K,J) B(A β , A Σ , a Σ ) + log 1 ∧ 1 .
This function is non-decreasing, and → φ (K,J) ( ) is non-increasing.

The root (K,J) is the solution of φ (K,J) ( (K,J) ) = √ n 2 (K,J) . With the expression of φ (K,J) , we get

2 (K,J) = D (K,J) n B(A β , A Σ , a Σ ) + log 1 (K,J) ∧ 1 . Nevertheless, we know that * = D (K,J) n B(A β , A Σ , a Σ ) minimizes (K,J) : we get 2 (K,J) ≤ D (K,J) n 2B 2 (A β , A Σ , a Σ ) + log 1 D (K,J) n B 2 (A β , A Σ , a Σ ) ∧ 1 . PROCEDURE Assumption K
We want to group models by their dimension.

Lemme 3.5.4. The quantity card{(K, J) ∈ N * × P({1, . . . , q} × {1, . . . , p}), D(K, J) = D} is upper bounded by   

2 pq if pq ≤ D -q 2 epq D-q 2 D-q 2
otherwise.

Proposition 3.5.5. Consider the weight family {w (K,J) } (K,J)∈K×J defined by

w (K,J) = D (K,J) log 4epq (D (K,J) -q 2 ) ∧ pq .
Then we have (K,J)∈K×J e -w (K,J) ≤ 2.

Appendix: technical results

In this appendix, we give more details for the proofs.

Bernstein's Lemma

Lemme 3.6.1 (Bernstein's inequality). Let (X 1 , . . . , X n ) be independent real valued random variables. Assume that there exists some positive numbers v and c such that n i=1 E(X 2 i ) ≤ v, and, for all integers

K ≥ 3, n i=1 E((X i ) K + ) ≤ K! 2 vc K-2 . Let S = n i=1 (X i -E(X i )).
Then, for every positive x, P (S ≥ √ 2vx + cx) ≤ exp(-x).

Proof of Lemma 3.5.2

This proof is adapted from Maugis and Meynet, [START_REF] Meynet | A sparse variable selection procedure in model-based clustering[END_REF]. First, let give some bounds.

Lemme 3.6.2. Let τ > 0. For all x > 0, consider

f (x) = x log(x) 2 , h(x) = x log(x) -x + 1, φ(x) = e x -x -1.
Then, for all 0 < x < e τ , we get

f (x) ≤ τ 2 φ(-τ ) h(x).
To prove this, we have to show that y → φ(y) y 2 is non-decreasing. We omit the proof here. We want to apply this inequality, in order to derive the Lemma 3.5.2. As log

s * sm ∞ ≤ τ , s * sm ∞ ≤ e τ ;
and we could apply the previous inequality to s * /s m . Indeed, for all x, for all y,

f s * (y|x) sm (y|x) ≤ τ 2 φ(-τ ) h s * (y|x)
sm (y|x) .

APPENDIX: TECHNICAL RESULTS

110

Integrating with respect to the density sm , we get that R q s * (y|.) sm (y|.) log s * (y|.) sm (y|.)

2 sm (y|.)dy

≤ R q
τ 2 e -τ -τ -1 s * (y|.) sm (y|.) log s * (y|.) sm (y|.) -s * (y|.) sm (y|.) + 1 sm (y|.)dy

⇐⇒ 1 n n i=1 s * (y|x i ) log s * (y|x i ) sm (y|x i ) 2 dy ≤ τ 2 e -τ -τ -1 1 n n i=1 s * (y|x i ) log s * (y|x i ) sm (y|x i ) dy.
It concludes the proof.

Determination of a net for the mean and the variance

In this subsection, we work with a Gaussian density, then β ∈ R q×p and Σ ∈ S ++ q . -Step 1: construction of a net for the variance Let ∈ (0, 1], and δ =

1 √ 2(p 2 q+ 3 4 q) . Let b j = (1 + δ) 1-j 2 A Σ . For 2 ≤ j ≤ N , we have [a Σ , A Σ ] = [b N , b N -1 ] . . . [b 3 , b 2 ],
where N is chosen to recover everything. We want that

a Σ = (1 + δ) 1-N/2 A Σ ⇔ log a Σ A Σ = 1 - N 2 log(1 + δ) ⇔ N = 2 log( A Σ a Σ √ 1 + δ) log(1 + δ) .
We want N to be an integer, then N =

2 log( A Σ a Σ √ 1+δ) log (1+δ)
. We get a net for the variance.

We could let B = diag(b i(1) , . . . , b i(q) ), close to Σ (and deterministic, independent of the values of Σ), where i is a permutation such that b

i(z)+1 ≤ [Σ] z,z ≤ b i(z) for all z ∈ {1, . . . , q}. Remember that b j+1 b j = 1 √ 1+δ . -Step 2:

construction of a net for the mean vectors

We select only the relevant variables detected by the Lasso estimator. For λ ≥ 0,

J λ = (z, j) ∈ {1, . . . , q} × {1, . . . , p}| βLasso z,j (λ) = 0 . Let f = ϕ(.|βx, Σ) ∈ F J .
-Definition of the brackets Define the bracket by the functions l and u: l(y, x) = (1 + δ) -p 2 q-3q/4 ϕ y|ν J x, (1 + δ) -1/4 B [1] ; u(y, x) = (1 + δ) p 2 q+3q/4 ϕ y|ν J x, (1 + δ)B [2] .

We have chosen i such that [B [1] ] z,z ≤ Σ z,z ≤ [B [2] ] z,z for all z ∈ {1, . . . , q}.

We need to define ν such that [l, u] is an -bracket for f . PROCEDURE -Proof that [l, u] is a bracket for f We are looking for a condition on ν J to have f u ≤ 1 and l f ≤ 1. We will use the following lemma to compute these ratios. Lemme 3.6.3. Let ϕ(.|µ 1 , Σ 1 ) and ϕ(.|µ 2 , Σ 2 ) be two Gaussian densities. If their variance matrices are assumed to be diagonal, with

Σ a = diag([Σ a ] 1,1 , . . . , [Σ a ] q,q ) for a ∈ {1, 2}, such that [Σ 2 ] z,z > [Σ 1 ] z,z > 0 for all z ∈ {1, . . . , q}, then, for all y ∈ R q , ϕ(y|µ 1 , Σ 1 ) ϕ(y|µ 2 , Σ 2 ) ≤ q z=1 [Σ 2 ] z,z [Σ 1 ] z,z e 1 2 (µ 1 -µ 2 ) t diag 1 [Σ 2 ] 1,1 -[Σ 1 ] 1,1 ,..., 1 [Σ 2 ]q,q -[Σ 1 ]q,q (µ 1 -µ 2 )
.

For the ratio f u we get:

f (y|x) u(y, x) = 1 (1 + δ) p 2 q+3q/4 ϕ(y|βx, Σ) ϕ(y|ν J x, (1 + δ)B [2] ) ≤ 1 (1 + δ) p 2 q+3q/4 q z=1 b z [Σ] z,z (1 + δ) q/2 × e 1 2 (βx-ν J x) t ((1+δ)B [2] -Σ) -1 (βx-ν J x) ≤(1 + δ) p 2 q-q/4 (1 + δ) q/4 e 1 2 (βx-ν J x) t (δB [2] ) -1 (βx-ν J x) ≤(1 + δ) p 2 q e 1 2δ (βx-ν J x) t [B [2] ] -1 (βx-ν J x) . (3.13)
For the ratio l f we get:

l(y, x) f (y|x) = 1 (1 + δ) p 2 q+3q/4 ϕ(y|ν J x, (1 + δ) -1/4 B [1] ) ϕ(y|βx, Σ) ≤ 1 (1 + δ) p 2 q+3q/4 q z=1 Σ z,z b z (1 + δ) q/8 × e 1 2 (βx-ν J x) t (Σ-B [1] ) -1 (βx-ν J x) ≤(1 + δ) -p 2 q-3q/8 (1 + δ) q/4 × e 1 2 (βx-ν J x) t ((1-(1+δ) -1/4 )B [1] ) -1 (βx-ν J x) ≤(1 + δ) -p 2 q-3q/8 e 1 2(1-(1+δ) -1/4 ) (βx-ν J x) t [B [1] ] -1 (βx-ν J x) . (3.14)
We want to bound the ratios (3.13) and (3.14) by 1. Put c = 5(1-2 -1/4 ) 8 , and develop these calculus. A necessary condition to obtain this bound is

||βx -ν J x|| 2 2 ≤ pqδ 2 (1 -2 -1/4 )A 2 Σ . Indeed, we want (1 + δ) -p 2 q-3q/8 e 1 2(1-(1+δ) -1/4 ) (βx-ν J x) t [B [2] ] -1 (βx-ν J x) ≤ 1 (1 + δ) -p 2 q e 1 2δA Σ (βx-ν J x) t [B [1] ] -1 (βx-ν J x) ≤ 1; which is equivalent to ||βx -ν J x|| 2 2 ≤ p 2 q δ 2 2 A 2 Σ ; ||βx -ν J x|| 2 2 ≤ p 2 q + 3 4 q δ 2 (1 -2 -1/4 )A Σ . As ||βx -ν J x|| 2 2 ≤ p||β -ν J || 2 2 ||x|| ∞ , and x ∈ [0, 1] p , we need to get ||β -ν J || 2 2 ≤ pqδ 2 (1 -2 -1/4 )A 2 Σ to have the wanted bound. Put U := Z ∩ -A β √ cδA Σ , A β √ cδA Σ .
For all (z, j) ∈ J, choose

u z,j = argmin v z,j ∈U β z,j - √ cδA Σ v z,j . (3.15)
Define ν by for all (z, j) ∈ J c , ν z,j = 0; for all (z, j) ∈ J , ν z,j = √ cδA Σ u z,j .

Then, we get a net for the mean vectors. -Proof that d H (l, u) ≤

We will work with the Hellinger distance.

d 2 H (l, u) = 1 2 R q ( √ l - √ u) 2 = 1 2 R q l + u -2 √ lu = 1 2 (1 + δ) -p 2 q-3q/4 + (1 + δ) p 2 q+3q/4 - R q √ ϕ l ϕ u = 1 2 (1 + δ) -p 2 q-3q/4 + (1 + δ) p 2 q+3q/4 - q z=1 2b i(z)+1 b i(z) (1 + δ) 1/2 (1 + δ) -1/8 (1 + δ)b i(z)+1 + (1 + δ) -1/4 b i(z) 2 1/2 * 1.
We have used the following lemma: Lemme 3.6.4. The Hellinger distance of two Gaussian densities with diagonal variance matrices is given by the following expression:

d 2 H (ϕ(.|µ 1 , Σ 1 ), ϕ(.|µ 2 , Σ 2 )) =2 -2 q z=1 2 [Σ 1 ] z,z [Σ 2 ] z,z [Σ 1 ] z,z + [Σ 2 ] z,z 1/2 × exp - 1 4 (µ 1 -µ 2 ) t diag 1 [Σ 1 ] 2 z,z + [Σ 2 ] 2 z,z z∈{1,...,q} (µ 1 -µ 2 ) As b i(z)+1 = (1 + δ) -1/2 b i(z) , we get that 2 (1 + δ) 3/8 b i(z) b i(z)+1 (1 + δ) -1/4 + (1 + δ) 1/2 (1 + δ) = 2 (1 + δ) 5/8 (1 + δ) -1/4 + (1 + δ) 3/2 = 2 (1 + δ) -7/8 + (1 + δ) 7/8 . Then d 2 H (l, u) = 1 2 (1 + δ) -(p 2 q+3q/4) + (1 + δ) p 2 q+3q/4 - 2 (1 + δ) -7/8 + (1 + δ) 7/8 q/2
= cosh((p 2 q + 3q/4) log(1 + δ)) -2 cosh(7/8 log(1 + δ)) -q/2 = cosh((p 2 q + 3q/4) log(1 + δ)) -1 + 1 -2 -q/2 cosh(7/8 log(1 + δ)) -q/2 . We want to apply the Taylor formula to f (x) = cosh(x) -1 to obtain an upper bound, and to g(x) = 1 -2 -q/2 cosh(x) -q/2 . Indeed, there exists c such that, on the good interval, f (x) ≤ cosh(c) x 2 2 and g(x) ≤ q 2 x 2 2 . Then, and because log(1 + δ) ≤ δ,

d 2 H (l, u) ≤ cosh((p 2 q + 3q/4) log(1 + δ)) -2 cosh(7/8 log(1 + δ)) -q/2
≤ (p 2 q + 3q/4) 2 δ 2 cosh(α) + 49 128

≤ 2(p 2 q + 3q/4) 2 δ 2 ≤ 2 ;
where ≥ √ 2(p 2 q + 3 4 q)δ. -Step 3: Upper bound of the number of -brackets for F J .

From Step 1 and Step 2, the family

B (F J ) =                    l(y, x) = (1 + δ) -(p 2 q+3q/4) ϕ(y|ν J x, (1 + δ) -1/4 B [1] ) u(y, x) = (1 + δ) p 2 q+3q/4 ϕ(y|ν J x, (1 + δ)B [2] ) B [a] = diag(b [a] i (1) , . . . , b [a] i(q)
) where i a is a permutation, for a ∈ {1, 2},

with      b [a] i(z) = (1 + δ) 1-ia(z)/2 A Σ for all z ∈ {1, . . . , q} ∀(z, j) ∈ J c , ν z,j = 0 ∀(z, j) ∈ J, ν z,j = √ cδA Σ u z,j                    (3.16
) is an -bracket for F J , for u z,j defined by (3.15). Therefore, an upper bound of the number of -brackets necessary to cover F J is deduced from an upper bound of the cardinal of B (F J ).

|B (F J )| ≤ N l=2 (z,j)∈J 2A β √ cδA Σ ≤ 2A β √ cδA Σ |J| N l=2 1 ≤ 2A β √ cδA Σ |J| (N -1). As N ≤ 2(A Σ /a Σ +1/2) δ , we get |B (F J )| ≤ 2 2A β √ cA Σ |J| A Σ a Σ + 1 2 δ -1-|J| .

Calculus for the function φ

From the Proposition 3.5.3, we obtain, for all > 0, 0

H [.] ( , S B (K,J) , d ⊗n H )d ≤ log(C) + D (K,J) ∧1 0 log 1 d (3.17)
We need to control 0 log 1 d , which is done in Maugis-Rabusseau and Meynet ([MMR12]).

Lemme 3.6.5. For all > 0,

0 log 1 d ≤ √ π + log 1 .
Then, according to (3.17), 3)

0 H [.] ( , S B (K,J) , d ⊗n H )d ≤ log(C) + D (K,J) ( ∧ 1) √ π + log 1 ∧ 1 ≤ D (K,J) log(C) D (K,J) + √ π + log 1 ∧ 1 Nevertheless, log(C) ≤ log(2) + log(K) + K 2 log(2πe) + K|J| log 2A β √ cA Σ + K log A Σ a Σ + 1 2 + (K -1) log(
≤D (K,J) log(2) + log( √ 2πe) + 1 + log(3) + log A Σ a Σ + 1 2 + log 2A β √ cA Σ ≤D (K,J) 1 + log A β A Σ A Σ a Σ + 1 2 + log 2 5/2 3 πe c . Then 0 H [.] ( , S B (K,J) , d ⊗n H )d ≤ D (K,J)   1 + log A β A Σ A Σ A Σ + 1 2 + log 2 5/2 3 πe c + √ π + log 1 ∧ 1 ≤ D (K,J) 1 + log A β A Σ A Σ a Σ + 1 2 + a + log 1 ∧ 1 ≤ D (K,J) B(A β , A Σ , a Σ ) + log 1 ∧ 1 ; with B(A β , A Σ , a Σ ) = 1 + log A β A Σ A Σ a Σ + 1 2 + a;
and a = √ π + log(2 5/2 3 πe c ).

3.6.5 Proof of the Proposition 3.5.5

We are interested in (K,J)∈K×J e -w (K,J) . Considering

w (K,J) = D (K,J) log 4epq (D (K,J) -q 2 ) ∧ pq , PROCEDURE
we could group models by their dimensions to compute this sum. Denote by C D the cardinal of models of dimension D.

K∈N * J∈P({1,...,q}×{1,...,p})

e -D (K,J) log 4epq (D (K,J) -q 2 )∧pq = D≥1 C D e -D log 4epq (D-q 2 )∧pq = pq+q 2 D=1 e -D log 4epq (D-q 2 ) epq D -q 2 D-q 2 + +∞ D=pq+q 2 +1 e -D log 4epq pq 2 pq = pq+q 2 D=1 4 -D epq D -q 2 -q 2 + +∞ D=pq+q 2 +1
e -D(log(4)+1)+pq log(2)

≤ pq+q 2 D=1 2 -D + +∞ D=pq+q 2 +1 2 -D = 2.
3.6.6 Proof of the Lemma 3.5.4

We know that D (K,J) = K -1 + |J|K + Kq 2 . Then,

C D = card{(K, J) ∈ N * × P({1, . . . , q} × {1, . . . , p}), D(K, J) = D} ≤ K∈N * 1≤z≤q 1≤j≤p pq |J| 1 K(|J|+q 2 +1)-1=D ≤ |J|∈N * pq |J| 1 |J|≤pq∧(D-q 2 ) . If pq < D -q 2 , |J|>0 pq |J| 1 |J|≤pq∧(D-q 2 ) = 2 pq .
Otherwise, according to the Proposition 2.5 in Massart, [START_REF] Massart | Concentration inequalities and model selection[END_REF],

|J|>0 pq |J| 1 |J|≤pq∧(D-q 2 ) ≤ f (D -q 2 )
where f (x) = (epq/x) x is an increasing function on {1, . . . , pq}. As pq is an integer, we get the result.

CHAPTER 4. AN ORACLE INEQUALITY FOR THE LASSO-RANK PROCEDURE

In this chapter, we focus on a theoretical result for the Lasso-Rank procedure. Indeed, we get the same kind of results as in the previous chapter, with rank constraint on the estimator. We get a theoretical penalty for which the model selected by a penalized criterion, among the collection, satisfies an oracle inequality. We also illustrate in more details benefits of this procedure with simulated and benchmark dataset including rank structure.

Introduction

The multivariate response regression model

Y = βX +
postulates a linear relationship between Y , the q×n matrix containing q responses for n subjects, and X, the p×n matrix on p predictor variables. The term is an q ×n matrix with independent columns, i ∼ N q (0, Σ) for all i ∈ {1, . . . , n}. The unknown q × p coefficient matrix β needs to be estimate. In a more general way, we could use finite mixture of linear model, which models the relationship between response and predictors, arising from different subpopulations: if the variable Y , conditionally to X, belongs to the cluster k, there exists

β k ∈ R q×p and Σ k ∈ S ++ q such that Y = β k X + , with ∼ N q (0, Σ k ).
If we use this model to deal with high-dimensional data, the number of variables could be quickly much larger than the sample size, because and predictors and response variables could be high-dimensional. To solve this problem, we have to reduce the parameter set dimension. One way to cope the dimension problem is to select relevant variables, in order to reduce the number of unknowns. Indeed, all the information should not be interesting for the clustering, and could even be harmful. In a density estimation way, we could cite Pan and Shen, in [START_REF] Pan | Penalized model-based clustering with application to variable selection[END_REF], who focus on mean variable selection, Meynet and Maugis in [START_REF] Meynet | A sparse variable selection procedure in model-based clustering[END_REF] who extend their procedure in high-dimension, Zhou et al., in [START_REF] Zhou | Penalized model-based clustering with unconstrained covariance matrices[END_REF], who use the Lasso estimator to regularize Gaussian mixture model with general covariance matrices, Sun et al., in [START_REF] Sun | Regularized k-means clustering of highdimensional data and its asymptotic consistency[END_REF], who propose to regularize the k-means algorithm to deal with high-dimensional data, Guo et al, in [START_REF] Guo | Pairwise variable selection for high-dimensional model-based clustering[END_REF], who propose a pairwise variable selection method, among others. In a regression framework, we could use the Lasso estimator, introduced by Tibshirani in [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF], which is a sparse estimator. It penalizes the maximum likelihood estimator by the 1 -norm, which achieves the sparsity, as the 0 -penalty, but leads also to a convex optimization. Because we work with the multivariate linear model, to deal with the matrix structure, we could prefer the group-Lasso estimator, variables grouped by columns, which selects columns rather than coefficients. This estimator was introduced by Zhou and Zhu in [START_REF] Zhou | Group variable selection via a hierarchical lasso and its oracle property[END_REF] in the general case. If we select |J| columns among the p possible, we have to estimate |J|q coefficients rather than pq for A, which could be smaller than nq if |J| is smaller enough. Another estimator which reduces the dimension, is the low rank estimator. Introduced by Izenman in [START_REF] Izenman | Reduced-rank regression for the multivariate linear model[END_REF] in the linear model, and more used the last decades, with among others Bunea et al. in [START_REF] Bunea | Joint variable and rank selection for parsimonious estimation of high-dimensional matrices[END_REF] and Giraud in [START_REF] Giraud | Low rank multivariate regression[END_REF], the regression matrix is estimated by a matrix of rank R, R < p ∧ q. Then, we have to estimate R(p + q -R) coefficients, which could be smaller than nq. In this chapter, we have chosen to mix these two estimators to provide a sparse and low rank estimator in mixture models. This method was introduced by Bunea et al. in [START_REF] Bunea | Joint variable and rank selection for parsimonious estimation of high-dimensional matrices[END_REF], in the case of linear model and known noise covariance matrix. They present different ways, more or less computational, with more or less good results in theory. They get an oracle inequality, which say that, among a model collection, they are able to choose an estimator with true rank and true relevant variables. For this model, Ma and Sun in [START_REF] Ma | Adaptive sparse reduced-rank regression[END_REF] get a minimax lower bound, which precise that they attain nearly optimal rates of convergence adaptively for square Schatten norm losses. In this chapter, we consider finite mixture of K linear models in high-dimension. This model is studied in details by Städler et al. for real response variable in [START_REF] Städler | 1 -penalization for mixture regression models[END_REF], and by Devijver for multivariate response variable in [START_REF] Devijver | Model-based clustering for high-dimensional data[END_REF]. We will estimate β k for all k ∈ {1, . . . , K} by a column sparse and low rank estimator. The Lasso estimator is used to select variables, whereas we refit the estimation by a low rank estimator, restricted on relevant variables. The procedure we propose is based on a modeling that recasts variable selection, rank selection, and clustering problems into a model selection problem. This procedure is developed in [START_REF] Devijver | Model-based clustering for high-dimensional data[END_REF], with methodology, computational issues, simulations and data analysis. In this chapter, we focus on theoretical point of view, and developed simulations and data analysis for the low rank issue. We construct a model collection, with models more or less sparse, and with vector of ranks varying with values more or less small. Among this collection, we have to select a model. We use the slope heuristic, which is a non-asymptotic criterion. In a theoretical way, in this chapter, we get an oracle inequality for the collection constructed by our procedure, which makes a performance comparison between our selected model and the oracle for a specified penalty. This result is an extension of the work of Bunea et al. in [START_REF] Bunea | Joint variable and rank selection for parsimonious estimation of high-dimensional matrices[END_REF], to mixture models and with unknown covariance matrices (Σ k ) 1≤k≤K . They ensure that mixing sparse estimator and low rank matrix could be interesting. Indeed, whereas we have to estimate q × p coefficients in each cluster for the regression matrix, we get only R(|J| + q -R) unknown variables, which could be smaller than the number of observations nq if |J| and R are small. Even if the oracle inequality we get in this chapter is an extension of Bunea et al. result, we use a really different way to prove it. Considering the model collection constructed, we want to select a model as good as possible. For that, we use the slope heuristic, which leads to construct a penalty, proportional to the dimension, and we select the model minimizing the penalized log-likelihood. Theoretically, we construct also a penalty, proportional to the dimension (up to a logarithm term). We provide an oracle inequality which compares, up to a constant, the Jensen-Kullback-Leibler divergence between our model and the true model to the Kullback-Leibler divergence between the oracle and the true model. Then, in estimation term, we do as well as possible. This oracle inequality is deduced from a general model selection theorem for maximum likelihood estimator of Massart developed in [START_REF] Massart | Concentration inequalities and model selection[END_REF]. Controlling the bracketing entropy of models, we could prove the result. Remark that we work in a regression framework, then we rather use an extension of this theorem proved in Cohen and Le Pennec article [START_REF] Cohen | Conditional density estimation by penalized likelihood model selection and applications[END_REF]. As our model collection is random, constructed by the Lasso estimator, we rather use an extension of this theorem proved in [START_REF] Devijver | Finite mixture regression: A sparse variable selection by model selection for clustering[END_REF]. To illustrate this procedure, in a computational way, we validate it on simulated dataset, and benchmark dataset. If the data have a low rank structure, we could easily find it with our methodology. This chapter is organized as follows. In the Section 4.2, we describe the finite mixture regression model used in this procedure, and the main step of the procedure. In the Section 4.3, we present the main result of this chapter, which is an oracle inequality for the procedure proposed. Finally, in Section 4.4, we illustrate the procedure on simulated and benchmark dataset. Proof details of the oracle inequality are given in Appendix.

The model and the model collection

We introduce our procedure of estimation by sparse and low rank matrix in the linear model, in Section 4.2.1, and extend it in Section 4.2.2 for mixture models. We also present the main algorithm used in this context, and we describe the procedure we propose in Section 4.2.4.

Linear model

We consider the observations (x i , y i ) 1≤i≤n which realized random variables (X, Y ), satisfying the linear model

Y = βX + ;
where Y ∈ R q are the responses, X ∈ R p are the regressors, β ∈ R q×p is an unknown matrix, and ∈ R q are random errors, ∼ N q (0, Σ), with Σ ∈ S ++ q a symmetric positive definite matrix. We will work in high-dimension, then q × p could be larger than the number of observations nq. We will construct an estimator which is sparse and low rank for β to cope with the high-dimension issue. Moreover, to reduce the covariance matrix dimension, we compute a diagonal estimator of Σ. The procedure we propose could be explained into two steps. First, we estimate the relevant columns of β thanks to the Lasso estimator, for λ > 0, using the estimator

βLasso (λ) = argmin β∈R q×p ||Y -βX|| 2 2 + λ||β|| 1 ; (4.1)
where ||β|| 1 = p j=1 q z=1 |β z,j |. We assume that the covariance matrix is unknown. For λ > 0, computing the Lasso estimator of βLasso (λ), we could deduce the relevant columns. Restricted to these relevant columns, in the second step of the procedure, we compute a low rank estimator of β, saying of rank at most R. Indeed, as explained in Giraud in [START_REF] Giraud | Low rank multivariate regression[END_REF], we restrict the maximum likelihood estimator to have a rank at most R, keeping only the R biggest singular values in the corresponding decomposition. We get an explicit formula. This two steps procedure leads to an estimator of β which is sparse and has a low rank. We have also reduced the dimension into two ways. We refit the covariance matrix estimator by the maximum likelihood estimator. This estimator is studied in Bunea et al. in [START_REF] Bunea | Joint variable and rank selection for parsimonious estimation of high-dimensional matrices[END_REF], in method 3. Let extend it in mixture models.

Mixture model

We observe n independent couples (x, y) = (x i , y i ) 1≤i≤n of random variables (X, Y ), with Y ∈ R q and X ∈ R p . We will estimate the unknown conditional density s * by a multivariate Gaussian mixture regression model. In our model, if the observation i belongs to the cluster k, we assume that there exists β k ∈ R q×p , and Σ k ∈ S ++ q such that y i = β k x i + i where i ∼ N q (0, Σ k ). Thus, the random response variable Y ∈ R q will be explained by a set of explanatory variables, written X ∈ R p , through a mixture of linear regression-type model. Give more precisions on the assumptions.

-The variables Y i are independent conditionally to X i , for all i ∈ {1, . . . , n} ;

THE MODEL AND THE MODEL COLLECTION

120

-we let

Y i |X i = x i ∼ s K ξ (y|x i )dy, with s K ξ (y|x) = K k=1 π k (2π) q/2 det(Σ k ) 1/2 exp - (y -β k x) t Σ -1 k (y -β k x) 2 (4.2) ξ = (π 1 , . . . , π k , β 1 , . . . , β k , Σ 1 , . . . , Σ K ) ∈ Ξ K = Π K × (R q×p ) K × (S ++ q ) K Π K = (π 1 , . . . , π K ); π k > 0 for k ∈ {1, . . . , K} and K k=1 π k = 1 S ++ q
is the set of symmetric positive definite matrices on R q .

For all k ∈ {1, . . . , K}, β k is the matrix of regression coefficients, and Σ k is the covariance matrix in the mixture component k. The π k s are the mixture proportions. For all k ∈ {1, . . . , K}, for all z ∈ {1, . . . , q}, [

β t k x] z = p j=1 [β k ] z,j
x j is the zth component of the mean of the mixture component k for the conditional density s K ξ (.|x). To detect the relevant variables, we generalize the Lasso estimator defined by (4.1) for mixture models. Indeed, we penalize the empirical contrast by an 1 -penalty on the mean parameters proportional to

||P k β k || 1 = p j=1 q z=1 |(P k β k ) z,j |,
where the Cholesky decomposition P t k P k = Σ -1 k defines P k for all k ∈ {1, . . . , K}. Then, we will consider ξLasso

K (λ) = argmin ξ=(π,β,Σ)∈Ξ K - 1 n n i=1 log(s K ξ (y i |x i )) + λ K k=1 π k ||P k β k || 1 . (4.3)
Remark that the penalty take into account the mixture weight. To reduce the dimension and simplify computations, we will estimate Σ k by a diagonal matrix, thus P k will be also estimated by a diagonal matrix, for all k ∈ {1, . . . , K}.

As in Section 4.2.1, we refit the estimator, restricted on releant columns, with low rank estimator. In Section 4.2.3, we will extend the EM algorithm to compute those two estimators.

Generalized EM algorithm

In a computational way, we will use two generalized EM algorithms, in order to deal with high-dimensional data and get a sparse and low rank estimator. Give some details about those algorithms.

Initially, the EM algorithm was introduced by Dempster et al. in [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF]. It alternates two steps until convergence, an expectation step to cluster data, and a maximization step to update estimation.

In our procedure, we want to know which columns are relevant, then we have to compute (4.3), and we want to refit the estimators by a maximum likelihood under low rank constraint estimator.

From the Lasso estimator (4.3), we could use a generalization of the EM algorithm described in [START_REF] Devijver | Model-based clustering for high-dimensional data[END_REF]. From the estimate of β, we could deduce which columns are relevant.

The second algorithm we use leads to determine β k restricted on relevant columns, for all k ∈ {1, . . . , K}, with rank R k . We alternate two steps, E-step and M-step, until relative convergence of the parameters and of the likelihood. We restrict the dataset to relevant columns, and construct an estimator of size q × |J| rather than q × p, where β k has for rank R k , for all k ∈ {1, . . . , K}. Explain the both steps at iteration (ite) ∈ N * .

CHAPTER 4. AN ORACLE INEQUALITY FOR THE LASSO-RANK PROCEDURE

-E-step: compute for k ∈ {1, . . . , K}, i ∈ {1, . . . , n}, the expected value of the loglikelihood function,

τ i,k = E θ (ite) ([Z i ] k |Y ) = γ k K l=1 γ l where γ l = π (ite) l det Σ (ite) l exp -1 2 y i -β (ite) l x i t (Σ -1 l ) (ite) y i -β (ite) l x i
for l ∈ {1, . . . , K}, and Z i is the component-membership variable for an observation i. -M-step:

-To get estimation in linear model, we assign each observation in its estimated cluster, by the MAP principle. We could compute this thanks to the E-step. Indeed,y i is assigned to the component number argmax k∈{1,...,K} τ i,k .

-Then, we could define βk (ite) = (x t |k x |k ) -1 x t |k y |k , in which x |k and y |k are the sample restriction to the cluster k. We decompose β(ite)

k
in singular values such that β(ite) k = U SV t with S = diag(s 1 , . . . , s q ) and s 1 ≥ s 2 ≥ . . . ≥ s q the singular values. Then, the estimator β(ite)

k is defined by β(ite) k = U S R k V t with S R k = diag(s 1 , . . . , s R k , 0, . . . , 0
). We do it for all k ∈ {1, . . . , K}.

The Lasso-Rank procedure

The procedure we propose, which is particularly interesting in high-dimension, could be decomposed into three main steps. First, we construct a model collection, with models more or less sparse and with more or less components. Then, we refit estimations with the maximum likelihood estimator, under rank constraint. Finally, we select a model with the slope heuristic.

Model collection construction Fix K ∈ K. To get various relevant columns, we construct a data-driven grid of regularization parameters G K , coming from EM algorithm formula. See [START_REF] Devijver | Model-based clustering for high-dimensional data[END_REF] for more details. For each λ ∈ G K , we could compute the Lasso estimator (4.3), and deduce relevant variables set, denoted by J (K,λ) . Then, varying λ ∈ G K , and K ∈ K, we construct J = ∪ K∈K ∪ λ∈G K J (K,λ) the set of relevant variables sets.

Refitting We could also define a low rank estimator ŝ(K,J,R) restricted to relevant variables detected by the Lasso estimator, indewed by J. From this procedure, we construct a model with K clusters, |J| relevant columns and matrix of regression coefficients of ranks R ∈ N K , as described by the next model S (K,J,R) .

S (K,J,R) = {y ∈ R q |x ∈ R p → s (K,J,R) ξ (y|x)} (4.4)
where

s (K,J,R) ξ (y|x) = K k=1 π k det(P k ) (2π) q/2 exp - 1 2 (y -(β R k k ) [J] x) t Σ -1 k (y -(β R k k ) [J] x) ; ξ = (π 1 , . . . , π K , β R 1 1 , . . . , β R K K , Σ 1 , . . . , Σ K ) ∈ Ξ (K,J,R) ; Ξ (K,J,R) = Π K × Ψ (K,J,R) × (S ++ q ) K ; Ψ (K,J,R) = ((β R 1 1 ) [J] , . . . , (β R K K ) [J] ) ∈ (R q×p ) K | for all k ∈ {1, . . . , K}, Rank(β k ) = R k .

ORACLE INEQUALITY 122

Varying K ∈ K ⊂ N * , J ∈ J ⊂ P({1, . . . , p}), and R ∈ R ⊂ {1, . . . , |J| ∧ q} K , we get a model collection with various number of components, relevant columns and matrix of regression coefficients.

Model selection Among this model collection, during the last step, a model has to be selected. As in Maugis and Michel in [START_REF] Maugis | A non asymptotic penalized criterion for Gaussian mixture model selection[END_REF], and in Maugis and Meynet in [START_REF] Meynet | A sparse variable selection procedure in model-based clustering[END_REF], among others, a non asymptotic penalized criterion is used. The slope heuristic was introduced by Birgé and Massart in [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF], and developed in practice by Baudry et al. in [START_REF] Baudry | Slope heuristics: overview and implementation[END_REF] with the Capushe package. To use it in our context, we have to extend theoretical result to determine the penalty shape in the high-dimensional context, with a random model collection, in a regression framework. The main result is described in the next section, whereas proof details are given in Appendix.

Oracle inequality

In a theoretical point of view, we want to ensure that the slope heuristic which penalizes the loglikelihood with the model dimension will select a good model. We follow the approach developed by Massart in [START_REF] Massart | Concentration inequalities and model selection[END_REF] which consists of defining a non-asymptotic penalized criterion, leading to an oracle inequality. In the context of regression, Cohen and Le Pennec, in [START_REF] Cohen | Conditional density estimation by penalized likelihood model selection and applications[END_REF], and Devijver in [START_REF] Devijver | Finite mixture regression: A sparse variable selection by model selection for clustering[END_REF], propose a general model selection theorem for maximum likelihood estimation. The result we get is a theoretical penalty, for which the model selected is as good as the best one, according to the Kullback-Leibler loss.

Framework and model collection

Among the model collection constructed by the procedure developed in Section 4.2.2, with various rank and various sparsity, we want to select an estimator which is close to the best one.

The oracle is by definition the model belonging to the collection which minimizes the contrast with the true model. In practice, we do not have access to the true model, then we could not know the oracle. Nevertheless, the goal of the model selection step of our procedure is to be nearest to the oracle. In this section, we present an oracle inequality, which means that if we have penalized the log-likelihood in a good way, we will select a model which is as good as the oracle, according to the Kullback-Leibler loss. We consider the model collection defined by (4.4).

Because we work in high-dimension, p could be big, and it will be time-consuming to test all the parts of {1, . . . , p}. We construct a sub-collection denoted by J L , which is constructed by the Lasso estimator, which is also random. This step is explained in more details in [START_REF] Devijver | Model-based clustering for high-dimensional data[END_REF]. Moreover, to get the oracle inequality, we assume that the parameters are bounded:

S B (K,J,R) = s (K,J,R) ξ ∈ S (K,J,R) ξ = (π, β, Σ), for all k ∈ {1, . . . , K}, (4.5)
Σ k = diag([Σ k ] 1,1 , . . . , [Σ k ] q,q ), for all z ∈ {1, . . . , q}, a Σ ≤ [Σ k ] z,z ≤ A Σ , for all k ∈ {1, . . . , K}, β R k k = R k r=1 [σ k ] r [u t k ] .,r [v k ] r,.
,

for all r ∈ {1, . . . , R k }, [σ k ] r < A σ .

CHAPTER 4. AN ORACLE INEQUALITY FOR THE LASSO-RANK PROCEDURE

Remark that it is the singular value decomposition of β k is the singular value decomposition, with ([σ k ] r ) 1≤r≤R k the singular values, and u k and v k unit vectors, for k ∈ {1, . . . , K}.

We also assume that covariates belong to an hypercube: without restrictions, we could assume that X ∈ [0, 1] p . Fixing K the possible number of components, J L the relevant columns set constructed by the Lasso estimator, and R the possible vector of ranks, we get a model collection We also define the Jensen-Kullback-Leibler divergence, first introduced in Cohen and Le Pennec in [START_REF] Cohen | Conditional density estimation by penalized likelihood model selection and applications[END_REF], by

K∈K J∈J L R∈R S B (K,J,R) . ( 4
JKL ρ (s, t) = 1 ρ KL(s, (1 -ρ)s + ρt)
for ρ ∈ (0, 1), s and t two densities. The tensorized one is defined by

JKL ⊗n ρ (s, t) = E 1 n n i=1 JKL ρ (s(.|x i ), t(.|x i )) .
Note that those divergences are not metrics, they do not satisfy the triangular inequality and they are not symmetric, but they are also wildly used in statistics to compare two densities.

Oracle inequality

Let state the main theorem.

Theorem 4.3.1. Assume that we observe (x i , y i ) 1≤i≤n ∈ ([0, 1] p ×R q ) n with unknown conditional density s * . Let M = K × J × R and M L = K × J L × R, where J L is constructed by the Lasso estimator. For (K, J, R) ∈ M, let s(K,J,R) ∈ S B (K,J,R) , where S B (K,J,R) is defined by (4.5), such that, for δ KL > 0, KL ⊗n (s * , s(K,J,R) ) ≤ inf

t∈S B (K,J,R) KL ⊗n (s * , t) + δ KL n
and there exists τ > 0 such that s(K,J,R) ≥ e -τ s * . (4.9)

For (K, J, R) ∈ M, consider the rank constraint log-likelihood minimizer ŝ(K,J,R) in S (K,J,R) , satisfying

ŝ(K,J,R) = argmin s (K,J,R) ξ ∈S B (K,J,R) - 1 n n i=1 log s (K,J,R) ξ (y i |x i ) .
Denote by D (K,J,R) the dimension of the model S B (K,J,R) . Let pen : M → R + defined by, for all (K, J, R) ∈ M,

pen(K, J, R) ≥ κ D (K,J,R) n 2B 2 (A σ , A Σ , a Σ ) -log D (K,J,R) n B 2 (A σ , A Σ , a Σ ) ∧ 1 + (1 ∨ τ ) log 4epq D (K,J,R)-q 2 ∧ pq + K k=1 R k
where κ > 0 is an absolute constant, and B(A σ , A Σ , a Σ ) is an absolute constant, depending on parameter bounds. Then, the estimator ŝ( K, Ĵ, R) , with

( K, Ĵ, R) = argmin (K,J,R)∈M L - 1 n n i=1 log(ŝ (K,J,R) (y i |x i )) + pen(K, J, R) , satisfies E JKL ⊗n ρ (s * , ŝ( K, Ĵ, R) ) ≤ C E inf (K,J,R)∈M L inf t∈S (K,J,R)
KL ⊗n (s * , t) + pen(K, J, R) + (1 ∨ τ ) n (4.10) some absolute positive constant C.

The proof of the Theorem 4.3.1 is given in Section 4.5. Note that condition (4.9) leads to control the random model collection. The mixture parameters are bounded in order to construct brackets over S (K,J,R) , and thus to upper bound the entropy. The inequality (4.10) not exactly an oracle inequality, since the Jensen-Kullback-Leibler risk is upper bounded by the Kullback-Leibler divergence. Note that we use the Jensen-Kullback-Leibler divergence rather than the Kullback-Leibler divergence, because it is bounded. This boundedness turns out to be crucial to control the loss of the penalized maximum likelihood estimator under mild assumptions on the complexity of the model and on parameters.

Because we are looking at a random sub-collection of models which is small enough, our estimator ŝ(K,J,R) is attainable in practice. Moreover, it is a non-asymptotic result, which allows us to study cases for which p increases with n.

We could compare our inequality with the bound of Bunea et al, in [START_REF] Bunea | Joint variable and rank selection for parsimonious estimation of high-dimensional matrices[END_REF], who computed a procedure similar to ours, for a linear model. According to consistent group selection for the group-Lasso estimator, they get adaptivity of the estimator to an optimal rate, and their estimators perform the bias variance trade-off among all reduced rank estimators. Nevertheless, their results are obtained according to some assumptions, for instance the mutual coherence on X t X, which postulates that the off-diagonal elements have to be small. Some assumptions on the design are required, whereas our result just needs to deal with bounded parameters and bounded covariates.

Numerical studies

We will illustrate our procedure with simulated and benchmark datasets, to highlight the advantages of our method. We adapt the simulations part of Bunea et al. article, [START_REF] Bunea | Joint variable and rank selection for parsimonious estimation of high-dimensional matrices[END_REF]. Indeed, we work in the same way, to get a sparse and low rank estimator. Nevertheless, we add a mixture framework to be consistent with our clustering method, and to have more flexibility.

Simulations

To illustrate our procedure, we use simulations adapted from the article of Bunea [START_REF] Bunea | Joint variable and rank selection for parsimonious estimation of high-dimensional matrices[END_REF], extended to mixture models. The design matrix X has independent and identically distributed rows X i , distributed from a multivariate Gaussian N q (0, Σ) with Σ = ρI, ρ > 0. We consider a mixture of 2 components. According to the cluster, the coefficient matrix β k has the form

β k = b k B 0 b k B 1 0 0
for k ∈ {1, 2}, with B 0 a J × R k matrix and B 1 a R k × q matrix. All entries in B 0 and B 1 are independent and identically distributed according to N (0, 1). The noise matrix has independent N (0, 1) entries. Let i denotes its ith row. The proportion vector π is defined by π = 1 2 , 1 2 , i.e. all clusters have the same probability. Each row Y i in Y is then generated as, if the observation i belongs to the cluster k, Y i = β k X i + i , for all i ∈ {1, . . . , n}. This setup contains many noise features, but the relevant ones lie in a low-dimensional subspace. We report two settings:

p > n: n = 50, |J| = 6, p = 100, q = 10, R = (3, 3), ρ = 0.1, b = (3, -3).

p < n: n = 200, |J| = 6, p = 10, q = 10, R = (3, 3), ρ = 0.01, b = (3, -3). The current setups show that variable selection, without taking the rank information into consideration, may be suboptimal, even if the correlations between predictors are low. Each model are simulated 20 times, and Table 4.1 summarizes our findings. We evaluate the prediction accuracy of each estimator β by the Kullback-Leibler divergence (KL) using a test sample at each run. We also report the median rank estimate (denoted by R) over all runs, rates of non included true variables (denoted by M for misses) and the rates of incorrectly included variables (F A for false actives). Ideally, we are looking for a model with low KL, low M and low F A. Over 20 simulations, we get almost exact clustering, and the Kullback-Leibler divergence between the model we construct and the true model is really low. In case of high-dimensional data, when p is larger than n, we get also good results. We find the good structure, selecting the relevant variables (our model will have false relevant variables, but no missed variables), and selecting the true ranks. We could remark that false relevant variables have low values. Comparing to another procedure which will not reduce the rank, we will perform the dimension reduction.

KL

Illustration on a real dataset

In this section, we apply our procedure to real data set. The Norwegian paper quality data were obtained from a controlled experiment that was carried out at a paper factory in Norway to uncover the effect of three control variables X 1 , X 2 , X 3 on the quality of the paper which was measured by 13 response variables. Each of the control variables X i takes values in {-1, 0, 1}.

To account for possible interactions and nonlinear effects, second order terms were added to the set of predictors, yielding

X 1 , X 2 , X 3 , X 2 1 , X 2 2 , X 2 3 , X 1 X 2 , X 1 X 3 , X 2 X 3
, and the intercept term. There were 29 observations with no missing values made on all response and predictor variables. The Box Behnken design of the experiment and the resulting data are described in Aldrin [START_REF] Aldrin | Moderate projection pursuit regression for multivariate response data[END_REF] and Izenman [START_REF] Izenman | Reduced-rank regression for the multivariate linear model[END_REF]. Moreover, Bunea et al. in [BSW12] also study this dataset. We always center the responses and the predictors. The dataset clearly indicates that dimension reduction is possible, making it a typical application for reduced rank regression methods. Moreover, our method will exhibit different clusters among this sample. We construct a model collection varying the number of clusters in K = {2, . . . , 5}. We select a model with 2 clusters. We select all variables except X 1 X 2 and X 2 X 3 , which is consistent with comments of Bunea et al. In the two clusters, we get two mean matrices, with ranks equal to 2 and 4. One cluster describes the mean comportment (with rank equals to 2), whereas the other cluster contains values more different.

Appendices

In those appendices, we present the details of the proof of the Theorem 4.3.1. It derives from a general model selection theorem, stated in Section 4.5.1, and proved in the Chapter 3. Then, the proof of the Theorem 4.3.1 could be summarized by satisfying assumptions H m , Sep m and K described in Section 4.5.1.

A general oracle inequality for model selection

Model selection appears with the AIC criterion and BIC criterion. A non-asymptotic theory was developed by Birgé and Massart in [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF]. With some assumptions detailed here, we get an oracle inequality for the maximum likelihood estimator among a model collection. Cohen and Le Pennec, in [START_REF] Cohen | Conditional density estimation by penalized likelihood model selection and applications[END_REF], generalize this theorem in regression framework. We have to use a generalization of this theorem detailed in [START_REF] Devijver | Finite mixture regression: A sparse variable selection by model selection for clustering[END_REF] because we consider a random collection of models. Let state the main theorem. We consider a model collection (S m ) m∈M , indexed by M. Let (X, Y ) ∈ X × Y. Begin by describe the assumptions. First, we impose a structural assumption. It is a bracketing entropy condition on the model with respect to the tensorized Hellinger divergence m with m the unique solution of

(d ⊗n H ) 2 (s, t) = E 1 n n i=1 d 2
1 φ m ( ) = √ n . (4.11)
Remark that the model complexity depends on the bracketing entropies not of the global models S m but of the ones of smaller localized sets. It is a weaker assumption.

For technical reasons, a separability assumption is also required.

Assumption Sep m . There exists a countable subset S m of S m and a set Y m with λ(Y \Y m ) = 0, for λ the Lebesgue measure, such that for every t ∈ S m , there exists a sequence (t l ) l≥1 of elements of S m such that for every x and every y ∈ Y m , log(t l (y|x)) goes to log(t(y|x)) as l goes to infinity.

According to this assumption, we could work with a countable subset. We also need an information theory type assumption on our model collection. We assume the existence of a Kraft-type inequality for the collection.

Assumption K. There is a family (w m ) m∈M of non-negative numbers such that m∈M e -wm ≤ Ω < +∞.

Then, we could write our main global theorem to get an oracle inequality in regression framework, with a random collection of models.

Theorem 4.5.1. Assume we observe

(x i , y i ) 1≤i≤n ∈ ([0, 1] p × R q ) n with
+ C 2 (1 ∨ τ ) Ω 2 n + η + η n .
Remark 4.5.2. We get that, among a random model collection, we are able to choose a model which is as good as the oracle, up to a constant C 1 , and some additive terms being around 1/n. This result is non-asymptotic, and gives a theoretical penalty to select this model.

Remark 4.5.3. The proof of this theorem is detailed in [START_REF] Devijver | Finite mixture regression: A sparse variable selection by model selection for clustering[END_REF]. Nevertheless, we could give the main ideas to understand the assumptions. From assumptions H m and Sep m , we could use maximal inequalities which lead to, except on a set of probability less than e -w m -w , for all w, a control of the ratio of the centered empirical process of log(ŝ m ) over the Hellinger distance between s * and ŝm , this control being around 1/n. Thanks to Bernstein inequality, satisfied according to the inequality (4.12), and thanks to the assumption K, we get the oracle inequality. Now, to prove Theorem 4.3.1, we have to satisfy assumptions H m and K, assumption Sep m being true for our conditional densities.

Assumption H m

Decomposition

As done in Cohen and Le Pennec [START_REF] Cohen | Conditional density estimation by penalized likelihood model selection and applications[END_REF], we could decompose the entropy by

H [.] ( , S B (K,J,R) , d ⊗n H ) ≤ H [.] ( , Π K , d ⊗n H ) + K k=1 H [.] ( , F (J,R k ) , d ⊗n H ) (4.15)
where

S B (K,J,R) =        y ∈ R q |x ∈ R p → s (K,J,R) ξ (y|x) = K k=1 π k ϕ y|(β R k k ) [J] x, Σ k ξ = π 1 , . . . , π K , (β R 1 1 ) [J] , . . . , (β R K K ) [J] , Σ 1 , . . . , Σ K ∈ Ξ (K,J,R) Ξ (K,J,R) = Π K × Ψ(K,J,R) × ((a Σ , A Σ ] q ) K        Ψ (K,J,R) = ((β R 1 1 ) [J] , . . . , (β R K K ) [J] ) ∈ (R q×p ) K |Rank(β k ) = R k ; Ψ(K,J,R) = ((β R 1 1 ) [J] , . . . , (β R K K ) [J] ) ∈ Ψ (K,J,R) for all k ∈ {1, . . . , K}, β R k k = R k r=1 σ r u t r v r , with σ r < A σ for all r ∈ {1, . . . , R k } Π K = (π 1 , . . . , π K ) ∈ (0, 1) K ; K k=1 π k = 1 ; F (J,R) = ϕ(.|(β R ) [J] X, Σ); β R = R r=1 σ r u t r v r , with σ r < A σ , Σ = diag(Σ 1,1 , . . . , Σ q,q ) ∈ [a Σ , A Σ ] q      ϕ the Gaussian density.
For the proportions, it is known that (see Wasserman and Genovese in [START_REF] Genovese | Rates of convergence for the Gaussian mixture sieve[END_REF])

H [.] ( , Π K , d ⊗n H ) ≤ log K(2πe) K/2 3 K-1 .
Look after the Gaussian entropy.

For the Gaussian

We want to bound the integrated entropy. For that, first we have to construct some brackets to recover S m . Fix f ∈ S m . We are looking for functions l and u such that l ≤ f ≤ u. Because f is a Gaussian, l and u are dilatations of Gaussian. We then have to determine the mean, the variance and the dilatation coefficient of l and u. We need the both following lemmas to construct these brackets.

Lemme 4.5.4. Let ϕ(.|µ 1 , Σ 1 ) and ϕ(.|µ 2 , Σ 2 ) be two Gaussian densities. If their variance matrices are assumed to be diagonal, with

Σ a = diag([Σ a ] 1,1 , . . . , [Σ a ] q,q ) for a ∈ {1, 2}, such that [Σ 2 ] z,z > [Σ 1 ] z,z > 0 for all z ∈ {1, . . . , q}, then, for all x ∈ R q , ϕ(x|µ 1 , Σ 1 ) ϕ(x|µ 2 , Σ 2 ) ≤ q z=1 [Σ 2 ] z,z [Σ 1 ] z,z e 1 2 (µ 1 -µ 2 ) t diag 1 [Σ 2 ] 1,1 -[Σ 1 ] 1,1 ,..., 1 [Σ 2 ]q,q -[Σ 1 ]q,q (µ 1 -µ 2 )
. Lemme 4.5.5. The Hellinger distance of two Gaussian densities with diagonal variance matrices 4.5. APPENDICES 130 is given by the following expression:

d 2 H (ϕ(.|µ 1 , Σ 1 ), ϕ(.|µ 2 , Σ 2 )) =2 -2 q z=1 2 [Σ 1 ] z,z [Σ 2 ] z,z [Σ 1 ] z,z + [Σ 2 ] z,z 1/2 × exp - 1 4 (µ 1 -µ 2 ) t diag 1 [Σ 1 ] z,z + [Σ 2 ] z,z z∈{1,...,q} (µ 1 -µ 2 )
To get an -bracket for the densities, we have to construct a δ-net for the variance and the mean, δ to be specify later.

-Step 1: construction of a net for the variance Let ∈ (0, 1], and δ

= √ 2q . Let b j = (1 + δ) 1-j 2 A Σ . For 2 ≤ j ≤ N , we have [a Σ , A Σ ] = [b N , b N -1 ] . . . [b 3 , b 2 ],
where N is chosen to recover everything. We want that

a Σ = (1 + δ) 1-N/2 A Σ ⇔ log a Σ A Σ = 1 - N 2 log(1 + δ) ⇔ N = 2 log( A Σ a Σ √ 1 + δ) log(1 + δ) .
We want N to be an integer, then N = 2 log(

A Σ a Σ √ 1+δ) log(1+δ)
. We get a regular net for the variance. We could let B = diag(b i(1) , . . . , b i(q) ), close to Σ (and deterministic, independent of the values of Σ), where i is a permutation such that b i(z)+1 ≤ Σ z,z ≤ b i(z) for all z ∈ {1, . . . , q}. -Step 2: construction of a net for the mean vectors

We use the singular decomposition of β, β = R r=1 σ r u t r v r , with (σ r ) 1≤r≤R the singular values, and (u r ) 1≤r≤R and (v r ) 1≤r≤R unit vectors. Those vectors are also bounded. We are looking for l and u such that d H (l, u) ≤ , and l ≤ f ≤ u. We will use a dilatation of a Gaussian to construct such an -bracket of ϕ. We let

l(x, y) = (1 + δ) -(p 2 qR+3q/4) ϕ(y|ν J,R x, (1 + δ) -1/4 B 1 ) u(x, y) = (1 + δ) p 2 qR+3q/4 ϕ(y|ν J,R x, (1 + δ)B 2 )
where B 1 and B 2 are constructed such that, for all z ∈ {1, . . . , q},

[B 1 ] z,z ≤ Σ z,z ≤ [B 2 ] z,z (see step 1).
The means ν J,R ∈ R q×p will be specified later. Just remark that J is the set of the relevant columns, and R the rank of ν J,R : we will decompose ν J,R = R r=1 σr ũt r ṽr , ũ ∈ R |J|×R , and ṽ ∈ R q×R . We get l(x, y) ≤ f (y|x) ≤ u(x, y)

if we have ||βx -ν J,R x|| 2 2 ≤ p 2 qR δ 2 2 a 2 Σ (1 -2 -1/4 ). PROCEDURE Remark that ||βx -ν J,R x|| 2 2 ≤ p||β -ν J,R || 2 2 ||x|| ∞ We need then ||β -ν J,R || 2 2 ≤ pqR δ 2 2 a 2 Σ (1 -2 -1/4 ) (4.16)
According to [START_REF] Devijver | Finite mixture regression: A sparse variable selection by model selection for clustering[END_REF], d H (l, u) ≤ 2(p 2 qR + 3q/4) 2 δ 2 , then, with

δ = √ 2(pqR + 3/4q)
we get the wanted bound. Now, explain how to construct ν J,R to get (4.16).

||β -ν J,R || 2 2 = p j=1 q z=1 R r=1
σ r u r,j v r,z -σr ũr,j ṽr,z

2 = p j=1 q z=1 R r=1 |σ r -σr ||u j,r v z,r | -σr |ũ r,j -u r,j ||ṽ z,r | -σr u r,j |v r,z -ṽr,z | 2 ≤ p j=1 q z=1 R r=1 |σ r -σr | + A σ |ũ r,j -u r,j | + A σ |v r,z -ṽr,z | 2 ≤ 2pqR max r |σ r -σr | 2 + A σ max r,j |ũ r,j -u r,j | 2 + A σ max r,z |ṽ r,z -v r,z | 2 We need ||β -ν J,R || 2 2 ≤ pqR δ 2 2 a 2 Σ (1 -2 -1/4
). If we choose σr , ũr,j and ṽr,z such that

|σ r -σr | ≤ δ √ 12 a Σ 1 -2 -1/4 , |u r,j -ũr,j | ≤ δ √ 12A σ a Σ 1 -2 -1/4 , |v r,z -ṽr,z | ≤ δ √ 12A σ a Σ 1 -2 -1/4 ,
then it works.

APPENDICES 132

To get this, we let, for . the floor function,

S = Z ∩   0,     A σ δ √ 12 a Σ 1 -2 -1/4       σr = argmin ς∈S σ r - δ √ 12 a Σ 1 -2 -1/4 ς , U = Z ∩   0,     A σ δ √ 12Aσ a Σ 1 -2 -1/4       ũr,j = argmin µ∈U u r,j - δ √ 12A σ a Σ 1 -2 -1/4 µ , V = Z ∩   0,     A σ δ √ 12Aσ a Σ 1 -2 -1/4       ṽz,r = argmin ν∈V v z,r - δ √ 12A σ a Σ 1 -2 -1/4 ν
for all r ∈ {1, . . . , R}, j ∈ {1, . . . , p}, z ∈ {1, . . . , q}.

Remark that we just need to determine the vectors ((ũ r,j ) 1≤j≤J-r ) 1≤r≤R and ((ṽ z,r ) 1≤z≤q-r ) 1≤r≤R because those vectors are unit. Then, we let for all j ∈ J c , for all z ∈ {1, . . . , p}, (ν J,R ) z,j = 0 for all j ∈ J, for all z ∈ {1, . . . , p}, (ν J,R ) z,j = R r=1 σr ũr,j ṽz,r -Step 3: Upper bound of the number of -brackets for F (J,R)

We have defined our bracket. Let c = 1-2 -1/4

12

. We want to control the entropy.

|B (F (J,R) )| ≤ N l=2 A σ δa Σ √ c R A 2 σ δa Σ √ c R( 2J-R-1 2 + 2q-R-1 2 ) ≤(N -1) A 2 σ δa Σ √ c R(J+q-R) A -R σ ≤C(a Σ , A Σ , A σ , J, R)δ -D (J,R) -1 with C(a Σ , A Σ , A σ , J, R) = 2 A Σ a Σ + 1 2 A 2 σ a Σ √ c R(J+q-R) A -R σ .
For the mixture Begin by computing the bracketing entropy: according to (4.15),

H [.] ( , S B (K,J,R) , d H ) ≤ log C 1 D (K,J,R)

CHAPTER 4. AN ORACLE INEQUALITY FOR THE LASSO-RANK PROCEDURE

where

C = 2 K K(2πe) K/2 3 K-1 A Σ a Σ + 1 2 K A σ √ 12 a 2 Σ 1 -2 -1/4 D (K,J,R) A -K k=1 R k σ (4.17) and D (K,J,R) = K k=1 R k (|J| + q -R k ). We have to determine φ (K,J,R) such that 0 H [.] ( , S B (K,J,R) (s (K,J,R) , ), d ⊗n H )d ≤ φ (K,J,R) ( ). ( 4

.18)

Let compute the integral.

0 H [.] ( , S B (K,J,R) (s (K,J,R) , ), d ⊗n H )d ≤ log(C) + D (K,J,R) 0 log 1 d ≤ D (K,J,R) √ π + log(C) D (K,J,R) + log 1 ∧ 1
with, according to (4.17),

log(C) =K log(2) + K 2 log(2πe) + (K -1) log(3) + K log A Σ a Σ + 1 2 + D (K,J,R) log A 2 σ √ 12 a Σ 1 -2 -1/4 + log(K) + K k=1 R k log( 1 A σ ) ≤D (K,J,R) log(2) + log(2πe) + log 3 + 1 + log A 2 σ √ 12 a Σ 1 -2 -1/4 + D (K,J,R) log A Σ a Σ + 1 2 ≤D (K,J,R) log 12 √ 12πe 1 -2 -1/4 + log A Σ a Σ + 1 2 A 2 σ a Σ Then, 0 H [.] ( , S B (K,J,R) (s (K,J,R) , ), d ⊗n H )d ≤ D (K,J,R) 2 + log A Σ a Σ + 1 2 A 2 σ a Σ + log 1 ∧ 1
Consequently, by putting

B = 2 + log A Σ a Σ + 1 2 A 2 σ a Σ ;
we get that the function φ (K,J,R) defined on R * + by

φ (K,J,R) ( ) = D (K,J,R) B + log 1 ∧ 1
4.5. APPENDICES 134 satisfies (4.18). Besides, φ (K,J,R) is nondecreasing and → φ (K,J,R) ( )/ is non-increasing, then φ (K,J,R) is convenient. Finally, we need to find an upper bound of * satisfying

φ (K,J,R) ( * ) = √ n 2 * . Consider * such that φ (K,J,R) ( * ) = 2 * . This is equivalent to solve * = D (K,J,R) n B + log 1 * ∧ 1
and then we could choose

2 * ≤ D (K,J,R) n 2B 2 + log 1 1 ∧ D (K,J,R) n B 2 . 4.5.3 Assumption K Let w (K,J,R) = D(K,J) log 4epq ( D(K,J) -q 2 )∧pq + k∈{1,...,K} R k , where D(K,J) = K(1 + |J|).
Then, we could compute the sum

(K,J,R) e -w (K,J,R) ≤ K≥1      1≤|J|≤pq e -D(K,J) log 4epq ( D(K,J) -q 2 )∧pq     R≥1 e -R   K    ≤ K≥1     1≤|J|≤pq e -D(K,J) log 4epq ( D(K,J) -q 2 )∧pq   1 K   ≤ 2
The last inequality is computed in Proposition 4.5 in [Dev14b] by 2. Then, (K,J,R) e -w (K,J,R) ≤ 2.

Introduction

New metering infrastructures as smart meters provide new and potentially massive informations about individual (household, small and medium enterprise) consumption. As an example, in France, ERDF (Electricite Reseau Distribution de France the French manager of the public electricity distribution network) deployed 250 000 smart meters, covering a rural and an urban territory and providing half-hourly household energy used each day. ERDF plans to install 35 millions of them over the French territory by the end of 2020 and exploiting such an amount of data is an exciting but challenging task (see http://www.erdf.fr/Linky). Many applications coming from individual data analysis can be found in the literature. The first and most popular one is load profiling. Understanding consumers time of use, seasonal patterns and the different features that drive their consumption is a fundamental task for electricity providers to design their offer and more generally for marketing studies (see e.g. [START_REF] Irwin | Statistical electricity demand modelling from consumer billing data[END_REF]). Energy agencies and states can also benefit from profiling for efficiency programs and improve recommendation policies. Customer segmentation based on load classification is a natural approach for that and [START_REF] Zhou | A review of electric load classification in smart grid environment[END_REF] proposes a nice review of the most popular methods, concluding that classification for smart grids is a hard task due to the complexity, massiveness, high dimension and heterogeneity of the data. Another problem pointed out is the dynamic structure of smart meters data and particularly the issue of portfolio variations (losses and gains of customers), the update of a previous classification when new customers arrive or the clustering of a new customer with very few observations on its load. In [START_REF] Kwac | Household energy consumption segmentation using hourly data[END_REF], the authors propose a segmentation algorithm based on K-means to uncover shape dictionaries that help to summarize information and cluster a large population of 250 000 households in California. However, the proposed solution exploits a quite long historic of data of at least 1 year. Recently, other important questions were raised by smart meters and the new possibility to send potentially complex signal to consumers (incentive payments, time varying prices...) and demand response program tailoring attracts a lot of attention (see [US 06], [H + 13]). Local optimization of electricity production and real time management of individual demand thus play an important role in the smart grid landscape. It induces a need for local electricity load forecasting at different levels of the grid and favorites bottom-up approaches based on a two stage process. First, it consists in building classes in a population such that each class could be sufficiently well forecast but corresponds to different load shapes or reacts differently to exogenous variables like temperature or prices (see e.g. [START_REF] Labeeuw | Potential of active demand reduction with residential wet appliances: A case study for belgium. Smart Grid[END_REF] in the context of demand response). The second stage consists in aggregating forecasts to forecast the total or any subtotal of the population consumption. For example, identify and forecast the consumption of a sub-population reactive to an incentive is an important need to optimize a demand response program. Surprisingly, few papers consider the problem of clustering individual consumption for forecasting and specially for forecasting at a disaggregated level (e. g. in each class). In [START_REF] Alzate | Improved electricity load forecasting via kernel spectral clustering of smart meters[END_REF], clustering procedures are compared according to the forecasting performances of their corresponding bottom-up forecasts of the total consumption of 6 000 residential customers and small-to-medium enterprises in Ireland. Even if they achieve nice performances at the end, the proposed clustering methods are quite independent to the VAR model used for forecasting. In [START_REF] Misiti | Optimized clusters for disaggregated electricity load forecasting[END_REF], a clustering algorithm is proposed that couples hierarchical clustering and multi-linear regression models to improve the forecast of the total consumption of a French industrial subset. They obtain a real forecasting gain but need a sufficiently long dataset (2-3 years) and the algorithm is computationally intensive. We propose here a new methodology based on high dimensional regression models. Our main contribution is that we focus on uncovering classes corresponding to different regression models. As a consequence, these classes could then be exploited for profiling as well as forecasting in each classes or for bottom-up forecasts in an unified view. More precisely, we consider regression models where Y d = X d β + ε d is typically an individual -high dimension-load curve for day d and X d could be alternatively Y d-1 or any other exogenous covariates. We consider a real dataset of Irish individual consumers of 4 225 meters, each with 48 halfhourly meter reads per day over 1 year: from 1st January 2010 up to 31st December 2010. These data have already been studied in [START_REF] Alzate | Improved electricity load forecasting via kernel spectral clustering of smart meters[END_REF] and [START_REF] Chaouch | Clustering-based improvement of nonparametric functional time series forecasting: Application to intra-day household-level load curves[END_REF] and we refer to those papers for a presentation of the data. For computational and time reasons, we draw a random sample of around 500 residential consumers among the 90% closest to the mean, to demonstrate the feasibility of our methods. We show that, considering only 2 days of consumption, we obtain physically interpretable clusters of consumers. According to the Fig. 5.1, deal with individual consumption curves is an hard task, because of the high variability.

Method

We propose to use model-based clustering and adopt the model selection paradigm. Indeed, we consider the model collection of conditional mixture densities,

S = s (K,J) ξ , K ∈ K, J ∈ J ,
where K denotes the number of clusters, J the set of relevant variables for clustering, and K and J being respectively the set of possible values of K and J.

The basic model we propose to use is a finite mixture regression of K multivariate Gaussian densities (see [START_REF] Städler | Comments on 1penalization for mixture regression models[END_REF] for a recent and fruitful reference), the conditional density being, for x ∈ R p , y ∈ R q , ϕ denoting the Gaussian density,

s (K,J) ξ (y|x) = K k=1 π k ϕ(β [J] k x, Σ k )
Such a model can be interpreted and used from two different viewpoints. First, from a clustering perspective, given the estimation ξ of the parameters ξ = (π, β, Σ), we could deduce data clustering from the Maximum A Posteriori principle: for each observation i, we compute the posterior probability τ i,k ( ξ) of each cluster k from the estimation ξ, and we assign the observation i to the cluster ki = argmax k∈{1,...,K} τ i,k ( ξ). Proportions of each cluster are estimated by π.

TYPICAL WORKFLOW USING THE EXAMPLE OF THE AGGREGATED CONSUMPTION 138

Second, in each cluster, the corresponding model is meaningful and its interpretation allows to understand the relationship between variables Y and X since it is of the form

Y = Xβ k + ,
the noise intensity being measured by Σ k . Parameters are estimated from the Lasso-MLE procedure, which is described in details in [START_REF] Devijver | Model-based clustering for high-dimensional data[END_REF], and theoretically approved in [START_REF] Devijver | Finite mixture regression: A sparse variable selection by model selection for clustering[END_REF]. To overcome the high-dimension issue, we use the Lasso estimator on the regression parameters and we restrict the covariance matrix to be diagonal. To avoid shrinkage, we estimate parameters by Maximum Likelihood Estimator on relevant variables selected by the Lasso estimator. Rather than selecting a regularization parameter, we present this issue at a model selection problem, considering a grid of regularization parameters. Indices of relevant variables for this grid of regularization parameters are denoted by J . Since we also have to estimate the number of components, we compute those models for different number of components, belonging to K. In this paper, K = {1, . . . , 8}. Among this collection, we could focus on a few models which seem interesting for clustering, depending on which characteristics we want to highlight. We propose to use the slope heuristics to extract potentially interesting models. The selected model minimizes the log-likelihood penalized by 2κD (K,J) /n, where D m denotes the dimension of the model m, and where κ is constructed from a completely data-driven procedure. In practice, we use the Capushe package, see [START_REF] Baudry | Slope heuristics: overview and implementation[END_REF].

In addition to this family of models, we need to have powerful tools to translate curves into variables. Rather than dealing with the discretization of the load consumption, we project it onto a functional basis to take into account the functional structure. Since we are interested in not only representing the curves into a functional basis, but also to benefit from a timescale interpretation of coefficients, we propose to use wavelet basis, see [START_REF] Mallat | A wavelet tour of signal processing[END_REF] for a theoretical approach, and [MMOP07] for a practical purpose. To simplify our presentation, we will focus on the Haar basis.

5.3 Typical workflow using the example of the aggregated consumption

General framework

The goal is to cluster electricity consumers using a regression mixture model. We will consider the consumption of the eve for the regressors, to explain the consumption of the day. Consider the daily consumption, where we observe 48 points. We project the signal onto the Haar basis at level 4. The signal could be decomposed in approximation, denoted by A 4 , and several details, denoted by D 4 , D 3 , D 2 , and D 1 . We illustrate it in Fig. 5.2, where in addition the decomposition in sum of orthogonal signals on the left, one can find a colored representation of the corresponding wavelet coefficients in the time scale plane. For an automatic denoising, we remove details of level 1 and 2, which correspond to high-frequency components. Two centerings will be considered:

-preprocessing 1: before projecting, we center each signal individually.

-preprocessing 2: we consider details coefficients of level 4 and 3. Here, we remove also a low-frequency approximation. Depending on the preprocessing, we will get different clusterings We observe the load consumption of n residentials over a year, denoted by (z i,t ) 1≤i≤n,t∈T . We consider -Z t = n i=1 z i,t the aggregated signal, The first one consider (Z d , Z d+1 ) 1≤d≤338 over time, and the results are easy to interpret. We take this opportunity to develop in details the steps of the method we propose from the model to the clusters via model visualization and interpretation. In the second one, we want to cluster consumers on mean days. Working with mean days leads to some stability. The last one is the most difficult, since we consider individuals curves (z i,d 0 , z i,d 0 +1 ) 1≤i≤n and we classify these individuals for the days (d 0 , d 0 + 1).

Cluster days on the residential synchronous curve

In this Section, we focus on the residential synchronous (Z t ) t∈T . We will illustrate our procedure step by step, and highlight some features of data. The whole analysis will be done for the preprocessing 2.

Model selection

Our procedure leads to a model collection, with various number of components and various sparsities. Let us explain how to select some interesting models, thanks to the slope heuristic. We define (K(κ), J(κ)) = argmin

(K,J) -γ n (ŝ (K,J) ) + 2κD (K,J) /n ,
where γ n is the log-likelihood function and ŝ(K,J) is the log-likelihood minimizer among the collection S (K,J) . We consider the step function κ → D K(κ),J(κ) , κ being the abscissa which leads to the biggest dimension jump. We select the model ( K, Ĵ) = (K(2κ), J(2κ)). To improve that, we could consider the points (D (K,J) , -γ n (s (K,J) ) + 2κD (K,J) /n) (K,J) , and select some models minimizing this criterion. According to Figs. 5.3 and 5.4, we could consider some κ which seem to create big jumps, and several models which seem to minimize the penalized log-likelihood.

Model visualization

Thanks to the model-based clustering, we have constructed a model in each cluster. Then, we could understand differences between clusters from β and Σ estimations. We represent it with an image, each coefficient being represented by a pixel. As we consider the linear model Y = Xβ k for each cluster, rows correspond to regressors coefficients and columns to response coefficients. Diagonal coefficients will explain the main part. The Figs. 5.5 and 5.6 explain the image construction, whereas we compute it for the model selected by the previous step in Fig. To highlight differences between clusters, we also plot β1 -β2 . First, we remark that β1 and β2 are sparse, thanks to the Lasso estimator. Moreover, the main difference between β1 and β2 is row 4, columns 1, 2 and 6. We could say that the procedure uses, depending on cluster, more or less the first coefficient of D 3 of X to describe coefficients 1 and 2 of D 3 and coefficient 3 of D 4 of Y . The Fig. 5.11 enlightens those differences between clusters.

5.7.

We represent the covariance matrix in Fig. 5.8. Because we estimate it by a diagonal matrix in each cluster, we just display the diagonal coefficients. We keep the same scale for all the clusters, to highlight which clusters are noisier.

Figure 5.8: For the model selected, we represent Σ in each cluster

Model-based clustering

We could compute the posterior probability for each observation. In Fig. 5.9, we represent it by boxplots. Closer there are to 1, more different are the clusters. In the present case, the two clusters are well defined and the clustering problem is quite easy, but see for example Fig. 5.13, in a different clustering issue, which presents a model with affectations less well separated.

Clustering

Now, we are able to try to interpret each cluster. In Fig. 5.10, we represent the mean curves for each cluster. We can also use functional descriptive statistics (see [START_REF] Shang | rainbow: An R Package for Visualizing Functional Time Series[END_REF]). Because clusters are done on the reliance between a day and its eve, we represent the both days.

Discussion

According to the preprocessing 2, we cluster weekdays and weekend days. The same procedure done with the preprocessing 1 shows the temperature influence. We construct four clusters, two of them being weekend days, and the two others are weekdays, differences made according to the temperature. In Fig. 5.11, we summarize clusters by the mean curves for this second model. According to Table 5.1, difference between both preprocessing is the temperature influence: when we center curves before projecting, we translate the whole curves, but when we remove the low frequency approximation, we skip the main features. Depending on the goal, each preprocessing could be interesting.

Clustering consumers

Another important approach considered in this paper is to cluster consumers. Before dealing with their daily consumption, in Section 5.4.2, which is an hard problem because of the irregularity of signals, we cluster consumers on mean days in Section 5.4.1. We focus here on Saturday/Sunday, for which there are different interesting clusters, see Fig. 5.12. Remark that we cannot summarize a cluster to its mean because of the high variability. The main differences between those two clusters are on differences between lunch time and afternoon, and on the Sunday morning. Notice that the big variability over the two days is not explained by our model, for which the variability is small, explaining the noise for the reliance between a day and its eve. On Sunday/Monday, we get also three different clusters. Even if we identify differences on the shape, the main difference is still on the mean consumption. On Friday/Saturday, we see differences between people who have the same consumptions, and people who have a really different comportment. However, because the selected model is again with one component, we think that consider the mean over days of each consumer cancel interesting effects, as the temperature and seasonal

Cluster consumers on mean days

Cluster consumers on individual curves

One major objective of this work is individual curves clustering. As already pointed in the introduction, this is a very challenging task that has various applications for smart grid management, going from demand response programs to energy reduction recommendations or household segmentation. We consider the complex situation where an electricity provider as access to very few recent measurements -2 days here-of each individual customers but need to classify them anyway. That could happen in a competitive electricity market where customers can change their electricity provider at any time without providing their past consumption curves. First, we focus on two successive days of electricity consumption measurements -Tuesday and Wednesday in winter: January 5th and 6th 2010-for our 487 residential customers. Note that we choose week days in winter following our experience on electricity data, as those days often bring important information about residential electricity usage (electrical heating, intra-day cycle, tariff effect...).

Selected mixture models

We apply the model-based clustering approach presented in Section 5.2 for preprocessing 1 and obtain two models minimizing the penalized log-likelihood corresponding to 2 and 5 clusters. Although these two classifications are based on auto-regression mixture model, we are able to analyze and interpret clusters in terms of consumption profiles and provide bellow a physical interpretation for the classes. In Fig. 5.13, we plot the proportions in each cluster for both models constructed by our procedure. The first remark is that this issue is harder than the one in Section 5.3.2. Nevertheless, even if there are a lot of outliers, for the model M1, a lot of affectations are well-separated. It is obviously less clear with the model M2, with 5 components. In Fig. 5.14, we plot the regression matrix to highlight differences between clusters. Indeed, those two matrices are different, for example more variables are needed to describe the cluster 2.

Analyzing clusters using classical profiling features

We first represent the main features of the cluster centres (the mean of all individual curves in a cluster). Fig. 5.15 represents daily mean consumptions of these centres along the year. We clearly see that the two classifications separate customers that have different mean level of consumption (small, -middle-and big residential consumers) and different ratio winter/summer consumption probably due to the amount of electrical heating among house usage. Let recall that the model based clustering is done on centered data and that the mean level discrimination is HIGH-DIMENSIONAL REGRESSION MIXTURE MODELS. This analysis is confirmed in Fig. 5.16 where we represent the daily mean consumptions of the cluster centres in function of the daily mean temperature for the two classifications. Points correspond to the mean consumption of one day and smooth curves are obtained with P-spline regression. We observe that for all classes, the temperature effect due to electrical heating starts at around 12˚C and that the different clusters have various heating profiles. The 2 clusters classification profiles confirm the observation of Fig. 5.15 that the population is divided into small and big customers with electrical heating. Concerning the 5 clusters classification, we clearly see on the small customer profile (purple points/curves) an inflexion at around 0˚C -this inflexion is also observed in the small customer cluster of the 2 clusters classificationcorresponding to e.g. an auxiliary heating device effect or at least an increase of consumption 5.4. CLUSTERING CONSUMERS 146 of the house for very low temperature. This is also what distinguishes the 2 middle customers classes (blue and green points/curves). The two big customers' clusters have similar heating profile, except that the green cluster correspond to higher electrical heating usage. q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q -5 0 Daily Consumption (kW) q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q -5 0 Daily Temperature (C°) q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q Figure 5.16: Daily mean consumptions of the cluster centres in function of the daily mean temperature for 2 (on the left) and 5 clusters (on the right)

Another interesting observation concern the weekly and daily profiles of the centres. We represent on Fig. 5.17 an average (over time) week of consumption for each centre of the two classifications. For the 2 clusters classification, we see again the difference in average consumption between the big customer cluster profile and the small customer one. They have similar shapes but the difference day/night and peak loads (at around 8h, 13h and 18h for weekdays), are more marked. For the 5 clusters curves, even if the weekly profiles are quite similar (no major difference in the week days/week ends profiles in each clusters), the daily shapes exhibit more differences. The day/night ratio could be very different as well as small variation along the day, probably related to different tariff options (see [Com11a] for a description of the tariffs).

CHAPTER 5. CLUSTERING ELECTRICITY CONSUMERS USING HIGH-DIMENSIONAL REGRESSION MIXTURE MODELS.

Cross analysis using survey data

To enrich this analysis, we also consider extra information providing in a survey realized by the Irish Commission for Energy Regulation. We summarize this large amount of information into 10 variables: ResidentialTariffallocation corresponds to the tariff option (see is a well known and tested non-parametric classification method that has the advantage to be quite automatic and easy to calibrate. In addition, it provides a nice summary of the previous covariate in terms of their importance for classification. On the Fig. 5.18 we represent the out of bag error of the random forest classifiers in function of the number of trees (one major parameter of the random forest classifier) for the two clusterings. That corresponds to a good estimate of what could be the classification error on a independent dataset. We have observed that, choosing a sufficiently large number of trees for the forest (300), the classification error rate attains 40% in the 2 clusters case and 75% in the 5 classes case which has to be compared to a random classifier who has respectively a 50% and 80% error rate. That means that the 10 previous covariates provide few but some information about the clusters.

Using model on one-day shifted data

To highlight advantages of our procedure, we compare prediction for the consumption of Thursday, 7th January, 2010. Indeed, even if the method is not designed for forecasting purposes, we want to show that model-based clustering is an interesting tool also for prediction. We will compare linear models, estimated on couples Tuesday, 5th January and Wednesday, 6th January, and we will predict Thursday from Wednesday. This is suggested by the clustering get in Section 5.3.2, showing that transitions between weekdays are similar. We then compare the following models. First, the most common is the linear model, without clustering. The second model is the first constructed by our procedure, described before, with 2 components. Moreover, we could use the clustering get by the models constructed by our procedure, but estimate parameters without variable selection, using full linear model in each component. We restrict here our study to one model to narrow the analysis, but everything is also computable with the models with 5 clusters.

For each consumer i, for each prediction procedure, we compute two prediction errors: the RMSE on Thursday prediction, and the RMSE of Wednesday prediction. Remind that RMSE, for a consumer i, is defined by We remark that if RMSE are almost the same for the three different models, the one estimated by our procedure leads to smaller median and interquartile range. For the three considered models, the median of the RMSE on Wednesday prediction (learning sample) and the RMSE on Thursday prediction (test sample) are close to each other, which means that the clustering remains good, even for one-day shifted data, of course as long as we remain in the class of working days, according to Section 5.3.2. To highlight this remark, we also compute our procedure on couple Wednesday/Thursday. Then, we select three different models, and involved clusterings are quite similar to clusterings given by models in Section 5.4.2. HIGH-DIMENSIONAL REGRESSION MIXTURE MODELS.

RM SE(i) =

Remarks on similar analyses

Alternatively, we make the same analysis on two successive weekdays of electricity consumption measurement in summer. We obtain three models, corresponding to 2, 3 and 5 clusters respectively. We compute, as in the Subsection 5.4.2, daily mean consumptions of the cluster center along the year, and in function of the daily mean temperature. The main difference is about the inflexion at around 0˚C. Because clustering is done for summer days, we do not distinguish cold effects. Moreover there are no cooling effects. We could remark again that clusterings are hierarchical, but different from those get in the winter study, as we expected.

q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q -5 0 5 10 15 20 Daily Consumption (kW) q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q Figure 5.20: Daily mean consumptions of the cluster centres in function of the daily mean temperature for 5 clusters, clustering done by observing Thursday and Wednesday in summer

We also study two successive weekend days of electricity consumption, in winter and in summer. We recognize different clusters, depending on behavior of consumers. We work with Friday/Saturday couples. The main thing we observe in summer is a cluster with no-consumption, consumers who leave their home. It could be useful to predict the Sunday consumption, but no more general for other weekend.

Discussion and conclusion

Massive information about individual (household, small and medium enterprise) consumption are now provided with new metering technologies and smart grids. Two major exploitations of individual data are load profiling and forecasting at different scales on the grid. Customer segmentation based on load classification is a natural approach for that and is a prolific way of research. We propose here a new methodology based on high-dimensional regression models.

DISCUSSION AND CONCLUSION 150

The novelty of our approach is that we focus on uncovering clusters corresponding to different regression models that could then be exploited for profiling as well as forecasting. We focus on profiling and show how, exploiting few temporal measurements of 500 residential customers consumption, we can derive informative clusters. We provide some encouraging elements about how to exploit these models and clusters for bottom up forecasting.

Figure 1 -

 1 Figure 1 -Exemple de données simulées où les classes en régression se comprennent par la réalisation graphique. On a ici envie de construire 3 classes, représentées par les différents symboles : * pour la classe 1, ♦ pour la classe 2, pour la classe 3. Les données sont en dimension un, (X, Y ) ∈ R × R. Les observations x sont issues d'une loi Gaussienne centrée réduite, et y est simulée suivant un mélange de Gaussiennes de moyennes β k x et de variance 0.25, où β = [-1, 0.1, 3].

Figure 3 -Figure 4 -

 34 Figure3-Illustration de l'heuristique des pentes : on estime κ par κ le plus grand saut de dimension. On sélectionne alors le modèle qui minimise la log-vraisemblance pénalisée par 2κ.

Figure 1 2 Figure 1 . 2 :

 1212 Figure 1.1: For one simulation, number of false relevant (in red color) and true relevant (in blue color) variables generated by the Lasso, by varying the regularization parameter λ in the grid G 2

3

 3

Figure 1 . 3 :

 13 Figure 1.3: For one simulation, slope graph obtain by our Lasso-Rank procedure. For large dimensions, we observe a linear part

For the model 2 ,

 2 according to the Figure (1.5) for the Kullback-Leibler divergence, and Figure (1.6) for the ARI, the Kullback-Leibler divergence is small and the ARI is close to 1, except for the MLE procedure. Boxplots are still readable with those values, but it is important to highlight that variable selection is needed, even in a model with reasonable dimension. The model collections are then well constructed. The model 3 is more difficult, because the noise is higher. That is why results, summarized in Figures (1.

  7) and (1.8), are not as good as for the model 2. Nevertheless, our procedures lead to the best ARI, and the Kullback-Leibler divergences are close to the one of the oracle. We could make the same remarks for the model 4. In this study, means are closer, according to the noise. Results are summarized in Figures (1.

  9) and (1.10). The model 5 is in high-dimension. Models selected by the BIC criterion are bad, in comparison with models selected by our procedures, or oracles. They are bad for estimation, according to the Kullback-Leibler divergence boxplots in Figure (1.11), but also for clustering, according to Figure (1.

  12

Figure 1 Figure 1 Figure 1 . 7 :Figure 1 . 8 : 3 1Figure 1 Figure 1 Figure 1 Figure 1

 11171831111 Figure1.5: Boxplots of the Kullback-Leibler divergence between the true model and the one selected by the procedure over the 20 simulations, for the Lasso-MLE procedure (LMLE), the Lasso-Rank procedure (LR), the oracle (Oracle), the BIC estimator (BIC) for the model 2

Figure 1 .

 1 Figure 1.13: Boxplots of the ARI over the 20 simulations, for the Lasso-MLE procedure (LMLE), the Lasso-Rank procedure (LR), the oracle (Oracle), the BIC estimator (Bic) and the MLE (MLE)

Figure 1 .

 1 Figure 1.14: Plot of the 70-sample of halfhour load consumption, on the two days

Figure 1 .

 1 Figure 1.16: Summarized results for the model 1. The graph on the left is a candidate for representing each cluster, constructed by the mean of reconstructed spectrum over an a posteriori probability greater than 0.6 On the right side, we present the boxplot of the fat values in each class, for observations with an a posteriori probability greater than 0.6.

Figure 1 .

 1 Figure1.17: Summarized results for the model 2. The graph on the left is a candidate for representing each cluster, constructed by the mean of reconstructed spectrum over an a posteriori probability greater than 0.6 On the right side, we present the boxplot of the fat values in each class, for observations with an a posteriori probability greater than 0.6.

. 7 )

 7 Denote by fm := -log ŝm s ξ 0 and fm := -log sm s ξ 0 . Let η ≥ 0 and fix m ∈ N * . We define the set M (m) by

CHAPTER 2 .

 2 AN 1 -ORACLE INEQUALITY FOR THE LASSO IN FINITE MIXTURE OF MULTIVARIATE GAUSSIAN REGRESSION MODELS Lemme 2.5.1. Let M n > 0. Let T = max i∈{1,...,n} max z∈{1,...,q}

  m(y i |x i )) + pen( m) ≤ inf m∈N *m (y i |x i )) + pen(m) + η; and M (m) = {m ∈ N * |P n (-log(ŝ m )) + pen(m ) ≤ P n (-log(ŝ m )) + pen(m) + η} . By definition, m ∈ M (m). Because for all m ∈ N * , for all m ∈ M (m), 1 -

Figure 3 . 1 :

 31 Figure 3.1: Boxplot of the Kullback-Leibler divergence between the true model and the one constructed by each procedure, the Lasso-MLE procedure and the Lasso estimator

CHAPTER 3 .Figure 3 . 2 :Figure 3 . 3 :

 33233 Figure 3.2: Boxplots of the Kullback-Leibler divergence between the true model and the one selected by the procedure over the 20 simulations, for the Lasso-MLE procedure (LMLE), the oracle (Oracle), the BIC estimator (BIC)

Theorem 3 .

 3 5.1. Assume we observe (x i , y i ) 1≤i≤n with unknown conditional density s * . Let the model collection S = (S m ) m∈M be at most countable collection of conditional density sets. Assume Assumption K holds, while assumptions H m and Sep m hold for every m ∈ M. Let δ KL > 0, and sm ∈ S m such that KL ⊗n (s * , sm ) ≤ inf t∈Sm KL ⊗n (s * , t) + δ KL n ; and let τ > 0 such that sm ≥ e -τ s * . (3.6) Introduce (S m ) m∈ M some random subcollection of (S m ) m∈M . Consider the collection (ŝ m ) m∈ M of η-log-likelihood minimizer in S m , satisfying, for all m ∈ M, n i=1 -log(ŝ m (y i |x i )) ≤ inf sm∈Sm n i=1 -log(s m (y i |x i )) + η.

  It only remains to sum up the tail bounds over all the possible values of m ∈ M and m ∈ M(m) by taking the union of the different sets of probability less than e -(w+wm+w m ) , m∈M m ∈M(m) e -(w+wm+w m ) ≤ e -w (m,m )∈M×M e -(wm+w m ) = e -w m∈M e -wm 2

Figure 5 . 1 :

 51 Figure 5.1: Load consumption of a sample of 5 consumers over a week in winter

CHAPTER 5 .Figure 5 . 2 : 2 -

 5522 Figure 5.2: Projection of a load consumption for one day into Haar basis, level 4. By construction, we get s = A 4 + D 4 + D 3 + D 2 + D 1 . On the left side, the signal is considered with reconstruction of dataset, the dotted being preprocessing 1 and the dotted-dashed being the preprocessing 2

5. 3 .Figure 5 . 4 :

 354 Figure 5.3: We select the model m using the slope heuristic

A 4 D 4 D 3 A 4 D 4 D 3 Figure 5 4 D 3 D 4 D 3 Figure 5

 443443543435 Figure 5.5: Representation of the regression matrix β k for the preprocessing 1.

Figure 5 . 7 :

 57 Figure 5.7: For the selected model, we represent β in each cluster. Absolute values of coefficients are represented by different colormaps, white for 0. Each color represents a cluster

Figure 5 . 9 :

 59 Figure 5.9: Assignment boxplots per cluster

Figure 5

 5 Figure 5.10: Clustering representation. Each curve is the mean in each cluster

Figure 5

 5 Figure 5.11: Clustering representation. Each curve is the mean in each cluster

Figure 5 .

 5 Figure 5.12: Saturday and Sunday load consumption in each cluster.

Figure 5 .

 5 Figure 5.13: Proportions in each cluster for models constructed by our procedure

Figure 5 Figure 5

 55 Figure 5.14: Regression matrix in each cluster for the model with 2 clusters

Figure 5

 5 Figure 5.18: Out of bag error of the random forest classifiers in function of the number of trees

Figure 5 .

 5 Figure 5.19: RMSE on Thursday prediction for each procedure over all consumers

Figure 5

 5 Figure 5.21: Daily mean consumptions of the cluster centres along the year for 3 clusters, clusering done on weekend observation

  Introduction sélection de variables, construire J λ correspond à sélectionner un paramètre de régularisation λ ≥ 0 ; dans les cas de ré-estimation par faible rang, on doit sélectionner le vecteur des rangs dans chaque composante. Dans chacun de ces trois cas, différentes méthodes existent dans la littérature. Pour le paramètre λ de régularisation du Lasso, on peut citer par exemple le livre de van de Geer et Bühlmann,[START_REF] Bühlmann | Statistics for High-Dimensional Data: Methods, Theory and Applications[END_REF], où une valeur λ proportionnelle à log(p)/n est considérée comme optimale. Pour le choix du nombre de classes K et du vecteur de rangs R, de nombreux auteurs se ramènent à un problème de sélection de modèles. Dans cette thèse, nous allons voir le problème de sélection de paramètres comme un problème de sélection de modèles, en construisant une collection de modèles, avec plus ou moins de classes, et plus ou moins de coefficients actifs. Il restera à choisir un modèle parmi cette collection. En toute généralité, notons S = (S

m ) m∈M la collection de modèles que l'on considère, indicée par M. Il est à noter que, contrairement à l'idée que l'on pourrait s'en faire, avoir une collection de modèles trop grande peut porter préjudice, par exemple en sélectionnant des estimateurs non consistants (voir Bahadur,

[START_REF] Bahadur | Examples of inconsistency of maximum likelihood estimates[END_REF]

) ou sous-optimaux (voir Birgé et Massart,

[START_REF] Birgé | Rates of convergence for minimum contrast estimators[END_REF]

). C'est ce qu'on appelle le paradigme du choix de modèle. On considère une fonction de contraste, notée γ, telle que s * = argmin t∈S E(γ(t)). La fonction de perte associée, notée l, est définie par

pour tout t ∈ S, l(s * , t) = E(γ(t)) -E(γ(s * )).

On définit aussi le contraste empirique γ n par pour tout t ∈ S, γ n (t) = 1 n n i=1 γ(t, x i , y i ) pour un échantillon (x, y) = (x i , y i ) 1≤i≤n . Pour le modèle m, on considère ŝm la densité qui minimise le contraste empirique γ n . C'est cette densité que l'on va utiliser pour représenter ce modèle. Par exemple, on peut prendre la log-vraisemblance comme contraste, et la divergence de Kullback-Leibler comme fonction de perte. Le but de la sélection de modèles est de sélectionner le meilleur estimateur parmi la collection (ŝ m ) m∈M . Le meilleur estimateur peut être défini comme l'estimateur qui minimise le risque avec la vraie densité notée s * . Cet estimateur, et le modèle correspondant, seront appelés dans cette thèse oracle (voir Donoho et Johnstone par exemple, [DJ94]). On note

  'est une des meilleures approximations de s * dans S m ). Or, pour minimiser le biais, il faut un modèle complexe, qui colle de très près aux données ; et pour minimiser la variance, il ne faut pas considérer des modèles trop complexes, pour ne pas surapprendre des données.

Une méthode pour rendre compte de cette remarque est de considérer le contraste empirique L n pénalisé par la dimension : on va pénaliser les modèles trop complexes, qui suraprennent de nos données. Soit pen : M → R + une pénalité à construire ; on va sélectionner

m = argmin m∈M {-γ n (ŝ m ) + pen(m)} .

Akaike et Schwarz ont introduit cette méthode pour l'estimation avec la vraisemblance, voir respectivement

[START_REF] Akaike | A new look at the statistical model identification[END_REF]

et

[START_REF] Schwarz | Estimating the Dimension of a Model[END_REF]

. Ils proposaient les critères désormais classiques AIC et BIC, où la pénalité vaut respectivement

pen AIC (m) = D m ; pen BIC (m) = log(n)D m 2 ;

où D m est la dimension du modèle m, et n est la taille de l'échantillon considéré. Ces critères sont fortement utilisés aujourd'hui. Il est à noter qu'ils sont basés sur des approximations asymptotiques (AIC sur le théorème de Wilks, et BIC sur une approche bayésienne), et on peut se méfier de leur comportement en non asymptotique. Ils supposent de plus, en toute rigueur, une collection de modèles fixée.

Mallows, au même moment, dans

[START_REF] Mallows | Some comments on Cp[END_REF]

, a étudié cette méthode dans le cadre de la régression linéaire. Il a obtenu

pen M allows (m) = 2D m σ 2 n

où σ 2 est la variance des erreurs, supposée connue. Birgé et Massart, dans

[START_REF] Birgé | Gaussian model selection[END_REF]

, ont introduit l'heuristique des pentes, qui est une méthodologie non asymptotique pour sélectionner un modèle parmi une collection de modèles. Décrivons les idées de cette heuristique. On cherche une pénalité proche de m ∈ M → l(s * , ŝm )γ n (ŝ m ). Comme on ne connaît pas s * , on va essayer d'approcher cette quantité :

  Introductionempirique. Si on choisit pen(m) = vm , on va choisir un modèle qui limite le biais mais pas la variance : on va sélectionner un modèle trop complexe. Cette pénalité est minimale : si on pose pen(m) = κv m , si κ < 1 on va choisir un modèle trop complexe, et si κ > 1, la dimension du modèle sera plus raisonnable. En fait, la pénalité optimale est le double de la pénalité minimale. Comme vm est la version empirique de v m , v m ≈ vm . Comme ∆ n (s m ) est d'espérance nulle, on peut contrôler ses fluctuations. On a donc envie de choisir pen(m) = 2v m . Ainsi, on peut trouver, sur un jeu de données, si on utilise l'heuristique des pentes, la pénalité qui nous permettra de sélectionner un modèle : soit on cherche le plus grand saut de dimension, soit on regarde la pente asymptotique de γ n (s m ), ce qui nous donne la pénalité minimale, et il suffit de la multiplier par deux pour obtenir la pénalité optimale. Les figures 3 et 4 illustrent ces idées.

  fonction, et un ensemble de détails. Pour une étude générale des ondelettes, on peut citer Daubechies,[START_REF] Daubechies | Ten Lectures on Wavelets[END_REF], ou Mallat,[START_REF] Mallat | A wavelet tour of signal processing[END_REF]. Il est à noter que, par données fonctionnelles, dans notre cadre de modèles de mélange en régression, on entend soit des régresseurs fonctionnels et une réponse vectorielle (typiquement, l'analyse spectrophotométrique de données, comme le jeu de données classiques Tecator où la proportion de gras d'un morceau de viande est exprimée en fonction de la courbe de spectrophotométrie), soit des régresseurs vectoriels et une réponse fonctionnelle, soit des régresseurs et une réponse fonctionnels (comme la régression de la consommation électrique d'un jour sur la veille).

On considère un échantillon de signaux (f i ) 1≤i≤n observés sur une grille de temps {t 1 , . . . , t T }. On peut considérer l'échantillon (f i (t j )) 1≤i≤n 1≤j≤p pour faire notre analyse (ces observations sont soit les régresseurs, soit la réponse, soit les deux suivant la nature des données), mais on peut aussi considérer les projections de l'échantillon sur une base orthonormée B = {α j } j∈N * . Dans ce cas, il existe (b j ) j∈N * tels que, pour tout f , pour tout t,

  R) with bounded parameters

	Dimensions
	p	number of regressors
	q	response size
	K	number of components
	n	sample size
	D m	dimension of the model S m
	D (K,J)

Table 1

 1

	.1.

  Table 1.2: Mean number {TR, FR} of true relevant and false relevant variables over the 20 simulations for each procedure, for models 2, 3 and 4. The standard deviations are put into parenthesis In Table (1.

			Model 2	Model 3		Model 4
	Procedure	TR	FR	TR	FR	TR	FR
	Lasso-MLE	8(0) 2.2(6.9)	8(0)	4.3(28.8) 8(0) 2(13, 5)
	Lasso-Rank	8(0)	24(0)	8(0)	24(0)	8(0)	24(0)
	Oracle	8(0) 1.5(3.3)	7.8(0.2) 2.2(11.7) 8(0) 0.8(2.6)
	BIC estimator 8(0) 2.6(15.8) 7.8(0.2) 5.7(64.8) 8(0) 2.6(11.8)

Table 1 .

 1 Results are summarized in Table1.3. 3: Mean absolute percentage of error of the predicted value, for each model, for the learning sample Thus, we work on the test sample. We use the response and the regressors to know the a posteriori of each observation. Then, using our models, we could compute the predicted fat values from the spectrometric curve, as before according to two ways, mixing or choosing the classes.

		Linear model in the class Mixing estimation
		with higher probability
	Model 1	0.200	0.198
	Model 2	0.055	0.056
		Linear model in the class Mixing estimation
		with higher probability
	Model 1	0.22196	0.21926
	Model 2	0.20492	0.20662

|(ŷ i -y i )/y i |.

Table 1 .

 1

4: Mean absolute percentage of error of the predicted value, for each model, for the test sample

  possible number of components, J a collection of subsets of {1, . . . , q} × {1, . . . , p}, and R the set of vectors of size K ∈ K with ranks values for each mean matrix. We could compute the MLE under the rank constraint thanks to an EM algorithm. Indeed, we could constrain the estimation of Φ k , for all k, to have a rank equal to R k , in keeping only the R k largest singular values. More details are given in Section 1.7.1. It leads to an estimator of the mean with row sparsity and low rank for each model. Functional datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 1.5.1 Functional regression model . . . . . . . . . . . . . . . . . . . . . . . . . 55 1.5.2 Two procedures to deal with functional datasets . . . . . . . . . . . . . 55 Projection onto a wavelet basis . . . . . . . . . . . . . . . . . . . . . . . 55 Our procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 1.5.3 Numerical experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 Simulated functional data . . . . . . . . . . . . . . . . . . . . . . . . . . 57 Electricity dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 Tecator dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 1.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 1.7 Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	Chapter 2
	An 1 -oracle inequality for the Lasso in
	finite mixture of multivariate Gaussian
	regression models

Contents 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 1.2 Gaussian mixture regression models . . . . . . . . . . . . . . . . . . . 42 1.2.1 Gaussian mixture regression . . . . . . . . . . . . . . . . . . . . . . . . . 42 1.2.2 Clustering with Gaussian mixture regression . . . . . . . . . . . . . . . 43 1.2.3 EM algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 1.2.4 The model collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 1.3 Two procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 1.3.1 Lasso-MLE procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 1.3.2 Lasso-Rank procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 1.4 Illustrative example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 1.4.1 The model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 1.4.2 Sparsity and model selection . . . . . . . . . . . . . . . . . . . . . . . . 49 1.4.3 Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 1.5 . . 61 1.7.1 EM algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 EM algorithm for the Lasso estimator . . . . . . . . . . . . . . . . . . . 61 EM algorithm for the rank procedure . . . . . . . . . . . . . . . . . . . 64 1.7.2 Group-Lasso MLE and Group-Lasso Rank procedures . . . . . . . . . . 64 Context -definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 Group-Lasso-MLE procedure . . . . . . . . . . . . . . . . . . . . . . . . 65

  Let M n > 0 and κ ≥ 36. Let C Mn = max 1 aπ

	.8)
	2.4.3 Proof of the Theorem 2.4.1 thanks to the Propositions 2.4.2 and 2.4.3

  get an oracle inequality for our clustering procedure, we have to use a general model selection theorem. Because the model collection constructed by our procedure is random, because of the Lasso estimator which select variables randomly, we have to generalize Cohen and Le Pennec Theorem. Begin by some general model selection theory. Before state the general theorem, begin by talking about the assumptions. We work here in a more general context, (X, Y ) ∈ X × Y, and (S m ) m∈M defining a model collection indexed by M. First, we impose a structural assumption on each model indexed by m ∈ M. It is a bracketing entropy condition on the model S m with respect to the Hellinger divergence, defined by

  Before state the main theorem which leads to the oracle inequality for the model collection (4.6), we need to define some metrics used to compare the conditional densities. First, the Kullback-Leibler divergence is defined by and t two densities. To deal with regression data, for observed covariates (x 1 , . . . , x n ), we define

								.6)
	4.3.2 Notations
	KL(s, t) =	  	R	log	s(y) t(y)	s(y)dy if sdy << tdy;	(4.7)
		 	+ ∞ otherwise;
	for s KL ⊗n (s, t) = E	1 n	n i=1	KL(s(.|x i ), t(.|x i ))	(4.8)
	for s and t two densities.

Table 4 .

 4 1: Performances of our procedure. Mean number {KL, R, M, F A, ARI} of the Kullback-Leibler divergence between the model selected and the true model, the estimated rank of the model selected in each cluster, the missed variables, the false relevant variables, and the ARI, over 20 simulations We could draw the following conclusions from Table4.1. When we work in low-dimensional framework, we get very good results. Even if we could use any estimator, because we do not have dimension problem, with our procedure we get the matrix structure involved by the model.

		R	M FA ARI
	p > n 19.03 [2.8,3] 0 20 0.95
	p < n 3.28	[3,3]	0 0.6 0.99

  H (s(.|x i ), t(.|x i )) ; for two densities s and t. A bracket [l, u] is a pair of functions such that for all (x, y) ∈ X × Y, For > 0, the bracketing entropy H [.] ( , S, d ⊗n H ) of a set S is defined as the logarithm of the minimum number of brackets [l, u] of width d ⊗n H (l, u) smaller than such that every densities of S belong to one of these brackets. Let m ∈ M.Assumption H m . There is a non-decreasing function φ m such that → 1/ φ m ( ) is nonincreasing on (0, +∞) and for every ∈ R + and every s m ∈ S m , 0H [.] ( , S m (s m , ), d ⊗n H )d ≤ φ m ( );where S m (s m , ) = {t ∈ S m , d ⊗n H (t, s m ) ≤ }. The model complexity D m is then defined as n 2

l(y, x) ≤ s(y|x) ≤ u(y, x).

  unknown conditional density s * . Let S = (S m ) m∈M be at most countable collection of conditional density sets. Let assumption K holds while assumptions H m and Sep m hold for every models S m ∈ S. Let δ KL > 0, and sm ∈ S m such that KL ⊗n (s * , sm ) ≤ inf Introduce (S m ) m∈ M some random sub-collection of (S m ) m∈M . Consider the collection (ŝ m ) m∈ Then, for any ρ ∈ (0, 1) and any C 1 > 1, there are two constants κ 0 and C 2 depending only on ρ and C 1 such that, as soon as for every index m ∈ M, , and where the model complexity D m is defined in (4.11), the penalized likelihood estimate ŝ m with m ∈ M such that

		pen(m) ≥ κ(D m + (1 ∨ τ )w m )	(4.13)
	with κ > κ 0 n		n
	i=1	-log(ŝ m(y i |x i )) + pen( m) ≤ inf m∈ M	i=1	-log(ŝ m (y i |x i )) + pen(m) + η
	satisfies
		E(JKL ⊗n ρ (s * , ŝ m)) ≤ C 1 E inf m∈ M inf t∈Sm	KL ⊗n (s * , t) + 2	pen(m) n	(4.14)
		t∈Sm	KL ⊗n (s * , t) +	δ KL n	;
	and let τ > 0 such that
		sm ≥ e -τ s * .	(4.12)

M

of η-log-likelihood minimizer in S m , satisfying

n i=1 -log(ŝ m (y i |x i )) ≤ inf sm∈Sm n i=1 -log(s m (y i |x i )) + η.

Table 5 .

 5 Table5.1, we summarize the both models considered according to the day type. 1: For each model selected, we summarize the proportion of day type in each cluster, and interpret it, T corresponding to the temperature.

	Interpretation	Mon. Tue. Wed. Thur. Fri. Sat. Sun.
	week	0.88 0.96 0.94	0.98 0.96	0	0
	weekend	0.12 0.04 0.06	0.02 0.04	1	1
	week, low T.	0.26 0.46 0.46	0.47 0.51	0	0
	weekend, low T.	0.1	0.02 0.03	0	0	0.2 0.65
	week, high T.	0.64 0.52	0.5	0.53 0.45	0	0
	weekend, high T.	0	0	0	0	0.04 0.79 0.35

  Define zi,d the mean signal over all days d for the residential i, over all days d ∈ {1, . . . , 7}. Then we consider couples (z i,d , zi,d+1 ) 1≤i≤n . If we look on the model collection constructed by our procedure for K = {1, . . . , 8}, we always select models with only one component, for every days d. Nevertheless, if we force the model to have several clusters, restricting K to K = {2, . . . , 8}, we get some interesting informations. All results get here are done for preprocessing 2. For weekdays couples, Monday/Tuesday, Tuesdays/Wednesday, Wednesday/Thursday, Thursday/Friday, we select models with two clusters, with same means and same covariance matrices: the model with one component is needed. The only difference on load consumption is on the mean comportment. It is relevant with clusterings obtained in Section 5.3.2.

  [Com11a], [Com11b]), Socialclass: AB, C1, C2, F in UK demographic classifications, Ownership whether or not a customer owns his/her house, ResidentialStimulusallocation the stimulus sends to the customer (see [Com11a], [Com11b]), Built.Year the year of construction of the building, Heat.Home and Heat.Water electrical or not, Windows.Doubleglazed the proportion of double gazed window in the house (none, quarter, half, 3 quarters, all), Home.Appliance.White.goods the number of white goods/major appliances of the household. To relate our clusters to those variables we consider the classification problem consisting in recovering model based classes with a random forest classifier. Random forest introduced in [Bre01]

  -z i,t ) 2 .

	48 t=1 (ẑ i,t 0 1 48 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6
	1 cluster	LMLE 2 clusters	LMLE 2 clusters bis

; où κ est une constante positive.

pour le Lasso. On suppose que les covariables, bien qu'aléatoires, sont bornées. Pour simplifier la lecture, on suppose que X ∈ [0, 1] p . Dans cette thèse, on obtient alors les deux théorèmes suivants.

Group-Lasso-Rank procedure . . . . . . . . . . . . . . . . . . . . . . . . 66

CHAPTER 3. AN ORACLE INEQUALITY FOR THE LASSO-MLE PROCEDURE

In this chapter, we focus on a theoretical result for the Lasso-MLE procedure. We will get a penalty which depends on the model complexity for which the model selected by the penalized criterion among the collection constructed satisfies an oracle inequality. This result is non-asymptotic. We derive it from a general model selection theorem, also detailed here, which is a generalization of Cohen and Le Pennec Theorem, [START_REF] Cohen | Conditional density estimation by penalized likelihood model selection and applications[END_REF], for a model collection constructed randomly.

Introduction

The goal of clustering methods is to discover a structure among individuals described by several variables. Specifically, in regression case, given n observations (x, y) = ((x 1 , y 1 ), . . . , (x n , y n )) which are realizations of random variables (X, Y ) with X ∈ R p and Y ∈ R q , one aims at grouping the data into clusters such that the observations Y conditionally to X in the same cluster are more similar to each other than those from the other clusters. Different methods could be considered, more geometric or more statistical. We are dealing with model-based clustering, in order to have a rigorous statistical framework to assess the number of clusters and the role of each variable. This method is known to have good empirical performance relative to its competitors, see in [START_REF] Thalamuthu | Evaluation and comparison of gene clustering methods in microarray analysis[END_REF]. Datasets are described by a lot of explicative variables, sometimes much more than the sample size. All the information should not be relevant for the clustering. To solve this problem, we propose a procedure which provide a data clustering from variable selection. In a density estimation way, we could also cite Pan and Shen, in [START_REF] Pan | Penalized model-based clustering with application to variable selection[END_REF], who focus on mean variable selection, Zhou and Pan, in [START_REF] Zhou | Penalized model-based clustering with unconstrained covariance matrices[END_REF], who use the Lasso estimator to regularize Gaussian mixture model with general covariance matrices, Sun and Wand, in [START_REF] Sun | Regularized k-means clustering of highdimensional data and its asymptotic consistency[END_REF], who propose to regularize the k-means algorithm to deal with high-dimensional data, Guo et al. in [START_REF] Guo | Pairwise variable selection for high-dimensional model-based clustering[END_REF] who propose a pairwise variable selection method. All of them deal with penalized model-based clustering. In a regression framework, the Lasso estimator, introduced by Tibshirani in [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF], is a classical tool in this context. Working well in practice, many efforts have been made recently on this estimator to get some theoretical results. Under a variety of different assumptions on the design matrix, we could have oracle inequalities for the Lasso estimator. For example, we can state the restricted eigenvalue condition, introduced by Bickel, Ritov and Tsybakov in [START_REF] Bickel | Simultaneous analysis of Lasso and Dantzig selector[END_REF], who get an oracle inequality with this assumption. For an overview of existing results, cite for example [START_REF] Van De Geer | On the conditions used to prove oracle results for the Lasso[END_REF].

Whereas focus on the estimation, the Lasso estimator could be used to select variables, and, for this goal, many results without strong assumptions are proved. The first result in this way is from Meinshausen and Bühlmann, in [START_REF] Meinshausen | Stability selection[END_REF], who show that, for neighborhood selection in Gaussian graphical models, under a neighborhood stability condition, the Lasso estimator is consistent. Under different assumptions, as the irrepresentable condition, described in [START_REF] Zhao | On model selection consistency of lasso[END_REF], one get the same kind of result: true variables are selected consistently. Thanks to those results, one could refit the estimation, after the variable selection, with an estimator with better properties. In this chapter, we focus on the maximum likelihood estimator on the estimated active set. In a linear regression framework, we could cite Massart and Meynet, [START_REF] Massart | The Lasso as an 1 -ball model selection procedure[END_REF], or Belloni and Chernozhukov, [START_REF] Belloni | Least squares after model selection in highdimensional sparse models[END_REF], or also Sun and Zhang,[START_REF] Sun | Scaled sparse linear regression[END_REF] for using this idea.

In our case of finite mixture regression, we propose a procedure which is based on a modeling that recasts variable selection and clustering problems into a model selection problem. This procedure is developed in [START_REF] Devijver | Model-based clustering for high-dimensional data[END_REF], with methodology, computational issues, simulations and with π1 = 0.89. Just a few of variables have been selected, which give to this model the understanding property of which coefficient are discriminating. We could discuss about those models. The first select only two clusters, but almost all variables, whereas the second model has more clusters, and less variables: there is a trade-off between clusters and variable selection for the dimension reduction.

Chapter 5

Clustering electricity consumers using high-dimensional regression mixture models.