Cesar Arnaud

Huynh Bao-Tuyen

Keywords: Mixture models, Mixture of Experts, Regularized Estimation, Feature Selection, Lasso, ℓ 1 -regularization, Sparsity, EM algorithm, MM Algorithm, Proximal-Newton, Coordinate Ascent, Clustering, Classification, Regression, Prediction

Many thanks to you, guys. It has been a pleasure to spend the last three years or more getting to know you.

Very special thanks to my teacher Dang Phuoc-Huy. Your enthusiasm for statistics, your wealth of knowledge, and your diligence inspires me to learn more and work harder, in an attempt to emulate your achievements. Without your help and support I cannot finalize this work. I also extend my thanks to anh Vu and cô Phuong

Scientific context

Nowadays, big data are collected and mined in almost every area of science, entertainment, business and industry. For example, medical scientists study the genomes of patients to decide the best treatments and to learn the underlying causes of their disease. Often, big data is heterogeneous, high-dimensional, unlabeled, involve missing characteristics, etc. Such complexity poses some issues on treating and analyzing this kind of data. Analyses include prediction on future data, and data exploration to summarize the main information within the data, or reveal hidden useful information, like groups of common characteristics, which can be achieved via optimized clustering techniques. Analyses also cover identifying redundant or usefulness features, in representing the data, for which feature selection methodologies can be helpful.

Such challenges impose new analysis strategies including pushing forward the modeling methodologies, along with optimized model inference algorithms. In such area of analysis, main state-of-the art methods rely either on learning automatic data analysis systems like neural networks [START_REF] Bishop | Neural networks for pattern recognition[END_REF], support vectors machines [START_REF] Vapnik | Statistical learning theory[END_REF], kernel machines [START_REF] Scholkopf | Learning with kernels: support vector machines, regularization, optimization, and beyond[END_REF], and also provable statistical generative learning models [START_REF] Friedman | The elements of statistical learning[END_REF], including statistical latent variable models. Statistical analysis of data is indeed an efficient tool that takes into account the randomness part of the data, measures uncertainty, and hence provides a nice framework for density estimation, prediction and unsupervised learning from data, including clustering. Among statistical models, mixture models [START_REF] Titterington | Statistical analysis of finite mixture distributions[END_REF]McLachlan and Peel., 2000) are the standard way for the analysis of heterogeneous data across a broad number of fields including bioinformatics, economics, machine learning, among many others. Such kind of models have been used for example applications including complex systems 1 1.1. SCIENTIFIC CONTEXT maintenance [START_REF] Chamroukhi | Switch mechanism diagnosis using a pattern recognition approach[END_REF](Chamroukhi et al., , 2009a;;Onanena et al., 2009;[START_REF] Chamroukhi | A dynamic probabilistic modeling of railway switches operating states[END_REF], bioacoustics [START_REF] Chamroukhi | Bayesian non-parametric parsimonious gaussian mixture for clustering[END_REF][START_REF] Bartcus | Hierarchical dirichlet process hidden markov model for unsupervised bioacoustic analysis[END_REF].

Thanks to their flexibility, mixture models can be used to estimate densities and cluster data arising from complex heterogeneous populations. They are also being used to conduct prediction analysis including regression, via their Mixtures of experts (MoE) extension [START_REF] Jacobs | Adaptive mixtures of local experts[END_REF]; [START_REF] Jordan | Hierarchical mixtures of experts and the EM algorithm[END_REF]. Basically, maximum likelihood estimation (MLE) [START_REF] Fisher | On an absolute criterion for fitting frequency curves[END_REF] with EM algorithms [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF][START_REF] Mclachlan | The EM algorithm and extensions[END_REF] is the common way to conduct parameter estimation of mixture models and MoE models. However, applying these methods directly on high-dimensional data set has some drawbacks. For example, estimating the covariance matrix in Gaussian mixture models in a high-dimensional scenario is "a curse of dimensionality" problem. Furthermore, a more serious problem occurs in the EM algorithm when computing the posterior probabilities in the E-step which require the inversion of the covariance matrices. The same problem can be found while applying the Newton-Raphson [START_REF] Boyd | Convex optimization[END_REF] procedure in the M-step of the EM algorithm namely for mixtures-ofexperts.

Actually, in a high-dimensional setting, the features can be correlated and the actual features that explain the problem reside in a low-dimensional intrinsic space. Hence, there is a need to conduct an adapted estimation and select a subset of the relevant features, that really explain the data. The Lasso or ℓ 1 -regularized linear regression method introduced by [START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF] is a successful method that has been shown to be effective ant in classical statistical analysis like regression on homogeneous data. Lasso provides two advantages: it yields sparse solution vectors, having only some coordinates that are nonzero and the convexity of the related optimization problem greatly simplifies the computation.

In this thesis, we focus on Mixtures of experts (MoE) and for generalized linear models. They go beyong density estimation and clustering of vectorial data, and provide an efficient framework to density estimation, clustering, and prediction of regression-type data, as well as supervised classification of heterogeneous data [START_REF] Jacobs | Adaptive mixtures of local experts[END_REF][START_REF] Yuksel | Twenty years of mixture of experts[END_REF]Jiang and Tanner, 1999a,b;[START_REF] Grün | Fitting finite mixtures of generalized linear regressions in r[END_REF]. MoE are used in several applications such as: predicting the daily electricity demand of France [START_REF] Weigend | Nonlinear gated experts for time series: Discovering regimes and avoiding overfitting[END_REF], generalizing the autoregressive models for time-series data [START_REF] Zeevi | Time series prediction using mixtures of experts[END_REF][START_REF] Wong | On a logistic mixture autoregressive model[END_REF], recognizing handwritten digits [START_REF] Hinton | Recognizing handwritten digits using mixtures of linear models[END_REF], segmentation and clustering time series with changes in regime [START_REF] Chamroukhi | Time series modeling by a regression approach based on a latent process[END_REF][START_REF] Samé | Model-based clustering and segmentation of time series with changes in regime[END_REF][START_REF] Chamroukhi | Model-based clustering and classification of functional data[END_REF], human activity recognition [START_REF] Chamroukhi | Joint segmentation of multivariate time series with hidden process regression for human activity recognition[END_REF], etc. Unfortunately, modeling with mixtures of experts models in the case of high-dimensional predictors is still limited. Our objectives here are therefore to study the estimation and feature selection in high-dimensional mixtures-of-experts models, and to: i) propose new estimation, and feature selection strategies, to overcome such limitations. We wish to achieve that by introducing new regularized estimation and feature selection in the MoE framework for generalized linear models based on Lasso-like penalties. We study CHAPTER 1. INTRODUCTION them for different family of mixtures of experts models, including MoE with Gaussian, Poisson and logistic expert distributions.

ii) develop efficient algorithms for estimate the parameters of these regularized models. The developed algorithms perform simultaneous feature estimation and selection, and should enjoy the capability of encouraging sparsity, and deal with some typical high-dimensional issues like by avoiding matrix inversion, so that they perform quite well in high-dimensional situations.

In the following section, we summarize the contributions of the thesis.

Contributions of the thesis

The manuscript is organized as follows. Chapter 2 is dedicated to state-of-the-art. Chapter 3 presents our first contribution to the estimation and feature selection of mixtures-of-experts models, for continuous data. Then, Chapter 4, presents our second contribution. It is on the estimation and feature selection of MoE models, for discrete data. Finally, in Chapter 5, we discuss our research, draw conclusions, and highlight some potential future avenues to pursue. Technical details related to the mathematical developments of our contributions are provided in Appendices A, B and C.

More specifically, first, in Chapter 2, we provide a substantial review of state-of-the art models and algorithms related to scientific subjects addressed in the thesis. We first focus on the general mixture modeling framework, as an appropriate choice of modeling heterogeneity in data. We describe its statistical modeling aspects and the related estimation strategies, and model selection techniques, with a particular attention given to the MLE via the EM algorithm. Then, we revisit this mixture modeling context, in the framework of regression problems on heterogeneous data, and present its extension to the framework of Mixtures of Experts models. At the next stage, we consider these models in a high-dimensional setting, including the case with Gaussian components, and describe the three main strategies to address the curse of dimensionality, i.e, the two-fold dimensionality reduction approach, the one of spectral decomposition of the model covariance matrices, and the one of feature selection via Lasso regularization techniques. We opt for this latter regularization strategy, and present it for the case of mixture of regression and MoE models. We review the regularized maximum likelihood estimation and feature selection for these models via adapted EM algorithms.

Then, in Chapter 3, we introduce a novel approach for the estimation and feature selection of mixtures-of-experts for regression with potentially high-dimensional predictors, and a heterogeneous population. The approach simultaneously performs parameter estimation, feature selection, clustering and regression on the heterogeneous regression data. It consists of regularized maximum-likelihood approach with a dedicated regularization that, on the one hand, encourages sparsity thanks to a Lasso-like regularization part, and on the other hand, efficient 1.2. CONTRIBUTIONS OF THE THESIS to handle due to the convexity of the ℓ 1 penalty. We propose an effective hybrid Expectation-Maximization (EM) framework, to efficiently solve the resulting optimization problem which monotonically maximizes the regularized log-likelihood. It results into three hybrid algorithms for maximizing the proposed objective function, that is, a Majorization-Maximization (MM) algorithm, a coordinate ascent, and a proximal Newton-type procedure. We show that the proposed approach does not require an approximate of the regularization term, and the three developed hybrid algorithms, allow to automatically select sparse solutions without any approximation on the penalty functions. We rely on a modified BIC criterion to achieve the model selection task, including the selection of the number of components, and the regularization tuning hyper-parameters. An experimental is then considered to compare the proposed approach to the main competitive state-of-art methods for the subject. Evaluation is made to assess the performance of the approach in terms of clustering, density estimation, regression, and sparsity, of heterogeneous data by MoE models. Extensive experiments on both simulations and real data, show that the proposed approach outperforms its competitive and is very encouraging to address the high-dimensional issue. This chapter has mainly led to the journal publication [START_REF] Chamroukhi | Regularized maximum likelihood estimation and feature selection in mixtures-of-experts models[END_REF] and conferences papers.

Next, in Chapter 4, we consider another family of MoE models, the one of discrete data, including MoE for counting data and MoE for classification, and introduce our main second contribution. We present a new regularized MLE strategy to the estimation and feature selection of dedicated mixtures of generalized linear expert models in a high-dimensional setting. We develop an efficient EM algorithm, which relies on a proximal Newton approximation, to monotonically maximize the proposed penalized log-likelihood criterion. The presented strategy simultaneously performs parameter estimation, feature selection, and classification on the heterogeneous discrete data. An advantage of the introduced proximal Newton-type strategy consists in the fact that one just need to solve weighted quadratic Lasso problems to update the parameters. Efficient tools such as coordinate ascent algorithm can be used to deal with these problems. Hence, the proposed approach does not require an approximate of the regularization term, and allow to automatically select sparse solutions without thresholding. Our approach is shown to perform well including in a high-dimensional setting and to outperform competitive state of the art regularized MoE models on several experiments on simulated and real data. The main publication related to this chapter is [START_REF] Huynh | Estimation and feature selection in mixtures of generalized linear experts models[END_REF] and other communication.

Finally, in Chapter 5, we discuss our developed research, draw some conclusions and open some future directions. The thesis research publications results, including R packages of opensource codes, are given in the list of publications and communications.

A substantial summary in French is provided in Résumé long en français at the end of the manuscript.

Chapter 2

State of the art Contents

FINITE MIXTURE MODELS

More specifically, we focus on FMMs for modeling, density estimation, clustering and regression. We also review the Expectation-Maximization (EM) algorithm [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF][START_REF] Mclachlan | The EM algorithm and extensions[END_REF] for maximum likelihood parameter estimation (MLE) of FMMs (McLachlan and Peel., 2000). However, the problem lies in the fact that applying these EM algorithms directly on large-scale data sets poses some drawbacks. In such situations, we require new techniques to handle these difficulties.

For data clustering with FMMs, several propositions were introduced in literature to deal with high-dimensional data sets such as: Parsimonious Gaussian mixture models [START_REF] Banfield | Model-based gaussian and non-gaussian clustering[END_REF][START_REF] Celeux | Gaussian parsimonious clustering models[END_REF], Mixture of factor analyzers [START_REF] Ghahramani | The EM algorithm for mixtures of factor analyzers[END_REF]McLachlan and Peel, 2000;[START_REF] Mclachlan | Modelling high-dimensional data by mixtures of factor analyzers[END_REF], etc. Recently, some authors suggested to use the regularized approaches for data clustering in high-dimensional setting. We take a survey on these approaches here.

Later, a review on regularized methods for variable selection in mixture of regression models, which form the core of this thesis, is presented. These works are mainly inspired by the Lasso regularization. We also give some discussions on the advantages and drawbacks of the approaches not only in term of modeling but also in term of parameter estimation approaches.

Most of the methods and approaches in this chapter are well-known in literature and are included to provide context for later chapters. This chapter is organized into four main parts. The first and second parts are concerned with FMMs for density estimation, clustering and regression tasks, and EM algorithms for parameter estimation. The third and last parts address the same problems but in high-dimensional scenario. Finally, we draw some conclusions and introduce our research directions in this thesis.

Finite mixture models

Let X ∈ X be a random variable where X ⊂ R p for some p ∈ N. Denote the probability density function of X by f (x) and let f k (x) (k = 1, 2, . . . , K) be K probability density functions over X . X is said to arise from a finite mixture model (FMM) if the density function of X is decomposed into a weighted linear combination of K component densities, (2.1) where π k > 0, for all k and K k=1 π k = 1. The parameters π 1 , . . . , π K are referred to as mixing proportions and f 1 , . . . , f K are referred to as component densities. We can characterize this model via a hierarchical construction by considering a latent variable Z, where Z represents a categorical random variable which takes its values in the finite set Z = {1, . . . , K} and P(Z = k) = π k . If we assume that the conditional density of X given CHAPTER 2. STATE OF THE ART Z = k equals f k (x), then the join density of (X, Z) can be written as following

f (x) = K k=1 π k f k (x),
f (x, z) = K k=1 [π k f k (x)] I(z=k) (2.2)
where I(z = k) is the indicator function and equals 1 when z = k and 0 otherwise. Hence, the marginal density of X is .3) This model was first proposed by [START_REF] Pearson | Contributions to the mathematical theory of evolution[END_REF]. In his work, Pearson modeled the distribution of the forehead breadth to body length ratios for 1000 Neapolitan crabs with a mixing of K = 2 univariate Gaussian distributions. However, as we have been seen through literature review, there are mainly three types of FMM (McLachlan and Peel. (2000), [START_REF] Nguyen | Finite mixture models for regression problems[END_REF]):

f (x) = K z=1 f (x, z) = K z=1 K k=1 [π k f k (x)] I(z=k) = K k=1 π k f k (x). ( 2
• Homogeneous-parametric FMMs: the component densities come from the same parametric family, such as Gaussian Mixture Models, t-distribution Mixture Models;

• Heterogeneous FMMs: the component densities come from the different parametric family, like zero inflated Poisson distribution (ZIP), uniform-Gaussian FMMs, etc;

• Nonparametric FMMs.

For the parametric FMMs, assuming that each of the K component densities typically consists of a relatively simple parametric model f k (x; θ k ) (such as for a homogeneous Gaussian mixture model

θ k = (µ k , Σ k )).
The FMMs are widely used in a variety of applications. For those of biology, economics, genetics interested readers can refer to [START_REF] Titterington | Statistical analysis of finite mixture distributions[END_REF]. Besides that, these models are also used in many modern facets of scientific research, most notably in bioinformatics, pattern recognition and machine learning.

In this thesis, we shall refer FMMs as homogeneous-parametric FMMs to distinguish them from heterogeneous FMMs and nonparametric FMMs. It turns out that many parameter estimation methods for FMMs have been proposed in literature, such as [START_REF] Pearson | Contributions to the mathematical theory of evolution[END_REF] used the method of moments to fit a mixture of two univariate Gaussian components, [START_REF] Rao | The utilization of multiple measurements in problems of biological classification[END_REF] used Fisher ′ s method of scoring to estimate a mixture of two homoscedastic Gaussian components, etc. However, the more common methods are the maximum likelihood (McLachlan and Peel. (2000) and the Bayesian methods (Maximum A Posteriori (MAP)) where a prior distribution 2.2. FINITE MIXTURE MODELS is assumed for the model parameters (see [START_REF] Stephens | Bayesian analysis of mixture models with an unknown number of componentsan alternative to reversible jump methods[END_REF]). Here, we consider the maximum likelihood framework. For more details on the method of point estimation, viewers can refer to [START_REF] Lehmann | Theory of point estimation[END_REF]. The optimization algorithm for performing the maximum likelihood parameter estimation is the Expectation-Maximization (EM) algorithm (see [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF] and also [START_REF] Mclachlan | The EM algorithm and extensions[END_REF]). In the next section, the use of the Expectation-Maximization algorithm for learning the parameters of FMMs will be discussed. The object is to maximize the log-likelihood as a function of the model parameters θ = (π 1 , . . . , π K-1 , θ 1 , . . . , θ K ), over the parameter space Ω.

Assume that x = (x 1 , . . . , x n ) is an sample generated from K clusters and each cluster follows a probability distribution f k (x i ; θ k ) (a common choice of f k (x i ; θ k ) can be Gaussian distribution N (x; µ k , Σ k )). A hidden variable Z i (i = 1, . . . , n) represents a multinomial random variable which takes its values in the finite set Z = {1, ..., K} and for each observation x i the probability that x i belongs to the kth cluster is given by P(Z i = k) = π k . Hence, the data set x can be interpreted as an i.i.d sample generated from a FMMs with the probability density function f (x; θ), where

f (x; θ) = K k=1 π k f k (x; θ k ); π k > 0, K k=1 π k = 1.
(2.4)

The observed-data log-likelihood function therefore is given by

L(θ; x) = log n i=1 f (x i ; θ) = n i=1 log K k=1
π k f k (x i ; θ k ). (2.5) In this case, the log-likelihood is a nonlinear function. Therefore, there is no way to maximize it in a closed form. However, it can be locally maximized using iterative procedures such as Newton-Raphson procedure or the EM algorithm. We mainly focus on the EM algorithm, which is widely used for FMMs. The next section presents the EM algorithm for finding local maximizers of the general parametric FMMs. We will then apply this algorithm to GMMs.

Maximum likelihood estimation for FMMs via EM algorithm

The Expectation-Maximization (EM) algorithms are a class of iterative algorithms that were first considered in [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF] and the book of McLachlan and Peel. (2000) provides a complete review on the topic. It is a broadly used for the iterative computation of the maximum likelihood estimates on the framework of latent models. In particular, an EM algorithm simplifies considerably the problem of fitting FMMs by the maximum likelihood. This can be described as follows.

CHAPTER 2. STATE OF THE ART

EM algorithm for FMMs

In the EM framework, the observed-data vector x = (x 1 , . . . , x n ) is viewed as being incomplete, where each x i is associated with the K-dimensional component label vector z i (which is also called the latent variable) and the kth element of z i , z ik = 1 or 0, according to whether x i did or did not arise from the kth component of the mixture (2.4) (i = 1, . . . , n; k = 1 . . . , K).

The component-label vectors (z 1 , . . . , z n ) are taken to be the realized values of the random vectors (Z 1 , . . . , Z n ). Thus Z i is distributed according to a multinomial distribution consisting of one draw on K categories with probabilities (π 1 , . . . , π K ) Z 1 , . . . , Z n i.i.d ∼ Mult(1; π 1 , π 2 , . . . , π K ).

(2.6)

The complete-data log-likelihood function over observed variable x and latent variable z = (z 1 , . . . , z n ), governed by parameters θ can be written as

L c (θ; x, z) = log f (x, z; θ) = log n i=1 K k=1 π k f k (x i ; θ k ) z ik = n i=1 K k=1 z ik log π k + log f k (x i ; θ k ) , (2.7)
where f (x, z; θ) is the complete-data probability density function.

The EM algorithm is an iterative algorithm where each iteration consists of two steps, the E-step (Expectation step) and the M-step (Maximization step). Let θ [q] denote the value of the parameter vector θ after the qth iteration, and let θ [0] be some initialization value. The EM algorithm starts with θ [0] and iteratively alternates between the two following steps until convergence:

E-step

This step (on the (q + 1)th iteration) consists the computation of the conditional expectation of L c (θ; X, Z) given x, using θ [q] for θ. This expectation, denoted Q(θ; θ [q] ), is given by Q(θ; θ [q] ) = E[L c (θ; X, Z)|x; θ [q]

] = n i=1 K k=1 E[Z ik |x; θ [q] ] log π k + log f k (x i ; θ k ) = n i=1 K k=1 τ [q] ik log π k + log f k (x i ; θ k ) , (2.8)
where τ

[q]

ik = E[Z ik |x; θ [q] ] = P(Z ik = 1|x; θ [q] ) =

π [q] k f k (x i ; θ [q] k ) K l=1 π [q] l f l (x i ; θ [q] l ) (2.9)
is the posterior probability that x i belongs to the kth component density. As shown in the expression of Q(θ; θ [q] ) above, the E-step simply requires computing of the conditional posterior probabilities τ

[q]

ik (i = 1, . . . , n; k = 1, . . . , K).

M-step

The M-step (on the (q + 1)th iteration) evaluates the estimate of θ by the value θ [q+1] that maximizes the function Q(θ; θ [q] ) with respect to θ over the parameter space Ω:

θ [q+1] = arg max θ∈Ω Q(θ; θ [q] ).

(2.10)

Rewrite Q in the separate form

Q(θ; θ [q] ) = n i=1 K k=1 τ [q] ik log π k + n i=1 τ [q]
ik log f k (x i ; θ k ).

(2.11)

The updated estimate π [q+1] = (π

[q+1] 1 , . . . , π [q+1]
K-1 ) T of the mixing proportions π k (k = 1, . . . , K -1) is calculated independently of the updated estimate η [q+1] = (θ

[q+1] 1 , . . . , θ [q+1] K
) of the parameter vector η = (θ 1 , . . . , θ K ) containing the unknown parameters in the component densities of the FMM (2.4).

Hence, maximizing the function Q(θ; θ [q] ) in (2.11) with respect to the parameter π can be performed using the method of Lagrange multiplier (by taking account of the constraint K k=1 π k = 1), we obtain the updated estimates of the mixing proportions π k π

[q+1] k = n i=1 τ [q]
ik n, (2.12) for k = 1, . . . , K.

Now, in forming the estimate of the parameter vector η on the M-step of the (q + 1)th iteration, it can be seen from (2.11) that η [q+1] is obtained as an appropriate root of

K k=1 n i=1 τ [q] ik ∂ log f k (x i ; θ k ) ∂η = 0.
(2.13)

One nice feature of the EM algorithm is that the solution of (2.13) often exists in closed form.

The E-and M-steps are alternated repeatedly until the difference between the (incompletedata) log-likelihood function L(θ [q+1] ; x) and the log-likelihood function L(θ [q] ; x) is small enough such as L(θ [q+1] ; x) -L(θ [q] ; x) < ε, CHAPTER 2. STATE OF THE ART for a small value of some threshold ε.

Generally, the E-and M-steps have simple forms when the complete-data probability density function is from the exponential family [START_REF] Mclachlan | The EM algorithm and extensions[END_REF]McLachlan and Peel., 2000). In cases where the M-step cannot be performed directly, other methods can be used to update θ

[q+1] k such as Newton-Raphson algorithm and other adapted extensions.

We now show that, after each iteration the value of the log-likelihood function L(θ; x) (2.5) is not decreased, in the sense that L(θ [q+1] ; x) = log g(x; θ [q+1] ) ≥ log g(x; θ [q] ) = L(θ [q] ; x), (2.14) for each q = 0, 1, . . . and g(x; θ) is the probability density function of the observed data vector x. This property can be found via the results from Section 10.3 of [START_REF] Lange | Numerical analysis for statisticians[END_REF] and is given in Appendix A.1.

In case that one can not find θ [q+1] = arg max θ Q(θ; θ [q] ), [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF] suggested to take θ [q+1] such that Q(θ [q+1] ; θ [q] ) ≥ Q(θ [q] ; θ [q] ). The result of this is an increasing array of the log-likelihood values, i.e, log g(x; θ [q+1] ) ≥ log g(x; θ [q] ). These algorithms are called generalized EM algorithms. The EM algorithm ensures that one will receive a monotonically increasing sequence L(θ [q] ; x). A complete proof on the convergence properties of EM algorithm can be found in [START_REF] Wu | On the convergence properties of the EM algorithm[END_REF].

Gaussian mixture models

The most popular homogeneous-parametric FMMs are the Gaussian mixture models (GMMs).

It is used for modeling the probability density function of random variables in R p . In this case, GMMs have density functions of the form

f (x; θ) = K k=1 π k N (x; µ k , Σ k ), (2.15)
where N (x; µ k , Σ k ) denotes the multivariate Gaussian density with mean vector µ k , covariance matrix Σ k and θ is the vector containing all the parameters in π = {π 1 , . . . , π K-1 }, µ = {µ 1 , . . . , µ K }, Σ = {Σ 1 , . . . , Σ K } that represents this model. Since many natural measurements and processes tend to have Gaussian distributions, it is not hard to incorporate the use of GMMs. In these cases, the populations containing subpopulations of such measurements tends to have densities resembling GMMs. Furthermore, GMMs are among the simplest FMMs to estimate and are thus straightforward to apply in practice. Another critical property of GMMs is that, for a given continuous density function f (x), one can use a GMM with K components to approximate f (x). This result can be done by using the following theorem (see Section 33.1 of DasGupta ( 2008)).

Theorem 2.2.1 (Denseness of FMMs). Let 1 ≤ p < ∞ and f (x) be a continuous density on

2.2. FINITE MIXTURE MODELS R p . If g(x)
is any continuous density on R p , then given ε > 0 and a compact set C ⊂ R p , there exists an FMM of the form

f (x) = K k=1 π k 1 σ p k g x -µ k σ k , such that sup x∈C |f (x) -f (x)| < ε, for some K ∈ N, where µ k ∈ R p and σ k > 0 (k = 1 . . . , K).
By taking g(x) = N (x; 0, ∆), where ∆ is a constant p × p matrix and is symmetric positive definite, we see that GMMs with component densities from the location-scale family of g,

1 σ p g x -µ σ : µ ∈ R p , σ > 0 = {N (x; µ, σ 2 ∆) : µ ∈ R p , σ > 0}
are dense in the class of continuous densities on R p .

Recent novel theoretical results about the approximation capabilities of mixture models are given in [START_REF] Nguyen | Approximation by finite mixtures of continuous density functions that vanish at infinity[END_REF]. In the next section, we present the EM algorithm for GMMs.

EM algorithm for GMMs

Consider the GMM case. Here, we assume that

x i i.i.d ∼ f (x; θ), with f (x; θ) is a mixture density function of the form (2.15) f (x; θ) = K k=1 π k N (x; µ k , Σ k ); π k > 0, K k=1 π k = 1, with normal components N (x; µ k , Σ k ) = 1 (2π) p/2 |Σ k | 1/2 exp - 1 2 (x -µ k ) T Σ -1 k (x -µ k ) (k = 1, . . . , K).
From (2.5) and (2.7), the observed-data log-likelihood function for the GMM is given by .16) and the complete-data log-likelihood is

L(θ; x) = n i=1 log K k=1 π k N (x i ; µ k , Σ k ), ( 2
L c (θ; x, z) = n i=1 K k=1 z ik log π k + log N (x i ; µ k , Σ k ) . (2.17) With an initial parameter θ [0] = (π [0] 1 , . . . , π [0] K-1 , θ [0] 1 , . . . , θ [0] K ) (k = 1, . . . , K) where θ [0] k = (µ [0] k , Σ [0]
k ), the corresponding EM algorithm is defined as follows:

E-step This step (on the (q +1)th iteration) consists of computing the conditional expectation of (2.17), given the observed data x, using the current value θ [q] for θ. In this case, we have from (2.8) that (2.18) where the posterior probabilities are given by (2.9)

Q(θ; θ [q] ) = E[L c (θ; X, Z)|x; θ [q] ] = n i=1 K k=1 τ [q] ik log π k + n i=1 K k=1 τ [q] ik log N (x i ; µ k , Σ k ),
τ [q] ik = π [q] k N (x i ; µ [q] k , Σ [q] k ) K l=1 π [q] l N (x i ; µ [q] l , Σ [q] l ) (i = 1, . . . , n; k = 1, . . . , K). (2.19)
This step therefore computes the posterior probabilities that x i belongs to the kth component density using the current parameter value θ [q] .

M-step

The M-step updates θ by the value θ [q+1] that maximizes the function Q(θ; θ [q] ) with respect to θ. Here, for the GMMs, the updated estimates of the mixing proportions π k are as given by (2.12) and the updates of the component means µ k are given by

µ [q+1] k = n i=1 τ [q] ik x i n i=1 τ [q] ik , (2.20)
and for the problem, known as heteroscedastic, the component covariance matrices Σ k are unequal. To model this, the updated estimate

Σ [q+1] k of Σ k is defined by Σ [q+1] k = n i=1 τ [q] ik (x i -µ [q+1] k )(x i -µ [q+1] k ) T n i=1 τ [q] ik , (2.21)
for k = 1, . . . , K.

For homoscedastic case, i.e, Σ k = Σ (k = 1, . . . , K), then Σ is updated by

Σ [q+1] = 1 n K k=1 n i=1 τ [q] ik (x i -µ [q+1] k )(x i -µ [q+1] k ) T . (2.22)
The above updated estimates of µ k and Σ k can be deduced in an analogous method to deriving the maximum likelihood estimates of the mean vector and the covariance matrix (from an multivariate Gaussian distribution), that have been discussed by [START_REF] Anderson | An introduction to multivariate statistical analysis[END_REF] and given in Appendix A.2.

Example 2.2.1. In this example, we generate 200 data points of a mixture of two Gaussian components with,

µ 1 = (0, 0) T , µ 2 = (4, 4) T , Σ 1 = Σ 2 = Σ = I 2 , π 1 = π 2 = 1/2. 13 2.

FINITE MIXTURE MODELS

The results after using EM algoritm for homoscedastic case and heteroscedastic case are clarified However, estimating the covariance matrix of GMMs in large case (p ≫ 1) is "a curse of dimensionality". For a p-variate GMM with K components, the maximum dimension of total number of parameters to estimate is equal to

(K -1) + Kp + Kp(p + 1)/2,
where (K -1), Kp and Kp(p + 1)/2 are respectively the numbers of free parameters for the proportions, the means and the covariance matrices. Hence, the number of parameters to estimate is a quadratic function of p in the case of GMM. A large number of observations will be necessary to correctly estimate those model parameters. Furthermore, a more serious problem occurs in the EM algorithm when computing the posterior probabilities τ ik = P[Z i = k|x i ; θ] which require the inversion of the covariance matrices Σ k , k = 1, . . . , K. In addition, if the number of observations n is smaller than p (n < p) then the estimates of the covariance matrix Σk are singular and the clustering methods cannot be used at all. We will mention some recent results that related to parameter reducing and feature selection for data clustering and classification in high dimension in Section 2.4. This also includes the GMMs for data clustering.

Alongside with GMMs, which is widely used for data clustering there has been an interest in the formulations of FMMs for random variables, utilizing non-Gaussian component densities. The multivariate t FMM was first considered in [START_REF] Mclachlan | Robust cluster analysis via mixtures of multivariate t-distributions[END_REF] and [START_REF] Peel | Robust mixture modelling using the t distribution[END_REF], for the clustering of data that arise from non-Gaussian subpopulations. To model data where the component densities have non-elliptical confidence sets some skewed densities in FMMs have been proposed such as skew-normal FMMs and skew-t FMMs. For those results related to these models one can refer to the work of (Lee and Mclachlan, 2013;Lee andMcLachlan, 2013, 2014), etc. A Bayesian inference approach for skew-normal and skew-t FMMs can be found in Frühwirth-Schnatter and Pyne (2010).

Determining the number of components

Until now, we have considered the FMMs and an EM algorithm for estimation the parameters in a context of a known number of components K. Therefore, in practice, a natural question arises: "How to choose the number of components from the data?". This problem can be seen as a model selection problem. In FMM literature, it is one of the most interesting research topics and pays a lot of attentions; see (McLachlan and Peel., 2000, Chapter 6), [START_REF] Celeux | An entropy criterion for assessing the number of clusters in a mixture model[END_REF], [START_REF] Fraley | How many clusters? which clustering method? answers via model-based cluster analysis[END_REF], [START_REF] Fonseca | Mixture-model cluster analysis using information theoretical criteria[END_REF] for reviews on the topic.

In the mixture model, the log-likelihood of the sample (x 1 , . . . , x n ) takes the form

L K (θ; x) = n i=1 log K k=1 π k f k (x i ; θ k ) .
The function L K (θ; x) cannot be used as a selection criterion which balances model fit and model complexity for choosing the number K of components in the mixture. The most popular methodology for the selection of K is to use a criterion (score function) that ensures the compromise between flexibility and over-fitting. In general, a score function that is explicitly composed of two components: a component that measures the goodness of fit of the model to the data (such as the log-likelihood value, the complete-data log-likelihood value, etc), and a penalty function that governs the model complexity. This yields an overall score function of the form of score(model) = error(model)+penalty(model) which will be minimized.

As in general, the complexity of a model is related to the number of its free parameters ν. Thus, the penalty function then involves the number of model parameters. Popular score functions for the selection of K are information criteria (IC) which take into account the loglikelihood value. An information criterion (IC) based selection process can be described as below.

Let f K = K k=1 π k f k (x; θ) be a FMM with component density function of form f k (x; θ) and suppose that (x 1 , . . . , x n ) is a data sample arising from a population with density f K 0 (x; θ 0 ) = K 0 k=1 π k f k (x; θ), where K 0 ∈ K ⊂ N. For each K ∈ K we estimate the MLE θK,n and compute the IC for n observations and K components as

IC(n, K) = -2L( θK,n ) + η(n, θK,n ),
where L( θK,n ) is the log-likelihood value, η(n, θK,n ) is the penalty function that depends on n and the number of free parameters ν. After that, an estimator for K 0 is selected by

K0 = arg min K∈K IC(n, K).
In this section, we cite the most commonly used criteria for model selection in FMM. The widely used IC are defined as follows:

• Akaike Information Criterion (AIC) [START_REF] Akaike | A new look at the statistical model identification[END_REF] AIC(n, K) = -2L( θK,n ) + 2ν( θK,n ).

• Bayesian Information Criterion (BIC) [START_REF] Schwarz | Estimating the dimension of a model[END_REF]

BIC(n, K) = -2L( θK,n ) + ν( θK,n ) log(n).
• Approximate weight of evidence (AWE) [START_REF] Banfield | Model-based gaussian and non-gaussian clustering[END_REF])

AWE(n, K) = -2L( θK,n ) + 2ν( θK,n )( 3 2 + log(n)).
• Integrated Classification Likelihood (ICL) [START_REF] Biernacki | Assessing a mixture model for clustering with the integrated completed likelihood[END_REF])

ICL(n, K) = -2L c ( θK,n ) + ν( θK,n ) log(n),
where L c ( θK,n ) is the complete-data log-likelihood for the model.

A general review and an empirical comparison of these criteria are presented in (McLachlan and Peel., 2000, Chapter 6).

CHAPTER 2. STATE OF THE ART

Mixture models for regression data

In applied statistics, tremendous number of applications deal with relating a random response variable Y to a set of explanatory variables or covariates X through a regression-type model. One can model the relationship between Y and X via the conditional density function of Y given X = x, say, f (y|x). Similar to the density estimation, regression analysis is commonly conducted via parametric models of the form of f (y|x; θ), where θ is the parameter vector. The homogeneous case assumes the regression coefficients are the same for every observation data point (X 1 , Y 1 ), . . . , (X n , Y n ). However, this assumption is often inadequate since parameters may change for difference subgroups of observations. Such heterogeneous data can be modeled with a mixture model for regression, as studied namely in [START_REF] Chamroukhi | Hidden process regression for curve modeling, classification and tracking[END_REF]Chamroukhi ( , 2016d,a),a).

As an alternative extension of FMMs for the regression data, we suppose there is a latent random variable Z with probability mass function

P(Z = k) = π k , k = 1 . . . , K and K k=1 π k = 1. Moreover, assuming that Y |X = x, Z = k has the density p k (y|x; θ k )
, where θ k is the parameter vector of the kth subgroup, then we have the mixture of regression model

f (y|x; θ) = K k=1 π k p k (y|x; θ k ) (2.23)
by the same argument as that of (2.2). This model was first introduced in [START_REF] Quandt | A new approach to estimating switching regressions[END_REF], since he studied the market for housing starts by modeling the conditional density function of Y given X = x by a mixture of univariate two component Gaussian linear regression models, i.e,

f (y|x; θ) = πN (y; x T β 1 , σ 2 1 ) + (1 -π)N (y; x T β 2 , σ 2
2 ), (2.24) where N (.; µ, σ 2 ) is the Gaussian density function with mean µ and variance σ 2 .

In this thesis, Y is assumed to be a univariate random variable. The next subsection presents the univariate Gaussian linear mixtures of regression models and the EM algorithm for estimating parameters.

Mixture of linear regression models

Let ((X 1 , Y 1 ), . . . , (X n , Y n )) be a random sample of n independent pairs (X i , Y i ), (i = 1, . . . , n) where Y i ∈ Y ⊂ R is the ith response given some vector of predictors X i ∈ X ⊂ R p . Let D = ((x 1 , y 1 ), . . . , (x n , y n )) be the observed data sample.

For mixture of linear regression models (MLR), we consider the conditional density of Y given X = x defined by a mixture of K Gaussian densities (2.25) where the vector β k = (β k1 , . . . , β kp ) T ∈ R p , and scalars β k0 and σ 2 k are the regression coefficients, intercepts and variances of the kth component density, respectively. π = (π 1 , . . . , π K ) T is the vector of mixing propositions.

f (y|x; θ) = K k=1 π k N (y; β k0 + x T β k , σ 2 k ),

MLE via the EM algorithm

The observed-data log-likelihood function for this model is given by

L(θ) = n i=1 log K k=1 π k N (y i ; β k0 + x T i β k , σ 2 k ).
(2.26)

The EM algorithm can be used to obtain the updated estimates of the parameter θ as in case of a FMM described in Subsection 2.2.1 (see also Subsection 2.2.2). We can therefore use a set {z 1 , . . . , z n } of latent variables where z ik ∈ {0, 1} in which, for each data point, all of the elements k = 1, . . . , K are 0 except for a single value of 1 indicating which regression model of the mixture was responsible for generating that data point.

The complete-data log-likelihood then takes the form

L c (θ) = n i=1 K k=1 z ik (log π k + log N (y i ; β k0 + x T i β k , σ 2 k )).
(2.27)

The EM algorithm starts with an initial parameter θ [0] and repeat the following steps until convergence:

E-step (on the (q + 1)th iteration) This step consists of computing the expectation of the complete-data log-likelihood:

Q(θ; θ [q] ) = E[L c (θ)|x, y; θ [q] ] = n i=1 K k=1 E[Z ik |x, y; θ [q] ](log π k + log N (y i ; β k0 + x T i β k , σ 2 k )) = n i=1 K k=1 τ [q] ik log π k + n i=1 K k=1 τ [q] ik log N (y i ; β k0 + x T i β k , σ 2 k ), (2.28)
where

τ [q] ik = E[Z ik |x, y; θ [q] ] = P[Z ik = 1|x, y; θ [q] ] = π [q] k N (y i ; β [q] k0 + x T i β [q] k , σ [q]2 k ) K l=1 π [q] l N (y i ; β [q] l0 + x T i β [q] l , σ [q]2 l ) (2.29)
is the posterior probability that (x i , y i ) belongs to the kth component given the current parameter estimation θ [q] , and x = (x 1 , . . . , x n ), y = (y 1 , . . . , y n ).

CHAPTER 2. STATE OF THE ART M-step (on the (q + 1)th iteration) In this step, the value of the parameter θ is updated by computing the parameter θ [q+1] that maximizing the Q-function with respect to θ. The Q-function in (2.28) is decomposed as

Q(θ; θ [q] ) = Q(π; θ [q] ) + K k=1 Q k (θ k ; θ [q]
), (2.30) where .31) and

Q(π; θ [q] ) = n i=1 K k=1 τ [q] ik log π k , Q k (θ k ; θ [q] ) = n i=1 τ [q] ik log N (y i ; β k0 + x T i β k , σ 2 k ), ( 2
θ k = (β k0 , β T k , σ 2
k ) T is the parameter vector of the kth component density function. The maximization of Q(θ; θ [q] ) with respect to θ is the performed by separately maximizing Q(π; θ [q] ) and Q k (θ k ; θ [q] ) (k = 1 . . . , K). Maximizing Q(π; θ [q] ) with respect to π subject to K k=1 π k = 1 consists of constrained optimization problem, which is solved using Lagrange multipliers and leading to the updated estimate for π k given by (2.12). The maximization of Q k (θ k ; θ [q] ) with respect to (β k0 , β T k ) has the same form as weighted linear regression and the estimated regression coefficients in this case are given by (β

[q+1] k0 , β [q+1]T k ) T = (X T R [q] k X) -1 X T R [q]
k y, (2.32) where R

[q] k = diag(τ

[q] 1k , . . . , τ [q]
nk ), X is the design matrix and y is the response vector. Finally, the estimation of the variance σ 2 k is given by maximizing

n i=1 τ [q] ik log 1 √ 2πσ k - (y i -β k0 -x T i β k ) 2 2σ 2 k with respect to σ 2
k and takes the form

σ [q+1]2 k = n i=1 τ [q] ik (y i -β [q+1] k0 -x T i β [q+1] k ) 2 n i=1 τ [q]
ik , (for heteroscedastic case) (2.33) and

σ [q+1]2 = 1 n K k=1 n i=1 τ [q] ik (y i -β [q+1] k0 -x T i β [q+1] k ) 2 .
(for homoscedastic case) (2.34)

Example 2.3.1. In this example, a simulation study was performed whereupon the data was

f (y|x; θ) = π 1 N (y; β 10 + β 11 x, σ 2 ) + π 2 N (y; β 20 + β 21 x, σ 2 ), ( 2

Mixtures of experts

In a more general context, it is useful to consider that the latent variable Z also has a conditional relationship with some covariate vector R ∈ R p . Commonly, R is also the explanatory variables vector, i.e, R = X. [START_REF] Jacobs | Adaptive mixtures of local experts[END_REF] modeled the conditional probability of Z = k|X = x via the logistic function (also called softmax function) and named these models as mixtures of experts.

The mixture of experts (MoE) model assumes that the observed pairs (x, y) are generated from K ∈ N (possibly unknown) tailored probability density components (the experts), governed by a hidden categorical random variable Z ∈ [K] = {1, . . . , K} that indicates the component from which a particular observed pair is drawn. The latter represents the gating network. Formally, the MoE decomposes the probability density of the observed data as a convex sum of a finite experts weighted by the gating network (typically a softmax function), and defined by CHAPTER 2. STATE OF THE ART the following semi-parametric probability density (or mass) function:

p(y|x; θ) = K k=1 π k (x; w)p k (y|x; θ k ) (2.36)
that is parameterized by the parameter vector defined by θ = (w T , θ T 1 , . . . , θ T K ) T , where w T = (w T 1 , . . . , w T K-1 ), and θ k (k = 1, . . . , K) is the parameter vector of the kth expert. The gating network is defined by the distribution of the hidden variable Z given the predictor x, i.e., π k (x; w) = P(Z = k|X = x; w), which is in general given by gating softmax function of the form: (Jiang and Tanner, 1999a). The experts network is defined by the conditional distribution p k (y i |x i ; θ k ) which is the short notation of p(y i |X = x, Z i = k; θ). The experts are chosen to sufficiently represent the data for each group k. Tailored regressors explaining the response y by the predictor x for continuous data, or multinomial experts for discrete data. For example, non-Normal MoE models [START_REF] Chamroukhi | Non-normal mixtures of experts[END_REF], including MoE for non-symmetric data [START_REF] Chamroukhi | Skew-normal mixture of experts[END_REF] and robust MoE [START_REF] Chamroukhi | Skew t mixture of experts[END_REF][START_REF] Chamroukhi | Robust mixture of experts modeling using the t-distribution[END_REF][START_REF] Nguyen | Laplace mixture of linear experts[END_REF], with public software (e.g see Chamroukhi et al. (2019a)), have been introduced. To avoid singularities and degeneracies of the MLE as highlighted namely in [START_REF] Stephens | Bayesian methods for mixtures of normal distributions[END_REF]; [START_REF] Fraley | Bayesian regularization for normal mixture estimation and model-based clustering[END_REF] one can regularize the likelihood through a prior distribution over the model parameter space. For a complete account of MoE, the types of gating networks and experts networks, reader can refere to [START_REF] Nguyen | Practical and theoretical aspects of mixture-ofexperts modeling: An overview[END_REF]. A comprehensive survey of the MoEs can be found in [START_REF] Yuksel | Twenty years of mixture of experts[END_REF].

π k (x; w) = P(Z = k|X = x; w) = exp(w k0 + x T w k ) 1 + K-1 l=1 exp(w l0 + x T w l ) (2.37) for k = 1, . . . , K -1 with w T k = (w k0 , w T k ) ∈ R p+1 and w T K = (w K0 , w T K ) = 0 for identifiability
Let Y 1 , . . . , Y n be an independent random sample, with corresponding covariates x i (i = 1, . . . , n), arising from a distribution with density of form (2.36), and let y i be the observed data of Y i (i = 1, . . . , n). The generative process of the data assumes the following hierarchical representation. Given the predictor x i , the categorical variable Z i follows the multinomial distribution:

Z i |x i ∼ Mult(1; π 1 (x i ; w), . . . , π K (x i ; w)) (2.38)
where each of the probabilities π z i (x i ; w) is given by the multinomial logistic function (2.37).

Then, conditioning on the hidden variable Z i = z i , given the covariate x i , a random variable Y i is assumed to be generated according to the following representation

Y i |Z i = z i , X i = x i ∼ p z i (y i |x i ; θ z i ) (2.39)
where

p z i (y i |x i ; θ z i ) = p(y i |Z i = z i , X i = x i ; θ z i )
is the probability density or the probability 2.3. MIXTURE MODELS FOR REGRESSION DATA mass function of the expert z i depending on the nature of the data (x i , y i ) within the group z i .

A common choice to model the relationship between the input x and the output Y is by considering regression functions. Within each homogeneous group Z i = z i , the response Y i , given the expert k, is modeled by the noisy linear model:

Y i = β z i 0 + β T z i x i + σ z i ε i
, where the ε i are standard i.i.d zero-mean unit variance Gaussian noise variables, the bias coefficient β k0 ∈ R and β k ∈ R p are the usual unknown regression coefficients describing the expert Z i = k, and σ k > 0 corresponds to the standard deviation of the noise. In such a case, the generative model (2.39)

of Y becomes Y i |Z i = z i , x i ∼ N (.; β z i 0 + β T z i x i , σ 2 z i )• (2.40)
Assuming that, conditionally to the x i s, the Y i s are independent distributed with densities f (y i |x i ; θ), respectively, in which each of these densities is a MoE of K Gaussian densities,

f (y i |x i ; θ) = K k=1 π k (x i ; w)N (y i ; β k0 + x T i β k , σ 2 k ),
where the parameter vector of the kth expert is

θ k = (β k0 , β T k , σ 2 k ) T (k = 1, . . . , K).
In the considered model for Gaussian regression, the incomplete-data log-likelihood function is given by

L(θ) = n i=1 log K k=1 π k (x i ; w)N (y i ; β k0 + β T k x i , σ 2 k ) , (2.41)
and the complete-data log-likelihood has the form of

L c (θ) = n i=1 K k=1 z ik log π k (x i ; w)N (y i ; β k0 + β T k x i , σ 2 k ) .
(2.42)

The EM algorithm for maximizing the log-likelihood function (2.41) can be described as follows (see [START_REF] Jacobs | Adaptive mixtures of local experts[END_REF]; [START_REF] Yuksel | Twenty years of mixture of experts[END_REF]; [START_REF] Nguyen | Practical and theoretical aspects of mixture-ofexperts modeling: An overview[END_REF]):

E-step This step requires computing the expectation of the complete-data log-likelihood under the parameter vector θ [q] . This is simply equivalence to computing the posterior probability that (x i , y i ) belongs to the kth expert given the current parameter θ [q] . The Q(θ; θ [q] ) function has the form of

Q(θ; θ [q] ) = Q(w; θ [q] ) + K k=1 Q k (θ k ; θ [q]
), (2.43) where

Q(w; θ [q] ) = n i=1 K k=1 τ [q]
ik log π k (x i ; w), (2.44)

Q k (θ k ; θ [q] ) = n i=1 τ [q] ik log N (y i ; β k0 + x T i β k , σ 2 k ), (2.45) 22 CHAPTER 2. STATE OF THE ART with τ [q] ik = E[Z ik |x, y; θ [q] ] = π k (x i ; w [q] )N (y i ; β [q] k0 + x T i β [q] k , σ [q]2 k ) K l=1
π l (x i ; w [q] )N (y i ; β

[q] l0 + x T i β [q] l , σ [q]2 l )
, (2.46) and π k (x i ; w [q] ) is defined by (2.37).

M-step This step updates θ with θ [q] by maximizing the Q-function can be done separately by maximizing Q(w; θ [q] ) and Q k (θ k ; θ [q] ). For the experts, the formula (2.32) can be used to update (β k0 , β k ) and the formulas (2.33), (2.34) are used to update σ 2 k , σ 2 , respectively.

Updating the parameters for the gating network requires to maximize Q(w; θ [q] ). This function is concave and cannot be maximized in a closed form. The Newton-Raphson algorithm is generally used to perform the maximization as well as other gradient-based techniques (see [START_REF] Minka | Algorithms for maximum-likelihood logistic regression[END_REF]). Here, we consider the use of the Newton-Raphson algorithm to maximize Q(w; θ [q] ) with respect to w. The Newton-Raphson algorithm is an iterative procedure, which consists of starting with an initial arbitrary solution w (0) = w [q] , and updating the estimation of w until the convergence criterion is reached. A single update is given by s) ; θ [q] )] -1 ▽Q(w (s) ; θ [q] ), (2.47) where ▽ 2 Q(w (s) ; θ [q] ) is the Hessian matrix of Q(w; θ [q] ) with respect to w (s) and ▽Q(w (s) ; θ [q] ) is the gradient vector of Q(w; θ [q] ) at w (s) . For the closed form formulas of the Hessian matrix and the gradient vector one can refer to [START_REF] Chamroukhi | Hidden process regression for curve modeling, classification and tracking[END_REF]. The main advantage of the Newton-Raphson lies in the fact that it is quadratic convergence [START_REF] Boyd | Convex optimization[END_REF]). [START_REF] Zeevi | Error bounds for functional approximation and estimation using mixtures of experts[END_REF] showed that under some regularity conditions, then for any target function f (x) belongs to the Sobolev class, f ∈ W r p (L)

w (s+1) = w (s) -[▽ 2 Q(w (
W r p (L) g(x) : g W r p = |α|≤r g (α) (x) p ≤ L, x ∈ I d where g (α) ≡ ∂ α 1 g ∂x α 1 1 x α 2 2 . . . x α d d , α ∈ N d and I d = [-1, 1] d , then sup f ∈W r p inf q K ∈Q K f (x) -q K (x) Lp(I d ,λ) ≤ c K r/d , 1 ≤ p ≤ ∞ (2.
Q K q(x) : q(x) = K k=1 c k σ(β k0 + x T β k ) K k=1 σ(β k0 + x T β k ) , c k , β k0 ∈ R, β k ∈ R d .
The ridge function σ(.) is chosen to satisfy Assumption 2 of [START_REF] Zeevi | Error bounds for functional approximation and estimation using mixtures of experts[END_REF]. By choosing the exponent ridge function σ(t) = e t , then the manifold Q K is of the form

Q K q(x) : q(x) = K k=1 exp(β k0 + x T β k ) K k=1 exp(β k0 + x T β k ) c k , c k , β k0 ∈ R, β k ∈ R d .
This shows that as K → ∞ and under regularity conditions, for a given function f , there exists a mean function q(x) of a MoE model that can approximate f , arbitrarily closely. Here,

q(x) = E[Y = y|X = x; θ] = K k=1 π k (x; w)(β k0 + x T β k ). ( 2
E(Y |X = x) = h -1 (β 0 + x T β), (2.50)
h(.) is the univariate invertible link function. In Table 2.1, we present characteristics of some common univariate GLMs which were considered in [START_REF] Nelder | Generalized linear models[END_REF]. For more details regarding to GLMs, see [START_REF] Mccullagh | Generalized linear models[END_REF].

Model dom(Y ) h -1 (β 0 + x T β) f (y) E(Y |X = x) Gaussian R β 0 + x T β N (y; µ, σ 2 ) µ Binomial {0, . . . , N } N exp(β 0 +x T β) 1+exp(β 0 +x T β) N y p y (1 -p) N -y N p Gamma (0, ∞) -1/(β 0 + x T β) Γ(y; a, b) a/b Poisson N exp(β 0 + x T β) exp(-λ)λ y y! λ Table 2
.1: Characteristics of some common univariate GLMs.

CHAPTER 2. STATE OF THE ART Consider the mixtures of GLMs. These models can be expressed as follows

f (y|x; θ) = K k=1 π k f k (y; g(β k0 + x T β k , ϕ k ), ϕ k ) (2.51)
where f k (.) belongs to the class of GLMs, and θ k = (β k0 , β T k , ϕ T k ) T is the parameter vector of the kth component density. The characterizations of f k (.), g(.) and ϕ k for common GLMs are presented in Table 2

.2. Model dom(Y ) ϕ k g(β k0 + x T β k , ϕ k ) f k (y; g(β k0 + x T β k , ϕ k ), ϕ k ) Gaussian R σ 2 k β k0 + x T β k N (y; β k0 + x T β k , σ 2 k ) Binomial {0, . . . , N } None exp(β 0 +x T β) 1+exp(β 0 +x T β) N y exp(β 0 +x T β) 1+exp(β 0 +x T β) y 1 1+exp(β 0 +x T β ) N -y Gamma (0, ∞) b k - b k β k0 +x T β k Γ(y; - b k β k0 +x T β k , b k ) Poisson N None exp(β 0 + x T β) exp(-exp(β 0 +x T β)) exp((β 0 +x T β)y) y!
Table 2.2: Parameter vectors of some common univariate GLMs.

The fitting of mixtures of GLMs has been considered by [START_REF] Jansen | Maximum likelihood in a generalized linear finite mixture model by using the EM algorithm[END_REF], [START_REF] Wedel | A mixture likelihood approach for generalized linear models[END_REF] via EM algorithms. Along with the original GLMs, [START_REF] Wedel | A mixture likelihood approach for generalized linear models[END_REF] also reported the inverse Gaussian regression models as an alternative extension to the gamma model. Other interesting works are [START_REF] Aitkin | A general maximum likelihood analysis of overdispersion in generalized linear models[END_REF][START_REF] Aitkin | A general maximum likelihood analysis of variance components in generalized linear models[END_REF]. For the multivariate version of mixture of GLMs, it is interesting to consider the work of [START_REF] Oskrochi | An EM-type algorithm for multivariate mixture models[END_REF].

Asides from the mixtures of GLMs, the MoEs based on GLMs have also been widely mentioned in literature. For example, (Jiang andTanner, 1999a,b,c, 2000) considered the use of GLM based MoEs from various perspectives. An R package f lexmix for estimating the parameters of the MoEs based on GLMs is also introduced in (Grün andLeisch, 2007, 2008). Jiang and Tanner (1999a) established the approximation error when using the MoEs based on GLMs. Actually, their theorem can be interpreted as below: Let Ω = [0, 1] p , the space of the predict x. Let Y ⊆ R be the space of the response y. Let (Y, F Y , λ) be a general measurable space, (Ω, F Ω , κ) be a probability space such that κ has a positive continuous density with respect to the Lebesgue measure on Ω and (Ω×Y, F Ω ⊗F Y , κ⊗λ) be the product measure space. Consider a random predictor-response pair (X, Y ). Suppose that X has a probability measure κ, and (X, Y ) has a probability density function (pdf) ϕ(., .) with respect to κ ⊗ λ, where ϕ is of the form ϕ(x, y) = π(h(x), y).

Here, π(., .) : R × Y → R

) is F Y -measurable. Assume that h ∈ W 2 ∞ (K 0 ), where W 2 ∞ (K 0
) is a ball with radius K 0 in a Sobolev space with sup-norm and second-order continuous differentiability. Denote the set of all probability density functions ϕ(., .) = π(h(.), .) as Φ.

Consider an approximator f in the MoE family is assumed to have the following form

f = f K (x, y; θ) = K k=1 π k (x; w)π(h k (x), y), (2.53)
where h k (x) = β k0 + x T β k and π(., .) has the form of (2.52). Then, under some conditions Jiang and Tanner (1999a) proved the following result:

Theorem 2.3.1. (Approximation rate in KL divergence) sup ϕ∈Φ inf f K ∈Π K,S KL(f K , ϕ) ≤ c K 4/p ,
where c is a constant independent of p, Π K,S is a restriction on the set of the parameters of f K , imposed by Jiang and Tanner (1999a) and KL(f K , ϕ) is the Kullback-Leibler divergence between f K and ϕ defined by

KL(f K , ϕ) = Ω×Y f K (x, y) log f K (x, y)
ϕ(x, y) dκ(x)dλ(y).

Clustering with FMMs

One of the most common applications of FMMs is to cluster heterogeneous data, i.e the so-called model-based clustering approach. Suppose that we have an observation X from a FMM of form (2.4). One can consider each of the component densities f k (x; θ k ) as a subpopulation density of the overall population defined by density (2.4). Hence, it is reasonable to think about the probability that X belongs to one of the K densities as clustering criterion.

Using the characterization of a FMM, we can consider the label Z = k as the true cluster that the observation X belongs to (i.e. X is generated from the kth component density f k (x; θ k )).

In addition, we also can use the Bayes' rule to compute the posterior probability that Z = k given the observation of Y (see also (2.9))

P(Z = k|X = x) = π k f k (x; θ k ) K l=1 π l f l (x; θ l ) , (k = 1, . . . , K). = τ k (x) (2.54)
A hard-clustering of X is given using the posterior probability (2.54) via the Bayes' assignment rule ẑ = arg max k τ k (x).

(2.55) ẑ is considered as the cluster that X is generated from. For more details on the properties of the Bayes' rule see [START_REF] Wasserman | All of statistics: a concise course in statistical inference[END_REF]. Now, assume that x = (x 1 , . . . , x n ) is a sample generated form a parametric K component FMM of form (2.4) with the parameter vector θ = (π 1 , . . . , π K-1 , θ 1 , . . . , θ K ). Let θ denotes the estimated parameter vector for θ ( θ can be estimated via EM algorithm). Hence, the clustering task via FMMs can be described as follows:

i) For each observation x i , compute the posterior probabilities τ ik ≡ τ k (x i ; θ) via (2.9).

ii) Use the Bayes' rule (2.55) to determine the cluster label of x i .

In GMMs framework, the equivalent form of (2.9) is given by (2.19). Similar strategies can be used to cluster the data for MLR models and MoE models. In these settings, one just need to replace the formula (2.9) with (2.29) and (2.46), respectively, to compute the posterior probabilities.

Clustering and classification in high-dimensional setting

In this part, we focus on some recent methods for model-based clustering in high-dimensional setting. The classical methods basically can be split into three families: dimensionality reduction, regularization and constrained and parsimonious models see for example [START_REF] Bouveyron | Model-based clustering of high-dimensional data: A review[END_REF] for a review. Aside with these methods, recent research also provides some interesting approaches to deal with high-dimensional data such as subspace clustering methods and feature selection methods. A helpful text for these methods is [START_REF] Bouveyron | Model-based clustering of high-dimensional data: A review[END_REF].

Classical methods

Dimensionality reduction approaches

For dimensionality reduction methods, one assumes that his data set with the number p of measured variables is too large and, implicitly, there is a small subspace of order d ≪ p contains most of his data. In a Gaussian mixture scenario, once the data is projected in this subspace, it is possible to apply the EM algorithm on the projected observations to obtain a partition of the original data (if d is small enough).

One of the most popular linear methods used for dimensionality reduction is the principal component analysis (PCA), which was introduction by [START_REF] Pearson | On lines and planes of closest fit to systems of point in space[END_REF]. Pearson described it as a linear projection that minimizes the average projection cost. [START_REF] Tipping | Probabilistic principal component analysis[END_REF] provided a probabilistic view of PCA by assuming the observed variables X ∈ R p , are conditionally 2.4. CLUSTERING AND CLASSIFICATION IN HIGH-DIMENSIONAL SETTING independent given the values of the latent (or unobserved) variables T ∈ R d . The relationship between X and T is of the linear form

X = ΛT + µ + ε.
(2.56)

The p × d matrix Λ relates the two sets of variable, while the parameter µ permits the model to have non-zero mean, ε ∼ N (0, σ 2 I p ). Hence, the T conditional probability distribution over X-space is given by X|T ∼ N (µ + ΛT , σ 2 I p ).

(2.57) Furthermore, if the marginal distribution over the latent variables T is Gaussian T ∼ N (0, I d ), then the marginal distribution of X is also Gaussian X ∼ N (µ, ΛΛ T + σ 2 I p ). Estimates for Λ and µ can also be obtained by iterating maximization of the log-likelihood function, such as by using EM algorithm.

One will then cluster the observations using T as the input data rather than X. [START_REF] Ghosh | Mixture modelling of gene expression data from microarray experiments[END_REF] used this strategy to cluster microarray datasets. An EM algorithm for clustering the latent variable T can also be found in this work. A Bayesian version for this approach was investigated by [START_REF] Liu | Bayesian clustering with variable and transformation selections[END_REF].

Factor analysis (FA) is another way to deal with dimensionality reduction. The only difference between PCA and FA lies in the fact that, with FA models, the distribution of ε in (2.56) with a Gaussian distribution given by ε ∼ N (0, Ψ), where Ψ is a diagonal matrix, Ψ = diag(σ 2 1 , . . . , σ 2 p ). Hence, it is to reduce the dimensionality of the space and to keep the observed covariance structure of the data. In the same context, [START_REF] Tamayo | Metagene projection for cross-platform, cross-species characterization of global transcriptional states[END_REF] suggested decomposing the data matrix X using the nonnegative matrix factorization [START_REF] Lee | Learning the parts of objects by non-negative matrix factorization[END_REF]) to reduce the dimension before clustering the data. However, all these approaches have a number of drawbacks. The resulting clustering is not sparse in the features, since each latent variable T h , h = 1, . . . , d is a linear combination of the full set of p features. Moreover, there is no guarantee that the new feature T h will contains the information that one is interested in detecting via clustering. In fact, [START_REF] Chang | On using principal components before separating a mixture of two multivariate normal distributions[END_REF] studied the effective of performing PCA to reduce the data dimension before clustering. He generated a data set from a mixture of two Gaussian distributions and found that using this procedure is not justified since the first component does not necessarily provide the best separation between subgroups.

Mixtures with regularization of the covariance matrix

As we have mentioned at the end of Section 2.2.1, it is possible to see the problem in clustering of high-dimensional data as a numerical problem in computing the matrix inversion of Σ k , which is CHAPTER 2. STATE OF THE ART used to compute the posterior probabilities τ ik and the log-likelihood function. From this point of view, a simple way to tackle this problem is to regularize the estimation of Σ k before their inversion. This can be done by using the ridge regression, which adds a small positive quantity δ k to the diagonal of Σk Σk = Σk + δ k I p .

A general version was introduced by [START_REF] Hastie | Penalized discriminant analysis[END_REF] while these authors studied the penalized discriminant analysis Σk = Σk + δΩ,

where Ω is a p × p symmetric and nonnegative definite matrix. This approach is different from the previous one since it also penalizes the correlations between the predictors. [START_REF] Friedman | Regularized discriminant analysis[END_REF] proposed a compromise between linear discriminant analysis (LDA) and quadratic discriminant analysis (QDA), which allows one to shrink the separate covariances of QDA to a common covariance as in LDA. The regularized covariance matrices have the form

Σk (λ) = λ(n k -1) Σk + (1 -λ)(n -K) Σ λ(n k -1) + (1 -λ)(n -K) , (2.58)
where Σ is the pooled covariance matrix as used in LDA and Σk is the covariance matrix of group kth used in QDA. λ ∈ [0, 1] allows a continuum of models between LDA and QDA.

A similar strategy allows Σk (λ) to be reduced to the scalar covariance. Combining these models leads to a more general family of covariances Σk (γ, λ), indexed by a pair of parameters

Σk (γ, λ) = γ Σk (λ) + (1 -γ) tr[ Σk (λ)] p I p ,
for λ ∈ [0, 1] and tr[ Σk (λ)]/p is the average eigenvalue of Σk (λ).

It is also possible to use the Moore-Penrose pseudo-inverse of Σ instead of the usual inverse Σ-1 . For the generalized inverse of matrices and its applications, the reader can refer to [START_REF] Rao | Boosting as a regularized path to a maximum margin classifier[END_REF]. A comprehensive overview of regularization techniques in classification can be found in [START_REF] Mkhadri | Regularization in discriminant analysis: an overview[END_REF].

Constrained and parsimonious mixture models

In this section, we consider another way to tackle the curse of dimensionality by consider it as a problem of over-parameterized modeling. As we have discussed in Section 2.2.1, the GMM requires lots of parameters to infer the model in high-dimension setting. The use of constrained and parsimonious models is another approach to reduce the number of parameters in modelbased clustering.

Constrained Gaussian mixture models. A classical method to reduce the number of free parameters of GMMs is to add some constrains on the model through their parameters. Mainly, we will add some restrictions on the covariance matrices Σ k , which commonly requires p(p+1)/2 parameters to construct one of them. A full GMM with K components needs Kp(p + 1)/2 real values for the covariance matrices. A simple way to reduce the parameter space is to constraint the K covariance matrices to be the same across all mixture components, i.e., using the homoscedastic case. It is also possible to assume that the variables are conditionally independent. This assumption implies that all the covariance matrices are diagonal, i.e., Σ k = diag(σ 2 k1 , . . . , σ 2 kp ) for all k = 1, . . . , K, and the associated model (Diag-GMM) has a low number of free parameters. In addition, one can reduce the free parameters by assuming that σ 2 kj = σ 2 k , ∀j = 1, . . . , p. He can also consider the homoscedastic case for those models. Table 2.3 provides the number of parameters which are used for those constrained models. However, it is hard to find a real data set, in which the features are conditionally independent. Therefore, it restricts the range of applications of these models. [START_REF] Banfield | Model-based gaussian and non-gaussian clustering[END_REF] and [START_REF] Celeux | Gaussian parsimonious clustering models[END_REF] developed a different criteria that are more general than the constrained models. The key to their approach is a reparameterization of the covariance matrix Σ k in term of its eigenvalue decomposition. [START_REF] Celeux | Gaussian parsimonious clustering models[END_REF] called them parsimonious Gaussian models.

Parsimonious Gaussian models. These models parametrize the covariance matrices from their eigenvalue decomposition (see (Bellman, 1960, Section 3.5))

Σ k = λ k D k A k D T k , (2.59)
where D k is the matrix of eigenvectors determines the orientation of the group kth, A k is a diagonal matrix with the eigenvalues of Σ k on the diagonal that explains its shape, and the parameter λ k determines the volume. By constraining the parameters λ k , D k and A k , Celeux and Govaert (1995) enumerated 14 different models which are listed in Table A.1. The second column of Table A.1 corresponds to the name used by [START_REF] Raftery | Variable selection for model-based clustering[END_REF]. The parsimonious models propose a trade-off between the perfect modeling and what one can correctly estimate in practice. The reader can refer to [START_REF] Celeux | Gaussian parsimonious clustering models[END_REF] for more details on these models including parameter estimation methods. Model selection can be achieved using the Bayesian information criterion (BIC) [START_REF] Schwarz | Estimating the dimension of a model[END_REF].

The drawback of all the classical approaches lies in the fact that all the features are kept while clustering the data. Hence, subspace clustering methods are good alternatives to dimensionality reduction approaches. Recent works have focus on reduce data dimensionality by selecting CHAPTER 2. STATE OF THE ART relevant variables for the clustering task. We take a survey on these works in next sections.

Subspace clustering methods

In this part, we focus on some subspace clustering methods in the point of view of modelbased approach. These methods are mostly related to the factor analysis model assuming the observation variable is linked to a latent variable through a linear relationship. EM algorithms for maximum likelihood factor analysis can be given through the work of [START_REF] Rubin | EM algorithms for ML factor analysis[END_REF].

Mixture of factor analyzers

The idea of Mixture of factor analyzers (MFA) was introduced by [START_REF] Ghahramani | The EM algorithm for mixtures of factor analyzers[END_REF] in which an EM algorithm was proposed for parameter estimation, and was first presented from the perspective of a method for model-based density estimation from high-dimensional data in McLachlan and Peel (2000). In PFA, for each cluster, the observed variable, X ∈ R p is linked with a latent variable Y ∈ R d (d < p) through a linear relationship which are different from each cluster. The model can be described as follows: Let {x 1 , . . . , x n } be n independent observations of a random vector X ∈ R p . Assume that X can be expressed from an unobserved random vector Y ∈ R d (d < p), named the factor. Moreover, assume that there are n unobserved partition {z 1 , . . . , z n }, which are assumed to be independent unobserved realizations of a random latent variable Z ∈ {1, . . . , K} where z i = k indicates that x i belongs to the kth factor analyzer.

Z ∼ Mult(1; π 1 , . . . , π K ),
where π k > 0, for all k and K k=1 π k = 1. The relationship between X and Y conditionally to Z is given by

X| Z=k = µ k + Λ k Y + ε, (2.60)
where µ k is the mean vector of the kth factor analyzer, Λ k is a p × d matrix. ε ∈ R p is assumed to be a center Gaussian distribution with a diagonal covariance matrix Ψ and is independent with Y , ε ∼ N (0, Ψ).

Besides, as in regular factor analysis, the factors are all assumed to be N (0, I d ) distributed, therefore,

Y | Z=k = Y ∼ N (0, I d ).
This implies that the conditional distribution of X given Y , Z is also Gaussian

X|Y , Z = k ∼ N (µ k + Λ k Y , Ψ),
X|Z = k ∼ N (µ k , Λ k Λ T k + Ψ).
The marginal distribution of X is thus a GMM .61) For this model, we need (K -1) parameters for the mixing proportions, Kp for the means {µ k } K k=1 . If d is chosen sufficiently smaller than p, then there are some constrains on the covariance matrices Λ k Λ T k + Ψ and thus reduces the number of free parameters to be estimated. In the case d > 1, there is an infinity of choices for Λ k since one can replace Λ k by Λ k C, where C is an orthogonal matrix, i.e., CC T = I d . Hence, a way to specify Λ k is to choose it in such that the d × d matrix Λ T k ΨΛ k is diagonal. Using this approach, [START_REF] Lawley | Factor analysis as a statistical method[END_REF] showed that the number of free parameters for a regular factor analysis model (without the mean vector) is p + pdd(d -1)/2. Therefore, Kd(p -(d -1)/2) + p free parameters are required to estimate the component covariance matrices in equation (2.61). The model complexity is (K -1) + Kp + Kd(p -(d -1)/2) + p. For example with K = 3, p = 50 and d = 5, then 1672 parameters have to be estimated. An EM algorithm for parameter estimation is also given in [START_REF] Ghahramani | The EM algorithm for mixtures of factor analyzers[END_REF].

f (x) = K k=1 p(Z = k)f k (x|Z = k) = K k=1 π k N (x; µ k , Λ k Λ T k + Ψ). ( 2
This model was generalized later by [START_REF] Mclachlan | Modelling high-dimensional data by mixtures of factor analyzers[END_REF] who removed the constraint of the noise ε in (2.60). Hence,

X| Z=k = µ k + Λ k Y + ε k , (2.62)
where ε k ∼ N (0, Ψ k ), and Ψ k = diag(σ 2 k1 , . . . , σ 2 kp ). In this case, the model complexity increases and takes the following value: (K -1) + Kp + Kd(p -(d -1)/2) + Kp. Regarding the model inference, [START_REF] Mclachlan | Modelling high-dimensional data by mixtures of factor analyzers[END_REF] proposed to make use of an alternating expectation-conditional maximization (AECM) algorithm [START_REF] Meng | The EM algorithm-an old folk-song sung to a fast new tune[END_REF]) to fit the mixture of factor analyzers by maximum likelihood. An interesting application of this model can be found in the work of [START_REF] Mclachlan | A mixture model-based approach to the clustering of microarray expression data[END_REF], where the authors used it to cluster gene expression microarray data.

To reduce the complexity of the MFA models, [START_REF] Baek | Mixtures of factor analyzers with common factor loadings: Applications to the clustering and visualization of high-dimensional data[END_REF] reparameterized the mixture model with restrictions on the means and covariance matrices, such that, for k = 1, . . . , K The orthonormal condition on the matrix A is to make sure the solution  for A is unique up to postmultiplication by an orthogonal matrix (see (Baek et al., 2009, Appendix) The MCFA models can be fitted via the EM algorithm given by their authors (see (Baek et al., 2009, Appendix)).

µ k = Aρ k (2.63) and Σ k = AΩ k A T + D, ( 2

Expanded parsimonious Gaussian mixture models

Inspired from MFA, [START_REF] Mcnicholas | Model-based clustering of microarray expression data via latent gaussian mixture models[END_REF] proposed a family of models known as the expanded parsimonious Gaussian mixture models (EPGMM) family. Each component factor the covariance matrix Σ k in (2.61) is given by

Σ k = Λ k Λ T k + Ψ k ,
where Λ k is the loading matrix and Ψ k is the diagonal variance matrix for the noise term of the kth factor. The matrix Ψ k can be further parameterized by writing

Ψ k = ω k ∆ k ,
where

ω k ∈ R and ∆ k = diag(δ k1 , . . . , δ kp ) such that |∆ k | = 1.
Therefore, the covariance matrix Σ k can be rewritten as

Σ k = Λ k Λ T k + ω k ∆ k .
By adding some constraints on the loading matrices Λ k , the constant ω k and the diagonal matrix Ψ k , McNicholas and Murphy (2010) obtained twelve Gaussian mixture models named EPGMM family. Nomenclature and the corresponding covariance structure of the members of the EPGMM are given in Table A.2. Parameter estimation is also carried out using AECM algorithm. Model selection can be achieved using the BIC. It is worth to mention that their AECM algorithm could be used for inferring most of the MFA models.

CLUSTERING AND CLASSIFICATION IN HIGH-DIMENSIONAL SETTING

Mixture of high-dimensional GMMs

Following the classical parsimonious GMMs, (Bouveyron et al., 2007a,b) constrained the GMM using the eigen-decomposition of the covariance matrix Σ k of the kth group (see equation 2.59)

Σ k = Q k Λ k Q T k , (2.65)
where Q k is a p × p orthogonal matrix contains the eigenvectors of Σ k and Λ k is a diagonal matrix of the associated eigenvalue sorted in a decreasing order. The key idea of these works lies in fact that the authors reparametrize the matrix Λ k , such that Λ k has only

d k + 1 different eigenvalues Λ k = diag(a k1 , . . . , a kd k , b k , . . . , b k p-d k ),
where d k first eigenvalues a k1 , . . . , a kd k parametrize the variance in the group-specific subspace and the pd k last terms, the b k 's model the variance of the noise (d k < p). The noise variance of each cluster k is isotropic and is contained in a subspace which is orthogonal to the subspace of the kth group. Bouveyron et al. (2007a) named this family the [

a kj b k Q k d k ].
If the symmetric positive definite matrix Σ k have p distinct eigenvalues then one can obtain the eigen-decomposition form (2.65). However, we now proof that if this symmetric positive definite matrix has only d + 1 different eigenvalues (d < p) with pd multiple characteristic roots then we still have the decomposition form (2.65).

Lemma 2.4.1. Let Σ be a p×p symmetric positive definite matrix with d+1 different eigenvalues {λ 1 , . . . , λ d , γ} where γ has p-d multiple characteristic roots. Then, there exits a p×p orthogonal matrix Q which contains the eigenvectors of Σ such that

Σ = QΛQ T , (2.66)
where

Λ = diag(λ 1 , . . . , λ d , γ, . . . , γ p-d
).

The proof of this Lemma is given in Appendix A.3.3.

The number of parameters required for the estimation of Σ (with the assumption that and(d + 1) are the number of parameters required for the estimation of Q and (λ 1 , . . . , λ d , γ). This can be computed using the trick provided in Appendix A.3.4. Given some constraints on a kj , b k , Q k and d k , the authors proposed 28 GMMs. Among them, 16 models have the closed-form estimators. The covariance structure, number of covariance parameters for these models are given in Table A.3. The inference for 16 GMMs listed in Table A.3 can be done using the EM algorithm since update formula for each parameter is closed-form.

λ i > γ for all i ∈ {1, . . . , d}) is d(p -(d + 1)/2) + d + 1, where d(p -(d + 1)/2)
The subspace clustering methods posed in this section belong to a huge family of GMMs. Besides that, there exists several links between these models, which were listed in (Bouveyron and CHAPTER 2. STATE OF THE ART Brunet-Saumard, 2014, Figure 4). These models in some sense can be described as a combination of the dimensionality reduction and parametric reduction methods. Some drawbacks can be found since all features still be used to construct the subspace through the loading matrices or through the component densities. In addition, the estimation procedures in some models can not be directly done using the EM algorithm because of the specific features of the latent subspace.

Among the related works, we may cite the discriminative latent mixture models, which were proposed by [START_REF] Bouveyron | Simultaneous model-based clustering and visualization in the fisher discriminative subspace[END_REF]. The robust versions of the MFA models rely on t-distributions can be referred to [START_REF] Andrews | Extending mixtures of multivariate t-factor analyzers[END_REF] and [START_REF] Mclachlan | Extension of the mixture of factor analyzers model to incorporate the multivariate t-distribution[END_REF].

Variable selection for clustering

In this section, we focus on some recent works which simultaneously cluster the data and select relevant variables for the clustering task. This task can be handle in two different ways. The first one tackles the problem of variable selection for model-based clustering as a model selection problem, while the other aims at introducing a penalty term in the log-likelihood function in order to obtain sparsity in the features.

Variable selection as a model selection problem [START_REF] Law | Simultaneous feature selection and clustering using mixture models[END_REF] considered the GMMs and assumed that the features are conditionally independent given the (hidden) component label, that is,

p(x|θ) = K k=1 π k p(x|θ k ) = K k=1 π k p j=1 p j (x j |θ kj ), (2.67)
where p j (x j |θ kj ) is the pdf of the jth feature in the kth component. In case of GMMs, this condition is equivalent to adopting diagonal covariance matrices. In addition, for the role of each feature, they suggest that the jth feature is irrelevant if its distribution is independent of the class label, i.e., if it follows a common density, denoted by q(x j |λ j ). So if we denote Ψ = (ψ 1 , . . . , ψ p ) be the set of binary variables, such that ψ j = 1 if feature jth is relevant and ψ j = 0 otherwise, then the mixture density in (2.67) can be rewritten as

p(x|Ψ, θ) = K k=1 π k p j=1 [p j (x j |θ kj )] ψ j [q(x j |λ j )] 1-ψ j .
(2.68)

CLUSTERING AND CLASSIFICATION IN HIGH-DIMENSIONAL SETTING

Hence, the joint density of (x, Ψ) is

p(x, Ψ; θ) = p(x|Ψ; θ)p(Ψ; {ρ j }) = K k=1 π k p j=1 [p j (x j |θ kj )] ψ j [q(x j |λ j )] 1-ψ j p j=1 ρ ψ j j (1 -ρ j ) 1-ψ j = K k=1 π k p j=1 [ρ j p j (x j |θ kj )] ψ j [(1 -ρ j )q(x j |λ j )] 1-ψ j ,
where ρ j = P(ψ j = 1). Therefore, the marginal density of x is given by

p(x|θ) = Ψ p(x, Ψ; θ) = K k=1 π k Ψ p j=1 [ρ j p j (x j |θ kj )] ψ j [(1 -ρ j )q(x j |λ j )] 1-ψ j = K k=1 π k p j=1 [ρ j p j (x j |θ kj ) + (1 -ρ j )q(x j |λ j )], (2.69)
with θ = {{π k }, {θ kj }, {λ j }, {ρ j }} is the set of all parameters of the model. The parameter estimation is done via the EM algorithm by treating Z (the hidden class labels) and Ψ are hidden variables.

However, assuming that the irrelevant variables are independent on both the clustering variables and the relevant variable seem to be unrealistic. To tackle these limitations, Maugis et al. (2009a) based on [START_REF] Raftery | Variable selection for model-based clustering[END_REF] to split the variables into two different sets: S and S c , where S denotes the set of relevant variables and S c is the set of irrelevant variables. In the proposed model, they assumed that the subset S c of irrelevant variables can be explained by a linear regression from a subset R of the clustering variables S. The model selection problem can be achieved by maximizing the quantity

crit( K, m, Ŝ, R) = arg max (K,m,S,R) BIC clust (x S |K, m) + BIC reg (x S c |x R ) , (2.70) where BIC clust (x S |K, m) = 2 log{p clust (x S |K, m, θ)} -λ S (K,m) log n, (2.71) with p clust (x S |K, m, θ) = K k=1 π k N (x S ; µ k , Σ k(m) ), θ = (π 1 , . . . , π K , µ 1 , . . . , µ K , Σ 1 , . . . , Σ K ) is the parameter vector, λ S (K,m)
is the number of free parameters of the (K, m) mixture model with card(S) variables. The BIC reg is given by

BIC reg (x S c |x R ) = 2 log{f reg (x S c |x R , B, Ω)} -ν (S,R) log n, (2.72)
( B, Ω) are the maximum likelihood estimate of the regression parameters and

ν (S,R) = (card(R) + 1)card(S c ) + card(S c )(card(S c ) + 1) 2
is the number of parameters for the regression model (in general case).

Later, [START_REF] Maugis | Variable selection in modelbased clustering: A general variable role modeling[END_REF] proposed a general variable role modeling. They split the features into three separated subsets of variable: the relevant variables (S), the irrelevant variables (U) which depend on a subset R of the relevant ones through a linear relationship and an other part (W) are independent of other variables. The data density can be decomposed into three parts as follows

p(x|K, m, r, l, V, θ) = K k=1 π k N (x S ; µ k , Σ k(m) ×N (x U ; a + x R B, Ω (r) ) × N (x W ; γ, Γ (l) ),
where θ is the full parameter vector and V = (S, R, U, W). The form of the regression covariance matrix Ω is denoted by r, which can be either spherical, diagonal or general. The form of the covariance matrix Γ of the independent variables W is denoted by l and can be spherical or diagonal. Thus, the model selection problem is solved by maximizing the following criterion (see also [START_REF] Maugis | Variable selection in modelbased clustering: A general variable role modeling[END_REF]

, Section 3)) crit(K, m, r, l, V ) = BIC clust (x S |K, m) + BIC reg (x U |x R , r) + BIC indep (x W |l), (2.73)
where BIC clust , BIC reg are given in (2.71) and (2.72). BIC indep represents the BIC criterion of the Gaussian model with the variables W.

For the identifiability of the SRUW model and the consistency of the variable selection, reader can refer to (Maugis et al., 2009b, Section 4). A procedure using embedded stepwise variable selection algorithms is used to identify the SRUW sets. Unfortunately, these procedures are limited as the number of variables is of the order of a few tens. Thus, it is not appropriate for large scale data.

Recently, [START_REF] Celeux | Variable selection in model-based clustering and discriminant analysis with a regularization approach[END_REF] proposed an alternative regularization approach of variable selection to overcome the drawback of the SRUW model. First, the variables are ranked with a Lasso-like procedure in order to avoid painfully slow stepwise algorithms. The variable roles are then determined using the stepwise procedures as in the SRUW model. In the next section, we will consider another way to combine variable selection and clustering by likelihood penalization which including the Lasso method used by [START_REF] Celeux | Variable selection in model-based clustering and discriminant analysis with a regularization approach[END_REF].

Variable selection by likelihood penalization

We take a survey on the penalized likelihood methods. These approaches aim at penalizing the clustering criteria in order to yield sparsity in the features. A general form for the penalized 2.4. CLUSTERING AND CLASSIFICATION IN HIGH-DIMENSIONAL SETTING log-likelihood function is as follows (2.74) where L(θ) is the log-likelihood function and p λ (θ) is the penalty function. Lasso [START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF], which will be described in the following Section 2.4.4, is one of the most widely used regularization method in literature. In GMM context, [START_REF] Pan | Penalized model-based clustering with application to variable selection[END_REF] proposed the Lassotype penalized log-likelihood to yield the sparsity of the mean vectors

P L p (θ) = L(θ) -p λ (θ),
P L(θ) = n i=1 log K k=1 π k N (x i ; µ k , Σ k ) -λ K k=1 µ k 1 , (2.75)
where

Σ k = Σ = diag(σ 2 1 , . . . , σ 2
p ) for all k ∈ {1, . . . , K} and λ is tuning parameter which decides the level of sparsity. Note that, the observations are standardized to have zero mean and unit variance for each variable j. If µ 1j = • • • = µ Kj = 0, then the variable jth cannot differentiate the components, hence deemed as noninformative and automatically excluded from clustering.

In a similar approach, Xie et al. (2008a) proposed a method dealing with the case of clustering specific diagonal covariance matrices leading to the following penalty function

p λ (θ) = λ 1 K k=1 µ k 1 + λ 2 K k=1 p j=1 |σ 2 kj -1|, (2.76)
with Σ k = diag(σ 2 k1 , . . . , σ 2 kp ) be the covariance matrix of the kth group. In this case, a second penalty term is used to force an estimate of σ 2 kj that is close to 1 to be exactly 1.

Finally, [START_REF] Zhou | Penalized model-based clustering with unconstrained covariance matrices[END_REF] considered a general covariance matrix Σ by relaxing the diagonal covariance matrix assumption. To facilitate estimating large and sparse covariance matrices, they employed the following penalty function

p λ (θ) = λ 1 K k=1 µ k 1 + λ 2 p j=1 p h=1 |Σ -1 jh |. (2.77)
This penalty function is then modified to permit varying cluster volumes and orientations

p λ (θ) = λ 1 K k=1 µ k 1 + λ 2 K k=1 j,h |Σ -1 k;jh |, (2.78)
where Σ k is the covariance matrix of the kth cluster. Note that, the penalty on the mean parameter is mainly for variable selection, while that for the covariance matrices is necessary for high-dimensional data. However, this leads to a difficult problem in estimating the covariance matrices, which are said to be positive-definite.

In the same spirit, [START_REF] Xie | Penalized mixtures of factor analyzers with application to clustering high-dimensional microarray data[END_REF] proposed a penalized MFA approach from the model introduced by [START_REF] Ghahramani | The EM algorithm for mixtures of factor analyzers[END_REF], where the noise covariance matrix is diagonal and common to all factor. By standardizing each variable to have variance at 1, one can treat these variables in a similar scale and thus penalize their mean parameters together by an ℓ 1 penalty. In addition, a variable j is irrelevant to all clusters if all µ jk , k = 1, . . . , K are 0 and all b kj. = (b k;j1 , . . . , b k;jq ), k = 1, . . . , K are required to be 0, where b k;jl is the value of the loading matrix Λ k at row jth and column lth. Hence, to realize more effective variable selection, it is natural to treat b k;j1 , . . . , b k;jq as a group of parameters, constructing a penalty that encourages all of them to be exactly 0. Therefore, the authors regularized the log-likelihood function of (2.61) with the following penalty function [START_REF] Pan | Penalized model-based clustering with application to variable selection[END_REF], is use to shrink a small estimate µ kj to be exactly 0, while p 2 (Λ), serving as a grouped variable penalty as in [START_REF] Xie | Variable selection in penalized model-based clustering via regularization on grouped parameters[END_REF] is used to shrink an estimate of factor loading vector b kj. that is close to 0 to be exactly 0.

p ( λ, γ)(θ) = λp 1 (µ) + γp 2 (Λ) = λ K k=1 µ k 1 + γ K k=1 p j=1 b kj. 2 , (2.79) where b kj. 2 = q l=1 b 2 k;jl . The ℓ 1 norm p 1 (µ), as in
As general, the tuning parameters of all these regularization methods are selected through a modified BIC criterion, which takes into account the level of sparsity in the model complexity term. However, the selection of the sparsity parameters is still an open issue. Among the related works, we refer the works of [START_REF] Witten | A framework for feature selection in clustering[END_REF] and [START_REF] Wang | Variable selection for model-based high-dimensional clustering and its application to microarray data[END_REF]. For the first approach, Witten and Tibshirani proposed a framework for sparse clustering based on a Lasso-type penalty to select the features. The method is then used to develop sparse K-means and sparse hierarchical clustering methods. For the remaining, Wang and Zhu suggested two regularization methods. The first one replaces the ℓ 1 -norm in [START_REF] Pan | Penalized model-based clustering with application to variable selection[END_REF] with the ℓ ∞ -norm to shrink the mean vectors and obtain a sparsity model

p λ (θ) = λ p j=1 (µ 1j , . . . , µ Kj ) ∞ , (2.80) where (µ 1j , . . . , µ Kj ) ∞ = max k (|µ 1j |, . . . , |µ Kj |).
Different from the ℓ 1 -norm, the ℓ ∞ -norm penalizes the maximum absolute value of µ kj , k = 1, . . . , K, for the jth variable. If the maximum of |µ kj | is reduced to 0, all µ kj are automatically reduced to 0. The jth variable is therefore irrelevant. However, this ℓ ∞ -norm penalty tends to reduce the µ kj , k = 1, . . . , K into the same magnitude. This motivates them to propose the second penalty function. First, µ kj is reparameterized in a more general form

µ kj = γ j η kj , k = 1, . . . , K; j = 1, . . . , p, (2.81)
where γ j ≥ 0. Secondly, they considered the following hierarchically penalized GMM (2.82) subject to γ j ≥ 0. If γ j is reduced to zero then all µ kj for the jth variable will be equal to zero. Otherwise, some of θ kj hence some of the µ kj , k = 1, . . . , K, still have the possibility of being zero; in this sense, the hierarchical penalty keeps the flexibility of the ℓ 1 -norm penalty.

P L(θ) = L(θ) -λ γ p j=1 γ j -λ η K k=1 p j=1 |η kj |,

Lasso regularization towards the mixture approach

In this section, we introduce the Lasso estimator for the linear regression problem, before considering it for the mixture of regression models. This method plays a key role in our research.

A method for solving the Lasso is also mentioned at the end of this section.

The Lasso regularization

Lasso is the ℓ 1 regularization of linear regression model, in which we assume that we are given n observations (x 1 , y 1 ), . . . , (x n , y n ), where the response variable y i ∈ R is related to the pdimensional vector of predictors x i ∈ R p via the linear combination of the predictors x i = (x i1 , . . . , x ip ) T as:

f (x i ) = β 0 + p j=1 β j x ij . (2.83)
Here, β 0 is the intercept term β 0 ∈ R and β ∈ R p is the vector of regression weights. Without loss of generality, assume that β 0 = 0.

The least-squares estimator for β is based on minimizing the square-error loss:

arg min β y -Xβ 2 2 , (2.84)
where y = (y 1 , . . . , y n ) T and

X =       x 11 x 12 . . . x 1p x 21 x 22 . . . x 2p . . . . . . . . . . . . x n1 x n2 . . . x np       .
It is not hard to check that the solution of (2.84), which is called the least-squares estimator, is given by βLS = (X T X) -1 X T y.

(2.85) However, it is well known in literature that if the square matrix X T X has one or more small eigenvalues, then the distance from the estimator βLS and the true parameter vector β will tend to be large. Especially, since n < p the least-squares estimator poses a problem: the solution is not unique because the rank of X T X is smaller than p.

To tackle the ill-conditional condition of X T X, [START_REF] Hoerl | Ridge regression: Biased estimation for nonorthogonal problems[END_REF] proposed an ℓ 2 regularization method which is named ridge regression. Basically, the ridge regression solves the follows problem βridge = arg min

β y -Xβ 2 2 , subject to β 2 ≤ t. (2.86)
Unfortunately, although ridge regression shrinks the regression coefficients but this method still poses a drawback since none of the ridge regression equals to zero. Therefore, all the features x j s still be used to construct the regression function. For more details on the ridge regression, readers can refer to [START_REF] Hoerl | Ridge analysis 25 years later[END_REF], [START_REF] Friedman | The elements of statistical learning[END_REF]. [START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF] introduced a minutely method to overcome the drawback of least square estimation and also the ridge regression. The idea lies in the fact that replacing the Euclidean norm in constraint of the ridge regression β 2 ≤ t with the ℓ 1 constraint β 1 ≤ t. This model is called Least Absolute Shrinkage and Selection Operator (Lasso). It was directly inspired by the nonnegative garrote estimator of [START_REF] Breiman | Better subset selection using the non-negative garotte[END_REF]. As its name, this method provides two advantages: first it shrinks some coefficients and secondly, it sets other coefficients to 0. Therefore it retains the good features of both subset selection and ridge regression. Figure 2.3 depicts the Lasso and ridge regression for the vector x is of the size 2. Both methods find the first point, where the elliptical contours hit the constraint region. Since the Lasso constraint region has corners; if the hit point is one of these corners then one of the two parameters is equal to zero.

It is interesting to look back to the classical feature selection method based on the L 0 norm as a natural approach for features selection, which is described as follows:

arg min β y -Xβ 2 2 , subject to β 0 ≤ t, (2.87)
where the L 0 norm is defined by

β 0 card{j ∈ {1, . . . , p} : β j = 0}.
This model has paid a lot of attentions in literature (see [START_REF] Foster | The risk inflation criterion for multiple regression[END_REF]). Unfortunately, the constraint is non-convex and the L 0 regularization method is proofed to be an N P -hard problem (see [START_REF] Natarajan | Sparse approximate solutions to linear systems[END_REF]). Hence, it is ineffective to use this method in application and solving (2.87) is still a challenge. Recently, [START_REF] Frommlet | An adaptive ridge procedure for l 0 regularization[END_REF] introduce an adaptive ridge procedure (AR), where iteratively weighted ridge problems are solved whose weights are updated in such a way that the procedure converges towards selection with L 0 penalties.

Going back to the ℓ 1 regularized method, the Lasso estimate requires solving an optimization problem, which can be described as arg min

β y -β 0 -Xβ 2 2 , subject to β 1 ≤ t. (2.88)
For convenience, assuming that all features are standardized to have mean zero and unit variance. We also assume that the outcome values y i have been centered. By doing this, we can omit the intercept β 0 . Hence, without loss of generally, the Lasso can be rewritten as arg min

β y -Xβ 2 2 , subject to β 1 ≤ t. (2.89)
It is often convenient to rewrite the Lasso problem in the Lagrange form, arg min

β 1 2 y -Xβ 2 2 + λ β 1 . (2.90)
The value of λ will decide the sparsity of the model. As proved in Section A.4.1, the smallest value of λ such that the regression coefficients estimated by Lasso (2.90) are all equal to zero is

λ max = max j | y, x j |,
where y = (y 1 , . . . , y n ) T , x j = (x 1j , x 2j , . . . , x nj ) T . y, x j is the inner product between y and x j , y, x j = y T x j .

The bound of t in (2.88) controls the complexity of the model: the larger values of t allow the model to adapt more closely to the training data. Conversely, the smaller values of t will restrict the parameters more and leading to sparser but the interpretable model fit the data less closely. Therefore, a natural question arises for the Lasso regression: "how to choose t or λ?". A procedure know as cross-validation is a common tool to decide the value of the tuning parameter λ. For more details, (Hastie et al., 2015, Sec. 2.3) is a helpful text for the cross-validation procedure.

Lemma 2.4.2 (Properties of the Lasso solutions [START_REF] Tibshirani | The Lasso problem and uniqueness[END_REF])). For some λ ≥ 0, suppose the β, η two solutions of (2.90) with common optimal value c * then:

• X β = Xη, i.e, the two solutions must yield the same predicted values.

• If λ > 0 then β 1 = η 1 .
The proof of this lemma can be found in Appendix A.4.2. As shown in Lemma 2.4.2, the fitted values X β are unique. However, the solution β may not be unique. Consider an example with two predictors x 1 , x 2 and response y. Assume that the Lasso solution coefficients β at λ are ( β1 , β2 ). If we include the third predictor x 3 which is an identical copy of the second predictor, i.e, x 3 = x 2 and consider the Lasso problem arg min

(β 1 ,β 2 ,β 3 ) 1 2 n i=1 (y i -β 1 x i1 -β 2 x i2 -β 3 x i3 ) 2 + λ 3 j=1 |β j |.
(2.91)

This optimization problem can be rewritten as

arg min {β} 1 2 n i=1 (y i -β 1 x i1 -(β 2 + β 3 )x i2 ) 2 + λ(|β j | + |β 2 + β 3 |) + λ(|β 2 | + |β 3 | -|β 2 + β 3 |) .
(2.92)

It turns out that, for any α ∈ [0, 1] the triplet ( β1 , α β2 , (1 -α) β2
) is a solution of (2.91). For this model, there is an infinite family of solutions.

In general, when λ > 0, [START_REF] Tibshirani | The Lasso problem and uniqueness[END_REF] showed that if the matrix X ∈ R n×p has columns in general position, then the Lasso solutions are unique. To be precise, the columns {x j } p j=1 of X are in general position if no k-dimensional subspace L ⊆ R n , for k < min{n, p}, contains more than k + 1 elements of the set {±x 1 , . . . , ±x p }, excluding antipodal pairs. This means the affine span of any k + 1 points σ 1 x j 1 , . . . , σ k+1 x j k+1 (j 1 , . . . , j k+1 ∈ {1, . . . , p}), for arbitrary signs σ 1 , . . . , σ k+1 ∈ {-1, +1} does not contain any element of {±x j : j = j 1 , . . . , j k+1 }. For the complete proof of this property see (Tibshirani, 2013, Lemma 2, 3). Hence, if the data matrix X are drawn from a continuous probability distribution, then with probability one the data are in general position and the Lasso solutions will be unique. The non-uniqueness of the Lasso solutions can occur only with discrete-valued data. In the previous example, the affine is generated from {±x 1 , ±x 2 } also contains ±x 3 . Therefore, the data are not in general position.

There has been an enormous amount theoretical works analyzing the performance of the Lasso. For the related works, some references are [START_REF] Bickel | Simultaneous analysis of Lasso and dantzig selector[END_REF]; [START_REF] Meinshausen | High-dimensional graphs and variable selection with the Lasso[END_REF]; [START_REF] Donoho | Stable recovery of sparse overcomplete representations in the presence of noise[END_REF]; [START_REF] Zhao | On model selection consistency of Lasso[END_REF]; [START_REF] Wainwright | Sharp thresholds for high-dimensional and noisy sparsity recovery using ℓ 1 -constrained quadratic programming (Lasso)[END_REF]; [START_REF] Massart | The Lasso as an ℓ 1 -ball model selection procedure[END_REF]; a helpful book for these kind of results is [START_REF] Bühlmann | Statistics for high-dimensional data: methods, theory and applications[END_REF].

Solving the Lasso optimization problem

The Lasso problem is a convex program, specifically a quadratic program with a convex constraint. As such, there are many sophisticated quadratic program methods for solving the Lasso. In this thesis, we use one of the simple but effective methods for solving the Lasso: coordinate descent algorithm. For more details on this method, viewers can the reader can be referred to [START_REF] Boyd | Convex optimization[END_REF]; [START_REF] Hastie | Statistical Learning with Sparsity: The Lasso and Generalizations[END_REF].

f (β) = g(β) + h(β)
where g(.) is a differentiable convex function, h(.) is convex but not necessarily differentiable.

Coordinate descent is an iterative algorithm that updates from β (t) to β (t+1) by choosing a single coordinate to update, and then performing a univariate minimization over this coordinate. If coordinate k is chosen at iteration t, then

β (t+1) k = arg min β k f (β (t) 1 , . . . , β (t) k-1 , β k , β (t) k+1 , . . . , β (t)
p ), (2.93) and β

(t+1) j = β (t+1) j
for j = k. The coordinate descent can get "stuck", and fail to reach the global minimum of f . A counter example is shown in (Hastie et al., 2015, Section 5.4). However, if the cost function f has the additive decomposition

f (β 1 , . . . , β p ) = g(β 1 , . . . , β p ) + p j=1 h j (β j ) (2.94)
where h j : R → R are convex, then with some conditions including the separate structure of h(β) [START_REF] Tseng | Convergence of a block coordinate descent method for nondifferentiable minimization[END_REF] and [START_REF] Wu | Coordinate descent algorithms for Lasso penalized regression[END_REF] show that the coordinate descent algorithm converges to the minimum point. For the Lasso problem, we have

f (β) = g(β) + p j=1 h(β j ), with g(β) = 1 2 n i=1 (y i -x T i β) 2 and h(β j ) = λ|β j |. Updating β (t+1) j
using the coordinate descent procedure is done by solving this problem

β (t+1) j = arg min β j 1 2 n i=1 β j x ij + ( k =j β (t) k x ik -y i ) 2 +λ|β j |. (2.95)
Then the solution for (2.95) satisfies

β (t+1) j = S λ n i=1 r j i x ij n i=1 x 2 ij , (2.96) with r j i = y i - k =j β (t)
k x ik and S λ (.) is a soft-thresholding operator defined by S λ (x) = sign(x)(|x|λ) + and (x) + a shorthand for max{x, 0}. Other parameters are kept while updating β j .

The coordinate descent procedure simply solves (2.95) in a cyclical way, iterating over j = 1, 2, . . . , p, 1, 2, . . . . In general, the coordinate descent algorithm for solving the Lasso can be interpreted as below:

Algorithm 1 Coordinate descent method for solving the Lasso

1: β (0) . 2: repeat 3: for j = 1 to p do 4:
Update β (t+1) j using the soft-thresholding operator in (2.96).

5:

For k = j then β

(t+1) k = β (t) k . 6: end for 7:
Evaluate the objective function. 8: until the stopping criterion is satisfied.

For the standard Lasso regression model arg min

(β 0 ,β) 1 2 n i=1 (y i -β 0 -x T i β) 2 + λ β 1 , (2.97)
one just need to modify the updating of β 0 at each step using the value

β (t+1) 0 = n i=1 (y i -x T i β (t+1) ) n •
Example 2.4.1. In this example, n = 150 data points were generated with p = 6 features,

X j i.i.d ∼ U [0, 1]. The respond variable Y is of the form Y = 1 + 2X 1 -X 4 + 0.5X 6 + ε,
where ε ∼ N (0, 0.5 2 ) and λ = 2.8. Starting from (β 0 , β T ) T = 0 with the objective function value 273.88630, after 187 loops the coordinate descent algorithm stops at ( β0 (λ), βT (λ)) T = (0.9877018, 1.7612036, 0, 0, -0.5732187, 0, 0.3337205) T , with the minimum value 26.51752. It is clear that, in this case the Lasso estimator provides the zero coefficients coincident with the true zero parameters of the model. For the non-zero coefficients (excepts the intercept), there are biases between these parameters with the true ones cause by penalty function. The changing of the objective function after each iteration can be observed from Figure 2.4.

Least angle regression (LARS) is another efficient algorithm for solving the Lasso, which is proposed by [START_REF] Efron | Least angle regression[END_REF]. It still requires to compute the inverse matrix in each loop, hence does not scale up to large problems as well as the coordinate descent method. An MM version for solving the regularized regression problems has been investigated in [START_REF] Hunter | Variable selection using MM algorithms[END_REF]. Chapter 5 of [START_REF] Hastie | Statistical Learning with Sparsity: The Lasso and Generalizations[END_REF] provides a general review for other optimization methods which can be used to solve the Lasso.

Regularized mixtures of regression models

In the applications of the mixture of regression models, many features are often used, and the contribution of each feature to the response variable Y is different from one component to another of the model. As the number of features and components in the mixture models increases the cost of computing the Akiake information criterion [START_REF] Akaike | A new look at the statistical model identification[END_REF] and the BIC also become more expensive. Hence, these methods may not be not idea in certain cases. In this section, we take some recent works that use penalized likelihood approaches for variable selection in mixture of regression models. Some authors such as [START_REF] Khalili | Variable selection in finite mixture of regression models[END_REF]Städler et al., 2010a;[START_REF] Khalili | Regularization in finite mixture of regression models with diverging number of parameters[END_REF][START_REF] Hui | Multi-species distribution modeling using penalized mixture of regressions[END_REF] tackle the problem of variable section for mixture of regressions models using Lasso-type regularization methods. In the similar context, [START_REF] Devijver | An ℓ 1 -oracle inequality for the Lasso in multivariate finite mixture of multivariate gaussian regression models[END_REF] considered the Lasso-penalized methods for mixture of multivariate Gaussian regression models. For the MoEs models, several works, such as [START_REF] Khalili | New estimation and feature selection methods in mixture-of-experts models[END_REF][START_REF] Peralta | Embedded local feature selection within mixture of experts[END_REF][START_REF] Jiang | Model selection for the localized mixture of experts models[END_REF] in particular, introduced a penalty term in the log-likelihood function in order to yield the sparsity in the features both for the experts and for the gating network. The reader can also refer to [START_REF] Khalili | An overview of the new feature selection methods in finite mixture of regression models[END_REF], which provides a comprehensive overview of the feature selection methods in mixture of regression models.

Regularized mixture of regression models

Khalili and Chen ( 2007) is considered as the earliest variable selection method in mixture of linear regression models (MLR) using penalized likelihood approach. Let (x 1 , y 1 ), . . . , (x n , y n ) be a sample of observations from the MLR model in (2.25). The (conditional) log-likelihood function of θ is given by

L(θ) = n i=1 log K k=1 π k N (y i ; β k0 + x T i β k , σ k ) . (2.98)
When the effect of a jth feature is not significant, the corresponding ordinary maximum likelihood estimate is often close to, but not equal to 0. To exclude the role of this feature, these author consider the penalized log-likelihood function as

P L(θ) = L(θ) - K k=1 π k p j=1
p n (β kj ; λ nk ), (2.99) where p n (β kj ; λ nk ) could be one of the following penalty functions:

• ℓ 1 -norm penalty (The Lasso):

p n (β kj ; λ nk ) = λ nk √ n|β kj |.
(2.100)

• SCAD penalty (Smooth clipped absolute deviation penalty [START_REF] Fan | Variable selection via nonconcave penalized likelihood and its oracle properties[END_REF]):

p nk (β kj ; λ nk ) √ n =          λ nk |β kj |, |β kj | ≤ λ nk -(β 2 kj -2aλ nk |β kj | + λ 2 nk )/[2(a -1)], λ nk < |β kj | ≤ aλ nk λ 2 nk (a + 1)/2, |β kj | > aλ nk , (2.101)
for some constant a > 2.

• HARD penalty (hard thresholding penalty Antoniadis (1997)):

p nk (β kj ) = λ 2 nk -( √ n|β kj | -λ nk ) 2 I( √ n|β kj | < λ nk ).
The hope is that with a proper choice of the penalty functions, if some of the regression coefficients β kj are zero, then their corresponding estimator βkj are also zero. Although the Lasso has many attractive properties, the shrinkage introduced by it results in significant bias toward 0 for large regression coefficients. The remains are nonconvex penalty functions. The SCAD has the so-called oracle property, i.e., in asymptotic sense, it performs as well as the oracle procedure in variable selection; namely, it works as well as if the correct submodel was known.

Note that the amount of penalty on each regression parameters β kj is proportional to the mixing probability π k , which is a common practice in relating the amount of penalty to the sample size. Under standard regularity conditions as well as the certain conditions on the penalty function, [START_REF] Khalili | Variable selection in finite mixture of regression models[END_REF] showed the estimator θn of (2.99) satisfies (see (Khalili and 2.5. REGULARIZED MIXTURES OF REGRESSION MODELS Chen, 2007, Theorem 1))

θn

-θ 0 = O p (n -1/2 (1 + b n )),
where . represents the Euclidean norm, θ 0 is the true vector of parameters in the true MLR model underlying the data and

b n = max k,j {|p ′ nk (β 0 kj )|/ √ n : β 0 kj = 0}.
In addition, if the penalty function guarantees that, for

N n = {β : 0 < β ≤ n -1/2 log n}, lim n→∞ inf β∈Nn p ′ nk (β)/ √ n = ∞,
then θn also provides the following properties (see also [START_REF] Khalili | Variable selection in finite mixture of regression models[END_REF], Theorem 2)):

• Sparsity: P( βk2 = 0) → 1, for k = 1, . . . , K, where βk2 is the vector of zero regression coefficients in the kth component of the model.

• Asymptotic normality:

√ n I(θ 01 ) - p ′′ (θ 01 ) n ( θn1 -θ 01 ) + p ′ (θ 01 ) n d → N (0, I(θ 01 )),
where p ′ (.) and p ′′ (.) are the first and second derivatives of the penalty function with respect to β kj 's, and I(θ 01 ) is the Fisher information matrix computed under the reduced model when all zero effects are removed.

To obtain these properties, the number of mixture component K of the model is known. In applications, one can use the BIC or other criteria to select K. Another strategy of regularization to perform simulatenous parameter estimation and selection of K is presented in [START_REF] Chamroukhi | Robust EM algorithm for model-based curve clustering[END_REF][START_REF] Chamroukhi | Unsupervised learning of regression mixture models with unknown number of components[END_REF].

Parameter estimation is done via the EM algorithm. To maximize the well-known Q-function in the M-step, the authors approximated the penalty function by a local quadratic approximation using [START_REF] Hunter | Variable selection using MM algorithms[END_REF] suggestion

p nk (β) ≈ p nk (β 0 ) + p ′ n k(β 0 ) 2(β 0 + ǫ) (β 2 -β 2 0 ), (2.102)
in a neighborhood of β 0 . A Newton-Raphson algorithm can then be used to maximize the penalized Q-function, where each iteration updates the local quadratic approximation. The drawback lies in the fact that using Newton-Raphson algorithm for large-scale data (p is large) may not be an appropriate choice. In addition, one must choose a threshold δ and declares a coefficient is zero if it values smaller than δ. Therefore, the selection of ǫ in (2.102) and δ affects the sparsity of the model and should be consider carefully. One important thing to mention is that maximizing the Q-function with respect to the mixing proportions π k , k = 1, . . . , K is still a challenge. In their work, the authors keep the same update of π k as a non penalized case and it still works quite well.

Later, [START_REF] Khalili | Regularization in finite mixture of regression models with diverging number of parameters[END_REF] proposed elastic-net type penalties for finite mixture regression models with a diverging number of covariates in each component. Their penalized log-likelihood function is as follows

P L n (θ) = L n (θ) -p n (θ) (2.103)
with the penalty function

p n (θ) = K k=1 π k p j=1 r n (β kj ; λ nk ) + τ n 2 K k=1 π k p j=1 β 2 kj , (2.104)
for some tuning parameter τ n ≥ 0, and r n (β kj ; λ nk ) is a nonnegative penalty function indexed by some tuning parameter λ nk ≥ 0. In fact, the authors considered three well-known regularization functions for r n (β kj ; λ nk ): the Lasso and SCAD as in (2.100), (2.101) and MCP (Zhang ( 2010)) which is given by

r n (β kj ; λ nk ) n =    λ nk |β kj | - β 2 kj 2γλ nk , |β kj | < γλ nk γλ 2 nk /2, |β kj | ≥ γλ nk . (2.105)
The parameter γ in MCP controls the concavity of the penalty. Moreover, if γ → ∞ the penalty becomes Lasso, if γ → 0 + then it becomes the L 0 penalty. Using the similar approach in [START_REF] Khalili | Variable selection in finite mixture of regression models[END_REF], the authors draw some asymptotic properties for their models. EM algorithms for parameter estimation are also given using the same strategy in [START_REF] Khalili | Variable selection in finite mixture of regression models[END_REF]. The methods are then applied to study the Gaussian and Poisson finite mixture of regression models.

Similarly, Städler et al. (2010a) applied the adaptive Lasso [START_REF] Zou | The adaptive Lasso and its oracle properties[END_REF]) with one tuning parameter for the mixture of linear regression models. In addition, some asymptotic theory and oracle inequalities are also presented. Conversely to this approach, [START_REF] Meynet | An ℓ 1 -oracle inequality for the Lasso in finite mixture gaussian regression models[END_REF] presented a complementary result to Städler et al. (2010a) by studying the Lasso for its ℓ 1 -regularization properties rather than considering it as a variable selection procedure. Meynet established oracle inequality for the Lasso in finite mixture Gaussian regression models, which can be seen as an extension of [START_REF] Massart | The Lasso as an ℓ 1 -ball model selection procedure[END_REF]. Devijver (2015) extended Meynet's ℓ 1 -oracle inequality for the mixture of multivariate Gaussian regression models. To exploit the grouped structure of covariates in the MLR models, [START_REF] Hui | Multi-species distribution modeling using penalized mixture of regressions[END_REF] introduced two regularization models MIXGL1 and MIXGL2 based on a modification of the group Lasso (Yuan and Lin (2006a)). With some regularity conditions, the oracle properties for their proposed models are also studied. Finally, it is interesting to present the work of Lloyd-Jones et al. ( 2018), who introduced an MM algorithm for maximizing the Lasso-penalized likelihood function of the MLR model.

Regularized mixture of experts models

A general framework for the regularized MLR model was also proposed by [START_REF] Khalili | New estimation and feature selection methods in mixture-of-experts models[END_REF] which focuses on MoE model. Since each mixing proportion is replaced by a softmax function of the variates, the effect of each feature appears both in the experts network and in the mixing proportion functions (the gating network). Hence, for feature selection, two extra penalty functions are applied to the ℓ 2 -penalized log likelihood function. His proposal penalized log-likelihood is then given by

P L(θ) = n i=1 log K k=1 π k (w; x i )N (y i ; β k0 + x T i β k , σ 2 k ) -p n (θ), (2.106)
with π k (w; x i ) is a softmax function in (2.37),and

p n (θ) = K k=1 p j=1 r n (β kj ; λ k ) + K-1 k=1 p j=1 r n (w kj ; γ k ) + τ n 2 K-1 k=1 w k 2 2 , (2.107)
where r n (.; .) is the Lasso or SCAD functions. Adding an ℓ 2 -norm of the mixing probabilities to the log-likelihood function controls the large positive and negative estimates of the regression coefficients corresponding to the correlated features. This behavior can be observed in logistic/multinomial regression when the number of potential features is large and highly correlated; see [START_REF] Park | Penalized logistic regression for detecting gene interactions[END_REF] and [START_REF] Bunea | Honest variable selection in linear and logistic regression models via ℓ 1 and ℓ 1 + ℓ 2 penalization[END_REF]. However, this also affects the sparsity of the regularization model.

By extending the theoretical developments for MLR models in [START_REF] Khalili | Variable selection in finite mixture of regression models[END_REF], a standard asymptotic theory for MoE models is established (see Theorem 2, Theorem 3 of [START_REF] Khalili | New estimation and feature selection methods in mixture-of-experts models[END_REF]). The asymptotic properties of the estimator θn depend on the choice of the penalty r n (.; .) and the tuning parameter τ n accompanying the ℓ 2 -norm. By taking an appropriate choice of these tuning parameters in SCAD, the estimator θn of the penalized log-likelihood is both consistent in feature selection and maintains root-n consistency. In contradiction with SCAD, for Lasso, the estimator θn cannot achieve both properties simultaneously. It is worth to point out that the proposed methodology of [START_REF] Khalili | New estimation and feature selection methods in mixture-of-experts models[END_REF] can be used for feature selection in the high-dimensional situations (p ≫ n). However, the statistical properties of the method in high-dimensional context such as consistency and oracle inequality still be open questions.

For maximizing the regularized log-likelihood, an appropriate EM algorithm combines with the Newton-Raphson method was introduced. Using the suggestion of [START_REF] Hunter | Variable selection using MM algorithms[END_REF], the penalty functions for both the gating network and the experts network are approximated with their local quadratic functions. After that, the Newton-Raphson can be used to update these coefficients in the M-step. However, some drawbacks from the EM algorithm for regularized MLR in [START_REF] Khalili | Variable selection in finite mixture of regression models[END_REF] still appear here. One must choose a threshold δ to declare zero coefficient and an addition parameter ǫ to avoid numerical instability of the algorithm due to the small values of some of the coefficient in the demominator of the approximation (2.102).

In a similar context, Peralta and Soto (2014) studied the ℓ 1 -regularized MoE for multinomial logit models. Hence, they assume that each expert component follows a multinomial distribution with R levels (R ≥ 2), i.e.,

Y i |Z i = z i , x i ∼ Mult(1; ϕ z i 1 (x i ; β z i ), . . . , ϕ z i R (x i ; β z i ))
where

ϕ kr (x i ; β k ) = P(y i = r|x i ; z i = k; β k ) = exp(β kr0 + x T i β kr ) R l=1 exp(β kl0 + x T i β kl ) , r ∈ {1, . . . , R}•
The ℓ 1 regularized observed data log-likelihood is given by

P L(θ) = n i=1 log K k=1 π k (w; x i )Mult(y i ; ϕ k1 (x i ; β k ), . . . , ϕ kR (x i ; β k )) -λ K-1 k=1 p j=1 |w kj | -γ K k=1 R-1 r=1 p j=1 |β krj |. (2.108)
For parameter estimation, these authors proposed an EM-based algorithm. Especially, for maximizing the penalized Q-function with respect to the gating network parameters in the M-step

Q(w; θ [q] ) = n i=1 K k=1 τ [q] ik log π k (x i ; w) - K-1 k=1 γ k w k 1 , (2.109)
where τ

[q]

ik is the conditional probability that the data pair (x i , y i ) is generated by the kth expert with respect to the current parameter vector θ [q] , they approximate (2.109) by using a transformation that implies inverting the softmax function. Hence, updating w

[q+1] k can be obtained by solving an approximated Lasso problem

w [q+1] k = arg min w k n i=1 (log τ [q] ik -w k0 -x T i w k ) 2 , subject to w k 1 ≤ λ. (2.110)
Unfortunately, this approximation does not guarantee that the penalized log-likelihood will increase after each update step. In is worth to noting that, the authors do not provide any evidence to prove this critical property of their EM algorithm.

Recently, to overcome the difficulty of updating the gating network, [START_REF] Jiang | Model selection for the localized mixture of experts models[END_REF] based on the work of [START_REF] Xu | An alternative model for mixtures of experts[END_REF] proposed a regularization method for the localized MoE models. Localized MoE assumes the latent discrete variable Z follows a multinomial distribution with

P(Z = k) = π k , k = 1, . . . , K and K k=1 π k = 1. 51 2.

REGULARIZED MIXTURES OF REGRESSION MODELS

Moreover, given the label variable Z = k, the random variate X follows a multivariate Gaussian distribution, i.e.,

X|Z = k ∼ N (µ k , Σ k ).
Then the estimation of localized MoE is based on joint distribution p(X, Y ), instead of the conditional distribution p(Y |X) in a standard MoE. The joint distribution p(X, Y ) is as follows

p(X, Y ) = K k=1 p(X, Y, Z = k) = K k=1 p(Z = k)p(X|Z = k)p(Y |X, Z = k) = K k=1 π k N (X; µ k , Σ k )N (y; β k0 + x T β k , σ k ).
(2.111)

Hence, the mixing proportion P(Z = k|X = x) has a different form as

P(Z = k|X = x) = P(Z = k)P(X = x|Z = k) K l=1 P(Z = l)P(X = x|Z = l) = π k N (X; µ k , Σ k ) K l=1 π l N (X; µ l , Σ l ) • (2.112)
Maximum likelihood estimator is done on the joint distribution

L(θ) = n i=1 log K k=1 π k N (x i ; µ k , Σ k )N (y i ; β k0 + x T i β k , σ k ) , (2.113)
where θ = (π 1 , . . . , π K-1 , µ 1 , . . . , µ K , Σ 1 , . . . , Σ K , (β 10 , β 1 ), . . . , (β K0 , β K ), σ 1 , . . . , σ K ) is the parameter vector of the model. An EM algorithm for maximizing L(θ) in (2.113) can be referred to [START_REF] Xu | An alternative model for mixtures of experts[END_REF], and a very recent algorithm for the regularized estimation is presented in [START_REF] Chamroukhi | MEteorits: Mixtures-of-ExperTs modEling for cOmplex non-noRmal dIsTributions. R package version 0[END_REF].

In order to simultaneously select the number of experts and influential variables of these experts, [START_REF] Jiang | Model selection for the localized mixture of experts models[END_REF] consider the following penalized log-likelihood function

P L(θ) = L(θ) -P 1 (π) -P 2 (β),
where the first penalty P 1 (.) is served to select the number of component. Following [START_REF] Huang | Model selection for gaussian mixture models[END_REF], they consider

P 1 (π) = nλ K k=1 log(ǫ + p λ,a 1 (π k )) -log ǫ) ,

52

CHAPTER 2. STATE OF THE ART and p λ,a 1 (π k )) is the SCAD penalty with the pair (λ, a 1 ). P 2 (β) is also a SCAD penalty for variable selection

P 2 (β) = K k=1 p j=1 p γ,a 2 (|β kj |),
with p γ,a 2 (.) is the SCAD penalty with the pair (λ, a 1 ).

It is important to mention that, in this regularization method, each expert's covariance matrix Σ k still updates as in normal. However, for a large value of the dimension p this task becomes a challenge. An efficient method for estimating covariance matrices Σ k should be considered. In a similar way, Morvan et al. ( 2019) considered a penalized likelihood method for localized MoE models with logistic expert. Here, they assume that Y being a binary response in {0, 1} and satisfies

Y |X = x, Z = k ∼ B 1, exp(β k0 + x T β k ) 1 + exp(β k0 + x T β k )
•

In their proposed model, an ℓ 1 -regularized estimation strategy is adopted to obtain sparse estimates of both the precision matrices Θ 1 , . . . , Θ K of the predictors and the regression coefficients

β 1 , . . . , β K , where Θ k = Σ -1 k .
Therefore, the penalized log-likelihood function is given by

P L(θ) = L(θ) - K k=1 λ k β 1 - K k=1 ρ k Θ k 1 ,
where β 1 = p j=1 |β kj | and Θ k 1 denotes the sum of the absolute values of Θ k . Note that, the penalty on the regression coefficients is mainly for variable selection, while that for the covariance matrices is necessary to have a sparse presentation of the inverse covariance matrices. However, this leads to a difficult problem in estimating these matrices, which are said to be positive-definite.

Conclusion

In this chapter, we reviewed FMMs used for modeling, clustering and regression for heterogeneous data. We also consider the EM algorithms with are widely use to estimate the parameters of these models.

In large-scale scenario, several approaches have been proposed for model-based clustering with FMMs. Classical methods such as parsimonious Gaussian models lay a lot of assumptions on the data and with many submodels it becomes a difficult problem to decide which model for a given data set. Also, they do not take into account that the structure of association among the variables can vary from one cluster to the other. Mixture of factor analyzers is another ways to deal with high dimensional data. Unfortunately, this method does not focus on the feature selection task which is one of the main missions in large-scale setting. In such a problem, variable selection methods such as SRUW model, penalized models seem to be potential 2.6. CONCLUSION approaches. However, there are some aspects should be study more such as an efficient method to have a sparse representation of the covariance matrices. Recently, [START_REF] Fop | Model-based clustering with sparse covariance matrices[END_REF] proposed an interesting method to follow in order to directly obtain sparse component covariance matrices.

For the regression problems, we can see that regularization methods for feature selection seem to be very promising in a MoE context. However, there are some open questions that should be answer. In the theoretical point of view, the statistical properties of the methods in highdimensional situations require more advanced theoretical developments, which is an interesting subject. From the application point of view, estimating the parameters using the EM algorithm requires a lot of advanced results from mathematical optimization, especially for solving the M-step with non-convex penalty functions. Some common methods aim at approximating the penalty function and combine with the Newton-Raphson procedure to update the parameters. Unfortunately, Newton-Raphson procedure is not an appropriate choice in high-dimensional setting. Regularization methods for localized MoE are interesting approaches to avoid updating the gating network parameters in MoE. However, estimating the component covariance matrices Σ k in large-scale situation should be studied more. In this context, a regularization approach likes Morvan et al. ( 2019) is an interesting suggestion. Another way to tackle this difficult is by considering the parsimonious Gaussian models [START_REF] Banfield | Model-based gaussian and non-gaussian clustering[END_REF][START_REF] Celeux | Gaussian parsimonious clustering models[END_REF] to parametrize the covariance matrices Σ k or to represent sparse covariance matrices via graph models [START_REF] Fop | Model-based clustering with sparse covariance matrices[END_REF]. Besides that, updating the experts network parameters with nonconvex penalty functions is not an easy task. One important thing should be reminded is that, to the best of our knowledge, until now there is no theoretical to support the regularization methods for localized MoE. Hence, establishing the statistical properties of the methods is the subject of future research.

INTRODUCTION

Introduction

In the previous chapter, we studied the Mixture of experts (MoE) introduced by [START_REF] Jacobs | Adaptive mixtures of local experts[END_REF] for regression, clustering and classification. MoE belong to the family of mixture models [START_REF] Titterington | Statistical analysis of finite mixture distributions[END_REF]McLachlan and Peel., 2000;[START_REF] Frühwirth-Schnatter | Finite Mixture and Markov Switching Models[END_REF] and is a fully conditional mixture model where both the mixing proportions, i.e, the gating network, and the components densities, i.e, the experts network, depend on the inputs. A general review of the MoE models and their applications can be found in [START_REF] Nguyen | Practical and theoretical aspects of mixture-ofexperts modeling: An overview[END_REF]. While the MoE modeling with maximum likelihood estimation (MLE) is widely used, its application in high-dimensional problems is still challenging due to the well-known problem of the ML estimator in such a setting. Indeed, in high-dimensional setting, the features can be correlated and thus the actual features that explain the problem reside in a low-dimensional space. Hence, there is a need to select a subset of the potentially large number of features, that really explain the data.

To avoid singularities and degeneracies of the MLE as highlighted namely in [START_REF] Stephens | Bayesian methods for mixtures of normal distributions[END_REF]; [START_REF] Snoussi | Degeneracy and likelihood penalization in multivariate gaussian mixture models[END_REF]; [START_REF] Fraley | Bayesian regularization for normal mixture estimation and model-based clustering[END_REF], one can regularize the likelihood through a prior distribution over the model parameter space. A better fitting can therefore be achieved by regularizing the objective function so that to encourage sparse solutions. However, feature selection by regularized inference encourages sparse solutions, while having a reasonable computational cost. Several approaches have been proposed to deal with the feature selection task, both in regression and in clustering.

For regression, the well-known Lasso method [START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF] is one of the most popular and successful regularization technique which utilizes the ℓ 1 penalty to regularize the squared error function, or by equivalence the log-likelihood in Gaussian regression, and to achieve parameter estimation and feature selection. This allows to shrink coefficients toward zero, and can also set many coefficients to be exactly zero.

In related mixture models for simultaneous regression and clustering, including mixture of linear regressions (MLR), where the mixing proportions are constant, [START_REF] Khalili | Variable selection in finite mixture of regression models[END_REF] proposed regularized ML inference, including MIXLASSO, MIXHARD and MIXSCAD and provided asymptotic properties corresponding to these penalty functions. Later [START_REF] Khalili | New estimation and feature selection methods in mixture-of-experts models[END_REF] extended his MLR regularization to the MoE setting, and provided a root-n consistent and oracle properties for Lasso and SCAD penalties, and developed an EM algorithm for fitting the models. However, as we will discuss it in Section 3.3, this is based on approximated penalty function, and uses a Newton-Raphson procedure in the updates of the gating network parameters, and thus requires matrix inversion.

In this chapter, we consider and improve the previous regularized MoE models from [START_REF] Khalili | New estimation and feature selection methods in mixture-of-experts models[END_REF]. We propose a new regularized maximum likelihood estimation approach with three hybrid algorithms for maximizing the proposed objective function. The proposed algorithms for fitting the model consist of an Expectation-Majorization-Maximization (EMM) algorithm, an EM algorithm with a coordinate ascent algorithm and an EM algorithm with proximal Newtontype method. The proposed approach does not require an approximate of the regularization MIXTURE OF EXPERTS FOR REGRESSION AND CLUSTERING term, and the three developed hybrid algorithms, allow to automatically select sparse solutions without thresholding. In addition, an evaluation of the effectiveness of an ℓ 2 penalty for the gating network in [START_REF] Khalili | New estimation and feature selection methods in mixture-of-experts models[END_REF] is also considered.

The remainder of this chapter is organized as follows. In Section 3.2 we present the modeling with MoE for heterogeneous data. Then, in Section 3.3, we present the regularized maximum likelihood strategy of the MoE model, and the three proposed EM-based algorithms. An experimental study, carried out on simulated and two real data sets, are given in Section 3.4. In Section 3.4.6, we discuss the effectiveness of our method in dealing with moderate dimensional problems, and consider an experiment which promotes its use in high-dimensional scenarios. Finally, in Section 3.5, we draw concluding remarks and mention future direction.

Mixture of experts model for continuous data

Let ((X 1 , Y 1 ), . . . , (X n , Y n )) be a random sample of n independently distributed pairs (X i , Y i ), (i = 1, . . . , n) where Y i ∈ Y ⊂ R is the ith response given some vector of predictors X i ∈ X ⊂ R p . Let D = ((x 1 , y 1 ), . . . , (x n , y n )) be an observed data sample. These data may be discrete or continuous. In this chapter, we emphasize on the continuous case and consider the MoE modeling in Section 2.3.3 for the analysis of a heterogeneous set of such data.

The model

The mixture of experts model assumes that the observed pairs (x, y) are generated from K ∈ N probability density components (the experts) governed by a hidden categorical random variable Z ∈ [K] = {1, . . . , K} that indicates the component from which a particular observed pair is drawn. The latter represents the gating network. Formally, the gating network is defined by the distribution of the hidden variable Z given the predictor x, i.e., π k (x; w) = P(Z = k|X = x; w), which is in general given by gating softmax functions of the form: (Jiang and Tanner, 1999a). The experts network is defined by the conditional densities f (y|x; θ k ) which is the short notation of f (y|X = x, Z = k; θ). Specially, for the regression problem in this chapter, the expert network is modeled by the noisy linear model:

π k (x; w) = P(Z = k|X = x; w) = exp(w k0 + x T w k ) 1 + K-1 l=1 exp(w l0 + x T w l ) (3.1) for k = 1, . . . , K -1 with w T k = (w k0 , w T k ) ∈ R p+1 and w T K = (w K0 , w T K ) = (0, 0) for identifi- ability
f (y|x; θ k ) = N (y; β k0 + x T β k , σ 2 k ),
where the bias coefficient β k0 ∈ R and β k ∈ R p are the regression coefficients describing the kth expert, and σ k > 0 corresponds to the standard deviation of the noise. Thus, in a MoE model for Gaussian regression with K components, the conditional density function of Y given x can 57 3.3. REGULARIZED MAXIMUM LIKELIHOOD PARAMETER ESTIMATION OF THE MOE be defined by the following semi-parametric probability density function:

f (y|x; θ) = K k=1 π k (x; w)f (y|x; θ k ) = K k=1 π k (x; w)N (y; β k0 + x T β k , σ 2 k ) (3.2)
that is parameterized by the parameter vector defined by θ = (w T 1 , . . . ,

w T K-1 , θ T 1 , . . . , θ T K ) T where θ k = (β k0 , β T k , σ 2 k ) T (k = 1, . . . , K)
is the parameter vector of the kth expert.

Maximum likelihood parameter estimation

Assume that, D = ((x 1 , y 1 ), . . . , (x n , y n )) is an observed data sample generated from the MoE (3.2) with unknown parameter θ. The parameter vector θ is commonly estimated by maximizing the observed data log-likelihood L(θ) = n i=1 log K k=1 π k (x i ; w)f (y i |x i ; θ k ) by using the EM algorithm which allows to iteratively find an appropriate local maximizer of the log-likelihood function (see Section 2.3.3 and Dempster et al. (1977); [START_REF] Jacobs | Adaptive mixtures of local experts[END_REF] for more details). In the considered model for Gaussian regression, the log-likelihood function is given by

L(θ) = n i=1 log K k=1 π k (x i ; w)N (y i ; β k0 + β T k x i , σ 2 k ) . (3.3)
However, it is well-known that the MLE may be unstable of even infeasible in high-dimension namely due to possibly redundant and correlated features. In such a context, a regularization of the MLE is needed.

Regularized maximum likelihood parameter estimation of the MoE

Regularized maximum likelihood estimation allows the selection of a relevant subset of features for prediction and thus encourages sparse solutions. In mixture of experts modeling, one may consider both sparsity in the feature space of the gates, and of the experts. We propose to infer the MoE model by maximizing a regularized log-likelihood criterion, which encourages sparsity for both the gating network parameters and the experts network parameters, and does not require any approximation, along with performing the maximization, so that to avoid matrix inversion. The proposed regularization combines a Lasso penalty for the experts parameters, and an elastic net like penalty for the gating network, defined by:

P L(θ) = L(θ) - K k=1 λ k β k 1 - K-1 k=1 γ k w k 1 - ρ 2 K-1 k=1 w k 2 2 .
(3.4) MIXTURE OF EXPERTS FOR REGRESSION AND CLUSTERING A similar strategy has been proposed in [START_REF] Khalili | New estimation and feature selection methods in mixture-of-experts models[END_REF] where the author proposed a regularized ML function like (3.4) but which is then approximated in the EM algorithm of the model inference. The EM algorithm for fitting the model follows indeed the suggestion of [START_REF] Fan | Variable selection via nonconcave penalized likelihood and its oracle properties[END_REF] to approximate the penalty function in a some neighborhood by a local quadratic function. Therefore, a Newton-Raphson can be used to update parameters in the M-step. The weakness of this scheme is that once a feature is set to zero, it may never reenter the model at a later stage of the algorithm. To avoid this numerical instability of the algorithm due to the small values of some of the features in the denominator of this approximation, [START_REF] Khalili | New estimation and feature selection methods in mixture-of-experts models[END_REF] replaced that approximation by an ǫ-local quadratic function. Unfortunately, these strategies have some drawbacks. First, by approximating the penalty functions with (ǫ-)quadratic functions, none of the components will be exactly zero. Hence, a threshold should be considered to declare a coefficient is zero, and this threshold affects the degree of sparsity. Secondly, using Newton-Raphson procedure for maximizing a concave function with large dimension p is not an appropriate choice related to the required Hessian matrix inversion. In addition, one has also to choose ǫ as an additional tuning parameter in practice. Our proposal overcomes these limitations.

The ℓ 2 term penalty is added in our model to take into account possible strong correlation between the features x j which could be translated especially on the coefficients of the gating network w because they are related between the different experts, contrary to the regression coefficients β. The resulting combination of ℓ 1 and ℓ 2 for w leads to an elastic net-like regularization, which enjoys similar sparsity of representation as the ℓ 1 penalty. The ℓ 2 term is not however essential especially when the main goal is to retrieve the sparsity, rather than to perform prediction.

Parameter estimation and feature selection with a dedicated block-wise EM

We propose three block-wise EM algorithms to monotonically find at least local maximizers of (3.4). The E-step is common to these algorithms, while in the M-step, three different algorithms are proposed to update the model parameters. More specifically, the first one relies on a MM algorithm, and the second one uses a coordinate ascent to update the gating network w parameters. Meanwhile, the last one uses proximal Newton-type procedure to update w. All these algorithm use the same coordinate ascent method to update the experts network β' parameters.

The EM algorithm for the maximization of (3.4) firstly requires the construction of the penalized complete-data log-likelihood

P L c (θ) = L c (θ) - K k=1 λ k β k 1 - K-1 k=1 γ k w k 1 - ρ 2 K-1 k=1 w k 2 2
(3.5)

where

L c (θ) = n i=1 K k=1 z ik log [π k (x i ; w)f (y i |x i ; θ k )]
is the standard complete-data log-likelihood, z ik is an indicator binary-valued variable such that z ik = 1 if Z i = k (i.e., if the ith pair (x i , y i ) 59 3.3. REGULARIZED MAXIMUM LIKELIHOOD PARAMETER ESTIMATION OF THE MOE is generated from the kth expert component) and z ik = 0 otherwise. Thus, the EM algorithm for the RMoE in its general form runs as follows. After starting with an initial solution θ [0] , it alternates between the two following steps until convergence (e.g., when there is no longer a significant change in the relative variation of the regularized log-likelihood).

E-step

The E-Step computes the conditional expectation of the penalized complete-data log-likelihood (3.5), given the observed data D under the current parameter vector θ [q] , q being the current iteration number of the block-wise EM algorithm:

Q(θ; θ [q] ) = E P L c (θ)|D; θ [q] = n i=1 K k=1 τ [q] ik log [π k (x i ; w)f k (y i |x i ; θ k )] - K k=1 λ k β k 1 - K-1 k=1 γ k w k 1 - ρ 2 K-1 k=1 w k 2 2
(3.6)

where

τ [q] ik = P(Z i = k|y i , x i ; θ [q] ) = π k (x i ; w [q] )N (y i ; β [q] k0 + x T i β [q] k , σ [q]2 k ) K l=1 π l (x i ; w [q] )N (y i ; β [q] l0 + x T i β [q] l , σ [q]2 l ) (3.7)
is the conditional probability that the data pair (x i , y i ) is generated by the kth expert. This step therefore only requires the computation of the conditional component probabilities τ

[q] ik (i = 1, . . . , n) for each of the K experts.

M-step

The M-Step updates the parameters by maximizing the Q function (3.6), which can be written as

Q(θ; θ [q] ) = Q(w; θ [q] ) + Q(β, σ; θ [q] ) (3.8) with Q(w; θ [q] ) = n i=1 K k=1 τ [q] ik log π k (x i ; w) - K-1 k=1 γ k w k 1 - ρ 2 K-1 k=1 w k 2 2 , (3.9)
and

Q(β, σ; θ [q] ) = n i=1 K k=1 τ [q] ik log N (y i ; β k0 + x T i β k , σ 2 k ) - K k=1 λ k β k 1 . (3.10)
The parameters w are therefore separately updated by maximizing the function

Q(w; θ [q] ) = n i=1 K-1 k=1 τ [q] ik (w k0 +x T i w k )- n i=1 log 1+ K-1 k=1 e w k0 +x T i w k - K-1 k=1 γ k w k 1 - ρ 2 K-1 k=1 w k 2 2 .
(3.11) MIXTURE OF EXPERTS FOR REGRESSION AND CLUSTERING

We propose and compare three approaches for maximizing (3.9) based on a MM algorithm, a coordinate ascent algorithm and proximal Newton-type method. These approaches have some advantages since they do not use any approximate for the penalty function, and have a separate structure which avoid matrix inversion.

MM algorithm for updating the gating network

In this part, we construct a MM algorithm to iteratively update the gating network parameters (w k0 , w k ). At each iteration step s of the MM algorithm, we maximize a minorizing function of the initial function (3.9). We begin this task by giving the definition of a minorizing function.

F (x) ≥ G(x|x m ) and F (x m ) = G(x m |x m ), ∀x.
In the maximization step of the MM algorithm, we maximize the surrogate function G(x|x m ), rather than the function F (x) itself. If x m+1 is the maximum of G(x|x m ), then we can show that the MM algorithm forces F (x) uphill, because

F (x m ) = G(x m |x m ) ≤ G(x m+1 |x m ) ≤ F (x m+1 ).
By doing so, we can find a local maximizer of F (x). If G(x m |x m ) is well constructed, then we can avoid matrix inversion when maximizing it. Next, we derive the surrogate function for Q(w; θ [q] ). We start by the following lemma.

Lemma 3.3.1. If x > 0, then the function f (x) =ln(1 + x) can be minorized by

g(x|x m ) = -ln(1 + x m ) - x -x m 1 + x m , at x m > 0.
By applying this lemma and following (Lange, 2013, page 211) we have

Theorem 3.3.1. The function I 1 (w) = - n i=1 log 1 + K-1 k=1 e w k0 +x T i w k is a majorizer of G 1 (w|w [s] ) = n i=1 - K-1 k=1 π k (x i ; w [s] ) p + 1 p j=0 e (p+1)x ij (w kj -w [s] kj ) -log C m i + 1 - 1 C m i ,
where

C m i = 1 + K-1 k=1 e w [s] k0 +x T i w [s]
G i (w|w [s] ) = -log 1 + K-1 k=1 e w [s] k0 +x T i w [s] k - K-1 k=1 (e w k0 +x T i w k -e w [s] k0 +x T i w [s] k ) 1 + K-1 k=1 e w [s] k0 +x T i w [s] k = -log C m i + 1 - 1 C m i - K-1 k=1 e w [s] k0 +x T i w [s] k C m i e (w k0 +x T i w k )-(w [s] k0 +x T i w [s] k ) •
Now, by using arithmetic-geometric mean inequality then

e (w k0 +x T i w k )-(w [s] k0 +x T i w [s] k ) = p j=0 e x ij (w kj -w [s] kj ) ≤ p j=0 e (p+1)x ij (w kj -w [s] kj ) p + 1 • (3.12) When (w k0 , w k ) = (w [s] k0 , w [s]
k ) the equality holds. Thus, I 1i (w) can be minorized by

G 1i (w|w [s] ) = - K-1 k=1 e w [s] k0 +x T i w [s] k (p + 1)C m i p j=0 e (p+1)x ij (w kj -w [s] kj ) -log C m i + 1 - 1 C m i = - K-1 k=1 π k (x i ; w [s] ) p + 1 p j=0 e (p+1)x ij (w kj -w [s] kj ) -log C m i + 1 - 1 C m i •
This leads us to the minorizing function G 1 (w|w [s] ) for I 1 (w)

G 1 (w|w [s] ) = n i=1 - K-1 k=1 π k (x i ; w [s] ) p + 1 p j=0 e (p+1)x ij (w kj -w [s] kj ) -log C m i + 1 - 1 C m i •
Therefore, the minorizing function G [q] (w|w [s] ) for Q(w; θ [q] ) is given by

G [q] (w|w [s] ) = n i=1 K-1 k=1 τ [q] ik (w k0 + x T i w k ) + G 1 (w|w [s] ) - K-1 k=1 γ k p j=1 |w kj | - ρ 2 K-1 k=1 p j=1 w 2 kj .
Now, let us separate G [q] (w|w [s] ) into each parameter for all k ∈ {1, . . . , K -1}, j ∈ {1, . . . , p}, we have:

G [q] (w k0 |w [s] ) = n i=1 τ [q] ik w k0 - n i=1 π k (x i ; w [s] ) p + 1 e (p+1)(w k0 -w [s] k0 ) + A k (w [s]
), (3.13) MIXTURE OF EXPERTS FOR REGRESSION AND CLUSTERING and

G [q] (w kj |w [s] ) = n i=1 τ [q] ik x ij w kj - n i=1 π k (x i ; w [s] ) p + 1 e (p+1)x ij (w kj -w [s] kj ) -γ k |w kj | - ρ 2 w 2 kj + B kj (w [s] ),
(3.14) where A k (w [s] ) and B kj (w [s] ) are only functions of w [s] . The update of w

[s] k0 is straightforward by maximizing (3.13) and given by w

[s+1] k0 = w [s] k0 + 1 p + 1 ln     n i=1 τ [q] ik n i=1 π k (x i ; w [s] )     . (3.15)
The function G [q] (w kj |w [s] ) is a concave function. Moreover, it is a univariate function with respect to w kj . We can therefore maximize it globally and with respect to each coeffcient w kj separately and thus avoid matrix inversion. Indeed, let us denote by

F [q] kjm (w kj ) = n i=1 τ [q] ik x ij w kj - n i=1 π k (x i ; w [s] ) p + 1 e (p+1)x ij (w kj -w [s] kj ) - ρ 2 w 2 kj + B kj (w [s] ),
hence, G [q] (w kj |w [s] ) can be rewritten as

G [q] (w kj |w [s] ) =          F [q] kjm (w kj ) -γ k w kj , if w kj > 0 F [q] kjm (0)
, if w kj = 0

F [q]
kjm (w kj ) + γ k w kj , if w kj < 0 .

We therefore have both F

[q]

kjm (w kj )-γ k w kj and F

[q]

kjm (w kj )+γ k w kj are smooth concave functions. Thus, one can use one-dimensional Newton-Raphson algorithm to find the global maximizers of these functions and compare with F

[q] kjm (0) in order to update w G [q] (w kj |w [s] ).

The update of w kj can then be computed by a one-dimensional generalized Newton-Raphson (NR) algorithm, which updates, after starting from and initial value w

[0] kj = w [s]
kj , at each iteration t of the NR, according to the following updating rule:

w [t+1] kj = w [t] kj - ∂ 2 G [q] (w kj |w [s] ) ∂ 2 w kj -1 w [t] kj
∂G [q] (w kj |w [s] )

∂w kj w [t] kj ,
where the first and the scalar gradient and hessian are respectively given by: ∂G [q] (w kj |w [s] )

∂w kj =    U (w kj ) -γ k , G [q] (w kj |w [s] ) = F [q] kjm (w kj ) -γ k w kj U (w kj ) + γ k , G [q] (w kj |w [s] ) = F [q] kjm (w kj ) + γ k w kj , 63 3.3. REGULARIZED MAXIMUM LIKELIHOOD PARAMETER ESTIMATION OF THE MOE and ∂ 2 G [q] (w kj |w [s] ) ∂ 2 w kj = -(p + 1) n i=1 x 2 ij π k (x i ; w [s] )e (p+1)x ij (w kj -w [s] kj ) -ρ, with U (w kj ) = n i=1 τ [q] ik x ij - n i=1 x ij π k (x i ; w [s] )e (p+1)x ij (w kj -w [s] kj ) -ρw kj .
Unluckily, while this method allows to compute separate univariate updates by globally maximizing concave functions, it has some drawbacks. First, we found the same behavior of the MM algorithm for this non-smooth function setting as in [START_REF] Hunter | Variable selection using MM algorithms[END_REF]: once a coefficient is set to be zero, it may never reenter the model at a later stage of the algorithm. Second, the MM algorithm can stuck on non-optimal points of the objective function. [START_REF] Schifano | Majorization-minimization algorithms for nonsmoothly penalized objective functions[END_REF] made an interesting study on the convergence of the MM algorithms for nonsmoothly penalized objective functions, in which they proof that with some conditions on the minorizing function (see Theorem 2.1 of Schifano et al. ( 2010)), then the MM algorithm will converge to the optimal value. One of these conditions requires the minorizing function must be strickly positive, which is not guaranteed in our method, since we use the arithmetic-geometric mean inequality in (3.12) to construct our surrogate function. Hence, we just ensure that the value of Q(w; θ [q] ) will not decrease in our algorithm. In the next section, we propose updating w k = (w k0 , w k ) by using coordinate ascent algorithm. This approach overcomes this weakness of the MM algorithm.

Coordinate ascent algorithm for updating the gating network

We now consider another approach for updating (w k0 , w k ) by using coordinate ascent algorithm. Indeed, based on [START_REF] Tseng | Coordinate ascent for maximizing nondifferentiable concave functions[END_REF][START_REF] Tseng | Convergence of a block coordinate descent method for nondifferentiable minimization[END_REF], with regularity conditions, then the coordinate ascent algorithm is successful in updating w. Thus, the w parameters are updated in a cyclic way, where a coefficient w kj (j = 0) is updated at each time, while fixing the other parameters to their previous values. Hence, at each iteration one just needs to update only one parameter. With this setting, the update of w kj is performed by maximizing the component (k, j) of (3.9) given by

Q(w kj ; θ [q] ) = F (w kj ; θ [q] ) -γ k |w kj |, (3.16)
where

F (w kj ; θ [q] ) = n i=1 τ [q] ik (w k0 + w T k x i ) - n i=1 log 1 + K-1 l=1 e w l0 +w T l x i - ρ 2 w 2 kj .
(3.17)

Hence, Q(w kj ; θ [q] ) can be rewritten as

Q(w kj ; θ [q] ) =          F (w kj ; θ [q] ) -γ k w kj , if w kj > 0 F (0; θ [q] ) , if w kj = 0 F (w kj ; θ [q] ) + γ k w kj , if w kj < 0 . [0] kj = w [q] kj
to find the maximizers of these functions and compare with F (0; θ [q] ) in order to update w = arg max

w kj Q(w kj ; θ [q] ),
where s denotes the sth loop of the coordinate ascent algorithm. The update of w kj is therefore computed iteratively after starting from and initial value w

[0] kj = w [s]
kj following the update equation

w [t+1] kj = w [t] kj - ∂ 2 Q(w kj ; θ [q] ) ∂ 2 w kj -1 w [t] kj ∂Q(w kj ; θ [q] ) ∂w kj w [t] kj , (3.18)
where t in the inner NR iteration number, and the one-dimensional gradient and hessian functions are respectively given by ∂Q(w kj ; θ [q] )

∂w kj =    U (w kj ) -γ k , if Q(w kj ; θ [q] ) = F (w kj ; θ [q] ) -γ k w kj U (w kj ) + γ k , if Q(w kj ; θ [q] ) = F (w kj ; θ [q] ) + γ k w kj , (3.19) and ∂ 2 Q(w kj ; θ [q] ) ∂ 2 w kj = - n i=1 x 2 ij e w k 0+x T i w k (C i (w kj ) -e w k0 +x T i w k ) C 2 i (w kj ) -ρ.
with

U (w kj ) = n i=1 x ij τ [q] ik - n i=1
x ij e w k0 +x T i w k

C i (w kj ) -ρw kj , and
C i (w kj ) = 1 + l =k e w l0 +x T i w l + e w k0 +x T i w k ,
is a univariate function of w kj when fixing other parameters. For other parameter we set w

[s+1] lh = w [s]
lh . Similarly, for the update of w k0 , a univariate Newton-Raphson algorithm with initial value w

[0] k0 = w [q]
k0 can be used to provide the update w

[s] k0 given by w [s+1] k0 = arg max w k0 Q(w k0 ; θ [q] ),
where Q(w k0 ; θ [q] ) is a univariate concave function given by

Q(w k0 ; θ [q] ) = n i=1 τ [q] ik (w k0 + x T i w k ) - n i=1 log 1 + K-1 l=1 e w l0 +x T i w l , (3.20) with ∂Q(w k0 ; θ [q] ) ∂w k0 = n i=1 τ [q] ik - n i=1 e w k0 +x T i w k C i (w k0 ) (3.21) 3.3. REGULARIZED MAXIMUM LIKELIHOOD PARAMETER ESTIMATION OF THE MOE and ∂ 2 Q(w k0 ; θ [q] ) ∂ 2 w k0 = - n i=1 e w k0 +x T i w k (C i (w k0 ) -e w k0 +x T i w k ) C 2 i (w k0 ) . (3.22)
The other parameters are fixed while updating w k0 . By using the coordinate ascent algorithm, we have univariate updates, and make sure that the parameters w kj may change during the algorithm even after they shrink to zero at an earlier stage of the algorithm.

In the next section, we propose a new procedure for updating the gating network parameters.

Proximal Newton-type procedure for updating the gating network Indeed, the developed MM and coordinate ascent algorithms for the estimation of the parameters of our model could be slow in a high-dimensional setting since we do not have the closed-form updates of the parameters of the gating network w at each step of the EM algorithm; while a univariate Newton-Raphson is derived to avoid matrix inversion operations, it is still slow in highdimension. However, this difficulty can be overcome by proximal Newton-type method. In this part, we propose two approaches for updating the gating network parameters w = {(w k0 , w k )} by maximizing Q(w; θ [q] ) based on the proximal Newton and the proximal Newton-type method.

The proximal Newton method approximates only the smooth part of (3.9) given by

I(w) = n i=1 K-1 k=1 τ [q] ik (w k0 + x T i w k ) - n i=1 log 1 + K-1 k=1 e w k0 +x T i w k (3.23)
with its Taylor expansion at current estimates Ĩt (w) = I(w (t) ) + ▽I(w (t) ) T (ww (t) ) + 1 2 (ww (t) ) T ▽ 2 I(w (t) )(ww (t) ), (3.24)

where ▽I(w (t) ), ▽ 2 I(w (t) ) are corresponding the gradient vector and the Hessian matrix of I(w) at w (t) . After that, the problem can be solved by an iterative algorithm with initial value w (0) where, at step (t + 1), it minimizes the proximal function

Qt (w) = Ĩt (w) - K-1 k=1 γ k w k 1 - ρ 2 K-1 k=1 w k 2 2
(3.25) instead of Q(w; θ [q] ) and then searches for the updating value w (t+1) based on the solution of (3.25) that improves the Q-function, i.e., Q(w (t) ; θ [q] ) < Q(w (t+1) ; θ [q] ) until the algorithm converges. This strategy has some advantages especially since I(.) does not have a quadratic form. First, by approximating I with its local quadratic form, several good methods can be used to solve (3.25) such as coordinate ascent, where updating one parameter in each step will avoid computing the inverse of a matrix. Second, one can obtain the closed-form update for each parameter at each iteration of the algorithm, hence, reduce the computational time of the algorithm. Finally, for searching w (t+1) , one can use the efficient backtracking line search MIXTURE OF EXPERTS FOR REGRESSION AND CLUSTERING strategy (see [START_REF] Boyd | Convex optimization[END_REF]) which is easy to setup.

However, the K -1 vectors for the gating network will not approximate I(w) with its Taylor expansion. Here, partial Newton steps are performed by forming a partial quadratic approximation to Q(w; θ [q] ) (Taylor expansion at the current estimates), allowing only (w k0 , w k ) to vary for a single class at a time. This algorithm is similar to the one in [START_REF] Friedman | Regularization paths for generalized linear models via coordinate descent[END_REF] except the fact that here after each outer loop that cycles over k, a backtracking line search is performed over the step size parameter t ∈ [0, 1]. The partial quadratic approximation to I(w) with respect to (w k0 , w k ) at w is given by (see Appendix C.2 for more details)

l I k (w k0 , w k ) = - 1 2 n i=1 d ik (c ik -w k0 -x T i w k ) 2 + C( w), (3.26)
where

c ik = wk0 + x T i wk + τ [q] ik -π k ( w; x i ) π k ( w; x i )(1 -π k ( w; x i ))
, (3.27)

d ik = π k ( w; x i )(1 -π k ( w; x i )), (3.28)
and C( w) is a function of w. After calculating the partial quadratic approximation l I k (w k0 , w k ) about the current parameters w, a coordinate ascent algorithm is used to solve the penalized weighted least-square problem max

(w k0 ,w k ) l I k (w k0 , w k ) -γ k w k 1 - ρ 2 w k 2 2 .
(3.29)

Using the soft-thresholding operator (see (Hastie et al., 2015, sec. 5.4)), one can obtain the closed-form update for w kj as follows

w m+1 kj = S γ k ( n i=1 d ik u m ikj x ij ) n i=1 d ik x 2 ij + ρ , (3.30) with u m ikj = c ik -w m k0 -x T i w m k + w m kj x ij and S γ k (.
) is a soft-thresholding operator defined by [S γ (u)] j = sign(u j )(|u j |γ) + and (x) + a shorthand for max{x, 0}. Here, m is defined as the mth step of the coordinate ascent algorithm. Note that, for each iteration of the coordinate ascent algorithm one parameter is updated while other are kept fixed, that means for h = j, w m+1 kh = w m kh . For w k0 , the closed-form update is given by

w m+1 k0 = n i=1 d ik (c ik -x T i w m k ) n i=1 d ik . (3.31)
Once the coordinate ascent algorithm converges, the new values of (w k0 , w k ) are taken into account for the next loop of the proximal Newton algorithm. Overall, the algorithm is summarized by pseudo-code 2.

Algorithm 2 Proximal Newton method for updating the gating network 1: w (0) = w [q] . 2: repeat 3:

for k = 1 to K -1 do 4:
Update the quadratic approximation l I k (w k0 , w k ) in (3.26) by using the current parameters.

5:

Solve the penalized weighted least-square problem in (3.29) by using coordinate ascent algorithm and compute the solution w(s) k according to (3.30), (3.31).

6:

Update (w k0 , w k ) by the new values.

7:

end for 8:

Set w (s+1) = (1t)w (s) + t w(s) , where t is found using a backtracking line-search.

9:

Evaluate the objective function Q(.; θ [q] ) at w (s+1) . 10: until the stopping criterion is satisfied.

The initial values for (w k0 , w k ) in this EM algorithm are set to 0 and the backtracking line-search is needed for algorithm to converge to the optimal solution. The proximal Newton method presented here can overcome the drawback of the coordinate ascent algorithm since at each step has a closed-form update update for each parameter. Hence, it improves the running time of the algorithm.

Even though in some cases the values of the probabilities π k (x i ; w) can become too small (or too close to 1), and the algorithm can get stuck while solving (3.29). To address this issue, we consider proximal Newton-type method as a proper choice for this situation. Proximal Newton-type methods use a symmetric negative definite matrix B ≈ ▽ 2 I( wk ) to model the curvature of I(w) at (w k0 , w k ). In this case, one can follow the suggestions of (Lange, 2013, sec. 8.7) and [START_REF] Gormley | A mixture of experts model for rank data with 152 BIBLIOGRAPHY applications in election studies[END_REF] by choosing a constant negative definite matrix B such as ▽ 2 I( wk ) > B. The proximal Newton-type algorithm here can be interpreted as a special case of the MM algorithm [START_REF] Hunter | A tutorial on MM algorithms[END_REF]. Specifically it is a minorize-maximize algorithm for updating the gating network and also the expert network in multinomial outputs case. Since,

∂ 2 I(w) ∂w kj ∂w kh = - n i=1 x ij x ih π k (x i ; w)(1 -π k (x i ; w)), ∀j, h,
then, using the fact that π(1π) ≤ 1/4, we can take B = -1/4 n i=1 x i x T i . Thus, instead of solving (3.29), one can solve the local quadratic model max ikπ k ( w; x i )), (3.34) Ĉ( w) is a function of w. Here, it is clear that this approach has some advantages. Firstly, we can avoid computing the Hessian matrix and can also avoid numerical instability caused by π k ( w; x i ). Secondly, the backtracking line search step is not necessary in this case. The increase of the Q(w; θ [q] ) after each loop is also guaranteed, since this algorithm is a proximal Newton-type algorithm and is a specific case of the MM algorithm.

(w k0 ,w k ) lI k (w k0 , w k ) -γ k w k 1 . ( 3

Updating the experts network

Now once we have these three methods to update the gating network parameters, we move on updating the experts network parameters ({β, σ 2 }). To do that, we first perform the update for (β k0 , β k ), while fixing σ k . This corresponds to solving K separated weighted Lasso problems. Hence, we choose to use a coordinate ascent algorithm to deal with this. Actually, in this situation the coordinate ascent algorithm can be seen as a special case of the MM algorithm; hence, this updating step is common to both of the proposed algorithms. More specifically, the update of β kj is performed by maximizing

Q(β, σ; θ [q] ) = n i=1 K k=1 τ [q] ik log N (y i ; β k0 + β T k x i , σ 2 k ) - K k=1 λ k β k 1 ; (3.35)
using a coordinate ascent algorithm, with initial values (β

[0] k0 , β [0] k ) = (β [q] k0 , β [q]
k ). We obtain closed-form coordinate updates that can be computed for each component following the results in (Hastie et al., 2015, sec. 5.4), and are given by

β [s+1] kj = S λ k σ (s)2 k n i=1 τ [q] ik r [s] ikj x ij n i=1 τ [q] ik x 2 ij , (3.36) with r [s] ikj = y i -β [s] k0 -β [s]T k x i + β [s] kj x ij and S λ k σ (s)2 k (.
) is a soft-thresholding operator defined by [S γ (u)] j = sign(u j )(|u j |γ) + and (x) + a shorthand for max{x, 0}. For h = j, we set β

[s+1] kh = β [s] kh . At each iteration m, β k0 is updated by β [s+1] k0 = n i=1 τ [q] ik (y i -β [s+1]T k x i ) n i=1 τ [q] ik • (3.37)
In the next step, we take (w

[q+2] k0 , w [q+2] k ) = (w [q+1] k0 , w [q+1] k ), (β [q+2] k0 , β [q+2] k ) = (β [q+1] k0 , β [q+1] k
), rerun the E-step, and update σ 2 k according to the standard update of a weighted Gaussian 69

3.4. EXPERIMENTAL STUDY regression σ 2[q+2] k = n i=1 τ [q+1] ik (y i -β [q+2] k0 -β [q+2] k T x i ) 2 n i=1 τ [q+1] ik • (3.38)
Each of these proposed algorithms is iterated until the change in P L(θ) is small enough. Moreover, we can directly get zero coefficients without any thresholding unlike in [START_REF] Khalili | New estimation and feature selection methods in mixture-of-experts models[END_REF]; [START_REF] Hunter | Variable selection using MM algorithms[END_REF]. These algorithms increase the penalised log-likelihood function (3.4) as shown in Appendix B.

Algorithm tuning and model selection

In practice, appropriate values of the tuning parameters (λ, γ, ρ) should be chosen. To select the tuning parameters, we propose a modified BIC with a grid search scheme, as an extension of the criterion used in [START_REF] Städler | l1-penalization for mixture regression models[END_REF] for regularized mixture of regressions. First, assume that K 0 ∈ {K 1 , . . . , K M } whereupon K 0 is the true number of expert components. For each value of K, we choose a grid of the tuning parameters. Consider grids of values {λ 1 , . . . , λ M 1 }, {γ 1 , . . . , γ M 2 } in the size of √ n and a small enough value of ρ ≈ O(log n) for the ridge turning parameter. ρ = 0.1 log n can be used in practice. For a given triplet (K, λ i , γ j ), we select the maximal penalized log-likelihood estimators θ K,λ,γ using each of our hybrid EM algorithms presented above. Then, the following modified BIC criterion,

BIC(K, λ, γ) = L( θ K,λ,γ ) -DF (λ, γ) log n 2 , (3.39)
where DF (λ, γ) is the estimated number of non-zero coefficients in the model, is computed. Finally, the model with parameters (K, λ, γ) = ( K, λ, γ) which maximizes the modified BIC value, is selected. While the problem of choosing optimal values of the tuning parameters for penalized MoE models is still an open research, the modified BIC performs reasonably well in our experiments.

Experimental study

We study the performance of our methods on both simulated data and real data. The results are compared among the regularized MoE using three proposed hybrid EM algorithms with Minorization-Maximization (MoE-Lasso+ℓ 2 (MM)), coordinate ascent (MoE-Lasso+ℓ 2 (CA)) and proximal Newton method (MoE-Lasso+ℓ 2 (PN)) with the following four methods: i) the standard non-penalized MoE (MoE), ii) the MoE with ℓ 2 regularization (MoE+ℓ 2 ), iii) the mixture of linear regressions with Lasso penalty (MIXLASSO), and iv) the MoE with BIC penalty for feature selection. We consider several evaluation criteria to assess the performance of the models, including sparsity, parameters estimation and clustering criteria.

In addition, to evaluate the affect of the ℓ 2 norm to the sparsity of the model, we also compare our results with the Lasso-regularized model. We use proximal Newton method (MoE-MIXTURE OF EXPERTS FOR REGRESSION AND CLUSTERING Lasso (PN)) to estimate parameters in this case.

Evaluation criteria

We compare the results of all the models for three different criteria: sensitivity/specificity, parameters estimation, and clustering performance for simulation data. The sensitivity/specificity is defined by

• Sensitivity: proportion of correctly estimated zero coefficients;

• Specificity: proportion of correctly estimated nonzero coefficients.

In this way, we compute the ratio of the estimated zero/nonzero coefficients to the true number of zero/nonzero coefficients of the true parameter for each component. In our simulation, the proportion of correctly estimated zero coefficients and nonzero coefficients have been calculated for each data set for the experts parameters and the gating parameters, and we present the average proportion of these criteria computed over 100 different data sets. Also, to deal with the label switching before calculating these criteria, we permuted the estimated coefficients based on an ordered between the expert parameters. If the label switching happens, one can permute the expert parameters and the gating parameters then replace the kth gating network vector with w per k = w kw K . By doing so, we ensure that the log-likelihood will not change, that means L( θ) = L( θper ) and these parameters satisfy the initialized condition w per K = 0. However, the penalized log-likelihood value can be different from the one before permutation. So this may result in misleading values of the sparsity criterion of the model when we permute the parameters. The regularized method tends to choose the model with small absolute values of the gating network. However, for K = 2, the log-likelihood function and the penalized loglikelihood function will not change since we have w per 1 = -w 1 . For the second criterion of parameter estimation, we compute the mean and standard deviation of both penalized parameters and non penalized parameters in comparison with the true value θ. We also consider the mean squared error (MSE) between each component of the true parameter vector and the estimated one, which is given by θ j -θj 2 .

For the clustering criterion, once the parameters are estimated and permuted, the provided conditional component probabilities τik defined in (3.7) represent a soft partition of the data. A hard partition of the data is given by applying the Bayes's allocation rule ẑi = arg

K max k=1 τ ik ( θ), (3.40)
where ẑi represents the estimated cluster label for the ith observation. Given the estimated and true cluster labels, we therefore compute the correct classification rate and the Adjusted Rand Index (ARI). Also, we note that for the standard MoE with BIC penalty, we consider a pool of 5 × 4 × 5 = 100 submodels. Our EM algorithm with coordinate ascent has been used with zero penalty coefficients and without updating the given zero parameters in the experts and the gating network to obtain the (local) MLE of each submodel. After that, the BIC criterion in (3.39) was used to choose the best submodel among 100 model candidates.

Simulation study

For each data set, we consider n = 300 predictors x generated from a multivariate Gaussian distribution with zero mean and correlation defined by corr(x ij , x ij′ ) = 0.5 |j-j′| . The response Y |x is generated from a normal MoE model of K = 2 expert components as defined by (2.38) and (2.40), with the following regression coefficients:

(β 10 , β 1 ) T = (0, 0, 1.5, 0, 0, 0, 1) T ;

(β 20 , β 2 ) T = (0, 1, -1.5, 0, 0, 2, 0) T ;

(w 10 , w 1 ) T = (1, 2, 0, 0, -1, 0, 0) T ;

and

σ 1 = σ 2 = σ = 1.
Histogram of a typical simulation data set can be seen from Figure 3.1. 100 data sets were generated for this simulation. The results will be presented in the following sections.

Sensitivity/specificity criteria Table 4.1 presents the sensitivity (S 1 ) and specificity (S 2 ) values for the experts 1 and 2 and the gates for each of the considered models. As it can be seen in the obtained results that the ℓ 2 and MoE models cannot be considered as model selection methods since their sensitivity MIXTURE OF EXPERTS FOR REGRESSION AND CLUSTERING almost surely equals zero. However, it is obvious that the MoE-Lasso+ℓ 2 , with the MM, the coordinate ascent and proximal Newton algorithms, performs quite well for experts 1 and 2.

The difference between the methods are not significant. The feature selection becomes more difficult for the gate π k (x; w) since there is correlation between features. While mixed Lasso+ℓ 2 regularized model using MM (MoE-Lasso+ℓ 2 (MM)) may get trouble in detecting non-zero coefficients in the gating network, the model using coordinate ascent (MoE-Lasso+ℓ 2 (CA)) and proximal Newton (MoE-Lasso+ℓ 2 (PA), and the Lasso regularized model (MoE-Lasso (PA)) perform quite well. There is a slightly difference between MoE-Lasso+ℓ 2 (PA) and MoE-Lasso (PA) in detecting the zero coefficients in the gating network. In this case, the model without the quadratic penalty performs better in retrieving the zero parameters in the gating network.

The MIXLASSO, can detect the zero coefficients in the experts but it will be shown in the later clustering results that this model has a poor result when clustering the data. Note that for the MIXLASSO we do not have gates, so variable "N/A" is mentioned in the results. Finally, while the BIC provides the best results in general, it is hard to apply BIC in reality since the number of submodels may be huge.

Method Expert 1 Expert 2 Gate S 1 S 2 S 1 S 2 S 1 S 2 MoE-BIC
0.920 1.000 0.930 1.000 0.850 1.000 MoE-Lasso+ℓ 2 (MM) 0.720 1.000 0.777 1.000 0.815 0.615 MoE-Lasso+ℓ 2 (CA) 0.700 1.000 0.803 1.000 0.853 0.945 MoE-Lasso+ℓ 2 (PN) 0.698 1.000 0.797 1.000 0.728 0.995 MoE-Lasso (PN) 0.700 1.000 0.790 1.000 0.748 0.995 MIXLASSO 0.775 1.000 0.693 1.000 N/A N/A Table 3.1: Sensitivity (S 1 ) and specificity (S 2 ) results for our proposed methods compared with the MoE model using BIC criterion for feature selection and the mixture of linear regression model (MLR).

Parameter estimation

The boxplots of all estimated parameters are shown in Figures 3. provide sparse results for the model, not only in the experts, but also in the gates. However, the method using MM algorithm in this situation forces the nonzero parameter w 14 toward zero, and this effects the clustering result. The MIXLASSO can also detect zero coefficients in the experts, but since this model does not have a mixture proportions that depend on the inputs, it is least competitive than others.

For the mean and standard derivation results shown in Table 3.2 and Table 3.3, we can see that the model using BIC for selection, the non penalized MoE, and the MoE with ℓ 2 penalty have better results, while regularized MoE models and MIXLASSO can cause bias to the estimated parameters, since the penalty functions are added to the log-likelihood function.

Beta_20 Beta_21 Beta_22 Beta_23 Beta_24 Beta_25 Beta_26 -2 -1 0 1 2 Beta_20 Beta_21 Beta_22 Beta_23 Beta_24 Beta_25 Beta_26 -2 -1 0 1 2 Beta_20 Beta_21 Beta_22 Beta_23 Beta_24 Beta_25 Beta_26 -1 0 1 2 MoE MoE+ℓ 2 MoE-BIC Beta_20 Beta_21 Beta_22 Beta_23 Beta_24 Beta_25 Beta_26 -1 0 1 2 Beta_20 Beta_21 Beta_22 Beta_23 Beta_24 Beta_25 Beta_26 -1 0 1 2 Beta_20 Beta_21 Beta_22 Beta_23 Beta_24 Beta_25 Beta_26 -1 0 1 2 MoE-Lasso + ℓ 2 (MM) MoE-Lasso + ℓ 2 (CA) MoE-Lasso + ℓ 2 (PN) Beta_20 Beta_21 Beta_22 Beta_23 Beta_24 Beta_25 Beta_26 -1 0 1 2 Beta_20 Beta_21 Beta_22 Beta_23 Beta_24 Beta_25 Beta_26 -1 0 1 2
MoE-Lasso (PN) MIXLASSO Figure 3.3: Boxplots of the expert 2's parameter estimates obtained by, respectively, Mixture-of-Experts (MoE) with standard MLE, MoE with ℓ 2 regularization, MoE with BIC procedure for model selection, and the proposed methods: MoE with mixed ℓ 1 (Lasso) and ℓ 2 regularization using EM algorithm with Minorization-Maximization (MM) updates, Coordinate Ascent (CA) updates, and Proximal Newton (PN) updates; MoE with Lasso regularization. In addition, the standard MIXLASSO (Mixture of regressions with Lasso regularization) is also considered.

In contrast, from Table 3.4 and Table 3.5, in terms of average mean squared error, the penalized MoE model and MIXLASSO provide a better result than MoE and the MoE with ℓ 2 penalty for estimating the zero coefficients. The BIC still provides the best result, but as we commented

Clustering of heterogeneous regression data

We calculate the accuracy of clustering of all these mentioned models for each data set. The results in terms of ARI and correct classification rate values are provided in Table 4.5. We can see that the MoE-Lasso+ℓ 2 (PN) model provides a good result for clustering data. The obtain results from this method is as good as the non penalized model. However, as we can see, the model without ℓ 2 penalty (MoE-Lasso (PN)) provides better result than MoE-Lasso+ℓ 2 (PN). The MoE-BIC model gives the best result but always with a very significant computational load. The difference between MoE-Lasso+ℓ 2 (CA) and MoE-BIC is smaller than 1%, while the MIXLASSO provides a poor result in terms of clustering. Here, we also see that the MoE-Lasso+ℓ 2 (MM) estimates the parameters in the experts quite well. However, the MM algorithm for updating the gate's parameter causes bad effect, since this approach forces the non-zero coefficient w 14 toward zero. Hence, this may decrease the clustering performance. Overall, we can clearly see the MoE-Lasso+ℓ 2 (CA), MoE-Lasso+ℓ 2 (PN) algorithm perform quite well to retrieve the actual sparse support; the sensitivity and specificity results are quite reasonable for the proposed Lasso+ℓ 2 regularization. If the data is not highly correlated, one can remove the ℓ 2 penalty to improve the sparsity of the gating network and also improve the accuracy of clustering of the regularization models. While the penalty function will cause bias to the 3. parameters, as shown in the results of the MSE, the algorithm can perform parameter density estimation with an acceptable loss of information due to the bias induced by the regularization. In terms of clustering, the methods using coordinate ascent and proximal Newton method work as well as two other MoE models and MoE-BIC, better than the MoE-Lasso+ℓ 2 (MM), MIXLASSO models. One important thing should be mention is that, with the same tuning parameters λ = 10, γ = 5 and ρ = 0.1 log n the regularized model using proximal Newton-type algorithm MoE-Lasso+ℓ 2 (PN) converges to better values of the penalized log-likelihood than those of the method using coordinate ascent MoE-Lasso+ℓ 2 (CA), -585.140 (12.990) compares with -585.410 (13.029) .

Lasso paths for the regularized MoE parameters

In this section, Lasso paths for the experts network are given based on one typical simulation data set. To do this two fix values of the tuning parameters γ are chosen γ ∈ {0, 5}, ρ = 0.1 log n as usual and a grid of values for λ is also provided λ ∈ {0, 1, . . Newton method (MoE-Lasso+ℓ 2 (PN)) is used to obtain the coefficients. The Figure 3.5 gives the Lasso paths for the experts network and also the changing of the gating network coefficients with different values of λ.

From Figure 3.5, one can see that the Lasso paths for each expert component perform quite consistent between different values of γ. Similar with the classical Lasso path for linear regression these entire Lasso paths are also piecewise-linear, excepts the trajectory of the intercept. However, the performance of the gating network coefficients with difference values of λ is hard to predict. Especially, the changing of λ also effect to the sparsity of the gating network in this case. This behavior can be found in the case γ = 5. At γ = 5 and λ ≥ 10 the third coefficient of the gating network is difference from it true value 0. Hence, it is interesting to provide theoretical results to explain this behavior of the regularized MoE.

Evaluation of the model selection via BIC

A small evaluation of the BIC criterion is given in this section. Here, we consider the effect of the BIC criterion to the sparsity of the model, the accuracy of clustering and the adjust rank index criteria. Also the effect of the ℓ 2 norm is also mention. For this reason, two grid of -513.28 (13.70) 0.6475 1.000 0.7533 1.000 0.7825 0.990 89.38% (1.63%) 0.6160 (.052) (10, 4, ρ) -512.94 (13.67) 0.6925 1.000 0.7900 1.000 0.6425 0.995 89.59% (1.68%) 0.6231 (.053) (10, 5, ρ) -512.81 (13.86) 0.6975 1.000 0.7967 1.000 0.7275 0.995 89.53% (1.65%) 0.6210 (.052) (10, 6, ρ) -513.17 (13.71) 0.6950 1.000 0.8000 1.000 0.7800 0.990 89.42% (1.65%) 0.6174 (.053) (11, 4, ρ) -513.25 (13.88) 0.7425 1.000 0.8333 1.000 0.6300 0.990 89.56% (1.69%) 0.6221 (.053) (11, 5, ρ) -513.19 (13.95) 0.7400 1.000 0.8300 1.000 0.7250 0.995 89.49% (1.71%) 0.6198 (.054) (11, 6, ρ) -513.54 (13.70) 0.7350 1.000 0.8300 1.000 0.7850 0.990 89.44% (1.68%) 0.6183 (.054) (10, 5, 0) -512.11 (13.75) 0.7000 1.000 0.7900 1.000 0.7475 0.995 89.56% (1.66%) 0.6222 (.053) (10, 6, 0) -512.53 (13.73) 0.7025 1.000 0.7967 1.000 0.7875 0.990 89.49% (1.70%) 0.6199 (.054) Table 3.7: Sparsity and average of the accuracy of clustering criteria on different tuning parameters.

(9, 4, 0.1 log n) provides the best result in term of clustering even better that the non penalized model. However, the Sensitive/Specificity criteria of this model is not as good as the others. The model which is used in our simulation (λ, γ, ρ) = (10, 5, 0.1 log n) provides balance results between sparsity criteria and accuracy of clustering criteria. In general, BIC is a good criterion for selecting model but base on the BIC criterion cannot ensure that one would obtain a best model.

Another thing can be observed form Table 3.7 is that adding an ℓ 2 norm in this case can reduce the sparsity and accuracy of clustering criteria. It seems that adding the ℓ 2 penalty will work quite well for the non-penalized model due to the fact that maximizing the log-likelihood function leads to using large positive and negative estimates for the regression coefficients, especially the gating network parameters (see [START_REF] Park | Penalized logistic regression for detecting gene interactions[END_REF]; [START_REF] Bunea | Honest variable selection in linear and logistic regression models via ℓ 1 and ℓ 1 + ℓ 2 penalization[END_REF] for the discussion on logistic regression). However, when the correlations of some features are not strong, and γ not deceases to zero the effect of the quadratic penalty with a small ρ is negligible.

Applications to real data sets

We analyze two real data sets as a further test of the methodology. Here, we investigate the housing data described on the website UC Irvine Machine Learning Repository and baseball salaries from the Journal of Statistics Education (www.amstat.org/publications/jse). This was done to provide a comparison with the work of [START_REF] Khalili | New estimation and feature selection methods in mixture-of-experts models[END_REF], [START_REF] Khalili | Variable selection in finite mixture of regression models[END_REF]. While in [START_REF] Khalili | Variable selection in finite mixture of regression models[END_REF] the authors used Lasso-penalized mixture of linear regression (MLR) models, we still apply penalized mixture of experts (to better represent the data than when using MRL models). We compare the results of each model-based upon two different criteria: the average mean squared error (MSE) between observation values of the response variable and the predicted values of this variable; we also consider the correlation of these values. After the parameters are estimated, we use two following values as predicted values for Y : MIXTURE OF EXPERTS FOR REGRESSION AND CLUSTERING

• Method 1: This method uses the following expected value under the estimated model

Ŷ = E θ(Y |x) = K k=1 π k (x; ŵ)E θ(Y |Z = k, x) = K k=1 π k (x; ŵ)( βk0 + x T βk ).
• Method 2: We use the Bayes's rule in (3.40) to cluster the data and the following value

to predict Y Ŷ = E θ(Y |x, ẑ = k) = βk0 + x T βk .
Note that, for the real data sets we do not consider the MoE model with BIC selection since it is computationally expensive. The red lines represent the estimated densities using kernel density estimation and the black lines show the estimated densities using a GMM.

Housing data

This data set concerns houses' value in the suburbs of Boston. It contains 506 observations and 13 features that may affect the house value. These features are: Per capita crime rate by town (x 1 ); proportion of residential land zoned for lots over 25, 000 sq.ft. (x 2 ); proportion of nonretail business acres per town (x 3 ); Charles River dummy variable (= 1 if tract bounds river; 0 otherwise) (x 4 ); nitric oxides concentration (parts per 10 million) (x 5 ); average number of rooms per dwelling (x 6 ); proportion of owner-occupied units built prior to 1940 (x 7 ); weighted distances to five Boston employment centers (x 8 ); index of accessibility to radial highways (x 9 ); full-value property-tax rate per $10, 000 (x 10 ); pupil-teacher ratio by town (x 11 ); 1000(Bk -0.63) 2 where Bk is the proportion of blacks by town (x 12 ); % lower status of the population (x 13 ). The columns of X were standardized to have mean 0 and variance 1. The response homes in variable of interest is the median value of owner occupied homes in $1000 ′ s, MEDV. [START_REF] Khalili | New estimation and feature selection methods in mixture-of-experts models[END_REF].

histogram of Y = MEDV/sd(MEDV) in Figure 3.6, where sd(MEDV) is the standard deviation of MEDV, [START_REF] Khalili | New estimation and feature selection methods in mixture-of-experts models[END_REF] separated Y into two groups of houses with "low" and "high" values.

Hence, a MoE model is used to fit the response Y ∼ π 1 (x; w)N (y;

β 10 + x T β 1 , σ 2 ) + (1 -π 1 (x; w))N (y; β 20 + x T β 2 , σ 2 ),
where π 1 (x; w) = e w 10 +x T w 1 1 + e w 10 +x T w 1 •

The parameter estimates of the MoE models obtained by MoE-Lasso+ℓ 2 (CA), MoE-Lasso+ℓ 2 (PN), MoE-Lasso (PN) and standard MLE are given in Table 3.8, Table 3.9. We compare our results with those of [START_REF] Khalili | New estimation and feature selection methods in mixture-of-experts models[END_REF] and the non-penalized MoE. In Table 3.10, we provide the result in terms of average MSE and the correlation between the true observation value Y and its predictions Ŷ . Our result provides a least sparse model than the one in [START_REF] Khalili | New estimation and feature selection methods in mixture-of-experts models[END_REF]. Some parameters in these methods have the same value. However, the MSE and the correlation from our methods are better than those [START_REF] Khalili | New estimation and feature selection methods in mixture-of-experts models[END_REF] Hence, in application one would consider the sparsity and the prediction of each estimated parameters. The Lasso +ℓ 2 (PN) and Lasso (PN) are successful in detecting the non-zero coefficient with respect to x 6 in the gating network similar with the result from the method in [START_REF] Khalili | New estimation and feature selection methods in mixture-of-experts models[END_REF] and there is a slightly difference in the gating network between MoE-Lasso +ℓ 2 (PN) and MoE-Lasso (PN) due to the addition of the ℓ 2 norm. With the same tuning parameters λ = 42, γ = 10 and ρ = 0.1 log n, the MoE-Lasso +ℓ 2 (PN) provides a better value of the penalized log-likelihood that the MoE-Lasso +ℓ Considering the case K = 3 and ρ = 0 as an extension. The estimated parameters, the average MSE, and the correlation between the true observation value Y and its prediction Ŷ (based on Method 2) for this case can be found in Table 3.11 and Table 3.12. It turns out that this model provides better results than those with K = 2 (with ρ = 0) in term of prediction.

The BIC criterion with K = 3 is also better than the case with K = 2, -246.844 compares with -292.822.

The histograms of Ŷ (using Method 2) for K = 2 and K = 3 are also compared with the histogram of Y (see Figure 3.6). It turns out that with K = 3 the histogram of Ŷ provides better result in approximating the histogram of the true value.

Histogram of the predicted value for Log of Salary

Histogram of Y Histogram of Ŷ Figure 3.7: Histograms of the log of salary and its predicted value for baseball data. The red lines represent the estimated densities using non-parametric kernel density estimation, and the black lines represents the estimated densities using a GMM. MIXTURE OF EXPERTS FOR REGRESSION AND CLUSTERING

Baseball salaries data

We now consider baseball salaries data set from the Journal of Statistics Education (see also [START_REF] Khalili | Variable selection in finite mixture of regression models[END_REF]) as a further test of the methodology. This data set contains 337 observations and 33 features. We compare our results with the non-penalized MoE models and the MIXLASSO models (see [START_REF] Khalili | Variable selection in finite mixture of regression models[END_REF]). [START_REF] Khalili | Variable selection in finite mixture of regression models[END_REF] used this data set in the analysis, which included an addition of 16 interaction features, making in total 32 predictors. The columns of X were standardized to have mean 0 and variance 1. Histogram of the log of salary (see Figure 3.7) shows multimodality making it a good candidate for the response variable under the MoE model with two components:

Y = log(salary) ∼ π 1 (x; w)N (y; β 10 + x T β 1 , σ 2 ) + (1 -π 1 (x; w))N (y; β 20 + x T β 2 , σ 2 ).
By taking all the tuning parameters to zero, we obtain the maximum likelihood estimator of the model. We also compare our result with MIXLASSO from [START_REF] Khalili | Variable selection in finite mixture of regression models[END_REF]. Table 3.13 and Table 3.14 present the estimated parameters for baseball salary data and Table 3.15 shows the results in terms of MSE, and R 2 between the true value of Y and its predicted values. These results suggest that the proposed algorithms with the Lasso+ℓ 2 penalty also shrink some parameters to zero and have acceptable results compared to MoE. It also shows that the proposed models provide better results than that of the MIXLASSO model. The two algorithms MoE-Lasso+ℓ 2 (CA) and MoE-Lasso+ℓ 2 (PN) converge to similar sets of parameters and the same penalized log-likelihood values, -292.9301 and -292.9294. Hence, they all provide similar results based on MSE and R 2 criteria. However, the method using proximal Newtontype procedure takes few seconds to obtain the estimate parameters, while for method using coordinate ascent few minutes are required to estimate these coefficients. We compare the CPU times of these methods in the next section.

3.4.6 CPU times and discussion for the high-dimensional setting

CPU times

For the CPU times, we compare two methods: the coordinate ascent algorithm (MoE-Lasso+ℓ 2 (CA)) and the proximal Newton algorithm (MoE-Lasso+ℓ 2 (PN)). We test these algorithms on different data sets. The first data set is one of 100 data sets used for the simulation study. With this data set, we run these algorithms 10 times with different number of clusters K = 2 and K = 3. The second data set is the baseball salaries. Finally, we also consider the residential building data set (UCI Machine Learning Repository) as a further comparison. The results for this data set is given in the next section. The computer used for this work has CPU Intel i5-6500T 2.5GHz with 16GB RAM.

The obtained results are given in Table 3.16. We can see that the algorithm for the residential data which has a quite high number of features, requires only few minutes and is thus has a very reasonable speed, and for moderate dimensional problems, is very fast. 3.13: Fitted models for baseball salary data with the standard MoE using MLE and MIXLASSO [START_REF] Khalili | Variable selection in finite mixture of regression models[END_REF].

Discussion for the high-dimensional setting

To evaluate the algorithm in a situation in which we have a high number of features, we consider the Residential Building Data Set (UCI Machine Learning Repository). This data set contains 372 and 108 features with the two response variables (V-9 and V-10), which represent the sale prices and construction costs. All the features are standardized to have zero-mean and unit-variance. We choose the V-9 variable (sale prices) as the response variable to be predicted.

- - - - - - x 1 * x 13 0.3218 - - 0.3203 - - x 1 * x 14 - - - - - - x 1 * x 15 - - - - - - x 1 * x 16 -0.0319 - - -0.0321 - - x 3 * x 13 - 0.0284 -0.5828 - 0.0254 -0.5876 x 3 * x 14 -0.0883 - - -0.0878 - - x 3 * x 15 - - - - - - x 3 * x 16 - - - - - - x 7 * x 13 - 0.004 - - 0.014 - x 7 * x 14 -0.1362 0.0245 - -0.1365 0.0260 - x 7 * x 15 - - - - - - x 7 * x 16 - - - - - - x 8 * x 13 - 0.2727 -0.3628 - 0.2801 -0.3621 x 8 * x 14 - 0.0133 - - 0.0109 - x 8 * x 15 0.3154 - - 0.3151 - - x 8 * x 16 0.0157 - - 0.0160 - -
Table 3.14: Fitted models for baseball salary data with our proposed algorithms: MoE-Lasso + ℓ 2 (CA) and MoE-Lasso + ℓ 2 (PN).

Histogram of this variable is given in Figure 3.8. A MoE model with K = 3 expert components is chosen to fix this data set. The MoE-Lasso +ℓ 2 (PN) method is used to estimate the parameters.

We provide the results of our algorithm with K = 3 and λ = 15, γ = 5. The estimated parameters are given in Table 3.17, Table 3.18 and Table 3.19.

The correlation and the mean squared error between the true value V-9 with its prediction can be found in Table 3.20. These results show that the proximal Newton method performs well 3. 4 Table 3.21: Results for the residential building data set by using the clustering method 2.

An experiment for p > n:

To consider the high-dimensional setting, we take the first n = 90 observations of the residential building data with all the p = 108 features. Histogram of the variable V-9 in this case is provided in Figure 3.9. We use a mixture of three experts and obtain the results by applying the MoE-Lasso+ℓ 2 (PN) algorithm. The parameter estimation results are provided in Table 3.24, Table 3.25 and Table 3.26. The results in terms of correlation and the mean squared error between the true value V-9 and its predictions are given in Table 3.22 and Table 3.23. From these Tables we can see that, in this high-dimensional setting, one still obtains acceptable results for the regularized MoE models and the EM algorithm using the proximal Newton method (MoE-Lasso+ℓ 2 (PN)) is an efficient tool for the parameter estimation. The running time in this experiment is about only few (∼ 8) minutes and the algorithm is quite effective in this setting.

Histogram of the predicted value for V-9 variable

Histogram of Y Histogram of Ŷ Figure 3.8: Histograms of the variable Y (V-9) and its predicted value for residential building data. The red lines represent the estimated densities using non-parametric kernel density estimation, and the black lines show the densities estimed using a GMM.

Histogram of the predicted value for V-9 variable

Histogram of Y Histogram of Ŷ Figure 3.9: Histograms of the variable Y (V-9) and its predicted value for the subset of residential building data. The red lines are the estimated densities using non-parametric kernel density estimation, and the black lines show the estimated densities using a GMM.

- x - - - - - x - - - - - x -0.01495 -0.00103 0.03774 - - x - - - - - x - -0.03344 - - - x - 0.06296 - - - x 0.04560 0.02466 - - - x 0.02368 0.03210 - - - x - -0.00546 -0.00398 - - x - -0.03934 - - - x - -0.04612 - - - x 0.01205 -0.00352 - - - x - - - - - x - 0.00409 - - - x - - - - - x - - 0.00047 - - x - - - - - x - 0.03494 0.04131 - - x - -0.00003 0.02288 - - x - - - - - x - - - - - x - 0.01468 -0.01095 - -
x - - - - - x - 0.00899 - - - x - 0.00061 - - - x -0.01694 -0.00559 - - - x 0.10214 0.02533 - 0.07086 - x 0.03770 - - - - x - -0.04162 - - - x - - - - - x - 0.00561 0.01148 - - x - 0.00770 - - - x - - - - - x - - - - - x -0.07316 0.03138 - - - x - 0.00493 -0.00183 - - x - 0.01320 - - - x -0.00076 -0.00041 - - 0.03819 x - - - - - x - - - - - x -0.00854 0.00077 - - - x - 0.00039 - - - x - - -0.11177 - - x - 0.00334 - - - x
0.04779 0.00405 0.00733 0.35226 x 0.06726 0.03743 0.02988 0.08489 -0.20694 x 0.02520 0.00128 0.01473

- - x - 0.00843 - - - x - 0.00034 - - - x - -0.00920 0.01184 - - x - 0.00002 - - - x - - - - - x - - - - - x -0.03840 - 0.02505 - - x - 0.00234 0.00238 - - x - - - - - x 0.06026 0.01750 0.05879 - - x - - - - -
x - -0.03636 - - - x - - -0.02932 - - x - - - - - x -0.02725 -0.02474 - - - x -0.01399 -0.16005 -0.08654 - - x - 0.00526 - - - x -0.05816 0.02821 - 0.01303 -0.35566 x - -0.00358 - 1.12522 - x -0.05416 - - - - x - - - - - x - - 0.04329 - - x - - - - - x - - - - - x - - - - - x - 0.00783 - - - x - - 0.01463 - - x 0.02337 0.03903 - - - x -0.04720 0.00909 - - - x - - - - - x - - - - - x -0.00070 -0.00626 -0.00458 - - x - - - - - x -0.00067 0.00309 - - - x - -0.00925 - - - x -0.00705 -0.00656 - - 0.03610 x - -0.00406 - - - x - 0.00714 0.01911 0.06610 - x - 0.00364 - - - x 100 - 0.00327 - - - x 101 - 0.02858 0.03974 - - x 102 0.01623 -0.01236 - - - x 103 - - - - - x 104 - - - - - x 105 - 0.00215 - - - x 106 -0.00006 -0.00129 - - - x 107 - 0.00851 - - -
Table 3.19: Fitted model parameters for residential building data obtained by our MoE-Lasso +ℓ 2 (PN) algorithm (part 3).

Conclusion and future work

- - x -0.03158 - - - - x - - -0.00001 - - x -0.02218 0.00987 -0.00527 - - x - - - - - x - - -0.10258 - - x -0.06036 - - - - x 0.03513 - -0.00602 - - x 0.01947 0.12495 0.07810 - - x -0.00347 0.01317 - - - x -0.03255 -0.00125 - - - x -0.06659 -0.00007 - - - x 0.03478 - 0.01314 - - x 0.01209 0.03787 -0.00287 - - x - - - - - x - - - - - x 0.06476 0.02369 -0.00461 - - x -0.01017 -0.00813 0.01805 - - x 0.03331 - - - - x -0.03870 0.01708 - - - x - - - - - x - - - - - x
0.02278 -0.02794 0.01933 --Table 3.24: Fitted model parameters for the subset of residential building data obtained by our MoE-Lasso +ℓ 2 (PN) algorithm (part 1).

regularized objective towards at least a local maximum. The proposed regularization does not require using approximations as in standard MoE regularization. The proposed algorithms are based on univariate updates of the model parameters via an MM, coordinate ascent, proximal Newton method, which allow to tackle matrix inversion problems and obtain sparse solutions.

CONCLUSION AND FUTURE WORK Features

Expert, σ = 0.0159 Gating network Exp.1 Exp.2 Exp.3 Gate.1 Gate.2 x - - - - - x -0.09359 - -0.06125 - - x - - -0.00356 - - x -0.11611 - -0.01973 - - x 0.21178 0.06134 0.13879 - - x 0.09095 - - - - x -0.03243 - - - - x -0.00032 - -0.01455 - - x -0.01643 - - - - x -0.03152 0.01812 -0.02303 - - x - - - - - x - - - - - x 0.13661 0.00862 - - - x 0.04914 0.06704 - - - x 0.00424 - -0.02954 - - x 0.04225 0.05518 -0.01411 - - x - - - - - x -0.01697 - - - - x 0.02922 0.00057 -0.00501 - - x - - - - - x - -0.02272 0.00131 - - x - - - - - x 0.11223 - 0.05349 - - x 0.23868 -0.00711 0.07830 - - x -0.07807 -0.05727 - - -0.02819 x -0.06729 - - - - x -0.02121 - - - - x -0.01886 0.04294 0.00548 - - x -0.01265 0.02236 - - - x - - - - - x - - - - - x -0.03609 - - - - x -0.07929 0.01190 -0.00001 - - x - - - - - x 0.09774 -0.01388 0.01683 - - x - - - - -
Table 3.25: Fitted model parameters for the subset of residential building data obtained by our MoE-Lasso +ℓ 2 (PN) algorithm (part 2).

The results in terms of parameter estimation, the estimation of the actual support of the sparsity, and clustering accuracy, obtained on simulated and three real data sets, confirm the effectiveness of our proposal at least for problems of moderate dimension. Namely, the model sparsity does not include significant bias in terms of parameter estimation nor in terms of recovering the actual

-0.06835 - - - - x - - -0.00166 - - x -0.07267 - - - - x -0.00061 0.02782 - - - x - - - - - x - - - - - x -0.02223 0.02194 0.03417 - - x 0.00029 - - - - x - - - - - x -0.06311 0.03682 -0.00977 - - x - - - - - x - - - - - x 0.06938 -0.03040 -0.00542 - - x - - - - - x 0.05246 - -0.00793 - - x -0.01214 - -0.00345 - - x - -0.06544 -0.00007 - - x 0.03763 - - - - x 0.04560 0.04346 0.00717 - - x 0.03892 - -0.01578 - - x 100 0.01633 - -0.01509 - - x 101 0.04869 0.01218 0.00076 - - x 102 -0.01996 - - - - x 103 - - - - - x 104 - - - - - x 105 -0.00248 - - - - x 106
-0.00344 -0.03221 0.01461 -x 107 -0.00779 -0.01415 0.00106 --Table 3.26: Fitted model parameters for the subset of the residential building data obtained by our MoE-Lasso +ℓ 2 (PN) algorithm (part 3).

clusters of the heterogeneous data. The obtained models with the proposed approach are sparse which promote its scalability to high-dimensional problems. The hybrid EM/MM algorithm is a potential approach. However, this model should be considered carefully, especially for nonsmooth penalty functions. The coordinate ascent approach for maximizing the M-step, however, works quite well although, while we do not have the closed form update in this situation. A proximal Newton extension is possible to obtain closed form solutions for an approximate of the M-step as an efficient method that is promoted to deal with high-dimensional data sets. First experiments on an example of a quite high-dimensional scenario with a subset of real data containing 90 observations and 108 features provide encouraging results. The next chapter will consist of investigating more the high-dimensional setting as well as considering MoE for discrete data.

For further research directions, recently, [START_REF] Jiang | Model selection for the localized mixture of experts models[END_REF] proposed a regularized approach for localized MoE models [START_REF] Xu | An alternative model for mixtures of experts[END_REF], in which the covariate variable X is assumed to have a mixture of Gaussian distributions and corresponds to each component the respond variable Y is described by a Gaussian regression function. The penalized part emphasizes on feature selection in the regression parts. Thus, it restricts their model since the covariance matrix of each mixture component cannot be estimated efficient in large-scale scenario. We can consider a sparse representations of these covariance matrices via the work of [START_REF] Fop | Model-based clustering with sparse covariance matrices[END_REF]. The method introduce in this chapter can directly applied for the hierarchical MoE models [START_REF] Jordan | Hierarchical mixtures of experts and the EM algorithm[END_REF]. However, theoritical results to support the penalized method in hierarchical MoE framework should be studied more. This chapter has lead to the following publications [START_REF] Huynh | Regularised maximum-likelihood inference of mixture of experts for regression and clustering[END_REF]; Chamroukhi and Huynh (2018Huynh ( , 2019)).

Introduction

The work presented in this chapter involves regularized MoE models for discrete data, including classification. Specifically, we consider the MoE models for counting data and classification data. While the MoE fitting by maximum likelihood (MLE) is widely used, the study of MoE in high-dimensional problems is still challenging due to the well-known problems of the ML 4.1. INTRODUCTION estimator in such a setting. Indeed, when the number of features in the data becomes being large, the features can be correlated and therefore the number of actual predictors/features that explain the problem are smaller. Additionally, numerical instability can also arise in the MLE of a MoE model in high-dimensional setting. For example in regression, maximizing the log-likelihood function leads to using large positive and negative estimates for the regression coefficients, corresponding to the correlated features when the number of features is moderate or large and highly correlated. This behavior can be observed in logistic regression; see [START_REF] Park | Penalized logistic regression for detecting gene interactions[END_REF] and [START_REF] Bunea | Honest variable selection in linear and logistic regression models via ℓ 1 and ℓ 1 + ℓ 2 penalization[END_REF] for more details. In a MoE scenario, estimating the parameters with moderate numbers of features and mixture components using MLE is challenging. A better fitting can indeed be achieved by regularizing the objective function so that to encourage sparse solutions. Feature selection by regularized inference encourages sparse solutions, with a reasonable computational cost.

Several approaches have been proposed to deal with the feature selection task. The wellknown Lasso method [START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF] is one of the most popular and successful regularization technique that encourages sparsity, which utilizes the ℓ 1 penalty to regularize the squared error function and achieve parameter estimation and feature selection. Extensions of the Lasso, based on penalized log-likelihood criteria with convex and nonconvex penalty functions has been proposed, including elastic net [START_REF] Zou | non supervisée, de telles situations de données à distributions complexes. Les modèles de mélange se sont affirmés comme l'approche de choix dans la modélisation de données hétérogènes, dans de nombreux problèmes de l'approche statistique de la science des données, y compris en estimation de densité et en classification[END_REF], group Lasso (Yuan and Lin, 2006b), adaptive Lasso [START_REF] Zou | The adaptive Lasso and its oracle properties[END_REF], smoothly clipped absolute deviation (SCAD) [START_REF] Fan | Variable selection via nonconcave penalized likelihood and its oracle properties[END_REF], minimax concave penalty (MCP) [START_REF] Zhang | Nearly unbiased variable selection under minimax concave penalty[END_REF]. Each method has its own advantages. The convex penalty functions are easy to handle due to the existence of efficient techniques from convex optimization to fit the models, while the nonconvex penalty functions involve practical challenges in fitting these models.

In a related model, [START_REF] Peralta | Embedded local feature selection within mixture of experts[END_REF] considered MoE with logistic regression model for the experts and proposed an EM algorithm based on inverting the soft-max function to estimate their Lasso regularized logistic MoE model. Unfortunately, the authors did not give any evidence that their EM algorithm improves the objective function after each iteration loop. To tackle the difficulty of updating the coefficients of the gating network, [START_REF] Jiang | Model selection for the localized mixture of experts models[END_REF] introduced a penalized likelihood method for the localized MoE models [START_REF] Xu | An alternative model for mixtures of experts[END_REF]. One limitation of their method lies in the fact that the local covariance matrix is updated normally in the M-step. Thus, it poses some disadvantages if one would like to apply their method in large scale scenario.

In this chapter, we propose an efficient regularized estimation and feature selection of Mixtures-of-Experts that encourages sparse solutions and consider MoE models for two common generalized linear models. We develop a proximal Newton-EM algorithm to maximize the proposed ℓ 1 -penalized log-likelihood function, in which a proximal Newton-type method for maximizing the M-step is used. An advantage of using proximal Newton-type method lies in the fact that one just need to solve weighted quadratic Lasso problems to update the parameters. Efficient tools such as coordinate ascent algorithm can be used to deal with these problems. DATA Hence, the proposed approach does not require an approximate of the regularization term, and allow to automatically select sparse solutions without thresholding. Our approach is shown to perform well including in a high-dimensional setting and to outperform competitive state of the art regularized MoE models on several experiments on simulated and real data. This chapter is organized as follows. Section 4.2 describes the modeling with MoE for heterogeneous data and maximum-likelihood parameter estimation. Then, in Section 4.3, the proposed regularized maximum likelihood strategy of the MoE models and the EM-based algorithm are developed. Our method is applied on simulated and real data sets in Section 4.4. Finally, we also discuss the effectiveness of our method in dealing with moderate dimensional problems and provide an experiment study to promotes our method in high-dimensional setting in Section 4.5.

Mixture of experts and maximum likelihood estimation for discrete data

Let ((X 1 , Y 1 ), . . . , (X n , Y n )) be a random sample of n independently distributed pairs (X i , Y i ), (i = 1, . . . , n) where Y i ∈ Y ⊂ N is the ith discrete response variable given some vector of p ∈ N predictors X i ∈ X ⊂ R p , and let D = ((x 1 , y 1 ), . . . , (x n , y n )) be an observed data sample. We consider the MoE modeling for the analysis of a heteregeneous set of such data (see Section 2.3.3). The model can be summarized as follows:

The mixture of experts model for discrete data

Assume that the observed pairs (x, y) are generated from K ∈ N parametric probability mass components (the experts) p z (y|x; θ), z ∈ [K] = {1, . . . , K}, governed by a gating network π z (x; w) represented by a hidden categorical random variable Z ∈ [K] that indicates the expert to which a particular observed pair belongs. In MoE framework, these gating network functions π z (x; w) are modeled by logistic functions. Hence, given the predictor or the input x i , the categorical variable Z i is generated according to the multinomial distribution:

Z i |x i ∼ Mult(1; π 1 (x i ; w), . . . , π K (x i ; w)) (4.1)
where

π k (x i ; w) = P(Z i = K|X i = x i ) = exp(w k0 + x T i w k ) 1 + K-1 l=1 exp(w l0 + x T i w l )
, such that w k = (w k0 , w T k ) T and w K = 0 is set to the null vector for identifiability (Jiang and Tanner, 1999a). Then, conditional on the hidden variable Z i = z i and x i , the observed random variable Y i is assumed to be generated from the expert z i distribution p z i (y i |x i ; θ z i ), that is:

Y i |Z i = z i , X i = x i ∼ p z i (y i |x i ; θ z i ) (4.2)
where p z i (y i |x i ; θ z i ) = p(y i |Z i = z i , X i = x i ; θ z i ) is the probability mass function of the expert z i depending on the nature of the data (x, y) within the group z i . Hence, formally, the MoE is defined by the following semi-parametric probability mass function:

p(y i |x i ; θ) = K k=1 π k (x i ; w)p k (y i |x i ; θ k ) (4.3)
with the parameter vector defined by θ = (w T 1 , . . . , w T K-1 , θ T 1 , . . . , θ T K ) T where θ k (k = 1, . . . , K) is the parameter vector of the kth expert.

For a complete account of MoE, types of gating networks and expert networks, the reader is refereed to [START_REF] Nguyen | Practical and theoretical aspects of mixture-ofexperts modeling: An overview[END_REF].

Maximum likelihood parameter estimation

For an observed data sample D = ((x 1 , y 1 ), . . . , (x n , y n )) generated from the MoE model ( 4.3), the unknown parameter vector θ is commonly estimated by maximizing the observed data loglikelihood

L(θ) = n i=1 log K k=1 π k (x i ; w)p k (y i |x i ; θ k ), (4.4)
by using the EM algorithm. Unfortunately, the MLE can be unstable or even infeasible in high-dimension due to possibly redundant and correlated features. In some cases, such as multilogistic model, this task becomes a challenge since the log-likelihood function becomes singular. Indeed, Rosset et al. (2004) showed that in logistic regression if the data are separable by the predictors the maximum likelihood solutions are undefined. In such cases, a regularization of the MLE is needed. For example, Park and Hastie (2007a) considered adding a small amount of quadratic penalization, and let the coefficients converge to the ℓ 2 penalized logistic regression solutions instead of infinity. Here, we consider the regularized models with take into account the well-known Lasso penalty.

Regularized maximum likelihood estimation

Regularized MLE allows the selection of a relevant subset of features for prediction and thus encourages sparse solutions. This approach also bounds the norm of the estimated parameters. Hence, it avoids the singularity of the penalized log-likelihood. In mixture-of-experts context, one may consider both sparsity in the feature space of the gates, and of the experts. We utilize Lasso penalty for the experts, and for detecting zero and nonzero coefficients in the mixing functions another Lasso penalty is added for the gating parameters. This approach has some advantages since the Lasso penalty functions are convex and convenient to handle. Moreover, by adding these penalty functions, we also force some parameters, not only in the experts but also in the gating networks, toward zero, to obtain a sparse model. The proposed regularization that combines two Lasso penalties for the experts parameters, and for the gating network is DATA defined by: (4.5) where v 1 = p j=1 |v j | is the ℓ 1 norm of a vector v ∈ R p , λ k ≥ 0 for all k = 1, . . . , K and γ k ≥ 0 for all k = 1, . . . , (K -1). The regularization parameters λ k and γ k control the amount of shrinkage on the parameters β k and w k . A similar strategy has been proposed in [START_REF] Khalili | New estimation and feature selection methods in mixture-of-experts models[END_REF] for Gaussian regression based on two well-known penalized techniques: Lasso [START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF] and SCAD [START_REF] Fan | Variable selection via nonconcave penalized likelihood and its oracle properties[END_REF] which are then approximated in the EM algorithm of the model inference. An ℓ 2 penalty function for the gating network is added to avoid wildly large positive and negative estimates of the regression coefficients corresponding to the mixing proportions. This behavior can be observed in logistic/multinomial regression when the number of potential features is large and they are highly correlated [START_REF] Park | Penalized logistic regression for detecting gene interactions[END_REF][START_REF] Bunea | Honest variable selection in linear and logistic regression models via ℓ 1 and ℓ 1 + ℓ 2 penalization[END_REF]. However, the ℓ 2 norm also affect the sparsity of the models (see Section 3.4). We therefore remove this ℓ 2 penalty in our proposal model. For parameter estimation, [START_REF] Khalili | New estimation and feature selection methods in mixture-of-experts models[END_REF] introduced an EM algorithm follows the suggestion of [START_REF] Hunter | Variable selection using MM algorithms[END_REF]. Unfortunately, this approach draws some inadequacies (see discussion in Section 3.3) which can be tackled by our proposals in Chapter 3.

P L(θ) = L(θ) - K k=1 λ k β k 1 - K-1 k=1 γ k w k 1 .
In a similar scenario, [START_REF] Peralta | Embedded local feature selection within mixture of experts[END_REF] suggested an EM algorithm for the regularized MoE of logistic regression, in which using a transformation that implies inverting the soft-max function. However, there is no evidence to ensure the increasing of their penalized log-likelihood values and this leads to the poor results from their approach. In our approach presented here, we propose and EM algorithm which relies on proximal Newton-type procedures in the M-step to overcome these limitations. We consider that in mixture of experts with two different models for the experts, that is Poisson, and logistic regressors.

Parameter estimation and feature selection via a proximal Newton-EM

For each of the two considered GLM for the MoE models, we propose an EM algorithm to monotonically find at least local maximizers of (4.5). The E-step is common to these models. For the M-step, two different algorithms are proposed to update the model parameters. Specifically, the first one relies on proximal Newton method, while the second one uses a proximal Newtontype method to update the gating network and expert's parameters. The difference between these algorithms is that the proximal Newton-type method we construct here to update the gating network can avoid the numerical instability of the proximal Newton method due to the small value of the mixing proportions. We discuss this difference in Section 4.3.2. The EM algorithm for the maximization of (4.5) requires the construction of the penalized complete-data log-likelihood, which is, in our context, given by

P L c (θ) = L c (θ) - K k=1 λ k β k 1 - K-1 k=1 γ k w k 1 (4.6)
where

L c (θ) = n i=1 K k=1 z ik log [π k (x i ; w)p k (y i |x i ; θ k )] (4.7)
is the standard complete-data log-likelihood for the MoE model where z ik an indicator binaryvalued variable such that z ik = 1 if Z i = k (i.e., if the ith pair (x i , y i ) is generated from the kth expert component) and z ik = 0 otherwise. Thus, the proposed EM algorithm for the regularized MoE model in its general form runs as follows. After starting with an initial solution θ [0] , it alternates between the two following steps until convergence (e.g., when there is no longer a significant change in the relative variation of (4.5)).

E-step:

The E-Step computes the conditional expectation of the penalized complete-data loglikelihood (4.6), given the observed data D and a current parameter vector θ [q] , q being the current iteration number of the block-wise EM algorithm:

Q(θ; θ [q] ) = E P L c (θ)|D; θ [q] = n i=1 K k=1 τ [q] ik log [π k (x i ; w)p k (y i |x i ; θ k )] - K k=1 λ k β k 1 - K-1 k=1 γ k w k 1 (4.8)
where τ

[q] ik = P(Z i = k|y i , x i ; θ [q] ) = π k (x i ; w [q] )p k (y i |x i ; θ

[q] k )/p(y i |x i ; θ [q] ) (4.9)

is the conditional probability that the data pair (x i , y i ) is generated by the kth expert. This step only requires the computation of the conditional component probabilities τ ik (i = 1, . . . , n) for each of the K experts.

M-step:

The M-Step updates the parameters by maximizing the Q function (4.8) with respect to θ. The Q-function can be written as:

Q(θ; θ [q] ) = Q(w; θ [q] ) + K k=1 Q k (θ k ; θ [q] ) (4.10) with Q(w; θ [q] ) = n i=1 K k=1 τ [q] ik log π k (x i ; w) - K-1 k=1 γ k w k 1 , = n i=1 K-1 k=1 τ [q] ik (w k0 + x T i w k ) - n i=1 log 1 + K-1 k=1 e w k0 +x T i w k - K-1 k=1 γ k w k 1 . (4.11)
and q] ) in (4.12), with

Q k (θ k ; θ [q] ) = n i=1 τ [q] ik log p k (y i |x i ; θ k ) -λ k β k 1 . (4.12) 4.3. REGULARIZED MAXIMUM LIKELIHOOD ESTIMATION of the function Q k ({β k0 , β k }; θ [
Q k ({β k0 , β k }; θ [q] ) = n i=1 τ [q] ik -exp(β k0 + x T i β k ) + y i (β k0 + x T i β k ) -log(y i !) P k ({β k0 ,β k };θ [q] ) -λ k β k 1 .
(4.14) This composite function is concave, nonsmooth and has a non quadratic form. Therefore, the proximal Newton method can be used to update β k . Following the strategy that was used to update the gating network, one needs to compute the quadratic approximation Pk ({β k0 , β k }; θ [q] ) of P k ({β k0 , β k }; θ [q] ) at ( βk0 , βk ). This function is given by (see Appendix C.3.1 for more details)

Pk ({β k0 , β k }; θ [q] ) = - 1 2 n i=1 a ik (b ik -β k0 -x T i β k ) 2 + D( βk0 , βk ), (4.15)
with

a ik = τ [q] ik exp( βk0 + x T i βk ); b ik = y i exp( βk0 + x T i βk ) -1 + βk0 + x T i βk ;
and D( βk0 , βk ) is a function of ( βk0 , βk ).

After that, the coordinate ascent algorithm with soft-thresholding operator is used to maximizing the penalized weighted least-square max

(β k0 ,β k ) Pk ({β k0 , β k }; θ [q] ) -λ k β k 1 . (4.16)
Then the solution is taken in account for the next update of the proximal Newton algorithm. This can be interpreted as in Algorithm 3.

Algorithm 3 Proximal Newton method for Poisson model

1: (β (0) k0 , β (0) k ) = (β [q] k0 , β [q] k ). 2: repeat 3:
Update the quadratic approximation Pk ({β k0 , β k }; θ [q] ) in (4.15) using the current parameters.

4:

Solve the penalized weighted least-square problem in (4.16) by using coordinate ascent algorithm and let ( β(s) k0 , β(s) k ) be the solution.

5:

Set (β

(s+1) k0 , β (s+1) k ) = (1-t)(β (s) k0 , β (s) k )+t( β(s) k0 , β(s) k )
, where t is found using a backtracking line-search.

6:

Evaluate the objective function Q k ({β k0 , β k }; θ [q] ) at (β

(s+1) k0 , β (s+1) k
). 7: until the stopping criterion is satisfied. DATA

Expert network with Multinomial outputs

Finally, for MoE for classification, assuming that each expert part is governed by a multinomial distribution with R (≥ 2) levels and the probability distribution of Y i given x i and z i becomes a multinomial-logistic distribution, i.e, (4.2) is defined by

Y i |Z i = z i , x i ∼ Mult(1; α z i 1 (x i ; β z i ), . . . , α z i R (x i ; β z i ))
where

α kr (x i ; β k ) = P(y i = r|x i ; z i = k) = exp(β kr0 + x T i β kr ) 1 + R-1 l=1 exp(β kl0 + x T i β kl )
, r ∈ {1, . . . , R} with (β kR0 , β kR ) = 0. Denote by U the n × R indicator response matrix with elements u ir = I(y i = r). Then Q k (β k ; θ [q] ) in (4.12) is written in the more explicit form

Q k (β k ; θ [q] ) = n i=1 τ [q] ik R-1 r=1 u ir (β kr0 + x T i β kr ) -log 1 + R-1 r=1 exp(β kr0 + x T i β kr ) I(β k ) - R-1 r=1 λ kr β kr 1 .
(4.17) The same strategy for updating the gating network by using proximal Newton method can be applied in this case. It is not hard to show that the local quadratic approximation Ĩr (β k ) of I(β k ) with respect to (β kr0 , β kr ) at βk is given by (see Appendix C.3.2)

Ĩr (β k ) = - 1 2 n i=1 τ [q]
ik d ikr (c ikrβ kr0x T i β kr ) 2 + E( βk ), (4.18) where The proximal Newton-type method can be suggested by replacing the Hessian matrix with the constant matrix B = -1/4 n i=1 τ

c ikr = βkr0 + x T i βkr + u ir -α kr ( βk ; x i ) α kr ( βk ; x i )(1 -α kr ( βk ; x i )) , (4.19)
d ikr = α kr ( βk ; x i )(1 -α kr ( βk ; x i )), ( 4
[q] ik x i x T i to avoid possible numerical instability. In such a case, instead of maximizing the weighted Lasso in (4.21) one will maximize a simple weighted ik (ĉ ikrβ kr0x T i β kr ) 2 + Ê( βk )λ kr β kr 1 , (4.22) where ĉikr = βkr0 + x T i βkr + 4(u irα kr ( βk ; x i )), and Ê( βk ) is a function of βk .

Algorithm tuning and model selection

The appropriate values of the tuning parameters (λ, γ) and the number of clusters K should be chosen carefully in practice. Here, we suggest to use a modified BIC with a grid search scheme introduced by [START_REF] Städler | l1-penalization for mixture regression models[END_REF] for this difficult task. This strategy was mentioned in Section 3.3.4. However, it is worth to note that choosing optimal values of the tuning parameters and the number of components for penalized MoE models is still an open research. Though, the BIC performs quite well for our simulation study.

Experimental study

The performance of these methods is studied on both simulated data and real data. The results of these algorithms are compared to the standard non-penalized MoE for the Poisson model (denoted by PMoE) and MoE with ℓ 2 regularization (LMoE+ℓ 2 ) for the logistic model since local maximum parameters that closed to the true value for the MoE of logistic model cannot found in some cases. Several evaluation criteria are used to assess the performance of the models, including sparsity, parameters estimation and clustering criteria. Our penalized models are denoted by PMoE-Lasso and LMoE-Lasso for the Poisson and logistic cases, respectively.

The R packages of codes of the developed algorithms and the documentation are publicly available on this link1 .

Evaluation criteria

To evaluate our methods, the results of all the models are compared based on three different criteria which were mentioned in Section 3.4.1. These criteria are: sensitivity/specificity, parameters estimation, and clustering performance. To deal with the label switching before calculating these criteria, we permuted the estimated coefficients based on an ordered between the expert parameters then replace the kth gating network vector with w per k = w kw K . Specially, for K = 2, the log-likelihood function and the penalized log-likelihood function will not change since we have w per 1 = -w 1 .

EXPERIMENTAL STUDY

Sensitivity/specificity criteria Table 4.1 presents the sensitivity (S 1 ), specificity (S 2 ) values for the experts 1 and 2, and the gates for each of the considered models. The non-penalized MoE models cannot be considered as model selection methods since their sensitivity almost surely equals zero, hence the results for these models are not provided. Especially, the estimated parameters for the logistic model with the standard MoE becomes challenging and unstable. For a typical data set, local maximum parameters that closed to the true value for the MoE of logistic model cannot found (see Table 4.2). Actually, in case of logistic regression Rosset et al. (2004) showed that if the data are separable by the predictors the maximum likelihood solutions are undefined. In such cases, by adding a small amount of quadratic penalization, Park and Hastie (2007a) let the coefficients converge to the ℓ 2 penalized logistic regression solutions instead of infinity as the Lasso tuning parameter λ approaches zero. Based on these ideas, we add two small quadratic penalizations for the MoE model with logistic outputs

ρ 2 K k=1 R-1 r=1 β kr 2 2 and ρ 2 K-1 k=1 w k 2 2
where ρ = 0.1 log(n) as in Chapter 3. Hence, the MoE with Lasso regularization model in the logistic case is compared with the ℓ 2 regularized MoE model. Here, the Lasso performs quite well for detecting non-zero coefficients both in the experts and in the gating network. By adding the penalty term, one can avoid the instability of the estimators. In the case with high correlation between features, one can consider adding ℓ 2 penalties for the experts and the gating network.

Model

Parameter estimation

The boxplots of all estimated parameters are shown in Figures 4.2 and 4.3. The boxplots are not provided for standard logistic model since the estimating parameter for this model is unstable in this case. It turns out that the PMoE and LMoE+ℓ 2 could not be considered as model selection methods. The Lasso provides sparse results for the model, both in the experts and in the gates. These Lasso models work quite well in detecting non-zero coefficients. However, in the logistic case, this becomes more challenging in the experts and in the gating network.

For the mean and standard derivation shown in Table 4.3 and Table 4.4. Notice that, in case of Poisson outputs the model using standard MoE give better results than PMoE-Lasso. This is because the regularized model can cause bias to the estimated parameters since the penalty functions are added to the log-likelihood function. On the other hand, the PMoE-Lasso provides better results than PMoE for estimating the zero coefficients in term of average mean squared error.

For the logistic model, it turns out that by adding two small quadratic penalty functions one can avoid the instability of estimation the MLE parameters. The LMoE+ℓ 2 has better results than the LMoE-Lasso for estimating the non-zero coefficients. Conversely, as in the Poisson model the Lasso is good at detecting zero coefficients and provides small MSE than LMoE+ℓ 2 at the zero parameters.

Clustering

The accuracy of clustering for all these mentioned models are calculated for each data set. The results in terms of ARI and correct classification rate values are provided in Table 4.5. For the Poisson case, the PMoE-Lasso model provides a better result than the PMoE model in accuracy of clustering term. However, the difference between these model is smaller than 1%. For the logistic case, it can be seen that, with LMoE-Lasso model we can obtain better results than LMoE+ℓ 2 model in this simulation.

In summary, it is clear that the regularized methods perform quite well in retrieving the actual sparse support; the sensitivity and specificity results are quite reasonable for the proposed models. Although the penalty function will cause bias to the parameters, as shown in the results of the MSE, the algorithm can perform parameter density estimation with an acceptable loss of information due to the bias induced by the regularization. In terms of clustering, the PMoE-Lasso works as well as PMoE models for the Poisson model. For logistic model, the LMoE-Lasso is successful in retrieving the actual parameters used for the model, while the non regularized method failed in this task. Hence, adding quadratic penalty functions in this case is necessary. A comparison between the LMoE-Lasso and the LMoE+ℓ 2 model shows that Lasso provides better results not only in terms of sparsity but also in terms of clustering.

Lasso paths for the regularized MoE parameters

In this part, we provide the Lasso paths for the experts network with Poisson and logistic outputs based on typical simulation data sets. To do this two fix values for the tuning parameters γ are chosen, and a grid of values for λ is also provided.

Lasso paths for the case of Poisson outputs

For the regularized MoE with Poisson regression, two fix values for γ are chosen 0 and 10; λ ∈ {0, 5, 10, . . . , 100}. The Figure 4.4 gives the Lasso paths for the experts network and also the changing of the gating network coefficients with different values of λ.

It turns out that the Lasso paths for each expert component perform quite consistent between different values of γ. The performance of the gating network coefficients with difference values of λ is quite similar between γ = 0 and γ = 10. The difference lies in the fact that with γ = 10 some parameters which are close to 0 are set to be 0. However, the changing of λ also effect to the sparsity of the gating network similar with the Gaussian model in Chapter 3. This behavior can be found in the case γ = 10, since λ ≥ 65 the third coefficient of the gating network is difference from it true value 0. 4.5: Average of the accuracy of clustering (correct classification rate and Adjusted Rand Index) results for the Poisson outputs with the non-penalized model (PMoE) and the proposed Lasso regularized model (PMoE-Lasso). The results are also considered for the logistic case with ℓ 2 penalization (LMoE+ℓ 2 ) and Lasso regularization (LMoE-Lasso).

Lasso paths for the case of logistic outputs

Finally, for the regularized MoE with logistic regression, we choose two non-zero values for γ, γ ∈ {1, 3} and a grid of 20 different values for λ, λ ∈ {0.5, 1.0, 1.5, . . . , 10}. Figure 4.5 gives the Lasso paths for the experts network and also the changing of the gating network coefficients with different values of λ.

Here, the Lasso paths for each expert part performs are almost identical with different values of γ. However, as λ decreases to zero, some coefficients of the experts network can grow to infinity. We therefore add two small quadratic penalizations for the regularized model with where ρ = 0.1 log(n) and consider addition elastic net paths in this case. This regularized model is denoted by LMoE-ElasticNet. Hence, the update of the gating network is done by using a similar proximal Newton-type procedure in Chapter 3. For the experts network, the equation It is interesting to see that the elastic net paths in this situation have similar shapes with the Lasso, except the ℓ 2 penalized solution is obtained instead of infinity as λ approaches zero. Thus, the instability of the solution can avoid. For λ > 1, the effect of the quadratic penalty with a small ρ is not appreciable. In application, depend on the data such as the correlations between features and the value of tuning parameters λ, γ, one can consider adding two small amount of quadratic penalizations or not. Otherwise, the model works quite well with just ℓ 1 penalizations.

Overall, the performance of Lasso paths for the experts network in both models seem to be consistence with different values of γ. Some open questions should be study more such as the effect of λ to the sparsity of the gating network.

Applications to real data sets

In this part, three real data sets are analyzed as a further test of the proposal methodology. Two data sets are for the logistic model and one for Poisson model. The obtain results are compared with other methods, which provided by [START_REF] Peralta | Embedded local feature selection within mixture of experts[END_REF]. The comparison are based upon three different criteria: the average mean squared error (MSE) between observation values and the predicted values of the response variable, the sparsity of each result, and the correlation of these values. After the parameters are estimated and the data are clustered, the following value under the estimated model Ŷ = mode p k (y|x; ẑ = k) = mode p k (y|x; θk ), is used as a predicted value for Y .

Regularized MoE model with Poisson outputs

A data set is used here to illustrate for the proposed regularized MoE of Poisson regression experts. The study used Cleveland Clinic Foundation heart disease data set that available at the website UC Irvine Machine Learning Repository. This data set includes 13 features and 297 In term of prediction, 65% of observations have the same values between their predictions and their response values. It is worth to consider the regularized MoE for ZIP model as an extended approach for this type of data. The changing of the penalized log-likelihood after each iteration is given in Figure 4.7.

Histogram of the variable Y and its predicted value for this data set is given in Figure 4.8.

Regularized MoE model with Multinomial outputs

For the logistic case, we consider the two data sets that were used by [START_REF] Peralta | Embedded local feature selection within mixture of experts[END_REF] in their work and compare the results between our approach with their method. We investigate the Ionosphere data and Musk-1 data which are described on the website UC Irvine Machine Learning Repository. The Ionosphere data contains 351 observations and 33 features. The Musk-1 data has 486 observations and 168 features. The variables with zero variance are removed. Hence, the Musk-1 data set remains with 167 features. Both data sets have two classes. All features are standardized to have mean zero and unit variance. K = 2 is taken as in [START_REF] Peralta | Embedded local feature selection within mixture of experts[END_REF]. The parameter estimates of the MoE models obtained by Lasso are given in Table 4.7 and Table 4.8, 4.9. The classification accuracy and percentage of features reduction results between the proposal with Peralta's work are found in Table 4.10. These results suggest that the proposed algorithm with Lasso provide better results than the remain method in term of data classification and features reduction. For Ionosphere dataset, Peralta used on average 78.1% of all dimensions while our approach just need 26.3%. For the Musk-1 dataset, the proposed Lasso method also increases the ratio of dimensionality reduction up to 10%. Consider the classification rate, on both data sets the proposal method increases this ratio up to 12% since comparing with Peralta's. One of the reasons for this improvement is that the approach of Peralta does not guarantee the increase of the penalized log-likelihood values after each loop of their EM algorithm. The changing of the penalized log-likelihood functions on these data set can be referred from Figure 4.9 and Figure 4.10.

Dataset name

Classification accuracy Dimensionality reduction Lasso (Peralta) LMoE-Lasso Lasso (Peralta) -0.75204 --0.28020

x 10 - - - x 11 - - - x 12 - - - x 13 - - - x 14 - - - x 15 - -0.15926 - x 16 - - - x 17 - -0.29576 - x 18 - - - x 19 - - - x 20 - - - x 21 0.41903 - - x 22 - - - x 23 -1.36138 1.48880 -1.83610 x 24 -0.41763 - - x 25 - - - x 26 - 0.20319 - x 27 - - - x 28 - - - x 29 - -0.02892 - x 30 - - - x 31 - - - x 32 - - - x 33
0.99009 -0.21365 -

- - - x1 - - - x43 - -0.32513 - x2 - - - x44 - - - x3 - - - x45 - - - x4 - - - x46 - - - x5 - - - x47 0.10696 0.13833 - x6 - - -1.15153 x48 -0.70925 - - x7 - - - x49 - 0.05006 - x8 - - -0.73044 x50 -0.10448 -0.20221 - x9 - - - x51 - - - x10 - - - x52 - - - x11 - - - x53 - - - x12 - - - x54 - - - x13 - - - x55 -0.10431 - - x14 - 0.35940 - x56 -0.53456 - - x15 - - - x57 - - - x16 - - - x58 - - - x17 - - - x59 -0.07893 - - x18 - - - x60 - - - x19 - - - x61 0.00010 - - x20 - - - x62 - - - x21 - - - x63 - - - x22 - - - x64 - - - x23 - - - x65 - - - x24 -0.31879 - - x66 - - - x25 - - - x67 - - - x26 - - - x68 - - - x27 - - - x69 - - - x28 - - - x70 0.18476 - - x29 - - - x71 - - - x30 - - - x72 - - - x31 - 0.56436 - x73 - - - x32 - - - x74 - - - x33 - - - x75 - - - x34 - - - x76 0.08573 0.45813 - x35 - - - x77 - - - x36 0.22055 0.31051 - x78 - - - x37 - 0.41421 - x79 - - - x38 - - - x80 - - - x39 - - - x81 - - - x40 - - - x82 - - - x41 - - - x83 -0.88481 - -
- x85 - - - x127 - - - x86 - - - x128 - - - x87 - - - x129 -0.57213 - - x88 - 0.20919 - x130 - - - x89 - - - x131 - - - x90 - - - x132 0.02409 - - x91 - - - x133 - - - x92 0.25523 0.03731 - x134 - - - x93 - - - x135 - - - x94 - - - x136 0.34955 - - x95 - - - x137 - - - x96 - - - x138 - - - x97 - 0.36352 - x139 - - - x98 - - - x140 - - - x99 - - - x141 -0.18019 - - x100 - - - x142 - - - x101 - - - x143 - - - x102 0.20188 - - x144 - - - x103 - - - x145 - - - x104 - - - x146 - - - x105 - - - x147 0.20336 0.51844 - x106 - - -0.88963 x148 - - - x107 - - - x149 - - - x108 - - - x150 - - - x109 0.13949 - - x151 0.56270 - - x110 - - - x152 - - - x111 - - - x153 - - - x112 - - - x154 - - - x113 - - - x155 - - - x114 - - - x156 - - - x115 - - - x157 - 0.23666 - x116 -0.21509 -0.39766 - x158 - - - x117 - - - x159 - - - x118 - - - x160 - - - x119 - - - x161 - - - x120 - - - x162 0.33300 0.62605 - x121 - - - x163 - 0.14212 - x122 -0.28134 - - x164 0.28869 - - x123 - - - x165 - -0.66940 - x124 - - - x166 - - - x125 - - -

Discussion for the high-dimensional setting

To evaluate our methods in high-dimensional setting p > n, we take 100 random observations of the Musk-1 data set with all 166 features. A MoE model with logistic outputs is used to modeling this data. Parameter estimation results are provided in Table 4.11 and Table 4.12.

A computer with CPU Intel i7-5500U 2.40GHz and 8GB RAM needs few (∼ 17) minutes to estimate all the parameters for the model with λ = 1.5 and γ = 1.5. The classification accuracy and percentage of features reduction results are given in Table 4.13.

From these tables, it turns out that the proposed algorithms perform quite well in this setting. We still obtain acceptable results for the regularized MoE models and the EM algorithm using the proximal Newton and proximal Newton-type methods are good tool for the parameter estimation. However, comparing with the parameters of the full data set there are big differences. For the full data set, 255 observations are clustered into the Class 1 and 221 observations belong to the Class 2. Meanwhile, in the subset data 89 observations belongs to Class 1 and the remaining is from Class 2. For the affect of each feature to the experts network and gating network, one can easily find that some features such as x 47 , x 50 , x 76 , x 83 , x 92 , and x 147 still keep their roles in the first expert part for both data set. While just only the feature x 84 appears in the gating network on two parameters estimation sets. 4.13: Classification accuracy and percentage of features reduction results for the subset of Musk-1 data using the proposed method LMoE-Lasso.

Conclusion and future work

In this chapter, we proposed a regularized MLE for the MoE model which encourages sparsity, and developed EM-based algorithms to monotonically maximize this regularized objective towards at least a local maximum, while they do not require using approximations as in standard MoE regularization. The proposed algorithms are based on proximal Newton-type methods and univariate updates of the model parameters via coordinate ascent, which allows to tackle matrix inversion problems and obtain sparse solutions. The results on the simulated and the real data sets in terms of parameter estimation, the estimation of the actual support of the sparsity, and clustering accuracy, confirm the effectiveness of this proposal, at least for problems with moderate dimension. The model sparsity does not include significant bias in terms of parameter estimation nor in terms of recovering the actual clusters of the heterogeneous data. A proximal Newton-type approach is possible to obtain closed form solutions for an approximate of the M-step as an efficient method that is promoted to deal with high-dimensional data sets.

A future work may consist of investigating more model selection experiments and considering hierarchical MoE of generalized linear models. Our proposed in this chapter can extend for the hierarchical MoE models with Lasso-like regularizations. As it can be seen through out this chapter, the Lasso regularized MoE models perform quite efficient in high-dimensional setting when p > n. However, theoretical results should be studied more to explain the behavior of the penalized methods in this context. In a different aspect, directly applying the regularized models for huge-scale data is not a reasonable approach. One should need more dimension reduction first, and our proposals should be consider as the terminal method to modeling these heterogeneous data. Recently work of [START_REF] Celeux | Variable selection in model-based clustering and discriminant analysis with a regularization approach[END_REF] is an open suggestion for dimension reduction. A hybrid method can be an efficient way to deal with these kinds of data.

In a similar spirit, [START_REF] Morvan | Prediction of the nash through penalized mixture of logistic regression models[END_REF] proposed a penalized mixture of logistic regression models. Actually, this is a regularized version of the localized MoE models [START_REF] Xu | An alternative model for mixtures of experts[END_REF] with logistic regression components. They suggested to penalize the experts network with the Lasso penalizations for feature selection task and another Lasso penalty function for sparse representation of the inverse component covariance matrices. The difficulty lies in the fact that they must guarantee that the estimate matrices are symmetric positive definite. In addition, the explanatory variable X can lied in a subspace for each mixture component. Hence, a method to interpret the distribution of X should be taken in account.

This chapter has led to the following publication [START_REF] Huynh | Estimation and feature selection in mixtures of generalized linear experts models[END_REF].

- - - x1 - - - x43 - - - x2 - - - x44 - - - x3 - - - x45 - - - x4 - - - x46 - - - x5 - - - x47 0.08585 - - x6 - - - x48 - - - x7 - - - x49 - - - x8 - - - x50 -0.69865 - - x9 - - - x51 -0.46418 - - x10 - - - x52 - - - x11 - - - x53 - - - x12 - - - x54 - - - x13 - - - x55 - - - x14 - - - x56 - - - x15 - - - x57 - - - x16 - - - x58 0.00085 - - x17 - - 0.42661 x59 - - - x18 - - - x60 - - - x19 - - - x61 - - - x20 - - - x62 - - - x21 -0.63760 - - x63 - - - x22 - - - x64 - - - x23 - - - x65 - - - x24 - - - x66 - - - x25 - - - x67 0.22870 - - x26 - - - x68 - - - x27 - - - x69 - - - x28 - - - x70 - - - x29 - - - x71 - - - x30 - - - x72 - - - x31 - - - x73 - - - x32 - - - x74 - - - x33 - - - x75 - - - x34 -0.70692 - - x76 0.49465 - - x35 - - - x77 - - - x36 - - - x78 - - - x37 - - -1.49579 x79 - - - x38 - - - x80 0.28250 - - x39 - - - x81 - - - x40 - - - x82 - - - x41 - - - x83 -0.87708 - -
- - - x85 - - - x127 - - - x86 - - - x128 - - - x87 - - - x129 - - - x88 0.07382 - - x130 - - - x89 -0.11238 - - x131 - - - x90 - - - x132 - - - x91 0.25618 - - x133 - - - x92 0.32106 - - x134 - - - x93 - - - x135 - - - x94 -0.46848 - - x136 - - - x95 - - - x137 - - - x96 0.40657 - - x138 - - - x97 - - - x139 - - - x98 - - - x140 - - - x99 - - - x141 - - - x100 - - - x142 - - - x101 - - - x143 - - - x102 - - - x144 - - - x103 - - - x145 - -0.45413 -0.45352 x104 - - - x146 0.44840 - - x105 - - - x147 0.53837 - - x106 - - - x148 - - - x107 - - - x149 - - - x108 - - - x150 - - - x109 - - - x151 - - - x110 - - - x152 - - - x111 - - - x153 - - - x112 - - - x154 - - - x113 - - - x155 - - - x114 - - - x156 - - - x115 - - - x157 0.38850 - - x116 - - - x158 - - - x117 - - - x159 - - - x118 - - - x160 - - - x119 - - - x161 1.00947 - - x120 - - - x162 - - - x121 - - - x163 - - -0.16575 x122 - - - x164 - - - x123 - - - x165 - - - x124 - - - x166 - - - x125
1.03320 --Table 4.12: Fitted regularized MoE model (LMoE-Lasso) to the subset of Musk-1 data (part 2).

Chapter 5

Conclusions and future directions

Conclusions

This thesis has been written regarding the regularized MoE models for regression data with various types of outputs. The developed approaches are particularly based on the Lasso-like penalty which simultaneously performs parameter estimation and feature selection on the heterogeneous regression data.

In Chapter 3, we first presented the MoE based for modeling heterogeneous regression data with continuous outputs. More specifically, we used MoE with Gaussian regressor experts for these data. Then, a regularized model, which contains two Lasso-like penalty functions, was proposed. These penalty parts concomitantly select the feature from the gating network as well as the experts network. For parameter estimation , three hybrid EM algorithms were introduced. Different from the state-of-art, our proposed methods avoid matrix inversion and also do not require any approximation on the penalty functions. Experiments on simulated data and real data showed the effectiveness of our approach. Furthermore, we believe that the proposed algorithm based on EM and proximal Newton-type procedure is a potential approach to deal with a large-scale data set. This method also can handle with the situation in which p > n, as discussed in Section 3.4.6.

In Chapter 4, we extended this regularized MoE modeling approach to model regression data with discrete outputs including counting data and classification data. The regularized models are obtained by adding two Lasso penalty functions: one the logistic mixing functions and the other for the regression components. By doing so, one can attain a sparse MoE model which can represent the data better than a classical model. Regularized MLE is done via hybrid EM algorithms with proximal Newton-type method. The algorithms inherit all the best characteristics of the previous one in Chapter 3, i.e, guarantee the monotonicity of the penalized log-likelihood after each iteration and also avoid computing the inverse matrices, and encourage sparsity. Experiments conducted on simulated data and real data demonstrated the relevance of the proposed approaches in terms of feature selection, clustering and prediction as compared to other alternatives. CHAPTER 5. CONCLUSIONS AND FUTURE DIRECTIONS Finally, the R packages for the proposed algorithms are constructed and can be downloaded via the link in List of Publications and Communications.

Future directions

Future work may consist of developing an extension of the regularized model for hierarchical MoE of generalized linear models [START_REF] Jordan | Hierarchical mixtures of experts and the EM algorithm[END_REF]Jiang and Tanner, 1999a). Here, the architecture is a tree in which the gating networks sit at the nonterminals of the tree and the expert networks sit at the leaves of the tree. Each expert produces an output vector for each input vector. These output vectors proceed up the tree, being blended by the gating network outputs. Hence, a regularized approach should be studied for this framework in preventing overfitting and to have a sparse representation model due to the large size of the input vector. Our proposals in this thesis can be seen as a potential approach for this problem.

In addition, penalized MoE models with multivariate response variables are also an interesting topic and has good theoretical properties (Nguyen et al., 2019a). To the best of our known, there is no research focus on these cases from the regularized estimation perspective.

For the localized MoE models [START_REF] Xu | An alternative model for mixtures of experts[END_REF], it is interesting to study more these models namely with multiple outputs. On the one hand, we should provide a sparse interpret in the experts networks and on the other hand, it also needed to have a sparse structure of the covariance matrices. A suggestion can be given by combining our methods with the work of [START_REF] Fop | Model-based clustering with sparse covariance matrices[END_REF].

Finally, as it has shown in experimental study, our methods work quite well with large-scale data, even for the case p > n. However, theoretical results to support our methods need to study more. Moreover, directly applying our methods for huge-scale data seems to be not a correct way. In such cases, the data should be reduce its dimension using other methods such as: SRUW, rough sets, etc, and then our approaches can be considered as a terminal step of the process.

A.3. COVARIANCE STRUCTURE OF GMMS IN HIGH-DIMENSIONAL SETTING

We now substitute (A.2) into (A.1) to give ik , G = Σ k and D = A k (for each k = 1, . . . , K), and using the fact that (µ

n i=1 K k=1 τ [q] ik log N (x i ; µ k , Σ k ) = - np 2 log(2π) + 1 2 K k=1 (- n i=1 τ [q] ik ) log |Σ k | -tr(Σ -1 k A k ) - 1 2 K k=1 ( n i=1 τ [q] ik )(µ [q+1] k -µ k ) T Σ -1 k (µ
[q+1] k - µ k ) T Σ -1 k (µ [q+1] k
-µ k ) ≥ 0 and is 0 if and only if µ k = µ A.3.3 Proof of Lemma 2.4.1

U C C C Σ k = Λ k Λ T k + ωI p K[pd -d(d -1)/2)] + 1 U C U C Σ k = Λ k Λ T k + ω k I p K[pd -d(d -1)/2)] + K C C C U Σ k = ΛΛ T + ω∆ [pd -d(d -1)/2)] + p C C U U Σ k = ΛΛ T + ω k ∆ [pd -d(d -1)/2)] + [K + (p -1)] U C C U Σ k = Λ k Λ T k + ω∆ K[pd -d(d -1)/2)] + p U C U U Σ k = Λ k Λ T k + ω∆ k K[pd -d(d -1)/2)] + [1 + K(p -1)] C U C U Σ k = ΛΛ T + ω∆ k [pd -d(d -1)/2)] + [1 + K(p -1)] C U U U Σ k = ΛΛ T + ω k ∆ k [pd -d(d -1)/2)] + Kp U U C U Σ k = Λ k Λ T k + ω∆ k K[pd -d(d -1)/2)] + [1 + K(p -1)] U U U U Σ k = Λ k Λ T k + ω k ∆ k K[pd -d(d -1)/2)] + Kp Table A.
Denote x 1 , . . . , x d are d eigenvectors associated with λ 1 , . . . , λ d such that x T i x i = 1 for all i ∈ {1, . . . , d}. Let {y 1 , . . . , y p-d } be pd linearly independent eigenvectors corresponding to the eigenvalue γ. Note that, for i = j then x T i x j = 0, and x T i y j = 0 for all i ∈ {1, . . . , d}, j ∈ {1, . . . , p -d}.

We now construct pd eigenvectors {ỹ 1 , . . . , ỹp-d } associated with γ from the system {y 1 , . . . , y p-d } which satisfy ỹT i ỹj = 0 (i = j). This can be done by using the Gram-Schmidt orthonormalization procedure. Finally, the updates of the experts' parameters {β} and {σ 2 } are performed by separate maximizations of Q(β, σ 2 ; θ [q] ). This function is concave and has the quadratic form. Hence, the coordinate ascent algorithm with soft-thresholding operator is successful to provide the updates β [q+1] = arg max β Q(β, σ [q] ; θ [q] ), (B.14) and σ [q+1] = arg max σ Q(β [q+1] , σ; θ [q] ) (B.15) and thus we have Q(β [q+1] ; θ [q] ) ≥ Q(β; θ [q] ) ≥ Q(β [q] ; θ [q] ), (B.16) and Q(σ [q+1] ; θ [q] ) ≥ Q(σ; θ [q] ) ≥ Q(σ [q] ; θ [q] ). (B.17 Choose H k , a positive definite approximation to the Hessian.

4:

Solve the subproblem for a search direction:

△x k ← arg min d ▽g(x k ) T d + 1 2 d T H k d + h(x k + d).
5:

Select t k with a backtracking line search.

6:

Update: x k+1 ← x k + t k △x k . 7: until a stopping condition is satisfied. In this part, the quadratic approximation for the function Q k ({β k0 , β k }; θ [q] ) of the Poisson model in (4.12) is constructed using Taylor expansion. This function is given by Q k ({β k0 , β k }; θ [q] ) = P k ({β k0 , β k }; θ [q] )λ k β k 1 , (C.8) List of Tables machines à noyaux [START_REF] Scholkopf | Learning with kernels: support vector machines, regularization, optimization, and beyond[END_REF], et aussi les méthode d'apprentissage statistique de modèles génératifs [START_REF] Friedman | The elements of statistical learning[END_REF], en particulier les modèles à variables latentes.

L'analyse statistique des données qui suppose que celles-ci sont des réalisations de variables aléatoires dont il s'agira de déterminer les distributions les plus adaptées aux données observées, est tient ainsi compte de la partie aléatoire des données et mesure l'incertitude, fournit un cadre agréable pour l'estimation de densité, la prédiction et l'apprentissage non supervisé, y compris en clustering. Parmi les modèles statistiques on distingue les modèles de mélange de densités [START_REF] Titterington | Statistical analysis of finite mixture distributions[END_REF]McLachlan and Peel., 2000) qui représentent le moyen commode pour l'analyse de données hétérogènes en statistique et en apprentissage machine, dans différents domaines d'application allant de la bioinformatique, à l'économie. Grâce à leur flexibilité, les modèles de mélange peuvent être utilisés pour l'estimation de densités et la classification de données provenant de populations hétérogènes complexes. Ils sont également utilisés pour effectuer des analyses de prédiction, y compris des analyses de régression, par le biais des mélanges d'experts [START_REF] Jacobs | Adaptive mixtures of local experts[END_REF]; [START_REF] Jordan | Hierarchical mixtures of experts and the EM algorithm[END_REF], l'une de leur extensions les plus flexibles. La question finale d'analyse sur la base de tels modèles, passe d'abord par l'ajustement de ces modèles aux données observées, c'est la question de l'estimation.

L'estimation par maximum de vraisemblance (EMV) [START_REF] Fisher | On an absolute criterion for fitting frequency curves[END_REF] avec les algorithmes EM [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF][START_REF] Mclachlan | The EM algorithm and extensions[END_REF] est le moyen courant d'estimer les paramètres des modèles de mélange et des modèles de mélange d'experts. Toutefois, l'application directe de ces méthodes à des ensembles de données de grande dimension présente certains inconvénients. Par exemple, il devient difficile d'estimer la matrice de covariance dans les modèles de mélange gaussiens avec un grand nom-telles que: prédiction de la demande quotidienne d'électricité [START_REF] Weigend | Nonlinear gated experts for time series: Discovering regimes and avoiding overfitting[END_REF], généralisation des modèles autorégressifs pour les séries temporelles [START_REF] Zeevi | Time series prediction using mixtures of experts[END_REF][START_REF] Wong | On a logistic mixture autoregressive model[END_REF], reconnaissance de chiffres manuscrits [START_REF] Hinton | Recognizing handwritten digits using mixtures of linear models[END_REF], segmentation et classification de séries temporelles avec changements de régime [START_REF] Chamroukhi | Time series modeling by a regression approach based on a latent process[END_REF][START_REF] Samé | Model-based clustering and segmentation of time series with changes in regime[END_REF][START_REF] Chamroukhi | Model-based clustering and classification of functional data[END_REF], reconnaissance d'activités humaines [START_REF] Chamroukhi | Joint segmentation of multivariate time series with hidden process regression for human activity recognition[END_REF] with Expectation-Maximization (EM) algorithms, and to simultaneously perform feature selection so that sparse models are encouraged in such a high-dimensional setting. We first introduce a mixtureof-experts' parameter estimation and variable selection methodology, based on ℓ 1 (lasso) regularizations and the EM framework, for regression and clustering suited to high-dimensional contexts. Then, we extend the method to regularized mixture of experts models for discrete data, including classification.

We develop efficient algorithms to maximize the proposed ℓ 1 -penalized observed-data log-likelihood function. Our proposed strategies enjoy the efficient monotone maximization of the optimized criterion, and unlike previous approaches, they do not rely on approximations on the penalty functions, avoid matrix inversion, and exploit the efficiency of the coordinate ascent algorithm, particularly within the proximal Newton-based approach.

Keywords: Mixture models; Mixture of Experts; Regularized Estimation; Feature Selection, Lasso, ℓ 1 -regularization; Sparsity; EM algorithm; MM Algorithm; Proximal-Newton; Coordinate Ascent; Clustering, Classification; Regression, Prediction.

  context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Contributions of the thesis . . . . . . . . . . . . . . . . . . . . . . . . . 3

Figure 2 . 1 :

 21 Figure 2.1: The results on testing data.

2 Figure 2 . 2 :

 222 Figure 2.2: Illustration of the MLR on two-component MLR model. The black and red lines are the fitted conditional expectations of the density components 1 and 2, respectively.

2. 3 .

 3 MIXTURE MODELS FOR REGRESSION DATA has the one-parameter exponential form π(h, y) = exp a(h)y + b(h) + c(y) , for y ∈ Y, (2.52) such that Y π(h, y)dλ(y) = 1 for each h ∈ R. a(.) and b(.) are analytic and have nonzero derivatives on R and c(.

2. 4 .

 4 CLUSTERING AND CLASSIFICATION IN HIGH-DIMENSIONAL SETTING and

Figure 2 . 3 :

 23 Figure 2.3: Estimation picture for the Lasso (left) and ridge regression (right) with constraint regions and the contours of the least squares error function. Picture from Friedman et al. (2001)

  function f (β) is presented as follows

Figure 2 . 4 :

 24 Figure 2.4: The changing of the objective function after each iteration.

  feature selection in mixture of experts for regression and clustering Contents 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Mixture of experts model for continuous data . . . . . . . . . . . . . 3.2.1 The model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Maximum likelihood parameter estimation . . . . . . . . . . . . . . . . . 3.3 Regularized maximum likelihood parameter estimation of the MoE 3.3.1 Parameter estimation and feature selection with a dedicated block-wise EM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 E-step . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.3 M-step . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.4 Algorithm tuning and model selection . . . . . . . . . . . . . . . . . . . 3.4 Experimental study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Evaluation criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Simulation study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.3 Lasso paths for the regularized MoE parameters . . . . . . . . . . . . . 3.4.4 Evaluation of the model selection via BIC . . . . . . . . . . . . . . . . . 3.4.5 Applications to real data sets . . . . . . . . . . . . . . . . . . . . . . . . 3.4.6 CPU times and discussion for the high-dimensional setting . . . . . . . 3.5 Conclusion and future work . . . . . . . . . . . . . . . . . . . . . . . .

  Definition 3.3.1. (see Lange (2013)) Let F (x) be a function of x. A function G(x|x m ) is called a minorizing function of F (x) at x m if and only if

k

  and x i0 = 1. 61 3.3. REGULARIZED MAXIMUM LIKELIHOOD PARAMETER ESTIMATION OF THE MOE Proof. Using Lemma 3.3.1, I 1i (w) =log 1 + K-1 k=1 e w k0 +x T i w k can be minorized by

Figure 3

 3 Figure 3.1: Histogram of the response variable Y for a MoE simulation data. The red line represents the estimated density using non-parametric kernel density estimation and the black line shows the estimated density using a GMM.

Figure 3 . 2 :

 32 Figure3.2: Boxplots of the expert 1's parameter estimates obtained by, respectively, Mixtureof-Experts (MoE) with standard MLE, MoE with ℓ 2 regularization, MoE with BIC procedure for model selection, and then our proposed methods: MoE with mixed ℓ 1 (Lasso) and ℓ 2 regularization using the EM algorithm with Minorization-Maximization (MM) updates, Coordinate Ascent (CA) updates, and Proximal Newton (PN) updates; MoE with Lasso regularization with hybrid EM/Proximal Newton method for parameter estimation. Finally, the standard MIXLASSO (Mixture of regressions with Lasso regularization) is also considered.

  Figure 3.4: Boxplots of the gate's parameter estimates obtained by Mixture-of-Experts (MoE) with standard MLE, MoE with ℓ 2 regularization, and with BIC procedure for model selection compare with boxplots of the proposed methods: MoE with mixed ℓ 1 (Lasso) and ℓ 2 regularization with Minorization-Maximization (MM) updates, Coordinate Ascent (CA) updates, and Proximal Newton (PN) updates; MoE with Lasso regularization bases on Proximal Newton updates.

  Figure 3.5: Lasso paths for the experts network and the changing of the gating network.

  Figure 3.6: Histograms of Y = MEDV/sd(MEDV) and its predicted value for housing data.The red lines represent the estimated densities using kernel density estimation and the black lines show the estimated densities using a GMM.

  .20) and E( βk ) is a function of βk . The corresponding Lasso form is described as following Ĩr (β k )λ kr β kr 1 .(4.21)Using a similar algorithm with Algorithm 2 by replacing the weighted Lasso in (3.29) with (4.21), one can obtain the kth expert's parameter vector.

Figure 4 . 2 :

 42 Figure 4.2: Boxplots of non-penalized MoE model (PMoE) and the proposed Lasso penalized MoE model (PMoE-Lasso) for Poisson case.

Figure 4 . 4 :

 44 Figure 4.4: Lasso paths for the experts network and the changing of the gating network for regularized MoE model (PMoE-Lasso) with Poisson outputs.

Figure 4 . 5 :

 45 Figure 4.5: Lasso paths for the experts network and the changing of the gating network for regularized MoE model (LMoE-Lasso) with logistic outputs.

( 4 .

 4 21) is replaced by Ĩr (β k )λ kr β kr 1coefficients. The elastic net paths are given in Figure4.6 for γ ∈ {0, 3}, λ ∈ {0, 0.5, . . . , 10} and ρ = 0.1 log(n).

Figure 4 . 6 :

 46 Figure 4.6: Elastic net paths for the experts network and the changing of the gating network for regularized MoE model (LMoE-ElasticNet) with logistic outputs.

Figure 4 . 7 :Figure 4 . 8 :

 4748 Figure 4.7: The penalized log-likelihood during the EM iterations when fitting the PMoE-Lasso model to the Cleveland Clinic Foundation heart disease data.

Figure 4 . 9 :

 49 Figure 4.9: The penalized log-likelihood during the EM iterations when fitting the LMoE-Lasso model to the Ionosphere data.

Figure 4 .

 4 Figure 4.10: The penalized log-likelihood during the EM iterations when fitting the LMoE-Lasso model to the Musk-1 data.

  use the following lemma (with a slightly change) (see Lemma 3.2.2 in[START_REF] Anderson | An introduction to multivariate statistical analysis[END_REF]).Lemma A.2.1. Let D be a symmetric positive definite matrix of order p, and α a positive real number. Then the maximum off (G) = -α log |G|tr(G -1 D) (A.4) with respect to symmetric positive definite matrices G exists, occurs at G = (1/α)D, and has the value f [(1/α)D] = pα log αα log |D|pα.To get the updated estimates of µ k and Σ k (k = 1, . . . , K), we apply Lemma A.2

  . . . , K). Then the maximum of the function (A.3) (with respect to µ k and Σ k (k = 1, . . . , K)) occurs at µ k = µ structure of GMMs in high-dimensional setting A.3.1 Covariance structure of the expanded parsimonious Gaussian models Model Name Number of parametersK = 3, p = 50 λ k D k A k D T k VVV (K -1) + Kp + Kp(p + 1)/2 3977 λD k A k D T k EVV (K -1) + Kp + Kp(p + 1)/2 -(K -1) 3975 λ k D k AD T k VEV (K -1) + Kp + Kp(p + 1)/2 -(K -1)(p -1) 3879 λD k AD T k EEV (K -1) + Kp + Kp(p + 1)/2 -(K -1)p 3877 λ k DA k D T VVE (K -1) + Kp + p(p + 1)/2 + (K -1)p 1527 λDA k D T EVE (K -1) + Kp + p(p + 1)/2 + (K -1)(p -1) 1525 λ k DAD T VEE (K -1) + Kp + p(p + 1)/2 + (

A. 3 . 2

 32 Covariance structure of the parsimonious Gaussian mixture modelsNumber of Λ k = Λ ∆ k = ∆ ω k = ω ∆ k = I p Covariance structure Covariance Parameters C C C C Σ k = ΛΛ T + ωI p [pdd(d -1)/2)] + 1 C C U C Σ k = ΛΛ T + ω k I p [pdd(d -1)/2)] + K

2 :

 2 The covariance structure, number of covariance parameters for each member of the EPGMM family McNicholas and Murphy (2010) (C = constrained, U = unconstrained).

  , ỹ1 = y 2 , ỹ1 -y 2 , ỹ1 ỹ1 , ỹ1 ỹ1 , ỹ1 = y 2 , ỹ1y 2 , ỹ1 = 0.

  ) Equations (B.11), (B.13), (B.16), and (B.17) and the increasing property of the proximal Newton-type methods show that (B.2) holds; hence, the penalised log-likelihood in monotonically increased by the proposed algorithms. C.3. QUADRATIC APPROXIMATION FOR THE EXPERTS NETWORK Algorithm 4 A generic proximal Newton-type procedure 1: Starting point x 0 ∈ domf .

xd

  By taking the first and second derivatives of I(w) with respect to (w k0 , w k )ikπ k (x i ; w))x ij , ij x ih π k (x i ; w)(1π k (x i ; w)), (C.4)for j, h ∈ {0, 1, . . . , p} with x i0 = 1, then the partial quadratic approximation to I(w) with respect to (w k0 , w k ) at ( wk0 , wk ) is given byl I k (w k0 , w k ) =ik (c ikw k0x T i w k ) 2 + C( ikπ k ( w; x i ) π k ( w; x i )(1π k ( w; x i )) , (C.6) d ik = π k ( w; x i )(1π k ( w; x i )), (C.7) C( w)is a function of w. C.3 Quadratic approximation for the experts network C.3.1 Quadratic approximation for the Poisson outputs

2. 1

 1 Characteristics of some common univariate GLMs. . . . . . . . . . . . . . . . . . 2.2 Parameter vectors of some common univariate GLMs. . . . . . . . . . . . . . . . 2.3 Number of free parameters to estimate for constrained GMMs. . . . . . . . . . . 3.1 Sensitivity (S 1 ) and specificity (S 2 ) results for our proposed methods compared with the MoE model using BIC criterion for feature selection and the mixture of linear regression model (MLR). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 The true value, mean and standard derivation for each coefficient of the estimated parameter vector via MoE, MoE+ℓ 2 , MoE-BIC and the MIXLASSO. . . . . . . 3.3 The true value, mean and standard derivation for each coefficient of the estimated parameter vector via the regularized models: MoE-Lasso+ℓ 2 (MM), MoE-Lasso+ℓ 2 (CA), MoE-Lasso+ℓ 2 (PN) and MoE-Lasso (PN). . . . . . . . . . . . . 3.4 The true value and mean squared error for each coefficient of the estimated parameter vector via MoE, MoE+ℓ 2 , MoE-BIC and the MIXLASSO. . . . . . . . . 3.5 The true value and mean squared error for each coefficient of the estimated parameter vector of the regularized models: MoE-Lasso+ℓ 2 (MM), MoE-Lasso+ℓ 2 (CA), MoE-Lasso+ℓ 2 (PN) and MoE-Lasso (PN). . . . . . . . . . . . . . . . . . 3.6 Average of the accuracy of clustering (correct classification rate and Adjusted Rand Index) comparison among the non-penalized MoE model, MoE with ℓ 2 regularization, MoE with the BIC criterion for model selection and the proposed regularized models. Finally, the results for the standard MIXLASSO (Mixture of regressions with Lasso regularization) is also given. . . . . . . . . . . . . . . . . . 3.7 Sparsity and average of the accuracy of clustering criteria on different tuning parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 Fitted models for housing data with MLE and MoE-Lasso+ℓ 2 (Khalili, 2010). . . 3.9 Fitted models for housing data with our proposed algorithms: MoE-Lasso+ℓ 2 (CA), MoE-Lasso+ℓ 2 (PN) and MoE-Lasso (PN). . . . . . . . . . . . . . . . . . . 3.10 Results for Housing data set with a two-component MoE model. . . . . . . . . . 163 LIST OF TABLES 3.11 Fitted models for housing data with MoE-Lasso (PN) method (K = 3). . . . . . 86 3.12 Results for Housing data set obtained by our MoE-Lasso (PN) (K = 3). . . . . . 86 3.13 Fitted models for baseball salary data with the standard MoE using MLE and MIXLASSO (Khalili and Chen, 2007). . . . . . . . . . . . . . . . . . . . . . . . 88 3.14 Fitted models for baseball salary data with our proposed algorithms: MoE-Lasso + ℓ 2 (CA) and MoE-Lasso + ℓ 2 (PN). . . . . . . . . . . . . . . . . . . . . . . . . 89 3.15 Results for Baseball salaries data set. . . . . . . . . . . . . . . . . . . . . . . . . . 90 3.16 Results for CPU times. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 3.20 Results for residential building data set by using the clustering method 1. . . . . 90 3.21 Results for the residential building data set by using the clustering method 2. . . 90 3.17 Fitted model parameters for residential building data obtained by our MoE-Lasso +ℓ 2 (PN) algorithm (part 1). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 3.22 Results for the subset of the residential building data set by using the clustering method 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 3.18 Fitted model parameters for residential building data obtained by our MoE-Lasso +ℓ 2 (PN) algorithm (part 2). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 3.23 Results for the subset of the residential building data set by using the clustering method 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 3.19 Fitted model parameters for residential building data obtained by our MoE-Lasso +ℓ 2 (PN) algorithm (part 3). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 3.24 Fitted model parameters for the subset of residential building data obtained by our MoE-Lasso +ℓ 2 (PN) algorithm (part 1). . . . . . . . . . . . . . . . . . . . . 95 3.25 Fitted model parameters for the subset of residential building data obtained by our MoE-Lasso +ℓ 2 (PN) algorithm (part 2). . . . . . . . . . . . . . . . . . . . . 96 3.26 Fitted model parameters for the subset of the residential building data obtained by our MoE-Lasso +ℓ 2 (PN) algorithm (part 3). . . . . . . . . . . . . . . . . . . 97 4.1 Sensitivity (S 1 ) and specificity (S 2 ) results of the Lasso regularized MoE models for Poisson outputs (PMoE-Lasso) and logistic ouputs (LMoE-Lasso). . . . . . . 110 4.2 Estimated parameters for a typical data set obtained with Lasso regularized MoE (LMoE-Lasso) and MLE for MoE with logistic observations. . . . . . . . . . . . 110 4.3 Estimated parameter vector obtained by PMoE and our PMoE-Lasso for Poisson outputs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 4.4 Estimated parameter vector obtained by LMoE+ℓ 2 and the proposed LMoE-Lasso models for logistic outputs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 tion de densité, prédiction et classification de telles données complexes car hétérogènes et de grande dimension. Nous proposons de nouvelles stratégies basées sur l'estimation par maximum de vraisemblance régularisé des modèles pour pallier aux limites des méthodes standards, y compris l'EMV avec les algorithmes d'espérance-maximisation (EM), et pour effectuer simultanément la sélection des variables pertinentes afin d'encourager des solutions parcimonieuses dans un contexte à grande dimension. Nous introduisons d'abord une méthode d'estimation régularisée des paramètres et de sélection de variables d'un mélange d'experts, basée sur des régularisations ℓ 1 (lasso) et le cadre de l'algorithme EM, pour la régression et la classification adaptés aux contextes de la grande dimension. Ensuite, nous étendons la stratégie à un mélange régularisé de modèles d'experts pour les données discrètes, y compris pour la classification. Nous développons des algorithmes efficaces pour maximiser la fonction de log-vraisemblance ℓ 1 -pénalisée des données observées. Nos stratégies proposées jouissent de la maximisation monotone efficace du critère optimisé, et contrairement aux approches précédentes, ne s'appuient pas sur des approximations des fonctions de pénalité, évitent l'inversion de matrices et exploitent l'efficacité de l'algorithme de montée de coordonnées, particulièrement dans l'approche proximale par montée de coordonnées. Mots-clés Modèles de mélange; Mélange d'experts; Estimation régularisée; Sélection de variables, Lasso, Régularisation; Parcimonie; Algorithme EM; Algorithme MM; Proximal-Newton; Montée de coordonnées; Clustering, classification; Régression; Prédiction.

  , etc. Malheureusement, la modélisation à l'aide des modèles de mélanges d'experts dans le cas des prédicteurs de grande dimension reste encore limitée. Nos objectifs ici sont donc d'étudier l'estimation et la sélection des variables dans des modèles de mélanges d'experts de grande dimension, et de: i) proposer de nouvelles stratégies d'estimation et de sélection de variables pour surmonter ces limites. Pour y parvenir, nous souhaitons introduire une nouvelle estimation régularisée et une sélection de variables dans le cadre de mélanges d'experts pour les modèles linéaires généralisés, basées sur des pénalités de type Lasso. Nous les étudions pour différentes familles d'experts, y compris le mélanges d'experts avec des distributions d'experts gaussiennes, de Poisson et logistiques. ii) ] développer des algorithmes efficaces pour estimer les paramètres de ces modèles régularisés. Les algorithmes développés effectuent simultanément l'estimation et la sélection de variables et devraient avoir la capacité d'encourager la parcimonie, et traiter certaines questions typiques du problème de grande dimension, en évitant l'inversion de matrice, de sorte qu'ils fonctionnent assez bien dans des situations de grande dimension. Dans la section suivante, nous résumons les contributions de la thèse. et décrivons les trois stratégies principales pour pallier au problème de la malédiction de la dimension, c'est-à-dire l'approche de réduction de dimension, celle de décomposition spectrale des matrices de covariance du modèle, et celle de sélection des variables par régularisation Lasso. Nous optons pour cette dernière stratégie de régularisation et la présentons dans le cas du mélange de modèles de régressions et du mélange d'experts. Nous passons en revue l'estimation et la sélection de variables par maximum de vraisemblance régularisé pour ces modèles au moyen d'algorithmes EM adaptés. Contribution du Chapitre 3 Puis, dans le chapitre 3, nous introduisons une nouvelle approche pour l'estimation et la sélection de variables de mélanges d'experts pour la régression avec des prédicteurs potentiellement de grande dimension, et une population hétérogène. L'approche effectue simultanément l'estimation des paramètres, la sélection des variables, le clustering et la régression sur des données de régression hétérogènes. Il s'agit d'une approche de maximum de vraisemblance régularisé avec une régularisation dédiée qui, d'une part, encourage la parcimonie grâce à une partie de régularisation de type Lasso, et d'autre part, un calcul facilité grâce à la convexité de la pénalité ℓ 1 . Nous proposons un cadre hybride efficace d'espérance-maximisation (EM) pour résoudre efficacement le problème d'optimisation qui en résulte, et qui maximise de façon monotone la log-vraisemblance régularisée. Il en résulte trois algorithmes hybrides pour maximiser la fonction objective proposée, i.e un algorithme de Majorisation-Maximisation (MM), une montée de coordonnées, et une procédure de type Newton proximal. Nous montrons que l'approche proposée ne nécessite pas d'approximation du terme de régularisation, et les trois algorithmes hybrides développés permettent de Estimation et sélection de variables dans les modèles de mélange d'experts de grande dimension Résumé: Cette thèse traite de la modélisation et de l'estimation de modèles de mélanges d'experts de grande dimension, en vue d'efficaces estimation de densité, prédiction et classification de telles données complexes car hétérogènes et de grande dimension. Nous proposons de nouvelles stratégies basées sur l'estimation par maximum de vraisemblance régularisé des modèles pour pallier aux limites des méthodes standards, y compris l'EMV avec les algorithmes d'espérance-maximisation (EM), et pour effectuer simultanément la sélection des variables pertinentes afin d'encourager des solutions parcimonieuses dans un contexte haute dimension. Nous introduisons d'abord une méthode d'estimation régularisée des paramètres et de sélection de variables d'un mélange d'experts, basée sur des régularisations ℓ 1 (lasso) et le cadre de l'algorithme EM, pour la régression et la classification adaptés aux contextes de la grande dimension. Ensuite, nous étendons la stratégie un mélange régularisé de modèles d'experts pour les données discrètes, y compris pour la classification. Nous développons des algorithmes efficaces pour maximiser la fonction de log-vraisemblance ℓ 1 -pénalisée des données observées. Nos stratégies proposées jouissent de la maximisation monotone efficace du critère optimisé, et contrairement aux approches précédentes, ne s'appuient pas sur des approximations des fonctions de pénalité, évitent l'inversion de matrices et exploitent l'efficacité de l'algorithme de montée de coordonnées, particulièrement dans l'approche proximale par montée de coordonnées. Mots-clés: Modèles de mélange ; Mélange d'experts ; Estimation régularisée ; Sélection de variables, Lasso, Régularisation ; Parcimonie ; Algorithme EM ; Algorithme MM ; Proximal-Newton ; Montée de coordonnées ; Clustering, classification ; Régression ; Prédiction. *** Estimation and Feature Selection in High-Dimensional Mixtures-of-Experts Models Abstract: This thesis deals with the problem of modeling and estimation of high-dimensional MoE models, towards effective density estimation, prediction and clustering of such heterogeneous and highdimensional data. We propose new strategies based on regularized maximum-likelihood estimation (MLE) of MoE models to overcome the limitations of standard methods, including MLE estimation

  48) with c is an absolute constant, the L p (I d , λ) norm is defined as f Lp(I d ,λ) ≡ I d |f | p dλ

	2.3. MIXTURE MODELS FOR REGRESSION DATA
	the manifold Q K is given by
	1/p and

  .64) where µ k and Σ k are respectively the mean vector and covariance matrix of kth cluster, ρ k CHAPTER 2. STATE OF THE ART is a d-dimensional vector, A is a p × d orthonormal matrix of loading on d unobserved factors (A T A = I d ), Ω k is a d × d positive definite symmetric matrix and D is a diagonal p × p matrix.

  2, 3.3 and 3.4. It turns out that the MoE and MoE+ℓ 2 models could not be considered as model selection methods. Besides that, by adding the ℓ 2 penalty functions, we can reduce the variance of the parameters in the gate. The BIC, MoE-Lasso+ℓ 2 (MM), MoE-Lasso+ℓ 2 (CA), MoE-Lasso+ℓ 2 (PN) and MoE-Lasso (PN)

  4: Boxplots of the gate's parameter estimates obtained by Mixture-of-Experts (MoE) with standard MLE, MoE with ℓ 2 regularization, and with BIC procedure for model selection compare with boxplots of the proposed methods: MoE with mixed ℓ 1 (Lasso) and ℓ

	Comp. True value	MoE	MoE+ℓ 2	MoE-BIC	MIXLASSO
		0	0.010 (.096)	0.009 (.097)	0.014 (.083)	0.043 (.093)
		0 1.5	-0.002 (.106) -0.002 (.107) -0.003 (.026) 1.501 (.099) 1.502 (.099) 1.495 (.075)	0.011 (.036) 1.404 (.086)
	Exp.1	0	0.000 (.099)	0.001 (.099)	0.000 (.037)	0.013 (.036)
		0 0 1	-0.022 (.102) -0.022 (.102) -0.001 (.097) -0.003 (.097) 1.003 (.090) 1.004 (.090)	0.002 (.020) 0.000 (.045) 0.998 (.077)	0.003 (.027) 0.013 (.040) 0.903 (.088)
		0 1	0.006 (.185) 1.007 (.188)	0.005 (.184) 1.006 (.188)	0.002 (.178) 1.002 (.187)	-0.063 (.188) 0.755 (.220)
	Exp.2	-1.5 0 0 2	-1.492 (.149) -1.494 (.149) -1.491 (.129) -1.285 (.146) -0.011 (.159) -0.012 (.158) -0.005 (.047) -0.023 (.071) 0.016 (.075) -0.010 (.172) -0.008 (.171) -0.006 (.079) 2.004 (.169) 2.005 (.169) 2.003 (.128) 1.891 (.159)
		0	0.008 (.139)	0.007 (.140)	0.008 (.053)	0.031 (.086)
		1	1.095 (.359)	1.008 (.306)	1.055 (.328)
		2	2.186 (.480)	1.935 (.344)	2.107 (.438)
	Gate	0 0 -1 0 0	0.007 (.287) -0.001 (.383) -0.031 (.222) 0.004 (.1.55) 0.038 (.250) -0.006 (.086) -1.131 (.413) -0.991 (.336) -1.078 (.336) -0.022 (.331) -0.033 (.281) -0.017 (.172) 0.025 (.283) 0.016 (.246) 0.005 (.055)	N/A
	σ	1	0.965 (.045)	0.961 (.045)	0.978 (.046)	1.000 (.053)
						2 regular-
	ization with Minorization-Maximization (MM) updates, Coordinate Ascent (CA) updates, and
	Proximal Newton (PN) updates; MoE with Lasso regularization bases on Proximal Newton
	updates.

before, it is hard to apply BIC in reality especially for high-dimensional data, since this involves a huge collection of model candidates. MIXTURE OF EXPERTS FOR REGRESSION AND CLUSTERING

Table 3 .

 3

2: The true value, mean and standard derivation for each coefficient of the estimated parameter vector via MoE, MoE+ℓ 2 , MoE-BIC and the MIXLASSO.

  4. EXPERIMENTAL STUDY Comp. True value MoE-Lasso+ℓ 2 MoE-Lasso+ℓ 2 MoE-Lasso+ℓ 2 MoE-Lasso

			(MM)	(CA)	(PN)	(PN)
		0	0.031 (.091)	0.026 (.089)	0.026 (.089)	0.026 (.089)
		0	0.009 (.041)	0.011 (.046)	0.010 (.045)	0.010 (.045)
		1.5	1.435 (.080)	1.435 (.080)	1.435 (.080)	1.434 (.080)
	Exp.1	0	0.012 (.042)	0.013 (.044)	0.013 (.045)	0.013 (.044)
		0	0.001 (.031)	0.000 (.032)	0.000 (.031)	0.000 (.032)
		0	0.013 (.044)	0.012 (.043)	0.012 (.043)	0.012 (.043)
		1	0.930 (.082)	0.930 (.082)	0.931 (.082)	0.931 (.082)
		0 1	-0.158 (.183) 0.661 (.209)	-0.162 (.177) 0.675 (.202)	-0.163 (.175) 0.675 (.200)	-0.165 (.175) 0.675 (.200)
	Exp.2	-1.5 0 0	-1.216 (.152) -0.018 (.055) 0.013 (.061)	-1.242 (.139) -0.018 (.055) 0.011 (.059)	-1.243 (.138) -0.018 (.055) 0.011 (.060)	-1.243 (.137) -0.018 (.055) 0.012 (.060)
		2	1.856 (.150)	1.876 (.149)	1.876 (.148)	1.876 (.148)
		0	0.022 (.062)	0.020 (.060)	0.020 (.060)	0.019 (.059)
		1	0.651 (.331)	0.759 (.221)	0.759 (.218)	0.778 (.224)
		2	1.194 (.403)	1.332 (.208)	1.333 (.206)	1.400 (.225)
		0	0.058 (.193)	0.024 (.068)	0.033 (.069)	0.028 (.067)
	Gate σ	0 -1 0 0 1	-0.025 (.214) -0.223 (.408) -0.082 (.243) -0.002 (.132) 1.000 (.052)	-0.011 (.039) -0.526 (.253) -0.032 (.104) -0.007 (.036) 0.989 (.050)	-0.014 (.068) -0.555 (.211) -0.040 (.106) -0.012 (.060) 0.989 (.050)	-0.014 (.072) -0.584 (.223) -0.039 (.111) -0.012 (.062) 0.989 (.050)

Table 3

 3

.3: The true value, mean and standard derivation for each coefficient of the estimated parameter vector via the regularized models: MoE-Lasso+ℓ 2 (MM), MoE-Lasso+ℓ 2 (CA), MoE-Lasso+ℓ 2 (PN) and MoE-Lasso (PN).

Table 3 .

 3

4:

The true value and mean squared error for each coefficient of the estimated parameter vector via MoE, MoE+ℓ 2 , MoE-BIC and the MIXLASSO.

  MIXTURE OF EXPERTS FOR REGRESSION AND CLUSTERING Mean squared error Comp. True value MoE-Lasso+ℓ 2 MoE-Lasso+ℓ 2 MoE-Lasso+ℓ 2 MoE-Lasso

			(MM)	(CA)	(PN)	(PN)
		0	0.0092 (.015)	0.0087 (.014)	0.0085 (.014)	0.0085 (.014)
		0	0.0018 (.005)	0.0022 (.008)	0.0021 (.007)	0.0021 (.007)
		1.5	0.0106 (.012)	0.0107 (.012)	0.0106 (.012)	0.0107 (.012)
	Exp.1	0	0.0019 (.005)	0.0021 (.006)	0.0022 (.006)	0.0022 (.006)
		0	0.0010 (.004)	0.0010 (.004)	0.0010 (.004)	0.0010 (.004)
		0	0.0021 (.007)	0.0020 (.006)	0.0020 (.006)	0.0020 (.006)
		1	0.0117 (.015)	0.0116 (.015)	0.0115 (.015)	0.0114 (.015)
		0	0.0585 (.072)	0.0575 (.079)	0.0575 (.077)	0.0580 (.077)
		1	0.1583 (.157)	0.1465 (.148)	0.1456 (.146)	0.1455 (.146)
	Exp.2	-1.5 0	0.1034 (.098) 0.0033 (.013)	0.0860 (.087) 0.0034 (.017)	0.0851 (.086) 0.0034 (.017)	0.0851 (.086) 0.0034 (.017)
		0	0.0039 (.019)	0.0037 (.020)	0.0037 (.020)	0.0037 (.020)
		2	0.0432 (.056)	0.0375 (.050)	0.0374 (.050)	0.0374 (.050)
		0	0.0043 (.017)	0.0040 (.015)	0.0039 (.015)	0.0039 (.015)
		1	0.2315 (.240)	0.1067 (.125)	0.1055 (.124)	0.0994 (.122)
		2	0.8123 (.792)	0.4890 (.277)	0.4881 (.275)	0.4111 (.269)
		0	0.0404 (.032)	0.0052 (.015)	0.0059 (.014)	0.0053 (.013)
	Gate	0	0.0501 (.050)	0.0017 (.007)	0.0049 (.028)	0.0054 (.031)
		-1 0	0.7703 (.760) 0.0656 (.066)	0.2885 (.295) 0.0120 (.062)	0.2428 (.212) 0.0128 (.061)	0.2226 (.213) 0.0137 (.068)
		0	0.0175 (.018)	0.0013 (.008)	0.0038 (.016)	0.0039 (.017)
	σ	1	0.0027 (.003)	0.0027 (.003)	0.0027 (.003)	0.0027 (.003)

Table 3

 3

	Model	C.rate	ARI
	MoE	89.57% (1.65%) 0.6226 (.053)
	MoE+ℓ 2	89.62% (1.63%) 0.6241 (.052)
	MoE-BIC	90.05% (1.65%) 0.6380 (.053)
	MoE-Lasso+ℓ 2 (MM) 87.76% (2.19%) 0.5667 (.067)
	MoE-Lasso+ℓ 2 (CA) 89.46% (1.76%) 0.6190 (.056)
	MoE-Lasso+ℓ 2 (PN) 89.53% (1.65%) 0.6210 (.052)
	MoE-Lasso (PN)	89.56% (1.66%) 0.6222 (.053)
	MIXLASSO	82.89% (1.92%) 0.4218 (.050)

.5: The true value and mean squared error for each coefficient of the estimated parameter vector of the regularized models: MoE-Lasso+ℓ 2 (MM), MoE-Lasso+ℓ 2 (CA), MoE-Lasso+ℓ 2 (PN) and MoE-Lasso (PN).

Table 3 .

 3 6: Average of the accuracy of clustering (correct classification rate and Adjusted Rand Index) comparison among the non-penalized MoE model, MoE with ℓ 2 regularization, MoE with the BIC criterion for model selection and the proposed regularized models. Finally, the results for the standard MIXLASSO (Mixture of regressions with Lasso regularization) is also given.

	Model			Sparsity criteria		Clustering criteria
	(λ, γ, ρ)	BIC	Exp.1	Exp.2		Gate	C.rate	ARI
		S1	S2	S1	S2	S1	S2
	(9, 4, ρ)	-512.75 (13.67) 0.6475 1.000 0.7500 1.000 0.6425 0.995 89.61% (1.65%) 0.6236 (.052)
	(9, 5, ρ)	-512.78 (13.88) 0.6450 1.000 0.7533 1.000 0.7250 0.995 89.50% (1.68%) 0.6202 (.053)
	(9, 6, ρ)

three difference values of λ and γ are used λ ∈ {9, 10, 11}, γ ∈ {4, 5, 6}. The tuning parameter ρ = 0.1 log n as in our simulation. These criteria are given on 100 simulation data sets.

From Table

3

.7 it shows that the model with highest BIC value with respect to (λ, γ, ρ) = 3.4. EXPERIMENTAL STUDY

Table 3 .

 3 8: Fitted models for housing data with MLE and MoE-Lasso+ℓ 2

	Based on the

Table 3 .

 3 2 (CA), -372.377 compares with -374.489. All the regularization methods give comparative results with the MLE. MIXTURE OF EXPERTS FOR REGRESSION AND CLUSTERING 9: Fitted models for housing data with our proposed algorithms: MoE-Lasso+ℓ 2 (CA), MoE-Lasso+ℓ 2 (PN) and MoE-Lasso (PN).

		MoE-Lasso+ℓ 2 (CA)	MoE-Lasso+ℓ 2 (PN)	MoE-Lasso (PN)
	Features		σ = 0.346		σ = 0.352		σ = 0.353
		Exp.1 Exp.2 Gate Exp.1 Exp.2 Gate Exp.1 Exp.2 Gate
	x 0	2.20	2.82	0.79	2.19	2.83	0.99	2.19	2.83	1.00
	x 1	-0.09	-	-	-0.09	-	-	-0.09	-	-
	x 2	-	0.07	-	-	0.06	-	-	0.06	-
	x 3	-	-	0.41	-	-	0.58	-	-	0.59
	x 4	0.05	0.06	-	0.04	0.06	-	0.04	0.06	-
	x 5	-0.08	-	-	-0.07	-	-	-0.07	-	-
	x 6	-	0.56	-	-	0.59 -0.25	-	0.59 -0.21
	x 7	-0.05	-	-	-0.04	-	-	-0.04	-	-
	x 8	-0.03	-0.19	-	-	-0.20	-	-	-0.19	-
	x 9	-	0.60	-	-	0.54	-	-	0.55	-
	x 10	-0.01	-	-	-0.003	-	-	-0.003	-	-
	x 11	-0.10	-0.08 0.28	-0.09	-0.06 0.39	-0.09	-0.06 0.39
	x 12	0.05	-	-	0.05	-	-	0.05	-	-
	x 13	-0.29	-0.57 1.05	-0.29	-0.51 1.26	-0.29	-0.51 1.36
						Predicted criteria
		Model		Method 1		Method 2
					R 2	MSE	R 2	MSE
			MoE		0.8457 0.1544 (.577) 0.8981 0.1019 (.236)
	MoE-Lasso+ℓ 2 (Khalili, 2010) 0.8094 0.2044 (.709) 0.8698 0.1371 (.286)
		MoE-Lasso+ℓ 2 (CA)	0.8221 0.1989 (.619) 0.8905 0.1104 (.254)
		MoE-Lasso+ℓ 2 (PN)	0.8180 0.1903 (.717) 0.8839 0.1172 (.281)
		MoE-Lasso (PN)		0.8204 0.1878 (.702) 0.8832 0.1178 (.282)

Table 3

 3

.10: Results for Housing data set with a two-component MoE model.

Table 3 .

 3 11: Fitted models for housing data with MoE-Lasso (PN) method (K = 3).

	3.4. EXPERIMENTAL STUDY

Table 3

 3

.12: Results for Housing data set obtained by our MoE-Lasso (PN) (K = 3).

Table

					3.4. EXPERIMENTAL STUDY
	Features	MLE, σ = 0.277	MIXLASSO, σ = 0.25
		Exp.1	Exp.2	Gate	Exp.1	Exp.2
	x 0	6.0472 6.7101 -0.3958	6.41	7.00
	x 1	-0.0073 -0.0197 0.1238	-	-0.32
	x 2	-0.0283 0.1377 0.1315	-	0.29
	x 3	0.0566 -0.4746 1.5379	-	-0.70
	x 4	0.3859 0.5761 -1.9359	0.20	0.96
	x 5	-0.2190 -0.0170 -0.9687	-	-
	x 6	-0.0586 0.0178 0.4477	-	-
	x 7	-0.0430 0.0242 -0.3682 -0.19	-
	x 8	0.3991 0.0085 1.7570	0.26	-
	x 9	-0.0238 -0.0345 -1.3150	-	-
	x 10	-0.1944 0.0412 0.6550	-	-
	x 11	0.0726 0.1152 0.0279	-	-
	x 12	0.0250 -0.0823 0.1383	-	-
	x 13	-2.7529 1.1153 -7.0559	0.79	0.70
	x 14	2.3905 -1.4185 5.6419	0.72	-
	x 15	-0.0386 1.1150 -2.8818	0.15	0.50
	x 16	0.2380 0.0917 -7.9505	-	-0.36
	3.3338 -0.8335 8.7834 x 1 * x 14 -2.4869 2.5106 -7.1692 x 1 * x 13 0.4946 -0.9399 2.6319 x 1 * x 15 x 1 * x 16 -0.4272 -0.4151 7.9715 0.7445 0.3201 0.5622 x 3 * x 13 x 3 * x 14 -0.0900 -1.4934 0.1417 x 3 * x 15 -0.2876 0.4381 -0.9124 x 3 * x 16 -0.2451 -0.2242 -5.6630 -0.18 -0.21 0.63 0.34 --0.14 -0.7738 0.1335 4.3174 -x 7 * x 13 -x 7 * x 14 -0.1566 1.2809 -3.5625 -x 7 * x 15 -0.0104 0.2296 -0.4348 0.5733 -0.2905 3.2613 -x 7 * x 16 0.29 x 8 * x 13 -1.6898 -0.0091 -8.7320 0.7843 -1.3341 6.2614 -0.14 x 8 * x 14 0.3711 -0.4310 0.8033 -x 8 * x 15 x 8 * x 16 -0.2158 0.7790 2.6731 -	------0.38 -0.74 --0.34 --0.46 ---

  Features MoE-Lasso+ℓ 2 (CA), σ = 0.345 MoE-Lasso+ℓ 2 (PN), σ = 0.345

		Exp.1	Exp.2	Gate	Exp.1	Exp.2	Gate
	x 0	5.9580 6.9297	0.0046	5.9570 6.9295	0.0037
	x 1	-0.0122	-	-	-0.0117	-	-
	x 2	-0.0064	-	-	-0.0068	-	-
	x 3	-	-	-	-	-	-
	x 4	0.4521 0.0749	-	0.4512 0.0751	-
	x 5	-	-	-	-	-	-
	x 6	-0.0051	-	-	-0.0050	-	-
	x 7	-	-	-	-	-	-
	x 8	-	0.0088	-	-	0.0066	-
	x 9	0.0135 0.0192	-	0.0140 0.0204	-
	x 10	-0.1146	-	-	-0.1143	-	-
	x 11	-0.0108 0.0762	-	-0.0107 0.0769	-
	x 12	-	-	-	-	-	-
	x 13	-	0.3855	-0.3946	-	0.3837	-0.3926
	x 14	0.0927 -0.0550	-	0.0932 -0.0540	-
	x 15	0.3268 0.3179	-	0.3273 0.3179	-
	x 16

Table 3 .

 3 15: Results for Baseball salaries data set.

	Data	No. features No. observations No. experts	CA	PN
	Simulation	7	300	2	45.34 (14.28) (s)	5.03 (1.09) (s)
	Simulation	7	300	3	7.94 (13.22) (m) 20.52 (9.23) (s)
	Baseball salaries	33	337	2	17.9 (15.87) (m) 46.76 (21.02) (s)
	Residential Data	108	372	3	N/A	3.63 (0.58) (m)

Table 3 .

 3 16: Results for CPU times. in this setting, in which it provides a sparse model and competitive criteria in prediction and clustering. We also provide the correlation and the mean squared error between those values after clustering the data in Table3.21.

		Predictive criteria		Number of zero coefficients
	Method	R 2	MSE	Exp.1 Exp.2 Exp.3 Gate.1 Gate.2
	Proximal Newton 0.991 0.0093 (.059)	71	38	75	97	101
	Table 3.20: Results for residential building data set by using the clustering method 1.
		Predictive criteria	Number of observations
	Method	R 2	MSE	Class 1 Class 2 Class 3
	Proximal Newton 0.9994 0.00064 (.0018)	59	292	21

Table 3 .

 3 17: Fitted model parameters for residential building data obtained by our MoE-Lasso +ℓ 2 (PN) algorithm (part 1).

		Predictive criteria		Number of zero coefficients
	Method	R 2	MSE	Exp.1 Exp.2 Exp.3 Gate.1 Gate.2
	Proximal Newton 0.9895 0.0204 (.056)	31	60	55	106	104

Table 3 .

 3 22: Results for the subset of the residential building data set by using the clustering method 1. MIXTURE OF EXPERTS FOR REGRESSION AND CLUSTERING

	Features	Expert, σ = 0.0255	Gating network
		Exp.1	Exp.2	Exp.3	Gate.1	Gate.2

Table 3 .

 3 18: Fitted model parameters for residential building data obtained by our MoE-Lasso +ℓ 2 (PN) algorithm (part 2).

		Predictive criteria	Number of observations
	Method	R 2	MSE	Class 1 Class 2 Class 3
	Proximal Newton 0.9999 0.00025 (.0014)	63	11	16
	Table 3.23: Results for the subset of the residential building data set by using the clustering
	method 2.

  In this chapter, we proposed a regularized MLE for the MoE model which encourages sparsity, and developed three versions of a blockwise EM algorithm to monotonically maximize this MIXTURE OF EXPERTS FOR REGRESSION AND CLUSTERING

	Features	Expert, σ = 0.0159	Gating network
		Exp.1	Exp.2	Exp.3	Gate.1	Gate.2
	x 0	0.09048 0.21992 0.05460 0.73646 -0.54048
	x 1	-	-	-	-	-
	x 2	0.00837	-	0.00112	-	-
	x 3	-	-	-	-	-
	x 4	0.04498 0.07325 0.00001	-	-
	x 5	0.08075 0.00807 0.00010	-	-
	x 6	-0.00836	-	-0.02235 0.02205	-
	x 7	0.01337 -0.00009 -0.00922	-	-
	x 8	0.02375 0.00443 0.00668	-	-
	x 9	0.02194 0.00379 -0.03344	-	-
	x	-0.01305 -0.00079 0.00560	-	-
	x	0.12763 0.01256 0.08537	-	0.16264
	x	1.08977 0.72843 1.04263	-	-
	x	0.00171 0.09792	-

  MIXTURE OF EXPERTS FOR REGRESSION AND CLUSTERING

	Features	Expert, σ = 0.0159	Gating network
		Exp.1	Exp.2	Exp.3	Gate.1 Gate.2
	x	-0.08791	-	-	-	-
	x	-0.06590 -0.13467 0.03526	-	-
	x	0.05718	-	-	-	-
	x	-0.14786 -0.03133	-	-	-
	x	-0.12865 -0.07620 -0.09485	-	-
	x	0.04578 0.04694	-	-	-
	x	0.01510 0.01860 0.08887	-	-
	x	-0.00755 0.00441 0.01526	-	-0.56947
	x

Table 4 .

 4 1: Sensitivity (S 1 ) and specificity (S 2 ) results of the Lasso regularized MoE models for Poisson outputs (PMoE-Lasso) and logistic ouputs (LMoE-Lasso).

				Expert 1	Expert 2	Gate
				S 1	S 2	S 1	S 2	S 1	S 2
		PMoE-Lasso 0.717 1.000 0.818 1.000 0.835 1.000
		LMoE-Lasso 0.693 0.960 0.835 0.805 0.780 0.980
		True value		LMoE-Lasso		MoE method
	Exp. 1 Exp. 2 Gate Exp. 1 Exp. 2	Gate		Exp. 1	Exp. 2	Gate
	0	0	1	-0.1184 -0.1470 0.5604 -2.5467 49.4886 0.4417
	-1	1	0	-0.6242	0	0		-1.8442 31.0822 -0.0505
	2	0	0	1.3393	0	0.0411		3.7090 -30.1612 -0.0523
	0	0	1	0	0	0.7802 -0.3482 48.1645 0.3263
	0	-2	0	0	-1.5576	0		0.9839 -66.4277 0.6738
	1.5	0	0	1.2773	0	-0.1194 2.7540	-9.4606 -0.7398
	0	0	-1.5	0.2138	0	-0.9343 -0.5401 -6.1314 -0.7966

Table 4 .

 4 2: Estimated parameters for a typical data set obtained with Lasso regularized MoE (LMoE-Lasso) and MLE for MoE with logistic observations.

Table 4 .

 4 3: Estimated parameter vector obtained by PMoE and our PMoE-Lasso for Poisson outputs.

Table 4 .

 4 4: Estimated parameter vector obtained by LMoE+ℓ 2 and the proposed LMoE-Lasso models for logistic outputs.

	Model	C.rate	ARI
	PMoE	88.85% (2.04%) 0.5965 (.063)
	PMoE-Lasso 88.96% (2.03%) 0.6004 (.063)
	LMoE+ℓ 2	81.02% (4.00%) 0.3813 (.092)
	LMoE-Lasso 82.06% (2.93%) 0.3985 (.078)

Table

Table 4 .

 4 6: Fitted regularized MoE model (PMoE-Lasso) to the heart disease data.observations. 160 observations among them have zero response value. Generally, an appropriate approach for this type of data is to use the zero inflated Poisson regression model (ZIP model). However, the regularized MoE of the Poisson regression is tested and observed on its behavior with this type of data. Taking K = 2 and focusing on the regularized MoE for Poisson regression, the model's estimated parameters are provided in Table4.6. There are two components, the first one has 108 objects and the second one has 189 objects. The second class contains 156 over 160 observations that have zero response value. In this case, it looks like the data is splitted into two parts, with one part contains mainly zero response value similar with the approach of ZIP.

	4.4. EXPERIMENTAL STUDY

Table 4 .

 4

	LMoE-Lasso

10: Classification accuracy and percentage of features reduction results compared between Peralta's method

[START_REF] Peralta | Embedded local feature selection within mixture of experts[END_REF]

and our proposed method (LMoE-Lasso).

Table 4

 4

	.7: Fitted regularized MoE model (LMoE-Lasso) to Ionosphere data.
	123

Table 4 .

 4 8: Fitted regularized MoE model (LMoE-Lasso) to Musk-1 data (part 1).

	Feature	Exp.1	Exp.2	Gate	Feature	Exp.1	Exp.2	Gate
	x84	-0.03139	0.55857	-1.21692	x126	0.36082	-

Table 4

 4

	.9: Fitted regularized MoE model (LMoE-Lasso) to Musk-1 data (part 2).
	125

  Hence, to interpret the model in highdimensional setting is an interesting question and should need more improvement in theoretical results. DATA

	Dataset name	Classification Nb. of zero coefficients Nb. of observations
		accuracy	Exp.1 Exp.2 Gate	Class 1	Class 2
	Musk-1 (Subset)	100%	145	165	161	89	11
	Table

Table 4 .

 4 11: Fitted regularized MoE model (LMoE-Lasso) to the subset of Musk-1 data (part 1).

	Feature	Exp.1	Exp.2	Gate	Feature	Exp.1	Exp.2	Gate
	x84	-	-	-0.27199	x126

Table A

 A

	λDAD T λ k B k λB k λ k B λB λ k I p λI p	EEE VVI EVI VEI EEI VII EII	(K -1) + Kp + p(p + 1)/2 (K -1) + Kp + Kp (K -1) + Kp + Kp -(K -1) (K -1) + Kp + p + (K -1) (K -1) + Kp + p (K -1) + Kp + K (K -1) + Kp + 1	K -1)	1429 1427 302 300 204 202 155 153

.1: Number of free parameters for parsimonious GMMs with K components and p variable.

•

  Huynh, B.T. (May, 2019). Estimation and feature selection in mixtures of generalized linear experts models. Talk at LMNO Lab. Software • R package of open-source codes available at https://github.com/fchamroukhi/prEMME: -RMoE: regularized MoE with Gaussian outputs; -PoissonRMoE: regularized MoE with Poisson outputs; -LogisticRMoE: regularized MoE with multinomial outputs.

https://github.com/fchamroukhi/prEMME

Acknowledgements

-1 0

DATA

The parameters w are therefore updated by maximizing the function (4.11). Here, the composite function Q(w; θ [q] ) is concave and does not have the weighted Lasso form. One can use coordinate ascent algorithm to update w since the penalty part has a separate structure (see [START_REF] Tseng | Convergence of a block coordinate descent method for nondifferentiable minimization[END_REF] for more details). However, this approach requires a lot of computing and is not suitable for large scale data (see [START_REF] Chamroukhi | Regularized maximum likelihood estimation and feature selection in mixtures-of-experts models[END_REF]). In this case, proximal Newton algorithm and proximal Newton-type algorithm are good choices to overcome these drawbacks. The principle of these methods are described in Appendix C.1. The idea of these approaches lies in the fact that they approximate the smooth part of Q(w; θ [q] ) with a local quadratic function. After that, one will solve a weighted Lasso regression problem, which has a closed-form update. The solution of this weighted Lasso regression a direction that one can choose to improve the value of Q(w; θ [q] ) using backtracking line search.

The methods for updating the gating network's parameters using proximal Newton, and proximal Newton-type method are described in the next section.

Proximal Newton-type procedure for updating the gating network

For updating the gating network, the similar approaches which were used to update these parameters for Gaussian outputs in Chapter 3 can be applied once again. Hence, the same proximal Newton and proximal Newton-type algorithms in Section 3.3.3 are the key methods to update w in this chapter. A little adjustment is provided by setting ρ = 0 in this case to have a closed-form update for w kj after each loop of the coordinate ascent algorithm. This is done since we remove the ℓ 2 the penalty in our model.

Proximal Newton-type procedure for updating the experts network

Expert network with Poisson outputs

In this case we consider the situation in which the response Y i is a count variable and the conditional probability distribution of Y i , given X i and Z i is described as a Poisson distribution. Therefore, the generative model (4.2) of Y is the one of Poisson expert regressor and is given by

Hence, the expert's distribution p k (y i |x i ; θ k ) becomes

.13)

If the count data Y is such that the probability of zero is large then the zero-inflated Poisson (ZIP) regression model should be considered. For the regularized zero-inflated regression models, we refer the reader to [START_REF] Buu | New variable selection methods for zeroinflated count data with applications to the substance abuse field[END_REF][START_REF] Wang | EM for regularized zero-inflated regression models with applications to postoperative morbidity after cardiac surgery in children[END_REF][START_REF] Tang | Risk factor selection in rate making: EM adaptive Lasso for zero-inflated poisson regression models[END_REF].

Updating the parameter vector for the kth Poisson regressor expert requires the maximization DATA

Simulation study

For this simulation, we generate n = 300 predictors x from a multivariate Gaussian distribution with zero mean and correlation structure given by corr(x ij , x ij′ ) = 0.5 |j-j′| . Meanwhile, the response Y |x is generated from a logistic model with two classes and a Poisson model of K = 2 expert components with the following regression coefficients:

• Simulation parameters for the Poisson model:

(w 10 , w 1 ) T = (1, 0, 0, 1, 0, -1.5, 0) T .

• Simulation parameters for the multinomial-logistic model (R = 2):

(β 110 , β 11 ) T = (0, -1, 2, 0, 0, 1.5, 0) T ;

(β 210 , β 21 ) T = (0, 1, 0, 0, -2, 0, 0) T ;

(w 10 , w 1 ) T = (1, 0, 0, 1, 0, 0, -1.5) T .

100 data sets were generated for each simulation. The results will be presented in the following sections. Histogram of the response variable for a typical simulated data in Poisson MoE model is presented in Figure 4.1.

Appendix A

Mathematics materials for State of the art

A.1 Monotonicity of the EM algorithm

In this section, we provide a proof for the monotonicity of the EM algorithm based on Information inequality theorem (Lange, 1998, Section 10.3).

Theorem A.1.1 (Information Inequality). Let f and g be probability densities with respect to a measure µ. Suppose f > 0 and g > 0 almost everywhere relative to µ. If E f denotes expectation with respect to the probability measure f dµ, then E f (log f ) ≥ E f (log g), with equality only if f = g almost everywhere relative to µ.

Proof. Becauselog(.) is a strictly convex function on (0, ∞), we can use Jensen ′ s inequality for the random variable g/f and get

be the conditional density of Z given X = x. Using Theorem A.1.1 we have Q(θ [q] ; θ [q] ) -L(θ [q] q+1] ; θ [q] ) -Q(θ [q] ; θ [q] ) ≥ 0, because θ [q+1] = arg max θ∈Ω Q(θ; θ [q] ).

A.2 Updated formulas for the convariance matrices of GMMs

To see these updated formulas, we can rewrite the second term on the right-hand side of (2.18) above as

and for heteroscedastic case, note that by taking µ

where tr(A) represents for the trace of a square matrix A, and the sum

) T is denoted as A k (for k = 1, . . . , K).

A.3. COVARIANCE STRUCTURE OF GMMS IN HIGH-DIMENSIONAL SETTING

In addition,

Thus, ỹ2 is an eigenvector associated with γ and ỹ2 , ỹ1 = 0. For all l < j, we have ỹj , ỹl = y j -

Since ỹi , ỹl = 0, ∀ i, l < j (i = l), then ỹj , ỹl = y j , ỹl -y j , ỹl ỹl , ỹl ỹl , ỹl = 0.

Moreover,

Therefore, ỹj is an eigenvector associated with γ and ỹi , ỹj = 0, ∀ i < j. Normalized the system {ỹ 1 , . . . , ỹp-d }, we obtain pd eigenvectors {ŷ 1 , . . . , ŷp-d } corresponded with γ such that ŷj , ŷj = 1, and ŷj , ŷi = 0, for all i, j ∈ {1, . . . , p -d} and i = j. Now, consider the p × p matrix Q, where

It is not hard to prove that Q is an orthogonal matrix and

First, we rewrite Σ as

where the p × d matrix Γ is given by

Then, the Equation (A.7) has the same formula which is used to construct the covariance matrices of the MFA in (2.61). Hence, based on the work of [START_REF] Lawley | Factor analysis as a statistical method[END_REF] (Bouveyron et al., 2007a).

A.4. MATHEMATICS MATERIALS FOR LASSO REGRESSION

A.4 Mathematics materials for Lasso regression A.4.1 Smallest value of λ to obtain all zero coefficients in Lasso regression

Consider the Lasso problem arg min

We have

The equality holds for β = 0.

A.4.2 Proof of the properties of the Lasso solutions

The proof of Lemma 2.4.2 is given in this section.

Assume that X β > Xη. Consider γ = ( β + η)/2. Using the fact that the function

Asides from this, the function g(a) = λ|a| (λ ≥ 0) is also convex. Thus,

From (A.9) and (A.10), we obtain this inequality

This contradicts with the assumption that β, η are two solutions with optimal value c * . Hence,

The last property is apparent.

Appendix B

Monotonicity of the penalized log-likelihood for Gaussian MoE

The proposed EMM algorithm maximizes the penalised log-likelihood function (4.5). To show that the penalized log-likelihood is monotonically improved, that is

we need to show that q+1] , θ [q] ) ≥ Q(θ [q] , θ [q] ). (B.2) Indeed, as in the standard EM algorithm algorithm for the non-penalised maximum likelihood estimation, by applying Bayes theorem we have

and by taking the conditional expectation with respect to the latent variables z, given the observed data D and the current parameter estimation θ [q] , the conditional expectation of the penalised completed-data log-likelhood is given by:

Since the penalised log-likelihood function P L(θ) does not depend on the variables z, its expectation with respect to z therefore still unchanged and we get the following relation:

H(θ,θ [q] ) .

(B.5) Thus, the value of change of the penalised log-likelihood function between two successive iterations is given by: q+1] , θ [q] ) -Q(θ [q] , θ [q] ) " -" H(θ [q+1] , θ [q] ) -H(θ [q] , θ [q] ) " . (B.6)

APPENDIX B. MONOTONICITY OF THE PENALIZED LOG-LIKELIHOOD FOR GAUSSIAN MOE

As in the standard EM algorithm, it can be easily shown, by using Jensen' inequality, that the second term H(θ [q+1] , θ [q] ) -H(θ [q] , θ [q] ) in the r.h.s of (B.6) is negative and we therefore just need to show that the first term Q(θ [q+1] , θ [q] ) -Q(θ [q] , θ [q] ) is positive.

In the following, we show that Q(θ [q+1] , θ [q] ) ≥ Q(θ [q] , θ [q] ). First, the Q-function is decomposed as

and is accordingly maximized separately with respect to w, {β k } and {σ 2 k }. To update w, first we use a univariate MM algirthm to iteratively maximize the minorizing function which satisfies Q(w; θ [q] ) ≥ G(w|w [q] ), ∀w (B.8) and Q(w [q] ; θ [q] ) = G(w [q] |w [q] ). (B.9)

In our situation, the minorizing function is concave and has a separate structure. We thus use a one-dimensional Newton Raphson algorithm to maximize it. Thus, the solution w

and hence we have q] ; θ [q] ). (B.11)

Hence, the MM algorithm leads to the improvement of the value of the Q(w; θ [q] ) function.

For the second version of the EM algorithm which uses the coordinate ascent algorithm to update w, we rely on the work of [START_REF] Tseng | Coordinate ascent for maximizing nondifferentiable concave functions[END_REF] and [START_REF] Tseng | Convergence of a block coordinate descent method for nondifferentiable minimization[END_REF], where it is proved that, if the nonsmooth part of Q(w; θ [q] ) has a separate structure, the coordinate ascent algorithm is successful in finding the w [q+1] = arg max w Q(w; θ [q] ). At each step of the coordinate ascent algorithm, within the M-step of the EM algorithm, we iteratively update the jth component, while fixing the other parameters to their previous values:

s being the current iteration of the coordinate ascent algorithm. The function Q(w kj ; θ [q] ) is concave, and the used iterative procedure to find w

[q+1] kj is the Newton Raphson algorithm.

Hence, the coordinate ascent leads to the improvement of the function Q(w; θ [q] ), that is q+1] ; θ [q] ) ≥ Q(w [q] ; θ [q] ). (B.13)

For the proximal Newton-type method, we based on the works of Lee et al. (2014) and [START_REF] Lee | Efficient l 1 regularized logistic regression[END_REF] to show that the value of Q(w; θ [q] ) is always increase after each loop. Specially, the proximal Newton-type method which uses a constant matrix B to approximate the Hessian matrix is proofed to be a special case of the MM algorithm (see (Lange, 2013, Sec. 8.7) and [START_REF] Gormley | A mixture of experts model for rank data with 152 BIBLIOGRAPHY applications in election studies[END_REF]). Hence, it always improve the values of Q(w; θ [q] ). For more details see Appendix C.1.

Appendix C

Proximal Newton-type methods

C.1 Proximal Newton-type methods

Assume that we want to solve an optimization problem given by

with a composite function f (x) where g is a convex, continuously differentiable loss function, and h is a convex but non differentiable penalty function. Such problems include the Lasso, elastic net, etc. Proximal Newton-type methods approximate only the smooth part g with a local quadratic function of the form:

where ▽g(x k ) is the gradient vector of g at x k and H k is an approximation to the Hessian matrix ▽ 2 g(x k ). If we choose H k = ▽ 2 g(x k ), we obtain the proximal Newton method. In this method, one uses an iterative algorithm with initial value x 0 and in which at step k minimizes the proximal function fk (x) instead of f and then searches for the next value x k+1 based on the solution of (C.2) that will improve the value of f , i.e., f (x k+1 ) < f (x k ) by using a back tracking line search until the algorithm converges. Lee et al. (2014) and [START_REF] Lee | Efficient l 1 regularized logistic regression[END_REF] studied convergence properties of proximal Newton methods. A generic proximal Newton-type method can be listed as in Algorithm 4 (see Lee et al. (2014)).

C.2 Partial quadratic approximation for the gating network

The Q(w; θ [q] ) function in (3.9) is given as following

where the concave, continuously differentiable function I(w) is

) is a concave, continuously differentiable function and

The first and second derivatives of P k ({β k0 , β k }; θ [q] ) with respect to (β k0 , β k ) can easily obtained. It is not hard to show that

for j, h ∈ {0, . . . , p} and x i0 = 1. Thus the quadratic approximation of P k ({β k0 , β k }; θ [q] ) at ( βk0 , βk ) is given as following

and D( βk0 , βk ) is a function of ( βk0 , βk ).

C.3.2 Partial quadratic approximation for the Multinomial outputs

Finally, we construct the quadratic approximation for the function Q k (β k ; θ [q] ) in (4.12), where as before

) is a concave, continuously differentiable function and

The first and second derivatives of I(β k ) with respect to (β kr0 , β kr ) are

for j, h ∈ {0, 1, . . . , p} and x i0 = 1. Hence, the partial quadratic approximation Ĩr (β k ) of I(β k ) with respect to (β kr0 , β kr ) at βk can be described as following

, (C.16)

List of Publications and Communications

Journal papers The red lines represent the estimated densities using kernel density estimation and the black lines show the estimated densities using a GMM. . . . . . . . . . .

3.7

Histograms of the log of salary and its predicted value for baseball data. The red lines represent the estimated densities using non-parametric kernel density estimation, and the black lines represents the estimated densities using a GMM.

3.8 Histograms of the variable Y (V-9) and its predicted value for residential building data. The red lines represent the estimated densities using non-parametric kernel density estimation, and the black lines show the densities estimed using a GMM.

3.9 Histograms of the variable Y (V-9) and its predicted value for the subset of residential building data. The red lines are the estimated densities using non-

Contexte scientifique

De nos jours, des données massives sont recueillies et exploitées dans presque tous les domaines de la science, de la technologie, et de l'industrie.

Cela concerne de très nombreux objectifs allant de l'optimisation de la per- [START_REF] Bishop | Neural networks for pattern recognition[END_REF], les machines à vecteurs de support [START_REF] Vapnik | Statistical learning theory[END_REF], les

LIST OF TABLES bre de variables, et c'est encore plus difficile dans un scénario à grande dimension. Cela devient en effet plus limitant avec l'algorithme EM dont le calcul de l'étape E nécessite l'inversion des matrices de covariance. Le même problème se pose lors de l'application de l'algorithme de Newton-Raphson [START_REF] Boyd | Convex optimization[END_REF] [START_REF] Jacobs | Adaptive mixtures of local experts[END_REF][START_REF] Yuksel | Twenty years of mixture of experts[END_REF]Jiang and Tanner, 1999a,b;[START_REF] Grün | Fitting finite mixtures of generalized linear regressions in r[END_REF]. Les mélanges d'experts sont utilisés dans plusieurs applications

Contributions de la thèse

Le manuscrit est organisé comme suit.