Cette Thèse

Je Voudrais Remercier Vivement Monsieur

Hein Matthias

Tran Bach

Minh Tam

Hoai Minh

Vinh Thanh

Xuan Thanh

Bich Thuy

Minh Thuy

Tran Thuy

, ... Anh Vu

Keywords:

Je souhaite en premier lieu exprimer ma profonde gratitude à Madame le Professeur LE THI Hoai An, pour la confiance qu'elle m'a accordée en acceptant d'encadrer cette thèse de doctorat. Elle a dirigé, avec une grande aisance, mes travaux de recherche avec une grande patience, rigueur et enthousiasme. Sa pédagogie et ses capacités dans la théorie fondamentale et appliquée sur un large spectre de domaines d'applications m'ont guidé sur le chemin de la réussite. Je lui suis très reconnaissant pour ses conseils, ses encouragements, ses soutiens permanents et ses aides très précieuses tant sur le plan professionnel que personnel pendant plus de trois ans. Sans eux, je ne serais certainement pas en mesure d'atteindre mon objectif. Je vais garder dans mon coeur tout le meilleur qu'elle m'a accordé. Je profite de cette riche expérience pour développer mes recherches dans l'avenir. J'adresse respectueusement mes sincères remerciements à Monsieur PHAM DINH Tao, professeur à l'INSA de Rouen pour ses conseils, et son suivi dans mes travaux de recherche. Je voudrais lui exprimer toute ma reconnaissance pour les discussions approfondies très intéressantes que nous avons eues et pour m'avoir suggéré de nouvelles voies de recherche.

Résumé

De nos jours, avec l'abondance croissante de données de très grande taille, les problèmes de classification de grande dimension ont été mis en évidence comme un challenge dans la communauté d'apprentissage automatique et ont beaucoup attiré l'attention des chercheurs dans le domaine. Au cours des dernières années, les techniques d'apprentissage avec la parcimonie et l'optimisation stochastique se sont prouvées être efficaces pour ce type de problèmes. Dans cette thèse, nous nous concentrons sur le développement des méthodes d'optimisation pour résoudre certaines classes de problèmes concernant ces deux sujets. Nos méthodes sont basées sur la programmation DC (Difference of Convex functions) et DCA (DC Algorithm) étant reconnues comme des outils puissants d'optimisation non convexe.

La thèse est composée de trois parties. La première partie aborde le problème de la sélection des variables. La deuxième partie étudie le problème de la sélection de groupes de variables. La dernière partie de la thèse liée à l'apprentissage stochastique.

Dans la première partie, nous commençons par la sélection des variables dans le problème discriminant de Fisher (Chapitre 2) et le problème de scoring optimal (Chapitre 3), qui sont les deux approches différentes pour la classification supervisée dans l'espace de grande dimension, dans lequel le nombre de variables est beaucoup plus grand que le nombre d'observations. Poursuivant cette étude, nous étudions la structure du problème d'estimation de matrice de covariance parcimonieuse et fournissons les quatre algorithmes appropriés basés sur la programmation DC et DCA (Chapitre 4). Deux applications en finance et en classification sont étudiées pour illustrer l'efficacité de nos méthodes.

La deuxième partie étudie la ℓ p,0 -régularisation pour la sélection de groupes de variables (Chapitre 5). En utilisant une approximation DC de la ℓ p,0 -norme, nous prouvons que le problème approché, avec des paramètres appropriés, est équivalent au problème original. Considérant deux reformulations équivalentes du problème approché, nous développons différents algorithmes basés sur la programmation DC et DCA pour les résoudre. Comme applications, nous mettons en pratique nos méthodes pour la sélection de groupes de variables dans les problèmes de scoring optimal et d'estimation de multiples matrices de covariance.

Dans la troisième partie de la thèse, nous introduisons un DCA stochastique pour des problèmes d'estimation des paramètres à grande échelle (Chapitre 6) dans lesquelles la

Conclusion Conclusion

A Appendix

A.1 Bounded optimal solution set of the problem (5.29) . . . . . . . . . . . .

List of Figures

5.2

Comparative results of ℓ 1,0 (DCA1), ℓ 1,0 (DCA2), ℓ 2,0 (DCA1), ℓ 2,0 (DCA2), SOS GLASSO and GS MSVM in terms of the average number of selected features and its standard deviation (upper row), and the average percentage of selected features and its standard deviation (lower row) over 10 training/test set splits. Bold fonts indicate the best results in each row. .

Notation

Throughout the thesis, we use uppercase letters to denote matrices, and lowercase letters for vectors or scalars. Vectors are also regarded as matrices with one column. The table below summarizes some of the notation used in the thesis.

• p ℓ p -norm (0 < p < ∞), x p = ( n i=1 |x i | p ) 1/p , x ∈ R n • vector ℓ 2 -norm/Euclidean norm, x = ( n i=1 |x i | 2 ) 1/2 , x ∈ R n matrix ℓ 2 -norm/spectral norm, X = max u∈R n , u =1 Xu , X ∈ R m×n • 0 ℓ 0 -'norm', x 0 = |{i : x i = 0}|, X 0 = |{(i, j) : X ij = 0}| • F Frobenius norm, X F = ( m i=1 n j=1 X 2 ij ) 1/2 , X ∈ R m×n , scalar product, X, Y = m i=1 n j=1 X ij Y ij , X, Y ∈ R m×n
X ij element located at the position (i, j) of X X IJ submatrix of X with row (resp. column) indices in I (resp. J) X T transpose of a matrix X, (X T ) ij = X ji |X| absolute of X, |X| ij = |X ij | for all (i, j) sgn(X) matrix of signs of X, (sgn(X)) ij = sgn(X ij ) = -1 if X ij < 0, 1 if X ij > 0, and 0 otherwise diag(x) diagonal matrix whose the main diagonal is the vector x λ min (X) the smallest eigenvalue of X λ max (X) the largest eigenvalue of X X -1 inverse of the matrix X det(X) determinant of the matrix X tr(X) trace of the matrix X ∈ R n×n , tr(X) = n i=1 X ii 19

Introduction générale

Cadre général et nos motivations L'émergence d'Internet ainsi que la croissance rapide de la science et de la technologie au cours de ces dernières années ont stimulé l'énorme volume des ressources d'informations disponibles, et il est encore en croissance à un rythme incroyablement rapide. Sans surprise, nous somme confrontés à l'immense quantité de données généralement appelées Big Data. Ainsi, les méthodes traditionnelles d'apprentissage et de fouille de données (Machine Learning and Data Mining -MLDM) deviennent inefficaces pour le traitement de ce genre de données. D'où, il est absolument nécessaire de développer des méthodes efficaces et robustes.

Dans cette thèse, nous nous concentrons sur deux challenges en MLDM dans le contexte du big data: apprentissage avec la parcimonie sur des données de très grande dimensions et apprentissage stochastique sur une énorme quantité de données. L'apprentissage avec la parcimonie permet d'avoir une meilleure interprétation et de réduise "overfitting" en supprimant les redondantes variables. Pour la conception de modèles d'apprentissage, la modélisation parcimonieuse est basée sur la norme zéro (la norme zéro d'un vecteur est définie comme le nombre de ses termes non nulles). C'est la façon la plus naturelle pour aborder la sélection des variables en MLDM, mais le problème d'optimisation correspondant est NP-difficile. C'est pourquoi, dans ces travaux, les problèmes d'optimisation incluent des doubles difficultés. En effet, la première est de savoir comment traiter la norme zéro et la seconde est causée par la non-convexité des problèmes originaux. La difficulté de la norme zéro peut être surmontée par son approximation via une fonction DC (Difference of Convex functions). Le problème résultant est encore difficile, mais il possède des propriétés intéressantes et peut être ainsi résolu par les méthodes basées sur l'optimisation DC. En outre, lorsque les données possèdent certaines structures de groupe, nous sommes naturellement intéressés à la sélection de groupes importants de variables plutôt que des individus. La régularisation générale est proposée pour obtenir la parcimonie groupée. Egalement, la difficulté de cette régularisation peut être surmontée en utilisant une approximation DC appropriée, le problème résultant est donc un problème d'optimisation DC. Enfin, la présence de techniques d'optimisation stochastiques est justifié être efficaces en MLDM pour résoudre des problèmes avec un grand nombre de points de données d'entraînement. Mais le véritable challenge est des problèmes d'estimation des 21 paramètres non-convexes dans lesquelles la fonction objectif est la somme d'une grande famille des fonctions DC. Par conséquent, DCA stochastique est introduite afin d'obtenir un faible coût de calcul à chaque itération.

Sur le plan algorithmique, la thèse a proposé une approche unifiée, fondée sur la programmation DC et DCA, des outils puissant d'optimisation non convexe qui connaît un grand succès, au cours de deux dernières décennies, dans la résolution de nombreux problèmes d'application dans diverses domaines de sciences appliquées, en particulier en MLDM. De nombreuses expérimentations numériques sur différents types de données (biologie, image, finace, ...) réalisées dans cette thèse ont prouvé l'efficacité, la scalabilité, la rapidité des algorithmes proposés et leur supériorité par rapports aux méthodes standards.

La programmation DC et DCA considèrent le problème DC de la forme α = inf{f (x) := g(x)h(x) : x ∈ R n } (P dc ), où g et h sont des fonctions convexes définies sur R n et à valeurs dans R ∪ {+∞}, semicontinues inférieurement et propres. La fonction f est appelée fonction DC avec les composantes DC g et h, et gh est une décomposition DC de f . DCA est basé sur la dualité DC et des conditions d'optimalité locale. La construction de DCA implique les composantes DC g et h et non la fonction DC f elle-même. Or chaque fonction DC admet une infinité des décompositions DC qui influencent considérablement sur la qualité (la rapidité, l'efficacité, la globalité de la solution obtenue,...) de DCA. Ainsi, au point de vue algorithmique, la recherche d'une "bonne" décomposition DC et d'un "bon" point initial est très importante dans le développement de DCA pour la résolution d'un programme DC.

L'utilisation de la programmation DC et DCA dans cette thèse est justifiée par de multiple arguments (Pham Dinh and Le Thi, 2014):

-La programmation DC et DCA fournissent un cadre très riche pour les problèmes de MLDM: MLDM constituent une mine des programmes DC dont la résolution appropriée devrait recourir à la programmation DC et DCA. En effet la liste indicative (non exhaustive) des références dans Le Thi (Website) témoigne de la vitalité la puissance et la percée de cette approche dans la communauté de MLDM. -DCA est une philosophie plutôt qu'un algorithme. Pour chaque problème, nous pouvons concevoir une famille d'algorithmes basés sur DCA. La flexibilité de DCA sur le choix des décomposition DC peut offrir des schémas DCA plus performants que des méthodes standard. -L'analyse convexe fournit des outils puissants pour prouver la convergence de DCA dans un cadre général. Ainsi tous les algorithmes basés sur DCA bénéficient (au moins) des propriétés de convergence générales du schéma DCA générique qui ont été démontrées. -DCA est une méthode efficace, rapide et scalable pour la programmation non convexe.

A notre connaissance, DCA est l'un des rares algorithmes de la programmation non convexe, non différentiable qui peut résoudre des programmes DC de très grande dimension. La programmation DC et DCA ont été appliqués avec succès pour la modélisation DC et la résolution de nombreux et divers problèmes d'optimisation non convexes dans différents domaines des sciences appliquées, en particulier en MLDM (voir par exemple la liste des références dans Le Thi (Website)).

Il est important de noter qu'avec les techniques de reformulation en programmation DC et les décompositions DC appropriées, on peut retrouver la plupart des algorithmes existants en programmation convexe/non convexe comme cas particuliers de DCA.

En particulier, pour la communauté de MLDM, les méthodes très connus comme Expectation-Maximisation (EM) [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF], Succesive Linear Approximation (SLA) [START_REF] Bradley | Feature selection via concave minimization and support vector machines[END_REF], ConCave-Convex Procedure (CCCP) [START_REF] Yuille | The concave-convex procedure (CCCP)[END_REF], Iterative Shrinkage-Thresholding Algorithms (ISTA) [START_REF] Chambolle | Nonlinear wavelet image processing: Variational problems, compression, and noise removal through wavelet shrinkage[END_REF] sont des versions spéciaux de DCA.

Nos contributions

La thèse vise à développer de nouveaux modèles et méthodes pour cinq classes de problèmes difficiles et importants en MLDM: analyse discriminante linéaire de Fisher parcimonieuse, scoring optimal parcimonieuse, estimation de matrice de covariance parcimonieuse, sélection de groupes de variables et des applications dans scoring optimal et estimation de matrices de covariance, estimation des paramètres du modèle à variables latentes. Nous commençons par décrire brièvement les principales réalisations de la thèse.

Dans le premier temps, nous considérons le problème de la classification supervisée dans l'espace de grande dimension, dans lequel le nombre de variables est beaucoup plus grand que le nombre d'observations. Dans de nombreuses applications telles que la recherche d'information, la reconnaissance faciale et l'analyse des microarrays, nous rencontrons souvent ce genre de problème. Parmi plusieurs méthodes de classification dans la littérature, l'analyse discriminante linéaire (LDA) est considérée comme l'une des méthodes les plus populaires en raison de son avantage de la réduction de dimension. L'objectif principal de LDA est de trouver une transformation linéaire qui distingue au mieux les différentes classes. La classification est alors effectuée dans l'espace transformé en utilisant des mesures de distance. Il existe trois approches différentes pour aborder LDA, qui sont basées sur la résolution du modèle normal, problème discriminant de Fisher et le problème de scoring optimal, respectivement. Nous développons donc une nouvelle approche pour la sélection des variables pour LDA basée sur le problème discriminant de Fisher et la norme zéro. Pour aborder la norme zéro, nous étudions les approches d'approximation DC. Parmi plusieurs fonctions induisant de la parcimonie existantes, nous utilisons le Capped-ℓ 1 et la fonction exponentielle concave par morceaux. Le choix du Capped-ℓ 1 est motivé par ses avantages tant sur le plan théorique qu'algorithmique. De plus, la fonction exponentielle concave par morceaux a été montrée pour être efficace via des résultats numériques dans de nombreux travaux. Les problèmes résultants sont formulés sous forme de programmes DC, puis quatre shémas de DCA sont proposés. Les résultats expérimentaux sur les deux données réelles et simulées démontrent l'efficacité des algorithmes proposés par rapport aux certaines méthodes standards.

Le deuxième problème abordé dans cette thèse est la sélection des variables dans le scoring optimal en utilisant la régularisation ℓ 2 +ℓ 0 , appelé le problème de scoring optimal parcimonieuse (SOS). La résolution de SOS comprend des doubles difficultés. La première est la façon de traiter la norme zéro et la seconde est causée par la non-convexité du problème original de scoring optimal. La difficulté de la norme zéro est surmontée en utilisant deux approximations DC. Les problèmes d'optimisation résultants sont encore difficiles, mais ils possèdent des propriétés intéressantes: quand w k est fixé, la solution optimale du problème par rapport à la variable θ k peut être calculée explicitement. Pour chaque θ k fixé, nous sommes confrontés à un programme DC par rapport à la variable w k , donc nous sommes suggérés d'utiliser des schémas alternatifs basés sur DCA pour les résoudre. Nous prouvons que les principaux algorithmes convergent vers un point critique des problèmes approchés. Les performances des algorithmes proposés sont soigneusement examinées en les comparant avec sept méthodes standards sur tous les deux données simulées et réelles de grande dimension.

Toujours dans le cadre de la sélection des variables, nous considérons le troisième problème -estimation de matrice de covariance parcimonieuse (SCME). L'objectif est d'estimer une matrice de covariance parcimonieuse sur la base d'un échantillon de vecteurs Gaussiens. Beaucoup d'analyses statistiques de données de grande dimension exige l'estimation d'une matrice de covariance ou son inverse, telles que la gestion de portefeuille et de l'évaluation des risques, l'analyse de l'indépendance et des relations d'indépendance conditionnelle entre les composants dans les modèles graphiques, analyse en composantes principales, et ainsi de suite. Le Capped-ℓ 1 et la fonction exponentielle concave par morceaux sont choisis de nouveau pour la modélisation parcimonieuse, cependant nous sommes toujours confrontés à la difficulté de la non-convexité de la fonction log-vraisemblance négative. Chapter 1

DC programming and DCA

This chapter summarizes some basis concepts and results that will be the groundwork of the thesis.

DC programming and DCA, which constitute the backbone of nonconvex programming and global optimization, were introduced by Pham Dinh Tao in their preliminary form in 1985. Important developments and improvements on both theoretical and computational aspects have been completed since 1993 throughout the joint works of Le Thi Hoai An and Pham Dinh Tao. In this section, we present some basic properties of convex analysis and DC optimization and DC Algorithm that computational methods of this thesis are based on. The materials of this section are extracted from [START_REF] Thi | Analyse numérique des algorithmes de l'optimization DC[END_REF][START_REF] Pham Dinh | Convex analysis approach to D.C. programming: Theory, algorithms and applications[END_REF][START_REF] Thi | The DC (difference of convex functions) programming and DCA revisited with DC models of real world nonconvex optimization problems[END_REF].

Throughout this section, X denotes the Euclidean space R n and R = R ∪ {±∞} is the set of extended real numbers.

Fundamental convex analysis

A subset C of X is said to be convex if (1-λ)x+λy ∈ C whenever x, y ∈ C and λ ∈ [0, 1].
Let f be a function whose values are in R and whose domain is a subset S of X. The set

{(x, t) : x ∈ S, t ∈ R, f (x) ≤ t}
is called the epigraph of f and is denoted by epif .

We define f to be a convex function on S if epif is convex set in X ×R. This is equivalent to that S is convex and

f ((1 -λ)x + λy) ≤ (1 -λ)f (x) + λf (y), ∀x, y ∈ S, ∀λ ∈ [0, 1]
The function f is strictly convex if the inequality above holds strictly whenever x and y are distinct in S and 0 < λ < 1.

The effective domain of a convex function f on S, denoted by domf , is the projection on X of the epigraph of f

domf = {x : ∃t ∈ R, (x, t) ∈ epif } = {x | f (x) < +∞} and it is convex. The convex function f is called proper if domf = ∅ and f (x) > -∞ for all x ∈ S.
The function f is said to be lower semi-continuous at a point x of S if

f (x) ≤ lim inf y→x f (y)
Denote by Γ 0 (X) the set of all proper lower semi-continuous convex function on X.

Let ρ ≥ 0 and C be a convex subset of X. One says that a function θ :

C → R ∪ {+∞} is ρ-convex if θ[λx + (1 -λ)y] ≤ λθ(x) + (1 -λ)θ(y) - λ(1 -λ) 2 ρ x -y 2
for all x, y ∈ C and λ ∈ (0, 1). It amounts to say that θ -

(ρ/2) . 2 is convex on C. The modulus of strong convexity of θ on C, denoted by ρ(θ, C) or ρ(θ) if C = X, is given by ρ(θ, C) = sup{ρ ≥ 0 : θ -(ρ/2) . 2 is convex on C} One say that θ is strongly convex on C if ρ(θ, c) > 0.
A vector y is said to be a subgradient of a convex function f at a point

x 0 if f (x) ≥ f (x 0 ) + x -x 0 , y , ∀x ∈ X
The set of all subgradients of f at x 0 is called the subdifferential of f at x 0 and is denoted by ∂f (x 0 ). If ∂f (x) is not empty, f is said to be subdifferentiable at x.

We also have notations

dom ∂f = {x ∈ X : ∂f (x) = ∅} and range ∂f (x) = ∪{∂f (x) : x ∈ dom ∂f } Proposition 1.1 Let f be a proper convex function. Then 1. ri(domf ) ⊂ dom ∂f ⊂ domf
where ri(domf ) stands for the relative interior of domf .

2. If f has a unique subgradient at x, then f is differentiable at x, and ∂f (x) = {∇f (x)}.

3.

x 0 ∈ argmin{f (x) : x ∈ X} if and only if 0 ∈ ∂f (x 0 ).

Conjugates of convex functions

The conjugate of a function f :

X → R is the function f * : X → R defined by f * (y) = sup x∈X { x, y -f (x)} Proposition 1.2 Let f ∈ Γ 0 (X). Then we have 1. f * ∈ Γ 0 (X) and f * * = f . 2. f (x) + f * (y) ≥ x, y , for any x, y ∈ X.
Equality holds if and only if y ∈ ∂f (x) ⇔ x ∈ ∂f * (y).

Polyhedral Functions

A polyhedral set is a closed convex set having form

C = {x ∈ X : x, b i ≤ β i , ∀i = 1, . . . , m}, where b i ∈ X and β i ∈ R for all i = 1, . . . , m. A function f ∈ Γ 0 (X) is said to be polyhedral if f (x) = max{ a i , x -α i : i = 1, . . . , k} + χ C (x), ∀x ∈ X (1.1)
where f is subdifferentiable at x, and ∂f (x) is a polyhedral convex set. In particular, if f is defined by

a i ∈ X, α i ∈ R for i = 1, . . . , k
(1.1) with C = X then ∂f (x) = co{a i : i ∈ I(x)} where I(x) = {i ∈ {1, . . . , k} : a i , x -α i = f (x)}. 2. The conjugate f * is a polyhedral convex function. Moreover, if C = X then domf * = co{a i : i = 1, . . . , k} f * (y) = inf k i=1 λ i α i k i=1 λ i a i = y, k i=1 λ i = 1, λ i ≥ 0, ∀i = 1, . . . , k In particular, f * (a i ) = α i , ∀i = 1, . . . , k
Difference of convex (DC) functions

A function f is called DC function on X if it has the form f (x) = g(x) -h(x), x ∈ X
where g and h belong to Γ 0 (X). One says that gh is a DC decomposition of f and g, h are its DC components. If g and h are in addition finite on all of X then one says that f = gh is finite DC function on X. The set of DC functions (resp. finite DC functions) on X is denoted by DC(X) (resp. DC f (X)).

Remark 1.1 Give a DC function f having a DC decomposition f = gh. Then for every θ ∈ Γ 0 (X) finite on the whole X, f = (g + θ) -(h + θ) is another DC decomposition of f . Thus, a DC function f has finitely many DC decompositions.

DC optimization

General DC program

In the sequel, we use the convention +∞ -(+∞) = +∞.

For g, h ∈ Γ 0 (X), a general DC program is that of the form

(P ) α = inf{f (x) = g(x) -h(x) : x ∈ X}
φ := g + χ C is in Γ 0 (X).
We will always keep the following assumption that is deduced from the finiteness of α dom g ⊂ dom h and dom h * ⊂ dom g * .

(1.2)

Polyhedral DC program

In problem (P ), if one of the DC components g and h is polyhedral function, we call (P ) polyhedral DC program. This is an important class of DC optimization. It is often encountered in practice and has worthy properties.

Consider problem (P ) where h is a polyhedral convex function given by

h(x) = max{ a i , x -α i : i = 1, . . . , k}
By Proposition 1.3, the dual problem (D) has the form

α * = inf{h * (y) -g * (y) : y ∈ X} = inf{h * (y) -g * (y) : y ∈ co{a i : i = 1, . . . , k}} = inf{α i -g * (a i ) : i = 1, . . . , k}
Note that, if g is polyhedral convex and h is not, then by considering the dual problem (D) we have the similar formulation as above since g * is polyhedral.

Optimality conditions for DC optimization

A point x * is said to be a local minimizer of g -h if x * ∈ dom g ∩ dom h (so, (g -h)(x * ) is finite) and there is a neighborhood U of x * such that g(x) -h(x) ≥ g(x * ) -h(x * ), ∀x ∈ U. (1.3)
A point x * is said to be a critical point of gh if it verifies the generalized Kuhn-Tucker condition ∂g(x * ) ∩ ∂h(x * ) = ∅ (1.4)

Let P and D denote the solution sets of problems (P ) and (D) respectively, and let

P ℓ = {x * ∈ X : ∂h(x * ) ⊂ ∂g(x * )}, D ℓ = {y * ∈ X : ∂g * (y * ) ⊂ ∂h * (y * )}
Below, we present some fundamental results on DC programming (Pham Dinh and Le [START_REF] Pham Dinh | Convex analysis approach to D.C. programming: Theory, algorithms and applications[END_REF].

Theorem 1.1 i) Transportation of global minimizers: ∪{∂h(x) : x ∈ P} ⊂ D ⊂ dom h * .
The first inclusion becomes equality if g * is subdifferentiable in D. In this case D ⊂ (dom ∂g * ∩ dom ∂h * ).

ii) Necessary local optimality: if x * is a local minimizer of gh, then x * ∈ P ℓ .

iii) Sufficient local optimality: Let x * is a critical point of g -h and y * ∈ ∂g(x * )∩∂h(x * ).

Let U be a neighborhood of x * such that (U ∩ dom g) ⊂ dom ∂h. If for any x ∈ U ∩ dom g, there is y ∈ ∂h(x) such that h * (y)g * (y) ≥ h * (y * )g * (y * ), then x * is a local minimizer of gh. More precisely,

g(x) -h(x) ≥ g(x * ) -h(x * ), ∀x ∈ U ∩ dom g iv) Transportation of local minimizers: Let x * ∈ dom ∂h be a local minimizer of g-h. Let y * ∈ ∂h(x * ) and a neighborhood U of x * such that g(x)-h(x) ≥ g(x * )-h(x * ), ∀x ∈ U ∩ dom g. If y * ∈ int(dom g * ) and ∂g * (y * ) ⊂ U then y * is a local minimizer of h * -g * .
Remark 1.3 a) By the symmetry of the DC duality, these results have their corresponding dual part. For example, if y is a local minimizer of h *g * , then y ∈ D ℓ . b) The properties i), iii) and their dual parts indicate that there is no gap between the problems (P ) and (D). They show that globally/locally solving the primal problem (P ) implies globally/locally solving the dual problem (D) and vice-versa. Thus, it is useful if one of them is easier to solve than the other. c) The necessary local optimality condition ∂h * (x * ) ⊂ ∂g * (x * ) is also sufficient for many important classes programs, for example (Le [START_REF] Thi | The DC (difference of convex functions) programming and DCA revisited with DC models of real world nonconvex optimization problems[END_REF], if h is polyhedral convex, or when f is locally convex at x * , i.e. there exists a convex neighborhood U of x * such that f is finite and convex on U. We know that a polyhedral convex function is almost everywhere differentiable, that is it is differentiable everywhere except on a set of measure zero. Thus, if h is a polyhedral convex function, then a critical point of gh is almost always a local solution to (P ). d) If f is actually convex on X, we call (P) a "false" DC program. In addition, if ri(domg) ∩ ri(domh) = ∅ and x 0 ∈ domg such that g is continuous at [START_REF] Thi | The DC (difference of convex functions) programming and DCA revisited with DC models of real world nonconvex optimization problems[END_REF]. Thus, in this case, the local optimality is also sufficient for the global optimality. Consequently, if in addition h is differentiable, a critical point is also a global solution.

x 0 , then 0 ∈ ∂f (x 0 ) ⇔ ∂h(x 0 ) ⊂ ∂g(x 0 ) (Le

DC Algorithm (DCA)

The DCA consists in the construction of the two sequences {x k } and {y k } (candidates for being primal and dual solutions, respectively) which are easy to calculate and satisfy the following properties:

i) The sequences (gh)(x k ) and (h *g * )(y k ) are decreasing.

ii) Their corresponding limits x ∞ and y ∞ satisfy the local optimality condition (x ∞ , y ∞ ) ∈ P ℓ × D ℓ or are critical points of gh and h *g * , respectively. From a given point x 0 ∈ dom g, the DCA generates these sequences by the scheme

y k ∈ ∂h(x k ) = arg min{h * (y) -y, x k : y ∈ X} (1.5a) x k+1 ∈ ∂g * (y k ) = arg min{g(x) -x, y k : x ∈ X}. (1.5b)
The interpretation of the above scheme is simple. At iteration k of DCA, we replace the second component h in the primal DC program by its affine minorant .6) where y k ∈ ∂h(x k ). Then the original DC program reduces to the convex program

h k (x) = h(x k ) + x -x k , y k , ( 1
(P k ) α k = inf{f k (x) := g(x) -h k (x) : x ∈ X}
that is equivalent to (1.5a). It is easy to see that f k is a majorant of f at x k . Similarly, by replacing g * with its affine minorant

g * k (y) = g * (y k-1 ) + y -y k-1 , x k , (1.7)
where x k ∈ ∂g * (y k-1 ), we lead to the convex problem

(D k ) inf{h * (y) -g * k (y) : y ∈ X} whose solution set is ∂h(x k ).
Convergence properties of DCA Theorem 1.2 Suppose that the sequences {x k } and {y k } are generated by DCA. Then we have

i) The sequences {g(x k )h(x k )} and {h * (y k )g * (y k )} are decreasing and

• g(x k+1 ) -h(x k+1 ) = g(x k ) -h(x k ) iff y k ∈ ∂g(x k ) ∩ ∂h(x k ), y k ∈ ∂g(x k+1 ) ∩ ∂h(x k+1 ) and [ρ(g) + ρ(h)] x k+1 -x k = 0. Moreover if g or h are strictly convex then x k = x k+1 .
In such a case DCA terminates at the k th iteration (finite convergence of DCA)

• h * (y k+1 ) -g * (y k+1 ) = h * (y k ) -g * (y k ) iff x k+1 ∈ ∂g * (y k ) ∩∂h * (y k ), x k+1 ∈ ∂g * (y k+1 ) ∩ ∂h * (y k+1
) and [ρ(g * ) + ρ(h * )] y k+1y k = 0. Moreover if g * or h * are strictly convex, then y k+1 = y k . In such a case DCA terminates at the k th iteration (finite convergence of DCA).

ii

) If ρ(g) + ρ(h) > 0 (resp. ρ(g * ) + ρ(h * ) > 0)) then the series { x k+1 -x k 2 (resp. { y k+1 -y k 2 } converges.
iii) If the optimal value α of problem (P ) is finite and the infinite sequences {x k } and {y k } are bounded then every limit point x * (resp. y * ) of the sequence

{x k } (resp. {y k }) is a critical point of g -h (resp. h * -g * ).
iv) DCA has a linear convergence for general DC programs.

v) In polyhedral DC programs, the sequences {x k } and {y k } contain finitely many elements and DCA has a finite convergence. Especially, if h is polyhedral convex and h is differentiable at x * , then x 8 is a local minimizer of (P ).

DCA's distinctive feature relies upon the fact that DCA deals with the convex DC components g and h but not with the DC function f itself. DCA is one of the rare algorithms for nonconvex nonsmooth programming. Moreover, a DC function f has infinitely many DC decompositions which have crucial implications for the qualities (convergence speed, robustness, efficiency, globality of computed solutions,...) of DCA. For a given DC program, the choice of optimal DC decompositions is still open. Of course, this depends strongly on the very specific structure of the problem being considered. In order to tackle the large-scale setting, one tries in practice to choose g and h such that sequences {x k } and {y k } can be easily calculated, i.e., either they are in an explicit form or their computations are inexpensive. Very often in practice, the solution of (D k ) to compute the sequence {y k } is explicit because the calculation of a subgradient of h is explicitly obtained by using the usual rules for calculating subdifferential of convex functions. But the solution of the convex program (P k ), if not explicit, should be achieved by efficient algorithms well-adapted to its special structure, in order to handle the large-scale setting.

Special DCA and proximal operator

The general scheme of DCA requires to solve a sequence of the subproblems of the form (1.5b) that might be not easy to solve. The design of an efficient DCA for a concrete problem should be based is a special structure. How to exploit the nice effect of DC decomposition is a crucial question to be studied for each DC program.

Before closing this chapter let us discuss the a special DC decomposition that can be very efficiently in many practical problems. Consider the convex constrained DC program of the form min{f (x) = g 1 (x) + g 2 (x)h(x) : x ∈ C}, (1.8) where C ⊂ R n is a convex set and g 1 , g 2 , h are convex functions.

We assume that there exists a nonnegative number ρ such that the function ρ 2 x 2 -g 2 (x) is convex. In many practical problems that ρ exists and can be computed according to the properties of the function g 2 . For instance, when g 2 is twice continuously differentiable. We now write the problem (1.8) in the form of DC program with the following decomposition:

ḡ = χ C (x) + g 1 (x) + ρ 2 x 2 , h = ρ 2 x 2 -g 2 (x) + h(x).
The DCA applied to the problem (1.8) with above decomposition can be described as follows:

SDCA (Special DCA): Let x 0 ∈ R n and set l ← 0. Repeat 1. Calculate y l ∈ ∂ ρ 2 . 2 -g 2 (.) + h(.) (x l ) 2. Calculate x l+1 by solving the convex problem min{χ C (x) + g 1 (x) + ρ 2 x 2 -x, y l : x ∈ R n } (1.9) i.e., x l+1 = prox χ C +g 1 ρ (y l /ρ). 3. l ← l + 1 Until convergence of {x l }.
Here, prox ϕ ρ stands for the proximal operator associated to ϕ defined by prox ϕ ρ (t) = arg min

x {ϕ(x) + ρ 2 x -t 2 }.
Remark 1.4 i) When g 1 ≡ 0, the proximal operator prox χ C ρ (y l /ρ) in the step 3 reduces to the orthogonal projection P C (y l /ρ) of y l /ρ on C. For certain cases of C, for example, box and ball, Algorithm SDCA is greatly less expensive than other algorithms, because the orthogonal projection on C in these cases is given in explicit form (see [START_REF] Thi | New and efficient DCA based algorithms for minimum sum-of-squares clustering[END_REF]; Pham Dinh and Le Thi (1998); [START_REF] Thi | A branch and bound method via d.c. optimization algorithms and ellipsoidal technique for box constrained nonconvex quadratic problems[END_REF]).

ii) In many application problems with sparsity-inducing norms, we will have g 1 (x) = λ x 1 . Hence, the proximal operator prox

χ C +λ . 1 ρ
(y l /ρ) can be computed by an inexpensive algorithm. Especially, if C = R n , x l+1 has a closed form:

x l+1 = S y l /ρ, λ/ρ , (1.10)
where S is a soft-thresholding operator.

iii) From Prop. 1 in Le [START_REF] Thi | New and efficient DCA based algorithms for minimum sum-of-squares clustering[END_REF], we obtain that ρ 2 x 2g 2 (x) is convex if ρ ≥ max{0, λ max (H g 2 (x))} for all x ∈ C, where λ max (H g 2 (x)) denotes the largest eigenvalue of the Hessian matrix H g 2 (x) of g 2 at x. iv) In practice, when g 2 is differentiable and the computation of its gradient is not difficult, and the proximal operator prox χ C +g 1 ρ (y l /ρ) can be inexpensively determined, the use of SDCA is highly recommended.

Part I Variable Selection and Classification

Introduction

The problem of classifying observations into Q classes (Q ≥ 2) has drawn considerable attention from researchers in machine learning, as it has been applied in many fields such as information retrieval or face recognition. In the literature, several notions have been used to formalize this classification problem. Let X be an n × p data matrix with observations x i (i = 1, ..., n) on the rows and features on the columns. Denote n i the number of observations in the class C i . We assume that the features have been standardized to have mean 0 and variance 1. To obtain an optimal classification rule, we need to know the class posterior probabilities Pr(k|x). We suppose that f k (x) is the classconditional density in the class k, and let π k be the prior probability of the class k, with Q k=1 π k = 1. Linear discriminant analysis (LDA) performs classification by assuming that the data within each class are normal distributed.

f k (x) = 1 (2π) p/2 |Σ w | 1/2 exp - 1 2 (x -µ k ) T Σ -1 w (x -µ k ) , (2.1)
where µ k is the mean vector of the k-th class and Σ w is the common within-class covariance matrix. In practice µ k , π k and Σ w are unknown, but they can be estimated from the training data by

µ k = 1/n k x i ∈C k x i , π k = n k /n and Σ w = 1 n Q k=1 x i ∈C k (x i -µ k )(x i -µ k ) T . (2.2)
Various approaches have been presented for solving the classification problem. Among these methods, Linear Discriminant Analysis (LDA) which was first introduced in Fisher ( 1936) is regarded as one of the most popular methods because of its advantage of dimension reduction. The primary purpose of LDA is to find a linear transformation that best discriminates between classes. The classification is then performed in the transformed space using some distance metrics. There are three different approaches to tackle LDA, which are based on solving the normal model, the Fisher's discriminant problem and the optimal scoring problem, respectively [START_REF] Hastie | Penalized discriminant analysis[END_REF][START_REF] Mardia | Multivariate Analysis[END_REF][START_REF] Hastie | The Elements of Statistical Learning[END_REF].

For the first approach, the LDA classification rule is obtained by using Bayes's rule to estimate the most likely class for a new observation, i.e., the predicted class for a new observation x is arg max

k P r(k|x), (2.3)
where a simple application of the Bayes's theorem gives us .4) In comparing two classes k and l, it is sufficient to look at the log-ratio:

P r(k|x) = f k (x)π k Q l=1 f l (x)π l . ( 2
ln P r(k|x) P r(l|x) = ln f k (x) f l (x) + ln π k π l = x T Σ -1 w µ k - 1 2 µ T k Σ -1 w µ k + ln π k -x T Σ -1 w µ l - 1 2 µ T l Σ -1 w µ l + ln π l . (2.5)
Therefore, the decision rule (2.3) is equivalent to

arg max k x T Σ -1 w µ k - 1 2 µ T k Σ -1 w µ k + ln n k . (2.6)
With the second approach, the main purpose of this method is seeking a low dimensional projection of the observations such that the between-class variance is large relative to the within-class variance, i.e. we seek discriminant vectors w 1 , ..., w Q-1 that successively minimize

min w k ∈R p -w T k Σ b w k : w T k Σ w w k = 1; w T k Σ w w l = 0, l = 1, .., k -1 , (2.7)
where

Σ b = 1 n Q k=1 n k µ k µ T
k is a standard estimate of the between-class covariance matrix. The problem (2.7) is a generalized eigen problem which has at most Q -1 non trivial solutions, since Σ b has rank at most Q -1, and hence at most Q -1 discriminant vectors. A classification rule is obtained by computing Xw 1 , ..., Xw Q-1 and assigning each observation to its nearest centroid in this transformation space. We can use only the first K ≤ Q -1 discriminant vectors in order to perform reduced rank classification.

For the last one, the rationality of this method derives from the fact that LDA can also be re-formulated as a regression problem via optimal scoring. This approach was discussed in detail by [START_REF] Hastie | Penalized discriminant analysis[END_REF]. Let Y ∈ R n×Q with Y ik = 1 if x i ∈ C k and 0 otherwise. To find K discriminant vectors w 1 , ..., w K , the optimal scoring criterion successively solves the problem min

w k ,θ k ||Y θ k -Xw k || 2 2 subject to 1 n θ T k Y T Y θ k = 1; θ T k Y T Y θ l = 0, l = 1, ..., k -1, (2.8)
where θ k is a Q-vector of scores.

In this chapter, we are interested in the Fisher's discriminant problem. To solve the generalized eigen problem (2.7), it typically requires the within-class covariance matrix Σ w to be nonsingular. However, this requirement is difficult to satisfy when the dimensionality is high, because the matrix Σ w is likely to be singular. In fact, in many applications such as information retrieval, face recognition and microarray analysis, we often encounter problems having a small number of observations but a very large number of features. In such cases, the classical LDA includes two great challenges. The first challenge is the singularity of the within-class covariance matrix of the features and the second one is the difficulty in interpreting the classification rule.

Numerous methods have been proposed to overcome the first challenge (see e.g. [START_REF] Hastie | Penalized discriminant analysis[END_REF][START_REF] Bickel | Some theory for fisher's linear discriminant function, naive bayes, and some alternatives when there are many more variables than observations[END_REF][START_REF] Krzanowski | Discriminant analysis with singular covariance matrices: methods and applications to spectroscopic data[END_REF][START_REF] Xu | Modified linear discriminant analysis approaches for classification of high-dimensional microarray data[END_REF]). These approaches use positive definite estimates of the within-class covariance matrix to deal with the singularity issue. Thus the problem (2.7) becomes

min w k ∈R p -w T k Σ b w k : w T k Σw w k = 1; w T k Σw w l = 0, l = 1, ..., k -1 , (2.9)
where Σw is a positive definite estimate for the within-class covariance matrix. The problem (2.9) is equivalent to the following problem (see [START_REF] Witten | Penalized classification using Fisher's linear discriminant[END_REF])

min w k ∈R p -w T k Σ k b w k : w T k Σw w k ≤ 1 , (2.10) where Σ k b = 1 n X T Y (Y T Y ) -1 2 P ⊥ k (Y T Y ) -1 2 Y T X. (2.11)
Here Y ∈ R n×Q with Y ij = 1 if i ∈ C j and 0 otherwise, P ⊥ 1 = I p (identity matrix), and

P ⊥ k (k > 1
) is an orthogonal projection matrix into the orthogonal space of the space generated by {(Y T Y ) -1 2 Y T Xw l : l = 1, ..., k -1}.

For the second challenge, the most suitable approach is feature selection. A sparse classifier leads to easier model interpretation and may reduce overfitting of the training data.

In the literature, there exist a number of works to extend LDA to the high-dimensional setting in such a way that the resulting classifier involves a sparse linear combination of the features. We here refer to the notable approaches. One of them is based on soft-thresholding in order to obtain a sparse classifier (see e.g. [START_REF] Tibshirani | Diagnosis of multiple cancer types by shrunken centroids of gene expression[END_REF][START_REF] Tibshirani | Class prediction by nearest shrunken centroids, with applications to DNA microarrays[END_REF]; [START_REF] Guo | Regularized linear discriminant analysis and its application in microarrays[END_REF]; [START_REF] Shao | Sparse linear discriminant analysis by thresholding for high dimensional data[END_REF]). Several authors use the ℓ 1 -norm to deal with sparsity. More precisely, the ℓ 1 -regularization is added to the objective function of the optimal scoring problem (2.8) (see e.g. [START_REF] Grosenick | Interpretable classifers for fmri improve prediction of purchases[END_REF][START_REF] Leng | Sparse optimal scoring for multiclass cancer diagnosis and biomarker detection using microarray data[END_REF][START_REF] Clemmensen | Sparse discriminant analysis[END_REF], and/or the Fisher's discriminant problem (2.10) [START_REF] Trendafilov | Dalass: Variable selection in discriminant analysis via the lasso[END_REF][START_REF] Wu | Sparse linear discriminant analysis for simultaneous testing for the significance of a gene set/pathway and gene selection[END_REF][START_REF] Witten | Penalized classification using Fisher's linear discriminant[END_REF]. In particular, [START_REF] Witten | Penalized classification using Fisher's linear discriminant[END_REF] proposed a biconvex formulation closely related to the sparse principal components analysis proposal of [START_REF] Witten | A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis[END_REF]. [START_REF] Mai | A direct approach to sparse discriminant analysis in ultra-high dimensions[END_REF]; Cai and Liu (2011) developed direct approaches for sparse discriminant analysis. [START_REF] Mai | A note on the connection and equivalence of three sparse linear discriminant analysis methods[END_REF] showed the connection between and the equivalence of three sparse discriminant analysis methods proposed in [START_REF] Wu | Sparse linear discriminant analysis for simultaneous testing for the significance of a gene set/pathway and gene selection[END_REF], [START_REF] Clemmensen | Sparse discriminant analysis[END_REF] and [START_REF] Mai | A direct approach to sparse discriminant analysis in ultra-high dimensions[END_REF].

A natural way to deal with feature selection in machine learning is using the ℓ 0 -norm in the regularization term for the problem (2.10). As a result, we propose the sparse Fisher linear discriminant (SFLD) problem

min w k ∈R p -w T k Σ k b w k + λ k ||w k || 0 : w T k Σw w k ≤ 1 , (2.12)
where ||w k || 0 denotes the ℓ 0 -norm of w k , i.e. the number of non-zero elements of vector w k , and λ k is a nonnegative tuning parameter.

Solving (2.12) is a formidable challenge since it is nonconvex, discontinuous and NP-hard. Optimization methods involving the ℓ 0 -norm can be divided into three categories according to the way to treat the ℓ 0 -norm: convex approximation, nonconvex approximation and nonconvex exact reformulation. We refer to Le [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF] for an excellent review on exact/approximation approaches to deal with the ℓ 0 -norm. The best known and widely used convex approximation of ℓ 0 -norm is ℓ 1 -norm called Lasso [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF]. For the problem (2.12), [START_REF] Witten | Penalized classification using Fisher's linear discriminant[END_REF] replaced the ℓ 0 -norm with the ℓ 1 -norm and applied the minorization-maximization (MM) approach for solving the resulting problem. This algorithm is in fact a version of difference of convex functions algorithm (DCA). DC approximation approaches for the ℓ 0 -norm have been studied extensively on both theoretical and practical aspects for the problem of feature selection in SVM (see e.g. [START_REF] Thi | A DC programming approach for feature selection in support vector machines learning[END_REF][START_REF] Sentana | The econometrics of mean-variance efficiency tests: a survey[END_REF][START_REF] Mairal | Incremental majorization-minimization optimization with application to large-scale machine learning[END_REF][START_REF] Collobert | Trading convexity for scalability[END_REF][START_REF] Neumann | Combined svm-based feature selection and classification[END_REF][START_REF] Ong | Learning sparse classifers with difference of convex cunctions algorithms[END_REF])), and linear regression (see e.g [START_REF] Chen | Lower bound theory of nonzero entries in solutions of l2-lp minimization[END_REF][START_REF] Gasso | Recovering sparse signals with a certain family of nonconvex penalties and DC programming[END_REF]). These works add the ℓ 0 -norm to a convex function and they only have a difficulty in treating the ℓ 0 -norm. In this chapter, solving (2.12) includes double difficulties. The first is how to treat the ℓ 0 -norm and the second is caused by the non-convexity of the original Fisher's discriminant problem. To tackle the ℓ 0 -norm we investigate DC approximation approaches. Among several existing sparse inducing functions we are using the piecewise linear function (called Capped-ℓ 1 ) and the piecewise exponential concave function introduced respectively in [START_REF] Peleg | A bilinear formulation for vector sparsity optimization[END_REF] and [START_REF] Bradley | Feature selection via concave minimization and support vector machines[END_REF]. This choice is motivated by the fact that the Capped-ℓ 1 has been proved theoretically in Le [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF] to be the tightest approximation while the piecewise exponential function has been showed to be efficient via the numerical results in numerous works (see e.g. [START_REF] Bradley | Feature selection via concave minimization and support vector machines[END_REF][START_REF] Thi | A DC programming approach for feature selection in support vector machines learning[END_REF][START_REF] Mairal | Incremental majorization-minimization optimization with application to large-scale machine learning[END_REF][START_REF] Thi | Feature selection for linear svms under uncertain data: robust optimization based on difference of convex functions algorithms[END_REF][START_REF] Ong | Learning sparse classifers with difference of convex cunctions algorithms[END_REF]). The resulting problems are formulated as DC programs, and then DCA are applied. We propose two DCA schemes for two different formulations of a common model to both the approximation functions.

The rest of chapter is organized as follows. In Section 2.2, we illustrate how to apply DCA to solve the problem (2.12). The numerical experiments are reported in Section 2.3. Finally, the conclusions are given in Section 2.4.

We are now going to present solution methods based on DC programming and DCA for solving the SFLD problem (2.12).

2.2 Solution methods via DC programming and DCA

DC approximations of ℓ 0 -norm

In this chapter, we consider two DC approximations of the ℓ 0 -norm. For an α > 0, let η α,1 and η α,2 be the functions given by

η α,1 (x) = 1 -e -α|x| , ∀x ∈ R, (2.13)
and

η α,2 (x) = min{1, α|x|}, ∀x ∈ R, (2.14)
respectively. Their graphs are illustrated in Figure 2.1.

The first DC approximation of the ℓ 0 -norm called the piecewise exponential concave function [START_REF] Bradley | Feature selection via concave minimization and support vector machines[END_REF] is defined by The second DC approximation of the ℓ 0 -norm is the piecewise linear approximation proposed in [START_REF] Peleg | A bilinear formulation for vector sparsity optimization[END_REF]. It is described as follows.

||w k || 0 ≈ p i=1 η α,1 (w ki ). ( 2
||w k || 0 ≈ p i=1 η α,2 (w ki ).
(2.16)

For simplification, we use the common notation η α to design both η α,1 and η α,2 . Then, the approximations (2.15) and (2.16) are rewritten as follows.

||w k || 0 ≈ p i=1 η α (w ki ).
(2.17)

Using the approximation (2.17), we can reformulate the SFLD problem (2.12) in the form min

F (w k ) = -w T k Σ k b w k + λ k p i=1 η α (w ki ) : w k ∈ Ω , (2.18)
where

Ω = w k ∈ R p : w T k Σw w k ≤ 1 .
Note that η α (w ki ) = η α (|w ki |) ∀w ki ∈ R and η α is increasing concave over [0, +∞], hence we get another equivalent form of (2.18)

min F (w k , z) = -w T k Σ k b w k + λ k p i=1 η α (z i ) : (w k , z) ∈ Ω 1 , (2.19)
where

Ω 1 = {(w k , z) : w k ∈ Ω, |w ki | ≤ z i ∀i = 1, ..., p}.
2.2.2 DCA for solving (2.18) The approximation η α can be presented as a DC function [START_REF] Thi | A DC programming approach for feature selection in support vector machines learning[END_REF][START_REF] Ong | Learning sparse classifers with difference of convex cunctions algorithms[END_REF]) 2.20) where g(x) = α|x|, h(x) = -1 + α|x| + e -α|x| if η α = η α,1 , and h(x) = -1 + max{1, α|x|} if η α = η α,2 . Note that the first DC decomposition g(x) is the same for the both η α,1 and η α,2 .

η α (x) = g(x) -h(x), (
Then, the problem (2.18) can be rewritten as follows.

min

{F (w k ) = G 1 (w k ) -H 1 (w k ) : w k ∈ R p } . (2.21)
where

G 1 (w k ) := χ Ω (w k ) + λ k p i=1
g(w ki ), (2.22) and

H 1 (w k ) := w T k Σ k b w k + λ k p i=1 h(w ki ) (2.23)
are clearly convex functions. According to the generic DCA scheme, at each iteration l, we have to compute a subgradient v l ∈ ∂H 1 (w l k ) and then solve the convex program of the form (P l ), namely min{G

1 (w k ) -v l , w k : w k ∈ R p }, (2.24)
For k = 1, , ..., K, DCA for solving (2.21) can be described as below.

DCA1 (DCA for solving (2.21))

Initialization: Let τ be a tolerance sufficient small, set l = 0 and choose

w 0 k ∈ Ω. repeat 1. Compute v l ∈ ∂H 1 (w l k ) 2. Solve the following convex problem to obtain w l+1 k min λ k α||w k || 1 -v l , w k : w T k Σw w k ≤ 1 . (2.25) 3. l ← l + 1. until ||w l k -w l-1 k || 2 ≤ τ ||w l-1 k || 2 + 1 or |F (w l k ) -F (w l-1 k )| ≤ τ |F (w l-1 k )| + 1 .
The implementation of DCA1 requires the computation of v l ∈ ∂H 1 (w l k ) in step 1, which depends on η α . More precisely, when η α = η α,1 , h is differentiable, so is h. Thus v l is computed by

v l i = 2 Σ k bi , w l k + λ k α(1 -e -αw l ki ) if w l ki ≥ 0 2 Σ k bi , w l k -λ k α(1 -e αw l ki ) if w l ki < 0 i = 1, ..., p, (2.26)
where Σ k bi stands for the ith row of Σ k b . When η α = η α,2 , v l is calculated as follows.

v l i = 2 Σ k bi , w l k + sgn(w ki )λ k α if α|w l ki | ≥ 1 2 Σ k bi , w l k otherwise i = 1, ..., p, (2.27)
where sgn(w ki ) is the sign of w ki , i.e., sgn(w ki ) = -1 if w ki < 0, 1 if w ki > 0, and 0 otherwise.

Remark 2.1

1. For solving the convex problem (2.25), we first solve the following convex problem.

d = arg min d∈R p d T Σw d + λ k α||d|| 1 -v l , d .
(2.28)

Then, the solution to (2.25) is ŵk = 0 if d = 0 and w k = d/ dT Σw d otherwise (see [START_REF] Witten | Penalized classification using Fisher's linear discriminant[END_REF]).

2. The problem (2.28) can be solved by using DCA (see [START_REF] Thi | An efficient algorithm for globally minimizing a quadratic function under convex quadratic constraints[END_REF]) with special DC components of its objective function as follows:

G(d) = µ 2 d 2 2 + λ k α d 1 -v l , d , H(d) = µ 2 d 2 2 -d T Σw d,
are convex functions when µ is larger than or equal to the largest eigenvalue of Σw .

At each iteration of this DCA, we can explicitly compute the solution to its convex subproblem by a soft-thresholding. We also note that the coordinate descent approach (see [START_REF] Friedman | Pathwise coordinate optimization[END_REF]) and the alternating direction method of multipliers (see [START_REF] Boyd | Distributed optimization and statistical learning via the alternating direction method of multipliers[END_REF]) can deal with this problem.

3. In fact, when the diagonal estimate of the within-class covarince matrix is used, the solution to the problem (2.28) is explicitly computed.

4. We observe that the problem (2.25) has ℓ 1 -perturbed form, hence this DCA scheme can be seen as ℓ 1 -perturbed algorithm (see [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF]).

, z) = G 2 (w k , z) -H 2 (w k , z) , (2.29) where G 2 (w k , z) := χ Ω 1 (w k , z), (2.30)
and

H 2 (w k , z) := w T k Σ k b w k + λ k p i=1 (-η α )(z i ).
(2.31)

DCA applied to (2.29) consist of first calculating a subgradient .32) and then solving the following convex program at the each iteration.

(v l , zl ) ∈ ∂H 2 (w l k , z l ), ( 2
(w l+1 k , z l+1 ) ∈ arg min G 2 (w k , z) -v l , w k -zl , z (2.33) When η α = η α,1 , zl is calculated by zl i = -λ k α exp(-αz l i ) ∀i = 1, ..., p, (2.34)
and when η α = η α,2 , we have

zl i = -λ k α if z l i ≤ 1/α,
w k ∈Ω -v l , w k - p i=1 zl i |w ki | z l+1 i = |w l+1 ki | ∀i.
(2.36)

Hence DCA applied on (2.29) is given by the algorithm below.

DCA2 (DCA for solving (2.29))

Initialization: Let τ be a tolerance sufficient small, set l = 0 and choose (w

0 k , z 0 ) ∈ Ω 1 . repeat 1. Compute v l = 2Σ k b w l k and zl i ∈ λ k ∂(-η α )(z l i )
min -v l , w k - p i=1 zl i |w ki | : w T k Σw w k ≤ 1 (2.37) 3. Compute z l+1 i = |w l+1 ki | ∀i = 1, ..., p. 4. l ← l + 1. until ||w l k -w l-1 k || 2 ≤ τ ||w l-1 k || 2 + 1 or | F (w l k , z l ) -F (w l-1 k , z l-1 )| ≤ τ | F (w l-1 k , z l-1 )| + 1 .
Remark 2.2 The problem (2.37) has ℓ 1 -reweighted form, hence this DCA scheme can be regarded as ℓ 1 -reweighted algorithm.

Theorem 2.2 (Convergence properties of DCA2) (i) DCA2 generates the sequence {(w l k , z l )} in Ω 1 such that { F (w l k , z l )} is decreasing. (ii) Every limit point (w * k , z * ) of the sequence {(w l k , z l )} is a critical point of the problem (2.

29)

Proof : The Theorem is direct consequences of convergence properties of general DC programs. ✷

Numerical experiments

We use the SFLD problem for supervised classification problems in high dimension. The SFLD problem transforms the set of labeled data points in the original space into a labeled set in a lower-dimensional space and selects relevant features. The classification rule is obtained by computing Xw 1 , ..., Xw s and assigning each observation to its nearest centroid in this transformation space, i.e. the predicted class for a test observation x is arg min

k ||x T W -µ T k W || 2 2 -2 ln(n k ), (2.38)
where the linear transformation W = [w 1 , ..., w K ] is computed by DCA1 or DCA2 and the second term is an adjustment term for unequal class sizes.

Comparative algorithms

We denote by DCA1-PiE and DCA1-Capped-ℓ 1 the DCA1 with η α = η α,1 and η α = η α,2 , respectively. The DCA2 with η α = η α,1 and η α = η α,2 are denoted by DCA2-PiE and DCA2-Capped-ℓ 1 , respectively. We will compare our proposed Algorithms with the methods proposed in [START_REF] Witten | Penalized classification using Fisher's linear discriminant[END_REF] (PLDA) and Guo et al. (2007) (RDA). We also compare with the method proposed in [START_REF] Mai | A direct approach to sparse discriminant analysis in ultra-high dimensions[END_REF] (DSDA) for the binary classification problems.

2.3.1.1 Penalized linear discriminant analysis (PLDA)

PLDA used the ℓ 1 -norm instead of the ℓ 0 -norm in the problem (2.12), that is

max w k ∈R p {w T k Σ k b w k -λ k ||w k || 1 : w T k Σw w k ≤ 1}, (2.39)
The problem (2.39) is nonconvex. [START_REF] Witten | Penalized classification using Fisher's linear discriminant[END_REF] used the MM approach for finding a local of this problem. In the experiments, the authors used the diagonal estimate diag(σ2 1 , ..., σ 2 p ) for Σw , where σ i is the within-class standard deviation for feature i. The R package penalizedLDA is available from CRAN 2 .

Shrunken centroids regularized discriminant analysis (RDA)

RDA is based on the same underlying model as LDA (see [START_REF] Guo | Regularized linear discriminant analysis and its application in microarrays[END_REF]) and it regularizes the within-class covariance matrix used by LDA (2.40) where 0 ≤ α ≤ 1. In order to perform feature selection, one can perform soft-thresholding of the quantity Σ-1

Σw = αΣ w + (1 -α)I p ,
w µ k . That is, we compute sgn( Σ-1 w µ k )(| Σ-1 w µ k | -δ) + , (2.41)
where δ is a nonnegative tuning parameter. The R package rda is available from CRAN.

(β,β 0 ) n i=1 (y i -β 0 -Xβ) 2 + λ β 1 , (2.42)
where λ is a tuning parameter. Then the classification rule is to assign

x to class 2 if [x -(μ 1 + μ2 )] T β + βT Σ β (μ 2 -μ1 ) T β -1 ln(n 2 /n 1 ) > 0,
where μ1 and μ2 are the sample mean vectors of classes 1 and 2, respectively, and Σ is the pooled sample covariance matrix.

Datasets

We evaluate the performance of DCA1-PiE, DCA1-Capped-ℓ 1 , DCA2-PiE and DCA2-Capped-ℓ 1 on three synthetic datasets and a collection of real world datasets. The description of three synthetic datasets are summarized in Table 2.1 and they are generated as follows (see [START_REF] Witten | Penalized classification using Fisher's linear discriminant[END_REF]):

For the first setup S1, we generate a four classes classification problem [START_REF] Alon | Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays[END_REF] and available at http://genomics-pubs. princeton.edu/oncology/.

SRBCT [START_REF] Khan | Classification and diagnostic prediction of cancers using expression profiling and artificial neural networks[END_REF] is the dataset of small, round blue cell tumors of childhood and can be downloaded at http://research.nhgri.nih.gov/microarray/Supplement/.

The training and test set consist of 83 samples spanning four classes.

The high-dimensional dataset consisting of multi-spectral imaging of three penicillium species: melanoconodium, polonicum and venetum. This data is studied in [START_REF] Clemmensen | A method for comparison of growth media in objective identification of penicillium based on multi-spectral imaging[END_REF].

Lung Cancer dataset, there are 181 tissue samples (31 MPM and 150 ADCA). Each sample is described by 12533 genes. It is used in [START_REF] Gordon | Translation of microarray data into clinically relevant cancer diagnostic tests using gege expression ratios in lung cancer and mesothelioma[END_REF] and can be downloaded at http://datam.i2r.a-star.edu.sg/datasets/krbd/.

Leukemia microarray dataset is published in [START_REF] Yeoh | Classification, subtype discovery, and prediction of outcome in pediatric lymphoblastic leukemia by gene expression profiling[END_REF] and available at http:// datam.i2r.a-star.edu.sg/datasets/krbd/. The data consisted of 12558 gene expression measurements for 248 samples belong to six cancer classes.

Nakayama data consisting of 86 samples from 5 types of soft tissue tumors, each with 22283 gene expression measurements [START_REF] Nakayama | Gene expression analysis of soft tissue sarcomas: characterization and reclassification of malignant fbrous histiocytoma[END_REF]. This data is available at Gene Expression Omnibus.

Sun data consisting of 180 samples and 54613 expression measurements (see [START_REF] Sun | Neuronal and glioma-derived stem cell factor induces angiogenesis within the brain[END_REF]). The samples are belong to four classes. It is available at Gene Expression Omnibus.

MNIST dataset is available at http://yann.lecun.com/exdb/mnist/. The training and test sets consist of 60000 and 10000 images of size 28 × 28 pixels, respectively.

Experimental setups

All algorithms are implemented in the R 3.0.2, and performed on a PC Intel i7 CPU3770, 3.40 GHz of 8GB RAM.

We use the same diagonal estimate Σw as PLDA. Its advantage for the data in which the number of features is much larger than the number of observations has been indicated in [START_REF] Bickel | Some theory for fisher's linear discriminant function, naive bayes, and some alternatives when there are many more variables than observations[END_REF]. Hence the solution d to the problem (2.28) is explicitly defined (2.43) where S is the soft-thresholding operator defined as

by di = 1 2σ 2 i S(v l i , λ k α) ∀i = 1, ..., d,
S(z, t) =      z -t if z > 0 and t < |z|, z + t if z < 0 and t < |z|, 0 if t ≥ |z|.
(2.44)

The value of λ k is taken as

λ k = λ.λ k 1 , where λ k 1 is the largest eigenvalue of Σ k b .
This can avoid penalizing each discriminant vector more than the previous discriminant vectors, since the objective value of problem (2.12) without the zero-norm is equal to the largest eigenvalue of Σ k b when the diagonal estimate of the within-class covariance matrix is used. From which it follows that an eigenvector of the matrix Σ k b corresponding to the largest eigenvalue is also a good starting point w 0 k of DCA, which seems to be natural (see [START_REF] Witten | Penalized classification using Fisher's linear discriminant[END_REF]).

The values of parameters λ and K (the number of used discriminant vectors) are chosen through a sixfold cross-validation procedure on training set from a set of candidates. The approximation parameter in (2.17) is fixed α = 5 as suggested in [START_REF] Bradley | Feature selection via concave minimization and support vector machines[END_REF]. Concerning the parameter α, from the theoretical point of view, the larger α is, the better approximation of the ℓ 0 -norm is. However, when we tried with larger α (up to 100), the result is not improved. The stop tolerance of DCA is τ = 10 -6 . We select relevant features as follows: feature i is deleted if |w ki | < 10 -6 for all k = 1, ..., K.

Numerical results on synthetic data

In this experiment, we generate training, tuning, and test sets in the same manner as described in Sect. 2.3.2. The tuning sets are used to choose the parameters λ and the number of discriminant vectors used K, while the test sets are used to measure the accuracy of various classifiers trained on the training sets. We perform 10 trials for each experimental setting. DSDA is only tested on the 2-class synthetic dataset S2.

The experimental results on synthetic data are given in Table 2.3. In this table, the average of percentage of selected features and its standard deviation (FS), the average of percentage of accuracy of classifiers and its standard deviation (ACC), the average of CPU time in second and its standard deviation (CPU) over 10 trials, as well as the number of discriminant vector used (K) are reported.

We observe from Table 2.3, in terms of feature selection, the DCA based algorithms give better results than PLDA, RDA and DSDA. DCA1-Capped-ℓ 1 gives the best results on 2/3 synthetic datasets. On average of 3 synthetic datasets, DCA1-PiE, DCA1-Capped-ℓ 1 , DCA2-PiE and DCA2-Capped-ℓ 1 respectively select 41.6%, 41.17%, 35.86% and 42.48% of features while PLDA and RDA respectively select 46.93% and 49.45% of features.

K S1 3 3 3 3 3 - - S2 1 1 1 1 1 - 1 S3 1 1 1 1 1 - -
The DCA based algorithms not only provide a good performance in terms of feature selection, but also give a high accuracy of classifiers. DCA1-PiE gives the best accuracy of classifiers on all the three synthetic datasets. On average of 3 synthetic datasets, DCA1-PiE, DCA1-Capped-ℓ 1 , DCA2-PiE, DCA2-Capped-ℓ 1 , PLDA and RDA respectively give the accuracy of classifiers 94.34%, 94.02%, 94.21%, 94.07%, 94.05% and 86.66%. On the second synthetic dataset, the DCA based algorithms are also better than DSDA in terms of accuracy of classifiers as well as feature selection.

In Table 2.3, we also see that DCA1-Capped-ℓ 1 and PLDA are fastest on all the three synthetic datasets.

Numerical results on real datasets

For the experiments on the first nine real datasets, we use the cross-validation scheme to validate the performance of various classifiers. Each real dataset is split into a training set containing 2/3 of the samples and a test set containing 1/3 of the samples. This process is repeated 10 times, each with a random choice of training set and test set. The parameter λ and the number of discriminant vectors K which is used are chosen via 6-fold cross-validation.

The computational results given by DCA1-PiE, DCA1-Capped-ℓ 1 , DCA2-PiE, DCA2-Capped-ℓ 1 , PLDA, RDA and DSDA are reported in Tables 2.4-2.5. We are interested in the efficiency (the sparsity and the accuracy of classifiers) as well as the rapidity of the algorithms. We notice that DSDA is only tested on the 2-class datasets.

We observe from computational results that:

Sparsity: On all the datasets, the classifiers obtained by the DCA based algorithms are sparser than those obtained by PLDA, RDA and DSDA. DCA1-Capped-ℓ 1 is the best on 5 out of 9 datasets and DCA1-PiE is the best on 2 out of 9 datasets. The DCA based algorithms select from 0, 09% to 69.74% of features while PLDA and RDA choose from 15.36% to 100% of features. Overall, DCA1-Capped-ℓ 1 realizes a better trade-off between accuracy and sparsity than other algorithms. It suppresses considerably the number of features (up to 99.9%) while the correctness of classification is quite good (from 72.33% to 100%).

Accuracy of classifiers:

In terms of the accuracy of classifiers, the DCA based algorithms attain better than PLDA, RDA and DSDA on 6/9 datasets (the gains vary from 0.05 to 11.45%). More specifically, DCA1-Capped-ℓ 1 is the best on 3/9 datasets, especially for the very large SUN data (54613 features), this approach only selects 9.02% of features but achieves the best accuracy of classifiers (72.33%). RDA is slightly better than the DCA based algorithms on the dataset NAK (the gains is 0.62%). However, RDA selects much more features than these approaches (the gain is 42.38% of features). On the two datasets ADV and GIS which the number of observations are larger than the number of features, RDA and DSDA are better than the DCA based algorithms. This can be explained that these approaches use the sample covariance matrix which can be appropriate for these datasets. To sum up, when the number of features are much larger than the number of observations, our proposed methods are especially more efficient than other algorithms in classification of high dimensional data. Thus, the proposed methods are highly recommended for this type of data.

Training time and discriminant vectors:

Training time and the numbers of discriminant vectors which are used are reported in Table 2.5. The DCA based algorithms and PLDA run very fast and they are comparable. RDA and DSDA are much slower than the DCA based algorithms (the ratios of gains are from 1.7 to 39660 times). The discriminant vectors can be used to visualize the datasets such as in The computational results of the MNIST dataset are reported on Table 2.6. Notably in this dataset, RDA is not able to perform since the amount of RAM is insufficient. From Table 2.6, we observe that the DCA based algorithms outperform PLDA in terms of sparsity as well as accuracy of classifiers. As for the training time, all five algorithms run very fast (less than 0.2 s).

Conclusion

We have proposed efficient approaches for solving the Sparse Fisher Linear Discriminant problem using the ℓ 0 -regularization. Among several sparse inducing functions of the ℓ 0 -norm, we have used two appropriate approximations functions and reformulated the resulting problems as DC programs. Overall, four DCA based algorithms have been developed in order to exploit the nice effect of DC decompositions/DC formulations on the one part, and the efficiency of the approximate functions on the other part. The robustness and the effectiveness of our DCA based algorithms have been demonstrated through the computational results on both the simulated and real datasets. Their efficiency has been compared with three standard algorithms which use the ℓ 1 -regularization (note again that this is the first work investigating ℓ 0 -regularization for the SFLD problem).

The research developed in this chapter permitted us to highlight the following comments/recommendations:

1. ℓ 1 -regularization versus ℓ 0 -regularization: similar to several works using ℓ 0regularization in learning with sparsity; once again, our work proved that ℓ 0regularization produces much better sparsity than ℓ 1 -regularization.

2. ℓ 1 -LDA (or more generally convex-LDA) versus ℓ 0 -LDA: unlike the convex regularization approach for several learning problems (for instance, the feature selection in SVM/linear regression, etc.) where convex regularizations result in convex optimization problems (which are so far easy to solve) and the convex-LDA problem is still nonconvex and then difficult. Hence, the quality of solutions (the sparsity and the accuracy of classifiers) depends on the efficiency of algorithms being investigated for these nonconvex programs. The same argument (both convex-LDA and ℓ 0 -LDA are nonconvex) does not necessarily imply that ℓ 0 -LDA algorithms are more time-consuming than the convex-LDA algorithms. Therefore, we recommend to use ℓ 0 -LDA in high-dimensional data classification, not only when sparsity is significantly desired, but also when high accuracy is requested.

3. PLDA, RDA and DSDA versus DCA: in our numerical experiments the DCA based algorithms are most of the time better than PLDA, RDA and DSDA. This superiority comes mainly from the arguments mentioned in ( 1) and ( 2) above. In another hand, the estimation of the within-class covariance matrix used in each method ( Σw ) influences also on its efficiency. A more detailed comparative analysis can be summarized as follows:

• PLDA versus DCA: these methods aim to solve the Fisher's discriminant problem but with two different regularizations to deal with sparsity: ℓ 1 (PLDA) and ℓ 0 (DCA). They use the same diagonal estimate matrix of the within-class covariance matrix Σw in the model, and their iterations are based on the same idea (the MM method used in PLDA is a special version of the general DCA scheme). Thanks to ℓ 0 -regularization, DCA always produce better sparsity than PLDA. By the nice effect of DC decompositions/DC formulations of the resulting nonconvex approximate problems, DCA give higher classification accuracy than PLDA. The training time of PLDA and DCA are comparable, but thanks again to the effect of DC decompositions, it is quite possible that DCA are faster than PLDA. Note also that, when the number of features is much larger than the number of observations, the diagonal estimate matrix Σw used in our DCA (and in PDLA) is good, and therefore, DCA produce high classification accuracy. Moreover, with this diagonal estimate matrix Σw , DCA are explicitly computed at each iteration, and by the way, they are very fast. Hence, the use of DCA for this type of data is highly recommended. • RDA versus DCA: RDA does not solve the Fisher's discriminant problem but uses the classification rule (2.6) which requires the matrix inversion (compute directly Σ-1 w ). Such a procedure is very time-consuming when the size of Σw (which is the number of features) is large. That is why DCA are always much faster than RDA, and the gain is more important in high-dimensional datasets. Note, however, that when the features are independent or when the number of sample is greater than the number of features, the estimate diagonal matrix used in DCA may not be appropriate, and then, it could happen that RDA produces better sparsity (e.g., the synthetic dataset S1 having independent features) or RDA gives better classification accuracy (e.g., the datasets ADV and GIS where the number of samples is greater than the number of features). For such types of data, the regularized matrix Σw defined in (2.40) by RDA is more appropriate. For other types of data, DCA considerably outperform RDA on both sparsity and classification accuracy.

• DSDA versus DCA: for binary classification DSDA requires solving the problem (2.42), while DCA's iterations are explicitly defined in a very simple formulation.

That is why DCA are faster than DSDA. By the same argument mentioned above concerning the efficiency of the estimate diagonal matrix used in DCA for datasets having the number of samples greater than the number of features, it could happen that DSDA gives higher classification accuracy than DCA (e.g., for the datasets ADV and GIS). As for sparsity, DCA are always better than DSDA which uses the ℓ 1 -regularization.

4. About the four versions of DCA for the SFLD problems: DCA1-Capped-ℓ 1 is the best, most of the time, on both sparsity and classification accuracy, and it always realizes the best trade-off between sparsity and classification accuracy. This confirms once again the results developed in Le [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF]: The Capped-ℓ 1 is the best nonconvex (DC) approximations, and the ℓ 1 -perturbed algorithm (DCA1) is more efficient than the ℓ 1 -reweighted algorithm (DCA2).

As a part of future work, we plan to study more extensive applications of the SFLD problem. We believe that the success of using DC approximation functions for the ℓ 0norm motivates and opens up a new avenue for the sparse Linear Discriminant Analysis (LDA) problem. In particular, we intend to apply different DC approximation functions as well as further explore other models for the sparse LDA problem.

Chapter 3

Sparse Optimal Scoring Problem

Abstract: Linear discriminant analysis (LDA) is a standard tool for classification and dimension reduction in many applications. However, the problem of high dimension is still a great challenge for the classical LDA. In this chapter we consider the supervised pattern classification in the high dimensional setting, in which the number of features is much larger than the number of observations and present a novel approach to the sparse optimal scoring problem using the zero-norm. The difficulty in treating the zero-norm is overcome by using appropriate continuous approximations such that the resulting problems are solved by alternating schemes based on DC (Difference of Convex functions) programming and DCA (DC Algorithms). The experimental results on both simulated and real datasets show the efficiency of the proposed algorithms compared to some state-of-the-art methods.

Introduction

Among several classification methods in the literature the LDA based approach is regarded as one of the most popular and is known to be efficient for problems having a small number of observations but a very large number of features. There are three different approaches to tackle LDA, which are based on solving the normal model, the Fisher's discriminant problem and the optimal scoring problem, respectively (see Chapter 2 for more details). In this chapter we do not directly consider the Fisher's discriminant problem. Instead, we are interested in the optimal scoring interpretation of LDA and develop a optimal scoring based approach, named the sparse optimal scoring problem.

Let X be an n × p data matrix with observations x i (i = 1, ..., n) on the rows and features on the columns. n i is denoted as the number of observations in the cluster C i , i = 1, ..., Q.

We assume that the features have been standardized to have mean 0 and variance 1. Let Y ∈ R n×Q with Y ik = 1 if x i ∈ C k and 0 otherwise. To find the linear transformation W , the optimal scoring criterion successively solves the problem min

w k ,θ k ||Y θ k -Xw k || 2 2 subject to 1 n θ T k Y T Y θ k = 1; θ T k Y T Y θ l = 0, l = 1, ..., k -1, (3.1)
where θ k is a Q-vector of scores.

LDA often performs quite well in simple and low-dimensional setting and it is known to fail when the number of features p is larger than the number of observations n. However, in many applications such as information retrieval, face recognition and microarray analysis, we often encounter problems having a small number of observations but a very large number of features. In such cases, one difficulty of the classical LDA is interpretation of the classifier, since the classification rule involves a linear combination of all p features. To overcome this, the most suitable approach is feature selection. A sparse classifier leads to easier model interpretation and may reduce overfitting of the training data. In the literature, several authors use the ℓ 1 -norm to deal with sparsity. More precisely, the ℓ 1 -regularization is added to the objective function of the optimal scoring problem (3.1) (see e.g. [START_REF] Grosenick | Interpretable classifers for fmri improve prediction of purchases[END_REF][START_REF] Leng | Sparse optimal scoring for multiclass cancer diagnosis and biomarker detection using microarray data[END_REF][START_REF] Clemmensen | Sparse discriminant analysis[END_REF]). [START_REF] Clemmensen | Sparse discriminant analysis[END_REF] replaced the ℓ 0 -norm with the ℓ 1 -norm and applied an alternating scheme for solving the resulting problem.

The most natural way to deal with feature selection in machine learning is using the ℓ 0 -norm in the regularization term. Using ℓ 2 + ℓ 0 regularization for the optimal scoring problem (3.1) leads us to consider the sparse optimal scoring (SOS) problem defined by min

w k ,θ k 1 2n ||Y θ k -Xw k || 2 2 + λ 1 -γ 2 ||w k || 2 2 + γ||w k || 0 subject to 1 n θ T k Y T Y θ k = 1; θ T k Y T Y θ l = 0, l = 1, ..., k -1. (3.2)
Here γ ∈ [0, 1] and λ ≥ 0 are tuning parameters, and ||w k || 0 denotes the ℓ 0 -norm of w k , i.e. the number of non-zero elements of vector w k .

In this chapter, solving (3.2) includes double difficulties. The first is how to treat the ℓ 0 -norm and the second is caused by the non-convexity of the original optimal scoring problem. To tackle the ℓ 0 -norm we investigate DC approximation approaches. As the previous chapter, we use two sparse inducing functions: the piecewise linear function (called Capped-ℓ 1 ) and the piecewise exponential concave function introduced respectively in Peleg and Meir (2008) and [START_REF] Bradley | Feature selection via concave minimization and support vector machines[END_REF]. Unfortunately, the resulting optimization problems are still difficult but they enjoy some interesting properties: when w k is fixed the optimal solution of the problem with respect to the variable θ k can be computed explicitly, while for each fixed θ k we are faced on a DC program with respect to the variable w k . We are then suggested to use alternating schemes based on DCA for solving them.

Our contributions are multiple. Using two DC approximations of the ℓ 0 -norm and consider two DC formulations of each resulting approximate SOS problem we propose alternating schemes for solving the four approximate problems. To deal with DC programs w.r.t w k in each step of the alternating algorithms, we investigate four DCA schemes. We prove that the main algorithms converge to a critical point of the approximate problems. The performance of the proposed algorithms are carefully examined in comparing with seven state of art methods on both simulated datasets and high-dimensional real datasets.

The rest of this chapter is organized as follows. In Section 3.2, we state the approximate problems and present the alternating schemes for solving them as well as the way to compute θ k in these schemes. In Section 3.3 we show how to apply DCA on the nonconvex subproblems to compute w k in the alternating schemes. Section 3.4 is devoted to the description of the main algorithms and their convergence analysis. The numerical experiments are reported in Section 3.5 and Section 3.6 concludes the chapter.

3.2 Alternating schemes for the approximate sparse optimal scoring problems

Approximate sparse optimal scoring problems

The discontinuity of the ℓ 0 -norm is overcome by using two DC approximations. For an α > 0, let η α,1 and η α,2 be the functions given by

η α,1 (x) = 1 -exp(-α|x|), ∀x ∈ R,
and η α,2 (x) = min{1, α|x|}, ∀x ∈ R,
The Capped-ℓ 1 approximation of the ℓ 0 -norm is defined by [START_REF] Peleg | A bilinear formulation for vector sparsity optimization[END_REF])

||w k || 0 ≈ p i=1 η α,1 (w ki ), (3.3)
and the piecewise exponential concave approximation [START_REF] Bradley | Feature selection via concave minimization and support vector machines[END_REF]) is

||w k || 0 ≈ p i=1 η α,2 (w ki ). (3.4)
For simplify the presentation, we use the common notation η α to design both η α,1 and η α,2 and consider the resulting approximate problem of the SOS problem (3.2) in the form min

(w k ,θ k )∈R p ×Ω k 1 2n ||Y θ k -Xw k || 2 2 + λ 1 -γ 2 ||w k || 2 2 + γ p i=1 η α (w ki ) , (3.5)
where

Ω k = {θ k ∈ R Q : θ T k Dθ k = 1; θ T k Dθ l = 0, l = 1, ..., k -1} and D = 1 n Y T Y . Observing that η α (w ki ) = η α (|w ki |) ∀w ki ∈ R and η α is increasing concave over [0, +∞],
we can deduce another equivalent form of (3.5) (see [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF] for more details) min

(w k ,z k )∈Λ k ,θ k ∈Ω k 1 2n ||Y θ k -Xw k || 2 2 + λ 1 -γ 2 ||w k || 2 2 + γ p i=1 η α (z ki ) , (3.6)
where

Λ k = {(w k , z k ) ∈ R p × R p + : |w ki | ≤ z ki ∀i = 1, ..., p}.
In the sequel we will investigate solution methods for solving the nonconvex problems (3.5) and (3.6).

For holding w k (resp. (w k , z k )) fixed, we can find an explicit solution θ k . However, for holding θ k fixed, the resulting problems are still nonconvex. Hence we will investigate alternating schemes based on DC programming and DCA for solving (3.5) and (3.6).

Alternating schemes for solving the approximate SOS problems

The alternating scheme for solving the problem (3.5) consists of holding θ k fixed and optimizing with respect to w k , and then holding w k fixed and optimizing with respect to θ k . More precisely:

Starting with w k ∈ R p and θ k ∈ Ω k , at each iteration we perform two steps: 1. Fix θ k and compute w k by solving min

w∈R p 1 2n ||Y θ k -Xw|| 2 2 + λ 1 -γ 2 ||w|| 2 2 + γ p i=1 η α (w i ) . (3.7)
2. Fix w k and compute θ k by solving min

θ∈Ω k ||Y θ -Xw k || 2 2 . (3.8)
Similarly, the alternating scheme for solving the problem (3.6) differs from the above scheme only on the step 1:

Starting with (w k , z k ) ∈ Λ k and θ k ∈ Ω k , at each iteration we perform two steps: 1. Fix θ k and compute w k by solving min

(w,z)∈Λ k 1 2n ||Y θ k -Xw|| 2 2 + λ 1 -γ 2 ||w|| 2 2 + γ p i=1 η α (z i ) . (3.9)
2. Fix w k and compute θ k by solving (3.8).

We will show below how to compute θ k in the step 2 of these alternating schemes.

Compute θ k in the alternating schemes

Let Q k-1 be the Q × (k -1) matrix whose columns are the previous k -1 solutions θ 1 , ..., θ k-1 consecutively. For solving (3.8), we state the following lemma.

Lemma 3.1 The problem (3.8) has a unique solution

θ k = s k / s T k Ds k , where s k = (I -Q k-1 Q T k-1 D)D -1 Y T Xw k .
Clemmensen et al. ( 2011) stated that θ k solves the problem (3.8), but they do not give the proof. We will prove that this solution is unique.

Proof : We assume that θ k is a solution of the problem (3.8). It follows that θ k satisfies the following KKT conditions:

2nDθ k -Y T Xw k + 2λ 1 Dθ k + DQ k-1 λ 2 = 0, (3.10) θ T k Dθ k = 1, (3.11) θ T k DQ k-1 = 0, (3.12)
where

λ 1 ∈ R and λ 2 ∈ R k-1 are Lagrange multipliers. Multiplying (3.10) by θ T k gives n + λ 1 = 1 2 θ T k Y T Xw k . (3.13)
On the other hand, substituting (3.13) into the objective function of the problem (3.8), we have

F w k (θ k ) = ||Y θ k -Xw k || 2 = n + w T k X T Xw k -4(n + λ 1 ). (3.14)
Thus, we only need to consider n + λ 1 > 0. From (3.10), solving for θ k leads to

θ k = 1 n + λ 1 D -1 (Y T Xw k - 1 2 DQ k-1 λ 2 ). (3.15)
The orthogonality constraints give

θ T k DQ k-1 = 0 ⇔ w T k X T Y Q k-1 - 1 2 λ T 2 Q k-1 DQ k-1 = 0 ⇒ λ 2 = 2Q T k-1 Y T Xw k .
Inserting this expression for λ 2 into equation (3.15) and simplifying gives

θ k = 1 n + λ 1 (I -Q k-1 Q T k-1 D)D -1 Y T Xw k = 1 n + λ 1 s k , (3.16)
where

s k = (I -Q k-1 Q T k-1 D)D -1 Y T Xw k . Finally, the constraint θ T k Dθ k = 1 gives n + λ 1 = s T k Ds k , then θ k = s k / s T k Ds k is unique solution of the problem (3.

8). ✷

We are now going to develop DCA based algorithms for solving the subproblems in the step 1 of alternating schemes. The approximation η α can be expressed as a DC function:

η α (x) = g(x) -h(x), (3.17)
where

g(x) = α|x|, h(x) = -1 + α|x| + exp(-α|x|) if η α = η α,1 , and h(x) = -1 + max{1, α|x|} if η α = η α,2 .
Therefore, the objective function of the problem (3.7) can be rewritten as follows.

F θ k (w) := G 1 (w, θ k ) -H 1 (w), (3.18)
where

G 1 (w, θ k ) : = 1 2n ||Y θ k -Xw|| 2 2 + λ 1 -γ 2 ||w|| 2 2 + γ p i=1 g(w i ) , H 1 (w) : = λγ p i=1 h(w i ),
are clearly convex functions. Hence a DC formulation of the problem (3.7) takes the form

min w∈R p {G 1 (w, θ k ) -H 1 (w)}. (3.19)
According to the generic DCA scheme, DCA applied on (3.19) consists of computing, at each iteration l, a subgradient v l ∈ ∂H 1 (w l ) and solving the convex program of the form (P l ), namely min

w∈R p {G 1 (w, θ k ) -v l , w }. (3.20)
The algorithm is described as follows.

DCA1

Initialization: Let τ be a tolerance sufficient small, set l = 0 and choose

w 0 ∈ R p . repeat 1. Compute v l ∈ ∂H 1 (w l ).
2. Solve the following convex problem to obtain w l+1 min

w∈R p 1 2n ||Y θ k -Xw|| 2 2 + λ 1 -γ 2 ||w|| 2 2 + γα||w|| 1 -v l , w . (3.21) 3. l ← l + 1. until w l -w l-1 2 ≤ τ ( w l-1 ) 2 + 1) or |F θ k (w l ) -F θ k (w l-1 )| ≤ τ (|F θ k (w l-1 )| + 1).
We see that the problem (3.21) has the ℓ 1 -perturbed form.

The implementation of DCA1 requires the computation of v l ∈ ∂H 1 (w l ) in the step 1, which depends on η α . More precisely, for η α = η α,1 , v l is computed by

v l i = λγα(1 -exp(-αw l i )) if w l i ≥ 0 -λγα(1 -exp(αw l i )) if w l i < 0 i = 1, ..., p. (3.22)
For η α = η α,2 , v l is calculated as follows.

v l i = sgn(w l i )λγα if α|w l i | ≥ 1 0 otherwise i = 1, ..., p. (3.23)
Remark 3.1 For solving the convex problem (3.21), we use the coordinate descent method [START_REF] Friedman | Pathwise coordinate optimization[END_REF]. This method is known to be one of the most efficient algorithms for solving unconstrained convex problems whose objective function has the form: quadratic + ℓ 1 . Its objective function is written as:

f (w) := 1 2n n i=1 (Y θ k ) i - m =j X im w m -X ij w j 2 + λ(1 -γ) 2 m =j w 2 m + λ(1 -γ) 2 w 2 j + λγα m =j |w m | + λγα|w j | - m =j v l m w m -v l j w j .
Suppose that we have estimates w m = wm for m = j, and we wish to partially optimize with respect to w j . The coordinate-wise update has the form

w j ←- S 1 n n i=1 X ij (Y θ) i - m =j X im wm + v l j , λγα 1 + λ(1 -γ) , (3.24)
where S(z, t) is the soft-thresholding operator with value

S(z, t) =      z -t if z > 0 and t < |z|, z + t if z < 0 and t < |z|, 0 if t ≥ |z|.
The update (3.24) is repeated for j = 1, 2, ..., p, 1, 2, ... until

w k -w k-1 2 ≤ ǫ or |f (w k ) -f (w k-1 )| ≤ ǫ,
where w k is the solution obtained at the k-th iteration.

For designing a DC formulation of (3.9) we observe that the function η α is concave on [0, +∞] and then -η α is convex on [0, +∞]. Therefore a DC formulation of (3.9) can be min

(w,z)∈R p ×R p Fθ k (w, z) := G 2 (w, z, θ k ) -H 2 (w, z) , (3.25)
where

G 2 (w, z, θ k ) : = 1 2n ||Y θ k -Xw|| 2 2 + λ(1 -γ) 2 ||w|| 2 2 + χ Λ k (w, z), H 2 (w, z) : = λγ p i=1 (-η α )(z i ),
are clearly convex functions. Here χ Λ k is the indicator function on Λ k , that is

χ Λ k (w, z) = 0 if (w, z) ∈ Λ k , +∞ otherwise.
Like DCA1, DCA applied on (3.25) consists of computing, at each iteration l, a subgradient (v l , zl ) ∈ ∂H 2 (w l , z l ), and then solving the following convex program to obtain (w l+1 , z l+1 ):

min (w,z)∈R p ×R p G 2 (w, z, θ k ) -v l , w -zl , z . (3.26) When η α = η α,1 , zl is calculated by zl i = -exp(-αz l i )λγα ∀i = 1, ..., p, (3.27)
and for η α = η α,2 , we have

zl i = -λγα if z l i ≤ 1/α, 0 otherwise, ∀i = 1, ..., p. (3.28)
We deduce from (3.27) and (3.28) that zl i ≤ 0 ∀i = 1, ..., p. Thus, the problem (3.26) is equivalent to

   w l+1 ∈ arg min w∈R p 1 2n ||Y θ k -Xw|| 2 2 + λ(1-γ) 2 ||w|| 2 2 - p i=1 zl i |w i | z l+1 i = |w l+1 i | ∀i.
(3.29) DCA for solving (3.25) is described as follow.

DCA2

Initialization: Let τ be a tolerance sufficient small, set l = 0 and choose

w 0 ∈ R p , z 0 i = |w 0 i | ∀i = 1, ..., p. repeat 1. Compute zl i ∈ λγ∂(-η α )(z l i ).
2. Solve the following convex problem to obtain w l+1 min

w∈R p 1 2n ||Y θ k -Xw|| 2 2 + λ(1 -γ) 2 ||w|| 2 2 + p i=1 (-z l i )|w i | (3.30) 3. Compute z l+1 i = |w l+1 i | ∀i = 1, ..., p. 4. l ← l + 1. until (w l , z l )-(w l-1 , z l-1 ) 2 ≤ τ ( (w l-1 , z l-1 ) 2 +1) or | Fθ k (w l , z l )-Fθ k (w l-1 , z l-1 )| ≤ τ | Fθ k (w l-1 , z l-1 )| + 1 .
In the step 2 of DCA2, we see that the problem (3.30) has the form of a ℓ 1 -regularization problem but with different weights on components of |w i |. So DCA2 iteratively solves the weighted-ℓ 1 problem (3.30) with an update of the weights -z l i at each iteration l.

Theorem 3.1 (i) DCA1 and DCA2 generate, respectively, the sequences

{w l } l in R p and {(w l , z l )} l in Λ k such that {F θ k (w l )} l and { Fθ k (w l , z l )} l are decreasing. (ii) If the sequence {w l } l (resp. {(w l , z l )} l ) is bounded, then every limit point w * (resp. (w * , z * )) of the sequence {w l } l (resp. {(w l , z l )} l ) is a critical point of the problem (3.19) (resp. (3.25)). (iii) In the case of Capped-ℓ 1 (η α = η α,2
), the sequence {w l } l and {(w l , z l )} l respectively convergence to w * and (w * , z * ) after a finite number of iterations. Moreover, the points w * and (w * , z * ) are critical points of the problems (3.19) and (3.25), respectively. If, in addition,

w * i / ∈ 1 α , - 1 α ∀i = 1, ..., p, (3.31)
(resp. z * i = 1/α ∀i = 1, ..., p) then w * (resp. (w * , z * )) is in fact a local minimizer of (3.19) (resp. (3.25)).
Proof : (i) and (ii) are direct consequences of convergence properties of general DC programs while the first part of (iii) is a convergence property of a DC polyhedral program.

For the second part of (iii), observing that the second DC component of (3.19) ✷

We can now describe our main algorithms based on alternating methods and DCA for solving the approximate SOS problems.

Description of the main algorithms and their convergence properties

Finally, the alternating scheme based on DCA for solving the problem (3.5) can be described as follows.

ADCA1 Alternating scheme based on DCA for the problem (3.5) for k = 1 to K, compute k-th discriminant vector w k as follows:

Initialization: w 0 k ∈ R p , θ 0 k = s 0 k / (s 0 k ) T Ds 0 k , where s 0 k = (I -Q k-1 Q T k-1 D)D -1 θ * k with θ * k ∈ R Q , and l = 0. repeat 1. For fixed θ l k , compute w l+1 k by DCA1 using w l k as initial point. 2. For fixed w l+1 k , compute s l+1 k = (I -Q k-1 Q T k-1 D)D -1 Y T Xw l+1 k and set θ l+1 k = s l+1 k / (s l+1 k ) T Ds l+1 k . 3. l ← l + 1. until Stopping criterion. end for
The stopping criterion of ADCA1 is given by

(w l k , θ l k ) -(w l-1 k , θ l-1 k ) 2 ≤ τ ( (w l-1 k , θ l-1 k ) 2 + 1), or |F (w l k , θ l k ) -F (w l-1 k , θ l-1 k )| ≤ τ (|F (w l-1 k , θ l-1 k )| + 1)
, where F (w k , θ k ) is the objective function of the problem (3.5).

For each algorithm, we use two approximations of the ℓ 0 -norm (η α,1 and η α,2 ). We denote by ADCA1-Exp (resp. ADCA1-Cap) ADCA1 using η α = η α,1 (resp. η α = η α,2 ). Furthermore, the alternating scheme using DCA for solving the problem (3.6) is given by the following algorithm.

ADCA2 Alternating scheme based on DCA for the problem (3.6) for k = 1 to K, compute k-th discriminant vector w k as follows:

Initialization:

w 0 k ∈ R p , θ 0 k = s 0 k / (s 0 k ) T Ds 0 k , where s 0 k = (I -Q k-1 Q T k-1 D)D -1 θ * k with θ * k ∈ R Q , and l = 0. repeat 1. For fixed θ l k , compute (w l+1 k , z l+1 k ) by DCA2 using w l k as initial point. 2. For fixed (w l+1 k , z l+1 k ), compute s l+1 k = (I -Q k-1 Q T k-1 D)D -1 Y T Xw l+1 k and set θ l+1 k = s l+1 k / (s l+1 k ) T Ds l+1 k . 3. l ← l + 1.
until Stopping criterion. end for

The stopping criterion of ADCA2 is given by

(w l k , z l k , θ l k ) -(w l-1 k , z l-1 k , θ l-1 k ) 2 ≤ τ ( (w l-1 k , z l-1 k , θ l-1 k ) 2 + 1), or | F (w l k , z l k , θ l k ) -F (w l-1 k , z l-1 k , θ l-1 k )| ≤ τ (| F (w l-1 k , z l-1 k , θ l-1 k )| + 1), where F (w k , z k , θ k ) is the objective function of the problem (3.6). ADCA2 using η α = η α,1 (resp. η α = η α,2 ) is denoted by ADCA2-Exp (resp. ADCA2-Cap).
The convergence properties of ADCA1 and ADCA2 are given by Theorem 3.2 below.

Theorem 3.2 (i) ADCA1 generates the sequences {(w l k , θ l k )} l in R p × Ω k , k = 1, ..., K such that {F (w l k , θ l k )} l is decreasing. (ii) If the sequence {(w l k , θ l k )} l generated by ADCA1 is bounded, then every limit point (w * k , θ * k ) of this sequence is a critical point of the problem (3.5).
Similarly, we have (iii) ADCA2 generates the sequences

{(w l k , z l k , θ l k )} l in Λ k × Ω k , k = 1, ..., K such that { F (w l k , z l k , θ l k )} l is decreasing. (iv) If the sequence {(w l k , z l k , θ l k )} l generated by ADCA2 is bounded, then every limit point (w * k , z * k , θ * k )
of this sequence is a critical point of the problem (3.6).

Proof :

The properties (i) and (iii) (resp. (ii) and (iv)) are proved analogously. Therefore we give here the proof for (i) and (ii) only.

For (i), we assume that {(w l k , θ l k )} l , k = 1, ..., K are generated by ADCA1. We have

{(w l k , θ l k )} l in R p × Ω k and F (w l+1 k , θ l+1 k ) -F (w l k , θ l k ) = F w l+1 k (θ l+1 k ) -F w l+1 k (θ l k ) + F θ l k (w l+1 k ) -F θ l k (w l k ).
By the Lemma 3.1, we have

F w l+1 k (θ l+1 k ) -F w l+1 k (θ l k ) ≤ 0. (3.32)
For fixed θ l k , w l+1 k is computed by DCA1 using w l k as initialization, then we reduce from (i) of Theorem 3.1 that

F θ l k (w l+1 k ) -F θ l k (w l k ) ≤ 0. (3.33) Thus, we have F (w l+1 k , θ l+1 k ) -F (w l k , θ l k ) ≤ 0.
(ii) We assume that the sequence

{(w l k , θ l k )} l is bounded. Let (w * k , θ * k ) be a limit point of {(w l k , θ l k )} l . Thus, there exists a subsequence (w lt k , θ lt k ) → (w * k , θ * k ) as t → +∞. We will prove that (w * k , θ * k ) is a critical point of the problem (3.5), i.e. ∅ = ∂ w k G 1 (w * k , θ * k ) ∩ ∂ w k H 1 (w * k ), (3.34) {θ * k } = arg min θ k ∈Ω k F w * k (θ k ). (3.35)
From the step 2 in ADCA1, we have

θ lt k = s lt k / (s lt k ) T Ds lt k , where s lt k = (I - Q k-1 Q T k-1 D)D -1 Y T Xw lt k . Taking the limit as t → +∞, we get θ * k = s * k / (s * k ) T Ds * k , with s * k = (I -Q k-1 Q T k-1 D)D -1 Y T Xw * k . By Lemma 3.1, it follows that the condition (3.35) holds. Since {θ lt-1 k } t is a subsequence of {θ l k } l , {θ lt-1 k
} t is also bounded. Without loss of generality, we can suppose (by extracting a subsequence if necessary) that the sequence

θ lt-1 k → θ * *
k as t → +∞. We know that {F (w l k , θ l k )} l is decreasing and it is bounded below by 0. Besides, we have

F (w l k , θ l k ) ≤ F (w l k , θ l-1 k ) ≤ F (w l-1 k , θ l-1 k ). Thus, lim l→+∞ F (w l k , θ l k ) = lim l→+∞ F (w l k , θ l-1 k ) = inf l F (w l k , θ l k ). Using the fact that F is continuous, we get F (w * k , θ * k ) = lim t→+∞ F (w lt k , θ lt k ) = lim t→+∞ F (w lt k , θ lt-1 k ) = F (w * k , θ * * k ). (3.36)
According to Lemma 3.1, the problem min θ k ∈Ω k F w * k (θ k ) has a unique solution. Hence, we deduce from (3.36)

that θ * * k = θ * k . From (ii) of Theorem 3.1, we have ∅ = ∂ w k G 1 (w lt k , θ lt-1 k ) ∩ ∂ w k H 1 (w lt k ). Therefore, there exists y lt k such that y lt k ∈ ∂ w k G 1 (w lt k , θ lt-1 k ) ∩ ∂ w k H 1 (w lt k ). (3.37)
From the computation of ∂ w k H 1 (w lt k ) it follows that the sequence {y lt k } t is bounded. Thus, without loss of generality, we can suppose that the sequence y lt k → y * as t → +∞. We have

y lt k ∈ ∂ w k G 1 (w lt k , θ lt-1 k ) ⇔ G 1 (w lt k , θ lt-1 k ) + G * 1 (y lt k , θ lt-1 k ) = w lt k , y lt-1 k , (3.38) where G * 1 (., θ lt-1 k ) is the conjugate function of G 1 (., θ lt-1 k
) and

y lt k ∈ ∂ w k H 1 (w lt k ) ⇔ H 1 (w lt k ) + H * 1 (y lt k ) = w lt k , y lt k . (3.39)
Taking t → +∞ and using Lemma 2 in Pham Dinh and Le [START_REF] Pham Dinh | Convex analysis approach to D.C. programming: Theory, algorithms and applications[END_REF] we obtain

G 1 (w * k , θ * k ) + G * 1 (y * k , θ * k ) = w * k , y * k , H 1 (w * k ) + H * 1 (y * k ) = w * k , y * k , and hence y * k ∈ ∂ w k G 1 (w * k , θ * k ) ∩ ∂ w k H 1 (w * k ).
The proof of (ii) in then complete. ✷ In thesemethods the ℓ 1 regularization is used to deal with sparsity. We are also interested in the comparison between the four versions of our algorithms to evaluate the efficiency of the two DC approximations of the ℓ 0 -norm as well as the two different DC formulations.

We also compare the proposed algorithms in this chapter with the ℓ 0 -sparse Fisher LDA (DCA1-Cap) proposed in previous chapter and ℓ 0 -sparse multiclass support vector machine proposed in Le Thi and Nguyen (2013) (SMSVM-Cap).

Penalized linear discriminant analysis (PLDA)

PLDA penalized the objective function of the Fisher's discriminant problem (2.7) with the ℓ 1 penalty on the discriminant vector, namely

max w k ∈R p {w T k Σ b w k -λ k ||w k || 1 : w T k Σw k = 1; w T k Σw l = 0, l = 1, ..., k -1}, (3.40)
where λ k is a nonnegative tuning parameter. The problem (3.40) is nonconvex. [START_REF] Witten | Penalized classification using Fisher's linear discriminant[END_REF] used the minorization-maximization approach for finding a local of this problem. This algorithm is in fact a version of DCA. The R package penalizedLDA is available from CRAN 2 .

Sparse discriminant analysis (SDA)

SDA using the ℓ 1 -norm was proposed by [START_REF] Clemmensen | Sparse discriminant analysis[END_REF], that is min

w k ,θ k { 1 n ||Y θ k -Xw k || 2 2 + γw T k Ωw k + λ||w k || 1 } subject to 1 n θ T k Y T Y θ k = 1; θ T k Y T Y θ l = 0, l = 1, ..., k -1, (3.41)
where Ω is a positive definite matrix, γ and λ are nonnegative tuning parameters. In [START_REF] Clemmensen | Sparse discriminant analysis[END_REF], the authors used an alternating scheme for finding a local optimum of this problem. The R package sparseLDA is available from CRAN.

Shrunken centroids regularized discriminant analysis (RDA)

RDA is based on the same underlying model as LDA (see [START_REF] Guo | Regularized linear discriminant analysis and its application in microarrays[END_REF]) and it regularizes the within-class covariance matrix used by LDA

Σ = αΣ + (1 -α)I p , (3.42)
where 0 ≤ α ≤ 1. In order to perform feature selection, one can perform soft-thresholding of the quantity Σ-1 µ k . That is, we compute

sgn( Σ-1 µ k )(| Σ-1 µ k | -δ) + , (3.43)
where δ is a nonnegative tuning parameter. The R package rda is available from CRAN.

Sparse partial least squares discriminant analysis (SPLS DA)

SPLS DA used the lasso to promote sparsity of a surrogate direction vector c instead of the original latent direction vector α, while keep α and c close (see [START_REF] Chun | Sparse partial least squares regression for simultaneous dimension reduction and variable selection[END_REF]). That is, the first SPLS DA direction vector solves min

α,c∈R p -κα T Mα + (1 -κ)(c -α) T M(c -α) + λ||c|| 1 + γ||c|| 2 subject to α T α = 1, (3.44)
where κ is a tuning parameter with 0 ≤ κ ≤ 1, and λ, γ are nonnegative tuning parameters. Performing the SPLS DA method obtains c 1 , ..., c s sparse surrogate direction vectors. Then, we obtain a classification rule by performing standard LDA on the low dimensional space (Xc 1 , ..., Xc s ). The R package spls is available from CRAN.

Sparse support vector machines (S SVM)

S SVM use the ℓ 1 -regularization for the multi-class support vector machine. This method is supported by the LIBLINEAR package [START_REF] Fan | Liblinear: A library for large linear classification[END_REF]. The R package LiblineaR is also available from CRAN. For the feature selection purpose, Le [START_REF] Le Thi | Efficient algorithms for feature selection in multi-class support vector machine[END_REF] proposes to use the ℓ 0ℓ 2 regularization for the MSVM model [START_REF] Weston | Support vector machines for multi-class pattern recognition[END_REF], that leads to the so called ℓ 2 -ℓ 0 -MSVM problem which is defined by min

(w,b,ξ)∈Ω C n i=1 k =y i ξ ik + β Q k=1 w k 2 2 + Q k=1 w k 0 , (3.46)
where C, β are nonnegative tuning parameters, and

Ω = (w, b, xi) ∈ R Q×d × R Q × R n×Q + : w y i -w k , x i + b y i -b k ≥ 1 -ξ ik , ∀1 ≤ i ≤ n, 1 ≤ k = y i ≤ Q .
Using the appropriate DC approximation functions of the ℓ 0 -norm, the resulting the problems are the DC programs which are solved by DCA (Le Thi and Nguyen, 2013).

Datasets

We evaluate the performance of comparative algorithms on three synthetic datasets and a collection of real world datasets. Three synthetic datasets are generated in the following ways.

For the first setup S1, we generate a three classes classification problem. Each class is assumed to have a multivariate normal distribution N(µ k , Σ), k = 1, 2, 3 with dimension p = 500. All elements on the main diagonal of covariance matrix Σ are equal to 1 and all other elements are equal to 0.6. The first 35 components of µ 1 are 0.7, µ 2j = 0.7 if 36 ≤ j ≤ 70 and µ 3j = 0.7 if 71 ≤ j ≤ 105 and 0 otherwise. For each class, we generate 100 training samples, 100 tuning samples and 500 test samples.

The second simulation setup S2 includes two classes of multivariate normal distributions N(µ 1 , Σ) and N(µ 2 , Σ), each of dimension p = 500. The components of µ 1 are assumed to be 0 and for µ 2 , µ 2j = 0.6 if j ≤ 200 and 0 otherwise. The covariance matrix Σ is the block diagonal matrix with five blocks of dimension 100 × 100 whose element (j, j ′ ) is 0.6 |j-j ′ | . For each class, 100 training samples, 100 tuning samples and 10000 test samples are generated.

For the last setup S3, we generate a three-class classification problem as follows: 784 60000/10000 10 data to zero mean and unit variance. The detailed information of these datasets is summarized in Table 3.1.

i ∈ C k then X ij ∼ N((k -1)/2, 1) if j ≤ 100, k = 1, 2, 3 and X ij ∼ N(0,

Experimental setups

The proposed algorithms were implemented in the Visual Studio 2012, and performed on a PC Intel i7 CPU3770, 3.40 GHz of 8GB RAM.

In our experiments, the tuning parameters are γ, λ and α in (3.3) and (3.4). We fixed α = 5 as suggested in [START_REF] Bradley | Feature selection via concave minimization and support vector machines[END_REF], and γ, λ are performed by 5fold cross-validation procedure on training or tuning set from sets of candidates given by Γ = {0.1, ..., 0.9, 1} and Λ = {0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.08, 0.012, 0.1, 0.15, 0.4, 0.6, 0.7, 0.9}, respectively. By this way, we avoid performing tuning parameter selection on a threedimensional grid.

The stop tolerance of DCA and ADCA is τ = 10 -5 while the stop tolerance of the coordinate descent method is ǫ = 10 -4 . The starting point (w 1]. w l k is a starting point of DCA at l-th iteration of ADCA. We select relevant features as follows: feature i is deleted if |w ki | < 10 -6 for all k = 1, ..., K.

0 k , θ 0 k ) of ADCA is computed by w 0 k = 0 and θ 0 k = s 0 k / (s 0 k ) T Ds 0 k , where s 0 k = (I -Q k-1 Q T k-1 D)D -1 θ * k with each element of θ * k is picked randomly in [-1,

Experiments on synthetic data

In this experiment, we generate training, tuning, and test sets in the same manner as described in Sect. 3.5.2. The tuning sets are used to choose the parameters λ, γ and the number of discriminant vectors used K, while the test sets are used to measure the accuracy of various classifiers trained on the training sets. We perform 10 trials for each experimental setting.

The experimental results on synthetic data are given in Table 3.2. In this table, the average number (#FS) and percentage (%FS) of selected features (standard deviations), the average percentage of accuracy of classifiers (ACC) and its standard deviation over 10 trials, the number of discriminant vectors used K (#DV), as well as training time in second (CPUs) are reported.

We observe from Table 3.2, in terms of feature selection, the DCA based algorithms are comparable and they give better results than PLDA, SDA, RDA, SPLS DA and S SVM. ADCA1-Cap, ADCA2-Exp and ADCA2-Cap give the best results on the S2, S2 and S3 datasets, respectively.

The DCA based algorithms not only provide a good performance in terms of feature selection, but also give a high accuracy of classifiers. ADCA1-Cap and ADCA2-Cap give the best accuracy of classifiers on 2/3 synthetic datasets.

The training time of all the algorithms is quite small: less than 2 seconds (except for the algorithm SDA).

Experiments on real datasets

For the experiments on the first eleven real datasets, we use the cross-validation scheme to validate the performance of various classifiers. Each real dataset is split into a training set containing 2/3 of the samples and a test set containing 1/3 of the samples. This process is repeated 10 times, each with a random choice of training set and test set. The parameter γ, λ and the number of discriminant vectors K which are used are chosen via 5-fold cross-validation.

The computational results given by ADCA1-Exp, ADCA1-Cap, ADCA2-Exp, ADCA2-Cap, PLDA, SDA, RDA and SPLS DA are reported in . We are interested in the efficiency (the sparsity and the accuracy of classifiers), the number of discriminant vectors used K, as well as the rapidity of these algorithms. The discriminant vectors can be used to visualize the datasets such as in Figure 3.1. Comments on computational results: Sparsity. In on all the datasets, the classifiers obtained by the DCA based algorithms are sparser than those obtained by PLDA, SDA, RDA and SPLS DA. The ADCA1-Exp approach is the best on 8 out of 11 datasets and ADCA2-Exp is the best on 2 out of 11 datasets. We also see that ADCA1-Cap achieves the best performance on the PRO dataset. Especially, on some datasets such as the COL, PEN, ALL, LUN and OVA datasets, ADCA1-Exp only selects a very small number of features (5.8, 2, 9.6, 3.7 and 4 features, respectively). ADCA1-Exp, ADCA1-Cap, ADCA2-Exp and ADCA2-Cap respectively select from 0, 02% to 20.31%, 0.18% to 21.09%, 0, 02% to 19.21% and 0.18% to 25.24% of features while PLDA, SDA, RDA, SPLS DA and S SVM respectively select from 0.59% to 63.47%, 0.05% to 31.24%, 0.62% to 100%, 0.39% to 99.03% and 0.37% to 22.48% of features.

Accuracy of classifiers. The accuracy of classifiers of the DCA based algorithms attain better than PLDA, SDA, RDA, SPLS DA and S SVM on 8/11 datasets. ADCA1-Cap is the best on 6/11 datasets. ADCA1-Exp and ADCA2-Exp are the best on 3/11 datasets. PLDA and RDA are slightly better than the DCA based algorithms on three datasets (LUN, LEU and PRO). This can be explained by the fact PLDA and RDA select much more features than the DCA based algorithms (the ratio is, respectively, 1042, 491 and 181 times on the LUN, LEU and PRO datasets).

Training time.

Training time of the DCA based algorithms is quite small and acceptable: less than 81 seconds (except for datasets LEU and MLL).

For the experiments on the MNIST dataset. his dataset is quite different, as it has a very large number of observations (70000) while the number of features is not large (784). The parameter γ, λ and the number of discriminant vectors K which are used are chosen via 5-fold cross-validation on the training set. The test set is used to measure the accuracy of Overall, the DCA based algorithms realize better a trade-off between accuracy and sparsity than the other algorithms. They suppress considerably the number of features (up to 99.98%) while the correctness of classification is quite good (from 78.17% to 100%).

3.5.6

Comparison with ℓ 0 -sparse Fisher LDA and ℓ 0 -sparse MSVM Before finishing this chapter, we compare ADCA1-Cap with the sparse Fisher linear discriminant analysis (DCA1-Cap) proposed in the previous chapter and the sparse multiclass support vector machine (SMSVM-Cap) using the ℓ 2 + ℓ 0 regularization (Le [START_REF] Le Thi | Efficient algorithms for feature selection in multi-class support vector machine[END_REF]. The all three methods use the Capped-ℓ 1 approximation function of the ℓ 0 -norm.

The comparative results of ADCA1-Cap, DCA1-Cap and SMSVM-Cap are reported in Table 3.6. We observe from this table that in terms of classification accuracy, ADCA1-Cap is better than DCA1-Cap while DCA1-Cap outperforms SMSVM-Cap. More precisely, ADCA1-Cap is the best on 10/14 datasets and DCA1-Cap is the best on 7/14 datasets in terms of classification accuracy. In terms of the sparsity, ADCA1-Cap and SMSVM-Cap give the best and second best results, respectively. Specifically, ADCA1-Cap, SMSVM-Cap and DCA1-Cap are respectively the best on 8/14, 4/14 and 2/14 datasets with respect to the sparsity. Concerning the running time, we notice that DCA1-Cap is the fast, following by ADCA1-Cap, while SMSVM-Cap runs much lower than DCA1-Cap and ADCA1-Cap. DCA1-Cap runs extremely quickly (less than 8s). This is explained by using the diagonal estimate matrix of the within-class covariance matrix in the model, hence DCA1-Cap is explicitly computed at each iteration. We also note that SMSVM-Cap is very expensive when p is quite large, because its convex quadratic sub-problem has 2Qp + Q + (Q -1)n variables and 2(Q -1)n + Qp constraints.

Conclusion

We have proposed efficient approaches for solving the Sparse Optimal Scoring (SOS) problem using the ℓ 0 regularization. Among several sparse inducing functions of the ℓ 0 -norm we used two appropriate approximations functions, and reformulated the SOS problem as continuous nonconvex optimization problems. We proposed two DC formulations of the approximate SOS problems and then investigated the alternating schemes based on DCA for them. Overall, four DCA based algorithms have been developed in order to exploit the nice effect of DC decompositions / DC formulations on one part, and the efficiency of the approximate functions on another part. The research developed in this chapter permitted us to highlight the following observations / recommendations:

• About the two DC formulations and their resulting DCA based algorithms: The second formulation seems to be more complicated as it needs an additional variable z. However, the use of z does not affect the complexity of DCA2: the two convex problems in DCA1 and DCA2 have the same form and the same dimension. Therefore, intuitively we can say that DCA1 and DCA2 have the same complexity. Regarding the behavior of algorithms, with the same approximate function, DCA2 seems to be more interesting in the sense that DCA2 is a re-weighted ℓ 1 type algorithm while DCA1 is simply a perturbed ℓ 1 algorithm (the coefficient of the ℓ 1 -term is fixed during DCA scheme). In fact, by updating the coefficients of the ℓ 1 -term in the convex problem at each iteration, DCA2 likely furnishes a better solution, consequently ADCA2 likely gives a better classification accuracy than ADCA1. This observation and the numerical results on the synthetic data suggest us to promote the use of DCA2 when we know that the data following a multivariate normal / Gaussian distribution. Meanwhile, averagely speaking, DCA1 and DCA2 are comparable on classification and CPU time. As for sparsity, it depends significantly on the approximate function.

• About the two approximate functions and their resulting DCA based algorithms: Intuitively, as has been discussed in Le [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF], Capped-ℓ 1 is more interesting than the exponential approximation (Exp) since the resulting DCA has a finite convergence and it gives in almost always cases a local minimum (whereas, theoretically, a solution obtained by DCA using Exp is only a critical point). It is easy to check the local optimality condition and in our experiments the DCA using Capped-ℓ 1 gives a local minimum in all test problems. This superiority of Capped-ℓ 1 versus Exp is confirmed by the numerical results in our experiments: generally speaking, in the same problem formulation, ADCA with Capped-ℓ 1 gives a better classification accuracy than ADCA using Exp. However, like numerical results in numerous previous works (see e.g. [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF], 2014a[START_REF] Leng | Sparse optimal scoring for multiclass cancer diagnosis and biomarker detection using microarray data[END_REF]; Ong and Le Thi, 2013a)), our numerical results show that Exp is more efficient than Capped-ℓ 1 when promoting sparsity (except for only one data set S3). Hence the users are recommended to use Capped-ℓ 1 (resp. Exp) when the classification accuracy (resp. the sparsity of classifier) is the most important criterion in the considered classification problem.

• Overall, the four versions of DCA based algorithms are comparable and the numerical results showed that ADCA1-Exp seems to be the most promising algorithm that realizes a good trade-off between accuracy and sparsity.

The efficiency of the four proposed methods have been compared with five standard algorithms which use the ℓ 1 regularization. The computational results show the robustness and the effectiveness of the DCA based algorithms and their superiority with respect to these standard approaches. It turns out that

• The ℓ 0 sparse optimal scoring methods are more efficient than other algorithms in classification of high dimensional data, especially when the number of features are much larger than the number of observations. The use of the proposed methods is strongly recommended for this type of data. It is worth noting that the SSVM is not suitable for such a data. Indeed, as has been seen in our numerical experiments, SVM using convex regularization like S SVM is not efficient to feature selection, and therefore nonconvex approximations of the ℓ 0 regularization is necessary to deal with sparsity. However, if the sparse MSVM is also treated by the Capped-ℓ 1 (SMSVM-Cap) and the exponential concave function as we did on the optimal scoring methods, then the convex sub-problems in DCA are quadratic which require second order methods (see [START_REF] Le Thi | Efficient algorithms for feature selection in multi-class support vector machine[END_REF]). Such methods are very expensive when p is quite large, because the convex quadratic sub-problem has 2Qp + Q + (Q -1)n variables and 2(Q -1)n + Qp constraints (note however that the sub-problems in our SOS methods have Kp(K < Q) variables).

• For any type of data, ℓ 0 regularization produces much better sparsity than ℓ 1 regularization. Hence we suggest to use the ℓ 0 sparse optimal scoring methods when the sparsity is significantly desired.

For future works, we plan to study more extensive applications of the SOS problem. In particular, we extend our works to more complex settings, such as the case where the observations from each class are drawn from a mixture of Gaussian distributions resulting in nonlinear separations between classes.

Introduction

The estimation of covariance matrix is a common statistical problem that emerges from many scientific applications, and it quickly becomes an active and fast growing field of research. Much statistical analysis of such high dimensional data requires estimating a covariance matrix or its inverse. Several applications in numerous domains such as portfolio management and risk assessment (Ledoit andWolf, 2003, 2004;[START_REF] Jagannathan | Risk reduction in large portfolios: Why imposing the wrong constraints helps[END_REF][START_REF] Kourtis | Parameter uncertainty in portfolio selection: Shrinking the inverse covariance matrix[END_REF][START_REF] Fan | Large covariance estimation by thresholding principal orthogonal complements[END_REF][START_REF] Xue | Positive-definite ℓ 1 -penalized estimation of large covariance matrices[END_REF][START_REF] Lai | Mean-variance portfolio optimization when means and covariances are unknown[END_REF][START_REF] Deng | Penalized covariance matrix estimation using a matrix-logarithm transformation[END_REF], high dimensional classification [START_REF] Guo | Regularized linear discriminant analysis and its application in microarrays[END_REF][START_REF] Witten | Penalized classification using Fisher's linear discriminant[END_REF][START_REF] Tibshirani | Class prediction by nearest shrunken centroids, with applications to DNA microarrays[END_REF], analysis of independence and conditional independence relationships between components in graphical models, statistical inference like controlling false discoveries in multiple testing [START_REF] Leek | A general framework for multiple testing dependence[END_REF][START_REF] Efron | Correlated z-values and the accuracy of large-scale statistical estimates[END_REF], finding quantitative trait loci based on longitudinal data [START_REF] Yap | Nonparametric modeling of longitudinal covariance structure in functional mapping of quantitative trait loci[END_REF][START_REF] Xiong | A flexible estimating equations approach for mapping function-valued traits[END_REF], testing the capital asset pricing model [START_REF] Sentana | The econometrics of mean-variance efficiency tests: a survey[END_REF], ... have reported success stories of using covariance matrix estimation. For instance, principal component analysis (PCA) applies the eigen-decomposition of the covariance matrix for dimension reduction.

In classification, linear discriminant analysis (LDA), quadratic discriminant analysis and other procedures exploit the inverse of a covariance matrix to compute the classification rule. In finance, portfolio optimization often uses the covariance matrix for minimizing the portfolio risk. More notably, graphical models are especially of interest in the analysis of gene expression data, since it is believed that genes operate in pathways, or networks. Graphical models based on gene expression data can provide a powerful tool for visualizing the relationships of genes and generating biological hypotheses [START_REF] Toh | Inference of a genetic network by a combined approach of cluster analysis and graphical gaussian modeling[END_REF][START_REF] Dobra | Sparse graphical models for exploring gene expression data[END_REF]Schafer and Strimmer, 2005a,b).

Let Y = (Y 1 , ..., Y p ) T be a p-dimensional random vector with the covariance matrix Σ = [Σ ij ] 1≤i,j≤p , where Σ ij is the covariance between Y i and Y j . Suppose that we observe a sample including n observational data points X 1 , ..., X n from a multivariate normal distribution N (0, Σ). The general purpose is to estimate Σ from this sample. This amounts to minimizing the negative log-likelihood function [START_REF] Mardia | Multivariate Analysis[END_REF], Chap. 4 defined by

ℓ(Σ) = n 2 log det Σ + tr(Σ -1 S) + p log 2π , (4.1)
where S = 1/n n i=1 X i X T i is the sample covariance matrix under the assumption that the data is normalized to zero mean.

The problem is that with the increasing abundance of high-dimensional datasets, the sample covariance matrix S becomes an extremely noisy estimator of the covariance matrix, and besides, the number of parameters used to estimate grows quadratically with the number of variables. Intuitively, the most suitable approach to cope with this problem is finding an estimate of the covariance matrix which is as sparse as possible, since the sparsity leads to the effective reduction in the number of parameters. In addition, the sparsity is visualized by the so-called covariance graph [START_REF] Chaudhuri | Estimation of a covariance matrix with zeros[END_REF]. In the covariance graph, each node presents a random variable in a random vector and these nodes are connected by bidirectional edges if the covariances between the corresponding variables are nonzero. Note that the two random variables Y i and Y j are marginally independent if and only if the covariance between Y i and Y j is zero. Hence the zeros in a covariance matrix correspond to marginal independencies between variables, and sparse estimation of the covariance matrix is equivalent to estimating a covariance graph having a small number of edges. Thus the sparsity of the covariance matrix or its inverse is useful to improve the estimation accuracy and/or to explore the structure of the covariance graphical model.

In recent years, in connection with the Big data phenomenon, the sparse covariance matrix estimation (SCME) problem attracts a lot of attention of researchers and constitutes a challenge for the machine learning community. Existing methods for the SCME problems follow two directions. The first direction is to estimate the sparse inverse covariance matrix. A popular method in this direction consists in adding the lasso penalty (ℓ 1 -norm regularization) on the entries of the inverse covariance matrix to the normal likelihood (see e.g. [START_REF] Meinshausen | High dimensional graphs and variable selection with the lasso[END_REF][START_REF] Yuan | Model selection and estimation in the gaussian graphical model[END_REF][START_REF] Banerjee | Model selection through sparse maximum likelihood estimation for multivariate gaussian or binary data[END_REF][START_REF] Friedman | Sparse inverse covariance estimationwith the graphical lasso[END_REF][START_REF] Rothman | Sparse permutation invariant covariance estimation[END_REF][START_REF] Danaher | The joint graphical lasso for inverse covariance estimation across multiple classes[END_REF]Cai et al., 2011;[START_REF] Zhang | Sparse precision matrix estimation via lasso penalized d-trace loss[END_REF]). The second direction is to estimate directly the sparse covariance matrix. In this chapter, we follow the latter one.

A natural way to deal with sparsity in machine learning is using the ℓ 0 -norm in the regularization term that leads to the following mathematical formulation of the SCME problem: min

Σ≻0 log det Σ + tr(Σ -1 S) + λ||Σ|| 0 , (4.2)
where λ is a nonnegative tuning parameter, the notation Σ ≻ 0 means that Σ is symmetric positive definite, and ||Σ|| 0 denotes the ℓ 0 -norm of Σ, i.e. the number of nonzero elements of matrix Σ. It is clear that the SCME problem is much more difficult than the classical estimation of covariance matrix problem. In view of optimization, the SCME problem includes a double difficulties: both the negative log-likelihood function and the ℓ 0 -norm are nonconvex.

Note that the solution to (4.2) is positive definite. This property is crucial for any covariance matrix estimator from both methodological and practical aspects. Positive definite covariance matrices are required in all statistical procedures that use the normal distribution (for example, the principal component analysis, the parametric bootstrap method and the linear or quadratic discriminant analysis). Even some important statistical methods that do not use the normal distribution still need positive definite covariance matrix estimators such as some portfolio optimization methods based on the celebrated Markowitz model.

If S is nonsingular, then the problem (4.2) is equivalent to the following problem: min

Σ δIp log det Σ + tr(Σ -1 S) + λ||Σ|| 0 , (4.3)
for some δ > 0 (see [START_REF] Bien | Sparse estimation of a covariance matrix[END_REF]). Here, I p denotes the p × p identity matrix, and the notation Σ δI p means that Σ -δI p is symmetric positive semidefinite. Note that if S is not full rank, we can replace S with S + ǫI p for some ǫ > 0. In this setting, the observed data reside in a lower dimensional subspace of R p , and the addition of S and ǫI p means the dataset with points that are not completely included in the span of the observed data is enhanced.

Optimization methods involving the ℓ 0 -norm can be divided into three categories according to the way treating the ℓ 0 -norm: convex approximation, nonconvex approximation, and nonconvex exact reformulation. We refer to Le [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF] for an excellent review on exact/approximation approaches to deal with the ℓ 0 -norm. When the objective function (besides the ℓ 0 -term) is convex, convex approximation techniques result to a convex optimization problem which is so far easy to solve. Unfortunately, due to the non-convexity of the negative log-likelihood function, the SCME problem (4.2) remains nonconvex with any approximation convex or nonconvex, of the ℓ 0 -norm. How to deal with the ℓ 0 -norm and how to treat the nonconvexity of the negative log-likelihood loss function are two crucial questions to be studied. Several works have been developed to the SCME problem, but designing an efficient method for it is still a challenge in this research field.

The approaches for solving the SCME problem can be divided into two groups: most of them are included in the first group that we name convex approach. Here one seeks to deter the non-convexity by replacing the negative log-likelihood function with a surrogate convex loss function and using the ℓ 1 -norm or ℓ 2 -norm instead of the ℓ 0 -norm to deal with sparsity (see e. 2012)). The resulting problems are then convex and so solvable, but it is unsurprising that the quality of solutions can be not good. In the second group including [START_REF] Bien | Sparse estimation of a covariance matrix[END_REF]; [START_REF] Lam | Sparsistency and rates of convergence in large covariance matrix estimation[END_REF], the negative log-likelihood function is keeped and the ℓ 1 -regularization is used to deal with sparsity. As mentioned above, due to the non-convexity of the negative log-likelihood function, the resulting problem is still nonconvex. In [START_REF] Bien | Sparse estimation of a covariance matrix[END_REF] the authors applied the minorizationmaximization (MM) approach for solving the resulting problem. Previous works on sparse optimization showed that the use of the ℓ 1 -norm as a convex approximation of the ℓ 0 -norm is not a good way in general (see [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF] and references therein). Instead, the approach approximating the ℓ 0 -norm by a nonconvex continuous function (actually a DC function) is more suitable from theoretical perspective (Le [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF]. The advantage of the ℓ 0 -regularization versus the ℓ 1 -regularization in learning with sparsity has been proved by several machine learning algorithms, however the difficulty caused by the ℓ 0 -norm prevents researchers use the ℓ 0 -regularization.

Our contributions. In this chapter, taking into account the advantages of some DC approximations of the ℓ 0 -norm developed in Le [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF], we seek the most natural way to tackle the sparsity -the ℓ 0 -regularization, and use these DC approximations to design two new models for the SCME problem. More precisely, we maintain the negative log-likelihood function in the problem (4.3) and replace the ℓ 0 -regularization by these DC approximations. The resulting approximate problems are far more difficult than the existing models, but they are DC programs. With our best knowledge, this is the first time in the literature the ℓ 0 -regularization is considered and its nonconvex approximations are used for the SCME problem. Furthermore, we tackle the resulting approximate SCME problem by DC programming and DCA, powerful tools in nonconvex programming framework. Our motivation is based on the fact that DCA is a fast and scalable approach which has been successfully applied to many large-scale (smooth or non-smooth) non-convex programs in various domains of applied sciences, in particular in data analysis and machine learning (see e.g. [START_REF] Collobert | Trading convexity for scalability[END_REF]; [START_REF] Krause | Leveraging the margin more carefully[END_REF]; Le [START_REF] Thi | The DC (difference of convex functions) programming and DCA revisited with DC models of real world nonconvex optimization problems[END_REF]; Le [START_REF] Thi | Exact penalty and error bounds in DC programming[END_REF][START_REF] Thi | New and efficient DCA based algorithms for minimum sum-of-squares clustering[END_REF][START_REF] Thi | Optimization based DC programming and DCA for hierarchical clustering[END_REF][START_REF] Thi | A DC programming approach for feature selection in support vector machines learning[END_REF]Thi et al. ( , 2014a)); Pham Dinh and Le [START_REF] Pham Dinh | Convex analysis approach to D.C. programming: Theory, algorithms and applications[END_REF]Thi ( , 1998Thi ( , 2014)); [START_REF] Hoai | Block clustering based on difference of convex functions (dc) programming and dc algorithms[END_REF]; [START_REF] Liu | Multicategory ψ-learning and support vector machine: Computational tools[END_REF]; Le [START_REF] Thi | Self-organizing maps by difference of convex functions optimization[END_REF] ) and the list of reference on http://lita.sciences.univ-metz.fr/ ~lethi/DCA.html). We note in passing that the MM based approach proposed in [START_REF] Bien | Sparse estimation of a covariance matrix[END_REF] for the SCME problem is also a version of DCA. Constituting the backbone of smooth/nonsmooth nonconvex programming and global optimization, DC programming and DCA address general DC programs of the form

α = inf{F (x) := G(x) -H(x) | x ∈ R n } (P dc ),
where G, H are lower semi-continuous proper convex functions on R n . Such a function F is called a DC function, and G -H a DC decomposition of F while G and H are the DC components of F . The general DCA scheme is a philosophy but not an algorithm.

There is not only one DCA but one family of DCAs for a considered problem. The main feature of DCA is that it is constructed from DC components but not the DC function F itself which has infinitely many DC decompositions, and there are as many DCA as there are DC decompositions. Such decompositions play a critical role in determining the speed of convergence, stability, robustness, and globality of sought solutions. Hence, what is a "good" DC decomposition is a crucial question when developing DCA for a DC program. The design of an efficient DCA for a concrete problem should be based on its special structure.

In this work, we propose two DC formulations of the approximate SCME problem based on two DC decompositions of its objective function. The first results from a natural DC decomposition while the second is introduced to exploit nice effects of DC decompositions. It turns out that the complexity of two corresponding DCA schemes are quite different, because that convex subproblems in the second DCA scheme can be solved by a very inexpensive algorithm. The ratio of gain between the two DCAs in terms of CPU times in our numerical experiments is up to 44 times. Among various existing sparse inducing functions, we are choosing, for implementing our algorithms, the piecewise linear approximation (Capped-ℓ 1 ) [START_REF] Peleg | A bilinear formulation for vector sparsity optimization[END_REF] and the piecewise exponential approximation [START_REF] Bradley | Feature selection via concave minimization and support vector machines[END_REF]. This choice is motivated by the fact that the Capped-ℓ 1 has been proved theoretically in Le [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF] to be the tightness approximation while the piecewise exponential function has been showed to be efficient via the numerical results in numerous works [START_REF] Bradley | Feature selection via concave minimization and support vector machines[END_REF][START_REF] Thi | A DC programming approach for feature selection in support vector machines learning[END_REF][START_REF] Mairal | Incremental majorization-minimization optimization with application to large-scale machine learning[END_REF][START_REF] Thi | Feature selection for linear svms under uncertain data: robust optimization based on difference of convex functions algorithms[END_REF][START_REF] Ong | Learning sparse classifers with difference of convex cunctions algorithms[END_REF] The discontinuity of the ℓ 0 -norm is overcome by using a DC approximation function. Define the step function s : R → R by s(t) = 1 for t = 0 and s(t) = 0 otherwise. Then x 0 = n i=1 s(x i ). The idea of approximation methods is to replace the discontinuous step function by a continuous approximation η α , where α > 0 is a parameter controlling the tightness of approximation. The approximation of the ℓ 0 -norm is then defined by

||Σ|| 0 ≈ i,j η α (Σ ij ). (4.4)
This leads to the approximation SCME problem of (4.3) which takes the form

min Σ∈Ω F (Σ) = log det Σ + tr(Σ -1 S) + λ i,j η α (Σ ij ) , (4.5)
where Ω = {Σ ∈ S p ++ : Σ δI p }.

We consider in this work two approximation functions η α,1 and η α,2 given by η α,1 (t) = 1exp(-α|t|) ∀t ∈ R, (the piecewise exponential concave approximation [START_REF] Bradley | Feature selection via concave minimization and support vector machines[END_REF]) and

η α,2 (t) = min{1, α|t|} ∀t ∈ R,
(the Capped-ℓ 1 [START_REF] Peleg | A bilinear formulation for vector sparsity optimization[END_REF]). In the sequel, for the sake of convenience, we use the common notation η α to design both η α,1 and η α,2 . It has been proved in [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF] that η α is a DC function verifying the Assumption 1 of Le [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF], and with a suitable value of α the Capped-ℓ 1 approximation SCME problem (say, (4.5) when η α = η α,2 ) is equivalent to the original SCME problem (4.3).

We now investigate DCA for solving the nonconvex problem (4.5).

4.2.2

The first DCA scheme for solving the approximation SCME problem (4.5)

The approximation η α can be presented as a DC function:

η α (t) = g(t) -h(t), with g(t) = α|t|, (4.6)
and

h(t) = -1 + α|t| + exp(-α|t|) if η α = η α,1 , h(t) = -1 + max{1, α|t|} if η α = η α,2 . (4.7)
In addition, we see that log det Σ is concave in Σ [START_REF] Boyd | Convex Optimization[END_REF] while tr(Σ -1 S) is convex. Indeed, we have

tr(Σ -1 S) = n i=1 X T i Σ -1 X i ,
and the function X T i Σ -1 X i is convex in Σ [START_REF] Boyd | Convex Optimization[END_REF]. It follows that tr(Σ -1 S) is convex. Consequently, the following DC decomposition of F (Σ) seems to be natural

F (Σ) = G 1 (Σ) -H 1 (Σ), (4.8)
where

G 1 (Σ) = tr(Σ -1 S) + λ i,j g(Σ ij ) + χ Ω (Σ),
and

H 1 (Σ) = -log det Σ + λ i,j h(Σ ij ),
are clearly convex functions. Now, the optimization problem (4.5) can be rewritten as:

min Σ∈R p×p {F (Σ) = G 1 (Σ) -H 1 (Σ)}.
(4.9)

According to the generic DCA scheme, at each iteration l, we have to compute V l ∈ ∂H 1 (Σ l ) and then solve the convex program of the form (P l ), namely

min Σ∈R p×p {F 1 (Σ) := G 1 (Σ) -V l , Σ }.
(4.10)

The computation of V l ∈ ∂H 1 (Σ l ) depends on η α . More precisely, for

η α = η α,1 , V l is computed by V l ij = -(Σ l ) -1 ij + sgn(Σ l ij )λα 1 -exp(-α|Σ l ij |) , (4.11)
where sgn(Σ l ij ) is the sign of Σ l ij . For η α = η α,2 , V l is calculated as

V l ij = -(Σ l ) -1 ij + sgn(Σ l ij )λα if α|Σ l ij | ≥ 1, -(Σ l ) -1 ij otherwise.
(4.12)

For solving the convex problem (4.10), we have to use an iterative method in convex programming. For instance, we use the generalized gradient descent (GGD) algorithm [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF]) that iteratively solves the following problem at each iteration k:

min Σ δIp l k (Σ) := 1 2ν ||Σ -Σ l,k + ν[-(Σ l,k ) -1 S(Σ l,k ) -1 -V l ]|| 2 F + λα||Σ|| 1 , (4.13)
where t > 0 is a suitable stepsize. And then the problem (4.13) can be solved by the alternating direction method of multipliers (ADMM) [START_REF] Boyd | Distributed optimization and statistical learning via the alternating direction method of multipliers[END_REF]. The augmented Lagrangian function of this problem is

L 1 (X, Y, Z) = 1 2ν ||X -Σ l,k + ν[-(Σ l,k ) -1 S(Σ l,k ) -1 -V l ]|| 2 F + λα||Y || 1 + + Z, X -Y + ρ 2 ||X -Y || 2 F ,
and ADMM solves the following problems at each iteration i:

X i+1 = arg min X δIp L 1 (X, Y i , Z i ), Y i+1 = arg min Y ∈R p×p L 1 (X i+1 , Y, Z i ), Z i+1 = X i + ρ(X i+1 -Y i+1 ).
Let S be the elementwise soft-thresholding operator defined by S(A,

B) ij = sgn(A ij )(|A ij | -B ij ) + .
Then, finally, DCA for solving (4.9) can be described as follows.

DCA1

Initialization: Σ 0 δI p , l = 0, τ > 0, and compute δ. repeat 1. Compute V l ∈ ∂H 1 (Σ l ) according to (4.11) (resp. 4.12)) when η α = η α,1 (resp. η α = η α,2 ). 2. Initialization (GGD): Σ l,0 = Σ l , k = 0, and ν, ǫ

2 > 0. repeat -Compute ∆ k = Σ l,k + ν[(Σ l,k ) -1 S(Σ l,k ) -1 + V l ].
-Initialization (ADMM): Y 0 = S ∆ k , λαν , Z 0 = 0, i = 0, and

ǫ 1 , ρ > 0. repeat -Compute X i+1 = UD δ U T where D δ = diag(max(D ii , δ)) and (∆ k + νρY i - νZ i )/(1 + νρ) = UDU T . -Compute Y i+1 = S (X i+1 + Z i /ρ, λα/ρ). -Compute Z i+1 = Z i + ρ(X i+1 -Y i+1 ). -i ← i + 1. until |l k (X i ) -l k (X i-1 )| ≤ ǫ 1 . -Σ l,k+1 = X i . -k ← k + 1. until Σ l,k -Σ l,k-1 1 ≤ ǫ 2 . 3. Σ l+1 = Σ l,k . 4. l ← l + 1. until ||Σ l -Σ l-1 || F ≤ τ ||Σ l-1 || F + 1 or |F (Σ l ) -F (Σ l-1 )| ≤ τ |F (Σ l-1 )| + 1 .
The complexity of one iteration of DCA1 is determined as follows. The computation of the subgradient of H 1 needs O(p 3 ) operations. The combined GGD-ADMM for solving convex subproblem requires O N GGD iter × N ADM M 1 iter × p 3 operations, where N GGD iter and N ADM M 1 iter are the number of iterations of the GGD and ADMM algorithms, respectively. Thus, the complexity of DCA1 is (4.14) where N DCA1 iter denotes the number of iterations of DCA1.

O N DCA1 iter × N GGD iter × N ADM M 1 iter × p 3 ,
We remark that DCA1 might be quite expensive because we have to use two iterative methods for solving the convex subproblem (4.10). This motivates us to consider another DC formulation of the problem (4.5).

4.2.3

The second DCA scheme for solving the approximation SCME problem (4.5) Observing that the convex problem (4.10) is difficult due to the presence of the term tr(Σ -1 S) in G 1 , we seek another DC decomposition by moving tr(Σ -1 S) to the second DC component. We then propose the following second DC formulation of the problem (4.5): min

Σ∈R p×p {F (Σ) = G 2 (Σ) -H 2 (Σ)}, (4.15)
where

G 2 (Σ) = µ 2 ||Σ|| 2 F + λ i,j g(Σ ij ) + χ Ω (Σ), (4.16) and H 2 (Σ) = µ 2 ||Σ|| 2 F -tr(Σ -1 S) -log det Σ + λ i,j
h(Σ ij ), (4.17) are convex functions when µ is large enough. For estimating µ, we state the following lemma.

Lemma 4.1 If µ ≥ 2||S|| 2 δ -3 , then H 2 (Σ) is convex.
Proof : Since the functionlog det Σ + λ i,j h(Σ ij ) is convex and the sum of two convex functions is also convex, it is sufficient to show that µ 2 ||Σ|| 2 Ftr(Σ -1 S) becomes convex, i.e. µ is greater than the spectral radius of the Hessian matrix of Λ(Σ) = tr(Σ -1 S). We have (4.18) where ρ(∇ 2 Λ(Σ)) and ||∇ 2 Λ(Σ)|| 2 are the spectral radius and the spectral norm of the Hessian matrix of Λ(Σ), respectively. The differential dΛ(Σ) of Λ(Σ) is defined by

ρ(∇ 2 Λ(Σ)) ≤ ||∇ 2 Λ(Σ)|| 2 ,
dΛ(Σ) = tr (dΣ -1 )S = tr -Σ -1 (dΣ)Σ -1 S = tr -Σ -1 SΣ -1 dΣ .
Hence, we get the gradient of Λ(Σ) as follows

∇Λ(Σ) = d dΣ Λ(Σ) = -Σ -1 SΣ -1 . (4.19)
We recall the product rule for matrix derivatives

d dΣ f (Σ)g(Σ) = g(Σ) T ⊗ I p d dΣ f (Σ) + (I p ⊗ f (Σ)) d dΣ g(Σ),
where f, g : R p×p → R p×p . Using this product rule, we have

d dΣ ∇Λ(Σ) = -(SΣ -1 ) T ⊗ I p d dΣ Σ -1 -I p ⊗ Σ -1 d dΣ SΣ -1 .
Moreover, by the product rule, we also obtain

d dΣ Σ -1 = -Σ -1 ⊗ Σ -1 and d dΣ SΣ -1 = (Σ -1 ) T ⊗ I p d dΣ S + (I p ⊗ S) d dΣ Σ -1 = (I p ⊗ S) -Σ -1 ⊗ Σ -1 .
Therefore, the Hessian of Λ(Σ) is

∇ 2 Λ(Σ) = Σ -1 S ⊗ I p Σ -1 ⊗ Σ -1 + I p ⊗ Σ -1 (I p ⊗ S) Σ -1 ⊗ Σ -1 .
Using the property (A ⊗ B)(C ⊗ D) = AC ⊗ BD, we finally get Applying DCA on (4.15), we have to compute V l ∈ ∂H 2 (Σ l ) and then solve the convex program of the form (P l ), say min

∇ 2 Λ(Σ) = Σ -1 SΣ -1 ⊗ Σ -1 + Σ -1 ⊗ Σ -1 SΣ -1 . ( 4
Σ δIp {F 2 (Σ) := µ 2 ||Σ|| 2 F + λα Σ 1 -V l , Σ }. (4.21)
The computation of V l is given by, for η α = η α,1 :

V l ij = µΣ l ij + (Σ l ) -1 S(Σ l ) -1 ij -(Σ l ) -1 ij + sgn(Σ l ij )λα 1 -exp(-α|Σ l ij |) , (4.22)
and for η α = η α,2 :

V l ij = µΣ l ij + (Σ l ) -1 S(Σ l ) -1 ij -(Σ l ) -1 ij + sgn(Σ l ij )λα if α|Σ l ij | ≥ 1, µΣ l ij + (Σ l ) -1 S(Σ l ) -1 ij -(Σ l ) -1 ij otherwise. . ( 4

.23)

For solving the convex subproblem (4.21), we use the ADMM algorithm. The augmented Lagrangian function of (4.21) is

L 2 (Σ, X, Y ) = µ 2 ||Σ|| 2 F -V l , Σ + λα||X|| 1 + Y, Σ -X + ρ 2 ||Σ -X|| 2 F .
More specifically, ADMM solves the following problems at each iteration k:

Σ l,k+1 = arg min Σ δIp L 2 (Σ, X k , Y k ) (4.24) X k+1 = arg min X∈R p×p L 2 (Σ l,k+1 , X, Y k ) (4.25) Y k+1 = Y k + ρ(Σ l,k+1 -X k+1 ). (4.26)
Hence, DCA for solving (4.15) can be described as follows.

DCA2

Initialization: Σ 0 δI p , τ > 0, l = 0, and compute δ, µ.

repeat 1. Compute V l ∈ ∂H 2 (Σ l ) according to (4.22) (resp. 4.23)) when η α = η α,1 (resp. η α = η α,2 ). 2. Initialization (ADMM): X 0 = S V l /µ, λα/µ , Y 0 = 0, k = 0, and ρ, ǫ 1 > 0. repeat 1. Compute Σ l,k+1 = UD δ U T where D δ = diag(max(D ii , δ)) and (V l -Y k + ρX k )/(µ + ρ) = UDU T . 2. Compute X k+1 = S Σ l,k+1 + Y k /ρ, λα/ρ . 3. Compute Y k+1 = Y k + ρ(Σ l,k+1 -X k+1 ) 4. k ← k + 1. until Σ l,k -Σ l,k-1 F ≤ ǫ 1 . 3. Σ l+1 = Σ l,k . 4. l ← l + 1. until ||Σ l -Σ l-1 || F ≤ τ ||Σ l-1 || F + 1 or |F (Σ l ) -F (Σ l-1 )| ≤ τ |F (Σ l-1 )| + 1 .
We observe that the convex subproblem (4.21) is easier to solve than the convex subproblem (4.10) in DCA1. It requires only one iterative algorithm (ADMM) which is explicit at each iteration. The complexity of DCA2 is (4.27) where N DCA2 iter and N ADM M 2 iter denote the number of iterations of DCA2 and ADMM, respectively.

O N DCA2 iter × N ADM M 2 iter × p 3 ,

Convergence analysis

Now we will prove the convergence properties of DCA1 and DCA2. For a DC program, the convergence of DCA is guaranteed when the optimal value is finite and the sequence generated by DCA is bounded.

Lemma 4.2

The optimal value of the problem (4.5) is finite.

Proof : Since Σ is positive definite and S is positive semidefinite, then tr(Σ -1 S) is always nonnegative. On the other hand, λ i,j η α (Σ ij ) > 0 and log det Σ ≥ p log δ for all Σ δI p . Hence F (Σ) > p log δ for all Σ δI p . The lemma has been proved. ✷ Lemma 4.3 Let A, B ∈ S p ++ be the positive definite matrices. We have (a)

λ min (A + B) ≥ λ min (A) + λ min (B). (b) λ min (AB) ≥ λ min (A)λ min (B). (c) λ min (A ⊗ B) ≥ λ min (A)λ min (B).
Proof : (a) The part (a) is a consequence of Theorem III.2.1 in Bhatia (1997).

(b) For any positive definite matrix A, we have

A = λ max (A),
where A denotes the operator norm of A. Hence, we get

λ max ((AB) -1 ) = B -1 A -1 ≤ B -1 A -1 = λ max (B -1 )λ max (A -1 ).
Since the eigenvalues of the inverse matrix are the inverse of the eigenvalues, we obtain

1 λ min (AB) ≤ 1 λ min (A)λ min (B) ⇒ λ min (AB) ≥ λ min (A)λ min (B).
(c) First of all, we will show that λ min (A ⊗ I p ) = λ min (A) and λ min (I p ⊗ B) = λ min (B).

Let A = Udiag(a 1 , ..., a p )U T and B = V diag(b 1 , ..., b p )V T be the eigendecomposition of A and B, respectively. We have

A ⊗ I p = (Udiag(a 1 , ..., a p )U T ) ⊗ I p = (U ⊗ I p )(diag(a 1 , ..., a p ) ⊗ I p )(U T ⊗ I p ).
Note that U T ⊗ I p = (U ⊗ I p ) T and diag(a 1 , ..., a p ) ⊗ I p is a p 2 × p 2 diagonal matrix that its diagonal entries are a 1 , ..., a 1 , ..., a p . Then, a 1 , ..., a 1 , ..., a p are also the eigenvalues of A ⊗ I p . It follows that λ min (A ⊗ I p ) = λ min (A). Similarly, we have On the other hand, we have A ⊗ B = (A ⊗ I p )(I p ⊗ B). Then, from the part (b) of the lemma, we obtain

I p ⊗ B = (I p ⊗ V )(I p ⊗ diag(b 1 , ..., b p ))(I p ⊗ V T ). Since I p ⊗ V T = (I p ⊗ V ) T and I p ⊗ diag(b 1 , ..., b p ) is a p 2 × p 2 diagonal
λ min (A ⊗ B) ≥ λ min (A ⊗ I p )λ min (I p ⊗ B) = λ min (A)λ min (B).
This completes the proof of Lemma 4.3. ✷

The convergence properties of DCA1 and DCA2 are given in the following theorem.

Theorem 4.1 Let {Σ l } be the sequence generated by DCA1 (resp. DCA2), we have

(a) {F (Σ l )} is decreasing. (b) {Σ l } is bounded. (c) +∞ l=0 Σ l -Σ l+1 2 F < +∞, and hence Σ l -Σ l+1 F → 0 as l → +∞. (d)
The sequence {Σ l } l has at least one limit point and every limit point of this sequence is a critical point of the problem (4.9) (resp. the problem (4.15)).

Proof : (a) is direct consequence of convergence properties of general DC programs.

(b) First, we will prove that the level set L = {Σ ∈ Ω :

F (Σ) ≤ F (Σ 0 )} is bounded, ∀Σ 0 ∈ Ω. Assume that this level set L is not bounded. Then, there is a sequence { Σk } ⊂ L such that Σk F → +∞ as k → +∞. Let λ 1 ( Σk ), ..., λ p ( Σk ) be the eigenvalues of Σk with λ max ( Σk ) := λ 1 ( Σk ) ≥ ... ≥ λ p ( Σk ) := λ min ≥ δ. We have Σk F ≤ √ pλ max ( Σk ) = √ pλ 1 ( Σk ).
Besides, we have Σk F → +∞ as k → +∞. Thus λ 1 ( Σk ) → +∞ as k → +∞. Since tr(( Σk ) -1 S) is always nonnegative and λ i,j η α ( Σk ) > 0, we have

F ( Σk ) > log det( Σk ) = p i=1 log λ i ( Σk ) ≥ log λ 1 ( Σk ) + (p -1) log δ.
(4.28)

It follows that F ( Σk ) → +∞ as k → +∞. But, the fact that Σk ∈ L implies F ( Σk ) ≤ F (Σ 0 ) for all k. Thus, we have a contradiction, i.e. the level set L = {Σ ∈ Ω :

F (Σ) ≤ F (Σ 0 )} is bounded, ∀Σ 0 ∈ Ω.
Since the sequence {F (Σ l )} is monotonically decreasing, we have {Σ l } ⊆ {Σ ∈ Ω : F (Σ) ≤ F (Σ 0 )} for some Σ 0 ∈ Ω. This and the boundness of the level set L imply (b).

(c) We will show that the first DC components (G 1 and G 2 ) are strongly convex. By the definition of

G 2 := µ 2 ||Σ|| 2 F + λ i,j g(Σ ij ) + χ Ω (Σ)
, it is obviously that G 2 is strongly convex. As for G 1 , it is sufficient to show that tr(Σ -1 S) is strongly convex. From the proof of Lemma 4.1, we have

∇ 2 Λ(Σ) = Σ -1 SΣ -1 ⊗ Σ -1 + Σ -1 ⊗ Σ -1 SΣ -1 .
(4.29)

Applying Lemma 4.3, we get

λ min (∇ 2 Λ(Σ)) ≥ λ min (Σ -1 SΣ -1 ⊗ Σ -1 ) + λ min (Σ -1 ⊗ Σ -1 SΣ -1 ) ≥ 2λ min (Σ -1 SΣ -1 )λ min (Σ -1 ) ≥ 2λ 3 min (Σ -1 )λ min (S) = 2λ min (S) λ 3 max (Σ)
.

As mentioned in Section 4.1, we assumed that S is positive definite. If S is not full rank, we replace S by S + ǫI p for some ǫ > 0. Thus λ min (∇ 2 Λ(Σ)) ≥ 2λ min (S)

λ 3 max (Σ) > 0. So Λ is strongly convex.
Let {Σ l } be the sequence generated by DCA1. If {Σ l } is generated by DCA2, then the part (c) of the theorem will be proved analogously. Recall that Σ l+1 is an optimal solution of the problem min

Σ∈R p×p G 1 (Σ) -V l , Σ , where V l ∈ ∂H 1 (Σ l ).
Then the first-order optimality condition holds at Σ l+1 , i.e, 0 ∈ ∂G 1 (Σ l+1 ) -V l and then

V l ∈ ∂G 1 (Σ l+1 ). (4.30)
Hence, we have

G 1 (Σ l ) ≥ G 1 (Σ l+1 ) + V l , Σ l -Σ l+1 + ρ(G 1 ) 2 Σ l -Σ l+1 2 F , (4.31)
where ρ(G 1 ) is the modulus of the strong convexity of G 1 . Since V l ∈ ∂H 1 (Σ l ), we also have

H 1 (Σ l+1 ) ≥ H 1 (Σ l ) + V l , Σ l+1 -Σ l . (4.32)
Combining (4.31) and ( 4.32), we have

G 1 (Σ l ) ≥ G 1 (Σ l+1 ) + H 1 (Σ l ) -H 1 (Σ l+1 ) + ρ(G 1 ) 2 Σ l -Σ l+1 2 F ⇒G 1 (Σ l ) -H 1 (Σ l ) ≥ G 1 (Σ l+1 ) -H 1 (Σ l+1 ) + ρ(G 1 ) 2 Σ l -Σ l+1 2 F ⇒F (Σ l ) -F (Σ l+1 ) ≥ ρ(G 1 ) 2 Σ l -Σ l+1 2 F .
Moreover, since G 1 is strongly convex, ρ(G 1 ) > 0. Hence, we obtain

Σ l -Σ l+1 2 F ≤ 2 ρ(G 1 ) F (Σ l ) -F (Σ l+1 ) . (4.33)
Let N be a positive integer. Summing (4.33) from l = 0 to N, we get

N l=0 Σ l -Σ l+1 2 F ≤ 2 ρ(G 1 ) F (Σ 0 ) -F (Σ N +1 ) . (4.34)
On the other hand, from the proof of Lemma 4.2 we see that F (Σ N +1 ) ≥ p log(δ). Combining this and (4.34) we get

N l=0 Σ l -Σ l+1 2 F ≤ 2 ρ(G 1 )
F (Σ 0 )p log(δ) .

Taking the limit as N → +∞, we obtain ), respectively. To study the performance of the proposed algorithms, we compare our algorithms with seven standard methods that cover all types of algorithms mentioned in Section 1.

+∞ l=0 Σ l -Σ l+1 2 F < +∞, ( 4
-Methods follow the first direction -estimate the sparse inverse covariance matrix: CLIME (Cai et al., 2011) and SPME [START_REF] Zhang | Sparse precision matrix estimation via lasso penalized d-trace loss[END_REF]. -Methods follow the second direction (estimate directly the sparse covariance matrix), in the first group (the convex approach), i.e. they use surrogate convex loss functions of the negative log likelihood function and the ℓ 1 -norm (resp. ℓ 2 -norm): PDSCE (Rothman, 2012) (resp. PCME [START_REF] Deng | Penalized covariance matrix estimation using a matrix-logarithm transformation[END_REF]). -Methods follow the second direction, in the second group -replace the ℓ 0 -norm by the ℓ 1 -norm: SPCOV1 and SPCOV2 [START_REF] Bien | Sparse estimation of a covariance matrix[END_REF].

Moreover, for the two real applications (classification and portfolio selection) we consider in addition a method using the sample covariance matrix: the Quadratic Discriminant Algorithm (QDA). We also compare the four DCA based algorithms to study the performance of the two DC approximations of the ℓ 0 -norm as well as the two different DC decompositions.

1 2 Σ -S 2 F -τ log det(Σ) + λ Σ 1 , (4.38)
where λ is a tuning parameter and τ > 0 is fixed at a small value. The author developed a blockwise coordinate descent algorithm to compute the solution to (4.38). The PDSCE algorithm is included in PDSCE package of R software.

Penalized Covariance Matrix Estimation using a matrix logarithm transformation (PCME)

The PCME method [START_REF] Deng | Penalized covariance matrix estimation using a matrix-logarithm transformation[END_REF] [START_REF] Bien | Sparse estimation of a covariance matrix[END_REF]. The R package spcov for SPCOV1 and SPCOV2 is available from CRAN (http://cran.r-project.org/).

W ij = 0 if i = j and W ij = 1 |S ij | otherwise)

Experimental setups

The proposed algorithms are implemented in R software and all algorithms are performed on a PC Intel i7 CPU3770, 3.40 GHz of 8GB RAM.

In experiments, we set the stop tolerance τ = 10 -4 for DCA based algorithms. The parameters of GGD and ADMM are chosen as proposed in [START_REF] Bien | Sparse estimation of a covariance matrix[END_REF]. The starting point Σ 0 of DCA is the sample covariance matrix S. The values of parameter λ and approximation parameter of the Capped-ℓ 1 are chosen through a 5-fold cross-validation procedure on training set. The approximation parameter of the piecewise exponential approximation function is chosen α = 5 as suggested in [START_REF] Bradley | Feature selection via concave minimization and support vector machines[END_REF].

The cross-validation procedure is described as follows [START_REF] Bien | Sparse estimation of a covariance matrix[END_REF]. For A ⊆ {1, ..., n}, let andA c i denotes the component of A. We divide {1, ..., n} into 5 subsets, A 1 , ..., A 5 , and then compute (4.41) where Σλ (S A c i ) is an estimate of the covariance matrix Σ with the parameter λ and S A c i , and ℓ Σλ (S A c i );

S A = |A| -1 i∈A X i X T i ,
f (λ) = 1 5 5 i=1 ℓ Σλ (S A c i ); S A i ,
S A i = -log det Σλ (S A c i ) -tr Σλ (S A c i ) -1
S A i . Finally, we choose λ = arg max λ f (λ).

Numerical results on synthetic datasets

We evaluate the performance of DCA1-CaP, DCA2-CaP, DCA1-PiE and DCA2-PiE on four synthetic datasets. We generate X = [X 1 , ..., X n ] from a multivariate normal distribution N p (0, Σ), where Σ is a sparse symmetric positive definite matrix. We consider three types of covariance graphs and a moving average model as follows [START_REF] Bien | Sparse estimation of a covariance matrix[END_REF]:

Cliques model: Σ = diag(Σ 1 , ..., Σ 5 ), where Σ 1 , ..., Σ 5 are dense matrices.

Hubs model: Σ = diag(Σ 1 , ..., Σ 5 ) again, however each submatrix Σ k is zero except elements in the last row and the last column. This corresponds to a graph with five connected components each of which has all nodes connected to one particular node.

Random model: in this model, we take Σ ij = Σ ji to be nonzero with the probability 0.02, independently of other elements.

First-order moving average model: we generate Σ i,i-1 = Σ i-1,i to be nonzero for i = 2, ..., p.

In the first three cases, the nonzero entries of matrix Σ are randomly drawn in the set {+1, -1}. In the moving average model, all nonzero values are set to be 0.4. In this experiment, for each covariance model, we generate 10 training sets with size n = 200, p = 100.

To evaluate the performance of each method, we consider three loss functions which are the root-mean-square error (RMSE), the entropy loss (EN), and the Kullback-Leibler loss (KL), respectively.

RMSE = || Σ -Σ|| F /p, EN = -log det( ΣΣ -1 ) + tr( ΣΣ -1 ) -p, KL = -log det( Σ-1 Σ) + tr( Σ-1 Σ) -p,
where Σ is a sparse estimate of the covariance matrix Σ.

The experimental results on synthetic datasets are given in Table 4.1. In this Table, the average of root-mean-square error (RMSE), entropy loss (EN), Kullback-Leibler loss (KL), number of nonzero elements (NZ), CPU time in seconds, and their standard deviations over 10 samples are reported.

We observe from Table 4.1 that in the cliques model, DCA2-CaP gives the lowest rootmean-square error while DCA2-PiE gives the best results in terms of the entropy loss and Kullback-Leibler loss. We further note that in terms of the sparsity, the number of the nonzero elements, the DCA based algorithms achieve much better performances than the other six approaches.

For the hubs and random models, the best and the second best performing methods with respect to the losses and the sparsity are DCA1-PiE and DCA2-PiE, respectively. Conversely, when using the moving model, DCA1-CaP outperforms the others with the loss and the sparsity measures. However, it provides slightly smaller improvement than DCA2-CaP. Here, notably the results obtained with DCA2-CaP and DCA2-PiE are still superior to that of SPCOV1, SPCOV2, CLIME, PCME, SPME, and PDSCE.

Regarding the training time, DCA2-CaP and DCA2-PiE are remarkably faster than the other algorithms in the nonconvex approach keeping the negative log-likelihood function, say DCA1-CaP, DCA1-PiE, SPCOV1 and SPCOV2, as well as the two methods estimating the inverse covariance matrix (CLIME and PCME). This can be explained by the fact that DCA2-CaP and DCA2-PiE lead to the sequences of convex problems which are easily solved by an explicit ADMM algorithm. As for the convex based approaches SPME and PDSCE, not suprisingly, they are faster than the DCA based algorithms but achieve much worse performances on all losses as well as sparsity. The ratio of gain of DCA varies from 1.1 to 168.6 in terms of losses and from 2 to 20.6 in terms of sparsity.

Numerical results on real datasets

We illustrate the use of the sparse covariance matrix estimation problem via two real applications: a classification problem and a portfolio optimization problem of stock data. These applications require an estimate of the covariance matrix.

Sparse quadratic discriminant analysis

Let X be an n × p training data matrix with observations x i (i = 1, ..., n) on the rows and features on the columns. We assume that the n i observations within the kth class C k are normal distributed N (µ k , Σ k ). We denote the prior probability of the kth class by π k . The quadratic discriminant function is

δ k (x) = - 1 2 log det Σ k - 1 2 (x -µ k ) T Σ -1 k (x -µ k ) + log π k . (4.42)
Then the predicted class for a new observation x is arg max k δ k (x). The decision boundary between each pair of classes k and l is described by a quadratic equation {x :

δ k (x) = δ l (x)}.
In practice we do not know π k , µ k , Σ k , and will need to estimate them using the training data:

π k = n k /n, µ k = 1/n k x i ∈C k x i
, and a natural way to estimate the Σ k is via maximum likelihood. Let

S k = 1/n k x i ∈C k (x i -µ k )(x i -µ k )
T is the sample covariance matrix for the class k, the negative log likelihood for the data takes the form (up to a constant) We propose to estimate Σ 1 , ..., Σ Q by minimizing the penalized negative log likelihood min

1 2 Q k=1 n k log det Σ k + tr(Σ -1 k S k ) . (4.43)
Σ k ≻0,k=1,...,Q Q k=1 log det Σ k + tr(Σ -1 k S k ) + λ k ||Σ k || 0 , (4.44)
where λ 1 , ..., λ Q are nonnegative tuning parameters. We refer to this classification method as sparse quadratic discriminant analysis.

Solving the problem (4.44) respect to Σ 1 , ..., Σ Q can be separated into Q independent subproblems of the same form. This leads to a potentially massive reduction in computational complexity. Recently, [START_REF] Sun | The application of sparse estimation of covariance matrix to quadratic discriminant analysis[END_REF] has also proposed the sparse quadratic discriminant analysis using a lasso penalty on the entries of the inverse covariance matrices. This work can be viewed as applications of the methods proposed in [START_REF] Rothman | Positive definite estimators of large covariance matrices[END_REF].

In our work, we directly estimate the covariance matrices Σ 1 , ..., Σ Q by using DCA1-CaP, DCA2-CaP, DCA1-PiE and DCA2-PiE. To study the performance of the proposed algorithms, we use SPCOV1, SPCOV2, CLIME, PCME, SPME and PDSCE to estimate Σ 1 , ..., Σ Q . We also compare with the quadratic discriminant analysis (QDA) replacing Σ k in (4.42) by the sample covariance matrix S k . Note that if S k is singular, then we replace it by S k + ǫI p , where ǫ is chosen through a 5-fold cross-validation on a set of candidates E = {10 -4 , ..., 10 -8 }.

For the experiment, we evaluate the proposed algorithms on three datasets: US Postal Service (USPS) dataset and two datasets from UCI Machine Learning Repository (Ionosphere and Waveform 2).

The US Postal Service task is still one of the most widely used reference dataset for handwritten digit recognition. Here we focus on a difficult sub-task, that of distinguishing handwritten 3s and 8s. All images are of size 16 × 16 pixels. There are 658 threes and 542 eights in the training set, and 166 test samples for each. In our experiment we name this data as 3s and 8s dataset. It can be downloaded at http://statweb.stanford.edu/ ~tibs/ElemStatLearn/data.html. A random selection is showed in Figure 4.1. We filter the data by replacing each non-overlapping 2 × 2 pixel block by its average. This reduces the dimension of the feature space from 256 to 64.

The detailed information of Johns Hopkins University Ionosphere and Waveform 2 datasets is summarized in Table 4.2. We use the cross-validation scheme to validate the performance of various approaches on these two datasets. The dataset is split into a training set containing 2/3 of the samples and a test set containing 1/3 of the samples. This process is repeated 10 times, each with a random choice of training set and test set. The parameter λ 1 , ..., λ Q are chosen via 5-fold cross-validation. The computational results are reported in . In Table 4.3, the classification results and the training time in second for 3s and 8s dataset are given, and we notice that the testing and training errors obtained with DCA2-CaP are the lowest.

In Table 4 we reported the average percentage of the testing and training errors, the training time in seconds as well as the standard deviations of Ionosphere and Waveform 2 datasets over 10 training/test set splits. We observe that, on the Ionosphere dataset, DCA2-CaP and DCA2-PiE outperform DCA1-CaP, DCA1-PiE, SPCOV1, QDA, CLIME, PCME, SPME and PDSCE in terms of the testing error. Notably in this dataset, SP-COV2 is not able to perform since there exists some zero elements in the sample covariance matrices. In the Waveform 2, the DCA based algorithms give slightly better testing error than the other approaches, and DCA1-PiE has the smallest testing error. However, QDA gives the best training error on both Ionosphere and Waveform 2 datasets. As for the training time, DCA2-CaP is the fastest on the Ionosphere dataset while DCA2-PiE is the best on the Waveform 2 dataset, and we further note that both of them are significantly faster than the other approaches on these two datasets. On the 3s and 8s dataset, DCA2-CaP and DCA2-PiE run slower than PDSCE but faster than the seven remaining approaches.

Portfolio optimization

In this section, we apply the sparse covariance matrix estimation problem on an application of portfolio optimization. The celebrated Markowitz portfolio selection problem [START_REF] Markowitz | Portfolio selection[END_REF] in finance is to construct the optimal mean-variance efficient portfolios by minimizing the following quadratic optimization problem:

min w∈R p w T Σw : w T µ = µ * ; w T e = 1; w ≥ 0 , (4.45)
where e denotes the p-dimensional vector of ones and µ * is the expected rate of return that is required on the portfolio. When short selling is allowed, the constraint w ≥ 0 in the problem (4.45) can be removed.

In the recent literature, many works aim to find an estimate of the covariance matrix to improve its portfolio strategy (see e.g. Ledoit andWolf (2003, 2004); Jagannathan and Ma ( 2003

We expect that an accurate covariance matrix estimate will lead to a better portfolio strategy. We use eleven different approaches to estimate Σ and to obtain w: our approaches (DCA1-CaP, DCA2-CaP, DCA1-PiE, DCA2-PiE), SPCOV1, SPCOV2, CLIME, PCME, SPME, PDSCE and the sample covariance matrix S.

We consider a stock dataset used in [START_REF] Deng | Penalized covariance matrix estimation using a matrix-logarithm transformation[END_REF] and it is also available from Yahoo!Finance (http://finance.yahoo.com). This dataset is the weekly returns of 30 components of the Dow Jones Industrial Index. The dataset of adjusted close prices of the weekly returns were extracted in the past three and a half years from January 8, 2007 to June 28, 2010. The dataset is divided in the same way as in [START_REF] Deng | Penalized covariance matrix estimation using a matrix-logarithm transformation[END_REF]. The first 50 observations of the weekly returns data is the training set, the next 50 observations is the tuning set, and the remaining data is the test set. The performance of a portfolio w is measured by the realized return

R(w) = x∈Xts w T x, (4.47)
the realized risk σ(w) = w T S ts w, (4.48) and the Sharpe ratio

S(w) = R(w) σ(w) , (4.49)
where X ts is the test set and S ts is the sample covariance matrix of X ts . In Table 5, the realized return, the realized risk and the Sharpe ratio of the eleven comparative methods DCA1-CaP, DCA2-CaP, DCA1-PiE, DCA2-PiE, SPCOV1, SPCOV2, CLIME, PCME, SPME, PDSCE and S are presented. These results indicate that DCA2-PiE is the best one. Although the realized risk produced by this approach is a little bit greater than DCA2-CaP, its realized return achieves the highest value. Hence, it follows that the largest Sharpe ratio is achieved by DCA2-PiE. We also notice that DCA2-PiE and DCA2-CaP are respectively the fastest and the second fastest in training time.

The next experiment, we evaluate the performance of portfolios in different periods. This experiment is the same as [START_REF] Deng | Penalized covariance matrix estimation using a matrix-logarithm transformation[END_REF] which is reviewed as follows: the first 50 observations of the weekly returns is the training set, the next 50 observations is the tuning set, and the third 50 observations of the weekly returns is the test set. By changing the starting week during the period from January 8, 2007 to August 20, 2007, we have the 33 different consecutive test periods. The realized return, the realized risk and the Sharpe ratio are calculated for each test period using the optimal portfolio w based on the corresponding training set. The computational results are shown in .

The results in Figures 4.2 and 4.3 show that the portfolios created by DCA2-CaP and DCA2-PiE have higher realized returns than the other approaches and these two methods also provide comparable realized risks with the others in almost all periods. Moreover, the results in Figure 4.4 show that the Sharpe ratios of the portfolios produced by DCA2-CaP and DCA2-PiE are larger than SPCOV1, SPCOV2, CLIME, PCME, SPME, PDSCE and S. Thus, it is convincing to conclude that DCA2-CaP and DCA2-PiE result in better portfolio strategies.

Conclusion

In this chapter, we have investigated DC approximation approaches and DCA for solving the sparse covariance matrix estimation problem using the ℓ 0 -norm. We propose two DC formulations for the approximation SCME problem and develop four DCA based algorithms, by using two appropriate DC approximation functions of the ℓ 0 -norm. The robustness and the effectiveness of our DCA based algorithms have been demonstrated through the computational results on both simulated and real datasets. The nice effect of DC decomposition has been exploited: the second DC decomposition seems to be very suitable since it leads to an efficient, fast and scalable DCA scheme. In the experimental results, DCA2-CaP and DCA2-PiE have obtained the best performance in terms of most of comparison criteria, and have taken the shortest time for training. The utility of our approaches have been illustrated via two important applications: classification by a quadratic discriminant function and portfolio optimization. Their superiority on seven state-of-the-art algorithms are proved via various numerical experiments.

We are convinced that the approaches developed in this chapter bring to researchers and practitioners new and efficient methods to treat an important and difficult problem that can be used to many applications in various domains.

As a part of future work, we plan to study more extensive applications of the sparse covariance matrix estimation problem. The need to select groups of variables arises in many statistical modeling problems and applications. In this chapter, we introduce a new regularization using the ℓ p,0 -norm for enforcing group sparsity. Using a DC (Difference of Convex functions) approximation of the ℓ p,0norm, we show that the approximate problem is equivalent to the original problem with suitable parameters. Considering two equivalent formulations of the approximate problem we develop DC programming and DCA (DC Algorithm) for solving them. When p = 1 (resp. p = 2), our algorithms include ℓ 1 -perturbed algorithm (resp. ℓ 2,1 -perturbed algorithm) and reweightedℓ 1 algorithm (resp. reweighted-ℓ 2,1 algorithm). It turns out that, among ℓ p,0 regularizations, the ℓ 1,0 is the most interesting regularization with several advantages in both theoretical and computational aspects. As applications, we implement the proposed algorithms for group variable selection in optimal scoring problem and estimation of multiple covariance matrices. In the first application, sparsity is obtained by using the ℓ p,0 -regularization that selects the same features in all discriminant vectors. The resulting sparse discriminant vectors provide a more interpretable low-dimensional representation of data. In the second application multiple covariance matrices sharing some common structures such as the locations or weights of non-zero elements, we combine the ℓ 0 -norm and the ℓ p,0 -norm for enforcing sparsity on each covariance matrix and across multiple covariance matrices, respectively. The experimental results on both simulated and real datasets demonstrate the efficiency of the proposed algorithms.

Introduction

Variable selection plays an important role in many applications and has drawn increased attention from many researchers. In the literature, the use of sparsity-inducing norms is a powerful technique for variable selection in the high-dimensional settings. In this direction, the ℓ 0 -norm has been studied extensively on both theoretical and practical aspects for individual variable selection in many practical problems. However, when the data possesses certain group structures, we are naturally interested in selecting important groups of variables rather than individual ones. For instance, in multi-factor analysis of variance, a factor with several levels may be expressed through a group of dummy variables. In nonparametric additive regression, each component can be represented by a linear combination of a set of basis functions. In genomic data analysis, the correlations between genes sharing the biological pathway can be high. Hence these genes should be considered as a group. In such cases, the selection of important factors/nonparametric components/groups of genes amounts to the selection of groups of variables. In recent years, there are many works based on regularization methods for group variable selection in various application domains such as machine learning, statistics, computational biology, signal processing, and other related areas. In this chapter, we introduce a natural approach for enforcing group sparsity by using the ℓ p,0 -regularization with p ≥ 1.

We define the step function s : R → R by s(t) = 1 if t = 0 and s(t) = 0 otherwise. Assume that x = (x 1 , ..., x d ) ∈ R d is partitioned into J non-overlapping groups x 1 , ..., x J , then the ℓ p,0 -norm of x is defined by

x p,0 = J j=1 s( x j p ).
The ℓ p,0 -regularized problem takes the form:

min f (x, y) + λ x p,0 : (x, y) ∈ K ⊂ R d × R m , (5.1)
where λ is a nonnegative tuning parameter.

Let us mention some important applications of group variable selection corresponding to the model (5.1).

Group variable selection in linear regression: Given n observations (x i , y i ), i = 1, ..., n, where y i is the response variable, x i = (x ′ i1 , ..., x ′ iJ ) ′ is the corresponding covariates with J groups of predictors and x ij is the d j -dimensional sub-covariate vector, j = 1, ..., J. Let X j = (x ′ 1j , ..., x ′ nj ) ′ be the n × d j design matrix corresponding to the j-th group and β j be the vector of the regression coefficients in the j-th group. The problem of selecting the important covariates and estimating the corresponding coefficient vector takes the form of (5.1):

min 1 2n y - J j=1 X j β j 2 2 + λ J j=1 s( β j p ) . (5.2)
Multi-task feature selection: Let T be the number of tasks. For the j-th task, the training set D j consists of n j labeled data points in the form of ordered pairs (x j i , y j i ), i = 1, ..., n j , with x j i ∈ R d and its corresponding output y j i ∈ R. Multi-task learning aims to estimate T functions f j (x) : R d → R, j = 1, ..., T , which well fit the data and are statistically predictive. Here, we focus on linear functions, i.e., f j (x) = w T j x + b j . The multi-task feature selection problem takes the form of (5.1):

min T j=1 n j i=1 L(y j i , w T j x j i + b j ) + λ |W p,0 , (5.3)
where L denotes the loss function, W = [w 1 , ..., w T ] ∈ R d×T and W p,0 denotes the ℓ p,0 -norm of the matrix W , i.e., W p,0 = d j=1 s( w j p ) with w j is the j-th row of W . In this problem, each row of W is regarded as a group. Note that the group variable selection problem in multiclass SVMs is a special case of the problem (5.3) where L is the hinge loss function given by

L(y j i , w T j x j i + b j ) = T l=1,l =j max 1 -(w T j x j i -w T l x j i + b j -b l ), 0 .
Group sparse principal component analysis (PCA): Let X ∈ R n×d be a data matrix which comprises n observations x i ∈ R d , where d is the number of features. We assume that the features have been centered to have mean 0. Denote by I k the k × k identity matrix. [START_REF] Zou | Sparse principal component analysis[END_REF] has transformed the PCA problem into a regression type optimization problem. min

A∈R p×k X -XAA T 2 F : A T A = I k , (5.4)
where the columns of A which minimize (5.4) are referred as the first k loading vectors of PCA. One way to obtain sparse loading vectors is imposing the ℓ p,0 penalty on the regression coefficients.

min

A,B∈R d×k X -XBA T 2 F + λ B p,0 : A T A = I k , (5.5)
where the columns β 1 , ..., β k of B correspond to the required sparse loading vectors.

Group sparse Fisher linear discriminant analysis (LDA): Let {(x i , y i ) : i = 1, ..., n} be a set of labeled training data with observation vector x i ∈ R d and label y i ∈ {1, ..., C}.

The original LDA formulation is known as the Fisher linear discriminant analysis [START_REF] Fisher | The use of multiple measurements in taxonomic problems[END_REF]. Fisher criterion aims to find a linear transformation W ∈ R d×L that maps the data in the d-dimensional space to a L-dimensional space (L ≤ C -1), in which the between-class variance is maximized while the within-class variance is minimized, i.e., max

W ∈R d×L tr (W T Σ w W ) -1 (W T Σ b W ) , (5.6)
where Σ b and Σ w are the between-class covariance matrix and the within-class covariance matrix, respectively. We use the ℓ p,0 -regularization to select the same features in all discriminant vectors w 1 , ..., w L which are the columns of W . The resulting sparse discriminant vectors can provide a more interpretable low-dimensional representation of data. The regularized problem takes the form of (5.1):

min

W ∈R d×L -tr (W T Σ w W ) -1 (W T Σ b W ) + λ W p,0 .
(5.7)

Joint sparse compressed sensing: Compressed sensing aims to recover the sparse signal w from a measurement vector b = Aw for a given matrix A. Compressed sensing can be extended to the multiple measurement vector in which the signals are represented as a set of jointly sparse vectors sharing a common set of the nonzero elements [START_REF] Cotter | Sparse solutions to linear inverse problems with multiple measurement vectors[END_REF][START_REF] Chen | Theoretical results on sparse representations of multiplemeasurement vectors[END_REF][START_REF] Sun | Efficient recovery of jointly sparse vectors[END_REF]. Joint compressed sensing considers the reconstruction of the signal represented by a matrix W , which is given by a dictionary A and a multiple measurement vector B such that B = AW . Since there usually exists noise in the data, the joint sparse compressed sensing can be formulated as the group sparse optimization problem of the form (5.1):

min W AW -B 2 F + λ W p,0 .
(5.8)

Other applications of group variable selection include multiple graphical models [START_REF] Danaher | The joint graphical lasso for inverse covariance estimation across multiple classes[END_REF], multi-task reinforcement learning [START_REF] Calandriello | Sparse multi-task reinforcement learning[END_REF], etc.

Existing works considered group variable selection as a natural extension of variable selection. The first approach, named the group Lasso [START_REF] Yuan | Model selection and estimation in regression with grouped variables[END_REF], is closely connected to the Lasso (ℓ 1 -norm) approximation of the ℓ 0 -norm. Works in this direction include ℓ ∞,1 -norm (Liu et al., 2009a;[START_REF] Quattoni | An efficient projection for ℓ ∞,1 -regularization[END_REF][START_REF] Zhang | Probabilistic multi-task feature selection[END_REF] and the ℓ 2,1 -norm which was widely used for group variable selection in multi-task learning [START_REF] Argyriou | Convex multi-task feature learning[END_REF][START_REF] Bi | An improved multi-task learning approach with applications in medical diagnosis[END_REF][START_REF] Liu | Multi-task feature learning via efficient ℓ 2,1 -norm minimization[END_REF][START_REF] Obozinski | Multi-task feature selection[END_REF][START_REF] Obozinski | Joint covariate selection and joint subspace selection for multiple classification problems[END_REF][START_REF] Zhang | Probabilistic multi-task feature selection[END_REF][START_REF] Nie | Efficient and robust feature selection via joint ℓ 2,1 -norms minimization[END_REF][START_REF] Lan | Joint sparse representation robust feature-level fusion for multi-cue visual tracking[END_REF], multiclass SVMs [START_REF] Blodel | Block coordinate descent algorithms for large-scale sparse multiclass classification[END_REF], PCA [START_REF] Kha | Joint group sparse pca for compressed hyperspectral imaging[END_REF], Fisher LDA [START_REF] Gu | Linear discriminant dimensionality reduction[END_REF], optimal scoring [START_REF] Leng | Sparse optimal scoring for multiclass cancer diagnosis and biomarker detection using microarray data[END_REF][START_REF] Merchante | An efficient approach to sparse linear discriminant analysis[END_REF], and compressed sensing [START_REF] Sun | Efficient recovery of jointly sparse vectors[END_REF]. In general, these convex regularization methods are not efficient, they may be not selection consistent and tend to select non important groups in the model. The second approach deals with nonconvex approximation and has been developed for the ℓ 2,0 -norm. More precisely, DC (difference of convex functions) approximation approaches based on the smoothly clipped absolute deviation penalty (SCAD) [START_REF] Fan | Variable selection via nonconcave penalized likelihood and its oracle properties[END_REF] and the minimax concave penalty (MCP) [START_REF] Zhang | Nearly unbiased variable selection under minimax concave penalty[END_REF] have been studied for the ℓ 2,0 -norm in linear regression problems (see e.g. [START_REF] Lee | Sparse optimization for nonconvex group penalized estimation[END_REF][START_REF] Huang | A selective review of group selection in high-dimensional models[END_REF][START_REF] Wang | Group scad regression analysis for microarray time course gene expression data[END_REF][START_REF] Wei | Group coordinate descent algorithms for nonconvex penalized regression[END_REF]). These works have proved that these DC approximation approaches for the ℓ 2,0 -norm are more efficient than the methods using the ℓ 2,1 -norm.

In this chapter, we investigate DC approximation approaches for the general case, i.e. the ℓ p,0 -norm with p ≥ 1. We consider the problem (5.1), where K is a compact polyhedral convex set in R d × R m and f is a finite DC function on R d × R m . The chapter makes the following contributions.

Firstly, basing on the piecewise linear function (called Capped-ℓ 1 ) introduced in [START_REF] Peleg | A bilinear formulation for vector sparsity optimization[END_REF], we approximate the ℓ p,0 -norm by a DC function. We prove that, with suitable parameters, the nonconvex approximate problem is equivalent to the original problem (5.1). This result gives an important mathematical foundation for our approximation method.

Secondly, we develop solution methods based on DC programming and DCA (DC Algorithms), a powerful technique in nonconvex optimization [START_REF] Thi | The DC (difference of convex functions) programming and DCA revisited with DC models of real world nonconvex optimization problems[END_REF][START_REF] Pham Dinh | Convex analysis approach to D.C. programming: Theory, algorithms and applications[END_REF], for solving the approximate problem. Considering two equivalent formulations of the approximate problem we propose two DCA. These two DCA schemes can be viewed as an ℓ p,1 -perturbed algorithm and a reweighted-ℓ p,1 algorithm. When p = 1 and p = 2, our algorithms include ℓ 2,1 -perturbed algorithm, reweighted-ℓ 2,1 algorithm, ℓ 1 -perturbed algorithm, and reweighted-ℓ 1 algorithm with different weights on groups and the same weight on each group.

Among ℓ p,0 -regularizations, we show that the ℓ 1,0 -regularization is the most interesting with several useful properties from both theoretical and computational aspects. The DCA schemes for solving the resulting approximate problem iteratively solve an ℓ 1perturbed/reweighted-ℓ 1 problem which can be separated into independent sub-problems in many applications. This interesting feature makes our proposed approach very efficient in terms of computational complexity.

Finally, as applications, we consider the problem of group variable selection in optimal scoring and estimation of multiple covariance matrices. We also perform a careful empirical experiment to study the performance of the proposed approaches.

The rest of the chapter is organized as follows. In Section 5.2, we present the approximate problems and show that these problems are equivalent to the original problem. We illustrate how to apply DCA to solve the approximate problems in Section 5.3. The application of the proposed algorithms for group variable selection in optimal scoring is described in Section 5.4 while the application of the proposed algorithms for group variable selection in estimation of multiple covariance matrices is described in Section 5.5. Section 5.6 concludes the chapter.

Throughout the chapter, for vectors u, v ∈ R n , the inner product of u and v is defined as

u, v = n i=1 u i v i . For every p ≥ 1, the ℓ p -norm of vector u is u p = ( n i=1 |u i | p ) 1
p . In addition, if u is partitioned into J non-overlapping groups u 1 , ..., u J , we recall that the ℓ p,0 -norm of u is defined as

u p,0 = J j=1 s( u j p ),
where s is the step function defined above. Similarly, the ℓ p,1 -norm of u is defined as

u p,1 = J j=1 u j p .
We denote the vector ( u 1 p , ..., u J p ) by |u J | p . For A ∈ R n×m , the Frobenius norm of A is given by ||A|| F = i,j A 2 ij .

DC approximate problems and the link with the original problem

We consider the approximate problem of (5.1) which takes the form min

F p α (x, y) := f (x, y) + λ J j=1 η α ( x j p ) : (x, y) ∈ K , (5.9)
where η α (t) = min{1, α|t|} is the Capped-ℓ 1 function [START_REF] Peleg | A bilinear formulation for vector sparsity optimization[END_REF], and α is a tuning parameter such that η α (t) approximates the step function s(t) as α tends to +∞.

Since x j ∞ ≤ x j p ≤ x j 1 ∀p ≥ 1 and η α is increasing on [0; +∞), we get

η α ( x j ∞ ) ≤ η α ( x j p ) ≤ η α ( x j 1 ) ≤ s( x j p ).
( 5.10) This shows that, with the same parameter α, η α ( . 1 ) is the closest to the step function s. We also consider another equivalent form of the problem (5.9) as follows:

min F α (x, y, z) := f (x, y) + λ J j=1 η α (z j ) : (x, y, z) ∈ K p , (5.11)
where K p = {(x, y, z) : (x, y) ∈ K, x j p ≤ z j , j = 1, ..., J}. Indeed, the problems (5.9) and (5.11) (5.11) then (x * , y * ) is a global solution to (5.9).

Proof : Since η α is an increasing function on [0, +∞), we have

F α (x, y, z) ≥ F α (x, y, |x J | p ) = F p α (x, y) ∀(x, y, z) ∈ K p .
Then the conclusion on global solutions is trivial. The result on local solutions can be deduced from the following remarks. Let a = max j d j ≥ 1, where d j is the number of variables of the j-th group. If 1 ≤ p < 2, we have

| x j p -(x * ) j p | ≤ x j -(x * ) j p ≤ d 1/p-1/2 j x j -(x * ) j 2 .
(5.12)

Combining d

1/p-1/2 j ≤ a and (5.12), we obtain

| x j p -(x * ) j p | ≤ a x j -(x * ) j 2 ∀j = 1, ..., J.
(5.13) 123

If p ≥ 2, we have

| x j p -(x * ) j p | ≤ x j -(x * ) j p ≤ x j -(x * ) j 2 .
(5.14) From ( 5.13)-( 5.14) and a ≥ 1, we have

|x J | p -|x * J | p 2 ≤ a x -x * 2 ∀p ≥ 1. Therefore, if (x, y) ∈ B((x * , y * ), δ/ √ a 2 + 1) then (x, y, |x J | p ) ∈ B((x * , y * , |x * | p ), δ)
. Moreover, if (x, y, z) ∈ B((x * , y * , z * ), δ) then (x, y) ∈ B((x * , y * ), δ). The proof is the complete. ✷

In this section, we introduce the following assumption:

Assumption 1 K is a compact polyhedral convex set. By Proposition 5.1, without the generality, we can assume that K p is a compact convex set.

Proposition 5.2 Under the assumption (1), with p = +∞. Then, there exists α 0 such that the approximate problem (5.9) is equivalent to the original problem (5.1) for all α > α 0 .

Proof : Since the ℓ ∞ -norm is polyhedral convex function, K ∞ = {(x, y, z) : (x, y) ∈ K, x j ∞ ≤ z j , j = 1, ..., J} is also a compact polyhedral convex set. We notice that the (5.11) is an approximate problem of the following problem including the ℓ 0 penalty on the vector z. min f (x, y) + λ J j=1 s(z j ) : (x, y, z) ∈ K ∞ .

(5.15)

Following the conclusion after Proposition 3 in Le [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF], there exists α 0 such that the problem (5.11) and (5.15) are equivalent for all α > α 0 . Moreover, we have

f (x, y) + λ J j=1 s(z j ) ≥ f (x, y) + λ J j=1 s( x j ∞ ) ∀(x, y, z) ∈ K ∞ .
It easily follows that the problem (5.15) is equivalent to the problem (5.1). By Proposition 5.1, we have the equivalence between the problem (5.11) and the problem (5.9). Hence, the approximate problem (5.9) is equivalent to the original problem (5.1). Thus, Proposition 5.2 is proved. ✷

For p = +∞, we have proved that the approximate problem (5.9) is equivalent to the problem (5.1) for all α > α 0 . We now have the general result for all p ≥ 1.

Proposition 5.3 Under the assumption (1), there exists α 0 such that the approximate problem (5.9) is equivalent to the original problem (5.1) for all α > α 0 and p ≥ 1.

Proof : From the inequality (5.10), we have

η α ( x j ∞ ) ≤ η α ( x j p ) ≤ s( x j p ).
(5.16)

Hence, we obtain

F ∞ α (x, y) ≤ F p α (x, y) ≤ f (x, y) + λ J j=1 s( x j ∞ ) ∀(x, y) ∈ K.
(5.17)

For p = +∞, by Proposition 5.2, the problems (5.9) and (5.1) are equivalent for all α > α 0 . Hence, let (x * , y * ) be a common optimal solution, we have

F ∞ α (x * , y * ) = F p α (x * , y * ) = f (x * , y * ) + λ J j=1 s( (x * ) j ∞ ) = f (x * , y * ) + λ J j=1 s( (x * ) j p ).
Combining with (5.17), for all (x, y) ∈ K, we have

f (x * , y * ) + λ J j=1 s( (x * ) j p ) = F p α (x * , y * ) ≤ F p α (x, y) ≤ f (x, y) + λ J j=1 s( x j p ).
Therefore, we can deduce that the approximate problem (5.9) is equivalent to the original problem (5.1). This completes the proof of the Proposition 5.

✷

We are now going to develop DCA based algorithms for solving the approximate problems (5.9)-(5.11).

Solution methods via DC programming and DCA

DCA for solving the first approximate problem

Firstly, we consider the approximate problem (5.9) and introduce a DCA scheme that includes algorithms of ℓ 1 -perturbed/ℓ 2,1 -perturbed types. The function η α (t) can be expressed as a DC function: η α (t) = α|t|r(t), (5.18) where r(t) = -1 + max{1, α|t|}. Hence, the objective function of the problem (5.9) can be rewritten as a DC function:

F p α (x, y) = G 1 (x, y) -H 1 (x, y), (5.19)
where

G 1 (x, y) = χ K (x, y) + g(x, y) + λ x p,1 H 1 (x, y) = h(x, y) + λ J j=1 r( x j p ),
and g, h are DC components of f , i.e., f = gh. Hence a DC formulation of the problem (5.9) takes the form min

(x,y) {G 1 (x, y) -H 1 (x, y)}.
(5.20)

Following the generic DCA scheme, DCA for solving the problem (5.20) can be described as follows.

DCA1

Initialization: Choose (x 0 , y 0 ) ∈ K, l ← 0 and let τ be a tolerance sufficient small. repeat 1. Compute (x l , ȳl ) ∈ ∂h(x l , y l ) and (v l ) j ∈ λ∂r( (x l ) j p ), j = 1, ..., J. 2. Compute (x l+1 , y l+1 ) by solving the problem: min (x,y)∈K g(x, y)xl , xȳl , y + λα x p,1v l , x .

(5.21)

3. l ← l + 1. until (x l , y l ) -(x l-1 , y l-1 ) 2 ≤ τ ( (x l-1 , y l-1 ) 2 + 1) or |F p α (x l , y l ) -F p α (x l-1 , y l-1 )| ≤ τ (|F p α (x l-1 , y l-1 )| + 1)
Remark 5.1 We see that the problem (5.21) has the form of an ℓ p,1 -perturbed problem. Thus, for p = 2, (5.21) is an ℓ 2,1 -perturbed problem. For p = 1, we have x 1,1 ≡ x 1 and the problem (5.21) can be rewritten as follows.

min (x,y)∈K g(x, y)xl , xȳl , y + λα x 1v l , x .

( 5.22) This problem has the form of an ℓ 1 -perturbed problem which can be found in many previous works (see e.g. [START_REF] Thi | A DC programming approach for feature selection in support vector machines learning[END_REF], 2014a[START_REF] Mairal | Incremental majorization-minimization optimization with application to large-scale machine learning[END_REF][START_REF] Ong | Learning sparse classifers with difference of convex cunctions algorithms[END_REF]). Thanks to the ℓ 1 -norm, if g(x, y) is separable in its variables, so is the problem (5.22). This leads to a potential massive reduction in computational complexity. In many applications such as multi-task feature learning, group sparse PCA, group sparse optimal scoring, joint sparse compressed sensing, etc, it requires to estimate a row-wise sparse matrix W , meanwhile, its objective function can be separated in columns. Hence, the problem (5.22) can be separated into independent sub-problems in these applications. Note, however, that the problem (5.21) is not separable if p = 1. Thus we can say that the ℓ 1,0 is the most interesting regularization for DCA.

DCA for solving the second approximate problem

In the following, we introduce a DCA scheme for solving the problem (5.11) and indicate its connection with reweighted-ℓ p,1 procedure which includes reweighted-ℓ 1 and reweighted-ℓ 2,1 as special cases. The problem (5.11) is a DC program of the form: min (x,y,z)

{G 2 (x, y, z) -H 2 (x, y, z)}, (5.23) where

G 2 (x, y, z) = g(x, y) + χ Kp (x, y, z), H 2 (x, y, z) = h(x, y) + λ J j=1 (-η α )(z j ).
Let (x l , y l , z l ) ∈ K p be the current solution at iteration l. DCA applied to the DC program (5.23) updates (x l+1 , y l+1 , z l+1 ) ∈ K p via two steps:

-Step 1: compute (x l , ȳl ) ∈ ∂h(x l , y l ) and v l j ∈ λ∂(-η α )(z l j ) ∀j = 1, ..., J by v l j = -λα if z l j ≤ 1/α 0 otherwise.

-Step 2: compute (x l+1 , y l+1 , z l+1 ) ∈ arg min (x,y,z)∈Kp g(x, y)xl , xȳl , yv l , z .

(5.24)

Since v l j ≤ 0 ∀j = 1, ..., J, the (5.24) is equivalent to

   (x l+1 , y l+1 ) = arg min (x,y)∈K g(x, y) -xl , x -ȳl , y + J j=1 (-v l j ) x j p , z l+1 j = (x l+1 ) j p ∀j = 1, ..., J.
Hence, DCA for solving the problem (5.11) can be described as follows.

DCA2

Initialization: Choose (x 0 , y 0 ) ∈ K, l ← 0 and let τ be a tolerance sufficient small. repeat 1. Compute (x l , ȳl ) ∈ ∂h(x l , y l ) and v l j ∈ λ∂(-η α )( (x l ) j p ) ∀j = 1, ..., J. (5.25)

3. l ← l + 1. until (x l , y l , |x l J | p ) -(x l-1 , y l-1 , |x l-1 J | p ) 2 ≤ τ ( (x l-1 , y l-1 , |x l-1 J | p ) 2 + 1) or |F α (x l , y l , |x l J | p ) -F α (x l-1 , y l-1 , |x l-1 J | p )| ≤ τ (|F α (x l-1 , y l-1 , |x l-1 J | p )| + 1)
Remark 5.2 If the function f is convex, we can choose DC components of f as g = f and h = 0. Then (x l , ȳl ) = 0 ∀l. In this case, the problem in step 2 becomes .26) We see that (5.26) has the form of an ℓ p,1 -regularization problem but with different weights on groups x j . Hence, DCA2 iteratively solves the weighted-ℓ p,1 problem (5.26) with weights (-v l j ) being updated at each iteration l.

min (x,y)∈K f (x, y) + J j=1 (-v l j ) x j p . ( 5
For p = 1, (5.26) becomes a group-weighted-ℓ 1 problem. Moreover, if f (x, y) is separable, problem (5.26) can be separated into independent sub-problems. This is the case in many applications.

For p = 2, (5.26) becomes a weighted-ℓ 2,1 problem and the DCA in this case is reweightedℓ 2,1 algorithm. Note that the adaptive group Lasso proposed in [START_REF] Wei | Consistent group selection in high-dimensional linear regression[END_REF] is also of this type. However, the adaptive group Lasso only processes in one step and heuristically computes weights by (-v j ) = 1/ xj 2 where x is an initial estimate of the solution.

Application to group variable selection in optimal scoring problem

In this section, we consider the problem of group variable selection in linear discriminant analysis (LDA). Instead of directly considering the Fisher formulation (5.6), we are interested in the optimal scoring interpretation of LDA [START_REF] Hastie | Flexible discriminant analysis by optimal scoring[END_REF][START_REF] Hastie | Penalized discriminant analysis[END_REF]. The rationality of the optimal scoring method derives from the fact that LDA can also be reformulated as a multiple regression problem via optimal scoring. Generally, the problem can be formulated as follows.

Let {(x i , y i ) : i = 1, ..., n} be a set of labeled training data with observation vector x i ∈ R d and label y i ∈ {1, ..., C}. The data matrix is denoted by X = [x 1 ; ...;

x n ] ∈ R n×d . Let Y ∈ R n×C with Y ik = 1 if x i belongs to the k-th class and 0 otherwise. To find the linear transformation W = [w 1 , ..., w L ] ∈ R d×L , that maps the data in the d-dimensional space to an L-dimensional space (L ≤ C -1), in which the between-class variance is maximized while the within-class variance is minimized, the optimal scoring criterion solves the problem min (5.27) where I L is the L × L identity matrix. [START_REF] Hastie | Flexible discriminant analysis by optimal scoring[END_REF] proposed an algorithm for solving the problem (5.27) described as below.

Θ∈R C×L ,W ∈R d×L 1 2n ||Y Θ -XW || 2 F subject to 1 n Θ T Y T Y Θ = I L ,
(1) Choose a score matrix Θ 0 such that

1 n Θ T 0 Y T Y Θ 0 = I L . ( 2
y = arg max k (x -u k ) T W * D 2 2 ,
where D is a diagonal matrix with k-th diagonal term

D kk = 1/ λ 2 k (1 -λ 2 k
) and u k is the mean vector of class k.

In high-dimensional settings, there are many irrelevant and/or redundant features. In addition, the classification rule involves a linear combination of the features. Hence, one difficulty of the LDA is data interpretation. The most suitable approach to overcome this difficulty is feature selection. The resulting sparse discriminant vectors can provide a more interpretable low-dimensional representation of data.

In the optimal scoring problem, a feature j-th is selected if at least a component in the row j-th of W is nonzero and vice versa. Therefore, it is reasonable to consider rows of W as groups. Denote by w j (j = 1, . . . , d) the row j-th of W . In the step 2, using ℓ p,0 -regularization for the problem (5.28) leads us to consider the group sparse optimal scoring problem. min

W ∈R d×L 1 2n ||Y Θ 0 -XW || 2 F + λ W p,0 , (5.29)
where λ ≥ 0 is a tuning parameter, and the ℓ p,0 -norm of W is defined by W p,0 = d j=1 s( w j p ), i.e., we consider W as a vector by concatenating its rows which are regarded as groups.

Observe that the problem (5.29) is a special case of (5.1) where the function f is given by

f (W ) = 1 2n ||Y Θ 0 -XW || 2 F ,
and the corresponding approximate problem takes the form: min 5.31) where K p = {(W, z) : w j p ≤ z j , j = 1, . . . , d}.

W ∈R d×L F p (W ) := 1 2n ||Y Θ 0 -XW || 2 F + λ d j=1 η α ( w j p ) . ( 5
F (W, z) := 1 2n ||Y Θ 0 -XW || 2 F + λ d j=1 η α (z j ) , (
In the appendix A.1, we will prove that the optimal solution set of the problem (5.29) is bounded. Hence, without loss of generality, we can only consider the problem (5.29) on a box K = [-M, M] d×L for sufficient large M. By Proposition 5.3, there exists α 0 such that the approximate problem (5.30) is equivalent to the original problem (5.29) for all α > α 0 .

Here f is a convex quadratic function, and DC components of f are taken as g = f and h = 0. According to DCA1 and DCA2 with p = 1, 2, we have four DCA based algorithms, ℓ 2,0 (DCA1), ℓ 2,0 (DCA2), ℓ 1,0 (DCA1) and ℓ 1,0 (DCA2), described as follows.

ℓ 1,0 (DCA1) (DCA1 with p = 1 for solving (5.30))

Initialization: Choose W 0 ∈ K, l ← 0 and let τ be a tolerance sufficient small. repeat 1. Set (v l ) j ∈ λ∂r( (w l ) j 1 ), j = 1, ..., d as

(v l ) l k = sgn((w l ) j k )λα if (w l ) j 1 > 1/α 0 otherwise, , k = 1, ..., L.
2. Compute W l+1 by solving the problem:

min W ∈K 1 2n ||Y Θ 0 -XW || 2 F + λα W 1,1 - d j=1
(v l ) j , w j .

(5.32) In our experiments, we use the cross-validation scheme to validate the performance of various classifiers. Each dataset is split into a training set containing 2/3 of the samples and a test set containing 1/3 of the samples. This process is repeated 10 times, each with a random choice of training set and test set.

3. l ← l + 1. until W l -W l-1 F ≤ τ ( W l-1 F + 1) or |F 1 (W l ) -F 1 (W l-1 )| ≤ τ (|F 1 (W l-1 )| + 1)
The tuning parameters are α, λ and L (number of used discriminant vectors). We fixed α = 5, and λ, L were chosen via 5-fold cross-validation procedure on the training set from the sets of candidates given by Λ = {0.002, 0.004, 0.006, 0.008, 0.01, 0.014, 0.016, 0.018, 0.02, 0.024, 0.028, 0.032}, and L = {1, ..., C -1}, respectively. By this way, we avoid performing tuning parameter selection on a three-dimensional grid. The test set is used to measure the accuracy of various classifiers given by the training procedure. The reported computational results are the average results over 10 runs with different training sets.

The stop tolerance of DCA is τ = 10 -5 while the starting point of DCA is zero. The bound M is set to 10 3 . We use the same stopping criterion of the (block) coordinate descent method in Chapter 3 with tolerance 10 -4 .

Numerical results

The computational results (the average results on 10 runs with different training sets) given by ℓ 1,0 (DCA1), ℓ 1,0 (DCA2), ℓ 2,0 (DCA1), ℓ 2,0 (DCA2), SOS GLASSO and GS MSVM are reported in Tables 5.2-5.3. We are interested in the efficiency (the sparsity and the accuracy of classifiers), the number of used discriminant vectors L, as well as the rapidity of these algorithms. The discriminant vectors can be used to visualize the datasets such as in Figure 5.1.

Comments on computational results.

Sparsity:

The classifiers obtained by ℓ 1,0 (DCA1), ℓ 1,0 (DCA2), ℓ 2,0 (DCA1) and ℓ 2,0 (DCA2) are sparser than those obtained by SOS GLASSO and GS MSVM on 10/11 datasets. SOS GLASSO is slightly better than the DCA based algorithms on one dataset (ORL). ℓ 1,0 (DCA1), ℓ 1,0 (DCA2), ℓ 2,0 (DCA1) and ℓ 2,0 (DCA2) respectively select from 0.2% to 7.01%, 0.09% to 3.7%, 0.19% to 14.29% and 0.41% to 9.79% of features while SOS GLASSO and GS MSVM select from 0.91% to 14.63% and from 0.71% to 19.78% of features, respectively. Comparing between the ℓ 1,0 -regularization and ℓ 2,0 -regularization, we see that ℓ 1,0 (DCA1) and ℓ 1,0 (DCA2) select less features than ℓ 2,0 (DCA1) and ℓ 2,0 (DCA2) on ten out of eleven datasets. We also observe that DCA1 and DCA2 are comparable. More precisely, ℓ 1,0 (DCA2) is better than ℓ 1,0 (DCA1) on 9/11 datatsets while ℓ 2,0 (DCA2) is better than ℓ 2,0 (DCA1) on 4/11 datasets.

Accuracy of classifiers:

The DCA based algorithms not only provide a good performance in terms of feature selection, but also give a high accuracy of classifiers. The accuracy of classifiers attained the DCA based algorithms are better than SOS GLASSO and GS MSVM on most of datasets. Comparing between the ℓ 1,0 -regularization and ℓ 2,0regularization, ℓ 1,0 (DCA1) and ℓ 1,0 (DCA2) are better than ℓ 2,0 (DCA1) and ℓ 2,0 (DCA2) on 7/11 datasets while ℓ 2,0 (DCA1) and ℓ 2,0 (DCA2) are better than ℓ 1,0 (DCA1) and ℓ 1,0 (DCA2) on 3/11 datasets. On the remaining dataset (PEN), they obtain equal performance. We also see that ℓ 1,0 (DCA1) and ℓ 1,0 (DCA2) are comparable. More precisely, ℓ 1,0 (DCA1) is the best on 7/11 datasets and ℓ 1,0 (DCA2) is the best on 3/10 datasets.

Training time: ℓ 1,0 (DCA1) and ℓ 1,0 (DCA2) are remarkable faster than the other algorithms (the ratios of gains are from 3 to 63 times). Especially, ℓ 1,0 (DCA1) and ℓ 1,0 (DCA2) run much faster than the other algorithms on the datasets with large number of features and large number of classes (face image datasets). This can be explained by the fact that ℓ 1,0 (DCA1) and ℓ 1,0 (DCA2) lead to the sequences of convex problems which are Table 5.2: Comparative results of ℓ 1,0 (DCA1), ℓ 1,0 (DCA2), ℓ 2,0 (DCA1), ℓ 2,0 (DCA2), SOS GLASSO and GS MSVM in terms of the average number of selected features and its standard deviation (upper row), and the average percentage of selected features and its standard deviation (lower row) over 10 training/test set splits. Bold fonts indicate the best results in each row. separated into independent sub-problems. Moreover, the sub-problems at each iteration are solved in parallel. We observe that SOS GLASSO runs faster than ℓ 2,0 (DCA1) and ℓ 2,0 (DCA2) on 8/11 datasets. This is because SOS GLASSO only solves a convex problem while ℓ 2,0 (DCA1) and ℓ 2,0 (DCA2) have to solve several problems of the same type as SOS GLASSO.

ℓ 1,0 (DCA1) ℓ 1,0 (DCA2) ℓ 2,0 (DCA1) ℓ 2,0 (

Application to estimation of multiple covariance matrices

In recent years, much interest has focused on estimating a covariance matrix on the basis of an n × d data matrix X, where n is the number of observations and d is the number of features. Suppose that the observations x 1 , ..., x n ∈ R d are independent and identically distributed N (0, Σ), where Σ is a positive definite d × d matrix. A natural way to estimate the covariance matrix Σ is via minimizing negative log-likelihood. The resulting optimization problem is min Σ≻0 log det Σ + tr(Σ -1 S) , (5.41) where S = 1/n n i=1 x i x T i is the sample covariance matrix and the notation Σ ≻ 0 means that Σ is symmetric positive definite.

As mentioned in Chapter 4, estimation of sparse covariance matrix plays an important role in various areas of statistical analysis. In that chapter, we have used the ℓ 0 -norm in the regularization term that leads to the following sparse covariance matrix estimation problem: min Σ≻0 log det Σ + tr(Σ -1 S) + Σ 0 , (5.42) where λ is a non-negative tuning parameter and ||Σ|| 0 denotes the ℓ 0 -norm of Σ, i.e., the number of nonzero elements of matrix Σ.

In this chapter we extend the results of Chapter 4 to deal with the case of multiple classes. Suppose that we have a dataset with Q classes. For the k-th class, let X k be an n k × d matrix consisting of n k observations with the number of features d common to all classes. Furthermore, we assume that the observations within each class are independent and identically distributed according to N (0, Σ k ). Let S k = 1 n k (X k ) T X k be the sample covariance matrix for the k-th class. The Q covariance matrices are estimated via minimizing negative log-likelihood 5.43) where Ω = {{Σ} := {Σ 1 , ..., Σ Q } : Σ k δ k I, k = 1, ..., Q}. Here, I denotes the d × d identity matrix, and the notation Σ k δ k I means that Σ kδ k I is symmetric positive semidefinite.

min {Σ}∈Ω Q k=1 n k log det Σ k + tr((Σ k ) -1 S k ) , (
We define the ℓ p,0 -norm of {Σ} by 5.44) where s is the step function defined by s(t) = 1 if t = 0 and s(t) = 0 otherwise. In this chapter, we propose to estimate sparse multiple covariance matrices using the ℓ 0 +ℓ p,0 regularization. The resulting group sparse multiple covariance matrix estimation problem takes the form:

{Σ} p,0 = i,j s( (Σ 1 ij , ..., Σ Q ij ) p ), (
min {Σ}∈Ω Q k=1 n k log det Σ k + tr((Σ k ) -1 S k ) + λ Σ k 0 + γ {Σ} p,0 , (5.45)
where λ and γ are non-negative tuning parameters. An ℓ 0 penalty is applied to the elements of the covariance matrices and an ℓ p,0 penalty is applied to the (i, j) element across all Q covariance matrices. The first regularization term encourages sparsity within each covariance matrix Σ k while the second one encourages a similar pattern of sparsity across all the covariance matrices. This is referred as bi-level variable selection in the estimation of multiple covariance matrices.

In the literature, there exists a number of methods that seek a sparse covariance matrix or its inverse (see e.g. [START_REF] Meinshausen | High dimensional graphs and variable selection with the lasso[END_REF]; Yuan and Lin ( 2007 Fan (2009)). These methods only estimate single sparse covariance matrix or its inverse. We can apply these methods to separately estimate each covariance matrix Σ k . However, these approaches can be less accurate than the jointly approaches. Recently, [START_REF] Danaher | The joint graphical lasso for inverse covariance estimation across multiple classes[END_REF]; [START_REF] Huang | Joint learning of multiple sparse matrix gaussian graphical models[END_REF]; [START_REF] Guo | Joint estimation of multiple graphical models[END_REF]; [START_REF] Lee | Joint estimation of multiple precision matrices with common structures[END_REF] have developed the methods based on group lasso or fused lasso for finding sparse multiple inverse covariance matrices.

In this section, we apply the proposed algorithms to solve the problem (5.42). We also study the convex approximation approaches of the ℓ 0 -norm and ℓ p,0 which are the ℓ 1 -norm (lasso) and the ℓ 2,1 -norm (group lasso), respectively. However, the resulting problem is still non-convex, and then we apply DCA to solve this problem. Among the ℓ p,0regularizations, we continue showing that ℓ 1,0 is the most interesting group regularization for DCA.

We observe that the problem (5.45) takes the form of (5.1) where the function f is given by

f ({Σ}) = Q k=1 n k log det Σ k + tr((Σ k ) -1 S k ) + λ Q k=1 ij η α (Σ k ij ),
and the corresponding approximate problem is min 5.46)

{Σ}∈Ω F p ({Σ}) = f ({Σ}) + γ ij η α ( (Σ 1 ij , ..., Σ Q ij ) p ) , (
Initialization: Choose {Σ 0 } ∈ Ω, l ← 0. repeat 1. Compute {V l } ∈ ∂H({Σ l }) with (V k ) l = µ k (Σ k ) l + n k [(Σ k ) l ] -1 S k [(Σ k ) l ] -1 -n k [(Σ k ) l ] -1 .
2. Compute {Σ l+1 } by ADMM:

Initialization: m = 0, {Y 0 } = 0, (X ij ) 0 = 0 if S (V ij ) l , λ/ max k µ k 2 ≤ γ/ max k µ k , and otherwise (X ij ) 0 = 1 - γ/ max k µ k S ((V ij ) l , λ/ max k µ k ) 2 S (V ij ) l , λ/ max k µ k . repeat + For k = 1, ..., Q compute Σ k,l,m+1 = UD δ k U T with D δ k = diag(max(D k ii , δ k )) and (V k ) l -(Y k ) m + ρ(X k ) m /(µ k + ρ) = UD k U T , + Set (X ij ) m+1 = 0 if S (Σ ij ) l,m+1 + (Y ij ) m /ρ, λ/ρ 2 ≤ λ/ρ, and otherwise (X ij ) m+1 = 1 - γ/ρ S ((Σ ij ) m+1 + (Y ij ) m /ρ, λ/ρ) 2 S (Σ ij ) l,m+1 + (Y ij ) m /ρ, λ/ρ . + {Y m+1 } = {Y m } + ρ({Σ l,m+1 } -{X m+1 }), + m ← m + 1. until Stopping criterion. 3. l ← l + 1. until Stopping criterion.
The stopping criterion of ℓ 1 /ℓ 2,1 (DCA) is used as in ℓ 0 /ℓ 2,0 (DCA1) Theorem 5.4 (Convergence properties of ℓ 1 /ℓ 2,1 (DCA)) Let {{Σ l }} be the sequence generated by ℓ 1 /ℓ 2,1 (DCA), we have

(a) {F ({Σ l })} is decreasing. (b) {{Σ l }} is bounded. (c) +∞ l=0 {Σ l } -{Σ l+1 } 2
F < +∞, and hence {Σ l } -{Σ l+1 } F → 0 as l → +∞. (d) The sequence {{Σ l }} has at least one limit point and every limit point of this sequence is a critical point of the problem (5.63).

Proof : The theorem is proved analogously to Theorem 4.1. ✷

Numerical experiments

The numerical experiments aim to evaluate the performance of the three approaches: ℓ 0 /ℓ 1,0 (DCA1), ℓ 0 /ℓ 2,0 (DCA1), and the standard approach based on the ℓ 1 + ℓ 2,1regularization model (5.62) (ℓ 1 /ℓ 2,1 (DCA)). In experiments, we set the stop tolerances ǫ = τ = 10 -4 . The starting point {Σ 0 } of DCA is the sample covariance matrices {S 1 , ..., S Q }. The values of parameter λ and γ are chosen through a 5-fold cross-validation procedure on training set. The approximation parameter α of the Capped-ℓ 1 is set 1. Note that the ℓ 1,0 regularization also promotes sparsity within the group. Hence, we set λ = 0 in ℓ 0 /ℓ 1,0 (DCA1) to avoid performing tuning this parameter.

Experiments on synthetic datasets

We evaluate the performance of ℓ 0 /ℓ 1,0 (DCA1) and ℓ 0 /ℓ 2,0 (DCA1) on two synthetic datasets. We consider two types of covariance graphs with three-class:

Model 1: We generate a covariance matrix for the first class as follows. Σ 1 = diag(Σ 1 , ..., Σ 5 ), where Σ 1 , ..., Σ 5 are dense matrices. We create Σ 2 by resetting one of its 5 sub-network blocks to the identity, i.e., Σ 2 = diag(I, Σ 2 , ..., Σ 5 ). Resetting an additional sub-network block to the identity, we have Σ 3 = diag(I, I, Σ 3 , ..., Σ 5 ). A example is showed in Figure 5.2.

Model 2: Σ 1 = diag(Σ 1 , ..., Σ 5 ) again, however each submatrix Σ k is zero except the elements in the last row and the last column. This corresponds to a sub-graph with five connected components each of which has all nodes connected to one particular node.

Similarly to model 1, we create Σ 2 = diag(I, Σ 2 , ..., Σ 5 ) and Σ 3 = diag(I, I, Σ 3 , ..., Σ 5 ). A example is showed in Figure 5.3.

The nonzero entries of matrices Σ k , k = 1, 2, 3 are randomly drawn in the set {+1, -1}.

Finally, for each class we generate independent, identically distributed observations X k = To evaluate the performance of each method, we consider three loss functions which are the average root-mean-square error (ARMSE), the average entropy loss (AEN), and the average Kullback-Leibler loss (AKL), respectively.

ARMSE = 1 Q Q k=1 || Σk -Σ k || F /d, AEN = 1 Q Q k=1 -log det( Σk (Σ k ) -1 ) + tr( Σk (Σ k ) -1 ) -d , AKL = 1 Q Q k=1 -log det(( Σk ) -1 Σ k ) + tr(( Σk ) -1 Σ k ) -d ,
where Σk is a sparse estimate of the covariance matrix Σ k .

The experimental results on synthetic datasets are given in Table 5.4. In this Table, the average of root-mean-square error (ARMSE), entropy loss (AEN), Kullback-Leibler loss (AKL), number of nonzero elements on each covariance matrix (NZ1, NZ2, NZ3) and their sum (NZ), CPU time in seconds, and their standard deviations over 10 samples are reported. We observe from Table 5.4 that in the both models, ℓ 0 /ℓ 1,0 (DCA1) gives the best results in terms of three losses. In terms of the sparsity, the number of the nonzero elements, this approach also achieves better performances than the other approaches. The second and third performing approaches with respect to the losses and the sparsity are ℓ 0 /ℓ 2,0 (DCA1) and ℓ 1 /ℓ 2,1 (DCA), respectively.

Regarding the training time, ℓ 0 /ℓ 1,0 (DCA1) is much faster than the other algorithms. This can be explained by the fact that ℓ 0 /ℓ 1,0 (DCA1) leads to the sequence of convex problems which can be separated into the independent sub-problems.

Experiments on real datasets

We illustrate the use of the sparse covariance matrix estimation problem via a real application: a classification problem based sparse quadratic discriminant analysis (SQDA). This application requires estimates of the covariance matrices.

Let X be an n × d training data matrix with observations on the rows and features on the columns. We assume that the n k observations x k i (i = 1, ..., n k ) within the k-th class C k are normal distributed N (µ k , Σ k ). We denote the prior probability of the k-th class by π k . The quadratic discriminant function is In practice we do not know π k , µ k , Σ k , and will need to estimate them using the training data. In this work, we directly estimate the covariance matrices Σ 1 , ..., Σ Q by using ℓ 0 /ℓ 1,0 (DCA), ℓ 0 /ℓ 2,0 (DCA), and ℓ 1 /ℓ 2,1 (DCA). Note that if S k is singular, then we replace it by S k + ǫI p , where ǫ is chosen through a 5-fold cross-validation.

δ k (x) = - 1 2 log det Σ k - 1 2 (x -µ k ) T Σ -1 k (x -µ k ) + log π k . ( 5
For the experiment, we evaluate the proposed algorithms on two datasets from UCI Machine Learning Repository (Ionosphere and Waveform 2). We use the cross-validation scheme to validate the performance of various approaches on these two datasets. The dataset is split into a training set containing 2/3 of the samples and a test set containing 1/3 of the samples. This process is repeated 10 times, each with a random choice of training set and test set. The parameter λ and γ are chosen via 5-fold cross-validation. The computational results are reported in Table 5.5. We observe that, on the Ionosphere dataset, ℓ 0 /ℓ 1,0 (DCA1) and ℓ 0 /ℓ 2,0 (DCA1) are comparable and better than ℓ 1 /ℓ 2,1 (DCA) in terms of the testing error and training error. On the Waveform 2, ℓ 0 /ℓ 1,0 (DCA1) gives better testing error and training error than the both algorithms ℓ 0 /ℓ 2,0 (DCA1) and ℓ 1 /ℓ 2,1 (DCA). In terms of training time, ℓ 0 /ℓ 1,0 (DCA1) is significantly faster than ℓ 0 /ℓ 2,0 (DCA1) and ℓ 1 /ℓ 2,1 (DCA) on these two datasets.

Conclusion

We have intensively studied DC programming and DCA for group variable selection problem including the ℓ p,0 -norm in the objective function. DC approximation approach has been investigated from both a theoretical and an algorithmic point of view. Using the Capped-ℓ 1 approximation function, we have proved that the Capped-ℓ 1 approximate problem is equivalent to the original problem with suitable parameter α. Considering the two equivalent formulations of the approximate problem we have developed DC programming and DCA for solving them. When p = 1 and p = 2, the four DCA based algorithms can be viewed as an ℓ 2,1 -perturbed algorithm, reweighted-ℓ 2,1 algorithm, ℓ 1 -perturbed algorithm, reweighted-ℓ 1 algorithm with different weights on groups and the same weight on each group.

Concerning the group variable selection in optimal scoring, three of four DCA schemes (ℓ 1,0 (DCA1), ℓ 1,0 (DCA2) and ℓ 2,0 (DCA2)) have the interesting convergence properties: they converge after a finite number of iterations to local solution. We have also showed several useful properties of the ℓ 1,0 -regularization for group variable selection. The achieved solutions by nonconvex methods using the ℓ 1,0 -regularization are sparser than that of the others. At each iteration, ℓ 1,0 (DCA1) and ℓ 1,0 (DCA2) solve independent convex sub-problems in parallel. Numerical experiments confirm the theoretical results: ℓ 1,0 (DCA1) and ℓ 1,0 (DCA2) have obtained the best performance in terms of accuracy of classifiers and feature selection, and have taken the shortest time for training.

In the estimation problem of multiple covariance matrices, we use two regularization terms in order to encourage simultaneously sparsity within each covariance matrix and across all the covariance matrices. Among the proposed DCA schemes, ℓ 0 /ℓ 1,0 (DCA1) is the best interesting. At iteration, this scheme solves independent convex sub-problems. We also propose explicitly algorithms for solving the convex sub-problems. Numerical experiments on both simulation and real datasets have shown that our methods are promising.

For the future works, we plan to study group variable selection for other applications. We believe that the success of the ℓ In this chapter, we introduce stochastic DCA for minimizing a large sum of nonconvex functions, a problem of utmost importance in machine learning. With appropriate DC components, we present two special versions of the stochastic DCA: stochastic proximal DCA and stochastic proximal Newton DCA. We also show that stochastic gradient descent algorithm and stochastic proximal descent algorithm are special versions of stochastic DCA. As an application, we apply the proposed algorithm to a log-linear model incorporating latent variables. Parameter estimation of this model often results in an optimization problem involving a rational function of mixtures of exponential terms. It is a non-convex and large-scale problem which is very hard to solve. Experiments on the some real datasets show the efficiency of the proposed algorithms.

Introduction

The increase of applications using big data causes the difficulty in computation, especially when both the number of features and samples are large. Among methods proposed to address this problem, stochastic has been recently widely used as an efficient technique. In this chapter, we introduce a stochastic scheme based on DCA for solving a large sum of non-convex functions:

min Θ∈Ω f (Θ) = 1 n n i=1 f i (Θ) , (6.1)
where f i : R d → R are DC functions, and Ω is a convex subset of R d . Θ represents some model parameters and each function f i measures the adequacy of the parameters Θ to an observed data point indexed by i.

There are many application problems of the form (6.1) in machine learning, signal processing and other domains. The problem (6.1) also arises in the minimization of an expected loss that depends on Θ and some random vector. Hence, the objective function is either an expected loss with respect to a discrete distribution or is a finite sample approximation to an expected loss. Let us mention some important application problems corresponding to the model (6.1).

Sparse logistic regression problem: Given n data points (y i , x i ) n i=1 where observation vector x i ∈ R d and label y i ∈ {-1, 1}. We consider the sparse logistic regression problem, which can be formulated as follows:

min Θ∈Ω 1 n n i=1 ℓ(y i , x T i Θ) + λ d i=1 η α (Θ i ) , (6.2)
where ℓ(y, x T Θ) = log(1 + exp(-yx T Θ)), and η α is a DC approximation function of the step function such as the Capped-ℓ 1 . This problem takes the form of (6.1) with f i (Θ) = ℓ(y i , x T i Θ) + nλ d j=1 η α (Θ j ).

Problem with many constraints: We consider the problem

min x∈X f (x)
subject to g i (x) ≤ 0, i = 1, ..., n, (6.3) where the number of constraints n is very large in many practical problems. One way can tackle this problem is to use a penalty function. The resulting penalty problem takes the form of (6. p(g i (x)), (6.4) where τ is a positive penalty parameter and p is a penalty function such as the nondifferentiable penalty p(t) = max{0, t}.

Minimizing an expected loss in stochastic programming: Considering the minimization problem of an expected loss min Θ∈Ω E[f (Θ, ξ)], (6.5) where f is a function of Θ and a random variable ξ. When ξ has a discrete distribution or we use the sample average approximation method, the problem can be rewritten of the form (6.1):

min Θ∈Ω 1 n n i=1
f (Θ, ξ i ), (6.6) where ξ 1 , ..., ξ n are independent samples of the random variables ξ.

Latent log-linear model: Let D train = {(x i , y i )} N i=1 be a set of labeled training data with observation vector x i ∈ R d and labels y i ∈ Y = {1, ..., Q}. A log-linear model with the parameters (θ i , λ i ) ∈ R d × R, i = 1, ..., Q is a model for the class-posterior probabilities of the form P (y|x) = exp ( x, θ y + λ y ) y ′ ∈Y exp ( x, θ y ′ + λ y ′ ) . (6.7)

where the index i l is randomly chosen from {1, ..., n} and γ l is a positive step size. At each iteration, this method only involves the computation of the gradient ∇f i l (Θ l ) corresponding to one sample, hence its per-iteration cost is very cheap. Other works are closely related to our works: online expectation-maximization algorithm [START_REF] Cappé | Online expectation-maximization algorithm for latent data models[END_REF], stochastic majorization-minimization algorithm [START_REF] Mairal | Stochastic majorization-minimization algorithms for large-scale optimization[END_REF], stochastic successive minimization method [START_REF] Razaviyayn | A stochastic successive minimization method for nonsmooth nonconvex optimization with applications to transceiver design in wireless communication networks[END_REF], incremental method [START_REF] Bertsekas | Incremental proximal methods for large scale convex optimization[END_REF][START_REF] Mairal | Incremental majorization-minimization optimization with application to large-scale machine learning[END_REF].

There are some practical and theoretical advantages of stochastic over batch approaches for solving the large-scale machine learning problems. However, in the fact that batch approaches possess some intrinsic advantages. Motivated by this we combine the best properties of the stochastic approach and one of the most powerful tools in optimization. We introduce in this chapter a stochastic scheme based on DCA called stochastic DCA for the problem (6.1) where Ω is a convex set in R d and f is a large sum of DC functions f i . At each iteration, the stochastic DCA requires to solve the convex surrogate problem of only one small subset of the DC functions, which results in obtaining a low computation cost per iteration. Secondly, we exploit the particular structure of the problem and provide two special versions of the stochastic DCA: stochastic proximal DCA and stochastic proximal Newton DCA. In many practical problems, these algorithms can provide an explicit solution at each iteration. In some cases of objective functions, we also point out that the stochastic gradient descent and stochastic proximal descent are special variants of the proposed algorithms. Finally, we apply the stochastic DCA to solve the latent log-linear model (6.11) and propose two DCA for this problem. In order to evaluate the performance of the proposed methods, an empirical experiment is conducted.

The rest of the chapter is organized as follows. In Section 6.2, we present a genetic stochastic DCA and show that the stochastic gradient descent algorithm is as a special version of the stochastic DCA. We illustrate how to apply stochastic DCA to solve the latent log-linear model in Section 6.3. The DCA based algorithms are also proposed in Section 6.4. The numerical experiments are reported in Section 6.5 and Section 6.6 concludes the chapter.

6.2 Solution method based on DCA

Stochastic DCA

In this section, we introduce a stochastic DCA that exploits the structure of the objective f being a large sum of n DC functions f i = g ih i , where g i and h i are convex functions. At each iteration l, a small number of functions is used, and an iteration of DCA is performed. Every function f i is DC function, hence we have a DC decomposition of In order to discuss the first special version of (6.1), we assume that there exists a positive number ρ i such that ρ i 2 Θ 2g i 1 (Θ) is convex, for example, g i 1 is differentiable with L -Lipschitz gradient. Therefore, we can choose DC components of ρl . End for. Remark 6.2 We consider a special case in which g i 2 ≡ 0, i = 1, ..., n, hence ḡl 2 ≡ 0. In the step 3 of the algorithm (6.2), the proximal operator prox χ Ω ρl v l ρl becomes a projection of v l ρl on Ω.

When Ω falls into one of the following cases: a simplex, a ball, a box, a hyperplane or the intersection of a box and a hyperplane, this projection is possible to be explicitly computed.

In many practical problems such as regularization problems, we have Ω = R d and g i 2 (Θ) = λ Θ 1 . The proximal map prox where S is the elementwise soft-thresholding operator defined by S(v, w) j = sgn(v j )(|v j |w j ) + .

6.2.2.2 Stochastic gradient descent and incremental proximal methods are special versions of SPDCA

When Ω = R d , g i 2 = 0, g i 1 , h i are differentiable, and at each iteration we randomly choose one index s l = {i l }, we have

v l = ∇ hl (Θ l ) = ρ i l Θ l -∇f i l (Θ l ).
Hence, the updated rule in step 3 can be rewritten as follows:

Θ l+1 = Θ l - 1 ρ i l ∇f i l (Θ l ).
This is the updated rule of the stochastic gradient descent algorithm with the step-size

1 ρ i l
at the iteration l. Therefore, the stochastic gradient descent algorithm is a special version of SPDCA.

When h i ≡ 0 and at each iteration we randomly choose one index s l = {i l }, we have

v l = ρ i l Θ l -∇g i l 1 (Θ l ),
where ∇g i l 1 (Θ l ) is an arbitrary subgradient of g i l 1 at Θ l . Hence, the updated rule in step 3 becomes

Θ l+1 = arg min Θ∈Ω g i l 2 (Θ) + ρ i l 2 Θ -Θ l + 1 ρ i l ∇g i l 1 (Θ l ) 2 .
This update rule can be rewritten as below.

Θ l+1 = arg min Θ∈Ω g i l 2 (Θ) + g i l 1 (Θ l ) + ∇g i l 1 (Θ l ), Θ -Θ l +

ρ i l 2 Θ -Θ l 2 .
This is the update rule of the incremental proximal method proposed in [START_REF] Bertsekas | Incremental proximal methods for large scale convex optimization[END_REF] for solving convex problems. The incremental proximal method is also special version of SPDCA.

Stochastic Proximal Newton DCA

The second special version of Algorithm 6.1 comes from the assumption that

1 2 Θ T H i Θ -g i 1 (Θ),
is convex for some positive definite matrix H i . We have a DC decomposition of 1 |s l | i∈s l f i as follows:

1 |s l | i∈s l f i (Θ) = ḡl (Θ) -hl (Θ), (6.17) DCA, DCA1 and DCA2 slightly outperform the stochastic DCA on the first two datasets where the number of observations is not large enough. However, on the remaining two datasets with a large number of observations, DCA1, DCA2 and the stochastic DCA are comparable in term of the classification accuracy. The training time of the stochastic DCA is quite short: less than 191 seconds. SDCA not only obtains the quite good results in term of the classification accuracy, but also takes a short time on the all datasets. More precisely, SDCA runs much faster than the other algorithms, and this can be explained by the fact that this approach leads to the sequence of the convex sub-problems which only use one mall subset of observations and have explicit solutions. The second best algorithm according to the running time is DCA2 in which the explicit solution is computed at each iteration.

Conclusion

In this chapter, we have introduced a stochastic scheme based DCA for solving large scale parameter estimation problems in which the objective function is a large sum of DC functions. At each iteration, we only use one small subset of the DC functions and run one iteration of the corresponding DC program. As an application, we have investigated the structure of the the latent log-linear model and proposed a stochastic DCA for solving it. At each iteration of this stochastic DCA, we can compute the explicit solution to the convex sub-problem. We have also investigated DC programming and DCA for solving the latent log-linear model. We propose two DC formulation of the latent log-linear model and develop two DCA based algorithms.

The robustness and the effectiveness of our algorithms have been demonstrated through the computational results on real datasets. The nice effect of DC decomposition has been exploited: the second DC decomposition seems to be very suitable since it leads to an efficient, fast and scalable stochastic DCA scheme. In the experimental results, the stochastic DCA have obtained the quite good results in terms of accuracy of classifiers, and have taken the shortest time for training. The second best algorithm in terms of the training time is DCA2 which have achieved the best performance in terms of accuracy of classifier.

Chapter 7

Conclusion

In this thesis, we have analyzed how the problems of the sparsity in high dimensional setting and the stochastic learning can be addressed from various aspects including theory, algorithms and applications. The main algorithmic methodologies applied in the thesis are DC (Difference of Convex functions) programming and DCA (DC Algorithms) which are considered one of the state of the arts and powerful tools in optimization.

In the first part of the thesis, we introduced and evaluated a methodology supporting variable selections by using DC programming and DCA. In particular, we have first investigated DC programming and DCA for the sparse Fisher linear discriminant analysis (SFLDA) problem using the ℓ 0 -regularization. In order to tackle the ℓ 0 -norm, we analyzed DC approximation approaches, and among several existing sparse inducing functions we decided to use the Capped-ℓ 1 and the piecewise exponential concave function. The resulting problems have been formulated as DC programs, and then DCA have been applied. Consequently, we have proposed two DCA schemes for two different formulations of a common model to both the approximation functions. The robustness and effectiveness of our DCA based algorithms have been demonstrated through the experiments conducted on both the simulated and real datasets, in which we compared our approaches with three standard algorithms that use the ℓ 1 -regularization.

Next, we concentrated on investigating alternating schemes based on DC programming and DCA for solving the sparse optimal scoring (SOS) problem. By using two DC approximations of the ℓ 0 -norm and considering two DC formulations of each resulting approximate SOS problem, we have proposed alternating schemes for solving the four approximate problems. In accordance, four DCA schemes have been studied to deal with DC programs w.r.t w k in each step of the alternating algorithms. The important point here, is that we have proved that the main algorithms converge to a critical point of the approximate problems. The efficiency of the four proposed methods have been compared with five standard algorithms which use the ℓ 1 regularization. The computational results have showed that the proposed algorithms have produced much better sparsity as well as higher classification accuracy than the standard algorithms on both simulated datasets and high-dimensional real datasets. Besides, we have provided recommendations on how to best use the proposed algorithms. We have also compared the proposed methods for SOS, SFLDA problems with the sparse multiclass support vector machine (SMSVM) using the ℓ 0 -regularization. Numerical results have showed that the proposed methods have outperformed SMSVM.

Our third contribution lies in the investigation of the sparse covariance matrix estimation (SCME) problem. Specifically, the capped-ℓ 1 and piecewise exponential concave functions are continued to be chosen for modeling the sparsity, however we face the difficulty in the non-convexity of the negative log-likelihood function. Thus, we have proposed two DC formulations of the approximate SCME problem based on two DC decompositions of its objective function to overcome the problem. The first results are obtained from a natural DC decomposition while the second is introduced to exploit nice effects of DC decompositions. It turns out that the complexity of two corresponding DCA schemes is significantly different and the ratio of gain between them in terms of CPU times in our numerical experiments is up to 44 times. This is explained by the fact that the convex subproblems in the second DCA scheme can be solved by an extremely inexpensive algorithm. Applying DCA on two DC formulations with two approximations, we then have four DCA based algorithms for the approximate SCME problem. Special convergence analysis results of our algorithms have been provided. Additionally, we have considered two important applications of the SCME problem in our experiments, which are respectively the quadratic discriminant analysis using sparse covariance matrices estimated by the proposed algorithms and the portfolio optimization problem. Numerical experiments have been carefully achieved on several test experiments on both simulated datasets and real datasets with eleven algorithms including seven state-of-the-art methods and the four proposed DCA schemes.

In the second part of this thesis, we turned our attention to the problem of group variable selection. We have studied the ℓ p,0 regularization (p ≥ 1) for enforcing group sparsity. Using a DC approximation of the ℓ p,0 -norm, we have indicated that the approximate problem is equivalent to the original problem with suitable parameters. By considering two equivalent formulations of the approximate problem we have developed DCA based algorithms to solve them. Among ℓ p,0 regularizations, we have show that the ℓ 1,0 is the most interesting regularization with several advantages in both theoretical and computational aspects. Regarding applications, we have implemented the proposed algorithms for group variable selection in optimal scoring problem and multiple covariance matrices estimation problem. In the first application, sparsity is obtained by using the ℓ p,0 regularization that can select the same variables in all discriminant vectors. The resulting sparse discriminant vectors have provided a more interpretable low-dimensional representation of data. In the second application where multiple covariance matrices share some common structures such as the locations or weights of non-zero elements, we have combined the ℓ 0 -norm and the ℓ p,0 -norm to enforce sparsity on each covariance matrix and across multiple covariance matrices, respectively.

Finally, we analyzed and applied the stochastic technique based DC programming and DCA to large scale parameter estimation problems in which the objective function is a large sum of DC functions. At each iteration, we only use one small subset of the DC functions and run one iteration of the corresponding DC program. We have also presented two special versions of the stochastic DCA: stochastic proximal DCA and stochastic proximal Newton DCA that regard some standard stochastic algorithms as special versions.

As application, we have investigated the structure of the the latent log-linear model and proposed a special stochastic DCA in which the solution to each convex sub-problem can be explicitly computed. We have also taken DC programming and DCA into account in order to solve the latent log-linear model.

This thesis has explored some issues relating to modeling sparsity and stochastic learning, and we believe several follow-up studies for the future can be derived from this research. First of all, concerning variable selection, the DCA based approaches presented in this thesis could be useful to develop efficient algorithms for other sparse optimization problems in high-dimensional setting. Moreover, in the scope of this thesis, we have just studied the Fisher's discriminant problem for the linear classification, so it is interesting to extend the proposed techniques to more complex settings, such as the case where the observations from each class are drawn from a mixture of Gaussian distributions resulting in nonlinear separations between classes. We also plan to study more extensive applications of these problems.

Regarding group variable selection, we believe that the success of the ℓ p,0 -regularization motivate and open up a new avenue for the group variable selection problems. To be more specific, we will study this regularization to other models such multiclass support vector machine, principal component analysis, compressed sensing, etc. By considering a common DC approximation of the ℓ p,0 -norm, we also intend to investigate the consistency between global minimums (local minimums) of approximate and original problems. Moreover, the combination of the ℓ 0 -norm and the ℓ p,0 -norm to obtain the sparsity at both group and individuals in group levels will be more explored.

Last but not least, our research study on the stochastic schemes based on DCA is just the beginning of the ongoing work. Therefore, in the future, the convergence properties of the generic stochastic DCA and its variants will be investigated along with the update rule (6.15). More concretely, we plan to study the stochastic DCA using this update rule and others, as well as explore the use of the stochastic DCA for solving the other models, especially, sparse SVM, spare logistic regression, sparse matrix factorization, group variable selection in latent log-linear model.

By construction of Ŵ and (A.2), we obtain

Y Θ 0 -X W = Y Θ 0 - i∈I\{i 0 } x i ( wi + δ i wi 0 ) T = Y Θ 0 -X Ŵ . (A.3)
Moreover, we notice that d i=1 s( ŵi p ) ≤ |I| -1 and d i=1 s( wi p ) = |I|. Then

1 2n Y Θ 0 -X Ŵ 2 F + λ d i=1 s( ŵi p ) ≤ 1 2n Y Θ 0 -X Ŵ 2 F + λ|I| -λ < 1 2n Y Θ 0 -X W 2 F + d i=1 s( wi p ).
This contradicts the hypothesis that W is an optimal solution of the problem (5.29).

The proof of Lemma A.1 is then completed. ✷

Given I ⊂ {1, 2, ..., d}, X I denotes the n × |I| matrix whose columns are x i , i ∈ I and W I denotes the |I| × L matrix whose rows is w i , i ∈ I. Let I W = {i : w i = 0} and λ I denotes the smallest eigenvalue of X T I X I . We denote S = {I ⊂ {1, 2, ..., d} : {x i } i∈I is linearly independent}.

It follows that ∀I ∈ S, X T I X I ∈ R |I|×|I| is positive definite, so λ I > 0. Since S is a finite set, then we obtain λ 0 = min{λ I : I ∈ S} > 0.

Proposition A.1 The optimal solution set of the problem (5.29) is bounded, i.e., ∀ W ∈ P then

W F ≤ 2 Y Θ 0 F √ λ 0 .
Proof : By Lemma A.1, if W is an optimal solution to the problem (5.29), then I W ∈ S.

We have

XW = i∈I W x i (w i ) T = X I W W I W . Hence XW 2 F = T r(W T I W X T I W X I W W I W ) ≥ λ I W W I W 2 F = λ I W W 2 F . (A.4)
Since W is an optimal solution to the problem (5.29), we have

1 2n Y Θ 0 -XW 2 F + λ d i=1 s( w i p ) ≤ 1 2n Y Θ 0 2 F .
Then, we get

Y Θ 0 -XW F ≤ Y Θ 0 F ⇒ XW F ≤ 2 Y Θ 0 F (A.5)
Combining (A.4) and (A.5), it follows that .6) This leads to

λ I W W F ≤ 2 Y Θ 0 F . (A
λ 0 W F ≤ 2 Y Θ 0 F ⇔ W F ≤ 2 Y Θ 0 F √ λ 0 .
The Proposition A.1 has been proved. ✷

Contents 5 . 6

 56 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . III Stochastic Learning 6 Stochastic DCA and Application to Latent Log-Linear Model 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Solution method based on DCA . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 Stochastic DCA . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.2 Special versions of stochastic DCA . . . . . . . . . . . . . . . . . 6.3 Application to latent log-linear model . . . . . . . . . . . . . . . . . . . . 6.4 DCA for solving the latent log-linear model . . . . . . . . . . . . . . . . 6.5 Numerical experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2. 1 3 . 3

 133 Graphs of approximation functions: Capped-ℓ 1 and exponential function 2.2 The SRBCT dataset was projected onto the first three sparse discriminant vectors. The samples in each class are shown by using a distinct symbol. 2.3 The Sun dataset was projected onto the first two sparse discriminant vectors. The samples in each class are shown by using a distinct symbol. . . 3.1 The penicillium data is projected onto the first two sparse discriminant vectors. The samples in each class are shown by using a distinct symbol. 4.1 Examples of digitized handwritten 3s and 8s. Each image is a 8 bit, 16×16 grayscale version of the original binary image. . . . . . . . . . . . . . . . 4.2 The comparison of the realized return in different test periods. . . . . . . 4.3 The comparison of the realized risk in different test periods. . . . . . . . 4.4 The comparison of the Sharpe ratio in different test periods. . . . . . . . 5.1 The TOX dataset was projected onto the first three sparse discriminant vectors. The samples in each class are shown by using a distinct symbol.5.2 Sparse multiple covariance matrices in Model 1 . . . . . . . . . . . . . . 5.3 Sparse multiple covariance matrices in Model 2 . . . . . . . . . . . . . . 13 List of Tables 2.1 Synthetic datasets used in experiments. . . . . . . . . . . . . . . . . . . . 2.2 Real datasets used in experiments. . . . . . . . . . . . . . . . . . . . . . 2.3 Comparative results of DCA1-PiE, DCA1-Capped-ℓ 1 , DCA2-PiE, DCA2-Capped-ℓ 1 , PLDA, RDA and DSDA on synthetic data. Bold fonts indicate the best results in each row. . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Comparative results of DCA1-PiE, DCA1-Capped-ℓ 1 , DCA2-PiE, DCA2-Capped-ℓ 1 , PLDA, RDA and DSDA in terms of the average of percentage of accuracy of classifiers and its standard deviation (upper row), and the average of percentage of selected features and its standard deviation (lower row) over 10 training/test set splits. Bold fonts indicate the best results in each row. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Comparative results of DCA1-PiE, DCA1-Capped-ℓ 1 , DCA2-PiE, DCA2-Capped-ℓ 1 , PLDA, RDA and DSDA in terms of the average of CPU time in second and its standard deviation (upper row) over 10 training/test set splits, the number of discriminant vectors used K (the data is projected onto a K-dimensional space) (lower row). Bold fonts indicate the best results in each row. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Comparative results of MNIST dataset in terms of the number of selected features, the percentage of accuracy of classifiers on the test set, and training time in second. Bold fonts indicate the best results in each column. . 3.1 Real datasets used in experiments. . . . . . . . . . . . . . . . . . . . . . 3.2 Comparative results of ADCA1-Exp, ADCA1-Cap, ADCA2-Exp, ADCA2-Cap, PLDA, SDA, RDA, SPLS DA and S SVM on the synthetic data. Bold fonts indicate the best results in each row. . . . . . . . . . . . . . . 15 Comparative results of ADCA1-Exp, ADCA1-Cap, ADCA2-Exp, ADCA2-Cap, PLDA, SDA, RDA, SPLS DA and S SVM in terms of the average number of selected features and its standard deviation (upper row), and the average percentage of selected features and its standard deviation (lower row) over 10 training/test set splits. Bold fonts indicate the best results in each row. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 ADCA1-Exp, ADCA1-Cap, ADCA2-Exp, ADCA2-Cap, PLDA, SDA, RDA, SPLS DA and S SVM in terms of the average of percentage of accuracy of classifiers and its standard deviation (first row) over 10 training/test set splits, the number of discriminant vectors used K (the data is projected onto a K-dimensional space) (second row), and the average of training time in second and its standard deviation (third row). Bold fonts indicate the best results in each row. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Comparative results of MNIST dataset in terms of the number (percentage) of selected features #FS (%FS), the percentage of accuracy of classifiers (ACC) on the test set, and training time in second. Bold fonts indicate the best results in each column. . . . . . . . . . . . . . . . . . . . . . . . 3.6 Comparative results of ADCA1-Cap, DCA1-Cap and SMSVM-Cap. Bold fonts indicate the best results. . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Comparative results in terms of the average of root-mean-square error (RMSE), entropy loss (EN), Kullback-Leibler loss (KL), number of nonzero elements, CPU time in second (and their standard deviations) over 10 runs. Bold fonts indicate the best result in each row. . . . . . . . . . . . . . . . 4.2 Two datasets from UCI repository used in experiments. . . . . . . . . . . 4.3 Digit classification results of 3s and 8s. Bold fonts indicate the best result in each column. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Comparative results of Ionosphere and Waveform 2 datasets in terms of the average of percentage of testing errors, training errors, training time in second and their standard deviations over 10 training/test set splits. The bold font indicates the best result in each column. . . . . . . . . . . . . . 4.5 The comparison of the realized return, realized risk and Sharpe ratio. Bold fonts indicate the best result in each column. . . . . . . . . . . . . . . . . 5.1 Real datasets used in experiments. . . . . . . . . . . . . . . . . . . . . .

5. 3

 3 ℓ 1,0 (DCA1), ℓ 1,0 (DCA2), ℓ 2,0 (DCA1), ℓ 2,0 (DCA2), SOS GLASSO and GS MSVM in terms of the average of percentage of accuracy of classifiers and its standard deviation (first row) over 10 training/test set splits, the number of used discriminant vectors L (the data is projected onto an L-dimensional space) (second row), and the average training time (in seconds) and its standard deviation (third row). Bold fonts indicate the best results in each row. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Comparative results of ℓ 0 /ℓ 1,0 (DCA1), ℓ 0 /ℓ 2,0 (DCA1), and ℓ 1 /ℓ 2,1 (DCA) in terms of the average of root-mean-square error (ARMSE), entropy loss (AEN), Kullback-Leibler loss (AKL), number of nonzero elements, CPU time in second (and their standard deviations) over 10 runs. Bold fonts indicate the best result in each row. . . . . . . . . . . . . . . . . . . . . . 5.5 Comparative results of Ionosphere and Waveform 2 datasets in terms of the average of percentage of testing errors, training errors, training time in second and their standard deviations over 10 training/test set splits. The bold font indicates the best result in each column. . . . . . . . . . . . . . 6.1 Real Datasets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Comparative results of DCA1, DCA2, L-BFGS, ProxCCCP and Kernel SVM. Bold fonts indicate the best results in each row. . . . . . . . . . . .

R

  set of real numbers R + set of nonnegative real numbers R ++ set of positive real numbers R n set of real column vectors of size n R m×n set of real matrices of size m -by -n R n + set of nonnegative real column vectors of size n R n ++ set of positive real column vectors of size n I n identity matrix of size n

  and C is a nonempty polyhedral set. Notation χ C stands for indicator function of C and is defined by χ C (x) = 0 if x ∈ C, and +∞ otherwise. It is clear that dom f = C. Proposition 1.3 Let f be a polyhedral convex function, and x ∈ domf . Then we have 1.

  α * = inf{h * (y)g * (y) : y ∈ X} There is a perfect symmetry between primal and dual programs (P ) and (D): the dual program to (D) is exactly (P ), moreover, α = α * . Remark 1.2 Let C be a nonempty closed convex set. Then, the constrained problem inf{f (x) = g(x)h(x) : x ∈ C} can be transformed into an unconstrained DC program by using the indicator function χ C , i.e., inf{f (x) = φ(x)h(x) : x ∈ X} where

Figure 2 . 1 :

 21 Figure 2.1: Graphs of approximation functions: Capped-ℓ 1 and exponential function

Figure 2 . 2 :

 22 Figure 2.2: The SRBCT dataset was projected onto the first three sparse discriminant vectors. The samples in each class are shown by using a distinct symbol.

Figure 2 . 3 :

 23 Figure 2.3: The Sun dataset was projected onto the first two sparse discriminant vectors. The samples in each class are shown by using a distinct symbol.

3. 3

 3 DCA based algorithms for solving nonconvex subproblems in alternating schemes 3.3.1 DC formulations and DCA based algorithms for nonconvex subproblems (3.7) and (3.9)

  (resp. (3.25)) is a polyhedral function. If the condition (3.31) (resp. z * i = 1/α ∀i = 1, ..., p) holds, then H 1 (resp. H 2 ) is differentiate at w * (resp. (w * , z * )). Using the DCA's convergence property (v) in Theorem 1.2, we deduce that w * (resp. (w * , z * )) is a local minimizer of(3.19) (resp. (3.25)) in the case of Capped-ℓ 1 .

3. 5

 5 Numerical experiments 3.5.1 Comparative algorithmsWe have four main algorithms based on alternating schemes and DCA, named ADCA1-Exp, ADCA1-Cap, ADCA2-Exp and ADCA2-Cap. To demonstrate the usefulness of the proposed ℓ 0 -sparse optimal scoring methods in view of many classification methods in the literature we will compare our algorithms with the five state-of-the-art algorithms: the SVM based method proposed in[START_REF] Fan | Liblinear: A library for large linear classification[END_REF] (S SVM) (note that SVM is the most popular classification method), the three LDA based approaches proposed in Witten and Tibshirani (2011) (PLDA),[START_REF] Clemmensen | Sparse discriminant analysis[END_REF] (SDA),[START_REF] Guo | Regularized linear discriminant analysis and its application in microarrays[END_REF] (RDA), and the sparse partial least squares discriminant analysis Chun and Keles (2010) (SPLS DA).

3. 5 . 1 . 6

 516 Sparse multiclass support vector machine (SMSVM-Cap)Given n training observations (x i , y i ), i = 1, ..., n, the classification rule of muticlass support vector machine (MSVM) is to classify an observation x to a class y defined by y = arg max w k , x + b k .(3.45)

Figure 3 . 1 :

 31 Figure 3.1: The penicillium data is projected onto the first two sparse discriminant vectors. The samples in each class are shown by using a distinct symbol.

  g. Deng and Tsui (2013); Liu et al. (2014); Rothman et al. (2009); Rothman (2012); Xue et al. (

Remark 4 . 1

 41 .20)We can deduce from (4.20) that ||∇ 2 Λ(Σ)|| 2 ≤ 2||S|| 2 δ -3 , and then by (4.18) we have ρ(∇ 2 Λ(Σ)) ≤ 2||S|| 2 δ -3 . The lemma is then proved.✷ From the Lemma 4.1, we can choose µ = 2||S|| 2 δ -3 .

  matrix with the diagonal entries b 1 , ..., b p , ..., b p , we get λ min (I p ⊗ B) = λ min (B).

  .35) and hence lim l→+∞ Σ l -Σ l+1 F = 0. (d) is deduced from (b), Lemma 4.2 and the DCA's convergence property (iii) of Theorem 1For each algorithm, we use two DC approximations of the ℓ 0 -norm. Hence we have four DCA based algorithms: DCA1-CaP, DCA1-PiE, DCA2-CaP and DCA2-PiE, where CaP and PiE denote the algorithm using the Capped-ℓ 1 (η α = η α,2 ) and the piecewise exponential approximation function (η α = η α,1

Figure 4 . 1 :

 41 Figure 4.1: Examples of digitized handwritten 3s and 8s. Each image is a 8 bit, 16 × 16 grayscale version of the original binary image.

  );[START_REF] Kourtis | Parameter uncertainty in portfolio selection: Shrinking the inverse covariance matrix[END_REF];[START_REF] Fan | Large covariance estimation by thresholding principal orthogonal complements[END_REF];[START_REF] Xue | Positive-definite ℓ 1 -penalized estimation of large covariance matrices[END_REF];[START_REF] Lai | Mean-variance portfolio optimization when means and covariances are unknown[END_REF];[START_REF] Deng | Penalized covariance matrix estimation using a matrix-logarithm transformation[END_REF]). We follow[START_REF] Jagannathan | Risk reduction in large portfolios: Why imposing the wrong constraints helps[END_REF];[START_REF] Kourtis | Parameter uncertainty in portfolio selection: Shrinking the inverse covariance matrix[END_REF];[START_REF] Fan | Large covariance estimation by thresholding principal orthogonal complements[END_REF];[START_REF] Xue | Positive-definite ℓ 1 -penalized estimation of large covariance matrices[END_REF];[START_REF] Deng | Penalized covariance matrix estimation using a matrix-logarithm transformation[END_REF] to focus on the global minimum variance portfolio is the minimum risk portfolio with weights that sum to unity, namely min w∈R p w T Σw : w T e = 1 .

Figure 4 . 2 :

 42 Figure 4.2: The comparison of the realized return in different test periods.

Figure 4 . 3 :Figure 4 . 4 :

 4344 Figure 4.3: The comparison of the realized risk in different test periods.

2 .

 2 Compute (x l+1 , y l+1 ) by solving the problem: min (x,y)∈K g(x, y)xl , xȳl , y +

3 )

 3 Compute eigenvector matrix V and corresponding eigenvaluesλ 1 , ..., λ L of Θ T 0 Y T X Ŵ . (4) Compute solution Θ * = Θ 0 V and W * = Ŵ V .The classification rule is to assign a new observation x to class y if

Figure 5 . 1 :

 51 Figure 5.1: The TOX dataset was projected onto the first three sparse discriminant vectors. The samples in each class are shown by using a distinct symbol.

  ); Banerjee et al. (2008); Friedman et al. (2008); Rothman et al. (2008); Danaher et al. (2014); Cai et al. (2011); Zhang and Zou (2014); Deng and Tsui (2013); Liu et al. (2014); Rothman et al. (2009); Rothman (2012); Xue et al. (2012); Bien and Tibshirani (2011); Lam and

Figure 5 . 2 :

 52 Figure 5.2: Sparse multiple covariance matrices in Model 1

Figure 5 . 3 :

 53 Figure 5.3: Sparse multiple covariance matrices in Model 2

  .64) Then the predicted class for a new observation x is arg max k δ k (x). The decision boundary between each pair of classes k and l is described by a quadratic equation {x : δ k (x) = δ l (x)}.

1

  |s l | i∈s l f i as follows: 1 |s l | i∈s l f i (Θ) = ḡl (Θ) -hl (Θ),6.2.2.1 Stochastic proximal DCA

  de ℓ 2,1 perturbé) et un algorithme de ℓ 1 repondéré (resp. algorithme de ℓ 2,1 repondéré). Il se trouve que, parmi les ℓ p,0 -régularisations, ℓ 1,0 est la régularisation la plus intéressante avec plusieurs avantages dans tous les deux aspects théoriques et computationels. En ce qui concerne les applications, nous appliquons les algorithmes proposés à la sélection de groupes de variables dans les problèmes de scoring optimal et estimation de matrices de covariance. Dans la première application, la parcimonie est obtenue en utilisant la ℓ p,0 -régularisation qui sélectionne les mêmes variables dans tous les vecteurs discriminants. Les vecteurs discriminants parcimonieux résultant fournissent une représentation de faible dimension plus interprétable des données. Dans la seconde application, les matrices de covariance partagent certaines structures communes telles que les emplacements ou les poids des éléments non nuls, nous combinons la ℓ 0 -norme et la ℓ p,0 -norme pour obtenir la parcimonie sur chaque matrice de covariance et à travers de multiples matrices de covariance, respectivement.

Ainsi, nous proposons deux formulations DC du problème approché SCME basé sur deux décompositions DC de sa fonction objectif. Le premier résultat est obtenu à partir d'une décomposition DC naturelle tandis que le second est introduit pour exploiter de beaux effets de décompositions DC. La complexité des deux schémas DCA correspondantes est sensiblement différente. Selon nos expériences numériques, le rapport de gain en temps de calcul entre les deux DCA est de 44 fois. En appliquant DCA à deux formulations DC avec deux approximations, nous avons alors quatre algorithmes basés sur DCA pour le problème approché SCME. Les résultats d'analyse de convergence spéciale de nos algorithmes sont fournis. En outre, nous considérons deux applications importantes du problème SCME dans nos expériences, qui sont respectivement l'analyse discriminante quadratique en utilisant des matrices de covariance parcimonieuses estimées par les algorithmes proposés et le problème d'optimisation de portefeuille.

Le quatrième problème abordé dans la thèse est la sélection de groupes de variables. La nécessité de sélectionner des groupes de variables se pose dans de nombreux domaines d'application tels que l'apprentissage, le statistique, la biologie computationnelle, le traitement du signal, et d'autres domaines connexes. Nous étudions la ℓ p,0 -régularisation pour obtenir la parcimonie groupée. En utilisant une approximation DC approprié du ℓ p,0norme, nous indiquons que le problème approché est équivalent au problème original avec les paramètres appropriés. En considérant deux reformulations équivalentes du problème approché, nous développons des algorithmes basés sur DCA pour les résoudre. Lorsque p = 1 (resp. p = 2), nos algorithmes comprennent un algorithme de ℓ 1 perturbé (resp. algorithme Finalement, nous analysons et appliquons le technique stochastique basé sur la programmation DC et DCA aux problèmes d'estimation des paramètres à grande échelle dans lesquelles la fonction objectif est la somme d'une grande famille des fonctions DC. A chaque itération, nous utilisons seulement un petit sous-ensemble des fonctions DC et exécutons une itération du programme DC correspondante. Comme une application, nous étudions la structure du modèle log-linéaire latent et proposons un schéma DCA stochastique spécial dans lequel la solution à chaque sous-problème convexe peut être explicitement calculée. Nous également étudions la programmation DC et DCA pour résoudre directement le modèle log-linéaire latent. Les expériences numériques montrent que nos algorithmes proposés offrent une bonne performance.

Organisation de la thèse

La thèse se compose en sept chapitres. Le premier chapitre décrit de manière succincte la programmation DC et DCA. Il présente les outils théoriques et algorithmiques servant des références aux autres chapitres. Les cinq chapitres suivants sont divisés en trois parties: la première partie (Chapitres 2, 3 et 4) aborde le problème de la sélection des variables. Plus précisément, nous présentons les approches basées sur DCA pour la sélection des variables dans le problème discriminant de Fisher (Chapitre 2), le problème de scoring optimal (Chapitre 3), et le dernier chapitre de cette partie (Chapitre 4) est consacré au problème d'estimation de matrice de covariance parcimonieuse. Ensuite, la deuxième partie (Chapitre 5), nous étudions le problème de sélection de groupes de variables et des applications dans les problèmes de scoring optimal et d'estimation de multiples matrices de covariance. La dernière partie traite un modèle à variables latentes en introduisant un DCA stochastique appliqué au modèle log-linéaire incorporant des variables latentes (Chapitre 6). Chapitre 7 fournit les conclusions et les perspectives de nos travaux.

Table 2 .

 2

	2: Real datasets used in experiments.
	Datasets	#Features #Samples	#Classes
	Internet Advertisement (ADV) 1558	3279	2
	Colon Tumor (COL)	2000	62	2
	SRBCT (SRB)	2308	83	4
	Pencillium (PEN)	3754	36	3
	Gisette (GIS)	5000	7000	2
	Lung Cancer (LUN)	12533	181	2
	Leukemia (LEU)	12558	248	6
	Nakayama (NAK)	22283	86	5
	Sun (SUN)	54613	180	4
	MNIST (MNI)	784	60000/10000 10

Table 2 .

 2 3: Comparative results of DCA1-PiE, DCA1-Capped-ℓ 1 , DCA2-PiE, DCA2-Capped-ℓ 1 , PLDA, RDA and DSDA on synthetic data. Bold fonts indicate the best results in each row.

			DCA1-PiE	DCA1-Capped-ℓ 1	DCA2-PiE	DCA2-Capped-ℓ 1	PLDA	RDA	DSDA
	ACC	S1 S2 S3	96.89 ± 0.035 96.66 ± 0.38 98.1 ± 0.38 97.9 ± 0.44 88.03 ± 1.61 87.51 ± 1.73	96.52 ± 0.28 98.1 ±0.38 88.03 ±0.1.16	96.45 ± 0.4 98.03 ± 0.23 87.72 ± 1.12	96.8 ± 0.39 98 ± 0.43 87.31 ± 1.31	96.34 ± 0.64 94.8 ± 1.01 68.81 ± 1.6	-91.62 ± 1.12 -
	Average	94.34	94.02	94.21	94.07	94.05	86.66	-
	FS	S1 S2	43.76 ± 12.44 55.3 ± 2.68	52.48 ± 9.99 45.64 ± 1.78	26.54 ± 0.77 55.3 ± 2.68	47.64 ± 1.53 51.76 ± 2.79	54.94 ± 7.03 51.76 ± 2.79	22.8 ± 0.95 79.68 ± 1.39	-59.45 ± 10.29
		S3	25.74 ± 1.38	25.4 ± 1.38	25.74 ± 1.38	28.04 ± 1.62	34.1 ± 1.38	45.86 ± 1.79	-
	Average	41.6	41.17	35.86	42.48	46.93	49.45	-
	CPU	S1 S2 S3	0.022 ± 0.009 0.001 ± 0.003 0.002 ± 0.006 0.02 ± 0.008 0.004 ± 0.006 0	0.06 ± 0.09 0.004 ±0.006 0.004 ± 0.006	0.032 ± 0.01 0.006 ± 0.008 0.01 ± 0.008	0.017 ± 0.01 0.328 ± 0.015 0.002 ± 0.006 0.03 ± 0.004 0.002 ± 0.006 0.034 ± 0.006	-0.1 ± 0.02 -
	Average	0.009	0.007	0.023	0.016	0.007	1.131	-

Table 2 .

 2 4: Comparative results of DCA1-PiE, DCA1-Capped-ℓ 1 , DCA2-PiE, DCA2-Capped-ℓ 1 , PLDA, RDA and DSDA in terms of the average of percentage of accuracy of classifiers and its standard deviation (upper row), and the average of percentage of selected features and its standard deviation (lower row) over 10 training/test set splits. Bold fonts indicate the best results in each row.

	Datasets	DCA1-PiE	DCA1-Capped-ℓ 1	DCA2-PiE	DCA2-Capped-ℓ 1	PLDA	RDA	DSDA
	ADV	94.17 ± 0.33	94.25± 0.29	94.17 ± 0.33	94.16 ± 0.28	94.23 ± 0.31	96.47 ± 0.26 96.1 ± 0.65
		25.84 ± 0.36	30.02 ± 1.13	25.84 ± 0.36	22.52 ± 0.21	33.18 ± 0.78	27.14 ± 0.92	26.56 ± 1.62
	COL	81.1 ± 4.89	81.57 ± 5.08	81.09 ± 4.89	82.52 ± 6.47	78.78 ± 6.54	71.07 ± 7.5	81.48 ± 6.36
		0.12 ± 0.09	0.1 ± 0.04	0.16 ± 0.08	0.29 ± 0.1	21.7 ± 7.99	96.59 ± 0.45	1.11 ± 0.14
	SRB	99.62 ± 1.15 99.27 ± 1.46	99.61 ± 1.15	98.92 ± 1.64	97.44 ± 3.66	99.29 ± 1.4	-
		9.84 ± 17.39 23.93 ± 29.23	14.28 ± 21.23	17.82 ± 24.44	57.4 ± 6.37	15.36 ± 0.4	-
	PEN	100 ± 0	100± 0	100± 0	96.66 ± 4.08	100± 0	96.66 ± 2.52	-
		0.09 ± 0.01	61.59 ± 2.12	0.09 ± 0.01	5.18 ± 4.25	63.47 ± 3.02	94.28 ± 0.03	-
	GIS	86.92 ± 0.47	87.45 ± 0.63	86.54 ± 0.66	86.88 ± 0.66	86.88 ± 0.63	84.52 ± 0.66	94.07 ± 0.33
		28.51 ± 4.28	50.56 ± 5.18	22.6 ± 0.17	28.26 ± 0.21	28.27 ± 0.21	98.67 ± 0.16	35.31 ± 0.4
	LUN	99.34 ± 0.81 99.34 ±0.81	99.34 ± 0.81 99.17 ± 0.83	99.17 ± 0.83	98.18 ± 1.38	94.86 ± 2.5
		20.01 ± 1.21	12.31 ± 10.1	20.81 ± 0.77	17.94 ± 5.98	20.86 ± 2.83	98.67 ± 0.12	20.82 ± 3.41
	LEU	96.87 ± 1.16	96.99 ± 1.32	96.87 ± 1.21	97.11 ± 1.33	96.86 ± 1.33	97.06 ± 1.59	-
		29.36 ± 12.51	28.21 ± 13.97	35.55 ± 0.24	30.56 ± 1.57	35.21 ± 0.23	83.6 ± 0.06	-
	NAK	87. 82 ± 6.18	89.23 ± 4.23	86.76 ± 7.08	87.1 ± 7.13	87.1 ± 7.13	89.85 ± 4.61 -
		69.05 ± 8.82	57.61 ± 29.22	69.33 ± 5.58	69.74 ± 6.13	68.4 ± 7.12	99.99 ± 0.006	-
	SUN	70.83 ± 4.07	72.33 ± 3.75	69.66 ± 4.72	69.66 ± 4.72	69.66 ± 4.72	67.67 ± 4.53	-
		16.24 ± 14.33	9.02 ± 13.83	33.93 ± 1.21	33.17 ± 4.62	35.34 ± 0.81	100 ± 0	-

Table 2 .

 2 5: Comparative results of DCA1-PiE, DCA1-Capped-ℓ 1 , DCA2-PiE, DCA2-Capped-ℓ 1 , PLDA, RDA and DSDA in terms of the average of CPU time in second and its standard deviation (upper row) over 10 training/test set splits, the number of discriminant vectors used K (the data is projected onto a K-dimensional space) (lower row). Bold fonts indicate the best results in each row.

	Datasets	DCA1-PiE	DCA1-Capped-ℓ 1	DCA2-PiE	DCA2-Capped-ℓ 1	PLDA	RDA	DSDA
	ADV	0.001 ± 0.003 0.003 ± 0.006	0.007 ± 0.007	0.008 ± 0.008	0.004 ± 0.007	58.24 ± 1.05	7.19 ± 0.32
		1	1	1	1	1	-	1
	COL	0.024 ± 0.019	0.001 ± 0.003	0.033 ± 0.017	0.004 ± 0.006	0.01 ± 0.014	0.067 ± 0.009	0.25 ± 0.009
		1	1	1	1	1	-	1
	SRB	0.141 ± 0.026	0.048 ± 0.037	0.169 ± 0.046	0.061 ± 0.04	0.079 ± 0.01	0.082 ± 0.008	-
		3	3	3	3	3	-	-
	PEN	0.046 ± 0.018	0.056 ± 0.023	0.051 ± 0.021	0.03 ± 0.01	0.033 ± 0.01	0.109 ± 0.009	-
		2	2	2	2	2	-	-
	GIS	0.004 ± 0.006 0.008 ± 0.012	0.014 ± 0.008	0.013 ± 0.016	0.044 ± 0.083	829.918± 63.815	158.64 ± 9.37
		1	1	1	1	1	-	1
	LUN	0.12 ± 0.045	0.036 ± 0.016	0.095 ± 0.032	0.101 ± 0.052	0.063 ± 0.012	1.052 ± 0.036	19.02 ± 0.39
		1	1	1	1	1	-	1
	LEU	0.39 ± 0.306	0.27 ± 0.089	0.248 ± 0.016	0.38 ± 0.021	0.21 ± 0.013 1.71± 0.085	-
		5	5	5	5	5	-	-
	NAK	0.971 ± 0.19	0.664 ± 0.246	0.907 ± 0.19	0.716 ± 0.19	0.749 ± 0.128	1.185 ± 0.972	-
		4	4	4	4	4	-	-
	SUN	1.527 ± 1.007	0.738 ± 0.593	1.173 ± 0.23	0.961 ± 0.197	0.996 ± 0.156	6.819 ± 3.843	-
		2	2	2	2	2	-	-

Table 2 .

 2 6: Comparative results of MNIST dataset in terms of the number of selected features, the percentage of accuracy of classifiers on the test set, and training time in second. Bold fonts indicate the best results in each column.

	Method	Selected features (%)	Accuracy of classifiers (%) CPU in second
	DCA1-PiE	65.17	88.79	0.09
	DCA1-Capped-ℓ 1	65.94	88.77	0.15
	DCA2-PiE	65.17	88.79	0.11
	DCA2-Capped-ℓ 1	65.94	88.77	0.13
	PLDA	80.09	81.11	0.05
	RDA	-	-	-

  1) otherwise, where N(µ, σ 2 ) denotes the Gaussian distribution with mean µ and variance σ 2 . A total of 300 training samples, 300 tuning samples and 1500 test samples are generated with equal probabilities for each class.

The real world datasets consist of two real datasets from UCI Machine Learning Repository and NIPS 2003 Feature Selection Challenge (Internet Advertisement, Gisette), nine real microarray gene expression datasets, and one dataset for handwritten character recognition (MNIST). All the datasets are preprocessed by normalizing each dimension of the

Table 3 .

 3

	1: Real datasets used in experiments.
	Datasets	#Features #Samples	#Classes
	Internet Advertisement (ADV) 1558	3279	2
	Colon Tumor 1 (COL)	2000	62	2
	SRBCT 2 (SRB)	2308	83	4
	Pencillium (PEN)	3754	36	3
	Gisette (GIS)	5000	7000	2
	ALL/AML 3 (ALL)	7129	72	3
	Lung Cancer 4 (LUN)	12533	181	2
	Leukemia 5 (LEU)	12558	248	6
	MLL-Leukemia 6 (MLL)	12582	72	3
	Protaste (PRO)	12600	136	2
	Ovarian Cancer 7 (OVA)	15154	253	2
	MNIST 8

Table 3 .

 3 2: Comparative results of ADCA1-Exp, ADCA1-Cap, ADCA2-Exp, ADCA2-Cap, PLDA, SDA, RDA, SPLS DA and S SVM on the synthetic data. Bold fonts indicate the best results in each row.

			ADCA1-Exp	ADCA1-Cap	ADCA2-Exp	ADCA2-Cap PLDA	SDA	RDA	SPLS DA	S SVM
	S1	ACC	100 (0)	100 (0)	100 (0)	100(0)	82.94 (9.77)	100 (0)	99.98 (0.04)	99.99 (0.02)	100 (0)
		#FS	105.6 (3.28)	107.30 (1.55)	97.1 (2.68)	108.4 (4.17)	170.2 (19.93)	114.09 (13.1)	104.9 (0.3)	140.6 (62.72)	154 (4.75)
		%FS	21.12 (0.67)	21.46 (0.31)	19.42 (0.56)	21.68 (0.83)	34.04 (3.98)	22.81 (2.62)	20.98 (0.06)	28.12 (12.54)	30.8 (0.95)
		#DV	2	2	2	2	2	2	-	-	-
		CPUs 1.15 (0.26)	1.32 (0.17)	0.79 (0.16)	1.02 (0.15)	0.42 (0.06)	4.93 (3.71)	0.17 (0.008)	0.11 (0.01)	0.23 (0.51)
	S2	ACC	92.62 (1.15)	94.82 (0.58)	93.57 (0.71)	95.19 (0.53)	96.62 (0.45)	94.41 (0.64)	92.26 (1.58)	95.36 (1.68)	92.4 (0.82)
		#FS	102.1 (4.76)	89.4 (4.35)	89.8 (3.05)	100.4 (3.29)	159 (9.34)	108.1 (4.57)	134.2 (13.64)	98 (21.12)	98.5 (6.92)
		%FS	20.1 (0.95)	17.88 (0.87)	17.96 (0.61)	20.08 (0.65)	31.9 (1.86)	21.62 (0.91)	26.84 (2.72)	19.6 (4.22)	19.7 (1.38)
		#DV	1	1	1	1	1	1	-	-	-
		CPUs 0.1 (0.03)	0.05 (0.01)	0.11 (0.01)	0.05 (0.009)	0.19 (0.12)	0.36 (0.04)	0.03 (0.004)	0.03 (0.01)	0.11 (0.02)
	S3	ACC	96.26 (0.65)	97.09 (0.56)	96.19 (0.81)	97.09 (0.65)	96.58 (0.34)	96.85 (0.59)	97 (0.6)	96.83 (0.14)	50.65 (1.68)
		#FS	110.3 (4.9)	116.6 (5.4)	96.8 (6.4)	92.6 (4.29)	293.7 (7.53)	113.6 (5.44)	240.7 (8.1)	123.8 (5.77)	170.5 (8.08)
		%FS	22.06 (0.98)	23.32 (1.08)	19.36 (1.28)	18.52 (0.85)	58.74 (1.5)	22.72 (1.08)	48.14 (1.62)	24.76 (1.15)	34.1 (1.61)
		#DV	1	1	1	1	1	1	-	-	-
		CPUs 0.91 (0.24)	0.37 (0.06)	0.67 (0.18)	0.31 (0.04)	0.49 (0.17)	11.64 (3.08)	0.03 (0.006)	0.02 (0.008)	0.12 (0.2.39)

Table 3 .

 3 3: Comparative results of ADCA1-Exp, ADCA1-Cap, ADCA2-Exp, ADCA2-Cap, PLDA, SDA, RDA, SPLS DA and S SVM in terms of the average number of selected features and its standard deviation (upper row), and the average percentage of selected features and its standard deviation (lower row) over 10 training/test set splits. Bold fonts indicate the best results in each row.

		ADCA1-Exp	ADCA1-Cap	ADCA2-Exp	ADCA2-Cap	PLDA	SDA	RDA	SPLS DA	S SVM
	ADV 34.7 (3.16)	253.3 (22.85)	144.5 (7.68)	259.8 (25.6)	516.9 (12.16)	321.9 (30.9)	426 (10.58)	1542 (7.93)	347 (44.55)
		2.22 (0.20)	16.11 (1.47)	9.27 (0.49)	16.67 (1.64)	33.17 (0.78)	20.66 (1.97)	27.34 (0.67)	99.03 (0.5)	22.27 (2.85)
	COL	5.80 (1.47)	23.9 (9.4)	6.5 (2.5)	29.5 (3.17)	434 (159.84)	31.5 (3.07)	1936.3 (11.39)	35 (5.23)	26.8 (4.66)
		0.29 (0.07)	1.19 (0.47)	0.32 (0.12)	1.47 (0.15)	21.7 (7.99)	1.57 (0.15)	96.81 (0.56)	1.75 (0.26)	1.34 (0.23)
	SRB	54.08(7.25)	70.6 (6)	32.3 (2.86)	73.7 (8.36)	1324.9 (147.15)	77.9 (3.91)	281 (5.16)	563.6 (179.14)	99.25 (1.56)
		2.37(0.31)	3.05 (0.26)	1.39 (0.12)	3.19 (0.36)	57.4 (6.37)	3.37 (0.16)	12.2 (0.22)	24.41 (7.76)	3.7 (0.25)
	PEN	2(0)	22.0 (4.42)	4.2 (2.56)	23.8 (5.13)	2383 (113.44)	4 (0)	3539.3 (2.23)	2679.1 (297.13)	26.5 (3.47)
		0.05 (0)	0.59 (0.12)	0.11 (0.06)	0.63 (0.13)	63.47 (3.02)	0.1 (0)	94.28 (0.06)	71.36 (7.91)	0.7 (0.09)
	GIS	1015.7 (24.56)	1054.6 (25.49)	960.5 (25.26)	1262.1 (21.97)	1413.7 (10.19)	1562.3 (24.68)	4933.5 (8.2)	3884.5 (23.76)	1124.4 (21.44)
		20.31 (0.49)	21.09 (0.5)	19.21 (0.5)	25.24 (0.43)	28.27 (0.21)	31.24 (0.49)	98.67 (0.16)	77.69 (0.47)	22.48 (0.42)
	ALL	9.6 (1.57)	63.1 (11.4)	14.9 (2.36)	60.4 (5.06)	2701 (183.54)	71.4 (4.45)	7128.9 (0.3)	90.6 (15.23)	81.2 (7.85)
		0.13 (0.02)	0.88 (0.16)	0.2 (0.03)	0.84 (0.07)	37.88 (2.57)	1 (0.06)	99.99 (0.004)	1.27 (0.21)	1.13 (0.11)
	LUN	3.7 (0.43)	27.8 (2.44)	3.8 (0.6)	37.7 (4.73)	2614.2 (354.48)	47.2 (5.38)	12341.1 (19.66)	223.6 (14.45)	46.6 (5.23)
		0.02 (0.003)	0.22 (0.01)	0.3 (0.004)	0.3 (0.03)	20.85 (2.82)	0.37 (0.04)	98.25 (0.15)	1.78 (0.11)	0.37 (0.04)
	LEU	21.8 (2.61)	37.9 (5.65)	22.4 (1.68)	52.6 (9.11)	4421.9 (29.04)	44 (5.47)	10500.2 (6.66)	1438.8 (7.39)	313.1 (15.78)
		0.17 (0.02)	0.3 (0.05)	0.17 (0.01)	0.41 (0.07)	35.21 (0.23)	0.35 (0.04)	83.61 (0.05)	11.45 (5.89)	2.49 (0.12)
	MLL	96.3 (9.62)	132.6 (15.09)	103.1 (8.98)	150.5 (9.64)	6296.4 (147.73)	103.2 (3.69)	12581.3 (0.78)	456.7 (86.43)	142.9 (7.56)
		0.76 (0.07)	1.05 (0.12)	0.81 (0.07)	1.19 (0.07)	50.04 (1.17)	0.82 (0.03)	99.99 (0.01)	3.62 (0.68)	1.13 (0.06)
	PRO	69.8 (12.41)	45.5 (9.68)	51.6 (5.12)	55.9 (3.8)	75.4 (3.38)	68.7 (3.74)	12600 (0)	11174.9 (9.95)	76.4 (7.83)
		0.55 (0.09)	0.36 (0.07)	0.4 (0.04)	0.44 (0.03)	0.59 (0.02)	0.54 (0.02)	100 (0)	9.32 (0.55)	0.6 (0.06)
	OVA	4 (0.47)	27.7 (2.35)	4.1 (0.3)	28.2 (2.44)	4160.8 (80.6)	8.1 (1.13)	93.8 (8.58)	60.4 (14.8)	56.8 (9.48)
		0.02 (0.003)	0.18 (0.01)	0.02 (0.001)	0.18(0.01)	27.45 (0.53)	0.05 (0.01)	0.62 (0.06)	0.39 (0.09)	0.37 (0.06)

Table 3 .

 3 4: ADCA1-Exp, ADCA1-Cap, ADCA2-Exp, ADCA2-Cap, PLDA, SDA, RDA, SPLS DA and S SVM in terms of the average of percentage of accuracy of classifiers and its standard deviation (first row) over 10 training/test set splits, the number of discriminant vectors used K (the data is projected onto a K-dimensional space) (second row), and the average of training time in second and its standard deviation (third row). Bold fonts indicate the best results in each row.

		ADCA1-Exp	ADCA1-Cap	ADCA2-Exp	ADCA2-Cap	PLDA	SDA	RDA	SPLS DA	S SVM
	ADV 96.94 (0.32)	97.27 (0.30)	97.28 (0.31)	97.27 (0.28)	94.23 (0.31)	97.23 (0.27)	96.45 (0.36)	97.14 (0.17)	97.25 (0.92)
		1	1	1	1	1	1	-	-	-
		2.89 (0.64)	2.41 (0.46)	2.31 (0.88)	2(0.3)	0.004(0.007) 47.97 (3.21)	62.55 (0.96)	31.74 (3.13)	2.03 (0.14)
	COL	84.32 (4.49)	86.32 (5.94)	83.35 (3.86)	84.87 (5.81)	78.78 (6.54)	82.43 (7.89)	79.39 (5.8)	85.3 (4.89)	78.73 (4.62)
		1	1	1	1	1	1	-	-	-
		0.95 (0.5)	0.23 (0.23)	1.36 (0.76)	0.33 (0.24)	0.01 (0.001)	0.34 (0.03)	0.23 (0.01)	0.01 (0.01)	0.06 (0.01)
	SRB	98.17 (2.60)	99.64 (1.12)	98.19 (3.72)	99.27 (1.54)	97.44 (3.66)	98.54 (1.78)	96.73 (4.69)	97.44 (3.99)	99.25 (1.56)
		3	3	3	3	3	3	-	-	-
		16.56 (3.22)	5.13 (1.01)	16.07 (3.74)	5.85 (1.58)	0.07 (0.01)	19.23 (6.81)	0.08 (0.01)	7.07 (2.08)	0.19 (0.01)
	PEN	100 (0)	100 (0)	100 (0)	100 (0)	100 (0)	100(0)	92.5 (10.83)	98.33 (3.33)	100 (0)
		2	2	2	2	2	2	-	-	-
		4.25 (1.95)	7.08 (1.88)	5.23 (1.96)	6.57 (4.91)	0.03 (0.01)	0.46 (0.36)	0.12 (0.01)	23.68 (3.62)	0.28 (0.03)
	GIS	97.64 (0.24)	97.59 ( 0.23)	97.39 (0.26)	97.51 (0.22)	86.88 (0.63)	97.58 (0.25)	84.52 (0.65)	96.39 (0.25)	97.6 (0.29)
		1	1	1	1	1	1	-	-	-
		30.11 (16.02)	48.51 (5.97)	24.03 (9.17)	32.56 (1.22)	0.04 (0.08)	3665.69 (51.12)	829.91 (63.81)	256.59 (41.76)	14.62 (2.57)
	ALL	95.43 (2.30)	97.06 (2.03)	94.58 (3.25)	95.83 (2.63)	92.13 (6.68)	95.83 (2.63)	95.44 (3.9)	96.26 (4.35)	94.88 (4.43)
		2	2	2	2	2	2	-	-	-
		39.55 (8.92)	48.8 (9.87)	58.25 (13.22)	52.66 (16.44)	0.05 (0.03)	52.2 (21.1)	0.42 (0.01)	31.24 (13.21)	0.47 (0.08)
	LUN	98.67 (1.05)	98.84 (1.12)	98.17 (1.88)	98.17 (1.74)	99.16 (0.73)	98.66 (1.45)	98 (1.45)	97.84 (1.29)	98.67 (1.52)
		1	1	1	1	1	1	-	-	-
		41.88 (6.59)	33.86 (3.15)	53.52 (25.82)	44.89 (6.15)	0.06 (0.01)	6.25 (0.77)	1.07 (0.05)	0.21 (0.02)	0.90 (0.04)
	LEU	94.69 (2.5)	95.52 (2.49)	94.08 (2.75)	95.05 (1.46)	96.86 (1.33)	95.88 (1.71)	98.42 (1.08)	92.25 (1.65)	96.49 (1.83)
		5	5	5	5	5	5	-	-	-
		284.41 (52.78)	106.05 (30.65)	205.98 (34.32)	117.65 (15.71)	0.21 (0.01)	53.17 (22.66)	1.66 (0.03)	168.37 (39.78)	2.23 (0.08)
	MLL	97.08 (2.34)	97.92 (2.19)	95.41 (4.73)	94.58 (4.58)	88.67 (7.41)	95.41 (5.08)	82.06 (17.27)	95.81 (3.72)	97.11 (4.4)
		2	2	2	2	2	2	-	-	-
		159.85 (33.32)	163.18 (51.26)	170.74 (42.5	159.44 (59.59)	0.15 (0.04)	157.96 (74.33)	0.49(0.01)	99.09 (41.01)	0.87 (0.07)
	PRO	81.52 (5.97)	81.51 (2.79)	78.21 (6.06)	80.83 (34.68)	78.17 (6.17)	79.76 (5.73)	87.91 (2.95)	82.62 (3.49)	80.5 (6.06)
		1	1	1	1	1	1	-	-	-
		57.21 (7.51)	30.58 (10.94)	63.77 (26.41)	27.11 (10.89)	1.73 (0.11)	14.3 (1.87)	0.77 (0.04)	0.31 (0.02)	0.62 (0.03)
	OVA	100 (0)	100 (0)	100 (0)	100 (0)	89.57 (4.27)	100 (0)	99.64 (0.54)	99.05 (0.88)	100 (0)
		1	1	1	1	1	1	-	-	-
		64.32 (25.95)	31.7 (7.27)	80.51 (30.12)	33.39 (7.93)	4.71 (0.1)	2.08 (0.24)	2.03 (0.06)	0.38 (0.02)	1.47 (0.05)

Table 3 .

 3 5: Comparative results of MNIST dataset in terms of the number (percentage) of selected features #FS (%FS), the percentage of accuracy of classifiers (ACC) on the test set, and training time in second. Bold fonts indicate the best results in each column. various classifiers trained on the training set. The computational results are reported on Table3.5. Notably in this dataset, RDA is not able to perform since the amount of RAM is insufficient. From Table3.5, we observe that the DCA based algorithms outperform PLDA, SDA, SPLS DA and S SVM in term of selected features and ADCA2-Exp is the best. In term of accuracy, the DCA based algorithms are also better than the other approaches. For the training time, PLDA is the fastest. Contrary to other datasets, here DCA based algorithm require much more training time, since the number of samples is very large.

	Method	#FS (%FS) ACC	Training time
	ADCA1-Exp	492 (62.75)	89.8	765.18
	ADCA1-Cap 529 (67.47)	89.82 937.98
	ADCA2-Exp	491 (62.62)	89.79	869.8
	ADCA2-Cap 530 (67.6)	89.69	733.35
	PLDA	610 (77.8)	81.59	100.84
	SDA	641 (81.76)	87.43	317143.43
	RDA	-	-	-
	SPLS DA	715 (91.19)	86.34	717.05
	S SVM	655 (83.54)	89.07	464.06

Table 3 .

 3 6: Comparative results of ADCA1-Cap, DCA1-Cap and SMSVM-Cap. Bold fonts indicate the best results.

		Accuracy of classifiers	Number/percentage of selected features		CPU time in second
		ADCA1-Cap	DCA1-Cap	SMSVM-Cap ADCA1-Cap	DCA1-Cap	SMSVM-Cap ADCA1-Cap	DCA1-Cap	SMSVM-Cap
	S1	100 (0)	100 (0)	99.99 (0.02)	107.3 (1.55)	110 (3)	115.4 (2.6)	1.32 (0.17)	0.012 (0.001)	66.6 (2.3)
					21.46 (0.31)	22 (0.6)	23 (0.5)
	S2	94.82 (0.58)	94.5 (0.8)	93.25 (0.5)	89.4 (4.35)	100.9 (6.5)	113.1 (3.9)	0.05 (0.01)	0.002 (0.001)	10.1 (0.3)
					17.88 (0.87)	20.1 (1.3)	22.6 (0.7)
	S3	97.09 (0.56)	96.66 (1.2)	50.77 (2)	116.6 (5.4)	114 (6.6)	135.3 (6.3)	0.37 (0.06)	0.005 (0.003)	41 (1)
					23.32 (1.08)	22.8 (1.3)	27 (1.2)
	ADV 97.27 (0.3)	94.25 (0.29)	96.14 (0.3)	253.3 (22.85)	467.71 (17.6)	46.5 (5.7)	2.41 (0.46)	0.003 (0.006)	10.4 (0.1)
					16.11 (1.47)	30.02 (1.13)	2.9 (0.3)
	COL	86.32 (5.94)	81.57 (5.08)	77.05 (6.1)	23.9 (9.4)	2 (0.8)	19 (2.5)	0.23 (0.23)	0.001 (0.003)	11.2 (1.9)
					1.19 (0.47)	0.1 (0.04)	0.9 (0.1)
	SRB	99.64 (1.12)	99.27 (1.46)	98.92 (1.6)	70.6 (6)	552.3 (674.62)	27.8 (3)	5.13 (1.01)	0.048 (0.037)	378.6 (28.8)
					3.05 (0.26)	23.93 (29.23)	1.2 (0.1)
	PEN	100 (0)	100 (0)	96 (5.8)	22.0 (4.42)	2312.08 (79.58)	31.5 (1.7)	7.08 (1.88)	0.056 (0.023)	89.1 (7.4)
					0.59 (0.12)	61.59 (2.12)	0.84 (0.04)
	GIS	97.59 (0.23)	87.45 (0.63)	97.04 (0.2)	1054.6 (25.49)	2528 (259)	627.1 (24.6)	48.51 (5.97)	0.008 (0.012)	4263 (2094.6)
					21.09 (0.5)	50.56 (5.18)	12.5 (0.4)
	ALL	97.06 (2.03)	96.24 (3.64)	92.27 (4.7)	63.1 (11.4)	2437.9 (268.14)	74.2 (7.8)	48.8 (9.87)	7.47 (0.05)	572.9 (283.2)
					0.88 (0.16)	34.19 (3.76)	1.04 (0.1)
	LUN	98.84 (1.12)	99.34 (0.81)	98.45 (1.8)	27.8 (2.44)	1542.81 (1265.83)	37.2 (4.7)	33.86 (3.15)	0.036 (0.016)	837.28 (201.73)
					0.22 (0.01)	12.31 (10.1)	0.29 (0.03)
	LEU	95.52 (2.49)	96.99 (1.32)	93.2 (3.1)	37.9 (5.65)	3549.38 (1748.89)	173.8 (28.54)	106.05 (30.65)	0.27 (0.089)	5482.48 (284.67)
					0.3 (0.05)	28.21 (13.9)	1.38 (0.22)
	MLL	97.92 (2.19)	98.48 (2.17)	92.85 (3.17)	132.6 (15.09)	6398 (388.61)	236.1 (8.26)	163.18 (51.28)	7.68 (0.13)	3719.3 (1725.52)
					1.05 (0.12)	50.85 (3.08)	1.87 (0.06)
	PRO	81.51 (2.79)	89.92 (2.1)	89.46 (4.3)	45.5(9.68)	5898.06 (2963.52)	26.4	30.58 (10.94)	0.82 (0.03)	486.9 (45.6)
					0.36 (0.07)	46.81 (23.52)	0.2 (0.02)
	OVA	100 (0)	100 (0)	100 (0)	27.7 (2.35)	4870.49 (2712.56)	30.1 (3.1)	31.7 (7.27)	1.36 (0.004)	3356.7 (146.6)
					0.18 (0.01)	32.14 (17.9)	0.2 (0.02)

  . Applying DCA on two DC formulations with two approximations, we have then four DCA based algorithms for the approximate SCME problem. Special convergence analysis results of our algorithms are provided. We consider two important applications of the SCME problem in our experiments.

	Section 4.3 and, finally, Section 4.4 concludes the chapter.
	4.2 DCA for solving the sparse covariance matrix es-
	timation SCME problem
	4.2.1 The approximation SCME problem
	The first
	is the quadratic discriminant analysis using sparse covariance matrices estimated by the
	proposed algorithms. The second is a portfolio optimization problem. Numerical exper-
	iments are carefully achieved on several test problems on both simulated datasets and
	real datasets with 11 algorithms including 7 state-of-the-art methods and the 4 proposed
	DCA schemes.

The rest of the chapter is organized as follows. The DCA based methods for solving the SCME problem is presented in Section 4.2. The numerical experiments are reported in

  4.3.1.1 Constrained ℓ 1 mInimization approach to sparse precision Matrix Estimation (CLIME)CLIME(Cai et al., 2011) estimated an inverse covariance matrix Ω = Σ -1 by solving the following problem minΩ∈R p×p { Ω 1 : SΩ -I ∞ ≤ λ} , (4.36)where λ is a tuning parameter. The convex program (4.36) can be further decomposed into p vector minimization problems which is solved by the primal dual interior method.The clime package for CLIME is also available from CRAN.SPME[START_REF] Zhang | Sparse precision matrix estimation via lasso penalized d-trace loss[END_REF]) used a surrogate loss function instead of the negative log likelihood function with the ℓ 1 penalty on the precision matrix Θ = Σ -1 , namely

	4.3.1.2 Sparse Precision Matrix Estimation via lasso penalized D-trace loss
	(SPME)
	min Θ ǫI Θ 4.3.1.3 1 2 Positive Definite Sparse Covariance Estimators (PDSCE)
	Rothman (2012) proposed the sparse covariance matrix estimator by solving the following
	problem
	min Σ≻0

2

, Str(Θ) + λ Θ 1,off ,

(4.37)

where λ is a tuning parameter and Θ 1,off = i =j |Θ ij |. Zhang and Zou (2014) used the alternating direction method of multipliers for solving the problem (4.37). The source code of this method is available on the author' homepage (https://math.cos.ucf.edu/ tengz/).

  used an estimate of covariance matrix Σ = is a tuning parameter. The MATLAB code for PCME is available on http:// www.tandfonline.com/doi/suppl/10.1080/10618600.2012.715556.

	4.3.1.5	Sparse estimation of a COVariance Matrix (SPCOV)
	Bien and Tibshirani (2011) considered the negative log-likelihood function with ℓ 1 penalty
	on the entries of the covariance matrix, namely
		min Σ≻0	log det Σ + tr(Σ -1 S) + λ||W • Σ|| 1 ,	(4.40)
	where W is an arbitrary matrix with nonnegative elements. The problem (4.40) is non-
	convex. Bien and Tibshirani (2011) used the MM approach for solving this problem (that
	is in fact a version of DCA). Denote by SPCOV1 (reps. SPCOV2) the MM approach for
	solving the problem (4.40) with W

∞ k=0

Âk k! ≡ exp( Â), where  solves the following problem min A∈S p tr(A) + tr[exp(-A)S] + λtr(A 2 ) ,

(4.39)

where λ ij = 0 if i = j and 1 otherwise (resp. with

Table 4 .

 4 1: Comparative results in terms of the average of root-mean-square error (RMSE), entropy loss (EN), Kullback-Leibler loss (KL), number of nonzero elements, CPU time in second (and their standard deviations) over 10 runs. Bold fonts indicate the best result in each row.

			DCA1-CaP	DCA2-CaP	DCA1-PiE	DCA2-PiE	SPCOV1	SPCOV2	CLIME	PCME	SPME	PDSCE
		RMSE	0.399	0.381	0.456	0.413	0.398	0.384	0.387	0.474	0.544	0.418
			(0.003)	(0.004)	(0.01)	(0.008)	(0.0.004)	(0.005)	(0.004)	(0.003)	(0.005)	(0.004)
		EN	15.37	14.15	14.99	13.54	15.77	16.83	83.54	27.85	31.81	80.42
	Cliques	KL NZ	(0.95) 23.23 (4.11) 2623	(0.61) 20.56 (1.22) 2419.6	(1.07) 15.77 (1.28) 1011.4	(0.7) 14.29 (0.59) 1018.2	(0.57) 31.43 (2.23) 3775.4	(0.53) 34.39 (2.04) 3565.8	(3.32) 21.33 (0.32) 8998.4	(0.36) 71.23 (1.94) 9847	(0.62) 47.12 (1.33) 2624.4	(2.68) 21.02 (0.21) 2755.6
			(510.08)	(209.34)	(30.21)	(18.59)	(156.13)	(94.57)	(18.37)	(15.72)	(821.28)	(72.65)
		CPUs	338.3	70.77	130.97	114.06	180.04	112.99	687.95	223.87	33.51	2.72
			(103.55)	(14.08)	(14.91)	(61.96)	(17.84)	(8.9)	(21.12)	(1.36)	(1.3)	(0.06)
		RMSE	0.085	0.077	0.062	0.072	0.073	0.073	0.194	0.237	0.183	0.109
			(0.004)	(0.004)	(0.008)	(0.006)	(0.0.003)	(0.004)	(0.084)	(0.002)	(0.007)	(0.003)
	Hubs	EN KL	3.32 (0.25) 4.44	3.09 (0.34) 3.83	1.53 (0.23) 1.72	1.98 (0.38) 2.2	3.54 (0.27) 5.79	2.63 ( 0.21) 3.6	264.8 (56.5) 18.94	29.03 (0.76) 64.92	61.46 (1.53) 22.41	141.61 (2.35) 13.99
			(0.38)	(0.35)	(0.27)	(0.34)	(0.73 )	(0.29)	(3.85)	(2.45)	(0.82)	(0.19)
		NZ	552.6	530.2	301.2	328.4	879	582.2	1174	9493.8	4640	597.6
			(25.02)	(21.36)	(5.38)	(12.51)	(33.06)	(24.5)	(124.21)	(30.68)	(1195.54)	(30.56)
		CPUs	96.1	53. 43	74.63	49.52	99.73	86.31	676.59	279.42	29.68	3.24
			(7.29)	(11.81)	(7.77)	(23)	(6.42)	(4.51)	(26.89)	(2.16)	(0.83)	(0.23)
		RMSE	0.096	0.086	0.051	0.052	0.086	0.066	0.089	0.177	0.125	0.083
			(0.001)	(0.002)	(0.003)	(0.003)	(0.0.002)	(0.002)	(0.002)	(0.001)	(0.003)	(0.002)
	Random	EN KL	5.42 (0.43) 5.68 (0.42)	3.91 (0.15) 4.53 (0.22)	1.58 (0.21) 1.7 (0.24)	1.61 (0.19) 1.74 (0.21)	3.9 (0.16) 5.07 (0.49)	2.47 (0.15) 3.02 (0.24)	32 (4.06) 7.41 (0.16)	23,57 (0.35) 52.61 (1.17)	16.21 (0.63) 12.17 (0.49)	27.69 (2.04) 6.67 (0.15)
		NZ	527.2	604.2	287.8	289	791.2	518.4	8972.6	9349.6	5932.8	614.22
			(26.73)	(31.6)	(4.68)	(5.31)	(46.64)	(20.09)	(449.65)	(37.82)	(943.41)	(37.28)
		CPUs	124.39	82.6	156.26	30.08	127.65	87.3	598.09	292.6	29.48	3.25
			(14.64)	(20.96)	(35.89)	(17.77)	(9)	(4.36)	(7.82)	(2.78)	(0.36)	(0.21)
		RMSE	0.009	0.01	0.015	0.012	0.038	0.015	0.025	0.05	0.044	0.021
			(0.0007)	(0.0004)	(0.0008)	(0.0008)	(0.0007)	(0.0009)	(0.0007)	(0.004)	(0.0009)	(0.001)
	Moving	EN KL	1.02 (0.11) 1.09 (0.19)	1.14 (0.09) 1.18 (0.15)	2.05 (0.15) 2.48 (0.22)	1.51 (0.15) 1.66 (0.21)	7.95 (0.38) 11.71 (0.82)	2.29 (0.17) 2.84 (0.25)	41.23 (1.65) 15.21 (0.3)	30.64 (11.84) 43.44 (8.2)	171.97 (4.23) 37.23 (0.89)	82.13 (3.5) 20.65 (0.63)
		NZ	298.4	298.4	451.4	361.6	1380	591.8	9986.4	8440.66	4311	640
			(1.2)	(1.2)	(26.3)	(14.41)	(53.4)	(25.16)	(6.24)	(213.4)	(2258.65)	(30.24)
		CPUs	48.91	6.67	86.4	27.81	110.86	78.98	679.42	258.07	29.45	8.85
			(5.28)	(1.39)	(6.13)	(4.1)	(6.31)	(6.59)	(6.41)	(27.93)	(0.34)	(0.41)

Table 4 .

 4 2: Two datasets from UCI repository used in experiments.

	Data	No. of features	No. of samples	No. of classes
	Ionosphere	34	351	2
	Waveform 2	40	5000	3

Table 4 .

 4 3: Digit classification results of 3s and 8s. Bold fonts indicate the best result in each column.

		Testing error (%) Training error (%) Training time (s)
	DCA1-CaP	4.81	2.08	129.53
	DCA2-CaP	2.71	1.91	34.97
	DCA1-PiE	4.51	3.08	113.13
	DCA2-PiE	3.01	2.16	39.5
	SPCOV1	4.81	2	110.43
	SPCOV2	4.51	3.16	87.39
	QDA	6.32	2.16	-
	CLIME	4.01	2.83	218.48
	PCME	4.31	2.25	64.18
	SPME	5.83	4.25	22.3
	PDSCE	5.32	3.33	7.45

Table 4 .

 4 4: Comparative results of Ionosphere and Waveform 2 datasets in terms of the average of percentage of testing errors, training errors, training time in second and their standard deviations over 10 training/test set splits. The bold font indicates the best result in each column.

			Testing error (%) Training error (%) Training time (s)
	Ionosphere	DCA1-CaP	6.58 ± 2.16	7.82 ± 1.5	18.95 ± 7.57
		DCA2-CaP	4.52 ± 1.66	6.23 ± 0.94	1.17 ± 0.15
		DCA1-PiE	6.75 ± 1.68	8.47 ± 1.49	15.08 ± 4.41
		DCA2-PiE	4.52 ± 1.62	6.58 ± 1.22	1.22 ± 0.23
		SPCOV1	6.41 ± 2.03	5.89 ± 0.87	21.44 ± 16.21
		SPCOV2	-	-	-
		QDA	11.62 ± 3.38	3.33 ± 0.49	-
		CLIME	10.96 ± 2.04	9.64 ± 1.19	33.96 ± 0.31
		PCME	11.25 ± 1.37	10.68 ± 0.43	19.6 ± 2.38
		SPME	7.17 ± 2.15	5.51 ± 0.68	8.62 ± 0.39
		PDSCE	10.38 ± 2.82	11.34 ± 1.91	3.97 ± 0.13
	Waveform 2	DCA1-CaP	14.33 ± 0.61	11.26 ± 0.68	12.3 ± 0.63
		DCA2-CaP	14.31 ± 0.36	10.87 ± 0.54	1.38 ± 0.87
		DCA1-PiE	14.27 ± 0.59	11.4 ± 0.69	13.5 ± 0.97
		DCA2-PiE	14.37 ± 0.52	11.73 ± 0.67	1.37 ± 0.57
		SPCOV1	14.91 ± 0.71	10.26 ± 0.24	9.75 ± 0.13
		SPCOV2	14.84 ± 0.67	10.28 ± 0.19	11.31 ± 0.35
		QDA	16.27 ± 0.57	9.09 ± 0.16	-
		CLIME	15.42 ± 0.84	10.45 ± 0.43	77.38 ± 0.36
		PCME	15.16 ± 0.73	9.79 ± 0.12	16.29 ± 0.78
		SPME	15.57 ± 0.65	14.68 ± 0.48	14.32 ± 0.73
		PDSCE	15.01 ± 0.93	10.89 ± 1.22	3.71 ± 0.11

Table 4 .

 4 5: The comparison of the realized return, realized risk and Sharpe ratio. Bold fonts indicate the best result in each column.

		R(w)	σ(w)	Sharpe ratio	Training time (s)
	DCA1-CaP	0.2398	0.0303	7.9038	14.52
	DCA2-CaP	0.2429	0.028	8.6613	0.33
	DCA1-PiE	0.2441	0.0309	7.8788	4.51
	DCA2-PiE	0.27	0.0303	8.886	0.22
	SPCOV1	0.2398	0.0303	7.9038	14.57
	SPCOV2	0.2567	0.0314	8.1566	4.66
	S	0.0593	0.0348	1.7045	-
	CLIME	0.1121	0.0237	4.73	13.77
	PCME	0.2172	0.0307	7.07	0.65
	SPME	0.2272	0.0302	7.51	1.09
	PDSCE	0.1258	0.0343	3.67	0.61

  are equivalent in the following sense. Proposition 5.1 A point (x * , y * ) ∈ K is a global (resp. local) solution of the problem (5.9) if and only if (x * , y * , |x * J | p ) is a global (resp. local) solution to the problem (5.11). Moreover, if (x * , y * , z * ) is a global solution to

Table 5 .

 5

		1: Real datasets used in experiments.
	Datasets	#Features #Samples #Classes Area
	SRBCT 1 (SRB)	2308	83	4	Microarray, Bio
	Pencillium (PEN)	3754	36	3	Microarray, Bio
	ALL/AML 2 (ALL)	7129	72	3	Microarray, Bio
	Leukemia 3 (LEU)	12558	248	6	Microarray, Bio
	MLL-Leukemia 4 (MLL) 12582	72	3	Microarray, Bio
	CLL-SUB-111 5 (CLL)	11340	111	3	Microarray, Bio
	TOX-171 5 (TOX)	5748	171	4	Microarray, Bio
	AR10P 5 (AR1)	2400	130	10	Image, Face
	PIX10P 5 (PIX)	10000	100	10	Image, Face
	PIE10P 5 (PIE)	2420	210	10	Image, Face
	ORL10P 5 (ORL)	10304	100	10	Image, Face
	5.4.1.3 Experimental setups
	All algorithms are implemented in the R 3.0.2, and performed on a PC Intel i7 CPU3770,
	3.40 GHz of 8GB RAM.

Table 5 .

 5 3: ℓ 1,0 (DCA1), ℓ 1,0 (DCA2), ℓ 2,0 (DCA1), ℓ 2,0 (DCA2), SOS GLASSO and GS MSVM in terms of the average of percentage of accuracy of classifiers and its standard deviation (first row) over 10 training/test set splits, the number of used discriminant vectors L (the data is projected onto an L-dimensional space) (second row), and the average training time (in seconds) and its standard deviation (third row). Bold fonts indicate the best results in each row.

		ℓ 1,0 (DCA1)	ℓ 1,0 (DCA2)	ℓ 2,0 (DCA1)	ℓ 2,0 (DCA2)	SOS GLASSO	GS MSVM
	SRB	100 (0)	100 (0)	99.1 (1.56)	99.64 (1.12)	99.61 (1.21)	97.47 (4.54)
		3	3	3	3	3	-
		0.94	0.93 (0.01)	30.81 (4.72)	25.48 (6.59)	15.17 (0.18)	71.93 (9.17)
	PEN	100 (0)	100 (0)	100 (0)	100 (0)	100 (0)	100 (0)
		2	2	2	2	2	-
		1.76 (0.03)	2.63 (0.016)	13.27 (8.07)	12.66 (8.21)	18.52 (7.94)	111.08 (52.06)
	ALL	95.89 (3.31)	96.24 (3.07)	95.37 (5.32)	95.76 (7.1)	95.34(2.42)	94.6 (3.43)
		2	2	2	2	2	-
		8.66 (0.06)	12.83 (0.06)	108.42 (31.32)	95.24 (21.17)	83.58 (10.89)	76.31 (12.08)
	LEU	98.38 (1.29)	96.84 (0.85)	98.07 (1.18)	95.97 (1.55)	97.09 (1.64)	74.27 (5.47)
		5	5	5	5	5	-
		47.14 (0.88)	56.93 (1.3)	349.45 (107.35)	128.32 (108.93)	193.92 (62.45)	74.54 (191.96)
	MLL	98.76(1.98)	97.12 (1.99)	95.86 (3.35)	94.87 (5.68)	97.08 (2.87)	88.67 (2.62)
		2	2	2	2	2	-
		28.44 (1.4)	28.14 (0.47)	406.58 (89.77)	337.22 (99.36)	252.2 (17.28)	318.75 (65.91)
	CLL	86.47 (3.26)	85.99 (1.59)	84.28 (3.49)	77.31 (5.4)	78.88 (4.59)	64.72 (5.13)
		2	2	2	2	2	-
		21.98 (1.04)	32.72 (1.66)	304.79 (196.73)	310.78 (78.83)	279.57 (63.55)	164.45 (15.08)
	TOX	91.92 (1.69)	92.45 (1.44)	94.73 (1.84)	91.92 (3.11)	92.45 (3.51)	92.98 (5.72)
		2	2	2	2	2	-
		8.21 (0.7)	8.1 (0.17)	88.91 (22.4)	36.12 (19.34)	83.76 (12.54)	339.39 (24.82)

Table 5 .

 5 4: Comparative results of ℓ 0 /ℓ 1,0 (DCA1), ℓ 0 /ℓ 2,0 (DCA1), and ℓ 1 /ℓ 2,1 (DCA) in terms of the average of root-mean-square error (ARMSE), entropy loss (AEN), Kullback-Leibler loss (AKL), number of nonzero elements, CPU time in second (and their standard deviations) over 10 runs. Bold fonts indicate the best result in each row.

			ℓ 0 /ℓ 1,0 (DCA1)	ℓ 0 /ℓ 2,0 (DCA1)	ℓ 1 /ℓ 2,1 (DCA)
		ARMSE	0.381 (0.004)	0.415 (0.007)	0.445 (0.002)
		AEN	12.97 (1.07)	17.53 (1.12)	18.08 (2.62)
		AKL	17.31 (2.24)	18.58 (3.1)	31.86 (2.75)
		NZ1	1903.2 (298.61)	2138.6 (313.7)	2242.4 (271.5)
	Model 1	NZ2 NZ3	1781.6 (307.96) 1728.2 (288.12)	2172.18 (281.6) 2058.37 (215.5)	1464.6 (316.4) 1918.8 (251.2)
		NZ	5413 (892.69)	6369.15 (810.8)	5625.8 (839.1)
		CPUs	642.11 (3.74)	3180.56 (7.92)	3471.66 (5.18)
		ARMSE	0.082 (0.003)	0.09 (0.007)	0.094 (0.005)
		AEN	3.57 (0.52)	18.02 (1.31)	28.08 (2.16)
		AKL	3.93 (0.56)	6.65 (1.66)	7.76 (1.82)
	Model 2	NZ1 NZ2	352.8 (13.51) 259 (13.61)	394.18 (52.47) 347.45 (38.52)	448.6 (52.3) 378.27 (36.1)
		NZ3	255.8 (11.18)	359.72 (12.98)	264.61 (31.6)
		NZ	867.6 (38.3)	1101.35 (103.97)	1091.48 (120)
		CPUs	267.22 (39.33)	3843.57 (27.37)	5593.15 (24.61)

Table 5 .

 5 5: Comparative results of Ionosphere and Waveform 2 datasets in terms of the average of percentage of testing errors, training errors, training time in second and their standard deviations over 10 training/test set splits. The bold font indicates the best result in each column.

			Testing error (%) Training error (%) Training time (s)
	Ionosphere	ℓ 0 /ℓ 1,0 (DCA1)	5.13 (1.3)	3.41 (0.48)	0.094 (0.02)
		ℓ 0 /ℓ 2,0 (DCA1)	5.13 (0.54)	3.84 (0.72)	0.94 (0.01)
		ℓ 1 /ℓ 2,1 (DCA)	6.79 (1.78)	4.27 (0.82)	0.97 (0.04)
	Waveform 2	ℓ 0 /ℓ 1,0 (DCA1)	13.01 (0.25)	11.41 (1.3)	3.28 (1.14)
		ℓ 0 /ℓ 2,0 (DCA1)	14.64 (0.38)	12.68 (0.32)	157.64 (23.99)
		ℓ 1 /ℓ 2,1 (DCA)	15.6 (1.01)	14.57 (0.39)	259.28 (84.06)

  1,0 -regularization motivates and opens up a new avenue for the group variable selection problems.

	Chapter 6
	Stochastic DCA and Application to
	Latent Log-Linear Model
	Part III
	Stochastic Learning

Abstract:

  i∈s l ρ i . Following the generic stochastic DCA 6.1, at each iteration l we have to compute v l ∈ ∂ hl (Θ l ) and Θ l+1 = arg min where χ Ω is defined by χ Ω (t) = 0 if t ∈ Ω and 0 otherwise, and proxḡl 2 +χ Ω ρldenotes the proximal operator associated to ḡl 2 +χ Ω . The first special version of the stochastic DCA is described in Algorithm 6.2. We can call this algorithm as the stochastic proximal DCA. Algorithm 6.2 Stochastic Proximal DCA (SPDCA) Initialization: Choose Θ 0 ∈ Ω. For l = 0, 1, ... do 1. Randomly choose a small subset s l ⊂ {1, ..., n}. 2. Compute v l ∈ ∂ hl (Θ l ). 3. Compute Θ l+1 = prox

										1 |s l |	i∈s l f i by
		ḡl (Θ) = ḡl 2 (Θ) + hl (Θ) = ρl 2 Θ 2 -ρl 2	Θ 2 , 1 |s l | i∈s l	g i 1 (Θ) +	1 |s l | i∈s l	h i (Θ),
	where ḡl 2 (Θ) = 1 |s l |	i∈s l g i 2 (Θ) and ρl = 1 |s l |
	Θ∈Ω	ḡl 2 (Θ) +	ρl 2	Θ 2 -v l , Θ = arg min Θ∈Ω	ḡl 2 (Θ) +	ρl 2	Θ -	ρl v l	2
							:= prox	ḡl 2 +χ Ω ρl	v l ρl	,
			ḡl 2 +χ Ω ρl	v l

Table 6 .

 6 2: Comparative results of DCA1, DCA2, L-BFGS, ProxCCCP and Kernel SVM. Bold fonts indicate the best results in each row.

			SDCA	DCA1	DCA2	L-BFGS	ProxCCCP Kernel SVM
		isoslet	93.17	95	96.02	94.99	95.02	95.95
	ACC (%)	usps ijcnn1	91.85 96.11	92.28 95.14	93.67 98.73	90.88 93.51	91.13 95.27	94.12 95.86
		webspam	98.72	97.69	98.25	95.19	96.02	97.5
		isoslet	4.93	232.37	156.45	183.62	797.37	161.46
	CPU (s)	usps ijcnn1	2.38 16.42	36.02 97.38	28.16 43.22	32.89 105.16	392.97 197.72	33.68 216.5
		webspam	190.28 2938.8	1518.68	2483.91	2173.13	6838.54

http://cran.r-project.org/

This chapter is published under the titles:[1] Hoai An Le Thi and Duy Nhat Phan. DC Programming and DCA for Sparse Optimal Scoring Problem. Neurocomputing 186: 170-181 (2016). [2] Hoai An Le Thi and Duy Nhat Phan. A DC Programming Approach for Sparse Optimal Scoring. Advances in Knowledge Discovery and Data Mining, Lecture Notes in Computer Science, Volume 9078,pp. 435-446, Springer (2015).

The material of this chapter is based on the following work:[1] Hoai An Le Thi and Duy Nhat Phan. Efficient Nonconvex Group Variable Selection and Application to Group Sparse Optimal Scoring. Submitted.

Remerciements

recommandation, ce qui m'a donné une chance de rencontrer et travailler avec Madame le Professeur LE THI Hoai An. Je remercie particulièrement le Docteur NGUYEN Manh Cuong pour les discussions s'adressent également au Gouvernement Vietnamien qui a financé

The problem (5.32) can be separated into L independent sub-problems of the same form: min (5.33) where (Y Θ 0 ) k denotes the k-th column of Y Θ 0 and v l k = ((v l ) 1 k , ..., (v l ) d k ). Hence, for solving the problem (5.32), we can solve L sub-problems in parallel. This shows the efficiency of the ℓ 1,0 -regularization. For solving problem (5.33), we use the coordinate descent method [START_REF] Friedman | Pathwise coordinate optimization[END_REF]. ℓ 1,0 (DCA2) (DCA2 with p = 1 for solving (5.31))

Initialization: Choose W 0 ∈ K, l ← 0 and let τ be a tolerance sufficient small. repeat 1. Set v l j = -λα if (w l ) j 1 ≤ 1/α and 0 otherwise ∀j = 1, ..., d. 2. Compute W l+1 by solving the problem:

(5.34)

The problem (5.34) can also be separated into L independent sub-problems of the same form, min 5.35) for which the coordinate descent method can be used.

The following result is a consequence of the convergence properties of DCA for DC polyhedral programs.

Theorem 5.1 The algorithm ℓ 1,0 (DCA1) (resp. ℓ 1,0 (DCA2)) terminates after a finite number of iterations, and the solution W * (resp. (W * , |W * d | 1 )) given by ℓ 1,0 (DCA1) (resp. ℓ 1,0 (DCA2)) is a critical point of the problem (5.30) (resp. (5.31)). Furthermore, if (w * ) j 1 = 1 α ∀j = 1, ..., d, then W * and (W * , |W * d | 1 ) are local solutions to (5.30) and (5.31), respectively.

Proof : Since the ℓ 1 -norm and the function d j=1 -η α (z j ) are polyhedral convex, the second DC component H 1 (resp. H 2 ) in (5.20) (resp. (5.23)) is polyhedral convex. Therefore, both (5.20) and (5.23) (5.30) and (5.31), respectively. ✷ ℓ 2,0 (DCA1) (DCA1 with p = 2 for solving (5.30))

Initialization: Choose W 0 ∈ K, l ← 0 and let τ be a tolerance sufficient small. repeat 1. Set (v l ) j ∈ λ∂r( (w l ) j 2 ), j = 1, ..., d as (v l ) j = λα (w l ) j 2 (w l ) j if (w l ) j 2 > 1/α 0 otherwise.

2. Compute W l+1 by solving the problem:

(v l ) j , w j .

(5.36)

Since f (W ) is not separable with respect to rows, the convex problem (5.36) cannot be separated into independent sub-problems as the previous cases. Here we apply a block coordinate descent algorithm for solving (5.36). We choose a row j to minimize, and consider the other rows as fixed. The resulting problem is min (5.37) where r (-

Combining the subgradient conditions with basic algebra, we get the solution ŵj of the problem (5.37) without constraint as follows.

ℓ 2,0 (DCA2) (DCA2 with p = 2 for solving (5.31))

1/α and 0 otherwise ∀j = 1, ..., d. 2. Compute W l+1 by solving the problem:

(5.38)

The problem (5.38) is also solved by a block coordinate descent algorithm. The update with respect to the j-th row has the form

Theorem 5.2 a) The sequence generated by ℓ 2,0 (DCA1) has at least one limit point and every limit point of this sequence is a critical point of the problem (5.30).

b) The algorithm ℓ 2,0 (DCA2) terminates after a finite number of iterations, and the solution

) is a local solution to (5.31).

Proof : a) (a) is a consequence of convergence properties of general DC programs and the facts that the objective function of (5.30) We will compare the proposed algorithms (ℓ 1,0 (DCA1), ℓ 1,0 (DCA2), ℓ 2,0 (DCA1) and ℓ 2,0 (DCA2)) with the standard algorithms using the ℓ 2,1 -regularization: SOS GLASSO [START_REF] Leng | Sparse optimal scoring for multiclass cancer diagnosis and biomarker detection using microarray data[END_REF][START_REF] Merchante | An efficient approach to sparse linear discriminant analysis[END_REF] and GS MSVM [START_REF] Blodel | Block coordinate descent algorithms for large-scale sparse multiclass classification[END_REF]. We also compare the four DCA based algorithms to study the performance of the two regularizations as well as the two different DC decompositions.

Sparse optimal scoring using group lasso (SOS GLASSO):

GLASSO used the ℓ 2,1 -norm instead of the ℓ p,0 -norm in the problem (5.29), that is min

(5.39)

In the experiments, this problem is solved by a block coordinate descent algorithm.

Group sparse multiclass support vector machine (GS MSVM):

Let W be a d×C matrix, where d and C represent the number of features and the number of classes, respectively. Denote by w k ∈ R d the k-th column of W . In multiclass support vector machine, an observation x is classified to one of the C classes using the following rule: y = arg max k∈{1,...,C} w T k x.

Given n training observations (x i , y i ), i = 1, ..., n, the goal of [START_REF] Blodel | Block coordinate descent algorithms for large-scale sparse multiclass classification[END_REF] is to estimate a row-wise sparse solution W by solving the following problem.

(5.40)

In [START_REF] Blodel | Block coordinate descent algorithms for large-scale sparse multiclass classification[END_REF], the authors use the block coordinate descent method for solving this problem. The code is published at https://github.com/mblondel/lightning.

Datasets

The real world datasets consist of seven real microarray gene expression datasets, and four face image datasets. The number of classes ranges from 3 to 10. All the datasets are preprocessed by normalizing each dimension of the data to zero mean and unit variance. The detailed information of these datasets is summarized in Table 5.1.

We note that log det Σ k is concave function while tr((Σ k ) -1 S k ) is convex in Σ k . Hence we also have a natural DC decomposition of f (see Chapter 4). However, in Chapter 4, we have showed the efficient of the special DC formulation by moving tr((Σ k ) -1 S k ) to the second DC component. We have a special DC decomposition of the function f as follows: 5.47) where

are convex functions in {Σ} when µ 1 , ..., µ Q are large enough. For estimating µ 1 , ..., µ Q , we have the following lemma.

is convex in Σ k . Moreover, we note that

is convex. Hence h({Σ}) is convex since the sum of convex functions is also convex. The proof of lemma is complete. ✷ Remark 5.3 From the Lemma 5.1, we can choose

5.5.1 DCA for solving the problem (5.46) with p = 1

According to DCA1 with p = 1, at each iteration l, we have to compute

, and {Σ l+1 } as a solution to the problem (5.48) where {V l } = {D l } + {C l }. Computing {D l } can be split into two parts as follows:

respectively computed by

The computation of {C l } is defined by

(5.51)

For solving each convex sub-problem (5.51), we use the alternating direction method of multipliers (ADMM) [START_REF] Boyd | Distributed optimization and statistical learning via the alternating direction method of multipliers[END_REF]. The augmented Lagrangian function of this problem is

More specifically, ADMM solves the following problems at each iteration m:

(5.54)

The solutions (5.52) and ( 5.53) can be explicitly computed as follows:

where S be the elementwise soft-thresholding operator defined by S(A,

DCA for solving (5.46) with p = 1 is summarized in the following algorithm.

ℓ 0 /ℓ 1,0 (DCA1) (DCA1 with p = 1 for solving (5.46))

5.5.2 DCA for solving the problem (5.46) with p = 2 According to DCA1 with p = 2, at each iteration l, we have to compute

, and {Σ l+1 } as a solution to the following problem: (5.55) where

Computing {D l } can be split into two parts as follows:

where {A l }, {B l } are respectively computed by using (5.49) and ( 5.50), and

The problem (5.55) cannot be separated into independent sub-problems as the previous case. Here we apply the ADMM algorithm for solving this problem. The augmented Lagrangian function of the problem (5.55) is

More specifically, ADMM solves the following problems at each iteration m:

(5.58)

The solution (5.56) can be explicitly computed as follows, k = 1, ..., Q,

The solution (5.57) can be computed as follows. We denote

Taking the norm of both sides, we obtain

(5.61)

Group Variable Selection

Substituting this expression for (X ij ) m+1 2 into Eq. ( 5.60) and simplifying gives

For summary, we describe the algorithm for (5.46) with p = 2 in the algorithm below.

ℓ 0 /ℓ 2,0 (DCA1) (DCA1 with p = 2 for solving (5.46))

Initialization: Choose {Σ 0 } ∈ Ω, l ← 0, and let τ, ǫ be tolerances sufficient small.

2. Compute {Σ l+1 } by ADMM:

Theorem 5.3 (Convergence properties of ℓ 0 /ℓ 1,0 (DCA1) and ℓ 0 /ℓ 2,0 (DCA1)) Let {{Σ l }} be the sequence generated by ℓ 0 /ℓ 1,0 (DCA1) (resp. ℓ 0 /ℓ 2,0 (DCA1)), we have

F < +∞, and hence {Σ l } -{Σ l+1 } F → 0 as l → +∞. (d) The sequence {{Σ l }} has at least one limit point and every limit point of this sequence is a critical point of the problem (5.46).

Proof :

The theorem is proved analogously to Theorem 4.1. ✷ 5.5.3 Group variable selection using ℓ 1 /ℓ 2,1 -regularization

In this section, we study the convex approximation approaches of ℓ 0 -norm and ℓ p,0 -norm which are respectively ℓ 1 -norm (lasso) and ℓ 2,1 -norm (group lasso). By replacing the ℓ 0 /ℓ p,0 -norm with the ℓ 1 /ℓ 2,1 -norm, the resulting problem is min

(5.62) This problem is still non-convex. We use DCA for solving it. Similar to the previous section, (5.62) where

and

DCA applied to DC program (5.63) is similar to ℓ 0 /ℓ 2,0 (DCA1). We simply replace the computation of {V l } ∈ ∂H({Σ l }) with {V l } = {A l } computed by (5.49). The DCA for solving (5.63) is described as follows.

ℓ 1 /ℓ 2,1 (DCA) (DCA for solving (5.63))

The predicted class for a new observation x is arg max

Therefore, the decision boundary between each pair of classes i and j is described by a linear equation {x : x, θ iθ j + λ iλ j = 0}. In the supervised learning setup with the labeled data D train , learning the log-linear model corresponds to maximizing the conditional log-likelihood on D train , namely max 1 N N i=1 log P (y i |x i ) . (6.9)

In order to extend this model is to create a latent log-linear model by incorporating a latent (or hidden) variable h [START_REF] Deselaers | Latent Log-linear Models for Handwritten Digit Classification[END_REF]. The latent log-linear model is an extension of log-linear model to increase the flexibility of the model. The posterior probability for a label y is defined by The problem (6.11) takes the form of (6.1) in which the DC function f i is defined by

Many other application problems in practice take the form of (6.1) such as sparse support vector machine, deep neutral networks, regularized least squares, sensor networks, other maximum likelihood problems, etc.

For solving the problem (6.1), batch approaches are more natural and well-known. However, when n is large, the above problem becomes more challenge in machine learning. The per-iteration costs of the batch approaches might be more expensive because they use all the functions f i . In the last few years, stochastic optimization techniques have proven to be useful in machine learning for solving problems with a large number n of training data points (s.g. [START_REF] Bottou | Stochastic Learning[END_REF][START_REF] Bottou | Online Learning in Neural Networks, chapter Online Learning and Stochastic Approximations[END_REF][START_REF] Duchi | Efficient online and batch learning using forward backward splitting[END_REF][START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF][START_REF] Mairal | Stochastic majorization-minimization algorithms for large-scale optimization[END_REF]). The typical stochastic optimization method is the stochastic gradient method. In the context minimizing a differentiable function f without constraint Ω = R d , the update step of the stochastic gradient method is given by Θ l+1 ← Θ lγ l ∇f i l (Θ l ), (6.13) where ḡl and hl are convex functions defined by

We propose a stochastic scheme that at each iteration l we randomly choose one small subset of the indexes s l ⊂ {1, ..., n} and compute v l ∈ ∂ hl (Θ l ), and then solve the following convex problem: min

The generic stochastic DCA is described in Algorithm 6.1.

Algorithm 6.1 Generic stochastic DCA Initialization: Choose Θ 0 ∈ Ω.

For l = 0, 1, ... do 1. Randomly choose a small subset s l ⊂ {1, ..., n}.

Compute v

End for.

Remark 6.1 We consider an extension of the stochastic DCA (6.1). In the step 3, the update rule of Θ l+1 can be replaced with • step 3'.

Special versions of stochastic DCA

In this section, we discuss about several versions of the stochastic DCA base on special DC decompositions of f i . The proposed algorithms in this section is very useful in practice because the solution to the convex problem (6.14) can be explicitly computed. We also show some existing stochastic algorithms are special versions of our algorithm. We now consider the first DC component g i expressed as follows: (6.16) where g i 1 and g i 2 are convex functions.

where

i∈s l H i is also a positive definite matrix. According to the genetic stochastic DCA scheme, at each iteration l, we have to compute v l ∈ ∂ hl (Θ l ) and

where prox

denotes the proximal Newton operator associated to ḡl 2 + χ Ω . Hence the second special version of (6.1) is similarly to the algorithm (6.2) in which the step 3 is replaced with the following rule:

The algorithm using this rule can be named as stochastic proximal Newton DCA. Note that the positive definite matrix Hl can be computed by using a quasi-Newton method.

Application to latent log-linear model

Latent variables have long been used to model observations in various models such as hidden Markov models (see e.g. [START_REF] Juang | A probabilistic distance measure for hidden markov models[END_REF]; [START_REF] Starner | Real-time american sign language recognition from video using hidden markov models[END_REF]; [START_REF] Durbin | Biological Sequence Analysis[END_REF]; [START_REF] Kim | Discriminative learning of mixture of bayesian network classifiers for sequence classification[END_REF]; [START_REF] Rabiner | A tutorial on hidden markov models and selected applications in speech recognition[END_REF]), hidden conditional random fields (see e.g. [START_REF] Lafferty | Conditional random fields: Probabilistic models for segmenting and labeling sequence data[END_REF]; [START_REF] Gunawardana | Hidden conditional random fields for phone classification[END_REF]; [START_REF] Wang | Hidden conditional random fields for gesture recognition[END_REF]; van der Maaten et al. ( 2011); [START_REF] Quattoni | Hidden conditional random fields[END_REF]) and log-linear model incorporated latent variables (see e.g. [START_REF] Deselaers | Latent Log-linear Models for Handwritten Digit Classification[END_REF]; [START_REF] Heigold | On the equivalence of gaussian and log-linear hmms[END_REF]; [START_REF] Tsiligkaridis | A difference of convex functions approach to large-scale log-linear model estimation[END_REF]). Recently, there are some works on structural SVMs with latent variables (see e.g. [START_REF] Yu | Learning structural svms with latent variables[END_REF]; [START_REF] Ping | Marginal structured svm with hidden variables[END_REF]). These latent variables models are motivated by numerous applications in areas, for examples, speech recognition, information retrieval, natural language processing, object recognition, gesture recognition, object detection, document-level sentiment classification and link prediction.

In this section, we apply the stochastic DCA for solving the latent log-linear model (6.11) which is an extension of log-linear model by incorporating a latent variable. This problem takes the form of (6.1): 6.18) where

Every f i is DC function, hence the problem (6.18) is a DC programming. A natural DC decomposition of f is 6.19) where

In [START_REF] Tsiligkaridis | A difference of convex functions approach to large-scale log-linear model estimation[END_REF], the authors have proposed a concave-convex procedure (CCCP) based on this DC decomposition and a proximal term (ProxCCCP) for solving the problem (6.18). Note that CCCP is a special version of DCA.

In this section, we exploit the particular structure of the problem ( 6.18) and propose a special stochastic DCA for solving it. At each iteration l, we randomly choose a small subset of functions f i and perform one iteration of DCA. We consider a special DC decomposition of 1

are convex function when ρl is large enough. For estimating ρl , we state the following lemma.

Since the function We have

and let (B, t) = (A, y), we also have

Hence we get the following inequality.

On the other hand, the spectral radius of the Hessian matrix of kl (Θ) is smaller than

It follows that ρ s l is larger than the spectral radius of the Hessian matrix. The lemma has been proved. ✷ Remark 6.3 From the Lemma 6.1, we can choose ρl = 2

Following the special stochastic DCA scheme, at each iteration l we have to randomly choose s l ⊂ {1, ..., n}, compute v l ∈ ∂ hl (Θ l ) and

Let δ be the function defined by

The computation of v l is described as follows.

The stochastic DCA for solving (6.18) is described in the following algorithm.

SDCA: Stochastic DCA for solving (6.18)

For l = 0, 1, ... do 1. Randomly choose a small subset s l ⊂ {1, ..., n}.

2. Compute v l ∈ ∂ hl (Θ l ) using (6.21).

3. Compute Θ l+1 = v l ρl . End for.

Founded on the results for the stochastic gradient descent (see [START_REF] Bottou | Online Learning in Neural Networks, chapter Online Learning and Stochastic Approximations[END_REF]), we prove the convergence properties of SDCA. Theorem 6.1 Assume that the data is bounded. 2 for some A ≥ 0 and B > 0.

Proof : First of all, we prove that the gradient ∇f i (Θ) and Hessian functions ∇ 2 f i (Θ) are bounded. We have

and

for (B, t) = (A, y). Hence we obtain the following inequalities (6.22) and

Therefore, we get

(||x i || 2 + 1). ( 6.23)

From the above results and similar arguments in [START_REF] Bottou | Online Learning in Neural Networks, chapter Online Learning and Stochastic Approximations[END_REF], the proof of theorem is completed. ✷ 6.4 DCA for solving the latent log-linear model

In this section, we propose two DCA for solving the problem (6.18). The first DCA bases on the natural DC decomposition (6.19) while the second DCA bases on a special DC decomposition of f . Firstly, the corresponding DC formulation of the problem (6.18) is min (6.24) According to the generic DCA scheme, DCA applied on (6.24) consists of computing, at each iteration l, a gradient V l = ∇H(Θ l ) and solving the convex program of the form

The computation of V l = ∇H(Θ l ) is described as follows.

The algorithm is described as follows.

DCA1: DCA for solving (6.24) Initialization: Let τ tolerance sufficient small, set l = 0 and choose Θ 0 . repeat 1. Compute V l = ∇H(Θ l ) by

2. Solve the following convex problem to obtain Θ l+1

For solving the convex subproblem (6.27), we use the Limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) optimization algorithm [START_REF] Nocedal | Numerical Optimization[END_REF].

We remark that DCA1 might be quite expensive because we have to use an iterative method for solving the convex subproblem (6.27). This motivates us to consider another DC formulation of the problem (6.18) that the corresponding DCA scheme has explicit solution at each iteration. We now consider a special DC decomposition of f (Θ) as follows.

f (Θ) = G ′ (Θ) -H ′ (Θ), (6.28) where (6.30) are convex functions when µ is large enough. Hence we have a DC formulation below. Like DCA1, DCA applied on (6.31) consist of computing, at each iteration l, a gradient V l = ∇H ′ (Θ l ), and then solving the following convex problem to obtain Θ l+1 .

min 6.32) whose solution is Θ l+1 = V l µ . The computation of V l is described as follows.

). ( 6.33)

DCA for solving (6.31) is described as follows.

DCA2: DCA for solving (6.31)

Initialization: Let τ tolerance sufficient small, set l = 0, compute µ and choose Θ 0 . repeat 1. Compute V l = ∇H ′ (Θ l ) using (6.33).

Compute Θ

The convergence properties of DCA1 and DCA2 are given in the following theorem. Theorem 6.2 (i) DCA1 (resp. DCA2) generates the sequence

2 < +∞ and Θ l -Θ l+1 → 0 as l → +∞. (iii) Every limit point of the sequence generated by DCA1 (resp. DCA2) is a critical point of the problem (6.24) (resp. the problem (6.31)).

Proof : (i) and (iii) are consequences of convergence properties of general DC programs and the facts that the objective function of (6.18) is bounded from below by 0.

(ii) From the step 1 of DCA2, we have V l = ∇H ′ (Θ l ). It follows that .34) By the step 3 of DCA2, we have V l = µΘ l+1 . Substituting this and G ′ (Θ) = µ 2 Θ 2 2 into (6.34), we obtain

Moreover, µ > 0. Hence, we have .35) Let N be a positive integer. Summing (6.35) from l = 0 to N, we get .36) On the other hand, we have f (Θ N +1 ) ≥ 0. Combining this and ( 6.36) we get

Taking the limit as N → +∞, we obtain (6.37) and hence lim l→+∞ Θ l -Θ l+1 2 = 0. ✷

Numerical experiments

We will compare the three proposed algorithms (one stochastic DCA scheme (SDCA) and two DCA schemes (DCA1 and DCA2)) to the two methods that aim to solve the problem (6.11) (L-BFGS [START_REF] Nocedal | Numerical Optimization[END_REF] and ProxCCCP [START_REF] Tsiligkaridis | A difference of convex functions approach to large-scale log-linear model estimation[END_REF]) and another state-of-the-art method for the classification problems (Kernel SVM).

The L-BFGS procedure is a well known optimization technique that directly optimizes the objective function of the problem (6.11). The L-BFGS procedure is performed by the lbfgs package of R software. where E cave (Θ) = -1 n n i=1 log h exp x i , θ h y i + λ h y i . In the experiments, we set c l = c = 5 × 10 -4 as suggested in [START_REF] Tsiligkaridis | A difference of convex functions approach to large-scale log-linear model estimation[END_REF].

Kernel SVM uses the Radial Basis (Gaussian) kernel function. It is included in the kernlab package of R software.

We have used four real-world datasets for our comparison, and their information is shown in Table 6 In all experiments, the number of latent variables is set to 10. We fix the number of indexes at each iteration. The test set is used to measure the accuracy of various classifiers trained on the training set.

The computational results of SDCA, DCA1, DCA2, L-BFGS, ProxCCCP and Kernel SVM are given in Table 6.2. We are interested in the accuracy of classifiers as well as the rapidity of the algorithms: the percentage of accuracy of classifiers and the training time in second are reported.

Comments on computational results:

The classification accuracy of SDCA, DCA1 and DCA2 is better than that of the compared algorithms on 3 out of 4 datasets. In comparison between stochastic DCA and Appendix A Appendix A.1 Bounded optimal solution set of the problem (5.29)

Let P be the optimal solution set of the problem (5.29). In the sequel, x 1 , x 2 , ..., x d denote the columns of the data matrix X. We will prove that P is bounded.

Lemma A.1 Assume that W ∈ P and let I = {i : wi = 0}. Then {x i } i∈I is linearly independent.

Proof : We suppose that {x i } i∈I is not linearly independent. Therefore, there exists i 0 ∈ I such that x i 0 can be represented by a linear combination of {x i } i∈I\{i 0 } . That is, there exists δ = (δ i ) i∈I\{i 0 } ∈ R |I|-1 such that

Hence, we have

x i ( wi + δ i wi 0 ) T .

(A.2)

We define Ŵ ∈ R d×L by ŵi = wi + δ i wi 0 if i ∈ I \ {i 0 } 0 otherwise.