Monsieur Je Souhaite Remercier Mon Co-Encadrant

Maître Ahmed Zidna

Je Souhaite Exprimer Ma Sincère Gratitude À Monsieur Tao

Pham Dinh

Je Souhaite Remercier Madame

Akiko Takeda

Je Souhaite Remercier Également Monsieur

Jérôme Darmont

Communications in national / International conferences

Je souhaite ici adresser un grand merci à toutes les personnes qui ont rendu cette thèse possible.

Contents

List of Figures

Abbreviations and Notations

Throughout the dissertation, we use uppercase letters to denote matrices, and lowercase letters for vectors or scalars. Vectors are also regarded as matrices with one column. Some of the abbreviations and notations used in the dissertation are summarized as follows.

DC

, R = R ∪ {±∞} • p p -norm (0 < p < ∞), x p = ( n i=1 |x i | p ) 1/p , x ∈ R n • Euclidean norm (or 2 -norm), x = ( n i=1 |x i | 2 ) 1/2 , x ∈ R n • ∞ ∞ -norm, x ∞ = max i=1,...,n |x i |, x ∈ R n •, • scalar product, x, y = n i=1 x i .y i , x, y ∈ R n χ C (•)
indicator function of a set C, χ C (x) = 0 if x ∈ C, +∞ otherwise co{C} convex hull of a set of points C Proj C (x) projection of a vector x onto a set C dom f effective domain of a function f ∇f (x) gradient of a function f at x ∂f (x) subdifferential of a function f at x P(X) probability of a random variable X

Résumé

Dans cette thèse, nous développons certaines techniques avancées d'apprentissage automatique dans le cadre de l'apprentissage en ligne et de l'apprentissage par renforcement ("reinforcement learning" en anglais -RL). L'épine dorsale de nos approches est la programmation DC (Difference of Convex functions) et DCA (DC Algorithm), et leur version en ligne, qui sont reconnues comme de outils puissants d'optimisation non convexe, non différentiable.

Cette thèse se compose de deux parties : la première partie étudie certaines techniques d'apprentissage automatique en mode en ligne et la deuxième partie concerne le RL en mode batch et mode en ligne. La première partie comprend deux chapitres correspondant à la classification en ligne (chapitre 2) et la prédiction avec des conseils d'experts (chapitre 3). Ces deux chapitres mentionnent une approche unifiée d'approximation DC pour différents problèmes d'optimisation en ligne dont les fonctions objectives sont des fonctions de perte 0-1. Nous étudions comment développer des algorithmes DCA en ligne efficaces en termes d'aspects théoriques et computationnels.

La deuxième partie se compose de quatre chapitres (chapitres 4, 5, 6, 7). Après une brève introduction du RL et ses travaux connexes au chapitre 4, le chapitre 5 vise à fournir des techniques efficaces du RL en mode batch basées sur la programmation DC et DCA. Nous considérons quatre différentes formulations d'optimisation DC en RL pour lesquelles des algorithmes correspondants basés sur DCA sont développés. Nous traitons les problèmes clés de DCA et montrons l'efficacité de ces algorithmes au moyen de diverses expériences. En poursuivant cette étude, au chapitre 6, nous développons les techniques du RL basées sur DCA en mode en ligne et proposons leurs versions alternatives. Comme application, nous abordons le problème du plus court chemin stochastique ("stochastic shortest path" en anglais -SSP) au chapitre 7. Nous étudions une classe particulière de problèmes de SSP qui peut être reformulée comme une formulation de minimisation de cardinalité et une formulation du RL. La première formulation implique la norme zéro et les variables binaires. Nous proposons un algorithme basé sur DCA en exploitant une approche d'approximation DC de la norme zéro et une technique de pénalité exacte pour les variables binaires. Pour la deuxième formulation, nous utilisons un algorithme batch RL basé sur DCA. Tous les algorithmes proposés sont testés sur des réseaux routiers artificiels.

Introduction générale

Cadre général et motivations

Au cours de ces dernières années, l'explosion quantitative des données a obligé les chercheurs à avoir de nouvelles visions pour analyser et explorer des données, ce qui est généralement mentionné comme un sujet de Big data. Dans ce contexte, l'inconvénient de l'utilisation d'approches classiques a été mis en évidence. Ainsi, il est nécessaire de recourir aux techniques avancées adaptées au Big data. À la question de "quelles sont les tendances clé des techniques dans Big data?", le développement des techniques innovantes d'apprentissage automatique ("machine learning" en anglais) est une réponse.

Dans cette thèse, nous nous concentrons sur deux challenges en apprentissage automatique dans le contexte du Big data: l'apprentissage automatique pour une grande quantité de données en mode en ligne et en mode batch. En fonction de la disponibilité des données, les techniques d'apprentissage automatique peuvent être considérées en mode batch ou en mode en ligne. En particulier, pour le mode batch, les techniques génèrent des modèles en apprenant sur l'ensemble des données d'apprentissage en une fois. L'apprentissage automatique en mode en ligne (appelé apprentissage en ligne, "online learning" en anglais) met à jour le modèle au fur et à mesure en fonction des nouvelles données. Pour une nouvelle donnée d'entrée à chaque itération, l'apprentissage en ligne effectue une prédiction en utilisant le modèle actuel, puis vérifie la qualité de sa prédiction qui est utilisée pour mettre à jour le modèle pour l'itération suivante. L'apprentissage en ligne joue un rôle important dans des multiples contextes : quand les données sont disponibles progressivement, ou les prédictions doivent être données en temps réel, ou l'apprenant doit s'adapter dynamiquement aux nouveaux types de données, ou il est irréalisable d'apprendre sur l'ensemble des données. Jusqu'alors, la conception d'algorithmes efficaces d'apprentissage en ligne a été influencée par l'optimisation convexe en ligne. Cependant, dans la plupart d'applications, la fonction de perte utilisée pour évaluer les prédictions ou le domaine des prédictions est non convexe. Le désavantage des approches d'optimisation convexe en ligne a été indiqué dans la littérature. Ainsi, il est indispensable de recourir à l'optimisation non convexe en ligne pour développer des algorithmes d'apprentissage en ligne efficaces. La difficulté de cette fonction de perte peut être surmontée en utilisant son approximation DC (Difference of Convex functions). Le problème résultant est encore difficile en raison de sa non convexité. Mais il peut être surmonté par des techniques

Introduction générale

basées sur la version en ligne de l'optimisation DC. Comme une contribution de cette thèse, nous développons les versions "en ligne" de DCA (DC Algorithm) standard pour les méthodes d'apprentissage (classification, prédiction) en ligne. La convergence des méthodes en ligne a été rigoureusement étudiée.

Parmi les principales catégories de techniques d'apprentissage automatique, nous nous intéressons à l'apprentissage par renforcement ("reinforcement learning" en anglais) en mode en ligne et en mode batch. L'apprentissage par renforcement est concerné par une classe de problèmes dans lesquels un agent doit apprendre un comportement décisionnel basé sur la rétroaction de récompense par les interactions avec un environnement dynamique. Les problèmes d'optimisation considérés en apprentissage par renforcement deviennent difficiles parce que leurs fonctions objectives sont des fonctions non convexes et non différentiables, plus précisément elles sont DC. Cependant, en exploitant les propriétés intéressantes de ces problèmes, ils peuvent être résolus efficacement par des techniques en ligne/batch basées sur DCA.

Sur le plan algorithmique, la thèse a proposé une approche unifiée, fondée sur la programmation DC et DCA, et leurs versions en ligne, des outils puissants d'optimisation non convexe qui connaissent un grand succès, au cours de trois dernières décennies, dans la modélisation et la résolution de nombreux problèmes d'application dans divers domaines de sciences appliquées, en particulier en apprentissage automatique et fouille de données ("data mining" en anglais) (voir par example [START_REF] Collobert | Trading convexity for scalability[END_REF][START_REF] Le | Sparse semi-supervised support vector machines by DC programming and DCA[END_REF][START_REF] Thi | New and efficient DCA based algorithms for minimum sum-of-squares clustering[END_REF][START_REF] Thi | Feature selection in machine learning: an exact penalty approach using a difference of convex function algorithm[END_REF][START_REF] Thi | Binary classification via spherical separator by DC programming and DCA[END_REF][START_REF] Thi | Long-Short Portfolio Optimization Under Cardinality Constraints by Difference of Convex Functions Algorithm[END_REF][START_REF] Thi | Self-organizing maps by difference of convex functions optimization[END_REF][START_REF] Thi | DCA based algorithms for feature selection in multi-class support vector machine[END_REF][START_REF] Thi | A DC programming approach for finding communities in networks[END_REF][START_REF] Thi | Solving a Class of Linearly Constrained Indefinite Quadratic Problems by D.C. Algorithms[END_REF][START_REF] Thi | The DC (Difference of Convex Functions) Programming and DCA Revisited with DC Models of Real World Nonconvex Optimization Problems[END_REF][START_REF] Thi | DC programming and DCA: thirty years of developments[END_REF][START_REF] Thi | DC approximation approaches for sparse optimization[END_REF][START_REF] Thi | Efficient approaches for 2 -0 regularization and applications to feature selection in svm[END_REF][START_REF] Thi | DC programming and DCA for sparse fisher linear discriminant analysis[END_REF][START_REF] Thi | Feature selection for linear SVMs under uncertain data: Robust optimization based on difference of convex functions algorithms[END_REF][START_REF] Thi | Efficient nonnegative matrix factorization by DC programming and DCA[END_REF][START_REF] Liu | Multicategory ψ-learning[END_REF][START_REF] Pham Dinh | A difference of convex functions algorithm for switched linear regression[END_REF][START_REF] Pham Dinh | Convex analysis approach to DC programming: theory, algorithms and applications[END_REF][START_REF] Pham Dinh | DC optimization algorithms for solving the trust region subproblem[END_REF][START_REF] Pham Dinh | Recent Advances in DC Programming and DCA[END_REF][START_REF] Yin | Minimization of 1-2 for compressed sensing[END_REF] et la liste des références dans [START_REF] Thi | DC Programming and DCA[END_REF]). De nombreuses expérimentations numériques sur différents types de données dans divers domaines (biologie, transport, physique . . . ) réalisées dans cette thèse ont prouvé l'efficacité, la scalabilité, la rapidité des algorithmes proposés et leur supériorité par rapport aux méthodes standards.

La programmation DC et DCA considèrent le problème DC de la forme α = inf{f (x) := g(x) -h(x) : x ∈ R n } (P dc ), où g et h sont des fonctions convexes définies sur R n et à valeurs dans R ∪ {+∞}, semi-continues inférieurement et propres. La fonction f est appelée fonction DC avec les composantes DC g et h, et g -h est une décomposition DC de f . DCA est basé sur la dualité DC et des conditions d'optimalité locale. La construction de DCA implique les composantes DC g et h et non la fonction DC f elle-même. Chaque fonction DC admet une infinité des décompositions DC qui influencent considérablement sur la qualité (la rapidité, l'efficacité, la globalité de la solution obtenue . . . ) de DCA. Ainsi, au point de vue algorithmique, la recherche d'une "bonne" décomposition DC et d'un "bon" point initial est très importante dans le développement de DCA pour la résolution d'un programme DC. L'utilisation de la programmation DC et DCA dans cette thèse est justifiée par de multiple arguments [START_REF] Pham Dinh | Recent Advances in DC Programming and DCA[END_REF]:

-La programmation DC et DCA fournissent un cadre très riche pour les problèmes d'apprentissage automatique et fouille de données: l'apprentissage automatique et fouille de données constituent une mine des programmes DC dont la résolution appropriée devrait recourir à la programmation DC et DCA. En effet la liste indicative (non exhaustive) des références dans [START_REF] Thi | DC Programming and DCA[END_REF] témoigne de la vitalité la puissance et la percée de cette approche dans la communauté d'apprentissage automatique et fouille de données. -DCA est une philosophie plutôt qu'un algorithme. Pour chaque problème, nous pouvons concevoir une famille d'algorithmes basés sur DCA. La flexibilité de DCA sur le choix des décomposition DC peut offrir des schémas DCA plus performants que des méthodes standard. -L'analyse convexe fournit des outils puissants pour prouver la convergence de DCA dans un cadre général. Ainsi tous les algorithmes basés sur DCA bénéficient (au moins) des propriétés de convergence générales du schéma DCA générique qui ont été démontrées. -DCA est une méthode efficace, rapide et scalable pour la programmation non convexe. A notre connaissance, DCA est l'un des rares algorithmes de la programmation non convexe, non différentiable qui peut résoudre des programmes DC de très grande dimension. La programmation DC et DCA ont été appliqués avec succès pour la modélisation DC et la résolution de nombreux et divers problèmes d'optimisation non convexes dans différents domaines des sciences appliquées, en particulier en apprentissage automatique et fouille de données (voir la liste des références dans [START_REF] Thi | DC Programming and DCA[END_REF]).

Il est important de noter qu'avec les techniques de reformulation en programmation DC et les décompositions DC appropriées, on peut retrouver la plupart des algorithmes existants en programmation convexe/non convexe comme cas particuliers de DCA. En particulier, pour la communauté d'apprentissage automatique et fouille de données, les méthodes très connus comme Expectation-Maximisation (EM) [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF], Succesive Linear Approximation (SLA) [START_REF] Bradley | Feature Selection via Concave Minimization and Support Vector Machines[END_REF], ConCave-Convex Procedure (CCCP) [START_REF] Yuille | The concave-convex procedure (CCCP)[END_REF], Iterative Shrinkage-Thresholding Algorithms (ISTA) [START_REF] Chambolle | Nonlinear wavelet image processing: Variational problems, compression, and noise removal through wavelet shrinkage[END_REF] sont des versions spéciales de DCA.

Nos contributions

Les principales contributions de la thèse consistent à développer les techniques avancées d'apprentissage automatique en utilisant l'approche d'optimisation. Nous étudions une approche basée sur la programmation DC et DCA, et leurs versions en ligne pour résoudre les problèmes d'apprentissage en ligne et, en particulier, d'apprentissage par renforcement. Tout au long de la thèse, les questions clés de DCA ont été étudiées : quelle est la bonne décomposition DC, quelle est la méthode de solution efficace pour les sous-problèmes convexes dans le schéma DCA et quel est le bon point initial. Aborder ces questions dépend fortement des structures spéciales de chaque problème d'optimisation considéré, ce qui est exploité dans la plupart de nos travaux dans les chapitres 2, 3, 5, 6. De plus, les chapitres 2 et 3 incluent les contributions à l'étude d'une version en ligne de DCA en termes d'aspects théoriques et computationnels. Les principales réalisations de la thèse sont décrites en détail comme suit.

Introduction générale

Tout d'abord, nous nous concentrons dans le chapitre 2 sur le développement de techniques d'apprentissage en ligne. L'apprentissage en ligne peut être décrit comme le processus consistant à prévoir une séquence d'échantillons basée sur la connaissance de la correction aux échantillons précédents et autres informations disponibles. La qualité des prédictions est évaluée par une fonction de perte, qui est souvent non convexe et/ou non différentiable. Dans nos travaux, nous étudions l'approche DC et nous proposons une version en ligne de DCA, appelée Online DCA, pour résoudre les problèmes en ligne correspondants. Au cas où chaque sous-problème convexe d'Online DCA ne peut pas être résolu explicitement ou n'est pas facile à résoudre, nous proposons un schéma particulier basé sur Online DCA, nommé ODCA, où chaque sous-problème est résolu en approximant par une itération de la méthode de sous-gradient classique. Nous analysons les propriétés de l'ODCA en termes de regret (c'est-à-dire la différence entre la perte cumulative subie et la plus petite perte cumulative tout au long du processus d'apprentissage). Nous indiquons également que les variantes de l'algorithme de gradient en ligne sont une version spéciale d'Online DCA. Comme application, nous considérons la classification en ligne où, à chaque itération, l'apprenant doit donner le classificateur à prédire l'étiquette correspondant à l'instance à venir basé sur l'étiquette correcte révélée plus tard et les classificateurs précédents. Dans ce cas, la qualité du classificateur est souvent mesurée par la fonction de perte 0-1 qui renvoie 1 si l'étiquette prédite est la même que l'étiquette correcte et 0 autrement. En effet, cette fonction est non convexe et non différentiable, mais elle peut être approximée par des fonctions DC. Dans nos travaux, nous proposons trois différentes fonctions d'approximation DC, y compris deux formes de fonction polyhédrale et une forme de fonction sigmoïde. En utilisant l'ODCA pour résoudre les problèmes d'optimisation résultant, nous proposons trois algorithmes en ligne correspondants et analysons le regret de chaque algorithme basé sur principaux résultats de l'analyse d'ODCA mentionnée ci-dessus. Les résultats de l'expérience sur une variété de ensembles de données de classification montrent l'efficacité de nos algorithmes proposés par rapport aux algorithmes de classification en ligne existants.

Le chapitre 3 concerne une autre classe de techniques d'apprentissage en ligne, à savoir la prédiction avec des conseils d'experts ("prediction with expert advice" en anglais).

Chapter 1 Preliminary

This chapter summarizes some basic concepts and results that will be the groundwork of the dissertation.

DC programming and DCA

DC programming and DCA, which constitute the backbone of nonconvex programming and global optimization, were introduced by Pham Dinh Tao in their preliminary form in 1985 [START_REF] Pham Dinh | Algorithms for Solving a Class of Nonconvex Optimization Problems[END_REF]. Important developments and improvements on both theoretical and computational aspects have been completed since 1994 throughout the joint works of Le Thi Hoai An and Pham Dinh Tao. In this section, we present some basic properties of convex analysis and DC optimization and DC Algorithm that computational methods of this dissertation are based on. The materials of this section are extracted from [START_REF] Thi | Contribution à l'optimisation non convexe et l'optimisation globale: Théorie, algorithmes et applications[END_REF][START_REF] Thi | Solving a Class of Linearly Constrained Indefinite Quadratic Problems by D.C. Algorithms[END_REF][START_REF] Thi | The DC (Difference of Convex Functions) Programming and DCA Revisited with DC Models of Real World Nonconvex Optimization Problems[END_REF][START_REF] Pham Dinh | Convex analysis approach to DC programming: theory, algorithms and applications[END_REF].

Throughout this section, X denotes the Euclidean space R n and R = R ∪ {±∞} is the set of extended real numbers.

Fundamental convex analysis

First, let us recall briefly some notions and results in convex analysis related to the dissertation (refer to the references [START_REF] Boyd | Convex Optimization[END_REF][START_REF] Pham Dinh | Convex analysis approach to DC programming: theory, algorithms and applications[END_REF][START_REF] Rockafellar | Convex analysis[END_REF] for more details).

A subset C of X is said to be convex if (1 -λ)x + λy ∈ C whenever x, y ∈ C and λ ∈ [0, 1].
Let f be a function whose values are in R and whose domain is a subset S of X. The set

{(x, t) : x ∈ S, t ∈ R, f (x) ≤ t}
is called the epigraph of f and is denoted by epif .

Chapter 1. Preliminary

We define f to be a convex function on S if epif is convex set in X × R. This is equivalent to that S is convex and

f ((1 -λ)x + λy) ≤ (1 -λ)f (x) + λf (y), ∀x, y ∈ S, ∀λ ∈ [0, 1].
The function f is strictly convex if the inequality above holds strictly whenever x and y are distinct in S and 0 < λ < 1.

The effective domain of a convex function f on S, denoted by domf , is the projection on X of the epigraph of f

domf = {x ∈ X : ∃t ∈ R, (x, t) ∈ epif } = {x ∈ X : f (x) < +∞}
and obviously, it is convex.

The convex function f is called proper if domf = ∅ and f (x) > -∞ for all x ∈ S.
The function f is said to be lower semi-continuous at a point x of S if

f (x) ≤ lim inf y→x f (y).
Denote by Γ 0 (X) the set of all proper lower semi-continuous convex functions on X.

Let ρ be a nonnegative number and C be a convex subset of X. One says that a function θ :

C → R ∪ {+∞} is ρ-convex if θ[λx + (1 -λ)y] ≤ λθ(x) + (1 -λ)θ(y) - λ(1 -λ) 2 ρ x -y 2
for all x, y ∈ C and λ ∈ (0, 1). It amounts to say that θ -

(ρ/2) • 2 is convex on C.
The modulus of strong convexity of θ on C, denoted by ρ(θ, C) or ρ(θ

) if C = X, is given by ρ(θ, C) = sup{ρ ≥ 0 : θ -(ρ/2) • 2 is convex on C}.
One says that θ is strongly convex on C if ρ(θ, C) > 0.

A vector y is said to be a subgradient of a convex function f at a point

x 0 if f (x) ≥ f (x 0 ) + x -x 0 , y , ∀x ∈ X.
The set of all subgradients of f at x 0 is called the subdifferential of f at x 0 and is denoted by ∂f (x 0 ). If ∂f (x) is not empty, f is said to be subdifferentiable at x.

For ε > 0, a vector y is said to be an ε-subgradient of a convex function f at a point

x 0 if f (x) ≥ (f (x 0 ) -ε) + x -x 0 , y , ∀x ∈ X.
The set of all ε-subgradients of f at x 0 is called the ε-subdifferential of f at x 0 and is denoted by

∂ ε f (x 0 ).
Let us describe two basic notations as follows.

dom ∂f = {x ∈ X : ∂f (x) = ∅} and range ∂f (x) = ∪{∂f (x) : x ∈ dom ∂f }.

Proposition 1.1. Let f be a proper convex function. Then 1. ∂ ε f (x) is a closed convex set, for any x ∈ X and ε ≥ 0.

2. ri(domf ) ⊂ dom ∂f ⊂ domf where ri(domf ) stands for the relative interior of domf .

3. If f has a unique subgradient at x, then f is differentiable at x, and ∂f (x) = {∇f (x)}.

4.

x 0 ∈ argmin{f (x) : x ∈ X} if and only if 0 ∈ ∂f (x 0 ).

Conjugates of convex functions

The conjugate of a function f :

X → R is the function f * : X → R, defined by f * (y) = sup x∈X { x, y -f (x)}. Proposition 1.2. Let f ∈ Γ 0 (X). Then we have 1. f * ∈ Γ 0 (X) and f * * = f . 2. f (x) + f * (y) ≥ x, y , for any x, y ∈ X.
Equality holds if and only if y ∈ ∂f (x) ⇔ x ∈ ∂f * (y).

y

∈ ∂ ε f (x) ⇐⇒ x ∈ ∂ ε f * (y) ⇐⇒ f (x) + f * (y) ≤ x, y + ε, for all ε > 0.

Polyhedral Functions

A polyhedral set is a closed convex set that is of form

C = {x ∈ X : x, b i ≤ β i , ∀i = 1, . . . , m}
where b i ∈ X and β i ∈ R for all i = 1, . . . , m.

A function f ∈ Γ 0 (X) is said to be polyhedral if f (x) = max{ a i , x -α i : i = 1, . . . , k} + χ C (x), ∀x ∈ X (1.1)
where

a i ∈ X, α i ∈ R for all i = 1, . . . , k and C is a nonempty polyhedral set. It is clear that dom f = C. Proposition 1.3.
Let f be a polyhedral convex function, and x ∈ domf . Then we have 1. f is subdifferentiable at x, and ∂f (x) is a polyhedral convex set. In particular, if f is defined by

(1.1) with C = X then ∂f (x) = co{a i : i ∈ I(x)} where I(x) = {i ∈ {1, . . . , k} : a i , x -α i = f (x)}. Chapter 1. Preliminary 2. The conjugate f * is a polyhedral convex function. Moreover, if C = X then domf * = co{a i : i = 1, . . . , k}, f * (y) = inf k i=1 λ i α i k i=1 λ i a i = y, k i=1 λ i = 1, λ i ≥ 0, ∀i = 1, . . . , k .
In particular, f * (a i ) = α i , ∀i = 1, . . . , k.

Difference of Convex (DC) functions

A function f is called DC function on X if it is of the form f (x) = g(x) -h(x), x ∈ X
where g and h belong to Γ 0 (X). One says that g -h is a DC decomposition of f and the functions g, h are its DC components. If g and h are in addition finite on all points of X then one says that f = g -h is finite DC function on X. The set of DC functions (resp. finite DC functions) on X is denoted by DC(X) (resp. DC f (X)).

Remark 1.1. Give a DC function f whose DC decomposition is f = g -h. Then for every θ ∈ Γ 0 (X) finite on the whole X, f = (g + θ) -(h + θ) is another DC decomposition of f . Thus, a DC function f has finitely many DC decompositions.

DC optimization DC program

In the sequel, we use the convention +∞ -(+∞) = +∞.

For g, h ∈ Γ 0 (X), a standard DC program is of the form

(P ) α = inf{f (x) = g(x) -h(x) : x ∈ X} and its dual counterpart (D) α * = inf{h * (y) -g * (y) : y ∈ X}.
There is a perfect symmetry between primal and dual programs (P ) and (D): the dual program to (D) is exactly (P ), moreover, α = α * .

Remark 1.2. Let C be a nonempty closed convex set. Then, the constrained problem

inf{f (x) = g(x) -h(x) : x ∈ C}
can be transformed into an unconstrained DC program by using the indicator function χ C , i.e., inf{f (x) = φ(x) -h(x) : x ∈ X} where φ := g + χ C belongs to Γ 0 (X).

We will always keep the following assumption that is deduced from the finiteness of α dom g ⊂ dom h and dom h * ⊂ dom g * .

(1.2)

Polyhedral DC program

In the problem (P ), if one of the DC components g and h is polyhedral function, we call (P ) a polyhedral DC program. This is an important class of DC optimization. It is often encountered in practice and has worthy properties.

Consider the problem (P ) where h is a polyhedral convex function given by

h(x) = max{ a i , x -α i : i = 1, . . . , k}.
By Proposition 1.3, the dual problem (D) can be expressed as follows.

α * = inf{h * (y) -g * (y) : y ∈ X} = inf{h * (y) -g * (y) : y ∈ co{a i : i = 1, . . . , k}} = inf{α i -g * (a i ) : i = 1, . . . , k}.
Note that, if g is polyhedral convex and h is not, then by considering the dual problem (D), we have the similar formulation as above since g * is polyhedral.

Optimality conditions for DC optimization

A point x * is said to be a local minimizer of g -h if x * ∈ dom g ∩ dom h (so, (g -h)(x * ) is finite) and there is a neighborhood U of x * such that

g(x) -h(x) ≥ g(x * ) -h(x * ), ∀x ∈ U. (1.3)
A point x * is said to be a critical point of g-h if it verifies the generalized Kuhn-Tucker condition ∂g(x * ) ∩ ∂h(x * ) = ∅.

(1.4)

Let P and D denote the solution sets of problems (P ) and (D) respectively, and let

P = {x * ∈ X : ∂h(x * ) ⊂ ∂g(x * )}, D = {y * ∈ X : ∂g * (y * ) ⊂ ∂h * (y * )}.
In the following, we present some fundamental results on DC programming [START_REF] Thi | Solving a Class of Linearly Constrained Indefinite Quadratic Problems by D.C. Algorithms[END_REF].

Theorem 1.1. i) Global optimality condition: x ∈ P if and only if

∂ ε h(x) ⊂ ∂ ε g(x), ∀ε > 0.
ii)

∈ ∂h(x) such that h * (y) -g * (y) ≥ h * (y * ) -g * (y * ), then x * is a local minimizer of g -h. More precisely, g(x) -h(x) ≥ g(x * ) -h(x * ), ∀x ∈ U ∩ dom g. v) Transportation of local minimizers: Let x * ∈ dom ∂h be a local minimizer of g -h. Let y * ∈ ∂h(x * ) and a neighborhood U of x * such that g(x) -h(x) ≥ g(x * ) -h(x * ), ∀x ∈ U ∩ dom g. If y * ∈ int(dom g * ) and ∂g * (y * ) ⊂ U then y * is a local minimizer of h * -g * .
Remark 1.3. a) By the symmetry of the DC duality, these results have their corresponding dual part. For example, if y is a local minimizer of h * -g * , then y ∈ D . b) The properties ii), v) and their dual parts indicate that there is no gap between the problems (P ) and (D). They show that globally/locally solving the primal problem (P ) implies globally/locally solving the dual problem (D) and vice-versa. Thus, it is useful if one of them is easier to solve than the other. c) The necessary local optimality condition ∂h * (x * ) ⊂ ∂g * (x * ) is also sufficient for many important classes programs, for example [START_REF] Thi | The DC (Difference of Convex Functions) Programming and DCA Revisited with DC Models of Real World Nonconvex Optimization Problems[END_REF], if h is polyhedral convex, or when f is locally convex at x * , i.e. there exists a convex neighborhood U of x * such that f is finite and convex on U . We know that a polyhedral convex function is almost everywhere differentiable, that is to say, it is differentiable everywhere except on a set of measure zero. Thus, if h is a polyhedral convex function, then a critical point of g -h is almost always a local solution to (P ). d) If f is actually convex on X, we call (P) a "false" DC program. In addition, if ri(dom g) ∩ ri(dom h) = ∅ and x 0 ∈ dom g such that g is continuous at x 0 , then 0 ∈ ∂f (x 0 ) ⇔ ∂h(x 0 ) ⊂ ∂g(x 0 ) [START_REF] Thi | The DC (Difference of Convex Functions) Programming and DCA Revisited with DC Models of Real World Nonconvex Optimization Problems[END_REF]. Thus, in this case, the local optimality is also sufficient for the global optimality. Consequently, if in addition h is differentiable, a critical point is also a global solution.

DC Algorithm (DCA)

The DCA consists in the construction of the two sequences {x k } and {y k } (candidates for being primal and dual solutions, respectively) which are easy to calculate and satisfy the following properties:

i) The sequences (g -h)(x k ) and (h * -g * )(y k ) are decreasing.

ii) Their corresponding limits x ∞ and y ∞ either satisfy the local optimality condition (x ∞ , y ∞ ) ∈ P × D or are critical points of g -h and h * -g * , respectively.

From a given initial point x 0 ∈ dom g, the DCA generates these sequences by the scheme y k ∈ ∂h(x k ) = arg min{h * (y) -y, x k : y ∈ X}, (1.5a)

x k+1 ∈ ∂g * (y k ) = arg min{g(x) -x, y k : x ∈ X}. (1.5b)
The interpretation of the above scheme is simple. At iteration k of DCA, one replaces the second component h in the primal DC program by its affine minorant

h k (x) = h(x k ) + x -x k , y k , (1.6)
where y k ∈ ∂h(x k ). Then the original DC program is reduced to the convex program

(P k ) α k = inf{f k (x) := g(x) -h k (x) : x ∈ X} that is equivalent to (1.5b). It is easy to see that f k is a majorant of f which is exact at x k i.e. f k (x k ) = f (x k ).
Similarly, by replacing g * with its affine minorant

g * k (y) = g * (y k-1 ) + y -y k-1 , x k (1.7)
where x k ∈ ∂g * (y k-1 ), it leads to the convex program

(D k ) inf{h * (y) -g * k (y) : y ∈ X} whose solution set is ∂h(x k ).

Well definiteness of DCA

DCA is well defined if one can construct two sequences {x k } and {y k } as described above from an arbitrary initial point. The following lemma is the necessary and sufficient condition for this property. Since for ϕ ∈ Γ 0 (X) one has ri(dom ϕ) ⊂ dom ∂ϕ ⊂ dom ϕ (Proposition 1.1). Moreover, under the assumptions dom g ⊂ dom h, dom h * ⊂ dom g * , one can say that DCA in general is well defined.

Convergence properties of DCA

Complete convergence of DCA is given in the following results [START_REF] Thi | Solving a Class of Linearly Constrained Indefinite Quadratic Problems by D.C. Algorithms[END_REF].

Theorem 1.2. Suppose that the sequences {x k } and {y k } are generated by DCA. Then we have

i) The sequences {g(x k ) -h(x k )} and {h * (y k ) -g * (y k )} are decreasing and

• g(x k+1 )-h(x k+1 ) = g(x k )-h(x k ) if and only if {x k , x k+1 } ⊂ ∂g * (y k )∩∂h * (y k ) and [ρ(h) + ρ(g)] x k+1 -x k = 0. • h * (y k+1 ) -g * (y k+1 ) = h * (y k ) -g * (y k ) if and only if {y k , y k+1 } ⊂ ∂g(x k ) ∩ ∂h(x k ) and [ρ(h * ) + ρ(g * )] y k+1 -y k = 0.
DCA terminates at the kth iteration if either of the above equalities holds.

ii

) If ρ(h) + ρ(g) > 0 (resp. ρ(h * ) + ρ(g * ) > 0), then the sequence { x k+1 -x k 2 } (resp. { y k+1 -y k 2 }) converges.
iii) If the optimal value α is finite and the sequences {x k } and {y k } are bounded, then every limit point x ∞ (resp. y ∞ ) of the sequence {x k } (resp.

{y k }) is a critical point of g -h (resp. h * -g * ).
iv) DCA has a linear convergence for general DC program.

v) In polyhedral DC programs, the sequences {x k } and {y k } contain finitely many elements and DCA has a finite convergence.

vi) If DCA converges to a point x * that admits a convex neighborhood in which the objective function f is finite and convex (i.e. the function f is locally convex at x * ) and if the second DC component h is differentiable at x * , then x * is a local minimizer to the problem (P ).

Remark 1.4. a) Finding y k , x k+1 based on the scheme 1.5 amounts to solving the problems (D k ) and (P k ). Thus, DCA works by reducing a DC program to a sequence of convex programs which can be solved efficiently. b) In practice, the calculation of the subgradient of the function h at a point x is usually easy if we know its explicit expression. But, the explicit expression of the conjugate of a given function g is unknown, so calculating x k+1 is done by solving the convex problem (P k ). For the large-scale setting, the solutions to the problem (P k ) should be either in an explicit form or achieved by efficient algorithms with inexpensive computations. c) When h is a polyhedral function, the calculation of the subdifferential ∂h(x k ) is explicit by Proposition 1.

Analysis of ODCA

In this section, we will present the fundamental concepts that are used to study the analysis of the ODCA based algorithms in the next chapters. First of all, we restate the ultimate goal mentioned previously as a performance metric of the series of points obtained by the algorithm in comparison with the best fixed point in terms of the cumulative objective value in hindsight, which is called a regret of the algorithm. Definition 1.1 ( [START_REF] Hazan | Introduction to online convex optimization[END_REF][START_REF] Zinkevich | Online convex programming and generalized infinitesimal gradient ascent[END_REF]). Assume that the sequence of the points {w 1 , w 2 , . . . , w T } is generated by an algorithm A. Formally, the regret of A until step T is defined as

Regret T A := T t=1 f t (w t ) -min w∈S T t=1 f t (w). (1.14)
We aim to show that the average regret of ODCA tends to zeros when T tends to infinity, that is to say, the performance of ODCA (1.9) is as good as that of the best fixed point in hindsight on the average. To attain that interesting property, we first indicate the regret bound of ODCA as a function of T . In fact, the analysis of ODCA depends largely on the update rule (1.13). Therefore, the different ODCA based algorithms lead to the different regret bounds, which will be stated in more detail in next chapters.

Chapter 1. Preliminary Remark 1.5. It would be interesting to note that if the sequence of points {w t } t=1,...,T is generated by the ODCA algorithms, then it is easy to obtain the following property:

T t=1 f t (w) ≤ T t=1 f C t (w), ∀w ∈ S, (1.15)
and, specially,

T t=1 f t (w t ) = T t=1 f C t (w t ), (1.16)
where f C t (w) = g t (w) -(h t (w t ) + z t , w -w t ) , t = 1, . . . , T , is a convex majorization of f t (w). From this fact, we can observe that by exploiting online convex algorithms to give the upper bound for the cumulative objective value corresponding to the convex loss function f C t , we will derive the upper bound for (1.9) with respect to the DC loss functions {f t } t=1,...,T , which is described in [START_REF] Ho | Online DC optimization for online binary linear classification[END_REF].

Another definition, namely online stability, is one of many quantities for any online algorithm [START_REF] Ross | Stability conditions for online learnability[END_REF][START_REF] Saha | The interplay between stability and regret in online learning[END_REF]. The online stability condition means that on the average, the difference between consecutive points generated by the online algorithm is eventually small.

Definition 1.2 ([104]

). Assume that the sequence of the points {w 1 , w 2 , . . . , w T , w T +1 } is generated by an algorithm A. The (cumulative) online stability of A is defined as

S T A := T t=1
w t -w t+1 .

(1.17) Abstract: This chapter concerns an approach based on an online version of DC (Difference of Convex functions) programming and DCA (DC Algorithm) for developing machine learning techniques in online mode (called online learning for short). Online learning is a class of techniques at the interface between machine learning and online optimization since its main goal is to minimize the cumulative loss between the predicted answer and the correct one along its learning process. There exist many efficient online learning techniques developed based on online convex optimization approaches. However, their disadvantages have recently been highlighted when facing the nonconvex answer domain or/and the nonconvex, nonsmooth loss function in most applications. In this chapter, we consider Online DC programming for online learning, propose an online version of DCA, called Online DCA, and indicate that Online DCA covers some variants of online gradient descent. In addition, we design a particular Online DCA based scheme, named ODCA, and study its analysis in terms of regret bound. As an application, we intensively investigate an Online DCA approach for the problems in the topic of online binary linear classification. Exploiting different DC approximation functions, we develop three corresponding ODCA based algorithms and analyze their regret bounds. Numerical experiments on a variety of benchmark datasets show the efficiency of our proposed algorithms in comparison with the state-of-the-art online classification algorithms.

An algorithm A is called to be online stable if S T A = o(T ) -that is, lim T →∞ 1 T T t=1 w t -w t+1 = 0. ( 1
Online Learning and Applications to Online Binary Linear Classification

Introduction

Background and related works

In recent years, it is neccessary to study many innovative techniques for treating the problems with an immense amount of data. Among certain techniques, developing machine learning techniques in online mode (online learning for short (OL)) occupies an important place. Online learning can be seen as the process of predicting an answer to the sequential arrival of questions based on the knowledge of the correct answers corresponding to previous questions and possibly other available information [START_REF] Shalev-Shwartz | Online learning and online convex optimization[END_REF]. Formally, at the step t, the learner receives a question relevant to an input data, described as a vector x t . Then he makes a prediction in order to answer that question.

In fact, his prediction is constructed based on a hypothesis, h t : X → Y, where X and Y are the space of questions and possible answers respectively. After that, he gets the correct answer to the question, cast as a vector y t ∈ Y. To assess the quality of his prediction, he suffers a loss between the predicted answer and the correct one, defined by t (h t , (x t , y t )), where the loss function t is used to update the hypothesis for the answer to the new question at the next step. The main goal for the learner is to generate the sequence of hypotheses which minimizes the cumulative suffered loss along its learning process.

Online learning plays a significant role in multiple contexts: when the data samples are available over time, the predictions must be made in real time, the learner is required to dynamically adapt to new data patterns, or even learning over the entire data at once is impossible in the computational aspect. Online learning has a wide range of applications such as online advertisement placement, online web ranking, online email categorization, prediction of stock prices and currency exchange, realtime recommendation [START_REF] Shalev-Shwartz | Online Learning: Theory, Algorithms, and Applications[END_REF][START_REF] Shalev-Shwartz | Online learning and online convex optimization[END_REF].

Due to its goal, online learning is seen as a general class of techniques at the interface between machine learning and online optimization. Up until now, most of the effective online learning algorithms have been derived from online convex optimization (see e.g. [START_REF] Shalev-Shwartz | Online Learning: Theory, Algorithms, and Applications[END_REF][START_REF] Shalev-Shwartz | Online learning and online convex optimization[END_REF][START_REF] Zinkevich | Online convex programming and generalized infinitesimal gradient ascent[END_REF] for more details). The paragidm of online convex optimization was introduced by Zinkevich [START_REF] Shalev-Shwartz | A primal-dual perspective of online learning algorithms[END_REF][START_REF] Zinkevich | Online convex programming and generalized infinitesimal gradient ascent[END_REF], where the prediction domain and the loss function are convex. In online convex optimization, there exists a common update rule to select the hypothesis at each step: Follow-the-Leader (FTL) [START_REF] Kalai | Efficient algorithms for online decision problems[END_REF] or/and its regularization form (RFTL) [START_REF] Shalev-Shwartz | Online Learning: Theory, Algorithms, and Applications[END_REF][START_REF] Shalev-Shwartz | A primal-dual perspective of online learning algorithms[END_REF]. In FTL, the learner chooses the hypothesis that has the minimum cumulative loss function over all past steps, whereas RFTL makes a stability for the predictions of online learning algorithms. From these, many effective online convex algorithms were proposed such as online gradient descent (with lazy/greedy projections) [START_REF] Zinkevich | Online convex programming and generalized infinitesimal gradient ascent[END_REF], exponentiated gradient [START_REF] Azoury | Relative loss bounds for on-line density estimation with the exponential family of distributions[END_REF][START_REF] Kivinen | Relative loss bounds for multidimensional regression problems[END_REF][START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF], p-norm [START_REF] Gentile | The robustness of the p-norm algorithms[END_REF]. However, the difficulties in online learning are that loss functions to assess the predictions are often nonsmooth, nonconvex (for example, 0-1 loss function returning 1 if the prediction is correct and 0 otherwise) or the domain of predictions is not convex. Hence solving the corresponding optimization problem becomes more intractable. Encountering the difficulty as just mentioned above, the disadvantage of using online convex optimization approaches has been highlighted recently. Thus, it is essential to resort to nonconvex optimization in online mode to overcome the difficulties, which is the main purpose of our works in this chapter. There exist several challenges that should be addressed when solving the online learning problems: what optimization models in online mode for these problems, how to design simple, efficient online learning techniques, what convergence properties of these techniques, etc.

Recently, several related works have been studied for the problems in online learning with nonconvex loss functions (see e.g. [START_REF] Ertekin | Nonconvex Online Support Vector Machines[END_REF][START_REF] Gasso | Batch and online learning algorithms for nonconvex neyman-pearson classification[END_REF][START_REF] Zhang | Online bandit learning for a special class of non-convex losses[END_REF]). In [START_REF] Ertekin | Nonconvex Online Support Vector Machines[END_REF], the authors proposed a nonconvex online algorithm for Support Vector Machine problems with ramp loss function based on the concave-convex procedure. [START_REF] Gasso | Batch and online learning algorithms for nonconvex neyman-pearson classification[END_REF] presented an online algorithm for nonconvex Neyman-Pearson classification problems using gradient method. However, the analysis of the online algorithms in both of them in terms of the bounds of regret have been not indicated yet, where the regret is defined by the difference between the cumulative loss suffered and the possibly smallest cumulative loss along its run. With a formal regret bound, an algorithm for online bandit learning problems with nonconvex loss function (where only the suffered loss is available) was proposed in [START_REF] Zhang | Online bandit learning for a special class of non-convex losses[END_REF]. However, algorithms for general online learning problems in full information setting (i.e. where the knowledge of loss function is used) with theoretical guarantees have been not investigated yet. This motivates us to develop a nonconvex optimization approach to fill up the family of online algorithms for such online learning problems.

The main algorithmic methodologies of this chapter are DC (Difference of Convex functions) programming and DCA (DC Algorithm) and their online version, which are well-known as powerful nonsmooth, nonconvex optimization tools. DC programming and DCA were introduced by Pham Dinh Tao in a preliminary form in 1985 and extensively developed since 1994 by Le Thi Hoai An and Pham Dinh Tao (see [START_REF] Thi | The DC (Difference of Convex Functions) Programming and DCA Revisited with DC Models of Real World Nonconvex Optimization Problems[END_REF][START_REF] Pham Dinh | Convex analysis approach to DC programming: theory, algorithms and applications[END_REF][START_REF] Pham Dinh | DC optimization algorithms for solving the trust region subproblem[END_REF] and the references therein). DCA has been successfully applied to many (smooth or nonsmooth) large-scale nonconvex programs in various domains of applied sciences, in particular in machine learning, for which they provide quite often a global solution and are proved to be more robust and efficient than the standard methods (see the list of references in [START_REF] Thi | DC Programming and DCA[END_REF]).

Our contributions

This chapter focuses on solving a class of problems in online learning by an optimization approach based on the online version of DC programming and DCA. Our contributions are multiple.

Firstly, we present Online DC programming for online learning problems where the loss function to assess the quality of predictions at each step is a DC function. Then, we develop an online version of DCA, namely Online DCA, with a generalized update rule. And next, we propose a particular Online DCA based scheme, namely ODCA, where the convex subproblem is solved by approximating by one iteration of the classic subgradient method.

Secondly, we thoroughly analyze the regret bounds for the proposed ODCA algorithm under some appropriate conditions for DC functions. Depending on the special properties of DC functions, the regret of ODCA grows sublinearly in the number of steps. Moreover, we are able to achieve a better regret bound, specifically logarithmic regret.

Thirdly, as an application, we focus on developing online learning techniques for solving the problems in an interesting topic of online classification, specifically online binary linear classification (OBLC). In this topic, instance becomes available in a sequential order and is used to update the linear classifier incrementally so as to predict the corresponding binary label for future instance. The loss function to assess the linear classifier at each step is usually 0-1 loss function, as defined above. In this chapter, we propose three DC approximation functions for this loss function, develop three corresponding ODCA based algorithms for solving these problems. Moreover, we also show that these proposed algorithms archive a logarithmic regret and as a result, the bounds of mistakes by predicting false labels are also given.

Finally, with the aim of evaluating the efficiency of our proposed online algorithms for solving the problems in OBLC, we conduct several numerical experiments on a variety of benchmark datasets in comparison with many state-of-the-art online algorithms.

The rest of the chapter is organized as follows. In Section 2.2, we first introduce Online DC programming and Online DCA for online learning, then indicate that online gradient descent algorithms are special cases of Online DCA and finally, show the formal regret bounds for a particular ODCA scheme. How to solve the OBLC problems by ODCA is described in Section 2.3. Section 2.4 reports the numerical results on several test problems which is followed by some conclusions in Section 2.5. In this section, we will show that some variants of online gradient descent for online convex programming are special cases of Online DCA. First of all, in online convex programming, the loss function at the step t, denoted by gt , is convex. In such a case, Online Learning and Applications to Online Binary Linear Classification the online gradient descent algorithm with greedy projection and with lazy projection, named OGD and OGD-L respectively, are described as follows [START_REF] Zinkevich | Online convex programming and generalized infinitesimal gradient ascent[END_REF].

Online

OGD: Online Gradient Descent with Greedy projection Initialization: let w 1 be an initial point in S, {η t } be the sequence of step sizes for t = 1, 2, . . . , T do 1. Compute st ∈ ∂g t (w t ).

2. Compute w t+1 by the update rule

w t+1 = Proj S (w t -η t st ). (2.4)
end for OGD-L: Online Gradient Descent with Lazy projection Initialization: let w 1 be an initial point in S, {η t } be the sequence of step sizes,

y 1 = 0 ∈ R n for t = 1, 2, . . . , T do 1. Compute st ∈ ∂g t (w t ).
2. Compute w t+1 by the update rule

y t+1 = y t -η t st , (2.5)
w t+1 = Proj S (y t+1 ).
(2.6)

end for

Now, we are going to indicate that in some cases of DC functions, the corresponding Online DCA algorithm is exactly the same as the versions of online gradient descent algorithm, which is presented in Propositions 2.1, 2.2.

Proposition 2.1. Assume that R = 0, t 0 = t and at the step t, there exists a positive number ρ t such that ρt 2 •2 -gt is convex on S. Then, OGD is a special case of Online DCA.

Proof. At the step t, we derive the following DC decomposition of gt :

gt (w) = g t (w) -h t (w),
where g t (w) = ρt 2 w 2 and h t (w) = ρt 2 w 2 -gt (w). Thus, solving the subproblem (2.3) in Online DCA amounts to solving the problem

min ρ t 2 w 2 -ρ t w t -st , w : w ∈ S .
It can be equivalently reformulated to min w -w t -1 ρ t st Thus, we deduce that

w t+1 = Proj S w t - 1 ρ t st ,
where Proj S (x) is the projection of x on S. It is exactly the same as the update rule (2.4) in OGD with step size of 1/ρ t . The proof is complete.

Remark 2.1. With the same assumptions in Proposition 2.1 but t 0 = 1 and S = R n , we also indicate that OGD is a special case of Online DCA. In particular, (2.3) would be

min t i=1 ρ i 2 w 2 - t i=1 ρ i w i -si , w : w ∈ R n or, equivalently, min    w - 1 ρ (t) t i=1 ρ i w i -si 2 : w ∈ R n    ,
where ρ (t) := t i=1 ρ i , ∀t. Thus, we obtain the following update rule

w t+1 = 1 ρ (t) t i=1 ρ i w i -si = 1 ρ (t) ρ t w t -st + ρ (t-1) w t = w t - 1 ρ (t) st . Proposition 2.2. Assume that t 0 = 1, R(w) = 1 2η w 2 2 (η > 0)
and at the step t, the loss function is linear i.e. f t (w) = w, st where st ∈ ∂g t (w t ). Then OGD-L is a special case of Online DCA.

Proof. We can see that DC components of f t are g t (w) = w, st and h t (w) = 0. According to Online DCA, (2.3) can be reformulated as

min 1 2η w 2 + t i=1 si , w : w ∈ S or, equivalently, min    w + η t i=1 si 2 : w ∈ S    .
Thus, we have

y t+1 = y t -ηs t , w t+1 = Proj S y t+1 . (2.7)
This update rule is the same as (2.5)-(2.6) in the algorithm OGD-L with step size of η. The proof is complete.

Next, we are going to present a particular Online DCA based scheme which is very efficient in practice problems.

Online Learning and Applications to Online Binary Linear Classification

ODCA: a proposed Online DCA based scheme

According to Section 1.2.3, we propose a particular scheme based on Online DCA where the covex subproblem (2.3) is solved by approximating by one iteration of subgradient method (see e.g. [START_REF] Shor | Minimization Methods for Non-Differentiable Functions[END_REF] for the definition). In particular, applying one iteration of the projected subgradient method at the point w t with step size of η t , we derive from (1.13) the following update rule:

w t+1 = Proj S (w t -η t s t ), (2.8)
where the vector s t ∈ ∂g t (w t ) -z t . Thus, a particular Online DCA based scheme (ODCA) applied to (2.2) can be described as follows.

ODCA: Online DCA-projected Subgradient scheme Initialization: let w 1 be an initial point, {η t } be the sequence of step sizes

for t = 1, 2, . . . , T do 1. Compute z t ∈ ∂h t (w t ). 2. Compute s t ∈ (∂g t (w t ) -z t ).
3. Compute w t+1 ∈ S by the update rule

w t+1 = Proj S (w t -η t s t ). (2.9)
end for Remark 2.2. It is worth noting that in the Online DCA scheme, the problem (1.12) can be solved approximating by k iterations of subgradient method at each step. In such a case, the quality of the solution w t+1 to the problem (1.12) may be improved, however one possible disadvantage is the expensive computation time. This chapter will address the case of one iteration. Studying the effect of the number of iterations of subgradient method on the efficiency of ODCA algorithms will be a part of future works.

Analysis of ODCA

In this section, we will present the regret of the sequence {w t } generated by ODCA.

Recall the regret of the algorithm A is defined by

Regret T A := T t=1 f t (w t ) -min w∈S T t=1 f t (w). (2.10)
First we make four necessary assumptions below and then we indicate the regret bound of ODCA, which is stated in Theorem 2.1 and Theorem 2.2.

Assumption 2.1. There exists a vector u * ∈ S such that for t = 1, . . . , T ,

u * ∈ arg min w∈S f t (w). (2.11)
Assumption 2.2. There exists a positive parameter α such that at the step t,

α 2 u * -w t 2 ≤ g t (w t ) -g t (u * ) -z t , w t -u * .
(2.12)

Assumption 2.3. There exists a nonnegative parameter β such that at the step t,

h t (u * ) -h t (w t ) -z t , u * -w t ≤ β 2 u * -w t 2 .
(2.13) Assumption 2.4. There exists a positive parameter γ such that at the step t,

g t (w t ) -g t (u * ) ≤ r t , w t -u * - γ 2 u * -w t 2 . ( 2
Regret T ODCA ≤ 3DL(α + β) √ T 2α .
Proof. From the definition of Regret T A (2.10), we have

Regret T ODCA = T t=1 f t (w t ) -min w∈S T t=1 f t (w) ≤ T t=1 f t (w t ) -min w∈S f t (w) . (2.16)
It derives from Assumption 2.1 that

f t (w t ) -min w∈S f t (w) = f t (w t ) -f t (u * ) = [g t (w t ) -g t (u * )] + [h t (u * ) -h t (w t )] = [g t (w t ) -g t (u * )] + [h t (u * ) -h t (w t ) -z t , u * -w t ].
(2.17)

From (2.16), (2.17) and Assumptions 2.2, 2.3, we obtain

Regret T ODCA ≤ 1 + β α T t=1 [g t (w t ) -g t (u * )]. (2.18)
Online Learning and Applications to Online Binary Linear Classification Since s t ∈ ∂g t (w t ), it follows

Regret T ODCA ≤ 1 + β α T t=1 s t , w t -u * . (2.19)
Similarly to the proof of Theorem 3.1 in [START_REF] Hazan | Introduction to online convex optimization[END_REF], we find an upper bound of s t , w t -u * , t ∈ {1, . . . , T }. In particular, by the definition of the sequence {w t } t=1,...,T as (2.8), we have

w t+1 -u * 2 ≤ (w t -η t s t ) -u * 2 = (w t -u * ) -η t s t 2 = w t -u * 2 -2η t s t , w t -u * + η 2 t s t 2 .
Thus,

s t , w t -u * ≤ w t -u * 2 -w t+1 -u * 2 2η t + η t 2 s t 2 .
(2.20)

It derives that

Regret T ODCA ≤ 1 + β α T t=1 w t -u * 2 1 2η t - 1 2η t-1 + L 2 2 T t=1 η t ≤ 1 + β α D 2 2 T t=1 1 η t - 1 η t-1 + L 2 2 T t=1 η t ≤ 1 + β α D 2 2η T - D 2 2η 0 + L 2 2 T t=1 η t .
Let us define 1 η 0 := 0 and η t = D L √ t for all t = 1, . . . , T . Then, we have

Regret T ODCA ≤ 1 + β α DL √ T 2 + DL 2 2 √ T -1 ≤ 3DL(α + β) √ T 2α .
The proof is complete.

From Theorem 2.1, we can say that Regret T ODCA grows sublinearly with the number of steps T , i.e. lim T →+∞

Regret T

ODCA /T = 0. However we can achieve a better regret bound, specifically logarithmic regret -O(log(T )) if Assumption 2.4 is also verified, which is stated in the following theorem.

Theorem 2.2. Suppose that ODCA generates the sequence {w t } t=1,...,T and Assumptions 2.1, 2.2, 2.3 and 2.4 are verified. Then, we have

Regret T ODCA ≤ L 2 (α + β) (1 + log(T )) 2αγ .
Proof. From (2.18) in Theorem 2.1 and Assumption 2.4, we derive that

Regret T ODCA ≤ 1 + β α T t=1 s t , w t -u * - γ 2 u * -w t 2 . (2.21)
Similarly to the proof of Theorem 3.3 in [START_REF] Hazan | Introduction to online convex optimization[END_REF], let us define 1 η 0 := 0 and η t = 1 γt for all t = 1, . . . , T . From (2.20) and (2.21), we have

Regret T ODCA ≤ 1 + β α T t=1 w t -u * 2 1 2η t - 1 2η t-1 - γ 2 =0 + L 2 2 T t=1 η t ≤ 1 + β α L 2 2γ T t=1 1 t ≤ L 2 (α + β) (1 + log(T )) 2αγ .
The proof is complete.

Remark 2.3. (Comments about Assumptions 2.1-2.4)
First of all, Assumption 2.1 can be replaced by the following assumption -that is, there exists a vector u * such that

u * ∈ arg min w∈S T t=1 f t (w).
In this case, the inequality (2.18) is guaranteed and we still yield the same regret bound.

In practice problems, Assumption 2.1 is verified more easily.

Next, Assumption 2.2 can be deduced from the inequality

µ 2 u * -w t 2 ≤ f t (w t ) -f t (u * )
where µ > β and β is defined in Assumption 2.3. Indeed, we have [START_REF] Pham Dinh | Convex analysis approach to DC programming: theory, algorithms and applications[END_REF]).

g t (w t ) -g t (u * ) = f t (w t ) -f t (u * ) -h t (u * ) -h t (w t ) -z t , u * -w t ≥ α 2 u * -w t 2 , α = µ -β > 0. Moreover, it is easy to verify Assumption 2.3 if h t is differentiable with β-Lipschitz gradient on S. Meanwhile, Assumption 2.4 is satisfied if g t is γ-convex on S (γ > 0) i.e. g t -(γ/2) • 2 is convex (see e.g.
In the sequel, we are going to present how to employ ODCA for solving the problems in the topic of online classification, specifically online binary linear classification.

Online Learning and Applications to Online Binary Linear Classification

Online DCA for Online Binary Linear Classification problems

In this section, we focus on handling the problems in the topic of online binary linear classification (OBLC) where the set of instances is X = R n , the set of linear classifiers is S = R n and the set of binary labels is Y = {-1, 1}. Formally, at the step t, after receiving an instance x t ∈ X , the learner will find a linear classifier w t ∈ S and then predict the corresponding binary label p t = p t (w t ) ∈ Y where the function p t is defined by

p t (w) := sign( w, x t ) = 1 if w, x t ≥ 0, -1 otherwise. (2.22)
After that, the correct label y t ∈ Y is revealed. In such a case, the quality of predictions is assessed by a loss function, specifically 0-1 loss function, which measures the difference between the predicted label and the correct one. The 0-1 loss function, denoted by t , is defined as

t (p t (w), y t ) := 1 {pt(w) =yt} (w) = 1 {yt w,xt ≤0} (w) = 1 if y t w, x t ≤ 0, 0 otherwise, (2.23)
where 1 X is the indicator function of X. Obviously, this loss function is nonconvex, nonsmooth.

There exist many online classification algorithms with different approaches such as Perceptron [START_REF] Novikoff | On convergence proofs for perceptrons[END_REF][START_REF] Rosenblatt | The perceptron: a probabilistic model for information storage and organization in the brain[END_REF][START_REF] Van Der Malsburg | Frank rosenblatt: Principles of neurodynamics: Perceptrons and the theory of brain mechanisms[END_REF], Approximate Maximal Margin Classification Algorithm (ALMA) [START_REF] Gentile | A new approximate maximal margin classification algorithm[END_REF], Relaxed Online Maximum Margin Algorithm (ROMMA) [START_REF] Li | The relaxed online maximum margin algorithm[END_REF], Passive-Aggressive learning algorithms (PA) [START_REF] Crammer | Online passive-aggressive algorithms[END_REF] and their variants. Hoi et al. [START_REF] Hoi | Libol: A library for online learning algorithms[END_REF] conducted a library of scalable and efficient online learning algorithms for large-scale online classification tasks. As the first works, [START_REF] Ho | Online DC optimization for online binary linear classification[END_REF] developed an online classification technique based on an online version of DCA where a DC function was explored to approximate the nonconvex, nonsmooth 0-1 loss function. Our works will thoroughly study more efficient OBLC algorithms with formal regret bounds.

First of all, we propose three different DC approximation functions f t on S instead of the 0-1 loss function t . Let us denote by M A the set of steps where the prediction of the algorithm A may be wrong in the sense that f t (w t ) > 0. If a step t / ∈ M A , then we observe the function f t (w) = 0, ∀w ∈ R n . Otherwise, we define the DC approximation function f t satisfying the following condition

f t (w t ) ≥ t (p t , y t ).
(2.24)

In particular, we suggest three DC functions: two piecewise linear functions (as in e.g. [START_REF] Ho | Online DC optimization for online binary linear classification[END_REF]) similar to ramp loss [START_REF] Collobert | Large scale transductive SVMs[END_REF][START_REF] Collobert | Trading convexity for scalability[END_REF] and one sigmoid function (see e.g. [START_REF] Mason | Boosting algorithms as gradient descent[END_REF]). In the following, we will present how to design online algorithms based on ODCA for these functions.

First piecewise linear function

At step t, we observe the function f

(1) t : R n → [0, 1] defined as follows [START_REF] Ho | Online DC optimization for online binary linear classification[END_REF]:

f (1)
t (w) := max 0, min

τ 2t τ 1t , -y t w, x t τ 1t (2.25)
where τ 2t , τ 1t are two positive parameters. It is evident that if τ 2t = τ 1t , then for any

w ∈ R n , 0 ≤ f (1) t (w) ≤ t (w) ≤ 1 and f (1) t (w) → t (w) as τ 1t → 0. Moreover, DC components of f (1) t are g (1)
t (w) = max 0, -y t w, x t τ 1t and h

(1)

t (w) = max 0, -τ 2t -y t w, x t τ 1t .
Remark that we have to choose the parameters τ 1t , τ 2t such that (2.24) is satisfied. However, it is never true if there exists a step t such that y t w t , x t = 0 since t (p t , y t ) = 1 and f

t (w t ) = 0. In such a case, we investigate another DC function, namely f

t , defined by

f (2) t (w) := max 0, min 1, τ 1t -y t w, x t τ 1t , (2.26)
and its DC decomposition is

f (2) t (w) = g (2)
t (w) -g

t (w) = max 0,

τ 1t -y t w, x t τ 1t .
Obviously, we have f

(2) t (w t ) = 1 = t (p t , y t ).
For these DC functions, let us define the set

M PiL1 := {t ∈ {1, . . . , T } : f (1) t (w t ) > 0 or y t w t , x t = 0} = {t ∈ {1, . . . , T } : y t w t , x t ≤ 0} (2.27)
and the set

N PiL1 := {t ∈ M PiL1 : y t w t , x t < 0}.
According to the ODCA algorithm, at step t, we compute first the vector z t ∈ ∂h

(1)

t (w t ) (resp. z t ∈ ∂g (1)
t (w t )), then the vector s t ∈ (∂g

(1) t (w t ) -z t ) (resp. s t ∈ (∂g (2) t (w t ) -z t )) when t ∈ N PiL1 (resp. t ∈ M PiL1 \ N PiL1
) and then the vector w t+1 using (2.8).

Compute ∂g

(1) t , ∂h

t (w) =            {0} if y t w, x t > 0, -y t x t τ 1t , 0 if y t w, x t = 0, -y t x t τ 1t if y t w, x t < 0, (2.28)
Online Learning and Applications to Online Binary Linear Classification

∂h (1) t (w) =            {0} if y t w, x t > -τ 2t , -y t x t τ 1t , 0 if y t w, x t = -τ 2t , -y t x t τ 1t if y t w, x t < -τ 2t , (2.29)
and ∂g

(2)

t (w) =            {0} if y t w, x t > τ 1t , -y t x t τ 1t , 0 if y t w, x t = τ 1t , -y t x t τ 1t if y t w, x t < τ 1t .
(2.30)

Here [a, b] is the line segment between a and b. When t ∈ N PiL1 , we choose z t ∈ ∂h

(1)

t (w t ), r t ∈ ∂g (1)
t (w t ) and

s t = r t -z t ∈ (∂g (1)
t (w t ) -z t ) as follows:

r t = -y t x t τ 1t , z t = -y t x t τ 1t if y t w t , x t < -τ 2t , 0 if -τ 2t ≤ y t w t , x t < 0,
and

s t = 0 if y t w t , x t < -τ 2t , -y t x t τ 1t if -τ 2t ≤ y t w t , x t < 0.
Similarly, when t ∈ M PiL1 \ N PiL1 , we have

r t = -y t x t τ 1t
, z t = 0 and s t = -y t x t τ 1t .

Choose the parameters τ 1t , τ 2t : in order to satisfy (2.24), for any t ∈ M PiL1 , we propose the following choices

τ 1t = min{τ 1 , -y t w t , x t } if t ∈ N PiL1 , τ 1 if t ∈ M PiL1 \ N PiL1 ,
where τ 1 is a positive tuning parameter, and

τ 2t = -y t w t , x t . (2.31)
Choose the learning rate η t : we consider the constant learning rate that is η t = η for all t. In this case, the prediction labels defined as (2.22) and the set M PiL1 is completely independent of the value of η (see [START_REF] Shalev-Shwartz | Online learning and online convex optimization[END_REF] for more details). Consequently, we can choose η t = 1 for all t.

Finally, ODCA with the first piecewise linear function is given by Algorithm 2.1 (ODCA-PiL1).

Algorithm 2.1 ODCA with first piecewise linear function (ODCA-PiL1)

Initialization: let τ 1 be a positive number, w 1 be an initial point

for t = 1, 2, . . . , T do if -τ 1 ≤ y t w t , x t < 0 then w t+1 = w t - x t w t , x t .
else if y t w t , x t = 0 or y t w t , x t < -τ 1 then

w t+1 = w t + y t x t τ 1 . else w t+1 = w t . end if end for
According to the analysis of ODCA in Section 2.2.4, we archive the logarithmic regret of the ODCA-PiL1 algorithm as stated in Theorem 2.2 for this nonconvex piecewise linear function. In order to get this result, we need to indicate that all four Assumptions 2.1-2.4 are satisfied for two cases: t ∈ M PiL1 \N PiL1 and t ∈ N PiL1 as the following lemmas. Throughout these lemmas, we further assume that at the step t ∈ M PiL1 , x t = 0. When x t = 0, the algorithm makes no update, which still holds for the update step in Algorithm 2.1 (corresponding to w t+1 = w t ). We also assume that there exists a vector u * ∈ R n such that for all t = 1, . . . , T ,

y t u * , x t ≥ 2τ 1 .
(2.32)

Lemma 2.1. For the DC function (2.25) and step t ∈ N PiL1 , there exist the parameters α, γ and the vector u * such that Assumptions 2.1, 2.2, 2.4 are satisfied. Moreover, Assumption 2.3 is satisfied for all β ≥ 0.

Proof. As for t ∈ N PiL1 , from (2.32), it is easy to check that Assumption 2.1 is satisfied. Moreover, it is worth noting that u * = w t for all t = 1, . . . , T . Indeed, assume the contrary that is u * = w t for some t ∈ N PiL1 , so y t u * , x t < 0 contradicting (2.32). Now, we will show that Assumptions 2.2-2.4 can be satisfied. In particular, as for Assumption 2.2, let us define the function g

t := g

t -z t , • and we have g

t (w t ) -g

t (u * ) = -

y t w t , x t τ 1t > 0.
Thus, the inequality (2.12) is satisfied with α ≤ min

t∈N PiL1 -2y t w t , x t τ 1t u * -w t 2 .
Regarding Assumption 2.3, from (2.31) and the definition of h

t , we have that for any t ∈ N PiL1 and β ≥ 0,

h (1) t (u * ) -h (1) t (w t ) -z t , u * -w t = 0 ≤ β 2 u * -w t 2 .
Online Learning and Applications to Online Binary Linear Classification Concerning Assumption 2.4, we have that for any t ∈ N PiL1 , g

t (u * ) -g

(1)

t (w t ) -r t , u * -w t = y t w t , x t τ 1t - -y t x t τ 1t , u * -w t = y t u * , x t τ 1t > 0.
Thus, the inequality (2.14) is satisfied with γ ≤ min

t∈N PiL1 2y t u * , x t τ 1t u * -w t 2 . The proof is complete.
Lemma 2.2. For step t ∈ M PiL1 \N PiL1 and the loss function (2.26), Assumptions 2.1, 2.2, 2.4 are satisfied with the suitable parameters α, γ and vector u * . Moreover, Assumption 2.3 is satisfied for all β ≥ 0.

Proof. In case t ∈ M PiL1 \ N PiL1 (that is y t w t , x t = 0), we can readily verify Assumption 2.1 by (2.32). We can see that if t ∈ M PiL1 \ N PiL1 , then u * = w t , since y t w t , x t = 0 while y t u * , x t ≥ 2τ 1 > 0.

We are going to show that Assumptions 2.2-2.4 are also satisfied. In particular, as for Assumption 2.2, let us define the function g

(2) t := g (2) t -z t , • for t ∈ M PiL1 \ N PiL1 and we have g (2)
t (w t ) -g

t (u * ) = 1 ≥ α 2 u * -w t 2 , (2)
where α ≤ min

t∈M PiL1 \N PiL1 2 u * -w t 2 .
Regarding Assumption 2.3, we have that for any t ∈ M PiL1 \ N PiL1 and β ≥ 0,

g (1) t (u * ) -g (1) t (w t ) -z t , u * -w t = 0 ≤ β 2 u * -w t 2 .
Concerning Assumption 2.4, we have that for any

t ∈ M PiL1 \ N PiL1 , g (2)
t (u * ) -g

(2)

t (w t ) -r t , u * -w t = -1 - -y t x t τ 1t , u * -w t = y t u * , x t τ 1t -1 ≥ γ 2 u * -w t 2 ,
where γ ≤ min

t∈M PiL1 \N PiL1 2 u * -w t 2 .
The proof is complete.

Consequently, we derive the following corollary for the regret bound of the ODCA-PiL1 algorithm (whose proof is straightforward with the choice β = α).

Corollary 2.1. Assume that ODCA-PiL1 generates the sequence {w t } t=1,...,T . Then, we have

Regret T ODCA-PiL1 ≤ L 2 (1 + log(T )) γ (2.33)
where L is defined as (2.15) and the positive parameter

γ ≤ min t∈M PiL1 2 min {τ 1t , ψ(y t u * , x t )} τ 1t u * -w t 2 , the real-value function ψ(x) = x if x > 0, +∞ otherwise.
The following theorem provides a bound on the number of prediction mistakes (which is called a mistake bound, for short) for the ODCA-PiL1 algorithm, which is defined as the bound on the number of steps where p t = y t .

Theorem 2.3. For any w ∈ R n , the number of prediction mistakes made by ODCA-PiL1 has an upper bound that is the root, x 1 , of the equation

x -a PiL1 -b PiL1 (1 + log(x)) = 0 where a PiL1 = t∈N PiL1 f (1)
t (w) + t∈M PiL1 \N PiL1 f (2) t (w), b PiL1 = L 2 /γ PiL1 , x 1 ≥ b PiL1 , the positive parameter γ PiL1 ≤ min {γ, L 2 }.
Proof. From the inequality (2.24), Corollary 2.1 and the definition of γ PiL1 , we derive that for any w ∈ R n ,

|M PiL1 | ≤ t∈N PiL1 f (1) t (w t ) + t∈M PiL1 \N PiL1 f (2) t (w t ) ≤ t∈N PiL1 f (1) t (w) + t∈M PiL1 \N PiL1 f (2) t (w) + L 2 (1 + log(|M PiL1 |)) γ PiL1 (2.34)
where |M PiL1 | is the number of steps of the set M PiL1 .

Using the definition of a PiL1 and b PiL1 , it is evident that a PiL1 ≥ 0, b PiL1 ≥ 1 and the inequality (2.34) can be rewritten as follows.

|M PiL1 | ≤ a PiL1 + b PiL1 (1 + log(|M PiL1 |)) .

The real function r

PiL1 : (0, +∞) → R, r PiL1 (x) = x-a PiL1 -b PiL1 (1 + log(x)) is strictly convex on (0, +∞). Since lim x→0 + r PiL1 (x) = lim x→+∞ r PiL1 (x) = +∞ and r PiL1 (b PiL1 ) ≤ 0, the equation r PiL1 (x) = 0 has two roots x 1 , x 2 such that 0 < x 2 ≤ b PiL1 ≤ x 1 .
The proof is complete.

ODCA applied to OBLC problems with this second piecewise linear loss function can be summarized in Algorithm 2.2 (ODCA-PiL2).

Algorithm 2.2 ODCA with second piecewise linear function (ODCA-PiL2)

Initialization: let w 1 be an initial point, {η t } be a sequence of learning rates, and {τ 1t }, {τ 2t } be a sequence of positive parameters mentioned above

for t = 1, 2, . . . , T do if -τ 2t ≤ y t w t , x t < τ 1t then w t+1 = w t + η t y t x t τ 1t . else w t+1 = w t . end if end for
In the following, we analyze the regret bound of the ODCA-PiL2 algorithm thanks to the analysis of ODCA in both cases t ∈ N PiL2 and t ∈ M PiL2 \ N PiL2 , as the following lemma. Before stating, we assume that x t = 0, ∀t and there exists a vector u * ∈ R n such that for all t = 1, . . . , T ,

y t u * , x t ≥ 2τ, (2.37)
where τ = max t=1,...,T τ 1t .

Lemma 2.3. For the loss function (2.35) and step t ∈ M PiL2 , the conditions (i)-(iv) are satisfied with the suitable parameters α, γ and vector u * . Moreover, the condition (iii) is satisfied for all β ≥ 0.

Proof. Similarly to Lemma 2.1 and Lemma 2.2, it is easy to check that u * ∈ arg min w∈R n f

t (w) for all t ∈ M PiL2 and if t ∈ N PiL2 , then u * = w t , since y t w t , x t < τ 1t while y t u * , x t ≥ 2τ 1t .

We will show that Assumptions 2.2-2.4 are satisfied for both cases t ∈ N PiL2 and t ∈ M PiL2 \ N PiL2 . Let us define the function g

(3) t := g (3) t -z t , • . • As for t ∈ N PiL2 , we have g (3) t (w t ) -g (3) t (u * ) = τ 1t -y t w t , x t τ 1t ≥ α 2 u * -w t 2 ,
where α ≤ min t∈N PiL2

2(τ 1t -y t w t , x t ) τ 1t u * -w t 2 . Thus, Assumption 2.2 is satisfied.

Regarding Assumption 2.3, we have that for β ≥ 0, h

t (u * ) -h

(1)

t (w t ) -z t , u * -w t = 0 ≤ β 2 u * -w t 2 .
Online Learning and Applications to Online Binary Linear Classification Concerning Assumption 2.4, we have

g (3) t (u * ) -g (3) t (w t ) -r t , u * -w t = g (2) t (u * ) -g (2) t (w t ) -r t , u * -w t = y t w t , x t τ 1t -1 + y t x t τ 1t , u * -w t = y t u * , x t τ 1t -1 ≥ 1. With γ ≤ min t∈N PiL2 2 u * -w t 2 , Assumption 2.4 is verified. • As for t ∈ M PiL2 \N PiL2 , it is evident to notice that Assumptions 2.2-2.4 are satisfied for all α > 0, β ≥ 0, γ > 0.
The proof is complete.

As a result of Lemma 2.3, the regret bound of the ODCA-PiL2 algorithm is stated as the following corollary.

Corollary 2.2. Assume that ODCA-PiL2 generates the sequence {w t } t=1,...,T . Then, we have

Regret T ODCA-PiL2 ≤ L 2 (1 + log(T )) γ
where the positive parameter γ ≤ min t∈N PiL2 2 u * -w t 2 and L is defined as (2.15).

We also archive the mistake bound for the ODCA-PiL2 algorithm as the following theorem.

Theorem 2.4. For any w ∈ R n , the number of prediction mistakes made by ODCA-PiL2 has an upper bound that is the root, x 1 , of the equation

x -a PiL2 -b PiL2 (1 + log(x)) = 0 where a PiL2 = t∈M PiL2 f (3) t (w), b PiL2 = L 2 /γ PiL2 , x 1 ≥ b PiL2 and the positive parameter γ PiL2 ≤ min {γ, L 2 }.
Proof. This theorem is proven similarly to Theorem 2.3 but remark that from (2.23) and (2.36), the number of prediction mistakes is bounded from above by |M PiL2 |.

Sigmoid function

We here consider a loss function which takes a form of sigmoid function [START_REF] Mason | Boosting algorithms as gradient descent[END_REF]. In particular, at step t, we observe the following loss function:

f (4) t (w) := max {1 -tanh(δ t ), 1 -tanh(κ t y t w, x t )} (2.38)
where δ t , κ t is positive parameters, the increasing function tanh : R → [-1, 1] is defined by tanh(s) = e s -e -s e s + e -s .

It is straightforward to see that (2.24) is satisfied for the sequence {f

t } t=1,...,T . Since f (4) t (w t ) > 0 at all steps, we have to update w t at all steps, which seems to be expensive in terms of rapidity. In order to alleviate this trouble, we propose a threshold ε ∈ [0, 1) to define the set M Sig as follows.

M Sig := {t ∈ {1, . . . , T } : 1 -tanh(κ t y t w t , x t ) > ε}.

In the case t / ∈ M Sig i.e. 1 -tanh(κ t y t w t , x t ) ≤ ε < 1, we obtain y t w t , x t > 0 that is to say t (p t , y t ) = 0. Thus, the condition (2.24) can be satisfied with the observed loss function f

(4) t = 0. When t ∈ M Sig , we see that f (4) t (w) is DC and its DC decomposition is f (4) t (w) = g (4) t (w) -h (4) t (w) where h (4) t (w) := 2e -4κtyt w,xt e -2κtyt w,xt + 1 , g (4)
t (w) := max 1 -tanh(δ t ) + h

t )(w t ) and then computing the prediction vector w t+1 as the update rule (2.8).

Compute ∂g

(4) t , ∂h (4)
t : the functions c t and h (4) t are differentiable and we have

∂c t (w) = {∇c t (w)} = {-2κ t y t x t c t (w)} , ∂h (4) t (w) = {∇h (4) t (w)} = - 4κ t y t x t e -2κtyt w,xt (2e 2κtyt w,xt + 1) (e 2κtyt w,xt + 1) 2 ,
and

∂g (4) t (w) =      {∇c t (w)} if δ t > κ t y t w, x t , ∇c t (w), ∇h (4)
t (w) if δ t = κ t y t w, x t , {∇h (4) t (w)} if δ t < κ t y t w, x t .
Online Learning and Applications to Online Binary Linear Classification Thus, for t ∈ M Sig , we choose z t ∈ ∂h

(4) t (w t ), r t ∈ ∂g (4)
t (w t ) and s t = r t -z t ∈ (∂g [START_REF] Azoury | Relative loss bounds for on-line density estimation with the exponential family of distributions[END_REF] t (w t ) -z t ) as follows:

z t = ∇h (4) t (w t ) = -2κ t y t x t c t (w t )m t , r t = ∇c t (w t ) if δ t > κ t y t w t , x t , ∇h (4) t (w t ) if δ t ≤ κ t y t w t , x t ,
where m t = 2e 2κtyt w t ,xt + 1 (e 2κtyt w t ,xt + 1) 2 , and

s t =    - 4κ t y t x t e 2κtyt w t ,xt
(e 2κtyt w t ,xt + 1)

2 if δ t > κ t y t w t , x t , 0 if δ t ≤ κ t y t w t , x t .
Choose the parameters δ t , κ t , ε and learning rate η t : we propose the following choice: for all t, κ t = κ/ x t and η t = C/ √ t where κ, C are positive tuning parameters, ε is a tuning threshold in [0, 1) and the positive number We are going to present the regret bound of the ODCA-Sig algorithm. First, we indicate in Lemma 2.4 that Assumptions 2.1-2.4 can be satisfied for the proposed loss function (2.38). Then, we can archive the logarithmic regret as presented in Corollary 2.3. Finally, the mistake bound of the ODCA-Sig algorithm is stated in Theorem 2.5.

δ t = κ t y t w t , x t -ln(m t )/2. ( 2
Before showing Lemma 2.4, we assume that x t = 0 ∀t and there exists a vector u * ∈ R n such that for all t = 1, . . . , T ,

κ t y t u * , x t ≥ δ, (2.40)
where δ = max t=1,...,T δ t .

Lemma 2.4. For the loss function (2.38) and t ∈ M Sig , there exist the parameters α > 0, β ≥ 0, γ > 0 and vector u * such that Assumptions 2.1-2.4 are satisfied.

Proof. From (2.40), we derive that f (4)

t (w) ≥ 1 -tanh(δ t ) = f (4)
t (u * ) for all w ∈ R n which means u * ∈ arg min w∈R n f (4) t (w). We also notice that u * = w t for all t ∈ M Sig since f

(4) t (w t ) > 1 -tanh(δ t ).
Let us define for t ∈ M Sig the function

g (4) t (w) := g (4) t (w) -z t , w = max 1 -tanh(δ t ) + h (4) t (w), c t (w) + 2c t (w t )m t κ t y t w, x t .
As for Assumption 2.2, we have

g (4) t (w t ) -g (4) t (u * ) = c t (w t ) -c t (u * ) + 2c t (w t )m t κ t y t w t -u * , x t = c t (w t ) 1 - c t (u * -w t ) 2 -2m t κ t y t u * -w t , x t = c t (w t ) 1 -e -2κtyt u * -w t ,xt -2m t κ t y t u * -w t , x t .
Due to (2.39), it is easy to prove that g

t (w t ) -g

t (u * ) > 0. By setting

α ≤ min t∈M Sig c t (w t ) -c t (u * ) + 2c t (w t )m t κ t y t w t -u * , x t u * -w t 2 , (2.41)
we derive that this assumption is verified.

Concerning Assumption 2.3, since h

(4) t is convex and differentiable we have

h (4) t (u * ) -h (4) t (w t ) -z t , u * -w t ≤ ∇h (4) t (u * ) -∇h (4) t (w t ), u * -w t ≤ ∇h (4) t (u * ) -∇h (4)
t (w t ) . u * -w t , and ∇h

t (u * ) -∇h

(4) t (w t ) = 8κ t x t c t (u * )[c t (-u * ) + 1] [c t (-u * ) + 2] 2 - c t (w t )[c t (-w t ) + 1] [c t (-w t ) + 2] 2 ≤ κ t x t 2 [c t (u * ) + 2][c t (-w t ) + 2] 2 -[c t (w t ) + 2][c t (-u * ) + 2] 2 ≤ κ t x t |c t (u * -2w t ) -c t (w t -2u * )| + 4|c t (u * -w t ) -c t (w t -u * )| +2|c t (u * ) -c t (w t )| + |c t (-2w t ) -c t (-2u * )| + 2|c t (-w t ) -c t (-u * )| .
For any x, y ∈ R n , we have

c t (x) -c t (y) = c t (x) [1 -c t (y -x)/2] = c t (x) 1 -e -2κtyt y-x,xt ≤ 2c t (x)κ t y t y -x, x t ≤ 2c t (x)κ t x t y -x .
Online Learning and Applications to Online Binary Linear Classification Thus, we readily derive that

|c t (x) -c t (y)| ≤ 2 max{c t (x), c t (y)}κ t x t y -x .
Let us define

K t = 2e 2κt(| u * -w t ,xt |+max{| u * ,xt |,| w t ,xt |}) .
It follows that

h (4) t (u * ) -h (4) t (w t ) -z t , u * -w t ≤ 26κ 2 t x t 2 K t u * -w t 2 .
Consequently, Assumption 2.3 is satisfied with

β ≥ max t∈M Sig 26κ 2 t x t 2 K t . (2.42)
Finally, regarding Assumption 2.4, we have

g (4) t (u * ) -g (4) t (w t ) -r t , u * -w t = c t (u * ) -c t (w t ) -∇c t (w t ), u * -w t = c t (u * ) -c t (w t ) + 2c t (w t )κ t y t u * -w t , x t = c t (w t ) c t (u * -w t ) 2 -1 + 2κ t y t u * -w t , x t = c t (w t ) e -2κtyt u * -w t ,xt -1 + 2κ t y t u * -w t , x t .
Obviously, e -2κtyt u * -w t ,xt + 2κ

t y t u * -w t , x t > 1. If γ ≤ min t∈M Sig c t (u * ) -c t (w t ) + 2c t (w t )κ t y t u * -w t , x t u * -w t 2 , (2.43)
then this assumption is verified.

The proof is complete.

Corollary 2.3. Assume that ODCA-Sig generates the sequence {w t } t=1,...,T . Then, we have

Regret T ODCA-Sig ≤ L 2 (α + β) (1 + log(T )) 2αγ .
where the parameters α, β, γ, L are defined as (2.41), (2.42), (2.43), (2.15), respectively.

Theorem 2.5. For any w ∈ R n , the number of prediction mistakes made by ODCA-Sig has an upper bound that is the root, x 1 , of the equation

x -a Sig -b Sig (1 + log(x)) = 0 where a Sig = t∈M Sig f (4) t (w), b Sig = L 2 (α + β)/(2αγ Sig ),
)
where T is the total number of steps (more exactly, the total number of instances) and n is the number of features.

Numerical Experiments

In the numerical experiments, we study the performance of the proposed Online DCA algorithms ODCA-PiL1, ODCA-PiL2, ODCA-Sig for OBLC problems and compare them with five state-of-the-art first-order learning algorithms: Perceptron [START_REF] Novikoff | On convergence proofs for perceptrons[END_REF][START_REF] Rosenblatt | The perceptron: a probabilistic model for information storage and organization in the brain[END_REF][START_REF] Van Der Malsburg | Frank rosenblatt: Principles of neurodynamics: Perceptrons and the theory of brain mechanisms[END_REF], Online Gradient Descent (OGD) [START_REF] Zinkevich | Online convex programming and generalized infinitesimal gradient ascent[END_REF], Relaxed Online Maximum Margin Algorithm (ROMMA) [START_REF] Li | The relaxed online maximum margin algorithm[END_REF], Approximate Maximal Margin Classification Algorithm (ALMA) [START_REF] Gentile | A new approximate maximal margin classification algorithm[END_REF], Passive-Aggressive learning algorithms (PA) [START_REF] Crammer | Online passive-aggressive algorithms[END_REF] which are summarized in the paper [START_REF] Hoi | Libol: A library for online learning algorithms[END_REF]. We tested on a variety of benchmark datasets from UCI Machine Learning Repository2 and LIBSVM website 3 . The datasets used in our experiments cover many areas (e.g. social sciences, biology, physics) and are shown in Table 2.1. Set up experiments: All experiments were implemented in MATLAB R2013b and performed on a PC Intel(R) Xeon(R) CPU E5-2630 v2, @ 2.60GHz of 32GB RAM. The open source MATLAB package for the state-of-the-art algorithms is available in [START_REF] Hoi | Libol: A library for online learning algorithms[END_REF]. The initial point of all algorithms is 0 ∈ R n . We are interested in the following criteria to evaluate the effectiveness of the proposed algorithms: the mistake rate (defined as the ratio of the number of mistakes to the number of instances T ) and the CPU time (in seconds). For a fair comparison, we follow a so-called validation procedure as described in [START_REF] Hoi | Libol: A library for online learning algorithms[END_REF] so as to choose the best parameters for different algorithms. In particular, we first perform each algorithm by running over one random permutation of the dataset with the different parameter values and then take the value corresponding to the smallest mistake rate. The ranges of parameters for the state-of-the-art algorithms are completely described in [START_REF] Hoi | Libol: A library for online learning algorithms[END_REF] while as for our algorithms, the best parameters τ 1 , τ 2 , C, κ and ε are searched from the range of {2 -4 , 2 -3 , . . . , 2 4 }, {1, 3, . . . , 9}, {2 -4 , 2 -3 , . . . , 2 4 }, {0.1, 0.2, . . . , 1} and {0, 0.1, . . . , 0.9}, respectively. After the validation procedure, each algorithm is conducted over 20 runs of different random permutations for each dataset with the best parameters chosen. The average results and their standard deviation over these 20 runs of all algorithms are reported in In summary, these orders of the algorithms in terms of the quality and the rapidity can state the fact that the better results the algorithm reaches, the more CPU time it needs. Among the comparative algorithms in these experiments, ODCA-Sig could be the algorithm which realizes the best trade-off between these two criteria.

Conclusion

In this chapter, we have intensively investigated an approach based on Online DC programming for the problems in online learning where the loss function at each online step is nonconvex and/or nonsmooth. We have developed an online version of DCA (Online DCA) and indicated that Online DCA covers some variants of well-known online gradient descent algorithm. Moreover, we have proposed a particular Online DCA based scheme (ODCA) where we have employed subgradient method with one iteration to treat each convex subproblem. We have analyzed the formal regret bound of ODCA, specially the logarithmic regret O(log(T )). As an application, we have considered a class of problems in the topic of online binary linear classification. In particular, we have proposed three different DC approximation functions (which are of the form of piecewise linear, sigmoid functions) and have developed the corresponding Chapter 3

Online DCA for Prediction with Expert Advice

Abstract: In this chapter, we are interested in developing an important class of online learning techniques, namely prediction with expert advice. It is characterized by the fact that, at each online step, making a prediction rests on the basis of experts' predictions. One common difficulty is that the loss function to assess the quality of the prediction at each step is often 0-1 (where 0-1 loss function returns 1 if the prediction is correct and 0 otherwise), more generally, nonconvex and nonsmooth. Thus, our works investigate a DC approximation approach to overcome this difficulty. We propose two particular schemes based on Online DCA, named ODCA-SG and ODCA-ESG. Specifically, each convex subproblem in two schemes is solved by approximating by one iteration of projected subgradient method and exponentiated subgradient method, respectively. We thoroughly study the analysis of ODCA-SG and ODCA-ESG schemes in terms of regret. In particular, ODCA-SG enjoys the logarithmic regret whereas ODCA-ESG yields the sublinear regret. As an application, we develop the techniques of prediction with expert advice for solving a class of online classification problems in which the experts are represented by the well-known online classification algorithms. Proposing DC approximation functions for 0-1 loss function, we derive two corresponding online algorithms based on ODCA-SG and ODCA-ESG. The performance of the proposed algorithms is verified on various benchmark classification datasets by comparing with two online convex algorithms and a well-known algorithm, namely weighted majority.

Online DCA for Prediction with Expert Advice

Introduction

Background and related works

Online learning is a general class of techniques at the interface between machine learning and online optimization [START_REF]Online Optimization of Large Scale Systems[END_REF][START_REF] Jaillet | Online Optimization -An Introduction[END_REF]. As an important topic in online learning, prediction with expert advice establishes the foundations to the theory of prediction of individual sequences [START_REF] Cesa-Bianchi | How to use expert advice[END_REF][START_REF] Cesa-Bianchi | Prediction, Learning, and Games[END_REF]. From the late 1980s, the framework of prediction with expert advice was first introduced as a model of online learning by DeSantis, Markowsky, and Wegman; Littlestone and Warmuth [START_REF] Desantis | Learning probabilistic prediction functions[END_REF][START_REF] Littlestone | The weighted majority algorithm[END_REF]. It is characterized by the fact that an online learner will directly take advice from a group of given experts. In particular, at each step, after receiving a question, the experts give their answers to that question. On their advice, the learner will predict the corresponding answer. After that, the correct answer is revealed. The quality of predictions is assessed by a loss function between the predicted answer and the correct one. The learner and each expert suffer corresponding losses. The goal for the learner is to make a sequence of predicted answers such that the cumulative regret with respect to each expert (that is the difference between the cumulative loss of the learner and that of the expert) is as small as possible.

In the literature, [START_REF] Littlestone | The weighted majority algorithm[END_REF] proposed a well-known algorithm, namely weighted majority (WM), to handle the problems in prediction with expert advice. Its prediction's principle is quite simple: all weights of experts are initially set to 1 and at each step, if any expert has the false prediction, then his weight is reduced. In fact, there are different prediction ways which result in the different online learning algorithms. For example, the framework of prediction with expert advice rests on the weighted average prediction strategy [START_REF] Cesa-Bianchi | Prediction, Learning, and Games[END_REF][START_REF] Zhao | Active learning with expert advice[END_REF]. In the past decades, such problems with different approaches have been exploited by many researchers (see e.g. [START_REF] Cesa-Bianchi | How to use expert advice[END_REF][START_REF] Cesa-Bianchi | Prediction, Learning, and Games[END_REF][START_REF] Desantis | Learning probabilistic prediction functions[END_REF][START_REF] Freund | A decision-theoretic generalization of on-line learning and an application to boosting[END_REF][START_REF] Haussler | Tight worst-case loss bounds for predicting with expert advice[END_REF][START_REF] Vovk | Aggregating strategies[END_REF]).

Another class of techniques of prediction with expert advice is derived from the online convex optimization approach. Many effective online convex algorithms were developed such as online gradient descent algorithm (with lazy/greedy projections) [START_REF] Zinkevich | Online convex programming and generalized infinitesimal gradient ascent[END_REF], exponentiated gradient algorithm [START_REF] Azoury | Relative loss bounds for on-line density estimation with the exponential family of distributions[END_REF][START_REF] Kivinen | Relative loss bounds for multidimensional regression problems[END_REF][START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF]. However, there is one common difficulty in many problems in prediction with expert advice: the prediction domain or the loss function is not convex. Although two convexification techniques, described in [START_REF] Shalev-Shwartz | Online learning and online convex optimization[END_REF], allow to overcome this difficulty, the disadvantage of online convex optimization approach has recently been highlighted. There exist several challenges that should be addressed when solving such problems: what optimization models for these problems, how to design simple, efficient prediction techniques, what convergence properties of these techniques, etc. These motivate us to investigate a new approach for such problems based on DC (Difference of Convex functions) and DCA (DC Algorithm), which are well-known as powerful nonconvex, nonsmooth optimization tools.

Our works develop the techniques of prediction with expert advice for solving the problems in the topic of online binary linear classification (OBLC) where each expert is represented by an OBCL algorithm. In OBLC, instance becomes available in a sequential order. For the incoming new instance, the learner must predict the corresponding binary label and then the correct label is revealed. The knowledge of the correct label and the loss function to assess the predicted label is used to make the prediction better for future instance. There exist many online classification algorithms with different ways to make the prediction and to define the loss functions such as perceptron [START_REF] Novikoff | On convergence proofs for perceptrons[END_REF][START_REF] Rosenblatt | The perceptron: a probabilistic model for information storage and organization in the brain[END_REF][START_REF] Van Der Malsburg | Frank rosenblatt: Principles of neurodynamics: Perceptrons and the theory of brain mechanisms[END_REF], relaxed online maximum margin algorithm (ROMMA) [START_REF] Li | The relaxed online maximum margin algorithm[END_REF], approximate maximal margin classification algorithm (ALMA) [START_REF] Gentile | A new approximate maximal margin classification algorithm[END_REF], passive-aggressive learning algorithms (PA) [START_REF] Crammer | Online passive-aggressive algorithms[END_REF], online gradient descent (OGD) [START_REF] Zinkevich | Online convex programming and generalized infinitesimal gradient ascent[END_REF] and their variants.

Our contributions

In this chapter, our contributions are multiple. Firstly, we present Online DC programming for prediction with expert advice where Online DCA can be applied. Secondly, we propose two particular Online DCA based schemes, namely ODCA-SG and ODCA-ESG, where each convex subproblem is solved by approximating by one iteration of projected subgradient method and exponentiated subgradient method, respectively. Thirdly, we study the analysis of these schemes in terms of regret (that is the difference between the cumulative suffered loss and the possibly smallest cumulative loss along the learning process). Specifically, the ODCA-SG scheme enjoys the logarithmic regret whereas the ODCA-ESG scheme yields the sublinear regret. Fourthly, we propose specific DC approximation functions for the 0-1 loss function and derive two corresponding online algorithms based on ODCA-SG and ODCA-ESG for solving the problems in OBLC. Finally, in order to evaluate the efficiency of our approach, we conduct some numerical experiments on many benchmark classification datasets and compare with two state-of-the-art online convex algorithms and the well-known WM algorithm.

The rest of the chapter is organized as follows. In Section 3.2, we describe a class of problems in prediction with expert advice and present Online DC programming for these problems where Online DCA can be applied. In Section 3.3, we propose two particular Online DCA based schemes and show the analysis of these schemes in terms of regret. How to develop the techniques of prediction with expert advice based on Online DCA for solving the problems in OBLC is shown in Section 3.4. Section 3.5 reports the numerical results on several test problems which is followed by some conclusions in Section 3.6.

Prediction with Expert Advice

In this section, we formally describe the prediction with expert advice for solving the problems in OBLC [START_REF] Shalev-Shwartz | Online learning and online convex optimization[END_REF]. In particular, at the step t, the learner receives an instance with n features, denoted by x t ∈ R n . He predicts a corresponding binary label, denoted by p t ∈ {0, 1}, based on the advice of d given experts, denoted by {k i } i=1,...,d . The function k i : R n → {0, 1} corresponds to the ith linear classifier, denoted by

u i ∈ R n ,
where

k i (x) := 1 u i ,x ≥0 (x) = 1 if u i , x ≥ 0, 0 otherwise.
In addition, the experts' advice is cast as a vector v t = (p i,t ) i=1,...,d ∈ {0, 1} d where its ith element pi,t := k i (x t ), i = 1, . . . , d.

Let us define the set of weight vectors

S := w ∈ R d : d i=1 w[i] = 1 and w[i] ≥ 0, i = 1, . . . , d
where w[i] is the weight assigned to the ith expert. Remark that S is the probability simplex in R d .

At the step t, based on the experts' advice v t and the updated weight vector w t ∈ S, the prediction label is chosen as p t = p t (w t ) where the function p t (w) = 1 w,vt ≥ρ (w) and ρ is a positive index. After predicting the label p t , the correct label, denoted by y t , is revealed. The predictions of the learner are qualified by the loss function, denoted by t , defined as follows.

t (p t (w), y t ) := 1 {pt(w) =yt} (w) = 1 if p t (w) = y t , 0 otherwise. (3.1)
Recall that the learner's goal is to make a sequence of weight vectors {w t } so as to minimize the cumulative regret with respect to all d experts, defined by ). Obviously, the difficulty is that the 0-1 loss function t is nonsmooth, nonconvex. Thus, in order to alleviate the difficulty, we suggest DC approximation functions for this loss function. In this case, we are able to deal with the resulting optimization problems by Online DC programming and Online DCA. In the next section, we are going to present how to develop the Online DCA based schemes and study the analysis of these schemes in terms of regret. [START_REF] Cesa-Bianchi | Analysis of Two Gradient-Based Algorithms for On-Line Regression[END_REF][START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF][START_REF] Shor | Minimization Methods for Non-Differentiable Functions[END_REF].

Regret d := T t=1 t (

Solution methods based on

Applying one iteration of the projected subgradient method at the point w t with constant step size of η, we have the following update rule.

w t+1 = Proj S (w t -ηs t ), (3.3)
where s t ∈ (∂g t (w t ) -z t ). Thus, the ODCA-SG scheme is described as follows.

ODCA-SG: Online DCA-projected SubGradient scheme Initialization: let w 1 be an initial point, η be the constant step size for t = 1, 2, . . . , T do 1. Compute z t ∈ ∂h t (w t ). 2. Compute s t ∈ (∂g t (w t ) -z t ).

3. Compute w t+1 ∈ S using (3.3). end for Similarly, for exponentiated subgradient method, we derive the following update rule

w t+1 [i] = w t [i]e -ηs t [i] d j=1 w t [j]e -ηs t [j] , i = 1, . . . , d, (3.4)
and the corresponding ODCA-ESG scheme is summarized as follows.

ODCA-ESG: Online DCA-Exponentiated SubGradient scheme Initialization: let w 1 be an initial point, η be the constant step size for t = 1, 2, . . . , T do 1. Compute z t ∈ ∂h t (w t ). 2. Compute s t ∈ (∂g t (w t ) -z t ).

3. Compute w t+1 ∈ S using (3.4). end for Before going in detail to design the Online DCA algorithms for prediction with expert advice, we present the analysis of ODCA-SG and ODCA-ESG in terms of regret.

Online DCA for Prediction with Expert Advice

Analysis of ODCA-SG and ODCA-ESG

We concentrate on analyzing the regret bound of ODCA-SG and ODCA-ESG. Recall that the regret of an algorithm A until step T is defined as (1.14):

Regret T A = T t=1 f t (w t ) -min w∈S T t=1 f t (w),
where the sequence {w 1 , w 2 , . . . , w T } is generated by the algorithm A.

In order to archive the regret bound of ODCA-SG and ODCA-ESG, we will work on four necessary assumptions about three parameters α > 0, β ≥ 0, γ > 0 and a vector u * ∈ S which are similar to Assumptions

Regret T ODCA-ESG ≤ (α + β) U 2 log(d)T α .
Proof. From the fact that Assumptions 2.1-2.3 are verified, we have

Regret T ODCA-ESG ≤ 1 + β α T t=1 s t , w t -u * .
Online DCA for Prediction with Expert Advice

79

For the update rule (3.4), we need to find the upper bound of the sequence { s t , w t -u * } t=1,...,T . To obtain that, we use the proof technique of Lemma 2 in [START_REF] Cesa-Bianchi | Analysis of Two Gradient-Based Algorithms for On-Line Regression[END_REF]. In particular, let

D(a b) = i a[i] log(a[i]/b[i])
be the Kullback-Leibler distance between any two real vectors a and b belonging to the probability simplex S. We have

D(u * w t ) -D(u * w t+1 ) = i u * [i] log(w t+1 [i]/w t [i]) = i u * [i] -ηs t [i] - i u * [i] log j w t [j]e -ηs t [j] = -η s t , u * -log j w t [j]e -ηs t [j]+η s t ,w t -η s t ,w t = -η s t , u * + η s t , w t -log j w t [j]e -ηs t [j]+η s t ,w t = η s t , w t -u * -log j w t [j]e -ηs t [j]+η s t ,w t ≥ η s t , w t -u * - η 2 U 2 2 .
The last inequality is obtained by the result of Lemma 12 in [START_REF] Cesa-Bianchi | Analysis of Two Gradient-Based Algorithms for On-Line Regression[END_REF] and the fact that the expectation E[s t [j]] = s t , w t . Thus, we yields

s t , w t -u * ≤ D(u * w t ) -D(u * w t+1 ) η + ηU 2 2 .
Summing up over t, we get

Regret T ODCA-ESG ≤ 1 + β α D(u * w 1 ) -D(u * w t+1 ) η + ηU 2 T 2 . Since D(a b) ≥ 0 for any a, b ∈ S and if b[i] = 1/d for i = 1, . . . , T then D(a b) ≤ log(d)
, for all a ∈ S, we derive that

Regret T ODCA-ESG ≤ 1 + β α log(d) η + ηU 2 T 2 .
Let us define η := 2 log(d)

U √ T
, it concludes the proof.

In the next section, we will present how to design two algorithms based ODCA-SG and ODCA-ESG for solving a class of OBLC problems and provide a bound on the number of prediction mistakes (called a mistake bound, for short) for these algorithms based on the regret bound.

Online DCA for Prediction with Expert Advice

Online DCA for prediction with expert advice

Recall that the loss function t defined by (3.1) is nonsmooth and nonconvex. At each step, we propose a DC approximation function f t on S instead of the loss function t .

Let us denote by M the set of steps where the prediction of the algorithm may be false in the sense that f t (w t ) > 0. If a step t / ∈ M, then we observe the function f t (w) = 0, ∀w ∈ R n . Otherwise, we define the DC function f t satisfying the following condition f t (w t ) ≥ t (p t (w t ), y t ).

(3.6)

In this chapter, we suggest appropriate DC functions with the form of piecewise linear function. It is noticeable that if y t = 0, then

t (p t (w t ), y t ) = p t (w t ) = 1 w,vt ≥ρ (w t ) since p t (w t ) ∈ Y. Otherwise, t (p t (w t ), y t ) = 1 -p t (w t ) = 1 w,vt <ρ (w t ).
Thus, there are two cases y t = 1 and y t = 0 corresponding to different DC functions. Let define τ 1t , τ 2t , τ 3t be the positive parameters and less than ρ for all t.

Case 1 (y t = 1): t (w) = 1 w,vt <ρ (w).

A natural DC function would be defined as

f (1) t : S → [0, 1] w → f (1)
t (w) := max 0, min 1,

ρ -w, v t ρ -τ 1t . (3.7)
Remark 3.1. It is easy to check that for w ∈ S, we have 0 ≤ f

t (w) ≤ t (w) ≤ 1 and f (1) t (w) → t (w) as τ 1t → ρ. A DC decomposition of f (1) t is proposed as follows f (1) t = g (1) t -h (1) t where g (1) t (w) = max 0, ρ -w, v t ρ -τ 1t , h (1)
t (w) = max 0,

τ 1t -w, v t ρ -τ 1t .
For this DC function and t ∈ {1, . . . , T }, we define that t ∈ M if and only if f

t (w t ) > 0 (which is equivalent to w t , v t < ρ). For simplifying the presentation, throughout the chapter, we will restrict to the case t ∈ M.

According to the ODCA-SG scheme (resp. ODCA-ESG scheme), we compute first the vector z t ∈ ∂h (1) t (w t ), s t ∈ (∂g (1) t (w t ) -z t ) and then the vector w t+1 using (3.3) (resp. (3.4)).

Compute ∂g

(1)

t , ∂h (1)
t : by the definition of g

t (w) =            -v t ρ -τ 1t if w, v t < ρ, -v t ρ -τ 1t , 0 if w, v t = ρ, {0} if w, v t > ρ,
and ∂h

t (w) =            -v t ρ -τ 1t if w, v t < τ 1t , -v t ρ -τ 1t , 0 if w, v t = τ 1t , {0} if w, v t > τ 1t . (1)
Here [a, b] is the line segment between a and b. We choose z t ∈ ∂h

(1)

t (w t ), r t ∈ ∂g (1)
t (w t ) and s t = r t -z t ∈ (∂g

(1)
t (w t ) -z t ) as follows:

r t = -v t ρ -τ 1t , z t =    -v t ρ -τ 1t if w t , v t < τ 1t , 0 if τ 1t ≤ w t , v t < ρ, and
s t =    0 if w t , v t < τ 1t , -v t ρ -τ 1t if τ 1t ≤ w t , v t < ρ.
• Choose the parameter τ 1t : in order to satisfy (3.6), we propose the following choices

τ 1t = max{ w t , v t , τ 1 }, t = 1, . . . , T, (3.8)
where τ 1 < ρ is a positive tuning parameter.

Case 2 (y t = 0): t (w) = 1 w,vt ≥ρ (w).

We adopt a loss function as follows

f (2) t : S → [0, 1] w → f (2)
t (w) := max 0, min 1,

w, v t -ρ ρ -τ 2t . (3.9) Remark 3.2.
It is evident that for any w ∈ S, we have 0 ≤ f

(2) t (w) ≤ t (w) ≤ 1 and f (2) t (w) → t (w) as τ 2t → ρ.
It is easy to see that f

t (w) = max 0, w, v t -ρ ρ -τ 2t , h (2)
t (w) = max 0, w, v t -2ρ + τ 2t ρ -τ 2t .
It is similar for the case w t , v t = ρ.

Compute ∂g

(3) t , ∂h (3)
t : similarly, we have

r t = v t ρ -τ 3t
, z t = 0 and s t = v t ρ -τ 3t .

• Choose the parameter τ 3t : we choose τ 3t = τ 3 where τ 3 < ρ is a positive tuning parameter.

Finally, from two ODCA-SG and ODCA-ESG schemes, Online DCA applied to prediction with expert advice is given by Algorithm 3.1 (ODCA-SG) and Algorithm 3.2 (ODCA-ESG) as follows.

Algorithm 3.1 ODCA-SG for prediction with expert advice (ODCA-SG)

Initialization: let w 1 be an initial point, η be the constant step size, τ 1 , τ 2 , τ 3 be positive numbers less than ρ.

for t = 1, 2, . . . , T do if y t = p t (w t ) then w t+1 = w t else if τ 1 ≤ w t , v t < ρ or ρ < w t , v t ≤ 2ρ -τ 2 then s t = -v t ρ -w t , v t else if w t , v t = ρ then s t = v t ρ -τ 3 else s t = 0 end if w t+1 = Proj S (w t -ηs t ) end if end for
Thanks to the analysis of both ODCA-SG and ODCA-ESG schemes in Section 3.3.2, we archive the logarithmic regret of the ODCA-SG algorithm and sublinear regret of the ODCA-ESG algorithm as stated in Theorem 3.2 and Theorem 3.3 for DC functions. In order to get this result, we need to indicate that all four Assumptions 2.1-2.4 are satisfied as in Lemma 3.1 and Lemma 3.2. Throughout the lemmas, we further assume that at each step t ∈ M, τ 1t = w t , v t (resp. τ 2t = 2ρ -w t , v t ) and v t = 0.

(3.11)

When v t = 0 ∈ Y d or τ 1t > w t , v t (resp. τ 2t > 2ρ-w t , v t ) (which means τ 1 > w t , v t (resp. τ 2 > 2ρ -w t , v t ))
, the algorithm makes no update, which still holds true for the update step in both algorithms (corresponding to s t = 0).

Thus, there exists α > 0 such that the condition (ii) is satisfied, in paricular, α ≤

min t∈M 2 u * -w t 2 2 .
• Assumption 2.3 is satisfied for any β ≥ 0. In particular, we have

h (1) t (u * ) -h (1) t (w t ) -z t , u * -w t = 0 ≤ β 2 u * -w t 2 2 .
• Concerning Assumption 2.4, we have that for any t ∈ M, g

t (u * ) -g

(1)

t (w t ) -r t , u * -w t = - ρ -w t , v t ρ -τ 1t - -v t ρ -τ 1t , u * -w t = u * , v t -ρ ρ -τ 1t ≥ γ 2 u * -w t 2 2 ,
where

γ ≤ min t∈M 2( u * , v t -ρ) (ρ -τ 1t ) u * -w t 2 2
. The proof is complete. Lemma 3.2. For the DC functions (3.9), (3.10) and step t ∈ M with y t = 0, Assumptions 2.1,2.2 and 2.4 are satisfied with the suitable parameters α, γ and vector u * . Moreover, Assumption 2.3 is satisfied for all β ≥ 0.

Proof. We readily derive from (3.12) that f

(2) t (u * ) = 0 (f (3)
t (u * ) = 0 when w t , v t = ρ). Thus, Assumption 2.1 for both functions is satisfied and u * = w t for all t. Similarly to Lemma 3.1, as for the function (3.9) (resp.

(3.10)), if α ≤ min t∈M 2 u * -w t 2 2 , β ≥ 0, γ ≤ min t∈M 2(ρ -v t , u * ) (ρ -τ 2t ) u * -w t 2 2 (resp. γ ≤ min t∈M 2(τ 3t -v t , u * ) (ρ -τ 3t ) u * -w t 2 2 when w t , v t = ρ),
then Assumptions 2.2-2.4 are satisfied, which concludes the proof.

Consequently, we derive the following corollaries for the regret bound of the ODCA-SG and ODCA-ESG algorithm.

Corollary 3.1. Assume that ODCA-SG generates the sequence {w t } t=1,...,T . Then, we have

Regret T ODCA-SG ≤ L 2 (1 + log(T )) γ
where L is defined as (3.5), the positive parameter

γ ≤ min t∈M 2 u * -w t 2 2 min κ(τ 3t -v t , u * ) ρ -τ 3t , κ(ρ -v t , u * ) ρ -τ 2t , κ( u * , v t -ρ) ρ -τ 1t , (3.13
) and the real function κ(x) = x if x > 0, +∞ otherwise.

Online DCA for Prediction with Expert Advice Corollary 3.2. Assume that ODCA-ESG generates the sequence {w t } t=1,...,T where w 1 [i] = 1/d, i = 1, . . . , d. Then, we have

Regret T ODCA-ESG ≤ 2U 2 log(d)T .
where U is defined as (3.5).

The following theorems provide a mistake bound for the ODCA-SG/ODCA-ESG algorithm that is the bound on the number of steps at which p t = y t . Before stating the theorems, at the step t ∈ M, we define the function Using the definition of a and c, it is evident that a, c ≥ 0 and the inequality (3.14) can be rewritten as follows.

f t (w) :=      f (1) t (w) if y t = 1, f (2) t (w) if y t = 0, w t , v t > ρ, f (3) t (w) if y t = 0, w t , v t = ρ.
|M| ≤ a + c |M|.

It leads to the bound

|M| ≤ c + √ c 2 + 4a 2 4 .
which concludes the proof.

Remark 3.3. (Comparison with some existing algorithms: Time complexity)

We consider three existing algorithms for prediction with expert advice: a well-known WM algorithm [START_REF] Littlestone | The weighted majority algorithm[END_REF] and two online convex algorithms, namely online gradient descent with greedy projection (OGD) [START_REF] Zinkevich | Online convex programming and generalized infinitesimal gradient ascent[END_REF] and normalized exponentiated gradient (NEG) [START_REF] Azoury | Relative loss bounds for on-line density estimation with the exponential family of distributions[END_REF][START_REF] Kivinen | Relative loss bounds for multidimensional regression problems[END_REF][START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF]. In these online convex algorithms, at the step t, the 0-1 function is approximated by hinge loss function f cv t [START_REF] Shalev-Shwartz | Online learning and online convex optimization[END_REF] which is defined as

f cv t (w) = 2| w, v t -y t | if t ∈ M, 0 otherwise.
From the work of [START_REF] Wang | Projection onto the probability simplex: An efficient algorithm with a simple proof, and an application[END_REF], the complexity of the projection algorithm Proj S (x) in R d is O(d log d) with d be the number of given experts. Thus, we derive the complexity of all comparative algorithms given in Table 3.1. In view of the fact that the number of experts is small in comparison with T (for example, d = 5 as in the following experiments), the computation-time complexity of our Online DCA based algorithms is the same as that of other algorithms -the worst-case complexity of O(T ).

Numerical experiments

With the aim of evaluating the performance of our proposed algorithms, we conduct online binary classification tasks with expert advice. In order to construct the group of experts, we used five well-known online classification algorithms (d = 5) mentioned in Section 3.1: Perceptron [START_REF] Novikoff | On convergence proofs for perceptrons[END_REF][START_REF] Rosenblatt | The perceptron: a probabilistic model for information storage and organization in the brain[END_REF][START_REF] Van Der Malsburg | Frank rosenblatt: Principles of neurodynamics: Perceptrons and the theory of brain mechanisms[END_REF], ROMMA [START_REF] Li | The relaxed online maximum margin algorithm[END_REF], ALMA [START_REF] Gentile | A new approximate maximal margin classification algorithm[END_REF], PA [START_REF] Crammer | Online passive-aggressive algorithms[END_REF] and OGD [START_REF] Zinkevich | Online convex programming and generalized infinitesimal gradient ascent[END_REF]. As mentioned previously, we compare our algorithms, namely ODCA-SG and ODCA-ESG, with the WM algorithm and two online convex algorithms, namely OGD and NEG. We tested on a variety of benchmark datasets from UCI Machine Learning Repository 1 and LIBSVM website 2 . The datasets used in our experiments cover many areas (e.g. social sciences, biology, physics, life sciences), which is shown in Table 3.2. Set up experiments: All algorithms were implemented in Visual C++ version 11.0 and run on a PC Intel(R) Core(TM) i5-3470 CPU 3.20GHz of 8GB RAM. All experts are the first-order learning algorithms for large-scale online classification tasks as surveyed in [START_REF] Hoi | Libol: A library for online learning algorithms[END_REF] and implemented in MATLAB R2013b. The open source MAT-LAB package for the expert algorithms is available in [START_REF] Hoi | Libol: A library for online learning algorithms[END_REF]. In our experiment, each dataset is randomly divided into two sets as follows. A so-called training set including 20% of the whole data is used by the system of experts to learn linear classifiers u i (i = 1, . . . , d), while a so-called test set consisting of the remaining dataset is adopted by all algorithms to make the predictions. The initial prediction vector w 1 ∈ S is set to (1/d, . . . , 1/d) . The positive index ρ is set to 0.5. The projection algorithm Proj S (•) is described in [START_REF] Wang | Projection onto the probability simplex: An efficient algorithm with a simple proof, and an application[END_REF]. We are interested in the following criteria to evaluate the effectiveness of the proposed algorithms: the percentage of regret (denoted by %regret in %) and CPU time (in seconds). Specifically, the %regret is computed as

%regret = Regret d T .100. (3.15)
where Regret d is defined as (3.2) and T is the number of steps (corresponding to the number of instances in the test set). This value means the distance between the number of mistakes of the algorithm and that of the best expert. The smaller the %regret is, the better the algorithm would be. For a fair comparison, we follow a so-called validation procedure on the test set which is described as in [START_REF] Hoi | Libol: A library for online learning algorithms[END_REF] so as to choose the best parameters for different algorithms. In particular, we first perform each algorithm by running over one random permutation of the dataset with the different parameter values and then take the value corresponding to the smallest mistake rate. The ranges of parameters for the expert algorithms and existing algorithms are completely described in [START_REF] Hoi | Libol: A library for online learning algorithms[END_REF] while as for our algorithms, the parameters τ 1 , τ 2 , τ 3 , η are chosen as follows. The parameters τ 1 , τ 2 , τ 3 are set to the same positive tuning parameter τ for all t (τ < ρ) and the constant step size η for ODCA-SG (resp. ODCA-ESG) is

η = C/ √ T (resp. η = C log(d)/ √ T )
where τ and C are searched from the range of {0.00, 0.02, . . . , 0.48} and {2 -4 , 2 -3 , . . . , 2 4 } respectively. The best values of τ of our algorithms on each dataset are shown in Table 3.3. After the validation procedure, each algorithm is conducted over 20 runs of different random permutations for each test set with the best parameters chosen. The average results and their standard deviation over these 20 runs of all algorithms are reported in Table 3.4 and Table 3.5. Figure 3.1 shows the number of mistakes of all algorithms along online process in the validation procedure on several notable datasets.

Comments on numerical results:

We observe from the numerical results of all algorithms that:

• In terms of %regret, ODCA-SG and ODCA-ESG are the most efficient. In particular, ODCA-SG is the best with the first best on 9/11 datasets and the second best on 1/11 datasets -the gain varies from 0.003% to 8.977% -especially, for the large datasets D2 (271617 instances) and D4 (581012 instances) (see Table 3.4). The second is ODCA-ESG which outperforms the existing algorithms on 9/11 datasets (6 for the first best and 3 for the second best) -the gain varies 0.005% to 8.415%. However, the %regret of ODCA-ESG is fairly comparable to that of ODCA-SG on 8/11 datasets with the difference from 0% to 0.076%. In addition, the values of %regret of our algorithms are actually small and stable (with the small standard deviation) on most datasets.

In fact, there are several datasets (e.g. D2, D5, D9, D10) on which ODCA-SG and ODCA-ESG provided the negative value of %regret. That is to say, our algorithms can make predictions even better than the best experts. Additionally, Figure 3.1 indicates that the number of mistakes of our algorithms is less than that of other algorithms along the online process, particularly in large datasets.

• Concerning CPU time: all algorithms run very fast. From Table 3 Online DCA for Prediction with Expert Advice discussed in Remark 3.3 and the fact that the learner only updates the predictions when one makes a false mistake on the previous round. Therefore, the more the algorithm makes the good predictions, the faster it would be.

• As for the value of τ , from Table 3.3, the best value for ODCA-SG and ODCA-ESG is almost near to ρ (ρ = 0.5) in the procedure of parameter validation. This is an interesting illustration for the particular property of the DC function as noticed in Remark 3.1 and Remark 3.2.

In summary, our Online DCA based algorithms work well in terms of the quality and the rapidity on these test problems. Specially, ODCA-SG seems to be preferred since it realizes the best trade-off between these two criteria.

Conclusions

In this chapter, we have intensively investigated an Online DC programming and Online DCA approach for developing a class of online learning techniques, namely prediction with expert advice. In particular, we have exploited DC approximation functions for the nonsmooth, nonconvex 0-1 loss function on each online step. The resulting optimization problems are solved by Online DCA. Solving each convex subproblem in Online DCA by approximating by one iteration of two variants of subgradient method, we have developed two particular Online DCA based schemes, namely ODCA-SG and ODCA-ESG. We have studied the analysis of both schemes in terms of regret, which enjoys the sublinear/logarithmic regret. As an application, we have derived two algorithms based on ODCA-SG and ODCA-ESG for OBLC. Numerical results on various benchmark classification datasets show the efficiency of our approach when comparing with the well-known existing algorithms.

Chapter 4 Reinforcement Learning: Introduction and Related Works

Abstract: In this chapter, we briefly introduce reinforcement learning which is a general class of machine learning techniques for dealing with sequential decision problems. The goal of reinforcement learning is to estimate the optimal learning policy in a dynamic environment typically formulated as a Markov decision process (with an incomplete model). It is wellknown that one can tackle this task through the problem of finding the zero of the so-called optimal Bellman residual, a classical concept of dynamic programming. The background of reinforcement learning and its related works which we concern in this dissertation are also presented.

Reinforcement Learning: Introduction and Related Works

Reinforcement learning (RL) is a general class of machine learning techniques where an agent must learn behavior through trial-and-error interactions with a dynamic environment which is typically modeled as a Markov Decision Process (MDP). RL is an intersection of many active research subfields including machine learning, statistical learning, behavior optimization, robotics, etc (see e.g. [START_REF]Reinforcement Learning: Stateof-the-Art, volume 12 of Adaptation, Learning, and Optimization[END_REF]). Currently, the applications of RL are very wide in various areas such as manufacturing, management, transportation.

The goal of RL is to estimate the optimal policy of an MDP without knowing its complete model. From the last two decades, the power of RL in dealing with the curse of dimensionality and the curse of modeling on large-scale and complex problems of dynamic optimization, in particular the MDP problem and its variants, has been more and more confirmed by various works. This success of RL is due to its strong mathematical tools in the principles of Dynamic Programming (DP). This dissertation contributes to enrich these tools by investigating a unified DC (Difference of Convex functions) programming framework and efficient DCA based approach for solving the problems of RL, in particular, in batch mode (i.e. a fixed set of learning experience is given a priori) or online mode (i.e. the learning experience is collected piece-by-piece through the interaction with the environment).

Background and related works

An MDP [START_REF] Bellman | A Markovian decision process[END_REF][START_REF] Bertsekas | Dynamic Programming: Deterministic and Stochastic Models[END_REF][START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF] can be described by the 5-tuple < S, A, R, P, γ > where S is a state space, A is an action space, R : S × A → R is a reward function on stateaction transitions, P : S × A × S → [0, 1] is a state transition probability function in which P(s |s, a) represents the probability of transition from s ∈ S to s ∈ S upon taking action a ∈ A and γ ∈ (0, 1) is a discount factor. To avoid the complication of systems having continuous state spaces and continuous action spaces, a finite MDP is often considered, i.e. the state space and the action space are finite, specifically S = {s i } i=1,...,N S , A = {a i } i=1,...,N A . A policy π of an MDP is called deterministic stationary if it specifies the same action each time a state is visited, i.e. π is a mapping from S to A (for short, π ∈ A S ) where π(s) is the action the agent takes in the state s. Broadly, the aim of the agent learning in this environment is to maximize the cumulative reward it receives (which can be the total reward, the average reward, or the total discounted reward).

RL methods [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF] are employed to address MDPs without knowledge of the perfect dynamic (a model of P or R). Hence, value functions are usually used in RL to measure how good is each state and/or action, and an RL agent may include one or more of three components: the policy (agent's behavior), the value function (which can be the state value function and/or the state-action value function), and the model (agent's representation of the environment). Generally, RL techniques can be divided into three categories: the value-based approach estimates first the optimal value function (which is the maximum value achievable under any policy) and then determines an optimal policy with relative easiness, the policy-based approach directly searches for the optimal policy, and the model-based approach builds a model of the environment. For an overview and a tutorial survey on RL, refer to [START_REF] Bertsekas | Neuro-Dynamic Programming[END_REF][START_REF] Buşoniu | Reinforcement Learning and Dynamic Programming Using Function Approximators[END_REF][START_REF] Gosavi | Reinforcement learning: A tutorial survey and recent advances[END_REF][START_REF] Sigaud | Markov Decision Processes in Artificial Intelligence[END_REF][START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF][START_REF] Szepesvári | Algorithms for Reinforcement Learning[END_REF].

In this work we focus on RL using the value function based approach to address the above finite MDP in an infinite time horizon, discounted reward setting. More precisely, given a fixed set of samples generated from state transitions in a finite MDP, the goal is to search for an optimal policy π * that maximizes a discounted, infinitehorizon optimality criterion (see e.g. [START_REF] Sigaud | Markov Decision Processes in Artificial Intelligence[END_REF][START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]):

π * ∈ arg max π∈A S E π ∞ t=0 γ t R(s t , π(s t )) ,
where E π [•] is the expectation given that the agent follows the policy π. Instead of directly finding π * over the space A S , we estimate its quality via the state-action value function Q π , viewed as a link between that criterion and π.

For a given policy π, the state-action value function under policy π is defined as

Q π : S × A → R, Q π (s, a) := E π ∞ t=0 γ t R(s t , π(s t )) s 0 = s, a 0 = a . (4.1)
The optimal state-action value function, denoted by Q * , is given by

Q * (s, a) := max Q π (s, a) : π ∈ A S . (4.2)
When Q * is obtained, the optimal policy is computed as π * (s) = arg max {Q * (s, a) : a ∈ A} .

Given a function Q ∈ R S×A , a policy π ∈ A S satisfying π(s) ∈ arg max a∈A Q(s, a), ∀s ∈ S is said to be greedy with respect to Q. Specially, π * is greedy with respect to Q * , say

Q * = Q π * .
RL methods are employed to solve two basic problems: the prediction problem and the control problem. The goal of the prediction problem is to compute the value function for an arbitrary given policy (or evaluate the given policy), while the control problem aims to estimate the optimal value function (or optimal policy) directly. In both problems, most of the RL techniques often improve the policy in different ways to realize the best trade-off between exploitation and exploration [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF] (for example, ε-greedy methods [126], softmax methods [START_REF] Luce | Individual Choice Behavior: A theoretical analysis[END_REF]). Our works concern RL techniques for control problems with ε-greedy method. [START_REF] Williams | Tight performance bounds on greedy policies based on imperfect value functions[END_REF]). Thus, to find the zero of BR (resp. OBR), the idea of minimizing a norm ( ∞ / p , p ≥ 1) of BR (resp. OBR) is natural. Moreover, as mentioned in [START_REF] Williams | Tight performance bounds on greedy policies based on imperfect value functions[END_REF], there is a very natural correspondence between Temporal Difference (TD) errors in TD approach [START_REF] Sutton | Learning to predict by the methods of temporal differences[END_REF] and the BR measures, and similarly, the individual components of the BR across the stateaction function Q are very closely related to what Q-learning [126] tries to reduce toward zero. In the earlier step of RL, the ∞ -norm and 2 -norm were often used, and later, the weighted p -norm was investigated.

= B * Q. The BR is then defined by B π Q -Q while the OBR is B * Q -Q (see e.g.
BR minimization approach is intensively studied in RL literature on both theoretical and algorithmic point of views (e.g. [3,5,[START_REF] Geist | Algorithmic Survey of Parametric Value Function Approximation[END_REF][START_REF] Maillard | Finite sample analysis of Bellman residual minimization[END_REF][START_REF] Munos | Performance Bounds in L p norm for Approximate Value Iteration[END_REF][START_REF] Scherrer | Should one compute the Temporal Difference fix point or minimize the Bellman residual? The unified oblique projection view[END_REF][START_REF] Schweitzer | Generalized polynomial approximations in markovian decision processes[END_REF][START_REF] Sigaud | Markov Decision Processes in Artificial Intelligence[END_REF][START_REF] Williams | Tight performance bounds on greedy policies based on imperfect value functions[END_REF]). Tight performance bounds on greedy policies were proved in [START_REF] Williams | Tight performance bounds on greedy policies based on imperfect value functions[END_REF] for ∞ -norm and in e.g. [START_REF] Maillard | Finite sample analysis of Bellman residual minimization[END_REF][START_REF] Munos | Performance Bounds in L p norm for Approximate Value Iteration[END_REF] for p -norm and weighted p -norm. On the algorithmic level, works in this direction can be divided into two groups. The first (OBR-based approach) minimizes directly a norm of OBR while the second (BR-based approach) works by alternating between the evaluation of the policy through BR minimization and the policy improvement. In fact, the function OBR is more complex than BR due to the max operator over the space A, and, as indicated in [START_REF] Sigaud | Markov Decision Processes in Artificial Intelligence[END_REF], minimizing a norm of the empirical OBR over the space of approximation functions is difficult, even in the case of linear approximation, since the resulting optimization problems are nonconvex, nonsmooth and thus solving them by global approaches is very hard in large-scale settings. Hence, the OBRbased approach was less worked than the BR-based approach, there exist some local optimization techniques, for example, neural networks for 2 -norm of OBR, although global convergence is not guaranteed.

In the literature, Baird [5] proposed residual algorithms which can be regarded as a weighted average of a direct algorithm with a residual gradient algorithm. The update step in the direct algorithm is the same as the TD(0) algorithm, whereas the residual gradient algorithm updates the weight in gradient descent on the 2 -norm of BR and OBR. Direct algorithms run fast but unstably, while residual gradient algorithms enjoy guaranteed convergence to a local optimum but slow. This residual algorithm takes advantage of the benefits of both direct and residual gradient algorithms in terms of convergence and its speed when using function approximator. Moreover, Geist and Pietquin [START_REF] Geist | Algorithmic Survey of Parametric Value Function Approximation[END_REF] have indicated that the RL methods via value-function approximation can be divided into three main approaches: bootstrapping, residual and projected fixed-point. Among them, residual approaches with gradient descent, the closest to our works, are represented in more detail for both BR and OBR in the state-action value function as the main works in [5]. However, as indicated in [START_REF] Geist | Algorithmic Survey of Parametric Value Function Approximation[END_REF], the computation of the gradient of sampled OBR functions is not straightforward since these functions are nonsmooth and nonconvex in the function-approximation space. Our works focus on developing RL techniques in both batch mode and online mode by addressing the different norms of empirical OBR via value-function linear approximation. Actually, most works in RL are considered in online mode since the sample is collected via the exploitation and exploration procedure with the environment after each step of updating the policy, meanwhile in batch mode, the set of samples is given a priori (see e.g. [START_REF] Ernst | Tree-based batch mode reinforcement learning[END_REF][START_REF] Lagoudakis | Least-squares policy iteration[END_REF][START_REF] Lange | Batch Reinforcement Learning[END_REF]). Thus, for simplicity, in the rest of this part, we mention the RL problems as those in online mode, and abbreviate the RL problems in batch mode by Batch RL.

In the context of Batch RL, there are some popular algorithms to deal with these problems such as Fitted Q-Iteration (FQI) [START_REF] Ernst | Tree-based batch mode reinforcement learning[END_REF][START_REF] Van Otterlo | Reinforcement Learning and Markov Decision Processes[END_REF], Least Square Policy Iteration (LSPI) [START_REF] Sigaud | Markov Decision Processes in Artificial Intelligence[END_REF][START_REF] Lagoudakis | Least-squares policy iteration[END_REF][START_REF] Van Otterlo | Reinforcement Learning and Markov Decision Processes[END_REF] which are developed from the standard algorithms of DP such as Approximate Value Iteration (AVI) [START_REF] Bellman | Functional approximation and dynamic programming[END_REF][START_REF] Samuel | Some studies in machine learning using the game of checkers[END_REF], Approximate Policy Iteration (API) [START_REF] Bertsekas | Neuro-Dynamic Programming[END_REF][START_REF] Sigaud | Markov Decision Processes in Artificial Intelligence[END_REF] respectively. These Batch RL algorithms are achieved by combining RL and DP methods with approximated function. While FQI can be seen as approximate value iteration for the state-action value functions, LSPI combines state-action value function approximation with linear representation and approximate policy iteration.

Reinforcement Learning: Introduction and Related Works

Motivation

It is worth remembering that the problem of minimizing a norm of the empirical OBR is nonconvex, nonsmooth. In the literature, there are the rare algorithms for nonconvex, nonsmooth programming framework. One of them is DC (Difference of Convex functions) programming and DCA (DC Algorithm) (see [START_REF] Thi | The DC (Difference of Convex Functions) Programming and DCA Revisited with DC Models of Real World Nonconvex Optimization Problems[END_REF][START_REF] Pham Dinh | Algorithms for Solving a Class of Nonconvex Optimization Problems[END_REF][START_REF] Pham Dinh | Convex analysis approach to DC programming: theory, algorithms and applications[END_REF][START_REF] Pham Dinh | DC optimization algorithms for solving the trust region subproblem[END_REF][START_REF] Pham Dinh | Recent Advances in DC Programming and DCA[END_REF] and the references in [START_REF] Thi | DC Programming and DCA[END_REF]) which are powerful, nonsmooth, nonconvex optimization tools. In this dissertation, we will tackle this resulting nonconvex problem by DC programming and DCA. Moreover, our work is also motivated by the fact that DCA has been successfully applied to many (smooth or nonsmooth) large-scale nonconvex programs in various domains of applied sciences, in particular in machine learning (see the list of references in [START_REF] Thi | DC Programming and DCA[END_REF]). As for the RL problems, the Online DC programming and Online DCA based approach is also investigated recently.

Recall briefly that DCA aims to solve a standard DC program that consists of minimizing a DC function f = g -h (with g and h being convex functions) over a convex set or on the whole space. Here g -h is called a DC decomposition of f , while the convex functions g and h are DC components of f . The main idea of DCA is approximating the second DC component h by its affine minorant and then solving the resulting convex subproblem at each iteration. The construction of DCA is relied on the convex DC components g and h but not the DC function f itself. Hence, for a DC program, each DC decomposition corresponds to a different version of DCA. Since a DC function f has an infinite number of DC decompositions which have crucial impacts on the qualities (speed of convergence, robustness, efficiency, globality of computed solutions,. . . ) of DCA, the search for a "good"DC decomposition is vital from the algorithmic point of views and is the most important key issue while designing DCA for a practical problem. Furthermore, how to efficiently solve convex subproblems in DCA is a crucial question as well (evidently the form of convex subproblems depends on the DC decomposition). In fact, although convex programming has been studied for about a century, an increasing amount of effort has been put recently into developing fast and scalable algorithms to deal with large-scale problems. Moreover, as DCA is a local approach, finding a good starting point is also an important matter to be studied.

In the next chapters, we will present how to develop efficient DCA based algorithms for large-scale Batch RL problems and design Online DCA based algorithms for RL problems with online mode.

Chapter 5

A unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning 1

Abstract: In this chapter, we investigate a powerful nonconvex optimization approach based on DC (Difference of Convex functions) programming and DCA (DC Algorithm) for reinforcement learning problems in batch mode (Batch RL) (i.e., a fixed set of learning experience is given a priori). These problems can be tackled through the problem of finding the zero of the so-called Optimal Bellman Residual. However, there exist so far a few works in the literature following this direction, knowing that it usually leads to a nonconvex optimization problem which is very hard to solve exactly. In this chapter we consider four optimization formulations of this problem that are the minimization of the p -norm with p ∈ {1, 2, +∞} of optimal Bellman residual and the new concave minimization to design these large-scale problems using linear value-function approximation. They are all formulated as DC programs for which four attractive DCA schemes are developed. Exploiting the special structure of the empirical optimal Bellman residual with linear value-function approximation we carefully address the key issues of DCA, namely the effect of DC decompositions, the efficiency of solution methods for the resulting convex subproblem, and the search for good starting points, when designing the four DCA based algorithms. Numerical experiments on various examples of the two benchmarks of Markov decision process problems -Garnet and Gridworld problems, show the efficiency of our approaches in comparison with two existing DCA based algorithms and two state-of-the-art reinforcement learning algorithms.

1. The material of this chapter is based on the following work: [1]. Hoai An Le Thi, Vinh Thanh Ho, Tao Pham Dinh. A unified DC Programming Framework and Efficient DCA based Approaches for Large Scale Batch Reinforcement Learning. Submitted to the Journal of Global Optimization.

A unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning

Our contributions

In this chapter, we investigate a unified DC (Difference of Convex functions) programming based approach in the context of Batch Reinforcement Learning (Batch RL), i.e. a fixed set of learning experience is given a priori. In particular, as mentioned in Chapter 4, we address the problem of finding the zero of the empirical Optimal Bellman Residual (OBR) via linear approximation under the different norms.

DC programming and DCA were studied in [START_REF] Piot | Difference of Convex Functions Programming for Reinforcement Learning[END_REF] for minimizing the p -norm of the empirical OBR via linear approximation (p ≥ 1), in which two DCA schemes, namely DCA 1 and DCA 2 , were developed for the case p = 1 and p = 2 respectively. However, interesting perspectives leaved by the authors of [START_REF] Piot | Difference of Convex Functions Programming for Reinforcement Learning[END_REF] are the choice of DC decompositions and the solution methods for convex subproblems. In this work, these key issues of DC programming and DCA are addressed in deeper ways to design new DCA based algorithms (for the same optimization problems considered in [START_REF] Piot | Difference of Convex Functions Programming for Reinforcement Learning[END_REF]) which are much more efficient than the DCA 1 and DCA 2 on both quality and rapidity. That is one among several contributions of this work.

Our contributions are multiple.

Firstly, we investigate more attractive DCA based algorithms for minimizing the pnorm of the empirical OBR in case p = 1 and p = 2. Exploiting the effect of DC decomposition and the special structure of the resulting convex subproblems, we propose two DCA schemes, named 1 -DCA and 2 -DCA in case p = 1 and p = 2 respectively, which require solving one linear program ( 1 -DCA) and one convex quadratic program ( 2 -DCA) at each iteration. It turns out that our algorithm 1 -DCA (resp. 2 -DCA) outperforms DCA 1 (resp. DCA 2 ) proposed in [START_REF] Piot | Difference of Convex Functions Programming for Reinforcement Learning[END_REF] in terms of both quality and rapidity. For medium MDPs having 100 states, the ratio of gain in terms of rapidity of 1 -DCA versus DCA 1 (resp. 2 -DCA versus DCA 2 ) is up to 11.5 (resp. 4.17) times, while the ratio of gain in terms of quality of 1 -DCA versus DCA 1 (resp. 2 -DCA versus DCA 2 ) is up to 70.2% (resp. 84%). Further, the gain increases considerably when the number of states increases.

Secondly, we consider the ∞ -norm of the empirical OBR and develop a DCA scheme ( ∞ -DCA) for solving this problem. The ∞ -DCA enjoys interesting convergence properties thanks to the fact that both DC components are polyhedral convex functions, and it requires also one linear program at each iteration. Numerical experiments show that the ∞ -DCA is slightly better than the existing DCA 1 and DCA 2 -the ratio of gain in terms of quality is up to 40.1%.

Thirdly, we propose a new formulation of the OBR without using the ∞ / p -norm.

We consider the problem as finding the zero of a function (the empirical OBR) which has a very special structure and highlight various possible formulations, the link between them and finally introduce a constrained optimization problem enjoying several advantages. This new formulation is in fact a concave minimization problem under linear constraints (which can also be interpreted as minimizing the restricted 1 -norm of the empirical OBR under a polytope). Fortunately, the addition of constraints A unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning 103

does not make the new problem more difficult than the above formulations which are unconstrained minimization problems. On contrary, the new DCA (named cc-DCA), consisting of solving at each iteration one linear program (whose the number of constraints is, interestingly, smaller than that in linear subproblems of 1 -DCA and ∞ -DCA), is more efficient than the existing DCA 1 and DCA 2 in terms of both quality and rapidity. The ratio of gain is up to 131 times on rapidity and 68.3% on quality.

Throughout the chapter, we carefully studied the key issues of DCA, namely the effect of DC decompositions, the efficiency of solution methods for the resulting convex subproblem, and the search for good starting points, when developing the four DCA based algorithms. We are particularly interested in specific DC programs, called polyhedral DC programs, where either g or h is polyhedral convex (i.e. the maximum of a finite family of affine functions), because that DC programs have interesting optimality and convergence properties. More precisely, we exploit the special structure of each objective function in an efficient way to propose a suitable DC decomposition. The 1 -norm, ∞ -norm and concave formulations of OBR are polyhedral DC programs where both DC components are polyhedral convex functions. The corresponding DCA enjoys, not only the finite convergence property, but also the local optimality of solutions. Furthermore, we study in a deep way the solution methods for the convex subproblems in DCA by deriving benefit from the specific structure of the convex subproblem in each DCA scheme. Actually, our techniques reduce to solving one linear program in 1 -DCA, ∞ -DCA, and cc-DCA, and one convex quadratic program in 2 -DCA. Thus, these problems can be efficiently solved by standard softwares. On another hand, we investigate an efficient strategy for finding good starting points for all our DCA based algorithms.

Finally, we provide several numerical experiments of the proposed algorithms on two benchmarks -the stationary Garnet problem and the Gridworld problem, compared with two existing DCA schemes and two state-of-the-art RL algorithms.

The rest of the chapter is organized as follows. Different optimization formulations of the empirical OBR are described in Section 5.2. Section 5.3 first presents a short introduction of DC programming and DCA and then shows how to apply DC programming and DCA to these optimization problems. Section 5.4 reports the numerical results on several test problems. Finally, Section 5.5 concludes the chapter.

Optimization formulations of the empirical OBR via linear function approximation

Linear function approximators are the most commonly used in function approximation techniques (for instance, SARSA, Q-learning, and Least-Squares Policy Iteration LSPI). A tutorial on linear function approximators for DP and RL was recently given in [START_REF] Geramifard | A tutorial on linear function approximators for dynamic programming and reinforcement learning[END_REF]. The basic idea is that, the full set of states is projected into a lower dimensional space where the value function is represented as a linear function. Let F denote the space of approximation functions, say

F := Q θ (s, a) = θ, φ(s, a) |θ ∈ R d .

p -norm formulation (p ≥ 1)

The p -norm (p ≥ 1) of OBR with the probability distribution µ is defined as (see e.g. [START_REF] Munos | Performance Bounds in L p norm for Approximate Value Iteration[END_REF]) .

J p,µ (Q) := ||B * Q -Q|| p,
Performance bounds of a policy greedy with respect to the state value function V in terms of the p -norm (1 ≤ p ≤ ∞) of its OBR were given in [START_REF] Munos | Performance Bounds in L p norm for Approximate Value Iteration[END_REF], a similar result was stated later in [START_REF] Piot | Difference of Convex Functions Programming for Reinforcement Learning[END_REF] when p ≥ 1 for the state-action value function Q. It is well known that Q * is the optimal solution to J p,µ and, obviously, if the minimum value of J p,µ is near zero, the corresponding greedy policy is close to the optimal policy.

With the linear approximation, the p -norm (p ≥ 1) of the empirical OBR over the space of linear approximation functions F is

J p,µ (Q θ ) = ||B * Q θ -Q θ || p,µ . (5.1)
In batch mode (Batch RL), estimating the optimal state-action value function rests on a fixed set of transition samples, i.e. the so-called sampling-based version of J p,µ in [START_REF] Munos | Performance Bounds in L p norm for Approximate Value Iteration[END_REF] is used. Specifically, one first selects N samples (S i , A i , S i ) i=1,...,N , then considers a nonbiased estimation of B * Q(S i , A i ) :

B * Q(S i , A i ) = R(S i , A i ) + γ max a ∈A Q(S i , a ) (5.2)
and finally minimizes the empirical p -norm of OBR [START_REF] Munos | Performance Bounds in L p norm for Approximate Value Iteration[END_REF][START_REF] Piot | Difference of Convex Functions Programming for Reinforcement Learning[END_REF]:

J p,µ N (Q) := || B * Q -Q|| p,µ N , (5.3)
where µ N is the empirical distribution given by µ N (s, a)

= 1 N N i=1 1 {(S i ,A i )=(s,a)} (s, a) and 1 X is the indicator function of X, i.e. 1 X (x) = 1 if x ∈ X, 0 otherwise.
The bound error between the minimized J p,µ and J p,µ N in terms of µ N and the capacity (the complexity) measure of the approximation function space (the Vapnik A unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning 105 consistency) was established in Statistical learning theory [START_REF] Vapnik | Statistical learning theory[END_REF]. In particular, the Vapnik-consistency of (5.1) with respect to µ N was deduced in [START_REF] Piot | Difference of Convex Functions Programming for Reinforcement Learning[END_REF] from the results of Theorem 5.3 of [START_REF] Vapnik | Statistical learning theory[END_REF].

Using this sampling-based technique with linear approximation operator, we have to minimize the empirical p -norm (p ≥ 1) of the empirical OBR that is

N i=1 B * Q θ (S i , A i ) -Q θ (S i , A i ) p 1 p
.

Finally the p -norm formulation of the empirical OBR via linear approximation takes the form min

θ∈R d F p,µ N (θ) := N i=1 R(S i , A i ) + γ max a ∈A θ, φ(S i , a ) -θ, φ(S i , A i ) p . (5.4)
Let f i , for i = 1, . . . , N , denote the real function on R d defined as

f i (θ) := R(S i , A i ) + γ max a ∈A θ, φ(S i , a ) -θ, φ(S i , A i ) . (5.5)
Clearly, f i is the maximal function of a finite family of affine functions and can be expressed as (remember that A is a finite space having N A elements)

f i (θ) := max j=1,...,N A A (i,j) , θ + b (i) , (5.6)
where A (i,j) := γφ(S i , a j ) -φ(S i , A i ), b (i) := R(S i , A i ), j = 1, . . . , N A .

Specifically, the 1 -norm optimization formulation is expressed as follows

min F 1 (θ) = N i=1 |f i (θ)| : θ ∈ R d , (5.7)
and the 2 -norm optimization formulation takes the form

min F 2 (θ) := N i=1 [f i (θ)] 2 : θ ∈ R d . (5.8)
As each f i is a polyhedral convex function, it is obvious that F 1 and F 2 are DC functions. In [START_REF] Piot | Difference of Convex Functions Programming for Reinforcement Learning[END_REF], the authors highlighted one DC formulation for each of above DC programs (5.7), (5.8) and proposed two DCA schemes for solving them. In Section 5.3, we will propose more attractive DCA based algorithms for these same problems.

∞ -norm formulation

With the aim of intensively studying DCA based approaches for the empirical OBR, we consider the ∞ -norm formulation of OBR, which is defined by

J ∞ (Q) = ||B * Q -Q|| ∞ , (5.9)
A unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning

where ||Q|| ∞ = max (s,a)∈S×A |Q(s, a)|. Evidently, the optimal state-action value function Q * is the optimal solution to (5.9).

Similarly to p -norm, after collecting N transition samples (S i , A i , S i ) and approximating the state-action value function Q ∈ R S×A by the weight vector θ ∈ R d on the space F, we obtain the ∞ -norm optimization formulation of the empirical OBR on F as follows min

θ∈R d F ∞ (θ) := max i=1,...,N B * Q θ (S i , A i ) -Q θ (S i , A i ) = max i=1,...,N |f i (θ)| . (5.10)
It is worth noting that the ∞ -norm in Q is "equivalent" to the p -norm on the finite state-action space S × A in the sense that for all Q ∈ R S×A , we have

C 1 p ||Q|| ∞ ≤ ||Q|| µ,p ≤ ||Q|| ∞ ,
where C = min{µ(s, a) : µ(s, a) > 0, ∀(s, a) ∈ S ×A}. Thus, the bound of (5.1) can be derived from the value of (5.9) and moreover the Vapnik-consistency for the ∞ -norm problem (5.9) can be guaranteed.

We will see later in Section 5.3 that (5.10) is also nonconvex, nonsmooth but a DC program.

New formulation: concave minimization under linear constraints

Recall that the main goal of MDP problems is to find the optimal state-action value function Q * such that B * Q * (s, a) -Q * (s, a) = 0 for all (s, a) ∈ S × A, which implies

(s,a) [B * Q * (s, a) -Q * (s, a)] = 0.
We are then suggested to consider the following optimization problem:

0 = min Q∈R S×A    m(Q) := - (s,a) [B * Q(s, a) -Q(s, a)] : B * Q(s, a) -Q(s, a) ≤ 0, ∀(s, a)    .
(5.11)

The following result (whose proof is straightforward) justifies our formulation.

Proposition 5.1. The optimal state-action value function Q * is a unique optimal solution to (5.11) with the optimal objective value being zero.

After collecting N transition samples (S i , A i , S i ) and approximating the state-action value function Q ∈ R S×A by the weight vector θ ∈ R d on the space of approximation function F, the problem (5.11) becomes 0 = min

θ∈R d F cc (θ) := - N i=1 f i (θ) : f i (θ) ≤ 0, ∀i = 1, . . . , N . (5.12)
A unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning 107 This problem can be equivalently reformulated as 0 = min

θ∈R d F cc (θ) := - N i=1 f i (θ) : f + i (θ) ≤ 0, ∀i = 1, . . . , N , (5.13)
where f + i is the function defined by

f + i (θ) := max {0, f i (θ)} .
As each f i is a polyhedral convex function, the function F cc is concave, and the problems (5.12) and (5.13) are concave minimization problems under polyhedral convex constraints (as will be seen later, they are in fact polyhedral DC programs). Obviously, the exact penalty holds for (5.13) (see [START_REF] Thi | Exact penalty and error bounds in DC programming[END_REF]), i.e. there exists the parameter τ 0 > 0 such that for all τ ≥ τ 0 , the problem (5.13) is equivalent to

0 = min θ∈R d - N i=1 f i (θ) + τ f + max (θ) ,
where f max is the function defined by f max (θ):= max{f i (θ) : i = 1, . . . , N }.

In this work we focus on the problem (5.12) because it enjoys several advantages for DCA as will be shown in the next section. Let us denote C θ the feasible set of (5.12). According to (5.6) we have

C θ := θ ∈ R d : max j=1,...,N A A (i,j) , θ + b (i) ≤ 0, ∀i = 1, . . . , N = θ ∈ R d : A (i,j) , θ + b (i) ≤ 0, ∀i = 1, . . . , N, ∀j = 1, . . . , N A .
C θ is so a polyhedral convex set defined by N.N A linear constraints, and (5.12) can be now expressed as

0 = min F cc (θ) := - N i=1 f i (θ) : A (i,j) , θ + b (i) ≤ 0, ∀i = 1, . . . , N, ∀j = 1, . . . , N A (5.14)
which is a concave minimization problem under linear constraints.

Remark 5.1. As |f i (θ)| = -f i (θ) for all θ ∈ C θ , we have F 1 (θ) = F cc (θ) for all θ ∈ C θ .
Hence, the formulation (5.12) can be viewed as the 1 -norm formulation restricted under C θ , in case µ is the uniform distribution.

From Remark 5.1 and Theorem 2 in [START_REF] Piot | Difference of Convex Functions Programming for Reinforcement Learning[END_REF], we have the Vapnik-consistency for the formulation (5.12) as showing the following lemma.

Lemma 5.1. Let F = Q θ ∈ F : ||Q θ || ∞ ≤ ||R||∞ 1-γ , B * Q θ -Q θ ≤ 0 , η ∈ (0, 1
) and consider the finite deterministic MDP, with probability at least 1 -η, we have:

∀Q θ ∈ F, 1 N S .N A m(Q θ ) ≤ 1 N F cc (θ) + 2||R|| ∞ 1 -γ ε(N ),
where ε(N ) =

h ln 2N h + 1 + ln 4 η N and h = 2N A (d + 1).
A unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning

Proof. Based on the Vapnik-consistency for the formulation (5.1) from Theorem 2 in [START_REF] Piot | Difference of Convex Functions Programming for Reinforcement Learning[END_REF], this lemma is obtained in case p = 1, µ is the uniform distribution over the whole state-action space S × A and µ N is the empirical distribution with respect to N transition samples.

Remark 5.2. The problem (5.12) is actually a concave minimization with linear constraints for which the use of DCA is worthy. In fact, DCA applied on linear constrained concave minimization, with the natural DC decomposition, consists of solving one linear program at each iteration, and has finite convergence property. We will show in Section 5.3.5 that (5.12) is more advantageous than the previous formulations and the corresponding DCA based algorithm is very efficient in terms of rapidity.

We are going to present DC programming and DCA for solving the above problems (5.7), (5.8), (5.10) and (5.12).

Solution methods by DC programming and DCA

5.3.1 DCA for solving the 1 -norm problem (

Remember that the problem (5.7) has the form

min F 1 (θ) = N i=1 |f i (θ)| : θ ∈ R d , where f i (θ) := max j=1,...,N A θ, A (i,j) + b (i) .
As mentioned previously, the function f i is polyhedral convex. By taking full advantage of this nice property, we construct a DC decomposition of |f i | as follows:

|f i | = 2f + i -f i ,
where

f + i := max{0, f i } is polyhedral convex too.
Let G 1 and H 1 be the functions defined by

G 1 (θ) := N i=1 2f + i (θ) and H 1 (θ) := N i=1 f i (θ), (5.15)
they are both polyhedral convex functions. Clearly,

F 1 (θ) = G 1 (θ) -H 1 (θ),
therefore the problem (5.7) can be now written in the form

min G 1 (θ) -H 1 (θ) : θ ∈ R d , (5.16)
which is a polyhedral DC program where both DC components are polyhedral convex.

A unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning 109

According to the generic DCA scheme, DCA applied on (5.16) consists of, at each iteration k, computing one subgradient w k ∈ ∂H 1 (θ k ), and then solving the following convex program:

min N i=1 2f + i (θ) -w k , θ : θ ∈ R d .
(5.17)

Thanks to the polyhedral convexity of f i defined by (5.5), we can reformulate the last problem as a linear program of the form

           min N i=1 2t i -w k , θ , s.t. θ ∈ R d , t i ≥ 0, ∀i = 1, . . . , N, t i ≥ A (i,j) , θ + b (i) , ∀i = 1, . . . , N, ∀j = 1, . . . , N A .
(5.18)

Compute ∂H 1 : by the definition of H 1 and according to the rule of computing the subdifferential of a function being the maximum of a finite family of convex functions [START_REF] Rockafellar | Convex analysis[END_REF] we have

∂H 1 (θ) = N i=1 ∂f i (θ) = N i=1 ∂ max j=1,...,N A A (i,j) , θ + b (i) = N i=1 co A (i,j i ) : j i ∈ I i (θ)}, I i (θ) = argmax j=1,...,N A A (i,j) , θ . (5.19)
In particular, we can choose w k ∈ ∂H 1 (θ k ) as follows: for i = 1, . . . , N , let j i ∈ I i (θ k ), then

w k = N i=1 A (i,j i ) . (5.20)
Define t = (t 1 , . . . , t N ) ∈ R N . DCA applied to (5.7) can be summarized in Algorithm 5.1 ( 1 -DCA) below.

Algorithm 5.1 DCA for solving (5.16) ( 1 -DCA)

Initialization: Let ε be a sufficiently small positive number. Let

θ 0 ∈ R d . Set k = 0. repeat 1. Compute w k ∈ ∂H 1 (θ k ) using (5.20).
2. Solve the linear program (5.18) to obtain (θ k+1 , t k+1 ).

3.

k = k + 1. until |F 1 (θ k ) -F 1 (θ k-1 )| ≤ ε(|F 1 (θ k-1 )| + 1) or ||θ k -θ k-1 || 2 ≤ ε(1 + ||θ k-1 || 2 ).
According to the convergence properties of DCA for polyhedral DC programs in Section 1.1, we deduce the following interesting convergence properties of 1 -DCA.

A unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning Theorem 5.1. Convergence properties of 1 -DCA i) 1 -DCA generates the sequence {θ k } such that the sequence {F 1 (θ k )} is decreasing.

ii) The sequence {θ k } converges to a critical point θ * of (5.16) after a finite number of iterations.

iii) θ * is almost always a local minimizer to (5.16). In particular, if I i (θ * ) is a singleton for all i = 1, ..., N , then θ * is a local minimizer to (5.16).

Proof. (i) and (ii) are, respectively, direct consequences of the convergence properties of general DC programs and polyhedral DC programs.

(iii) Since H 1 is a polyhedral convex function, the necessary local optimality condition ∂H 1 (θ * ) ⊂ ∂G 1 (θ * ) is also sufficient. This inclusion holds when H 1 is differentiable at θ * . As a polyhedral convex function is almost always differentiable, we deduce that θ * is almost always a local minimizer to (5.16). In particular, from the computation of ∂H 1 we see that if I i (θ * ) is a singleton for all i = 1, ..., N then H 1 is differentiable. The proof is then complete.

Remark 5.3. Our 1 -DCA algorithm is much more advantageous than DCA 1 in [98]:

1 -DCA solves one linear program at each iteration while DCA 1 uses the iterative subgradient method to deal with nonsmooth convex subproblems.

5.3.2 DCA for solving the 2 -norm problem (

Recall that the 2 -norm problem (5.8) is of the form:

min θ∈R d F 2 (θ) := N i=1 [f i (θ)] 2 .
Similarly, we thoroughly exploit the convexity of the function f i to design DCA. For each i = 1, . . . , N , with a i ∈ dom ∂f i and b i ∈ ∂f i (a i ), we define the affine minorization of f i as follows

l i (θ) = f i (a i ) + θ -a i , b i . Let l -(θ) = max{0, -l(θ)}. For θ ∈ R d , we have f i (θ) = f i (θ) + l - i (θ) -l - i (θ), which implies [f i (θ)] 2 = 2 f i (θ) + l - i (θ) 2 + l - i (θ) 2 -f i (θ) + 2l - i (θ) 2 .
Hence, we derive the following DC decomposition of F 2 :

F 2 (θ) = G 2 (θ) -H 2 (θ), (5.21)
where

G 2 (θ) = N i=1 2 f i (θ) + l - i (θ) 2 + l - i (θ) 2 , H 2 (θ) = N i=1 f i (θ) + 2l - i (θ) 2 . (5.22)
It is evident that f i + l - i and l - i are nonnegative and convex on R d , so are G 2 and H 2 . Hence (5.8) is a DC program of the form

min{G 2 (θ) -H 2 (θ) : θ ∈ R d }.
(5.23)

A unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning 111 Applying DCA on (5.23) leads us to compute two sequences {w k } and {θ k } such that w k ∈ ∂H 2 (θ k ) and θ k+1 solves the following convex program of the form (P k ) min

N i=1 2 f i (θ) + l - i (θ) 2 + l - i (θ) 2 -w k , θ : θ ∈ R d .
The last problem can be reformulated as min

N i=1 (2τ 2 i ) + N i=1 (2t 2 i ) -w k , θ : t i ≥ l - i (θ), ∀i = 1, . . . , N, τ i ≥ t i + f i (θ), ∀i = 1, . . . , N which is in fact a quadratic program of the form                min N i=1 (2τ 2 i ) + N i=1 (2t 2 i ) -w k , θ , s.t. θ ∈ R d , t i ≥ 0, ∀i = 1, . . . , N, t i ≥ -l i (θ), ∀i = 1, . . . , N, τ i ≥ t i + θ, A (i,j) + b (i) , ∀i = 1, . . . , N, ∀j = 1, . . . , N A .
(5.24)

Compute ∂H 2 : from the definition of H 2 and l - i , we have

∂H 2 (θ) = N i=1 ∂ f i (θ) + 2l - i (θ) 2 ,
and for i = 1, . . . , N,

∂l - i (θ) =    {0} if l i (θ) > 0, 0, -b i if l i (θ) = 0, {-b i } if l i (θ) < 0.
Here [0, -b i ] is the line segment between 0 and -b i . Hence, we can take a subgradient w k ∈ ∂H 2 (θ k ) as follows

w k = 2 N i=1 f i (θ k ) + 2l - i (θ k ) A (i,j i ) -b i .1 {l i (θ)<0} (θ k ) , (5.25)
where j i ∈ I i (θ k ), ∀i = 1, . . . , N and I i (θ) is defined in (5.19).

Define τ = (τ 1 , . . . , τ N ) ∈ R N . Finally, DCA applied to (5.23) can be described as Algorithm 5.2 ( 2 -DCA).

As direct consequences of convergence properties of DCA in Section 1.1, we have Theorem 5.2. Convergence properties of 2 -DCA i) 2 -DCA generates the sequence {θ k } such that the sequence {F 2 (θ k )} is decreasing.

ii) If the optimal value of problem (5.23) is finite then the sequence {θ k } converges to θ * that is a critical point of (5.23).

A unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning Algorithm 5.2 DCA for solving (5.23) ( 2 -DCA)

Initialization: Let ε be a sufficiently small positive number. Let θ 0 ∈ R d . Set k = 0. repeat 1. Compute w k ∈ ∂H 2 (θ k ) using (5.25).

2. Solve the quadratic program (5.24) to obtain (θ k+1 , τ k+1 , t k+1 ).

3.

k = k + 1. until |F 2 (θ k ) -F 2 (θ k-1 )| ≤ ε(|F 2 (θ k-1 )| + 1) or ||θ k -θ k-1 || 2 ≤ ε(1 + ||θ k-1 || 2 ).
Remark 5.4. The DC decomposition (5.21)-(5.22) enjoys several advantages: firstly, the affine minorization l i of f i is easily computed, therefore the DC components G 2 and H 2 are defined in a simple way. Such a DC decomposition is possible thanks to the convexity of f i . Secondly, the function l i can be updated at each iteration to better approximate f i , which means we can use a local DC decomposition of F 2 at each iteration. In our numerical experiments (Section 5.4), a i and b i are updated during iterations as follows: at iteration k of 2 -DCA, we take a i = θ k and b i = A (i,j i ) , where j i ∈ I i (θ k ), ∀i = 1, . . . , N and I i is defined as (5.19). Thirdly, the resulting convex subproblem is reformulated as a quadratic program that can be solved by standard softwares. Such a reformulation is feasible by dint of the polyhedral convexity of f i . We thus see a good exploitation of the properties of f i in the construction of DC decomposition and of the corresponding DCA. By the way, from a numerical point of view, our 2 -DCA algorithm is more efficient than DCA 2 in [START_REF] Piot | Difference of Convex Functions Programming for Reinforcement Learning[END_REF] which uses the subgradient method to solve nonsmooth convex subproblems.

DCA for solving the ∞ -norm problem (5.10)

The ∞ -norm problem (5.10) is written as follows.

min F ∞ (θ) = max i=1,...,N |f i (θ)| : θ ∈ R d .
For θ ∈ R d and i = 1, . . . , N , we have

|f i (θ)| = 2f + i (θ) -f i (θ) = 2f + i (θ) + N j=1,j =i f j (θ) - N j=1 f j (θ) . Let G 3 (θ) := max i=1,...,N 2f + i (θ) + N j=1,j =i f j (θ) .
Then G 3 is convex and therefore we get the following DC formulation of (5.10):

min{F ∞ (θ) := G 3 (θ) -H 1 (θ) : θ ∈ R d }, (5.26)
where both DC components are polyhedral convex. Hence, the problem (5.10) is a polyhedral DC program. Similarly, DCA applied to (5.26) consists of, at each iteration A unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning 113 k, first calculating a subgradient w k ∈ ∂H 1 (θ k ) and then solving the following convex program: min

θ∈R d max i=1,...,N 2f + i (θ) + N j=1,j =i f j (θ) -w k , θ . (5.27)
The problem (5.27) can be equivalently reformulated as the next linear program:

                     min t -w k , θ , s.t. θ ∈ R d , N j=1,j =i t j -t ≤ 0, ∀i = 1, . . . , N, 2t i + N j=1,j =i t j -t ≤ 0, ∀i = 1, . . . , N, θ, A (i,j) + b (i) ≤ t i , ∀i = 1, . . . , N, ∀j = 1, . . . , N A .
(5.28)

Finally, DCA applied to (5.26) is described in Algorithm 5.3 ( ∞ -DCA) below.

Algorithm 5.3 DCA for solving (5.26) ( ∞ -DCA)

Initialization: Let ε be a sufficiently small positive number. Let

θ 0 ∈ R d . Set k = 0. repeat 1. Compute w k ∈ ∂H 1 (θ k ) using (5.20).
2. Solve the linear program (5.28) to obtain (θ k+1 , tk+1 , t k+1 ).

3.

k = k + 1. until |F ∞ (θ k ) -F ∞ (θ k-1 )| ≤ ε(|F ∞ (θ k-1 )| + 1) or ||θ k -θ k-1 || 2 ≤ ε(1 + ||θ k-1 || 2 ).
Similarly to Theorem 5.1, we have

Theorem 5.3. Convergence properties of ∞ -DCA i) ∞ -DCA generates the sequence {θ k } such that the sequence {F ∞ (θ k )} is decreasing.
ii) The sequence {θ k } converges to a critical point θ * of (5.26) after a finite number of iterations. iii) θ * is almost always a local minimizer to (5.26). In particular, if I i (θ * ) is a singleton for all i = 1, ..., N , then θ * is a local minimizer to (5.26).

DCA applied on the new concave minimization formulation (5.14)

As (5.14) is a concave minimization problem with linear constraints, the following DC formulation is the most natural

min{F cc (θ) := χ C θ (θ) -H 1 (θ) : θ ∈ R d }, (5.29)
Obviously, the problem (5.29) is a polyhedral DC program where both DC components are polyhedral convex functions.

A unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning DCA applied to (5.29) consists in computing two sequences {w k } and {θ k } such that

w k ∈ ∂H 1 (θ k ), θ k+1 ∈ argmin χ C θ (θ) -w k , θ : θ ∈ R d ⇔ θ k+1 ∈ argmin -w k , θ : A (i,j) , θ + b (i) ≤ 0, i = 1, . . . , N, j = 1, . . . , N A . (5.30)
Algorithm 5.4 DCA for solving (5.29) (cc-DCA)

Initialization: Let ε be a sufficiently small positive number. Let

θ 0 ∈ R d . Set k = 0. repeat 1. Compute w k ∈ ∂H 1 (θ k ) using (5.20).
2. Solve the linear program (5.30) to obtain θ k+1 .

3.

k = k + 1. until |F cc (θ k ) -F cc (θ k-1 )| ≤ ε(|F cc (θ k-1 )| + 1) or ||θ k -θ k-1 || 2 ≤ ε(1 + ||θ k-1 || 2 ).
As (5.29) is a polyhedral DC program where both DC components are polyhedral convex, we have Theorem 5.4. Convergence properties of cc-DCA i) cc-DCA generates the sequence {θ k } such that the sequence {F cc (θ k )} is decreasing.

ii) The sequence {θ k } converges to a critical point θ * of (5.29) after a finite number of iterations. iii) θ * is almost always a local minimizer to (5.29). In particular, if I i (θ * ) is a singleton for all i = 1, ..., N , then θ * is a local minimizer to (5.29).

Remark 5.5. Similarly to the 1 -norm problem, our DC formulations for the ∞norm and the concave minimization problems have the same advantages: both DC components are convex polyhedral, consequently the three algorithms 1 -DCA, ∞ -DCA and cc-DCA enjoy the same interesting convergence properties.

Performance analysis on different DCA based algorithms

For the same 1 -norm (resp. 2 -norm) problem, our proposed algorithm 1 -DCA (resp.

2 -DCA) is more efficient than DCA 1 (resp. DCA 2 ) developed in [START_REF] Piot | Difference of Convex Functions Programming for Reinforcement Learning[END_REF] (see remarks 5.3 and 5.4 above). These advantages show once again the great effect of DC decompositions as well as the one of solution methods for convex subproblems in DCA. Here the convexity and the polyhedral structure of f i are the key for designing 1 -DCA and 2 -DCA.

Among the four proposed DC formulations, those for the 1 -norm, the ∞ -norm and the concave minimization are similar: they have the same second DC component H 1 and both DC components are polyhedral convex. Consequently the three resulting A unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning 115 these three algorithms are more efficient than 2 -DCA which consists in solving iteratively one convex quadratic program. Note also that 2 -DCA does not have the finite convergence property as 1 -DCA, ∞ -DCA and cc-DCA. Meanwhile, the DC decomposition (5.23) and the use of local DC decompositions in 2 -DCA (see Remark 5.4) are quite interesting, which can result in higher quality of solutions. This observation is confirmed by numerical results in our experiments.

Since the three DC programs of the 1 -norm, the ∞ -norm and the concave minimization problems share the same DC component H 1 , the complexity of the resulting DCAs differs from one of others by the complexity of linear subproblems. In this sense, cc-DCA is more advantageous than 1 -DCA and ∞ -DCA. In fact, the linear program in cc-DCA has the smallest size with d variables, N A .N constraints; the one in 1 -DCA comes next with d + N variables, (N A + 1).N constraints and finally, the one in ∞ -DCA has largest size with d + N + 1 variables, (N A + 2).N constraints. Thus, cc-DCA should be the fastest, 1 -DCA is the next and the last is ∞ -DCA.

Starting points for DCA

Finding a good starting point is an important issue when designing DCA. For this purpose we convexify the concave function F cc by ignoring the difficult term max a ∈A θ, φ(S i , a ) in each f i and then consider the convex problem

min F cv (θ) := N i=1 Q θ (S i , A i ) : f i (θ) ≤ 0, ∀i = 1, . . . , N (5.31)
which can be reformulated as a linear program. When (5.31) admits an optimal solution, we take it as starting point for DCA. Instead, in case (5.31) has no optimal solution, we choose a feasible solution.

Numerical experiments

In the numerical experiments, we study the performance of the proposed DCA algorithms 1 -DCA, 2 -DCA, ∞ -DCA, cc-DCA and compare them with two existing DCA algorithms DCA 1 , DCA 2 [START_REF] Piot | Difference of Convex Functions Programming for Reinforcement Learning[END_REF], and two standard RL algorithms named Fitted Q-Iteration (FQI) [START_REF] Ernst | Tree-based batch mode reinforcement learning[END_REF] and Least Square Policy Iteration (LSPI) [START_REF] Lagoudakis | Least-squares policy iteration[END_REF]. Table 5.1 summarizes the comparative algorithms.

Our experiment is composed of two parts. In the first experiment we study the efficiency of the six DCA based algorithms: the four proposed DCA and the two existing DCA 1 , DCA 2 [START_REF] Piot | Difference of Convex Functions Programming for Reinforcement Learning[END_REF] on the Garnet problem [START_REF] Bhatnagar | Natural actor-critic algorithms[END_REF] which has been tested in [START_REF] Piot | Difference of Convex Functions Programming for Reinforcement Learning[END_REF]. For a fair comparison we use the same procedure to generate Garnet instances as in [START_REF] Piot | Difference of Convex Functions Programming for Reinforcement Learning[END_REF]. We also study the effect of starting points on the proposed DCAs.

A unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning In the second experiment we compare notable DCA algorithms ( 1 -DCA, 2 -DCA, and cc-DCA) with the FQI, LSPI algorithms on Garnet problems and the Gridworld problem [1,[START_REF] Pashenkova | Value iteration and policy iteration algorithms for markov decision problem[END_REF][START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF].

Description of Garnet and Gridworld problems

Let us first present the two problems concerning our computational experiments: Garnet and Gridworld.

Garnet: Stationary Garnet problems, a class of randomly constructed finite MDP (the state space S and the action space A are finite), were introduced in [START_REF] Bhatnagar | Natural actor-critic algorithms[END_REF]. The typical features of a stationary Garnet problem, denoted by Garnet(N S ,N A ,N B ), are three parameters: the number of states N S , the number of actions N A and the number of next states N B for each state-action pair. We consider here a particular type of Garnet, given in [START_REF] Piot | Difference of Convex Functions Programming for Reinforcement Learning[END_REF], which is a topological structure relative to real dynamical systems. In this Garnet, the state space is considered as S = {s i } i=1,...,N S where each state s i = (s (j) i ) j=1,...,m is an m-tuple (m = 2) and each component s (j) i is chosen out of all integer numbers between 1 and x i (x i = K, ∀i = 1, . . . , N S ). Thus, the number of states is N S = m i=1 x i = K 2 . All parameters of Garnet problems are set similarly as in [START_REF] Piot | Difference of Convex Functions Programming for Reinforcement Learning[END_REF]. Specifically, for each state-action pair (s, a), the next states are randomly chosen via the multivariate normal distribution N m (s, I m ) where I m is the identity matrix of size m. The transition probability of going to each next state is generated by dividing the interval [0, 1] at (N B -1) random points. The reward for each stateaction pair is generated uniformly between -1 and 1. The discount factor γ is set to 0.95. All algorithms use the exact tabular representation and thus, the number of basis functions is d = N S .N A . For each Garnet problem with the transition probability function and the reward function generated as above, the optimal policy π * is obtained by the policy iteration algorithm (see, e.g., [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]). We test on N G different Garnet(N S , N A , N B ) problems {G p } 1≤p≤N G and, for each Garnet G p , construct N data datasets {D p,q } 1≤q≤N data of N transition samples (S i , A i , S i ) i=1,...,N drawn uniformly and independently.

Gridworld: We use a 100 × 100 Gridworld with sparse rewards as described in A unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning 117 [1,[START_REF] Pashenkova | Value iteration and policy iteration algorithms for markov decision problem[END_REF][START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. A cell of the grid corresponds to a state and thus the number of states is N S = 10 4 . At each state, there are four possible actions: North, South, East, and West (thus, N A = 4), which make an agent move one cell in the corresponding direction on the grid with some probability p, or in each of three unintended directions with probability 1-p 3 . If the agent gets out of the grid, its place will be unchanged. We divide the grid into non-overlapping regions of 10 × 10 cells, called macrocells, and thus the number of macrocells is N reg = 100. For each i ∈ {1, . . . , N reg } and j ∈ {1, . . . , N A }, φ i+Nreg(j-1) (s, a) = 1 if the state s resides in the ith macrocell and the action a coincides with a j , 0 otherwise. In this case, the number of basis functions is d = N reg .N A . The reward function is given by R = w φ, where the weight w ∈ R d is generated randomly as follows: for i ∈ {1, . . . , d}, w i = 0 with probability of 0.9 and w i is sampled uniformly from [0, 1] with probability of 0.1. Finally, w is renormalized such that i=1,...,d w i = 1. Instances with fewer than two nonzero entries in w are not interesting and discarded. The discounted factor γ and the probability p are set to 0.99 and 0.7, respectively. For each Gridworld problem, we also obtain the optimal policy π * by the policy iteration algorithm (see, e.g., [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]). We run on N G Gridworlds {Gr p } 1≤p≤N G and, for each Gr p , build N data datasets {D p,q } 1≤q≤N data of N transition samples (S i , A i , S i ) i=1,...,N generated uniformly and independently.

Set up experiments

All experiments were implemented in MATLAB R2013b and performed on a PC Intel(R) Xeon(R) CPU E5-2630 v2, @ 2.60GHz of 32GB RAM. The software CPLEX 12.6 was used for solving linear and/or convex quadratic programs. The functions lsqlin and mldivide in MATLAB were utilized for solving constrained linear leastsquares problems in FQI and the system of equations in LSPI, respectively.

The stopping criteria of all DCA based algorithms considered are either

||θ k -θ k-1 || 2 ≤ ε(1 + ||θ k-1 || 2 ) or |F (θ k ) -F (θ k-1 )| ≤ ε(|F (θ k-1 )| + 1)
or the CPU time exceeds 1000 seconds for 1 -norm/ ∞ -norm/concave problems and 2000 seconds for 2 -norm problems. The default tolerance is ε = 10 -4 .

The stopping criteria of the subgradient method for solving convex subproblems in DCA 1 and DCA 2 are either s k 2 ≤ ε (s k is a subgradient of the objective function at θ k ) or the number of iterations does not exceed M . A small value of M implies a premature stopping of subgradient algorithm, in such a case DCA 1 and DCA 2 are not stable in the sense that the sequence of objective function values may not be decreasing, and then the obtained solution may be worse. When M is large, the subgradient method runs very slowly. By compromising, we set M to 100.

We are interested in the following criteria to evaluate the effectiveness of the proposed algorithms: the error performance and its standard deviation (denoted by T A and std A A unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning respectively), the number of iterations (denoted by Iter), the CPU time (in seconds) and the value of OBR (for p -norm problems with p = 1 and p = 2). T A and std A are defined in [START_REF] Piot | Difference of Convex Functions Programming for Reinforcement Learning[END_REF] as follows.

T A = 1 N G .N data N G p=1 N data q=1 T p,q A , T p,q A = E ρ [V π * -V π p,q A ] E ρ [|V π * |] ,
where ρ is the uniform distribution over S and the policy π p,q A is greedy with respect to the output Q θ of the algorithm A and V π is the state value function under policy π. For the first purpose, we test our four DCA schemes ( 1 -DCA, 2 -DCA, cc-DCA, ∞ -DCA) and two existing ones (DCA 1 , DCA 2 ) on 10 Garnet(100,5,5) problems {G p } 1≤p≤10 and 5 sample datasets {D p,q } 1≤q≤5 for each Garnet G p as described in Section 5.4.1. The starting point of DCA 1 , DCA 2 is set to zero as in [START_REF] Piot | Difference of Convex Functions Programming for Reinforcement Learning[END_REF]. The average results of T A , std A , Iter and CPU obtained by the algorithms on 50 different runs corresponding to 50 pairs of (G p , D p,q ) with different values of N (N ∈ {500, 800, 1000, 1300, 1500, 1800, 2000, 2300, 2500}) are shown in Table 5.2. In Fig. 5.1 we plot the same results concerning T A and CPU time. We also test the algorithms on 5 different Garnet(N S , 5, 5) problems (N S ∈ {225, 324}) and 5 datasets of N transition samples for each Garnet with different values of N (N ∈ {4000, 5000, 6000, 7000}). The comparative results are given in Table 5.3.

std A = 1 N G N G p=1 1 N data N data q=1 T p,q A - 1 N data N data k=1 T p,k A 2 1 2
For the second purpose, we perform four algorithms • For the medium Garnet problems (N S = 100) in Table 5. ii) The CPU time of DCA 2 usually exceeds the maximum time (2000 seconds) while that of 2 -DCA is about 155 seconds (when N = 5000) -the gain varies from 3.51 to 11.7 times.

In summary, we can say that for these medium Garnet problems, 1 -DCA could be the algorithm realizing the best trade-off between the quality and the rapidity while 2 -DCA could be the best choice when considering the larger (in terms of the state space A unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning and sample datasets) Garnet problems. Moreover, our DCA algorithm 1 -DCA (resp.

2 -DCA) is more efficient than the existing one DCA 1 (resp. DCA 2 ) in all criteria.

Effect of starting points on our four DCA algorithms

In this experiment, we study the effect of our starting point θ (1) (as in Section 5.3.6) and the zero starting point θ (2) on our algorithms 1 -DCA, 2 -DCA, ∞ -DCA and cc-DCA in terms of the error performance T A on 10 different Garnet(100,5,5) problems and 5 transition sample datasets for each Garnet (N ∈ {200, 400, 600, 800, 1000}).

The comparative results are shown in Table 5.5.

N 1 -DCA 2 -DCA ∞ -DCA cc-DCA θ (1) θ (2) θ (1) θ (2) θ (1) θ (2)
θ (1) θ (2) 200 0.847 0.862 0.972 0.990 0.864 0.877 0.850 0.859 400 0.576 0.627 0.837 0.860 0.747 0.816 0.648 0.734 600 0.408 0.537 0.607 0.640 0.587 0.741 0.508 0.783 800 0.286 0.513 0.443 0.459 0.425 0.694 0.391 0.812 1000 0.221 0.487 0.269 0.300 0.409 0.740 0.288 0.832 From Table 5.5 we see that, for all the algorithms, the starting point θ (1) is more efficient than θ (2) in all cases -the ratio of gain on T A of 1 -DCA, 2 -DCA, ∞ -DCA, cc-DCA varies from 1.74% to 54.6%, from 1.81% to 10.3%, from 1.48% to 44.7%, from 1.04% to 65.3%, respectively. This is an illustration of the effectiveness of our proposed starting point. This experiment aims to make a comparison between our notable DCA algorithms 1 -DCA, cc-DCA, 2 -DCA and the standard RL algorithms FQI, LSPI on Garnet problems. We test the algorithms on 50 different Garnet [START_REF] Randour | Variations on the stochastic shortest path problem[END_REF]5,5) problems and 10 datasets of N transition samples for each Garnet (N ∈ {500, 1000, 1500, 2000, 2500}). The comparative results of these algorithms in terms of T A , std A and CPU are presented in Table 5.6. In Fig. 5.2 we plot the same results concerning T A and CPU time.

A unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning 125 A unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning

Comments on numerical results

For medium Garnet problems (Table 5.6 and Fig. 5.2), LSPI and FQI give the best error performance T A , except for the case N = 500 where FQI completely fails to find a good solution and its T A is the largest versus all algorithms. Meanwhile, to reach these results LSPI and FQI need much more CPU time than our DCAs. The T A furnished by 1 -DCA is close to the ones of LSPI but 1 -DCA is much faster than LSPI.

The ratio of gain varies from 4.3 to 25.7 times. Similarly, 1 -DCA is faster than FQI, from 3.5 to 5.6 times. As for 2 -DCA and cc-DCA, their error performances T A are comparable with 1 -DCA. More precisely, 2 -DCA gives the smaller T A when N = 2000 and N = 2500 and the slightly larger T A in other three cases, but it consumes more time than 1 -DCA (from 4.1 to 11.8 times). The T A given by cc-DCA is slightly larger than that of 1 -DCA but it is faster from 1.5 to 5.5 times. Thus, not surprisingly, cc-DCA is the fastest algorithm, in particular, the ratio of gain versus LSPI (resp. FQI) varies from 23.4 to 39.4 (resp. from 7 to 19.6) times. In summary, 1 -DCA could be the algorithm realizing the best trade-off between the quality and the rapidity.

Gridworld problems

In this experiment, we study the performance of 1 -DCA, 2 -DCA, cc-DCA in comparing with LSPI, FQI on 10 different Gridworld problems and 10 random datasets of N transition samples for each Gridworld (N ∈ {900, 1100,. . ., 2100}) as described in Section 5.4.1. The comparative results in terms of T A , std A and CPU time are reported in Table 5. In sum, we can say that for these Gridworld problems 2 -DCA is the most efficient algorithm which realizes the best trade-off between the quality and the rapidity.

Conclusions

We have intensively investigated DC programming and DCA for Batch RL by finding the zero of the empirical Optimal Bellman Residual via linear approximation. Besides the existing p -norm formulations of OBR (p ∈ {1, 2}) we have considered the ∞norm, and also proposed the new concave minimization formulation without using the p -norm of OBR. These optimization formulations are reformulated as DC programs for which four DCA schemes have been developed. Exploiting the special structure of the considered problems we have carefully addressed the two key issues of DCA -the nice effect of DC decompositions and the solution methods of resulting convex subproblems so as to design efficient DCA based algorithms. It turns out that, for the same 1norm (resp. 2 -norm) formulation, the proposed 1 -DCA (resp. 2 -DCA) is much more efficient than the existing DCA 1 (resp. DCA 2 ) on both quality and rapidity. Among the four proposed DCA, the cc-DCA (corresponding to the new concave formulation) is the fastest, and the three 1 -DCA, cc-DCA and ∞ -DCA enjoy interesting convergence properties of polyhedral DC programs while 2 -DCA is of good quality thanks to the update of DC decompositions during the algorithm. Comparing with the two standard approaches for Batch RL, 1 -DCA (resp. 2 -DCA) realizes the best trade-off between the quality and the rapidity for Garnet problems (resp. Gridworld problems).

Chapter 6

Online DCA for Reinforcement Learning 1

Abstract: This chapter focuses on developing reinforcement learning (RL) techniques in online mode via the optimal Bellman residual (OBR) minimization approach. Based upon the transition sample collected from the interaction at each step, an agent tries to improve the policy via the state-action value function Q by optimizing the sampled OBR functions with p -norm (p ≥ 1). In fact, each p -norm optimization formulation of OBR is a DC (Difference of Convex functions) program. In this chapter, we investigate Online DCA (DC Algorithm) for solving the 2 -norm formulations of OBR via value-function linear approximation. Exploiting an appropriate DC decomposition, we develop a corresponding Online DCA based algorithm which enjoys the online stability property. We also indicate that the well-known residual gradient algorithm for RL is a special case of our Online DCA algorithm. We propose an alternating version of this algorithm where the value function at each step is alternatively updated. Numerical experiments on two benchmarks -mountain car and pole balancing problems -show the effectiveness of our approaches in comparison with some standard RL algorithms (Q-learning, SARSA).

Online DCA for Reinforcement Learning

Our contributions

We develop RL techniques in online mode based on Online DC programming and Online DCA by addressing the p -norm of empirical OBR via value-function linear approximation. Firstly, we consider an 2 -norm optimization formulation of OBR for which Online DC programming is investigated. Exploiting an appropriate DC decomposition for DC function at each step, we develop the corresponding Online DCA based algorithm, which enjoys the online stability property. Secondly, we suggest an alternating version of our algorithm where the update rule of value function is computed alternatively. Finally, we provide several numerical experiments of the proposed algorithms on two benchmarks -mountain car problems and pole balancing problems -in comparison with two well-known RL algorithms, namely Q-learning and SARSA.

The rest of the chapter is organized as follows. Optimization formulations of the empirical OBR is described in Section 6.2. Section 6.3 first presents an introduction of Online DC programming and Online DCA and then shows how to investigate Online DC programming and Online DCA for RL. Section 6.4 reports the numerical results on several test problems which is followed by some conclusions in Section 6.5. Let F denote the space of approximation functions, say

Optimization formulations

F := Q θ (s, a) = θ, φ(s, a) |θ ∈ R d .
The p -norm (p ≥ 1) of OBR with the probability distribution µ is defined as (see e.g. [START_REF] Munos | Performance Bounds in L p norm for Approximate Value Iteration[END_REF]) .

J p,µ (Q) := ||B * Q -Q|| p,
Thus, with the linear approximation, the p -norm (p ≥ 1) of the empirical OBR over the space of linear approximation functions F is

J p,µ (Q θ ) = ||B * Q θ -Q θ || p,µ .
In RL, estimating the optimal state-action value function at each step is based on the transition sample through the interaction with the environment. Specifically, at the step t, one first collects a sample (S t , A t , S t , r t ) via ε-greedy strategy, then considers a nonbiased estimation of B * Q(S t , A t ) :

B * Q(S t , A t ) = r t + γ max a ∈A Q(S t , a )
Finally, at the final step T , one minimize the corresponding empirical p -norm of OBR:

J p,T (Q) := T t=1 B * Q(S t , A t ) -Q(S t , A t ) p 1/p .
Using this sampling-based technique with linear approximation operator, we have to minimize the empirical p -norm (p ≥ 1) of the empirical OBR that is

T t=1 B * Q θ (S t , A t ) -Q θ (S t , A t ) p 1 p
and finally the p -norm formulation of the empirical OBR via linear approximation takes the form min

θ∈R d F p,T (θ) := T t=1 r t + γ max a ∈A θ, φ(S t , a ) -θ, φ(S t , A t ) p . (6.1)
Let p t denote the real-value function on R d defined as

p t (θ) := r t + γ max a ∈A θ, φ(S t , a ) -θ, φ(S t , A t ) .
Clearly, p t is the maximal function of a finite family of affine functions, expressed as

p t (θ) = max j=1,...,N A A (t,j) , θ + b (t)
where A (t,j) := γφ(S t , a j ) -φ(S t , A t ), b (t) := r t , j = 1, . . . , N A .

In this chapter, we consider the 2 -norm optimization formulation at the final step T taking the form

min F 2,T (θ) := T t=1 [p t (θ)] 2 : θ ∈ R d . (6.2)
As each p t is a polyhedral convex function, F 2,T is a DC function. Assume that ODCA generates the sequence {θ t } t=1,...,T +1 with the step size η t = η for all t and the sequence {v t } is upper bounded by V , i.e. v t ≤ V for all t. If η = 1/ √ T , then the ODCA algorithm is online stable, that is to say,

lim T →∞ 1 T T t=1 θ t -θ t+1 = 0.
Proof. We know that the update rule (6.3) is equivalent to

θ t+1 ∈ arg min θ∈R d t i=1 θ, v i + 1 η θ 2 .
Since the function • 2 is a strongly convex function on R d , we derive from (6.3.1) that

t i=1 θ t , v i + 1 η θ t 2 - t i=1 θ t+1 , v i + 1 η θ t+1 2 ≥ 1 η θ t -θ t+1 2 . (6.9)
Similarly to the step t -1, we have

t-1 i=1 θ t+1 , v i + 1 η θ t+1 2 - t-1 i=1 θ t , v i + 1 η θ t 2 ≥ 1 η θ t+1 -θ t 2 . (6.10)
The sum, side by side, of (6.9) and (6.10) yields

2 η θ t+1 -θ t 2 ≤ θ t -θ t+1 , v t ≤ θ t+1 -θ t v t , ∀t.
Thus, we have

S T ODCA = 1 T T t=1 θ t -θ t+1 ≤ ηV 2 ,
which concludes the proof.

Remark 6.1. By choosing the suitable parameters, we see that the class of residual gradient (RG) algorithms in [5] is a special case of ODCA. Indeed, if δ t = 0 or a t = θ t for all t, then

θ t+1 = θ t -η t A (t,jt) , θ t + b (t) A (t,jt) ,
which is exactly the same as the update step in RG algorithms, which is summarized in Algorithm 6.2 (ODCA1).

Computational results

Two purposes of our experiments in this chapter are to, first, give a comparison of four algorithms: ODCA, ODCA1, AODCA, AODCA1 in mountain car problems and, second, compare between the notable algorithms AODCA1 and Q-learning, SARSA in both problems.

Concerning the first purpose, we tune the parameter η for each algorithm over 5 different runs and, with the best value of η, we test the algorithm over 50 runs. Fig. 6.2 shows the average number of steps of four ODCA based algorithms over 50 runs in the number of episodes in mountain car problems. In addition, each bar in Fig. 6.2 represents the value of standard deviation number of steps over 50 runs divided by 8 (this makes the figures clearer). Table 6.1 reports the average/standard deviation number of steps at the last episode and CPU time (in seconds) of the four algorithms. Online DCA for Reinforcement Learning

For the second purpose, similarly, we tune the parameter over 10 different runs and run each algorithm over 100 runs in both mountain car and pole balancing problems. The average/standard deviation results of AODCA1 and Q-learning, SARSA over these 100 runs are reported in Fig. 6.3 and Fig. 6.4. These results at the last episode and CPU time for mountain car problems are given in Table 6.2. Table 6.3 presents the total number of steps of all episodes, CPU time (in seconds) and the ratio of CPU time to the total number of steps of the three algorithms in pole balancing problems. The lower the curve, the better the performance.

Online DCA for Reinforcement Learning

In summary, we can say that the alternating versions of ODCA based algorithms are more efficient than the original versions in all criteria. In addition, for both considered problems, AODCA1 could be the algorithm realizing the best trade-off between the quality and the rapidity.

Conclusions

We have investigated Online DC programming and Online DCA for RL in online mode via the OBR minimization approach. Considering the 2 -norm formulations of OBR, we have developed an Online DCA based algorithm (ODCA) which enjoys the online stability property. We have indicated that the classic residual gradient algorithm for RL is a special case of our algorithm. To exploit the knowledge of transition samples, we have proposed its alternating versions. Comparing with the two standard RL algorithms (Q-learning and SARSA) on two benchmarks -mountain car and pole balancing problems, our ODCA based approach is proven to be effective due to its best trade-off between the quality and the rapidity. In the future, we plan to extend our results to develop more efficient RL techniques, for example, RL with eligibility traces.

Chapter 7

Applications to Stochastic Shortest Path problems: DCA Approaches via Cardinality Minimization and Reinforcement Learning 1

Abstract: The chapter concerns the Stochastic Shortest Path (SSP) problem for a single independent vehicle on a road network using the probability tail model-based criterion. The SSP problem aims to search an optimal path that maximizes the probability of reaching destination before a particular deadline. There exist two different reformulations of this problem: first, a cardinality minimization formulation (cardinality of a vector is the number of nonzero elements in that vector) and second, an RL formulation. For the first formulation, the maximization problem is reformulated as a cardinality minimization problem with zero-one variables. Recently, some algorithms have been proposed by approximating the cardinality term due to its discontinuity, however, without treating zero-one variables. In this chapter, we develop a DC programming and DCA based approach for solving this cardinality problem. We investigate first a DC approximation approach for the cardinality term and then an exact penalty technique for the zero-one variables. The resulting optimization reformulation can be expressed as a DC program for which DCA is applied. We propose a DCA based algorithm, namely Card-DCA, for solving this SSP problem. Numerical experiments in an artificial road network with different given deadlines show the efficiency of the proposed algorithm in terms of both quality and rapidity when compared with the existing algorithms. For the RL formulation, we take into account the 1 -norm optimization problem in which the given set of samples is defined based on the travel time data on the road network and hence a DCA based algorithm, namely RL-DCA, is proposed for these SSP problems. In comparison with Card-DCA, our proposed algorithm RL-DCA is capable of improving the probability of reaching destination.

1. The material of this chapter is developed from the following work: [1]. Vinh Thanh Ho, Hoai An Le Algorithm) based approach for the 0 -norm problem and indicated that corresponding DCA schemes cover all standard algorithms in nonconvex approximation approaches as special versions. Although there are many studies for solving the 0 -norm problems, these problems are actually hard to solve due to 0 -norm. In this chapter, we follow the continuous DC approximation function in [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF] to tackle the discontinuous Card term.

MDP reformulation

In the literature, there are many SSP problems which are analyzed by means of the MDP framework (see e.g. [START_REF] Bertsekas | Dynamic Programming: Deterministic and Stochastic Models[END_REF][START_REF] Bertsekas | Stochastic Optimal Control: The Discrete Time Case[END_REF][START_REF] Bertsekas | An analysis of stochastic shortest path problems[END_REF]). These SSP problems can be seen as a well-known class of MDP problems where their different MDP formulations are considered (see [START_REF] Bertsekas | Dynamic Programming and Optimal Control[END_REF][START_REF] Cao | Maximizing the Probability of Arriving on Time: A Practical Q-Learning Method[END_REF][START_REF] Neu | The adversarial stochastic shortest path problem with unknown transition probabilities[END_REF][START_REF] Randour | Variations on the stochastic shortest path problem[END_REF]). Concerning the PT model-based SSP problem, its MDP reformulation has been proposed in [START_REF] Cao | Maximizing the Probability of Arriving on Time: A Practical Q-Learning Method[END_REF]. In particular, the state space, denoted by S, is defined as the set of the pair of intersections (or nodes in the graph) and time-to-deadlines:

S := {s = (v, τ ) : v ∈ V, τ ∈ Γ}
where Γ is the set of time-to-deadlines. Let us assume that the set Γ is finite and denote by N S , N Γ respectively the size of the space S, the set Γ. Thus, one has N S = n.N Γ .

The action space, denoted by A, is defined as the set of directions:

A := {a i } i=1,...,N A .

The state transition probability P(s = (v , τ )|s = (v, τ ), a) represents the distribution of the random travel time, denoted by t v,v , on the road link (or arc) (v, v ) ∈ A r by the direction a ∈ A that takes the vehicle (or the agent) from the intersection v with the time-to-deadline to the next intersection v with the remaining time-to-deadline τ computed by τ = τ -t v,v . The reward function, denoted by R, at each state-action transition (s, a) represents the expected immediate reward received after taking the direction a for the intersection s with the time-to-deadline τ . The discount factor is denoted by γ.

In this chapter, we concern these SSP problems in which the full knowledge of MDP in terms of P and R is not given in advance, and thus the aforementioned RL approach via state-action value functions will be investigated. By setting γ to be 1 and the immediate reward to be 1 if the action moves the vehicle from the current intersection to the deadline before the deadline, and 0 otherwise, the optimal state-action value function defined by (4.1) and (4.2) is exactly the same as the probability of arriving on time defined by (7.1). In the recent work [START_REF] Cao | Maximizing the Probability of Arriving on Time: A Practical Q-Learning Method[END_REF], the authors use Q-learning to approximate the optimal state-action value function for discrete/continuous deadlines via a neural network approximation where the sample at each step is chosen through e.g. the Softmax strategy. In this chapter, we focus on estimating the probability of arriving on time by reinforcement learning (RL) techniques in batch mode (Batch RL), i.e. based on the given fixed set of transition samples (more precisely the travel time samples on arcs) with discrete deadlines.

Our contributions

Our contributions are multiple, which can be classified into two parts corresponding to two reformulation problems just mentioned previously. Concerning the cardinality problem (7.2), we develop a DCA approach for solving it via two following main steps with respect to the Card term and the zero-one variables. First, we use a DC approximation of the Card term, which results in approximating the problem (7.2) to a DC minimization problem with zero-one variables. Second, we exploit an exact penalty technique in DC programming for treating these zero-one variables, which has been widely studied in, e.g., [START_REF] Thi | Exact penalty and error bounds in DC programming[END_REF][START_REF] Thi | Exact Penalty in DC Programming[END_REF][START_REF] Pham Dinh | Recent Advances in DC Programming and DCA[END_REF]. Consequently, the resulting combinatorial optimization problem is equivalently reformulated as a continuous problem of the standard form of a DC program. Thus, a DCA based algorithm, namely Card-DCA is designed for solving the considered SSP problem. In addition, as an illustrative experiment for the arriving on time problem, we demonstrate the efficiency of our approach for the problem (7.2) in terms of both the quality of obtained paths and the rapidity on a road network in comparison with the 1 -norm algorithm and the reweighted-1 algorithm. As for the Batch RL based SSP problem, we take into account the 1 -norm optimization problem in which the given samples are defined based on the travel time data on the network and hence another DCA based algorithm, namely RL-DCA, is proposed for these SSP problems. Several numerical experiments on the artificial road networks are conducted in order to compare two DCA approaches for these problems, in particular Card-DCA and RL-DCA algorithms.

The rest of the chapter is organized as follows. In Section 7.2, we present two DCA approaches for SSP problems in which we show how to express the cardinality minimization problem as well as the 1 -norm optimization problem as DC programs, and then design DCA based algorithms for the resulting optimization reformulation. Section 7.3 reports the numerical results on artificial road networks which is followed by some conclusions in Section 7.4. Comments on numerical results (Tables 7.1, 7.2): we observe from experiments that with suitable penalty parameters, our Card-DCA algorithm always furnishes the zero-one solutions, which is an advantage when solving the continuous problems. Moreover, in terms of the accuracy, our Card-DCA algorithm is almost always better than the existing algorithms with the different deadline coefficients and different random distributions. The quality of our solution is very high with the accuracy of above 95% and as for normal/gamma distribution, even above 99%, that is to say, our algorithm usually finds the optimal path as the enumeration method. Moreover, in most cases of our Card-DCA, the larger the training set is, the more the accuracy is improved, however for normal/gamma distribution, the accuracy in case S 1 = 500 is sufficiently high. Regarding CPU time, Card-DCA is faster than the existing algorithms 1 , re-1 , respectively, up to 16.2, 18.6 times, which can be explained by the fact that our Card-DCA algorithm only requires solving successive linear programs, while the existing algorithms solve MILPs. In addition, with the good obtained accuracy, Card-DCA runs much faster than the enumeration method up to 211 times.

Experiment 2: Comparison between DCA approaches

In this experiment, we make a comparison between two DCA approaches, namely RL-DCA and Card-DCA, on an artificial road network with discrete deadlines which is described in [START_REF] Cao | Maximizing the Probability of Arriving on Time: A Practical Q-Learning Method[END_REF]. In particular, the road network is a grid with n × n intersections and thus, the number of intersections (or nodes) is n = n 2 and the number of links (or arcs) is m = 2n(n -1). At each intersection, there are four possible travel directions: east, west, south and north (thus, N A = 4). Set up experiments: The parameters n, S, β are set as follows: n ∈ {20, 30, 40}, S ∈ {200, 500}, β ∈ {1.05, 1.10, 1.15}. The number of (o,d) pairs selected is 100. The experiment was implemented in MATLAB R2016a on a PC Intel(R) Xeon(R) CPU E5-2630 v2, @ 2.6GHz of 32GB RAM. The software CPLEX 12.6 was used for solving linear programs. The default tolerance is ε = 10 -4 . The parameters of Card-DCA are set the same as the previous experiment. Let us define the basic function vector φ as follows: for any state s = (v, τ ) and any action a, i = 1, . . . , n, j = -1, . . . , T , k = 1, . . . , N A , we have the basic function φ i+(j+1)n+n(k-1)(N Γ +2) (s, a) = 1 if the action a, the node v coincide respectively a k ∈ A, i in V and the time-to-deadline τ satisfies that τ < 0 when j = -1, τ = j when j = 0, . . . , T -1, τ ≥ T when j = T , and 0 otherwise. Thus, the number of basic functions is l = nN A (T + 2). The starting point for RL-DCA is a zero vector in R d .

Comparison criteria of algorithms: In our experiment, for any (o,d) pair, we cannot guarantee to find the path which has the maximum number of times of not being late by enumerating all the possible paths during a reasonable time (e.g. more than 3600 seconds). Thus, here we are interested in the following aspects: accuracy (in %) and CPU time (in seconds). Comments on numerical results (Tables 7.3, 7.4): for different grids n × n and number of samples S, the probability of arriving on time of RL-DCA is always larger than that of Card-DCA in all cases of deadline parameters β -the gain of RL-DCA versus Card-DCA varies from 0.08% to 5.85%. The CPU time of both algorithms is small in all data sets: less than 12 seconds -in particular, the Card-DCA is faster than RL-DCA when n = 20, 30 and moderately comparable when n = 40. In our experiments, we can summarize that within an acceptable time, RL-DCA gives the better accuracy than Card-DCA. Concerning the RL problem, we have explored the DCA based RL techniques in batch mode for solving these SSP problems. The numerical results on artificial road networks show that our approach for the cardinality problem is efficient in comparison with the existing algorithms in terms of the quality and the rapidity whereas the RL techniques can obtain the effectiveness to improve the accuracy.

Chapter 8

Conclusions

In the dissertation, we have developed the machine learning techniques in both theoretical and computational aspects. The backbones of our approaches are DC (Difference of Convex functions) and DCA (DC Algorithm), and their online version, which are best known as powerful nonsmooth, nonconvex optimization tools. Over the last three decades, DCA has been thoroughly studied and has enjoyed a lot of great success in a variety of domains in applied science. Thanks to DCA, we have recently developed its online version in the dissertation. This new version allows to encounter most of the large-scale optimization problems nowadays.

In the first part of the dissertation, we have intensively developed online learning (OL) techniques for a class of online problems where the loss function at each step is nonconvex and/or nonsmooth. By the DC approach, we have proposed a specific online version of DCA, named ODCA, where each subproblem is solved by approximating by one iteration of classical subgradient method. We have thoroughly studied the analysis of ODCA in the terms of regret -ODCA enjoys the sublinear/logarithmic regret. As an application, we have considered online binary linear classification. In particular, we have designed three corresponding ODCA based algorithms, all of which enjoy the logarithmic regret. Through the numerical experiments on various benchmark classification datasets, the effectiveness of our algorithms in terms of the quality and the rapidity has been demonstrated by comparing with the five state-of-the-art online classification algorithms.

Continuing the previous works, we have developed another class of OL techniques, namely prediction with expert advice, where the prediction at each step is made based on the basis of experts' predictions. We have exploited different DC approximation functions, which results in two particular Online DCA based schemes. Each convex subproblem in the schemes is solved based on two variants of subgradient method. Several analyses of both schemes in terms of regret have been studied thoroughly.

The performance of our approach in efficiency, rapidity and scalability respects has been verified when compared with three existing standard algorithms on a variety of benchmark datasets.

Conclusions

Next, we have been interested in reinforcement learning (RL) techniques in both online mode and batch mode. As for batch RL, we have intensively investigated DC programming and DCA for the problem of finding the zero of the empirical optimal Bellman residual via linear approximation. We have considered four different DC optimization formulations for which four corresponding DCA schemes have been developed. To design efficient DCA algorithms, we have carefully addressed the three key issues of DCA by exploiting the special structure of the considered problems. It points out that, our proposed algorithm are more efficient than the existing DCA based algorithms on both quality and rapidity. Comparing with the two standard approaches for Batch RL, our algorithm realizes the best trade-off between the quality and the rapidity in several numerical experiments. Concerning RL in online mode, we have proposed a particular online DCA based algorithm which enjoys the online stability property. Moreover, we have suggested efficient alternating versions of the proposed online algorithm. Some experiment results on the classic mountain car and pole balancing problems have shown the efficiency of our approach.

As an application, we have studied DCA approaches for solving a class of stochastic shortest path (SSP) problems in vehicle routing. These SSP problems can be expressed as a cardinality optimization problem with binary variables as well as an RL problem.

Regarding the cardinality problem, we have employed a DC approximation approach and an exact penalty technique in DC programming to develop a fast, efficient DCA scheme for the resulting problem. As for the RL problem, we have explored the DCA based RL techniques in batch mode for solving these problems. The numerical results on artificial road networks show that our DCA approaches give the best accuracy.

In our ongoing works, it is necessary to address some following issues.

Concerning OL, it is known that OL plays a very important role among machine learning techniques for solving the problems which are encountered more and more in various day-to-day applications. In this context, it is essential to create new, efficient tools for the development of the novel OL techniques. Our research on the online version of DC programming and DCA has just started but from the encouraging results, it promises to be an effective tool. In the future, we are making progress in further development of Online DC programming and Online DCA in theoretical and algorithmic respects. In particular, we plan to explore the more efficient DC approximation functions, exploit the fast and scalable solution methods for solving convex subproblems in the Online DCA scheme, study the thorough analysis of the corresponding scheme. We continue investigating this tool to solve a variety of problems in OL.

Regarding RL, our study on RL in batch/online mode suggests that DCA is an efficient approach. That motivates us to develop DC programming and DCA for applications of RL in several areas. Moreover, we intend to extend our results to develop more efficient RL techniques, for example, RL with eligibility traces.

As for SSP problems, we plan to explore the property of incident matrix in the cardinality formulation to guarantee the binary property of our obtained solutions. In addition, the first study on the way to treat the cardinality optimization problem with Conclusions 161 binary variables based on DC programming and DCA opens up promising directions: not only how to develop efficient DCA schemes, but also the consistency between the approximate problem and the original problem should be investigated. On another hand, we are able to investigate these SSP problems based on the nonconvex exact reformulation approach for the cardinality term using the exact penalty techniques. In regard to RL approach for these SSP problems, it will be interesting to know whether our RL techniques can deal with this cardinality optimization problem or not. In the scope of the dissertation, we have considered linear approximation with simple basic function vectors, so the effect of basic functions on the performance of the proposed algorithms as well as how to develop DCA for RL with different approximation functions (e.g. neural network) should also be addressed in the future.

3. 1 121 5. 5 124 5. 6 125 5. 7

 1121512461257 The number of mistakes of all five algorithms with respect to the best value of parameters in the validation procedure on five notable datasets 5.1 Average results of T A and CPU in seconds obtained by the six versions of DCA on 50 runs (corresponding to 10 different Garnet(N S = 100,N A = 5,N B = 5) problems and 5 transition sample datasets for each Garnet) with different numbers of transition samples N . . . . . . . . . . . . . . 5.2 Average results of T A and CPU in seconds obtained by 1 -DCA, 2 -DCA, cc-DCA and LSPI, FQI on 500 runs (corresponding to 50 different Garnet(100,5,5) problems and 10 transition sample datasets for each Garnet) with different numbers of transition samples N . . . . . . . . .5.3 Average results ofT A and CPU in seconds obtained by 1 -DCA, 2 -DCA, cc-DCA and LSPI, FQI on 100 runs (corresponding to 10 different Gridworld problems and 10 transition sample datasets for each Gridworld) with different numbers of transition samples N . . . . . . . . . . . . . . 6.1 Procedure 1: compute θ t+1 in the alternating version of ODCA . . . . . 6.2 Average/standard deviation number of steps of two ODCA based algorithms and their alternating version in the number of episodes over 50 runs in mountain car problems. Each bar represents the value of standard deviation number of steps divided by 8. The lower the curve, the better the performance. . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Average/standard deviation number of steps of AODCA1 and Qlearning, SARSA in the number of episodes over 100 runs in mountain car problems. The lower the curve, the better the performance. . . . . 6.4 Average/standard deviation number of steps of AODCA1 and Qlearning, SARSA in the number of episodes over 100 runs in pole balancing problems. The upper the curve, the better the performance. . . 7.1 A road network with 123 road links and 65 intersections [19] . . . . . . List of Tables 5.3 Average results of T A , std A , Iter and CPU in seconds obtained by the six versions of DCA on 25 runs (corresponding to 5 different Garnet(N S ,5,5) problems (N S ∈ {225, 324}) and 5 transition sample datasets for each Garnet) with different numbers of transition samples N . Bold values indicate the best results. . . . . . . . . . . . . . . . . . . . . . . . . . . 120 5.4 Average results of T A , std A , Iter, CPU in seconds and OBR obtained by 1 -DCA and DCA 1 (resp. 2 -DCA and DCA 2 ) on 20 runs (corresponding to 10 Garnet(400,5,5) problems and 2 transition sample datasets for each Garnet) with different numbers of transition samples N . Bold values indicate the best results in the same 1 -norm (resp. 2 -norm) problem. . Comparative results of the proposed starting point θ (1) and the zero starting point θ (2) for our algorithms 1 -DCA, 2 -DCA, ∞ -DCA and cc-DCA in terms of T A on 50 runs (corresponding to 10 different Garnet(100,5,5) problems and 5 transition sample datasets for each Garnet) with different numbers of transition samples N . Bold values indicate the best results in each algorithm. . . . . . . . . . . . . . . . . . . . . . Average results of T A , std A and CPU in seconds obtained by 1 -DCA, 2 -DCA, cc-DCA and LSPI, FQI on 500 runs (corresponding to 50 different Garnet(100,5,5) problems and 10 transition sample datasets for each Garnet) with different numbers of transition samples N . Bold values indicate the best results. . . . . . . . . . . . . . . . . . . . . . . . . . . Average results of T A , std A and CPU in seconds obtained by 1 -DCA, 2 -DCA, cc-DCA and LSPI, FQI on 100 runs (corresponding to 10 different Gridworld problems and 10 transition sample datasets for each Gridworld) with different numbers of transition samples N . Bold values indicate the best results. . . . . . . . . . . . . . . . . . . . . . . . . . . 127 6.1 Average/standard deviation number of steps at the last episode and CPU time (in seconds) of two ODCA based algorithms and their alternating version over 50 runs in mountain car problems. Bold values indicate the best results. . . . . . . . . . . . . . . . . . . . . . . . . . . 139 6.2 Average/standard deviation number of steps at the last episode and CPU time (in seconds) of AODCA1 and Q-learning, SARSA over 100 runs in mountain car problems. Bold values indicate the best results. . 140 6.3 Total number of steps of all episodes and CPU time (in seconds) and the ratio of CPU time to the total number of steps (denoted by CPU/step) of AODCA1 and Q-learning, SARSA over 100 runs in pole balancing problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 7.1 Accuracy (%) of Card-DCA and 1 -norm algorithm ( 1 ), reweighted-1 algrorithm (re-1 ) on 100 runs with the different deadline coefficients κ and random distributions . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 CPU time (in seconds) of enumeration method, Card-DCA, 1 and re-1 on 100 runs with different random distributions in case κ = 1.2 . . . . 7.3 Accuracy (%) of RL-DCA and Card-DCA on 100 (o,d) pairs with the different grids n × n, the different number of samples on each link S, the different deadline parameters β. Bold values are the best results. . 7.4 CPU time (in seconds) of RL-DCA and Card-DCA on 100 (o,d) pairs with the different grids n × n, the different number of samples on each link S, the different deadline parameters β. Bold values are the best results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Lemma 1 . 1 (

 11 [START_REF] Thi | Solving a Class of Linearly Constrained Indefinite Quadratic Problems by D.C. Algorithms[END_REF]). The sequences {x k } and {y k } in DCA are well defined if and only if dom ∂g ⊂ dom ∂h and dom ∂h * ⊂ dom ∂g * .

  the definition of g

( 4 )

 4 t (w), c t (w) , and the function c t : R n → R is defined by c t (w) := 2e -2κtyt

x 1 ≥

 1 b Sig and the positive parameter γ Sig ≤ min {γ, L 2 (α + β)/2α}. Remark 2.4. (Time complexity) All three proposed ODCA algorithms have the same time complexity of O(nT

  is a DC function with the following DC components g

Theorem 3 . 4 .Theorem 3 . 5 . 2 + 4a 2 / 4

 3435224 For w ∈ S, the number of prediction mistakes made by ODCA-SG has an upper bound that is the root, x 1 , of the equationx -a -b (1 + log(x)) = 0 where a := t∈M f t (w), b := L 2 /γ SG , x 1 ≥ b, the positive parameter γ SG ≤ min {γ, L 2 }.Proof. From the inequality (3.6), Corollary 3.1 and the definition of γ SG , we derive that for any w ∈ S,|M| ≤ t∈M f t (w t ) ≤ t∈M f t (w) + L 2 (1 + log(|M|)) γ SG ,where |M| is the number of steps of the set M.Using the definition of a and b, it is evident that a ≥ 0, b ≥ 1 and the inequality (3.14) can be rewritten as follows.|M| ≤ a + b (1 + log(|M|)) .The real function r: (0, +∞) → R, r(x) = x -a -b (1 + log(x)) is strictly convex on (0, +∞). Since lim x→0 + r(x) = lim x→+∞ r(x) = +∞ and r(b) ≤ 0, the equation r(x) = 0 has two roots x 1 , x 2 such that 0 < x 2 ≤ b ≤ x 1 .The proof is complete. For any w ∈ S, the number of prediction mistakes made by ODCA-ESG is upper bounded by c + √ c where c := 2U 2 log(d). Proof. From the inequality (3.6), Corollary 3.2, we get that for any w ∈ S, |M| ≤ t∈M f t (w t ) ≤ t∈M f t (w) + 2U 2 log(d)|M|. (3.14)

Table 3 . 1 -

 31 Time complexity of comparative algorithms with T and d be the number of rounds and the number of experts respectively. ODCA-SG ODCA-ESG OGD NEG WM O(T d log d) O(T d) O(T d log d) O(T d) O(T d)

Figure 3 . 1 -

 31 Figure 3.1 -The number of mistakes of all five algorithms with respect to the best value of parameters in the validation procedure on five notable datasets

  , B π Q(s, a) = R(s, a) + γ s ∈S P(s |s, a)Q(s , π(s )), ∀(s, a) ∈ S × A, and the Optimal Bellman operator on state-action functions is defined by (see e.g. [128]) B * : R S×A → R S×A , B * Q(s, a) = R(s, a) + γ s ∈S P(s |s, a) max a ∈A Q(s , a ), ∀(s, a) ∈ S × A. The concepts of Bellman Residual (BR) and Optimal Bellman Residual (OBR) arise from the standard results of the theory of DP that are, for any policy π the value function Q π is the unique solution of the Bellman equation Q = B π Q, and the optimal value function Q * is the unique solution of the Bellman equation Q

A

  unified DC programming framework and efficient DCA based approaches for large scale batch Reinforcement Learning Given a basic function vector φ(•) = (φ 1 (•), φ 2 (•)..., φ d (•)) ∈ R d where φ i : S × A → R, i = 1, . . . , d are basic functions on S × A. Then the linear approximation of the function Q, denoted by Q θ , is characterized by the weight vector θ ∈ R d according to the relation Q θ (s, a) := θ, φ(s, a) , ∀(s, a) ∈ S × A.

  µ , where ||Q|| p,µ :=   (s,a)∈S×A µ(s, a)|Q(s, a)| p   1 p

. 1 :

 1 The value of OBR for p -norm problems (p = 1 and p = 2) is given by OBR Comparison between DCA based algorithms 5.4.3.1 Comparative results of the six versions of DCA Two purposes of our experiments in this part are to, first, give a comparison of six versions of DCA and, second, compare carefully between 1 -DCA (resp. 2 -DCA) and DCA 1 (resp. DCA 2 ).

Figure 5 . 1 -

 51 Figure 5.1 -Average results of T A and CPU in seconds obtained by the six versions of DCA on 50 runs (corresponding to 10 different Garnet(N S = 100,N A = 5,N B = 5) problems and 5 transition sample datasets for each Garnet) with different numbers of transition samples N .

5. 4 . 4

 44 Experiment 2: Comparison with LSPI, FQI 5.4.4.1 Garnet problems

Figure 5 . 2 -

 52 Figure 5.2 -Average results of T A and CPU in seconds obtained by 1 -DCA, 2 -DCA, cc-DCA and LSPI, FQI on 500 runs (corresponding to 50 different Garnet(100,5,5) problems and 10 transition sample datasets for each Garnet) with different numbers of transition samples N .

Figure 5 . 3 -

 53 Figure 5.3 -Average results of T A and CPU in seconds obtained by 1 -DCA, 2 -DCA, cc-DCA and LSPI, FQI on 100 runs (corresponding to 10 different Gridworld problems and 10 transition sample datasets for each Gridworld) with different numbers of transition samples N .

Similarly to Section 5 . 2 ,

 52 we briefly restate the optimization formulation of p -norm (p ≥ 1) of OBR, in particular when p = 2. Given a basic function vector φ(•) = (φ 1 (•), φ 2 (•), ..., φ d (•)) ∈ R d where φ i : S × A → R, i = 1, . . . , d are basic functions on S×A. Then the linear approximation of the function Q, denoted by Q θ , is characterized by the weight vector θ ∈ R d according to the relation Q θ (s, a) := θ, φ(s, a) , ∀(s, a) ∈ S × A.

  µ , where ||Q|| p,µ :=

Figure 6 . 2 -

 62 Figure 6.2 -Average/standard deviation number of steps of two ODCA based algorithms and their alternating version in the number of episodes over 50 runs in mountain car problems. Each bar represents the value of standard deviation number of steps divided by 8. The lower the curve, the better the performance.

  Applications to Stochastic Shortest Path problems: DCA Approaches via Cardinality Minimization and Batch Reinforcement Learning Data set: For each link, we randomly generate S travel time instances with mean µ and standard deviation σ = 0.3µ where the value of µ is randomly generated with mean of 15 and standard deviation of 3. The origin o and the destination d are chosen from the set of intersections V. The deadline T is defined by T = βT e where β is a deadline parameter (β > 1), T e is the least expected travel time from o to d with the shortest path of length l e . All values of travel time, deadline are rounded to the nearest integer. In batch mode, the samples are collected from the origin o along any path of length l e based on all S travel time data. Thus, the number of batch samples is N = S.l e .

  For each (o,d) pair and each value of β, we compute the accuracy of each algorithm by the probability of arriving on time of the path obtained by the algorithm on the data set. Descriptions of tables of results: The average accuracy and average CPU time obtained by two algorithms RL-DCA, Card-DCA on 100 (o,d) pairs with the different grids n × n, the different number of samples on each link S, the different deadline parameters β are reported in Tables 7.3, 7.4. Bold values in these tables represent the best results.

  Le paradigme de la prédiction avec les conseils d'experts est introduit comme modèle d'apprentissage en ligne. Il se caractérise par la prise d'une prédiction repose sur la base des prédictions des experts via le vecteur de poids attribué aux experts. De même que le chapitre 2, quand la fonction de perte évaluant la qualité de la prédiction est non convexe et/ou non différentiable, nous pouvons étudier les approches d'approximation DC. En fait, le vecteur de poids appartient souvent à un ensemble particulier (par exemple, simplexe des probabilités), et donc nous proposons deux schémas en ligne en particulier basés sur ODCA, nommés ODCA-SG et DCA-ESG, où chaque sousproblème est résolu en approximant par une itération de la méthode de sous-gradient projeté et la méthode de sous-gradient exponentié, respectivement. Nous analysons également les deux schémas en termes de regret. Nous développons les techniques SG, DCA-ESG, et analysons la borne de regret pour ces algorithmes. La performance des algorithmes proposés est vérifiée en comparant de trois algorithmes standards existants sur différents ensembles de données benchmark.Les chapitres 4, 5, 6 se concentrent sur l'étude des techniques d'apprentissage par renforcement en mode batch et en mode en ligne. Le chapitre 4 présente brièvement l'apprentissage par renforcement et ses travaux connexes. L'apprentissage par renforcement vise à estimer la politique optimale dans un environnement dynamique qui est généralement formulé comme un processus de décision markovien ("Markov decision process" en anglais) avec un modèle incomplet. Il est bien connu que nous pouvons aborder cette tâche par le problème de trouver le zéro du soi-disant résidu optimal de Bellman ("optimal Bellman residual" en anglais), un concept classique de programmation dynamique. Il existe quelques travaux dans la littérature suivant cette Introduction générale d'un vecteur est le nombre d'éléments non nuls dans ce vecteur) et une formulation de l'apprentissage par renforcement. Pour la première approche, il existe une double difficulté: le terme de cardinalité et les variables binaires. Certains algorithmes ont récemment été proposés en approximant le terme de cardinalité, cependant, sans traiter les variables binaires. Ainsi, dans ce travail, nous étudions une approche d'approximation DC pour le terme de cardinalité, et employons une technique de pénalité exacte pour les variables binaires. La formulation d'optimisation résultante peut être exprimée sous la forme d'un programme DC pour lequel l'algorithme basé sur DCA, nommé Card-DCA, est proposé. Les résultats de l'expérience montrent l'efficacité de Card-DCA en termes de qualité et rapidité par rapport aux algorithmes existants. Au regard de la formulation de l'apprentissage par renforcement, nous tenons compte du problème d'optimisation de 1 -norm dans lequel l'ensemble d'échantillons donné est défini en fonction des données de temps de déplacement sur le réseau routier. Nous donc proposons un algorithme basé sur DCA, nommé RL-DCA, pour les problèmes du plus court chemin stochastique. Plusieurs expériences numériques sur les réseaux routiers artificiels sont menées afin de comparer deux approches DCA pour ces problèmes, en particulier les algorithmes Card-DCA et RL-DCA.

de prédiction avec des conseils d'experts pour résoudre les problèmes de classification en ligne où les experts sont représentés par les algorithmes de classification en ligne bien connus. Avec une fonction d'approximation DC, nous obtenons deux algorithmes en ligne basés sur ODCAdirection, sachant qu'il résulte à un problème d'optimisation non convexe qui est très difficile à résoudre exactement. Dans le chapitre 5, nous considérons quatre formulations d'optimisation de ce problème qui minimisent le p -norm du résidu optimal de Bellman avec p ∈ {1, 2, +∞} pour développer les techniques d'apprentissage par renforcement en mode batch (c'est-à-dire un ensemble fixe d'expériences d'apprentissage est donné a priori) en utilisant l'approximation linéaire de la fonction de valeur. Ils sont formulés comme programmes DC pour lesquels quatre schémas DCAs sont développés. En exploitant la structure spéciale du résidu optimal de Bellman empirique avec approximation linéaire, nous abordons les questions clés de DCA, en particulier la effet des décompositions DC, l'efficacité des méthodes de solution pour résoudre le sous-problème convexe résultant, et la recherche de bons points initiaux, lors de la conception des quatre algorithmes basés sur DCA. Expériences numériques sur deux benchmarks des problèmes de processus de décision markovien -le problème de Garnet et Gridworld -montrent l'efficacité de nos approche en comparaison avec deux algorithmes existants basés sur DCA et deux algorithmes d'apprentissage par renforcement. En poursuivant ces travaux, le chapitre 6 vise à développer des techniques d'apprentissage par renforcement en mode en ligne via une formule d'optimisation de Dans le chapitre 7, nous utilisons la technique d'apprentissage par renforcement basée sur DCA pour aborder un des problèmes classiques dans le domaine de tournées de véhicules qui est le problème du plus court chemin stochastique ("stochastic shortest path" en anglais). Nos travaux concernent ce problème pour un seul véhicule indépendant sur un réseau routier en utilisant le critère basé sur le modèle de probabilité à queue. En particulier, ce problème du plus court chemin stochastique vise à rechercher un chemin optimal qui maximise la probabilité d'arriver à la destination avant un délai donné. Il existe deux approches qui formulent ce problème, respectivement, comme une formulation de minimisation de cardinalité (la cardinalité

Organisation de la Thèse

La thèse est composée de huit chapitres. Le premier chapitre décrit brièvement les concepts fondamentaux et les principaux résultats de l'analyse convexe, la programmation DC et DCA, et sa version en ligne, ce qui fournit la base théorique et algorithmique pour les chapitres suivants. Les six chapitres suivants sont divisés en deux parties. La première partie, y compris les chapitres 2 et 3, concerne les techniques d'apprentissage en ligne. Plus précisément, dans le chapitre 2, nous présentons l'approche basée sur Online DCA pour les problèmes d'apprentissage en ligne et développons des techniques en ligne correspondantes pour une classe de problèmes de classification en ligne. En poursuivant cette direction, le chapitre 3 se concentre sur la conception d'une autre technique d'apprentissage en ligne, à savoir la prédiction avec des conseils d'experts. La deuxième partie (chapitres 4, 5, 6, 7) concerne en particulier les techniques d'apprentissage par renforcement. Sa brève introduction et ses travaux connexes sont présentés au chapitre 4. Les techniques d'apprentissage par renforcement basées sur DCA en mode batch et en mode en ligne sont développées respectivement au chapitre 5 et au chapitre 6. Dans le chapitre 7, nous considérons une classe de problèmes du plus court chemin stochastique via des approches DCA. Les conclusions et les perspectives de nos travaux sont données au chapitre 8.

  Online DC programming consists of a feasible set X and a sequence of the function {f t } t=1,...,T where each f t : X → R is a DC function whose DC decomposition is g t -h t where g t and h t ∈ Γ 0 (X). For simplicity, instead of X we consider the feasible set is a nonempty, bounded, closed, convex subset of X, denoted by S. , G t and H t depend on previous DC functions f i , g i and h i , i = 1, . . . , t -1, respectively. Then, one observes a new DC function f t for that step. DCA approximates the concave part -h by its affine majorization (by choosing a subgradient of h at x k ) and minimizes the resulting convex function. Founded on the idea of DCA, we derive an online version of standard DCA, which is called Online DCA. In particular, at the step t, one replaces the second DC component h t of DC loss function f t by its affine minorization (corresponding to taking z t ∈ ∂h t (w t )) and then updates the point w t+1 for the next step t + 1 by solving the convex subproblem(1.11).

	Chapter 1. Preliminary
	1.2 Online DC programming and Online DCA the function R : S → R is a convex regularization function, and t 0 ∈ {1, t}. where z t ∈ ∂h t (w t ).
	Obviously, F t is a DC function. Provided that at the step t, a DC decomposition of Approximating (1.11) by one iteration of the subgradient methods at the point w t with
	Online DC programming and Online DCA is seen as an online version of DC pro-F t is defined as the step size of η t , we derive the following update rule
	gramming and DCA. While DC programming aims to search a point to minimize a DC function, in Online DC programming, the DC programming problems are solved F t = G t -H t , t w t+1 = Ψ(η t , w t , s t ), (1.13) t iteratively as an online process. In recent years, the more and more emergence of the online process of nonconvex, nonsmooth optimization problems in various domains of where the function G t (w) = i=t 0 g i (w) + R(w) and H t (w) = i=t 0 h i (w). where s t ∈ (∂g t (w t ) -z t ) and Ψ is the appropriate operator of subgradient methods
	applied sciences, specially in machine learning, motivates us to develop a novel ap-(for instance the projection operator Proj S (•) of the point w t -η t s t corresponding to
	proach based on Online DC programming and Online DCA. In this section, we present the projected subgradient method).
	some notations and fundamental definitions relating to the Online DC programming 1.2.2 Online DCA Thus, a particular Online DCA based scheme (ODCA) applied to the Online DC and Online DCA framework. programming problem (1.10) can be described as follows. Recall that for solving a DC program, DCA has the quite simple principle: at each iter-Throughout this section, T denotes the total number of steps in online process, t denotes the step index and X denotes the Euclidean space R n . 1.2.1 Online DC optimization Online DC programming ODCA: Online DCA-Subgradient scheme Initialization: let w 1 be an initial point in S, {η t } be the sequence of step sizes. for t = 1, 2, . . . , T do 1. Compute z t ∈ ∂h t (w t ). 2. Compute s t ∈ (∂g t (w t ) -z t ). 3. Compute w t+1 using (1.13). ation k, Online DCA end for
	In this case, an Online DC programming problem can be described as follows. At the step t, one tries to find Initialization: let w 1 be an initial point in S for t = 1, 2, . . . , T do the best feasible point in S, denoted w t , by solving a DC program of form Compute z t ∈ ∂h t (w t ).
	3. With a fixed choice of subgradients of h, the sequence inf{F t (w) := G t (w) -H t (w) : w ∈ S} (1.8) Compute w t+1 , an optimal solution to the convex program
	{y k } is discrete i.e. it has only finitely many different elements. This leads to finite convergence of DCA. d) DCA's distinctive feature relies upon the fact that DCA deals with the convex DC components g and h but not with the DC function f itself. Moreover, a DC function f has infinitely many DC decompositions which have crucial min G t (w) -t i=t 0 z i , w : w ∈ S . (1.11) where the functions F t The ultimate goal is to generate the sequence {w t } t=1,...,T that minimizes the cumula-tive objective value until step T , defined as end for
	implications for the qualities (e.g. convergence speed, robustness, efficiency, T
	globality of computed solutions) of DCA. For a given DC program, the choice f t (w t ). (1.9)
	of optimal DC decompositions is still open. Of course, this depends strongly on the very specific structure of the problem being considered. t=1 1.2.3 ODCA: a proposed Online DCA based scheme Minimizing (1.9) depends on the update rule to generate the sequence of {w t } t=1,...,T , e) Similarly to the effect of DC decompositions on DCA, searching the good initial points for DCA is also an open question to be studied. more precisely what DC optimization problem is considered at each step. In this work, In Online DCA scheme, at each step, we need to solve the convex subproblem (1.11). we propose the following update rule However, in some practice problems, (1.11) cannot be solved explicitly or is solved
	w t+1 ∈ arg min with expensive computaions. Thus, we propose a particular Online DCA based scheme F t (w), t = 1, . . . , T, (1.10)
	w∈S where each convex subproblem (1.11) is solved by approximating by one iteration of
	where the objective function subgradient methods (see e.g. [111] for more details about subgradient methods).
	F t (w) := Particularly, at the step t, considering t 0 = t and R(w) = 0, the subproblem (1.11) t i=t 0 becomes f i (w) + R(w), min g t (w) -z t , w : w ∈ S , (1.12)

  f i and previous hypothesis vectors w i , i = 1, . . . , t -1. Then he observes a new DC function f t to assess the quality of w t . Finally he suffers a loss between the predicted answer and the correct one, defined by f t (w t ). The main goal of the Online DC programming problem is to generate the sequence of {w t } t=1,...,T that minimizes the cumulative suffered loss nonconvexity of F t . Let us assume that for i = 1, . . . , T , a DC decomposition of f i is g i -h i . It is evident to show that F t is a DC function with a DC decomposition defined asF t = G t -H t ,

	DC programming and Online DCA for Online learning 2.2.1 An introduction to Online DC programming and Online DCA for Online learning Throughout this chapter, we will formally describe Online DC programming and On-line DCA in the context of online learning, which is similar to Section 1.2.1. We further assume that T is the number of steps and the set of hypotheses, denoted by S, is closed convex in R n . In this context, an Online DC programming problem involves the set of hypotheses S and a sequence of loss functions {f t } t=1,...,T where each real-value func-tion f t : S → R is a DC function. In particular, at step t, when a question is incoming, the learner must choose a hypothesis vector w t ∈ S to predict an answer. The vector w t is computed as an optimal solution to a DC program, which depends mainly on t=1 f t (w t ). (2.1) As Section 1.2.1, in this chapter, we consider the generalized update rule as follows. w t+1 ∈ arg min w∈S F t (w), t = 1, . . . , T, (2.2) where the objective function F t (w) := t i=t 0 f i (w) + R(w), the function R : S → R is a convex regularization function, and t 0 ∈ {1, t}. It is worth noting that the update rule (2.2) covers existing update rules in online convex optimization such as FTL, RFTL mentioned in Section 2.1. However, (2.2) is difficult due to the where the function G t (w) = t i=t 0 g i (w) + R(w) and H t (w) = t i=t 0 h i (w). Thus, we derive the online version of DCA, namely Online DCA, which is summarized as follows. Online DCA Initialization: let w 1 be an initial point in S. for t = 1, 2, . . . , T do 1. Compute z t ∈ ∂h t (w t ). 2. Compute w t+1 ∈ arg min t i=t 0 g i (w) + R(w) -t i=t 0 z i , w : w ∈ S . (2.3) end for previous loss functions T 2.2.2 Online gradient descent: special version of Online DCA

  w,xt .

	It is evident that g t (4)	and h (4)

t are convex, positive functions on R n . With the sigmoid function (2.38), ODCA consists of, at the step t, first calculating the vector s t ∈ (∂g (4) t -∂h

Table 2 .

 2

		1 -Datasets used in our experiments
	Dataset Name	# Instances (T ) # Features (n)
	D1	a8a	32561	123
	D2	cod-rna	271617	8
	D3	colon-cancer	62	2000
	D4	covtype	581012	54
	D5	diabetes	768	8
	D6	ijcnn1	141691	22
	D7	magic04	19020	10
	D8	splice	3175	60
	D9	svmguide1	7089	4
	D10	w7a	49749	300

Table 2 .

 2 2 and Table 2.3.

Table 2 . 2 -

 22 Average mistake rate (upper row) and its standard deviation (lower row) obtained by ODCA-PiL1, ODCA-PiL2, ODCA-Sig and Perceptron, ROMMA, ALMA, OGD, PA. Bold (resp. underlining) values indicate the first best (resp. second best) results. In terms of the mistake rate: from Table2.2, we observe that ODCA-Sig is the best algorithm, ODCA-PiL2 is the second and ODCA-PiL1 is slightly more effi-

	Dataset ODCA-PiL1 ODCA-PiL2 ODCA-Sig Perceptron ROMMA ALMA OGD	PA
	D1	0.2088	0.1575	0.1574	0.2100	0.2249	0.1581 0.1577 0.2108
		0.001	0.001	0.001	0.001	0.002	0.001	0.001	0.002
	D2	0.1739	0.1176	0.1149	0.1749	0.1517	0.1994 0.1657 0.2074
		0.001	0.001	0.000	0.001	0.065	0.001	0.000	0.001
	D3	0.3088	0.2379	0.2379	0.3137	0.3771	0.4435 0.3032 0.2637
		0.039	0.046	0.050	0.043	0.086	0.056	0.060	0.041
	D4	0.4697	0.4237	0.4231	0.4697	0.4804	0.4839 0.4676 0.4835
		0.001	0.001	0.000	0.001	0.011	0.001	0.001	0.000
	D5	0.3194	0.2615	0.2621	0.3265	0.3072	0.2655 0.2586 0.3346
		0.015	0.008	0.007	0.013	0.015	0.010	0.007	0.016
	D6	0.1045	0.0705	0.0740	0.1062	0.1008	0.0699 0.0767 0.1023
		0.024	0.001	0.018	0.000	0.001	0.001	0.001	0.001
	D7	0.3593	0.2786	0.2775	0.3645	0.3365	0.3636 0.3557 0.3835
		0.007	0.002	0.001	0.002	0.034	0.003	0.003	0.003
	D8	0.2969	0.2329	0.2150	0.2732	0.2684	0.2283 0.2168 0.2617
		0.056	0.006	0.003	0.004	0.009	0.006	0.004	0.007
	D9	0.2492	0.2723	0.2026	0.2560	0.3037	0.2564 0.2466 0.3130
		0.007	0.116	0.002	0.004	0.032	0.004	0.010	0.005
	D10	0.1147	0.1005	0.1005	0.1151	0.1094	0.1028 0.1037 0.1051
		0.012	0.000	0.000	0.000	0.001	0.001	0.001	0.000

ii) Concerning CPU time: all algorithms run very fast and can be classified as follows: Perceptron and ODCA-PiL1 are the fastest algorithms, three algorithms ROMMA, ALMA and PA come next and finally, ODCA-Sig, ODCA-PiL2 and OGD. More specifically, Perceptron runs the fastest on all 10 datasets while ODCA-PiL1 is comparable with Perceptron -the ratio of gain of Perceptron versus ODCA-PiL1 varies from 1 to 1.036 times. As for ODCA-Sig and ODCA-PiL2, although they are the slowest, their CPU time is fairly small and acceptable on all datasets -the ratio of gain of Perceptron versus ODCA-Sig (resp. ODCA-PiL2) varies from 1.315 to 1.526 (resp. from 1.309 to 1.666) times.

  Euclidean space R d . According to the Online DCA scheme in Section 1.2.2, at each step, we need to solve a convex subproblem(1.11). In this chapter, we propose two particular Online DCA based schemes in which(1.11) is solved by approximating by one iteration of projected subgradient method and exponentiated subgradient method respectively

	Online DC pro-
	gramming and Online DCA
	3.3.1 ODCA-SG and ODCA-ESG: ODCA schemes for Pre-
	diction with Expert Advice

Recall that at step t, we have a DC function, denoted by f t , whose DC decomposition is g t -h t and the set of predictions S is the nonempty probability simplex in the

  2.1-2.4 in the Section 2.2.4. Suppose that L, U and D are respectively positive upper bounds of the sequence {s t } t=1,...,T in 2 -norm, ∞ -norm and {w t -u * } t=1,...,T in 2 -norm that is Next, the regret bound of ODCA-ESG can be archived as stated in Theorem 3.3. Theorem 3.3. Assume that ODCA-ESG generates the sequence {w t } t=1,...,T where Assumptions 2.1-2.3 are satisfied and w 1 [i] = 1/d, i = 1, . . . , d. Then, we have

	max t∈{1,...,T }	s t	2 ≤ L, sup
			Regret T ODCA-SG ≤	3DL(α + β) 2α	√ T	.
			Regret T ODCA-SG ≤	L 2 (α + β) (1 + log(T )) 2αγ	.

t∈{1,...,T } s t ∞ ≤ U and sup t∈{1,...,T } w t -u * 2 ≤ D. (3.5) Under these assumptions, the results of the regret bound of ODCA-SG can be achieved similarly in the analysis of ODCA in Section 2.2.4 (Theorems 2.1, 2.2), which is stated as the following theorems for ODCA-SG. Theorem 3.1. Assume that ODCA-SG generates the sequence {w t } t=1,...,T where Assumptions 2.1-2.3 are verified. Then, we have Theorem 3.2. Assume that ODCA-SG generates the sequence {w t } t=1,...,T where Assumptions 2.1-2.4 are verified. Then, we have

Table 3 . 2 -

 32 Datasets used in our experiments.

	Dataset Name	# Instances # Features (n)
	D1	a8a	32561	123
	D2	cod-rna	271617	8
	D3	colon-cancer	62	2000
	D4	covtype	581012	54
	D5	diabetes	768	8
	D6	german.number	1000	24
	D7	ionosphere	351	34
	D8	madelon	1549	500
	D9	mushrooms	8124	112
	D10	spambase	4601	57
	D11	svmguide1	7089	4

Table 3 . 3 -

 33 The best value of τ for ODCA-SG and ODCA-ESG during the parameter validation (τ < 0.5)

	Data	ODCA-SG ODCA-ESG
	a8a	0.40	0.40
	cod-rna	0.48	0.48
	colon-cancer	0.48	0.48
	covtype	0.38	0.38
	diabetes	0.48	0.48
	german.number	0.00	0.40
	ionosphere	0.40	0.26
	madelon	0.14	0.22
	mushrooms	0.40	0.18
	spambase	0.48	0.48
	svmguide1	0.38	0.38

  When an approximate function is used to learn a value function, tight performance bounds on greedy policies are needed to get good resulting control. Bellman Residual minimization (used at the first time in the work of Schweitzer and Seidman in 1985[START_REF] Schweitzer | Generalized polynomial approximations in markovian decision processes[END_REF] for computing approximate state value functions assuming the full knowledge of a finite MDP) is an efficient approach in RL which can meet such requirement (see e.g.[START_REF] Maillard | Finite sample analysis of Bellman residual minimization[END_REF][START_REF] Munos | Performance Bounds in L p norm for Approximate Value Iteration[END_REF][START_REF] Williams | Tight performance bounds on greedy policies based on imperfect value functions[END_REF]). In fact, a common practice of RL applications is to minimize the Bellman residual and then use the corresponding greedy policy.

	Reinforcement Learning: Introduction and Related Works
	survey on these techniques is given in [17, 127, 129]. In the framework of the value
	function based approach, several value function approximation algorithms using dif-
	ferent function approximators were proposed. These RL algorithms either determine
	(incrementally) a value for each state through satisfaction of the Bellman equation
	(e.g. approximate Q-learning [115, 118, 120, 126], approximate SARSA algorithms
	[114, 117], adaptive learning) or minimize directly a norm of empirical Bellman resid-
	ual/Optimal Bellman residual (the direct Bellman residual minimization approach,
	e.g. [3, 5, 38, 81, 86, 98, 106, 107, 113]).

To deal with large state and action spaces in MDPs, RL methods with function approximation techniques have been intensively studied in the literature, a comprehensible The Bellman operator is a widely used concept in DP. In RL, the Bellman operator is defined on the set of state value functions and/or the set of state-action value functions. As we focus on state-action value functions, all notations presented here concern only state-action value functions (similar notations exist for state value functions). The Bellman operator on state-action functions is defined by B π : R S×A → R S×A

Table 5 . 1 -

 51 Summary of all comparative algorithms

	Algorithm	Considered problem	Reference
	1 -DCA	1 -norm (5.7)	our proposal
	2 -DCA	2 -norm (5.8)	our proposal
	∞ -DCA	∞ -norm (5.10)	our proposal
	cc-DCA	concave (5.12)	our proposal
	DCA 1	1 -norm (5.7)	[98]
	DCA 2	2 -norm (5.8)	[98]
	FQI	-	[32]
	LSPI	-	[52]

Table 5 . 2 -

 52 1 -DCA, 2 -DCA and DCA 1 , DCA 2 on 10 different Garnet(400,5,5) problems and 2 transition sample datasets for each Garnet with the same tolerance ε = 10 -6 and the same starting point as proposed in Section 5.3.6. The average results of T A , std A , Iter, CPU and OBR obtained by the algorithms with different numbers of transition samples N (N ∈ {5000, 6000, 7000, 8000}) are reported in Table5.4. Average results of T A , std A , Iter and CPU in seconds obtained by the six versions of DCA on 50 runs (corresponding to 10 different Garnet(N S = 100,N A = 5,N B = 5) problems and 5 transition sample datasets for each Garnet) with different numbers of transition samples N . Bold values indicate the best results.

	A unified DC programming framework and efficient DCA based approaches
	for large scale batch Reinforcement Learning			119
	N		1 -DCA	2 -DCA	∞ -DCA	cc-DCA	DCA 1	DCA 2
	500	T A	0.474	0.758	0.639	0.562	0.481	0.536
		std A	0.06	0.09	0.10	0.07	0.06	0.06
		CPU	0.84	10.8	14.4	0.65	7.58	8.72
		Iter	7.10	17.3	4.78	6.98	6.30	5.86
	800	T A	0.286	0.425	0.425	0.391	0.416	0.485
		std A	0.05	0.09	0.08	0.07	0.05	0.07
		CPU	1.57	14.7	38.7	1.01	16.2	16.1
		Iter	5.78	14.6	3.66	5.72	7.40	5.92
	1000	T A	0.221	0.275	0.409	0.288	0.393	0.458
		std A	0.05	0.07	0.07	0.07	0.05	0.08
		CPU	3.06	16.6	27.7	1.27	20.4	20.9
		Iter	5.26	12.5	3.98	4.64	7.60	5.68
	1300	T A	0.152	0.190	0.321	0.192	0.383	0.450
		std A	0.03	0.05	0.05	0.04	0.06	0.08
		CPU	4.21	23.6	32.4	1.80	17.8	45.8
		Iter	4.68	10.9	3.56	4.08	7.64	5.84
	1500	T A	0.141	0.150	0.287	0.173	0.386	0.428
		std A	0.03	0.04	0.06	0.04	0.06	0.07
		CPU	5.78	27.2	43.9	1.36	16.1	50.6
		Iter	4.18	10.5	3.50	4.10	7.22	6.24
	1800	T A	0.136	0.112	0.310	0.177	0.356	0.444
		std A	0.03	0.03	0.09	0.04	0.04	0.09
		CPU	7.33	41.9	92.9	1.79	48.3	108
		Iter	5.08	9.90	3.82	3.80	7.58	5.82
	2000	T A	0.126	0.103	0.294	0.173	0.369	0.432
		std A	0.03	0.02	0.06	0.04	0.06	0.08
		CPU	9.11	34.9	129	1.90	61.5	117
		Iter	5.34	9.82	3.26	3.48	8.18	6.06
	2300	T A	0.125	0.084	0.275	0.155	0.358	0.411
		std A	0.03	0.02	0.07	0.03	0.06	0.07
		CPU	5.69	35.0	201	2.25	65.5	146
		Iter	5.06	9.50	3.58	3.26	8.00	5.92
	2500	T A	0.112	0.069	0.259	0.137	0.376	0.433
		std A	0.03	0.02	0.07	0.04	0.06	0.08
		CPU	6.07	51.7	269	2.44	50.4	165
		Iter	5.12	9.36	4.08	2.86	7.48	6.38

Table 5 . 3 -

 53 Average results of T A , std A , Iter and CPU in seconds obtained by the six versions of DCA on 25 runs (corresponding to 5 different Garnet(N S ,5,5) problems (N S ∈ {225, 324}) and 5 transition sample datasets for each Garnet) with different numbers of transition samples N . Bold values indicate the best results.

	Garnet	N	1 -DCA 2 -DCA ∞ -DCA cc-DCA DCA 1 DCA 2
	(N S , N A , N B )
	(225,5,5) 4000 T A	0.225 0.162	0.572	0.360 0.369 0.370
		std A	0.03	0.03	0.09	0.04 0.03 0.05
		CPU	38.0	77.8	793	9.57	216	361
		Iter	9.84	9.36	3.64	4.76 9.88 9.24
		5000 T A	0.208 0.145	0.640	0.328 0.354 0.359
		std A	0.02	0.03	0.08	0.05 0.03 0.06
		CPU	20.1	126	1175	8.49	250	374
		Iter	10.4	9.00	2.84	4.48 10.0 8.44
		6000 T A	0.188 0.145	0.632	0.307 0.360 0.328
		std A	0.03	0.03	0.13	0.04 0.04 0.06
		CPU	22.4	199	1136	9.40	204	505
		Iter	8.84	9.28	2.00	3.76 9.84 8.92
		7000 T A	0.197 0.135	0.614	0.308 0.352 0.336
		std A	0.03	0.02	0.10	0.05 0.03 0.07
		CPU	27.5	143	1765	11.3	298	585
		Iter	8.68	9.16	2.00	3.64 9.40 8.72
	(324,5,5) 4000 T A	0.271 0.181	0.697	0.425 0.353 0.325
		std A	0.03	0.02	0.12	0.04 0.04 0.04
		CPU	65.2	153	773	10.0	753 1307
		Iter	11.2	13.4	4.44	6.80 12.0	9.6
		5000 T A	0.249 0.148	0.654	0.397 0.355 0.346
		std A	0.04	0.03	0.07	0.05 0.03 0.06
		CPU	93.9	231	994	10.9	748 1259
		Iter	10.3	12.4	3.04	5.48 10.6 8.52
		6000 T A	0.241 0.121	0.730	0.384 0.345 0.295
		std A	0.03	0.02	0.08	0.04 0.04 0.04
		CPU	116	210	1150	11.0	789 1377
		Iter	9.48	11.3	2.00	4.44 10.2 9.60
		7000 T A	0.234 0.106	0.712	0.368 0.358 0.300
		std A	0.03	0.02	0.07	0.04 0.03 0.05
		CPU	133	300	1830	12.3	842 1428
		Iter	9.96	11.0	2.00	4.08 9.44 10.0

Table 5 .

 5 4 -Average results of T A , std A , Iter, CPU in seconds and OBR obtained by 1 -DCA and DCA 1 (resp. 2 -DCA and DCA 2 ) on 20 runs (corresponding to 10 Garnet(400,5,5) problems and 2 transition sample datasets for each Garnet) with different numbers of transition samples N . Bold values indicate the best results in the same 1 -norm (resp. 2 -norm) problem.

	N		1 -DCA DCA 1	2 -DCA DCA 2
	5000	T A	0.294	0.459	0.217	0.404
		std A	0.02	0.02	0.02	0.03
		CPU	100	400	155	1813
		Iter	13.8	12.6	15.6	11.6
		OBR	0.070	0.118	0.211	0.318
	6000	T A	0.294	0.480	0.175	0.398
		std A	0.02	0.02	0.01	0.02
		CPU	172	395	261	1902
		Iter	14.9	10.8	16.4	11.1
		OBR	0.073	0.129	0.205	0.299
	7000	T A	0.298	0.480	0.145	0.400
		std A	0.02	0.02	0.01	0.02
		CPU	236	378	406	1741
		Iter	12.6	8.60	15.6	13.7
		OBR	0.073	0.132	0.215	0.292
	8000	T A	0.280	0.488	0.139	0.395
		std A	0.02	0.03	0.01	0.03
		CPU	319	431	541	1901
		Iter	12.6	8.75	15.0	13.1
		OBR	0.074	0.138	0.220	0.304

  2, 1 -DCA and 2 -DCA are the two best algorithms, cc-DCA is the third, ∞ -DCA is the next and DCA 1 , DCA 2 are the worst. In particular, 1 -DCA is the best in 5/9 cases (when N ≤ 1500) while 2 -DCA is the best in other four cases. The ratio of gain on T A of 1 -DCA versus cc-DCA, ∞ -DCA, DCA 1 , DCA 2 and 2 -DCA varies from 15.6% to 27.1%, from 25.8% to 57.1%, from 1.45% to 70.2%, from 11.5% to 74.1% and from 6% to 37.4%, respectively. As for 2 -DCA, it outperforms cc-DCA, ∞ -DCA, DCA 1 in 7/9 cases (except when N = 500 and N = 800) and DCA 2 in 8/9 cases (except when N = 500) -the gain versus cc-DCA, ∞ -DCA, DCA 1 , DCA 2 varies from 1.04% to 49.6%, from 32.7% to 73.3%, from 30% to 81.6%, from 12.3% to 84%, respectively. cc-DCA is more efficient than DCA 1 , DCA 2 in 8/9 cases (except when N = 500) and ∞ -DCA in all casesthe gain versus DCA 1 , DCA 2 , ∞ -DCA varies from 6.01% to 63.5%, from 19.3% to 68.3%, from 8% to 47.1%, respectively. Finally, ∞ -DCA also outperforms DCA 1 (resp. DCA 2 ) in 6/9 cases (resp. 8/9 cases) -the gain varies from 13.9% to 31.1% (resp. from 10.6% to 40.1%).• For the larger Garnet problems (N S ∈ {225, 324} and N ≥ 4000) in Table5.3, the quality of the algorithms is classified in the following order: 2 -DCA, 1 -DCA, cc-DCA, DCA 1 , DCA 2 , ∞ -DCA. Particularly, 2 -DCA gives the best T A in all cases -the ratio of gain versus 1 -DCA, cc-DCA, ∞ -DCA, DCA 1 , DCA 2 varies from 22.8% to 54.7%, from 52.7% to 71.1%, from 71.6% to 85.1%, from 48.7% to 70.3%, from 44.3% to 64.6%, respectively. As for 1 -DCA, the gain on T A versus cc-DCA, ∞ -DCA, DCA 1 , DCA 2 varies from 36% to 38.7%, from 60.6% to 70.2%, from 23.2% to 47.7%, from 16.6% to 42.6%, respectively. The T A of cc-DCA is better than that of both DCA 1 and DCA 2 in 4/8 cases with the ratio of gain from 2.43% to 14.7% and larger in other four cases with the ratio of loss from 2.71% to 23.5%. ∞ -DCA seems to be inefficient in these large problems.ii) Concerning CPU time: cc-DCA is the fastest algorithm, 1 -DCA and 2 -DCA come next, and then DCA 1 , DCA 2 and finally ∞ -DCA. For the medium Garnet problems with N S = 100 (see Table5.2), the ratio of gain of cc-DCA versus 1 -DCA, 2 -DCA, ∞ -DCA varies, respectively, from 1.29 to 4.79, from 13.1 to 23.4, from 18 to 110.2 times. The gain is more important versus DCA 1 (resp. DCA 2 ): it varies from 9.88 to 32.3 (resp. from 13.4 to 67.6) times. 1 -DCA is also fast -the gain versus DCA 1 varies from 2.78 to 11.5 times. Not surprisingly, DCA 2 , 2 -DCA and ∞ -DCA consume much more time due to the complexity of convex subproblems, but 2 -DCA runs faster than DCA 2 and ∞ -DCA in most cases -the gain of 2 -DCA versus DCA 2 varies from 1.09 to 4.17 times in 8/9 cases (when N ≥ 800). For the larger Garnet problems with N S ∈ {225, 324} (see Table 5.3), the ratio of gain of cc-DCA is larger -the gain versus 1 -DCA, 2 -DCA, ∞ -DCA, DCA 1 , DCA 2 varies, respectively, from 2.36 to 10.8, from 8.13 to 24.3, from 82.8 to 156.1, from 21.7 to 75.3, from 37.7 to 131 times. As for 1 -DCA, the gain versus 2 -DCA, ∞ -DCA, DCA 1 , DCA 2 varies, respectively, from 1.81 to 8.88, from 9.91 to 64.1, from 5.68 to 12.4, from 9.5 to 22.5 times. 2 -DCA runs faster than ∞ -DCA, DCA 1 , DCA 2 with the ratio of gain from 4.3 to 12.3, from 1.02 to 4.92, from 2.53 to 8.54 times respectively. Finally, DCA 1 , ∞ -DCA, DCA 2 run slowly and sometimes exceed the maximum CPU time. b) 1 -DCA versus DCA 1 (Table 5.4) i) Regarding the quality of solutions: 1 -DCA is much more efficient than DCA 1 for all Garnet problems in terms of both T A and std A . The ratio of gain on T A varies from 35.9% to 42.6%. Moreover, the OBR of 1 -DCA is always smaller than that of DCA 1 . ii) As for CPU time: 1 -DCA runs faster than DCA 1 in all considered Garnet problems -the ratio of gain varies from 1.35 to 4 times. c) 2 -DCA versus DCA 2 (Table 5.4) i) 2 -DCA enhances T A and OBR more significantly than DCA 2 in all cases. Particularly, the ratio of gain on T A of 2 -DCA versus DCA 2 varies from 46.2% to 64.8%.

Table 5 .

 5 5 -Comparative results of the proposed starting point θ (1) and the zero starting point θ (2) for our algorithms 1 -DCA, 2 -DCA, ∞ -DCA and cc-DCA in terms of T A on 50 runs (corresponding to 10 different Garnet(100,5,5) problems and 5 transition sample datasets for each Garnet) with different numbers of transition samples N . Bold values indicate the best results in each algorithm.

Table 5 . 6 -

 56 Average results of T A , std A and CPU in seconds obtained by 1 -DCA, 2 -DCA, cc-DCA and LSPI, FQI on 500 runs (corresponding to 50 different Garnet(100,5,5) problems and 10 transition sample datasets for each Garnet) with different numbers of transition samples N . Bold values indicate the best results.

	N		1 -DCA	2 -DCA cc-DCA	FQI	LSPI
	500	T A	0.492	0.762	0.578	1.007	0.478
		std A	0.07	0.09	0.09	0.00	0.08
		CPU	0.89	10.5	0.58	4.10	22.9
	1000	T A	0.233	0.288	0.301 0.210	0.211
		std A	0.05	0.07	0.06	0.05	0.05
		CPU	3.04	16.5	1.38	11.4	40.3
	1500	T A	0.139	0.146	0.173 0.126	0.127
		std A	0.03	0.04	0.05	0.04	0.04
		CPU	6.26	25.9	1.32	22.3	44.6
	2000	T A	0.130	0.100	0.166 0.085	0.085
		std A	0.03	0.03	0.04	0.02	0.03
		CPU	10.5	44.3	1.89	37.0	45.0
	2500	T A	0.122	0.081	0.148 0.069	0.069
		std A	0.03	0.02	0.04	0.02	0.02
		CPU	5.63	48.5	2.48	31.7	58.0

Table 5 . 7 -

 57 Average results of T A , std A and CPU in seconds obtained by 1 -DCA, 2 -DCA, cc-DCA and LSPI, FQI on 100 runs (corresponding to 10 different Gridworld problems and 10 transition sample datasets for each Gridworld) with different numbers of transition samples N . Bold values indicate the best results.Comments on numerical results (Table5.7 and Fig.5.3)In terms of quality, 2 -DCA gives the best error performance T A in all cases, LSPI and FQI come next, and afterward 1 -DCA, then cc-DCA. In particular, the gain on T A of 2 -DCA versus LSPI (resp. FQI) varies from 10.3% to 22.7% (resp. from 8.97% to 22.3%). As for the rapidity, cc-DCA is always the fastest algorithm, especially the ratio of gain versus LSPI (resp. FQI) varies from 33.4 to 125.9 (resp. from 12.96 to 14.8) times. In all cases, LSPI consumes much more time than other algorithms. Among three DCA based algorithms 2 -DCA is the slowest, meanwhile it is faster than LSPI (resp. FQI) from 2.7 to 16.27 (resp. from 1.14 to 1.67) times.

	N		1 -DCA cc-DCA	2 -DCA	FQI LSPI
	900	T A	0.820	0.826	0.629 0.691 0.702
		std A	0.02	0.02	0.06	0.06	0.07
		CPU	0.75	0.69	5.34	8.94	86.9
	1100	T A	0.820	0.829	0.610 0.695 0.705
		std A	0.02	0.01	0.06	0.07	0.06
		CPU	0.98	0.78	7.50	11.0	83.3
	1300	T A	0.814	0.827	0.587 0.688 0.690
		std A	0.02	0.01	0.05	0.06	0.06
		CPU	1.31	0.90	9.94	12.9	79.2
	1500	T A	0.810	0.826	0.577 0.712 0.710
		std A	0.03	0.00	0.04	0.06	0.06
		CPU	1.77	1.01	11.1	14.8	69.1
	1700	T A	0.812	0.825	0.569 0.711 0.708
		std A	0.02	0.00	0.04	0.06	0.07
		CPU	1.65	1.12	13.4	16.6	57.1
	1900	T A	0.818	0.825	0.561 0.710 0.724
		std A	0.02	0.00	0.03	0.06	0.06
		CPU	1.93	1.33	14.9	18.4	51.3
	2100	T A	0.815	0.825	0.556 0.716 0.720
		std A	0.02	0.00	0.02	0.06	0.07
		CPU	2.24	1.44	17.8	20.3	48.1

Table 6 . 1 -

 61 Average/standard deviation number of steps at the last episode and CPU time (in seconds) of two ODCA based algorithms and their alternating version over 50 runs in mountain car problems. Bold values indicate the best results.

	Algorithms Number of steps at the last episode CPU
	ODCA	295.4.2 ± 111.3	52.6
	ODCA1	219.1 ± 54.1	42.5
	AODCA	278.9 ± 90.84	43.3
	AODCA1	182.3 ± 39.1	25.5

Table 6 .

 6 2 -Average/standard deviation number of steps at the last episode and CPU time (in seconds) of AODCA1 and Q-learning, SARSA over 100 runs in mountain car problems. Bold values indicate the best results.

	Algorithms Number of steps at the last episode CPU
	SARSA	210.2 ± 54.2	24.1
	Q-learning	222.1 ± 55.7	26.1
	AODCA1	184.8 ± 35.0	26.8

Table 6 . 3 -

 63 Total number of steps of all episodes and CPU time (in seconds) and the ratio of CPU time to the total number of steps (denoted by CPU/step) of AODCA1 and Q-learning, SARSA over 100 runs in pole balancing problems.

	Algorithms Total number of steps CPU CPU/step
	SARSA	637 899	26.0	4.0e-5
	Q-learning	780 088	37.3	4.7e-5
	AODCA1	791 213	37.9	4.7e-5

Figure 6.3 -Average/standard deviation number of steps of AODCA1 and Qlearning, SARSA in the number of episodes over 100 runs in mountain car problems.

  Thi. A DCA Approach for the Stochastic Shortest Path Problem in Vehicle Routing. Accepted by IESM 2017: 7th International Conference on Industrial Engineering and Systems Management. Applications to Stochastic Shortest Path problems: DCA Approaches via Cardinality Minimization and Batch Reinforcement Learning main categories: convex approximation, nonconvex approximation and nonconvex exact reformulation. Le Thi et al. [68] developed a unified DC (Difference of Convex functions) approximation and DCA (DC

Table 7 .

 7 7.2 DCA Approaches for the reformulations of PT model-based SSP problems 7.2.1 The first reformulation: cardinality problem (7.2) A DC formulation of the cardinality problem (7.2) By introducing the slack variable y, the following proposition (whose proof is evident) will justify our considered optimization formulation (7.3) whose feasible set is bounded. Applications to Stochastic Shortest Path problems: DCA Approaches via Cardinality Minimization and Batch Reinforcement Learning Table 7.1 -Accuracy (%) of Card-DCA and 1 -norm algorithm ( 1 ), reweighted-1 algrorithm (re-1 ) on 100 runs with the different deadline coefficients κ and random distributions S 1 is the size of training set on each arc. Bold values are the best results in each column. Applications to Stochastic Shortest Path problems: DCA Approaches via Cardinality Minimization and Batch Reinforcement Learning 155 2 -CPU time (in seconds) of enumeration method, Card-DCA, 1 and re-1 on 100 runs with different random distributions in case κ = 1.2 DCA 1 re-1 Card-DCA 1 re-1 Card-DCA 1 re-1 DCA 1 re-1 Card-DCA 1 re-1 Card-DCA 1 re-1 DCA 1 re-1 Card-DCA 1 re-1 Card-DCA 1 re-1

	Normal distribution
		S 1 = 100		S 1 = 500		S 1 = 1000
	κ Card-DCA	1	re-1 Card-DCA	1	re-1 Card-DCA	1	re-1
	0.2	99.16	99.14 99.14	99.60	99.60 99.60	99.60	99.59 99.59
	0.4	99.54	99.53 99.53	99.90	99.89 99.89	99.90	99.89 99.89
	0.6	99.73	99.72 99.72	99.98	99.98 99.98	99.98	99.98 99.98
	0.8	99.81	99.80 99.80	100	100 100	100	100 100
	1.0	99.87	99.87 99.87	100	100 100	100	100 100
	1.2	99.91	99.90 99.90	100	100 100	100	100 100
	Gamma distribution
		S 1 = 100		S 1 = 500		S 1 = 1000
	κ Card-DCA	1	re-1 Card-DCA	1	re-1 Card-DCA	1	re-1
	0.2	99.77	99.71 99.71	99.87	99.61 99.61	99.87	99.83 99.83
	0.4	99.82	99.78 99.78	99.75	99.71 99.71	99.92	99.71 99.71
	0.6	99.90	99.85 99.85	99.84	99.80 99.80	99.84	99.80 99.80
	0.8	99.94	99.90 99.90	99.90	99.86 99.86	99.90	99.86 99.86
	1.0	99.86	99.92 99.92	99.96	99.92 99.92	99.96	99.92 99.92
	1.2	100	99.94 99.94	99.98	99.94 99.94	99.98	99.94 99.94
	Lognormal distribution
		S 1 = 100		S 1 = 500		S 1 = 1000
	κ Card-DCA	1	re-1 Card-DCA	1	re-1 Card-DCA	1	re-1
	0.2	95.98	95.89 95.89	96.48	95.66 95.66	97.84	96.99 96.99
	0.4	96.43	96.21 96.22	97.01	96.32 96.32	98.18	97.41 97.41
	0.6	96.88	96.65 96.67	97.50	96.80 96.80	98.51	97.72 97.72
	0.8	97.32	97.06 97.07	97.74	97.15 97.15	98.05	97.99 97.99
	1.0	96.82	97.30 97.31	97.37	97.48 97.48	98.19	98.24 98.24
	1.2	97.28	96.47 96.48	97.63	97.28 97.28	98.38	97.91 97.91

Proposition 7.1. If x * is an optimal solution to the problem (7.2), then (x * , y * ) is an optimal solution to the following problem *

Table 7 .

 7 Applications to Stochastic Shortest Path problems: DCA Approaches via Cardinality Minimization and Batch Reinforcement Learning 157 3 -Accuracy (%) of RL-DCA and Card-DCA on 100 (o,d) pairs with the different grids n × n, the different number of samples on each link S, the different deadline parameters β. Bold values are the best results.

		n = 20, S = 200	n = 30, S = 200	n = 40, S = 200
	β	RL-DCA Card-DCA RL-DCA Card-DCA RL-DCA Card-DCA
	1.05	75.23	74.78	84.29	83.34	83.19	77.34
	1.10	88.85	88.59	97.14	93.60	97.22	97.14
	1.15	95.46	92.70	99.50	99.0	99.64	99.243
		n = 20, S = 500	n = 30, S = 500	n = 40, S = 500
	β	RL-DCA Card-DCA RL-DCA Card-DCA RL-DCA Card-DCA
	1.05	80.69	77.43	78.88	78.03	80.48	76.38
	1.10	95.66	93.05	95.65	91.82	96.45	94.04
	1.15	99.20	98.11	99.32	98.13	99.38	98.65

Table 7 .

 7 4 -CPU time (in seconds) of RL-DCA and Card-DCA on 100 (o,d) pairs with the different grids n × n, the different number of samples on each link S, the different deadline parameters β. Bold values are the best results.We have studied nonconvex, nonsmooth programming approaches based on DC programming and DCA for solving a class of SSP problems in vehicle routing for a single independent vehicle. In fact, the considered SSP problems can be reformulated as a cardinality optimization problem with zero-one variables and an RL problem. As for Applications to Stochastic Shortest Path problems: DCA Approaches via Cardinality Minimization and Batch Reinforcement Learning the cardinality problem, we have used a DC approximation approach for the cardinality term. Exploiting an exact penalty technique in DC programming, we have dealt with the zero-one variables and obtained an exact reformulation of the corresponding approximate problem. Employing DCA for the resulting reformulation problem, we have developed a fast, efficient DCA scheme for the considered cardinality problem.

		n = 20, S = 200	n = 30, S = 200	n = 40, S = 200
	β	RL-DCA Card-DCA RL-DCA Card-DCA RL-DCA Card-DCA
	1.05	1.35	0.17	3.53	1.67	9.96	10.9
	1.10	1.33	0.18	3.68	1.25	10.1	9.36
	1.15	1.37	0.16	3.74	0.83	10.3	6.23
		n = 20, S = 500	n = 30, S = 500	n = 40, S = 500
	β	RL-DCA Card-DCA RL-DCA Card-DCA RL-DCA Card-DCA
	1.05	3.91	0.82	5.17	5.28	11.0	11.7
	1.10	4.05	0.69	5.29	4.28	11.3	9.16
	1.15	4.21	0.50	5.45	2.97	11.7	6.72
	7.4 Conclusions

p -norm (p = 2). Nous proposons un algorithme en ligne basé sur DCA (ODCA) qui a la propriété de stabilité en ligne. Nous indiquons qu'une classe d'algorithmes de gradient en RL est un cas particulier de notre schéma ODCA. Nous suggérons également une version alternative d'ODCA pour exploiter la connaissance des échantillons. Les résultats numériques sur deux problèmes benchmark -le problème de Mountain car et Pole balancing -indiquent l'efficacité de nos approches en comparaison avec deux algorithmes standard d'apprentissage par renforcement.

: w ∈ S .

http://www.ics.uci.edu/ ~mlearn/MLRepository.html

https://www.csie.ntu.edu.tw/ ~cjlin/libsvmtools/datasets/

-DCA, ∞ -DCA and cc-DCA have the same convergence properties (almost always converge to local solutions after a finite number of iterations) and they require solving one linear program at each iteration. Hence, in terms of complexity, we can say that

Remerciements

Conclusions

Bibliography

Cette thèse a été réalisée au sein du

List of Figures

List of Tables

Online Learning and Applications to Online Binary Linear Classification

Second piecewise linear function

In this section, we propose another DC approximation function which is also of form of piecewise linear function. Specifically, at step t, we observe the function f

(3) t : R n → [0, 1] defined as follows:

where τ 2t , τ 1t are two positive parameters. Obviously, (2.24) is always true for the function f

(3)

t . The set M PiL2 is defined by

t (w t ) > 0} = {t ∈ {1, . . . , T } : y t w t , x t < τ 1t } (2.36) and the set N PiL2 := {t ∈ {1, . . . , T } : -τ 2t ≤ y t w t , x t < τ 1t }.

is a DC function whose DC decomposition is

t (w) -h

t (w), w ∈ R n , where g

t (w) := g

t (w) if t ∈ N PiL2 ,

t (w) := h

(1)

Applying ODCA with this loss function leads us at step t ∈ N PiL2 to compute z t ∈ ∂h

t (w t ), s t ∈ (∂g

t (w t ) -z t ) and w t+1 based on (2.8).

Compute ∂g

t : from (2.29) and (2.30), for t ∈ M PiL2 , we choose z t ∈ ∂h (3) t (w t ), r t ∈ ∂g (3) t (w t ) and s t = r t -z t ∈ (∂g (3) t (w t ) -z t ) as follows: z t = 0,

Choose the parameters τ 1t , τ 2t and learning rate η t : we propose the following choice: τ 1t = τ 1 x t , τ 2t = τ 2 x t and η t = C/ √ t for all t where τ 1 , τ 2 and C are positive tuning parameters.

Online DCA for Prediction with Expert Advice

Remark that we have to choose the parameters τ 2t such that (3.6) holds true. However, it is never true if there exists a step t such that w t , v t = ρ, since t (p t , y t ) = 1 and f (2) t (w t ) = 0. In such a case, we investigate another DC function, namely f

and its DC decomposition is

t (w) -h

t (w), where g

t (w) = max 0,

t (w) = max 0, w, v t -ρ ρ -τ 3t .

Obviously, we have f

t (w t ) = 1 = t (p t , y t ). For both DC functions, we define that the step t belongs to M if and only if w t , v t ≥ ρ.

Compute ∂g

(2)

t : we have ∂g (2)

Thus, we choose s t = r t -z t ∈ (∂g

t -∂h

t )(w t ) as follows:

• Choose the parameter τ 2t : in order to satisfy (3.6), we propose the following choices

where τ 2 < ρ is a positive tuning parameter.

Online DCA for Prediction with Expert Advice Algorithm 3.2 ODCA-ESG for prediction with expert advice (ODCA-ESG)

Initialize: let w 1 be an initial point, η be the constant step size, τ 1 , τ 2 , τ 3 be positive numbers less than ρ. for t = 1, 2, . . . , T do if y t = p t (w t ) then

end for end if end for

We also assume that there exists a vector u * ∈ S such that for all t = 1, . . . , T ,

where the set Proof. For each t ∈ M, since v t = 0, the set I t is not empty. From (3.12), it is easy to check that u * ∈ arg min w∈S f

t (w) and thus, Assumption 2.1 is satisfied. It is worth noting that u * = w t for all t. Indeed, assume the contrary that is some step t such that u * = w t , since t ∈ M then u * , v t < ρ which contradicts (3.12) that u * , v t ≥ ρ. Now, we will show that Assumptions 2.2-2.4 can be satisfied.

• As for Assumption 2.2, let us define the function g

t -z t , • . From (3.11) and the definition of g

t , we derive that

Online DCA for Prediction with Expert Advice

Taking a t ∈ dom ∂p t , b t ∈ ∂p t (a t ), we define the affine minorization of the polyhedral convex function p t as follows

It is evident that p t + l - t and l - t are nonnegative and convex on R d , so are g t and h t .

According to a particular Online DCA based scheme in Section 1.2.3, at the step t, we solve the subproblem (1.11) by approximating by one iteration of subgradient method. Thus, θ t+1 will be updated as

where η t is a step size, the vector v t ∈ (∂g t (θ t ) -z t ), z t ∈ ∂h t (θ t ).

Compute ∂h t (θ): from the definition of h t and l - t , we have

= co A (t,jt) : j t ∈ I t (θ)},

Hence, we can take a subgradient z t ∈ ∂h t (θ t ) as the following procedure.

+ Let j t be an index in I t (θ t ) defined by (6.4), then b t = A (t,jt) .

+ Let j t be an index in I t (a t ), then b t = A (t,j t ) .

Online DCA for Reinforcement Learning 133

+ z t is computed by

Compute ∂g t (θ): from the definition of g t , we have

Similarly, we can take a subgradient u t ∈ ∂g t (θ t ) as the following procedure.

+ u t is computed by

From (6.5) and (6.6), v t ∈ (∂g t (θ t ) -z t ) is chosen as

where

At last, Online DCA applied to (6.2) can be described as Algorithm 6.1 (ODCA).

Algorithm 6.1 Online DCA for solving (6.2) (ODCA)

Initialization: let {η t } be the sequence of step sizes, θ 1 be the initial point for t = 1, 2, . . . do 1. Choose j t ∈ I t (θ t ) and j t ∈ I t (a t ) using (6.4). 2. Compute v t ∈ (∂g t (θ t ) -z t ) with z t ∈ ∂h t (θ t ) using (6.8).

3. Compute θ t+1 using (6.

3). end for

The proposed ODCA algorithm enjoys an interesting property as stated Theorem 6.1. The proof technique of this theorem rests partly on the works [START_REF] Saha | The interplay between stability and regret in online learning[END_REF]. Algorithm 6.2 ODCA when a t = θ t for all t (ODCA1) Initialization: let {η t } be the sequence of step sizes, θ 1 be the initial point for t = 1, 2, . . . , T do 1. Choose j t ∈ I t (θ t ) using (6.4).

2. Compute v t ∈ (∂g t (θ t ) -z t ) with z t ∈ ∂h t (θ t ):

2. Compute θ t+1 using (6.3). end for

Alternating Online DCA versions

To exploit the knowledge of transition sample (S t , A t , S t , r t ) for each update step (6.3), we propose alternating versions of ODCA. Indeed, the computation of v t (6.8) can be rewritten explicitly as

When we change the transition at the step t that is (S t , a) = (S t , A t ), the procedure to update θ t+1 can be performed alternatively in terms of φ(S t , A t ), φ(S t , a jt ), φ(S t , a j t )

as Figure 6.1. In particular, the vector θ t+1 for the current state-action (S t , A t ) is used to update θ t+1 for the next state-action (S t , a). Thus, the quality of θ t+1 can be improved.

Consequently, we derive an alternating version of ODCA (resp. ODCA1) given by Algorithm 6.3 (AODCA) (resp. Algorithm 6.4 (AODCA1)).

Algorithm 6.3 Alternating Online DCA for solving (6.2) (AODCA)

Initialize: let {η t } be the sequence of step sizes, θ 1 be the initial point for t = 1, 2, . . . , T do 1. Choose j t ∈ I t (θ t ) and j t ∈ I t (a t ) using (6.4).

2. Compute θ t+1 using Procedure 1. end for Online DCA for Reinforcement Learning 1: Input: sample (S t , A t , S t , r t ), basic function φ, weight θ t , indices j t , j t , and

10:

end if 12: else if a j t = a jt then 13: 14:

18:

else if (S t , a j t ) = (S t , A t ) then 26:

else 29:

end if 33: end if Figure 6.1 -Procedure 1: compute θ t+1 in the alternating version of ODCA Algorithm 6.4 AODCA when a t = θ t for all t (AODCA1) Initialize: let {η t } be the sequence of step sizes, θ 1 be the initial point for t = 1, 2, . . . , T do 1. Choose j t ∈ I t (θ t ) using (6.4).

2. Compute θ t+1 by the following procedure if (S t , a jt ) = (S t , A t ) then

end if end for

Numerical experiments

In the numerical experiments, we describe the performance of the proposed online algorithms ODCA, ODCA1 and their alternating versions AODCA, AODCA1, and compare our notable algorithm with some standard RL algorithms: Q-learning [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]126], SARSA [START_REF] Singh | Convergence results for single-step on-policy reinforcement-learning algorithms[END_REF][START_REF] Sutton | Generalization in reinforcement learning: Successful examples using sparse coarse coding[END_REF]. Our experiment consists of two well-known benchmarks: mountain car problems [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF] and pole balancing problems [START_REF] Barto | Neuronlike adaptive elements that can solve difficult learning control problems[END_REF][START_REF] Michie | BOXES: An experiment in adaptive control[END_REF] described as follows.

Descriptions of mountain car and pole balancing problems

Mountain car problem: a continuous control task of driving an underpowered car up a steep mountain road [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. The state of the car at step t is characterized by its position, denoted by q t , and its velocity, denoted by qt . The simplified physics which the car moves according to is described in detail in [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. In this case, the state space is continuous and S c = {(q, q) : bl q ≤ q ≤ br q , bl q ≤ q ≤ br q} where the left bounds bl q = -1.5, bl q = -0.07 and the right bounds br q = 0.45, br q = 0.07. If the next position q t+1 ≤ bl q , then the next velocity qt+1 is set to 0. When q t+1 ≥ br q , the goal at the top of the mountain is reached. Each episode of the task starts from "the foot of the mountain" (i.e. q = -0.5, q = 0) and ends when either the goal or the maximum number of steps per episode is reached. The action space is A = {-1, 0, 1}, corresponding to three actions: full throttle forward, full throttle reverse and zero throttle. The reward r t is set to 100 if the goal is reached, -1 otherwise. In our experiment, we discretize the continuous state space S c into a discrete state space denoted by S. In order to convert a continuous state s ∈ S c into S, we check which state in S is the closest to s and return it as a state discretization of s. As for basis functions, for each (s i , a j ) ∈ S × A, i = 1, . . . , N S and j = 1, . . . , N A , we define φ(s i , a j ) = (φ k (s i , a j )) k=1,...,N S .N A (6.13)

Online DCA for Reinforcement Learning where

.N S , 0 otherwise, (6.14) and thus, the number of basis functions is d = N S .N A .

Pole balancing problem: a control task which consists in stabilizing a pole hinged to a cart by applying forces to the cart [START_REF] Barto | Neuronlike adaptive elements that can solve difficult learning control problems[END_REF][START_REF] Michie | BOXES: An experiment in adaptive control[END_REF][START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. In this problem, the state space S c = {(q, q, x, ẋ) ∈ R 4 } is continuous, where q, q are respectively the vertical angle (in radians), the angular velocity (in radians per second) of the pole, and x, ẋ are respectively the cart position (in meters) from the center and its velocity (in meters per second). The general physics of the cart and pole is presented in [START_REF] Barto | Neuronlike adaptive elements that can solve difficult learning control problems[END_REF]. In our experiments, each episode starts from the "zero" state (i.e. q = q = x = ẋ = 0) and ends when the pole falls (i.e. |q| > π/4) or the cart moves off the track (i.e. |x| > 4). The action space is A = {-10, -9, . . . , 9, 10}, corresponding to the forces (in newtons) to the cart. At step t, the reward is defined as r t = -10000 -50|q t | -100|x t | if the corresponding episode is terminated, and r t = 10 -1000x 2 -5|q t | -10 ẋt otherwise. Similarly to mountain car problem, we also consider a discrete state space S and the similar basis functions.

Set up experiments

All experiments were implemented in MATLAB R2013b and performed on a PC Intel(R) Xeon(R) CPU E5-2630 v2, @ 2.60GHz of 32GB RAM. All parameters for the simulation of both above problems are set the same as in the implementation by José Antonio Martín H. and the source codes of Q-learning, SARSA are available on his homepage 2 . The initial point for all algorithms is θ 1 = 0 ∈ R d . As mentioned in Section 4.1, we here consider the ε-greedy method to make the trade-off between exploitation and exploration. In particular, the exploration schedule is ε t = ε 0 .ε epi (t) where the decay rate ε = 0.99, the initial exploration probability ε 0 = 0.01 and the function epi(t) returns the index of episode which the step t belongs to. The discount factor for mountain car problems (resp. pole balancing problems) is γ = 0.97 (resp. γ = 0.95). The maximum number of episodes is set to 1000. The maximum number of steps per episode is set to 1000. The learning rate is η t = η for all t where η is chosen from the set {0.1, 0.2, . . . , 2} such that it gives the smallest (resp. largest) averaged number of steps at the last episode for mountain car problems (resp. pole balancing problems). As for the ODCA, AODCA algorithms, the value δ t = -0.5 and a t = θ 1 for all t. For mountain car problems, the criteria to evaluate the effectiveness of the proposed algorithms are the average number of steps at the last episode and CPU time (in seconds), while for pole balancing problems we are interested in the total number of steps of all episodes and the CPU time per step (in seconds).

2. https://jamh-web.appspot.com/download.htm#Reinforcement_Learning: The upper the curve, the better the performance.

Comments on computational results:

Concerning mountain car problems, the smaller the number of steps, the better the algorithm (due to the fact that one needs to drive the car such that it can move up to the top of the mountain as soon as possible). From Fig. 6.2 and Table 6.1, we see that the alternating versions of ODCA algorithms are more efficient than their original versions. Among our four proposed algorithms, AODCA1 is the best in both quality of solutions and rapidity. From Fig. 6.3 and Table 6.2, our online algorithm AODCA1 gives the average number of steps better than SARSA, Q-learning algorithms when the number of episodes is greater than 400, and more precisely, the policy generated by AODCA1 is more efficient, stable than SARSA and Q-learning. Although the complexity of AODCA1 at each iteration is larger than that of SARSA and Q-learning, the CPU time of AODCA1 is comparable with SARSA and Q-learning (see Table 6.2). This can be explained by the fact that the number of steps obtained by AODCA1 is smaller than SARSA and Q-learning.

For pole balancing problems, from the fact that one wants to stabilize the pole as long as possible, it derives that the larger the number of steps, the better the algorithm. From Fig. 6.4 and Table 6.3, we observe that AODCA1 keeps the pole balancing longer than SARSA and Q-learning. Specifically, when the number of episodes is less than 500, AODCA1 is the best in terms of number of steps, SARSA is the worst and when the number of steps is greater than or equal to 500, the three algorithms are comparable. Concerning CPU time per step, AODCA1 is fairly similar to both SARSA and Q-learning although the complexity of AODCA1 at each step is larger. This fact is because the number of steps of AODCA1 is the best in most cases.

Applications to Stochastic Shortest Path problems: DCA Approaches via Cardinality Minimization and Batch Reinforcement Learning

Introduction

The Stochastic Shortest Path (SSP) problem in vehicle routing on a given road network is the problem of finding the optimal path in which the length of road links is random [START_REF] Frank | Shortest Paths in Probabilistic Graphs[END_REF][START_REF] Sigal | The Stochastic Shortest Route Problem[END_REF][START_REF] Martin | Distribution of The Time Through A Directed, Acyclic Network[END_REF]. As one fundamental problem in network studies, the SSP problem has attracted attention from researchers in many areas of, e.g., transportation engineering, computer science and operations research. In this chapter, we consider the SSP problem for a single independent vehicle where the length of road links is represented by their travel time. In fact, the randomness of travel time is due to many uncertain traffic conditions, e.g., the quality of vehicle, traffic jam, weather and so on. In such a stochastic environment, there are many different criteria for an optimal path, e.g., least expected travel [START_REF] Miller-Hooks | Least Expected Time Paths in Stochastic, Time-Varying Transportation Networks[END_REF], mean-risk model [START_REF] Nikolova | Approximation Algorithms for Reliable Stochastic Combinatorial Optimization[END_REF], path optimality index [START_REF] Sigal | The Stochastic Shortest Route Problem[END_REF], probability tail (PT) model [START_REF] Frank | Shortest Paths in Probabilistic Graphs[END_REF][START_REF] Nikolova | Stochastic Shortest Paths via Quasi-convex Maximization[END_REF][START_REF] Fan | Arriving on Time[END_REF], which are reviewed in detail in [START_REF] Cao | Maximizing the Probability of Arriving on Time : A Stochastic Shortest Path Problem[END_REF]. Because of perspicuity for drivers and solving daily issues, we are interested in SSP problems using the PT model-based criterion where the optimal path is the one that maximizes the probability of reaching destination before a given deadline, which are known as the arriving-on-time problems. An optimization formulation of PT model-based SSP problems on a road network can be expressed as the following. In this chapter, we use bold letters to denote matrices and vectors, normal letter to denote scalars.

An optimization formulation of PT model-based SSP problems

A road network can be modeled as a directed graph G = (V, A r ) where V = {1, . . . , n} represents the set of nodes and

represents the set of arcs with the size of m, (v 1 , v 2 ) represents an arc from node v 1 to node v 2 . As mentioned above, the goal of the PT model-based SSP problem is to find the path x that maximizes the probability of arriving at the destination d from the origin o not later than a given deadline T (more precisely, T is the remaining time to deadline) based on the distributed travel time samples w. The corresponding optimization formulation can be written as follows [START_REF] Cheng | Stochastic Shortest Path Problem with Uncertain Delays[END_REF].

) is on the concerned path. This formulation is equivalent to the following problem: where M ∈ R n×m is a node-arc incidence matrix (see, e.g., [2] for more details), b ∈ R n whose elements are zero except the two elements corresponding to node o and node d, which are 1 and -1, respectively.

Related works

Most related studies for solving the problem (7.2) are based on the common assumptions about specific distributions of travel times, correlations between travel times on different road links, deadlines (see, e.g., [START_REF] Nikolova | Stochastic Shortest Paths via Quasi-convex Maximization[END_REF][START_REF] Lim | Stochastic Motion Planning and Applications to Traffic[END_REF][START_REF] Frank | Shortest Paths in Probabilistic Graphs[END_REF][START_REF] Fan | Arriving on Time[END_REF][START_REF] Nie | Shortest Path Problem Considering On-Time Arrival Probability[END_REF] for more details). However, on real road networks, these assumptions are hard to be satisfied. Without working on these assumptions, Cao et al. [START_REF] Cao | Finding The Shortest Path in Stochastic Vehicle Routing: A Cardinality Minimization Approach[END_REF] proposed a data-driven approach which is only based on the travel time samples of all road links. In this chapter, to solve the problem (7.2), we consider two reformulations of these PT model-based SSP problems based on the data-driven approach: cardinality minimization and Markov Decision Process (MDP). In the following, we present the related works of these two reformulations.

Cardinality minimization reformulation

By reformulating (7.2), a cardinality minimization problem with the zero-one variables is described as follows (see [START_REF] Cao | Finding The Shortest Path in Stochastic Vehicle Routing: A Cardinality Minimization Approach[END_REF]).

where Card(z) is the number of nonzero elements in z, vector C(x) = [w 1 x -T ] + , . . . , [w S x -T ] + , {w i } i=1,...,S denotes the set of travel time samples on all arcs with the size of S and [•] + = max{0, •}. There exists a double difficulty in solving this cardinality minimization problem: how to treat, first, the cardinality term and, second, the zero-one variables. In [START_REF] Cao | Finding The Shortest Path in Stochastic Vehicle Routing: A Cardinality Minimization Approach[END_REF], the authors approximated the cardinality term by p -norm (0 < p ≤ 1) functions, the logarithmic approximation function and their combination, and thus, proposed the so-called 1 -norm algorithm and reweighted-1 algorithms. The 1 -norm algorithm only solves one Mixed Integer Linear Program (MILP), while the reweighted-1 algorithms require solving successive MILPs. There is one limitation of this work: expensive computations in solving MILPs. In [START_REF] Cao | Improving The Efficiency of Stochastic Vehicle Routing: A Partial Lagrange Multiplier Method[END_REF], the authors avoided solving MILPs by relaxing them to linear programs as well as using partial Lagrange multiplier techniques and a property of incidence matrix M to guarantee that their obtained solutions are zero-one. Concerning the choice of good approximation of the cardinality term, it is known that there exist approximation functions in [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF] which have been proved to be more efficient than those in [START_REF] Cao | Finding The Shortest Path in Stochastic Vehicle Routing: A Cardinality Minimization Approach[END_REF][START_REF] Cao | Improving The Efficiency of Stochastic Vehicle Routing: A Partial Lagrange Multiplier Method[END_REF].

In the optimization and machine learning community, the cardinality minimization problem (7.2) with the continuous variables instead of zero-one variables is known as an 0 -norm problem (see, e.g, [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF]) where 0 -norm is defined as Card. A variety of works relating to 0 -norm problems have been recently reviewed in the seminal work of Le Thi et al. [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF]. According to the way to treat the 0 -norm, they are divided into three

where y = (y 1 , . . . , y S ) ∈ R S , y * i = [w i x * -T ] + , ∀i = 1, . . . , S. Conversely, if (x * , y * ) is an optimal solution to (7.3), then x * is an optimal solution to (7.2).

Note that the upper bounds for the variable y are added in (7.3) due to the fact that for all x ∈ {0, 1} m , 0

This boundedness of the feasible set is useful for the exact penalty techniques as well as the nonconvex exact reformulation approach for the problem (7.2) as discussed in Section 7.1.2.

In this chapter, we use a continuous DC approximation function of the Card term, specifically the exponential concave approximation [START_REF] Bradley | Feature Selection via Concave Minimization and Support Vector Machines[END_REF], which is proven to be efficient in pratice (see, e.g., [START_REF] Thi | A DC Programming Approach for Feature Selection in Support Vector Machines Learning[END_REF][START_REF] Bradley | Feature Selection via Concave Minimization and Support Vector Machines[END_REF][START_REF] Thi | DC approximation approaches for sparse optimization[END_REF][START_REF] Thi | DC Programming and DCA for Sparse Optimal Scoring Problem[END_REF]), as follows.

where the function r η (t) = 1-e -η|t| for t ∈ R and the positive approximation parameter η. Thus, we derive the following approximate problem

In fact, solving the problem (7.4) is still difficult due to zero-one variables. Using the exact penalty techniques to treat these variables leads to an exact continuous reformulation of (7.4) which is of the standard form of a DC program for which DCA is applied. Let p : [0, 1] m → R be the penalty function defined by p(x) := m i=1 min{x i , 1 -x i }. The problem (7.4) can be rewritten as follows:

is the bounded polyhedral convex set defined by n + S linear constraints and upper/lower bound constraints. It leads to the corresponding penalized problem (τ being the positive penalty parameter) α(τ ) := min {f (x, y) + τ p(x) : (x, y) ∈ K} .

It follows from Theorem 7 in [START_REF] Thi | Exact penalty and error bounds in DC programming[END_REF] that there exists τ 0 > 0 such that for all τ > τ 0 , the two problems (7.4) and (7.5) are equivalent, in the sense that they have the same optimal value and the same solution set.

Applications to Stochastic Shortest Path problems: DCA Approaches via Cardinality Minimization and Batch Reinforcement Learning 149

Considering the problem (7.5) with a sufficiently large number τ , we use the following DC decomposition of r η [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF]:

where g(t) = η|t| and h(t) = η|t| -1 + e -η|t| . Thus, a DC reformulation of the problem (7.5) can be expressed as follows.

min

where G(x, y) = χ K (x, y) + S i=1 g(y i ) and H(x, y) = S i=1 h(y i ) -τ p(x). Obviously, (7.7) is a polyhedral DC program where the first DC component G(x, y) is polyhedral convex.

DCA for solving the polyhedral DC program (7.7)

According to the generic DCA scheme, at each iteration k, after computing one subgradient (x k , y k ) ∈ ∂H(x k , y k ), the calculation of (x k+1 , y k+1 ) is reduced to solve the following convex program min G(x, y) -(x k , y k ), (x, y) : (x, y) ∈ R m+S , which is equivalent to a linear program min (-x k , ηe -y k ), (x, y) : (x, y) ∈ K , (

where e is the vector of ones in the appropriate vector space.

Let q(y) := S i=1 h(y i ). We present how to compute a subgradient (x k , y k ) ∈ ∂H(x k , y k ). Since y k ≥ 0, we can choose x k ∈ τ ∂(-p)(x k ) and y k ∈ ∂q(y k ) as follows. For i = 1, . . . , m,

and for j = 1, . . . , S,

DCA applied to (7.7) can be given by Algorithm 7.1 (Card-DCA).

Algorithm 7.1 DCA for solving the cardinality problem (7.7) (Card-DCA) Initialization: Let (x 0 , y 0 ) ∈ R m+S , k = 0. repeat 1. Compute (x k , y k ) ∈ ∂H(x k , y k ) using (7.9)-(7.10). 2. Compute (x k+1 , y k+1 ), an optimal solution to (7.8).

3.

Applications to Stochastic Shortest Path problems: DCA Approaches via Cardinality Minimization and Batch Reinforcement Learning

According to the convergence properties of the generic DCA scheme for polyhedral DC programs in Section 1.1, we deduce the following convergence properties of Card-DCA.

Theorem 7.1. Convergence properties of Card-DCA i) Card-DCA generates the sequence {(x k , y k )} containing finitely many elements such that the sequence {F (x k , y k )} is decreasing.

ii) The sequence {(x k , y k )} converges to a critical point (x * , y * ) of (7.7) after a finite number of iterations.

iii) For a sufficiently large number τ , if there exists an iteration k such that x k ∈ {0, 1} m , then x l ∈ {0, 1} m for all l > k.

Proof. The properties i) and ii) are direct consequences of the convergence properties of DCA for a polyhedral DC program. The property iii) is proved similarly as Theorem 1 in [START_REF] Thi | Feature selection in machine learning: an exact penalty approach using a difference of convex function algorithm[END_REF] and Theorem 2 in [START_REF] Thi | Exact Penalty in DC Programming[END_REF].

The second reformulation: Batch RL problem

Remember that for the MDP reformulation as described in Section 7.1.2.2, the different optimization formulations of Optimal Bellman Residual (OBR) which were thoroughly studied in Chapter 5 can be made use of for solving these SSP problems. In this chapter, we concentrate on the 1 -norm optimization formulation, which is briefly presented as follows.

-norm optimization formulation

In batch mode, we collect N samples (S i , A i , S i , R i ) i=1,...,N where R i is the immediate reward received by taking the action A i which moves the agent from the current state S i to the next state S i for i = 1, . . . , N . In this case, we have the 1 -norm optimization formulation expressed as min F (θ) :

where the real function

the matrix A and the vector c are defined as A (i,j) := γφ(S i , a j ) -φ(S i , A i ), c (i) := R i for i = 1, . . . , N , j = 1, . . . , N A , φ is an l-dimensional basic function vector.

where j i ∈ argmax j=1,...,N A A (i,j) , θ k for i = 1, . . . , N .

Solve the linear program

to obtain (θ k+1 , t k+1 1 , . . . , t

In addition, RL-DCA enjoys the following convergence properties. Theorem 7.2. Convergence properties of RL-DCA i) RL-DCA generates the sequence {θ k } such that the sequence {F (θ k )} is decreasing.

ii) The sequence {θ k } converges to a critical point θ * of (7.12) after a finite number of iterations. iii) θ * is almost always a local minimizer to (7.12). In particular, if I i (θ * ) is a singleton for all i = 1, ..., N , then θ * is a local minimizer to (7.12). (N A + 1).N constraints, and the one in Card-DCA has m + S variables, n + 2m + 3S constraints where l is the number of the basic functions, N is the number of batch samples, N A is the number of actions, m is the number of arcs, n is the number of nodes, S is the number of travel time data for each link.

Numerical experiments

In this section, we study the performance of the proposed DCA algorithms RL-DCA, Card-DCA and compare them with existing algorithms for solving these SSP problems which will be listed in each experiment, on the arriving-on-time problem (i.e. maximizing the probability of arriving at the destination d from the origin o not later than the deadline T ) for a single vehicle on road networks.

Our experiments consist of two parts. In the first experiment, we only concern the cardinality problem, in particular we consider the efficiency of Card-DCA in comparison with two existing algorithms in [START_REF] Cao | Finding The Shortest Path in Stochastic Vehicle Routing: A Cardinality Minimization Approach[END_REF]: 1 -norm, reweighted 1 -norm minimization algorithms (we call 1 , re-1 respectively for short). The 1 algorithm replaces the Card term by the 1 -norm function and solves one MILP. The re-1 algorithm uses the combination of the p -norm (0 < p < 1) and logarithmic approximation functions and requires solving successive MILPs. In fact, the approximation in the re-1 algorithm is a special version of our DC approximation approach (see [START_REF] Thi | DC approximation approaches for sparse optimization[END_REF] for more details). It has been shown in [START_REF] Cao | Finding The Shortest Path in Stochastic Vehicle Routing: A Cardinality Minimization Approach[END_REF] that when comparing between re-1 and 1 , in most cases, re-1 (resp. 1 ) is better than the other in terms of the quality of solutions (resp. the rapidity). In the second experiment, we aim to compare between two DCA approaches for these PT model-based SSP problems, in particular RL-DCA and Card-DCA on artificial networks.

Experiment 1: Card-DCA for the cardinality problem

This experiment aims to make a comparison between the proposed Card-DCA algorithm and the existing algorithms ( 1 , re-1 ) for the cardinality problem, which is implemented on a road network with 123 road links and 65 intersections of roads as in [START_REF] Cao | Finding The Shortest Path in Stochastic Vehicle Routing: A Cardinality Minimization Approach[END_REF], corresponding to a directed graph with 123 arcs and 65 nodes respectively as Fig. 7.1 and the length of arcs is represented by the travel time of road links (in seconds).

Data set: Numerical experiments are conducted on the data sets (including the training set of size S 1 and test set of size S 2 ) of travel time samples on each link generated by some independent random distribution functions (e.g., normal, gamma and lognormal distributions). The (o, d) pair is chosen randomly from the given set. The deadline T is defined by T = T 1 + κ(T 2 -T 1 ) where κ is a deadline coefficient, T 2 is the minimum longest travel time for all paths of (o,d) and T 1 is the shortest travel time with respect to the same path (see [START_REF] Cao | Finding The Shortest Path in Stochastic Vehicle Routing: A Cardinality Minimization Approach[END_REF] for more details). Comparison criteria of algorithms: We are interested in the following aspects: accuracy (in %) and CPU time (in seconds). The accuracy of each algorithm is computed on the test set by the percentage of the probability of arriving on time of the path obtained by the algorithm relative to the maximum probability of all possible paths by the enumeration method.

Descriptions of tables of results: The average accuracy and average CPU time obtained by three algorithms Card-DCA,