Keywords: Apprentissage statistique, prévision séquentielle, suites individuelles, agrégation PACbayésienne, pondération exponentielle, régression parcimonieuse, grande dimension, calibration automatique, vitesses minimax, regret externe, regret interne, sélection de modèles Statistical learning, online learning, machine learning, individual sequences, regret bounds, PAC-Bayesian aggregation, exponential weighting, high-dimensional regression, sparsity, parameter tuning, minimax rates, external regret, internal regret, swap regret, model selection. AMS Classification: 68Q32, 62J02, 62J05, 62C20 . . . . . . . . . . . . . . . . oracle inequalities . . . . . . . . . . . . 87

Prédiction de suites individuelles et cadre statistique classique : étude de quelques liens autour de la régression parcimonieuse et des techniques d'agrégation Résumé : Cette thèse s'inscrit dans le domaine de l'apprentissage statistique. Le cadre principal est celui de la prévision de suites déterministes arbitraires (ou suites individuelles), qui recouvre des problèmes d'apprentissage séquentiel où l'on ne peut ou ne veut pas faire d'hypothèses de stochasticité sur la suite des données à prévoir. Cela conduit à des méthodes très robustes. Dans ces travaux, on étudie quelques liens étroits entre la théorie de la prévision de suites individuelles et le cadre statistique classique, notamment le modèle de régression avec design aléatoire ou fixe, où les données sont modélisées de fac ¸on stochastique. Les apports entre ces deux cadres sont mutuels : certaines méthodes statistiques peuvent être adaptées au cadre séquentiel pour bénéficier de garanties déterministes ; réciproquement, des techniques de suites individuelles permettent de calibrer automatiquement des méthodes statistiques pour obtenir des bornes adaptatives en la variance du bruit. On étudie de tels liens sur plusieurs problèmes voisins : la régression linéaire séquentielle parcimonieuse en grande dimension (avec application au cadre stochastique), la régression linéaire séquentielle sur des boules 1 , et l'agrégation de modèles non linéaires dans un cadre de sélection de modèles (régression avec design fixe). Enfin, des techniques stochastiques sont utilisées et développées pour déterminer les vitesses minimax de divers critères de performance séquentielle (regrets interne et swap notamment) en environnement déterministe ou stochastique.

sans vous ? Merci pour votre aide et votre disponibilité.

Je souhaiterais par ailleurs saluer toutes les personnes que j'ai pu croiser au cours de séminaires ou conférences, pour les discussions mathématiques que nous avons pu entretenir, mais aussi pour les bons moments passés au soleil (ou sous la pluie) : Joseph, Odalric, Sébastien, Sylvain, Christophe, Antoine, Guillaume, pour n'en mentionner que quelques-uns. Et une mention spéciale pour Mohamed : un grand merci pour ta relecture minitieuse de mon introduction et pour les conseils que tu as pu m'apporter au cours de ma thèse. Merci aussi à Nicolò pour son accueil chaleureux à Milan : les quelques jours de recherche avec vous en mars dernier ont été très stimulants ! Enfin, je souhaiterais remercier tous mes proches pour leur bienveillance et tous les bons moments passés ensemble. Je pense évidemment à la petite troupe de Centrale : Benoît, Christophe, Manu, Toutoune et Thibaut ; merci d'avoir supporté mes digressions mathématiques ! Merci aussi à Cédric, Wafaa et Fabrizio ; les sorties ciné et crêperie à Montparnasse ont été un régal ! Franc ¸ois, je suis content que nos parcours parallèles depuis le collège nous aient conduits tous deux à Orsay ; nous aurons même soutenu à deux semaines d'intervalle ! Pour finir, je tiens à remercier chaleureusement mes parents, mes grands-parents et ma soeur pour le soutien inconditionnel dont ils ont fait preuve tout au long de cette thèse ; ce travail vous est entièrement dédié.

Prévision de suites individuelles

Considérons la tâche de prévision séquentielle suivante. Un statisticien cherche à prévoir tour après tour les valeurs inconnues d'une suite d'observations y 1 , y 2 , . . . ∈ Y à partir de prévisions (ou décisions) a 1 , a 2 , . . . ∈ D. (Les espaces d'observation Y et de décision D peuvent différer.) En théorie statistique classique de prévision séquentielle, il est d'usage de supposer que la suite y 1 , y 2 , . . . est la réalisation d'un certain processus stochastique, par exemple ergodique stationnaire. De telles hypothèses permettent d'estimer séquentiellement les caractéristiques du processus sous-jascent, et ainsi de construire des méthodes de prévision performantes quand le modèle statistique choisi décrit bien les données en jeu. Cela peut en revanche s'avérer irréaliste dans certaines situations difficilement modélisables de fac ¸on statistique, par exemple, lorsque la suite y 1 , y 2 , . . . évolue et réagit aux décisions a 1 , a 2 , . . . comme c'est le cas pour la détection de courriels frauduleux ou pour l'investissement sur le marché boursier.

Dans la théorie dite de prévision de suites individuelles, aucune hypothèse de stochasticité n'est faite sur la fac ¸on dont est générée la suite des observations y 1 , y 2 , . . .. Toutes les suites possibles sont considérées et des garanties théoriques sont disponibles pour chacune d'elles -d'où le nom de prévision de suites individuelles.

Dans un cadre aussi général, il est irréaliste de chercher à prévoir correctement l'observation y t à chaque date t et sur le seul fondement des observations passées. En revanche, si le statisticien dispose à chaque instant t de prévisions de base (ou avis d'experts) a θ,t ∈ D, θ ∈ Θ, alors un but raisonnable consiste à prévoir presque aussi bien que le meilleur des experts sur le long terme. Ce problème générique, qualifié de prévision avec avis d'experts, est celui considéré dans cette thèse. Une description sous la forme d'un jeu répété entre le statisticien et l'environnement est donnée en figure 1.2.

Paramètres : espace de décision convexe D, espace d'observation Y, fonction de perte : D × Y → R, et ensemble Θ des indices d'experts.

A chaque date t ∈ N * {1, 2, . . .}, 1. l'environnement choisit les avis d'experts a θ,t ∈ D pour tout θ ∈ Θ ; ils sont révélés au statisticien ;

2. le statisticien prend une décision a t ∈ D, qu'il garde confidentielle ou révèle a à l'environnement ;

3. l'environnement choisit et révèle l'observation y t ∈ Y ; 4. le statisticien encourt la perte a t , y t et chaque expert θ ∈ Θ encourt la perte a θ,t , y t .

a Si l'environnement n'a pas accès aux décisions at du statisticien, il est qualifié d'oublieux. Si, à l'inverse, l'environnement peut réagir aux décisions passées du statisicien, il est qualifié d'antagoniste. Ces deux cadres sont équivalents lorsque l'algorithme de prévision utilisé est déterministe ; cf. section 2.3.1.

FIGURE 1.2 -Prévision avec avis d'experts.

Dans ce cadre, la qualité des prévisions du statisticien après T tours de prévision est mesurée par sa perte cumulée T t=1 ( a t , y t ). Un objectif classique est alors de faire en sorte que, malgré la contrainte de prévision séquentielle, cette perte cumulée soit presque aussi petite que celle du meilleur expert a posteriori. Cela correspond à minimiser la différence T t=1 ( a t , y t ) -inf θ∈Θ T t=1 (a θ,t , y t ) .

Cette différence est appelée regret (externe). D'autres formes de regret en lien avec la théorie des jeux (par ex., regret interne, regret swap) sont considérées au chapitre 5 -cf. section 1.4. Dans l'essentiel de cette thèse, nous suivons l'approche des suites individuelles, i.e., nous étudions des stratégies du statisticien dont le regret (externe) est "petit" uniformément en toutes les suites y 1 , y 2 , . . . ∈ Y. Par "petit", il convient d'entrendre sous-linéaire en T (puisqu'une vitesse linéaire en T est triviale quand est bornée). Cela correspond à un regret moyen dans le pire des cas qui est asymptotiquement négatif quand T → +∞, i.e., sup y 1 ,...,y T ∈Y (a θ,1 ) θ ,...,(a θ,T

) θ ∈D Θ 1 T T t=1 ( a t , y t ) -inf θ∈Θ 1 T T t=1 (a θ,t , y t )
o(1) quand T → +∞ .

Une telle garantie indique qu'en moyenne, le statisticien prévoit presque aussi bien que le meilleur des experts a posteriori. Quand Θ est fini de cardinal K, des ordres de grandeur typiques pour le regret moyen dans le pire des cas sont (ln K)/T lorque la perte est bornée et convexe ou (ln K)/T lorsque la perte est exp-concave.

On suppose ci-après que Θ = {1, . . . , K}. Un exemple classique et fondamental d'algorithme séquentiel atteignant les vitesses mentionnées ci-dessus est le prédicteur par pondération exponentielle introduit en machine learning par [START_REF] Littlestone | The weighted majority algorithm[END_REF] et [START_REF] Vovk | Aggregating strategies[END_REF]. A chaque date t 1, la prévision de cet algorithme est donnée par la combinaison convexe a t K j=1 p j,t a j,t , où (p 1,1 , . . . , p K,1 ) = (1/K, . . . , 1/K) et où, pour tout t 2, p i,t exp -η t-1 s=1 (a i,s , y s ) K j=1 exp -η t-1 s=1 (a j,s , y s )

, 1 i K , où η > 0 est un paramètre de l'algorithme. Le théorème suivant indique que pour une calibration judicieuse de η, le regret de cet algorithme est au plus de l'ordre de √ T ln K ou de ln(K) selon que la fonction de perte est convexe bornée ou exp-concave. Ce résultat est prouvé au chapitre 2 aux théorèmes 2.1 et 2.2, qui sont dus respectivement à [START_REF] Cesa-Bianchi | Analysis of two gradient-based algorithms for on-line regression[END_REF] (cf. aussi [CBL06, théorème 2.2] et [CBFH + 97, CBL99]) et à [START_REF] Kivinen | Averaging expert predictions[END_REF]. (A2) La fonction : D × Y → R est η 0 -exp-concave en son premier argument pour un certain η 0 > 0, i.e., la fonction a → e -η 0 (a,y) est concave sur D pour tout y ∈ Y.

Alors, pour tout T ∈ N * et toute suite d'avis d'experts a i,t ∈ D et d'observations y t ∈ Y, le regret du prédicteur par pondération exponentielle calibré avec η > 0 vérifie :

• Sous (A1), T t=1 a t , y t -min

1 i K T t=1 a i,t , y t ln K η + ηT (B 2 -B 1 ) 2 8 .
Cette borne, minimisée en η = (B 2 -B 1 ) -1 8(ln K)/T , devient (B 2 -B 1 ) (T /2) ln K.

• Sous (A2) et lorsque η ∈ (0, η 0 ],

T t=1 a t , y t -min

1 i K T t=1
a i,t , y t ln K η .

La calibration du paramètre η est un problème crucial. En effet, les valeurs suggérées pour η dépendent de quantités potentiellement inconnues au début de la tâche de prévision comme l'étendue des pertes B 2 -B 1 et l'horizon de prévision T sous l'hypothèse (A1). Sous l'hypothèse (A2), la valeur optimale suggérée pour η est η 0 , qui est également inconnue en général -par exemple, la perte carrée : [-B, B] × [-B, B] → R définie par (a, y) = (y -a)2 est 1/(8B 2 )exp-concave en son premier argument ; l'amplitude des observations et des avis d'experts B est généralement inconnue.

Il est possible de calibrer séquentiellement η de fac ¸on totalement automatique, tout en garantissant des bornes de regret quasiment identiques (à de petits facteurs multiplicatif et additif près). Une technique générale due à [START_REF] Auer | Adaptive and self-confident on-line learning algorithms[END_REF] puis à [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF] consiste à redéfinir à chaque date t les poids exponentiels (p 1,t , . . . , p K,t ) à l'aide d'un paramètre η t choisi en fonction des observations passées y s et des avis d'experts passés a i,s , s = 1, . . . , t -1. De telles procédures de calibration séquentielle sont décrites en détail au chapitre 2 (cf. section 2.2.2). Nous en développons aux chapitres 3 et 4 pour la perte carrée (cf. sections 1.2 et 1.3).

Liens avec le cadre statistique classique

On décrit ci-après des liens qu'entretient la prévision de suites individuelles avec des cadres satistiques plus classiques comme le modèle de régression avec plan d'expérience (design) fixe ou aléatoire, où les données observées sont cette fois modélisées de fac ¸on stochastique.

Considérons le problème générique de prévision suivant. Soit D un espace de décision convexe, Z un espace d'observation 1 , et : D × Z → R une fonction de perte convexe en son premier argument. Au début de la tâche de prévision, le statisticien observe T copies indépendantes Z 1 , . . . , Z T de Z ∈ Z, de loi commune inconnue. Le but du statisticien est de prévoir l'observation suivante 2 Z T +1 ∼ Z presque aussi bien que le meilleur élément (constant) d'un ensemble Θ ⊂ D. Plus précisément, il s'agit de construire une décision a T ∈ D mesurable en l'échantillon (Z 1 , . . . , Z T ) de sorte à minimiser une quantité appelée excès de risque en espérance3 E a T , Z -inf a∈Θ E a, Z , où l'espérance de gauche est prise par rapport à (Z 1 , . . . , Z T ) et Z. Un exemple classique de telle tâche de prévision est donnée par le modèle de régression avec design aléatoire.

Exemple 1.1 (Agrégation dans le modèle de régression avec design aléatoire). Soit (X , B) un espace mesurable et Θ un ensemble de fonctions mesurables de X vers R. Le statisticien observe T copies indépendantes (X 1 , Y 1 ), . . . , (X T , Y T ) d'un couple aléatoire (X, Y ) ∈ X × R de loi inconnue, avec E[Y 2 ] < ∞. Dans ce cadre, un objectif de prévision consiste à estimer la fonction de régression f : x ∈ X → E[Y |X = x] presque aussi bien que le meilleur élément de Θ. Plus précisément, il s'agit de construire un estimateur f T : X → R à partir de l'échantillon (X 1 , Y 1 ), . . . , (X T , Y T ) de sorte à minimiser l'excès de risque en espérance

E f (X) -f T (X) 2 -inf g∈Θ E f (X) -g(X) 2 ,
où les espérances sont prises par rapport à (X 1 , Y 1 ), . . . , (X T , Y T ) et X. Or, par de simples manipulations (en développant les carrés et en conditionnant par (X 1 , Y 1 ), . . . , (X T , Y T ), X), l'excès de risque précédent est égal à

E Y -f T (X) 2 -inf g∈Θ E Y -g(X) 2 .
Par conséquent, ce problème d'agrégation correspond au cadre décrit ci-dessus avec Z = X × R, avec D égal à l'ensemble des fonctions mesurables de X vers R, et avec : D × Z → R définie par g, (x, y) = y -g(x)

2 .

La tâche de prévision décrite précédemment n'est pas séquentielle -on la qualifie de batch en anglais car toutes les observations sont disponibles d'emblée. Cela n'interdit pas en revanche de traiter l'échantillon de fac ¸on séquentielle. On rappelle ci-dessous une technique standard qui permet de convertir un algorithme séquentiel encourant un faible regret pour des suites individuelles en une méthode stochastique encourant un petit excès de risque en espérance pour des suites i.i.d.. Soit ( a t ) t 1 un algorithme séquentiel, i.e., dans ce cadre, une suite de fonctions mesurables a t : Z t-1 → D ( a 1 est déterministe). L'échantillon Z 1:T (Z 1 , . . . , Z T ) est traité de fac ¸on séquentielle de la date 1 à la date T : l'algorithme ( a t ) t 1 produit séquentiellement les décisions a t (Z 1:t-1 ) ∈ D mesurables en Z 1:t-1 (Z 1 , . . . , Z t-1 ), t = 1, . . . , T . Le résultat suivant est dû à [START_REF] Cesa-Bianchi | On the generalization ability of on-line learning algorithms[END_REF] (cf. aussi [START_REF] Littlestone | From on-line to batch learning[END_REF]) ; nous le reprouvons au chapitre 2, proposition 2.5. Il indique qu'une fac ¸on simple de convertir l'algorithme séquentiel ( a t ) t 1 en méthode stochastique est de considérer la moyenne a T (Z 1:T ) = 1 T T t=1 a t (Z 1:t-1 ) .

(1.1)

Proposition 1.1 (Conversion online to batch, cf. proposition 2.5). Soit D un espace de décision convexe, Z un espace d'observation, et : D × Z → R une fonction de perte convexe en son premier argument. Soit ( a t ) t 1 un algorithme séquentiel et (R T ) T 1 une suite de réels telle que, pour tout T 1 et tous z 1 , . . . , z T ∈ Z, T t=1 ( a t , z t ) -inf a∈Θ T t=1 (a, z t ) R T .

Alors la conversion (1.1) appliquée à l'algorithme ( a t ) t 1 donne une procédure a T telle que, pour tout échantillon i.i.d. (Z 1 , . . . , Z T ) ∈ Z T ,

E a T , Z -inf a∈Θ E a, Z R T T ,
où les espérances sont prises par rapport à (Z 1 , . . . , Z T , Z), avec Z ∈ Z une variable aléatoire indépendante de (Z 1 , . . . , Z T ) et de même loi que Z 1 .

Sans surprise, la proposition précédente montre que tout algorithme séquentiel qui bénéficie de garanties déterministes peut être converti en une méthode statistique bénéficiant de garanties en espérance. Des bornes avec grande probabilité ont également été obtenues par [START_REF] Cesa-Bianchi | Improved risk tail bounds for on-line algorithms[END_REF] dans un cadre non nécessairement convexe et par [START_REF] Zhang | Data dependent concentration bounds for sequential prediction algorithms[END_REF][START_REF] Kakade | On the generalization ability of online strongly convex programming algorithms[END_REF] dans un cadre "très convexe" (perte carrée ou pertes fortement convexes). En régression, la conversion précédente est adaptée au modèle de régression avec design aléatoire (cf. exemple 1.1). Le cas du design fixe peut, dans une certaine mesure, être traité avec des techniques similaires ; voir la section 3.4.2.

La conversion précédente -qualifiée de online to batch en anglais -établit un lien de la prévision de suites individuelles vers le cadre statistique classique. Il s'avère que les apports de ces deux domaines sont en fait réciproques.

• Des méthodes statistiques classiques, conc ¸ues et étudiées sous des hypothèses stochastiques, peuvent aussi, moyennant quelques adaptations, s'avérer performantes dans un cadre de suites individuelles. C'est le cas de la méthode de régression ridge4 de [START_REF] Hoerl | Ridge regression: biased estimation for nonorthogonal problems[END_REF], initialement analysée dans le modèle de régression avec design fixe, qui a ensuite été adaptée et étudiée pour des suites individuelles par [START_REF] Azoury | Relative loss bounds for on-line density estimation with the exponential family of distributions[END_REF] et [START_REF] Vovk | Competitive on-line statistics[END_REF]. Les algorithmes de descente de gradient stochastique bénéficient également de garanties déterministes comme l'ont montré, par ex., [START_REF] Cesa-Bianchi | Worst-case quadratic loss bounds for prediction using linear functions and gradient descent[END_REF][START_REF] Zinkevich | Online convex programming and generalized infinitesimal gradient ascent[END_REF]. C'est le cas également du prédicteur par pondération exponentielle, qui a été étudié parallèlement en machine learning et en statistique 5 , des travaux fondateurs étant respectivement [START_REF] Littlestone | The weighted majority algorithm[END_REF][START_REF] Vovk | Aggregating strategies[END_REF] et [START_REF] Catoni | Universal aggregation rules with exact bias bounds[END_REF][START_REF] Yang | Combining different procedures for adaptive regression[END_REF][START_REF] Yang | Adaptive regression by mixing[END_REF]. Dans tous les cas, analyser des méthodes statistiques dans un cadre de suites individuelles permet d'en comprendre le coeur déterministe et d'en évaluer la robustesse (i.e., de jauger à quel point les hypothèses stochastiques sont nécessaires). Au chapitre 3, notre algorithme séquentiel SeqSEW est inspiré de la méthode statistique Sparse Exponential Weighting [START_REF] Dalalyan | Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity[END_REF][START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF]. Ainsi, notre analyse déterministe de l'algorithme SeqSEW indique que la méthode de [START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF] fonctionne essentiellement pour des raisons déterministes.

• D'après la remarque précédente, la théorie de la prévision de suites individuelles hérite d'idées fructueuses venant du cadre statistique classique, puisque ce dernier lui fournit de sérieux candidats pour la conception de nouveaux algorithmes séquentiels.

• En retour, les algorithmes séquentiels nouvellement conc ¸us peuvent être rapatriés dans le cadre statistique classique via la conversion online to batch (1.1). Cela permet de construire des méthodes statistiques calibrées automatiquement en fonction des données (par des techniques de suites individuelles) et qui sont adaptatives. Nous illustrons cet intérêt au chapitre 3 où l'on déduit des bornes de risque en design aléatoire similaires à [START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF], mais qui sont adaptatives en la variance inconnue du bruit (à un facteur logarithmique près) quand ce dernier est gaussien.

Nous venons d'évoquer des liens algorithmiques entre la prévision de suites individuelles et le cadre statistique classique. Cela induit notamment des similarités au niveau des techniques de preuve. Par exemple, la formule de dualité pour la divergence de Kullback-Leibler rappelée en annexe A.1 est un outil clé tant dans le cadre déterministe (chapitres 3 et 4) que dans le cadre stochastique (chapitre 6, où l'on considère le modèle de régression avec design fixe). Au chapitre 4, nous adaptons également un argument statistique classique connu sous le nom d'argument à la Maurey, qui permet de déterminer la qualité de l'approximation d'une discrétisation adéquate du simplexe en dimension quelconque. Comme en témoigne la preuve du théorème 4.2, cet argument s'adapte directement au cadre déterministe. D'autres similarités dans les techniques de preuve apparaissent aussi pour l'obtention de bornes inférieures. Une fac ¸on d'obtenir des bornes inférieures non asymptotiques en suites individuelles repose en effet sur l'utilisation d'outils de théorie de l'information comme le lemme de Fano ou l'inégalité de Pinsker (cf. annexe A.7), comme en statistique classique. La vitesse minimax du regret externe peut ainsi être obtenue via une variante du lemme de Fano ; l'analyse correspondante est due à [ACBFS02, CBLS05] et rappelée en section 2.3.2. En s'inspirant de ces techniques et de [Sto05, théorème 3.3], nous obtenons au chapitre 5 une borne inférieure sur le regret swap via l'inégalité de Pinsker. Enfin, des techniques stochastiques peuvent être utilisées à des fins purement déterministes. C'est le cas de la randomisation, que nous exploitons pour l'argument à la Maurey mentionné ci-dessus ou pour obtenir des bornes inférieures (cf. section 2.1.3 pour plus de détails). C'est le cas aussi des inégalités de concentration, telle l'inégalité de Hoeffding ou l'inégalité de Bernstein, qui permettent de déduire des bornes de regret pour des suites déterministes -cf. section 2.2.1. Au chapitre 5, nous développons également une technique stochastique qui permet de majorer le regret minimax pour des suites individuelles (relativement à diverses formes de regret, par ex., externe, interne, swap). Cette technique repose en partie sur l'utilisation d'une inégalité élémentaire de concentration de martingales, l'inégalité de Hoeffding-Azuma (cf. annexe A.5).

Les liens entre suites individuelles et cadre statistique classique évoqués précédemment sont partiellement représentés en figure 1.1. Nous détaillons ci-après les contributions principales de cette thèse, qui correspondent aux chapitres 3 à 6.

Bornes de parcimonie en régression linéaire séquentielle

Au cours de la dernière décennie, le phénomène de parcimonie -ou sparsité -a fait l'objet de nombreux travaux dans le cadre statistique classique. Parmi les outils introduits à cet effet, la notion d'inégalité oracle de sparsité -ou sparsity oracle inequality en anglais -joue un rôle fondamental. En régression linéaire, de telles bornes impliquent que la tâche consistant à prévoir presque aussi bien qu'un vecteur inconnu de grande dimension est statistiquement faisable pourvu que ce vecteur ait peu de coordonnées non nulles.

Au chapitre 3, on introduit un équivalent séquentiel déterministe de la notion d'inégalité oracle de sparsité. Nous prouvons de telles bornes pour un algorithme séquentiel appelé SeqSEW qui procède par pondération exponentielle et par troncature dépendante des données. Dans un second temps seulement, on applique une version totalement automatique de cet algorithme au cas particulier de suites i.i.d.. Les bornes de risque obtenues sont similaires à celles de [START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF] mais répondent à deux questions soulevées par les auteurs. En particulier, nos bornes sont adaptatives en la variance inconnue du bruit (à un facteur logarithmique près) si ce dernier est gaussien. Nous traitons aussi le cas du design fixe comme dans [START_REF] Dalalyan | Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity[END_REF].

Les contributions principales du chapitre 3 sont détaillées ci-après.

Cadre et enjeux

Contexte : régression linéaire séquentielle pour des suites individuelles

Le cadre principal du chapitre 3 est celui de la régression linéaire séquentielle pour des suites individuelles. Il s'agit d'un cas particulier du problème de la prévision avec avis d'experts décrit en figure 1.2. Un statisticien doit prévoir de fac ¸on séquentielle, à chaque tour t = 1, 2, . . ., la valeur y t ∈ R d'une suite inconnue d'observations en fonction d'une valeur d'entrée x t ∈ X et de prédicteurs de base ϕ j : X → R, 1 j d, à partir desquels il formule sa propre prévision y t ∈ R (la famille (ϕ j ) 1 j d est qualifiée de dictionnaire). La qualité des prévisions est évaluée avec la perte carrée. L'objectif du statisticien est de prévoir presque aussi bien que le meilleur prédicteur linéaire u • ϕ d j=1 u j ϕ j , où u ∈ R d , i.e., de satisfaire, uniformément sur toutes les suites individuelles (x t , y t ) 1 t T , une borne de regret de la forme (Par soucis de clarté, on omet les dépendances de ∆ T,d (u) en les amplitudes max 1 t T x t ∞ et max 1 t T |y t |.)

Hypothèse de parcimonie

Dans le cadre décrit ci-dessus, une variante6 de l'algorithme séquentiel ridge étudiée par [START_REF] Azoury | Relative loss bounds for on-line density estimation with the exponential family of distributions[END_REF] et [START_REF] Vovk | Competitive on-line statistics[END_REF] assure, lorsqu'elle est calibrée illégalement comme suggéré en section 2.4.2, un regret d'ordre au plus d ln T . Quand la dimension ambiante d est bien plus grande que le nombre de tours de prévision T , cette dernière borne de regret est bien supérieure à T et est donc en quelque sorte triviale. Puisque la borne d ln T est optimale en un certain sens (cf. [Vov01, théorème 2]), des hypothèses supplémentaires sont nécessaires pour garantir des performances théoriques intéressantes. Une hypothèse naturelle, qui a déjà été maintes fois étudiée dans le cadre stochastique, est qu'il existe une combinaison linéaire parcimonieuse u * (sparse en anglais, i.e., avec s T /(ln T ) coordonnées non nulles) dont la perte cumulée est petite. Si le statisticien connaissait à l'avance le support J(u * ) {j : u * j = 0} de u * , il pourrait appliquer le même algorithme de prévision séquentielle que précédemment mais seulement au sous-espace vectoriel de dimension s donné par u ∈ R d : ∀j / ∈ J(u * ), u j = 0 . Le regret de cet "oracle" serait alors au plus de l'ordre de s ln T et donc sous-linéaire en T . Sous cette hypothèse de parcimonie, un regret sous-linéaire semble donc possible, même si, bien sûr, la borne de regret s ln T peut seulement être utilisée comme une borne idéale de référence (puisque le support de u * est inconnu).

Au chapitre 3, on montre qu'il est possible d'atteindre une borne de regret proportionnelle à s (à un facteur logarithmique près). On prouve ainsi en corollaire 3.1 (cf. proposition 1.2 ci-dessous) et ses raffinements (cf., par ex., proposition 1.3 ci-dessous) des bornes de regret de la forme T t=1 (y t -y t ) 2 inf

u∈R d T t=1 y t -u • ϕ(x t ) 2 + u 0 + 1 g T,d u 1 , ϕ ∞ , (1.2)
où u 0 désigne le nombre de coordonnées non nulles de u et où g est croissante mais croît au plus logarithmiquement en T , d, u 1 d j=1 |u j |, et ϕ ∞ sup x∈X max 1 j d |ϕ j (x)|. Nous appellerons bornes de regret de sparsité -ou bornes de parcimonie -les bornes de regret de la forme précédente.

Travaux connexes dans les cadres stochastique et déterministe

La borne de regret (1.2) peut être vue comme un équivalent séquentiel déterministe des inégalités oracle de sparsité introduites dans le cadre statistique classique au cours de la dernière décennie. Un exemple typique de telles bornes de risque est

R u T (1 + a) inf u∈R d R(u) + C(a) u 0 ln d + 1 T (1.3)
en espérance ou avec grande probabilité, où R(u) désigne le risque L 2 de u • ϕ si le design est aléatoire (i.e., R(u) = E f (X) -u • ϕ(X)

2 ) ou le risque empirique de u • ϕ si le design est fixe (i.e., R(u) = T -1 T t=1 f (x t ) -u • ϕ(x t ) 2 sur le design (x 1 , . . . , x T )). Ainsi, les inégalités oracle de sparsité expriment un compromis entre le risque R(u) et le nombre de coordonnées nonnulles u 0 de tout vecteur u ∈ R d . De telles bornes ont été obtenues par [START_REF] Birgé | Gaussian model selection[END_REF] via des arguments de sélection de modèles et ont ensuite été développées, entre autres, par [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF][START_REF] Bunea | Aggregation for Gaussian regression[END_REF] dans le modèle de régression avec design fixe et par [START_REF] Bunea | Aggregation for regression learning[END_REF] dans le modèle de régression avec design aléatoire. Une introduction plus détaillée avec de plus amples références est proposée au chapitre 2 (section 2.6). Mentionnons néanmoins que, récemment, depuis les travaux de [START_REF] Dalalyan | Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity[END_REF], des inégalités oracle de sparsité avec constante 1 devant l'infimum7 ont été prouvées sans presque aucune hypothèse sur le dictionnaire (ϕ j ) j , et pour des méthodes pouvant être approchées numériquement à un coût algorithmique raisonnable pour de grandes valeurs de la dimension ambiante d. Ces méthodes procèdent par pondération exponentielle ; cf. [DT07, DT08, RT11, AL11] pour le modèle de régression avec design fixe et [START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF][START_REF] Alquier | PAC-Bayesian bounds for sparse regression estimation with exponential weights[END_REF] pour le modèle de régression avec design aléatoire.

Quant au cadre séquentiel déterministe, à notre connaissance, les propositions 1.2 et 1.3 cidessous (cf. aussi théorème 3.1 au chapitre 3) fournissent les premiers exemples de borne de regret de sparsité au sens de (1.2). De récents travaux [LLZ09, SST09, Xia10, DSSST10] en optimisation convexe séquentielle ont certes abordé la question de la sparsité, mais sous un tout autre angle. Dans le cas de la régularisation 1 sous la perte carrée, ces travaux proposent des algorithmes qui prédisent comme une combinaison linéaire parcimonieuse y t = u t • ϕ(x t ) des prévisions de base (i.e., u t 0 est petit), alors que de telles garanties ne semblent pas pouvoir être montrées pour notre algorithme SeqSEW. En revanche, ces travaux prouvent des bornes sur le regret 1 -régularisé de la forme

T t=1 (y t -u t • x t ) 2 + λ u t 1 inf u∈R d T t=1 (y t -u • x t ) 2 + λ u 1 + ∆ T,d (u) ,
pour un terme de regret ∆ T,d (u) qui croît beaucoup plus rapidement (comme une puissance et non logarithmiquement) en la dimension ambiante d, en la norme u 1 ou en T . Les bornes prouvées pour ces algorithmes sont donc sous-optimales dans le cadre qui nous intéresse ici (prévision sur des boules 0 de petit diamètre). Cela contraste avec les bornes de regret de la forme (1.2) que vérifie, par exemple, notre algorithme SeqSEW.

On reprend ci-après à grands traits les algorithmes et résultats principaux du chapitre 3, d'abord dans le cadre déterministe (section 1.2.2) puis dans le cadre stochastique (section 1.2.3).

Bornes de sparsité en suites individuelles

Pour simplifier l'analyse, on suppose d'abord que, au début du jeu, le statisticien a accès au nombre T de tours de prévision, à une borne B y sur l'amplitude des observations |y 1 |, . . . , |y T | et à une borne B Φ sur la trace de la matrice de Gram empirique, i.e., La première version de notre algorithme est définie en figure 1.3. Nous l'appelons SeqSEW puisqu'il s'agit d'une variante adaptée aux suites individuelles de l'algorithme Sparse Exponential Weighting introduit dans le cadre statistique classique par [START_REF] Dalalyan | Aggregation by exponential weighting and sharp oracle inequalities[END_REF][START_REF] Dalalyan | Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity[END_REF].

En utilisant un lemme PAC-Bayésien déterministe dû à [START_REF] Audibert | Fast learning rates in statistical inference through aggregation[END_REF] et la forme particulière du prior π τ (à queue lourde), on montre que cet algorithme vérifie la borne de regret suivante.

Proposition 1.2 (cf. corollaire 3.1). Supposons que, pour des constantes connues B y , B Φ > 0, les (x 1 , y 1 ), . . . , (x T , y T ) sont tels que y 1 , . . . , y T ∈ [-B y , B y ] et d j=1 T t=1 ϕ 2 j (x t ) B Φ . Alors, l'algorithme SeqSEW B,η

τ calibré avec B = B y , η = 1/(8B 2 y ) et τ = 16B 2 y /B Φ vérifie T t=1 (y t -y t ) 2 inf u∈R d T t=1 y t -u • ϕ(x t ) 2 + 32 B 2 y u 0 ln 1 + √ B Φ u 1 4 B y u 0 + 16B 2 y .
Remarquons que, si ϕ ∞ sup x∈X max 1 j d |ϕ j (x)| est fini, alors la proposition précédente fournit une borne de regret de sparsité au sens de (1.2). En effet, dans ce cas, on peut prendre B Φ = d T ϕ 2 ∞ , ce qui donne une borne de regret proportionnelle à u 0 et qui croît logarithmiquement en d, T , u 1 et ϕ ∞ .

Paramètres : seuil B > 0, température inverse η > 0 et résolution τ > 0 à laquelle on associe la loi a priori π τ sur R d défini par π τ (du) Initialisation : p 1 π τ .

A chaque tour de prévision t 1, Si le statisticien n'a pas accès à une borne a priori B y sur les observations, il peut s'adapter séquentiellement à cette borne inconnue en tronquant les prévisions u • ϕ(x t ) de fac ¸on dépendante des données. L'algorithme plus sophistiqué SeqSEW * τ produit ainsi les prévisions

1
y t R d u • ϕ(x t ) Bt p t (du) , où B t inf √ 2 k ; k ∈ Z ∩ max 1 s t-1
|y s |, +∞ , et où la probabilité a posteriori p t sur R d est définie comme précédemment mais en remplac ¸ant la température η par η t 1/(8B 2 t ) et le seuil B par B s pour chaque indice s de la somme dans (1.4). Une analyse PAC-Bayésienne plus approfondie (cf. lemme 3.2) conduit à la borne suivante.

Proposition 1.3 (cf. proposition 3.2). Pour tout τ > 0, l'algorithme SeqSEW * τ précédent vérifie

T t=1 (y t -y t ) 2 inf u∈R d T t=1 y t -u • ϕ(x t ) 2 + 64 max 1 t T y 2 t u 0 ln 1 + u 1 u 0 τ + τ 2 d j=1 T t=1
ϕ 2 j (x t ) + 32 max

1 t T y 2 t .
Au vu de la dernière proposition, la calibration de τ requiert encore la connaissance a priori d'une borne B Φ sur d j=1 T t=1 ϕ 2 j (x t ). Cela peut être évité au moyen d'une technique classique appelée doubling trick, qui donne lieu à une borne similaire (cf. théorème 3.1 et corollaire 3.4) à un facteur logarithmique près, mais pour un algorithme cette fois totalement automatique.

Adaptativité en la variance pour des données i.i.d.

Dans cette sous-section, on applique l'algorithme SeqSEW au modèle de régression avec design aléatoire (le cas du design fixe peut, dans une moindre mesure, être traité avec des techniques simi-laires, cf. section 3.4.2). Le statisticien a accès à T copies indépendantes (X 1 , Y 1 ), . . . , (X T , Y T ) de (X, Y ) ∈ X × R de loi inconnue. On suppose que E[Y 2 ] < ∞ ; l'objectif du statisticien est d'estimer la fonction de régression f : X → R définie par f (x) E[Y |X = x] pour tout x ∈ X . On pose aussi h L 2 (E[h(X) 2 ]) 1/2 pour toute fonction mesurable h : X → R.

On emploie la conversion online to batch décrite en proposition 1.1. L'échantillon (X t , Y t ) T t=1 est traité de fac ¸on séquentielle, en appliquant l'algorithme SeqSEW * τ de la date 1 à la date T avec τ = 1/ √ dT . L'estimateur f T : X → R retenu est défini par

f T (x) 1 T T t=1 R d u • ϕ(x) Bt p t (du) .
Contrairement à de nombreux travaux en statistique comme [Cat04, BN08, DT11], l'estimateur f T est totalement automatique : il ne dépend d'aucune connaissance a priori sur la loi inconnue de (X, Y ) telle que la variance du bruit E (Y -f (X)) 2 ou les normes ϕ j ∞ ou f -ϕ j ∞ (nous ne supposons d'ailleurs pas que ces dernières quantités sont finies). Nous prouvons au chapitre 3 une borne de risque pour f T valable sous de faibles hypothèses sur la loi de Y (cf. théorème 3.2 et corollaire 3.5). Nous mentionnons seulement le cas sous-gaussien ci-dessous.

Proposition 1.4 (cf. corollaire 3.6). Supposons que f ∞ < +∞ et que, pour une constante σ 2 > 0 inconnue, E e λ(Y -f (X)) X e λ 2 σ 2 /2 p.s.. Alors, pour tout T 2,

E f -f T 2 L 2 inf u∈R d f -u • ϕ 2 L 2 + 128 f 2 ∞ + 2σ 2 ln(2eT ) u 0 T ln 1 + √ dT u 1 u 0 + 1 dT d j=1 ϕ j 2 L 2 + 64 T f 2 ∞ + 2σ 2 ln(2eT ) .
Cette borne est comparable à la proposition 1 prouvée par Dalalyan et Tsybakov [START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF]. Elle vaut néanmoins sur R d tout entier au lieu de boules 1 de rayons finis, ce qui résout une question laissée ouverte dans [DT11, section 4.2]. Par ailleurs, notre algorithme ne requiert pas la connaissance a priori du facteur de variance σ 2 > 0 du bruit, ce qui résout une seconde question soulevée dans [DT11, section 5.1, remarque 6].

Régression linéaire séquentielle optimale et adaptative sur des boules 1

Au chapitre 4, nous abordons un problème proche de celui du chapitre 3 : la régression linéaire séquentielle sur des boules 1 . Nous en détaillons ci-après le cadre et nos principales contributions.

Cadre et objectif de prévision

On considère la formulation suivante 8 du problème de régression linéaire séquentielle pour des suites individuelles (cf. section 2.4 pour une introduction à ce cadre). Au début de la tâche de prévision, l'environnement choisit une suite d'observations (y t ) t 1 dans R et une suite de vecteurs de prévisions de base (x t ) t 1 dans R d , toutes deux initialement cachées au statisticien. A chaque date t ∈ N * = {1, 2, . . .}, l'environnement révèle le vecteur x t ∈ R d , le statisticien formule ensuite sa propre prévision y t ∈ R, puis l'environnement révèle l'observation y t ∈ R ; le statisticien encourt alors la perte carrée (y t -y t ) 2 .

Objectif de prévision

Etant donné un rayon U > 0 et un horizon de prévision T 1, l'objectif du statisticien est ici de prévoir presque aussi bien que le meilleur prédicteur linéaire

x ∈ R d → u • x d j=1 u j x j tel que u 1 d j=1 |u j |
U . Autrement dit, il s'agit de minimiser le regret sur la boule

1 B 1 (U ) u ∈ R d : u 1 U défini par T t=1 (y t -y t ) 2 -min u∈B 1 (U ) T t=1 (y t -u • x t ) 2 .
Cet objectif de prévision généralise la tâche d'agrégation convexe ; il est d'ailleurs possible d'être compétitif vis-à-vis de toutes les boules B 1 (U ), simultanément pour tout U > 0 (cf. fin de la section 1.3.3). Cette tâche peut s'avérer utile quand les observations y t sont correctement approchées par une combinaison linéaire u ∈ R d des prévisions de base x j,t , j = 1, . . . , d, avec une petite norme u 1 -ce qui peut être le cas, par exemple, si u est approximativement parcimonieuse. Notons enfin que la dimension ambiante d peut être petite ou grande relativement à l'horizon de prévision T : on considère tous les cas.

Dans la suite, on présente des algorithmes et des bornes sur leur regret qui valent uniformément en toutes les individuelles 9 (x t , y t ) 1 t T telles que x t ∞ X and |y t | Y pour tout t = 1, . . . , T , où X, Y > 0. Ces bornes de regret dépendent de quatre quantités importantes : U , X, Y et T , lequelles peuvent être connues ou inconnues du statisticien.

On présente ci-après les principales contributions du chapitre 4 : la première concerne la détermination de la vitesse minimax du regret (section 1.3.2), la seconde a trait à l'adaptation en les quantités X, Y , T et U lorsqu'elles sont inconnues (section 1.3.3), et la troisième consiste en un raffinement des bornes de regret via une technique appelée lipschitzification des pertes (section 1.3.4).

Vitesse optimale

Notre première contribution consiste en la détermination de l'ordre de grandeur du regret minimax sur B 1 (U ) pour des données bornées par X et Y défini par

inf ( yt) t 1 sup x 1 ∞ ,..., x T ∞ X |y 1 |,...,|y T | Y T t=1 (y t -y t ) 2 -min u 1 U T t=1 (y t -u • x t ) 2 , (1.5)
sont en fait équivalentes ; celle choisie ici est plus classique en suites individuelles, alors que celle avec ϕ était plus adaptée pour le passage au design aléatoire. 9 En fait, nos résultats sont aussi valables quand (xt, yt) t 1 est engendrée par un environnement antagoniste puisque nous ne considérons que des algorithmes déterministes. Cf. section 2.3.1 pour de plus amples détails. où le supremum est pris sur toutes les suites individuelles (x 1 , y 1 ), . . . , (x T , y T ) ∈ R d × R telles que x t ∞ X et |y t | Y pour tout t = 1, . . . , T , et où l'infimum est pris sur tous les prédicteurs séquentiels ( y t ) t 1 , i.e., toutes les suites de fonctions y t : (R d × R) t-1 × R d → R associant aux données passées (x s , y s ), 1 s t -1, et à la donnée d'entrée x t la prévision au temps t, encore notée y t par un léger abus de notation. Le regret minimax (1.5) correspond donc à la meilleure performance possible d'un prédicteur séquentiel, lorsque cette performance est évaluée en termes du regret sur B 1 (U ) dans le pire des cas.

Notre premier résultat est une borne supérieure sur le regret minimax qui, selon la valeur de U , améliore légèrement la borne de regret de l'algorithme séquentiel EG ± de [START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF] ou du prédicteur séquentiel ridge de [START_REF] Azoury | Relative loss bounds for on-line density estimation with the exponential family of distributions[END_REF][START_REF] Vovk | Competitive on-line statistics[END_REF]. En particulier, la deuxième borne améliore la borne de l'algorithme EG ± d'un facteur au plus de l'ordre de ln d. Cette borne découle d'un argument à la Maurey, qui consiste à discrétiser la boule B 1 (U ) et à montrer -par le biais d'une randomisation auxiliaire -qu'il est sensiblement équivalent de minimiser le regret sur B 1 (U ) ou sur une discrétisation judicieuse. Cet argument est classique en statistique et a été utilisé, par exemple, par [Nem00, Tsy03, BN08, SSSZ10] ; la preuve du résultat suivant montre qu'il s'adapte directement au cadre déterministe. (y t -y t ) 2 -min

u 1 U T t=1 (y t -u • x t ) 2            3 U XY 2T ln(2d) si U < Y X ln(1+2d)
T ln 2 ,

26 U XY T ln 1 + 2dY √ T U X si Y X ln(1+2d) T ln 2 U 2dY √ T X , 32 dY 2 ln 1 + √ T U X dY + dY 2 si U > 2dY X √ T .
La borne de regret précédente peut être réécrite en termes de d, Y et d'une quantité intrinsèque κ √ T U X/(2dY ) qui relie la dimension ambiante d à √ T U X/(2Y ). On obtient :

inf ( yt) t 1 sup x 1 ∞ ,..., x T ∞ X |y 1 |,...,|y T | Y T t=1 (y t -y t ) 2 -min u 1 U T t=1 (y t -u • x t ) 2        6 dY 2 κ 2 ln(2d) si κ < √ ln(1+2d) 2d √ ln 2 , 52 dY 2 κ ln(1 + 1/κ) si √ ln(1+2d) 2d √ ln 2
κ 1 , 32 dY 2 ln(1 + 2κ) + 1 si κ > 1 .

(1.6)

En petite dimension, on remarque une transition d'une borne de regret de l'ordre de √ T à une borne de l'ordre de ln T autour du point κ = 1. Cette transition est absente en grande dimension : pour d ωT , avec ω 32(ln(3) + 1) -1 , la borne de regret 32 dY 2 ln(1 + 2κ) + 1 est supérieure à la borne triviale T Y 2 pour tout κ 1. Quand κ ln(1 + 2d)/(2d √ ln 2), la borne (1.6) correspond (à une division par T près) à la vitesse optimale d'agrégation convexe dans le modèle de régression gaussienne avec de-sign aléatoire [START_REF] Tsybakov | Optimal rates of aggregation[END_REF] et à la vitesse optimale d'estimation sur des boules 1 dans le modèle de régression gaussienne avec design fixe et matrice identité10 [BM01a] (cf. aussi [START_REF] Donoho | Minimax risk over p -balls for q -error[END_REF] et [START_REF] Raskutti | Minimax rates of estimation for high-dimensional linear regression over q -balls[END_REF]). Ce fait indique que la régression linéaire sur des boules 1 n'est pas plus difficile dans un cadre de suites individuelles que dans un cadre statistique classique.

Ces deux tâches de prévision (stochastique et déterministe) sont en fait de même complexité (à des facteurs logarithmiques près) : d'après la conversion online to batch décrite à la proposition 1.1, la borne inférieure de [START_REF] Tsybakov | Optimal rates of aggregation[END_REF] pour l'agrégation convexe en design aléatoire implique une borne inférieure en suites individuelles (on exploite aussi la bornitude d'un bruit gaussien avec grande probabilité). Le résultat suivant indique que pour tout d ∈ N * , Y > 0 et κ ln(1 + 2d)/(2d √ ln 2), la borne (1.6) ne peut être améliorée de plus d'un facteur logarithmique. Cette borne inférieure étend celles de [START_REF] Cesa-Bianchi | Analysis of two gradient-based algorithms for on-line regression[END_REF][START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF], qui valent seulement pour κ petit de l'ordre de 1/d. (y t -y t ) 2 -min

u 1 U T t=1 (y t -u • x t ) 2      c 1 ln 2+16d 2 dY 2 κ ln (1 + 1/κ) si √ ln(1+2d) 2d √ ln 2 κ 1 , c 2 ln 2+16d 2 dY 2 si κ > 1 ,
où c 1 , c 2 > 0 sont des constantes absolues.

Adaptation aux paramètres du problème

Certains prédicteurs séquentiels utilisés dans la preuve du théorème 1.2 n'admettent pas de mise en oeuvre algorithmique efficace en grande dimension d (par exemple, celui utilisé conjointement avec l'argument à la Maurey), et tous utilisent la connaissance a priori de X, Y , T ou U . Une fac ¸on de surmonter ces limites est fournie par le self-confident p-norm algorithm de [START_REF] Auer | The nonstochastic multi-armed bandit problem[END_REF] ; pour p = 2 ln d, le regret de ce prédicteur sur B 1 (U ) est borné par11 8U XY

√

eT ln d + 32e U 2 X 2 ln d .

Cet algorithme est efficace et notre borne inférieure montre qu'il est optimal à un facteur logarithmique près dans le régime κ 1, i.e., en dimension d √ T U X/(2Y ), et ce sans utiliser de connaissance a priori sur X, Y et T (voir une remarque ci-après pour le régime κ > 1).

Notre deuxième contribution consiste à montrer que des propriétés d'adaptativité similaires peuvent être obtenues par pondération exponentielle, et ce pour un même coût algorithmique (linéaire en d). Plus précisément, on étudie une variante du prédicteur séquentiel EG ± de [START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF] calibré avec un paramètre évolutif η t choisi en fonction des données.

Cet algorithme séquentiel, que nous appelons algorithme EG ± adaptatif, dépend du rayon U de la boule considérée. A , 1 i d , γ ∈ {+, -} , avec ∇ i t (u) = -2(y t -u • x t )x i,t pour la perte carrée12 . Lorsque l'on choisit la suite (η t ) t 1 selon la calibration automatique (fonction de la variance cumulée du prédicteur) introduite par [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF] et définie précisément en section 2.4.3, l'algorithme EG ± adaptatif vérifie la borne de regret suivante. Cette dernière indique avec les théorèmes 1.2 et 1.3 que l'algorithme EG ± adaptatif est effectivement adaptatif (à un facteur logarithmique près) en les paramètres X, Y et T dans le régime κ 1, i.e., en dimension d √ T U X/(2Y ).

Théorème 1.4 (cf. corollaire 2.2). Soit U > 0. L'algorithme EG ± adaptatif calibré avec U et la suite (η t ) t 1 préconisée par [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF] vérifie, pour toute suite (x 1 , y 1 ), . . . , (x T , y

T ) ∈ R d × R, T t=1 (y t -u t • x t ) 2 -min u 1 U T t=1 (y t -u • x t ) 2
8U X L * T ln(2d) + 137 ln(2d) + 24 U XY + U 2 X 2 8U XY T ln(2d) + 137 ln(2d) + 24 U XY + U 2 X 2 , (1.7)

où L * T min u 1 U T t=1 (y t -u • x t ) 2 , X max 1 t T x t ∞ et Y max 1 t T |y t | sont inconnus du statisticien.
Cette borne de regret est similaire à celle de l'algorithme EG ± de [START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF], mais est obtenue sans la connaissance a priori de X, Y , ou T . Elle est également de la même forme que la borne du self-confident p-norm algorithm de [START_REF] Auer | Adaptive and self-confident on-line learning algorithms[END_REF], ce qui corrobore la proximité déjà observée par [START_REF] Gentile | The robustness of the p-norm algorithms[END_REF] entre l'algorithme p-norm et le prédicteur EG ± (avant calibration adaptative).

Au chapitre 4, nous ne détaillons l'adaptation en X, Y et T que dans le régime κ 1 (i.e., d √ T U X/(2Y )). Il est néanmoins également possible d'obtenir une borne adaptative (à un facteur logarithmique près) dans le régime κ > 1. Pour ce faire, il suffit de modifier l'algorithme de prévision ridge séquentiel défini en section 2.4.2 en tronquant ses prévisions et en le calibrant au moyen d'une technique générique appelée doubling trick. Cet algorithme peut être mis en oeuvre avec une complexité algorithmique égale à celle de l'algorithme ridge séquentiel, donc au plus de l'ordre de d 3 à chaque tour de prévision.

Remarquons que la connaissance de U est encore requise par l'algorithme EG ± adaptatif. Il est en fait possible de s'adapter à U , i.e., d'atteindre approximativement le regret minimax simultanément sur toutes les boules B 1 (U ), U > 0. A cette fin, on suppose d'abord pour simplifier que X, Y et T sont connus. Il suffit alors d'agréger plusieurs instances de l'algorithme EG ± adaptatif calibrées avec différentes valeurs de U -celles associées à une grille exponentielle de la forme {U r = U 0 2 r : r = 0, . . . , R}, où U 0 Y / X T ln(2d) ; cf. théorème 4.4. Nous expliquons ensuite, en section 4.5, comment étendre cette méthode pour construire un algorithme totalement adaptatif, i.e., qui vérifie une borne de regret similaire à (1.7) pour tout U > 0 et qui ne dépend d'aucune connaissance a priori sur U , X, Y ou T .

Une amélioration : la lipschitzification des pertes

Notre troisième contribution consiste en l'introduction d'une technique générique appelée lipschitzification des pertes. On transforme les fonctions de perte u → (y t -u • x t ) 2 (ou u → y t -u • x t α si les prévisions sont évaluées avec la perte α , α 2) en des fonctions t : R L'intérêt de la lipschitzification des pertes peut être illustré avec l'algorithme EG ± adaptatif. En effet, lorsque l'on applique ce prédicteur séquentiel aux fonctions de perte lipschitzifiées t , il vérifie la borne de regret suivante (cf. théorème 4.3) :

T t=1 (y t -y t ) 2 inf u 1 U T t=1 t (u) + 8U X inf u 1 U T t=1 t (u) ln(2d) + 153 ln(2d) + 58 U XY + U 2 X 2 + 12Y 2 .
Les deux termes principaux de cette borne améliorent légèrement ceux de la borne obtenue sans lipschitzification des pertes, puisqu'on a toujours t (u) (y t -u

• x t ) 2 pour tout u ∈ R d .
L'intérêt de la lipschitzification est plus clair pour des fonctions de perte de plus grande courbure, par exemple, x → |y t -x| α avec α > 2. Dans ce cas, l'algorithme EG ± adaptatif avec pertes lipschitzifiées a un regret qui croît au plus linéairement en U , alors que la borne de l'algorithme EG ± adaptatif sans lipschitzification donne (au premier abord du moins) une borne naïve en U α/2 . Voir la remarque 4.1 pour de plus amples détails.

Vitesses minimax des regrets interne et swap

Au chapitre 5, on étudie une instance du protocole de prévision avec avis d'experts correspondant à des pertes linéaires sur le simplexe. Ce problème de décision séquentielle est dû à [START_REF] Freund | A decision-theoretic generalization of on-line learning and an application to boosting[END_REF] et peut être décrit comme suit. A chaque date t ∈ N * = {1, 2, . . .}, le statisticien choisit un vecteur de poids p t = (p 1,t , . . . , p K,t ) sur K 2 actions différentes, i.e., p t appartient au simplexe

X K x ∈ R K + , K i=1 x i = 1 .
L'environnement révèle ensuite le vecteur de pertes t ( i,t ) 1 i K ∈ [0, 1] K ; chaque action i ∈ {1, . . . , K} encourt la perte i,t et le statisticien encourt la perte linéaire (ou perte moyenne) p t • t = K i=1 p i,t i,t . Après T 1 pas de temps, la perte cumulée du statisticien vaut T t=1 p t • t , et son objectif premier est de la minimiser.

On suppose que la suite ( t ) t 1 est fixée à l'avance par l'environnement, et on étudie les deux situations suivantes : ( t ) t 1 est déterministe arbitraire13 (i.e., c'est une suite individuelle), ou ( t ) t 1 est aléatoire i.i.d. de loi inconnue.

Les vecteurs de poids p t sont choisis en fonction des vecteurs de pertes passées et peuvent donc être vus comme des valeurs de fonctions p t ( 1 , . . . , t-1 ). On appelle stratégie (du statisticien) toute suite (p t ) t 1 de fonctions boréliennes p t : [0, 1] K(t-1) → X K . Pour simplifier les notations, on omettra souvent les dépendences et p t ( 1 , . . . , t-1 ) sera simplement noté p t .

Regret interne et regret swap

Jusqu'à présent (chapitres 3 et 4), nous avons évalué la qualité d'un prédicteur séquentiel par son regret externe. Dans le cadre considéré ici, le regret externe d'une stratégie S = (p t ) t 1 pour une suite 1:T ( 1 , . . . , T ) est défini par

R ext T (S, 1:T ) T t=1 p t • t -min 1 j K T t=1 j,t .
(1.8) Au chapitre 5, on étudie deux autres notions de regret qui jouent un rôle important en théorie des jeux : les regrets interne et swap. L'ensemble des stratégies auxquelles est comparée la stratégie S n'est plus externe comme dans (1.8) (où les stratégies de référence sont les stratégies constantes (δ i ) t 1 , avec δ i la masse de Dirac en i ∈ {1, . . . , K}) ; au contraire, il est composé de modifications de la stratégie S elle-même.

La notion de regret interne a été introduite et étudiée par [FV97, FV98, FV99] (cf. aussi [FL99, HMC00, HMC01]). Pour une stratégie S = (p t ) t 1 et une suite finie 1 , . . . , T ∈ [0, 1] K , le regret interne R int T (S, 1:T ) d'une stratégie S associé à 1:T ( 1 , . . . , T ) est défini par

R int T (S, 1:T ) T t=1 p t • t -min 1 i =j K T t=1 p i→j t • t , (1.9)
où le vecteur de poids modifié p i→j t ∈ X K est obtenu à partir de p t en remplac ¸ant l'action i par l'action j. Plus précisément, pour tout k = 1, . . . , K, la k-ème composante de p i→j t est définie par

(p i→j t ) k =      0 si k = i, p i,t + p j,t si k = j, p k,t si k / ∈ {i, j}.
Ainsi, le regret interne mesure le "regret" qu'encourt le statisticien à n'avoir pas choisi l'action j à chaque fois qu'il a choisi l'action i, et ce pour tous les couples (i, j) possibles, i = j. Intuitivement, si le statisticien minimise son regret interne, alors il bénéficie de propriétés de stabilité. Cela a été illustré en théorie des jeux : [FV97, FV99] ont montré que, dans un jeu répété randomisé entre un nombre fini de joueurs, si tous les joueurs suivent une stratégie dont le regret interne est sous-linéaire en T , la distribution empirique jointe de leurs actions converge vers un ensemble d'équilibres appelé l'ensemble des équilibres corrélés du jeu (cf. aussi [FL95, HMC00, SL07]). Le regret interne a aussi des liens historiques avec une autre branche de la théorie des jeux appelée calibration en anglais (à ne pas confondre avec parameter tuning). Ainsi, l'existence de stratégies assurant un regret interne sous-linéaire en T implique l'existence d'algorithmes de prévision bien calibrés (calibrated forecasters, cf. [START_REF] Foster | Asymptotic calibration[END_REF] par ex.).

La notion de regret swap a été introduite par [START_REF] Blum | From external to internal regret[END_REF] (voir aussi [START_REF] Greenwald | A general class of no-regret learning algorithms and gametheoretic equilibria[END_REF] pour la notion plus générale de Φ-regret). Le regret swap R sw T (S, 1:T ) d'une stratégie S = (p t ) t 1 pour une suite finie 1 , . . . , T ∈ [0, 1] K est défini par

R sw T (S, 1:T ) T t=1 p t • t -min F ∈F K T t=1 p F t • t ,
où F K désigne l'ensemble des fonctions F : {1 . . . , K} → {1 . . . , K} et où le vecteur de poids modifié p F t ∈ X K est obtenu à partir de p t en remplac ¸ant chaque action i par l'action F (i), i.e., sa j-ème composante est définie par (p F t ) j = i:F (i)=j p i,t , 1 j K.

En particulier, le regret swap d'une stratégie est plus grand que ses regrets externe et interne.

Bornes existantes sur les regrets interne et swap

Dans la suite, on détaille les contributions principales du chapitre 5, qui concernent principalement les vitesses minimax des regrets interne et swap en environnement stochastique ou déterministe.

Pour le regret interne, on s'intéresse aux deux quantités minimax suivantes. D'après la borne inférieure de [Sto05, théorème 3.3] et la borne supérieure de [START_REF] Stoltz | Internal regret in on-line portfolio selection[END_REF], on sait que14 le regret interne minimax pour des vecteurs de pertes i.i.d. (à gauche ci-dessous) et le regret interne minimax pour des suites individuelles [SL05, théorème 3] (à droite ci-dessous) vérifient l'encadrement suivant :

√ T / 64 √ 3 inf S sup Q E Q ⊗T R int T (S, 1:T ) inf S sup 1 ,..., T ∈[0,1] K R int T (S, 1:T ) √ T ln K ,
où la première inégalité vaut pour tout T K 2 /192 (les deux autres inégalités sont valides pour tout T 1), où les deux infima sont pris sur toutes les stratégies S = (p t ) t 1 , où le supremum sup Q s'étend sur toutes les probabilités sur [0, 1] K (muni de sa tribu borélienne), et où dans l'espérance, les vecteurs de pertes 1 , . . . , T ∈ [0, 1] K sont supposés i.i.d. de loi Q. On remarque un facteur

√

ln K manquant entre les bornes inférieure et supérieure.

Quant au regret swap, [START_REF] Blum | From external to internal regret[END_REF] ont construit une stratégie dont le regret swap est majoré par (T /2)K ln K uniformément sur toutes les suites individuelles (cf. aussi [START_REF] Stoltz | Internal regret in on-line portfolio selection[END_REF]). Ainsi, le regret swap minimax pour des suites individuelles est majoré comme suit :

inf S sup 1 ,..., T ∈[0,1] K R sw T (S, 1:T ) (T /2)K ln K .
Une borne inférieure sur le regret swap en suites individuelles de l'ordre de √ T K a été exhibée par [START_REF] Blum | From external to internal regret[END_REF], mais seulement dans un sens assez faible : leur borne inférieure est prouvée dans un cadre randomisé antagoniste et pour une quantité plus grande que le regret swap stricto sensu ; de plus, elle n'est prouvée que lorsque T est sous-exponentiel en K.

Vitesse minimax du regret interne dans un environnement stochastique

La première contribution du chapitre 5 est la détermination de la vitesse minimax du regret interne en environnement stochastique, qui est de l'ordre de

√

T et est donc indépendante de la dimension ambiante K.

Théorème 1.5 (cf. corollaire 5.1). Il existe des constantes absolues c 1 , c 2 , c 3 > 0 telles que, pour tout K 2 et tout T c 1 K 2 , le regret interne minimax pour des vecteurs de pertes i.i.d. vérifie

c 2 √ T inf S sup Q E Q ⊗T R int T (S, 1:T ) c 3 √ T .
En particulier, le résultat est vrai pour c 1 = 1/192 et c 2 = 1/(64 √ 3) ; la constante c 3 peut être calculée explicitement et directement à partir des fins des preuves des théorème 5.2 et corollaire 5.1, mais sa valeur n'a pas été optimisée.

La borne supérieure est obtenue de fac ¸on constructive : on montre en section 5.3 qu'une stratégie procédant par pondération exponentielle et par estimation séquentielle des espérances des pertes encourt un regret interne au plus de l'ordre de

√

T avec grande probabilité (via des outils élémentaires de concentration, par ex., l'inégalité de Hoeffding). La forme des poids choisie impose une répartition uniforme de la masse entre des pertes d'espérances proches ; cela s'avère clé pour supprimer la dépendance du regret interne minimax en la dimension ambiante K. Théorème 1.6 (cf. théorème 5.3). Il existe une constante absolue c > 0 telle que, pour tous K 2 et T max 128c 2 K 5 , K , le regret swap minimax pour des suites individuelles vérifie

inf S sup 1 ,..., T ∈[0,1] K T t=1 p t • t -min F ∈F K T t=1 p F t • t c √ T K .
En particulier, on prouve le théorème pour c = 1/ 16 128 ln(4/3) .

Cette borne inférieure est plus forte que celle de [BM07b, théorème 9], puisqu'elle vaut pour le regret swap lui-même plutôt qu'une variante randomisée dans un cadre antagoniste (qui rend l'obtention de la borne inférieure plus simple). Cela résout ainsi un problème ouvert de [BM07b, section 9]. De plus, nous nous sommes affranchis de l'hypothèse d'un horizon T sous-exponentiel en K.

Comme remarqué en section 5.4.2, notre borne inférieure de l'ordre de √ T K pointe une différence essentielle entre les regrets externe et swap. En effet, alors que le regret externe minimax est du même ordre de grandeur pour des pertes i.i.d. ou pour des suites individuelles (en l'occurence,

√

T ln K, cf. chapitre 2), le regret swap est bien plus difficile à minimiser pour des suites individuelles que pour des vecteurs de pertes i.i.d. (comparer la borne inférieure en

√

T K ci-dessus avec la vitesse minimax en

√

T ln K pour des pertes i.i.d. prouvée en section 5.4.2). La quantité minimax associée au (ψ, ϕ)-regret peut être ré-interprétée de fac ¸on stochastique à l'aide du théorème minimax suivant.

Une technique stochastique pour des majorations en suites individuelles

Théorème 1.7 (cf. théorème 5.4). Soit E un espace vectoriel réel, ψ = (ψ t ) t 1 une suite de fonctions convexes ψ t : E → R, et ϕ : R K × R K → E une fonction bi-affine. Alors, le (ψ, ϕ)regret vérifie la formule de dualité suivante :

inf S sup 1 ,..., T ∈[0,1] K ψ T T t=1 ϕ(p t , t ) = sup Q∈M + 1 ([0,1] KT ) inf S E Q ψ T T t=1 ϕ(p t , t ) ,
où les deux infima sont pris sur toutes les stratégies S = (p t ) t 1 , où M + 1 [0, 1] KT désigne l'ensemble de toutes les probabilités sur [0, 1] KT , et où l'espérance E Q [ • ] est prise par rapport aux variables aléatoires 1 , . . . , T ∈ [0, 1] K supposées de loi jointe Q.

Le théorème ci-dessus fournit un moyen non constructif de majorer le regret minimax (à gauche) ; la quantité maximin (à droite) est d'apparence plus simple à majorer car la stratégie S choisie peut dépendre de la loi jointe Q des pertes. Cette technique permet de retrouver de fac ¸on stochastique les meilleures bornes supérieures connues sur les regrets externe, interne et swap minimax, à savoir (T /2) ln K,

√

T ln K et (T /2)K ln K (cf. proposition 5.4). On l'utilise également à la proposition 5.5 pour prouver une borne supérieure de l'ordre de

√

T ln K sur le regret makespan (utile pour modéliser des problèmes de planification de tâches ou de répartition de charges), améliorant ainsi la borne d'ordre ln(K)

√

T obtenue par [START_REF] Dar | Online learning for global cost functions[END_REF].

Mentionnons qu'une technique similaire a été étudiée indépendamment par [START_REF] Rakhlin | Online learning: beyond regret[END_REF]. Puisque nous travaillons dans un cadre beaucoup plus restreint, il nous est possible d'obtenir des constantes explicites (et même optimales dans le cas du regret externe). Notre preuve du théorème 1.7 s'appuie sur des arguments simples comme la technique de bernoullisation de [START_REF] Schlag | How to minimize maximum regret in repeated decision-making[END_REF] -qui permet de recourir à une version du théorème minimax de von Neumann sans considérations topologiques fines. La majoration de la quantité maximin (à droite) repose sur des outils élémentaires de concentration de martingales comme l'inégalité de Hoeffding-Azuma, que nous combinons avec une inégalité maximale pour des variables aléatoires sous-gaussiennes. Nous renvoyons le lecteur à la section 5.5.1 pour une comparaison plus détaillée avec la littérature.

Notons que quelques questions importantes restent encore ouvertes. Tout d'abord, même si la technique stochastique décrite précédemment est utile pour mieux comprendre le problème de prévision sous-jascent (puisqu'elle permet de majorer le regret minimax associé), elle n'est pas constructive -tout comme dans [START_REF] Rakhlin | Online learning: beyond regret[END_REF]. Il est donc important, pour la suite, d'exhiber des algorithmes explicites qui atteignent les bornes supérieures nouvellement prouvées (par ex., existe-t-il un algorithme efficace dont le regret makespan est au plus de l'ordre de

√

T ln K ?). Par ailleurs, la question du facteur logarihmique manquant √ ln K entre les bornes inférieure et supérieure des regrets interne et swap est toujours partiellement ouverte. On a prouvé que le facteur √ ln K n'était pas nécessaire pour le regret interne en environnement stochastique, mais la question de savoir si cela est aussi le cas pour des suites individuelles n'est pas encore résolue.

Agrégation de modèles non linéaires

Au chapitre 6, on étudie un problème de régression en considérant des estimateurs reposant sur des techniques d'agrégation, comme aux chapitres 3 et 4, mais dans un cadre de sélection de modèles. Ce chapitre est un travail en cours.

Cadre et objectif de prévision

Le cadre considéré au chapitre 6 est un modèle linéaire gaussien généralisé introduit par [START_REF] Birgé | Gaussian model selection[END_REF] et qui inclut les modèles de régression avec design fixe et le modèle de bruit blanc gaussien. Par souci de simplicité, on se concentre ci-après sur le cas fini-dimensionnel, mais tous les résultats énoncés sont aussi prouvés dans le cas où R n est remplacé par un espace de Hilbert séparable.

Considérons ainsi le modèle de régression gaussienne avec design fixe : le statisticien observe le vecteur (Y 1 , . . . , Y n ) ∈ R n donné par

Y i = s i + σξ i ∈ R , 1 i n ,
où les variables aléatoires ξ 1 , . . . , ξ n sont i.i.d. de loi N (0, 1), où σ > 0 est le niveau de bruit supposé connu, et où s = (s 1 , . . . , s n ) ∈ R n est un vecteur déterministe inconnu.

L'objectif du statisticien est d'estimer s en fonction de

Y = (Y 1 , . . . , Y n ) ∈ R n . La perfor- mance d'un estimateur s ∈ R n est évaluée via son risque quadratique (empirique) s -s 2 n , où l'on pose u 2 n n -1 n i=1 u 2 i pour tout u ∈ R n .
Afin d'estimer s, le statisticien a accès à une famille au plus dénombrable (S m ) m∈M de parties non vides de R n (appelées modèles15 ci-après) ; il dispose alors des estimateurs des moindres carrés 16s

m ∈ argmin t∈Sm Y -t 2 n , m ∈ M .
(1.10) La tâche de prévision consiste à construire un estimateur s de s presque aussi bon que le meilleur des estimateurs parmi { s m : m ∈ M}. Par exemple, on dit que c'est le cas lorsque

E s s -s 2 n C inf m∈M E s s m -s 2 n ,
où C 1 est une constante (qui peut dépendre de la "taille" de M) et où E s désigne l'espérance prise par rapport à Y (dont la loi dépend de s). La borne de risque précédente est qualifiée d'inégalité oracle selon la terminologie de [START_REF] Donoho | Ideal spatial adaptation by wavelet shrinkage[END_REF][START_REF] Birgé | Gaussian model selection[END_REF].

Sélection et agrégation de modèles linéaires

On suppose dans cette sous-section que les modèles S m sont linéaires, i.e., qu'il s'agit de sousespaces vectoriels de S m . On rappelle ci-après -à très grands traits17 -deux approches alternatives : sélection de modèles et agrégation de modèles. Une caractéristique commune de ces deux approches est que les estimateurs s m sont combinés (ou sélectionnés) via les mêmes données que celles ayant servi à leur construction ; on ne fait pas de sample splitting, qui n'est pas adapté au cas du design fixe.

La procédure de sélection de modèles par pénalisation de [BM01a] estime s avec s = s m , où l'indice m sélectionné est défini par

m ∈ argmin m∈M Y -s m 2 n + pen(m) , avec pen(m) ( ) K σ 2 D m n 1 + 2L m 2 ;
dans l'expression ci-dessus, D m dim(S m ) désigne la dimension de S m , et K > 1 ainsi que (L m ) m∈M sont des paramètres de la procédure tels que Σ m:Dm>0 e -LmDm < ∞. Comme l'ont montré [START_REF] Birgé | Gaussian model selection[END_REF], la procédure précédente vérifie l'inégalité oracle

E s s m -s 2 n C K inf m∈M d 2 (s, S m ) + pen(m) + σ 2 n (Σ + 1) 1 + sup m∈M L m C K,Σ inf m∈M E s s m -s 2 n si égalité dans ( ), où d 2 (s, S m ) inf t∈Sm s -t 2 n et où C K , C K,Σ > 1 sont deux constantes dépendant unique- ment de K et (K, Σ) respectivement.
Ces résultats et la procédure de sélection de modèles associée ont ensuite été étendus par [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF] au cas de modèles non linaires via une notion de dimension généralisée (cf. section 1.5.3).

Plus récemment, [START_REF] Leung | Information theory and mixing least-squares regressions[END_REF] ont étudié une variante bayésienne de la procédure de sélection de modèles. Au lieu de retenir

s m , où m ∈ argmin m∈M Y -s m 2 n + pen(m) , ils considèrent la combinaison convexe m∈M ρ (η) m s m , où ρ (η) m = exp -η Y -s m 2 n + pen (η) (m) m ∈M exp -η Y -s m 2 n + pen (η) (m ) , m ∈ M ,
(1.11) avec une pénalité pen (η) qui peut maintenant dépendre de η (afin de prendre en compte une probabilité a priori sur les modèles). Comme l'ont montré [START_REF] Leung | Information theory and mixing least-squares regressions[END_REF], si η n/(4σ 2 ) et pen (η) (m) = 2σ 2 D m /n + x m /η, où les x m 0 sont tels que Σ m∈M e -xm < +∞, alors

E s m∈M ρ (η) m s m -s 2 n inf m∈M E s s m -s 2 n + x m η + ln Σ η .
(1.12)

Lorsque η = n/(4σ 2 ) et sup m x m < ∞, la borne de risque précédente est une inégalité oracle exacte, i.e., avec constante 1 devant l'infimum.

Agrégation de modèles non linéaires : contributions

La borne de risque (1.12) de [START_REF] Leung | Information theory and mixing least-squares regressions[END_REF] a été obtenue sous l'hypothèse que les modèles S m ⊂ R n sont linéaires et que les s m sont les estimateurs des moindres carrés associés (i.e., les projecteurs orthogonaux de Y ∈ R n sur les S m ). Ces travaux ont été étendus dans deux directions. D'une part, le cas de la variance inconnue a été traité par [START_REF] Giraud | Mixing least-squares estimators when the variance is unknown[END_REF]. D'autre part, [START_REF] Dalalyan | Optimal aggregation of affine estimators[END_REF] ont remplacé la famille de projecteurs orthogonaux s m m∈M par une famille quasi-arbitraire d'estimateurs affines ; cette large classe d'estimateurs inclut, par ex., les filtres diagonaux et la régression ridge à noyau. Au chapitre 6, on étend les travaux de [START_REF] Leung | Information theory and mixing least-squares regressions[END_REF] dans une troisième direction : on considère toujours des estimateurs par projection (cf. (1.10)), mais les modèles S m ⊂ R n peuvent être quasi-arbitraires (ou non linéaires). Dans une telle généralité, l'emploi de la formule d'estimation sans biais du risque de Stein [START_REF] Stein | Estimation of the mean of a multivariate distribution[END_REF] à la manière de [LB06, DT08, DS11] semble difficile. On suit à la place l'approche par concentration de [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF] pour obtenir des inégalités de type oracle avec grande probabilité (mais avec une constante devant l'infimum supérieure à 1).

Dans le même esprit que [START_REF] Leung | Information theory and mixing least-squares regressions[END_REF], on procède par pondération exponentielle : on considère l'estimateur de type Gibbs s (η) m∈M ρ

ϕ m : R + → R + est croissante, continue et telle que x ∈ R * + → x -1 ϕ m (x) est décroissante et, en posant ξ (ξ 1 , . . . , ξ n ) et <u, v> n n -1 n i=1 u i v i pour tous u, v ∈ R n , ∀u ∈ S m , ∀x > 0, 2 √ n E sup t∈Sm <ξ, t> n -<ξ, u> n t -u 2 n + x 2 x -2 ϕ m (x) .
Comme l'a montré [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF], D m mesure la taille du modèle S m (D m est liée à la notion d'entropie métrique). Par exemple, si S m est linéaire, alors on peut choisir

D m = dim(S m ) ; si S m est fini de cardinal |S m |, alors on peut choisir D m = 8 ln |S m | ; enfin, si S m = d j=1 u j ϕ j : u ∈ R d , u 1 U m pour un dictionnaire ϕ 1 , . . . , ϕ d ∈ R n , alors on peut choisir D m proportionnel à U m [MM11].
En combinant l'analyse par concentration de [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF] avec une formule de dualité sur la divergence de Kullback-Leibler (déjà utilisée au chapitre 3), on obtient l'inégalité de type oracle avec grande probabilité suivante. Comme à la section précédente, on pose d 2 (s, S m ) inf t∈Sm s -t 2 n pour tout m ∈ M.

Théorème 1.8 (cf. théorème 6.2 et remarque 6.1). Soit η > 0,

K > 1, et (x m ) m∈M ∈ R M + tel que Σ m∈M e -xm < ∞. Fixons pen (η) : M → R + telle que ∀m ∈ M, pen (η) (m) Kσ 2 n D m + √ 2x m 2 + x m η .
Alors, pour une constante

C K > 1 dépendant uniquement de K, l'estimateur s (η) = m∈M ρ (η) m s m défini en (1.10) -(1.11) vérifie, pour tout s ∈ R n et tout z > 0, avec probabilité au moins égale à 1 -Σ 2 e -z , s (η) -s 2 n C K inf m∈M d 2 (s, S m ) + pen (η) (m) + ln Σ η + σ 2 n (z + 1) -J ( ρ (η) ) , (1.13) où J (ρ) m∈M ρ m s m -s 2 n - m∈M ρ m s m -s 2 n 0.
En intégrant la borne précédente, on peut en déduire une borne en espérance. Le théorème 6.2 fournit une autre borne en espérance un peu plus fine, de type PAC-bayésien ; cf. (6.21).

Le théorème précédent pointe un lien naturel entre agrégation de modèles et sélection de modèles : notre inégalité de type oracle est valide pour un continuum d'estimateurs s (η) : η > 0 qui s'étend de l'agrégation de modèles classique (où η est au plus de l'ordre de n/σ 2 ) à la sélection de modèles (où η = +∞).

En particulier, notre estimateur agrégé m∈M ρ (η) m s m converge presque sûrement vers l'estimateur sélectionné s m de [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF] quand η → ∞ (si m est unique), et on retrouve la même borne de risque que celle de [Mas07, théorème 4.18] lorsqu'on passe à la limite dans (1.13) quand η → +∞ (cf. corollaire 6.1). Par ailleurs, pour η assez grand (au moins de l'ordre de n/σ2 ), la borne du théorème 1.8 est du même ordre de grandeur que celle de [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF]théorème 4.18]. Les mêmes applications que celles traitées par [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF][START_REF] Massart | The Lasso as an 1 -ball model selection procedure[END_REF] peuvent donc être considérées : par ex., modèles linéaires, modèles finis, ellipsoïdes de Besov, boules 1 . On en traite quelques-unes en section 6.4.1.

Nous n'avons pas encore eu le temps d'étudier en détails si l'agrégation possède de meilleures performances que la sélection de modèles pour des modèles non linéaires classiques comme ceux cités précédemment. En revanche, la borne du théorème 1.8 suggère que cela puisse être le cas à cause de la présence du terme positif J ( ρ (η) ), qui est une différence dans une inégalité de Jensen. Une autre motivation en faveur de l'agrégation est que, même dans le cas simple de modèles linéaires, il existe des situations de fort biais pour lesquelles l'agrégation est plus robuste que la sélection en termes d'excès de risque. On prouve ainsi en proposition 6.1 le fait suivant : il existe une collection de modèles linéaires

(S m ) m∈M avec |M| = 2 telle que, pour tout n 16/( √ 2 -1) 2 , ∀s ∈ R n , E s m∈M ρ (η) m s m -s 2 n inf m∈M E s s m -s 2 + 4 ln(2) σ 2 n , (1.14) ∀ m, ∃s ∈ R n , E s s m -s 2 inf m∈M E s s m -s 2 + σ 2 4 √ n , (1.15) où ρ (η)
m est défini en (1.11) avec η = n/(4σ 2 ) et pen (η) (m) = 2 dim(S m )σ 2 /n (il s'agit de l'estimateur de [START_REF] Leung | Information theory and mixing least-squares regressions[END_REF] avec la plus grande température inverse autorisée, cf. (1.12)), et où (1.15) est valide pour toute fonction de sélection m : R n → M mesurable en les données.

Les modèles linéaires S 1 , S 2 ⊂ R n et le vecteur s ∈ R n exhibés dans la preuve de (1.15) sont tels que les estimateurs des moindres carrés s 1 et s 2 associés à S 1 et S 2 possèdent un fort biais (de l'ordre de la variance du bruit σ 2 ), ont un risque E s s m -s bornes avec grande probabilité -ou de la généralité des modèles ? En particulier, quand les modèles sont linéaires, il pourrait être intéressant de retrouver via une analyse unifiée les bornes plus fines de [START_REF] Leung | Information theory and mixing least-squares regressions[END_REF] et de [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF] obtenues respectivement pour l'agrégation et la sélection de modèles.

• La question importante de la calibration du paramètre η est ouverte. Est-il possible d'identifier -au moins pour des problèmes classiques -un choix optimal de η ? Si tel est le cas, peut-on calibrer η de fac ¸on automatique et quasi-optimale ?

• Enfin, l'étude d'exemples classiques de modèles non linéaires (par ex., ellipsoïdes de Besov, boules 1 , réseaux de neurones) pourrait permettre de mieux comparer la procédure de sélection de modèles de [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF] avec les méthodes d'agrégation.

1.6 Perspectives de recherche dans la droite lignée des travaux de cette thèse

Ces travaux de thèse soulèvent plusieurs questions que nous projetons d'aborder par la suite ; nous les présentons brièvement ci-après. Ces problèmes sont à la frontière entre l'apprentissage séquentiel de suites individuelles et l'apprentissage dans un cadre statistique plus classique.

Régression linéaire séquentielle parcimonieuse

Au chapitre 3, nous importons la notion d'inégalité oracle de sparsité dans un cadre de suites déterministes arbitraires et traitons des problèmes d'adaptativité (dans un cadre déterministe dans un premier temps, puis, en corollaire, dans un cadre statistique classique). Ces résultats pourraient être prolongés de la fac ¸on suivante.

Peut-on modifier l'algorithme séquentiel SeqSEW pour produire des combinaisons linéaires parcimonieuses ? Le prédicteur séquentiel SeqSEW construit au chapitre 3 vérifie des bornes de regret de sparsité, mais ses prévisions séquentielles y t ne sont en général pas -au premier abord du moins -des combinaisons linéaires parcimonieuses des prévisions de base, i.e., des prévisions de la forme y t = u t • x t avec u t 0 T . En fait, les prévisions de l'algorithme SeqSEW ont une forme un peu plus élaborée car elles font intervenir l'opérateur de troncature avant le mélange convexe ; elles sont en effet de la forme : Peut-on prouver des bornes de parcimonie pour des algorithmes séquentiels parcimonieux ? Une autre piste de recherche, actuellement en cours, consiste à tenter de prouver des bornes de parcimonie pour des algorithmes séquentiels dont on sait qu'ils produisent des combinaisons parcimonieuses. Un exemple de tel algorithme est donné par une variante séquentielle de l'estimateur Lasso [START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF][START_REF] Donoho | Ideal spatial adaptation by wavelet shrinkage[END_REF] ; cette variante produit la prévision y t = u t • x t , où u t est donné par

y t R d u • ϕ(x t ) Bt p t (du
u t ∈ argmin u∈R d t-1 s=1 (y s -u • x s ) 2 + λ u 1
pour un paramètre de régularisation λ > 0 à calibrer judicieusement. En plus de produire des combinaisons parcimonieuses, l'algorithme précédent a l'avantage de pouvoir être implémenté18 avec un coût algorithmique faible. Cela contraste ainsi avec notre prédicteur théorique SeqSEW, qui pourrait certes être approché numériquement par des méthodes de Langevin Monte-Carlo étudiées par [START_REF] Dalalyan | Sparse regression learning by aggregation and Langevin Monte-Carlo[END_REF] dans le cadre stochastique, mais qui ne jouit pour l'instant pas de garanties théoriques quant à la précision de cette approximation.

Notons qu'on pourrait également tenter de prouver des bornes de parcimonie pour les algorithmes séquentiels de [LLZ09, SST09, Xia10, DSSST10] mentionnés en section 1.2.1.

Regret interne

Comme précisé en section 1.4.3, plusieurs questions relatives au regret interne et au regret swap sont encore ouvertes. Ainsi, la question du facteur logarithmique manquant √ ln K entre les bornes inférieure et supérieure des regrets interne et swap nécessite sans doute des techniques plus fines que celles utilisées jusqu'à présent. On décrit en section 5.6 des pistes de majoration ou de minoration.

Par ailleurs, la technique stochastique développée à la fin du chapitre 5 est utile d'un point de vue théorique (puisqu'elle permet de majorer le regret minimax), mais elle n'est pas constructive. Nous souhaiterions donc nous pencher sur la construction d'algorithmes (p t ) t 1 explicites (et efficaces) atteignant les bornes supérieures nouvellement prouvées, par exemple pour le regret makespan, dont on a majoré la valeur minimax par une quantité de l'ordre de √ T ln K. Une construction générique traitant d'emblée le regret généralisé défini en section 5.5.1 (lequel inclut regrets externe, interne, swap et makespan) serait idéale.

Agrégation de modèles non linéaires

Le dernier chapitre de la thèse présente des travaux en cours sur l'agrégation de modèles non linéaires. Comme précisé en section 1.5.4, plusieurs questions importantes sont encore ouvertes. Nous projetons ainsi d'étudier la possibilité d'obtenir des inégalités de type oracle exactes pour des modèles S m non linéaires (au moins sur des exemples classiques comme, par exemple, les ellipsoïdes de Besov et les réseaux de neurones). Ces exemples pourraient permettre de mieux comparer les procédures d'agrégation de modèles et de sélection de modèles.

Enfin, nous souhaiterions aborder la question -cruciale en pratique -de la calibration du paramètre η. Il s'agira probablement d'étudier l'existence d'une calibration optimale -au moins sur des exemples classiques -puis de chercher à imiter cette calibration à l'aide des données seulement. On pourrait par exemple mêler des arguments de calibration séquentielle proches de ceux du chapitre 3 et des idées propres à l'heuristique de pente introduite par [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF][START_REF] Arlot | Data-driven calibration of penalties for least-squares regression[END_REF].

Autres liens entre suites individuelles et sélection de modèles

Dans le cadre de la prévision avec un nombre fini d'avis d'experts (cf. figure 1.2 avec Θ = {1, . . . , K}), et pour des fonctions de perte convexes et bornées, on connaît depuis plus d'une décennie des procédures d'agrégation optimales au sens minimax. En particulier, le regret dans le pire des cas des stratégies optimales correspondantes ne peut pas être amélioré (même d'un facteur multiplicatif, cf. remarque 2.3 page 61). En revanche, les travaux plus récents de [FS97, ACBG02, ANN04, CBMS07, HK08] ont montré qu'il existe des algorithmes qui, dans des cas favorables (donc loin du pire des cas considéré pour la quantité minimax), possèdent des performances bien meilleures. Les bornes associées ont été qualifiées de bornes du premier ou second ordre (on en présente une introduction en section 2.2.2).

Dans ce cadre, un problème encore ouvert -formulé par [CBMS07] -consiste en l'obtention de bornes de regret de type oracle, i.e., des bornes de regret du second ordre qui sont un analogue séquentiel des inégalités de type oracle en sélection de modèles. Plus précisément, pour des fonctions de perte : D × Y → R bornées et convexes en leur premier argument, il s'agirait de prouver des bornes de regret de la forme T t=1 a t , y t min

1 i K T t=1 a i,t , y t + γ 1 Q i,T ln K + γ 2 E ln K , (1.17)
où γ 1 , γ 2 > 0 sont des constantes, où E max 1 t T max 1 i,j K (a i,t , y t ) -(a j,t , y t ) désigne l'étendue des pertes jusqu'à la date T , et où Q i,T est une quantité du second ordre, par exemple,

Q i,T = T t=1
2 (a i,t , y t ) ou, mieux, un terme de variance empirique

Q i,T = T t=1 (a i,t , y t ) -µ i,T 2 , avec µ i,T 1 T T t=1 (a i,t , y t ) .
Une borne de la forme (1.17) permettrait de réaliser un compromis de type biais-variance entre les experts : la perte cumulée T t=1 (a i,t , y t ) du i-ème expert joue le rôle d'une erreur d'approximation, alors que la quantité γ 1 Q i,T ln K est une mesure de la difficulté séquentielle d'estimation (et joue donc le rôle d'un terme de variance).

Les exemples de quantités du second ordre Q i,T mentionnés ci-dessus furent introduits par [CBMS07, HK08], mais ces deux travaux ne prouvent une borne de la forme (1.17) qu'au prix d'une très forte connaissance a priori sur la suite des données à prévoir (en l'occurrence, pour obtenir la borne (1.17), il convient de calibrer leurs algorithmes en fonction de la quantité Q i * T ,T , où i * T réalise le minimum dans (1.17) ; en l'absence d'un tel a priori, leurs bornes sont plus faibles). Notre objectif est donc de prouver une borne du type (1.17) pour un algorithme n'utilisant pas un tel fort a priori. Il est vraisemblable que de nouvelles techniques de calibration soient nécessaires.

Introduction

In this thesis we study sequential prediction problems that can all be cast into the following setting. A decision-maker -or forecaster -has to predict in a sequential fashion the values of an unknown sequence y 1 , y 2 , . . . of elements of an outcome space Y. His decisions a t -or predictions -belong to a decision space D, which we assume to be a convex subset of a vector space. Even if the case when D = Y is easier to interpret, D may be different from Y. The prediction task is sequential: the outcomes are only revealed one after another; at time t, the forecaster guesses the next outcome y t right before it is revealed.

In the classical statistical theory of sequential prediction, some stochastic assumptions are made on the way the sequence y 1 , y 2 , . . . is generated. For example, it may be assumed to be the realization of an ergodic stationary process. Such assumptions enable to sequentially estimate the properties of the underlying stochastic process and therefore to design statistical methods that work well when the statistical model properly describes the data at hand. This however may be unrealistic in practical problems where the process is hard to model from a statistical viewpoint and may even react to the forecaster's decisions -the last situation occurs, e.g., in computer security and computational finance.

The theory of prediction of individual sequences addresses the sequential prediction problem from a quite different angle. No stochastic assumptions whatsoever are made on the sequence of outcomes y 1 , y 2 , . . . to be predicted. Therefore, all outcome sequences are considered and we look for prediction methods that are robust in the sense that they work well even in the worst case. The name individual sequences comes from the fact that performance guarantees are proved for any arbitrary deterministic sequence y 1 , y 2 , . . . ∈ Y.

Without any stochastic model, it is not immediately clear how the prediction problem can be made meaningful and which goals are reasonable. One popular possibility is to measure the performance of a decision-maker by the loss he has accumulated in the long run, where the losses are scored by a loss function : D × Y → R. The goal of the forecaster is to minimize his cumulative loss T t=1 ( a t , y t ). Since no stochastic assumptions are made on the outcome sequence, a classical approach consists in comparing the forecaster's performance to that of reference forecasters -also called experts. Namely, we assume that at each time t, the forecaster has access to base forecasts a θ,t ∈ D, θ ∈ Θ, where Θ is a fixed index set -the a θ,t are called the experts' predictions or expert advice. Then we look for methods that guarantee that the regret

T t=1 ( a t , y t ) -inf θ∈Θ T t=1 (a θ,t , y t )
is small uniformly over all outcome sequences y 1 , y 2 , . . . ∈ Y and all expert advice sequences (a θ,1 ) θ∈Θ , (a θ,2 ) θ∈Θ , . . . ∈ D Θ . Further comments on the regret are made in Section 2.1.2 below.

The expert advice can be of quite different nature. They can correspond to statistical methods designed under different assumptions on the underlying stochastic process. Minimizing the regret above then ensures that, in the long run, the predictions of the forecaster are almost as good as that of the method associated with the unknown best statistical model (in this respect, regret minimization can be seen as a meta-statistical problem). The expert advice can also be truly deterministic predictions based on scientific modelling. Such situations occur, e.g., in daily ozone forecasting, where the experts may be numerical simulations computed from chemico-physical PDE models (electricity consumption forecasting is another fruitful example). Even worse, the experts can also be malicious opponents that react to the forecaster's decisions -as, e.g., in finance or in spam email detection problems. Since the prediction guarantees of the forecasting methods presented in the sequel hold uniformly over all individual sequences, all the examples above can be handled by the theory at hand.

A few notations

Throughout this chapter, N = {0, 1, . . .} and N * {1, 2, . . . , } denote the sets of nonnegative and positive integers respectively, and e exp(1) denotes Euler's number. Vectors are denoted by bold letters. Additional notations will be stated explicitely when necessary.

Prediction with expert advice: main framework

The problem of prediction with expert advice mentioned in the introductory paragraphs can be formulated as a repeated game between the forecaster and the environment; see At each time round t ∈ N * , 1. the environment chooses the expert advice a θ,t ∈ D for all θ ∈ Θ; they are revealed to the forecaster; 2. the forecaster chooses a point a t ∈ D, which may be kept secret or revealed a to the environment;

3. the environment chooses and reveals the outcome y t ∈ Y;

4. the forecaster incurs the loss a t , y t and each expert θ ∈ Θ incurs the loss a θ,t , y t .

a If the environment does not have access to the forecaster's predictions at, then we say that it is oblivious to the forecaster's predictions. But if the environment can react to the forecaster's past moves, then we say that it is adversarial. See Section 2.3.1 for further comments. This formulation as a repeated game is convenient to make clear all the dependences between the quantities at hand. For example, the prediction a t ∈ D of the forecaster at time t is a function of the past expert advice a θ,s θ∈Θ ∈ D Θ and outcomes y s ∈ Y, 1 s t -1, and of the current expert advice a θ,t θ∈Θ ∈ D Θ . More formally, in this setting, we call strategy of the forecaster any sequence ( a t ) t 1 of functions a t : D Θ × Y t-1 × D Θ → D. Though we most often omit these dependences for notational convenience, keeping them in mind is crucial to properly define the problem and the associated optimal performance guarantees such as the minimax regret (see Section 2.3).

Next we give some examples of decision spaces, outcome spaces, and loss functions that have been extensively studied in the online prediction protocol of Figure 2 • the Hellinger loss, which corresponds to

D = Y = [0, 1] and (a, y) = (1/2)( √ a - √ y) 2 + (1/2)( √ 1 -a - √ 1 -y) 2 ;
• the absolute loss, which corresponds to

D = [0, 1], Y = {0, 1} or Y = [0, 1],
and (a, y) = |y -a|.

• the linear loss (or mixture loss), which corresponds to D = X K , Y = [0, 1] K (for some K ∈ N * ), and (a, y) = K i=1 a i y i , where X K denotes the simplex of order K:

X K x ∈ R K + : K i=1 x i = 1 . (2.1)

A performance criterion: the (external) regret

Since the sequences of outcomes and expert advice can be totally arbitrary, it is in general unrealistic for the forecaster to try to incur at each time t the smallest possible loss inf a∈D (a, y t ) or even the loss inf θ∈Θ (a θ,t , y t ) of the best expert at time t (which may change at each round t). However, in the long run (i.e., if the forecaster and the experts are scored through their cumulative loss), predicting almost as well as the best fixed expert in hindsight is a realistic goal. Stated otherwise, it corresponds to minimizing the regret

T t=1 ( a t , y t ) -inf θ∈Θ T t=1 (a θ,t , y t ) ,
where the time horizon T may be known or unknown to the forecaster. Regret is sometimes called external regret in contrast with other forms of regret such as internal and swap regrets (the last two performance criteria are studied in Chapter 5). It can be thought of as the regret that the forecaster feels after T time steps for not following the advice of the best expert in hindsight.

The notion of regret can be interpreted as an estimation error in the statistical terminology. As noted in [START_REF] Stoltz | Agrégation séquentielle de prédicteurs : méthodologie générale et applications à la prévision de la qualité de l'air et à celle de la consommation électrique[END_REF], the cumulative loss of the forecaster up to time T can be decomposed as .

The first term is an online counterpart of an approximation error given by the cumulative loss incurred by the best expert in hindsight, while the second quantity -the regret -is an online counterpart of an estimation error, which measures the difficulty of the forecaster to mimic the best expert in hindsight while being compelled to output predictions in a sequential fashion.

To minimize his cumulative loss, the forecaster should control both the approximation and the estimation error terms. Minimizing the approximation error is an important problem both in pratice and in theory: the experts should be carefully chosen for their approximation properties (they can be, e.g., statistical estimators associated to different functional bases with various approximation properties, or numerical simulations associated with different physical models or different numerical approximation schemes). In this thesis, we focus on the other quantity -the estimation error -and study methods whose regret is small uniformly over all outcome sequences y 1 , y 2 , . . . ∈ Y and all expert advice sequences (a θ,1 ) θ∈Θ , (a θ,2 ) θ∈Θ , . . . ∈ D Θ .

When the loss function is nonnegative and bounded, the regret grows at most linearly in the number of time rounds T . Therefore, a first reasonable goal is to ensure a sublinear regret, i.e., to guarantee a vanishing worst-case per-round regret sup y 1 ,...,y T (a θ,1 ) θ ,...,(a θ,T ) θ

1 T T t=1 ( a t , y t ) -inf θ∈Θ 1 T T t=1 (a θ,t , y t ) o(1) as T → +∞ ,
where the supremum is taken over all outcome sequences y 1 , y 2 , . . . in Y and over all expert advice sequences (a θ,1 ) θ∈Θ , . . . , (a θ,T ) θ∈Θ in D Θ (the outcomes and the expert advice can be chosen adversarially, see Section 2.3.1). The above guarantee indicates that, on the average, the forecaster predicts almost as well as the best fixed expert in hindsight. As we show in Section 2.2, when Θ is finite with cardinality |Θ|, typical rates of the per-round regret are (ln |Θ|)/T and (ln |Θ|)/T .

The fact that the forecaster's and experts' predictions are scored through their cumulative losses contrasts with the stochastic batch setting (where the forecaster is given an i.i.d. sample (y 1 , . . . , y T ) from an unknown distribution, as, e.g., in the regression model with random design). In the latter setting, the performance of a statistical predictor constructed on (y 1 , . . . , y T ) are instead assessed on a new outcome y T +1 ∈ Y. The usual criterion to be minimized is the risk, i.e., the expected loss of the predictor on the next outcome (where the expectation is with respect to the distribution of the outcome). The two performance criteria -cumulative loss or risk -are of different nature. However, a close connection exists between them: we explain in Section 2.5 how to convert an online forecaster into a method suitable for a stochastic batch setting.

On the use of randomization

In this thesis all the strategies of the forecaster that we consider are deterministic. This is sufficient since both the decision space D and the functions (•, y t ) are assumed to be convex. Such assumptions are natural in many applications: online linear regression, classification with the absolute loss, sequential probability assignement, online portfolio optimization, among other examples. Next we mention important situations where these convexity assumptions are however not satisfied; in such cases, randomization is useful since it is a way to convexify the problem.

An example where D is not convex is given by the binary classification problem, where D = Y = {0, 1} and where (a, y) = I {a =y} . Assume that at each time t = 1, . . . , T , the forecaster is given two expert advice: a 1,t = 0 and a 2,t = 1. Note that whatever the forecaster plays, there is always an individual sequence y 1 , . . . , y T ∈ {0, 1} such that T t=1 ( a t , y t ) = T (it is given by y t = 1 -a t ). Since in addition one of the two experts predicts correctly for at least half of the rounds, we get that min 1 i 2 T t=1 (a i,t , y t ) T /2. Therefore, the regret of the forecaster is lower bounded by T t=1 ( a t , y t ) -min

1 i 2 T t=1 (a i,t , y t ) T 2 .
Thus, in this very simple but non-convex setting, a regret sublinear in T cannot be achieved. Had the decision space (and the loss function) been convex, e.g., had we considered D = [0, 1] and (a, y) = |a -y| instead, then the forecaster would have been allowed to choose his decisions a t as convex combinations of the expert advice a 1,t and a 2,t (since [a 1,t , a 2,t ] ⊂ D). Such weighted average predictions are key to get a sublinear regret (see the next sections), but are forbidden in the non-convex setting D = {0, 1} described above.

A way to compensate for the lack of convexity of D or (•, y t ) is to resort to a randomized strategy: at each time t, the forecaster chooses a probability distribution p t on Θ, draws an expert index θ t ∈ Θ at random from p t , and outputs the decision a t = a θt,t ∈ D. The environment has access to p t before choosing the outcome y t , but only gets to see the decision a t after revealing y t (contrary to the prediction protocol of Figure 2.1). This way, even if the environment uses the knowledge of p t to react to the forecaster's decisions, the forecaster can counteract the environment's possible diabolic movements thanks to randomization; in many applications, it yields a regret sublinear in T with high probability. This is essentially because the conditional expected loss

E ( a t , y t ) | a 1 , . . . , a t-1 = Θ (a θ,t , y t ) p t (dθ)
is linear and thus convex in the probability distribution p t : randomization is a way to convexify the problem. (Unsurprisingly, many randomized strategies are based on the ideas presented in this chapter for deterministic prediction with convex decision spaces and loss functions; the variability introduced by randomization is then handled via martingale concentration inequalities.)

The setting of randomized prediction with expert advice (with finite Θ) has been extensively studied since the seminal works of [START_REF] Blackwell | An analog of the minimax theorem for vector payoffs[END_REF] and [START_REF] Hannan | Approximation to Bayes risk in repeated play[END_REF]; see, eg, [FMG92, FV99, CBL99] and [CBL06, Chapter 4] for a thorough introduction. More recently it has been analysed under various restrictions on the information available to the forecaster. Well-known problems include:

• bandit games, where the forecaster has only access to the loss (a θt,t , y t ) of his own decision, but not to that of the other experts a θ,t , θ = θ t ; see, e.g., [Rob52, ACBF02, ACBFS02, AB09, BMSS11] and [CBL06, Chapter 6] for a detailed overview;

• label-efficient prediction, where the forecaster has only access to the outcomes y t at a small number of rounds; see, eg, [HP97, CBLS05];

• sequential prediction under partial monitoring, where the forecaster does not have access to the past outcomes y t but only to a feedback signal; see, e.g., [START_REF] Rustichini | Minimizing regret: The general case[END_REF][START_REF] Cesa-Bianchi | Regret minimization under partial monitoring[END_REF][START_REF] Lugosi | Strategies for prediction under imperfect monitoring[END_REF].

We also refer the reader to [START_REF] Audibert | Regret bounds and minimax policies under partial monitoring[END_REF] for a detailed account on minimax strategies under (combinations of) some of the above restrictive assumptions.

In this thesis, we only consider the full information setting: at the beginning of each time round t 1, the whole history (a θ,1 ) θ∈Θ , y 1 , . . . , (a θ,t-1 ) θ∈Θ , y t-1 is available to the forecaster. Moreover, since we focus on cases where both the decision space D and the functions (•, y t ) are convex, it is enough to consider only deterministic strategies.

However, even if our setting and our strategies are deterministic, we sometimes use randomization for the sake of mathematical analysis. Combined with Fano's lemma or Pinsker's inequality, randomization indeed turns out to be useful to derive lower bounds on the minimax regret (cf. Section 2.3 for the external regret and Chapter 5, Section 5.4 for the swap regret). We also use ideas based on randomization to derive upper bounds (cf. Chapter 4, Section 4.2 where we use a Maurey-type argument and Chapter 5, Section 5.5 where we restrict our attention to Bernoulli losses through a simple randomization argument).

Prediction with expert advice

In this section we present some basic results on the theory of prediction with expert advice. We consider the prediction protocol of Figure 2.1. For the sake of clarity, we assume thereafter that Θ is finite. Therefore, up to a one-to-one relabelling, we have Θ = {1, . . . , K} for some K ∈ N * . For all t 1, we index the expert advice a i,t with i ∈ {1, . . . , K}.

In the sequel, for all t 1 and all i ∈ {1, . . . , K}, we denote by i,t (a i,t , y t ) the loss of expert i at time t and by L i,t t s=1 i,s its cumulative loss up to time t (by convention, we also set L i,0 0).

In the next subsections, we focus on the celebrated exponentially weighted average forecaster and its refined variants. This forecaster benefits from interesting properties both in the online and stochastic settings. For other algorithms belonging to the more general family of weighted average forecasters or related to more specific problems, we refer the reader to the monograph [START_REF] Cesa-Bianchi | Prediction, Learning, and Games[END_REF].

The exponentially weighted average forecaster

Next we recall one of the most famous algorithms in prediction with expert advice called the exponentially weighted average forecaster. In machine learning theory this algorithm was introduced by [START_REF] Littlestone | The weighted majority algorithm[END_REF] and [START_REF] Vovk | Aggregating strategies[END_REF].

The statement of this algorithm is given in Figure 2.2. Note that the initial weight vector p 1 = (1/K, . . . , 1/K) is the uniform weight vector and that the weight p i,t assigned to expert i at each time t 2 is a smooth nonincreasing function of its past cumulative loss L i,t-1 t-1 s=1 i,s .

Parameter: η > 0.

Initialization: L i,0 0 for all i = 1, . . . , K.

At each time round t 1, 1. Receive the expert advice a 1,t , . . . , a K,t ∈ D; 2. Compute the weight vector p t = p i,t 1 i K ∈ X K defined by

p i,t
e -ηL i,t-1 K j=1 e -ηL j,t-1

, 1 i K .

3. Output the prediction a t K i=1 p i,t a i,t ∈ D;

4. Receive the outcome y t ∈ Y, compute the instantaneous losses i,t (a i,t , y t ), and update the cumulative losses L i,t L i,t-1 + i,t for all i = 1, . . . , K. Theorem 2.1. Assume that the loss function : D × Y → R is convex in its first argument and takes its values in [B 1 , B 2 ] for some constants B 1 < B 2 ∈ R. Then, for all T ∈ N * and all η > 0, and for all sequences of expert advice a i,t ∈ D and of outcomes y t ∈ Y, the regret of the exponentially weighted average forecaster with fixed parameter η is upper bounded by T t=1 a t , y t -min

1 i K T t=1 a i,t , y t ln K η + ηT (B 2 -B 1 ) 2 8 .
This bound is minimized at η = (B 2 -B 1 ) -1 8(ln K)/T and becomes (B 2 -B 1 ) (T /2) ln K.

Proof: We set W t (1/K) K i=1 e -ηL i,t-1 for all t = 1, . . . , T + 1 (recall that L i,0 0 for all i = 1, . . . , K by convention, so that W 1 = 1). Next we bound the key quantity ln(W T +1 /W 1 ) from below and above. On the one hand,

ln W T +1 W 1 = ln K i=1
e -ηL i,T -ln K ln max

1 i K e -ηL i,T -ln K = -η min 1 i K L i,T -ln K .
(2.2)

On the other hand, we can rewrite ln(W T +1 /W 1 ) as a telescopic sum and get

ln W T +1 W 1 = T t=1 ln W t+1 W t = T t=1 ln K i=1 e -ηL i,t-1 e -η i,t K i=1 e -ηL i,t-1 = T t=1 ln K i=1 p i,t e -η i,t ,
(2.3) where we used the definition of p i,t in Figure 2.2. But, by Hoeffding's lemma (see Lemma A.4 in Appendix A.5) and by the fact that the loss function is [B 1 , B 2 ]-valued, we get that, for all t = 1, . . . , T ,

ln K i=1 p i,t e -η i,t -η K i=1 p i,t i,t + η 2 (B 2 -B 1 ) 2 8 .
Substituting the last inequality in (2.3), we get

ln W T +1 W 1 -η T t=1 K i=1 p i,t i,t + η 2 T (B 2 -B 1 ) 2 8 . (2.4)
Combining the last inequality with (2.2) and dividing by η yields

T t=1 K i=1 p i,t i,t -min 1 i K L i,t ln K η + ηT (B 2 -B 1 ) 2 8 .
We conclude the proof by noting that a t , y t K i=1 p i,t a i,t , y t = K i=1 p i,t i,t for all t = 1, . . . , T (by definition of a t and by convexity of in its first argument).

The above theorem shows that for all loss functions that are bounded and convex in their first argument, the regret of the exponentially weighted average forecaster is at most of order √ T ln K. The next theorem shows that if the loss function : D×Y → R is exp-concave in its first argument (which implies convexity, but not necessarily boundedness), then the regret of the same forecaster is at most of the order of ln K (with a properly chosen η).

More precisely, following Appendix A.2, we say that a loss function : D × Y → R is η 0exp-concave in its first argument for some η 0 > 0 if the function a → e -η 0 (a,y) is concave on D for all y ∈ Y.

Among the loss functions listed in Example 2.1 above, the following are η 0 -exp-concave: the square loss on [-B, B] × [-B, B] (with η 0 = 1/(8B 2 )), the relative entropy loss (with η 0 = 1), and the Hellinger loss (with η 0 = 1). On the contrary, the linear loss and the absolute loss are not η 0 -exp-concave for any value of η 0 > 0 and therefore do not satisfy the assumptions of the following theorem. We refer the reader to [START_REF] Vovk | A game of prediction with expert advice[END_REF][START_REF] Vovk | Competitive on-line statistics[END_REF] and [START_REF] Haussler | Sequential prediction of individual sequences under general loss functions[END_REF][START_REF] Kivinen | Averaging expert predictions[END_REF] for further details on exp-concavity. Finally, note that though exp-concavity implies convexity (cf. Appendix A.2), it does not necessarily imply boundedness (e.g., the relative entropy is not bounded).

Theorem 2.2. Assume that for some η 0 > 0, the loss function : D × Y → R is η 0 -exp-concave in its first argument. Then, for all T 1, the exponentially weighted average forecaster tuned with any η ∈ (0, η 0 ] satisfies, uniformly over all sequences of expert advice a i,t ∈ D and of outcomes y t ∈ Y, T t=1 a t , y t -min

1 i K T t=1 a i,t , y t ln K η .
The above theorem is due to [START_REF] Kivinen | Averaging expert predictions[END_REF] (though stated in a slightly different form). See also [START_REF] Cesa-Bianchi | Potential-based algorithms in on-line prediction and game theory[END_REF] for an analysis in terms of the exponential potential u ∈ R K → η -1 ln K i=1 e ηu i . The following proof is however closer in spirit to [START_REF] Vovk | Competitive on-line statistics[END_REF].

Proof: We follow the same lines as for Theorem 2.1, i.e., we start from (2.2) and (2.3) and simply replace the call to Hoeffding's lemma by a sharper argument. Indeed, we upper bound the righthand side of (2.3) by noting that, for all t = 1, . . . , T ,

K i=1 p i,t e -η a i,t ,yt exp -η K i=1 p i,t a i,t , y t = exp -η a t , y t , (2.5)
where the inequality follows by concavity of a → e -η (a,yt) on D (since, by assumption, is η 0 -exp-concave in its first argument and therefore η-exp-concave for all η ∈ (0, η 0 ]; see Appendix A.2).

Taking the logarithms of both sides of the last inequality, summing it over t = 1, . . . , T , and combining it with (2.2) and (2.3), we get -η min

1 i K L i,T -ln K -η T t=1 a t , y t .
Dividing the last inequality by η and rearranging terms, we conclude the proof.

Remark 2.1 (The Aggregating Algorithm). In the proof above, the only place where the particular form of a t = K i=1 p i,t a i,t is used is in (2.5). In particular, the bound of Theorem 2.2 would have remained true for any a t ∈ D such that (2.5) holds, i.e., such that

∀y t ∈ Y , a t , y t - 1 η ln K i=1
p i,t e -η a i,t ,yt .

Any algorithm that outputs predictions a t ∈ D satisfying the above inequality is called an aggregating algorithm (see [START_REF] Vovk | Aggregating strategies[END_REF][START_REF] Vovk | A game of prediction with expert advice[END_REF][START_REF] Vovk | Competitive on-line statistics[END_REF]). An example is given by the exponentially weighted average forecaster when the loss function is η-exp-concave in its first argument. But for some loss functions, such an algorithm may exist even for values of η > 0 for which the loss function is not η-exp-concave (e.g., for the square loss on [-B, B], there is an aggregating algorithm with η = 1/(2B 2 ) while the square loss is only 1/(8B 2 )-exp concave). The existence of an aggregating algorithm is ensured by a weaker assumption than exp-concavity that is called mixability in [CBL06, Sections 3.5 and 3.6]; see [Vov90, Vov98, HKW98, KW99, Vov01] for further details.

In the subsequent chapters, we could sometimes directly address the more general aggregating algorithm instead of studying only the exponentially weighted average forecaster. This is the case, e.g., in Chapter 3 where, for the square loss, we could also use the aggregating algorithm to get similar bounds (with actually a leading constant better by a factor of 4, and without any additional difficulties). We however chose to focus on the exponentially weighted average forecaster for its popularity, its wide use in practice, its nice theoretical performance, and the various parameter tunings that have already been proposed so far.

Parameter tuning techniques

The value of the parameter η minimizing the upper bound of Theorem 2.1 depends on the range B 2 -B 1 and on the time horizon T , which may be unknown in practice. Similarly, Theorem 2.2 suggests to take η = η 0 , which may depend on quantities that are not known beforehand. For example, the square loss on D × Y = [-B, B] 2 is 1/(8B 2 )-exp-concave in its first argument (see Appendix A.2); in this case, η 0 = 1/(8B 2 ) depends on the possibly unknown range B of the outcomes and the expert advice.

Next we introduce tuning techniques to choose η in an adaptive way, i.e., that do not require any a priori knowledge on the data to be predicted (or less knowledge), while still ensuring regret bounds of the same order of magnitude.

The doubling trick

We consider the setting of Theorem 2.1; we also assume for the moment that the range B 2 -B 1 is known beforehand. A way to adapt to the unknown time horizon T is the so-called doubling trick, whose first precise analysis in machine learning theory can probably be dated back to [CBFH + 97] and [START_REF] Vovk | A game of prediction with expert advice[END_REF].

The idea underlying the doubling trick is to partition time into periods (or regimes) of exponentially increasing lengths. Traditionnally we take regimes with doubling lengths, i.e., time intervals of the form {2 r , . . . , 2 r+1 -1}, r ∈ N. At the beginning of each regime r, the exponentially weighted average forecaster is run with η tuned optimally as a function of the length of the period (i.e., η = (B 2 -B 1 ) -1 8(ln K)/2 r ). When the regime ends, the algorithm is re-initialized 1 and run on the next regime with a new value of η = (B 2 -B 1 ) -1 8(ln K)/2 r+1 . Theorem 2.3 (Adaptation to T via a doubling trick in T ). Assume that the loss function : D × Y → R is convex in its first argument and takes its values in [B 1 , B 2 ] for some known constants B 1 < B 2 ∈ R. Then, the doubling version of the exponentially weighted average forecaster described above satisfies, for all T ∈ N * and for all choices of the experts' predictions a i,t ∈ D and outcomes y t ∈ Y, T t=1 a t , y t -min

1 i K T t=1 a i,t , y t √ 2 √ 2 -1 (B 2 -B 1 ) T 2 ln K + (B 2 -B 1 ) .
A version of the above theorem can be found, e.g., in [Sto05, Theorem 2.2]. As mentioned therein, the classical choice of 2 for the regimes' lengths ratio is not optimal but close to the optimum. Note that it leads to a regret bound whose main term is within a factor of √ 2/( √ 2 -1) of the best bound of Theorem 2.1.

The main idea underlying the proof of Theorem 2.3 as well as all other applications of the doubling trick is the following. By superadditivity of the minimum, the regret on {1, . . . , T } is smaller than the sum of the regrets on each regime {2 r , . . . , 2 r+1 -1} ∩ [1, T ] such that 2 r T , i.e., such that r R log 2 T . Therefore, by Theorem 2.1, the regret on {1, . . . , T } is at most of the order of R r=0 √ 2 r ln K. The last sum is geometric and is therefore of the order of √ 2 R ln K, which is smaller than √ T ln K by definition of R. This yields the desired result.

Time-varying tunings

Next we present an improved tuning technique that does not require that the exponentially weighted average forecaster be restarted repeatedly, which is more desirable in practice. This technique consists in tuning the exponentially weighted average forecaster with a time-varying parameter η t that can depend on t but also on the whole information available to the forecaster at the beginning of the t-th round.

Parameter: sequence of functions a (η t ) t 2 where η t :

D K × Y t-1 × D K → (0, +∞).
Initialization: L i,0 0 for all i = 1, . . . , K.

At each time round t 1, 1. Access the experts' advice a 1,t , . . . , a K,t ∈ D;

2. Compute the weight vector p t = p i,t 1 i K ∈ X K defined by

p i,t e -ηtL i,t-1 K j=1 e -ηtL j,t-1
, 1 i K .

3. Output the prediction a t K i=1 p i,t a i,t ∈ D;

4. Receive the outcome y t ∈ Y, compute the instantaneous losses i,t (a i,t , y t ), and update the cumulative losses L i,t L i,t-1 + i,t for all i = 1, . . . , K.

a The parameter ηt > 0 can be chosen as a function of t and of the information available to the forecaster at the beginning of the t-th round. The corresponding algorithm is stated in Figure 2.3. The following lemma upper bounds its regret when the sequence (η t ) t 2 is nonincreasing.

Lemma 2.1 (Time-varying parameter).

Let : D × Y → R be any loss function and (η t ) t 2 be any nonincreasing sequence of positive real numbers (possibly chosen as a function of the past). Then, the exponentially weighted average forecaster tuned with η t as in Figure 2.3 satisfies, for all T ∈ N * , for all choices of the experts' predictions a i,t ∈ D and outcomes y t ∈ Y, and for all η 1 η 2 , T t=1 a t , y t -min

1 i K T t=1 a i,t , y t ln K η T +1 + T t=1 1 η t ln K i=1 p i,t e -ηt i,t -( at,yt) . (2.6)
Moreover, if is convex in its first argument, then, setting ¯ t K i=1 p i,t i,t for all t = 1, . . . , T , the regret can be further upper bounded by T t=1 a t , y t -min

1 i K T t=1 a i,t , y t ln K η T +1 + T t=1 1 η t ln K i=1 p i,t e -ηt i,t -¯ t .
(2.7)

A general way to use the above lemma is to choose a nonincreasing sequence (η t ) t 2 such that (ln K)/η T +1 is of the order the desired bound for all T 1. As for the remaining sum, note that the logarithm in (2.7) is a log-moment generating function. In the case of general convex and bounded loss functions, it can be upper bounded via Hoeffding's lemma (see Proposition 2.1 below) or via Bernstein's inequality (see Theorem 2.4 below). In the case of exp-concave loss functions, it is preferable to use (2.6) since the corresponding logarithm is nonnegative if (•, y t ) is η t -exp concave. In Chapter 3 we detail for the square loss how to choose η t so that (•, y t ) is indeed η t -exp-concave for most time rounds t (the other ones only accounting for a small regret). Lemma 2.1 above is essentially due to an argument of [START_REF] Auer | Adaptive and self-confident on-line learning algorithms[END_REF], which was then adapted by [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF]. However, it slightly improves on [CBMS07, Lemma 3] in two ways. First, the term (ln K)/η T +1 above replaces the quantity (2/η T +1 -1/η 1 ) ln K of [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF]. Our term is always smaller (since (η t ) t 1 is nonincreasing) and can be up to twice as small. See also Remark 2.2 below for a consequence of this fact. Second, the following proof, which is essentially due to [GO07, Lemma 1], is much shorter and simply relies on Jensen's inequality (see also [START_REF] Audibert | Fast learning rates in statistical inference through aggregation[END_REF]Theorem D.1] for a similar result in the batch stochastic setting under very generic assumptions).

Proof: We adapt the beginning of the proof of Theorem 2.1 to handle the case of a time-varying parameter. More precisely, instead of controlling the telescopic sum T t=1 ln(W t+1 /W t ) and dividing the resulting bounds by η, we directly control T t=1 (ln W t+1 )/η t+1 -(ln W t )/η t , where W t

(1/K) K i=1 e -ηtL i,t-1 for all t = 1, . . . , T + 1 (recall that L i,0 0 for all i = 1, . . . , K by convention, so that W 1 = 1). One the one hand, we get as in (2.2) that

ln W T +1 η T +1 - ln W 1 η 1 = 1 η T +1 ln K i=1 e -η T +1 L i,T - ln K η T +1 -min 1 i K L i,T - ln K η T +1 . (2.8)
On the other hand, we can rewrite (ln W T +1 )/η T +1 -(ln W 1 )/η 1 as a telescopic sum and get

ln W T +1 η T +1 - ln W 1 η 1 = T t=1 ln W t+1 η t+1 - ln W t η t = T t=1 ln W t+1 η t+1 - ln W t+1 η t at + 1 η t ln W t+1 W t bt ,
(2.9) where W t+1 is obtained from W t+1 by replacing η t+1 with η t , i.e., W t+1 (1/K) K i=1 e -ηtL i,t .

Let t ∈ {1, . . . , T }. As noted in [GO07, Lemma 1], the first term a t is non-positive by Jensen's inequality. Indeed, by concavity of x → x η t+1 /ηt on R * + (since 0 < η t+1 η t by assumption), we get that

W t+1 1 K K i=1 e -η t+1 L i,t = 1 K K i=1 e -ηtL i,t η t+1 /ηt 1 K K i=1 e -ηtL i,t η t+1 η t W t+1 η t+1 η t .
Taking the logarithms of both sides of the last inequality and dividing it by η t+1 , we get that (ln W t+1 )/η t+1 (ln W t+1 )/η t , so that a t 0. As for the second term b t , we get as in (2.3) that

b t 1 η t ln W t+1 W t = 1 η t ln K i=1 e -ηtL i,t-1 e -ηt i,t K i=1 e -ηtL i,t-1 = 1 η t ln K i=1 p i,t e -ηt i,t ,
where the last equality follows by definition of p i,t in Figure 2.3. Therefore, substituting the last upper bounds on a t and b t in (2.9), and combining the latter inequality with (2.8), we get that -min

1 i K L i,T - ln K η T +1 T t=1 1 η t ln K i=1 p i,t e -ηt i,t .
Adding T t=1 a t , y t to both sides of the last inequality and rearranging terms yields (2.6). As for (2.7), it follows from the fact that ( a t , y t ) ¯ t by definition of a t and by convexity of (•, y t ). This concludes the proof.

The above lemma can be used for several adaptation purposes. Next we derive a result similar to Theorem 2.3 but for the exponentially weighted average forecaster tuned with a time-varying parameter η t . In view of Theorem 2.1, this parameter is chosen as η t = (B 2 -B 1 ) -1 c ln(K)/t for some constant c > 0 (we have in mind c = 8, but it turns out that c = 4 leads to a better bound).

The next proposition is a variant of [CBL06, Theorem 2.3]. Our only modification is that we use Lemma 2.1 instead of [CBMS07, Lemma 3]. This yields an improvement of a √ 2 multiplicative factor (see the comments below).

Proposition 2.1 (Adaptation to T via a time-varying parameter). Let c > 0. Assume that the loss function : D × Y → R is convex in its first argument and takes its values in [B 1 , B 2 ] for some known constants B 1 < B 2 ∈ R. Then, the exponentially weighted average forecaster with time-varying parameter η t = (B 2 -B 1 ) -1 c ln(K)/t of Figure 2.3 satisfies, for all T ∈ N * and for all sequences of expert advice a i,t ∈ D and of outcomes y t ∈ Y, T t=1 a t , y t -min

1 i K T t=1 a i,t , y t (B 2 -B 1 ) 1 √ c + √ c 4 √ T ln K + ln K c .
This upper bound is approximately minimized with c = 4 and becomes

(B 2 -B 1 ) √ T ln K + (B 2 -B 1 ) √ ln K/2.
Note that the main term (B 2 -B 1 ) √ T ln K of the above regret bound with c = 4 is within a mutiplicative factor of √ 2 of the best bound of Theorem 2.1 (which is minimax optimal; cf. Remark 2.3 in Section 2.3). Therefore, adaptation to the unknown time horizon T is possible at the price of a multiplicative factor at most of √ 2. In particular, the above bound improves on Theorem 2.3 obtained via a doubling trick, where the price was a factor of √ 2/( √ 2 -1) ≈ 3.41 (more importantly, the forecaster is no longer repeatedly re-initialized, which may lead to better performance in practice). It also improves on the best bound known so far2 of [CBL06, Theorem 2.3], where the price was a factor of 2.

Proof: By (2.7) in Lemma 2.1, we have T t=1 a t , y t -min

1 i K T t=1 a i,t , y t ln K η T +1 + T t=1 1 η t ln K i=1 p i,t e -ηt
1 η t ln K i=1 p i,t e -ηt i,t -¯ t T t=1 1 η t η 2 t (B 2 -B 1 ) 2 8 = (B 2 -B 1 ) √ c ln K 8 T t=1 1 √ t (B 2 -B 1 ) √ c T ln K 4 ,
where the first line follows by definition of

η t = (B 2 -B 1 ) -1 c ln(K)/t,
1 i K T t=1 a i,t , y t (B 2 -B 1 ) (T + 1) ln K c + √ c T ln K 4 (B 2 -B 1 ) 1 √ c + √ c 4 √ T ln K + ln K c ,
where we used the upper bound √ T + 1 √ T + 1. This concludes the proof.

Lemma 2.1 can also be used to adapt simultaneously to the unknown time horizon T and the unknown range B 2 -B 1 . Time-varying tunings achieving this task have been proposed by at least two papers so far3 : [ACBG02] and then [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF]. The key idea in both papers is to use a sharper inequality than Hoeffding's lemma to upper bound the log-moment generating function appearing in (2.7). Next we recall the result of [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF] who use a Bernstein-type inequality to upper bound the log-moment generating function.

The most sophisticated tuning of [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF] for the exponentially weighted average forecaster relies on the following two time-varying quantities. First, for all t 1, the effective range of the losses i,s up to time t is approximated (and upper bounded) by

E t inf 2 k : k ∈ Z, 2 k max 1 s t max 1 i,j K i,s -j,s .
Second, the authors keep track of the cumulative variance of the forecaster up to time t defined by

V t t s=1 K i=1 p i,s   i,s - K j=1 p j,s j,s   2 .
Then, setting C 2( √ 2 -1)/(e -2), the time-varying parameter η t is chosen for all t 2 as η t min 1

E t-1 , C ln K V t-1 .
(2.11)

Note that η t depends on the forecaster's past predictions (through V t-1 ) and is totally parameter-free. The next theorem bounds the regret of the corresponding exponentially weighted average forecaster in terms of the cumulative variance V T .

Theorem 2.4 (Theorem 6 and Corollary 1 of [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF]).

Assume that the loss function : D×Y → R is convex in its first argument. Then, the exponentially weighted average forecaster with time-varying parameter η t defined by Figure 2 Moreover, the regret bound in (2.12) may improve significantly over the worst-case bound (B 2 -B 1 ) (T /2) ln K of Theorem 2.1. Though the latter is minimax optimal for some loss functions (see Remark 2.3 in Section 2.3), the bound in (2.12) can be much smaller in situations that are more favorable than the worst case, and in particular, when the cumulative variance V T of the forecaster is small. Note that this property is natural: if the forecaster is confident enough to rapidly concentrate its mass around the experts of smallest losses (which corresponds to a small cumulative variance), then its regret should be small. This is close in spirit to the self-confident forecasters of [START_REF] Auer | Adaptive and self-confident on-line learning algorithms[END_REF].

More generally, regret bounds that are minimax optimal (up to multiplicative factors) and that can be significantly smaller than the worst-case bound in some favorable situations are called re-fined regret bounds. Following the terminology of [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF], the bound (2.12) is called a secondorder regret bound (since the main term depends on second-order quantities like the squared losses). Previous refined regret bounds were obtained by [START_REF] Freund | A decision-theoretic generalization of on-line learning and an application to boosting[END_REF][START_REF] Auer | Adaptive and self-confident on-line learning algorithms[END_REF] and [START_REF] Allenberg-Neeman | Full information game with gains and losses[END_REF]. These papers provide first-order regret bounds, i.e., regret bounds expressed in terms of the quantities

T t=1 | i,t |, 1 i K.
In particular, in the case of nonnegative losses 0 i,t

E, [START_REF] Freund | A decision-theoretic generalization of on-line learning and an application to boosting[END_REF] showed that (a properly tuned version of) the exponentially weighted average forecaster satisfies an improvement for small losses, i.e., a regret bound of the order of EL * T ln K + E ln K, where L * T min 1 i K L i,T is the smallest cumulative loss up to time T . If L * T is much smaller than T E, then the latter regret bound is much smaller than the zero-order bound E

√

T ln K of Theorem 2.1 (hence the name of the improvement).

An improvement for small losses can actually also be derived from (2.12) above. Indeed, [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF] show in Corollary 3 therein that for nonnegative losses i,t ∈ [0, E], T t=1 a t , y t -min

1 i K T t=1 a i,t , y t 4 EL * T ln K + 39E max{1, ln K} .
(2.15)

An online PAC-Bayesian-style analysis

In the last section we mentioned several refined regret bounds satisfied by the exponentially weighted average forecaster when the latter is properly tuned. It turns out that, even for the basic exponentially weighted average forecaster (with constant parameter η > 0), another type of refinement in the regret bounds is possible -see Proposition 2.2 below. It uses the notion of Kullback-Leibler divergence and resembles risk bounds from the PAC-Bayesian literature.

We first need the following definitions. Given a measurable space (E, B), we denote by M + 1 (E) the set of all probability distributions on (E, B). Moreover, for all ρ, π ∈ M + 1 (E), the Kullback-Leibler divergence K(ρ, π) between ρ and π is defined by between two elements p = (p 1 , . . . , p K ) and q = (q 1 , . . . , q K ) of the simplex X K reads:

K(ρ, π)    E ln dρ
K(p, q) = K i=1 p i ln p i q i ,
where by convention 0 ln(0/x) = 0 for all x 0 and x ln(x/0) = +∞ for all x > 0.

As recalled in Appendix A.1, the Kullback-Leibler divergence satisfies the following key duality formula (see, e.g., [START_REF] Catoni | Statistical learning theory and stochastic optimization[END_REF] for a proof of it): for all functions h : E → [a, +∞) lower bounded by some constant a ∈ R,

-ln E e -h dπ = inf ρ∈M + 1 (E) E h dρ + K(ρ, π) .
(2.16) Moreover, the last infimum is achieved at ρ = π exp -h , where π exp -h ∈ M + 1 (E) is absolutely continuous with respect to π and is given by

dπ exp -h e -h E e -h dπ dπ .
(2.17)

The elementary equality (2.16) proves to be very useful in most papers of the PAC-Bayesian literature (see, e.g., the monographs [Cat04, Cat07, Aud04a] and the references therein for the stochastic batch setting; see also [Aud09, Section 4.2] for the online deterministic setting). We use it as a key tool in Chapters 3 and 6 for online and batch purposes respectively.

An improvement over Theorem 2.1

The above duality formula can be used to refine the regret bound of Theorem 2.1 for general convex and bounded loss functions. The next upper bound will be (somewhat abusively) called a PAC-Bayesian bound.

Proposition 2.2. Assume that the loss function : D × Y → R is convex in its first argument and takes its values in [B 1 , B 2 ] for some constants B 1 < B 2 ∈ R. Then, for all T ∈ N * and all η > 0, and for all sequences of expert advice a i,t ∈ D and outcomes y t ∈ Y, the exponentially weighted average forecaster with fixed parameter η satisfies

T t=1 a t , y t inf q∈X K K i=1 q i T t=1 a i,t , y t + K(q, p 1 ) η + ηT (B 2 -B 1 ) 2 8 ,
where p 1 = (1/K, . . . , 1/K) ∈ X K is the initial weight vector of the forecaster.

The above proposition (together with the next remarks) is essentially due to [START_REF] Freund | Using and combining predictors that specialize[END_REF] (see also [START_REF] Kivinen | Averaging expert predictions[END_REF][START_REF] Cesa-Bianchi | Analysis of two gradient-based algorithms for on-line regression[END_REF]), whose analysis is based on a telescopic argument involving the progress K(q, p t ) -K(q, p t+1 ) for any vector q ∈ X K .

More recently, [START_REF] Audibert | Fast learning rates in statistical inference through aggregation[END_REF] proved a PAC-Bayesian result on individual sequences for general losses and prediction sets. Combined with Hoeffding's lemma, [Aud09, Theorem 4.6] also yields the above proposition. As in [START_REF] Audibert | Fast learning rates in statistical inference through aggregation[END_REF], the next proof relies on the duality formula (2.16). Our analysis is however slightly simpler since we only work in a particular case of [Aud09, Theorem 4.6].

Proof: The improvement over Theorem 2.1 appears at the beginning of the proof: instead of lower bounding the sum K i=1 e -ηL i,T by max 1 i K e -ηL i,T in (2.2), we use the duality formula (2.16) with E = {1, . . . , K} and the prior p 1 = (1/K, . . . , 1/K) ∈ X K to get

ln W T +1 W 1 = ln 1 K K i=1 e -ηL i,T = -inf q∈X K η K i=1 q i L i,t + K(q, p 1 ) .
Combing the above equality with (2.4) in the proof of Theorem 2.1 and rearranging terms, we conclude the proof.

The bound of Proposition 2.2 improves on the regret bound of Theorem 2.1. To see this, it suffices to take the Dirac probability distribution q = δ i * at i * ∈ argmin 1 i K T t=1 a i,t , y t and to use that K(q, p 1 ) = ln K since p 1 = (1/K, . . . , 1/K).

Since the bound of Theorem 2.1 is minimax optimal for some loss functions (see Section 2.3), the improvement of Proposition 2.2 over Theorem 2.1 is not significant in all cases. However, it indicates that the regret term (ln K)/η of Theorem 2.1 can actually be made smaller when several experts have a cumulative loss close to the minimal one. For example, assume that the set J * of optimal experts (i.e., the experts whose cumulative loss up to time T is minimal) contains at least k 2 experts. Then, taking q = I {i∈J * } /k 1 i K as the uniform weight vector over J * , we can see from Proposition 2.2 that the term ln(K)/η can be replaced with the smaller term ln(K/k)/η. Therefore, the PAC-Bayesian bound of Proposition 2.2 better reflects the complexity of the family of experts.

Another interesting consequence of the duality formula (2.16) and of the form of the minimizer (2.17) is that, at each time t 1, the exponentially weighted average forecaster is seen to choose exactely the convex combination p t ∈ X K that minimizes the upper bound of Proposition 2.2 at time t -1, i.e., it satisfies that

p t ∈ argmin q∈X K K i=1 q i t-1 s=1 a i,s , y s + K(q, p 1 ) η .
(Put differently, p t minimizes the linearized past cumulative loss q → K i=1 q i L i,t-1 regularized by the Kullback-Leibler divergence.) Note also that the above analysis obviously remains the same if we allow the initial vector of the exponentially weighted average forecaster to be arbitrary (instead of p 1 = (1/K, . . . , 1/K)). More precisely, for any prior π ∈ X K , if we define the weights p i,t by

p i,t π i e -ηL i,t-1 K j=1 π j e -ηL j,t-1 instead of p i,t = (1/K)e -ηL i,t-1 K j=1 (1/K)e -ηL j,t-1
, then the bound of Proposition 2.2 remains true with the initial weight vector p 1 = π.

Other improvements

Following the same lines as in the proof of Proposition 2.2 above, it is also possible to improve the regret bounds of Theorem 2.2 and Lemma 2.1. The resulting improvements consist in replacing in the upper bounds the quantity

min 1 i K T t=1
a i,t , y t + ln K η by the smaller quantity

inf q∈X K K i=1 q i T t=1
a i,t , y t + K(q, p 1 ) η .

(For Lemma 2.1, the above bound is obtained for η = η T +1 .) Such improvements are not new. The improvement over Theorem 2.2 mentioned above is a consequence of [Aud09, Theorem 4.6]. As for the improvement over Lemma 2.1, a similar result in the stochastic batch setting (and that can be straightforwardly adapted to our deterministic online setting) can be found in [Aud06, Theorem D.1].

We also note that the aforementioned PAC-Bayesian-type upper bounds readily extend to the case where Θ is an arbitrary measurable space (possibly uncountably infinite); we only dealt with the finite case to be consistent with the previous sections.

Minimax regret

In this section we first define two notions of minimax regret -associated with adversarial or oblivious environments respectively -and show that these quantities are actually equal. We then prove a lower bound on the minimax regret with the linear loss that matches the upper bound of Theorem 2.1.

Let D be a decision space, Y be an outcome space, and : D × Y → R be any loss function. We consider the same setting as in Section 2.2, i.e., the prediction protocol of Figure 2.1 with a finite set of experts Θ (we use the same notations). We consider the next two definitions, which are associated with adversarial or oblivious environments respectively. Definition 2.1. We call minimax regret with an adversarial environment the quantity

inf S sup A T t=1 a t , y t -min 1 i K T t=1 a i,t , y t , (2.18)
where the infimum is taken over all strategies S = ( a t ) t 1 of the forecaster and where the supremum is taken over all strategies A = (a i,t ) 1 i K , y t t 1 of the environment. More precisely, the functions a t : D K × Y t-1 × D K → D associate5 with the past expert advice and outcomes and with the current expert advice the prediction of the forecaster at time t. The experts a i,t : D t-1 → D associate to the past predictions of the forecaster their advice at time t. The outcomes are chosen as functions y t : D t → Y of the forecaster's past and current predictions. In this case, the environment is said to be adversarial (i.e., it can react adversarially to the forecaster's predictions). where the infimum is taken over all strategies S = ( a t ) t 1 of the forecaster and where the supremum is taken over all sequences of elements (a i,1 ) 1 i K , . . . , (a i,T ) 1 i K ∈ D K and y 1 , . . . , y T ∈ Y. In this case, the environment does not react to the forecaster's past moves: it is said oblivious to the forecaster's predictions. The sequences of fixed-in-advance elements (a i,t ) 1 i K , y t t 1 are called individual sequences.

On the equivalence between oblivious and adversarial environments

In Proposition 2.3 below, which is now folklore knowledge in the theory of prediction with expert advice, we show that the quantities (2.18) and (2.19) are equal. In other words, in our setting, adversarial environments are not harder to beat than oblivious ones in a minimax sense. (This is no longer true in general when the forecaster is allowed to resort to randomization, see below.)

For notational convenience, we write a t = (a i,t ) 1 i K for all t = 1, . . . , T . In the first claim below, we also write explicitely the dependencies of the predictions a t of the forecaster on the available data (a s , y s ) s t-1 and a t . However, in the second claim, we make some slight abuse of notations by dropping these dependencies for the sake of readability.

Proposition 2.3 (Oblivious and adversarial environments are equivalent in deterministic games). Consider the prediction protocol of Figure 2.1. Let S = ( a t ) t 1 be any strategy of the forecaster. Then the following claims hold true.

• For all strategies A = (a i,t ) 1 i K , y t t 1 of the environment, the regret of S against A equals the regret of S on a particular individual sequence of expert advice and outcomes (a 1 , y 1 ), . . . , (a T , y t ) ∈ D K × Y, i.e., T t=1 a t (a s , y s ) s t-1 , a t , y t -min

1 i K T t=1 a i,t , y t ,
where the quantities a t = (a i,t ) 1 i K ∈ D K and y t ∈ Y are the values of the functions (a i,t ) 1 i K and y t evaluated at the forecasters' past predictions (see (2.20) and (2.21)).

• As a consequence, the worst-case regrets of S against adversarial environments and on individual sequences are equal:

sup A T t=1
a t , y t -min

1 i K T t=1 (a i,t , y t ) = sup a i,t ,yt T t=1
a t , y t -min

1 i K T t=1 (a i,t , y t ) ,
where the first supremum is taken over all strategies of the environment while the second supremum is restricted to the set of all individual sequences (see Definitions 2.1 and 2.2).

• Therefore, the minimax quantities (2.18) and (2.19) are equal. Their common value may be simply refered to as the minimax regret.

Proof: The third claim is straightforward, so that we only prove the first two ones.

First claim. The first claim is just a matter of rewriting things properly, i.e., with all dependencies on the past data. More formally, the sequence (a 1 , y 1 ), . . . , (a T , y t ) ∈ D K × Y is defined by a 1 a 1 and y 1 y 1 a 1 (a 1 ) and, by induction, for all t ∈ {2, . . . , T }, a i,t a i,t a 1 (a 1 ), . . . , a t-1 ((a s , y s ) s t-2 , a t-1 ) , 1 i K ,

(2.20) (2.22) For this purpose, let A = (a i,t ) 1 i K , y t t 1 be any strategy of the environment (i.e., a sequence of functions). But, by the first claim, the regret of S against the environment's strategy A satisfies T t=1 a t , y t -min

y
1 i K T t=1 a i,t , y t sup a i,t ,y t T t=1
a t , y t -min

1 i K T t=1 a i,t , y t ,
where we made a slight abuse of notation by not writing all dependencies explicitely. The last inequality yields (2.22), which concludes the proof.

In all subsequent chapters, we only consider individual sequences (i.e., the prediction game is described as if the environment were oblivious to the forecaster's predictions). By the above proposition, this is actually not a restriction and we could just as well assume that the environment were adversarial. Our choice however leads to a simpler presentation.

We stress that this equivalence is due to the fact that we only consider deterministic strategies of the forecaster. Indeed, if the forecaster were allowed to resort to randomization (cf. Section 2.1.3), then its worst-case expected regret could differ whether it were computed against adversarial environements or against individual sequences. More formally, if at each time t, the forecaster picks I t ∈ {1, . . . , K} at random according to a probability distribution p t ∈ X K built on the past data and predicts as a It,t , then there are situations for which

sup A E T t=1 a It,t , y t -min 1 i K T t=1 (a i,t , y t ) > sup a i,t ,yt E T t=1 a It,t , y t -min 1 i K T t=1 (a i,t , y t )
where the first supremum is taken over all strategies of the environment 6 while the second supre-mum is restricted to the set of all individual sequences.

Lower bound on the minimax regret

Next we show that the upper bound of Theorem 2.1 cannot be improved for the linear loss. In other words, the minimax regret associated with the linear loss is exactly of the order of √ T ln K.

More precisely, we consider the prediction protocol of Figure 2.1 with the linear loss function :

X K × [0, 1] K → R defined by (a, y) = K i=1 a i y i (see Example 2.1
). In the sequel the standard inner product between u, v ∈ R K is denoted by u • v. We also denote by

δ i = (I {j=i} ) 1 j K ∈ X K the Dirac probability distribution at i ∈ {1, . . . , K}.
Note that is convex in its first argument and takes its values in [0, 1]. Therefore, by Theorem 2.1, the minimax regret for the linear loss is at most of (T /2) ln K. In the next theorem we show that this upper bound cannot be improved by more than a constant factor (see also Remark 2.3 below about the tightness of the constant c 2 ).

Theorem 2.5 (Minimax lower bound for the linear loss).

There exist two absolute constants c 1 , c 2 > 0 such that the following holds true. Let K 1 and T c 1 ln K. Consider the prediction protocol of Figure 2.1 with D = X K , Y = [0, 1] K , and the linear loss (a, y) = a • y. Then, the minimax regret for the linear loss is lower bounded by inf S sup a i,t ,y t T t=1 a t , y t -min

1 i K T t=1 a i,t , y t c 2 T 2 ln K , (2.23)
where the infimum is taken over all strategies of the forecaster and where the supremum is taken over all individual sequences such that a i,t ∈ X K and y t ∈ [0, 1] K . In particular, we prove the theorem for c 1 40e/(2e + 1) ∈ [16.8, 16.9] and c 2 [2/(2e + 1)] e/[5(2e

+ 1)] ∈ [0.09; 0.1].
The above theorem is essentially due to [START_REF] Cesa-Bianchi | Minimizing regret with label efficient prediction[END_REF] and uses techniques of [START_REF] Auer | The nonstochastic multi-armed bandit problem[END_REF]. It relies on a probabilistic method: we lower bound the supremum of the regret over all individual sequences (y t ) t 1 by the expected regret on a suitably chosen i.i.d. random sequence (y t ) t 1 , which is at least of c 2 (T /2) ln K; see Lemma 2.2 below.

Since the upper bound of Theorem 2.1 was obtained for individual sequences, our lower bound on i.i.d. sequences mentioned above indicates that, surprisingly at first sight, minimizing the regret on individual sequences is just as hard as minimizing it on i.i.d. sequences. As we show in Chapter 5, this property in no longer true for refined notions of regret such as swap regret.

The first lower bounds on the minimax regret were also derived through a probabilistic method but were asymptotic; e.g., for the absolute loss defined in Example 2.1, [CBFH + 97] proved that

lim inf K→+∞ lim inf T →+∞ 1 (T /2) ln K inf S sup a i,t ,y t T t=1 a t , y t -min 1 i K T t=1 a i,t , y t 1 .
outcome yt, but only gets to see the decision aI t ,t after revealing yt.

Remark 2.3 (On the tightness of the constant 1/ √ 2). The asymptotic lower bound above indicates that the constant 1/ √ 2 of the upper bound of Theorem 2.1 is asymptotically tight for the absolute loss. Since the minimax regret associated with the linear loss is at least as large as that associated with the absolute loss (by convexity), it implies that the constant c 2 in (2.23) can be chosen as close to 1 as desired provided that K and T are large enough.

We refer the reader to [START_REF] Haussler | Sequential prediction of individual sequences under general loss functions[END_REF] for asymptotic lower bounds with other loss functions (via a probabilistic method) and for other deterministic techniques to derive lower bounds on individual sequences (e.g., by induction). We also refer to [START_REF] Cesa-Bianchi | On prediction of individual sequences[END_REF] and [CBL06, Chapter 8] for lower bounds associated with the absolute loss and particular families of experts (via tools from empirical process theory). See also [START_REF] Rakhlin | Online learning: Random averages, combinatorial parameters, and learnability[END_REF] for generic lower bounds on the regret (in a randomized game) in terms of combinatorial parameters or sequential Rademacher averages.

Theorem 2.5 is a straightforward consequence of the following lemma, the proof of which is essentially due to [START_REF] Cesa-Bianchi | Minimizing regret with label efficient prediction[END_REF][START_REF] Stoltz | Prédiction de suites individuelles[END_REF] and is postposed to Section 2.A. It is yet another application of Fano's lemma (see Appendix A.7), which has already proved very useful in nonparametric statistics.

Lemma 2.2. There exist two absolute constants c 1 , c 2 > 0 such that the following holds true. Let K 1 and T c 1 ln K. Consider the prediction protocol of Figure 2.1 with D = X K , Y = [0, 1] K , and the linear loss (a, y) = a • y. Then, for constant expert advice given by a i,t t 1 = δ i t 1 for all i ∈ {1, . . . , K}, we have

inf S sup (Yt)t i.i.d. E T t=1 a t , Y t -min 1 i K T t=1 δ i , Y t c 2 T 2 ln K , (2.24)
where in the last expectation, Y t 1 t T is an i.i.d. random sequence in {0, 1} K , and where the supremum is taken over all possible distributions for Y 1 (i.e., over all probability distributions on {0, 1} K ). In particular, we prove the lemma with the constants c 1 40e/(2e + 1) ∈ [16.8, 16.9] and c 2 [2/(2e + 1)] e/[5(2e + 1)] ∈ [0.09; 0.1].

Proof (of Theorem 2.5):

The proof follows straightforwardly from Lemma 2.2. Indeed, for any strategy S = a t t 1 of the forecaster, the worst-case regret of S over all individual sequences is lower bounded by

sup a i,t ,y t T t=1 a t , y t -min 1 i K T t=1 a i,t , y t sup y t T t=1 a t , y t -min 1 i K T t=1 δ i , y t E T t=1 a t , Y t -min 1 i K T t=1 δ i , Y t ,
where in the last expectation, (Y t ) t 1 is any i.i.d. sequence in {0, 1} K . We conclude the proof by using Lemma 2.2.

Remark 2.4. In the above proof, Lemma 2.2 provides an i.i.d. sequence such that the expected regret is at least of the order of √ T ln K. The distribution of Y 1 may depend on the strategy of the forecaster (note that the sup is after the inf in the statement of Lemma 2.2). However, it is also possible to construct explicitely a random sequence (Y t ) 1 t T yielding a similar lower bound but whose distribution is independent of the forecaster (but the Y T may be no longer i.i.d.). For further details, see Remark 2.5 in Section 2.A.

Theorem 2.5 indicates that the upper bound of order

√

T ln K of Theorem 2.1 cannot be improved uniformly over all convex and bounded loss functions. This does not mean that the rate √ T ln K is minimax optimal for any bounded and convex loss function. For instance, by Theorem 2.2, the minimax regret associated with exp-concave and bounded loss functions (which are in particular convex and bounded) is at most of the order of ln K and is therefore much smaller. For such losses, lower bounds of the order of ln K can usually be derived -see, e.g., [HKW98, Theorems 3.19 and 3.22] for the square loss, the relative entropy loss, and the Hellinger loss.

Online linear regression

In this section we introduce the setting of online linear regression, which we study in Chapter 3 under a sparsity scenario and in Chapter 4 for the problem of aggregation over 1 -balls.

In the sequel, u • v denotes the standard inner product between u, v ∈ R d , and we set u ∞ max 1 j d |u j | and u p d j=1 |u j | p 1/p for all p ∈ [1, +∞).

Framework

The online linear regression framework, also known as prediction with side information under the square loss (cf. [CBL06, Chapter 11]), is a particular case of the framework of prediction with expert advice that unfolds as follows. A forecaster has to predict in a sequential fashion the values y t ∈ R of an unknown sequence of observations given some input data x t ∈ R d . At each time t 1, on the basis of the newly revealed input data x t and on the past information (x s , y s ) 1 s t-1 , he outputs a prediction y t ∈ R, which is finally compared to the new observation y t through the square loss. A precise description of this repeated game is given in Figure 2.4.

In this setting the goal of the forecaster is to predict almost as well as the best linear forecaster x ∈ R d → u • x, where u ∈ R d , i.e., to satisfy, uniformly over all individual sequences (x t , y t ) 1 t T , a regret bound of the form

T t=1 y t -y t 2 inf u∈R d T t=1 y t -u • x t 2 + ∆ T,d (u) ,
for some regret term ∆ T,d (u) that should be as small as possible and, in particular, sublinear in T (actually, ∆ T,d (u) may also depend on the amplitudes of the individual sequence (x t , y t ) 1 t T such as max 1 t T x t ∞ and max 1 t T |y t |). Sublinearity in T ensures that the regret bound is non trivial: it is indeed easy to ensure a regret of T B 2 y if the observations all lie in a bounded interval [-B y , B y ] -this regret is achieved by, e.g., the constant predictions y t = 0. Moreover, Initial step: the environment chooses a sequence of observations (y t ) t 1 in R and a sequence of input data (x t ) t 1 in R d but the forecaster has not access to them.

At each time round t ∈ N * , 1. The environment reveals the input data x t ∈ R d .

2. The forecaster chooses a prediction y t ∈ R (possibly as a linear function of x t , but this is not necessary).

3. The environment reveals the observation y t ∈ R.

Each linear forecaster

x → u • x, for u ∈ R d , incurs the loss y t -u • x t 2
and the forecaster incurs the loss (y t -y t ) 2 .

Figure 2.4: The online linear regression setting.

dividing both sides by T , the above regret bound becomes

1 T T t=1 y t -y t 2 inf u∈R d 1 T T t=1 y t -u • x t 2 + ∆ T,d (u) T .
Therefore, sublinearity of ∆ T,d (u) in T implies that, on the average, the loss of the forecaster is smaller than that of each linear forecaster x → u•x up to a vanishing remainder term ∆ T,d (u)/T . The similarity of the above regret bound with risk bounds in the stochastic batch setting is exploited in Section 2.5.

The next two comments are of qualitative nature; they aim to at better comparing the different frameworks considered in this manuscript. Therefore, the reader only interested in online linear regression can skip them and go directly to Section 2.4.2.

A first comment

The setting of Figure 2.4 is a particular case of the prediction protocol of Figure 2.1 (cf. page 41) with decision and outcome spaces7 D = Y = R, with the square loss function (a, y) → (y -a) 2 , and with experts indexed by Θ = R d and predicting a u,t = u • x t at each time t 1 for all u ∈ Θ.

Another way to cast online linear regression into the prediction protocol of Figure 2.1 is the following. At each round t 1, the prediction y t is chosen as a function of the new input x t and the past data (x s , y s ) 1 s t-1 , so that the forecaster can be thought of, before observing x t , as choosing a function f t : R d × (R d × R) t-1 → R; its t-th prediction is then given by y t = f t x t ; (x s , y s ) 1 s t-1 . Therefore, another way to cast online linear regression into the prediction protocol of Figure 2.

1 is to consider Y = R d × R (the set of all pairs (x, y)), D = R R d (the set of all functions from R d to R), : D × Y → R defined by f, (x, y) y -f (x) 2 , Θ = x ∈ R d → u • x : u ∈ R d
, and constant experts' advice a θ,t = θ. This description is closer to works from the stochastic setting. We use a similar description in Section 2.5.2 for the online to batch conversion.

A second comment

The setting of Figure 2.4 is studied in Chapter 4. In Chapter 3 we consider the following generalized variant (more suited for a comparison with the stochastic setting, see Section 2.5.2 and Chapter 3). Instead of repeatedly observing pairs (x t , y t ) ∈ R d × R, the forecaster observes inputoutput pairs (x t , y t ) ∈ X × R where X is an arbitrary measurable set. At the beginning of the game, the forecaster is also given a dictionary ϕ = (ϕ 1 , . . . , ϕ d ) of base forecasters ϕ j : X → R, 1 j d (the ϕ j can be, e.g., elements of a suitably chosen functional basis or estimators associated with different statistical models). The goal of the forecaster is then to predict almost as well as the best forecaster u • ϕ d j=1 u j ϕ j for u ∈ R d . The last setting is clearly a generalization of the prediction protocol of Figure 2.4 (consider the particular case where X = R d and ϕ is the identity function). However, if the input data x t are only used through the base predictions ϕ(x t ) ∈ R d , then the two settings are equivalent.

Among the many papers that addressed the online linear regression framework, the first individual sequence analyses can be dated back to [Fos91, LLW95, CBLW96, KW97]. Next we recall a few basic algorithms in a non-chronological order together with their regret guarantees.

The sequential ridge regression forecaster

In the online linear regression framework described above, we can consider the following online analogue of the ridge regression method that [START_REF] Hoerl | Ridge regression: biased estimation for nonorthogonal problems[END_REF] introduced in the stochastic setting (linear regression model with fixed design). Its predictions are of the form y t = u t • x t , where u 1 = 0 ∈ R d and where, at each time t 2, the linear combination u t ∈ R d is the solution of the following optimization problem:

u t ∈ argmin u∈R d t-1 s=1 (y s -u • x s ) 2 + λ u 2 2 .
(2.25)

In the above equation, λ > 0 is a parameter of the algorithm. The regularization term λ u 2 2 ensures that the solution u t is unique and, more importantly, that it cannot be too far away from the null vector 0. This shrinking property (or, to see it from an online convex optimization perspective, the strong convexity of • 2 2 with respect to the norm • 2 ) is important to get non-trivial regret guarantees.

The following theorem was proved by [START_REF] Azoury | Relative loss bounds for on-line density estimation with the exponential family of distributions[END_REF] via a key telescoping argument involving linear algebra calculations (see also [START_REF] Cesa-Bianchi | Prediction, Learning, and Games[END_REF]Theorem 11.7], which combines such arguments with other ideas of [START_REF] Forster | On relative loss bounds in generalized linear regression[END_REF]).

Theorem 2.6 (Theorem 4.6 of [START_REF] Azoury | Relative loss bounds for on-line density estimation with the exponential family of distributions[END_REF]). For all λ > 0, the online algorithm described above satisfies

T t=1 y t -u t • x t 2 inf u∈R d T t=1 y t -u • x t 2 + λ u 2 2 + 4B 2 d ln 1 + T B 2 x λ ,
where

B x = max |x j,t | : 1 j d, 1 t T and B = max |y t |, | u t • x t | : 1 t T .
A drawback of the previous bound is that the quantity B depends on the amplitude of the predictions of the algorithm

| u t • x t |. It is not difficult to see that | u T • x T | 2
can be as large as d(T -1)/(4λ), so that the above regret bound only implies regret bounds that have a large dependence in T (e.g., a slow rate

√

T with λ chosen of the order of √ T , instead of a fast rate ln T as in Theorem 2.7 below). Such a situation occurs, e.g., when

y 1 = . . . , y T -1 = 1 , x 1 , . . . , x T -1 = (α, . . . , α) ∈ R d ,
and

y T = 0 , x T = (1, . . . , 1) ∈ R d .
Indeed, we can see by (2.25) and by symmetry of the problem that

u T = (T -1)α λ + (T -1)dα 2 (1, . . . , 1) .
Therefore, choosing α = λ/ d(T -1) ensures that all observations y t and base predictions x i,t lie in [-1, 1] as soon as T λ/d + 1, and that

( u T • x T ) 2 = d(T -1)/(4λ).
Fortunately it turns out that a key modification of the previous algorithm no longer suffers from this drawback. The next algorithm is due to [START_REF] Azoury | Relative loss bounds for on-line density estimation with the exponential family of distributions[END_REF] and [START_REF] Vovk | Competitive on-line statistics[END_REF]. We call8 it the sequential ridge regression forecaster throughout this manuscript; it should not be confused with the algorithm defined by (2.25). Its predictions are of the form y t = u t • x t , where u 1 = 0 ∈ R d and where, at each time t 2, the linear combination u t ∈ R d is the solution of the following optimization problem:

u t ∈ argmin u∈R d t-1 s=1 (y s -u • x s ) 2 + (u • x t ) 2 + λ u 2 2 .
(2.26)

Note that the modification consists in adding (u • x t ) 2 . This quantity can be interpreted as the proxy loss at time t of the linear forecaster x → u • x, where the unknown observation y t is replaced by 0.

Theorem 2.7 (Theorem 5.6 of [START_REF] Azoury | Relative loss bounds for on-line density estimation with the exponential family of distributions[END_REF] and Theorem 1 of [START_REF] Vovk | Competitive on-line statistics[END_REF]). For all λ > 0, the sequential ridge regression forecaster defined in (2.26) satisfies

T t=1 y t -u t • x t 2 inf u∈R d T t=1 y t -u • x t 2 + λ u 2 2 + B 2 y d ln 1 + T B 2 x λ ,
where B x = max |x j,t | : 1 j d, 1 t T and B y = max |y t | : 1 t T .

The above theorem was proved independently by [START_REF] Azoury | Relative loss bounds for on-line density estimation with the exponential family of distributions[END_REF] and by [START_REF] Vovk | Competitive on-line statistics[END_REF] via quite different arguments (see also [START_REF] Forster | On relative loss bounds in generalized linear regression[END_REF]). The proof of [AW01, Theorem 5.6] uses a key telescopic lemma combined with linear algebra calculations. As for the analysis of [Vov01, Theorem 1], it consists in interpreting the sequential ridge regression forecaster as an aggregating algorithm with continuous weights on R d and a Gaussian prior. The regret of this aggregating algorithm is then upper bounded via an analysis close to the online PAC-Bayesian-style analysis carried out in Section 2.2.3. But instead of upper bounding the right-hand side of the duality formula (2.16) via the choice of a suitable probability distribution ρ ∈ M 1 + (R d ), Vovk uses exact calculations in [Vov01, Appendix A.2] to compute the log-moment generating function appearing on the left-hand side.

The optimal a posteriori tuning of the ridge regression forecaster

We end this subsection with a comment on the tuning of the sequential ridge regression forecaster. We explain below that an (ideal) tuning of λ leads to a regret bound that depends logarithmically -as opposed to linearly -in u 2 2 . This tuning is first carried out in an illegal way (i.e., it depends the whole data sequence (x t , y t ) 1 t T ); we explain in the last two paragraphs of this subsection how to overcome this limitation.

In the sequel we set, for all λ > 0 and all u ∈ R d ,

B λ (u) T t=1 y t -u • x t 2 + λ u 2 2 + B 2 y d ln 1 + T B 2 x λ ,
so that the upper bound of Theorem 2.7 reads inf u∈R d B λ (u). To minimize it over λ ∈ R * + , we first note that

inf λ>0 inf u∈R d B λ (u) = inf u∈R d inf λ>0 B λ (u) = inf u∈R d T t=1 y t -u • x t 2 + inf λ>0 λ u 2 2 + B 2 y d ln 1 + T B 2 x λ .
By elementary calculations (i.e., derivation at the first order), the last infimum over λ > 0 can be seen to be achieved at λ = λ * (u), where

λ * (u) T B 2 x 2 -1 + 1 + 4dB 2 y T u 2 2 B 2 x .
Substituting the last expression into the previous equality, we can see that the optimal a posteriori (and therefore illegal) choice of λ leads to the ideal upper bound:

inf λ>0 inf u∈R d B λ (u) = inf u∈R d T t=1 y t -u • x t 2 + T B 2 x u 2 2 2 -1 + 1 + 4dB 2 y T u 2 2 B 2 x + B 2 y d ln 1 + 2 -1 + 1 + (4dB 2 y )/(T u 2 2 B 2 x )
.

The above upper bound does not depend linearly in u 2 2 as in Theorem 2.7 but only logarithmi-cally when u 2 → +∞ (because of the equivalent √ 1 + x ∼ 1 + x/2 when x → 0). To see it perhaps more simply, we can use the suboptimal but simpler tuning λ(u) B 2 y d/ u 2 2 to get the ideal (illegal) bound:

inf λ>0 inf u∈R d B λ (u) inf u∈R d B λ(u) (u) = inf u∈R d T t=1 y t -u • x t 2 + B 2 y d + B 2 y d ln 1 + T B 2 x u 2 2 B 2 y d . (2.27)
Therefore, if the linear forecaster u ∈ R d with smallest cumulative loss has a "small" 2 -norm (say, at most of the order of T γ for some γ > 0), then the regret on R d of the sequential ridge regression forecaster with optimal a posteriori tuning is roughly upper bounded by dB 2 y ln(T ) -note that a linear dependence in u 2 2 would yield a much worse regret bound. The bound dB 2 y ln(T ) is used in Chapter 3 to motivate the notion of sparsity regret bound.

As we already mentioned at the beginning of this subsection, the upper bound (2.27) is ideal since the optimal tuning of λ depends on the sequence (x t , y t ) 1 t T . (To be more precise, to get (2.27), it suffices to choose λ = λ( u) = B 2 y d/ u 2 2 , where u minimizes the right-hand side of (2.27) over u ∈ R d ; this tuning however still depends on the data sequence through u 2 .) One way around this is to take a grid {U r = 2 2 r : r = 0, 1, . . .} of R + and to associate with each

2 -ball {u ∈ R d : u 2 U r } a sequential ridge regression forecaster u (r) t t 1 tuned with the quasi-optimal parameter λ r = B 2 y d/U 2 r .
Then, by Theorem 2.7, we get, for all r 0,

T t=1 y t -u (r) t • x t 2 inf u 2 Ur T t=1 y t -u • x t 2 + B 2 y d + B 2 y d ln 1 + T B 2 x U 2 r B 2 y d
.

Using an exponentially weighted average forecaster (with a so-called clipping technique) to combine the sub-algorithms u (r) t t 1 , we can construct a single algorithm that almost achieves the last upper bounds uniformly over all r ∈ N, and that therefore satisfies a bound similar to (2.27). The main argument is that the square loss is exp-concave on bounded intervals. For further details, see Chapter 4, Section 4.4, where a similar double mixture is carried out for adaptation purposes.

The Exponentiated Gradient forecaster

Various gradient-based forecasters have been proposed for online linear regression, and, more generally, for online convex optimization: the gradient-descent algorithm [WH60, CBLW96, KW97, CB99], the Exponentiated Gradient forecaster [KW97, CB99], the p-norm algorithms9 [GL99, Gen03], and unifying forecasters such as the general additive algorithms10 [WJ98, KW01], the mirror descent algorithm [START_REF] Nemirovski | Problem Complexity and Method Efficiency in Optimization[END_REF][START_REF] Beck | Mirror descent and nonlinear projected subgradient methods for convex optimization[END_REF], and the composite objective mirror descent algorithm [START_REF] Duchi | Composite objective mirror descent[END_REF]. Next we recall the basic properties of the Exponentiated Gradient forecaster, which will be used in Chapter 4.

The Exponentiated Gradient forecaster was designed by [START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF] to be competitive against any vector of the simplex X d -or, by a simple trick detailed later, of 1 -balls of arbitrary radii.

We present below a generic version of this algorithm suited not only for the square loss but for general convex and differentiable loss functions11 : at each time t, the forecaster chooses a linear combination u t ∈ R d (denoted below by p t since it belongs to X d ), then the environment chooses and reveals a convex and differentiable loss function t : R d → R, and the forecaster incurs the loss t ( u t ). In online linear regression, the loss functions are given by t

(u) = (y t -u • x t ) 2 .
Let (η t ) t 2 be a sequence of nonnegative parameters that can be chosen in a sequential fashion (strictly speaking, η t is a function of 1 , . . . , t-1 ). Then, the Exponentiated Gradient forecaster tuned with (η t ) t 2 predicts at each time t as y t = p t • x t , where the weight vector

p t = (p i,t ) 1 i d ∈ X d is defined by p 1 (1/d, . . . , 1/d),
and, for all t 2, by

p i,t exp -η t t-1 s=1 ∇ i s (p s ) d j=1 exp -η t t-1 s=1 ∇ j s (p s ) , 1 i d ,
(2.28)

where ∇ i t (u) denotes the first-order partial derivative of t : R d → R in its i-th variable at the point u -e.g., for the square loss

t (u) = (y t -u • x t ) 2 , we have ∇ i t (u) = -2(y t -u • x t )x i,t .
An automatic tuning for the Exponentiated Gradient algorithm with general loss functions Several regret bounds have been derived for the Exponentiated Gradient algorithm. Originally analysed with the square loss by [START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF], it was later studied for more general loss functions by [START_REF] Cesa-Bianchi | Analysis of two gradient-based algorithms for on-line regression[END_REF]. A general and simple analysis for arbitrary differentiable convex loss functions can also be found in [CBL06, Section 2.5]. The latter analysis probably gives the best intuition on the Exponentiated Gradient algorithm. It relies on the fact that this algorithm is nothing but an exponentially weighted average forecaster applied to the loss vectors ∇ t (p t )

∇ i t (p t ) 1 i d ∈ R d , t 1.
In the next corollary we use the fully automatic exponentially weighted average forecaster of [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF]; it yields an Exponentiated Gradient algorithm for which (η t ) t 2 is tuned in a fully automatic way. More precisely, replacing the losses with the gradients of the losses in (2.11), we set, for all t 2, η t min 1

E t-1 , C ln K V t-1 , (2.29) where C 2( √ 2 -1)/(e -2)
and where

E t-1 inf k∈Z 2 k : 2 k max 1 s t-1 max 1 j,k d ∇ j s (p s ) -∇ k s (p s ) , V t-1 t-1 s=1 d j=1 p j,s ∇ j s (p s ) - d k=1 p k,s ∇ k s (p s ) 2 .
The next proposition is a direct consequence of [CBMS07, Corollary 1] (see Theorem 2.4). We denote the gradient of t at any u ∈ R d by ∇ t (u) ∇ 1 t (u), . . . , ∇ d t (u) . The reason why we do not bound ∇ t (p t ) 2

∞ uniformly over all t = 1, . . . , T within the square root will become clear in Corollary 2.2 (application to the square loss).

Proposition 2.4 (The Exponentiated Gradient algorithm with automatic tuning).

Assume that the forecaster is repeatedly given a convex differentiable12 loss function t : R d → R, and that he uses the Exponentiated Gradient algorithm defined in (2.28) with (η t ) t 2 given by Equation (2.29). Then, for all T 1 and all sequences 1 , . . . , T ,

T t=1 t (p t ) -min q∈X d T t=1 t (q) 4 T t=1 ∇ t (p t ) 2 ∞ ln d + (8 ln d + 12) max 1 t T ∇ t (p t ) ∞ .
Proof: Since t : R d → R is differentiable and convex for all t = 1, . . . , T , we get that

T t=1 t (p t ) -min q∈X d T t=1 t (q) = max q∈X d T t=1 t (p t ) -t (q) max q∈X d T t=1 ∇ t (p t ) • (p t -q)
= max

1 i d T t=1 ∇ t (p t ) • (p t -e i ) (2.30) = T t=1 d i=1 p i,t ∇ i t (p t ) -min 1 i d T t=1 ∇ i t (p t ) , (2.31)
where (2.30) follows from the fact that q → T t=1 ∇ t (p t ) • (p t -q) is affine (convexity is sufficient) on the polytope X d , the vertices of which are denoted by e i I {j=i} 1 j d . But, by definition of p t and η t above, we can apply Theorem 2.4 with the linear loss (cf. Example 2.1 on page 42), the observations y t = ∇ t (p t ) ∈ R d , and the constant expert advice a i,t = e i ∈ X d to get that

T t=1 d i=1 p i,t ∇ i t (p t ) -min 1 i d T t=1 ∇ i t (p t ) 4 T t=1 ∇ t (p t ) 2 ∞ ln d + (4 ln d + 6) 2 max 1 t T ∇ t (p t ) ∞ ,
where we used the fact that, in our case, the effective ranges

E t max 1 i,j d ∇ i t (p t ) - ∇ j t (p t ) in (2.13) are upper bounded as E t 2 ∇ t (p t ) ∞ .
This concludes the proof.

Extension to 1 -balls

In the previous paragraphs, we showed that the Exponentiated Gradient algorithm is competitive against the whole simplex X d . A limitation is that the vectors of X d are restricted to have nonnegative components and an 1 -norm bounded by 1. Next, we overcome this limitation via a trick due to [START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF] and that transforms the Exponentiated Gradient forecaster into an algorithm which is competitive against all vectors of the 1 -ball

B 1 (U ) u ∈ R d : u 1 U for a given U > 0.
In the general case of convex and differentiable loss functions t : R d → R, the trick of [KW97] unfolds as follows. First note that the polytope B 1 (U ) is the convex hull of its 2d vertices ±U e i , i = 1, . . . , d (recall that e i I {j=i} 1 j d ). Therefore, for all u ∈ B 1 (U ), there exists13 a convex combination p

= (p + 1 , p - 1 , . . . , p + d , p - d ) ∈ X 2d such that u = d i=1 p + i (U e i ) + p - i (-U e i ) = U d i=1 p + i -p - i e i .
The last remark suggests to apply the Exponentiated Gradient algorithm defined in (2.28)-(2.29) to the augmented loss function

(U ) t : R 2d → R defined for all v = (v + 1 , v - 1 , . . . , v + d , v - d ) ∈ R 2d by (U ) t (v) t U d i=1 v + i -v - i e i .
(2.32)

Denote the resulting weight vectors by

p t = (p + 1,t , p - 1,t , . . . , p + d,t , p - d,t ) ∈ X 2d
. Then, we call adaptive EG ± algorithm on B 1 (U ) the forecaster that outputs the linear combinations u t ∈ B 1 (U ) given by

u t = U d i=1 p + i,t -p - i,t e i , t = 1, . . . , T .
A formal definition of the adaptive EG ± algorithm on B 1 (U ) is given in Figure 2.5. Note that this forecaster takes as input parameter the radius U of the 1 -ball B 1 (U ). The form of the update (2.33) follows from the fact that, for all t = 1, . . . , T and all

v = (v + 1 , v - 1 , . . . , v + d , v - d ) ∈ R 2d , d (U ) t dv + j (v) = U ∇ j t U d i=1 v + i -v - i e i and d (U ) t dv - j (v) = -U ∇ j t U d i=1 v + i -v - i e i ,
so that, by definition of u t above, we have, for all j ∈ {1, . . . , d} and all γ ∈ {+, -},

d (U ) t dv γ j (p t ) = γU ∇ j t ( u t ) , (2.34)
where, by a slight abuse of notation, the symbols "+" and "-" also denote the values +1 and -1 respectively (e.g., γU should be understood as -U if γ = -). We can use Proposition 2.4 to bound the regret of the adaptive EG ± algorithm as follows.

Corollary 2.1 (The adaptive EG ± algorithm for general convex and differentiable loss functions).

Let U > 0. Then, the adaptive EG ± algorithm on B 1 (U ) defined in Figure 2.5 satisfies, for all T 1 and all sequences of convex and differentiable14 loss functions 1 , . . . , T : R d → R,

T t=1 t ( u t ) -min u: u 1 U T t=1 t (u) 4U T t=1 ∇ t ( u t ) 2 ∞ ln(2d) + U 8 ln(2d) + 12 max 1 t T ∇ t ( u t ) ∞ .
In particular, the regret is bounded by 4U max 1 t T ∇ t ( u t ) ∞ T ln(2d) + 2 ln(2d) + 3 .

Parameter: radius U > 0.

Initialization:

p 1 = (p + 1,1 , p - 1,1 , . . . , p + d,1 , p - d,1 ) 1/(2d), . . . , 1/(2d) ∈ R 2d .
At each time round t 1,

1. Output the linear combination u t U d j=1 p + j,t -p - j,t e j ∈ B 1 (U );
2. Get the (convex and differentiable) loss function t : R d → R and define

z + j,s U ∇ j s ( u s ) and z - j,s -U ∇ j s ( u s ) , j = 1, . . . , d, s = 1, . . . , t , E t inf k∈Z    2 k : 2 k max 1 s t max 1 j,k d γ,µ∈{+,-} z γ j,s -z µ k,s    , V t t s=1 1 j d γ∈{+,-} p γ j,s     z γ j,s - 1 k d µ∈{+,-} p µ k,s z µ k,s     2 ;
3. Update the parameter η t+1 according to

η t+1 min 1 E t , C ln K V t , where C 2( √ 2 -1)/(e -2) ;
4. Update the weight vector

p t+1 = (p + 1,t+1 , p - 1,t+1 , . . . , p + d,t+1 , p - d,t+1
) ∈ X 2d defined for all j = 1, . . . , d and γ ∈ {+, -} by

p γ j,t+1 exp -η t+1 t s=1 γU ∇ j s ( u s ) 1 k K µ∈{+,-} exp -η t+1 t s=1 µU ∇ k s ( u s )
.

(2.33)

Figure 2.5: The adaptive EG ± algorithm on B 1 (U ) for general convex and differentiable loss functions (cf. Corollary 2.1).

Proof: The result follows straightforwardly from Proposition 2.4 by noting that, on the one hand, by definitions of (U ) t , p t , and u t ,

T t=1 t ( u t ) = T t=1 (U )
t (p t ) and min

u: u 1 U T t=1 t (u) = min q∈X 2d T t=1 (U ) t (q) ,
and, on the other hand, ∇ (U )

t (p t ) ∞ = U ∇ t ( u t )
∞ for all t = 1, . . . , T (by (2.34)).

An improvement for small losses under the square loss

In the particular case of the square loss t (u) = (y t -u•x t ) 2 , the gradients are given by ∇ t (u) = -2(y t -u • x t ) x t for all u ∈ R d . Applying Corollary 2.1, we get the following improvement for small losses.

Corollary 2.2 (An improvement for small losses under the square loss).

Let U > 0. Consider the online linear regression setting. Then, the adaptive EG ± algorithm on B 1 (U ) defined in Figure 2.5 with the loss functions t : u → (y t -u • x t ) 2 satisfies, for all sequences of (x 1 , y 1 ), . . . ,

(x T , y T ) ∈ R d × R, T t=1 (y t -u t • x t ) 2 L * T + 8U X L * T ln(2d) + 137 ln(2d) + 24 U XY + U 2 X 2 ,
where the quantities We point out that the large constants 137 and 24 above can be improved. This can be done, e.g., by using the tighter bound of Remark 2.2 on page 54, instead of the original bound of [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF]. Note that this also reduces the constant 8 of the main regret term to 8 × 1.32/2 = 5.28.

L * T min {u∈R d : u 1 U } T t=1 (y t -u • x t ) 2 , X max 1 t T x t ∞ ,
The above bound is comparable to the bound 2U X 2B ln(2d) + 2U 2 X 2 ln(2d) implied by [KW97, Theorem 5.11], where B is a known upper bound on L * T . Note that our main term is larger than that of [START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF] by a multiplicative factor of 2 √ 2; our lower-order term is also larger by (quite large 15 ) multiplicative factors and by an additional term of the order of U XY ln(2d). However, to get their bound, [START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF] tuned the EG ± algorithm as a function of B and of two known upper bounds X and Y on the input data and the observations (cf. the choice of η = ( ln(2d))/(U X √ 2B + 2U 2 X 2 ln(2d)) therein). On the contrary, the version of the EG ± algorithm we use does not require the knowledge of B, X, and Y .

Thus, Corollary 2.2 is of the same flavor as the regret bound of [ACBG02, Theorem 3.1] for the self-confident p-norm algorithm. Indeed, for a given parameter U > 0 and for p = 2 ln d, [START_REF] Auer | Adaptive and self-confident on-line learning algorithms[END_REF] show that the cumulative loss L T of the self-confident p-norm algorithm 16 satisfies, 15 See the comment on the constants 137 and 24 above. 16 The vectors output by the self-confident p-norm algorithm with parameter U lie in the q -ball {u ∈ R d : u q U }, where q is the conjugate of p, i.e., q = p/(p -1)

≈ 1 + 1/(2 ln d) if p = 2 ln d. This q -ball contains the 1 -ball B1(U ) but is only a slight overapproximation of it since e -1 • 1 • q • 1 .
for some nonnegative quantity k T (2e ln d)U 2 X 2 ,

L T L * T + 8k T + 8 (k T L * T )/2 + k 2 T L * T + 8U X (e ln d) L * T + (32e ln d) U 2 X 2 ,
where the quantities

L * T min {u∈R d : u 1 U } T t=1 (y t -u • x t ) 2 , X max 1 t T x t ∞ ,
and Y max 1 t T |y t | are unknown to the forecaster. The fact that we got a similar bound is not surprising because the p-norm algorithm is known to share many properties with the EG ± algorithm (in the limit p → +∞ with an appropriate initial weight vector, or for p of the order of ln d with a zero initial weight vector, cf. [START_REF] Gentile | The robustness of the p-norm algorithms[END_REF]). The bound of Corollary 2.2 corroborates this similarity.

Proof (of Corollary 2.2): We apply Corollary 2.1 with the square loss

t (u) = (y t -u • x t ) 2 : T t=1 t ( u t ) -min u: u 1 U T t=1 t (u) 4U T t=1 ∇ t ( u t ) 2 ∞ ln(2d) + U 8 ln(2d) + 12 max 1 t T ∇ t ( u t ) ∞ .
Using the equality ∇ t (u) = -2(y t -u • x t ) x t for all u ∈ R d , we get that, on the one hand, by the upper bound x t ∞ X,

∇ t ( u t ) 2 ∞ 4X 2 t ( u t ) , (2.35)
and, on the other hand, max 1 t T ∇ t ( u t ) ∞ 2(Y + U X)X (indeed, by Hölder's inequality,

u t • x t u t 1 x t ∞ U X).
Substituting the last two inequalities in the bound of Corollary 2.1, setting L T T t=1 t ( u t ), and recalling that

L * T min {u∈R d : u 1 U } T t=1 t (u), we get that L T L * T + 8U X L T ln(2d) + 16 ln(2d) + 24 U XY + U 2 X 2 C .
Solving for L T via Lemma A.2 in Appendix A.4, we get that

L T L * T + C + 8U X ln(2d) L * T + C + 8U X ln(2d) 2 L * T + 8U X L * T ln(2d) + 8U X C ln(2d) + 64U 2 X 2 ln(2d) + C .
Using that

U X C ln(2d) = U X ln(2d) 16 + 24/ ln(2d) U XY + U 2 X 2 U 2 X 2 + U XY ln(2d) 16 + 24/ ln(2) U XY + U 2 X 2 = 16 + 24/ ln(2) U XY + U 2 X 2 ln(2d)
and performing some simple upper bounds concludes the proof.

In the last corollary we used the adaptive EG ± algorithm with the square loss functions

t : u → (y t -u • x t ) 2 .
In Chapter 4 we use yet another instance of the adaptive EG ± algorithm that we call the Lipschitzifying Exponentiated Gradient (LEG) algorithm. It corresponds to the adaptive EG ± algorithm applied not to the square loss but to a Lipschitz continuous modification t : R d → R of the square loss. Both the modified loss function t and the threshold used to perform an additional clipping are updated as a function of the available data only. Applying Corollary 2.1 again, we show in Theorem 4.3 of Chapter 4 that the cumulative loss L T of the LEG algorithm is upper bounded by

L T L * T + 8U X L * T ln(2d) + 153 ln(2d) + 58 U XY + U 2 X 2 + 12Y 2 ,
where

L * T min {u∈R d : u 1 U } T t=1 t (u)
is the optimal cumulative Lipschitzified loss within B 1 (U ). The main two terms of the last bound slightly improve on those of Corollary 2.2 since, by Figure 4.2 of Chapter 4, we always have

L * T min u∈R d : u 1 U T t=1 (y t -u • x t ) 2 .
As explained therein, the improvement brought about by the Lipschitzification step is more significant for loss functions with higher curvature than the square loss, e.g., loss functions of the form u → y t -u • x t α with α > 2.

From online to batch bounds

In this section we explain how to convert an online algorithm into a method suitable for a probabilistic batch setting. We then point out that, contrary to a common belief, online methods can be used in the regression model with random design even if the outputs are unbounded.

The online-to-batch conversion

Let D be a convex decision space, Z be an outcome space17 , and : D ×Z → R be a loss function convex in its first argument. In the sequel we consider the following batch stochastic setting. The forecaster is given at the beginning of the game T independent random copies Z 1 , . . . , Z T of Z ∈ Z, whose common distribution is unknown. The goal of the forecaster is to predict the next outcome Z T +1 ∼ Z almost as well as does any fixed element in a non-empty subset Θ ⊂ D. More precisely, its goal is to output a decision a T ∈ D based on the sample (Z 1 , . . . , Z T ) so as to minimize its excess expected risk

E a T , Z -inf a∈Θ E a, Z , (2.36)
where the expectation on the left is taken with respect to all sources of randomness (i.e., with respect to (Z 1 , . . . , Z T ) and Z).

We illustrate in the sequel the links between this batch18 setting and the online framework of prediction with expert advice studied in the previous sections. More precisely, consider the online protocol of Figure 2.1 when the outcome space is Z and when the expert advice are constant and given by a θ,t = θ for all θ ∈ Θ and all t 1. In this particular setting, the online protocol of Figure 2.1 can be rewritten as follows: a forecaster repeatedly outputs a decision a t ∈ D, observes the new outcome z t ∈ Z, and incurs the loss ( a t , z t ); after T time steps, its regret against the set of experts Θ reads:

T t=1 ( a t , z t ) -inf a∈Θ T t=1 (a, z t ) .
(2.37) (The above online protocol is known as online convex optimization when Θ = D; see, e.g., [START_REF] Zinkevich | Online convex programming and generalized infinitesimal gradient ascent[END_REF][START_REF] Shalev-Shwartz | Stochastic convex optimization[END_REF].)

Next we show that any online algorithm-i.e., any strategy of the forecaster-( a t ) t 1 that has a small regret (2.37) in the above online protocol can be converted into a method a T that has a small excess expected risk (2.36) in the batch stochastic setting.

The following online-to-batch conversion is a standard trick in the machine learning community that can be traced back to around [START_REF] Littlestone | From on-line to batch learning[END_REF] (see also the earlier references given in [START_REF] Dekel | Data-driven online to batch conversions[END_REF]). High-probability data-dependent risk bounds for arbitrary convex decision spaces and convex and bounded loss functions were derived by [START_REF] Cesa-Bianchi | On the generalization ability of on-line learning algorithms[END_REF]. The latter paper also addresses the case when either the decision space or the (bounded) loss function is not convex via a more sophisticated online-to-batch conversion. Several improved high-probability risk bounds were then obtained by [START_REF] Cesa-Bianchi | Improved risk tail bounds for on-line algorithms[END_REF] in the possibly non-convex setting and by [START_REF] Zhang | Data dependent concentration bounds for sequential prediction algorithms[END_REF][START_REF] Kakade | On the generalization ability of online strongly convex programming algorithms[END_REF] for convex decision spaces and "Bernstein-friendly" loss functions (e.g., strongly convex losses).

The conversion consists in treating the sample Z 1:T (Z 1 , . . . , Z T ) in a sequential fashion: even if all the Z t are known at the beginning of the game, they are only used one at a time from round 1 to round T , that is, the online algorithm ( a t ) t 1 sequentially outputs its decisions a t (Z 1:t-1 ) ∈ D based on the past data Z 1:t-1 (Z 1 , . . . , Z t-1 ), t = 1, . . . , T ( a 1 is deterministic). Finally, the simplest way to define a T (Z 1:T ) when D is convex is to consider the average:

a T (Z 1:T ) = 1 T T t=1 a t (Z 1:t-1 ) .
Proposition 2.5. Let D be a convex decision space, Z be an outcome space, and : D × Z → R be a loss function convex in its first argument. Let ( a t ) t 1 be any online algorithm and (R T ) T 1 be any real-valued sequence such that, for all T 1, uniformly over all z 1 , . . . , z T ∈ Z,

T t=1 ( a t , z t ) -inf a∈Θ T t=1 (a, z t ) R T .
(2.38)

Then the above online to batch conversion applied to ( a t ) t 1 yields a procedure a T such that, for all i.i.d. samples

(Z 1 , . . . , Z T ) ∈ Z T , E a T , Z -inf a∈Θ E a, Z R T T ,
where the expectations are taken with respect to both the sample (Z 1 , . . . , Z T ) and a random variable Z ∈ Z independent of (Z 1 , . . . , Z T ) and distributed as Z 1 .

Proof: In the sequel we explicitely write all the dependencies a t (Z 1:t-1 ) and a T (Z 1:T ). By assumption, the regret bound (2.38) holds uniformly over all individual sequences (z 1 , . . . , z T ) ∈ Z T . Therefore, almost surely,

T t=1 a t (Z 1:t-1 ), Z t inf a∈Θ T t=1 (a, Z t ) + R T .
(2.39)

In particular the last inequality holds in expectation, so that, dividing by T and using the fact that

E inf a∈Θ T t=1 (a, Z t ) inf a∈Θ E T t=1 (a, Z t ) , we get 1 T E T t=1 a t (Z 1:t-1 ), Z t 1 T inf a∈Θ E T t=1 (a, Z t ) + R T T = inf a∈Θ E (a, Z) + R T T , (2.40)
where the last equality follows from the fact that Z t and Z are identically distributed for all t = 1, . . . , T . We conclude the proof via Jensen's inequality: by definition of a T (Z 1:T ) and by convexity of in its first argument, we have

E a T (Z 1:T ), Z = E 1 T T t=1 a t (Z 1:t-1 ), Z 1 T T t=1 E a t (Z 1:t-1 ), Z = 1 T T t=1 E a t (Z 1:t-1 ), Z t ,
where we used the fact that (Z 1:t-1 , Z) and (Z 1:t-1 , Z t ) are identically distributed (since Z and Z t are identically distributed and both independent of Z 1:t-1 ). Combining the last inequality with (2.40) concludes the proof.

Note that Z T was not used to construct the procedure a T (Z 1:T ). Taking instead the average a T (Z 1:T ) (T + 1) -1 T +1 t=1 a t (Z 1:t-1 ) up to time T + 1, we can see from the above analysis that it has an excess risk upper bounded by R T +1 /(T + 1), which is usually smaller than R T /T . (This was carried out, e.g., in [START_REF] Audibert | Fast learning rates in statistical inference through aggregation[END_REF][START_REF] Bartlett | Online prediction[END_REF].) More importantly, the following improvements or extensions are possible:

(a) We assumed that R T was a uniform upper bound on the regret of ( a t ) t 1 (cf. (2.38)). The above analysis also works if R T = R T (z 1 , . . . , z T ) is data-dependent. The risk bound becomes:

E a T , Z -inf a∈Θ E a, Z E R T (Z 1:T ) T .
(b) Proposition 2.5 tranforms the almost-sure regret bound (2.39) into a risk bound in expectation. It is also possible to transform (2.39) into a high-probability data-dependent risk bound via standard martingale concentration tools (e.g., the Hoeffding-Azuma inequality or Bernstein's inequality for martingales). See [START_REF] Cesa-Bianchi | Improved risk tail bounds for on-line algorithms[END_REF] for a different conversion suited for general decision spaces and bounded loss functions (not necessarily convex) and [START_REF] Zhang | Data dependent concentration bounds for sequential prediction algorithms[END_REF][START_REF] Kakade | On the generalization ability of online strongly convex programming algorithms[END_REF] for the same conversion as the one studied above but in the particular case of "Bernsteinfriendly" losses (e.g., strongly convex).

(c) As showed in [START_REF] Bartlett | Online prediction[END_REF], the online-to-batch conversion described above can be slightly modified to handle sequences (Z 1 , . . . , Z T , Z) that are generated by a stationary processindependence is no longer required. This can be achieved by defining

a T (Z 1:T ) = 1 T + 1 T t=0 a T -t+1 (Z t+1:T ) .
In this case the online algorithm is repeatedly re-initialized: for each t = 0, . . . , T , the algorithm is restarted and run on the sub-sample Z t+1:T (Z t+1 , . . . , Z T ) of length T -t (so that the corresponding decision function is a T -t+1 ). By similar arguments, this procedure is seen to satisfy the bound

E ( a T , Z) -inf a∈Θ E (a, Z) R T +1 T + 1 .

Application: regression model with random design and unbounded outputs

In this section, we focus on the online-to-batch conversion from the online linear regression setting to the regression model with random design. The case of a fixed design is not addressed here, but it can to some extent be dealt with via similar techniques (see Section 3.4.2 in Chapter 3). In the sequel, we first make some preliminary comments and introduce the regression model with random design together with some related aggregation problems. Afterward we study the onlineto-batch conversion in the easy case of bounded outputs. We then discuss a trick of [START_REF] Bunea | Sequential procedures for aggregating arbitrary estimators of a conditional mean[END_REF] to deal with unbounded outputs when the forecaster has access to some prior knowledge on the regression function and on the dictionary at hand. We finally explain how to overcome the last limitation, i.e., how to design a fully automatic online algorithm whose batch conversion satisfies adaptivity properties.

Preliminary comments

In the last section we showed that, unsurprisingly, worst-case regret bounds imply risk bounds in expectation. In this respect, individual sequence prediction methods can be thought of as being more robust than standard batch methods as they can be lead to controls under almost no assumption on the data at hand. The only assumptions are on the loss function (e.g., boundedness and convexity, exp-concavity) and on the set of experts' indices Θ (e.g., finiteness, convexity).

A major criticism that has however sometimes been issued as far as the square loss is concerned is that only bounded outputs can be dealt with in online (deterministic) linear regression, while this restriction fails even in a stochastic setting as simple as the regression model with random design and Gaussian noise. It actually turns out that, with additional care, individual sequence methods can still be used in this setting. This was illustrated by [START_REF] Bunea | Sequential procedures for aggregating arbitrary estimators of a conditional mean[END_REF], and we make further progress in this direction.

Note that most methods studied in the batch stochastic setting already handle unbounded outputs (at least in the Gaussian case, or more generally, when the regression function is uniformly bounded and when the deviation of the output from the regression function has a bounded exponential moment). See, e.g., [START_REF] Nemirovski | Topics in Non-Parametric Statistics[END_REF][START_REF] Catoni | Universal aggregation rules with exact bias bounds[END_REF][START_REF] Tsybakov | Optimal rates of aggregation[END_REF][START_REF] Audibert | Aggregated estimators and empirical complexity for least square regression[END_REF] and [Cat04, Chapters 3 and 4]; see also [START_REF] Audibert | Robust linear least squares regression[END_REF] for PAC-Bayesian methods under even weaker assumptions. It also turns out that some of these methods have an online nature: e.g., the progressive mixture rule is nothing else than the exponentially weighted average forecaster with constant parameter η combined with the online-to-batch conversion. Its theoretical properties with unbounded outputs were derived in [Cat04, Chapter 3] using arguments different from that of the individual sequence framework (see also [BN08, Theorem 1-(a)]). The corresponding risk bounds were proved when the parameter η is tuned as a function of a known uniform bound on the regression function and on the base regressors and of a known bound related to the moment-generating function of the noise. It is not clear whether the latter tuning ensures non-trivial regret bounds for individual sequences. Next we focus on online algorithms (e.g., properly tuned variants of the exponentially weighted average forecaster) that have provable guarantees for individual sequences and for which the standard online-to-batch conversion yields interesting risk bounds in the stochastic setting. This individual sequence approach not only ensures that the resulting method is robust in some sense, but also provides adaptivity results to unknown quantities in the stochastic setting.

Regression model with random design

Next we introduce the regression model with random design and related aggregation problems. In this batch setting the forecaster is given at the beginning of the game T independent random copies (X 1 , Y 1 ), . . . , (X T , Y T ) of (X, Y ) ∈ X × R whose common distribution is unknown. The random variables Y t , 1 t T , are called outputs or (somewhat abusively) observations. We assume thereafter that E[Y 2 ] < ∞; the goal of the forecaster is to estimate the regression function

f : X → R defined by f (x) E[Y |X = x] for all x ∈ X . The quality of a regressor f T : X → R based on the sample (X 1 , Y 1 ), . . . , (X T , Y T ) is measured by its L 2 -risk f -f T 2 L 2 ,
where, denoting the distribution of X by P X , we set, for all measurable functions h : X → R,

h L 2 X h(x) 2 P X (dx) 1/2 = E h(X) 2 1/2 .
In the sequel we focus on the expected

L 2 -risk E f -f T 2 L 2 . Nota: Setting ε t Y t -f (X t
) for all t = 1, . . . , T , the regression model can be rewritten as

Y t = f (X t ) + ε t , 1 t T ,
where the pairs andE[ε 1 |X 1 ] = 0 almost surely (note that ε 1 and X 1 are not necessarily independent). This equivalent description is sometimes chosen to state the regression model with random design.

(X 1 , ε 1 ), . . . , (X T , ε T ) are i.i.d. with E[ε 2 1 ] < ∞, E[f 2 (X 1 )] < ∞,
We consider the following aggregation problem. The forecaster is given a dictionnary ϕ = (ϕ 1 , . . . , ϕ d ) of base regressors ϕ j : X → R, 1 j d (the ϕ j can be, e.g., elements of a suitably chosen functional basis or estimators computed on an independent sample). The goal of the forecaster is to output a regressor f T : X → R based on the sample (X 1 , Y 1 ), . . . , (X T , Y T ) and whose expected L 2 -risk is almost as small as that of the best linear regressor u • ϕ d j=1 u j ϕ j in a given reference class {u • ϕ : u ∈ U}, where U ⊂ R d . Namely, its goal is to satisfy a risk bound of the form

E f -f 2 L 2 inf u∈U f -u • ϕ 2 L 2 + ψ T,d,U (u) (2.41)
for a remainder term ψ T,d,U (u) that should be as small as possible. Note that ψ T,d,U (u) depends on the sample size T , the ambient dimension d, and the comparison set U. To be more rigorous, it may actually also depend on the joint distribution P of (X, Y ) (through, e.g., the noise level

E[(Y -f (X)) 2 ]
) and on the dictionary ϕ (through, e.g., some norm ϕ ). Following [Nem00, Chapter 5] and [START_REF] Tsybakov | Optimal rates of aggregation[END_REF] (see also [START_REF] Lounici | Generalized mirror averaging and D-convex aggregation[END_REF]), the comparison set U ⊂ R d can be taken as, e.g., the set of the vertices of the simplex in R d (which corresponds to the problem of model-selection aggregation), the whole simplex (which corresponds to convex aggregation), or the whole R d space (which corresponds to linear aggregation).

From online to batch: straightforward bounds for bounded outputs

Next we discuss the applicability of algorithms designed for online linear regression (cf. Section 2.4) to the regression model with random design. For the three aggregation problems mentioned above, there are online algorithms 19 ( f t ) t 1 that satisfy regret bounds of the following form: uniformly over all sequences (x 1 , y 1 ), . . . ,

(x T , y T ) ∈ X × [-B y , B y ], T t=1 y t -f t (x t ) 2 inf u∈U T t=1 y t -u • ϕ(x t ) 2 + ∆ T,
ϕ ∞ max 1 j d sup x∈X ϕ j (x) .
In their basic forms, most online algorithms satisfying (2.42) are tuned as a function of the bound B y (e.g., the exponentially weighted average forecaster for model-selection aggregation, 19 By online algorithm, we mean here any sequence ( ft) t 1 of functions such that ft : R d ×(R d ×R) t-1 → R maps at time t the new input xt and the past data (xs, ys) 1 s t-1 to a prediction ft xt; (xs, ys) 1 s t-1 , also denoted by ft xt) or by yt for notational convenience. 20 As mentioned above, we recall that, for the sake of clarity, we only write the more important dependencies in T , d, and By, but the regret term ∆ T,d,By (u) usually also depends on the dictionary ϕ (through, e.g., some norm ϕ ).

the Exponentiated Gradient algorithm for convex aggregation, or the sequential ridge regression forecaster with the ideal tuning (2.27) of the end of Section 2.4.2 for linear aggregation).

We can then use the online to batch conversion of Section 2.

( f t ) t 1 into data-based regressors f T 1 T T t=1 f t that satisfy the risk bound E Y -f T (X) 2 inf u∈U E Y -u • ϕ(X) 2 + ∆ T,d,By (u) T .
Elementary manipulations then yield the desired risk bound (see the proof of Theorem 3.2 in Chapter 3, Appendix 3.A.3 for more details):

E f -f T 2 L 2 inf u∈U f -u • ϕ 2 L 2 + ∆ T,d,By (u) T .
What about unbounded outputs Y t ? -The method of [START_REF] Bunea | Sequential procedures for aggregating arbitrary estimators of a conditional mean[END_REF] under some prior knowledge on the regression function f and the dictionary ϕ.

In the previous paragraphs, we assumed that the output Y lied in some bounded interval

[-B y , B y ].
Assume now that Y is unbounded in the sense that

∀B > 0, P |Y | > B > 0 .
(2.43)

In this case, one method suggested by [START_REF] Bunea | Sequential procedures for aggregating arbitrary estimators of a conditional mean[END_REF] is to truncate the outputs Y t to some threshold γ = b or γ = b ln T (up to constant factors) for some known bound b > 0 on the infinity norms of the regression function f and the base regressors ϕ j , 1 j d, i.e.,

b max f ∞ , ϕ 1 ∞ , . . . , ϕ d ∞ .
The authors then apply an exponentially weighted average forecaster to the truncated outputs

Y t = [Y t ] γ , 1 t T ,
where [x] γ min γ, max{-γ, x} denotes the truncation (or clipping) of x ∈ R to the threshold level γ. They then prove a risk bound for f T in terms of the truncated output Y = [Y ] γ , from which they derive by an approximation argument a risk bound in terms of the non-truncated (and possibly unbounded) output Y .

A drawback of the previous approach is that the bound b is assumed to be known in advance. Besides, the base forecasters u • ϕ, u ∈ U, are not uniformly bounded if ϕ = 0 and U = R d , so that the approach followed in [START_REF] Bunea | Sequential procedures for aggregating arbitrary estimators of a conditional mean[END_REF] for model-selection or convex aggregation does not readily apply to linear aggregation. For these two reasons, we truncate the base forecasters u • ϕ instead of the outputs Y t (see below); truncation is carried out in an automatic way in the sense that no a priori knowledge is required.

The idea of truncating the base forecasts was used many times in the past; see, e.g., [START_REF] Vovk | Competitive on-line statistics[END_REF] for the online linear regression setting, [GKKW02, Chapter 10] for the regression problem with random design, and [GO07, BBGO10] for sequential prediction of unbounded time series under the square loss. A key ingredient in our work (i.e., in Chapters 3 and 4) is to perform truncation with respect to a data-driven threshold.

Online The above remark suggests to design algorithms ( f t ) t 1 that satisfy regret bounds of the form (2.42) without knowing the random bound B y = max 1 t T |Y t | in advance. Indeed, by Remark (a) at the end of Section 2.5.1 and by elementary manipulations carried out in the proof of Theorem 3.2 (Chapter 3), the resulting batch method f T

1 T T t=1 f t satisfies E f -f T 2 L 2 inf u∈U f -u • ϕ 2 L 2 + E ∆ T,d,By (u) T .
As can be seen from the examples of ∆ T,d,By (u) following Equation (2.42), the regret term ∆ T,d,By (u) usually (roughly) scales as B 2 y or B y . Hence the term E ∆ T,d,By (u) scales as

E[B 2 y ] = E max 1 t T Y 2 t or as E[B y ] = E max 1 t T |Y t | .
The last two quantities can be both upper bounded under general assumptions on the distribution of Y , e.g., when Y -E[Y ] satisfies a tail assumption such as boundedness, a subgaussian tail, or a bounded exponential moment (see Corollary 3.5 in Chapter 3). In a word, an online adaptation to the unknown bound B y is key to handle unbounded outputs with individual sequence techniques.

In Chapter 3, Section 3.4, we design such an algorithm ( f t ) t 1 when U = R d , i.e., it satisfies a regret bound of the form (2.42) without knowing the random bound B y . This algorithm is a properly tuned exponentially weighted average forecaster (with continuous weights on R d ) applied to truncated base forecasts u • ϕ(X t )] Bt -note that, contrary to [BN08, Corollary 1 and Theorem 1-(b)], we truncate the base forecasts instead of truncating the outputs Y t . Ideally we would like to choose the threshold B t equal to the unknown random bound max 1 t T |Y t |, since this can only improve prediction. The actual truncation (and the tuning of η t ) is thus performed with respect to a time-varying threshold B t that adapts to max 1 t T |Y t | via parameter-tuning techniques provided by [START_REF] Auer | Adaptive and self-confident on-line learning algorithms[END_REF][START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF].

Adaptation means here that we are able to prove regret bounds within constant factors of (quasi-optimal) bounds that could be proved if max 1 t T |Y t | was known in advance by the forecaster. By a remark above on the upper bounding of E max 1 t T Y 2 t , this adaptivity property in the online deterministic setting leads to sparsity oracle inequalities in the stochastic setting that are adaptive to the unknown variance of the noise at least whenever the latter is Gaussian -see Chapter 3, page 113 .

Sparsity oracle inequalities in the stochastic setting

Sparsity has been extensively studied in the stochastic setting over the past decade. Among the tools introduced for this purpose the notion of sparsity oracle inequality plays a fundamental role. In high-dimensional linear regression, such inequalities indicate that the task consisting in predicting almost as well as an unknown target vector is still statistically feasible if the target vector has only few non-zero coordinates. Such theoretical guarantees and the associated statistical methods have proved useful in many contemporary applications such as computational biology (e.g., analysis of DNA sequences), collaborative filtering (e.g., Netflix, Amazon), satellite and hyperspectral imaging, and high-dimensional econometrics (e.g., cross-country growth regression problems).

In this section we recall the basic ideas underlying the notion of sparsity oracle inequality in the stochastic batch setting. In Chapter 3 we use similar ideas in the framework of individual sequences to introduce a new type of (deterministic) regret bounds under a sparsity scenario.

Framework

We first consider the (generalized) linear regression model with fixed or random design. The forecaster observes independent random pairs (X 1 , Y 1 ), . . . , (X T , Y T ) ∈ X × R given by

Y t = u * • ϕ(X t ) + ε t , 1 t T , (2.44)
where the X t ∈ X are either i.i.d random variables (random design) or fixed elements (fixed design), denoted in both cases by capital letters in this section, where ϕ = (ϕ 1 , . . . , ϕ d ) is a dictionary of base regressors ϕ j : X → R, 1 j d, where u * ∈ R d is the unknown linear combination (recall that u * • ϕ d j=1 u * j ϕ j ), and where the ε t are i.i.d. square-integrable real random variables with zero mean (conditionally on the X t if the design is random).

Three main statistical problems arise in this linear regression framework:

• prediction: estimating u * • ϕ(X t ) 1 t T (fixed design) or u * • ϕ (random design);
• estimation: estimating u * ;

• support estimation: estimating the set of the non-zero coordinates of u * . These three tasks have been extensively studied over the past decade in the high-dimensional setting under a sparsity scenario, namely, when

u * 0 T d , (2.45)
where u * 0 {j : u * j = 0} denotes the number of non-zero coordinates of u * . In this thesis, we focus on the prediction problem. As we will see later, this problem can be addressed in rather general regression models both in the stochastic batch setting (see (2.46) below) and in the deterministic online setting (see Chapter 3).

In the stochastic setting, most risk bounds for the prediction problem under a sparsity scenario take the form of sparsity oracle inequalities. We explain below the basic idea that underlies such risk bounds, as well as their main consequences.

An ideal ordinary least-squares estimator

In the (generalized) linear regression model with fixed or random design (2.44), the ordinary least squares estimator

u T ∈ argmin u∈R d 1 T T t=1 Y t -u • ϕ(X t )
2 has an expected risk E R u T at most of the order of d/T (see [START_REF] Györfi | A distribution-free theory of nonparametric regression[END_REF] and the references therein for the fixed design, and the more recent advances in [START_REF] Audibert | Robust linear least squares regression[END_REF] for the random design under weak assumptions on the output distribution). Here, we defined the risk R(u) of any linear combination u ∈ R d by

R(u)        u * • ϕ -u • ϕ 2 L 2 (random design), 1 T T t=1 u * • ϕ(X t ) -u • ϕ(X t ) 2 (fixed design),
where, in the random design case, we set

h L 2 E[h(X 1 ) 2 ] 1/2 for all measurable functions h : X → R such that E[h(X 1 ) 2 ] < ∞.
When the ambient dimension d is much larger than the sample size T , a direct minimization of the least-squares criterion on R d can lead to overfitting, which is reflected in the non-vanishing upper bound d/T . However, as suggested by the following remark, it is still possible to achieve a small risk under the additional assumption u * 0 = s T . Indeed, if the support J(u * ) j ∈ {1, . . . , d} : u * j = 0 of u * was known in advance, the oracle applying the ordinary least squares estimator to the linear subspace u ∈ R d : ∀j / ∈ J(u * ), u j = 0 would have a risk at most of the order of s/T 1. This suggests that the prediction task in high dimension is still feasible under a sparsity scenario. However this ordinary least-squares is ideal since the support J(u * ) is unknown in practice; it is closely related to the notion of oracle in the terminology of model selection (see Chapter 6 for further details).

Adaptivity to the unknown sparsity by model selection

The rate s/T of the above ideal least-squares estimator can actually be achieved up to a ln d factor without the prior knowledge of the set J(u * ) nor even of its cardinality s = u * 0 . This was first done by adding a 0 -complexity penalty to the least-squares criterion:

u pen T ∈ argmin u∈R d 1 T T t=1 Y t -u • ϕ(X t ) 2 + pen(u) ,
where the penalty pen(u) is proportional to the number u 0 of non-zero coefficients of u (see [START_REF] Akaike | Information theory and an extension of the maximum likelihood principle[END_REF] for the AIC criterion, [START_REF] Mallows | Some Comments on Cp[END_REF] for Mallows' C p , and [START_REF] Schwarz | Estimating the dimension of a model[END_REF][START_REF] Foster | The risk inflation criterion for multiple regression[END_REF] for the BIC criterion).

In the fixed design case, [START_REF] Birgé | Gaussian model selection[END_REF] proved via model-selection arguments that for a penalty pen(u) of the order of u 0 1 + ln(d/ u 0 ) , the penalized least-squares estimator u pen T "mimics" the 0 -oracle in the sense that its risk is at most of the order of s ln(d/s) T .

For a detailed proof of this fact with a Bayesian variant of the estimator u pen T , see Chapter 6, Section 6.4.1. The above rate follows from (6.41) therein.

Numerous works addressed this model-selection-type problem: see, e.g., [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF][START_REF] Abramovich | Adapting to unknown sparsity by controlling the false discovery rate[END_REF][START_REF] Bunea | Aggregation for Gaussian regression[END_REF] for the fixed design setting and [START_REF] Bunea | Aggregation for regression learning[END_REF] for the random design setting. Further references can be found, e.g., in [START_REF] Abramovich | Adapting to unknown sparsity by controlling the false discovery rate[END_REF] and in [AGS11, Chapter 4].

The above rate holds without any assumption on the dictionary ϕ = (ϕ 1 , . . . , ϕ d ) and without any prior knowledge on the support J(u * ). Note that the rate contains an additional multiplicative factor ln(d/s) compared to the upper bound s/T satisfied by the ideal least-squares estimator of the previous section. This logarithmic factor is the price to pay for not knowing J(u * ) in advance. Indeed, [START_REF] Raskutti | Minimax rates of estimation for high-dimensional linear regression over q -balls[END_REF][START_REF] Verzelen | Minimax risks for sparse regressions: Ultra-high-dimensional phenomenons[END_REF] proved that the rate s ln(d/s)/T is minimax optimal on 0 -balls for fixed or Gaussian random designs (see also [START_REF] Birgé | A generalized Cp criterion for Gaussian model selection[END_REF] in the infinite-dimensional Gaussian sequence model). In this respect, methods with a risk at most of order s ln(d/s)/T are termed adaptive to the unknown sparsity s of u * .

The risk bounds proved in [BM01a, [START_REF] Bunea | Aggregation for regression learning[END_REF][START_REF] Bunea | Aggregation for Gaussian regression[END_REF] are actually stronger, since they are stated in a more general regression model already encountered in the previous sections:

Y t = f (X t ) + ε t , 1 t T , (2.46)
where the X t are either i.i.d. random variables (random design) or deterministic elements (fixed design). Since f is not necessarily assumed to be of the form f = u * •ϕ, the risk bounds mentioned earlier

E R u T = O s ln(d/s) T are replaced with bounds on the differences E R u T -R(u) for all u ∈ R d , where the risk R(u) is now defined by R(u)        f -u • ϕ 2 L 2 (random design) 1 T T t=1 f (X t ) -u • ϕ(X t ) 2 (fixed design).
The risk bounds proved in [START_REF] Bunea | Aggregation for regression learning[END_REF][START_REF] Bunea | Aggregation for Gaussian regression[END_REF] are indeed of the form

E R u T (1 + a) inf u∈R d R(u) + C(a) u 0 T ln e d max u 0 , 1 , (2.47)
where C(a) ∼ a -1 → +∞ as a → 0. The above upper bound is a typical example of what is called a sparsity oracle inequality, i.e., in the prediction problem, a risk bound involving a trade-off between the risk R(u) and the number of non-zero coordinates u 0 of all u ∈ R d .

Other methods: 1 -regularization and exponential weighting

We end this section with a brief computational efficiency-oriented overview of alternatives to 0regularization. Indeed, a major drawback of 0 -regularization is that the corresponding non-convex minimization problems are not computationally tractable. This complexity issue has been handled by replacing the 0 -penalty with a 1 -penalty (proportional to the sum of the absolute values of the coefficients). 1 -regularization can indeed be seen as a 'convex relaxation' of 0 -regularization, i.e., from a geometrical viewpoint, it behaves similarly to the 0 -penalty but leads to convex and thus computationally tractable minimization problems. It was first proposed by [START_REF] Tibshirani | Regression shrinkage and selection via the Lasso[END_REF] for the so-called Lasso estimator and by [START_REF] Donoho | Ideal spatial adaptation by wavelet shrinkage[END_REF] for a soft thresholding-based estimator in the context of wavelet regression. In its dual form, the simplest version of the Lasso is defined by

u Lasso T ∈ argmin u∈R d 1 T T t=1 Y t -u • ϕ(X t ) 2 + λ u 1 ,
for some tuning parameter λ usually taken of the order of σ ln(d)/T if ε t ∼ N (0, σ 2 ). The 0 -oracle properties of the Lasso (and variants of the Lasso), i.e., risk bounds typically of the form s ln(d)/T , have then been extensively studied over the past decade. A list of few referencesbut far from being comprehensive -includes [BTW07b, CT07,

. Until very recently all sparsity oracle inequalities proved for the Lasso had a leading constant strictly larger than 1. This apparent drawback (as compared to exponential weighting algorithms mentioned below) was overcome in [START_REF] Koltchinskii | Nuclear norm penalization and optimal rates for noisy low rank matrix completion[END_REF], who derived a sharp sparsity oracle inequality for the Lasso, i.e., a sparsity oracle inequality with leading constant equal to 1.

We also mention that [START_REF] Massart | The Lasso as an 1 -ball model selection procedure[END_REF] recently addressed the 1 -oracle properties of the Lasso estimator from a different viewpoint. They analyze the Lasso not as a variable selector but as a model selector among a countably infinite collection of 1 -balls. Their oracle-type inequalities follow from the general model selection theorem for nonlinear models of [Mas07, Theorem 4.18]. For further details, see Chapter 6 where we analyze a Bayesian extension of this model-selection procedure based on exponential weighting.

Despite their computational efficiency, the aforementioned 1 -regularized methods still suffer from a drawback: their 0 -oracle properties hold under rather restrictive assumptions on the (fixed or random) design; namely, that the covariates should be nearly orthogonal. We refer the reader to [START_REF] Van De Geer | On the conditions used to prove oracle results for the Lasso[END_REF] for a detailed discussion on these assumptions.

Recently an attempt has thus been made to reach a compromise between strong theoretical guarantees (that hold under very weak assumptions on the design) and computational efficiency. In this respect [DT07, DT08, DT11] proposed an aggregation algorithm which is based on exponential weighting, which satisfies sharp sparsity oracle inequalities on a fixed or random design under almost no assumption on the dictionary, and which can be approximated numerically at a reasonable computational cost for large values of the ambient dimension d (cf. [START_REF] Dalalyan | Sparse regression learning by aggregation and Langevin Monte-Carlo[END_REF] who use Langevin Monte-Carlo methods). This is the algorithm from which our online forecaster SeqSEW of Chapter 3 in the deterministic setting is inspired.

More recently [START_REF] Rigollet | Exponential Screening and optimal rates of sparse estimation[END_REF][START_REF] Alquier | PAC-Bayesian bounds for sparse regression estimation with exponential weights[END_REF] designed aggregation algorithms that achieve optimal rates of sparse aggregation in the regression model with fixed design (in the sense of [START_REF] Rigollet | Exponential Screening and optimal rates of sparse estimation[END_REF]). [START_REF] Alquier | PAC-Bayesian bounds for sparse regression estimation with exponential weights[END_REF] also addressed the regression model with random design but the corresponding risk bounds depend as in [START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF] on the logarithms of u * 1 and T (but their bound holds with large probability). In both papers [START_REF] Rigollet | Exponential Screening and optimal rates of sparse estimation[END_REF][START_REF] Alquier | PAC-Bayesian bounds for sparse regression estimation with exponential weights[END_REF] the corresponding algorithms were shown to be well approximated by MCMC methods with conclusive experimental results.

Some interesting consequences of sparsity oracle inequalities

As detailed in [BTW06, BTW07a, DT08], sparsity oracle inequalities have interesting consequences. They indeed imply that:

• In the high-dimensional linear regression model (2.44), prediction is still statistically feasible under a sparsity scenario (this is the main motivation we chose to introduce the notion of sparsity oracle inequality).

• Statistical procedures satisfying such sparsity oracle inequalities can be used to perform adaptive nonparametric regression (i.e., for an appropriately well chosen basis, these procedures are adaptive to the unknown smoothness of the regression function f ). See also [START_REF] Birgé | Gaussian model selection[END_REF][START_REF] Massart | Concentration Inequalities and Model Selection[END_REF].

• Statistical procedures satisfying sharp sparsity oracle inequalities achieve (quasi-)optimal rates of model-selection, convex, and linear aggregation in the sense of [START_REF] Nemirovski | Topics in Non-Parametric Statistics[END_REF][START_REF] Tsybakov | Optimal rates of aggregation[END_REF]. Namely, up to some small remainder terms, these procedures predict at least as well as the best among the base predictors ϕ j (model-selection aggregation), the best convex combination of the ϕ j (convex aggregation), and the best linear combination of the ϕ j (linear aggregation); the corresponding remainder terms are the smallest possible ones. Further details can be found, e.g., in [RT11, Section 6] (note that there are also other types of aggregation than the three ones mentioned above, such as D-convex aggregation [Lou07]).

2.A Proofs

Proof (of Lemma 2.2): Note that the lower bound is trivial21 if K = 1. Therefore, we assume in the sequel that K 2. In the sequel Ber(q) denotes the Bernoulli distribution with parameter q ∈ [0, 1].

The proof technique is due to [START_REF] Cesa-Bianchi | Minimizing regret with label efficient prediction[END_REF][START_REF] Stoltz | Prédiction de suites individuelles[END_REF] and relies on arguments of [START_REF] Auer | The nonstochastic multi-armed bandit problem[END_REF]. Consider the space Ω = ({0, 1} K ) T endowed with its discrete σ-algebra. For all 1 t T , define the random variable Y t : ({0, 1} K ) T → {0, 1} K as the t-th coordinate mapping on ({0, 1} K ) T . We equip Ω with a family of probability distributions Q ⊗T j 1 j K , where we set

Q j = K i=1
Ber 1/2 -ε I {i=j} for some ε ∈ (0, 1/2) to be determined by the analysis. The proof is dedicated to show that inf S max

1 j K E Q ⊗T j T t=1 a t • Y t -min 1 i K T t=1 Y i,t c 2 T 2 ln K .
(2.48)

This will then conclude the proof since

( a t , Y t ) = a t • Y t and (δ i , Y t ) = δ i • Y t = Y i,t
for all i ∈ {1, . . . , K} and t ∈ {1, . . . , T } almost surely.

Note that, by construction, the random vectors Y t = (Y 1,t , . . . , Y K,t ), 1 t T , are such that for all j = 1, . . . , K, under Q ⊗T j ,

• the real random variables Y i,t , 1 i K, 1 t T , are independent;

• for all i ∈ {1, . . . , K}, the random sequence Y i,t 1 t T associated with the i-th action is an i.i.d. Bernoulli sequence with parameter 1/2

(if i = j) or 1/2 -ε (if i = j).
By Fano's lemma, we show next that if ε is small enough, then the forecaster cannot identify the best action j too quickly uniformly over all distributions Q ⊗T j , and therefore incurs a regret at least of the order of

√ T ln K for at least one distribution Q ⊗T j .
Let S = a t t 1 be any strategy of the forecaster. We split below the expected regret into two parts. On the one hand, denoting by Y 1:t-1 Y 1 , . . . , Y t-1 ) the whole22 information available to the forecaster before making its prediction at time t, and noting that a t • Y t = K i=1 a i,t Y i,t , we get by the tower rule that, for all j ∈ {1, . . . , K},

E Q ⊗T j T t=1 a t • Y t = T t=1 K i=1 E Q ⊗T j E Q ⊗T j a i,t Y i,t Y 1:t-1 = T t=1 K i=1 E Q ⊗T j a i,t 1 2 -εI {i=j} (2.49) = T 2 -ε T t=1 E Q ⊗T j a j,t , (2.50)
where (2.49) follows from the fact that a i,t is Y 1:t-1 -measurable (recall that the experts' advice a i,t = δ i are deterministic) and from the fact that, under

Q ⊗T j , Y i,t ∼ Ber 1/2 -ε I {i=j} is independent of Y 1:t-1 .
As for (2.50), it follows from the almost sure equality K i=1 a i,t = 1 (since a t ∈ X K ).

Next we introduce an external randomization (as in [START_REF] Cesa-Bianchi | Minimizing regret with label efficient prediction[END_REF][START_REF] Stoltz | Prédiction de suites individuelles[END_REF]). Let (Ω ext , B ext , Q ext ) be a probability space, and let I 1 , . . . , I T ∈ {1, . . . , K} be random variables defined on the augmented space ({0, 1} K ) T × Ω ext such that23 I t is measurable with respect to the σ-field σ(Y 1 , . . . , Y t-1 ) ⊗ B ext , and, for all j ∈ {1, . . . , K},

∀t ∈ {1, . . . , T }, ∀i ∈ {1, . . . , K}, Q ⊗T j ⊗ Q ext I t = i (Y 1 , I 1 ), . . . , (Y t-1 , I t-1 ) = a i,t .
(Recall that K i=1 a i,t = 1 almost surely.) By the property above, (2.50) can be rewritten for all j ∈ {1, . . . , K} as

E Q ⊗T j T t=1 a t • Y t = T 2 -ε T t=1 Q ⊗T j ⊗ Q ext I t = j .
(2.51)

On the other hand, by Jensen's inequality and by definition of Q j , we get, for all j ∈ {1, . . . , K},

E Q ⊗T j min 1 i K T t=1 Y i,t min 1 i K E Q ⊗T j T t=1 Y i,t = E Q ⊗T j T t=1 Y j,t = T 2 -T ε .
Combining the last inequality with (2.51), we can lower bound the expected regret under each probability distribution Q ⊗T j and get that max

1 j K E Q ⊗T j T t=1 a t • Y t -min 1 i K T t=1 Y i,t T ε 1 -min 1 j K 1 T T t=1 Q ⊗T j ⊗ Q ext I t = j .
(2.52)

To conclude the proof of (2.48), it suffices to lower bound the minimum in the parentheses by a positive absolute constant for ε of the order of (ln K)/T . But, by the extension of Fano's lemma to convex combinations due to [START_REF] Cesa-Bianchi | Minimizing regret with label efficient prediction[END_REF] (see Lemma A.10 in Appendix A.7) and by the fact that K 2, we get

min 1 j K 1 T T t=1 Q ⊗T j ⊗ Q ext I t = j max 2e 2e + 1 , K ln K , (2.53)
where

K 1 K -1 K j=2 T t=1 1 T K Q ⊗T j ⊗ Q ext , Q ⊗T 1 ⊗ Q ext = T K -1 K j=2 K Q j , Q 1 .
(2.54)

In the last equality, we used the fact that

K Q ⊗T j ⊗Q ext , Q ⊗T 1 ⊗Q ext = T K Q j , Q 1 by
the chain rule for the Kullback-Leibler divergence. But, noting that for all j = 2, . . . , K, the probability distributions

Q j = ⊗ K i=1 Ber 1/2-ε I {i=j} and Q 1 = ⊗ K i=1
Ber 1/2-ε I {i=1} only differ on the two actions 1 and j by ε, we have, using again the chain rule for the Kullback-Leibler divergence,

K Q j , Q 1 = K Ber(1/2), Ber(1/2 -ε) + K Ber(1/2 -ε), Ber(1/2) 5ε 5 , (2.55)
where the last inequality is proved in [Sto05, Lemma A.5] for all 0 ε 1/10. Putting (2.53), (2.54), and (2.55) together, we get that, for all 0 < ε 1/10, min

1 j K 1 T T t=1 Q ⊗T j ⊗ Q ext I t = j max 2e 2e + 1 , 5T ε 2 ln K = 2e 2e + 1 , (2.56)
where the last equality follows from the choice of Remark 2.5. In the proof above, we showed via a version of Fano's lemma that there exists a probability distribution under which the random vectors Y t , 1 t T , are i.i.d. and such that the expected regret is at least of the order of √ T ln K (see (2.48)). This probability distribution is of the form Q ⊗K j * , where j * ∈ {1, . . . , K} minimizes the left-hand side of (2.53). Therefore, it depends on the strategy of the forecaster ( a t ) t 1 through (I t ) t 1 .

ε = 2e 2e + 1 ln K 5T ,
Note that we could have used another variant of Fano's lemma 24 for K 3 or Pinsker's inequality (see Appendix A.7) for K = 2 to prove that the expected regret under (1/K) K i=1 Q ⊗T j is also at least of the order of √ T ln K for any strategy of the forecaster. Interestingly, the probability distribution

(1/K) K i=1 Q ⊗T j
is now independent of the forecaster. Besides, the aforementioned √ T ln K lower bound on the expected regret under (1/K) K i=1 Q ⊗T j yields a lower bound similar to (2.48), at the price of worst constants though.

Chapter 3

Sparsity regret bounds for individual sequences in online linear regression

We consider the problem of online linear regression on arbitrary deterministic sequences when the ambient dimension d can be much larger than the number of time rounds T . We introduce the notion of sparsity regret bound, which is a deterministic online counterpart of the so-called sparsity oracle inequalities from the stochastic setting. We prove such regret bounds for an onlinelearning algorithm called SeqSEW and based on exponential weighting and data-driven truncation. In a second part we apply a parameter-free version of this algorithm to the regression model with random design (i.i.d. data) and derive risk bounds of the same flavor as in [START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF] but which solve two questions left open therein. In particular our risk bounds are adaptive (up to a logarithmic factor) to the unknown variance of the noise if the latter is Gaussian. We also address the regression model with fixed design as in [START_REF] Dalalyan | Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity[END_REF].

NOTA: This chapter is the full version (with extensive proofs) of a conference paper [START_REF] Gerchinovitz | Sparsity regret bounds for individual sequences in online linear regression[END_REF] that appeared in the proceedings of COLT 2011. Corollary 3.5 and Section 3.4.2 are published here for the first time.

Introduction

Sparsity has been extensively studied in the stochastic setting over the past decade. Among the theoretical tools introduced for this purpose, the notion of sparsity oracle inequality plays a fundamental role. In high-dimensional linear regression, such inequalities indicate that the task consisting in predicting almost as well as an unknown target vector is still statistically feasible if the target vector has only few non-zero coordinates. A detailed motivation of such risk bounds and some bibliographic references are provided in Section 2.6 of Chapter 2.

In this chapter, we bring the notion of sparsity oracle inequality into the framework of prediction of individual sequences (of deterministic nature). The corresponding deterministic inequalities are called sparsity regret bounds. We prove such bounds for an online-learning algorithm called SeqSEW which is inspired from the Sparse Exponential Weighting algorithm introduced in the stochastic setting by [START_REF] Dalalyan | Aggregation by exponential weighting and sharp oracle inequalities[END_REF]. Thanks to individual sequences techniques (e.g., online truncation and online tuning), the most sophisticated version of our algorithm is fully automatic in the sense that no a priori knowledge is needed for the choice of the tuning parameters.

The second contribution of this chapter deals with fruitful connections between the framework of individual sequences and the stochastic setting. More precisely, we show that, via the standard online to batch trick, the online truncation and parameter tuning performed by the algorithm Se-qSEW for deterministic purposes yield, in the regression model with random or fixed design, sparsity oracle inequalities with leading constant 1 which are of the same flavor as in [START_REF] Dalalyan | Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity[END_REF][START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF]. In addition our bounds are adaptive to the unknown variance σ 2 of the noise (up to a logarithmic factor) at least whenever the latter is Gaussian; weaker bounds are also proved under weaker assumptions. Therefore, in the batch stochastic setting, individual sequence techniques appear to be useful for adaptation purposes.

In the next paragraphs, we introduce our main setting and motivate the notion of sparsity regret bound from an online learning viewpoint (this motivation can be paralleled to that of Section 2.6.1 in Chapter 2). We then detail our main contributions with respect to the statistical literature and the machine learning literature.

Introduction of a deterministic counterpart of sparsity oracle inequalities

We consider the problem of online linear regression on arbitrary deterministic sequences. A forecaster has to predict in a sequential fashion the values y t ∈ R of an unknown sequence of observations given some input data x t ∈ X and some base forecasters ϕ j : X → R, 1 j d, on the basis of which he outputs a prediction y t ∈ R. The quality of the predictions is assessed by the square loss. The goal of the forecaster is to predict almost as well as the best linear forecaster u • ϕ d j=1 u j ϕ j , where u ∈ R d , i.e., to satisfy, uniformly over all individual sequences (x t , y t ) 1 t T , a regret bound of the form

T t=1 y t -y t 2 inf u∈R d T t=1 y t -u • ϕ(x t ) 2 + ∆ T,d (u)
for some regret term ∆ T,d (u) that should be as small as possible and, in particular, sublinear in T . (For the sake of introduction, we omit the dependencies of ∆ T,d (u) on the amplitudes max 1 t T x t ∞ and max 1 t T |y t |.)

In this setting the version of the sequential ridge regression forecaster1 studied by [START_REF] Azoury | Relative loss bounds for on-line density estimation with the exponential family of distributions[END_REF] and [START_REF] Vovk | Competitive on-line statistics[END_REF] and tuned with the illegal optimal tuning of Section 2.4.2 has a regret ∆ T,d (u) of order at most d ln T u 2 2 ; see (2.27) in the aforementioned section. When the ambient dimension d is much larger than the number of time rounds T , the bound d ln T may unfortunately be larger than T and is thus somehow trivial. Since the regret bound d ln T is optimal in a certain sense (see [Vov01, Theorem 2]), additional assumptions are needed to get interesting theoretical guarantees.

A natural assumption, which has already been extensively studied in the stochastic setting, is that there is a sparse linear combination u * (i.e., with s T /(ln T ) non-zero coefficients) which has a small cumulative square loss. If the forecaster knew in advance the support J(u * ) {j : u * j = 0} of u * , he could apply the same forecaster as above but only to the s-dimensional linear subspace u ∈ R d : ∀j / ∈ J(u * ), u j = 0 . The regret bound of this "oracle" would be roughly of order s ln T and thus sublinear in T . Under this sparsity scenario, a sublinear regret thus seems possible, though, of course, the aforementioned regret bound s ln T can only be used as an ideal benchmark (since the support of u * is unknown).

In this chapter, we prove that a regret bound proportional to s is achievable (up to logarithmic factors). In Corollary 3.1 and its refinements (Corollary 3.2 and Theorem 3.1), we indeed derive regret bounds of the form

T t=1 (y t -y t ) 2 inf u∈R d T t=1 y t -u • ϕ(x t ) 2 + u 0 + 1 g T,d u 1 , ϕ ∞ , (3.1)
where u 0 denotes the number of non-zero coordinates of u and where g is increasing but grows at most logarithmically in T , d, u 1 d j=1 |u j |, and ϕ ∞ sup x∈X max 1 j d |ϕ j (x)|. We call regret bounds of the above form sparsity regret bounds.

This work is in connection with several papers that belong either to the statistical or to the machine learning literature. Next we discuss these papers and some related references.

Related works in the stochastic setting

The above regret bound (3.1) can be seen as a deterministic online counterpart of the so-called sparsity oracle inequalities introduced in the stochastic setting in the past decade. The latter are risk bounds expressed in terms of the number of non-zero coefficients of the oracle vector. Such inequalities were derived by [START_REF] Birgé | Gaussian model selection[END_REF] through model selection arguments and later developed by, e.g., [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF][START_REF] Bunea | Aggregation for Gaussian regression[END_REF] in the regression model with fixed design and by [START_REF] Bunea | Aggregation for regression learning[END_REF] for the case of a random design. An introduction to the notion of sparsity oracle inequality can be found in Section 2.6 (Chapter 2); we refer the reader to this section for further references.

We only mention that, recently, sparsity oracle inequalities with leading constant equal to 1

were proved for procedures based on exponential weighting; see [DT07, DT08, RT11, AL11] for the regression model with fixed design and [START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF][START_REF] Alquier | PAC-Bayesian bounds for sparse regression estimation with exponential weights[END_REF] for the regression model with random design. These papers show that a trade-off can be reached between strong theoretical guarantees (as with 0 -regularization) and computational efficiency (as with 1 -regularization). They indeed propose aggregation algorithms which satisfy sparsity oracle inequalities under almost no assumption on the base forecasters (ϕ j ) j , and which can be approximated numerically at a reasonable computational cost for large values of the ambient dimension d.

Our online-learning algorithm SeqSEW is inspired from [START_REF] Dalalyan | Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity[END_REF][START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF]. Following the same lines as in [START_REF] Dalalyan | Sparse regression learning by aggregation and Langevin Monte-Carlo[END_REF], it is possible to slightly adapt its statement to make it computationally tractable by means of Langevin Monte-Carlo approximation while not affecting its statistical properties. The technical details are however omitted in this chapter, which only focuses on the theoretical guarantees of the algorithm SeqSEW.

Previous works on sparsity in the framework of individual sequences

To the best of our knowledge, Corollary 3.1 and its refinements (Corollary 3.2 and Theorem 3.1) provide the first examples of sparsity regret bounds in the sense of (3.1). To comment on the optimality of such regret bounds and compare them to related results in the framework of individual sequences, note that (3.1) can be rewritten in the equivalent form: For all s ∈ N and all U > 0,

T t=1 (y t -y t ) 2 -inf u 0 s u 1 U T t=1 y t -u • ϕ(x t ) 2 s + 1 g T,d U, ϕ ∞ ,
where g grows at most logarithmically in T , d, U , and ϕ ∞ . When s T , this upper bound matches (up to logarithmic factors) the lower bound of order s ln T that follows in a straightforward manner from [Vov01, Theorem 2] or [START_REF] Cesa-Bianchi | Prediction, Learning, and Games[END_REF]Chapter 11]. Indeed, if s T , X = R d , and ϕ j (x) = x j , then for any forecaster, there is an individual sequence (x t , y t ) 1 t T such that the regret of this forecaster on u ∈ R d : u 0 s and u 1 d is bounded from below by a quantity of order s ln T . Therefore, up to logarithmic factors, any algorithm satisfying a sparsity regret bound of the form (3.1) is minimax optimal on intersections of 0 -balls (of radii s T ) and 1 -balls. This is in particular the case for our algorithm SeqSEW, but this contrasts with related works discussed below.

Recent works in the field of online convex optimization addressed the sparsity issue in the online deterministic setting, but from a quite different angle. They focus on algorithms which output sparse linear combinations, while we are interested in algorithms whose regret is small under a sparsity scenario, i.e., on 0 -balls of small radii. See, e.g., [LLZ09, SST09, Xia10, DSSST10] and the references therein. All these articles focus on convex regularization. In the particular case of 1 -regularization under the square loss, the aforementioned works propose algorithms which predict as a sparse linear combination y t = u t • ϕ(x t ) of the base forecasts (i.e., u t 0 is small), while no such guarantee can be proved for our algorithm SeqSEW. However they prove bounds on the 1 -regularized regret of the form

T t=1 (y t -u t • x t ) 2 + λ u t 1 inf u∈R d T t=1 (y t -u • x t ) 2 + λ u 1 + ∆ T,d (u) , (3.2)
for some regret term ∆ T,d (u) which is suboptimal on intersections of 0 -and 1 -balls as explained below. The truncated gradient algorithm of [LLZ09, Corollary 4.1] satisfies 2 such a regret bound with ∆ T,d (u) at least of order ϕ ∞ √ dT when the base forecasts ϕ j (x t ) are dense in the sense that max

1 t T d j=1 ϕ 2 j (x t ) ≈ d ϕ 2 ∞ .
This regret bound grows as a power of and not logarithmically in d as is expected for sparsity regret bounds (recall that we are interested in the case when d T ). The three other papers mentioned above do prove (some) regret bounds with a logarithmic dependence in d, but these bounds do not have the dependence in u 1 and T we are looking for. For p -1 ≈ 1/(ln d), the p-norm RDA method of [START_REF] Xiao | Dual averaging methods for regularized stochastic learning and online optimization[END_REF] and the algorithm SMIDAS of [START_REF] Shalev-Shwartz | Stochastic methods for 1 -regularized loss minimization[END_REF] -the latter being a particular case of the algorithm COMID of [START_REF] Duchi | Composite objective mirror descent[END_REF] specialized to the pnorm divergence -satisfy regret bounds of the above form (3.2) with ∆ T,d (u) ≈ µ u 1

√

T ln d, for some gradient-based constant µ. Therefore, in all three cases, the function ∆ grows at least linearly in u 1 and as

√

T . This is in contrast with the logarithmic dependence in u 1 and the fast rate O(ln T ) we are looking for and prove, e.g., in Corollary 3.1.

Note that the suboptimality of the aforementioned algorithms is specific to the goal we are pursuing, i.e., prediction on 0 -balls (intersected with 1 -balls). On the contrary the rate u 1 √ T ln d is more suited and actually nearly optimal for learning on 1 -balls (see Chapter 4). Moreover, the predictions output by our algorithm SeqSEW are not necessarily sparse linear combinations of the base forecasts. A question left open is thus whether it is possible to design an algorithm which both ouputs sparse linear combinations (which is statistically useful and sometimes essential for computational issues) and satisfies a sparsity regret bound of the form (3.1).

PAC-Bayesian analysis in the framework of individual sequences

To derive our sparsity regret bounds, we follow a PAC-Bayesian approach combined with the choice of a sparsity-favoring prior. We do not have the space to review the PAC-Bayesian literature in the stochastic setting and only refer the reader to [START_REF] Catoni | Statistical learning theory and stochastic optimization[END_REF] for a thorough introduction to the subject. As for the online deterministic setting, PAC-Bayesian-type inequalities were proved in the framework of prediction with expert advice, e.g., in [START_REF] Freund | Using and combining predictors that specialize[END_REF] and [START_REF] Kivinen | Averaging expert predictions[END_REF], or in the same setting as ours with a Gaussian prior in [START_REF] Vovk | Competitive on-line statistics[END_REF]. More recently, [START_REF] Audibert | Fast learning rates in statistical inference through aggregation[END_REF] proved a PAC-Bayesian result on individual sequences for general losses and prediction sets. The latter result relies on a unifying assumption called the online variance inequality, which holds true, e.g., when the loss function is exp-concave. In the present chapter, we only focus on the particular case of the square loss. We first use Theorem 4.6 of [START_REF] Audibert | Fast learning rates in statistical inference through aggregation[END_REF] to derive a non-adaptive sparsity regret bound. We then 2 The bound stated in [LLZ09, Corollary 4.1] differs from (3.2) in that the constant before the infimum is equal to

C = 1/(1 -2c 2 d η), where c 2 d ≈ max 1 t T d j=1 ϕ 2 j (xt) d ϕ 2
∞ , and where a reasonable choice for η can easily be seen to be η ≈ 1/ 2c provide an adaptive online PAC-Bayesian inequality to automatically adapt to the unknown range of the observations max 1 t T |y t |.

Open questions by Dalalyan and Tsybakov

In Section 3.4.1 we apply a parameter-free version of our algorithm SeqSEW on i.i.d. data and derive a risk bound of the same flavor as in [START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF]. However, our risk bound holds on the whole

R

Outline of the chapter

This chapter is organized as follows. In Section 3.2 we describe our main (deterministic) setting as well as our main notations. In Section 3.3 we prove the aforementioned sparsity regret bounds for our algorithm SeqSEW, first when the forecaster has access to some a priori knowledge on the observations (Sections 3.3.1 and 3.3.2), and then when no a priori information is available (Section 3.3.3), which yields a fully automatic algorithm. In Section 3.4 we apply the algorithm SeqSEW to the regression model with random design (Section 3.4.1) and to the regression model with fixed design (Section 3.4.2). Some technical tools are finally given in appendix.

Setting and notations

The main setting considered in this chapter is an instance of the game of prediction with expert advice called prediction with side information (under the square loss) or, more simply, online linear regression. This online protocol is described in Figure 3.1. An introduction to this setting is provided in Section 2.4 of Chapter 2.

Note that our online protocol is described as if the environment were oblivious to the forecaster's predictions. Actually, since we only consider deterministic forecasters, all regret bounds of this chapter also hold when (x t ) t 1 and (y t ) t 1 are chosen by an adversarial environment. See Section 2.3.1 of Chapter 2 for further details.

Two stochastic batch settings are also considered later in this chapter. See Section 3.4.1 for the regression model with random design, and Section 3.4.2 for the regression model with fixed design.

Some notations

We now define some notations. Vectors in R d will be denoted by bold letters. For all u, v ∈ R d , the standard inner product in R d between u = (u 1 , . . . , u d ) and v = (v 1 , . . . , v d ) will be denoted Parameters: input data set X , base forecasters ϕ = (ϕ 1 , . . . , ϕ d ) with ϕ j : X → R, 1 j d.

Initial step: the environment chooses a sequence of observations (y t ) t 1 in R and a sequence of input data (x t ) t 1 in X but the forecaster has not access to them.

At each time round t ∈ N * , 1. The environment reveals the input data x t ∈ X .

2. The forecaster chooses a prediction y t ∈ R (possibly as a linear combination of the ϕ j (x t ), but this is not necessary).

3. The environment reveals the observation y t ∈ R.

Each linear forecaster

u • ϕ d j=1 u j ϕ j , u ∈ R d , incurs the loss y t -u • ϕ(x t )
2 and the forecaster incurs the loss (y t -y t ) 2 . .

The set of all probability distributions on a set Θ (endowed with some σ-algebra, e.g., the Borel σalgebra when Θ = R d ) will be denoted by M + 1 (Θ). For all ρ, π ∈ M + 1 (Θ), the Kullback-Leibler divergence between ρ and π is defined by

K(ρ, π)    R d ln dρ dπ dρ if ρ is absolutely continuous with respect to π; +∞ otherwise,
where dρ dπ denotes the Radon-Nikodym derivative of ρ with respect to π.

For all x ∈ R and B > 0, we denote by x the smallest integer larger than or equal to x, and by [x] B its thresholded (or clipped) value:

[x] B      -B if x < -B; x if -B x B; B if x > B.
Finally, we will use the (natural) conventions 1/0 = +∞, (+∞) × 0 = 0, and 0 ln(1 + U/0) = 0 for all U 0. Any sum 0 s=1 a s indexed from 1 up to 0 is by convention equal to 0.

Sparsity regret bounds for individual sequences

In this section we prove sparsity regret bounds for different variants of our algorithm SeqSEW. We first assume in Section 3.3.1 that the forecaster has access in advance to a bound B y on the observations |y t | and a bound B Φ on the trace of the empirical Gram matrix. We then remove these requirements one by one in Sections 3.3.2 and 3.3.3.

Known bounds B y on the observations and B Φ on the trace of the empirical Gram matrix

To simplify the analysis, we first assume that, at the beginning of the game, the number of rounds T is known to the forecaster and that he has access to a bound B y on all the observations y 1 , . . . , y T and to a bound B Φ on the trace of the empirical Gram matrix, i.e., y 1 , . . . , y T ∈ [-B y , B y ] and

d j=1 T t=1 ϕ 2 j (x t ) B Φ .
The first version of the algorithm studied in this chapter is defined in Figure 3.2 (adaptive variants will be introduced later). We name it SeqSEW for it is a variant of the Sparse Exponential Weighting algorithm introduced in the stochastic setting by [START_REF] Dalalyan | Aggregation by exponential weighting and sharp oracle inequalities[END_REF][START_REF] Dalalyan | Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity[END_REF] which is tailored for the prediction of individual sequences.

The choice of the heavy-tailed prior π τ is due to [START_REF] Dalalyan | Aggregation by exponential weighting and sharp oracle inequalities[END_REF]. The role of heavy-tailed priors to tackle the sparsity issue was already pointed out earlier; see, e.g., the discussion in [See08, Section 2.1]. In high dimension, such heavy-tailed priors favor sparsity: sampling from these prior distributions (or posterior distributions based on them) typically results in approximately sparse vectors, i.e., vectors having most coordinates almost equal to zero and the few remaining ones with quite large values.

Proposition 3.1. Assume that, for a known constant B y > 0, the (x 1 , y 1 ), . . . , (x T , y T ) are such that y 1 , . . . , y T ∈ [-B y , B y ]. Then, for all B B y , all η 1/(8B 2 ), and all τ > 0, the algorithm SeqSEW B,η

τ satisfies T t=1 (y t -y t ) 2 inf u∈R d T t=1 y t -u • ϕ(x t ) 2 + 4 η u 0 ln 1 + u 1 u 0 τ +τ 2 d j=1 T t=1 ϕ 2 j (x t ) .
(y t -y t ) 2 inf u∈R d T t=1 y t -u • ϕ(x t ) 2 + 32 B 2 y u 0 ln 1 + √ B Φ u 1 4 B y u 0 + 16B 2 y .
(3.4)

Parameters: threshold B > 0, inverse temperature η > 0, and prior scale τ > 0 with which we associate the sparsity prior π τ ∈ M + 1 (R d ) defined by

π τ (du) d j=1 (3/τ ) du j 2 1 + |u j |/τ 4 .
Initialization: p 1 π τ .

At each time round t 1,

1. Get the input data x t and predict a as y t

R d u • ϕ(x t ) B p t (du) ;
2. Get the observation y t and compute the posterior distribution

p t+1 ∈ M + 1 (R d ) as p t+1 (du) exp -η t s=1 y s -u • ϕ(x s ) B 2 W t+1 π τ (du) ,
where

W t+1 R d exp -η t s=1 y s -v • ϕ(x s ) B 2 π τ (dv) .
a The clipping operator [•]B is defined in Section 3.2. Note that, if ϕ ∞ sup x∈X max 1 j d |ϕ j (x)| is finite, then the last corollary provides a sparsity regret bound in the sense of (3.1). Indeed, in this case, we can take B Φ = d T ϕ 2 ∞ , which yields a regret bound proportional to u 0 and that grows logarithmically in d, T , u 1 , and ϕ ∞ .

To prove Proposition 3.1, we first need the following deterministic PAC-Bayesian inequality which is at the core of our analysis. It is a straightforward consequence of Theorem 4.6 of [START_REF] Audibert | Fast learning rates in statistical inference through aggregation[END_REF] when applied to the square loss (see also Appendix 3.A.1 for a self-contained proof). An adaptive variant of this inequality will be provided in Section 3.3.2. Lemma 3.1. Assume that for some known constant B y > 0, we have y 1 , . . . , y T ∈ [-B y , B y ]. For all τ > 0, if the algorithm SeqSEW B,η τ is used with B B y and η 1/(8B 2 ), then

T t=1 (y t -y t ) 2 inf ρ∈M + 1 (R d ) R d T t=1 y t -u • ϕ(x t ) B 2 ρ(du) + K(ρ, π τ ) η (3.5) inf ρ∈M + 1 (R d ) R d T t=1 y t -u • ϕ(x t ) 2 ρ(du) + K(ρ, π τ ) η . ( 3
∈ R d → u • ϕ(•) B ∈ G.
To apply the aforementioned theorem, recall from Appendix A.2 that the square loss is 1/(8B 2 )exp-concave on [-B, B] and thus η-exp-concave4 (since η 1/(8B 2 ) by assumption). Therefore, by Theorem 4.6 of [START_REF] Audibert | Fast learning rates in statistical inference through aggregation[END_REF] with the variance function δ η ≡ 0 (see the comments following Remark 4.1 therein), we get

T t=1 (y t -y t ) 2 inf µ∈M + 1 G G T t=1 y t -g(x t ) 2 µ(dg) + K(µ, π) η inf ρ∈M + 1 (R d ) R d T t=1 y t -u • ϕ(x t ) B 2 ρ(du) + K( ρ, π) η ,
where the last inequality follows by restricting the infimum over M + 1 G to the subset ρ : Remark 3.1. As can be seen from the previous proof, Lemma 3.1 still holds when π τ is replaced with any prior π ∈ M + 1 (R d ) (both in the statement of the lemma and in the definition of the algorithm SeqSEW). This fact is standard in the PAC-Bayesian approach; see, e.g., [START_REF] Catoni | Statistical learning theory and stochastic optimization[END_REF] and [START_REF] Dalalyan | Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity[END_REF]. As a consequence, any algorithm satisfying (3.6) will also satisfy Proposition 3.1 and Corollary 3.1.

ρ ∈ M + 1 (R d ) ⊂ M + 1 G , where ρ ∈ M + 1 G denotes the probability distribution induced by ρ ∈ M + 1 (R d ) via the mapping u ∈ R d → u • ϕ(•) B ∈ G.
Proof (of Proposition 3.1): Our proof mimics the proof of Theorem 5 in [START_REF] Dalalyan | Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity[END_REF]. We thus only write the outline of the proof and stress the minor changes that are needed to derive Inequality (3.3). The key technical tools provided in [START_REF] Dalalyan | Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity[END_REF] are reproduced in Appendix 3.B.1 for the convenience of the reader.

Let u * ∈ R d . Since B B y and η 1/(8B 2 ), we can apply Lemma 3.1 and get T t=1

(y t -y t ) 2 inf ρ∈M + 1 (R d ) R d T t=1 y t -u • ϕ(x t ) 2 ρ(du) + K(ρ, π τ ) η R d T t=1 y t -u • ϕ(x t ) 2 ρ u * ,τ (du)
(1)

+ K(ρ u * ,τ , π τ ) η (2)
.

(3.7)

In the last inequality, ρ u * ,τ is taken as the translated of π τ at u * , namely,

ρ u * ,τ (du) dπ τ du (u -u * ) du = d j=1 (3/τ ) du j 2 1 + |u j -u * j |/τ 4 .
The two terms (1) and (2) can be upper bounded as in the proof of Theorem 5 in [START_REF] Dalalyan | Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity[END_REF]. By a symmetry argument recalled in Lemma 3.3, the first term (1) can be rewritten as

R d T t=1 y t -u • ϕ(x t ) 2 ρ u * ,τ (du) = T t=1 y t -u * • ϕ(x t ) 2 + τ 2 d j=1 T t=1 ϕ 2 j (x t ) . (3.8)
As for the term (2), we have, as is recalled in Lemma 3.4,

K(ρ u * ,τ , π τ ) η 4 η u * 0 ln 1 + u * 1 u * 0 τ . ( 3
(y t -y t ) 2 inf u∈R d T t=1 y t -u • ϕ(x t ) 2 + 4 η u 0 ln 1+ u 1 u 0 τ + τ 2 d j=1 T t=1 ϕ 2 j (x t ) inf u∈R d T t=1 y t -u • ϕ(x t ) 2 + 4 η u 0 ln 1 + u 1 u 0 τ + τ 2 B Φ , (3.10) since d j=1 T t=1 ϕ 2 j (x t )
B Φ by assumption. The particular choices for η and τ given in the statement of the corollary then yield the desired inequality (3.4).

We end this subsection with a remark on the choices of B, η, and τ suggested in Corollary 3.1. The best choice of (B, η) that satisfies the assumptions of Proposition 3.1 is B = B y and η = 1/(8B 2 y ). As for the choice of τ , it approximately minimizes the upper bound given in (3.10). Indeed, for all C 1 , C 2 , C 3 > 0, the function f : (0, +∞) → R defined by

f (τ ) C 1 ln C 2 τ + C 3 τ 2 has a derivative equal to f (τ ) = -C 1 /τ + 2C 3 τ = τ -1 (2C 3 τ 2 -C 1 )
, which is negative on 0, C 1 /(2C 3 ) and positive on C 1 /(2C 3 ), +∞ . The function f thus admits a global minimum in τ = C 1 /(2C 3 ). Since the sum of the last two terms of (3.10) is approximately of the form of f (τ ) with5 C 1 = 4/η = 32B 2 y and C 3 = B Φ , a reasonable choice for τ is given by

τ = 32B 2 y 2B Φ = 16B 2 y B Φ .
3.3.2 Unknown bound B y on the observations but known bound B Φ on the trace of the empirical Gram matrix

In the previous section, to prove the upper bounds stated in Lemma 3.1 and Proposition 3.1, we assumed that the forecaster had access to a bound B y on the observations |y t | and to a bound B Φ on the trace of the empirical Gram matrix. In this section, we remove the first requirement and prove a sparsity regret bound for a variant of the algorithm SeqSEW B,η τ which is adaptive to the unknown bound B y = max 1 t T |y t |; see Proposition 3.2 and Remark 3.2 below.

Parameter: prior scale τ > 0 with which we associate the sparsity prior

π τ ∈ M + 1 (R d ) defined by π τ (du) d j=1 (3/τ ) du j 2 1 + |u j |/τ 4 .
Initialization: B 1 0, η 1 +∞, and p 1 π τ .

At each time round t 1,

1. Get the input data x t and predict a as y t

R d u • ϕ(x t ) Bt p t (du);
2. Get the observation y t and update:

• the threshold B t+1 2 log 2 max 1 s t y 2 s 1/2 ,
• the inverse temperature η t+1 1/ 8B 2 t+1 , • and the posterior distribution

p t+1 ∈ M + 1 (R d ) as p t+1 (du) exp -η t+1 t s=1 y s -u • ϕ(x s ) Bs 2 W t+1 π τ (du) ,
where

W t+1 R d exp -η t+1 t s=1 y s -v • ϕ(x s ) Bs 2 π τ (dv) .
a The clipping operator [•]B is defined in Section 3.2.

Figure 3.3: The algorithm SeqSEW * τ .

For this purpose we consider the algorithm of Figure 3.3, which we call SeqSEW * τ thereafter. It differs from SeqSEW B,η τ defined in the previous section in that the threshold B and the inverse temperature η are now allowed to vary over time and are chosen at each time round as a function of the data available to the forecaster.

The idea of truncating the base forecasts was used many times in the past; see, e.g., [START_REF] Vovk | Competitive on-line statistics[END_REF] for the online linear regression setting, [GKKW02, Chapter 10] for the regression problem with random design, and [GO07, BBGO10] for sequential prediction of unbounded time series under the square loss. A key ingredient in the present chapter is to perform truncation with respect to a data-driven threshold. The online tuning of this threshold is based on a pseudo-doubling-trick technique provided in [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF]. (We use the prefix pseudo since the algorithm does not restart at the beginning of each new regime.) Proposition 3.2. For all τ > 0, the algorithm SeqSEW * τ satisfies T t=1

(y t -y t ) 2 inf u∈R d T t=1 y t -u • ϕ(x t ) 2 + 32B 2 T +1 u 0 ln 1 + u 1 u 0 τ (3.11) + τ 2 d j=1 T t=1 ϕ 2 j (x t ) + 16B 2 T +1 ,
where

B 2 T +1 2 log 2 max 1 t T y 2 t 2 max 1 t T y 2 t .
Remark 3.2. In view of Proposition 3.1, the algorithm SeqSEW * τ satisfies a sparsity regret bound which is adaptive to the unknown bound B y = max 1 t T |y t |. The price for the automatic tuning with respect to B y consists only of a multiplicative factor smaller than 2 and the additive factor 16B 2 T +1 which is smaller than 32B 2 y .

As in the previous section, several corollaries can be derived from Proposition 3.2. If the forecaster has access beforehand to a quantity B Φ > 0 such that d j=1 T t=1 ϕ 2 j (x t ) B Φ , then a suboptimal but reasonable choice of τ is given by τ = 1/ √ B Φ ; see Corollary 3.2 below. The simpler tuning6 τ = 1/ √ dT of Corollary 3.3 will be useful in the stochastic batch setting (cf. Section 3.4). The proofs of the next corollaries are immediate.

Corollary 3.2. Assume that, for a known constant B Φ > 0, the (x 1 , y 1 ), . . . , (x T , y T ) are such that Corollary 3.3. Assume that T is known to the forecaster at the beginning of the prediction game. Then, when used with τ = 1/ √ dT , the algorithm SeqSEW * τ satisfies T t=1

d j=1 T t=1 ϕ 2 j (x t ) B Φ . Then, when used with τ = 1/ √ B Φ , the algorithm SeqSEW * τ satisfies T t=1 (y t -y t ) 2 inf u∈R d T t=1 y t -u • ϕ(x t ) 2 + 32B 2 T +1 u 0 ln 1 + √ B Φ u 1 u 0 (3.12) + 16B 2 T +1 + 1 , where B 2 T +1 2 log 2 max 1 t T y 2 t 2 max 1 t T y 2 t . √ d T ) -which corresponds to BΦ = d T ϕ 2 ∞ -but it
(y t -y t ) 2 inf u∈R d T t=1 y t -u • ϕ(x t ) 2 + 32B 2 T +1 u 0 ln 1 + √ dT u 1 u 0 (3.13) + 1 dT d j=1 T t=1 ϕ 2 j (x t ) + 16B 2 T +1 ,
where

B 2 T +1 2 log 2 max 1 t T y 2 t 2 max 1 t T y 2 t .
As in the previous section, to prove Proposition 3.2, we first need a key PAC-Bayesian inequality. The next lemma is an adaptive variant of Lemma 3.1. Lemma 3.2. For all τ > 0, the algorithm SeqSEW * τ satisfies

T t=1 (y t -y t ) 2 inf ρ∈M + 1 (R d ) R d T t=1 y t -u • ϕ(x t ) Bt 2 ρ(du) + 8B 2 T +1 K(ρ, π τ ) + 8B 2 T +1 (3.14) inf ρ∈M + 1 (R d ) R d T t=1 y t -u • ϕ(x t ) 2 ρ(du) + 8B 2 T +1 K(ρ, π τ ) + 16B 2 T +1 , (3.15)
where B 2

T +1 2 log 2 max 1 t T y 2 t 2 max 1 t T y 2 t .
Proof (of Lemma 3.2): The proof is based on arguments that are similar to those underlying Lemma 3.1, except that we now need to deal with B and η changing over time. In the same spirit as in [ACBG02, CBMS07, GO07], our analysis relies on the control of (ln W t+1 )/η t+1 -(ln W t )/η t where W 1 1 and, for all t 2,

W t R d exp -η t t-1 s=1 y s -u • ϕ(x s ) Bs 2 π τ (du) .
On the one hand, we have

ln W T +1 η T +1 - ln W 1 η 1 = 1 η T +1 ln R d exp -η T +1 T t=1 y t -u • ϕ(x t ) Bt 2 π τ (du) - 1 η 1 ln 1 = - inf ρ∈M + 1 (R d ) R d T t=1 y t -u • ϕ(x t ) Bt 2 ρ(du) + K(ρ, π τ ) η T +1 , (3.16)
where the last equality follows from a convex duality argument for the Kullback-Leibler divergence (cf., e.g., [Cat04, p. 159]) which we recall in Proposition A.1 in Appendix A.1.

On the other hand, we can rewrite (ln W T +1 )/η T +1 -(ln W 1 )/η 1 as a telescopic sum and get

ln W T +1 η T +1 - ln W 1 η 1 = T t=1 ln W t+1 η t+1 - ln W t η t = T t=1 ln W t+1 η t+1 - ln W t+1 η t (1) + 1 η t ln W t+1 W t (2)
, (3.17) where W t+1 is obtained from W t+1 by replacing η t+1 with η t ; namely,

W t+1 R d exp -η t t s=1 y s -u • ϕ(x s ) Bs 2 π τ (du) .
Let t ∈ {1, . . . , T }. The first term (1) is non-positive by Jensen's inequality (note that x → x η t+1 /ηt is concave on R * + since η t+1 η t by construction). As for the second term (2), by definition of W t+1 ,

1 η t ln W t+1 W t = 1 η t ln R d exp -η t y t -u • ϕ(x t ) Bt 2 exp -η t t-1 s=1 y s -u • ϕ(x s ) Bs 2 W t π τ (du) = 1 η t ln R d exp -η t y t -u • ϕ(x t ) Bt 2 p t (du) (3.18) -(y t -y t ) 2 if B t+1 = B t ; -(y t -y t ) 2 + (2B t+1 ) 2 if B t+1 >
R d e -ηt yt-u•ϕ(xt) B t 2 p t (du) exp -η t y t - R d u • ϕ(x t ) Bt p t (du) 2 = e -ηt(yt-yt) 2 ,
where the last equality follows by definition of y t . Taking the logarithms of both sides of the last inequality and dividing by η t , we get (3.19) when B t+1 = B t .

As for the rounds t such that B t+1 > B t , the square loss x → (y t -x) 2 is no longer 1/(8B

ln W T +1 η T +1 - ln W 1 η 1 - T t=1 (y t -y t ) 2 + 4 T t=1 t:B t+1 >Bt B 2 t+1 - T t=1 (y t -y t ) 2 + 8B 2 T +1 , (3.20)
where, setting K log 2 max 1 t T y 2 t , we bounded the geometric sum T t:B t+1 >Bt B 2 t+1 from above by K k=-∞ 2 k = 2 K+1 2B 2 T +1 in the same way as in Theorem 6 of [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF].

Putting Equations (3.16) and (3.20) together, we get the PAC-Bayesian inequality Proof (of Proposition 3.2): The proof follows the exact same lines as in Proposition 3.1 except that we apply Lemma 3.2 instead of Lemma 3.1. Indeed, using Lemma 3.2 and restricting the infimum to the ρ u * ,τ , u * ∈ R d (cf. (3.43)), we get that

T t=1 (y t -y t ) 2 inf ρ∈M + 1 (R d ) R d T t=1 y t -u • ϕ(x t ) Bt 2 ρ(du) + K(ρ, π τ ) η T +1 + 8B 2 T +1 ,
T t=1 (y t -y t ) 2 inf u * ∈R d R d T t=1 y t -u • ϕ(x t ) 2 ρ u * ,τ (du) + 8B 2 T +1 K(ρ u * ,τ , π τ ) + 16B 2 T +1 inf u * ∈R d T t=1 y t -u * • ϕ(x t ) 2 + 32B 2 T +1 u * 0 ln 1 + u * 1 u * 0 τ + τ 2 d j=1 T t=1 ϕ 2 j (x t ) + 16B 2 T +1 ,
where the last inequality follows from Lemmas 3.3 and 3.4.

A fully automatic algorithm

In the previous section, we proved that adaptation to B y was possible. If we also no longer assume that a bound B Φ on the trace of the empirical Gram matrix is available to the forecaster, then we can use a doubling trick on the nondecreasing quantity

γ t ln   1 + t s=1 d j=1 ϕ 2 j (x s )  
and repeatedly run the algorithm SeqSEW * τ of the previous section for rapidly-decreasing values of τ . This yields a sparsity regret bound with extra logarithmic multiplicative factors as compared to Proposition 3.2, but which holds for a fully automatic algorithm; see Theorem 3.1 below.

More formally, our algorithm SeqSEW * * is defined as follows. The set of all time rounds t = 1, 2, . . . is partitioned into regimes r = 0, 1, . . . whose final time instances t r are data-driven. Let t -1 0 by convention. We call regime r, r = 0, 1, . . ., the sequence of time rounds (t r-1 + 1, . . . , t r ) where t r is the first date t t r-1 + 1 such that γ t > 2 r . At the beginning of regime r, we restart the algorithm SeqSEW * τ defined in Figure 3.3 with the parameter τ = τ r , where τ r is the solution of the equation 2 r = ln(1 + 1/τ ), i.e., τ r 1/ exp(2 r ) -1 .

Theorem 3.1. Without requiring any preliminary knowledge at the beginning of the prediction game, SeqSEW * * satisfies, for all T 1 and all (x 1 , y 1 ), . . . , (x T , y T ) ∈ X × R,

T t=1 (y t -y t ) 2 inf u∈R d T t=1 y t -u • ϕ(x t ) 2 + 256 max 1 t T y 2 t u 0 ln   e + T t=1 d j=1 ϕ 2 j (x t )   + 64 max 1 t T y 2 t A T u 0 ln 1 + u 1 u 0 + 1 + 38 max 1 t T y 2 t A T ,
where

A T 2 + log 2 ln e + T t=1 d j=1 ϕ 2 j (x t ) .
On each regime r, the current instance of the algorithm SeqSEW * τr only uses the past observations y s , s ∈ {t r-1 + 1, . . . , t -1}, to perform the online trunction and to tune the inverse temperature parameter. Therefore, the algorithm SeqSEW * * is fully automatic. Note however that two possible improvements could be addressed in the future. From a theoretical viewpoint, can we contruct a fully automatic algorithm with a bound similar to Theorem 3.1 but without the extra logarithmic factor A T ? From a practical viewpoint, is it possible to perform the adaptation to B Φ without restarting the algorithm repeatedly (just like we did for B y )? A smoother time-varying tuning (τ t ) t 2 might enable to answer both questions. This would be very probably at the price of a more involved analysis (e.g., if we adapt the PAC-Bayesian bound of Lemma 3.2, then a third approximation term would appear in (3.17) since π τt changes over time).

Proof sketch (of Theorem 3.1): The proof relies on the application of Proposition 3.2 with τ = τ r on all regimes r visited up to time T . Summing the corresponding inequalities over r then concludes the proof. See Appendix 3.A.2 for a detailed proof. Theorem 3.1 yields the following corollary. It upper bounds the regret of the algorithm SeqSEW * * uniformly over all u ∈ R d such that u 0 s and u 1 U , where the sparsity level s ∈ N and the 1 -diameter U > 0 are both unknown to the forecaster. The proof is postponed to Appendix 3.A.2.

Corollary 3.4. Fix s ∈ N and U > 0. Then, for all T 1 and all (x 1 , y 1 ), . . . , (x T , y T ) ∈ X ×R, the regret of the algorithm SeqSEW * * on u : u 0 s u :

u 1 U is bounded by T t=1 (y t -y t ) 2 -inf u 0 s u 1 U T t=1 y t -u • ϕ(x t ) 2 256 max 1 t T y 2 t s ln   e + T t=1 d j=1 ϕ 2 j (x t )   + 64 max 1 t T y 2 t A T s ln 1 + U s + 1 + 38 max 1 t T y 2 t A T ,
where A T 2 + log 2 ln e + T t=1 d j=1 ϕ 2 j (x t ) .

Adaptivity to the unknown variance in the stochastic setting

In this section, we apply the online algorithm SeqSEW * τ of Section 3.3.2 to two related stochastic settings: the regression model with random design (Section 3.4.1) and the regression model with fixed design (Section 3.4.2). The sparsity regret bounds proved for this algorithm on individual sequences imply in both settings sparsity oracle inequalities with leading constant 1. These risk bounds are of the same flavor as in [START_REF] Dalalyan | Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity[END_REF][START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF] but they are adaptive (up to a logarithmic factor) to the unknown variance σ 2 of the noise if the latter is Gaussian. In particular, we solve two questions left open in [START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF] in the random design case.

In the sequel, just like in the online deterministic setting, we assume that the forecaster has access to a dictionary ϕ = (ϕ 1 , . . . , ϕ d ) of measurable base regressors ϕ j : X → R, j = 1, . . . , d.

Regression model with random design

In this section we apply the algorithm SeqSEW * τ to the regression model with random design. In this batch setting the forecaster is given at the beginning of the game T independent random copies (X 1 , Y 1 ), . . . , (X T , Y T ) of (X, Y ) ∈ X × R whose common distribution is unknown. We assume thereafter that E[Y 2 ] < ∞; the goal of the forecaster is to estimate the regression function

f : X → R defined by f (x) E[Y |X = x] for all x ∈ X . Setting ε t Y t -f (X t ) for all t = 1, . . . , T , note that Y t = f (X t ) + ε t , 1 t T ,
and that the pairs (X 1 , ε 1 ), . . . , (X T , ε T ) are i.i.d. and such that E[ε 2 1 ] < ∞ and E[ε 1 |X 1 ] = 0 almost surely. In the sequel, we denote the distribution of X by P X and we set, for all measurable functions h : X → R,

h L 2 X h(x) 2 P X (dx) 1/2 = E h(X) 2 1/2 .
Next we construct a regressor f T : X → R based on the sample (X 1 , Y 1 ), . . . , (X T , Y T ) that satisfies a sparsity oracle inequality, i.e., its expected

L 2 -risk E f -f T 2 L 2 is almost as small as the smallest L 2 -risk f -u • ϕ 2 L 2 , u ∈ R d , up
to some additive term proportional to u 0 .

Algorithm and main result

Even if the whole sample (X 1 , Y 1 ), . . . , (X T , Y T ) is available at the beginning of the prediction game, we treat it in a sequential fashion. We run the algorithm SeqSEW * τ of Section 3.3.2 from time 1 to time T with τ = 1/ √ dT (note that T is known in this setting). Using the standard online to batch conversion (cf. Section 2.5 in Chapter 2), we define our data-based regressor f T : X → R as the uniform average

f T 1 T T t=1 f t (3.21)
of the regressors f t : X → R sequentially built by the algorithm SeqSEW * τ as

f t (x) R d u • ϕ(x) Bt p t (du) . (3.22)
Note that, contrary to much prior work from the statistics community such as [Cat04, BN08, DT11], the regressors f t : X → R are tuned online. Therefore, f T does not depend on any prior knowledge on the unknown distribution of the (X t , Y t ), 1 t T , such as the unknown variance E (Y -f (X)) 2 of the noise, the ϕ j ∞ , or the f -ϕ j ∞ (actually, the ϕ j and the f -ϕ j do not even need to be bounded in ∞ -norm).

In this respect, as explained in Section 2.5.2 (Chapter 2), this work improves on [START_REF] Bunea | Sequential procedures for aggregating arbitrary estimators of a conditional mean[END_REF] who tune their online forecasters as a function of max 1 j d ϕ j ∞ . The major technique difference is that we truncate the base forecasts u • ϕ(X t ) instead of truncating the observations Y t . In particular, this enables to aggregate the base regressors u•ϕ for all u ∈ R d , i.e., in the whole R d space.

The next sparsity oracle inequality is the main result of this section. It follows from the deterministic regret bound of Corollory 3.3 and from Jensen's inequality. Two corollaries are to be derived later.

Theorem 3.2. Assume that (X 1 , Y 1 ), . . . , (X T , Y T ) ∈ X × R are independent random copies of (X, Y ) ∈ X × R, where E[Y 2 ] < +∞ and ϕ j 2 L 2 E[ϕ j (X) 2 ] < +∞ for all j = 1, . . . , d. Then, the data-based regressor f T defined in (3.21)-(3.22) satisfies E f -f T 2 L 2 inf u∈R d f -u • ϕ 2 L 2 + 64 E max 1 t T Y 2 t T u 0 ln 1 + √ dT u 1 u 0 + 1 dT d j=1 ϕ j 2 L 2 + 32 E max 1 t T Y 2 t T .
Note that our risk bounds are stated in expectation (which already improves on existing results in the stochastic setting, see the next section). However, by convexity (and closedness) of all sets of the form u • ϕ : J(u) ⊂ J 0 , u 1 U }, where U 0 and J 0 ⊂ {1, . . . , d}, and where J(u) {j : u j = 0}, it is possible to use [Zha05, Theorem 8] to transform our results into risk bounds with high probability (at least when the output Y is bounded, but similar results should hold true under reasonable assumptions on the output distribution).

Proof sketch (of Theorem 3.2): By Corollary 3.3 and by definition of f t above and y t f t (X t ) in Figure 3.3, we have, almost surely,

T t=1 (Y t -f t (X t )) 2 inf u∈R d T t=1 Y t -u • ϕ(X t ) 2 + 64 max 1 t T Y 2 t u 0 ln 1 + √ dT u 1 u 0 + 1 dT d j=1 T t=1 ϕ 2 j (X t ) + 32 max 1 t T Y 2 t .
Taking the expectations of both sides and applying Jensen's inequality yields the desired result. For a detailed proof, see Appendix 3.A.3.

Theorem 3.2 above can be used under several assumptions on the distribution of the output Y . In all cases, it suffices to upper bound the amplitude E max 1 t T Y 2 t . We present below a general corollary and explain later why our fully automatic procedure f T solves two questions left open by [START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF] (see Corollary 3.6).

A general corollary

Using Lemmas 3.5, 3.6, and 3.7 in Appendix 3.B to upper bound the two terms E max 1 t T Y 2 t of Theorem 3.2, we get the following sparsity oracle inequality. The proof is postponed to Appendix 3.A.3. • SG(σ 2 ) : ∆Y is subgaussian with variance factor σ 2 > 0, that is, E e λ∆Y e λ 2 σ 2 /2 for all λ ∈ R;

Corollary 3.5. Assume that (X 1 , Y 1 ), . . . , (X T , Y T ) ∈ X × R are independent random copies of (X, Y ) ∈ X × R, that sup 1 j d ϕ j 2 L 2 < +∞, that E|Y | < +∞,
• BEM(α, M) : ∆Y has a bounded exponential moment, that is, E e α|∆Y | M for some given constants α > 0 and M > 0;

• BM(α, M) : ∆Y has a bounded moment, that is, E |∆Y | α
M for some given constants α > 2 and M > 0.

Then, the data-based regressor f T defined above satisfies

E f -f T 2 L 2 inf u∈R d f -u • ϕ 2 L 2 + 128 E[Y ] 2 T + ψ T u 0 ln 1 + √ dT u 1 u 0 + 1 dT d j=1 ϕ j 2 L 2 + 64 E[Y ] 2 T + ψ T ,
where

ψ T 1 T E max 1 t T Y t -E[Y t ] 2                          B 2 T under Assumption BD(B) , 2σ 2 ln(2eT ) T under Assumption SG(σ 2 ) , ln 2 (M + e)T α 2 T under Assumption BEM(α, M) , M 2/α T (α-2)/α
under Assumption BM(α, M) .

Several comments can be made about Corollary 3.5. We first stress that, if T 2, then the two "bias" terms E[Y ] 2 /T above can be avoided, at least at the price of a multiplicative factor of 2T /(T -1) 4. This can be achieved via a slightly more sophisticated online clipping -see Remark 3.4 in Appendix 3.A.3.

Second, under the assumptions BD(B) , SG(σ 2 ) , or BEM(α, M) , the key quantity ψ T is respectively of the order of 1/T , ln(T )/T and ln 2 (T )/T . Up to a logarithmic factor, this corresponds to the classical fast rate of convergence 1/T obtained in the random design setting for different aggregation problems (see, e.g., [START_REF] Catoni | Universal aggregation rules with exact bias bounds[END_REF][START_REF] Juditsky | Learning by mirror averaging[END_REF][START_REF] Audibert | Fast learning rates in statistical inference through aggregation[END_REF] for model-selection-type aggregation and [START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF] for linear aggregation). However, the rate T -(α-2)/α we proved under the bounded moment assumption BM(α, M) does not match the faster rate T -α/(α+2) obtained in [START_REF] Juditsky | Learning by mirror averaging[END_REF][START_REF] Audibert | Fast learning rates in statistical inference through aggregation[END_REF] under a similar assumption. In particular the bound of Corollary 3.5 goes to M > 0 as α → 2, while the optimal rate for α = 2 in similar situations is

T -1/2 [Aud09].
The minor logarithmic difference under Assumptions SG(σ 2 ) or BEM(α, M) and the clear difference in the rates under Assumption BM(α, M) come from the fact that our online algorithm SeqSEW * τ was primarily designed for bounded individual sequences with an unknown bound. As remarked in Section 2.5.2 (Chapter 2), the finite i.i.d. sequence Y 1 , . . . , Y T is almost surely uniformly bounded by the random bound max 1 t T |Y t |. Our individual sequence techniques adapt sequentially to this random bound, yielding a regret bound that scales as max 1 t T Y 2 t . As a result, the risk bounds obtained after the online to batch conversion scale as E max 1 t T Y 2 t /T . If the distribution of the output Y is bounded or lightly-tailed, then we can (almost) recover for free the fast rate of convergence 1/T (the extra logarithmic factor coming from the "slight" non-boundedness of Y ). But if the distribution of Y is heavy-tailed, then a different tuning of η t or a more sophisticated online truncation seem necessary. Doing so without requiring any prior knowledge on the output distribution such as the quantities σ 2 , α, M ... -as is the case for our fully automatic procedure f T -is a challenging task.

Third, several variations on the assumptions are possible. First note that several classical assumptions on Y expressed in terms of f (X) and ε Y -f (X) are either particular cases of the above corollary or can be treated similarly. Indeed, each of the four assumptions above on ∆Y

Y -E[Y ] = f (X) -E[f (X)
] + ε is satisfied as soon as both the distribution of f (X) -E[f (X)] and the conditional distribution of ε (conditionally on X) satisfy the same type of assumption. For example, if f (X) -E[f (X)] is subgaussian with variance factor σ 2 X and if ε is subgaussian conditionally on X with a variance factor uniformly bounded by a constant σ 2 ε , then ∆Y is subgaussian with variance factor σ 2 X + σ 2 ε (see also Remark 3.5 in Appendix 3.A.3 to avoid conditioning).

The assumptions on f (X) -E[f (X)] and ε can also be mixed together. For instance, as explained in Remark 3.5 in Appendix 3.A.3, under the classical assumptions

f ∞ < +∞ and E e α|ε| X M a.s. (3.23) or f ∞ < +∞ and E e λε X e λ 2 σ 2 /2 a.s., ∀λ ∈ R , (3.24)
the key quantity ψ T in the corollary can be bounded from above by

ψ T        8 f 2 ∞ T + 2 ln 2 (M + e)T α 2 T under the set of assumptions (3.23), 8 f 2 ∞ T + 4σ 2 ln(2eT )
T under the set of assumptions (3.24).

In particular, under the set of assumptions (3.24), our procedure f T solves two questions left open in [START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF]. We discuss below our contributions in this particular case.

Questions left open by Dalalyan and Tsybakov

In this subsection we focus on the case when the regression function f is bounded (by an unknown constant) and when the noise ε Y -f (X) is subgaussian conditionally on X in the sense that, for some (unknown) constant σ 2 > 0,

f ∞ < +∞
and E e λε X e λ 2 σ 2 /2 a.s., ∀λ ∈ R .

(3.25)

In this case, the two terms E max 1 t T Y 2 t of Theorem 3.2 can be upper bounded in a simpler and slightly tighter way as compared to the proof of Corollary 3.5 (we only use the inequality (x + y) 2 2x 2 + 2y 2 once, instead of twice). It yields the following sparsity oracle inequality.

Corollary 3.6. Assume that (X 1 , Y 1 ), . . . , (X T , Y T ) ∈ X × R are independent random copies of (X, Y ) ∈ X × R such that the set of assumptions (3.25) above holds true. Then, the data-based regressor f T defined in

(3.21)-(3.22) satisfies E f -f T 2 L 2 inf u∈R d f -u • ϕ 2 L 2 + 128 f 2 ∞ + 2σ 2 ln(2eT ) u 0 T ln 1 + √ dT u 1 u 0 + 1 dT d j=1 ϕ j 2 L 2 + 64 T f 2 ∞ + 2σ 2 ln(2eT ) .
Proof: We apply Theorem 3.2 and bound E max 1 t T Y 2 t from above. By the elementary inequality (x + y) 2 2x 2 + 2y 2 for all x, y ∈ R, we get

E max 1 t T Y 2 t = E max 1 t T f (X t ) + ε t 2 2 f 2 ∞ + E max 1 t T ε 2 t 2 f 2 ∞ + 2σ 2 ln(2eT ) ,
where the last inequality follows from Lemma 3.5 in Appendix 3.B and from the fact that, for all 1 t T and all λ ∈ R, we have E e λεt = E e λε = E E e λε X e λ 2 σ 2 /2 by (3.25). (Note that the assumption of conditional subgaussianity in (3.25) is stronger than what we need, i.e., subgaussianity without conditioning.) This concludes the proof.

The above bound is of the same order (up to a ln T factor) as the sparsity oracle inequality proved in Proposition 1 of [START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF]. For the sake of comparison we state below with our notations (e.g., β therein corresponds to 1/η in this chapter) a straightforward consequence of this proposition, which follows by Jensen's inequality and the particular 7 choice τ = 1/ √ dT .

7 Proposition 1 of [START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF] may seem more general than Theorem 3.2 at first sight since it holds for all τ > 0, but this is actually also the case for Theorem 3.2. The proof of the latter would indeed have remained true had we replaced τ = 1/ √ dT with any value of τ > 0. We however chose the reasonable value τ = 1/ √ dT to make our algorithm parameter-free. As noted earlier, if ϕ ∞ sup x∈X max 1 j d |ϕj(x)| is finite and known by the forecaster, another simple and easy-to-analyse tuning is given by τ

= 1/( ϕ ∞ √ d T ).
Proposition 3.3 (A consequence of Prop. 1 of [START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF]).

Assume that sup 1 j d ϕ j ∞ < ∞ and that the set of assumptions (3.25) above hold true. Then, for all R > 2 d/T and all η η(R)

2σ 2 + 2 sup u 1 R u • ϕ -f 2 ∞ -1
, the mirror averaging aggregate f T : X → R defined in [DT11, Equations (1) and (3)] satisfies

E f -f T 2 L 2 inf u 1 R-2 √ d/T f -u • ϕ 2 L 2 + 4 η u 0 T + 1 ln 1 + √ dT u 1 u 0 + 4 dT d j=1 ϕ j 2 L 2 + 1 (T + 1)η .
We can now discuss the two questions left open by [START_REF] Dalalyan | Mirror averaging with sparsity priors[END_REF]. Despite the similarity of the two bounds, the sparsity oracle inequality stated in Proposition 3.3 above only holds for vectors u within an 1 -ball of finite radius R -2 d/T , while our bound holds over the whole R d space. Moreover, the parameter R above has to be chosen in advance, but it cannot be chosen too large since 1/η 1/η(R), which grows as R 2 when R → +∞ (if ϕ = 0). The authors asked in [DT11, Section 4.2] whether it was possible to get a bound with 1/η < +∞ such that the infimum in Proposition 3.3 extends to the whole R d space. Our results show that, thanks to datadriven truncation, the answer is positive. Note that it is still possible to transform the bound of Proposition 3.3 into a bound over the whole R d space if the parameter R is chosen (illegally) as R = u * 1 + 2 d/T (or as a tight upper bound of the last quantity), where u * ∈ R d minimizes over R d the regularized risk

f -u • ϕ 2 L 2 + 4 η( u 1 + 2 d/T ) u 0 T + 1 ln 1 + √ dT u 1 u 0 + 4 dT d j=1 ϕ j 2 L 2 + 1 (T + 1)η( u 1 + 2 d/T ) .
For instance, choosing R = u * 1 + 2 d/T and η = η(R), we get from Proposition 3.3 that the expected L 2 -risk E f -f T 2 L 2 of the corresponding procedure is upper bounded by the infimum of the above regularized risk over all u ∈ R d . However, this parameter tuning is illegal since u * 1 is not known in practice. On the contrary, thanks to data-driven truncation, the prior knowledge of u * 1 is not required by our procedure.

The second open question, which was raised in [DT11, Section 5.1, Remark 6], deals with the prior knowledge of the variance factor σ 2 of the noise. The latter is indeed required by their algorithm for the choice of the inverse temperature parameter η. The authors thus asked whether adaptivity to σ 2 was possible. Corollary 3.6 above provides a positive answer (up to a ln T factor).

Regression model with fixed design

In this section, we consider the regression model with fixed design. In this batch setting the forecaster is given at the beginning of the game a T -sample (x 1 , Y 1 ), . . . , (x T , Y T ) ∈ X × R, where the x t are deterministic elements in X and where

Y t = f (x t ) + ε t , 1 t T, (3.26)
for some i.i.d. sequence ε 1 , . . . , ε T ∈ R (with unknown distribution) and some unknown function f : X → R.

In this setting, just like in Section 3.4.1, our algorithm and the corresponding analysis are a straightforward consequence of the general results on individual sequences developed in Section 3.3. As in the random design setting, the sample (x 1 , Y 1 ), . . . , (x T , Y T ) is treated in a sequential fashion. We run the algorithm SeqSEW * τ defined in Figure 3.3 from time 1 to time T with the particular choice of τ = 1/ √ dT . We then define our data-based regressor f T : X → R by

f T (x)        1 n x 1 t T t:xt=x f t (x) if x ∈ {x 1 , . . . , x T } , 0 if x / ∈ {x 1 , . . . , x T } , (3.27)
where n x t : x t = x = T t=1 I {xt=x} , and where the regressors f t : X → R sequentially built by the algorithm SeqSEW * τ are defined by

f t (x) R d u • ϕ(x) Bt p t (du) .
(3.28)

In the particular case when the x t are all distinct, f T is simply defined by f T (x) f T (x) if x ∈ {x 1 , . . . , x T } and by f T (x) = 0 otherwise. The next theorem is the main result of this subsection. It follows as in the random design setting from the deterministic regret bound of Corollory 3.3 and from Jensen's inequality. The proof is postponed to Appendix 3.A.4. Theorem 3.3. Consider the regression model with fixed design described in (3.26). Then, the data-based regressor f T defined in (3.27)-(3.28) satisfies

E 1 T T t=1 f (x t ) -f T (x t ) 2 inf u∈R d 1 T T t=1 f (x t ) -u • ϕ(x t ) 2 + 64 E max 1 t T Y 2 t T u 0 ln 1 + √ dT u 1 u 0 + 1 dT 2 d j=1 T t=1 ϕ 2 j (x t ) + 32 E max 1 t T Y 2 t T .
As in Section 3.4.1, the amplitude E max 1 t T Y 2 t can be upper bounded under various assumptions. The proof of the following corollary is postponed to Appendix 3.A.4. Corollary 3.7. Consider the regression model with fixed design described in (3.26). Assume that one of the following assumptions holds on the distribution of ε 1 .

• BD(B) : |ε 1 | B almost surely for a given constant B > 0;

• SG(σ 2 ) : ε 1 is subgaussian with variance factor σ 2 > 0, that is, E e λε 1 e λ 2 σ 2 /2 for all λ ∈ R;

• BEM(α, M) : ε has a bounded exponential moment, that is, E e α|ε| M for some given constants α > 0 and M > 0;

• BM(α, M) : ε has a bounded moment, that is, E |ε| α M for some given constants α > 2 and M > 0.

Then, the data-based regressor f T defined in (3.27)-(3.28) satisfies

E 1 T T t=1 f (x t ) -f T (x t ) 2 inf u∈R d 1 T T t=1 f (x t ) -u • ϕ(x t ) 2 + 128 max 1 t T f 2 (x t ) T + ψ T u 0 ln 1 + √ dT u 1 u 0 + 1 dT 2 d j=1 T t=1 ϕ 2 j (x t ) + 64 max 1 t T f 2 (x t ) T + ψ T ,
where

ψ T 1 T E max 1 t T ε 2 t                        B 2 T if Assumption BD(B) holds, 2σ 2 ln(2eT ) T if Assumption SG(σ 2 ) holds, ln 2 ((M + e)T ) α 2 T if Assumption BEM(α, M) holds, M 2/α T (α-2)/α if Assumption BM(α, M) holds.
The above bound is of the same flavor as that of [DT08, Theorem 5]. It has one advantage and one drawback. On the one hand, we note two additional "bias" terms max 1 t T f 2 (x t ) /T as compared to the bound of [DT08, Theorem 5]. As of now, we have not been able to remove them using ideas similar to what we did in the random design case (see Remark 3.4 in Appendix 3.A.3). On the other hand, under Assumption SG(σ 2 ) , contrary to [START_REF] Dalalyan | Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity[END_REF], our algorithm does not require the prior knowledge of the variance factor σ 2 of the noise. Proof (of Inequality (3.5) of Lemma 3.1): As is usually done in the online learning setting for the study of the exponentially weighted average forecaster, this proof relies on the control of t η -1 ln(W t+1 /W t ) where we recall that W 1 1 and, for all t 2,

3.A Proofs

W t R d exp -η t-1 s=1 y s -u • ϕ(x s ) B 2 π τ (du) .
On the one hand, we have

1 η ln W T +1 W 1 = 1 η ln R d exp -η T t=1 y s -u • ϕ(x s ) B 2 π τ (du) - 1 η ln 1 = - inf ρ∈M + 1 (R d ) R d T t=1 y t -u • ϕ(x t ) B 2 ρ(du) + K(ρ, π τ ) η , (3.29)
where the last equality follows from a convex duality argument for the Kullback-Leibler divergence (cf., e.g., [Cat04, p. 159]) which we recall in Proposition A.1 in Appendix A.1.

On the other hand, we rewrite η -1 ln(W T +1 /W 1 ) = T t=1 η -1 ln(W t+1 /W t ) as a telescopic sum and note that, for all t = 1 . . . , T ,

1 η ln W t+1 W t = 1 η ln R d exp -η y t -u • ϕ(x t ) B 2 exp -η t-1 s=1 y s -u • ϕ(x s ) B 2 W t π τ (du) = 1 η ln R d exp -η y t -u • ϕ(x t ) B 2 p t (du) , (3.30)
where (3.30) follows from the definition of p t .

Let t ∈ {1, . . . , T }. First note that by assumption y t ∈ [-B y , B y ] ⊂ [-B, B] so that both y t and u•ϕ(x t ) B are [-B, B]-valued for all u ∈ R d . Moreover, from Proposition A.2 in Appendix A.2, the square loss is 1/(8B 2 )-exp-concave on [-B, B] and thus η-exp-concave (since η 1/(8B 2 ) by assumption). Therefore, by Jensen's inequality,

R d e -η yt-u•ϕ(xt) B 2 p t (du) exp -η y t - R d u • ϕ(x t ) B p t (du) 2 .
Taking the logarithms of both sides of the inequality yields ln

R d e -η yt-u•ϕ(xt) B 2 p t (du) -η y t - R d u • ϕ(x t ) B p t (du) 2 = -η(y t -y t ) 2 .
(3.31)

Dividing the latter inequality by η, summing over t ∈ {1, . . . , T } and combining with Equa-tion (3.30), we get

1 η ln W T +1 W 1 - T t=1 (y t -y t ) 2 .
We conclude the proof of (3.5) of Lemma 3.1 by combining the last inequality with (3.29).

3.A.2 Proofs of Theorem 3.1 and Corollary 3.4

Before proving Theorem 3.1, we first need the following comment. Since the algorithm SeqSEW * τ is restarted at the beginning of each regime, the threshold values B t used on regime r by the algorithm SeqSEW * τ are not computed on the basis of all past observations y 1 , . . . , y t-1 but only on the basis of the past observations y t , t ∈ {t r-1 + 1, . . . , t -1}. To avoid any ambiguity, we set

B r,t 2 log 2 max t r-1 +1 s t-1 y 2 s 1/2 , t ∈ {t r-1 + 1, . . . , t r } . (3.32)
Proof (of Theorem 3.1): We denote by R min{r ∈ N : T t r } the index of the last regime. For notational convenience, we re-define t R T (even if γ T 2 R ).

We upper bound the regret of the algorithm SeqSEW * * on {1, . . . , T } by the sum of its regrets on each time interval. To do so, first note that

T t=1 (y t -y t ) 2 = R r=0 tr t=t r-1 +1 (y t -y t ) 2 = R r=0   (y tr -y tr ) 2 + tr-1 t=t r-1 +1 (y t -y t ) 2   R r=0   2(y 2 tr + B 2 r,tr ) + tr-1 t=t r-1 +1 (y t -y t ) 2   (3.33) R r=0   tr-1 t=t r-1 +1 (y t -y t ) 2   + 6(R + 1)y * T 2 , ( 3
2 log 2 max t r-1 +1 t tr -1 y 2 t 2 max t r-1 +1 t tr-1 y 2 t 2 y * T 2 .
But, for every r = 0, . . . , R, the trace of the empirical Gram matrix on {t r-1 + 1, . . . , t r -1} is upper bounded by

tr-1 t=t r-1 +1 d j=1 ϕ 2 j (x t ) tr-1 t=1 d j=1 ϕ 2 j (x t ) (e 2 r -1) 2 ,
where the last inequality follows from the fact that γ tr-1 2 r (by definition of t r ). Since in addition τ r 1/ (e 2 r -1) 2 , we can apply Corollory 3.2 on each period {t r-1 + 1, . . . , t r -1}, r = 0, . . . , R, with B Φ = (e 2 r -1) 2 and get from (3.34) the upper bound

T t=1 (y t -y t ) 2 R r=0 inf u∈R d    tr-1 t=t r-1 +1 y t -u • ϕ(x t ) 2 + ∆ r (u)    + 6(R + 1)y * T 2 , (3.35)
where

∆ r (u) 32B 2 tr u 0 ln 1 + e 2 r -1 u 1 u 0 + 16B 2 r,tr + 1 . (3.36)
Since the infimum is superadditive and since y tr -u • ϕ(x tr ) 2 0 for all r = 0, . . . , R, we get from (3.35) that T t=1 Second case: R 1 Since R 1, we have, by definition of t R-1 ,

(y t -y t ) 2 inf u∈R d R r=0   tr t=t r-1 +1 y t -u • ϕ(x t ) 2 + ∆ r (u)   + 6(R + 1)y * T 2 = inf u∈R d T t=1 y t -u • ϕ(x t ) 2 + R r=0 ∆ r (u) + 6(R + 1)y * T 2 . (3.37) Let u ∈ R d . Next we bound R r=0 ∆ r (u
2 R-1 < γ t R-1 ln   1 + t R-1 t=1 d j=1 ϕ 2 j (x t )   ln   e + T t=1 d j=1 ϕ 2 j (x t )   .
The last inequality entails that 2 R+1 -1 4 • 2 R-1 4 ln e + Substituting the last two inequalities in (3.37) and noting that y * T 2 = max 1 t T y 2 t concludes the proof.

Proof (of Corollary 3.4):

The proof is straightforward. In view of Theorem 3.1, we just need to check that the quantity (continuously extended in s = 0) 256 max

1 t T y 2 t s ln   e + T t=1 d j=1 ϕ 2 j (x t )   + 64 max 1 t T y 2 t A T s ln 1 + U s is non-decreasing in s ∈ R + and in U ∈ R + .
This is clear for U . The fact that it also non-decreasing in s comes from the following remark. For all U 0, the function s ∈ (0, +∞) → s ln(1 + U/s) has a derivative equal to

ln 1 + U s - U/s 1 + U/s
for all s > 0 .

From the elementary inequality

ln(1 + u) = -ln 1 1 + u - 1 1 + u -1 = u 1 + u ,
which holds for all u ∈ (-1, +∞), the above derivative is nonnegative for all s > 0 so that the continuous extension s ∈ R + → s ln (1 + U/s) is non-decreasing.

3.A.3 Proofs of Theorem 3.2 and Corollary 3.5

In this subsection, we set ε Y -f (X), so that the pairs (X 1 , ε 1 ), . . . , (X T , ε T ) are independent copies of (X, ε) ∈ X × R. We also define σ 0 by

σ 2 E ε 2 = E (Y -f (X)) 2 .
Proof (of Theorem 3.2): By Corollory 3.3 and the definitions of f t above and y t f t (X t ) in Figure 3.3, we have, almost surely,

T t=1 (Y t -f t (X t )) 2 inf u∈R d T t=1 Y t -u • ϕ(X t ) 2 + 64 max 1 t T Y 2 t u 0 ln 1+ √ dT u 1 u 0 + 1 dT d j=1 T t=1 ϕ 2 j (X t ) + 32 max 1 t T Y 2 t .
It remains to take the expectations of both sides with respect to (X 1 , Y 1 ), . . . , (X T , Y T ) . First note that for all t = 1, . . . , T , since ε t Y t -f (X t ), we have

E Y t -f t (X t ) 2 = E ε t + f (X t ) -f t (X t ) 2 = σ 2 + E f (X t ) -f t (X t ) 2 , since E ε 2 t = E ε 2
σ 2 one the one hand, and, on the other hand, f t is a measurable function of (X s , Y s ) 1 s t-1 and E ε t (X s , Y s ) 1 s t-1 , X t = E ε t X t = 0 (from the independence of (X s , Y s ) 1 s t-1 and (X t , Y t ) and by definition of f ).

In the same way,

E Y t -u • ϕ(X t ) 2 = σ 2 + E f (X t ) -u • ϕ(X t ) 2 .
Therefore, by Jensen's inequality and the concavity of the infimum, the last inequality becomes, after taking the expectations of both sides,

T σ 2 + T t=1 E f (X t ) -f t (X t ) 2 inf u∈R d T σ 2 + T t=1 E f (X t ) -u • ϕ(X t ) 2 + 64 E max 1 t T Y 2 t u 0 ln 1 + √ dT u 1 u 0 + 1 dT d j=1 T t=1 E ϕ 2 j (X t ) + 32 E max 1 t T Y 2 t .
Noting that the T σ 2 cancel out, dividing the two sides by T , and using the fact that X t ∼ X in the right-hand side, we get

1 T T t=1 E f (X t ) -f t (X t ) 2 inf u∈R d f -u • ϕ 2 L 2 + 64 E max 1 t T Y 2 t T u 0 ln 1 + √ dT u 1 u 0 + 1 dT d j=1 ϕ j 2 L 2 + 32 E max 1 t T Y 2 t T .
The right-hand side of the last inequality is exactly the upper bound stated in Theorem 3.2. To conclude the proof, we thus only need to check that f -f T 2 L 2 is bounded from above by the left-hand side. But by definition of f T and by convexity of the square loss we have

E f -f T 2 L 2 E f (X) - 1 T T t=1 f t (X) 1 T T t=1 E f (X) -f t (X) 2 = 1 T T t=1 E f (X t ) -f t (X t ) 2 .
The last equality follows classically from the fact that, for all t = 1, . . . , T , (X s , Y s ) 1 s t-1 (on which f t is constructed) is independent from both X t and X and the fact that X t ∼ X.

Remark 3.3. The fact that the inequality stated in Corollary 3.3 has a leading constant equal to 1 on individual sequences is crucial to derive in the stochastic setting an oracle inequality in terms of the (excess

) risks E f -f T 2 L 2 and f -u • ϕ 2 L 2 .
Indeed, if the constant appearing in front of the infimum was equal to C > 1, then the T σ 2 would not cancel out in the previous proof, so that the resulting expected inequality would contain a non-vanishing additive term (C -1)σ 2 . Proof (of Corollary 3.5): We can apply Theorem 3.2. Then, to prove the upper bound on

E f -f T 2 L 2 , it suffices to show that E max 1 t T Y 2 t T 2 E[Y ] 2 T + ψ T . (3.39) Recall that ψ T 1 T E max 1 t T Y t -E[Y t ] 2 = 1 T E max 1 t T (∆Y ) 2 t ,
where we defined

(∆Y ) t Y t -E[Y t ] = Y t -E[Y ] for all t = 1, . . . , T .
From the elementary inequality (x + y) 2 2x 2 + 2y 2 for all x, y ∈ R, we have

E max 1 t T Y 2 t E max 1 t T E[Y ] + (∆Y ) t 2 2 E[Y ] 2 + 2 E max 1 t T (∆Y ) 2 t .
(3.40)

Dividing both sides by T , we get (3.39).

As for the upper bound on ψ T , since the (∆Y ) t , 1 t T , are distributed as ∆Y , we can apply Lemmas 3.5, 3.6, and 3.7 in Appendix 3.B.2 to bound ψ T from above under the assumptions SG(σ 2 ) , BEM(α, M) , and BM(α, M) respectively (the upper bound under BD(B) is straightforward):

E max 1 t T (∆Y ) 2 t                B 2
if Assumption BD(B) holds,

σ 2 + 2σ 2 ln(2eT ) if Assumption SG(σ 2 ) holds, ln 2 (M + e)T α 2 if Assumption BEM(α, M) holds, (M T ) 2/α if Assumption BM(α, M) holds .
Remark 3.4. If T 2,then the two "bias" terms E[Y ] 2 /T appearing in Corollary 3.5 can be avoided, at least at the price of a multiplicative factor of 2T /(T -1) 4. It suffices to use a slightly more sophisticated online clipping defined as follows. The first round t = 1 is only used to observe Y 1 . Then, the algorithm SeqSEW * τ is run with τ = 1/ d(T -1) from round 2 up to round T with the following important modification: instead of truncating the predictions to [-B t , B t ], which is best suited to the case E[Y ] = 0, we truncate them to the interval

Y 1 -B t , Y 1 + B t , where B t 2 log 2 max 2 s t-1 |Ys-Y 1 | 2 1/2 .
If η t is changed accordingly, i.e., if η t = 1/(8B t 2 ), then it easy to see that the resulting procedure

f T 1 T -1 T s=2 f s (where f 2 , . . . , f T are the regressors output by SeqSEW * τ ) satisfies E f -f T 2 L 2 inf u∈R d f -u • ϕ 2 L 2 + 128 Var[Y ] T -1 + ψ T -1 u 0 ln 1 + √ dT u 1 u 0 + 1 dT d j=1 ϕ j 2 L 2 + 64 Var[Y ] T -1 + ψ T -1 ,
where Var

[Y ] E (Y -E[Y ]) 2 .
Comparing the last bound to that of Corollary 3.5, we note that the two terms E[Y ] 2 /T are absent, and that we loose a multiplicative factor at most of 4 since Var

[Y ] E max 2 t T (Y t -E[Y t ]) 2 (T -1)ψ T -1 so that Var[Y ] T -1 + ψ T -1 2ψ T -1 2 T T -1 ψ T 4ψ T .
Remark 3.5. We mentioned after Corollary 3.5 that each of the four assumptions on ∆Y is fulfilled as soon as both the distribution of f (X) -E[f (X)] and the conditional distribution of ε (conditionally on X) satisfy the same type of assumption. It actually extends to the more general case when the conditional distribution of ε given X is replaced with the distribution of ε itself (without conditioning). This relies on the elementary upper bound

E max 1 t T (∆Y ) 2 t = E max 1 t T f (X t ) -E[f (X)] + ε t 2 2 E max 1 t T f (X t ) -E[f (X)] 2 + 2 E max 1 t T ε 2 t .
From the last inequality, we can also see that assumptions of different nature can be made on f (X) -E[f (X)] and ε, such as the assumptions given in (3.23) or in (3.24).

3.A.4 Proofs of Theorem 3.3 and Corollary 3.7

Proof (of Theorem 3.3): The proof follows the sames lines as in the proof of Theorem 3.2. We thus only sketch the main arguments. In the sequel, we set

σ 2 E ε 2 1 ].
Applying Corollory 3.3 we have, almost surely,

T t=1 Y t -f t (x t ) 2 inf u∈R d T t=1 Y t -u • ϕ(x t ) 2 + 64 max 1 t T Y 2 t u 0 ln 1 + √ dT u 1 u 0 + 1 dT d j=1 T t=1 ϕ 2 j (x t ) + 32 max 1 t T Y 2 t .
Taking the expectations of both sides, expanding the squares

Y t -f t (x t ) 2 and Y t -u • ϕ(x t ) 2 ,
noting that two terms T σ 2 cancel out, and then dividing both sides by T , we get

E 1 T T t=1 f (x t ) -f t (x t ) 2 inf u∈R d 1 T T t=1 f (x t ) -u • ϕ(x t ) 2 + 64 E max 1 t T Y 2 t T u 0 ln 1 + √ dT u 1 u 0 + 1 dT 2 d j=1 T t=1 ϕ 2 j (x t ) + 32 E max 1 t T Y 2 t T .
The right-hand side is exactly the upper bound stated in Theorem 3.3. We thus only need to check that

E 1 T T t=1 f (x t ) -f T (x t ) 2 E 1 T T t=1 f (x t ) -f t (x t ) 2 . (3.42)
This is an equality if the x t are all distinct. In general we get an inequality which follows from the convexity of the square loss. Indeed, by definition of n x , we have, almost surely,

T t=1 f (x t ) -f T (x t ) 2 = x∈{x 1 ,...,x T } 1 t T t:xt=x f (x t ) -f T (x t ) 2 = x∈{x 1 ,...,x T } n x f (x) -f T (x) 2 = x∈{x 1 ,...,x T } n x f (x) - 1 n x 1 t T t:xt=x f t (x) 2 x∈{x 1 ,...,x T } n x 1 n x 1 t T t:xt=x f (x) -f t (x) 2 = T t=1 f (x t ) -f t (x t ) 2 ,
where the second line is by definition of f T and where the last line follows from Jensen's inequality. Dividing both sides by T and taking their expectations, we get (3.42), which concludes the proof.

Proof (of Corollary 3.7): First note that

E max 1 t T Y 2 t E max 1 t T f (x t ) + ε t 2 2 max 1 t T f 2 (x t ) + E max 1 t T ε 2 t .
The proof then follows the exact same lines as for Corollary 3.5 with the sequence (ε t ) instead of the sequence (∆Y ) t .

3.B Tools

3.B.1 Some tools to exploit our PAC-Bayesian inequalities

In this section, we recall two results needed for the derivation of Proposition 3.1 and Proposition 3.2 from the PAC-Bayesian inequalities (3.6) and (3.15). The proofs are due to [DT07, DT08] and we only reproduce 8 them for the convenience of the reader.

For any u * ∈ R d and τ > 0, define ρ u * ,τ as the translated of π τ at u * , namely,

ρ u * ,τ dπ τ du (u -u * ) du = d j=1 (3/τ ) du j 2 1 + |u j -u * j |/τ 4 .
(3.43) Lemma 3.3. For all u * ∈ R d and τ > 0, the probability distribution ρ u * ,τ satisfies

R d T t=1 y t -u • ϕ(x t ) 2 ρ u * ,τ (du) = T t=1 y t -u * • ϕ(x t ) 2 + τ 2 d j=1 T t=1 ϕ 2 j (x t ) .
Lemma 3.4. For all u * ∈ R d and τ > 0, the probability distribution ρ u * ,τ satisfies

K(ρ u * ,τ , π τ ) 4 u * 0 ln 1 + u * 1 u * 0 τ .
Proof (of Lemma 3.3): For all t ∈ {1, . . . , T } we expand the square

y t -u • ϕ(x t ) 2 = y t -u * • ϕ(x t ) + (u * -u) • ϕ(x t )
2 and use the linearity of the integral to get

R d T t=1 y t -u • ϕ(x t ) 2 ρ u * ,τ (du) (3.44) = T t=1 y t -u * • ϕ(x t ) 2 + T t=1 R d (u * -u) • ϕ(x t ) 2 ρ u * ,τ (du) + T t=1 2 y t -u * • ϕ(x t ) R d (u * -u) • ϕ(x t ) ρ u * ,τ (du)

=0

The last sum equals zero by symmetry of ρ u * ,τ around u * , which entails that

R u ρ u * ,τ (du) = u * .
As for the second sum of the right-hand side, it can be bounded from above similarly. Indeed, ex- 8 The notations are however slightly modified because of the change in the statistical setting and goal. The target predictions (f (x1), . . . , f (xT )) are indeed replaced with the observations (y1, . . . , yT ) and the prediction loss f -fu 2 n is replaced with the cumulative loss T t=1 yt -u • ϕ(xt) 2 . Moreover, the analysis of the present proof is slightly simpler since we just need to consider the case L0 = +∞ according to the notations of Theorem 5 in [START_REF] Dalalyan | Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity[END_REF].

panding the inner product and then the square (u * -u) • ϕ(x t ) 2 we have, for all t = 1, . . . , T ,

(u * -u) • ϕ(x t ) 2 = d j=1 (u * j -u j ) 2 ϕ 2 j (x t ) + 1 j =k d (u * j -u j )(u * k -u k ) ϕ j (x t ) ϕ k (x t ) .
By symmetry of ρ u * ,τ around u * and the fact that ρ u * ,τ is a product-distribution, we get Proof (of Lemma 3.4): By definition of ρ u * ,τ and π τ , we have

T t=1 R d (u * -u) • ϕ(x t ) 2 ρ u * ,τ (du) = T t=1 d j=1 ϕ 2 j (x t ) R d (u * j -u j ) 2 ρ u * ,τ (du) + 0 = T t=1 d j=1 ϕ 2 j (x t ) R (u * j -u j ) 2 (3/τ ) du j 2 1 + |u j -u * j |/τ 4 (3.45) =τ 2 T t=1 d j=1 ϕ 2 j (x t ) R 3t 2 dt 2(1 + |t|) 4 (3.46) =τ 2 T t=1 d j=1 ϕ 2 j (x t ) . ( 3
K(ρ u * ,τ , π τ ) R d ln dρ u * ,τ dπ τ (u) ρ u * ,τ (du) = R d   ln d j=1 1 + |u j |/τ 4 1 + |u j -u * j |/τ 4   ρ u * ,τ (du) = 4 R d d j=1 ln 1 + |u j |/τ 1 + |u j -u * j |/τ ρ u * ,τ (du) . (3.48)
But, for all u ∈ R d , by the triangle inequality,

1 + |u j |/τ 1 + |u * j |/τ + |u j -u * j |/τ 1 + |u * j |/τ 1 + |u j -u * j |/τ ,
so that Equation (3.48) yields the upper bound

K(ρ u * ,τ , π τ ) 4 d j=1 ln 1 + |u * j |/τ = 4 j:u * j =0 ln 1 + |u * j |/τ .
We now recall that u * 0 {j : u * j = 0} and apply Jensen's inequality to the concave function

x ∈ (-1, +∞) -→ ln(1 + x) to get

j:u * j =0 ln 1 + |u * j |/τ = u * 0 1 u * 0 j:u * j =0 ln 1 + |u * j |/τ u * 0 ln 1 + j:u * j =0 |u * j | u * 0 τ u * 0 ln 1 + u * 1 u * 0 τ .
This concludes the proof.

3.B.2 Some maximal inequalities

In this section, we prove three maximal inequalities needed for the derivation of Corollaries 3.5 and 3.7 from Theorems 3.2 and 3.3 respectively. Their proofs are quite standard but we provide them for the convenience of the reader.

Lemma 3.5. Let Z 1 , . . . , Z T be T 1 (centered) real random variables such that, for a given constant ν 0, we have ∀t ∈ {1, . . . , T }, ∀λ ∈ R, E e λZt e λ 2 ν/2 . (3.49)

Then,

E max 1 t T Z 2 t 2ν ln(2eT ) .
Lemma 3.6. Let Z 1 , . . . , Z T be T 1 real random variables such that, for some given constants α > 0 and M > 0, we have ∀t ∈ {1, . . . , T }, E e α|Zt| M .

Then,

E max 1 t T Z 2 t ln 2 (M + e)T α 2 .
Lemma 3.7. Let Z 1 , . . . , Z T be T 1 real random variables such that, for some given constants α > 2 and M > 0, we have

∀t ∈ {1, . . . , T }, E |Z t | α M .
Then,

E max 1 t T Z 2 t (M T ) 2/α .
Proof (of Lemma 3.5): Let t ∈ {1, . . . , T }. From the subgaussian assumption (3.49) it is wellknown (see, e.g., [Mas07, Chapter 2]) that for all x 0, we have ∀t ∈ {1, . . . , T } , P |Z t | > x 2e -x 2 /(2ν) .

Let δ ∈ (0, 1). By the change of variables x = 2ν ln(2T /δ), the last inequality entails that, for all t = 1, . . . , T , we have |Z t | 2ν ln(2T /δ) with probability at least 1 -δ/T . Therefore, by a union bound, we get, with probability at least 1 -δ, ∀t ∈ {1, . . . , T } , |Z t | 2ν ln(2T /δ) .

As a consequence, with probability at least 1 -δ,

max 1 t T Z 2 t 2ν ln(2T /δ) 2ν ln(1/δ) + 2ν ln(2T ) .
Integrating the last high-probability bound via Lemma A.7 in Appendix A.6 (cf. Example A.1 with the change of variables z = ln(1/δ)), we get that E max 1 t T Z 2 t 2ν + 2ν ln(2T ), which concludes the proof.

Proof (of Lemma 3.6): We first need the following definitions. Let ψ α : R + → R be a convex majorant of x → e α √ x on R + defined by

ψ α (x) e if x < 1/α 2 , e α √ x if x 1/α 2 .
We associate with ψ α its generalized inverse ψ -1 α : R → R + defined by

ψ -1 α (y) = 1/α 2 if y < e , ( ln
y) 2 /α 2 if y e .
Elementary manipulations show that:

• ψ α is nondecreasing and convex on R + ;

• ψ -1 α is nondecreasing on R;

• x ψ -1 α ψ α (x) for all x ∈ R + .
The proof is based on a Pisier-type argument as is done, e.g., in [Mas07, Lemma 2.3] to prove the maximal inequality E[max 1 t T ξ t ] √ 2ν ln T for all subgaussian real random variables ξ t , 1 t T , with common variance factor ν 0 (see Lemma A.3 in Appendix A.5).

From the inequality x ψ -1 α ψ α (x) for all x ∈ R + we have

E max 1 t T Z 2 t ψ -1 α ψ α E max 1 t T Z 2 t ψ -1 α E ψ α max 1 t T Z 2 t = ψ -1 α E max 1 t T ψ α Z 2 t ,
where the last two inequalities follow by Jensen's inequality (since ψ α is convex) and the fact that both ψ -1 α and ψ α are nondecreasing.

Since ψ α 0 and ψ -1 α is nondecreasing we get

E max 1 t T Z 2 t ψ -1 α E T t=1 ψ α Z 2 t = ψ -1 α T t=1 E ψ α Z 2 t ψ -1 α T t=1 E e α|Zt| + e ψ -1 α M T + eT = ln 2 M T + eT α 2 ,
where the second line follows from the inequality ψ α (x) e + e α

√

x for all x ∈ R + , and where the last line follows from the bounded exponential moment assumption and the definition of ψ -1 α . It concludes the proof.

Proof (of Lemma 3.7): As in the previous proof, we have, by Jensen's inequality and the fact that x → x α/2 is convex and nondecreasing on R + (since α > 2), E max

1 t T Z 2 t E max 1 t T Z 2 t α/2 2/α = E max 1 t T Z t α 2/α E T t=1 Z t α 2/α (M T ) 2/α
by the bounded moment assumption, which concludes the proof.

Chapter 4

Adaptive and optimal online linear regression on 1 -balls

We consider the problem of online linear regression on individual sequences. The goal in this paper is for the forecaster to output sequential predictions which are, after T time rounds, almost as good as the ones output by the best linear predictor in a given 1 -ball in R d . We consider both the cases where the dimension d is small and large relative to the time horizon T . We first present regret bounds with optimal dependencies on d, T , and on the sizes U , X and Y of the 1 -ball, the input data and the observations. The minimax regret is shown to exhibit a regime transition around the point d = √ T U X/(2Y ). Furthermore, we present efficient algorithms that are adaptive, i.e., that do not require the knowledge of U , X, Y , and T , but still achieve nearly optimal regret bounds.

NOTA: This chapter is the full version of a conference paper [START_REF] Gerchinovitz | Adaptive and optimal online linear regression on 1 -balls[END_REF] to be presented at ALT 2011. Some improved bounds are published here for the first time (Theorem 4.3 and Remark 4.1).

Contents

Introduction

In this chapter, we consider the problem of online linear regression against arbitrary sequences of input data and observations, with the objective of being competitive with respect to the best linear predictor in an1 -ball of arbitrary radius. This extends the task of convex aggregation. We consider both low-and high-dimensional input data. Indeed, in a large number of contemporary problems, the available data can be high-dimensional-the dimension of each data point is larger than the number of data points. Examples include analysis of DNA sequences, collaborative filtering, astronomical data analysis, and cross-country growth regression. In such high-dimensional problems, performing linear regression on an 1 -ball of small diameter may be helpful if the best linear predictor is sparse. Our goal is, in both low and high dimensions, to provide online linear regression algorithms along with bounds on 1 -balls that characterize their robustness to worst-case scenarios.

Setting

We consider the online version of linear regression, which unfolds as follows (see also Section 2.4 for an introduction to this setting). First, the environment chooses a sequence of observations (y t ) t 1 in R and a sequence of input vectors (x t ) t 1 in R d , both initially hidden from the forecaster. At each time instant t ∈ N * = {1, 2, . . .}, the environment reveals the data x t ∈ R d ; the forecaster then gives a prediction y t ∈ R; the environment in turn reveals the observation y t ∈ R; and finally, the forecaster incurs the square loss (y t -y t ) 2 . The dimension d can be either small or large relative to the number T of time steps: we consider both cases.

In the sequel, u • v denotes the standard inner product between u, v ∈ R d , and we set u ∞ max 1 j d |u j | and u 1 d j=1 |u j |. The 1 -ball of radius U > 0 is the following bounded subset of R d :

B 1 (U ) u ∈ R d : u 1 U .
Given a fixed radius U > 0 and a time horizon T 1, the goal of the forecaster is to predict almost as well as the best linear forecaster in the reference set

x ∈ R d → u • x ∈ R : u ∈ B 1 (U ) , i.e.,
to minimize the regret on B 1 (U ) defined by

T t=1 (y t -y t ) 2 -min u∈B 1 (U ) T t=1 (y t -u • x t ) 2 .
We shall present algorithms along with bounds on their regret that hold uniformly over all sequences 1 (x t , y t ) 1 t T such that x t ∞ X and |y t | Y for all t = 1, . . . , T , where X, Y > 0. These regret bounds depend on four important quantities: U , X, Y , and T , which may be known or unknown to the forecaster.

Contributions and related works

Next we detail the main contributions of this chapter in view of related works in online linear regression.

Our first contribution consists of a minimax analysis of online linear regression on 1 -balls in the arbitrary sequence setting. We first provide a refined regret bound expressed in terms of Y , d, and a quantity κ = √ T U X/(2dY ). This quantity κ is used to distinguish two regimes: we show a distinctive regime transition2 at κ = 1 or d =

√

T U X/(2Y ). Namely, for κ < 1, the regret is of the order of √ T , whereas it is of the order of ln T for κ > 1.

The derivation of this regret bound partially relies on a Maurey-type argument used under various forms with i.i.d. data, e.g., in [Nem00, Tsy03, BN08, SSSZ10] (see also [START_REF] Yang | Aggregating regression procedures to improve performance[END_REF]). We adapt it in a straightforward way to the deterministic setting. Therefore, this is yet another technique that can be applied to both the stochastic and individual sequence settings.

Unsurprisingly, the refined regret bound mentioned above matches the optimal risk bounds for stochastic settings3 [BM01a, Tsy03] (see also [START_REF] Raskutti | Minimax rates of estimation for high-dimensional linear regression over q -balls[END_REF]). Hence, linear regression is just as hard in the stochastic setting as in the arbitrary sequence setting. Using the standard online to batch conversion, we make the latter statement more precise by establishing a lower bound for all κ at least of the order of √ ln d/d. This lower bound extends those of [CB99, KW97], which only hold for small κ of the order of 1/d.

The algorithm achieving our minimax regret bound is both computationally inefficient and non-adaptive (i.e., it requires prior knowledge of the quantities U , X, Y , and T that may be unknown in practice). Those two issues were first overcome by [START_REF] Auer | Adaptive and self-confident on-line learning algorithms[END_REF] via an automatic tuning termed self-confident (since the forecaster somehow trusts himself in tuning its parameters). They indeed proved that the self-confident p-norm algorithm with p = 2 ln d and tuned with U has a cumulative loss L T = T t=1 (y t -y t ) 2 bounded by

L T L * T + 8U X (e ln d) L * T + (32e ln d) U 2 X 2 8U XY √ eT ln d + (32e ln d) U 2 X 2 , where L * T min {u∈R d : u 1 U } T t=1 (y t -u • x t ) 2 T Y 2 .
This algorithm is efficient, and our lower bound in terms of κ shows that it is optimal up to logarithmic factors in the regime κ 1 without prior knowledge of X, Y , and T . In Section 4.3, we study the adaptivity possibilities of a closely related forecaster due to [START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF] and called the EG ± algorithm. A detailed presentation of this forecaster can be found in Section 2.4.3 (Chapter 2). As proved in [KW97, Theorem 5.11], when tuned as a function of U , X, and a known upper bound B on L * T , this algorithm has a regret bounded from above by 2U X 2B ln(2d) + 2U 2 X 2 ln(2d). Again, this algorithm is efficient and nearly optimal in the regime κ 1. However, the EG ± algorithm requires prior knowledge of U , X, and B -or, alternatively, U , X, Y , and T .

Our second contribution -already detailed in Chapter 2 -is a generic version of the EG ± algorithm for general convex loss functions. When applied to the square loss and combined with the variance-based tuning of [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF], the corresponding adaptive EG ± algorithm satisfies a regret bound comparable to that of the self-confident p-norm algorithms (Corollary 2.2 in Section 2.4.3). In particular this algorithm adapts automatically to X, Y , and T when U is known.

Our third contribution is a generic technique called loss Lipschitzification. It transforms the loss functions u → (y t -u • x t ) 2 (or u → y t -u • x t α if the predictions are scored with the α-loss, α 2) into Lipschitz continuous functions. We illustrate this technique by applying the generic adaptive EG ± algorithm to the modified loss functions. When the predictions are scored with the square loss, this yields an algorithm (the LEG algorithm) whose main regret term can only improve on that derived for the adaptive EG ± algorithm without Lipschtizification. The benefits of this technique are clearer for loss functions with higher curvature: if α > 2, the resulting regret bound roughly grows as U instead of a naive U α/2 . Finally, we provide a simple way to achieve minimax regret uniformly over all 1 -balls B 1 (U ) for U > 0. This method aggregates instances of an algorithm that require prior knowledge of U . For the sake of simplicity, we assume that X, Y , and T are known, but explain in the discussions how to extend the method to a fully adaptive algorithm that requires the knowledge neither of U , X, Y , nor T . This chapter is organized as follows. In Section 4.2, we establish our refined upper and lower bounds in terms of the intrinsic quantity κ. In Section 4.3, we present an efficient and adaptive algorithm -the adaptive EG ± algorithm with or without loss Lipschitzfication -that achieves the optimal regret on B 1 (U ) when U is known. In Section 4.4, we use an aggregating strategy to achieve an optimal regret uniformly over all 1 -balls B 1 (U ), for U > 0, when X, Y , and T are known. Finally, in Section 4.5, we discuss as an extension a fully automatic algorithm that requires no prior knowledge of U , X, Y , or T . Some proofs and additional tools are postponed to the appendix.

Optimal rates

In this section, we first present a refined upper bound on the minimax regret on B 1 (U ) for an arbitrary U > 0. In Corollary 4.1, we express this upper bound in terms of an intrinsic quantity κ √ T U X/(2dY ). The optimality of the latter bound is shown in Section 4.2.2.

We first consider the following definition to avoid any ambiguity. We call online forecaster any sequence F = ( f t ) t 1 of functions such that f t : R d × (R d × R) t-1 → R maps at time t the new input x t and the past data (x s , y s ) 1 s t-1 to a prediction f t x t ; (x s , y s ) 1 s t-1 . Depending on the context, the latter prediction may be simply denoted by f t x t ) or by y t .

Upper bound

Theorem 4.1 (Upper bound). Let d, T ∈ N * , and U, X, Y > 0. The minimax regret on B 1 (U ) for bounded base predictions and observations satisfies inf

F sup xt ∞ X, |yt| Y T t=1 (y t -y t ) 2 -inf u 1 U T t=1 (y t -u • x t ) 2            3U XY 2T ln(2d) if U < Y X ln(1+2d) T ln 2 , 26 U XY T ln 1 + 2dY √ T U X if Y X ln(1+2d) T ln 2 U 2dY √ T X , 32 dY 2 ln 1 + √ T U X dY + dY 2 if U > 2dY X √ T ,
where the infimum is taken over all forecasters F and where the supremum extends over all sequences (x t , y t )

1 t T ∈ (R d × R) T such that |y 1 |, . . . , |y T | Y and x 1 ∞ , . . . , x T ∞ X.
Theorem 4.1 improves the bound of [KW97, Theorem 5.11] for the EG ± algorithm. First, our bound depends logarithmically-as opposed to linearly-on U for U > 2dY /( √ T X). Secondly, it is smaller by a factor ranging from 1 to

√ ln d when Y X ln(1 + 2d) T ln 2 U 2dY √ T X . (4.1)
Hence, Theorem 4.1 provides a partial answer to a question4 raised in [START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF] about the gap of ln(2d) between the upper and lower bounds.

) reads: inf F sup xt ∞ X, |yt| Y T t=1 (y t -y t ) 2 -inf u 1 U T t=1 (y t -u • x t ) 2        6 dY 2 κ 2 ln(2d) if κ < √ ln(1+2d) 2d √ ln 2 , 52 dY 2 κ ln(1 + 1/κ) if √ ln(1+2d) 2d √ ln 2 κ 1 , 32 dY 2 ln(1 + 2κ) + 1 if κ > 1 .
The upper bound of Corollary 4.1 is shown in The constant c is chosen to ensure continuity at κ = 1, and ω 32(ln(3) + 1) -1 . We define:

κ min = ln(1 + 2d)/(2d √ ln 2
) and κ max = (e (T /d-1)/c -1)/2.

We now prove Theorem 4.1. The main part of the proof relies on a Maurey-type argument. Although this argument was used in the stochastic setting [Nem00, Tsy03, BN08, SSSZ10], we adapt it to the deterministic setting. This is yet another technique that can be applied to both the stochastic and individual sequence settings. T X . We use a Maurey-type argument to refine the regret bound (4.2). This technique was used under various forms in the stochastic setting, e.g., in [Nem00, Tsy03, BN08, SSSZ10]. It consists of discretizing B 1 (U ) and looking at a random point in this discretization to study its approximation properties. We also use clipping to get a regret bound growing as U instead of a naive U 2 .

Proof (of

More precisely, we first use the fact that to be competitive against B 1 (U ), it is sufficient to be competitive against its finite subset 5 As proved in Lemma 4.4, the regret bound (4.2) is achieved either by the EG ± algorithm, the algorithm SeqSEW B,η τ of Chapter 3 (we could also get a slightly worse bound with the sequential ridge regression forecaster), or the trivial null forecaster.

B U,m    k 1 U m , . . . , k d U m : (k 1 , . . . , k d ) ∈ Z d , d j=1 |k j | m    ⊂ B 1 (U ) ,
where m α with α U X Y T (ln 2)/ ln 1 + 2dY √ T U X .

By Lemma 4.6 in appendix, and since m > 0 (see below), we indeed have

inf u∈ B U,m T t=1 (y t -u • x t ) 2 inf u∈B 1 (U ) T t=1 (y t -u • x t ) 2 + T U 2 X 2 m inf u∈B 1 (U ) T t=1 (y t -u • x t ) 2 + 2 √ ln 2 U XY T ln 1 + 2dY √ T U X , (4.3)
where (4.3) follows from m α α/2 since α 1 (in particular, m > 0 as stated above).

To Therefore it only remains to exhibit an algorithm which is competitive against B U,m at an aggregation price of the same order as the last term in (4.3). This is the case for the standard exponentially weighted average forecaster applied to the clipped predictions

u • x t Y min Y, max -Y, u • x t , u ∈ B U,m ,
and tuned with the inverse temperature parameter η = 1/(8Y 2 ). More formally, this algorithm predicts at each time t = 1, . . . , T as

y t u∈ B U,m p t (u) u • x t Y ,
where p 1 (u) 1/ B U,m (denoting by B U,m the cardinality of the set B U,m ), and where the weights p t (u) are defined for all t = 2, . . . , T and u ∈ B U,m by

p t (u) exp -η t-1 s=1 y s -[u • x s ] Y 2 v∈ B U,m exp -η t-1 s=1 y s -[v • x s ] Y 2 .
By Lemma 4.5 in appendix, the above forecaster tuned with η = 1/(8Y 2 ) satisfies T t=1

(y t -y t ) 2 -inf u∈ B U,m T t=1 (y t -u • x t ) 2 8Y 2 ln B U,m 8Y 2 ln e(2d + m) m m (4.4) = 8Y 2 m 1 + ln(1 + 2d/m) 8Y 2 α 1 + ln(1 + 2d/α) (4.5) = 8Y 2 α + 8Y 2 α ln   1 + 2dY √ T U X ln 1 + 2dY /( √ T U X) ln 2   8Y 2 α + 16Y 2 α ln 1 + 2dY √ T U X (4.6) 8 √ ln 2 + 16 √ ln 2 U XY T ln 1 + 2dY √ T U X . ( 4

Lower bound

Corollary 4.1 gives an upper bound on the regret in terms of the quantities d, Y , and κ √ T U X/(2dY ). We now show that for all d ∈ N * , Y > 0, and κ ln(1 + 2d)/(2d √ ln 2), the upper bound can not be improved6 up to logarithmic factors.

(y t -y t ) 2 -inf u 1 U T t=1 (y t -u • x t ) 2      c 1 ln 2+16d 2 dY 2 κ ln (1 + 1/κ) if √ ln(1+2d) 2d √ ln 2 κ 1 , c 2 ln 2+16d 2 dY 2 if κ > 1 ,
where c 1 , c 2 > 0 are absolute constants. The infimum is taken over all forecasters F and the supremum extends over all sequences (x t , y t )

1 t T ∈ (R d × R) T such that |y 1 |, . . . , |y T | Y and x 1 ∞ , . . . , x T ∞ X.
The above lower bound extends those of [CB99, KW97], which hold for small κ of the order of 1/d. The proof is postponed to Appendix 4.A.1. We perform a reduction to the stochastic batch setting-via the standard online to batch conversion, and employ a version of a lower bound of [START_REF] Tsybakov | Optimal rates of aggregation[END_REF].

Adaptation to unknown X, Y and T via exponential weights

Although the proof of Theorem 4.1 already gives an algorithm that achieves the minimax regret, the latter takes as inputs U , X, Y , and T , and it is inefficient in high dimensions. In this section, we present a new method that achieves the minimax regret both efficiently and without prior knowledge of X, Y , and T provided that U is known. Adaptation to an unknown U is considered in Section 4.4. Our method consists of modifying an underlying linear regression algorithm such as the EG ± algorithm [START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF] or the sequential ridge regression [START_REF] Vovk | Competitive on-line statistics[END_REF][START_REF] Azoury | Relative loss bounds for on-line density estimation with the exponential family of distributions[END_REF]. Next, we show that automatically tuned variants of the EG ± algorithm -the first of which was introduced in Section 2.4.3 -nearly achieve the minimax regret for the regime d

√

T U X/(2Y ). A similar modification could be applied to the ridge regression forecaster to achieve a nearly optimal regret bound of order dY 2 ln 1 + d

√ T U X dY 2 in the regime d < √ T U X/(2Y )
. The latter analysis is more technical and hence is omitted.

An adaptive EG ± algorithm

The second algorithm of the proof of Theorem 4.1 is computationally inefficient because it aggregates approximately d √ T experts. In contrast, the EG ± algorithm has a manageable computational complexity that is linear in d. In Section 2.4.3 of Chapter 2 we introduced a version of the EG ± algorithm -called the adaptive EG ± algorithm -that does not require prior knowledge of X, Y and T (as opposed to the original EG ± algorithm of [START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF]). This version uses the automatic tuning of [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF]. As proved in Corollary 2.2 of Chapter 2 -a consequence of [CBMS07, Corollary 1] -the adaptive EG ± algorithm on B 1 (U ) defined in Figure 2.5 with t (u) = (y t -u • x t ) 2 satisfies, for all choices of (x 1 , y 1 ), . . . ,

(x T , y T ) ∈ R d × R, T t=1 (y t -u t • x t ) 2 L * T + 8U X L * T ln(2d) + 137 ln(2d) + 24 U XY + U 2 X 2 , (4.8)
where the quantities

L * T min {u∈B 1 (U )} T t=1 (y t -u • x t ) 2 , X max 1 t T x t ∞ ,
(y t -u t • x t ) 2 -min u∈B 1 (U ) T t=1 (y t -u • x t ) 2 8U XY T ln(2d) + 137 ln(2d) + 24 U XY + U 2 X 2 .
Therefore, our version of the EG ± algorithm is efficient and adaptive in X, Y , and T . It achieves approximately the regret bound of Theorem 4.1 in the regime κ 1, i.e., d

√

T U X/(2Y ).

Another way to perform the adaptation to X, Y , and T in an efficient way is provided by the self-confident p-norm algorithm of [START_REF] Auer | Adaptive and self-confident on-line learning algorithms[END_REF] with p = 2 ln d. As commented on after the statement of Corollary 2.2 in Chapter 2, this algorithm satisfies an improvement for small losses similar to (4.8). The fact that we got a similar bound is not surprising because the p-norm algorithms are known to share many properties with the EG ± algorithm (in the limit p → +∞ with an appropriate initial weight vector, or for p of the order of ln d with a zero initial weight vector, cf. [START_REF] Gentile | The robustness of the p-norm algorithms[END_REF]). The bound of Corollary 2.2 corroborates this similarity.

In the next subsections, we use yet another instance of the adaptive EG ± algorithm that we call the Lipschitzifying Exponentiated Gradient (LEG) algorithm. It corresponds to the adaptive EG ± algorithm applied not to the square loss but to a Lipschitz continuous modification t : R d → R of the square loss.

Lipschitzification of the loss function

Our key technique consists of transforming the loss functions u → (y t -u • x t ) 2 into functions t that are Lipschitz continuous with respect to • 1 . Afterward, adaptation to the unknown Lipschitz constants ∇ t ∞ is carried out using the techniques of [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF].

We point out that our Lipschitzification method can be applied to other convex loss functions with higher curvature, see Remark 4.1 later. Moreover, this technique is not specific to the pair of dual norms ( • 1 , • ∞ ) and to the EG ± algorithm; it could be used with other pairs ( • q , • p ) (with 1/p + 1/q = 1) and other gradient-based algorithms, such as the p-norm algorithm [Gen03, ACBG02] and its regularized variants (SMIDAS and COMID) [START_REF] Shalev-Shwartz | Stochastic methods for 1 -regularized loss minimization[END_REF][START_REF] Duchi | Composite objective mirror descent[END_REF].

The Lipschitzification proceeds as follows. At each time t 1, using adaptivity-oriented ideas from Chapter 3, we set

B t 2 log 2 (max 1 s t-1 y 2 s ) 1/2
, so that B t satisfies |y s | B t for all s = 1, . . . , t -1. The modified (or Lipschitzified) loss function t : R d → R is constructed as follows:

• if |y t | > B t , then t (u) 0 for all u ∈ R d ;
• if |y t | B t , then t is the convex function that coincides with the square loss when u•x t B t and is linear elsewhere. This function is shown in Figure 4.2 and can be formally defined as

t (u)      (y t -u • x t ) 2 if u • x t B t , (y t -B t ) 2 + 2(B t -y t )(u • x t -B t ) if u • x t > B t , (y t + B t ) 2 + 2(-B t -y t )(u • x t + B t ) if u • x t < -B t .
Observe

∇ t ∞ 2 y t -[u • x t ] Bt x t ∞
(4.10)

2 |y t | + B t x t ∞ 2 1 + √ 2 x t ∞ max 1 s t |y s | , (4.11)
where we used the fact that B t √ 2 max 1 s t-1 |y s |. We can also glean from Figure 4.2 that, when |y t | B t , the modified loss function t : R d → R lies in between the square loss and its clipped version:

∀u ∈ R d , y t -[u • x t ] Bt 2 t (u) y t -u • x t 2 . ( 4

Lipschitzifying Exponentiated Gradient algorithm

In this section we illustrate the Lipschitzification technique described above with the adaptive EG ± algorithm. We denote by (e j ) 1 j d the canonical basis of R d and by ∇ j the j-th component of the gradient.

Parameter: radius U > 0.

Initialization:

B 1 0, p 1 = (p + 1,1 , p - 1,1 , . . . , p + d,1 , p - d,
exp -η t+1 t s=1 γU ∇ j s ( u s ) 1 k K µ∈{+,-} exp -η t+1 t s=1 µU ∇ k s ( u s ) . 6. Update the threshold B t+1 2 log 2 (max 1 s t y 2 s ) 1/2
.

a For all γ ∈ {+, -}, by a slight abuse of notation, γU denotes U or -U if γ = + or γ =respectively. It is yet another instance of the adaptive EG ± algorithm on B 1 (U ) (cf. Figure 2.5 of Section 2.4.3) applied not to the square loss but to the Lipschitzified loss functions t , t 1. In particular the LEG algorithm uses as a blackbox the exponentially weighted majority forecaster of [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF] on 2d experts-namely, the vertices ±U e j of B 1 (U )-as in [START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF]. It adapts to the unknown Lipschitz constants ∇ t ∞ by the particular choice of η t due to [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF] and defined for all t 2 by

η t = min 1 E t-1 , C ln K V t-1 , (4.13)
where C 2( √ 2 -1)/(e -2) and where we set, for all t = 1, . . . , T ,

z + j,s U ∇ j s ( u s ) and z - j,s -U ∇ j s ( u s ) , j = 1, . . . , d, s = 1, . . . , t , E t inf k∈Z    2 k : 2 k max 1 s t max 1 j,k d γ,µ∈{+,-} z γ j,s -z µ k,s    , V t t s=1 1 j d γ∈{+,-} p γ j,s     z γ j,s - 1 k d µ∈{+,-} p µ k,s z µ k,s     2 .
Note that E t-1 approximates the range of the z γ j,s up to time t -1, while V t-1 is the corresponding cumulative variance of the forecaster.

The next theorem bounds the regret of the LEG algorithm on B 1 (U ). As with the square loss, this algorithm is efficient and adaptive in X, Y , and T ; it achieves approximately the regret bound of Theorem 4.1 in the regime κ 1, i.e., d √ T U X/(2Y ). The proof is postponed to Appendix 4.A.2. It follows from the bound on the adaptive EG ± algorithm for general convex loss functions that we proved in Corollary 2.1 (Section 2.4.3).

Theorem 4.3. Let U > 0. Then, the Lipschitzifying Exponentiated Gradient algorithm tuned with U satisfies, for all T 1 and all individual sequences (x 1 , y 1 ), . . . ,

(x T , y T ) ∈ R d × R, T t=1 (y t -y t ) 2 inf u 1 U T t=1 t (u) + 8U X inf u 1 U T t=1 t (u) ln(2d) + 153 ln(2d) + 58 U XY + U 2 X 2 + 12Y 2 ,
where none of the three quantities inf {u∈R d : u 1 U } T t=1 t (u), X max 1 t T x t ∞ , and Y max 1 t T |y t | is known to the forecaster.

The first two terms of the bound of Theorem 4.3 slightly improve on those obtained without Lipschitzification (cf. (4.8)) since we always have

inf u 1 U T t=1 t (u) inf u∈R d : u 1 U T t=1 (y t -u • x t ) 2 , (4.14)
where we used the key property t (u) (y t -u • x t ) 2 for all u ∈ R d and all t = 1, . . . , T (by (4.12) if |y t | B t , obvious otherwise). Though the improvement in the regret bound entailed by (4.14) is usually only of minor importance, Lipschitzification can be useful in at least two ways.

Remark 4.1 (Application to other convex loss functions with higher curvature). Lipschitzification can be used in a much more general setting than the one studied in this paper, i.e., when the loss functions are of the form x → f |y t -x| for an increasing twice differentiable function f : R + → R such that f 0. Assume, e.g., that f (x) = x α for all x 0 and some α 2. As explained below, the benefits of Lipschitzification become clear when α > 2: it yields a regret bound that depends linearly in U , instead of the rate U α/2 that would follow from a similar analysis for the adaptive EG ± algorithm without loss Lipschitzification.

Next we assume that for some α 2, the predictions y t of the forecaster and the base predictions u • x t are scored with the loss functions x → |y t -x| α , t 1. Correspondingly, we set t (u) = |y t -u • x t | α for all u ∈ R d . The Lipschtizification step of Section 4.3.2 can easily be extended to this case. The main two changes consist of the following:

• The adaptive clipping level is defined by B t 2 log 2 (max 1 s t-1 |ys| α ) 1/α .

• At every round t such that |y t | B t , the Lipschitzified loss function t : R d → R is defined by

t (u)      y t -u • x t α if u • x t B t , y t -B t α + α y t -B t α-1 (u • x t -B t ) if u • x t > B t , y t + B t α -α y t + B t α-1 (u • x t + B t ) if u • x t < -B t .
Consider the adaptive EG ± algorithm of Section 2.4.3 applied to the Lipschitzified loss functions t . To analyse its performance, it suffices to follow the same lines as in the proof of Theorem 4.3. Again, a key property is that7

∇ t ( u t ) = -α sgn y t -[ u t •x t ] Bt y t -[ u t •x t ] Bt α-1 x t =
-α sgn y t -y t y t -y t α-1 x t . This entails that

∇ t ( u t ) 2 ∞ α 2 X 2 y t -y t 2α-2 = α 2 X 2 y t -y t α-2 y t -y t α α 2 X 2 (B t+1 + B t ) α-2 y t -y t α α 2 X 2 (1 + 2 1/α )Y α-2 y t -y t α .
Then, following the same lines as in the proof of Theorem 4.3, we can see that the adaptive EG ± algorithm applied to ( t ) t 1 has a cumulative loss T t=1 y t -y t α at most of

L * T + c 1 (α)U XY α 2 -1 L * T ln(2d) + c 2 (α) U XY α-1 + U 2 X 2 Y α-2 ln(2d) , where L * T inf {u∈R d : u 1 U } T t=1 t (u)
T Y α and where c 1 (α), c 2 (α) > 0 are constants depending only on α (e.g., c 1 (α) = 4α 1 + 2 1/α α/2-1 ). This bound improves on the bound we would have obtained via the same analysis for the adaptive EG ± algorithm applied to the original losses

t (u) = |y t -u • x t | α : L * T + c 3 (α)U X(Y + U X) α 2 -1 L * T ln(2d) + c 4 (α) αU X(Y + U X) α-1 + α 2 U 2 X 2 (Y + U X) α-2 ln(2d) , where we set L * T inf {u∈R d : u 1 U } T t=1 |y t -u • x t | α ,
and where c 3 (α), c 4 (α) > 0 are constants depending only on α. The main difference between the two regret bounds above lies in the dependence in U : our main regret term scales as U XY α/2-1 while the one obtained without Lipschitzification scales as U X(Y + U X) α/2-1 . The first term grows linearly in U while the second grows as U α/2 , hence a clear improvement for α > 2.

Remark 4.2 (A simpler analysis for the minimax regret).

Another benefit of Lipschitzification is that all online convex optimization regret bounds expressed in terms of the maximal dual norm of the gradients -i.e., max 1 t T ∇ t ∞ in our case -can be used fruitfully with the Lipschitzified loss functions t . For instance, using the very simple bound (2.13) of Theorem 2.4 (a consequence of Corollary [CBMS07, Corollary 1], see Section 2.2.2), we can prove that

T t=1 (y t -y t ) 2 -inf u 1 U T t=1 (y t -u • x t ) 2 c 1 U XY T ln(2d) + 8 ln(2d) + c 2 Y 2 ,
(y t -y t ) 2 -inf u 1 U T t=1 (y t -u • x t ) 2 T t=1 t ( u t ) -inf u 1 U T t=1 t (u) + c 2 Y 2 4U max 1 t T ∇ t ∞ T ln(2d) + 2 ln(2d) + 3 + c 2 Y 2 (4.

Adaptation to unknown U

In the previous section, the forecaster is given a radius U > 0 and asked to ensure a low worst-case regret on the 1 -ball B 1 (U ). In this section, U is no longer given: the forecaster is asked to be competitive against all balls B 1 (U ), for U > 0. Namely, its worst-case regret on each B 1 (U ) should be almost as good as if U were known beforehand. For simplicity, we assume that X, Y , and T are known: we discuss in Section 4.5 how to simultaneously adapt to all parameters. We define

R log 2 (2T /c) + and U r Y X 2 r
T ln(2d) , for r = 0, . . . , R , (4.17)

where c > 0 is a known absolute constant and

x + min k ∈ N : k x for all x ∈ R .
The Scaling algorithm of Figure 4.4 works as follows. We have access to a sub-algorithm A(U )

Parameters: X, Y, η > 0, T 1, and c > 0 (a constant).

Initialization: R = log 2 (2T /c) + , w 1 = 1/(R + 1) ∈ R R+1 . For time steps t = 1, . . . , T :

1. For experts r = 0, . . . , R:

• Run the sub-algorithm A(U r ) on the ball B 1 (U r ) and obtain the prediction y (r) t .

Output the prediction y

t = R r=0 w (r) t R r =0 w (r ) t y (r) t Y . 3. Update w (r) t+1 = w (r) t exp -η y t -y (r) t Y
2 for r = 0, . . . , R. which we run simultaneously for all U = U r , r = 0, . . . , R. Each instance of the sub-algorithm A(U r ) performs online linear regression on the 1 -ball B 1 (U r ). We employ an exponentially weighted forecaster to aggregate these R + 1 sub-algorithms to perform online linear regression simultaneously on the balls B 1 (U 0 ), . . . , B 1 (U R ). The following regret bound follows by expconcavity of the square loss.

Theorem 4.4. Suppose that X, Y > 0 are known. Let c, c > 0 be two absolute constants. Suppose that for all U > 0, we have access to a sub-algorithm A(U ) with regret against B 1 (U ) of at most cU XY T ln(2d) + c Y 2 for T T 0 , (4.18)

uniformly over all sequences (x t ) and (y t ) bounded by X and Y . Then, for a known T T 0 , the Scaling algorithm with η = 1/(8Y 2 ) satisfies T t=1

(y t -y t ) 2 inf u∈R d T t=1 (y t -u • x t ) 2 + 2c u 1 XY T ln(2d) + 8Y 2 ln log 2 (2T /c) + + 1 + (c + c )Y 2 . (4.19)
In particular, for every U > 0, where the last inequality follows by assumption (4.18), and where we set

T t=1 (y t -y t ) 2 inf u∈B 1 (U ) T t=1 (y t -u • x t ) 2 + 2cU XY T ln(2d) + 8Y 2 ln log 2 (2T /c) + + 1 + (c + c )Y 2 .
z 8Y 2 ln(R + 1) + c Y 2 . Let u * T ∈ arg min u∈R d T t=1 (y t -u • x t ) 2 + 2c u 1 XY T ln(2d)
. Next, we proceed by considering three cases:

U 0 < u * T 1 < U R , u * T 1 U 0 , and u * T 1 U R . Case 1: U 0 < u * T 1 < U R . Let r * min r = 0, . . . , R : U r u * T 1 . Note that r * 1 since u * T 1 > U 0 . By (4.20) we have T t=1 (y t -y t ) 2 inf u∈B 1 (U r * ) T t=1 (y t -u • x t ) 2 + cU r * XY T ln(2d) + z T t=1 (y t -u * T • x t ) 2 + 2c u * T 1 XY T ln(2d) + z ,
where the last inequality follows from u * T ∈ B 1 (U

y t ∈ [-Y, Y ], so that T t=1 (y t -y t ) 2 4Y 2 T T t=1 (y t -u * T • x t ) 2 + 2cU R XY T ln(2d) T t=1 (y t -u * T • x t ) 2 + 2c u * T 1 XY T ln(2d) ,
where the second inequality follows by 2cU R XY T ln(2d) = 2cY 2 2 R 4Y 2 T (since 2 R 2T /c by definition of R), and the last inequality uses the assumption u * T 1 U R . We finally get (4.19) by definition of u * T . This concludes the proof of the first claim (4.19). The second claim follows by bounding u 1 U .

Extension to a fully adaptive algorithm and other discussions

The Scaling algorithm of Section 4.4 uses prior knowledge of Y , Y /X, and T . In order to obtain a fully automatic algorithm, we need to adapt efficiently to these quantities. Adaptation to Y is possible via a technique already used for the LEG algorithm, i.e., by updating the clipping range B t based on the past observations |y s |, s t -1.

In parallel to adapting to Y , adaptation to Y /X can be carried out as follows. We replace the exponential sequence {U 0 , . . . , U R } by another exponential sequence {U 0 , . . . , U R }:

U r 1 T k 2 r
T ln(2d) , r = 0, . . . , R , (4.22)

where R R + log 2 T 2k = log 2 (2T /c) + + log 2 T 2k , and where k > 1 is a fixed constant. On the one hand, for T T 0 max (X/Y ) 1/k , (Y /X) 1/k , we have (cf. (4.17) and (4.22)),

[U 0 , U R ] ⊂ [U 0 , U R ] .
Therefore, the analysis of Theorem 4.4 applied to the grid {U 0 , . . . , U R } yields8 a regret bound of the order of U XY

√

T ln d + Y 2 ln(R + 1). On the other hand, clipping the predictions to [-Y, Y ] ensures the crude regret bound 4Y 2 T 0 for small T < T 0 . Hence, the overall regret for all T 1 is of the order of

U XY √ T ln d + Y 2 ln(k ln T ) + Y 2 max (X/Y ) 1/k , (Y /X) 1/k .
Adaptation to an unknown time horizon T can be carried out via a standard doubling trick on T . However, to avoid restarting the algorithm repeatedly, we can use a time-varying exponential sequence {U -R (t) (t), . . . , U R (t) (t)} where R (t) grows at the rate of k ln(t). This gives9 us an algorithm that is fully automatic in the parameters U , X, Y and T . In this case, we can show that the regret is of the order of

U XY √ T ln d + Y 2 k ln(T ) + Y 2 max √ T X/Y 1/k , Y /( √ T X) 1/k ,
where the last two terms are negligible when T → +∞ (since k > 1).

Next we discuss another possible improvement. There is a logarithmic gap between the upper bound of Theorem 4.1 and the lower bound of Theorem 4.2. This gap comes from a concentration argument on a specific sequence of (unbounded) normal random variables in the proof of the lower bound. We think that in the interval κ cd (for some large enough absolute constant c > 0), we can recover the missing ln(1 + 2κ) in our lower bound by using the argument of [Vov01,

Theorem 2] instead. As for the interval κ cd, we could use a different sequence of random variables with bounded support, and, e.g., Assouad's Lemma.

4.A Proofs

4.A.1 Proof of Theorem 4.2

To prove Theorem 4.2, we perform a reduction to the stochastic batch setting (via the standard online to batch trick), and employ a version of the lower bound proved in [START_REF] Tsybakov | Optimal rates of aggregation[END_REF] for convex aggregation.

We first need the following notations. Let T ∈ N * . Let (S, µ) be a probability space for which we can find an orthonormal family 10 (ϕ j ) 1 j d with d elements in the space of square-integrable functions on S, which we denote by L 2 (S, µ) thereafter. For all u ∈ R d and γ, σ > 0, denote by

P γ,σ
u the joint law of the i.i.d. sequence (X t , Y t ) 1 t T such that

Y t = γϕ u (X t ) + σε t ∈ R , (4.23)
where ϕ u d j=1 u j ϕ j , where the X t are i.i.d points in S drawn from µ, and where the ε t are i.i.d standard Gaussian random variables such that (X t ) 1 t T and (ε t ) 1 t T are independent.

The next lemma is a direct adaptation of [Tsy03, Theorem 2], which we state with our notations in a slightly more precise form (we make clear how the lower bound depends on the noise level σ and the signal level γ).

Lemma 4.1 (An extension of Theorem 2 of [START_REF] Tsybakov | Optimal rates of aggregation[END_REF]). Let d, T ∈ N * and γ, σ > 0. Let (S, µ) be a probability space for which we can find an orthonormal family (ϕ j ) 1 j d in L 2 (S, µ), and consider the Gaussian linear model (4.23). Then there exist absolute constants c 4 , c 5 , c 6 , c 7 > 0 such that

inf f T sup u∈R d + j u j 1 E P γ,σ u f T -γϕ u 2 µ      c 4 dσ 2 T if d √ T c 5 γ σ , c 6 γσ 1 T ln 1 + dσ √ T γ if c 5 γ σ < d √ T c 7 γd σ √ ln(1+d)
, where the infimum is taken over all estimators 11 f T : S → R, where the supremum is taken over all nonnegative vectors with total mass at most 1, and where f 2 µ S f (x) 2 µ(dx) for all measurable functions f : S → R.

Note that the lower bound we stated in Theorem 4.2 is very similar to T times the above lower bound with γ ∼ X and σ ∼ Y (recall that κ √ T U X/(2dY )). The main difference is that 10 An example is given by S = [-π, π], µ(dx) = dx/(2π), and ϕj(x) = √ 2 sin(jx) for all 1 j d and x ∈ [-π, π]. We will use this particular case later.

11 As usual, an estimator is a measurable function of the sample (Xt, Yt) 1 t T , but the dependency on the sample is omitted. the latter holds for unbounded observations, while we need bounded observations y t , 1 t T . A simple concentration argument will show that these observations lie in [-Y, Y ] with high probability, which will yield the desired lower bound. The proof of Theorem 4.2 thus consists of the following steps:

• step 1: reduction to the stochastic batch setting;

• step 2: application of Lemma 4.1;

• step 3: concentration argument.

Proof (of Theorem 4.2): We first assume that ln(1 + 2d)/ 2d √ ln 2 κ 1. The case when κ > 1 will easily follow from the monotonicity of the minimax regret in κ (see the end of the proof). We set

T 1 + (4dκ) 2 , U 1 , and X 2dκY √ T , (4.24) so that T 2, √ T U X/(2dY ) = κ, and X Y /2 (since √ T 4dκ).
Step 1: reduction to the stochastic batch setting. First note that by clipping to [-Y, Y ], we have inf

( ft)t sup xt ∞ X |yt| Y T t=1 y t -f t (x t ) 2 -inf u 1 U T t=1 (y t -u • x t ) 2 = inf ( ft)t | ft| Y sup xt ∞ X |yt| Y T t=1 y t -f t (x t ) 2 -inf u 1 U T t=1 (y t -u • x t ) 2 , (4.25)
where the first infimum is taken over all online forecasters 12 ( f t ) t , where the second infimum is restricted to online forecasters ( f t ) t which output predictions in [-Y, Y ], and where both suprema are taken over all individual sequences (x t , y t )

1 t T ∈ (R d × R) T such that |y 1 |, . . . , |y T | Y and x 1 ∞ , . . . , x T ∞ X.
Next we use the standard online to batch conversion (cf. Section 2.5.1) to bound from below the right-hand side of (4.25) by T times the lower bound of Lemma 4.1, which we apply to the particular case where S = [-π, π], µ(dx) = dx/(2π), and

ϕ j (x) = √ 2 sin(jx) for all 1 j d and x ∈ [-π, π]. Let γ c 8 X and σ c 9 Y √ ln T , (4.26)
for some absolute constants c 8 , c 9 > 0 to be chosen by the analysis.

Let ( f t ) t 1 be any online forecaster whose predictions lie in [-Y, Y ], and consider the estimator 12 Recall that an online forecaster is a sequence of functions ( ft) t 1 , where ft : [-π, π] drawn from the uniform distribution µ(dx) = dx/(2π), and where the ε t are i.i.d standard Gaussian random variables such that (X t ) t and (ε t ) t are independent. All the expectations below are thus taken with respect to the probability distribution P γ,σ u * .

R d × (R d × R)
By standard manipulations (e.g., using the tower rule and Jensen's inequality), we get the following lower bound. A detailed proof can be found after the proof of the present theorem (page 154).

Lemma 4.2 (Reduction to the batch setting).

With ( f t ) 1 t T , f T , and u * defined above, we have

E T t=1 y t -f t (x t ) 2 -inf u 1 1 T t=1 y t -u • x t 2 T E f T -γϕ u * 2 µ .
Step 2: application of Lemma 4.1. Next we use Lemma 4.1 to prove that, for some absolute constants c 9 , c 11 > 0,

T E f T -γϕ u * 2 µ c 11 c 2 9 ln 2 + 16d 2 dY 2 κ ln(1 + 1/κ) . (4.29)
By Lemma 4.1 and by definition of u * , we have

E f T -γϕ u * 2 µ      c 4 dσ 2 T if d √ T c 5 γ σ , c 6 γσ 1 T ln 1 + dσ √ T γ if c 5 γ σ < d √ T c 7 γd σ √ ln(1+d) .        c 4 c 2 9 T (ln T ) dY 2 if d √ T c 5 γ σ , c 6 c 8 c 9 √ ln T U XY 1 T ln 1 + c 9 dY c 8 √ T (ln T )U X if c 5 γ σ < d √ T c 7 γd σ √ ln(1+d) , (4.30)
where the last inequality follows from (4.26) and from U = 1.

The above lower bound is only meaningful if the following condition holds true:

d √ T c 7 γd σ ln(1 + d) . (4.31)
But, by definition of T 1 + (4dκ) 2 and by the assumption ln(1 + 2d)/ 2d √ ln 2 κ, elementary manipulations show that (4.31) actually holds true whenever 14 c 9 c 7 c 8 c 10 , where

c 10 1 2 inf x 2 ln 3 ln 2 x √ 1+ x 2
(note that c 10 > 0).

Therefore, if c 9 c 7 c 8 c 10 , then (4.30) entails that

E f T -γϕ u * 2 µ min    c 4 c 2 9 T (ln T ) dY 2 , c 6 c 8 c 9 √ ln T U XY 1 T ln 1 + c 9 dY c 8 T (ln T )U X    . (4.32) Moreover, note that if c 9 c 8 2 √ ln 2, then c 8 c 9 /(2 √ ln 2) c 9 /(2 √ ln T ). In this case, since x → x ln(1 + A/x) is nondecreasing on R *
+ for all A > 0, we can replace c 8 with c 9 /(2 √ ln T ) in the next expression and get

c 6 c 8 c 9 √ ln T U XY 1 T ln 1 + c 9 dY c 8 T (ln T )U X c 6 c 2 9 2 ln T U XY 1 T ln 1 + 2dY √ T U X = c 6 c 2 9 T (ln T ) dY 2 κ ln(1 + 1/κ) ,
where we used the definition of κ √ T U X/(2dY ). In the sequel we will choose the absolute constants c 8 and c 9 such that c 9 c 7 c 8 c 10 and c 9 c 8 2

√ ln 2 . (4.33)
Therefore, by the above remarks, by the fact that ln T ln 1 + (4dκ) 2 ln 2 + 16d 2 (since κ 1 by assumption), and multiplying both sides of (4.32) by T , we get

T E f T -γϕ u * 2 µ min c 4 c 2 9 ln 2 + 16d 2 dY 2 , c 6 c 2 9 ln 2 + 16d 2 dY 2 κ ln(1 + 1/κ) c 11 c 2 9 ln 2 + 16d 2 dY 2 κ ln(1 + 1/κ) ,
where we set c 11 min c 4 / √ ln 2, c 6 , and where used the fact that x → x ln(1 + 1/x) is nondecreasing on R * + , so that its value at x = κ 1 is smaller than √ ln 2. This concludes the proof of (4.29).

Combining Lemma 4.2 and (4.29), we get

E T t=1 y t -f t (x t ) 2 -inf u 1 1 T t=1 y t -u • x t 2 c 11 c 2 9 ln 2 + 16d 2 dY 2 κ ln(1 + 1/κ) . (4.34)
Step 3: concentration argument. At this stage it would be tempting to conclude by using (4.34) to assert that since the expectation is lower bounded, then there is at least one individual sequence with the same lower bound. However, we have no boundedness guarantee about such individual sequence since the random observations y t lie outside of [-Y, Y ] with positive probability. Next we prove that the probability of the event

A T t=1 |y t | Y
is actually close to 1, and that

E I A L T -inf u 1 1 L T (u) 1 2 c 11 c 2 9 ln 2 + 16d 2 dY 2 κ ln(1 + 1/κ) . (4.35)
(Note a missing factor of 2 between (4.34) and (4.35).) The last lower bound will then enable to conclude the proof of this theorem.

Set L T T t=1 y t -f t (x t ) 2 and L T (u) T t=1 y t -u • x t 2 for all u ∈ R d . Denote by A c
the complement of A, and by I A and I A c the corresponding indicator functions. We have

E I A L T -inf u 1 1 L T (u) = E L T -inf u 1 1 L T (u) -E I A c L T -inf u 1 1 L T (u) c 11 c 2 9 ln 2 + 16d 2 dY 2 κ ln(1 + 1/κ) -E I A c L T , (4.36)
where the last inequality follows by (4.34) and by the fact that L T (u) 0 for all u ∈ R d . The rest of the proof is dedicated to upper bounding the above quantity E I A c L T by half the term on its left. This way, we will have proved (4.35).

First note that

E I A c L T E I A c T t=1 y t -f t (x t ) 2 E I A c T t=1 4Y 2 I {|yt| Y } + y t -f t (x t ) 2 I {|yt|>Y } (4.37) 4T Y 2 P A c + T t=1 E y t -f t (x t ) 2 I {|εt|> Y 2σ } , (
E y t -f t (x t ) 2 I {|εt|> Y 2σ } E Y /2 + σ|ε t | + Y 2 I {|εt|> Y 2σ } 2 3Y 2 2 P |ε t | > Y 2σ + 2σ 2 E ε 2 t I {|εt|> Y 2σ } (4.40) 9Y 2 2 P |ε t | > Y 2σ + 2σ 2 √ 3 P 1/2 |ε t | > Y 2σ (4.41) 9Y 2 T -1/(8c 2 9 ) + 2 c 2 9 Y 2 ln 2 √ 6 T -1/(16c 2 9 ) , (4.42)
where we used the following arguments. Inequality (4.40) follows by the elementary inequality (a + b) 2 2(a 2 + b 2 ) for all a, b ∈ R. To get (4.41) we used the Cauchy-Schwarz inequality and the fact that E ε 4 t = 3 (since ε t is a standard Gaussian random variable). Finally, (4.42) follows by definition of σ c 9 Y / √ ln T c 9 Y / √ ln 2 and from the fact that, since ε t is a standard Gaussian random variable,

P |ε t | > Y 2σ 2e -1 2 ( Y 2σ ) 2 = 2e -1 2 √ ln T 2c 9 2 = 2T -1/(8c 2 9 ) .
Using the fact that

P A c T t=1 P |y t | > Y T t=1 P |ε t | > Y /(2σ) 2T 1-1/(8c 2 9
) by the inequality above and substituting (4.42) in (4.38), we get

E I A c L T 8Y 2 T 2-1/(8c 2 9 ) + 9Y 2 T 1-1/(8c 2 9 ) + 2c 2 9 √ 6 ln 2 Y 2 T 1-1/(16c 2 9 ) 8Y 2 2 2-1/(8c 2 9 ) + 9Y 2 2 1-1/(8c 2 9 ) + 2c 2 9 √ 6 ln 2 Y 2 2 1-1/(16c 2 9 ) , (4.43)
where the last inequality follows from the fact that T α 2 α for all α < 0 (since T 2) and from a choice of c 9 such that c 9 < 1/4 (which we assume thereafter).

In order to further upper bound E I A c L T , we use the following technical lemma, which is proved after the proof of the present theorem (see page 155).

Lemma 4.3. There exists an absolute constant c 13 > 0 such that, for all c 9 ∈ (0, c 13 ),

8Y 2 2 2-1/(8c 2 9 ) + 9Y 2 2 1-1/(8c 2 9 ) + 2c 2 9 √ 6 ln 2 Y 2 2 1-1/(16c 2 9 )
1 2

c 11 c 2 9 ln 2 + 16d 2 dY 2 κ ln(1 + 1/κ) .
We can now fix the values of the constants c 8 and c 9 and conclude the proof. Choosing c 9 and c 8 max c 9 /(2 √ ln 2), c 9 /(c 7 c 10 ) such that c 9 < 1/ √ 2 (condition (4.39)), c 9 < 1/4, and c 9 c 13 , then the condition (4.33) also holds, and (4.43) combined with Lemma 4.3 entails that

E I A c L T 1 2 c 11 c 2 9 ln 2 + 16d 2 dY 2 κ ln(1 + 1/κ) .
Substituting the last inequality in (4.36), we get that

E I A L T -inf u 1 1 L T (u) 1 2 c 11 c 2 9 ln 2 + 16d 2 dY 2 κ ln(1 + 1/κ) .
By the above lower bound and the fact that, P γ,σ u * -almost surely, x t ∞ γ √ 2 X for all t = 1, . . . , T (since γ c 9 X and c 9 1/ √ 2), we get that

sup x 1 ∞ ,..., x T ∞ X y 1 ,...,y T ∈R I A L T -inf u 1 1 L T (u) 1 2 c 11 c 2 9 ln 2 + 16d 2 dY 2 κ ln(1 + 1/κ) .

Therefore, by definition of

A T t=1 |y t | Y , of L T T t=1 y t -f t (x t )
2 , and of L T (u) T t=1 (y t -u • x t ) 2 , we get that, for all online forecasters

( f t ) t 1 whose predictions lie in [-Y, Y ], sup x 1 ∞ ,..., x T ∞ X |y 1 |,...,|y T | Y T t=1 y t -f t (x t ) 2 -inf u 1 U T t=1 (y t -u • x t ) 2 1 2 c 11 c 2 9 ln 2 + 16d 2 dY 2 κ ln(1 + 1/κ) .
Combining the last lower bound with (4.25) and setting c 1 c 11 c 2 9 /2 concludes the proof under the assumption ln(1 + 2d)/ 2d √ ln 2 κ 1.

Assume now that κ > 1.

The stated lower bound follows from the case when κ = 1 and by monotonicity of the minimax regret in κ (when d and Y are kept constant).

More formally, by the first part of this proof (when κ = 1), we can fix T 1, U 1 > 0, and

X > 0 such that √ T U 1 X/(2dY ) = 1 and inf ( ft)t sup xt ∞ X |yt| Y T t=1 y t -f t (x t ) 2 -inf u 1 U 1 T t=1 (y t -u • x t ) 2 c 1 ln 2 + 16d 2 dY 2 √ ln 2 ,
where the infimum is taken over all online forecasters ( f t ) t 1 , and where the supremum is taken over all individual sequences bounded by X and Y . Now take κ > 1, and set U κU 1 > U 1 , so that

√ T U X/(2dY ) = κ (since √ T U 1 X/(2dY ) = 1
). Moreover, for all individual sequences bounded by X and Y , the regret on B 1 (U ) is at least as large as the regret on B 1 (U 1 ) (since U > U 1 ). Combining the latter remark with the lower bound above and setting c 2 c 1 √ ln 2 concludes the proof.

Proof (of Lemma 4.2): We use the same notations as in Step 1 of the proof of Theorem 4.2. Let (X , y ) be a random copie of (X 1 , y 1 ) independent of (X t , y t ) 1 t T , and define the random vector x γϕ 1 (X ), . . . , γϕ d (X ) . By the tower rule, we have

E (y t -f t (x t ) 2 = E E (y t -f t (x t )) 2 (x s , y s ) s t-1 = E (y -f t (x ) 2 ,
where we used the fact that f t is built on the past data (x s , y s ) s t-1 and that (x , y ) and (x t , y t ) are both independent of (x s , y s ) s t-1 and are identically distributed. Similarly E (y t -u•x t ) 2 = E (y -u • x ) 2 . Using the last equalities and the fact that E inf{. . .} inf E {. . .} , we get

E T t=1 y t -f t (x t ) 2 -inf u 1 1 T t=1 y t -u • x t 2 T 1 T T t=1 E y -f t (x ) 2 -inf u 1 1 E y -u • x 2 T E y -f T (X ) 2 -inf u 1 1 E y -u • x 2 (4.44) = T E γϕ u * (X ) -f T (X ) 2 (4.45) = T E f T -γϕ u * 2 µ .
Inequality (4.44) follows by definition of f T T -1 T t=1 f t (see (4.27)) and by Jensen's inequality. As for Inequality (4.45), it follows by expanding the square

y -f t (X ) 2 = γϕ u * (X ) -f T (X ) + y -γϕ u * (X ) 2 ,
by noting that E y -γϕ u * (X ) X = 0 (via (4.28)) and by the fact that

inf u 1 1 E y -u • x 2 = E y -γϕ u * (X ) 2 ,
where we used u * 1 1 (by definition of u * ) and u • x = γϕ u (X ). This concludes the proof.

Proof (of Lemma 4.3): We use the same notations and assumptions as in the proof of Theorem 4.2. Since the function x → x ln(1 + 1/x) is nondecreasing on R * + and since κ κ min ln(1 + 2d)/(2d √ ln 2) by assumption, we have

c 11 c 2 9 ln 2 + 16d 2 dY 2 κ ln(1 + 1/κ) c 11 c 2 9 ln 2 + 16d 2 dY 2 κ min ln(1 + 1/κ min ) = c 11 c 2 9 2 √ ln 2 Y 2 ln(1 + 2d) ln 1 + 2d √ ln 2/ ln(1 + 2d) ln 2 + 16d 2 (4.46) c 11 c 2 9 2 √ ln 2 Y 2 c 12 , (4.47)
where c 12 denotes the infimum of the last fraction of (4.46) over all d 1; in particular, c 12 > 0.

It is now easy to see that by choosing the absolute constant c 13 > 0 small enough (where c 13 can be expressed in terms of c 11 and c 12 ), we have, for all c 9 ∈ (0, c 13 ),

8 • 2 2-1/(8c 2 9 ) + 9 • 2 1-1/(8c 2 9 ) + 2c 2 9 √ 6 ln 2 2 1-1/(16c 2 9 ) c 11 c 2 9 2 √ ln 2 c 12 .
Multiplying both sides of the last inequality by Y 2 and combining it with (4.47) concludes the proof.

4.A.2 Proof of

(B t+1 + B t ) 2 T t=1 t:|yt| Bt t ( u t ) + 1 + 1 √ 2 2 T t=1 t:B t+1 >Bt B 2 t+1 T t=1 t ( u t ) + 4 1 + 1 √ 2 2 Y 2 , (4.48)
where the second inequality follows from the fact that:

• if |y t | B t then (y t -[ u t • x t ] Bt ) 2 t ( u t
) by Equation (4.12);

• if |y t | > B t , which is equivalent to B t+1 > B t by definition of B t+1 , then B t B t+1 / √ 2, so that B t+1 + B t 1 + 1/ √ 2 B t+1 .
As for the third inequality above, we used the non-negativity of t ( u t ) and upper bounded the geometric sum T t:B t+1 >Bt B 2 t+1 in the same way as in [CBMS07, Theorem 6], i.e., setting K log 2 max 1 t T y 2 t ,

T t:B t+1 >Bt B 2 t+1 K k=-∞ 2 k = 2 K+1 4Y 2 .
To bound (4.48) further from above, we now use the fact that, by construction, the LEG algorithm is the adaptive EG ± algorithm applied to the modified loss functions t . Therefore, we get from Corollary 2.1 (cf. Chapter 2) that

T t=1 t ( u t ) inf u 1 U T t=1 t (u) + 4U T t=1 ∇ t ( u t ) 2 ∞ ln(2d) + U 8 ln(2d) + 12 max 1 t T ∇ t ( u t ) ∞ .
(4.49)

We can now follow the same lines as in Corollary 2.2, except that we use the particular shape of the Lipschitzified losses. First note from (4.11) that max

1 t T ∇ t ( u t ) ∞ 2 1 + √ 2 XY .
Moreover, using (4.9) and the definition of y t in Figure 4.3, we can see that the gradients satisfy

∇ t ( u t ) = -2 y t -[ u t • x t ] Bt x t = -2(y t -y t ) x t .
Combining the last equality with the upper bound x t ∞ X, we get that

∇ t ( u t ) 2 ∞ 4X 2 (y t -y t ) 2 .
Substituting the last two inequalities in (4.49) and combining the resulting bound with (4.48), we get

T t=1 y t -y t 2 inf u 1 U T t=1 t (u) + 8U X T t=1 y t -y t 2 ln(2d) + 16 ln(2d) + 24 1 + √ 2 U XY + 4 1 + 1/ √ 2 2 Y 2 . Setting C 16 ln(2d) + 24 1 + √ 2 U XY + 4 1 + 1/ √ 2 2 Y 2 , L T T t=1 (y t -y t ) 2 , and L * T min {u∈R d : u 1 U } T t=1 t (u)
, the previous inequality can be simply rewritten as

L T L * T + C + 8U X L T ln(2d) .
Solving for L T via Lemma A.2 in Appendix A.4, we get that

L T L * T + C + 8U X ln(2d) L * T + C + 8U X ln(2d) L * T + 8U X L * T ln(2d) + 8U X C ln(2d) + 64U 2 X 2 ln(2d) + C . (4.50) But, rewriting C = C 1 + C 2 with C 1 16 ln(2d) + 24 1 + √ 2 U XY and C 2 4 1 + 1/ √ 2 2 Y 2 , we get that U X C ln(2d) U X C 1 ln(2d) + U X C 2 ln(2d) = U X C 1 ln(2d) + 2 1 + 1/ √ 2 U XY ln(2d) , (4.51)
where

U X C 1 ln(2d) = U X ln(2d) 16 + 24/ ln(2d) 1 + √ 2 U XY U 2 X 2 + U XY ln(2d) 16 + 24/ ln(2) 1 + √ 2 U XY + U 2 X 2 = 16 + 24/ ln(2) 1 + √ 2 U XY + U 2 X 2 ln(2d) .
Combining (4.50) with (4.51) and the last inequality and performing some simple upper bounds, we conclude the proof.

4.B Lemmas

The next lemma is useful to prove Theorem 4.1. At the end of this section, we also provide an elementary lemma about the exponentially weighted average forecaster combined with clipping.

Lemma 4.4. Let d, T ∈ N * , and U, X, Y > 0. The minimax regret on B 1 (U ) for bounded base predictions and observations satisfies inf

F sup xt ∞ X, |yt| Y T t=1 (y t -y t ) 2 -inf u 1 U T t=1 (y t -u • x t ) 2 min 3U XY 2T ln(2d), 32 dY 2 ln 1 + √ T U X dY + dY 2 ,
where the infimum is taken over all forecasters F and where the supremum extends over all sequences (x t , y t )

1 t T ∈ (R d × R) T such that |y 1 |, . . . , |y T | Y and x 1 ∞ , . . . , x T ∞ X.
Proof: We treat each of the two terms in the above minimum separately.

Step 1: We prove that their exists a forecaster F whose worst-case regret on B 1 (U ) is upper bounded by 3U XY 2T ln(2d).

First note that if U (Y /X) T /(2 ln(2d)), then the upper bound 3U XY 2T ln(2d) 3T Y 2 T Y 2 is trivial (by choosing the forecaster F which outputs y t = 0 at each time t).

We can thus assume that U < (Y /X) T /(2 ln(2d)). Consider the EG ± algorithm as given in [KW97, Theorem 5.11], and denote by u t ∈ B 1 (U ) the linear combination it outputs at each time t 1. Then, by the aforementioned theorem, this forecaster satisfies, uniformly over all individual sequences bounded by X and Y , that

T t=1 (y t -u t • x t ) 2 -inf u 1 U T t=1 (y t -u • x t ) 2 2U XY 2T ln(2d) + 2U 2 X 2 ln(2d) 2U XY 2T ln(2d) + 2 Y T 2 ln(2d) U X ln(2d) (4.52) 3U XY 2T ln(2d) ,
where (4.52) follows from the assumption U X < Y T /(2 ln(2d)). This concludes the first step of this proof.

Step 2: We prove that their exists a forecaster F whose worst-case regret on B 1 (U ) is upper bounded by 32 dY 2 ln 1 +

√ T U X dY + dY 2 .
Such a forecaster is given by the algorithm SeqSEW B,η τ of Section 3.3.1 (Chapter 3) tuned with B = Y , η = 1/(8Y 2 ) and τ = Y /( √ T X). Indeed, by Proposition 3.1 of Chapter 3, the cumulative square loss of this algorithm is upper bounded by

inf u∈R d T t=1 y t -u • x t 2 + 32 u 0 Y 2 ln 1 + √ T X u 1 u 0 Y + dY 2 inf u 1 U T t=1 y t -u • x t 2 + 32dY 2 ln 1 + √ T XU dY + dY 2 ,
where the last inequality follows by monotonicity15 in u 0 and u 1 of the second term of the left-hand side. This concludes the proof.

Next we recall a regret bound satisfied by the standard exponentially weighted average forecaster applied to clipped base forecasts. Assume that at each time t 1, the forecaster has access to K 1 base forecasts y (k) t ∈ R, k = 1, . . . , K, and that for some known bound Y > 0 on the observations, the forecaster predicts at time t as

y t K k=1 p k,t y (k) t Y .

In the equation above, [x] Y

min{Y, max{-Y, x}} for all x ∈ R, and the weight vectors p t ∈ R K are given by p 1 = (1/K, . . . , 1/K) and, for all t = 2, . . . , T , by

p k,t exp -η t-1 s=1 y s -y (k) s Y 2 K j=1 exp -η t-1 s=1 y s -y (j) s Y 2 , 1 k K ,
for some inverse temperature parameter η > 0 to be chosen below. The next lemma is a straigthforward consequence of Theorem 2.2 in Chapter 2.

Lemma 4.5 (Exponential weighting with clipping). Assume that the forecaster knows beforehand a bound Y > 0 on the observations |y t |, t = 1, . . . , T . Then, the exponentially weighted average forecaster tuned with η 1/(8Y 2 ) and with clipping

[ • ] Y satisfies T t=1 y t -y t 2 min 1 k K T t=1 y t -y (k) t 2 + ln K η .
Proof (of Lemma 4.5): The proof follows straightforwardly from Theorem 2.2 in Chapter 2. To apply the latter result, recall from Appendix A.2 that the square loss is 1/(8Y 2 )-exp-concave on [-Y, Y ] and thus η-exp-concave16 (since η 1/(8Y 2 ) by assumption). Therefore, by definition of our forecaster above, Theorem 2.2 yields

T t=1 y t -y t 2 min 1 k K T t=1 y t -y (k) t Y 2 + ln K η .
To conclude the proof, note for all t = 1, . . . , T and k = 1, . . . , K that |y t | Y by assumption, so that clipping the base forecasts to [-Y, Y ] can only improve prediction, i.e., y t -y

(k) t Y 2 y t -y (k) t 2 .

4.C Additional tools

The next approximation argument is originally due to Maurey, and was used under various forms, e.g., in [Nem00, Tsy03, BN08, SSSZ10].

Lemma 4.6 (Approximation argument). Let U > 0 and m ∈ N * . Define the following finite subset of B 1 (U ):

B U,m    k 1 U m , . . . , k d U m : (k 1 , . . . , k d ) ∈ Z d , d j=1 |k j | m    ⊂ B 1 (U ) .
Then, for all (x t , y t )

1 t T ∈ R d × R T such that max 1 t T x t ∞ X, inf u∈ B U,m T t=1 (y t -u • x t ) 2 inf u∈B 1 (U ) T t=1 (y t -u • x t ) 2 + T U 2 X 2 m .
Proof: The proof is quite standard and follows the same lines as [Nem00, Proposition 5.2.2] or [BN08, Theorem 2] who addressed the aggregation task in the stochastic setting. We rewrite this argument below in our online deterministic setting.

Fix u * ∈ argmin u∈B 1 (U ) T t=1 (y t -u•x t ) 2 . Define the probability distribution π = (π -d , . . . , π d ) ∈ R 2d+1 + by π j                    (u * j ) + U if j 1; (u * j ) - U if j -1; 1 - d j=1 |u * j | U if j = 0 .
Let J 1 , . . . , J m ∈ {-d, . . . , d} be i.i.d. random integers drawn from π, and set

u U m m k=1 e J k ,
where (e j ) 1 j d is the canonical basis of R d , where e 0 0, and where e -j -e j for all 1 j d. Note that u ∈ B U,m by construction. Therefore,

inf u∈ B U,m T t=1 (y t -u • x t ) 2 E T t=1 (y t -u • x t ) 2 .
(4.53)

The rest of the proof is dedicated to upper bounding the last expectation. Expanding all the squares

(y t -u • x t ) 2 = (y t -u * • x t + u * • x t -u • x t ) 2 , first note that E T t=1 (y t -u • x t ) 2 = T t=1 (y t -u * • x t ) 2 + T t=1 E (u * • x t -u • x t ) 2 + 2 T t=1 (y t -u * • x t ) E u * • x t -u • x t . (4.54)
But by definition of u and π,

E u = U E e J 1 = U d j=-d π j e j = U d j=1 u * j + U e j + u * j - U (-e j ) = U d j=1 u * j U e j = u * ,
so that E u • x t = u * • x t for all 1 t T . Therefore, the last sum in (4.54) above equals zero, and

E u * • x t -u • x t 2 = Var u • x t = U 2 m 2 m k=1 Var e J k • x t U 2 X 2 m ,
where the second equality follows from u • x t = (U/m) m k=1 e J k • x t and from the independence of the J k , 1 k m, and where the last inequality follows from |e

J k • x t | e J k 1 x t ∞ X for all 1 k m.
Combining (4.54) with the remarks above, we get

E T t=1 (y t -u • x t ) 2 T t=1 (y t -u * • x t ) 2 + T U 2 X 2 m = inf u∈B 1 (U ) T t=1 (y t -u • x t ) 2 + T U 2 X 2 m ,
where the last line follows by definition of u * . Substituting the last inequality in (4.53) concludes the proof.

The combinatorial result below (or variants of it) is well-known; see, e.g., [START_REF] Tsybakov | Optimal rates of aggregation[END_REF][START_REF] Bunea | Sequential procedures for aggregating arbitrary estimators of a conditional mean[END_REF]. We reproduce its proof for the convenience of the reader. We use the notation e exp(1).

   (k 1 , . . . , k d ) ∈ Z d : d j=1 |k j | m       (k -d , . . . , k d ) ∈ N 2d+1 : d j=-d k j = m    = 2d + m m (4.55) e(2d + m) m m . (4.56)
To get inequality (4.55), we used the (elementary) fact that the number of 2d + 1 integer-valued tuples summing up to m is equal to the number of lattice paths from (1, 0) to (2d + 1, m) in N 2 , which is equal to 2d+1+m-1 m . As for inequality (4.56), it follows straightforwardly from a classical combinatorial result stated, e.g., in [Mas07, Proposition 2.5].

Chapter 5

Minimax rates of internal and swap regrets

Within the framework of prediction with expert advice under linear losses, we study the minimax rates of two performance criteria related to game theory: internal regret and swap regret. We first prove the exact rates

√

T and √ T ln K respectively for internal and swap regrets when the loss vectors are i.i.d.. This shows that the missing √ ln K factor between the known upper and lower bounds of [START_REF] Stoltz | Internal regret in on-line portfolio selection[END_REF] and [START_REF] Stoltz | Incomplete information and internal regret in prediction of individual sequences[END_REF] on internal regret is unnecessary in the stochastic i.i.d. setting. Second, in the game with arbitrary deterministic loss vectors, we provide a lower bound of order

√

T K on the swap regret; it improves on a lower bound of [START_REF] Blum | From external to internal regret[END_REF]. Finally, we develop a generic technique that enables to reinterpret known deterministic regret bounds from a stochastic viewpoint, but also to derive a new regret bound in the problem of learning with global cost functions.

Introduction

In this chapter, we consider a decision-theoretic variant1 of the framework of prediction with expert advice due to [START_REF] Freund | A decision-theoretic generalization of on-line learning and an application to boosting[END_REF]. The problem is stated as a repeated game between a forecaster and an environment. At each time round t ∈ N * = {1, 2, . . .}, the forecaster chooses a weight vector p t = (p 1,t , . . . , p K,t ) over K 2 different actions, i.e., p t belongs to the simplex

X K {x ∈ R K + , K i=1 x i = 1}.
The environment then reveals a loss vector t ( i,t ) 1 i K in [0, 1] K ; each action i ∈ {1, . . . , K} incurs the loss i,t and the forecaster incurs the linear loss p t • t = K i=1 p i,t i,t . After T 1 time rounds, the cumulative loss of the forecaster equals T t=1 p t • t , and his primary goal is to minimize it. In the sequel, we assume for simplicity that the loss sequence 1 , . . . , T is fixed in advance by the environment. However, by Section 2.3.1 (cf. Chapter 2), since we only consider deterministic forecasters, all upper bounds stated for individual sequences also hold true for adversarial environments.

The weight vectors p t are chosen on the basis of the past loss vectors and can therefore be seen as values of functions p t ( 1 , . . . , t-1 ). We call strategy (of the forecaster) any sequence (p t ) t 1 of Borel functions p t : [0, 1] K(t-1) → X K . For notational convenience, we often omit the dependency in ( 1 , . . . , t-1 ) and write p t instead of p t ( 1 , . . . , t-1 ).

A classical way to assess the quality of a strategy S = (p t ) t 1 on a finite loss sequence 1:T ( 1 , . . . , T ) is to compare its cumulative loss to that of the best action in hindsight. The difference between these two quantities, i.e.,

R ext

T (S, 1:T )

T t=1 p t • t -min 1 j K T t=1 j,t , (5.1)
is called the external regret of the forecaster and measures his difficulty to mimic the best action in hindsight while being compelled to output decisions in a sequential fashion. This performance criterion was introduced in Chapter 2 and studied in Chapters 3 and 4.

In this chapter, we study two stronger notions of regret called internal regret and swap regret, which play an important role in the theory of repeated games. Like for the external regret, the cumulative loss of the forecaster is compared to that of the best strategy (in hindsight) in a given reference class. However, the reference strategies are not external as for the external regret, but are instead given by consistent modifications of the forecaster's own strategy -hence the term internal.

The notion of internal regret was first studied by [FV97, FV98, FV99] (see also [START_REF] Fudenberg | Universal conditional consistency[END_REF][START_REF] Hart | A simple adaptive procedure leading to correlated equilibrium[END_REF][START_REF] Hart | A general class of adaptive strategies[END_REF]). For any strategy S = (p t ) t 1 and any finite loss sequence 1 , . . . , T ∈ [0, 1] K , the internal regret R int T (S, 1:T ) of the strategy S on the sequence 1:T ( 1 , . . . , T ) is defined by

R int T (S, 1:T ) T t=1 p t • t -min 1 i =j K T t=1 p i→j t • t , (5.2)
where the modified weight vector p i→j t ∈ X K is obtained from p t by replacing action i with action j. Namely, for all k = 1, . . . , K, the k-th component of p i→j t equals

(p i→j t ) k =      0 if k = i, p i,t + p j,t if k = j, p k,t if k / ∈ {i, j}. (5.3)
Internal regret thus measures for all pairs (i, j), i = j, the regret the forecaster feels for not choosing action j each time he chose action i (all loss vectors being equal). Intuitively, if a forecaster has a small internal regret then he enjoys some stability properties. This has indeed been illustrated in game theory: [START_REF] Foster | Calibrated learning and correlated equilibrium[END_REF][START_REF] Foster | Regret in the on-line decision problem[END_REF] showed that if all players of a finite randomized game choose a strategy whose internal regret is almost surely sublinear in T , then the joint empirical frequencies of play converge almost surely to an equilibrium set called the set of correlated equilibria (see also [START_REF] Fudenberg | Universal consistency and cautious fictitious play[END_REF][START_REF] Hart | A simple adaptive procedure leading to correlated equilibrium[END_REF][START_REF] Stoltz | Learning correlated equilibria in games with compact sets of strategies[END_REF]). Internal regret also has some historical connections with another branch of game theory called calibration: the existence of strategies with sublinear internal regret implies the existence of calibrated forecasters (see [START_REF] Foster | Asymptotic calibration[END_REF] and the other references given in [CBL06,

Chapter 4]).

The notion of swap regret was introduced by [START_REF] Blum | From external to internal regret[END_REF] (see also [START_REF] Greenwald | A general class of no-regret learning algorithms and gametheoretic equilibria[END_REF] for the broader notion of Φ-regret). The swap regret of a strategy is larger than its internal regret, since the pool of modified strategies {(p i→j t ) t 1 , i = j} to which the forecaster's strategy is compared is extended to all linear modifications {(ϕ(p t )) t 1 }, where ϕ extends over all linear mappings from the simplex X K into itself. As is done in [Sto05, Chapter 3] via the Krein-Millman theorem, we can equivalently define the swap regret R sw T (S, 1:T ) of a strategy S = (p t ) t 1 on a finite loss sequence

1 , . . . , T ∈ [0, 1] K by R sw T (S, 1:T ) T t=1 p t • t -min F ∈F K T t=1 p F t • t , (5.4)
where F K denotes the set of all mappings F : {1 . . . , K} → {1 . . . , K} and where the modified weight vector p F t ∈ X K is obtained from p t by replacing each action i with the action F (i). Namely, for 1 j K, the j-th component of p F t is defined by

(p F t ) j = i:F (i)=j p i,t .
(5.5)

Known upper and lower bounds on internal and swap regrets

The existence of strategies with a small (sublinear in T ) internal regret on individual sequences was first shown by [START_REF] Foster | Calibrated learning and correlated equilibrium[END_REF]; see also [START_REF] Fudenberg | Universal conditional consistency[END_REF][START_REF] Hart | A simple adaptive procedure leading to correlated equilibrium[END_REF][START_REF] Hart | A general class of adaptive strategies[END_REF]. More precisely, [START_REF] Foster | Calibrated learning and correlated equilibrium[END_REF] designed an algorithm whose internal regret on all individual sequences 1 , . . . , T ∈ [0, 1] K is upper bounded by √ 2KT . The dependence in T is correct (see below), but the dependence in K is far from being optimal. Indeed, using an argument of [START_REF] Hart | A general class of adaptive strategies[END_REF], [START_REF] Cesa-Bianchi | Potential-based algorithms in on-line prediction and game theory[END_REF] proved the upper bound 2

√

T ln K for a suitable exponentially weighted average forecaster. This bound was later lowered by [START_REF] Stoltz | Internal regret in on-line portfolio selection[END_REF] to √ T ln K via a new analysis based on a fixed-point property. Therefore, up to now, the best2 upper bound on the minimax internal regret for individual sequences reads:

inf S sup 1 ,..., T ∈[0,1] K R int T (S, 1:T ) √ T ln K , (5.6)
where the infimum is taken over all strategies S = (p t ) t 1 of the forecaster. As shown by [START_REF] Stoltz | Incomplete information and internal regret in prediction of individual sequences[END_REF], the last upper bound cannot be improved more than by a logarithmic factor

√ ln K. Indeed, it is proved in [Sto05, Theorem 3.3] that, for all K
2 and all T K 2 /192, the minimax internal regret for i.i.d. loss vectors is bounded from below by

inf S sup Q E Q ⊗T R int T (S, 1:T ) √ T / 64 √ 3 , (5.7)
where the infimum is taken over all strategies S = (p t ) t 1 , where the supremum is taken over all probability distributions on [0, 1] K (endowed with its Borel σ-algebra), and where the loss vectors

In the same spirit, [START_REF] Blum | From external to internal regret[END_REF] proved that there exists an (efficient) algorithm whose swap regret is upper bounded by (T /2)K ln K uniformly over all individual sequences; see also [START_REF] Stoltz | Internal regret in on-line portfolio selection[END_REF] for an alternative proof with a less efficient algorithm (of combinatorial nature). Therefore, as of now, the best upper bound on the minimax swap regret for individual sequences reads:

inf S sup 1 ,..., T ∈[0,1] K R sw T (S, 1:T ) (T /2)K ln K .
The last upper bound was shown to be almost optimal by [START_REF] Blum | From external to internal regret[END_REF], who exhibited a lower bound of the order of √ T K. Their lower bound has however two limitations: first, it is proved in a randomized and adversarial setting for a quantity larger than the swap regret stricto sensu. Second, it is proved only for rounds T that are sub-exponential in K. See Section 5.4 for further details.

Main contributions

The main contributions of this chapter are the following. The first one is related to the stochastic protocol (i.i.d. loss vectors) while the other two ones are related to the deterministic protocol (arbitrary loss vectors).

• In the stochastic protocol, we derive the exact minimax rates of internal regret and swap regret for i.i.d. loss vectors. Using the same notations as above, we show that they are respectively of the order3 of √ T and √ T ln K:

inf S sup Q E Q ⊗T R int T (S, 1:T ) √ T and inf S sup Q E Q ⊗T R sw T (S, 1:T ) √ T ln K .
In particular, when the loss vectors are i.i.d., the optimal rate of internal regret is independent of the ambient dimension K.

• We prove a lower bound of the order of √ T K on the minimax swap regret for individual sequences: setting c 1/ 16 128 ln(4/3) , we show that, for all K 2 and all T max 128c 2 K 5 , K ,

inf S sup 1 ,..., T ∈[0,1] K R sw T (S, 1:T ) c √ T K .
This lower bound is stronger than that of [BM07b, Theorem 9] since it holds for the swap regret itself instead of a randomized variant of it (see Section 5.4). This solves a question left open in [BM07b, Section 9]. Besides, we do not need their assumption that T be subexponential in K.

Moreover, our lower bound of order √ T K highlights a major difference between external and swap regrets. On the one hand, as recalled in Chapter 2, the external regret behaves similarly on i.i.d. loss vectors and on individual sequences. Indeed, combining Theorem 2.1 and Lemma 2.2 therein (cf. pages 46 and 62), we get that, using the same notations as above, for all K 1 and all T 40e/(2e + 1) ln K,

2 2e + 1 e T ln K 5(2e + 1) inf S sup Q E Q ⊗T R ext T (S, 1:T ) inf S sup 1 ,..., T ∈[0,1] K R ext T (S, 1:T ) T 2 ln K .
On the other hand, contrary to external regret, swap regret is much harder to minimize with individual sequences than with i.i.d. losses (compare the rates √ T K and √ T ln K above).

• We develop a stochastic technique to derive upper bounds on a generalized form of regret including external, internal, and swap regrets. This technique provides a new insight on the rates of these three types of regret and can be used to recover the best upper bounds known so far. We also derive -in a non-constructive way -a new upper bound of order √ T ln K on the makespan regret, thus improving on the known bound of order ln(K)

√ T of [EDKMM09].
As is detailed in Section 5.5.1, a similar stochastic technique was independently studied in [START_REF] Rakhlin | Online learning: beyond regret[END_REF]. Since we work in a much more specific setting, we are able to get (sometimes tight) explicit constants. Our proof relies on arguments such as Bernoullization and an elementary maximal inequality for subgaussian random variables.

This work in progress raises some important questions. First, though the aforementioned stochastic technique is useful to better understand the problem at hand (since it provides an upper bound on the minimax regret), it is not constructive. Designing explicit algorithms that achieve the obtained upper bounds is an important task to be addressed in the future (e.g., is there any efficient algorithm with a makespan regret at most of order √ T ln K?). Note that the same issue arises in [START_REF] Rakhlin | Online learning: beyond regret[END_REF]. Second, the question of the missing logarithmic factor √ ln K between the lower and upper bounds on the internal regret is still (partially) open. We did prove that the √ ln K factor is unnecessary for i.i.d. loss vectors, but we still do not know whether this is also the case for arbitrary deterministic loss vectors.

Outline of the chapter

The rest of the chapter is organized as follows. In Section 5.2 we formally describe our setting and notations and state some basic facts. In Section 5.3 we derive the √ T minimax rate of internal regret in a stochastic environment. In Section 5.4 we prove a lower bound of the order of √ T K on the minimax swap regret with individual sequences, together with the optimal rate √ T ln K for the minimax swap regret with i.i.d. loss vectors. The results in Sections 5.3 and 5.4 are obtained in a constructive way. In Section 5.5 we provide a general (non-constructive) stochastic technique to derive upper bounds with individual sequences on the quantities studied before and on other ones. Finally, some technical proofs can be found in Section 5.A while some elementary lemmas are provided in Section 5.B.

Setting, notations, and basic properties

Setting and notations

We give in Figure 5.1 a formal description of our repeated game. We consider two different assumptions on the way the loss vectors t are chosen before the beginning of the game: the t can either be drawn by a stochastic environment or they can form an individual sequence.

In the sequel, we denote by X K

x ∈ R K + : K i=1 x i = 1 the simplex of order K and by F K the set of all functions from {1, . . . , K} to {1, . . . , K}. We also set, for all x ∈ R,

x sup k ∈ Z : k x and x inf k ∈ Z : k x .

Basic properties

Next we recall some basic properties of external, internal, and swap regrets and well-known inequalities to compare them.

Equivalent definitions of external, internal, and swap regrets

In view of (5.1), the external regret can be rewritten as follows:

R ext T (S, 1:T ) = max

1 j K K i=1 T t=1 p i,t ( i,t -j,t ) .
(5.8)

Initial step: the environment chooses a sequence of loss vectors ( t ) t∈N * , where the t = ( i,t ) i ∈ [0, 1] K will be revealed round after round. Two different assumptions are considered:

• stochastic environment: ( t ) t 1 is an i.i.d. sequence;

• individual sequence: ( t ) t 1 is an arbitrary deterministic sequence.

At each time round t ∈ N * , 1. the forecaster chooses a convex combination p t ∈ X K ;

2. the environment reveals the loss vector t ∈ [0, 1] K ;

3. each action i incurs the loss i,t and the forecaster incurs the linear loss As for the internal regret, note that since the weight vectors p i→j t and p t only differ in at most two coordinates, many terms cancel out in the difference (5.2). Therefore, we get that R int T (S, 1:T ) = max

p t • t = K i=1 p i,t i,t .
1 i =j K T t=1 p i,t ( i,t -j,t ) .
(5.9)

Finally, in view of (5.4) and (5.5), and noting that p F t • t = K i=1 p i,t F (i),t , the swap regret can be rewritten as follows:

R sw T (S, 1:T ) = max F ∈F K K i=1 T t=1 p i,t i,t -F (i),t = K i=1 max 1 j K T t=1 p i,t i,t -j,t . (5.10)

Comparison of external, internal, and swap regrets

The three notions of external, internal, and swap regret are closely related. Equations (5.1), (5.2), and (5.4) show that

R ext T R sw T and R int T R sw T . (5.11)
Internal and swap regrets are of the same order of magnitude in T since R sw T K (R int T ) + by (5.9) and (5.10), so that

R int T R sw T K (R int T ) + .
On the contrary, external and internal regrets are not necessarily of the same order of magnitude in T . On the one hand, we can see from (5.8) and (5.9) that

R ext T (K -1) (R int T ) + ,
so that minimizing internal regret is a more difficult task than minimizing external regret (up to the constant factor K -1). On the other hand, we cannot upper bound the internal regret by a constant times the external regret in the general case. Indeed, as shown by [START_REF] Stoltz | Internal regret in on-line portfolio selection[END_REF], there exists an algorithm whose external regret is sublinear in T but whose internal regret grows linearly in T .

A new elementary upper bound on the internal regret

Next we design a strategy whose internal regret is almost bounded by min √ T ln K, T /K uniformly over all individual sequences 1 , . . . , T ∈ [0, 1] K . The latter bound interpolates the √ T ln K bound of [START_REF] Stoltz | Internal regret in on-line portfolio selection[END_REF] and the trivial T /K bound satisfied by the forecaster choosing constant weight vectors p t = (1/K, . . . , 1/K). Such an interpolation improves on the bound

√ T ln K for large values of K since T /K √ T ln K when K √ ln K √ T .
If T is known in advance, the bound min √ T ln K, T /K can be easily achieved. Indeed, it suffices to predict either with uniform weight vectors if K √ ln K √ T or with a strategy attaining the upper bound

√ T ln K if K √ ln K < √
T (e.g., the strategy described in [SL05, Theorem 3.1]). Next we design a simple trick which, up to a factor of √ 2 and a small remainder term, achieves the bound min

√

T ln K, T /K without knowing T in advance.

Let S be any strategy whose internal regret after T time steps is upper bounded by √ T ln K. We denote its t-th weight vector by p S t ( 1 , . . . , t-1 ) when applied to the loss sequence 1 , 2 , . . . ∈ [0, 1] K . Our meta-strategy (p t ) t 1 is built on S as follows. We split the whole time interval N * into two periods {1, . . . , T 0 } and {T 0 + 1, T 0 + 2, . . .}, where

T 0 K 2 ln K + 1 . Note that T 0 approximately satisfies K √ ln K ≈ √ T 0 , so that K √ ln K √ T for all T T 0 while K √ ln K √
T for all T T 0 + 1. The last comment combined with the above remark about the case when T is known in advance suggests to define our meta-strategy as follows. On the first period, our meta-strategy outputs uniform weights, i.e., p t (1/K, . . . , 1/K) , for all t ∈ {1, . . . , T 0 } .

(5.12)

On the second period, we start the strategy S at time T 0 + 1 from scratch (i.e., the past information ( 1 , . . . , T 0 ) is not used) and we output the same weight vectors as S, i.e., p t p S t-T 0 ( T 0 +1 , . . . , t-1 ) , for all t T 0 + 1 .

(5.13) Proposition 5.1 (A new elementary upper bound on the internal regret). Let K 2. Then, the internal regret of the strategy defined in (5.12) and (5.13) satisfies, for all T 1 and all

1 , . . . , T ∈ [0, 1] K , max 1 i =j K T t=1 p i,t ( i,t -j,t ) √ 2 min T /K, √ T ln K + 1/K .
Remark 5.1. Up to a factor of √ 2 and a small remainder term, the above bound interpolates the two aforementioned bounds √ T ln K and T /K. In particular, it improves on the √ T ln K bound for large values of K. Note that it is achieved by a strategy that does not use prior knowledge of T .

Proof: The case T T 0 is straightforward: since the weight vectors p t are uniform for all t T 0 , max

1 i =j K T t=1 p i,t ( i,t -j,t ) T /K √ 2 min T /K, √ T ln K + 1/K .
We can thus assume that T > T 0 . Since the maximum is subadditive, we have

max 1 i =j K T t=1 p i,t ( i,t -j,t
) max

1 i =j K T 0 t=1 p i,t ( i,t -j,t ) + max 1 i =j K T t=T 0 +1 p i,t ( i,t -j,t ) .
In the last inequality, the first term of the right-hand side is bounded from above by T 0 /K since the first T 0 weights are uniform. In view of the requirement imposed on the strategy S, the second term is bounded from above by (T -T 0 ) ln K. Therefore, we get that max

1 i =j K T t=1 p i,t ( i,t -j,t ) T 0 K + (T -T 0 ) ln K K 2 ln K + 1 K + (T -K 2 ln K) ln K (5.14) = 1/K + K 2 ln 2 K + T ln K -K 2 ln 2 K 1/K + 2(T ln K) , (5.15)
where (5.14) follows from the fact that T 0 -1 K 2 ln K T 0 (by definition of T 0 ), and where (5.15) follows from the elementary inequality √ x + √ y 2(x + y) that holds for all x, y 0 . Since T T 0 K 2 ln K, we have

√ T ln K T /K, so that 2(T ln K) √ 2 min T /K, √ T ln K .
This concludes the proof.

Since the bound T /K is easy to achieve, in the sequel, we focus on the most interesting regime (i.e., the second one, when T is large enough). For example, we show in the next section that when the loss vectors are i.i.d., the upper bound

√

T ln K can be lowered to √ T .

Minimax rate of internal regret in a stochastic environment

In this section we derive the optimal rate of internal regret in a stochastic environment. Namely, we consider the repeated game of Figure 5.1 with i.i.d. loss vectors: the t ∈ [0, 1] K , t 1, are drawn independently at random from a common distribution

Q ∈ M + 1 [0, 1] K .
In the sequel, all expectations are taken with respect to Q ⊗T , where T 1 is a fixed time horizon.

We set m i E[ i,1 ] = . . . = E[ i,T ] for all 1 i K and define the gaps ∆ i by

∆ i m i -m i * , where i * ∈ argmin 1 j K m j .
Next we prove upper bounds for the internal regret that are of the order of √ T with high probability. This entails that the optimal rate of the expected internal regret against i.i.d. loss vectors is

√

T and therefore does not depend on the ambient dimension K. For the sake of clarity, we first assume that the distribution Q of the loss vectors is known to the forecaster, and then extend the analysis to the case when it is unknown.

Known distribution

In this subsection we assume that the distribution Q of the loss vectors is known to the forecaster. We explain below why it is possible to achieve an internal regret independent of the ambient dimension K. A key remark is that, contrary to the external regret where the weights p i,t appear additively over i ∈ {1, . . . , K}, the internal regret max 1 i =j K T t=1 p i,t ( i,tj,t ) scales as a single p i,t (the sum over i is replaced by a maximum). Therefore, if several actions i are almost optimal (i.e., if they almost minimize m i ), then the probability mass of p t should be well spread among those actions.

Let us illustrate the above remark with the toy situation where the losses i,t , i = 1, . . . , K, t = 1, . . . , T , are i.i.d. Bernoulli random variables with parameter 1/2. In this case, the strategy that constantly outputs the Dirac probability distribution p t = δ i * at some i * ∈ argmin 1 i K m i has an expected internal regret of E max

1 i =j K T t=1 p i,t ( i,t -j,t ) = E T t=1 i * ,t -min 1 j K T t=1 j,t = E max 1 j K T t=1 1 2
j,t .

By central limit arguments4 , the last expectation is of the order of √ T ln K. On the contrary, the strategy that outputs uniform weight vectors p t = (1/K, . . . , 1/K) has an expected internal regret E max

1 i =j K T t=1 p i,t ( i,t -j,t ) = 1 K E max 1 i =j K T t=1 ( i,t -j,t )
of the order of √ T ln K/K. Therefore, though tempting at first sight, the naive strategy (δ i * ) t 1 is suboptimal against i.i.d. vectors, while averaging over the actions leads to a

√

T -internal regret.

Consider now the still ideal but slightly more difficult situation where the distribution Q of the loss vectors is such that, for some K ∈ {1, . . . , K},

m 1 = . . . = m K < m K +1 = . . . = m K where m K +1 -m K ln K T .
In this case assigning uniform weights to all the actions is clearly a bad choice, and a suitable trade-off between "selecting the good actions" and "spreading the probability mass sufficiently" is necessary. A simple and reasonable strategy consists in assigning zero weights to the suboptimal actions i ∈ {K + 1, . . . , K} and uniform weights to the optimal actions i ∈ {1, . . . , K }; more formally, p 1 = . . . = p K = 1/K and p K +1 = . . . = p K = 0 .

Then, due to the averaging, the expected internal regret is not of order T ln(K ) (as would be the case with (δ i * ) t 1 ) but at most of order T ln(K )/K , hence a √ T -rate again. The latter statement is proved for a more general (smoothed) strategy in Theorem 5.1 and can be roughly explained as follows: decomposing T t=1 p i ( i,tj,t ) into a bias term and a deviation term, E max

1 i =j K T t=1 p i ( i,t -j,t ) = E max 1 i =j K p i T (m i -m j ) + p i T t=1 i,t -j,t -(m i -m j ) ≈ E max 1 i =j K p i T (m i -m j ) + p i T t=1 i,t -j,t -(m i -m j ) = 1 K E max 1 i =j K T t=1 i,t -j,t -(m i -m j ) , (5.16)
where the approximation above (i.e., the restriction of the minimum to {1, . . . , K }) follows from the fact that p i = 0 for all i > K and from the fact that, for all i K and j > K ,

T (m i -m j ) - √ T ln K + T t=1 i,t -j,t -(m i -m j ) √ T ln K 0 ,
where we used the assumption m i -m j -(ln K)/T and the fact that, by Hoeffding-Azuma inequality (cf. Lemma A.6 in Appendix A.5), the deviations T t=1 i,tj,t -(m i -m j ) are at most of the order of

√

T ln K with high-probability. But, by a well-known maximal inequality stated in [Mas07, Lemma 2.3] (see Lemma A.3 in Appendix A.5), the expectation in (5.16) is at most of the order of T ln(K )/K .

Parameters: Q ∈ M + 1 [0, 1] K and T 1. At each time round t = 1, . . . , T , Output the same weight vector p int (Q) = p int i (Q) 1 i K defined by p int i (Q) e - √ T m i K j=1 e - √ T m j , 1 i K , (5.17)
where

m i E Q [ i,1
] for all i ∈ {1, . . . , K}. If the distribution Q of the loss vectors is arbitrary, then the trade-off between "selecting the good actions" and "spreading the probability mass sufficiently" can be carried out in a continuous way. The above explanation suggests to take constant weigth vectors p t = p 1 such that p i,1 decreases continuously with the gap ∆ i m i -m i * . Such a choice is given by the exponential weights defined in Figure 5.2. Note that the corresponding weights p int i (Q) can be rewritten as

p int i (Q) = e - √ T ∆ i K j=1 e - √ T ∆ j = e - √ T ∆ i K eff where K eff K j=1 e - √ T ∆ j ∈ [1, K] .
K eff is a smooth generalization of the number K considered in the example above; it can be thought of as the effective number of good actions.

The next theorem provides an (optimal)

√ T -high-probability upper bound on the internal regret of the simple strategy of Figure 5.2. In particular, it is independent of the ambient dimension K.

Theorem 5.1 (A √ T -internal regret when the distribution Q of the loss vectors is known). Let K 2 and T 1. Assume that the loss vectors t ∈ [0, 1] K , 1 t T , are drawn independently at random from a common known distribution

Q ∈ M + 1 ([0, 1] K ).
Then, for all δ ∈ (0, 1), with probability at least 1 -δ, the internal regret of the constant sequence p int (Q) t 1 defined in (5.17) is upper bounded by max

1 i =j K T t=1 p int i (Q)( i,t -j,t ) 3 K eff T ln 3K eff δ 3 T ln 3 δ ,
where

K eff K i=1 e - √ T ∆ i ∈ [1, K
] can be interpreted as the effective number of good actions.

The proof is postponed to Appendix 5.A. As noted in Remark 5.6 therein, a weighted unionbound is key to derive an upper bound of the order of √ T .

Unknown distribution

In this section the distribution Q of the loss vectors is no longer assumed to be known in advance by the forecaster. We adapt the simple strategy of Figure 5.2 to this setting by a plug-in method: the expectations m i E Q [ i,1 ] are sequentially estimated over exponentially growing epochs {1} and 2 r-1 + 1, . . . , 2 r , r ∈ N * . The resulting strategy is defined in Figure 5.3. We prove in Theorem 5.2 that it still achieves a √ T -internal regret with high probability.

Theorem 5.2 (A √ T -internal regret when the distribution Q of the loss vectors is unknown). There is an absolute constant c 0 > 0 such that the following holds true. Let K 2 and T 1. Assume that the loss vectors t ∈ [0, 1] K , 1 t T , are drawn independently at random from an unknown distribution Q ∈ M + 1 ([0, 1] K ). Then, for all δ ∈ (0, 1), with probability at least 1-δ, the internal regret of the strategy defined in Figure 5.3 is upper bounded by max

1 i =j K T t=1 p i,t ( i,t -j,t ) c 0 √
T exp 2 2 ln(4/δ) + 1 .

Parameter: none.

Initialization: output the weight vector p 1 = (1/K, . . . , 1/K) ∈ X K .

For each regime r ∈ N * ,

1. Set m (r) i = 1 2 r-1 2 r-1 t=1 i,t for all i ∈ {1, . . . , K};
2. At each time round t ∈ 2 r-1 + 1, . . . , 2 r , output the same weight vector p t p (r) = p (r) i

1 i K defined by p (r) i = exp - √ 2 r-1 m (r) i K j=1 exp - √ 2 r-1 m (r) j
, 1 i K .

(5.18) Figure 5.3: An internal-regret-minimizing strategy when the distribution Q of the loss vectors is unknown (cf. Theorem 5.2).

The proof of Theorem 5.2 is postponed to Appendix 5.A. It is a simple adaptation of that of Theorem 5.1. The only but important additional tool is a "backward weighted union-bound" carried out at the end of the proof.

Note that an explicit upper bound on the absolute constant c 0 can be computed at the end of the proof. However, since for the sake of clarity, we sometimes performed crude upper bounds, its value may be far from optimal.

Though Q is unknown, the bound of the above theorem is still independent of the ambient dimension K. Moreover, even if the deviation factor exp 2 2 ln(4/δ) above is much larger than the more standard factor T ln(3/δ) of Theorem 5.1, it is still small enough to yield a bound of order √ T in expectation. It suffices to integrate the above high-probability bound (see Section A.6) and to combine it with the lower bound of [START_REF] Stoltz | Incomplete information and internal regret in prediction of individual sequences[END_REF] to get the following.

Corollary 5.1 (Minimax rate of internal regret with i.i.d. loss vectors).

There exist absolute constants c 1 , c 2 , c 3 > 0 such that the following holds true. Let K 2 and T c 1 K 2 . Then, the minimax internal regret with i.i.d. loss vectors satisfies

c 2 √ T inf S sup Q∈M + 1 ([0,1] K ) E Q ⊗T max 1 i =j K T t=1 p i,t ( i,t -j,t ) c 3 √ T ,
where the infimum is taken over all strategies S = (p t ) t 1 of the forecaster, and where, in the last expectation, the loss vectors 1 , . . . , T are i.i.d. with common distribution Q.

The proof of Corollary 5.1 is postponed to Appendix 5.A. Again, explicit bounds on the absolute constants c 1 ,c 2 ,c 3 can easily be computed at the end of the proof, but their values have not been optimized.

Lower bound on the swap regret with individual sequences

In this section we prove a lower bound of order √ T K on the minimax swap regret with individual sequences. This lower bound solves a question left open in [START_REF] Blum | From external to internal regret[END_REF] -see below. It also highlights a major difference between external and swap regrets: contrary to external regret, swap regret is much harder to minimize with individual sequences than with i.i.d. losses -see Section 5.4.2.

Main result

The main result of this section is the following.

Theorem 5.3 (Lower bound on the minimax swap regret).

There exists an absolute constant c > 0 such that the following holds true. Let K 2 and T max 128c 2 K 5 , K . Then the minimax swap regret with individual sequences satisfies

inf S sup 1 ,..., T ∈[0,1] K T t=1 p t • t -min F ∈F K T t=1 p F t • t c √ T K ,
where the infimum is taken over all strategies S = (p t ) t 1 of the forecaster, where F K denotes the set of all functions from {1, . . . , K} to {1, . . . , K}, and where the transformed weight vector p F t is defined in (5.5). In particular, we prove the theorem for c = 1/ 16 128 ln(4/3) .

The above theorem solves an open problem stated in [BM07b, Section 9]. The latter authors already proved a lower bound of order √ T K but only in a weak sense:

• Their lower bound was stated in a randomized and adversarial setting for a quantity larger than the swap regret stricto sensu (which makes the lower bound easier to prove). Their adversarial setting is defined recursively as follows. The environment -or adversary -has a strategy: it chooses a sequence (π t ) t 1 of conditional probability distributions on [0, 1] K such that π t d t ( s , p s , I s ) s t-1 , p t is the law of t conditionally on the available data ( s , p s , I s ) s t-1 , p t . At each time t, the forecaster picks I t ∈ {1, . . . , K} at random such that, conditionally on the past data ( s , I s ) 1 s t-1 , the random variables I t and t are independent and I t = i with probability p i,t . The weight vectors p t are now measurable functions of ( s , I s ) 1 s t-1 , and the corresponding sequence of functions (p t ) t 1 is called a randomized strategy. Then, setting δ It (I {It=i} ) 1 i K , Theorem 9 of [START_REF] Blum | From external to internal regret[END_REF] provides a lower bound on the quantity

sup (πt) t 1 inf S rand E T t=1 δ It • t -min F ∈F K T t=1 δ F It • t ,
where the supremum is taken over all adversaries (π t ) t 1 , where the infimum is taken over all randomized strategies S, and where the expectation is taken with respect to all sources of randomness (i.e., ( t , I t ) 1 t T ). By Jensen's inequality and by the fact that E δ It ( s , I s ) s t-1 , t = p t , the above quantity is larger than

sup (πt) t 1 inf S rand E T t=1 p t • t -min F ∈F K T t=1 p F t • t sup Q∈M + 1 ([0,1] K ) inf S E Q T t=1 p t • t -min F ∈F K T t=1 p F t • t (5.19) = inf S sup 1 ,..., T ∈[0,1] K T t=1 p t • t -min F ∈F K T t=1 p F t • t , (5.20)
where the supremum in (5.19) is restricted 5 to the set of all probability distributions on [0, 1] KT (the corresponding expectation is taken with respect to ( 1 , . . . , T ) with joint distribution Q), and where (5.20) follows from minimax duality (cf. Theorem 5.4 of the next section).

Therefore, our lower bound is stronger than the one of [BM07b, Theorem 9]. It solves a question left open in [BM07b, Section 9]: the authors showed that, in the randomized setting described above, it was not possible to ensure a worst-case swap regret of εT in a number of rounds T sublinear in K. They asked whether such impossiblity result remained true in the distributional setting (i.e., in our own non-randomized setting, with p t instead of δ F It ), where the task of the forecaster seems easier. Our lower bound provides a positive answer: a worst-case swap regret of εT is only possible for T at least of the order of K/ε 2 .

• The lower bound √ T K/160 -1 of [BM07b, Theorem 9] is only stated for K T exp(K/288)/ √ K, therefore, only for rounds T that are sub-exponential in K. On the contrary, our lower bound holds for all T max 128c 2 K 5 , K . The proof of Theorem 5.3 is postponed to Appendix 5.A.2. We use the key equality (5.10) of Section 5.2.2 to rewrite the swap regret as a sum of K K/2 internal regrets on time subintervals of length T /K . The

√

T -lower bound on the internal regret of [Sto05, Theorem 3.3] then yields a lower bound on the swap regret of order K T /K = √ T K . We make this statement more precise by using techniques borrowed from [Sto05, Theorem 3.3]. Namely, we use a reduction to stochastic losses for which, at each time t, only two of them are small, and then use Pinsker's inequality. However, due to the larger complexity of swap regret, our analysis is more involved than for internal regret -see the construction by induction in Appendix 5.A.2.

A major difference with classical works on external regret

In this section we point out a major difference between external and swap regrets: contrary to external regret, swap regret is much harder to minimize with individual sequences than with i.i.d. losses6 . Indeed, on the one hand, all known lower bounds on the minimax external regret with individual sequences are proved with i.i.d. loss sequences (whose distribution may depend on the strategy of the forecaster). This is the case in the full information setting (see Section 2.3.2 in Chapter 2), but also in the bandit setting (cf. [ACBFS02, Theorem 5.1]), or in the label-efficient prediction setting (cf. [CBLS05, Theorem 13]).

On the other hand, there is a large gap between the minimax swap regret for individual sequences and the minimax swap regret for i.i.d. loss vectors. Indeed, as shown below, if the loss vectors are i.i.d. with common distribution Q ∈ M + 1 [0, 1] K , then the expected swap regret can be made as small as

√

T ln K (up to a constant factor) uniformly over all distributions Q ∈ M + 1 [0, 1] K . On the contrary, we proved in Theorem 5.3 that the minimax swap regret for individual sequences is at least of the order of √ T K: setting c 1/ 16 128 ln(4/3) , we showed that, for all K 2 and all T max 128c 2 K 5 , K ,

inf S sup 1 ,..., T ∈[0,1] K T t=1 p t • t -min F ∈F K T t=1 p F t • t c √ T K ,
where the infimum is taken over all strategies S = (p t ) t 1 of the forecaster. This lower bound was derived with piecewise i.i.d. loss vectors. Therefore, the lack of stationarity in the loss sequence deteriorates the ability of the forecaster to minimize his swap regret. This is in contrast with the external regret, for which arbitrary loss sequences are as easy to control as i.i.d. loss sequences.

Next we prove the aforementioned √ T ln K bound: we design a simple strategy whose expected swap regret is at most of the order of

√ T ln K uniformly over all distributions Q ∈ M + 1 [0, 1] K .
This upper bound is optimal (up to constant factors). Indeed, by the lower bound of order

√

T ln K on the external regret proved in Lemma 2.2 of Chapter 2 (which a fortiori implies a lower bound on the swap regret by (5.11)), we get that, for all K 1 and all T 40e/(2e + 1) ln K,

inf S sup Q∈M + 1 ([0,1] K ) E Q ⊗T T t=1 p t • t -min F ∈F K T t=1 p F t • t 2 2e + 1 e T ln K 5(2e + 1) , (5.21)
where the infimum is taken over all strategies S = (p t ) t 1 of the forecaster and where in the last expectation, the loss vectors 1 , . . . , T are i.i.d. with common distribution Q.

For the sake of simplicity, we first assume that the distribution Q is known in advance by the forecaster. In this case, we set m i E Q [ i,1 ] for all i = 1, . . . , K and consider the simple strategy that constantly outputs the Dirac probability distribution

p t = δ i * , 1 t T , where i * ∈ argmin 1 i K m i .
The following proposition indicates that this simple strategy, which we proved to be suboptimal for internal regret with i.i.d. loss vectors (cf. Section 5.3), is however sufficient to attain the optimal rate of swap regret.

Proposition 5.2. Let K 2 and T 1. Assume that the loss vectors t ∈ [0, 1] K , 1 t T , are drawn independently at random from a common known distribution

Q ∈ M + 1 ([0, 1] K ).
Then the swap regret of the constant strategy (δ i * ) t 1 described above satisfies

E Q ⊗T T t=1 δ i * • t -min F ∈F K T t=1 δ F i * • t T 2 ln K .
Proof: The proof uses arguments that are similar to the ones of Theorem 5.1, but is even simpler due to the simpler form of the weight vector δ i * . Therefore, we only sketch below the main lines.

First note that since

δ F i * = δ F (i * ) for all F ∈ F K , we get that min F ∈F K T t=1 δ F i * • t = min 1 j K T t=1 j,t
, so that the swap regret reduces to the external regret:

E Q ⊗T T t=1 δ i * • t -min F ∈F K T t=1 δ F i * • t = E Q ⊗T T t=1 i * ,t -min 1 j K T t=1 j,t
T m i * -min

1 j K T m j = 0 + E Q ⊗T max 1 j K T t=1 (m j -j,t ) √ (T /2) ln K ,
where the last inequality follows from the fact that min i a i -

min i b i max i (a i -b i ) for all (a i ) i , (b i ) i ∈ R K ,
and where the upper bound by (T /2) ln K follows from Hoeffding's inequality combined with an elementary maximal inequality for subgaussian random variables (cf. Lemmas A.5 and A.3 respectively in Appendix A.5). This concludes the proof.

We only stated a result in expectation. Note that a similar bound of the order of T ln(2K/δ) can be seen to hold true with probability at least 1 -δ. Moreover, if the distribution Q of the loss vectors is unknown to the forecaster, then we can also derive a bound of the order of T ln(2K/δ) by adapting the above strategy via a plug-in method based on a doubling trick -in the same spirit as in Section 5.3.2, except that the weights are much simpler here. More precisely, set p 1 = (1/K, . . . , 1/K) ∈ X K and set p t = p (r) for all t ∈ {2 r-1 + 1, . . . , 2 r }, r 1, where p (r) δ ir , with i r ∈ argmin

1 j K m (r) j = argmin 1 j K    1 2 r-1 2 r-1 t=1 j,t    .
Then, adapting the proof of Proposition 5.2 through the use a "backward weighted union-bound" as in the proof of Theorem 5.2, we could prove the following7 (the proof is omitted for the sake of concision). Proposition 5.3. Let K 2 and T 1. Assume that the loss vectors t ∈ [0, 1] K , 1 t T , are drawn independently at random from a common unknown distribution

Q ∈ M + 1 ([0, 1] K ).
Then, for some absolute constant c 4 > 0, the swap regret of the strategy defined above satisfies, for all δ > 0, with probability at least 1 -δ,

T t=1 p t • t -min F ∈F K T t=1 p F t • t c 4 T ln 2K δ .
Integrating the last upper bound (via Lemma A.7 in Appendix A.6) and combining it with the lower bound (5.21), we get the next corollary.

Corollary 5.2 (Minimax rate of swap regret with i.i.d. loss vectors).

There exist absolute constants c 5 , c 6 , c 7 > 0 such that the following holds true. Let K 2 and T c 5 ln K. Then, the minimax swap regret with i.i.d. loss vectors satisfies

c 6 √ T ln K inf S sup Q∈M + 1 ([0,1] K ) E Q ⊗T T t=1 p t • t -min F ∈F K T t=1 p F t • t c 7 √ T ln K ,
where the infimum is taken over all strategies S = (p t ) t 1 of the forecaster and where in the last expectation, the loss vectors 1 , . . . , T are i.i.d. with common distribution Q.

A stochastic technique for upper bounds with individual sequences

In this second part of the chapter, we develop a general stochastic technique to upper bound the minimax regret on arbitrary deterministic sequences. This technique is non-constructive but can be used for more general forms of regret than the ones studied before. It relies on a minimax duality theorem that enables to rewrite the minimax regret as a maximin regret where the loss vectors are random with a known joint distribution (see Sections 5.5.1 and 5.5.2 below). In Section 5.5.3 we then use this technique to recover known upper bounds on the external, internal and swap regrets. Finally, in Section 5.5.4, we derive a new upper bound of order √ T ln K on the makespan regret, thus improving on the known bound of order ln(K)

√ T of [EDKMM09].
As is detailed in Section 5.5.1, page 183, a similar technique has been independently studied in [START_REF] Rakhlin | Online learning: beyond regret[END_REF]. Since we work in a much more specific setting, we are able to get (sometimes tight) explicit constants. Our proofs rely on less involved arguments (e.g., the Bernoullization technique of [START_REF] Schlag | How to minimize maximum regret in repeated decision-making[END_REF] and an elementary maximal inequality for subgaussian random variables of [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF]).

We also stress that, though this stochastic technique is useful to better understand the problem at hand (since it provides an upper bound on the minimax regret), it is non-constructive. Designing explicit algorithms that achieve the obtained upper bounds is an important task to be addressed in the future (e.g., an efficient algorithm with a

√

T ln K makespan regret). Note that the same issue arises in [START_REF] Rakhlin | Online learning: beyond regret[END_REF].

Definitions and sketch of the stochastic technique

Next we introduce a generalized form of regret that includes as special cases the external, internal, and swap regrets, as well as the regret associated to global cost functions of [START_REF] Dar | Online learning for global cost functions[END_REF]. We then sketch the stochastic technique we use in the following sections to upper bound the minimax rate of such regrets on arbitrary deterministic sequences.

A generalized form of regret Definition 5.1 ((ψ, ϕ)-regret). Let E be a real vector space, let ψ = (ψ t ) t 1 be a sequence of convex functions ψ t : E → R, and let ϕ : R K × R K → E be a bi-affine function, that is, ϕ(u, •) and ϕ(•, v) are affine8 for all u, v ∈ R K .

Then, for any strategy S = (p t ) t 1 of the forecaster, we define its (ψ, ϕ)-regret after T time rounds on any loss sequence 1 , . . . , T ∈ [0, 1] K by ψ T T t=1 ϕ(p t , t ) .

Example 5.1 (External regret). In view of (5.1), external regret corresponds to E = R K , ψ t : R K → R, and ϕ : R

K × R K → R K defined by ψ t (x) = max 1 i K x i and by ϕ(u, v) = (u • v -v i ) 1 i K .
Example 5.2 (Internal regret). By (5.9), internal regret corresponds to E = R K×K , ψ t : R K×K → R, and ϕ : R

K × R K → R K×K defined by ψ t (x i,j ) i,j = max 1 i =j K x i,j and by ϕ(u, v) = u i (v i -v j ) 1 i,j K .
Example 5.3 (Swap regret). By (5.10), swap regret corresponds to E = R K×K , ψ t : R K×K → R, and ϕ : R

K ×R K → R K×K defined by ψ t (x i,j ) i,j = K i=1 max 1 j K x i,j and by ϕ(u, v) = u i (v i -v j ) 1 i,j K .
Example 5.4 (Online learning with global cost functions).

The framework of online learning with global cost functions recently introduced9 by [EDKMM09] can also be cast into our generalized setting. More precisely, let C : R K + → R be a convex function such that C * : R K + → R defined by

C * (x 1 , . . . , x K ) min α∈X K C(α 1 x 1 , . . . , α K x K )
is concave, and where

X K x ∈ R K + : K i=1 x i = 1
. Typical examples of C include the makespan (C(x 1 , . . . , x K ) = max i x i ) and the d-norm cost (C(x 1 , . . . , x K ) = ( i x d i ) 1/d ). Then, for any strategy S = (p t ) t 1 and any loss sequence 1 , . . . , T ∈ [0, 1] K , [START_REF] Dar | Online learning for global cost functions[END_REF] define the regret of S on ( 1 , . . . , T ) with respect to the global cost function C by

C 1 T T t=1 p 1,t 1,t , . . . , 1 T T t=1 p K,t K,t -C * 1 T T t=1 1,t , . . . , 1 T T t=1 K,t
.

This regret corresponds to the (ψ, ϕ)-regret when

E = R K × R K and when ψ t : R K × R K → R and ϕ : R K × R K → R K × R K are defined by ψ t (u, v) = C t -1 u -C * t -1 v and by ϕ(u, v) = (u i v i ) 1 i K , v .

Analysis from a stochastic viewpoint

In the sequel we derive upper bounds on the minimax (ψ, ϕ)-regret defined by

inf S sup 1 ,..., T ∈[0,1] K ψ T T t=1 ϕ(p t , t )
for at least all (ψ, ϕ) corresponding to either external, internal, swap, or makespan regret (see the examples above). The infimum is taken over all strategies S = (p t ) t 1 of the forecaster. We explain below how the minimax (ψ, ϕ)-regret -of deterministic nature -can be re-interpreted as a quantity involving random variables whose joint distribution is known and that are therefore easily manageable.

Step 1: Using minimax duality. The first step consists in using a minimax duality theorem (see Theorem 5.4 below) to exchange the infimum and the supremum in the sense that

inf S sup 1 ,..., T ∈[0,1] K ψ T T t=1 ϕ(p t , t ) = sup Q∈M + 1 ([0,1] KT ) inf S E Q ψ T T t=1 ϕ(p t , t ) ,
where the supremum in the right-hand-side is taken over all probability distributions on [0, 1] KT and where its expectation is taken with respect to random variables 1 , . . . , T ∈ [0, 1] K with joint distribution Q.

The left-hand-side quantity corresponds to a minimax game: the goal of the forecaster is to choose a strategy S whose worst-case regret is the smallest possible. In particular, the forecaster does not have any prior knowledge on the loss sequence to be dealt with. On the contrary, the right-hand-side quantity corresponds to a maximim game: the forecaster is first given the joint distribution Q of the future loss sequence ( 1 , . . . , T ) and then chooses a strategy S accordingly.

Step 2: Upper bounding the maximin regret. By the equality above, we can see that the two aforementioned games are equally difficult. Therefore, the minimax regret can be upper bounded through its maximin counterpart. The last quantity is generally easier to control: at each time t, the forecaster knows the distributions of all future losses conditionnally on the past (since he knows Q and the past loss vectors). Therefore, the forecaster can minimize a "conditional variant" of the (ψ, ϕ)-regret given by ψ T T t=1 ϕ p t , E t 1:t-1 .

Deviations of the true (ψ, ϕ)-regret from this "conditional variant" are often small enough; in our proofs, we will control them via standard martingale concentration arguments.

One simple strategy for dealing with this "conditional variant" consists in putting at each round t a unit mass at the index I * t minimizing the next expected loss (conditionnally on the past). This corresponds to the strategy S * (Q) = (p Q t ) t 1 defined by

p Q t δ I * t , with I * t ∈ argmin 1 i K E Q i,t 1:t-1 , (5.22)
where δ i ∈ X K denotes the Dirac distribution at i ∈ {1, . . . , K} and where 1:T is assumed to be drawn at random with joint distribution Q ∈ M + 1 [0, 1] KT . Note that p Q t depends on Q as suggested above. We will use S * (Q) later to revisit known upper bounds on external, internal, and swap regrets.

Comparison to the literature

The use of minimax duality to analyse the minimax rate of external regret was first exploited by [START_REF] Abernethy | A stochastic view of optimal regret through minimax duality[END_REF] and later by [START_REF] Rakhlin | Online learning: Random averages, combinatorial parameters, and learnability[END_REF]. The analysis of the aforementioned papers is generic enough to cover various loss functions but relies in a somewhat crucial way on the fact that the weight vectors p t appear additively in the external regret10 .

In the present chapter, we focus on the linear loss but extend the analysis of [START_REF] Abernethy | A stochastic view of optimal regret through minimax duality[END_REF] to other types of regret that do not satisfy this additivity property, e.g., the internal, swap, and makespan regrets. Such an extension has been independently carried out by [START_REF] Rakhlin | Online learning: beyond regret[END_REF] via the so-called "Triplex Inequality" and the control of "sequential Rademacher complexities". The setting considered in the last paper is much broader than ours: their analysis covers a wider sprectrum of cases and our notion of (ψ, ϕ)-regret resembles the regret of [START_REF] Rakhlin | Online learning: beyond regret[END_REF] in a particular situation that they called "when B is a function of the average" -see Section 3.2 therein. However, in our simpler setting, our analysis relies on related but simpler tools such as Bernoullization and an elementary maximal inequality for subgaussian random variables (cf. Lemma 5.1 below and Lemma A.3 in Appendix A.5). The Bernoullization step allows to resort directly to a version of von Neumann's minimax theorem without the need to write the minimax regret as a cumbersome sequence of multiple infima and suprema -this is in contrast with [AABR09, RST10, RST11]. As for the aforementioned maximal inequality, it replaces a Dudley-entropytype upper bound. The last tool is much more general, but it is unncessarily involved in our case since we only consider finite reference classes. Moreover, our approach yields explicit sharp constants that exactely recover the best constants known so far for the external, internal, and swap regrets.

A minimax theorem for the (ψ, ϕ)-regret

The next minimax duality theorem is the main result of this section. Theorem 5.4 (A minimax theorem for the (ψ, ϕ)-regret). Let E be a real vector space, let ψ = (ψ t ) t 1 be a sequence of convex functions ψ t : E → R, and let ϕ : R K × R K → E be a bi-affine function (cf. Definition 5.1). Then the (ψ, ϕ)-regret satisfies the following duality formula:

inf S sup 1 ,..., T ∈[0,1] K ψ T T t=1 ϕ(p t , t ) = sup Q∈M + 1 ([0,1] KT ) inf S E Q ψ T T t=1 ϕ(p t , t ) ,
where both infima are taken over all strategies S = (p t ) t 1 , where M + 1 [0, 1] KT denotes the set of all probability distributions on [0, 1] KT , and where the expectation E Q [ • ] is taken with respect to the random variables 1 , . . . , T ∈ [0, 1] K with joint distribution Q.

The proof of Theorem 5.4 consists of a careful application of a version of von Neumann's minimax theorem (stated as Lemma A.1 in Appendix A.3). To use it in a convenient way, we will first use the next technical lemma that enables a reduction to binary losses and relies on a technique due to [START_REF] Schlag | How to minimize maximum regret in repeated decision-making[END_REF]. With this reduction, the compacity and continuity assumptions of Lemma A.1 are then satisfied.

Lemma 5.1 (Bernoullization).

Let E be a real vector space, let ψ = (ψ t ) t 1 be a sequence of convex functions ψ t : E → R, and let ϕ : R K × R K → E be a bi-affine function (cf. Definition 5.1). Then the minimax (ψ, ϕ)-regret can be reduced to binary losses 1 , . . . , T ∈ {0, 1} K in the sense that

inf S sup 1 ,..., T ∈[0,1] K ψ T T t=1 ϕ(p t , t ) = inf S sup 1 ,..., T ∈{0,1} K ψ T T t=1 ϕ(p t , t ) ,
where both infima are taken over all strategies S = (p t ) t 1 .

Note that the above lemma would be immediate if the considered strategies S were static, i.e., such that p t ( 1:t-1 ) = q t for all t 1 and 1:t-1

( 1 , . . . , t-1 ) and for some fixed sequence (q t ) t 1 in X K . Indeed, the (ψ, ϕ)-regret of such strategies is convex on the polytope [0, 1] KT and thus achieves its supremum on the hypercube {0, 1} KT . The above lemma shows that, even for non-static strategies, the hypercube is in some sense sufficient to assess the performance of any strategy (see also Remark 5.2 below).

Proof (of Lemma 5.1): Our proof is based on a Bernoullization argument of [START_REF] Schlag | How to minimize maximum regret in repeated decision-making[END_REF] which we slightly simplify via the use of Jensen's inequality. Let U i,t , 1 i K, 1 t T , be independent real random variables uniformly distributed on [0, 1]. We set U t (U i,t ) 1 i K and U 1:t (U s ) 1 s t for all t ∈ {1, . . . , T }. In this proof, we write E U 1:t or E U t when the expectation is taken over U 1:t or U t respectively. To avoid any ambiguity, we also explicitly see the weights as functions p t : [0, 1] K (t-1) → X K of the past loss vectors 1:t-1 ( 1 , . . . , t-1 ), and hence write p t ( 1:t-1 ).

We first introduce the following key definitions. We associate with any deterministic loss sequence 1 , . . . , T ∈ [0, 1] K its randomly thresholded version 1 , . . . , T ∈ {0, 1} K defined for all t by i,t

I { i,t U i,t } , 1 i K.
Moreover, we associate with any strategy S = (p t ) t 1 its bernoullized variant S = ( p t ) t 1 defined for all 1 , . . . , T ∈ [0, 1] K by

p t ( 1:t-1 ) E U 1:t-1 p t 1:t-1 = E U 1:t-1 p t I { i,s U i,s } 1 i K 1 s t-1
.

(5.23) Thus, at each round t, the Bernoullized strategy S first transforms the past losses i,s , 1 i K, 1 s t -1, into independent Bernoulli random variables i,s with respective parameters i,s , then applies the function p t to them, and finally averages the result out.

As noted in Remark 5.2 after the present proof, for any strategy S, the Bernoullized variant S has a lower worst-case regret than S. However, computing p t ( 1:t-1 ) in practice requires to evaluate the function p t at the 2 K(t-1) vertices of the hypercube {0, 1} K(t-1) , so that S is only of theoretical interest.

To prove the lemma it suffices to show that

inf S sup 1 ,..., T ∈[0,1] K ψ T T t=1 ϕ p t ( 1:t-1 ), t inf S sup 1 ,..., T ∈{0,1} K ψ T T t=1
ϕ p t ( 1:t-1 ), t .

(5.24) First, restricting the infimum of the right-hand side to the set of all Bernoullized strategies S = ( p t ) t 1 , we get that

inf S sup 1 ,..., T ∈[0,1] K ψ T T t=1 ϕ(p t ( 1:t-1 ), t ) inf S sup 1 ,..., T ∈[0,1] K ψ T T t=1
ϕ p t ( 1:t-1 ), t .

(5.25) Let S = (p t ) t 1 and 1 , . . . , T ∈ [0, 1] K . The definition of S in (5.23) and the equality

E U t i,t = E U t I { i,t U i,t } = i,
t for all i = 1, . . . , K and t = 1, . . . , T yield

ψ T T t=1 ϕ p t ( 1:t-1 ), t = ψ T T t=1 ϕ E U 1:t-1 p t 1:t-1 , E U t t = ψ T T t=1 E U 1:t ϕ p t 1:t-1 , t (5.26)
= ψ T E U 1:T T t=1 ϕ p t 1:t-1 , t E U 1:T ψ T T t=1 ϕ p t 1:t-1 , t , (5.27)
where (5.26) follows from the fact that ϕ : R K ×R K → E is bi-affine (cf. Definition 5.1) and from Fubini's theorem (since U 1:t-1 and U t are independent), and where (5.27) follows from Jensen's inequality (since ψ T : E → R is convex). Note that all expectations above are actually taken over a finite number of points since 1:T ∈ {0, 1} KT almost surely. Therefore no additional assumption on the real vector space E was needed to ensure that these expectations are well-defined or to apply Fubini's theorem and Jensen's inequality.

Since 1:T ∈ {0, 1} KT , the expectation in (5.27) is upper bounded by a supremum over {0, 1} KT . Therefore we have proved that for all S = (p t ) t 1 ,

sup 1 ,..., T ∈[0,1] K ψ T T t=1 ϕ p t ( 1:t-1 ), t sup 1 ,..., T ∈{0,1} K ψ T T t=1
ϕ p t ( 1:t-1 ), t .

(5.28)

Combining the last inequality with (5.25) immediately yields (5.24). This concludes the proof.

Remark 5.2 (Bernoullization can only help).

Note from (5.28) above that the Bernoullized variant S of any strategy S performs always better than S in a worst-case sense. More precisely, since the weights p t ( 1:t-1 ) of S and the weights p t ( 1:t-1 ) of S coincide for all 1 , . . . , T ∈ {0, 1} KT , Inequality (5.28) is actually an equality:

sup 1 ,..., T ∈[0,1] K ψ T T t=1 ϕ p t ( 1:t-1 ), t = sup 1 ,..., T ∈{0,1} K ψ T T t=1 ϕ p t ( 1:t-1 ), t sup 1 ,..., T ∈[0,1] K ψ T T t=1 ϕ p t ( 1:t-1 ), t .
Therefore, the Bernoullization of any strategy can only improve its worst-case regret on [0, 1] KT . Even better, by the above equality, the worst-case regret of S on [0, 1] KT equals that of S on the restricted set {0, 1} KT and is therefore not influenced by the (potentially bad) performance of S outside of the hypercube {0, 1} KT . However, as mentioned earlier, the strategy S is unfortunately only of theoretical interest because of its exponential computational complexity.

Proof (of Theorem 5.4): The proof consists of a careful application of a version of von Neumann's minimax theorem. We first use a reduction to binary losses (via Lemma 5.1), then apply the aforementioned version of von Neumann's minimax theorem, and finally get back to [0, 1]-valued losses. To avoid any ambiguity, we write all dependencies p t ( 1:t-1 ) explicitely. By Lemma 5.1, we have

inf S sup 1 ,..., T ∈[0,1] K ψ T T t=1 ϕ(p t ( 1:t-1 ), t ) = inf S sup 1 ,..., T ∈{0,1} K ψ T T t=1 ϕ(p t ( 1:t-1 ), t ) = inf S sup Q∈M + 1 ({0,1} KT ) E Q ψ T T t=1
ϕ(p t ( 1:t-1 ), t )

F (Q,S)
(5.29)

= sup Q∈M + 1 ({0,1} KT ) inf S E Q ψ T T t=1 ϕ(p t ( 1:t-1 ), t ) , (5.30)
where in the last two equalities, the expectations E Q [ • ] are taken with respect to the random variables 1 , . . . , T ∈ [0, 1] K with joint distribution Q, and where we used the following arguments.

• (5.29) is elementary: the inequality " " follows by considering the Dirac probability distributions Q = δ ( 1 ,..., T ) for all ( 1 , . . . , T ) ∈ {0, 1} KT ; the inequality " " follows from the fact that any expectation is smaller than or equal to the supremum of its integrand.

• As for (5.30), the equality inf

S sup Q F (Q, S) = sup Q inf S F (Q, S)
F (Q, S) E Q ψ T T t=1 ϕ p t ( 1:t-1 ), t 11
We make a slight abuse of notation by denoting with the same symbols 1, . . . , T either random variables (with joint distribution Q) or fixed elements of {0, 1} K .

12 Recall that a strategy is a sequence S = (p t ) t 1 of Borel functions p t : [0, 1] K(t-1) → XK .

=

1:T ∈{0,1} KT Q { 1:T } ψ T T t=1 ϕ p t ( 1:t-1 ), t .
The aforementioned version of von Neumann's minimax theorem is recalled in Lemma A.1 (Appendix A.3). Its assumptions are immediately satisfied in this finite-dimensional setting: M + 1 ({0, 1} KT ) ≡ X 2 KT is a convex, Haussdorff, and compact subset of R 2 KT (under the Euclidean topology), S is clearly convex, and F satisfies:

for all S ∈ S, Q → F (Q, S) is linear on M + 1 ({0, 1} KT ) ≡ X 2 KT (and thus concave and continuous);

-for all Q ∈ M + 1 ({0, 1} KT ), S → F (Q, S) is convex on S since ψ T is convex and ϕ(•, v) is affine for all v ∈ R K .
We can thus apply Lemma A.1, which yields (5.30).

To conclude the proof, we get back to [0, 1]-valued losses by noting that

sup Q∈M + 1 ({0,1} KT ) inf S E Q ψ T T t=1 ϕ(p t , t ) sup Q∈M + 1 ([0,1] KT ) inf S E Q ψ T T t=1 ϕ(p t , t ) inf S sup Q∈M + 1 ([0,1] KT ) E Q ψ T T t=1 ϕ(p t , t ) = inf S sup 1 ,..., T ∈[0,1] K ψ T T t=1 ϕ(p t , t ) ,
where the second line follows from the standard inequality sup inf inf sup, and where the last equality follows from arguments similar to those used for (5.29). By (5.30), all the previous inequalities are equalities, which concludes the proof.

Remark 5.3 (Why use Bernoullization?).

Bernoullization enables a reduction to simple topological spaces -e.g., M + 1 ({0, 1} KT ) lies in a finite-dimensional space. In particular, there is no need here to use finer topological notions such as weak topology.

Rederivation of known bounds on external, internal, and swap regret

In this section we use the above minimax theorem to rederive known regret bounds on individual sequences from a stochastic viewpoint. A similar treatment will be carried out in the next section to derive a new bound on the makespan regret.

For all forms of regret considered below -e.g., external, internal, and swap regret -we use the distribution-dependent strategy S * (Q) defined in (5.22). Recall that this strategy assigns at each time t a unit mass at the random index

I * t ∈ argmin 1 i K E Q i,t 1:t-1 .
Let Φ ⊂ F K be a set of functions from {1, . . . , K} to {1, . . . , K}; we denote its cardinality by |Φ|. The next proposition provides an upper bound on the minimax Φ-regret defined by

inf S sup 1 ,..., T ∈[0,1] K T t=1 p t • t -min F ∈Φ T t=1 (p t ) F • t ,
where the infimum is taken over all strategies S = (p t ) t 1 of the forecaster and where (p t ) F is the weight vector induced by p t via the mapping F (cf. (5.5)). The notion of Φ-regret was introduced by [START_REF] Greenwald | A general class of no-regret learning algorithms and gametheoretic equilibria[END_REF]; it includes as special cases the external, internal, and swap regrets (see below). Though we are mainly interested in those three particular cases, our analysis is generic enough to cover the cases of all subsets Φ ⊂ F K (see also [START_REF] Rakhlin | Online learning: beyond regret[END_REF]).

Proposition 5.4 (Φ-regret from a stochastic viewpoint).

Let Q ∈ M + 1 [0, 1] KT .
Then, the strategy S * (Q) defined in (5.22) satisfies

E Q T t=1 p t • t -min F ∈Φ T t=1 (p t ) F • t T 2 ln |Φ| , (5.31)
where 1:T is drawn at random from the joint distribution Q ∈ M + 1 [0, 1] KT . As a consequence, the minimax Φ-regret on individual sequences satisfies

inf S sup 1 ,..., T ∈[0,1] K T t=1 p t • t -min F ∈Φ T t=1 (p t ) F • t T 2 ln |Φ| ,
where the infimum is taken over all strategies of the forecaster S = (p t ) t 1 .

Before proving the last proposition, note that, as a corollary, we can recover the best upper bounds known so far for the external, internal, and swap regrets.

• External regret corresponds to the transformation set Φ = {F i : i = 1, . . . , K}, where F i is defined by F i (k) = i for all k = 1, . . . , K. Since |Φ| = K, the above proposition entails that the minimax external regret is upper bounded by (T /2) ln K.

• Internal regret corresponds to the transformation set Φ = {F i,j : 1 i = j K}, where F i,j is defined by F i,j (k) = k for all k = i and by F i,j (i) = j. Since |Φ| = K(K-1) K 2 , the above proposition entails that the minimax internal regret is upper bounded by √ T ln K.

• Swap regret corresponds to the whole transformation set Φ = F K , whose cardinality equals K K . Therefore, by the above proposition, the minimax swap regret is upper bounded by (T /2)K ln K.

Remark 5.4. For the external regret, the stochastic viewpoint not only enables to get the optimal rate √ T ln K but also the asymptotically optimal constant 1/ √ 2 (cf. Remark 2.3 in Chapter 2, Section 2.3.2). As for the internal and swap regrets, the bounds proved above are the best known so far, and we know from (5.7) and Theorem 5.3 that they are rate-optimal up to a factor at most of the order of √ ln K (see also the next remark).

Remark 5.5. For the particular case of internal regret, we do not know yet whether the strategy S * (Q) is sufficient to get the optimal individual sequence rate (i.e., whether the missing √ ln K factor is necessary), but we do know from Section 5.3 that S * (Q) is suboptimal if the loss vectors are i.i.d.. Indeed, in the last case, a less aggressive strategy that appropriately spreads the probability mass among the actions achieves a √ T -upper bound (while S * (Q) does not). A natural question which should be addressed in the future is whether S * (Q) can be refined in the same spirit as in Section 5.3 to get an internal regret at most of the order of √ T on individual sequences (if such bound is possible). See Section 5.6 for some suggestions.

Proof (of Proposition 5.4): In the sequel we set ¯ i,t E Q i,t 1:t-1 for all i = 1, . . . , K and all t = 1, . . . , T (the dependence in Q is omitted). Since for all t = 1, . . . , T , the weight vector p Q t defined in (5.22) is measurable with respect to 1:t-1 , we get that

E Q T t=1 p Q t • t = T t=1 K i=1 E Q p Q i,t E Q i,t 1:t-1 = E Q T t=1 p Q t • ¯ t ,
where we set ¯ t = ( ¯ 1,t , . . . , ¯ K,t ). Using the last equality and the fact that min i a i -

min i b i max i (a i -b i ) for all (a i ) i , (b i ) i ∈ R K , we get that E Q T t=1 p Q t • t -min F ∈Φ T t=1 p Q t F • t E Q T t=1 p Q t • ¯ t -min F ∈Φ T t=1 p Q t F • ¯ t 0 a.s. + E Q max F ∈Φ T t=1 p Q t F • ¯ t -t . (5.32)
The first expectation of the right-hand side corresponds to what we called the "conditional variant" of the regret in Section 5.

p Q t • t = T t=1 min 1 i K ¯ i,t min F ∈Φ T t=1 p Q t F • ¯ t .
The last expectation of (5.32), which is a deviation term, can be upper bounded via a classical maximal inequality for subgaussian random variables that can be found, e.g., in [Mas07, Lemma 2.3 and Section 6.1.1] and that we recall in Appendix A.5. Note indeed that for all F ∈ Φ, the random sequence

(p Q t ) F • ( ¯ t -t ) t 1
is a martingale difference sequence with respect to the filtration generated by the t . Moreover, it takes its values in the predictable intervals [A t , A t + 1], where A t (p Q t ) F • ¯ t -1 (since the losses are [0, 1]-valued). Therefore, by the Hoeffding-Azuma inequality (cf. Lemma A.6 in Appendix A.5), the random variables T t=1 (p Q t ) F •( ¯ tt ), F ∈ Φ, are subgaussian with common variance factor v = T /4. Hence, by Lemma A.3 in Appendix A.5,

E Q max F ∈Φ T t=1 p Q t F • ¯ t -t 2 T 4 ln |Φ| = T 2 ln |Φ| .
Substituting the last inequality in (5.32), we get (5.31), which concludes the first part of the proposition. As a consequence, for all

Q ∈ M + 1 [0, 1] KT , inf S E Q T t=1 p Q t • t -min F ∈Φ T t=1 p Q t F • t T 2 ln |Φ| ,
where the infimum is taken over all strategies of the forecaster S = (p t ) t 1 . Therefore, to prove the second part of the proposition, it suffices to use the minimax duality result of Theorem 5.4 to get that

inf S sup 1 ,..., T ∈[0,1] K T t=1 p Q t • t -min F ∈Φ T t=1 p Q t F • t = sup Q∈M + 1 ([0,1] KT ) inf S E Q T t=1 p Q t • t -min F ∈Φ T t=1 p Q t F • t T 2 ln |Φ| .
This concludes the proof.

A new bound on the makespan regret

In this section, we derive a new bound on the minimax makespan regret. Following13 [START_REF] Dar | Online learning for global cost functions[END_REF] (see Example 5.4), we define the makespan regret of any strategy S = (p t ) t 1 on any loss sequence 1 , . . . , T ∈ [0, 1] K by max

1 i K T t=1 p i,t i,t -min q∈X K max 1 i K T t=1 q i i,t .
This notion of regret is useful, e.g., to model job scheduling or load balancing problems. In such settings, a decision-maker repeatedly distributes a job to K machines; p i,t denotes the proportion of the t-th job assigned to the i-th machine and i,t denotes the loss (or load) per job unit incurred by this machine at time t (so that the decision-maker incurs the weighted loss p i,t i,t on this machine). The goal of the decision-maker is to minimize the worst cumulative weighted loss max 1 i K T t=1 p i,t i,t over the K machines, and his performance is compared to that of the best static allocation q ∈ X K .

Using the same non-constructive stochastic viewpoint as in the previous subsection, we prove next that the minimax makespan regret is upper bounded by √ T + T ln(2K)/2. This improves on the bound of order ln(K)

√

T initially obtained by [START_REF] Dar | Online learning for global cost functions[END_REF] through an explicit algorithm. The design of an explicit algorithm with the better rate

√

T ln K should of course be addressed in the future.

We mention that a similar upper bound of the order of √ T ln K was derived independently by [START_REF] Rakhlin | Online learning: beyond regret[END_REF]; see Section 5.5.1, page 183, for further details.

We first define a new distribution-dependent strategy. Let Q ∈ M + 1 [0, 1] KT and set ¯ i,t E Q i,t 1:t-1 for all i = 1, . . . , K and t = 1, . . . , T (where 1:T has joint distribution Q). We associate with Q the strategy S mk (Q) whose weight vectors (p t ) t 1 are recursively defined by

p t ∈ argmin q∈X K max 1 i K t-1 s=1 p i,s ¯ i,s + q i ¯ i,t , 1 t T .
(5.33)

Note that S mk (Q) is a greedy-type strategy minimizing the makespan regret associated with the conditional losses ¯ i,s . But, by an elementary induction and by Lemma 5.2 in Appendix 5.B, we can see that, for all t = 1, . . . , T and all i = 1, . . . , K,

p i,t ¯ i,t = 1 K j=1 1/ ¯ j,t and p i,t = 1/ ¯ i,t K j=1 1/ ¯ j,t if ¯ i,t > 0 , (5.34)
with the convention that 1/0 = +∞, 1/(+∞) = 0, and x + (+∞) = +∞ for all x ∈ R + .

Proposition 5.5 (A new bound on the makespan regret).

Let Q ∈ M + 1 [0, 1] KT .
Then, the makespan regret of the strategy S mk (Q) defined in (5.33) satisfies

E Q max 1 i K T t=1 p i,t i,t -min q∈X K max 1 i K T t=1 q i i,t √ T + T ln(2K)/2 , (5.35)
where 1:T is drawn at random from the joint distribution Q ∈ M + 1 [0, 1] KT . As a consequence, the minimax makespan regret on individual sequences satisfies

inf S sup 1 ,..., T ∈[0,1] K max 1 i K T t=1 p i,t i,t -min q∈X K max 1 i K T t=1 q i i,t √ T + T ln(2K)/2 .
Proof: First note that, by subadditivity of the maximum, we can upper bound the makespan regret by the sum of its "conditional variant" and two deviation terms, i.e., almost surely,

max 1 i K T t=1 p i,t i,t -min q∈X K max 1 i K T t=1 q i i,t max 1 i K T t=1 p i,t ¯ i,t -min q∈X K max 1 i K T t=1 q i ¯ i,t
+ max

1 i K T t=1 p i,t i,t -¯ i,t + min q∈X K max 1 i K T t=1 q i ¯ i,t -min q∈X K max 1 i K T t=1
q i i,t .

(5.36)

Next we upper bound each of the three terms of the righ-hand side separately.

Term 1.

The first term is non-positive since, by (5.34),

max 1 i K T t=1 p i,t ¯ i,t = T t=1 1 K j=1 1/ ¯ j,t 1 K j=1 1/ T t=1 ¯ j,t = min q∈X K max 1 j K q j T t=1 ¯ j,t ,
(5.37) where the inequality above follows from Lemma 5.3 in Appendix 5.B applied to the vectors ( ¯ j,t ) 1 j K ∈ R K + , t = 1, . . . , T , and where the last equality follows from Lemma 5.2 in Appendix 5.B.

Term 2.

The second term is upper bounded by √ T in expectation. Indeed, by the elementary maximal inequality

E[max i Z i ] i E[Z 2 i ]
1/2 that holds for all integrable random vectors

(Z i ) 1 i K ∈ R K , we get that E Q max 1 i K T t=1 p i,t i,t -¯ i,t   K i=1 E Q   T t=1 p i,t i,t -¯ i,t 2     1/2 = K i=1 T t=1 E Q p 2 i,t i,t -¯ i,t 2 1/2
(5.38)

T t=1 E Q K i=1 p 2 i,t 1/2 √ T , (5.39)
where (5.38) follows from the Pythagorean theorem since, for every i = 1, . . . , K, the random sequence T t=1 p i,t i,t -¯ i,t 1 t T is a square-integrable martingale and therefore has orthogonal increments. As for (5.39)

+ → R by f (x) min q∈X K max 1 i K q i x i .
Since f is 1-Lipschitz continuous with respect to the infinity norm • ∞ (essentially because the minimum and maximum functions are also 1-Lipschitz continuous), the last term of (5.36) reads, almost surely,

min q∈X K max 1 i K T t=1 q i ¯ i,t -min q∈X K max 1 i K T t=1 q i i,t = f ( LT ) -f (L T ) LT -L T ∞ = max 1 i K T t=1 ¯ i,t -i,t .
Taking the expectations of both sides of the inequality above, we get, using again the Hoeffding-Azuma inequality and an elementary maximal inequality for subgaussian random variables (cf.

Lemmas A.6 and A.3 in Appendix A.5),

E Q min q∈X K max 1 i K T t=1 q i ¯ i,t -min q∈X K max 1 i K T t=1
q i i,t T 2 ln(2K) .

(5.40)

Putting everything together.

We conclude the proof of (5.35) by substituting the upper bounds (5.37), (5.39), and (5.40) in (5.36). As for the second part of the proposition, it follows again by using the minimax duality property of Theorem 5.4.

Future works

We recall that the present chapter is a work in progress, which raises some important questions. First, though the stochastic technique of Section 5.5 is useful to better understand the problem at hand (since it provides an upper bound on the minimax regret), it is non-constructive. Designing explicit algorithms that achieve the obtained upper bounds is an important task to be addressed in the future. For instance, is there any efficient algorithm with a makespan regret at most of order √ T ln K?

Another fundamental question that remains open is related to the missing logarithmic factor between the known lower and upper bounds on internal regret (of the order of

√

T and √ T ln K respectively). Is this logarithmic factor necessary or not? We proved that it is unncessary for i.i.d. loss vectors, we also recovered the best known upper bound

√

T ln K on individual sequences (with the best known constant) through a new viewpoint, but we still do not know whether the √ ln K factor is necessary for individual sequences. We briefly sketch below some ideas to tackle either the lower bound or the upper bound. (Note that both directions could be useful in case the order of magnitude of the minimax internal regret for individual sequences lies strictly between √ T and √ T ln K.) Note that similar questions arise about the minimax swap regret for individual sequences (the rate of which lies between √ T K and √ T K ln K). The next suggestions are however more suited for the internal regret.

= Q ⊗T , Q ∈ M + 1 ([0, 1] K )).
In that setting, it can indeed be refined through exponential weighting to yield a

√

T -upper bound. Is such an improvement also possible for all joint distributions Q on [0, 1] KT ? We could study a smooth variant of S * (Q) given, e.g., by the exponential weights p t = (p i,t ) 1 i K defined as

p i,t = e -c √ T ¯ i,t K j=1 e -c √ T ¯ j,t , 1 i K ,
where ¯ i,t E Q i,t 1:t-1 for all i ∈ {1, . . . , K} and t 1, and where c > 0 is an absolute constant. The above strategy generalizes that of Section 5.3 for i.i.d. loss vectors when T is known in advance (which is the case here since T is a parameter of the minimax rate).

Note that the weight vectors p t suggested above are no longer constant over time in general. This prevents from factorizing the internal regret by p i as we did in Section 5.3. Therefore, in this more general setting, the use of Bernstein's inequality for martingales (see [START_REF] Freedman | On tail probabilities for martingales[END_REF]) seems more appropriate than the Hoeffding-Azuma inequality. (Indeed, the conditional variances of the random variables p i,t ( i,tj,t ) should be taken into account to see that the deviations of T t=1 p i,t ( i,tj,t ) from T t=1 p i,t ( ¯ i,t -¯ j,t ) scale as

T t=1 p 2 i,t instead of √ T .)
Refinement of the √ T lower bound on internal regret?

Another direction consists in proving a larger lower bound (if such improvement is possible). In view of all known lower bounds on the external regret, it could be tempting to try to construct a suitable i.i.d. sequence (possibly depending on the strategy of the forecaster) for which the internal regret of the forecaster is at least of the order of T f (K) with f (K) → +∞ when K → +∞.

The results of Section 5.3 provide a negative answer by indicating that this is not possible (note that mixtures of i.i.d. sequences are banned as well). More sophisticated stochastic sequences could thus be studied in the future, e.g., piecewise i.i.d. Bernoulli sequences.

5.A Proofs

In this section we provide the proofs of Theorem 5.1, Theorem 5.2, and Corollary 5.1 (internal regret in a stochastic environment), as well as the proof of Theorem 5.3 (swap regret with individual sequences).

5.

A.1 Proofs related to internal regret in a stochastic environment Proof (of Theorem 5.1):

In the sequel we write p i = p int i (Q) for notational convenience. First note that max

1 i =j K T t=1 p i ( i,t -j,t ) max 1 i,j K T t=1 p i ( i,t -j,t ) = max 1 i K p i T t=1 i,t -min 1 j K T t=1 j,t , (5.41)
where the last equality follows from the fact that the weights p i are constant over time.

Next we bound with high probability T t=1 i,t from above and min 1 j K T t=1 j,t from below. Since for all i ∈ {1, . . . , K}, the i,t , 1 t T , are independent, [0, 1]-valued, and have common mean m i , Hoeffding's inequality (see Lemma A.5 in Appendix A.5) entails that ∀i ∈ {1, . . . , K}, ∀δ i ∈ (0, 1) ,

P T t=1 i,t -T m i > T 2 ln 2 δ i δ i .
Now, let δ ∈ (0, 1) and α 1 , . . . , α K > 0 such that K i=1 α i = 1 (the α i will be determined by the analysis). Combining the above inequality with a union bound, we get that, with probability at least 1

-K i=1 α i δ = 1 -δ, ∀i ∈ {1, . . . , K}, T m i - T 2 ln 2 α i δ T t=1 i,t T m i + T 2 ln 2 α i δ . (5.42)
Therefore, with probability at least 1 -δ, for all 1 i K, T t=1 i,t -min

1 j K T t=1 j,t T m i + T 2 ln 2 α i δ -min 1 j K T m j - T 2 ln 2 α j δ = 2T m i -T m i + T 2 ln 2 α i δ + max 1 j K -T m j + T 2 ln 2 α j δ 2T m i + 2 max 1 j K -T m j + T 2 ln 2 α j δ .
Substituting the last inequality in (5.41) and using the fact that m i = m i * + ∆ i by definition, we get that, with probability at least 1 -δ,

max 1 i =j K T t=1
p i ( i,tj,t ) max

1 i K p i 2T ∆ i + 2 max 1 j K -T ∆ j + T 2 ln 2 α j δ 2 √ T max 1 i K p i √ T ∆ i + 2 max 1 i K p i max 1 j K -T ∆ j + T 2 ln 2 α j δ , (5.43)
where in the last inequality, we used the subadditivity of the maximum and the fact the last maximum is nonnegative (since -T ∆ i * = 0). But, multiplying the numerator and the denominator of (5.17) by exp √ T m i * , the weights p i can be rewritten for all i ∈ {1, . . . , K} as

p i = e - √ T ∆ i K j=1 e - √ T ∆ j = e - √ T ∆ i K eff where K eff K j=1 e - √ T ∆ j ∈ [1, K] .
(5.44)

Next we combine (5.43) with (5.44). First note from (5.44) that p i

√ T ∆ i = K -1 eff e - √ T ∆ i √ T ∆ i so that p i √ T ∆ i K -1 eff sup x 0 e -x x = 1/(e K eff )
. Second, we get max 1 i K p i = 1/K eff from (5.44). Subtituting the last two upper bounds in (5.43), we get, with probability at least 1 -δ,

max 1 i =j K T t=1 p i ( i,t -j,t ) 2 √ T e K eff + 2 K eff max 1 i K -T ∆ i + T 2 ln 2 α i δ .
(5.45)

It turns out that a convenient choice of the α i is enough to get the claimed bound. Note that it is such that the lower confidence bounds of (5.42) on the quantities T t=1 i,t , 1 i K, are approximately equalized; see (5.47) below. More precisely, we set

α i e -2T ∆ 2 i K j=1 e -2T ∆ 2 j , 1 i K . (5.46)
Substituting the definition of the α i in (5.45), we get, with probability at least 1 -δ,

max 1 i =j K T t=1 p i ( i,t -j,t ) 2 √ T e K eff + 2 K eff max 1 i K      -T ∆ i + T 2 ln e 2T ∆ 2 i + T 2 ln   2 δ K j=1 e -2T ∆ 2 j        2 √ T e K eff + 1 K eff 2T ln   2 δ K j=1 e -2T ∆ 2 j   , (5.47)
where the last inequality follows from the elementary upper bound √ x + y √ x + √ y for all x, y 0 (so that -T ∆ i and T ∆ i cancel out). But note that for all 1 j K, we have

e -2T ∆ 2 j e - √ T ∆ j sup x 0 e -2x 2 +x = e - √ T ∆ j e 1/8 , so that K j=1 e -2T ∆ 2 j e 1/8 K j=1 e - √ T ∆ j = e 1/8 K eff .
(5.48)

Substituting the last inequality in (5.47), we get, with probability at least 1 -δ,

max 1 i =j K T t=1 p i ( i,t -j,t ) 2 √ T e K eff + 1 K eff 2T ln 2 e 1/8 K eff δ 2 e + √ 2 1 K eff T ln 3K eff δ 3 K eff T ln 3K eff δ ,
where the second inequality follows from the fact that 2 e 1/8 3 and that ln(3K eff /δ) 1 (since K eff 1 and δ 1). As for the last inequality, it follows from the elementary upper bound 2/e + √ 2 3.

We have just proved the first inequality of the theorem. The second one follows from the fact that the function x → x -1 ln(3x/δ) is nonincreasing 14 on 1, +∞ and from the inequality K eff 1. This concludes the proof.

Remark 5.6. The weighted union bound with (α 1 , . . . , α K ) in the previous proof is key to derive an upper bound of order √ T . It will also be useful in the case when Q is unknown -see below.

14 Indeed, the first derivative of x → x -2 ln(x) is equal to x -3 1 -2 ln(x) , which is non-positive on e 1/2 , +∞ .

Therefore, the function x → x -1 ln(3x/δ) is non-increasing on e 1/2 δ/3, +∞ and in particular on [1, +∞) (since e 1/2 δ/3 e 1/2 /3 1).

Proof (of Theorem 5.2): The proof uses the same key arguments as that of Theorem 5.1 -in particular, careful weighted union bounds are central to our analysis. Therefore, we omit some details already encountered in the previous proof and only stress the major changes (namely, we now need to control the deviations of the estimates m (r) i from their expectations m i and to deal with the change of regimes).

In the sequel, we set R log 2 T , t -1 0, t r 2 r for all r ∈ {0, . . . , R -1}, and t R T . Therefore {t r-1 + 1, . . . , t r } = {2 r-1 + 1, . . . , 2 r } ∩ [1, T ] for all 1 r R and we have the partition {1, . . . , T } = R r=0 {t r-1 + 1, . . . , t r }.

First, rewriting the sum T t=1 = R r=0 tr t=t r-1 +1 and using the subadditivity of the maximum, we get that, almost surely,

max 1 i =j K T t=1 p i,t ( i,t -j,t ) R r=0 max 1 i,j K tr t=t r-1 +1 p i,t ( i,t -j,t ) 1 + R r=1 max 1 i K    p (r) i   tr t=t r-1 +1 i,t -min 1 j K tr t=t r-1 +1 j,t      . (5.49)
To get the last equality, we upper bounded the summand at r = 0 by 1 (since the losses lie in [0, 1] and since t 0 -t -1 = 1), and we used for r 1 the fact that p i,t = p (r) i for all t ∈ {t r-1 +1, . . . , t r }.

Next we control the deviations of the sums tr t=t r-1 +1 i,t and of the estimators m (r) i around their expectations, uniformly over all r = 1, . . . , R. Let δ ∈ (0, 1), and fix β 1 , . . . , β R > 0 such that R r=1 β r 1. Fix also α 1,r , . . . , α K,r > 0 and α 1,r , . . . , α K,r > 0 such that K i=1 α i,r = 1 and K i=1 α i,r = 1 for all r = 1, . . . , R (the β r , α i,r , and α i,r will be determined by the analysis).

Following the same lines that led to (5.42), we get that, by Hoeffding's inequality and by several union bounds: on some event Ω δ of probability at least 1-δ, for all 1 r R and all

1 i K, τ r m i - τ r 2 ln 4 β r α i,r δ tr t=t r-1 +1 i,t τ r m i + τ r 2 ln 4 β r α i,r δ (5.50) and 2 r-1 m i - 2 r-1 2 ln 4 β r α i,r δ 2 r-1 t=1 i,t 2 r-1 m i + 2 r-1 2 ln 4 β r α i,r δ , (5.51)
where we set τ r t r -t r-1 . Note that, by definition of the t r , we have τ 0 = 1, τ r = 2 r-1 for all r ∈ {1, . . . , R -1}, and τ R = T -2 R-1 . In particular, τ r 2 r-1 for all 1 r R.

By (5.50) and by the same upper bounding that led to (5.43), we get, on Ω δ , for all r ∈ {1, . . . , R},

max 1 i K    p (r) i   tr t=t r-1 +1 i,t -min 1 j K tr t=t r-1 +1 j,t      2 √ τ r max 1 i K p (r) i √ τ r ∆ i + 2 max 1 i K p (r) i max 1 i K -τ r ∆ i + τ r 2 ln 4 β r α i,r δ 2 √ 2 r-1 max 1 i K p (r) i √ 2 r-1 ∆ i + 2 max 1 i K p (r) i τ r 2 ln   4 β r δ K j=1 e -2τr∆ 2 j   , (5.52)
where the last inequality follows from the fact that τ r 2 r-1 for all 1 r R and from a choice of α i,r similar to (5.46), i.e.,

α i,r e -2τr∆ 2 i K j=1 e -2τr∆ 2 j , 1 i K , 1 r R .
(For the moment, we do not upper bound τ r by 2 r-1 in the last term of (5.52); see the tighter bound in (5.56).)

Contrary to the proof of Theorem 5.1, the weights p

K (r) eff K j=1 exp - √ 2 r-1 3∆ j 2 , 1 r R .
(5.53)

For this purpose, we choose the α i,r in a way similar to the α i,r :

α i,r e -2 r-1 ∆ 2 i /2 K j=1 e -2 r-1 ∆ 2 j /2 , 1 i K , 1 r R .
By definition of m (r) i = 2 -(r-1) 2 r-1 t=1 i,t and of ∆ i m i -m i * , it is easy to see from (5.51) and from the choice of α i,r above that, on Ω δ , for all i ∈ {1, . . . , K},

m i * + ∆ i 2 -B r (δ) m (r) i m i * + 3∆ i 2 + B r (δ) ,
where we set

B r (δ) 1 2 r ln   4 β r δ K j=1 e -2 r-1 ∆ 2 j /2   .
Subtituting the last inequalities in the definition of p (r) i (cf. Figure 5.3) and using the definition of

K (r)
eff in (5.53), we get that, on Ω δ , for all i ∈ {1, . . . , K},

p (r) i exp 2 √ 2 r-1 B r (δ) exp - √ 2 r-1 ∆ i /2 K (r) eff
.

(5.54)

Before combining (5.52) with the above inequalities, note that, following the same lines that led to (5.48) and using the elementary equality sup x 0 e -x 2 /2+3x/2 = e 9/8 , we get the upper bound

K j=1 e -2 r-1 ∆ 2 j /2 e 9/8 K (r)
eff , so that, by definition of B r (δ) above,

2 √ 2 r-1 B r (δ) 2 ln 4 e 9/8 K (r) eff β r δ .
(5.55)

In the same way, the elementary equality sup x 0 e -2x 2 +3x/2 = e 9/32 yields the upper bound (5.56) where the last inequality follows from the bound τ r 2 r-1 and from the fact that x → x ln a b 1/x is nondecreasing on R * + for all a 1 and all b > 0 (note that 4K

eff /(β r δ) 1). Substituting the upper bounds (5.54), (5.55), and (5.56) in (5.52), we get that, on Ω δ , for all r ∈ {1, . . . , R},

max 1 i K    p (r) i   tr t=t r-1 +1 i,t -min 1 j K tr t=t r-1 +1 j,t      2 K (r) eff exp   2 ln 4 e 9/8 K (r) eff β r δ     √ 2 r-1 sup x 0 e -x/2 x + 2 r-1 2 ln 4 e 9/32 K (r) eff β r δ   c √ 2 r-1 exp 2 ln 4 β r δ ln 4 β r δ , (5.57)
where

c sup K 1    2 K exp 2 ln e 9/8 K   sup x 0 e -x/2 x + 1 √ 2 + ln e 9/32 K 2      .
Elementary manipulations show that c 2 e 3/2 2/e + 1/ √ 2 + 1 e 2 162. Substituting (5.57) in (5.49) and using the fact that e x x e 2x for all x ∈ R, we get that, on Ω δ , max

1 i =j K T t=1 p i,t ( i,t -j,t ) 1 + R r=1 c √ 2 √ 2 r-1 exp 2 2 ln 4 β r δ 1 + c √ 2 exp 2 2 ln 4/δ R r=1 √ 2 r-1 exp 2 2 ln 1/β r .
Recall that the β r are parameters of the analysis and can therefore be chosen at our own convenience. To avoid any extra exp √ ln T factor, we carry out a "backward weighted union bound" by choosing β r 6/π 2 (R -r + 1) 2 for all 1 r R, so that R r=1 β r 1. This yields the bound

R r=1 √ 2 r-1 exp 2 2 ln 1/β r exp 2 2 ln π 2 /6 R r=1 √ 2 r-1 exp 4 ln(R -r + 1) 8 √ 2 R R k=1 √ 2 -k exp 4 ln(k) c 0 <∞ 8c 0 √ 2T ,
where the second inequality follows from the change of variables k = R-r +1, and where the last inequality follows from the fact that R log 2 (T ) log 2 (T ) + 1. Combining the inequalities above, we conclude the proof by setting c 0 8 c c 0 < ∞.

Proof (of

E Q ⊗T max 1 i =j K T t=1 p i,t ( i,t -j,t ) c 3 √ T + 1 (c 3 + 1) √ T ,
where, using the constant c 0 of Theorem 5.2, we set c 3 c 0 exp 2 2 ln(4) e 16 + 1 (this constant can be improved). We conclude the proof by setting c 3 c 3 + 1.

5.A.2 Proof of the lower bound on the swap regret

Proof (of Theorem 5.3): In the sequel we first assume that T is a mutiple of K/2 max k ∈ N : k K/2 (the general case will follow by monotonicity of the minimax swap regret in Tsee the end of the proof). Under this assumption, we prove in what follows that

sup 1 ,..., T ∈[0,1] K T t=1 p t • t -min F ∈F K T t=1 p F t • t c √ 2T K .
(5.58)

We use the standard reduction to stochastic losses. Let S = (p t ) t 1 be any strategy of the forecaster. First note that

sup 1 ,..., T ∈[0,1] K T t=1 p t • t -min F ∈F K T t=1 p F t • t sup Q∈M + 1 [0,1] KT E Q T t=1 p t • t -min F ∈F K T t=1 p F t • t , (5.59)
where M + 1 [0, 1] KT denotes the set of all probability distributions on [0, 1] KT (endowed with its Borel σ-algebra) and where the loss vectors 1 , . . . , T ∈ [0, 1] K appearing inside the expectation

E Q [ • ]
are drawn at random with joint distribution Q.

Next we consider a finite family (Q γ ) γ of probability distributions on [0, 1] KT under which the random15 vectors t ∈ [0, 1] K are piecewise i.i.d.. The time interval {1, . . . , T } is divided into K/2 sub-intervals {t r-1 + 1, . . . , t r } such that t 0 0 < t 1 < . . . < t K/2 T and t r -t r-1 = T / K/2 for all r = 1, . . . , K/2 . Then, for all γ = (γ r ) 1 r K/2 ∈ {0, 1} K/2 , we define the probability distribution Q γ on [0, 1] KT such that, under Q γ :

• the (real-valued) losses i,t , 1 i K, 1 t T are independent;

• on each sub-interval {t r-1 + 1, . . . , t r } (1 r K/2 ), the loss vectors t ∈ [0, 1] K are i.i.d. and

     i,t = 1 a.s. if i / ∈ {2r -1, 2r} , i,t ∼ Ber(1/2 -γ r ε) if i = 2r -1 , i,t ∼ Ber(1/2 -(1 -γ r )ε) if i = 2r ,
where Ber(q) denotes the Bernoulli distribution with parameter q ∈ [0, 1] and where ε ∈ (0, 1/2) will be chosen by the analysis. We also set i r (γ) 2r -(1 -γ r ) and j r (γ) 2r -γ r , so that i r (γ), j r (γ) = {2r -1, 2r} and, for all t ∈ {t r-1 + 1, . . . , t r },

E Qγ ir(γ),t = 1/2, E Qγ jr(γ),t = 1/2 -ε, E Qγ k,t = 1, ∀k / ∈ {2r -1, 2r} . (5.60)
Note also that ∀t / ∈ {t r-1 + 1, . . . , t r }, ir(γ),t = jr(γ),t = 1 a.s.

(5.61)

Next we use an induction argument and Pinsker's inequality to show that, for at least one γ in the hypercube {0, 1} K/2 , the expected swap regret under Q γ is at least of the order of √ T K. The lower bound on individual sequences will then follow by (5.59). First note from the key equality (5.10) of Section 5.2.2 that, for all γ ∈ {0,

1} K/2 , E Qγ T t=1 p t • t -min F ∈F K T t=1 p F t • t = E Qγ K i=1 max 1 j K T t=1 p i,t ( i,t -j,t ) E Qγ   K/2 r=1 i∈{2r-1,2r} max 1 j K T t=1 p i,t ( i,t -j,t )   (5.62) E Qγ   K/2 r=1 tr t=t r-1 +1
p ir(γ),t ( ir(γ),tjr(γ),t )   .

(5.63) Inequality (5.62) follows from the fact that the pairs {2r -1, 2r}, r = 1, . . . , K/2 , are mutually disjoint subsets of {1, . . . , K} and from the nonnegativity of max 1 j K T t=1 p i,t ( i,tj,t ) for i = K (useful when K is odd). As for (5.63), we only kept for each r = 1, . . . , K/2 the term corresponding to i = i r (γ) (the other one being nonnegative) and used the fact that ir(γ),tjr(γ),t = 1 -1 = 0 a.s. for all t / ∈ {t r-1 + 1, . . . , t r } (by (5.61)).

But, by (5.60) and by conditioning on ( 1 , . . . , t-1 ), we have E Qγ p ir(γ),t ( ir(γ),tjr(γ),t ) = ε E Qγ p ir(γ),t . Substituting the last equality in (5.63), we get

E Qγ T t=1 p t • t -min F ∈F K T t=1 p F t • t ε K/2 r=1 tr t=t r-1 +1 E Qγ p ir(γ),t = ε K/2 r=1 tr t=t r-1 +1 E Qγ   1 -p jr(γ),t - k / ∈{2r-1,2r} p k,t   ε K/2 r=1 T K/2        1 - K/2 T tr t=t r-1 +1 E Qγ p jr(γ),t Ar,γ - K/2 T k / ∈{2r-1,2r} tr t=t r-1 +1 E Qγ [p k,t ] Br,γ        , (5.64)
where we used the fact that t r -t r-1 = T / K/2 for all r = 1, . . . , K/2 .

Next we show that, in non-trivial situations, A r,γ 3/4 and B r,γ 1/8 for an appropriate choice of γ ∈ {0, 1} K/2 . We start with B r,γ . By (5.60) and yet another use of the tower rule, we have, for all r ∈ 1, . . . , K/2 and all k / ∈ {2r -1, 2r}, 1 2

tr t=t r-1 +1 E Qγ [p k,t ] = E Qγ   tr t=t r-1 +1 p k,t k,t -ir(γ)   .
Following an argument of [Sto05, Theorem 3.3], note that the last expectation can be assumed to be smaller than c √ 2T K for all r ∈ 1, . . . , K/2 and all k / ∈ {2r -1, 2r}. Otherwise, the lower bound of (5.58) would follow straightforwardly by using (5.59) on the sub-interval {t r-1 + 1, . . . , t r } and by monotonicity 16 of the minimax swap regret in T . Therefore, we can assume that, for all r ∈ 1, . . . , K/2 ,

K/2 T k / ∈{2r-1,2r} tr t=t r-1 +1 E Qγ [p k,t ] K/2 T (K -2)2 c √ 2T K 2c 2 K 5 T 1 8 , (5.65)
where the last inequality follows from the assumption T 128c 2 K 5 .

As for the term A r,γ , we show in what follows that, by iteratively using Pinsker's inequality, there 16 To see why the minimax swap regret is nondecreasing in T , it suffices to show that the worst-case swap regret of any strategy S is nondecreasing in T . The latter fact is elementary by associating with each loss sequence ( 1, . . . , T ) the loss sequence ( 1, . . . , T , 0). exists γ ∈ {0, 1} K/2 such that, for all r ∈ 1, . . . , K/2 , K/2 T

⊗ Q ext I t = i ( 1 , I 1 ), . . . , ( t-1 , I t-1 ) = p i,t .
By the property above, (5.66) is equivalent to

K/2 T tr t=t r-1 +1 Q γ ⊗ Q ext I t = j r (γ) 3 4 , i.e., K/2 T tr t=t r-1 +1 Q (γ 1 ,...,γr,•) ⊗ Q ext I t = 2r -γ r 3 4 , (5.67)
where we used the definition of j r (γ) 2r -γ r , and where Q (γ 1 ,...,γr,•) denotes the joint distribution of ( 1 , . . . , tr ) (note that for all t t r , I t is measurable with respect to σ( 1 , . . . , t-1 )⊗B ext and a fortioti to σ( 1 , . . . , tr ) ⊗ B ext ).

Next we define γ * = γ * 1 , . . . , γ * K/2 ∈ {0, 1} K/2 such that the condition (5.67) holds for all r ∈ 1, . . . , K/2 . Fix

γ * 1 ∈ argmin γ 1 ∈{0,1} t 1 t=t 0 +1 Q (γ 1 ,•) I t = 2 -γ 1 ,
and, by induction,

γ * r ∈ argmin γr∈{0,1}    tr t=t r-1 +1 Q (γ * 1 ,...,γ * r-1 ,γr,•) ⊗ Q ext I t = 2r -γ r    .
The definition of γ * r above is motivated by the use of Pinsker's inequality -see (5.73) below. Let r = 1, . . . , K/2 . Then, using Pinsker's inequality at each t ∈ {t r-1 + 1, . . . , t r } (see Lemma A.8 in Appendix A.7), and averaging the resulting bounds, we get, for all γ r ∈ {0, 1},

K/2 T tr t=t r-1 +1 Q (γ * 1 ,...,γ * r-1 ,γr,•) ⊗ Q ext I t = 2r -γ r K/2 T tr t=t r-1 +1 P (γ * 1 ,...,γ * r-1 ,•) ⊗ Q ext I t = 2r -γ r + K/2 T tr t=t r-1 +1 Kr,t (γ r ) 2 , (5.68)
Combining it with (5.64) and (5.65), we finally get

E Q γ * T t=1 p t • t -min F ∈F K T t=1 p F t • t εT /8 .
Choosing ε = 8c 2K/T with c 1/ 16 128 ln(4/3) , we can check that 16T ln(4/3)ε 2 /K 1/16 and that ε 1/4 (since by assumption T 128c 2 K 5 (32c) 2 2K because K 4 2 4 ). This choice of ε yields the lower bound c √ 2T K under Q γ * , which in turn yields (5.58) by (5.59). This concludes the proof of the theorem when T is a multiple of K/2 . General case: We no longer assume that T is a multiple of K/2 . We use a reduction to the previous case. Denote by T ∈ N the largest multiple of K/2 smaller than or equal to T . Then, by monotonocity of the worst-case swap regret in T (see Footnote 16 on Page 202), we have

sup 1 ,..., T ∈[0,1] K T t=1 p t • t -min F ∈F K T t=1 p F t • t sup 1 ,..., T ∈[0,1] K T t=1 p t • t -min F ∈F K T t=1 p F t • t c √ 2T K ,
where the last inequality follows from the previous analysis (see (5.58)). We conclude the proof by noting that, by definition of T ,

T > T -K/2 T /2
(since K/2 T /2 by the assumption T K).

5.B Elementary lemmas

The first lemma of this section follows from elementary manipulations and can be found, e.g., in [EDKMM09, Lemma 3].

Lemma 5.2. Let K 1 and x 1 , . . . , x K ∈ R + . Then, using the conventions 1/0 = +∞, 1/(+∞) = 0, and x + (+∞) = +∞ for all x ∈ R + , we have

min q∈X K max 1 i K q i x i = 1 K i=1 1/x i .
Moreover, for every minimizer q ∈ X K of the above expression, we have, for all i = 1, . . . , K,

q i x i = 1 K j=1 1/x j so that q i = 1/x i K j=1 1/x j if x i > 0 .
The second lemma can be obtained from elementary calculations or directly seen as a consequence of the concavity of (u 1 , . . . , u K ) ∈ (R * + ) K → K j=1 1/u j -1 proved, e.g., in [EDKMM09,

Lemma 22]. It indicates that the harmonic mean is superadditive.

Introduction

In this section we briefly introduce our framework and our statistical procedure. We also discuss our main contributions and some related works. For the sake of clarity, we omit some technical details, which are postponed to Section 6.2.

We consider the generalized linear Gaussian framework introduced in [BM01a], i.e., one observes the whole stochastic process Y ε (t) t∈H given by

Y ε (t) = <s, t> + εW (t) , t ∈ H , (6.1)
where H, <•, •> is some separable Hilbert space, where W is an isonormal process on H (i.e., an isometry from H onto a centered Gaussian space 1 ), where the noise level ε > 0 is assumed to be known, and where s ∈ H is the unknown vector to be estimated. See Examples 6.1 and 6.2.

To estimate the unknown vector s, the statistician is given a family of least-squares estimators associated with different models, and his goal is to mimic the best of them. More precisely, following the same lines as [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF], we fix some at most countable collection (S m ) m∈M of nonempty subsets of H, which will be refered to as the models thereafter. For each m ∈ M, the closest point to s in S m (when it exists) is s m ∈ argmin t∈Sm t -s 2 = argmin t∈Sm t 2 -2<s, t> . Therefore, a natural estimator of s within the model S m is a least-squares estimator 2 s m ∈ S m defined by s m ∈ argmin t∈Sm γ ε (t) , where γ ε (t) t 2 -2Y ε (t) . (6.2)

In this setting, a natural goal is to construct an estimator s ∈ H of s which is almost as good as the best least-squares estimator in the family s m m∈M . This is the case, when, e.g., s satisfies a risk bound of the form

E s s -s 2 C inf m∈M E s s m -s 2 , (6.3)
where E s denotes the expectation with respect to the law of Y ε (t) t∈H (which depends on the unknown vector s), and where C 1 is a constant.

Example 6.1 (Gaussian regression framework with fixed design). In this setting, we observe

Y i = s i + σξ i , 1 i n ,
where the ξ i are independent standard normal random variables, where the noise level σ > 0 is assumed to be known, and where s ∈ R n is the vector to be estimated. As explained in [START_REF] Birgé | Gaussian model selection[END_REF], this setting corresponds to H = R n endowed with <u, v> = n -1 n i=1 u i v j , together with Y (t) = <Y, t>, W (t) = √ n <ξ, t>, s = (s 1 , . . . , s n ), and ε = σ/ √ n, where Y (Y 1 , . . . , Y n ) and ξ (ξ 1 , . . . , ξ n ).

1 Equivalently, W = W (t) t∈H is a family of real random variables such that, for all p 1 and all t1, . . . , tp ∈ H, the random vector W (t1), . . . , W (tp) is Gaussian with zero mean and covariance matrix <ti, tj> 1 i,j p .

2 In this introduction we assume that a least-squares estimator sm exists for all m ∈ M. The standard extension to approximate least-squares estimators is presented in Section 6.2.2.

Note that in this setting, the least-squares criterion can be rewritten as γ ε (t) = t 2 -2<Y, t> = Y -t 2 -Y 2 . Therefore, the least-squares estimators take the more standard form:

s m ∈ argmin t∈Sm Y -t 2 .
Example 6.2 (White noise framework). In this setting, we observe the whole path of the stochastic process ζ ε (u) 0 u 1 defined by

ζ ε (u) = u 0 s(x)dx + εB(u) ,
where B(u) u 0 is a standard Brownian motion, and where s ∈ L 2 ([0, 1], dx) is an unknown square-integrable function on [0, 1]. As noted in [START_REF] Birgé | Gaussian model selection[END_REF], this setting corresponds to H = L 2 ([0, 1], dx), W (t) = 1 0 t(x)dB(x), and Y ε (t) = 1 0 t(x)dζ ε (x) provided that H is equipped with the usual inner product <s, t> = 1 0 s(x)t(x)dx.

Model selection

One way to obtain risk bounds of the form (6.3) is to employ the celebrated model selection via penalization procedure of [START_REF] Birgé | Gaussian model selection[END_REF][START_REF] Massart | Concentration Inequalities and Model Selection[END_REF]. In the setting described above, this procedure chooses the estimator s = s m , where m minimizes some penalized least-squares criterion over M. When the models S m are finite-dimensional linear subspaces of H, and for an appropriately well-chosen penalty function, the estimator s = s m is shown to satisfy a risk bound of the form (6.3) with a constant C 1 depending on the collection (S m ) m∈M (C can be large if there are many models of the same dimension). Following the terminology of [START_REF] Donoho | Ideal spatial adaptation by wavelet shrinkage[END_REF], the latter risk bound is called an oracle inequality, since it indicates that the estimator s = s m mimics the oracle s m * (s) , where m * (s) ∈ argmin m∈M E s s m -s 2 corresponds to the unknown best model in (S m ) m∈M .

First consider the case where all the models S m are finite-dimensional linear subspaces of H (in the sequel, such models are refered to as linear models). As detailed in [START_REF] Birgé | Gaussian model selection[END_REF][START_REF] Massart | Concentration Inequalities and Model Selection[END_REF], this includes as important examples the problems of variable selection, curve estimation, and change points detection. In this setting [START_REF] Birgé | Gaussian model selection[END_REF] associate with each model S m a "weight" L m 0 such that Σ m∈M:Dm>0 e -LmDm < ∞, where D m denotes the dimension of the linear space S m . They then select the model m as a minimizer of a penalized least-squares criterion:

m ∈ argmin m∈M γ ε ( s m ) + pen(m) , where pen(m) Kε 2 D m 1 + 2L m 2
for some constant K > 1 to be chosen by the statistician. Note that the lower bound on the penalty pen(m) is proportional to the dimension D m . As shown in [START_REF] Birgé | Gaussian model selection[END_REF], for some constant C K > 1 depending only on K, the selected estimator s m satisfies the oracle-type inequality

E s s m -s 2 C K inf m∈M s m -s 2 + pen(m) + (Σ + 1) ε 2 (6.4) C K,Σ 1 + sup m∈M L m inf m∈M E s s m -s 2 , ( 6.5)
where the last inequality holds true if pen(m) = Kε 2 D m 1 + √ 2L m 2 , and where C K,Σ > 1 is a constant depending only on K and Σ. In particular (6.5) yields an oracle inequality of the form (6.3) if we can choose the L m such that sup m L m < ∞ and Σ m∈M:Dm>0 e -LmDm < ∞. Examples include the problems of ordered variable selection (where the L m can be chosen such that sup m L m is independent of the number of variables) and of complete variable selection (where an optimal choice of the L m is such that sup m L m scales as the logarithm of the number of variables); see Section 6.4.1 for further details. Another important consequence of (6.4) is that, as shown in [START_REF] Birgé | A generalized Cp criterion for Gaussian model selection[END_REF], the model selection procedure s m can be used to perform adaptive estimation in an (approximately) minimax sense for various problems (e.g., variable selection, curve estimation).

The case where the models S m are not necessarily linear was addressed by [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF] via a notion of generalized dimension D m (defined through a suitable weighted empirical process -see Section 6.2.2). The family of models (S m ) m∈M is associated with "weights" x m 0 such that Σ m∈M e -xm < ∞. The selected model m is then defined by

m ∈ argmin m∈M γ ε ( s m ) + pen(m) , where pen(m) Kε 2 D m + √ 2x m 2
for some constant K > 1 to be chosen by the statistician. As shown in [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF], the selected estimator s m satisfies the risk bound

E s s m -s 2 C K inf m∈M s m -s 2 + pen(m) + ε 2 (Σ + 1) .
The above risk bound cannot be rewritten in the form (6.3) in general. It is thus called an oracletype inequality. Note however that it still leads to adaptivity properties (see, e.g., [Mas07, Section 4.4.2] about adaptation to Besov ellipsoids). In this chapter, we derive risk bounds of the same form but for a more general procedure based on exponential weighting.

Some references on model selection

The model selection via penalization approach of [START_REF] Birgé | Gaussian model selection[END_REF] was inspired from the pioneering paper [START_REF] Barron | Minimum complexity density estimation[END_REF]. It was first introduced by [START_REF] Birgé | From model selection to adaptive estimation[END_REF] in the context of density estimation and later developed in [START_REF] Barron | Risk bounds for model selection via penalization[END_REF][START_REF] Birgé | Gaussian model selection[END_REF] for the density estimation and regression problems. Several extensions and refinements were later addressed, e.g., by [START_REF] Baraud | Model selection for regression on a fixed design[END_REF] in non-Gaussian settings, by [START_REF] Baraud | Model selection for regression on a random design[END_REF][START_REF] Birgé | Model selection for Gaussian regression with random design[END_REF] for the random-design case, by [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF][START_REF] Arlot | Data-driven calibration of penalties for least-squares regression[END_REF] for the study of minimal penalties and the corresponding slope's heuristics, by [START_REF] Baraud | Gaussian model selection with an unknown variance[END_REF] in the case of an unknown variance, and by [START_REF] Baraud | Estimator selection in the Gaussian setting[END_REF] for the wider problem of estimator selection. We refer the reader to [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF] for a comprehensive introduction to the topic. Detailed historical references can also be found in [START_REF] Mcquarrie | Regression and Time Series Model Selection[END_REF].

Aggregation: main contributions and related works

In this chapter we are interested in the connections between model selection and aggregation, in the same vein as in [START_REF] Foster | Calibration and empirical Bayes variable selection[END_REF] and, more recently, [START_REF] Leung | Information theory and mixing least-squares regressions[END_REF]. The key idea, already pointed out in [BM01a, Section 3.4], is to interpret the selected model m in Theorem 6.1 as the mode of some posterior probability distribution in a Bayesian context3 . More precisely, putting all convergence issues aside (see Section 6.3), we have

m ∈ argmax m∈M ρ (η) m ,
where, for some inverse temperature parameter η > 0, the posterior probability distribution ρ

(η) = ρ (η)
m m∈M is defined by

ρ (η) m = exp -η γ ε ( s m ) + pen(m) m ∈M exp -η γ ε ( s m ) + pen(m )
, m ∈ M . (6.6)

In this chapter, we consider the following Bayesian variant of the model selection procedure of [START_REF] Birgé | Gaussian model selection[END_REF][START_REF] Massart | Concentration Inequalities and Model Selection[END_REF]. Instead of estimating s with s m where m is the mode of ρ (η) , we estimate s with the convex combination of the estimators s m given by ρ (η) , i.e., with the estimator

s (η) = m∈M ρ (η) m s m .
(6.7) (In the sequel, we also allow pen to depend on η.)

Aggregation (or mixing) via exponential weighting has now quite a long history in both the machine learning and the statistical literatures. In machine learning, the exponentially weighted average forecaster has received a considerable attention from the seminal works [START_REF] Vovk | Aggregating strategies[END_REF][START_REF] Littlestone | The weighted majority algorithm[END_REF] to more recent parameter-tuning-oriented papers such as [START_REF] Cesa-Bianchi | Improved second-order bounds for prediction with expert advice[END_REF]; see Chapter 2 and [START_REF] Cesa-Bianchi | Prediction, Learning, and Games[END_REF] for an introduction to the subject.

As for the statistical literature, progressive mixture rules4 for the regression model with random design have been thoroughly studied by [START_REF] Catoni | Universal aggregation rules with exact bias bounds[END_REF][START_REF] Catoni | Statistical learning theory and stochastic optimization[END_REF] and later, e.g., by [Yan00, Yan01, Yan03, Yan04] and [START_REF] Audibert | Progressive mixture rules are deviation suboptimal[END_REF]. In this batch i.i.d. setting, aggregation via exponential weighting can also be carried out in a non-sequential way, i.e., by computing the exponential weights only once, with the whole sample -as in (6.6)-(6.7). Versions of such procedures were first analysed under the name of Gibbs estimators by [START_REF] Catoni | Statistical learning theory and stochastic optimization[END_REF], where they are proved to satisfy sharp oracle-type inequalities (i.e., with a leading constant equal to 1). Subsequent contributions include, among other papers, [START_REF] Audibert | Aggregated estimators and empirical complexity for least square regression[END_REF] and [START_REF] Alquier | PAC-Bayesian bounds for randomized empirical risk minimizers[END_REF][START_REF] Alquier | PAC-Bayesian bounds for sparse regression estimation with exponential weights[END_REF]. Most results of the aforementioned works are obtained for families of base estimators that are deterministic -or random, but independent of the sample used for the aggregation task (the last situation corresponds to the so-called sample splitting trick).

In the regression framework with fixed design (cf. Example 6.1), sharp PAC-Bayesian oracletype inequalities for deterministic base estimators were derived by [START_REF] Dalalyan | Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity[END_REF] under weak assumptions on the noise distribution. However, if the aggregated estimators are random and constructed on a fraction of the whole sample, it is not possible in general to combine them via the remaining subsample so as to ensure a low expected empirical squared error on the whole sample (since the two subsamples are no longer identically distributed). In a word, the sample splitting trick is not appropriate in this setting.

The model selection via penalization procedure of [START_REF] Birgé | Gaussian model selection[END_REF] is a way to overcome this limitation: the selected model m is chosen as a function of the same sample on which the leastsquares estimators s m are constructed. Another way is to use the Bayesian variant introduced above. A key contribution in this respect was carried out by [START_REF] Leung | Information theory and mixing least-squares regressions[END_REF] when the (finite) family of estimators consists of least-squares estimators on linear models S m (still in the framework of Example 6.1). Using our notations, and denoting by D m the dimension of S m , their procedure is of the form of (6.6)-(6.7) with pen(m) = 2D m σ 2 /n + x m /η, where the x m are such that Σ m∈M e -xm < +∞. Then, [START_REF] Leung | Information theory and mixing least-squares regressions[END_REF] show that for all η n/(4σ 2 ), the estimator s (η) defined in (6.6)-(6.7) satisfies the sharp oracle-type inequality

E s m∈M ρ (η) m s m -s 2 inf m∈M E s s m -s 2 + x m η + ln Σ η . (6.8)
The above risk bound has proved useful, e.g., in high-dimensional linear regression. As shown by [LB06, RT11, AL11], for a proper choice of the prior weights e -xm /Σ, this risk bound leads to sharp sparsity oracle inequalities without any assumption on the dictionary at hand (cf. Chapter 2, Section 2.6).

First contribution: aggregation of nonlinear models

The risk bound (6.8) of [START_REF] Leung | Information theory and mixing least-squares regressions[END_REF] was derived under the assumption that the models S m ⊂ R n are linear and that the s m are the associated least-squares estimators (i.e., the orthogonal projections of Y ∈ R n on the S m ). This work was further extended in two directions. On the one hand, the case of an unknown variance was addressed by [START_REF] Giraud | Mixing least-squares estimators when the variance is unknown[END_REF]. On the other hand, [START_REF] Dalalyan | Optimal aggregation of affine estimators[END_REF] replaced the family of projection estimators s m m∈M with an arbitrary family of affine estimators; this wider class of estimators includes, e.g., diagonal filters, kernel ridge regression, and multiple kernel learning.

In this chapter, we extend the work of [START_REF] Leung | Information theory and mixing least-squares regressions[END_REF] in a third direction: we still consider projection estimators, but the models S m ⊂ R n can be almost arbitrary (or nonlinear). (Actually, we consider the more general case S m ⊂ H, where H is possibly infinite-dimensional). In such generality, the use of the key Stein's unbiased risk formula of [START_REF] Stein | Estimation of the mean of a multivariate distribution[END_REF] as carried out in [LB06, DT08, DS11] seems difficult. Instead we follow the concentration approach of [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF] to derive oracle-type inequalities with high probability (with, however, leading constants larger than 1).

Second contribution: continuum of estimators from model aggregation to model selection

This work exhibits a natural connection between model aggregation and model selection: our oracle-type inequalities hold for a continuum of estimators s (η) : η > 0 ranging from classical model aggregation (where η is at most of the order5 of 1/ε 2 ) to model selection (where η = +∞).

In particular, for an appropriate choice of the penalty, our aggregating estimator s (η) converges almost surely to the selected estimator s m as η → ∞ (if m is unique); see Corollary 6.1. This fact is not surprising since our analysis is mostly based on the arguments of [Mas07, Theorem 4.18]the main change, however, consists in using a key duality formula for the Kullback-Leibler divergence instead of the definition of m. The present chapter thus shows that the analysis of [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF] for nonlinear models can be extended to arbitrary values of η > 0, instead of just taking η = +∞.

As of now, we do not know whether aggregation outperforms model selection for classical nonlinear models (e.g., Besov ellipsoids, 1 -balls, neural networks). However, our risk bounds suggest that it might be the case because of the presence of a Jensen-type nonnegative term -see Corollary 6.2. Another reason is that, even in the case of linear models, there are situations where aggregation is more robust than model selection in terms of excess risks -see Section 6.4.2 for a lower bound on model selection procedures. This lower bound suggests that aggregation might benefit from a similar advantage with nonlinear models.

Outline of the chapter

This chapter is organized as follows. In Section 6.2 we formally describe our statistical framework together with the model-selection and aggregation procedures at hand. In Section 6.3 we prove our main oracle-type inequality for aggregation of nonlinear models. Then, in Section 6.4, we derive several corollaries in classical examples and explain in which situations convexification may be useful (as compared to model selection). Finally some important questions raised by this work in progress are stated in Section 6.5.

Framework and statistical procedures at hand

In this section we recall the framework mentioned in the introduction and give a precise description of the collection of models at hand. Then we recall with our notations the main theorem of [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF] for model selection with nonlinear models. Finally we define our aggregation procedure.

The generalized linear Gaussian framework

We consider the generalized linear Gaussian framework introduced in [BM01a], i.e., one observes the whole stochastic process Y ε (t) t∈H given by Y ε (t) = <s, t> + εW (t) , t ∈ H , (6.9) where H, <•, •> is some separable Hilbert space, where W is an isonormal process on H (i.e., an isometry from H onto a centered Gaussian space6 ), where the noise level ε > 0 is assumed to be known, and where s ∈ H is the unknown vector to be estimated.

In the sequel, P s denotes the law of Y ε (t) t∈H (which depends on the unknown vector s), and E s denotes the corresponding expectation.

Collection of models

We follow below the same lines as in [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF]. Fix some at most countable collection (S m ) m∈M of subsets of H, which will be refered to as the models thereafter. We assume that for every m ∈ M, there exists some almost-surely continuous version of the isonormal process W on the closure S m of S m , still denoted by W .

[Mas07] associates with each model S m a generalized dimension D m defined as follows (see Section 6.4.1 for some examples). We assume that for all m ∈ M, their exists a nondecreasing continuous function ϕ

m : [0, +∞) → R + such that x → x -1 ϕ m (x) is nonincreasing on R * + and 7 2 E sup t∈Sm W (t) -W (u) t -u 2 + x 2 x -2 ϕ m (x) , (6.10)
for all x > 0 and all u ∈ S m . We let τ m = 1 if S m is closed and convex and τ m = 2 otherwise. Under the assumptions above, we associate with each model S m a generalized dimension D m 0 defined by:

• if ϕ m ≡ 0, then D m 0;

• if ϕ m (x 0 ) > 0 for some x 0 0, then D m is the unique positive solution of the equation

ϕ m τ m ε D m = εD m . (6.11)
Legitimate definition: Next we explain why D m is well-defined when there is x 0 0 such that ϕ m (x 0 ) > 0. Note that (6.11) has a unique solution D m > 0 if and only if the equation x -2 ϕ m (x) = 1/(τ 2 m ε) has a unique solution x > 0 (by the change of variables

x = τ m ε √ D m ).
But, since x → x -1 ϕ m (x) is nonincreasing on (0, x 0 ] and since ϕ m is nondecreasing on [x 0 , +∞), we can see that ϕ m (x) > 0 for all x > 0.

Therefore, x → x -2 ϕ m (x) is a product of two continuous, positive, and nonincreasing functions x → x -1 ϕ m (x) and x → x -1 , one of them being decreasing. The function x → x -2 ϕ m (x) is thus continuous and decreasing on (0, +∞). Since in addition

lim x→0 x -2 ϕ m (x) = +∞ and lim x→+∞ x -2 ϕ m (x) = 0 (because x → x -1 ϕ m (x)
is positive and nonincreasing), we get from the intermediate value theorem that the equation x -2 ϕ m (x) = 1/(τ 2 m ε) has a unique solution x > 0. This implies that (6.11) has a unique solution D m > 0.

We associate with each m ∈ M a real number x m 0 such that Σ m∈M e -xm < ∞. The sequence (e -xm /Σ) m∈M can be seen as a prior probability distribution on the sequence of models (S m ) m∈M . We will denote thereafter by ∆(M) the set of all probability distributions on 7 To avoid any measurability issues, the supremum sup t∈Sm in (6.10) should be understood as a supremum sup t∈Am over any at most countable dense subset Am ⊂ Sm. In the same way, the infimum inft∈S m in (6.13) can be replaced with an infimum inft∈A m ; the resulting weaker assumption ensures that their always exists ( sm)m∈M such that sm and W ( sm) are measurable. Though we do not focus on measurablity issues in this chapter, all stated results remain true under the aforementioned slight modification (by density of Am and since W admits an almost-sure continuous version on Sm).

M (endowed with its discrete σ-algebra).

We also set, for all t ∈ H, γ ε (t) = t 2 -2Y ε (t) . (6.12)

Given µ > 0, we associate with this empirical contrast some collection of µ-least-squares estimators ( s m ) m∈M (or µ-LSEs for short). This means that for all m ∈ M,

s m ∈ S m and γ ε ( s m ) inf t∈Sm γ ε (t) + µ almost surely. (6.13)
Finally, for all m ∈ M, we denote by d(s, S m ) inf t∈Sm s -t the distance between s and the model S m .

Model selection via penalization

In the framework described above, the general model selection theorem by [Mas07, Theorem 4.18] reads as follows. Note that it holds for possibly nonlinear models S m (i.e., the S m ⊂ H are not necessarily linear subspaces of H).

Theorem 6.1 (A general model selection theorem by [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF] for nonlinear models).

Consider the framework given in (6.9). Let K > 1 be some constant and pen : M → R + be such that, for all m ∈ M,

pen(m) Kε 2 D m + √ 2x m 2 , (6.14)
where D m is defined in (6.10) -(6.11) and where (x m ) ∈ R M + is such that Σ m∈M e -xm < ∞. Then, almost surely, there is a minimizer m ∈ argmin m∈M γ ε ( s m ) + pen(m) .

(6.15)

Defining a penalized µ-LSE as s = s m , we have, for some constant C K > 1 depending only on K, for all s ∈ H,

E s s -s 2 C K inf m∈M d 2 (s, S m ) + pen(m) + ε 2 (Σ + 1) + µ . (6.16)

Our aggregation procedure

Our aggregation procedure is defined as follows. We refer the reader to the introduction for related references.

Given an inverse temperature parameter η > 0 and a penalty function pen (η) : M → R + , we define the associated Gibbs posterior ρ

(η) = ρ (η) m m∈M ∈ ∆(M) by ρ (η) m = exp -η γ ε ( s m ) + pen (η) (m) m ∈M exp -η γ ε ( s m ) + pen (η) (m )
, m ∈ M . (6.17)

We will later see sufficient conditions for the almost sure convergence of the series in the denominator above, in which case ρ (η) is well-defined. If this is the case, and if almost surely the next series converges absolutely in H, then we define our estimator s (η) by

s (η) = m∈M ρ (η) m s m . (6.18)
Note that ρ (η) and s (η) are always well-defined when M is finite.

Model aggregation with nonlinear models

In this section we prove an oracle-type inequality for the aggregation procedure s (η) . To avoid convergence technicalities, all the proofs in this section are given for the case of a finite collection M; the extensions to any at most countable collection are postponed to Section 6.A.

We first fix some notations. We set ln + (x) max ln(x), 0 for all x ∈ R. Moreover, for all ρ, π ∈ ∆(M), the Kullback-Leibler divergence K(ρ, π) between ρ and π is defined by

K(ρ, π)    m∈M ρ m ln ρ m π m if ρ is absolutely continuous with respect to π; +∞ otherwise.
Finally, for all ρ ∈ ∆(M) such that m∈M ρ m s m < ∞ almost surely (which is the case when, e.g., M is finite), we set

J (ρ) m∈M ρ m s m -s 2 - m∈M ρ m s m -s 2 . (6.19)
Note that J (ρ) 0 by Jensen's inequality.

The main result of this section is the following theorem. See also Remark 6.1 below for a risk bound with high probability. Theorem 6.2. Consider the framework given in (6.9). Assume that M is at most countable. Let η > 0 and K > 1 be some constants and take pen (η) : M → R + such that, for all m ∈ M,

pen (η) (m) Kε 2 D m + √ 2x m 2 + x m η , (6.20)
where D m is defined in (6.10)-(6.11) and where (x m ) ∈ R M + is such that Σ m∈M e -xm < ∞. Then, almost surely,

•

m∈M exp -η γ ε ( s m ) + pen (η) (m) < ∞, so that ρ (η) is well defined in (6.17);

• m∈M ρ (η) m s m < ∞, so that s (η) = m∈M ρ (η)
m s m is well defined in (6.18).

Moreover, defining the prior π ∈ ∆(M) by π m e -xm /Σ for all m ∈ M, we have, for some constant C K > 1 depending only on K (cf. (6.34)), for all s ∈ H,

E s s (η) -s 2 C K inf ρ∈∆(M) m∈M ρ m d 2 (s, S m ) + pen (η) (m) - x m η + K(ρ, π) η + ε 2 (ln + (Σ) + 1) + µ -E s J ρ (η) (6.21) C K inf m∈M d 2 (s, S m ) + pen (η) (m) + ln Σ η + ε 2 (ln + (Σ) + 1) + µ -E s J ρ (η) .
(6.22) Remark 6.1 (High-probability bound).

The above oracle-type inequalities are only stated in expectation. The proof of Theorem 6.2 also leads to the following high-probability bound: for all z > 0, with probability at least 1 -Σ 2 e -z ,

s (η) -s 2 C K inf m∈M d 2 (s, S m ) + pen (η) (m) + ln Σ η + ε 2 (z + 1) + µ -J ρ (η)
for some absolute constant C K > 0 depending only on K (cf. (6.35)). See Remark 6.2 later for a proof of this bound. Note that it implies a bound in expectation similar to (6.22) -via Lemma A.7 in Appendix A.6 -but that is in general not comparable to (6.21).

Before proving Theorem 6.2, we first make some comments on the procedure s (η) and state two corollaries of Theorem 6.2. Next we compare s (η) with the standard model selection procedure s m recalled in Theorem 6.1; we assume for simplicity that M is finite. For this purpose, let pen : M → R + be a penalty function, and consider the case when pen (η) (m) = pen(m) + x m /η for all m ∈ M. Assume that, for all m ∈ M, we have pen(m)

Kε 2 √ D m + √ 2x m 2 , or, equivalently, pen (η) (m) Kε 2 √ D m + √ 2x m 2 + x m /η. Then, by the fact that ρ (η) m is proportional to exp -η γ ε ( s m ) + pen (η) (m) = exp -η γ ε ( s m ) + pen(m) -x m , (6.23)
it is easy to see that, as η → +∞, the probability distribution ρ (η) converges almost surely to

ρ (∞) ∈ ∆(M) ⊂ R |M| defined by ∀m ∈ M , ρ (∞) m e -xm / Z if m ∈ M , 0 if m / ∈ M , (6.24)
where M argmin m∈M γ ε ( s m )+pen(m) ⊂ M and where Z m∈ M e -xm . In particular, if there is a unique minimizer m ∈ argmin m∈M γ ε ( s m ) + pen(m) , then ρ (η) tends to the Dirac distribution at m, so that s (η) → s m almost surely. This is stated formally in Corollary 6.1 below.

On the contrary, when η → 0, we can see from (6.23) that, almost surely, the probability distribution ρ (η) tends to the prior π (e -xm /Σ) m∈M , so that s (η) → m∈M π m s m . Therefore, the continuous family s (η) : η > 0} contains Bayesian variants of the estimator s m ranging from pure model aggregation (when η → 0) to pure model selection (when η → ∞).

How to choose the value of the tuning parameter η? Though this important question is beyond the scope of the present chapter, two values of η seem reasonable. As indicated above (and formally stated in Corollary 6.1 below), letting η → +∞ is never a bad choice, since we recover the theoretical guarantees of the standard model selection procedure s m stated in Theorem 6.1. Another interesting choice is suggested in Corollary 6.2. For smaller values of η ≈ 1/ε 2 , the estimator s (η) still satisfies a risk bound comparable to that of s m , but it can also benefit from the convexification phenomenon due to aggregation -see the brief comment after Corollary 6.2 and the discussion in Section 6.4.2.

Note that for the two choices of η mentioned above, the estimator s (η) is built as a function of the noise level ε, which may be unknown in practice. In the case η = +∞, adaptation to ε was first tackled by [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF][START_REF] Arlot | Data-driven calibration of penalties for least-squares regression[END_REF] for histogram-based regression via the so-called slope's heuristics. As for smaller values of η ≈ 1/ε 2 , adaptation to ε was addressed by [START_REF] Giraud | Mixing least-squares estimators when the variance is unknown[END_REF] for linear models with a Mallows' C p -type penalty. However, it is not clear whether the choices η = +∞ or η ≈ 1/ε 2 mentioned above are the best ones, so that two important questions remain open. First, can we identify an optimal choice of η (in a reasonable sense) at least for classical prediction problems? Second, if such an optimal (and theoretical) choice is identified, is it possible to tune our aggregating procedure in an automatic and near-optimal way? Corollary 6.1 (Choice of η → +∞). Consider the framework given in (6.9). Assume that M is at most countable. Let K > 1 be a constant and take pen :

M → R + such that pen(m) Kε 2 √ D m + √ 2x m
2 for all m ∈ M, where D m is defined in (6.10) -(6.11) and

where

(x m ) ∈ R M + is such that Σ m∈M e -xm < ∞.
Set pen (η) (m) pen(m) + x m /η. Then, as η → +∞, the estimator s (η) defined in (6.17) -(6.18) converges almost surely to the estimator

s (∞) m∈M ρ (∞) m s m ,
where ρ (∞) is defined in (6.24) and has almost surely a finite support M. Moreover, for some constant C K > 1 depending only on K, we have, for all s ∈ H,

E s s (∞) -s 2 C K inf m∈M d 2 (s, S m ) + pen(m) + ε 2 (ln + (Σ) + 1) + µ . (6.25)
The proof of the last corollary is immediate when M is finite: the almost sure convergence s (η) -s (∞) → 0 was already explained in the previous paragraphs, and the risk bound above follows from Fatou's lemma and from (6.22) in Theorem 6.2. The proof in the general case of an at most countable collection M is postponed to Appendix 6.A.2.

Note that the oracle-type inequality above is identical to that of [Mas07, Theorem 4.18] recalled in Theorem 6.1 (except for the term ln + (Σ) that is smaller than Σ, but this improvement can also be made for Theorem 6.1). This is not surprising since our proof of Theorem 6.2 follows the same lines as that of Theorem 6.1. The present chapter thus shows that the analysis of [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF] for nonlinear models can be extended to arbitrary values of η > 0, instead of just taking η = +∞. Corollary 6.2 (Choice of η = c/ε 2 ). Consider the framework given in (6.9). Assume that M is at most countable. Let c > 0 and K > 1 be some constants and take pen c : M → R + such that, for all m ∈ M,

pen c (m) Kε 2 D m + √ 2x m 2 + c -1 ε 2 x m , (6.26)
where D m is defined in (6.10) -(6.11) and where

(x m ) ∈ R M + is such that Σ m∈M e -xm < ∞.
Then, for some constant C K > 1 depending only on K, the estimator s (c/ε 2 ) defined in (6.17) -(6.18) satisfies, for all s ∈ H,

E s s (c/ε 2 ) -s 2 C K inf m∈M d 2 (s, S m ) + pen c (m) + ε 2 ln Σ c + ε 2 (ln + (Σ) + 1) + µ -E s J ρ (c/ε 2 ) .
The last corollary is an immediate consequence of Theorem 6.2, so the proof is omitted. Note that the risk bound above is at most of the same order 8 as the bound of Corollary 6.1 if in each corollary the penalty functions are chosen as the smallest penalties allowed by the assumptions. However, choosing η = c/ε 2 instead of η → ∞ enables to reduce the risk bound by the additive term E s J ρ (c/ε 2 ) . We have not investigated yet the extent to which the above risk bound improves -via the term E s J ρ (c/ε 2 ) -on the bound of Theorem 6.1 for model selection. We however briefly explain in Section 6.4.2 in which situations the convexification phenomenon due to aggregation may be useful.

Proof (of Theorem 6.2): To avoid any convergence technicalities, we assume in the sequel that M is finite (in particular, the first two claims of the theorem are straightforward). The proof in the countably infinite case is postponed to Appendix 6.A.1.

The proof follows the same lines as the general model selection theorem of [Mas07, Theorem 4.18] recalled in Theorem 6.1. The key point is to replace the line stating that m minimizes some penalized empirical risk over M by the fact that ρ (η) minimizes some penalized average empirical risk over ∆(M). Indeed, first note that ρ (η) defined in (6.17) can be rewritten as

ρ (η) m = exp -η γ ε ( s m ) + pen (η) 2 (m) π m m ∈M exp -η γ ε ( s m ) + pen (η) 2 (m ) π m , m ∈ M ,
where π m e -xm /Σ and pen (η) 2 (m) pen (η) (m)-x m /η for all m ∈ M. We can thus use a key duality formula on the Kullback-Leibler divergence proved, e.g., in [Cat04, pp. 159-160] and that 8 Indeed, for all m ∈ M, the bound of Corollary 6.2 is larger than the bound of Corollary 6.1 by at most the additive term c -1 ε 2 xm + c -1 ε 2 ln Σ. Therefore, the overall bound of Corollary 6.2 is smaller than (1 + c -1 ) times that of Corollary 6.1 (note that c

-1 ε 2 xm c -1 Kε 2 √ Dm + √ 2xm 2 since K > 1 1/2).
we recall in Proposition A.1 of Appendix A.1. Applying Proposition A.1 with E = M and the (random) function h : M → R defined by h(m) = η γ ε ( s m ) + pen (η) 2 (m) for all m ∈ M, we can see that h is almost surely bounded on M (since M is finite) and that ρ (η) = π exp -h ; therefore, for all ρ ∈ ∆(M) and all families (s m ) m∈M of elements in H, almost surely,

m∈M ρ (η) m γ ε ( s m ) + pen (η) 2 (m) + K ρ (η) , π η m∈M ρ m γ ε ( s m ) + pen (η) 2 (m) + K(ρ, π) η m∈M ρ m γ ε (s m ) + pen (η) 2 (m) + K(ρ, π) η + µ , (6.27)
where the last inequality follows by definition of s m in (6.13) and by the fact that m∈M ρ m = 1.

In the sequel we fix δ > 0 and choose s m ∈ S m such that s -s m 2 d 2 (s, S m ) + δ 2 for all m ∈ M. At the end of the proof, we will let δ → 0.

We also fix ρ ∈ ∆(M). We can assume that K(ρ, π) < ∞ (since otherwise, ρ does not participate to the infimum in (6.21)). Therefore, by (6.27), we also have K ρ (η) , π < ∞, so that all terms in (6.27) are finite. Moreover, note that for all t ∈ H, by definition of γ ε (t) and Y ε (t),

γ ε (t) t 2 -2Y ε (t) = t 2 -2 <s, t> + εW (t) = t -s 2 -s 2 -2εW (t) . (6.28)
Substituting the last equality in (6.27), and noting that two terms -s 2 cancel out, we get

m∈M ρ (η) m s m -s 2 m∈M ρ m s m -s 2 + pen (η) 2 (m) + K(ρ, π) η - K ρ (η) , π η + µ + 2ε m∈M ρ (η) m W ( s m ) - m∈M ρ m W (s m ) - m∈M ρ (η) m pen (
(η) m W ( s m ) - m∈M ρ m W (s m ) = m∈M ρ (η) m W ( s m ) -W (s ρ ) m∈M ρ (η) m w m ( s m ) sup t∈Sm W (t) -W (s ρ ) w m (t) ,
where, for all m ∈ M and t ∈ H, we set w m (t) (1/2) s -s ρ + s -t 2 + y 2 m for some real numbers y m to be chosen later. Substituting the above inequality in (6.29), and neglecting the term K ρ (η) , π /η 0, we get, almost surely,

m∈M ρ (η) m s m -s 2 m∈M ρ m s m -s 2 + pen (η) 2 (m) + K(ρ, π) η + µ + m∈M ρ (η) m 2εw m ( s m ) sup t∈Sm W (t) -W (s ρ ) w m (t) Vm -pen (η)
2 (m) .

(6.30)

The rest of the proof follows the same lines as the proof of [Mas07, Theorem 4.18] (recalled in Theorem 6.1). First, we use the fact that with large probability, for all m ∈ M, the penalty pen

(η)
2 (m) is large enough to annihilate the fluctuations 2εw m ( s m )V m . This is proved in [Mas07, Theorem 4.18] between Equations (4.78) and (4.82); Lemma 6.2 in Appendix 6.B.2 is a straightforward variant of this argument. We apply it with a = s ρ .

To that end, let z > 0 and K > 1, and set

y m K ε √ D m + √ 2x m + (2π) -1/2 + √ 2z
for all m ∈ M. Then, Lemma 6.2 indicates that on some event Ω z,K of probability

P s Ω z,K 1 -Σe -z , for all m ∈ M, 2εw m ( s m )V m K 2 ε 2 D m + √ 2x m 2 + 2K 2 ε 2 K -1 1 2π + 2z + 1 √ K s -s m 2 + s -s ρ 2 √ K -1 . Now, we choose K √ K.
Therefore, from the last inequality and from the fact that pen

(η) 2 (m) pen (η) (m) -x m /η Kε 2 √ D m + √ 2x m 2
by Assumption (6.20), we can see that on the event

Ω z,K , for all m ∈ M, 2εw m ( s m )V m -pen (η) 2 (m) 2Kε 2 √ K -1 1 2π + 2z + 1 K 1/4 s -s m 2 + s -s ρ 2 K 1/4 -1 .
(6.31) Substituting the last inequality in (6.30), and using again the fact that m ρ

(η) m = 1, we get, on the event Ω z,K , m∈M ρ (η) m s m -s 2 m∈M ρ m s m -s 2 + pen (η) 2 (m) + K(ρ, π) η + µ + 1 K 1/4 m∈M ρ (η) m s -s m 2 + 2Kε 2 √ K -1 1 2π + 2z + s -s ρ 2 K 1/4 K 1/4 -1 .
Noting that s -s ρ 2 m∈M ρ m s -s m 2 by definition of s ρ and by convexity of • 2 , and reordering the terms of the inequality above, we get, with probability at least 1 -Σe -z ,

1 - 1 K 1/4 m∈M ρ (η) m s m -s 2 (1 + A K ) m∈M ρ m s m -s 2 + pen (η) 2 (m) + K(ρ, π) η + µ + 2Kε 2 √ K -1 1 2π + 2z , (6.32)
where we set A K 1/ K 1/4 (K 1/4 -1) . Dividing both sides by 1 -K -1/4 , recalling that s -s m 2 d 2 (s, S m ) + δ 2 for all m ∈ M, and integrating the resulting high-probability bound via Lemma A.7 in Appendix A.6 (see Example A.1), we get that

E s m∈M ρ (η) m s m -s 2 1 + A K 1 -K -1/4 m∈M ρ m d 2 (s, S m ) + δ 2 + pen (η) 2 (m) + K(ρ, π) η + µ + 2Kε 2 1 -K -1/4 √ K -1 1 2π + 2 ln + (Σ) + 1 . (6.33)
Since the last inequality holds for all δ > 0 and all ρ ∈ ∆(M) such that K(ρ, π) < ∞, we conclude the proof of (6.21) by letting δ → 0, by recalling that pen

(η) 2 (m) = pen (η) (m) -x m /η, by setting C K max 1 + A K 1 -K -1/4 , 2K 1 -K -1/4 √ K -1 1 2π + 2 1 , (6.34)
and by using the definitions of J ρ (η) and of s (η) .

The upper bound (6.22) then follows straightforwardly by restricting the infimum over all ρ ∈ ∆(M) to the Dirac distributions ρ = δ m at m ∈ M, and by noting that K(δ m , π) = ln(1/π m ) = x m + ln Σ (since π m e -xm /Σ). Remark 6.2 (Bound with high probability). We can derive oracle-type inequalities with high probability instead of risk bounds in expectation (as in [START_REF] Arlot | Data-driven calibration of penalties for least-squares regression[END_REF][START_REF] Massart | The Lasso as an 1 -ball model selection procedure[END_REF] for instance). Indeed, note that (6.32) in the proof above holds with probability at least Σe -z for any fixed ρ ∈ ∆(M), and in particular for any fixed Dirac distribution δ m at m ∈ M. Therefore, by a union-bound over M and by the equality K(δ m , π) = x m + ln Σ, we get that, with probability at least 1 -Σ 2 e -z , for all m ∈ M,

m∈M ρ (η) m s m -s 2 1 + A K 1 -K -1/4 s m -s 2 + pen (η) 2 (m ) + x m + ln Σ η + µ + 2Kε 2 1 -K -1/4 √ K -1 1 2π + 2(z + x m ) .
Therefore, using again s -s m 2 d 2 (s, S m ) + δ 2 and pen (η) 2 (m ) = pen (η) (m ) -x m /η, and letting9 δ → 0, we get that, with probability at least 1 -Σ 2 e -z , for all m ∈ M,

m∈M ρ (η) m s m -s 2 1 + A K 1 -K -1/4 d 2 (s, S m ) + pen (η) (m ) + ln Σ η + µ + A K 2Kε 2 x m 1 -K -1/4 + 2Kε 2 1 -K -1/4 √ K -1 1 2π + 2z ,
where we set A K 2/ √ K -1 . Finally, noting that 2Kε 2 x m pen (η) (m ) (by Assumption (6.20)), using the definitions of J ρ (η) and of s (η) , and setting

C K max 1 + A K + A K 1 -K -1/4 , 4K 1 -K -1/4 √ K -1 , (6.35)
we get the bound stated in Remark 6.1: with probability at least 1 -Σ 2 e -z ,

s (η) -s 2 C K inf m ∈M d 2 (s, S m ) + pen (η) (m ) + ln Σ η + ε 2 (z + 1) + µ -J ρ (η) .

Examples

In this section we apply Theorem 6.2 to classical problems such as aggregation of linear models, of finite models, and of 1 -balls. The resulting oracle-type inequalities are comparable to those obtained with the model selection procedure of [Mas07, Theorem 4.18] (cf. Theorem 6.1). In a second part, we briefly explain why aggregation might outperform model selection in some situations where convexification is useful.

Application to some classical problems

Next we derive several corollaries of Theorem 6.2 in classical settings. They follow in a straightforward manner from the latter theorem and from the computations of the various generalized dimensions D m that are carried out in [Mas07, Chapter 4]. We only present a few of them (linear models, finite models, and 1 -balls) but all examples treated in [Mas07, Chapter 4] could also be addressed here (e.g., aggregation of Besov ellipsoids).

Aggregation of linear models

As explained in [START_REF] Birgé | Gaussian model selection[END_REF][START_REF] Massart | Concentration Inequalities and Model Selection[END_REF], the particular case of linear models already includes important practical problems such as variable selection, curve estimation, and change points detection. The next corollary is an immediate consequence of Theorem 6.2 (with µ = 0 and η c/ε 2 ) and of the fact that, for any finite dimensional linear subspace S m of H, its generalized dimension D m coincides with its (classical) dimension -see [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF]p. 130]. Corollary 6.3 (Linear models). Fix some constant c > 0. Consider the framework given in (6.9), and assume that (S m ) m∈M is an at most countable collection of linear subspaces of H with finite dimensions D m respectively. For all m ∈ M, let s m ∈ argmin t∈Sm t 2 -2Y ε (t) be the least-squares estimator on S m . Finally, let η c/ε 2 and K > 1 be some constants and take pen (η) : M → R + such that, for all m ∈ M,

pen (η) (m) Kε 2 D m + √ 2x m 2 + x m η , (6.36)
where

(x m ) ∈ R M + is such that Σ m∈M e -xm < ∞.
Then, for some constant C K > 1 depending only on K, the estimator s (η) defined in (6.17 A possible choice for the constant C K above is given in (6.34). This choice, which follows from our analysis for nonlinear models, can of course be tightened in the particular case of linear models. A tighter analysis for refined penalties was indeed carried out for selection of linear models in [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF]. As for the aggregation of linear models, [LB06, Corollary 6] proved an oracle-type inequality with leading constant 1 (instead of C K > 1) for an inverse temperature parameter corresponding to η 1/(4ε 2 ). They use a Mallow's C p -type penalty (i.e., pen (η) (m) = 2ε 2 D m + x m /η) and analyse it through Stein's unbiased risk formula [START_REF] Stein | Estimation of the mean of a multivariate distribution[END_REF].

An interesting open question is thus the following: in the particular case of linear models, and for a proper choice of the penalty, can we extend the refined risk bounds of [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF][START_REF] Leung | Information theory and mixing least-squares regressions[END_REF] through a single analysis to all η > 0 (instead of η = +∞ or η 1/(4ε 2 ))? In particular, what are the performance of our aggregation procedure with a Mallow's C p -type penalty when η > 1/(4ε 2 )? Stein's unbiased risk formula could also be useful in this case.

Next we rewrite the risk bound above for the particular problems of ordered and complete variable selection in the Gaussian regression framework with fixed design. Recall from Example 6.1 in Section 6.1 that, in this setting, we observe

Y i = s i + σξ i , 1 i n ,
where the ξ i are independent standard normal random variables, where the noise level σ > 0 is assumed to be known, and where s ∈ R n is the vector to be estimated. Recall also from Example 6.1 that for all m ∈ M, the least-squares estimator s m ∈ argmin t∈Sm t 2 -2Y ε (t) can be rewritten in a more standard way:

s m ∈ argmin t∈Sm Y -t 2 .
Let (ϕ j ) 1 j p be a family of linearly independent vectors in H = R n . Let M be a finite collection of subsets of {1, . . . , p}. For all m ∈ M, define S m as the linear span of {ϕ j : j ∈ m}, i.e., where S (D) Dm=D S m . Note that the above risk bound can be rewritten in a way similar to (6.39) but with a leading constant of the order of ln p. However, if (ϕ j ) 1 j p is an orthonormal system in R n , then, by the lower bound for complete variable selection of [Mas07, Corollary 4.12], the risk bound above cannot be improved on any S (D) more than by constant factors. In particular, the estimator s (η) is minimax optimal (up to constant factors) on each S (D) , D = 1, . . . , p.

Note that (6.40) yields the following risk bound. This bound, which is due to [START_REF] Birgé | Gaussian model selection[END_REF] for η = +∞, is one of the first sparsity oracle inequalities -see Section 2.6 in Chapter 2 for an introduction. Recall that u 0 {j : u j = 0} for all u ∈ R p .

E s s (η) -s √ n concludes the proof of (6.41).

Remark 6.3. The linear independence assumption on the dictionary (ϕ j ) 1 j p is not necessary to derive the sparsity oracle inequality (6.41). This assumption is only useful to reinterpret (6.38) as an oracle inequality of the form (6.3), e.g., in (6.39). If (ϕ j ) 1 j p is abitrary, then the penalty pen (η) (m) = Kε 2 |m| + 2x (|m|) 2 + x (|m|)/η with x (D) αD + D ln(ep/D) still satisfies (6.36)-(6.37) (since |m| D m and since D → x(D) is nondecreasing and such that x (D) x(D)). Applying Corollary 6.3 then also yields (6.41).

Aggregation of finite models

Next we rewrite Theorem 6.2 in the case of an at most countable collection of finite models. The interest of such models is commented on after the proof of the corollary. Corollary 6.4 (Finite models). Consider the framework given in (6.9). Assume that (S m ) m∈M is an at most countable collection of non-empty finite subsets of H, and denote their cardinalities by |S m |. Let η > 0 and K > 1 be some constants and take pen (η) : M → R + such that, for all m ∈ M,

pen (η) (m) Kε 2 8 ln |S m | + √ 2x m 2 + x m η ,
where (x m ) ∈ R M + is such that Σ m∈M e -xm < ∞.

Then, for some constant C K > 1 depending only on K, the estimator s (η) defined in (6.17 Proof: In view of Theorem 6.2 the only thing to prove is that for all m ∈ M, a valid value12 for the generalized dimension D m of S m is given by D m = 8 ln |S m |. By (6.11) it is sufficient to prove that the assumption (6.10) is satisfied with ϕ m (x) = x 2 ln |S m |.

For this purpose, let x > 0 and u ∈ S m . Then, by the elementary inequality a 2 + b 2 2ab, and by linearity13 of t → W (t), we get

2 E sup t∈Sm W (t) -W (u) t -u 2 + x 2 x -1 E sup t∈Sm W (t) -W (u) t -u = x -1 E max t∈Sm W t -u t -u x -1 2 ln |S m | = x -2 x 2 ln |S m | ,
where the last upper bound follows from a maximal inequality for subgaussian random variables stated in [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF]Lemma 2.3]. This fact is recalled in Lemma A.3 in Appendix A.5; we used it here with T = |S m | 1 and with v = 1 (since the random variables W (t -u)/ t -u all have standard Gaussian distribution).

Therefore, the choice of ϕ m (x) = x 2 ln |S m | satisfies (6.10), which in turns yields D m = 8 ln |S m | by (6.11). This concludes the proof. Note that the above proof slightly improves on a computation carried out in [Mas07, Section 4.4.3] through a peeling argument. The author first remarks that Dudley's bound for metric entropy combined with this peeling argument immediately yields (6.10) with ϕ m (x) = κx ln |S m | for some absolute constant κ > 0. He then mentions that this is true for κ = 8 √ 2 by Lemma A.3. The proof above shows that, unsurprisingly, we get a better constant κ = √ 2 via a global argument (i.e., without using a peeling argument, which is unnecessarily involved here). This is in the same spirit as in [MM11, Theorem A.1].

Finite models can be useful in at least two situations. First, we can see from the above corollary that deterministic estimators s m of s are associated with the 0-dimensional nonlinear models S m = {s m }. Thus, to aggregate deterministic or frozen estimators in the Gaussian regression framework with fixed design, it is sufficient to use the penalty pen (η) (m) = 2 Kε 2 + 1/η x m . In this case, up to a small additive term of the order of ln + (Σ), the price pen (η) (m) to pay for aggregation is only proportional to the logarithm of the inverse of the prior probability mass e -xm /Σ assigned to m ∈ M. This is what is expected when aggregating deterministic estimators (see, e.g., [START_REF] Dalalyan | Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity[END_REF]).

The second situation for which finite models are useful (at least from a theoretical viewpoint) is when the models at hand are arbitrary compact subsets of H. In this case, each model can be approximated by a finite set, so that selecting the best model in the collection approximately amounts to selecting the best associated finite set. This remark is one of the ideas that underly the metric point of view advocated by [START_REF] Birgé | Model selection via testing: an alternative to (penalized) maximum likelihood estimators[END_REF] for adaptive estimation. We refer the reader to [Mas07, Section 4.4.3] for further details.

Aggregation of 1 -balls

Next we derive another corollary of Theorem 6.2 when the models are associated to 1 -balls. We consider the Gaussian regression framework with fixed design described in Example 6.1. Let p 1 and ϕ = (ϕ j ) 1 j p be a family of vectors in R n . We denote u 1 Let (U m ) m∈M be an at most countable family of positive real numbers. Following (6.17 A key contribution of the last paper was to interpret the Lasso estimator as a selected projection estimator s m and hence to derive 1 -oracle-type inequalities on the Lasso without any assumption on the dictionary ϕ. Next we show that, unsurprisingly, the Bayesian variant m∈M ρ (η) m u(U m ) • ϕ satisfies a similar risk bound. We however do not claim that this estimator should be prefered to the Lasso since, unlike this efficient algorithm, it involves the computation of possibly many exponential weights14 . Corollary 6.5 (Aggregation of 1 -balls). Consider the Gaussian regression framework with fixed design described in Example 6.1. Let (ϕ j ) 1 j p be a family of vectors in R n , and (U m ) m∈M be an at most countable family of positive real numbers. Let η > 0 and K > 1 be some constants and take pen (η) : M → R + such that, for all m ∈ M, pen (η) (m) 4KU m γσ 2 ln(2p) n + 4Kσ 2 n + 1 η x m , (6.44)

where (x m ) m∈M ∈ R M + is such that Σ m∈M e -xm < ∞ and where γ max 1 j p ϕ j = max 1 j p n -1 n i=1 ϕ 2 i,j

1/2 .
Then, the estimator s (η) = m∈M ρ (η) m u(U m ) • ϕ given by (6.42)-(6.43) is well-defined and satisfies, for some constant C K > 1 depending only on K, for all s ∈ R n , E s s (η) -s Before proving the corollary, note that if the penalty is chosen as the right-hand side of (6.44) and if η is at least of the order of n/σ 2 , then the risk bound of Corollary 6.5 scales for each m ∈ M approximately as U m γσ ln(2p)/n. It is therefore very similar to the regret bounds derived on 1balls in the online linear regression setting (see Chapters 2 and 4). This similarity is not surprising √ n) and for xm = m, the infinite sum in (6.43) and the estimator s (η) can be computed exactly and with a computational complexity which is linear in log 2 u LSE 1 γ

√ n/σ .
in view of the connections between the online linear regression setting and the stochastic batch setting (e.g., as shown in Section 4.2.1, a Maurey-type argument can be used in both settings). The connections between these settings are actually deeper. Indeed, take M = N and set U m 2 m σ/( √ nγ) and x m m for all m ∈ N in a way similar to [START_REF] Massart | The Lasso as an 1 -ball model selection procedure[END_REF]; choose the penalty as the right-hand side of (6.44). Then, we can see that for all η at least of the order of n/σ 2 , Corollary 6.5 yields a risk bound of the form

E s s (η) -s 2 C K inf u∈R p u • ϕ -s 2 + u 1 γσ ln(2p) n + σ 2 n (ln + (Σ) + 1) ,
where C K > 0 is an absolute constant depending only on K. In Chapter 4 we proved a regret bound of a similar form as far as adaptation to U was concerned (see Section 4.4).

Proof (of Corollary 6.5): First note that the estimators u(U m ) • ϕ ∈ R n are nothing but the projection estimators associated to the models

S Um u • ϕ : u ∈ R p , u 1 U m ,
i.e., u(U m ) • ϕ ∈ argmin t∈S Um Y -t 2 for all m ∈ M. Note also that, by (6.44) and by the elementary inequality 2(a 2 + b 2 ) (a + b) 2 for all a, b ∈ R, we have, for all m ∈ M,

pen (η) (m) K σ 2 n 2U m γ σ 2n ln(2p) + √ 2x m 2 + x m η .
Theorefore, in view of Theorem 6.2, the only thing to prove is that for all m ∈ M, a valid value for the generalized dimension D m of S Um is given by D m = (2U m γ/σ) 2n ln(2p). By (6.11) and by the fact that ε = σ/ √ n, it is sufficient to prove that the assumption (6.10) is satisfied with ϕ m (x) = 2U m γ 2 ln(2p). This fact is essentially proved in [MM11, Theorem 3.1] via the linearity of t → W (t) on the polytope S Um and via the maximal inequality for subgaussian random variables of [Mas07, Lemma 2.3] (cf. Lemma A.3 in Appendix A.5).

A situation where convexification is useful

In all the examples addressed in the previous section, we always neglected the nonnegative term E s J ρ (η) appearing in the risk bound of Theorem 6.2. As a consequence, all bounds of the previous section are comparable to the bounds that would derive from Theorem 6.1 for the model selection procedure, but they do not show any improvement over them. Since the nonnegative term E s J ρ (η) is a gap in a Jensen-type inequality, it suggests that in some favorable situations, combining the base estimators s m instead of selecting one of them may result in better performance. Next we describe a typical situation in which convexification is indeed useful and we prove a simple toy lower bound for model selection indicating that the latter is less robust than aggregation in large-bias situations.

The benefits of convexification were already pointed out in various settings in the past (see, e.g., the introduction of [START_REF] Yang | Adaptive regression by mixing[END_REF]). For example, the bagging method introduced by [START_REF] Breiman | Bagging predictors[END_REF] was shown to improve the performance of unstable base estimators. For the regression problem, the latter technique consists in computing several predictions over independent bootstrap samples, and then in averaging the resulting predictions. As indicated by the author, this averaging is useful when the base predictions are unstable (because of a large gap in a Jensen-type inequality).

The improvement of model aggregation over model selection was formally proved by, e.g., [Cat99, Section 8] for a distribution estimation problem and by [Cat04, Section 4.7] and [Aud07, JRT08, LM09] for the regression problem. A key idea in these works is that aggregation procedures can improve over selection procedures if the base (deterministic) estimators at hand are far away from the prediction target and if several of them are quasi-optimal and well-separated. The aforementioned works address the case of deterministic (frozen) base estimators.

Next we show that a similar improvement is also possible in our setting, where the base estimators s m are random (recall that no sample splitting is allowed in this fixed-design context). where s (η) is defined in (6.17)-(6.18) with η = n/(4σ 2 ) and pen (η) (m) = 2 dim(S m )σ 2 /n (i.e., s (η) is the aggregating estimator of [START_REF] Leung | Information theory and mixing least-squares regressions[END_REF] with the largest allowed inverse temperature parameter), and where (6.46) holds for all measurable functions m : R n → M (i.e., all data-driven selectors).

Therefore, there are situations where the aggregation procedure of [START_REF] Leung | Information theory and mixing least-squares regressions[END_REF] has a risk smaller than that of the oracle up to an additive term at most of the order of 1/n, while any model selection procedure cannot beat the oracle at a rate faster than 1/ √ n uniformly over all s ∈ R n .

In the toy example exhibited in the subsequent proof, the bias of the estimators s 1 and s 2 is large (of the order of σ 2 ). Therefore, the lower bound (6.46) does not contradict the fact that model selection procedures are minimax optimal (up to constant factors) in many classical problems for which the prediction target s lies within a model (see, e.g., [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF]Chapter 4]). However, this lower bound indicates that, at least for linear models, if the target vector is far from all the models at hand (hence a large bias) and if a few models are nearly-optimal and well-separated, then there is an aggregation procedure whose excess risk is much smaller than that of any model selection procedure (compare the rates 1/n and 1/ √ n). In this sense, model aggregation can be thought of as more robust than model selection.

In the general case of nonlinear models, all our oracle-type inequalities were obtained with leading constants larger than 1 (contrary to [START_REF] Leung | Information theory and mixing least-squares regressions[END_REF]). Therefore, it is not clear for the moment whether the aforementioned robustness property also holds true for our aggregation procedure. However, in view of the simplicity of the example exhibited in the following proof, we tend to think that aggregation might benefit from a similar advantage with nonlinear models. This important question remains open.

Proof (of Proposition 6.1): The upper bound (6.45) follows directly from [LB06, Corollary 6]. As for the lower bound (6.46), it can be proved with the following toy example, which is inspired from a lower bound of [START_REF] Catoni | Statistical learning theory and stochastic optimization[END_REF] in a slightly different setting (in our case, the base estimators s m are random). Define the two models S 1 ⊂ R n and S 2 ⊂ R n by S 1 R × {0} × . . . × {0} and S 2 {0} × R × {0} × . . . × {0} .

Consider the following two potential target vectors (one of which is going to be badly estimated by the model selection procedure m at hand):

s α σ √ n 1 + cα √ n , 1 - cα √ n , 0, . . . , 0 ∈ R n , α ∈ {-1, 1} ,
where c ∈ (0, 1) is an absolute constant to be determined by the analysis. If the true vector is s α , the statistician observes the n-dimensional vector Y = s α + σξ with ξ = (ξ 1 , . . . , ξ n ), where the ξ i are i.i.d. standard normal random variables (cf. Example 6.1). In the sequel we denote the law of s α + σξ by P sα (i.e., P sα = N (s α , σ 2 I n )) and the corresponding expectation by E sα .

Let m : R n → {1, 2} be any measurable function. The rest of the proof is dedicated to show that, for all c ∈ (0, 1), max α∈{-1,1}

E sα s m -s α 2 -inf m∈{1,2} E sα s m -s α 2 4cσ 2 √ n 1 2 - c √ 2 - 1 4c √ n .
(6.47) First note that if Y = s α + σξ, then, by definition of s m ∈ argmin t∈Sm Y -t 2 , s 1 = σ √ n 1 + cα √ n + σξ 1 , 0, 0, . . . , 0 , s 2 = 0, σ √ n 1 -cα √ n + σξ 2 , 0, . . . , 0 .

Recall from Example 6.1 that we set u 2 n -1 n i=1 u 2 i for all u ∈ R n . By the two equalities above, if Y = s α + σξ, then for all m ∈ {1, 2},

s m -s α 2 =            σ 2 ξ 2 1 n + σ 2 1 - cα √ n 2 if m = 1 , σ 2 1 + cα √ n 2 + σ 2 ξ 2 2 n if m = 2 .
The last equality yields, on the one hand,

I { m=1} σ 2 1 - cα √ n 2 + I { m=2} σ 2 1 + cα √ n 2 = σ 2 1 + c √ n 2 -σ 2 1 + c √ n 2 -1 - c √ n 2 I { m=mα} ,
where we set m 1 1 and m -1 2 and where used the fact that I { m=1} + I { m=2} = 1 almost surely. Taking the expectations of both sides of the last inequality, substracting (6.48), and using the fact that where the last inequality follows from the fact that { m = m 1 } ∩ { m = m 2 } = ∅ and from the elementary equalities K P sα , P s 0 = n i=1 s i,α -s i,0 2 /(2σ 2 ) = 2σ 2 c 2 /(2σ 2 ) = c 2 .

1 + c/ √ n 2 -1 -c/ √ n 2 = 4c/ √ n,
Substituting the last upper bound in the right-hand side of (6.49) directly yields the lower bound (6.47). We conclude the proof of (6.46) by choosing c = 1/(2 √ 2) and by using the assumption that n 16/( √ 2 -1) 2 .

Future works

As mentioned earlier, this chapter is a work in progress. In particular, important open questions remain open. Among the issues raised throughout this chapter, we ask the following:

• Our oracle-type inequalities are only obtained with leading constants larger than 1. Is this a consequence of the concentration approach -which however yields risk bounds with high probability -or of the generality of the models? In particular, when the models are linear, it could be interesting to recover via a single analysis the tighter bounds of [START_REF] Leung | Information theory and mixing least-squares regressions[END_REF] and of [START_REF] Birgé | Minimal penalties for Gaussian model selection[END_REF] obtained for model aggregation and model selection respectively.

• The important problem of the tuning of η is left open. Is it possible to identify -at least for classical problems -an optimal choice of η? If so, can we tune η in an automatic and nearly-optimal way?

• Finally, investigating classical examples of nonlinear models (e.g., Besov ellipsoids, 1balls, neural networks) could help to compare the model selection procedure of [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF] with our aggregation procedure.

6.A Proofs 6.A.1 Proof of Theorem 6.2 when M is countably infinite Theorem 6.2 is stated for an at most countable collection M. In Section 6.3 we only proved it under the assumption that M is finite. Next we provide a proof in the other and more technical case, i.e., when M is countably infinite.

Proof (of Theorem 6.2, M countably infinite): We assume in the sequel that M in countably infinite. The proof consists of three steps. In Steps 1 and 2, we check that the two sums over M appearing in the definition of s (η) are convergent. In Step 3, we then employ a reduction to the case of a finite collection to prove the oracle-type inequality (6.21).

Step 1: We prove that Z m∈M exp -η γ ε ( s m ) + pen (η) (m) < ∞ almost surely.

Recall from (6.28) that γ ε ( s m ) can be rewritten as γ ε ( s m ) = s m -s 2 -s 2 -2εW s m , m ∈ M . (6.50)

We also set pen 2 (m) with high probability. We follow the same arguments that led to (6.31) in the proof of Theorem 6.2 for a finite collection (see Section 6.3). Namely, let z > 0, and set, for all m ∈ M and t ∈ H,

y m √ Kε D m + √ 2x m + (2π) -1/2 + √ 2z , w m (t) 1 2 s + s -t 2 + y 2 m .
Applying Lemma 6.2 in Appendix 6.B.2 with K = √ K, a = 0, and V m = sup t∈Sm W (t)/w m (t) , we get, on some event Ω z,K of probability P s Ω z,K 1 -Σe -z , that for all m ∈ M,

2εW ( s m ) 2εw m ( s m )V m Kε 2 D m + √ 2x m 2 + 2Kε 2 √ K -1 1 2π + 2z + K -1/4 s -s m 2 + s 2 K 1/4 -1 .
Therefore, from the last inequality and from the fact that pen s -s m 2 + s 2 K 1/4 -1 . (6.52)

Substituting the last inequality in (6.51), we get, on the event Ω z,K , that for all m ∈ M, exp -η γ ε ( s m ) + pen (η) (m)

A η,ε,s,K exp 4ηKε 2 z √ K -1 e -xm exp -η 1 -K -1/4 s m -s 2 , (6.53)

where we set A η,ε,s,K e η s 2 exp π -1 ηKε 2 / √ K -1 + ηK -1/4 s 2 / K 1/4 -1 . Upper bounding the last exponential in (6.53) by 1 (since K > 1), summing the resulting inequality over m ∈ M, and using the assumption m∈M e -xm < ∞ (see Section 6.2.2), we can see that

Z m∈M exp -η γ ε ( s m ) + pen (η) (m)
is finite on all events Ω z,K , z > 0. Applying, e.g., Borel-Cantelli's lemma to the family of complementary events Ω c 2 ln k k∈N * , we deduce from ∞ k=1 P s Ω c 2 ln k ∞ k=1 Σ/k 2 < ∞ that, almost surely, the defining conditions of the events Ω 2 ln k are satisfied for all k large enough. Therefore, Z is almost surely finite, which proves that ρ (η) is well-defined.

Step 2: We prove that m∈M ρ where the last inequality follows from (6.53). Using the fact that sup t∈R + e -At 2 t < ∞ for all A > 0, and in particular for A η 1 -K -1/4 (note that A > 0 since K > 1), we get that m∈M ρ (η) m s m < ∞ on all events Ω z,K , z > 0 .

Therefore, using the same argument as at the end of Step 1 (e.g., Borel-Cantelli's lemma), we can see that m∈M ρ (η) m s m is almost surely finite. Since H is complete (by definition of a Hilbert space), this proves that s (η) = m∈M ρ (η) m s m is well-defined.

Step 3: Reduction to a finite collection.

Since M is countably infinite, we can assume (up to a one-to-one mapping) that M = N. In order to derive (6.21), we employ a reduction to the finite collections (S m ) 0 m M , where M ∈ N. Applying Theorem 6.2 in the finite case will then conclude the proof.

To that end, we set, for all M ∈ N and all m ∈ {0, . . . , M }, where the infimum in (6.58) is taken over all ρ ∈ ∆(N) whose support supp(ρ) {m ∈ N : ρ m > 0} is finite ((6.58) is straightforward), and where (6.59) follows from Explanation 2 below. Combining (6.59) with the definition of J ρ (η) concludes the proof. where the last inequality follows from (6.63). Letting δ → 0, we get (6.62), which in turn yields (6.61). The last sum above is convergent for all η > 0 (see, e.g., Step 2 in the proof of Theorem 6.2). Since in addition the summand decreases to 0 as η → ∞ (by (6.66)), we get by Lebesgue's dominated Then, on some event Ω z,K of probability P s Ω z,K 1 -Σe -z , we have, for all m ∈ M,

ρ (η,M )
2εw m ( s m )V m K 2 ε 2 D m + √ 2x m 2 + 2K 2 ε 2 K -1 1 2π + 2z + 1 √ K s -s m 2 + s -a 2 √ K -1 .
Proof: This upper bound is proved in [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF]Theorem 4.18] between Equations (4.78) and (4.82) therein 18 . We only recall its proof for the convenience of the reader.

We first make the following assumption. The general case will be addressed at the end of the proof.

Assumption 6.1. S m is closed for all m ∈ M.

Step 1: High-probability bound on εV m . Recall that we fixed z > 0. Next we apply a well-known concentration inequality for the suprema of Gaussian processes that can essentially be found, e. .

But we have w m (t) t -a y m by definition of w m (t) (1/2) s -a + s -t 2 + y 2 m , by the triangle inequality, and by the the fact that 2ab a 2 + b 2 for all a, b ∈ R. Therefore, v m y -2 m for all m ∈ M. Substituting the latter inequality in (6.68) and using a union-bound over M, we get, on some event Ω z,K of probability where the last inequality follows from the lower bound y m ε √ D m and from the fact that x → x -1 ϕ m τ m (1 + δ)x in nonincreasing on R * + (since it is the case for x → x -1 ϕ m (x) by assumption). As for the second term in (6.70), we can see from w m (t) (1/2) s -a + s -t 2 + y 2 m , from s -t

P s Ω z,K 1 -Σe -z , ∀m ∈ M , V m E[V m ] +
(1 + δ) -1 s -s m for all t ∈ S m (by (6.71)), and from the triangle inequality that

s -s m + s -a 2 √ K s -s m 2 + s -a 2 √ K -1 ,
and, on the other hand (first with θ = K -1, and then with θ = 1),

y 2 m K 2 ε 2 D m + √ 2x m + (2π) -1/2 + √ 2z 2 K 2 ε 2 K D m + √ 2x m 2 + 2K K -1 1 2π + 2z .
Combining the two inequalities above, we get

2w m ( s m ) √ K s -s m 2 + s -a 2 √ K -1 + K 3 ε 2 D m + √ 2x m 2 + 2K 3 ε 2 K -1 1 2π + 2z .
(6.76)

Step 3: Putting everything together. Combining (6.75) and (6.76), we get

2εw m ( s m )V m 1 √ K s -s m 2 + s -a 2 √ K -1 + K 2 ε 2 D m + √ 2x m 2 + 2K 2 ε 2 K -1 1 2π + 2z ,
which conludes the proof under Assumption 6.1.

General case: We no longer assume that S m is closed for all m ∈ M. We employ a reduction to the case studied above. Namely, we use the above analysis to the modified collection of models S m m∈M .

By contruction, the collection S m m∈M satisfies Assumption 6.1. Let us check that it also satisfies the assumptions of Lemma 6.2. First, the existence of an almost-sure continuous version of W on S m = S m is straightforward. Moreover, (6.10) holds true for S m since it holds on the restricted set S m (by assumption) and since W has an almost-sure continuous version on S m . Second, setting τ m = 1 if S m is convex and τ m = 2 otherwise, we note that τ m τ m . Therefore, the generalized dimension D m of S m defined as in (6.11) satisfies D m D m . (The last inequality follows from the lower bound 1/(τ 2 m ε) 1/(τ 2 m ε) and from the same arguments as those used after (6.11), e.g., from the fact that x → x -2 ϕ m (x) is nonincreasing on R * + .). We can thus apply the conclusion of Step 3 to the collection S m m∈M . To conclude the proof, it suffices to use the fact that D m D m for all m ∈ M and to note that

V m V m sup t∈Sm W (t) -W (a)
w m (t) .

In the sequel we use the following notation. For all probability distributions ρ, π on a given measurable space (E, B), the Kullback-Leibler divergence K(ρ, π) between ρ and π is defined by A.1 A duality formula for the Kullback-Leibler divergence

We recall below a key duality formula satisfied by the Kullback-Leibler divergence and whose proof can be found, e.g., in [START_REF] Catoni | Statistical learning theory and stochastic optimization[END_REF] (see also [START_REF] Dembo | Large deviations techniques and applications[END_REF]p. 264]).

Proposition A.1. For any measurable space (E, B), any probability distribution π on (E, B), and any measurable function h : E → [a, +∞) bounded from below (by some a ∈ R), we have

-ln E e -h dπ = inf ρ∈M + 1 (E) E h dρ + K(ρ, π) ,
where M + 1 (E) denotes the set of all probability distributions on (E, B), and where the expectations E h dρ ∈ [a, +∞] are always well defined since h is bounded from below. Moreover, the above infimum is achieved at ρ = π exp -h , where π exp -h ∈ M + 1 (E) is absolutely continuous with respect to π and is given by dπ exp -h e -h E e -h dπ dπ .

The above duality formula can be equivalently reformulated as follows (just apply it with -h). For any measurable function h : E → (-∞, a] bounded from above (by some a ∈ R), This more classical statement indicates that the log-moment generating function can be thought of as the Legendre transform of the Kullback-Leibler divergence.

A.2 Exp-concavity of the square loss

Next we recall the notion of exp-concavity and the elementary fact that the square loss is 1/(8B 2 )exp-concave on [-B, B]. See, e.g., [START_REF] Kivinen | Averaging expert predictions[END_REF] or [CBL06, Chapter 3] for a reference on exp-concave losses.

Definition A.1 (Exp-concavity).

Let D be a convex subset of a real vector space. A function h : D → R is said to be exp-concave for a given η > 0 (or simply η-exp-concave) if the function H η e -η h is concave on D.

Noting that H η = H η /η η and h = -1 η ln H η , we can see that if h : D → R is η-exp-concave, then

• h is η -exp-concave for all 0 < η η (since x → x η /η is concave and non-decreasing);

• h is convex (since x → -1 η ln x is convex and non-increasing).

The next elementary result can be found, e.g., in [KW99, Proof of Theorem 2] or in [Vov01, Remark 3].

Proposition A.2. The square loss is 1/(8B 2 )-exp-concave on [-B, B] in the sense that, for all y ∈ [-B, B], the function x ∈ [-B, B] → (y -x) 2 is 1/(8B 2 )-exp-concave. (Moreover, the constant 1/(8B 2 ) is not improvable.)

A.3 A version of von Neumann's minimax theorem

We recall below a version of von Neumann's minimax theorem due to [START_REF] Kneser | Sur un théorème fondamental de la théorie des jeux[END_REF] and [START_REF] Fan | Minimax theorems[END_REF]. The next statement is a straightforward consequence of [Fan53, Theorem 2] (see also [Sio58, Theorem 4.2]). Our assumptions are slightly stronger (concave/convex instead of concave-like/convexlike, and continuous instead of upper semi-continuous), but they are sufficient for our purposes.

Lemma A.1 (A version of von Neumann's minimax theorem).

Let X and Y be convex subsets of vector spaces and let f : X × Y → R be a function such that f ( • , y) is concave for all y ∈ Y, and f (x, • ) is convex for all x ∈ X . Assume also that X is endowed with a topology that makes it Haussdorff and compact, and that f ( • , y) is continuous for all y ∈ Y. Then, sup A.5 Some concentration inequalities and a maximal inequality

The next maximal inequality was proved by [Mas07, Lemma 2.3] through an argument of [START_REF] Pisier | Some applications of the metric entropy condition to harmonic analysis[END_REF] to control the expected supremum of random variables that belong to a given Orlicz space.

Lemma A.3 (A maximal inequality). Let Z t , 1 t T , be centered real random variables for which there exists v ∈ R + such that E e λZt e λ 2 v/2 for all t ∈ {1, . . . , T } and all λ > 0 (we say that the Z t are subgaussian with common variance factor v). Then,

E max 1 t T Z t √ 2 v ln T .
The next two lemmas are due to [START_REF] Hoeffding | Probability inequalities for sums of bounded random variables[END_REF]. The first lemma is stated for a single random variable. The second lemma is a direct extension of the first one by independence of the random variables Z t , t = 1, . . . , T . As a consequence, for all δ ∈ (0, 1),

P   T t=1 Z t -E[Z t ] > 1 2 T t=1 (b t -a t ) 2 ln 1 δ   δ
Example A.1. Let a, Σ > 0 and b ∈ R. Let Z be a real random variable such that Z az + b with probability at least 1 -Σe -z for all z > 0. This example corresponds to f (z) = z with the notations above. Therefore we get that E[Z] a ln + (Σ) + 1 + b.

Example A.2. Let a, c > 0 and b ∈ R. Let Z be a real random variable such that, for all δ ∈ (0, 1), we have Z a exp c ln(1/δ) + b with probability at least 1 -δ. This example corresponds to f (z) = a exp c √ z + b and Σ = 1 with the notations above. Therefore, we get from (A.3) and from elementary manipulations that E[Z] a exp(2c 2 ) + 1 + b.

Proof (of Lemma A.7): First note from the intermediate value theorem that since f is increasing and continuous, it is a one-to-one mapping from R + to f (0), lim

A.7 Some information-theoretic tools

The next inequality is due to Pinsker [START_REF] Pinsker | Information and information stability of random variables and processes[END_REF] (and to [START_REF] Csiszar | Information Theory: Coding Theorems for discrete Memory-less Systems[END_REF] for the optimal constant 1/ √ 2).

Lemma A.8 (Pinsker's inequality).

Let P and Q be two probability distributions on a given measurable space (E, B). Then,

P -Q TV K(P, Q) 2 ,
where P -Q TV sup B∈B P (B) -Q(B) is the total variation distance between P and Q.

  FIGURE 1.1 -Structure générale de cette thèse : dépendences entre les chapitres 1 à 6.

Théorème 1. 1 .

 1 Supposons que l'une des deux hypothèses suivantes soit vérifiée : (A1) La fonction : D × Y → R est convexe en son premier argument et est bornée à valeurs [B 1 , B 2 ], où B 1 < B 2 ∈ R.

  u • ϕ(x t ) 2 + ∆ T,d (u) ,pour un terme de regret ∆ T,d (u) aussi petit que possible et, en particulier, sous-linéaire en T .

  y 1 , . . . , y T ∈ [-B y , B y ] et t ) B Φ .

  τ ) du j 2 1 + |u j |/τ 4 .

  Théorème 1.2 (cf. théorème 4.1). Soit d, T 1 et U, X, Y > 0. Le regret minimax sur B 1 (U ) pour des données bornées par X et Y vérifie inf ( yt) t 1 sup x 1 ∞ ,..., x T ∞ X |y 1 |,...,|y T | Y T t=1

  Théorème 1.3 (cf. théorème 4.2). Pour tous d ∈ N * , Y > 0 et κ √ ln(1+2d) 2d √ ln 2 , il existe T 1, U > 0 et X > 0 tels que √ T U X/(2dY ) = κ et inf ( yt) t 1 sup x 1 ∞ ,..., x T ∞ X |y 1 |,...,|y T | Y T t=1

1. 4 . 2

 42 Borne inférieure sur le regret swap pour des suites individuelles Notre deuxième contribution consiste en l'obtention d'une borne inférieure de l'ordre de √ T K sur le regret swap minimax pour des suites individuelles. La preuve du théorème suivant s'appuie sur l'inégalité de Pinsker (cf. annexe A.7) et repose en partie sur des techniques de borne inférieure séquentielle développées par [ACBG02, CBLS05] et [Sto05, Theorem 3.3].

m

  s m défini en (1.10) -(1.11). La pénalité pen (η) (m) est quant à elle choisie en fonction d'une dimension généralisée D m du modèle S m introduite par [Mas07] : D m est la solution dans R * + de l'équation ϕ m τ m σ D m /n = σD m / √ n, où τ m 1 si S m est fermé et convexe et τ m 2 sinon, et où ϕ m : R + → R + vérifie l'hypothèse suivante. Hypothèse 1.1.

  Figure 2.1. Parameters: convex decision space D, outcome space Y, loss function : D × Y → R, and set Θ of expert indices.

Figure 2 . 1 :

 21 Figure 2.1: Prediction with expert advice.

. 1 .

 1 Example 2.1. Typical examples of loss functions include (note that D and Y may be different): • the square loss for bounded outcomes and predictions, which corresponds to D = Y = [-B, B] (for some B > 0) and (a, y) = (y -a) 2 ; • the relative entropy loss, which corresponds to D = Y = [0, 1] and (a, y) = y ln(y/a) + (1 -y) ln (1 -y)/(1 -a) ; this loss is called the logarithmic loss when Y = {0, 1};

Figure 2 . 2 :

 22 Figure 2.2: The exponentially weighted average forecaster.

Figure 2 . 3 :

 23 Figure 2.3: The exponentially weighted average forecaster with time-varying parameter.

  and Y max 1 t T |y t | are unknown to the forecaster.

  5.1 with Z = X × [-B y , B y ], D being the set of all measurable functions from X to [-B y , B y ], : D × Z → R defined by f, (x, y) y -f (x) 2 , and Θ = {u • ϕ : u ∈ U}. If the output Y lies almost surely in [-B y , B y ], this online to batch conversion transforms the aforementioned online algorithms

  Contents 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 3.2 Setting and notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 3.3 Sparsity regret bounds for individual sequences . . . . . . . . . . . . . . . . . . . . . 98 3.3.1 Known bounds B y on the observations and B Φ on the trace of the empirical Gram matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 3.3.2 Unknown bound B y on the observations but known bound B Φ on the trace of the empirical Gram matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 3.3.3 A fully automatic algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 3.4 Adaptivity to the unknown variance in the stochastic setting . . . . . . . . . . . . . . 108 3.4.1 Regression model with random design . . . . . . . . . . . . . . . . . . . . . . . . 108 3.4.2 Regression model with fixed design . . . . . . . . . . . . . . . . . . . . . . . . . 113 3.A Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 3.A.1 Another proof of Lemma 3.1 (Section 3.3.1) . . . . . . . . . . . . . . . . . . . . . 115 3.A.2 Proofs of Theorem 3.1 and Corollary 3.4 . . . . . . . . . . . . . . . . . . . . . . 117 3.A.3 Proofs of Theorem 3.2 and Corollary 3.5 . . . . . . . . . . . . . . . . . . . . . . 119 3.A.4 Proofs of Theorem 3.3 and Corollary 3.7 . . . . . . . . . . . . . . . . . . . . . . 122 3.B Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 3.B.1 Some tools to exploit our PAC-Bayesian inequalities . . . . . . . . . . . . . . . . 124 3.B.2 Some maximal inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

  d space instead of 1 -balls of finite radii, which solves one question left open by [DT11, Section 4.2]. Besides, our algorithm does not need the a priori knowledge of the variance factor of the noise when the latter is subgaussian, which solves a second question raised in [DT11, Section 5.1, Remark 6].

Figure 3 . 1 :0 d j=1 I

 31j=1 Figure 3.1: The online linear regression setting.

(3. 3 )

 3 Corollary 3.1. Assume that, for some known constants B y > 0 and B Φ > 0, the (x 1 , y 1 ), . . . , (x T , y T ) are such that y 1 , . . . , y T ∈ [-B y , B y ] and d j=1 T t=1 ϕ 2 j (x t ) B Φ .Then, when used with B = B y ,

Figure 3 . 2 :

 32 Figure 3.2: The algorithm SeqSEW B,η τ .

  .6) Proof (of Lemma 3.1): Inequality (3.5) is a straightforward consequence of Theorem 4.6 of[START_REF] Audibert | Fast learning rates in statistical inference through aggregation[END_REF] when applied to the square loss, the set of prediction functions Gx → u • ϕ(x) B : u ∈ R d , and the prior 3 π on G induced by the prior π τ on R d via the mapping u

  Inequality (3.5) then follows from the fact that for all ρ ∈ M + 1 (R d ), we have K( ρ, π) K(ρ, π τ ) by joint convexity of K(•, •). As for Inequality (3.6), it follows from (3.5) by noting that ∀y ∈ [-B, B], ∀x ∈ R, y -[x] B |y -x| . Therefore, truncation to [-B, B] can only improve prediction under the square loss if the observations are [-B, B]-valued, which is the case here since by assumption y t ∈ [-B y , B y ] ⊂ [-B, B] for all t = 1, . . . , T .

  .9) Combining (3.7), (3.8), and (3.9), which all hold for all u * ∈ R d , we get Inequality (3.3).Proof (of Corollary 3.1): Applying Proposition 3.1, we have, since B B y and η 1/(8B 2 ), T t=1

  requires that ϕ ∞ sup x∈X max 1 j d |ϕj(x)| be finite. Note that the last tuning satisfies the scale-invariant property pointed out in [DT11, Remark 4].

  and that one of the following assumptions holds on the distribution of ∆Y Y -E[Y ]. • BD(B) : |∆Y | B almost surely for a given constant B > 0;

  3.A.1 Another proof of Lemma 3.1 (Section 3.3.1)In Section 3.3.1 we already provided a short proof of Lemma 3.1 via the use of [Aud09, Theorem 4.6]. Below is an alternative self-contained proof of Inequality (3.5) that we only provide for the convenience of the reader.

2

  ) and 6(R + 1)y * T 2 from above. First note that, by the upper bound B 2 and by the elementary inequality ln(1 + xy) ln ((1 + x)(1 + y)) = ln(1 + x) + ln(1 + y) with x = e 2 r -1 and y = u 1 / u 0 , (3.36) yields ∆ r (u) 64 y * T 2 u 0 2 r + 64 y * Summing over r = 0, . . . , R, we get R r=0 ∆ r (u) 64 2 R+1 -1 y * T 2 u 0 +(R+1) 64 y * 38) in (3.37), we conclude the proof by noting that A T 2 + log 2 1 1 and that

  j (x t ) and that R + 1 2 + log 2 ln e + T t=1 d j=1 ϕ 2 j (x t ) A T . Therefore, one the one hand, via (3.38),

  .47) Equation (3.45) follows from the definition of ρ u * ,τ . Equation (3.46) is obtained by the change of variables t = (u j -u * j )/τ . As for Equation (3.47), it follows from the equality R proved by integrating by parts. Combined with (3.47), Equation (3.44) yields the desired equality.

  Before proving the theorem (see below), we state the following immediate corollary. It expresses the upper bound of Theorem 4.1 in terms of an intrinsic quantity κ √ T U X/(2dY ) that relates √ T U X/(2Y ) to the ambient dimension d. Corollary 4.1 (Upper bound in terms of an intrinsic quantity). Let d, T ∈ N * , and U, X, Y > 0. The upper bound of Theorem 4.1 expressed in terms of d, Y , and the intrinsic quantity κ √ T U X/(2dY

  Figure 4.1. Observe that, in low dimension (Figure 4.1(b)), a clear transition from a regret of the order of√T to one of ln T occurs at κ = 1. This transition is absent for high dimensions: for d ωT , where ω 32(ln(3) + 1) -1 , the regret bound 32 dY 2 ln(1 + 2κ) + 1 is worse than a trivial bound of T Y 2 when κ 1.

  Low dimension d < ωT .

Figure 4 . 1 :

 41 Figure 4.1: The regret bound of Corollary 4.1 over B 1 (U ) as a function of κ = √T U X/(2dY ). The constant c is chosen to ensure continuity at κ = 1, and ω 32(ln(3) + 1) -1 . We define:

T

  Theorem 4.1): First note from Lemma 4.4 in Appendix 4.B that the minimax regret on B 1 (U ) is upper bounded 5 by min 3U XY 2T ln(2d), 32 dY 2 ln 1 ln 2 and the third case U > dY X √ T are straightforward. Therefore, we assume in the sequel that Y X ln(1+2d) T ln 2 U 2dY √

  Theorem 4.2 (Lower bound). For all d ∈ N * , Y > 0, and κ √ ln(1+2d) 2d √ ln 2 , there exist T 1, U > 0, and X > 0 such that √ T U X/(2dY ) = κ and inf F sup xt ∞ X, |yt| Y T t=1

Figure 4 . 3 :

 43 Figure 4.3: The Lipschitzifying Exponentiated Gradient (LEG) algorithm.

Figure 4

 4 Figure 4.4: The Scaling algorithm.

Lemma 4. 7 (

 7 An elementary combinatorial upper bound). Let m, d ∈ N * . Denoting by |E| the cardinality of a set E, we have    (k 1 , . . . , k d ) ∈ Z d : Lemma 4.7): Setting (k -j , k j ) (k j ) -, (k j ) + for all 1 j d, and k 0 md j=1 |k j |, we have

Figure 5 . 1 :

 51 Figure 5.1: Description of the online protocol. The environment can be either stochastic (i.i.d. sequence) or deterministic (individual sequence).

Figure 5 .

 5 Figure 5.2: A simple internal-regret-minimizing strategy when the distribution Q of the loss vectors is known (cf. Theorem 5.1).

  Refinement of the √ T ln K upper bound on internal regret? By minimax duality, to prove a √ T -upper bound on the minimax internal regret for individual sequences, it is sufficient to prove a √ T -upper bound in the maximin game of Section 5.5.3. For arbitrary joint distributions Q on [0, 1] KT , the strategy S * (Q) achieves a √ T ln K-upper bound. The results of Section 5.3 indicate that this strategy is suboptimal in the particular case of i.i.d. loss vectors (i.e., for Q of the form Q

i

  are not exactly of the form e - √ 2 r-1 ∆ i /K (r) eff (since the gaps ∆ i are now estimated by their empirical counterparts m (r) i ). Next we show that p (r) i C r (δ)e - √ 2 r-1 ∆ i /2 /K (r) eff on Ω δ , where C r (δ) is small enough, and where

  Corollary 5.1): The lower bound with the constants c 1 1/192 and c 2 1/ 64 √ 3 follows straightforwardly from the proof of [Sto05, Theorem 3.3]. As for the upper bound, it follows by integrating the high-probability bound of Theorem 5.2. More precisely, combining Theorem 5.2 and Example A.2 in Appendix A.6, we get that inf S sup Q∈M + 1 ([0,1] K )

  since W : H → L 2 P s is linear (by definition of an isonormal process), we have, almost surely, m∈M ρ m W (s m ) = W m∈M ρ m s m = W s ρ , where we set s ρ m∈M ρ m s m . Therefore, using the fact that m ρ (η) m = 1, we get, almost surely, m∈M ρ

  ) -(6.18) satisfies, for all s ∈ H,E s s (η) -s 2 C K inf m∈M d 2 (s, S m ) + pen (η) (m) + ln Σ η + ε 2 (ln + (Σ) + 1) C K inf m∈M d 2 (s, S m ) + pen (η) (m) + ε 2 (1 + c -1) ln + (Σ) + 1 .

u

  j ϕ j : u ∈ R p ; ∀j / ∈ m , u j = 0    , and denote by D m = |m| the dimension of S m . Following the same lines as [BM01a, Mas07], we take x m x(D m ) defined by x(D) αD + ln M D , 0 D p , (6.37) where M D {m ∈ M : D m = D} and where α > 0 is an absolute constant. Then we get Σ= p D=0 M D e -x(D) = 1/ 1 -e -α < ∞. With this choice of (x m ) m∈M , the choice of pen (η) (m) = Kε 2 √ D m + 2x(D m ) 2 + x(D m )/η, and the choice of η c/ε 2 (as in Corollary 6.2), we get the risk boundE s s (η) -s 2 C K min 0 D p d 2 s, S (D) + ε 2 D + ε 2 ln M D + 1 , (6.38)for some constant C K > 1 depending only on (K, α, c). In the last inequality, S (D)m∈M D S m so that d 2 s, S (D) = inf m∈M:Dm=D d 2 (s, S m ) ,where the right-hand side equals +∞ by convention if no model S m has dimension D.In the problem of ordered variable selection, M {1, . . . , D} : D = 1, . . . , p , so that, for all m = {1, . . . , D} ∈ M, we have S m = span ϕ 1 , . . . , ϕ D and D m = D. Moreover, M D = 1 for all D = 1, . . . , p. Therefore, the risk bound (6.38) readsE s s (η) -s 2 C K min 1 D p d 2 s, span(ϕ 1 , . . . , ϕ D ) + ε 2 D + ε 2 = C K min m∈M E s s m -s 2 + ε 2 . (6.39) The last equality follows from the well-known bias-variance decomposition E s s m -s 2 = d 2 s, span(ϕ 1 , . . . , ϕ D ) + ε 2 D for all m = {1, . . . , D}, D = 1, . . . , p (see, e.g., [Mas07, Section 4.2]). It indicates that the estimator s (η) mimics the oracle s m * (s) , where m * (s) ∈ argmin m∈M E s s m -s 2 . It is thus an oracle inequality. We now turn to the problem of complete variable selection, where M P {1, . . . , N } . For all D = 0, . . . , p, we have 10 ln M D = ln p D D ln(ep/D) by, e.g., [Mas07, Proposition 2.5]. Therefore, the risk bound (6.38) reads E s s (η) -s 2 C K min 0 D p d 2 s, S (D) + ε 2 D 2 + ln p D + ε 2 , (6.40)

  To see why (6.40) leads to (6.41), it suffices to note that for all u ∈ R d , we have u ∈ S m(u) where m(u) {j : u j = 0}. Then, upper bounding the minimum in (6.40) by its argument in D m(u) |m(u)| u 0 , noting that x → x 2 + ln(p/x) is nondecreasing 11 on [0, ep] (and a fortiori on [0, p]), and using u ∈ S m(u) ⊂ S (D m(u) ) and ε = σ/

  ) -(6.18) satisfies, for all s ∈ H,E s s (η) -s 2 C K inf m∈M d 2 (s, S m ) + pen (η) (m) + ln Σ η + ε 2 (ln + (Σ) + 1) .

p

  j=1 |u j | and u • ϕ p j=1 u j ϕ j ∈ R n for all u ∈ R p . For all U > 0 we set u(U ) ∈ argmin u∈R p : u 1 U Y -u • ϕ 2 .(6.42)

  ), given η > 0 and a penalty function pen(η) : M → R, we set, for all m ∈ M,ρ (η) m = exp -η Y -u(U m ) • ϕ 2 + pen (η) (m) m ∈M exp -η Y -u(U m ) • ϕ 2 + pen (η) (m ).(6.43)The next corollary upper bounds the risk of the aggregated estimator m∈M ρ (η) m u(U m ) • ϕ . As in the previous corollaries, it essentially relies on the computation of generalized dimensions carried out by [MM11, Theorem 3.1].

  Um u • ϕ -s 2 + pen (η) (m) + ln Σ η + σ 2 n (ln + (Σ) + 1) .

  the other hand, almost surely, s m -s α 2

  pen (η) (m) -x m /η for all m ∈ M. Therefore, we have, for all m ∈ M,exp -η γ ε ( s m ) + pen (η) (m) = e η s 2 e -xm exp -η s m -s 2 exp η 2εW s m -pen (η) 2 (m) .(6.51)Next we use Lemma 6.2 to upper bound the quantity 2εW s m -pen (η)

  pen (η) (m) -x m /η Kε 2 √ D m + √ 2x m 2by Assumption (6.20), we can see that, on the event Ω z,K , for all m ∈ M, 2εW ( s m ) -pen

  s m < ∞ almost surely.We follow the same lines as in Step 1. Noting that ρ(η) m = Z -1 exp -η γ ε ( s m ) + pen (η) (m)where Z > 0 is the random normalization constant studied above, we get, by the triangle inequality and by the equality m∈M ρ(η) m = 1, that on the event Ω z,K introduced in Step 1, exp -η γ ε ( s m ) + pen (η) (m) s m -s s + A η,ε,s,K Z exp 4ηKε 2 z √ K -1 m∈M e -xm exp -η 1 -K -1/4 s m -s 2 s m -s ,

m

  exp -η γ ε ( s m ) + pen(m) M m =0 exp -η γ ε ( s m ) + pen(m ) ∈ ∆ {0, . . . , M } is the Gibbs distribution associated with the finite collection (S m ) 0 m M (compare to (6.17)), and π(M ) π(M ) m 0 m M ∈ ∆ {0, .. . , M } is the associated prior. The corresponding estimator is defined by s (η,M ) = the conclusions of Theorem 6.2 to the finite collection (S m ) 0 m M . Setting B m d 2 (s, S m ) + pen (η) (m) -x m /η for all m ∈ N, we get from (6.21) and by definition of J ρ (η,M ) that for allM ∈ N, m + K ρ, π (M ) η + ε 2 ln + Σ (M ) + 1 + µ C K inf ρ∈∆({0,...,M }) +∞ m=0 ρ m B m + K ρ, π η + ε 2 (ln + (Σ) + 1) + µ , (6.57)where we set Σ(M ) (see Explanation 1 below), we get from (6.57) and from Fatou's lemma 15 that m + K ρ, π η+ C K ε 2 (ln + (Σ) + 1) + µ = C K inf ρ∈∆(N) supp(ρ)<∞ +∞ m=0 ρ m B m + K(ρ, π) η + C K ε 2 (ln + (Σ) + 1) + µ (6.58) = C K inf ρ∈∆(N) +∞ m=0ρ m B m + K(ρ, π) η + C K ε 2 (ln + (Σ) + 1) + µ , (6.59)

Explanation 1 :ρ

 1 We show below that, almost surely,  +∞ m =0 exp -η γ ε ( s m ) + pen(m ) M m =0 exp -η γ ε ( s m ) + pen(m ) m s m -s 2 isalmost surely finite, the last upper bound converges almost surely to 0 as M → +∞.Therefore, combining (6.60) with the above remarks, we get that, almost surely,M m=0 ρ (η,M ) m s m -s 2 → +∞ m=0 ρ (η) m s m -s 2 as M → +∞.Explanation 2: We show thatinf ρ∈∆(N) |supp(ρ)|<∞ +∞ m=0 ρ m B m + K(ρ, π) η = inf ρ∈∆(N) +∞ m=0 ρ m B m + K(ρ, π) η .(6.61) Note that it suffices to show that, for all ρ ∈ ∆(N), ρ ∈ ∆(N) and δ > 0. We can assume that K(ρ, π) < ∞ (otherwise, the inequality obviously holds true). Then, since +∞ m=0 ρ m = 1 and +∞ m=0 ρ m ln(ρ m /π m ) = K(ρ, π) < ∞, we can fix M ∈ N such that S m ln(ρ m /π m ) -δ . (6.63) Define ρ ∈ ∆(N) by ρ m ρ m /S for all m ∈ {0, . . . , M } and by ρ m 0 for all m M + 1, so that |supp(ρ )| < ∞. Moreover, we have +∞ m=0 ρ m B m + K(ρ , π)

6.A. 2

 2 Proof of Corollary 6.1 Proof (of Corollary 6.1): We prove that s (η) -→ η→∞ s (∞) almost surely. The bound (6.25) will then follow directly from Fatou's lemma by letting η → ∞ in (6.22) of Theorem 6.2. First note that the random set M argmin m∈M γ ε ( s m ) + pen(m) ⊂ M is almost surely non-empty and finite. This fact is proved in [Mas07, p. 130] under the assumption that pen(m) where we set B min m∈M γ ε ( s m ) + pen(m) . By definition of B and M, the quantity γ ε ( s m ) + pen(m) -B is equal to zero if and only if m ∈ M, and it is positive otherwise. Therefore, we get, almost surely, for all m ∈ M, exp -η γ ε ( s m ) + pen(m) -B -x m -→ η→∞ e -xm if m ∈ M , 0 if m / ∈ M . (6.66) Since the exponential above is nonincreasing in η and is bounded by e -xm for all m ∈ M, we get by Lebesgue's dominated convergence theorem that, almost surely, m∈M exp -η γ ε ( s m ) + pen(m) -B -x m   m∈ M e -xm Z as η → ∞ . (6.67) Combining (6.65), (6.66), and (6.67), we get that, almost surely, ρ (η) m -→ η→∞ e -xm / Z for all m ∈ M. Since M is almost surely finite, we can conclude that m∈ M ρ (η) m -e -xm / Z s m -→ -η γ ε ( s m ) + pen(m) -B -x m s m .

  g., in[START_REF] Ledoux | The concentration of measure phenomenon[END_REF] Theorem 7.1] or [Mas07, Proposition 3.19], and that is recalled in Lemma 6.1 above. By definition of V m and since the centered Gaussian process [W (t) -W (a)]/w m (t) t∈Sm is almost surely continuous on S m (by the almost-sure continuity of t → W (t) on S m ), Lemma 6.1 ensures that, for all m ∈ M,P V m E[V m ] + 2v m (x m + z)1 -e -xm e -z ,

  E sup t∈Sm W (t) -W (s m ) w m (t) y -2 m ϕ m τ m (1 + δ)y m y -1 m ε D m -1 ϕ m τ m (1 + δ)ε D m ,(6.73)

  inf t∈Sm w m (t) (1/2) (1 + δ) -2 a -s m 2 + y 2 m (1 + δ) -1 s m -a y m ,where the last inequality follows from the fact that 2ab a 2 + b 2 for all a, b ∈ R. Therefore,E W (s m ) -W (a) + inf t∈Sm w m (t) (1 + δ) y -1 m E W (s m ) -W (a) s m -a + = (1 + δ) y -1 m (2π) -1/2 ,where, to get the last equality, we used the fact that W (s m ) -W (a) / s m -a is a standard normal random variable. Substituting the last upper bound and (6.73) in (6.70), we getE[V m ] y -1 m ε D m -1 ϕ m τ m (1 + δ)ε D m + (1 + δ)y -1 m (2π) -1/2 .Letting δ → 0, we get by continuity of ϕ m on R + thatE[V m ] y -1 m ε D m -1 ϕ m τ m ε D m + y -1 m (2π) -1/2 y -1 m D m + (2π) -1/2 , (6.74)where the last inequality follows from the fact that ϕ m τ m ε √ D m = εD m by definition of D m (see (6.11)). Substituting the last upper bound in (6.69), we get, on the eventΩ z,K , ∀m ∈ M , εV m εy -1 m D m + (2π) -1/2 + √ 2x m + 2z K -1 ,(6.75)where the last inequality follows by definition ofy m K ε √ D m + √ 2x m + (2π) -1/2 + √ 2z .Step 2: Upper bound on 2w m ( s m ).Let m ∈ M. Next we bound from above the quantity 2w m ( s m ) s -a + s -s m 2 + y 2 m . Using repeatedly the elementary inequality(a + b) 2 (1 + θ)a 2 + (1 + θ -1 )b 2 , a, b ∈ R ,for various values of θ > 0, we get, on the one hand (with θ = √ K -1),

  absolutely continuous with respect to π;+∞otherwise.

h

  dρ -K(ρ, π) .

  x∈X inf y∈Y f (x, y) = inf y∈Y sup x∈X f (x, y) .

A. 4

 4 An elementary lemma to solve for the cumulative lossThe next elementary lemma is due to [CBLS05, Appendix III]. It is useful to compute an upper bound on the cumulative loss L T of a forecaster when L T satisfies an inequality of the form (A.1).Lemma A.2. Let a, b 0. Assume that x 0 satisfies the inequality

  Lemma A.4 (Hoeffding's lemma). Let Z be a real random variable such that a Z b almost surely, where a < b ∈ R are deterministic constants. Then Z -E[Z] is subgaussian with variance factor (b -a) 2 /4, i.e.,∀λ ∈ R, E exp λ Z -E[Z] exp λ 2 8 (b -a) 2 .The above bound can also be rewritten as ln E e λZ λE[Z] + λ 2 (b -a) 2 /8.

Lemma

∞fP

  . Moreover, setting x + max{x, 0 for all x ∈ R, we haveE Z > ax + b dx (A.4) f ln + (Σ) + lim ∞ f f (ln + (Σ)) Σ exp -f -1 (x) dx , (A.5)where (A.4) follows from the fact that ((Z -b)/a) + > x = (Z -b)/a > x for all x > 0 and where we proceeded as follows to get (A.5). We split the integral into three terms and upper bounded P Z > ax + b separately. We first used the crude 1 upper bound P Z > ax + b 1 for all 0 x f ln + (Σ) . Second, for x ∈ f (ln + (Σ)), lim ∞ f we used the fact that P Z > ax + b = P Z > af f -1 (x) + b Σ exp -f -1 (x) by assumption (A.2). Finally, by (A.2) again, and by the fact that f is increasing, we get that, for all x > lim ∞ f = sup u 0 f (u),P Z > ax + b inf z>0 P Z > af (z) + b inf z>0 Σe -z = 0 . Since x → exp -f -1 (x) is integrable on f (0), lim ∞ f , then E Z + E (Z -b) + +b + < +∞ by (A.5), so that E Z ∈ [-∞, +∞) is well-defined. Using the fact that (Z-b)/a ((Z-b)/a) + and rearranging the terms of (A.5) concludes the proof.

  . Recevoir la donnée x t et prévoir y t R d u • ϕ(x t ) B p t (du) , où [x] B max -B, min{B, x} ; 2. Recevoir l'observation y t et calculer la probabilité a posteriori p t+1 sur R d via l'expression (W t+1 est une constante de renormalisation)

	p t+1 (du)	1 W t+1	exp -η

t s=1 y s -u • ϕ(x s ) B 2 π τ (du) .

(1.4) FIGURE 1.3 -Définition de l'algorithme SeqSEW B,η τ .

  (U e i ) + p - i,t (-U e i ) , où (e 1 , . . . , e d ) désigne la base canonique de R d et où les poids p + i,t et p - i,t sont définis par

		d
	u t exp -η t t-1 s=1 i,t p γ i=1 p + γU ∇ i s ( u s ) i,t t-1
	exp -η t	µU ∇ j s ( u s )
	1 j d	s=1
	µ∈{+,-}

chaque date t 1, sa prévision est donnée par y t = u t • x t avec u t ∈ B 1 (U ) défini par

  Ce dernier cas correspond graphiquement à la courbe en pointillés sur la figure 1.4.

	16
					Square loss
	14				Lipschitzified
					Clipped
	12
	10
	8
	0 2 4 6	4	2 ¡ B t	0	u •x t y t	2	B t	4	6
	FIGURE 1.4 -La perte carrée (y

d → R convexes et lipschitziennes sur R d . A chaque date t 1, la lipschitzification est effectuée le long de la direction de x t et au-delà d'un seuil adaptatif B t 2 log 2 max 1 s t-1 y 2 s 1/2 ≈ max 1 s t-1 |y s |, que nous avions déjà utilisé auparavant (cf. section 1.2.2). Plus précisément, si y t / ∈ [-B t , B t ], alors on pose t ≡ 0, sinon, on définit t comme étant la plus petite fonction convexe coïncidant avec la perte carrée (y t -u • x t ) 2 lorsque |u • x t | B t (elle est donc affine en dehors de [-B t , B t ]). t -u • x t ) 2 , sa version tronquée y t -[u • x t ] Bt 2 -clipped en anglais -et sa version lipschitzifiée t (u) sont tracées en fonction de u • x t .

  aux composantes de la combinaison linéaire produite) ; voir [DT09, section 5.2.1]. Dans notre cadre séquentiel, on pourrait envisager d'étudier si de telles propriétés sont vraies (d'un point de vue pratique ou théorique) pour une modification appropriée de l'algorithme SeqSEW. Autrement dit, peut-on approcher les prévisions y t définies par (1.16) par des prévisions de la forme u t • x t avec u t 0 petit (par ex., u t 0 t) et telles que la perte cumulée encourue soit proche ?

) .

(1.16) En grande dimension, produire des prévisions qui sont des combinaisons linéaires parcimonieuses des prévisions de base pourrait pourtant être utile d'un point de vue statistique (à des fins de sélection de variables) et algorithmique (pour diminuer l'espace mémoire nécessaire). On pourrait envisager de modifier notre prédicteur SeqSEW en remarquant que la probabilité a priori π τ (du) choisie sur R d (et donc, dans une moindre mesure, les probabilités a posteriori p t (du) associées) charge(nt) davantage les combinaisons linéaires u approximativement parcimonieuses. Dans le modèle de régression avec design fixe,

[START_REF] Dalalyan | Sparse regression learning by aggregation and Langevin Monte-Carlo[END_REF]

remarquent ainsi sur des simulations que leur algorithme exponentiel sélectionne correctement les variables pertinentes (pourvu qu'une troncature raisonnable soit appliquée

  for all t = 1, . . . , T . But, as in Theorem 2.1, by Hoeffding's lemma and by the fact that i,t ∈ [B 1 , B 2 ] by assumption,

	where ¯ t i=1 p i,t i,t T K
	t=1
	i,t -¯ t	,	(2.10)

  t y t a 1 (a 1 ), . . . , a t (a s , y s ) s t-1 , a t .

										(2.21)
	The first claim then follows by definition of the regret.
	Second claim.
	Since the set of all strategies of the environment is larger than the set of all individual sequences,
	we only need to prove that
		T		T			T		T
	sup A	t=1	a t , y t -min 1 i K	t=1	a i,t , y t	sup a i,t ,yt	t=1	a t , y t -min 1 i K	t=1	a i,t , y t	.

  adaptation to the almost-sure bound max 1 t T |Y t | -Dealing with all previous issues. Another way to handle the case where the output Y is unbounded is to note that, almost surely, the finite sequence Y 1 , . . . , Y T lies in the bounded interval [-B y , B y ] where B y max 1 t T |Y t | (B y < +∞ a.s. since E |Y | < +∞ by assumption). The almost-sure boundedness of the Y t should not be confused with the boundedness of Y in the sense of (2.43).

  2 t )-expconcave on [-B t , B t ]. In this case (3.19) follows from the cruder upper bound (1/η t ) ln(W t+1 /W t ) 0 -(y t -y t ) 2 + (2B t+1 ) 2 (since |y t |, | y t | B t+1 ). Summing (3.19) over t = 1, . . . , T , Equation (3.17) yields

  The other PAC-Bayesian inequality (3.15), which is stated for non-truncated base forecasts, follows from (3.14) by the fact that truncation to B t can only improve prediction if |y t | B t . The remaining t's such that |y t | > B t then just account for an overall additional term at most equal to T t:B t+1 >Bt 2B t+1

	which yields (3.14) by definition of η T +1	1/(8B 2 T +1 ). 2	8B 2 T +1 , which
	concludes the proof.

  Lipschitzifying Exponentiated Gradient algorithm . . . . . . . . . . . . . . . . . 139 4.4 Adaptation to unknown U . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 4.5 Extension to a fully adaptive algorithm and other discussions . . . . . . . . . . . . . . 146 4.A Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 4.A.1 Proof of Theorem 4.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 4.A.2 Proof of Theorem 4.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 4.B Lemmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 4.C Additional tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 4.1.1 Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 4.1.2 Contributions and related works . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 4.2 Optimal rates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 4.2.1 Upper bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 4.2.2 Lower bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 4.3 Adaptation to unknown X, Y and T via exponential weights . . . . . . . . . . . . . . 137 4.3.1 An adaptive EG ± algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 4.3.2 Lipschitzification of the loss function . . . . . . . . . . . . . . . . . . . . . . . . 138 4.3.3

  and Y max 1 t T |y t | are unknown to the forecaster.

	The above regret bound is an improvement for small losses (cf. (2.15) in Section 2.2.2). By the
	elementary inequality L * T	T Y 2 (since 0 ∈ B 1 (U )), it yields the zero-order regret bound
	T
	t=1

  that in both cases |y t | > B t and |y t | B t , the function t is continuously differen-tiable. Moreover, if |y t | B t , then ∀u ∈ R d , ∇ t (u) = -2 y t -[u • x t ] Bt x t , (4.9) where we define the clipping operator [•] B by [x] B min B, max{-B, x} for all x ∈ R and all B > 0. Therefore, in both cases |y t | > B t and |y t | B t , the function t is Lipschitz continuous with respect to • 1 with Lipschitz constant

  Example when |y t | B t . The square loss (y t -u • x t ) 2 , its clipped version y t -[u • x t ] Bt 2 and its Lipschitzified version t (u) are plotted as a function of u • x t .

									.12)
	16
					Square loss
	14				Lipschitzified
					Clipped
	12
	10
	8
	0 2 4 6	4	2 ¡ B t	0	u •x t y t	2	B t	4	6
	Figure 4.2:

  1 ) 1/(2d), . . . , 1/(2d) ∈ R 2d . Get x t ∈ R d and output the clipped prediction y t u t • x t Bt ; 3. Get y t ∈ R and define the modified loss function t : R d → R as in Section 4.3.2;

	At each time round t 1,
	d
	1. Compute the linear combination u t U	p + j,t -p -j,t e j ∈ B 1 (U );
	j=1
	2. 4. Update the parameter η t+1 according to (4.13);
	5. Update the weight vector p t+1 = (p + 1,t+1 , p -1,t+1 , . . . , p + d,t+1 , p -d,t+1 ) ∈ X 2d defined for all j = 1, . . . , d and γ ∈ {+, -} by a
	p γ j,t+1

  By the key property (4.12) that holds for all rounds t such that |y t | B t (the other rounds accounting only for an additional total loss at most of c 2 Y 2 , see (4.48) in Appendix 4.A.2), we get that

	where c 1 8	√	2+1 and c 2 4 1 + 1/ √	2	2 . The bound is no longer an improvement for small
	losses, but the analysis is even more straightforward (no need to solve a quadratic inequality); see
	below.
	Proof (of Remark 4.2): T
			t=1

  T defined for each sample (X t , Y t ) 1 t T and each new input X by f T X ; (X t , Y t ) 1 t T 1 T , . . . , ϕ d ), and where we explicitely wrote all the dependencies 12 of the f t , t = 1, . . . , T . Take u * ∈ R d + achieving the supremum 13 in Lemma 4.1 for the estimator f T . Note that u * Besides, consider the i.i.d. random sequence (x t , y t ) 1 t T in R d × R defined for all t = 1, . . . , T by x t γϕ 1 (X t ), . . . , γϕ d (X t ) and y t γϕ u * (X t ) + σε t , (so that y t = u * • x t + σε t for all t), where the X t are i.i.d points in

		T
		f t γϕ(X ); (γϕ(X s ), Y s ) 1 s t-1 ,	(4.27)
		t=1
	where ϕ	(ϕ 1 1	1.
			(4.28)
	where ϕ u *	d j=1 u * j ϕ j

t-1 → R maps at time t the new input xt and the past data (xs, ys) 1 s t-1 to a prediction ft xt; (xs, ys) 1 s t-1 . However, unless mentioned otherwise, we omit the dependency in (xs, ys) 1 s t-1 , and only write ft(xt).

f

  4.38) where (4.37) follows from the fact that the online forecaster ( f t ) t outputs its predictions in [-Y, Y ]. As for (4.38), note by definition of y t that |y t | u * 1 γ ϕ(X t ) ∞ + σ|ε t | γ = 1, . . . , d and x ∈ R. Therefore, by definition of γ c 9 X, and since X Y /2 (by definition of X), we get |y t | c 9

	u *	1	1 and |ϕ j (x)|	| √	2 sin(jx)|	√	√ 2 for all j √ 2 Y /2+σ|ε t | 2 + σ|ε t | since
	Y /2 + σ|ε t | provided that
						c 9	1 √ 2	,	(4.39)

which we assume thereafter. The above remarks show that {|y t | > Y } ⊂ {|ε t | > Y /(2σ)}, which entails (4.38). By the same comments and since | f t | Y , we have, for all t = 1, . . . , T ,

  The proof follows directly from Corollary 2.2 of Chapter 2 and from the fact that the Lipschitzified losses are larger than their clipped versions. Indeed, first note that, by definition of y t and B t+1 |y t |, we have

	T	y t -y t	2	T	y t -u t • x t Bt	2	+	T
	t=1			t=1				t=1
				t:|yt| Bt				t:|yt|>Bt

Theorem 4.3 Proof (of Theorem 4.3):

  5.1. It is non-positive since, by definition of the weight vector p Q

	t	δ I * t
	and of the index I * t ∈ argmin 1 i K E Q i,t 1:t-1 , we have, almost surely,

T t=1

  it follows from the boundedness property | i,t -¯ i,t | 1 and from

	the inequality K i=1 p 2 i,t	K i=1 p i,t = 1.
	Term 3.
	Set LT	T t=1	¯ i,t 1 i K and L T	T t=1 i,t 1 i K , and define the function f : R K

  ext , B ext , Q ext ) be a probability space, and let I 1 , . . . , I T ∈ {1, . . . , K} be random variables defined on the augmented space [0, 1] T K × Ω ext such that 17 I t is measurable with respect to the σ-field σ( 1 , . . . , t-1 ) ⊗ B ext , and, for all γ ∈ {0, 1} K/2 , ∀t ∈ {1, . . . , T }, ∀i ∈ {1, . . . , K}, Q γ

	tr t=t r-1 +1	E Qγ p jr(γ),t	3 4	.	(5.66)
	For this purpose, we introduce an external randomization (as in [Sto05, Theorem 3.3]). Let
	(Ω

  Proposition 6.1 (A lower bound on model selection). Consider the regression framework with fixed design described in Example 6.1. Then, there exists a collection of linear models (S m ) m∈M in R n with |M| = 2 such that for all n 16/( √ 2 -1) 2 , ∀s ∈ R n , E s s (η) -s

	2	inf m∈M	E s s m -s	2 +	4 ln(2) σ 2 n	,	(6.45)
				2 +	σ 2 4 √ n	,		(6.46)

∀ m, ∃s ∈ R n , E s s m -s 2 inf m∈M E s s m -s

  To prove (6.47), it suffices to upper bound the minimum in the last inequality. But, using Pinsker's inequality (cf. Lemma A.8 in Appendix A.7), we can see that, if P s 0 denotes the law of the ndimensional random vector s 0 + σξ with s 0 σ √ n(1, 1, 0, . . . , 0), then s 0 m = m α + max α∈{-1,1} K P sα , P s 0

							we get
		max α∈{-1,1}	E sα s m -s α	2 -inf m∈{1,2}	E sα s m -s α	2
		4cσ 2 √ n	1 -min α∈{-1,1}	P sα m = m α -	σ 2 n	.	(6.49)
								
	min α∈{-1,1}	P sα m = m α	min α∈{-1,1}	  P s 0 m = m α +	K P sα , P s 0 2	 
				min α∈{-1,1}	2
				1 2	+	c √ 2	,

P

  The rest of this step is dedicated to upper bounding the expectation E[V m ]. First note that by definition of V m , We upper bound each term of the right-hand side separately. Let δ > 0. Recall that, by definition, τ m = 1 if S m is closed and convex, and τ m = 2 otherwise. In all cases, by Assumption 6.1, we can fix for all m ∈ M a point s m ∈ S m such that the two following conditions are satisfied: -t τ m (1 + δ) s -t , for all t ∈ S m . (6.72)Details:Indeed, if S m is closed and convex, then we can take s m as the projection of s onto S m , so that s -s m = d(s, S m ). Moreover, since the projection is a contraction, we have s m -t s -t for all t ∈ S m , which yields (6.72) with τ m = 1.But if the S m are arbitrary, then there always exists s m ∈ S m such that (6.71) holds true (by Assumption 6.1 if d(s, S m ) = 0, obvious if d(s, S m ) > 0). The property (6.72) then follows by noting that, for allt ∈ S m , s m -t s m -s + s -t 2(1 + δ) s -t .By definition of w m (t) and by (6.72), we have 2w m (t) τ -2 m (1 + δ) -2 t -s m 2 + y 2 m . Using the assumption (6.10) with u = s m , we get

			y -1 m	2(x m + z) .		(6.69)
	E[V m ] E sup t∈Sm	W (t) -W (s m ) w m (t)	+ E	W (s m ) -W (a) + inf t∈Sm w m (t)	.	(6.70)

s -s m (1 + δ) d(s, S m ) ,

(6.71)

s m

  A.5 (Hoeffding's inequality). Let Z t , 1 t T , be independent real random variables such that Z t ∈ [a t , b t ] a.s. for all t ∈ {1, . . . , T }, where a t , b t ∈ R are some constants. Then the sum T t=1 Z t -E[Z t ] is subgaussian with variance factor T t=1 (b t -a t ) 2 /4, i.e.

	∀λ ∈ R,	E exp λ	T t=1	Z t -E[Z t ]	exp	λ 2 8	T t=1	(b

t -a t ) 2 .

On utilise la notation Z au lieu de Y pour éviter toute ambiguité avec le modèle de régression avec design aléatoire, où Yt désigne uniquement la sortie alors que le statisticien observe le couple Zt = (Xt, Yt) ∈ X × R. Dans ce cadre, Z = X × R.

La variable aléatoire ZT +1 ∈ Z est indépendente de (Z1, . . . , ZT ) et de même loi que Z.

Le risque en espérance de la procédure aT correspond quant à lui à la quantité E aT , Z .

Rappelons qu'il s'agit d'une méthode de régression des moindres carrés régularisés par la norme 2 .

Dans le modèle de régression avec design aléatoire, la méthode statistique résultant d'un prédicteur par pondération exponentielle via la conversion (1.1) est qualifiée de progressive mixture rule en anglais ; cf.[START_REF] Catoni | Statistical learning theory and stochastic optimization[END_REF].

Ce prédicteur séquentiel est rappelé au chapitre 2 ; cf. (2.26) en section 2.4.2.

Un exemple de telles bornes est donné par (1.3) avec a = 0. Ces bornes permettent de majorer les excès de risque R uT -inf { u 0 s} R(u) pour tout s ∈ {0, . . . , d}.

La description du cadre de régression linéaire séquentielle diffère très légèrement de celle de la section 1.2 : le vecteur des prévisions de base à l'instant t est xt alors qu'il s'agissait de ϕ(xt) en section 1.2. Les deux descriptions

Ce modèle est connu sous le nom de Gaussian sequence framework en anglais.

Ce prédicteur vérifie en fait une borne plus fine, du même type que celle du théorème 1.4.

De même que dans[START_REF] Cesa-Bianchi | Analysis of two gradient-based algorithms for on-line regression[END_REF] et [CBL06, section 2.5], l'algorithme EG ± adaptatif est générique et peut être utilisé avec toute suite ( t)t 1 de fonctions de pertes convexes et différentiables sur R d . Nous l'utilisons dans un premier temps avec les fonctions de perte t : u ∈ R d → (yt -u • xt) 2 .

En fait, dans le cadre de suites individuelles, on pourrait plus généralement supposer que les pertes t sont choisies par un environnement antagoniste, i.e., qui réagit aux décisions du statisticien (cf. section 2.3.1).

Voir l'introduction du chapitre 5 pour de plus amples références.

Le terme modèle recouvre plusieurs significations : il est utilisé à la fois pour désigner le cadre (modèle de régression) et une partie de R n .

On suppose pour simplifier que de tels estimateurs existent ; en toute généralité, on peut considérer des estimateurs approchés des moindres carrés -cf. section 6.2.2.

Une introduction plus détaillée (avec bien plus de références) est proposée au chapitre 6.

n proche et sont suffisamment séparés l'un de l'autre. Les deux bornes (1.14) et (1.15) ci-dessus indiquent que l'estimateur agrégé de[START_REF] Leung | Information theory and mixing least-squares regressions[END_REF] a un excès de risque au plus de l'ordre de 1/n uniformément en s, alors que toute méthode de sélection de modèles encourt dans au moins une situation de fort biais (et telle que décrite précédemment) un excès de risque au moins de l'ordre de 1/ √ n. En ce sens, l'agrégation de modèles est plus robuste que la sélection de modèles.La borne inférieure précédente est prouvée avec des modèles linéaires, mais sa simplicité suggère que l'agrégation de modèles pourrait bénéficier d'une propriété de robustesse similaire pour des modèles non linéaires classiques ; cette question ouverte sera abordée prochainement.

De surcroît, une implémentation du type LARS[START_REF] Efron | Least angle regression[END_REF] permet de calculer le chemin entier de régularisation, ce qui est utile à des fins de calibration.

In particular, the experts' losses on the past regimes are no longer used and the weight vector of the forecaster is reset to (1/K, . . . , 1/K).

We compare existing bounds in the case when B2 -B1 is known but T is unknown.

Time-varying tunings have also been designed for other frameworks or for other types of algorithms than the exponentially weighted average forecaster: see, e.g., [BHR08, MS10, DHS10] for time-varying tunings in online convex optimization.

By Remark 2.2 above, the constant 2 √ 2 ≈ 2.83 can actually be improved. More precisely, the mutiplicative price to pay for adaptation to T and B2 -B1 is smaller than 2.64/2 = 1.87.

We do not consider any dependence between the forecaster's current prediction and its past predictions: this is useless since the forecaster does not randomize and can thus at each time t re-compute all its past predictions (at least theoretically). A similar comment holds for the environment's moves, that only depend on the forecaster's predictions.

As recalled in Section 2.1.3, in this randomized setting, the environment has access to pt before choosing the

In the sequel, we may restrict Y to a bounded interval [-By, By] to emphasize the fact that the performance of the online algorithm under analysis are assessed for bounded observations yt ∈ [-By, By].

This algorithm is also called the Vovk-Azoury-Warmuth forecaster in [CBL06, Section 11.8].

See also[START_REF] Grove | General convergence results for linear discriminant updates[END_REF] in the classification context.

Same comment.

This corresponds to the online convex optimization setting.

If the convex loss functions t : R d → R are not differentiable, gradients can be replaced with subgradients.

The corresponding vector p ∈ X 2d is not unique. An example is given by p + i = (ui)+/U + U -u 1 /(2dU ) and by p - i = (ui)-/U + U -u 1 /(2dU )

.14 Again, gradients can be replaced with subgradients in case of non-differentiability.

We use the notation Z instead of Y to avoid any ambiguity with the regression model with random design where Yt only denotes the output while we observe the whole pair Zt = (Xt, Yt) ∈ X × R. In this setting, Z = X × R.

"Batch" means that all outcomes Z1, . . . , ZT are available at the beginning of the game-as opposed to the online setting.

Indeed, the expected regret is nonnegative for the i.i.d. sequence Y1 = . . . = YT = 0, and (T /2) ln 1 = 0.

Since the expert advice are constant and known to the forecaster (they are given by ai,t = δi), the only useful information at time t is Y1:t-1 = Y1, . . . , Yt-1).

The random variables It can be constructed as follows: at each time t = 1, . . . , T , pick It ∈ {1, . . . , K} at random such that It = i with probability ai,t (conditionally on the past data (Y1, I1), . . . , (Yt-1, It-1)).

This forecaster is recalled in Chapter

2; see (2.26) in Section 2.4.2.

The set G is endowed with the σ-algebra generated by all the coordinate mappings g ∈ G → g(x) ∈ R, x ∈ X (where R is endowed with its Borel σ-algebra).

This means that for all y ∈ [-B, B], the function x → exp -η(y -x) 2 is concave on [-B, B].

We omit the factor u 0 in C1, since the 0 -norm of the minimizer u of (3.10) is unknown and "small" under a sparsity scenario. This approximation leads to a reasonable tuning as can be seen from Corollary 3.1.

The tuning τ = 1/ √ dT only uses the knowledge of T , which is known by the forecaster in the stochastic batch setting. In that framework, another simple and easy-to-analyse tuning is given by τ = 1/( ϕ ∞

Actually our results hold whether (xt, yt) t 1 is generated by an oblivious environment or a non-oblivious opponent since we consider deterministic forecasters. See

Section 2.3.1 in Chapter 2 for further details.

In high dimensions (i.e., when d > ωT , for some absolute constant ω > 0), we do not observe this transition (cf. Figure4.1).

For example, (xt, yt) 1 t T may be i.i.d. , or xt can be deterministic and yt = f (xt) + εt for an unknown function f and an i.i.d. sequence (εt) 1 t T of Gaussian noise.

The authors of[START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF] asked: "For large d there is a significant gap between the upper and lower bounds. We would like to know if it possible to improve the upper bounds by eliminating the ln d factors."

For T sufficiently large, we may overlook the case κ < ln(1 + 2d)/(2d √ ln 2) or √ T < (Y /(U X)) ln(1 + 2d)/ ln 2. Observe that in this case, the minimax regret is already of the order of Y 2 ln(1 + d) (cf. Figure4.1).

For all x ∈ R, sgn(x) equals 1 (resp. -1, 0) if x > 0 (resp. x < 0, x = 0).

The proof remains the same by replacing 8Y 2 ln(R + 1) with 8Y 2 ln(R + 1).

Each time the exponential sequence (U r ) expands, the weights assigned to the existing points U r are appropriately reassigned to the whole new sequence.

If the supremum in Lemma 4.1 is not achieved, then we can instead take an ε-almost-maximizer for any ε > 0. Letting ε → 0 in the end will conclude the proof.

Note that for all A > 0, the function x → x ln(1 + A/x) (continuously extended at x = 0) has a nonnegative first derivative and is thus nondecreasing on R+.

This means that for ally ∈ [-Y, Y ], the function x → exp -η(y -x) 2 is concave on [-Y, Y ].

This prediction protocol corresponds to the framework of prediction with expert advice described in Figure

2.1 (Chapter 2) with D = XK , Y = [0, 1] K , the linear loss (p, ) ∈ XK × [0, 1] K → p • , and constant expert advice ai,t = δi (I {j=i} ) 1 j K , i = 1, . . . , K, t = 1, . . . , T .

, . . . , T ∈ [0, 1] K are i.i.d. with common distribution Q. Since the minimax internal regret for individual sequences is larger than the minimax internal regret with i.i.d. loss vectors, the inequalities above show that the orders of magnitude in T and K of both minimax quantities lie between √ T and √ T ln K. We note a missing √ ln K factor between the lower and upper bounds.

To be exact, the best upper bound known so far equals T 2 ln[K(K -1)] which is smaller but close to

We write aT,K bT,K if and only if there exist two absolute constants c1, c2 > 0 and a sequence (tK ) K 1 in N * such that c1bT,K aT,K c2bT,K for all K 1 and all T tK .

See ,e.g., [CBFH + 97, Section 3.2] or [CBL06, Theorem 3.7] where asymptotic lower bounds of order √ T ln K are derived on the minimax external regret.

Since in(5.19) the environment is oblivious to the forecaster's past moves, the infimum inf {S rand} can be restricted to non-randomized strategies, i.e., such that p t is a measurable function of ( s)1 s t-1 (by Jensen's inequality).

As of now, we do not know if their is such difference for the internal regret. In any case, contrary to swap regret, such difference cannot be too large since the minimax internal regret for individual sequences is at most a factor of √ ln K larger than the minimax internal regret for i.i.d. loss vectors -cf. (5.6) and (5.7).

Contrary to Theorem 5.2, we are able to prove a bound that grows root-logarithmically in 1/δ. This is due to the simpler form of the weights (Dirac probability distributions) compared to those of Figure5.3 (exponential weights).

A function f : R K → R is affine if and only if f -a is linear for some a ∈ R.

This setting was motivated by load balancing and job scheduling applications. See Section 5.5.4.

The fact that the p t appear additively in the external regret has been used many other times. An example is in the water-filling technique used to derive the exact minimax external regret in the binary prediction problem under the absolute loss; see, e.g., [CBL06, Section 8.2] and the references therein.

Note that, contrary to[START_REF] Dar | Online learning for global cost functions[END_REF], we chose not to normalize the sums T t=1 pi,t i,t and T t=1 qi i,t by T . This definition is of course equivalent to that of[START_REF] Dar | Online learning for global cost functions[END_REF], but it is more consistent with the other definitions of regret considered in this chapter (none of which is normalized).

With a slight abuse of notation, we denote the identity function on [0, 1] T K by 1:T = ( 1, . . . , T ). The coordinate mappings t : [0, 1] T K → [0, 1] K can be seen as random vectors defined on the measurable space [0, 1] T K (endowed with its Borel σ-algebra).

The random variables It can be constructed as follows: at each time t = 1, . . . , T , pick It ∈ {1, . . . , K} at random such that It = i with probability pi,t (conditionally on the past data ( 1, I1), . . . , ( t-1, It-1)).

Note that the minimum of two quantities is smaller than their means, and that {It = 2r} ∩ {It = 2r -1} = ∅.

When H = R n ,[START_REF] Birgé | Gaussian model selection[END_REF] show in Section 3.4 that m is the mode of the posterior probability distribution in a Bayesian framework with an improper "uniform" prior on each model Sm and a prior on the collection M which is proportional to exp -pen(m)/(2ε 2 ) , m ∈ M. See also [FG00, Theorem 1],[START_REF] Abramovich | MAP model selection in Gaussian regression[END_REF], and [AGS11, Part IV].

In the terminology of online learning, a progressive mixture rule is the result of the standard online-to-batch conversion (see Section 2.5.1) when applied to an exponentially weighted average forecaster computed on the sample.

In the Gaussian regression framework with fixed design,[START_REF] Leung | Information theory and mixing least-squares regressions[END_REF][START_REF] Dalalyan | Optimal aggregation of affine estimators[END_REF] assume that η n/(4σ 2 ), where n and σ 2 denote the sample size and the variance of the noise respectively. This condition can be rewritten in our setting (6.1) as η 1/(4ε 2 ) (cf. the correspondence ε = σ/ √ n in Example

6.1).

Equivalently, W = W (t) t∈H is a family of real random variables such that, for all p 1 and all t1, . . . , tp ∈ H, the random vector W (t1), . . . , W (tp) is Gaussian with zero mean and covariance matrix <ti, tj> 1 i,j p .

Note that the probability Σ 2 e -z is independent of δ and that P[Z > a + δ] ↑ P[Z > a] as δ ↓ 0 for any real random variable Z and any constant a ∈ R.

We use the natural convention 0 ln(A/0) = 0 for all A > 0.

See Footnote 10.

If Sm has cardinality one, then τm = 1 so that the value 8 ln |Sm| exhibited below is only an upper bound on the solution of (6.11). This is not an issue since Theorem 6.2 only assumes a lower bound on the penalty.

More precisely, we use the fact that, for all t ∈ Sm, W (t) -W (u) / t -u = W (t -u)/ t -u almost surely (since by definition of an isonormal process t → W (t) is a linear function from H into a space of squareintegrable random variables). Since Sm is finite, the latter equality holds almost surely simultaneously for all t ∈ Sm.

Note however that, letting u LSE be any least-squares estimator in R p , we can choose u(Um) = u LSE for all Um > u LSE 1 . Thus, for Um of the form Um = 2 m σ/(γ

Note that we only work in expectation here. However, it is of course also possible to follow similar arguments to derive a high-probability bound from the high-probability bound obtained with finite collections (Remark 6.2).

We use sup t∈R + e -At 2 t 2 < ∞ for all A > 0 instead of sup t∈R + e -At 2 t < ∞.

As previously mentioned, if sup t∈S Xt is not measurable, we consider sup t∈A Xt instead, where A is any at most countable dense subset of S; the last two suprema are almost surely equal by the almost-sure continuity of t → Xt.

The slight improvement with respect to [Mas07, Theorem 4.18] (with however, the exact same proof) is that we let a be arbitrary.

Note that if ln+(Σ) > 0, then this crude upper bound is smaller than Σ exp -f -1 (x) > 1 for all x < f ln+(Σ) since f -1 is increasing.

Remerciements

where P (γ * 1 ,...,γ * r-1 ,•) is defined similarly to Q (γ * 1 ,...,γ * r-1 ,γr,•) except that 2r-1,t and 2r,t are both Ber(1/2) under P (γ * 1 ,...,γ * r-1 ,•) on the regime {t r-1 + 1, . . . , t r }, and where we set Kr,t (γ r ) K P (γ * provided that ε 1/4. In (5.69) we denote by

Inequality (5.70) follows by joint convexity of K(•, •). To get (5.71), we used the chain rule for the Kullback-Leibler divergence, the independence of the losses i,t , 1 i K, 1 t T , and the fact that P (γ * 1 ,...,γ * r-1 ,•) and Q (γ * 1 ,...,γ * r-1 ,γr,•) only differ on the regime {t r-1 + 1, . . . , t r } (of length T / K/2 ) at i = 2r -γ r . Finally, (5.72) follows from the fact that K Ber(1/2), Ber(1/2 -ε) = -ln(1 -4ε 2 )/2 and that -ln(1 -x) 4 ln(4/3) x for all x ∈ (0, 1/4) (see, e.g., [START_REF] Cesa-Bianchi | Prediction, Learning, and Games[END_REF]), and from the elementary inequality K/2 K/4 (since K 2).

Substituting the upper bound of (5.72) (that does not depend on γ r ) in (5.68) and using the definition of γ * r , we get

where the upper bound by 1/2 in (5.73) follows from the fact that 18 min γr∈{0,1} tr t=t r-1 +1

and where (5.74) holds provided that ε 1/4 and 16T ln(4/3)ε 2 /K 1/16. For such an ε, and for the choice of γ * γ * 1 , . . . , γ * K/2 above, we have just proved (5.67) or, equivalently, (5.66).

Lemma 5.3. Let K, T 1 and x 1 , . . . , x T ∈ R K + . Then, using the same conventions as above, we have

Chapter 6

Aggregation of nonlinear models

We consider the generalized linear Gaussian framework introduced in [START_REF] Birgé | Gaussian model selection[END_REF], which includes as special cases the Gaussian regression model with fixed design and the white noise framework. Given a collection of subsets (or nonlinear models) in a separable Hilbert space, the goal is to estimate the unknown vector almost as well as the best of the least squares estimators associated with the models in the collection. In this setting we analyse a Bayesian variant of the celebrated general model selection procedure of [START_REF] Birgé | Gaussian model selection[END_REF][START_REF] Massart | Concentration Inequalities and Model Selection[END_REF]. As in [START_REF] Leung | Information theory and mixing least-squares regressions[END_REF], our procedure is based on exponential weighting, but the models at hand can be arbitrary. In such generality, we use the concentration approach of [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF] and derive (non-sharp) oracle-type inequalities with high probability. This work exhibits a natural connection between model aggregation and model selection: our oracle-type inequalities hold for a continuum of estimators ranging from classical model aggregation (where the inverse temperature parameter is small enough) to model selection (where the inverse temperature parameter is infinite). We finally prove a lower bound indicating that aggregation is more robust that model selection in case of linear models. This lower bound suggests that aggregation might benefit from a similar advantage with nonlinear models. DISCLAIMER: This chapter is a work in progress. In particular, important questions remain open (see Section 6.5). The preliminary results stated thereafter were presented at the workshop Stat-MathAppli 2011 [START_REF] Gerchinovitz | Aggregation of nonlinear models[END_REF]. Putting everything together, we get from (6.64) that, almost surely, s (η) → s (∞) as η → ∞.

Contents

6.B Useful lemmas

In this section we recall two results that prove to be useful throughout this chapter.

6.B.1 A classical concentration inequality for Gaussian processes

We recall below a well-known concentration inequality for the suprema of Gaussian processes that follows straightforwardly from [Led01, Theorem 7.1] by the almost-sure continuity assumption and by separability arguments (see also [Mas07, Proposition 3.19]). Lemma 6.1. Let (X t ) t∈S be a centered Gaussian process indexed by a separable topological space S such that t → X t is almost surely continuous on S. Then, for all x 0,

where ν sup t∈S E X 2 t , where E sup t∈S X t ∈ (-∞, +∞] is always well-defined 17 (since X t 0 is integrable for any t 0 ∈ S), and where we used the conventions +∞ +∞ and (+∞) + x = +∞ for all x ∈ R.

6.B.2 An upper bound on some fluctuations

Next we recall a result due to [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF]. We use it in Theorem 6.2 to show that with large probability, for all m ∈ M, the penalty pen Lemma 6.2. As in Section 6.2.2, we assume that for every m ∈ M, there exists some almost-surely continuous version of the isonormal process W on the closure S m of S m and that (6.10) holds true for some nondecreasing continuous function

We let τ m = 1 if S m is closed and convex and τ m = 2 otherwise and define the generalized dimension D m of S m as in (6.11).

Let z > 0 and K > 1, and set

The next lemma is an extension of Hoeffding's inequality to martingales with zero-mean and bounded increments. It is due to [START_REF] Hoeffding | Probability inequalities for sums of bounded random variables[END_REF] and [START_REF] Azuma | Weighted sums of certain dependent random variables[END_REF].

We first need the following definition. A sequence of random variables (X t ) t∈N * on a probability space (Ω, A, P) is a martingale difference sequence with respect to a filtration (F t ) t∈N if and only if, for all t 1, X t is F t -measurable, integrable, and satisfies, almost surely,

Lemma A.6 (The Hoeffding-Azuma inequality). Let (X t ) t∈N * be a martingale difference sequence with respect to a filtration (F t ) t∈N . Assume that for all t 1, there exists a F

A.6 Integration of high-probability bounds

The next elementary lemma is useful to derive bounds in expectation from bounds in high probability. We then specialize it to two examples that are used throughout the manuscript.

Lemma A.7 (Integration of high-probability bounds).

Let Z be a real random variable such that, for some constants a, Σ > 0 and b ∈ R, and for some increasing and continuous function f : R + → R + , whose inverse we denote by f -1 ,

where ln + (Σ) max ln(Σ), 0 .

The next lemma is a version of Fano's lemma due to [START_REF] Birgé | A new lower bound for multiple hypothesis testing[END_REF] (see also [Mas07, Corollary 2.18] for the statement given below). We denote by N * = {1, 2, . . .} the set of positive integers. Lemma A.9 (Fano's lemma -Birgé's version).

Let (E, B) be a measurable space and N ∈ N * . Let (A 0 , . . . , A N ) be a measurable partition of (E, B) and (P 0 , . . . , P N ) be a family of probability distributions on (E, B). Then we have

,

where κ > 0 is an absolute constant such that κ 2e/(2e + 1) and where K 1

The next lemma is an extension of Lemma A.9 to convex combinations of probability distributions. This extension was proved (with different constants) in [CBLS05, Lemma 18] through a simple adaptation of the arguments of [START_REF] Birgé | A new lower bound for multiple hypothesis testing[END_REF]. Another way to prove it is to use Lemma A.9 on the augmented space Ω × {1, . . . , S} and to rewrite the resulting bound via the law of total probability and the chain rule for the Kullback-Leibler divergence.

Lemma A.10 (Fano's lemma for convex combinations).

Let (E, B) be a measurable space and N, S ∈ N * . Let (A s,0 , . . . , A s,N ) : s = 1, . . . , S be a family of measurable partitions of (E, B) and P s,j : s = 1, . . . , S, j = 1, . . . , N be a family of probability distributions on (E, B). Let α 1 , . . . , α S ∈ R + be such that S s=1 α s = 1. Then, where κ > 0 is an absolute constant such that κ 2e/(2e + 1) and where

α s K(P s,i , P s,0 ) .