Emilie Kaufmann

Directeur De

Olivier Cappe

Aurélien Garivier

Rémi Munos

Professeur Jury M Gérard Biau

BONALD M Thomas

M Olivier Cappe

M Olivier Catoni

M Nicolò Cesa-Bianchi

Professeur M Aurélien Garivier

Professeur, I3M Jean-Michel Marin

MUNOS M Rémi

Spécialité « Signal et Images » présentée et soutenue publiquement par

étudiés . . . . . . .

L'heure est venue de mettre la touche finale à ce manuscrit, en remerciant toutes les personnes sans qui ce dernier n'existerait pas, et qui ont fait de ces trois années de thèse une expérience inoubliable.

Tout d'abord un grand merci à mes directeurs de thèse. Vous avez su me proposer un sujet passionnant, et doser avec justesse encadrement et liberté de recherche tout au long de ces trois ans. Aurélien, merci d'avoir pu me consacrer des journées entières lors de nos séances de travail à Toulouse. Merci pour ton accueil chaleureux et pour tous tes conseils, qu'ils soient scientifiques ou non. Olivier, merci pour toutes nos discussions face à un tableau, un bon repas ou même une caméra, et ton aide au quotidien (même informatique) malgré ton emploi du temps chargé. Merci à vous deux pour vos encouragements, votre disponibilité et vos relectures attentives. Rémi, merci pour ton accueil fréquent à Lille lors de mes deux premières années de thèse, et pour notre travail sur le Thompson Sampling. Tu m'a aussi permis de collaborer avec Nathan, que je remercie dans ces lignes pour nos échanges nombreux et fructueux. I also want to thank Shivaram for pointing a new problem to me at ICML in 2012. It was great working with you, even from far, and I hope we will see each other again at some other conference! Nicolò and Olivier, I am really honored that you accepted to review this manuscript and I warmly thank you for your careful reading. Merci également à Gérard, Thomas et Jean-Michel de s'être intéressés à mon travail, et d'avoir pris le temps d'assister à cette soutenance.

Le chemin de l'apprenti chercheur est fait de grandes joies (finir une preuve !) mais aussi de grandes frustrations (elle était fausse...). Pendant ces trois ans, cela aura été un plaisir de les partager (ou de les oublier) avec mes collègues doctorants et jeunes docteurs du 37, rue Dareau. Un merci tout particulier à mes co-bureau du DA320 pour leur bonne humeur. Certains sont maintenant docteurs (Joffrey, Alexandre, Sylvain, Yao), d'autres le seront bientôt (Adrien, Yasir et Andrés) et à Claire à qui je passe le flambeau, je souhaite trois belles années. Il me serait difficile de citer, sans oublier personne, tous ceux avec qui j'ai partagé pauses café, puis pauses thé, gâteaux ou autres bières post-séminaire, soirées gastronomiques ou encore un week-end incroyable en Roumanie. Ils se reconnaîtront, et sauront comme tout cela aura été précieux pour moi ! Une mention spéciale à Cristina, pour avoir été une colloc' géniale en plus d'une collègue formidable. A Amandine, qui m'a fait découvrir des thés délicieux (tout comme Ruocong), re-découvrir Hermann (et d'autres recettes), et aura toujours été d'une grande aide dans les formalités administratives. A Andrés pour une séance shopping mémorable (et d'autres à venir). A Emilie, pour ses astuces LaTeX et une séance de bricolage fort amusante rue Dareau. Au club des fans de Question Pour Un Champion (Sylvain 2 et Olivier). A Eric: non il n'y aura pas de concours de pot de thèse ! En plus de mes collègues, je souhaiterais remercier Janique et Laurence, interlocutrices toujours agréables et efficaces pour les ordres de mission, Fabrice et Sophie-Charlotte pour leur aide informatique (ainsi que Nicolas !), et Florence pour son aide dans tous les domaines ainsi que son soutien aux activités du Bureau des Doctorants. Un clin d'oeil à Georges, Simon, Dong-Bach et tous les autres pour cette belle aventure associative.

Ces trois années ont été enrichissantes en terme de rencontres et de voyages. D'abord au travers des séminaires et autres conférences (souvent) au soleil, où j'ai eu la chance de rencontrer des jeunes chercheurs fort sympatiques. Je pense en particulier aux doctorants et post-doc de l'équipe Sequel, aux randonneurs d'Aussois (et à ceux des îles Canaries), aux skieurs de Grenade, aux compagnons de tapas à Barcelone ou aux organisateurs d'YSP toujours motivés. J'ai aussi eu la chance de passer trois mois à l'Université de Princeton, ce qui m'a donné l'occasion de prendre la mesure de la recherche outre-Atlantique. Merci à Sébastien de m'avoir accueillie. J'ai beaucoup apprécié nos discussions et j'espère que nous aurons l'occasion de travailler ensemble dans le futur. Merci à tous les amis "américains" (qui ne le sont pas pour la plupart) qui ont rendu ce séjour agréable. Thanks to Rofoldo and Theo for making me feel at home during these three months. Thanks Aga for having been such a great office mate, but also Andrew, Flavia and Tana for our little Princeton gang. Thanks to the welcoming 'Frenchies' I met there and also to some graduate students at ORFE with whom we shared nice moments.

Mes remerciements vont bien sûr aussi à mes amis "parisiens" (qui ne le sont pas pour la plupart) et alsaciens (qui le sont pour la plupart) qui m'ont cotoyée et soutenue pendant cette aventure. Je pense notamment à Camille, Arthur et Sonia, Timon et Tiphaine (et Elise qui nous a rejoints en cours de route), Nina, Laure, Vincent, Elsa, Guillaume, Sarah (C.) et son Cricri, Pierre, Jean, et les autres amis rencontrés pendant les années Cachanaises, avec qui discuter de sa thèse est presque un passage obligé. Aux collocs, Evelyne et Yacine, pour les bons moments au 13, rue Bellier Dedouvre. A Sarah (L.), une autre alsacienne conquise par la capitale. A mes amis d'enfance (ou presque) pour qui la lecture d'une page au hasard de ce document est hilarante (et qui me croient désormais membre de la mafia avec tous ces bandits...), merci d'être toujours présents quand je rentre en Alsace ! Les "filles": Agnès, Marie-Madeleine, Martine, Marion, Marie, Céline, et tous ceux qui se sont rajoutés avec bonheur à notre petit groupe, c'est toujours un plaisir de vous retrouver ! Je ne serais pas arrivée jusqu'ici sans l'appui de ma famille. A mon papa, qui m'a familiarisée dès mon plus jeune âge avec les problèmes de remplissage de baignoires qui fuient ou autres croisements de trains, et à ma maman, qui a toujours eu peur que ses enfants détestent l'école, je veux dire un grand merci. Merci pour tout ce que vous m'avez transmis, pour votre soutien durant ces longues années d'études, et pour votre présence dans ce moment important. Merci à Stéphane pour cette complicité que nous partageons, et pour ta capacité à toujours croire en ta grande soeur. Tu as avant moi fait l'expérience de l'écriture d'un manuscrit qui intéressera à coup sûr de plus nombreux lecteurs que celuici ( [Kaufmann, 2014]). Longue vie à l'imaginaire du roi Barry, et à ta muse Ombline. Plus largement, merci à tous les membres de ma famille pour tous ces bons moments partagés, autour d'une tarte flambée, d'une partie de carte ou autre, et pour tous ceux à venir. Un merci ému à Pépé et Mémé.

Romain, c'est à toi que s'adressent ces derniers mots. Quand nous nous sommes rencontrés tu achevais l'écriture de ta thèse, et une autre thèse plus tard -la mienne cette fois-ta présence à mes côtés est devenue une évidence. Merci pour tous ces beaux moments et pour ton soutien de chaque instant.

CONTENTS

Introduction et présentation des résultats

Cette thèse s'est déroulée au sein du LTCI (Laboratoire Traitement et Communication de l'Information) à Telecom ParisTech, sous la co-direction d'Olivier Cappé et d'Aurélien Garivier. Elle a été agrémentée de quelques visites à l'Université Paul Sabatier à Toulouse, où Aurélien Garivier est désormais professeur. Cette thèse a également été co-encadrée par Rémi Munos, ce qui m'a amenée à travailler ponctuellement avec lui à l'INRIA Lille.

L'objectif de cette thèse est de proposer et d'analyser de nouvelles stratégies optimales pour des problèmes d'allocation séquentielle de ressources dans un environnement aléatoire. L'environnement est constitué de plusieurs options (certaines étant meilleures que d'autres) qui peuvent être testées, et produisent des résultats aléatoires. Nos ressources correspondent aux tests que nous pouvons effectuer, et le but est de déterminer des stratégies d'allocation de ce budget de test qui permettent de réaliser certains objectifs (par exemple identifier les meilleures options). Un modèle statistique naturel pour de telles situations est le modèle de bandit stochastique à plusieurs bras. L'objectif de ce chapitre est d'introduire les problèmes de bandits que nous avons considérés, et de présenter nos contributions. Celles-ci seront détaillées dans les chapitres suivants, où on trouvera les preuves des résultats énoncés, ainsi que des éléments bibliographiques plus précis.

La section 1 est consacrée à la présentation des modèles de bandit et des deux problèmes de bandit étudiés dans cette thèse : la maximisation des récompenses d'une part, et l'identification des meilleurs bras d'autre part. Dans chaque cas, nous nous attacherons en particulier à la définition d'un critère d'optimalité. La section 2 présente nos contributions relatives à la maximisation des récompenses. Dans des modèles de bandit paramétriques simples, ce problème est bien compris puisqu'il existe une borne inférieure asymptotique sur le regret d'un algorithme efficace, ainsi que des algorithmes atteignant cette borne. Bien que le regret soit une mesure de performance fréquentiste, nous montrons que deux algorithmes d'inspiration bayésienne, Bayes-UCB et Thompson Sampling, sont également (asymptotiquement) optimaux du point de vue du regret, sont plus simples d'implémentation que les algorithmes optimaux existants et se généralisent facilement à des modèles plus complexes, comme les modèles dits contextuels. La section 3 présente nos contributions à l'identification des meilleurs bras (ou exploration pure). Nous présentons et analysons deux algorithmes, KL-LUCB et KL-Racing, basés sur des intervalles de confiance construits à l'aide de la divergence de Kullback-Leibler, transposant au cadre de l'exploration pure des améliorations récentes obtenues pour la minimisation du regret. Nous proposons également une borne inférieure sur le nombre moyen d'échantillons des bras nécessaires pour identifier les m meilleurs bras, qui ne permet toutefois pas de prouver l'optimalité des algorithmes proposés. La complexité de l'identification des meilleurs bras est en effet moins bien comprise que celle de la minimisation du regret. Nous introduisons ici une notion de complexité, qui nous permettra, pour des exemples importants de modèles de bandit à deux bras, d'identifier des algorithmes optimaux.

Dans notre présentation, nous tâcherons de souligner les outils théoriques utilisés. Nous distinguons CONTENTS principalement ceux liés à l'analyse d'algorithmes (inégalités de déviations, construction d'intervalles de confiance), et ceux liés à l'obtention de bornes inférieures sur la performances de ces algorithmes (liés aux changements de loi).

1 Présentation des problèmes de bandit étudiés

Un modèle de bandit stochastique à plusieurs bras (ou simplement modèle de bandit dans la suite) est une collection de K lois de probabilités ν = (ν 1 , . . . , ν K ) supposées indépendantes, que l'on désigne par ≪ bras ≫ ou ≪ options ≫. On note µ a la moyenne du bras a, c'est-à-dire de la distribution associée ν a , et on introduit µ * = max a µ a et a * tel que µ * = µ a * . Un agent, qui ne connaît pas ν, peut interagir avec ce modèle de bandit. Il choisit à chaque instant t un bras A t et observe une réalisation de la loi sous-jacente, X t ∼ ν At . Le bras A t est choisi en fonction des observations passées de l'agent, A 1 , X 1 , . . . , A t-1 , X t-1

La suite de variables aléatoires (A t ) t∈N * est la stratégie d'échantillonnage des bras adoptée par l'agent, parfois appelée politique ou algorithme de bandit. Elle est notée A dans la suite. Bien évidemment, cette stratégie va fortement dépendre de l'objectif de l'agent, c'est-à-dire du problème de bandit considéré. Dans cette thèse il sera question de deux objectifs différents, qui visent globalement à identifier les meilleurs bras, mais sous des contraintes différentes.

Le terme ≪ bandit ≫ provient des bandits manchots, qui désignent les machines à sous. Le cadre probabiliste décrit ci-dessus peut en effet modéliser un casino, où l'on suppose que lorsqu'on tire son bras, chaque machine à sous délivre une récompense qui suit une certaine loi de probabilité (inconnue du joueur, bien évidemment). Un objectif naturel pour l'agent (le joueur) est de maximiser la somme des récompenses obtenues pendant son interaction avec le modèle de bandit (les machines à sous). Cette somme de récompenses étant aléatoire, un objectif raisonnable est de s'attacher à construire une stratégie maximisant pour un horizon T donné (le temps de jeu) l'espérance de la somme des récompenses obtenues. C'est cet objectif qui est considéré dans l'article de [Thompson, 1933] présentant le premier algorithme de bandit. Le cadre applicatif des essais cliniques qui y est décrit a véritablement motivé l'étude d'algorithmes de bandits (au contraire de l'exemple du casino, qui n'est qu'un prête-nom) et nous le présentons ici. Un médecin possède pour un symptôme donné K traitements possibles. Une probabilité de guérison inconnue, p a , est associée au traitement a. Lorsque qu'il donne traitement A t au t-ème patient, il observe la réponse du patient X t , qui vaut 1 s'il guérit, 0 sinon (et qui constitue en quelque sorte la 'récompense' du médecin). On peut faire l'hypothèse que X t suit une loi de Bernoulli de moyenne p At , et que les réponses des différents patients sont indépendantes. L'objectif du médecin est de construire une stratégie d'allocation des traitements (A t ) qui maximise l'espérance du nombre de patients guéris, soit l'espérance de la somme de ses récompenses. La stratégie du médecin sera alors un compromis entre exploration (essai des traitements peu donnés pour estimer leur efficacité) et exploitation (tendance à privilégier le traitement qui a paru le plus efficace jusque-là).

Le fait d'interpréter les observations des bras comme des récompenses nous place naturellement dans le cadre plus général de l'apprentissage par renforcement, où l'interaction d'un agent avec son environnement est modélisée par un Processus Décisionnel de Markov (MDP pour Markov Decision Process en anglais). Un MDP est un quadruplet (X , A, P, R) où X désigne l'espace d'états, A l'espace d'actions, P ∶ X , A → M 1 (X ) le noyau de transition et R ∶ X , A → M 1 (X ) le noyau de récompenses. Lorsqu'il est dans l'état x, et qu'il choisit l'action a, l'agent rec ¸oit une récompense r ∼ R(⋅ x, a) et effectue une transition vers un état y ∼ P(⋅ x, a). Le but de l'agent est de trouver une politique (une fonction indiquant quelle action choisir dans un état donné) qui maximise l'espérance de ses récompenses dans un MDP de paramètres inconnus. Le problème de bandit décrit ci-dessus peut donc être vu comme le problème d'apprentissage par renforcement le plus simple : on a un seul état, x 0 , et le le noyau de récompenses est donné par R(⋅ x 0 , a) = ν a . Nous verrons au chapitre 3 que certains des algorithmes proposés dans cette thèse peuvent se généraliser au cadre de l'apprentissage par renforcement. Par ailleurs, la théorie des Processus Décisionnels de Markov nous sera utile au chapitre 1, où nous verrons que la formulation bayésienne d'un problème de bandit peut être modélisée par un MDP.

Un objectif différent peut être envisagé par l'agent : celui d'identifier le ou les meilleurs bras (c'està-dire ceux qui ont les moyennes les plus élevées), mais sans la contrainte de maximiser la somme des réalisations des bras X t obtenues. Ces dernières ne sont alors plus perc ¸ues comme des récompenses. Pour bien comprendre la différence avec le problème précédent, considérons un autre exemple qui motive actuellement l'étude des modèles de bandits : celui de la publicité en ligne.

Un site Internet dispose d'un (ou plusieurs) emplacements publicitaires et son gestionnaire peut choisir les annonces qu'il veut y mettre parmi un panel de K publicités. Il est payé par les annonceurs en fonction du nombre de clics sur leur publicité. On peut modéliser de manière simple la réponse X t du t-ème visiteur du site (clic ou non-clic sur la publicité qu'on lui présente) par une variable aléatoire de Bernoulli de moyenne p a si on lui présente la publicité a. Le gestionnaire du site peut choisir de maximiser le nombre de clics -la somme des X t -sans chercher à estimer précisément la probabilité de clic sur chacune des publicités, ce qui revient à maximiser ses récompenses dans le modèle de bandit associé. Le nombre de visiteurs du site étant grand, il peut aussi décider de procéder en deux phases : d'abord déterminer le (ou les) publicités ayant les probabilité de clic les plus élevées (en acceptant de présenter des 'mauvaises' publicités, et donc de perdre de l'argent pendant cette phase). Au terme de cette phase, il ne présentera plus que les meilleurs publicités sur son site. S'il adopte une telle stratégie, le gestionnaire du site dissocie la phase d'exploration de la phase d'exploitation.

Lorsque l'objectif de l'agent est d'identifier le(s) meilleur(s) bras, on parlera en effet d'exploration pure, par opposition au compromis entre exploration et exploitation auquel il faut parvenir lorsque l'objectif est la maximisation des récompenses. Si la maximisation des récompenses peut être vue comme un problème d'apprentissage par renforcement, l'identification des meilleur bras est plutôt un cas particulier de problème d'optimisation d'une fonction bruitée. Nous allons voir que les algorithmes pour ces deux objectifs, ainsi que leur complexité, sont de nature différente. CONTENTS en fonction de leurs moyennes :

d(µ, µ ′ ) = KL(ν ḃ-1 (µ) , ν ḃ-1 (µ ′ ) ), (2)
où KL(p, q) désigne la divergence de Kullback-Leibler (ou KL-divergence) entre les distributions p et q, définie par KL(p, q) = ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ ∫ log dp dq (x) dp(x) si q ≪ p, +∞ sinon.

Les modèles de bandit binaires sont un cas particulier des modèles de bandit exponentiels, si la loi de Bernoulli de moyenne µ est paramétrée par son paramètre naturel θ = log(µ (1 -µ)). Pour les bandits binaires, on a d(x, y) = KL(B(x), B(y)) = x log x y + (1 -x) log 1 -x 1 -y .

1.1 Maximisation des récompenses : mesure de performance et objectifs Soit ν = (ν 1 , . . . , ν K ) un modèle de bandit. On rappelle que µ * = max a µ a désigne la moyenne du meilleur bras. Un algorithme de bandit A = (A t ) t∈N qui maximise les récompenses minimise de manière équivalente une quantité appelée regret, qui mesure l'écart entre la récompense moyenne obtenue si on n'avait tiré que le meilleur bras et la récompense moyenne effectivement obtenue par la stratégie. Le regret d'une stratégie A à l'horizon T est défini par

R ν (T, A) = E ν T µ * - T t=1 X t .
Le regret peut aussi se réécrire de la manière suivante en introduisant N a (t), le nombre de tirages du bras a entre les instants 1 et t : (

) 3
La notion de regret a été introduite par [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF], qui donnent également une borne inférieure sur le regret d'une stratégie A vérifiant R ν (T, A) = o(T α ) pour tout α ∈]0, 1[ et tout modèle de bandit dans la classe M possédant un unique bras optimal. Un telle stratégie est dite uniformément efficace. Le résultat de Lai et Robbins est valable pour certaines classes de modèles de bandit paramétriques telles que les distributions des bras dépendent d'un paramètre réel. En particulier, il est vrai pour M une classe de modèles de bandit exponentiels, et nous l'énonc ¸ons dans ce cadre, en rappelant que la fonction d(µ, µ ′ ) désigne la divergence de Kullback-Leibler (dans une famille exponentielle donnée) entre les distributions de moyennes µ et µ ′ .

Théorème 1. Soit A un algorithme uniformément efficace. Pour tout modèle de bandit ν ∈ M, et tout bras sous-optimal a, lim inf

T →∞ E ν [N a (T )] log(T ) ≥ 1 d(µ a , µ * )
En utilisant (3), on obtient la borne inférieure suivante sur le regret :

lim inf T →∞ R ν (T, A) log(T ) ≥ a∶µa<µ * (µ * -µ a ) d(µ a , µ * ) . (4)
Un algorithme dont le regret atteint la borne (4) est dit asymptotiquement optimal. Des raffinements successifs dans l'analyse des algorithmes de bandit ont conduit à l'introduction d'algorithmes asymptotiquement optimaux pour lesquels une analyse à horizon fini est proposée, c'est-à-dire une majoration non asymptotique de leur regret. Un exemple de tel algorithme est l'algorithme KL-UCB, introduit par [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF]]. Cet algorithme s'inscrit dans la lignée des politiques d'indices dites de type UCB, c'est-à-dire utilisant un sommet d'intervalle de confiance (Upper Confidence Bound en anglais) (voir par exemple [Auer et al., 2002a, Audibert et al., 2009]). Une politique d'indices calcule pour chaque bras un indice ne dépendant que des observations passées de ce bras, et choisit le bras d'indice maximal. KL-UCB choisit à l'instant t+1 le bras A t+1 = argmax a u a (t), où u a (t) est l'indice suivant, qui apparaît comme le sommet d'un intervalle de confiance basé sur la divergence de Kullback-Leibler : u a (t) = sup{q ≥ μa (t) ∶ N a (t)d (μ a (t), q) ≤ β(t)}, avec β(t) = log t + 3 log log t (5) où μa (t) désigne la moyenne empirique des observations issues du bras a collectées entre les instants 1 et t, et d est la fonction de divergence associée à la famille exponentielle considérée (voir ( 2)). Dans cette thèse, nous avons cherché à proposer de nouveaux algorithmes de bandits qui, tout en conservant la propriété d'optimalité asymptotique dans des modèles simples, ont de meilleures performances pratiques et un meilleur pouvoir de généralisation. Pour ce faire, nous avons adopté une approche bayésienne, discutée à la section 2.

Identification des meilleurs bras : mesure de performance et objectifs

Fixons m ∈ {1, . . . , K} et supposons maintenant que l'agent cherche à identifier les m meilleurs bras. On note (µ [1] , . . . , µ [K] ) le réarrangement décroissant des moyennes des bras, et on considère des classes M m de modèles de bandit telles que pour tout ν ∈ M m , µ [m] > µ [m+1] , de sorte que l'ensemble S * m des m bras ayant les plus grandes moyennes est défini sans ambiguïté.

Comme dans le cadre de la maximisation des récompenses, l'agent choisit séquentiellement les bras dont il veut obtenir des échantillons, selon une règle d'échantillonnage (A t ) t∈N , qui détermine quel bras tirer en fonction des observations passées. Mais il lui appartient aussi de décider quand arrêter son échantillonnage, selon une règle d'arrêt τ , qui est un temps d'arrêt par rapport à la filtration F t = σ(A 1 , X 1 , . . . , A t , X t ), et de choisir un ensemble de m bras Ŝm (F τ -mesurable) selon une règle de recommandation. Le triplet A = ((A t ), τ, Ŝm ) constitue sa stratégie.

La stratégie de l'agent peut être adaptée à deux contraintes différentes considérées dans la littérature, que l'on comprend bien à travers un exemple pratique. Imaginons qu'une entreprise cherche à déterminer les m meilleurs produits parmi K possibles afin de les lancer sur le marché, et puisse pendant une phase de test les proposer à des clients pour observer leur réaction (la réponse à chaque produit étant modélisée par des échantillons d'une distribution qui lui est associée). Pendant cette phase de test, l'entreprise perd de l'argent puisqu'elle accepte de présenter des mauvais produits à ses clients. Pour des raisons économiques, on peut donc imaginer qu'elle fixe le nombre de clients participants à l'étude de marché, et cherche alors à minimiser la probabilité de ne pas trouver les m meilleurs produits. Si ce ≪ budget ≫ est choisi trop petit, la probabilité d'erreur ne pourra peut-être pas être rendue très petite. Un autre type de contrainte possible est que l'entreprise fixe un seuil pour la probabilité d'erreur. Elle veut identifier les m meilleurs produits avec une probabilité supérieure à 0.95 par exemple : son but est alors d'atteindre ce seuil en minimisant le nombre de clients impliqués dans l'étude.

La formulation mathématique de ces deux contraintes est la suivante. Dans le cadre de l'identification des meilleurs bras à budget fixé (fixed-budget setting en anglais), le nombre de tirages des bras τ est fixé CONTENTS à l'avance (τ = t, où t est le budget) et on cherche à trouver une règle d'échantillonnage et une règle de recommandation qui minimisent la probabilité d'erreur, que l'on note p t (ν) ∶= P ν ( Ŝm ≠ S * m ). Dans ce cadre, une stratégie est dite consistante si pour tout problème de bandit ν ∈ M m , p t (ν) tend vers 0 lorsque t tend vers l'infini. Dans le cadre de l'identification des meilleurs bras à niveau de confiance fixé (fixed-confidence setting en anglais), on cherche à construire des stratégies dites δ-PAC (pour Probably Approximately Correct en anglais), dont la probabilité d'erreur est majorée par δ sur tous les modèles de bandit : ∀ν ∈ M m , P ν ( Ŝm ≠ S * m ) ≤ δ. Dans ce cadre, l'objectif est de construire des stratégies δ-PAC qui minimisent le nombre moyen d'observations utilisées, E ν [τ ].

Par analogie avec le problème de minimisation du regret considéré plus haut, nous nous sommes posé la question suivante : comment définir des algorithmes optimaux pour l'identification des meilleurs bras avec un budget ou un niveau de confiance fixé ? La notion d'optimalité asymptotique via-à-vis du regret est en effet bien justifiée par la borne de Lai et Robbins, et on a montré dans la section précédente que, pour M une classe de modèles de bandit exponentiels, pour tout ν ∈ M,

inf stratégies A lim sup T →∞ R ν (T, A) log(T ) = a∶µa<µ * (µ * -µ a ) d(µ a , µ * ) .
Le terme de droite peut s'interpréter comme un terme de complexité qui dépend du modèle de bandit ν et qui fait intervenir un quantité informationnelle (la divergence de Kullback-Leibler). Nous avons vu qu'il existe des algorithmes atteignant cette complexité (c'est-à-dire réalisant l'infimum ci-dessus), que l'on a qualifié d'asymptotiquement optimaux. Pour l'identification des meilleurs bras, nous proposons les termes de complexité suivants pour un budget fixé (κ B (ν)) et un niveau de confiance fixé (κ C (ν)) :

κ B (ν) = inf A consistant lim sup t→∞ - 1 t log p t (ν) -1 et κ C (ν) = inf A δ-PAC lim sup δ→0 E ν [τ ] log 1 δ . (6)
De manière heuristique, pour atteindre une probabilité d'erreur δ, un algorithme optimal (au sens des complexités ci-dessus) à budget fixé devrait utiliser un budget t ≃ κ B (ν) log(1 δ) et un algorithme optimal à niveau de confiance fixé nécessite un nombre moyen d'échantillons E ν [τ ] ≃ κ C (ν) log(1 δ).

On peut donc naturellement se poser la question de la comparaison entre κ B (ν) et κ C (ν). La littérature abondante sur l'identification des meilleurs bras (nous renvoyons le lecteur au chapitre 5 pour une bibliographie détaillée) fournit des bornes supérieures et inférieures sur la probabilité d'erreur d'un algorithme consistant (pour un budget fixé) ou le nombre moyen d'échantillons utilisés par un algorithme δ-PAC (pour un niveau de confiance fixé), qui conduisent naturellement à des encadrement de κ B (ν) et κ C (ν) respectivement. Toutefois, un écart persiste entre les bornes supérieures et inférieures, ce qui ne permet pas d'identifier les complexités. Ces bornes font intervenir des constantes multiplicatives (non nécessairement explicites) et elles sont obtenues pour des modèles de bandit où les bras sont des distributions sous-gaussiennes1 , faisant intervenir la quantité

H(ν) = K a=1 1 ∆ 2 a avec ∆ a = ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ µ a -µ [m+1] pour a ∈ S * m , µ [m] -µ a
pour a ∈ (S * m ) c . Pour des bandits binaires (qui forment un cas particulier de distribution 1/4 sous-gaussiennes), l'écart entre les moyennes de deux bras (µ a -µ ′ a ) 2 apparait d'après l'inégalité de Pinsker 2 comme une approximation de la divergence de Kullback-Leibler entre B(µ a ) et B(µ ′ a ). Par analogie avec la minimisation du regret, on pourrait donc s'attendre à des termes de complexité κ B (ν) et κ C (ν) qui dépendent de quantités informationnelles. Nous présentons en section 3 nos contributions relatives à l'identification et la comparaison des complexités de l'identification des meilleurs bras pour un budget ou un niveau de confiance fixés. Nous chercherons en particulier à identifier les quantités informationnelles caractéristiques des problèmes d'identification des meilleurs bras.

Des algorithmes bayésiens pour la maximisation des récompenses

Cette section présente nos contributions relatives à l'analyse d'algorithmes de bandit d'inspiration bayésienne pour l'objectif de maximisation des récompenses. Celles-ci seront détaillées dans les chapitres 2 à 5 de ce document. La section 2.1 présente les modèles de bandit bayésiens, ainsi que le critère de performance qui leur est associé. Nous nous intéressons également à des approximations de la solution bayésienne du problème de bandit, comme une approximation basée sur les indices de Gittins à horizon fini. Notre objectif était de proposer de nouveaux algorithmes asymptotiquement optimaux du point de vue du regret. Bien que ce dernier soit une quantité fréquentiste, nous avons pu obtenir de telles garanties pour deux algorithmes bayésiens, Bayes-UCB et Thompson Sampling : nous présentons les analyses proposées en section 2.2. Ces deux algorithmes peuvent également être utilisés dans des modèles de bandit plus généraux, et nous discutons en particulier l'exemple des modèles contextuels linéaires à la section 2.3.

Deux approches probabilistes d'un problème de bandit

Nous considérons dans cette section des modèles de bandit paramétriques, de la forme ν = ν θ = (ν θ 1 , . . . , ν θ K ), où la distribution ν a du bras a dépend d'un paramètre θ a ∈ Θ. On note θ = (θ 1 , . . . , θ K ) ∈ Θ K le paramètre global du modèle. Comme dans tout modèle paramétrique, deux approches sont possibles : l'approche fréquentiste où l'on considère que θ est un paramètre inconnu, et l'approche bayésienne, où l'on considère que θ est une variable aléatoire, qui suit une loi a priori Π 0 .

Modèle de bandit fréquentiste

Modèle de bandit bayésien θ ∈ Θ K est un paramètre inconnu -θ est tiré sous Π 0 , une loi a priori sur Θ K -∀a, (X a,t ) est i.i.d. de loi ν θa et de moyenne µ a -∀a, conditionnellement à θ a , (X a,t ) est i.i.d. de loi ν θa et de moyenne µ a -(X a,t ) a,t est une famille indépendante -conditionnellement à θ, (X a,t ) a,t est une famille indépendante Dans les deux modèles, l'agent choisit à l'instant t un bras A t à tirer et observe la récompense X t = X At,t issue du bras choisi. En introduisant la filtration F t = σ(A 1 , X 1 , . . . , A t , X t ), si l'agent adopte une stratégie déterministe, la variable aléatoire A t est F t-1 -mesurable, alors que s'il adopte une stratégie randomisée, A t est tirée selon une loi p t sur {1, . . . , K}, et c'est le vecteur de probabilités p t qui est F t-1 -mesurable.

La notion de regret introduite en section 1.1 est une mesure de performance associée au modèle fréquentiste, puisqu'il dépend du modèle ν, ou de manière équivalente, du paramètre θ dans notre cadre P θ et E θ désignent la probabilité et l'espérance sous le modèle fréquentiste (qui dépend de θ). Notons E Π 0 et P Π 0 la probabilité et l'espérance sous le modèle bayésien. Une stratégie qui maximise l'espérance de la somme des récompenses sous le modèle bayésien minimise de manière équivalente le risque bayésien (ou regret bayésien), défini par

BR Π 0 (T, A) = E Π 0 T µ * - T t=1 X t = E Π 0 [R θ (T, A)] .
La borne inférieure de [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF]] nous a permis de définir des stratégies asymptotiquement optimales par rapport au regret, et nous en avons donné des exemple en section 1.1. Du point de vue du risque bayésien on peut aller plus loin et montrer qu'il existe une stratégie optimale, qui minimise le risque bayésien parmi toutes les stratégies possibles.

Ces deux formulations fréquentiste et bayésienne d'un même problème de bandit (celui de maximiser l'espérance de la somme des récompenses) peuvent être dissociées des outils qui leur sont associés. Dans la littérature liée à la minimisation du regret, les algorithmes proposés sont basés sur des estimateurs du maximum de vraisemblance des paramètres inconnus des bras, et sur des intervalles de confiances, que nous pouvons qualifier d'outils fréquentistes. A l'inverse, nous appellerons algorithmes bayésiens des algorithmes qui pour choisir le bras A t+1 se basent sur la loi a posteriori de θ, Π t (θ) = L(θ A 1 , X 1 , . . . , A t , X t ), qui est la loi conditionnelle de θ sachant les observations obtenues jusqu'à l'instant t. Nous pouvons ainsi nous intéresser au risque bayésien d'un algorithme 'fréquentiste', et inversement, et c'est ce qui sera au coeur de cette thèse, à la performance d'algorithmes bayésiens évaluée en terme de regret.

Dans un premier temps, nous allons nous demander si la solution optimale du problème bayésien ellemême, ou certaines de ses approximations, peuvent fournir de nouveaux algorithmes asymptotiquement optimaux du point de vue du regret. Nous commenc ¸ons par décrire ces premiers algorithmes bayésiens.

La solution bayésienne du problème de bandit. L'existence d'une solution à la minimisation du risque bayésien vient du fait qu'elle peut s'interpréter comme un problème de planification dans un Processus Décisionnel de Markov (MDP) associé. Pour simplifier la présentation, nous allons exhiber cette solution pour un modèle de bandit binaire, avec des loi a priori uniformes indépendantes sur chaque moyenne.

Soit ν = (B(µ 1 ), . . . , B(µ K ) un modèle de bandit binaire. On fait l'hypothèse que pour tout a, θ a ∼ U([0, 1]) et que les θ a sont indépendantes. La loi a posteriori sur θ = (µ 1 , . . . , µ K ) à l'instant t prend la forme d'un produit de K marginales indépendantes Π t = (π t 1 , . . . , π t K ) et la loi a posteriori sur µ a est donnée par π t a = Beta(S a (t) + 1, N a (t) -S a (t) + 1), où S a (t) est la somme des récompenses obtenues du bras a entre les instants 1 et t, et on le rappelle N a (t) est le nombre de tirages du bras a entre les instants 1 et t. La loi Beta(a, b) admet pour densité par rapport à la mesure de Lebesgue

f (a,b) (x) = Γ(a + b) Γ(a)Γ(b) x a-1 (1 -x) b-1 1 [0,1] (x).
L'histoire du jeu de bandit peut donc être résumée par une table (A a , B a ) K a=1 ∈ (N × N) K qui indique les paramètres des lois a posteriori Beta courantes associées à chaque bras. Cet état évolue dans le Processus Décisionnel de Markov dont l'espace d'état est X = (N × N) K , l'espace d'action est A = {1, . . . , K} et la transition est la suivante.

Supposons qu'à l'instant t on soit dans l'état S t = (A a , B a ) K a=1 , ce qui signifie que la loi a posteriori π t-1 a sur la moyenne du bras a est Beta(A a , B a ). Si l'action A t = a est choisie, une récompense X t = X a,t est tirée selon B(µ a ) et l'état est mis à jour de la manière suivante :

A a ← A a + X t B a ← B a + (1 -X t ).
et pour i ≠ a, A i et B i ne changent pas. Les fonctions de transition et de récompense dans ce MDP sont connues puisqu'on a

P Π 0 (X t S t = (A i , B i ) K i=1 , A t = a) = A a A a + B a
(qui est la moyenne d'une loi Beta de paramètre A a et B a ). La théorie des MDP (voir par exemple [START_REF] Sigaud | Processus Decisionnels de Markov et Intelligence artificielle[END_REF]) nous dit alors qu'il existe une politique φ * ∶ X × [0, T ] → A, indiquant quelle action choisir en fonction de l'état et de l'instant de jeu, qui maximise la somme des récompenses jusqu'à l'horizon T , E φ Π 0 ∑ T t=1 X t . C'est-à-dire que la stratégie d'échantillonnage A t = φ * (S t , t) est solution du problème de bandit bayésien considéré. Cette politique φ * est solution d'une équation de programmation dynamique, et peut dans le cas d'une horizon fini être calculée par récurrence.

Indices de Gittins et politique optimale. Pour des bandits binaires, le calcul de la politique optimale est théoriquement possible par récurrence, mais il est très couteux du fait de la taille de l'espace d'état et ne pourra être effectif que pour des petits horizons. Plus généralement, pour tout modèle de bandit paramétrique dans lequel les lois a priori sur les paramètres θ a sont indépendantes, [Gittins, 1979] a montré que lorsqu'on cherche à maximiser la somme des récompenses actualisées, c'est-à-dire

E Π 0 ∞ t=1 α t-1 X t ,
pour un certain coefficient d'actualisation α ∈]0, 1[, la solution bayésienne se réduit à une politique d'indices. Chacun de ces indices de Gittins G α (π) (qui dépend d'une loi a posteriori π et du coefficient d'actualisation) peut être obtenu comme solution d'une équation de programmation dynamique dans un espace d'état réduit. La définition des indices de Gittins pour un critère actualisé peut naturellement être transposée à un critère à horizon fini, et nous introduisons les indices de Gittins à horizon fini, G(π, n), qui dépendent de π, loi a posteriori sur un bras et du temps restant n = Tt + 1. Ces deux types d'indices peuvent être définis de la manière suivante : [τ ] .

G α (π) = sup τ >0 E θ∼π ∑ τ t=1 α τ -1 Y t E θ∼π [∑ τ t=1 α τ -1 ] et G(π, n) = sup 0<τ ≤n E θ∼π [∑ τ t=1 Y t ] E θ∼π
où conditionnellement à θ, la suite (Y t ) est i.i.d. de loi ν θ et où le supremum porte sur l'ensemble des temps d'arrêt τ par rapport aux (Y t ), bornés par n dans le second cas. G(π, n) représente la récompense moyenne par unité de temps que l'on peut obtenir d'un bras dont la loi a posteriori courante est π, si on peut en collecter au plus n réalisations.

CONTENTS

Au chapitre 1, nous présenterons une définition équivalente des indices de Gittins, basée sur un problème de calibration associé à un bras. Cette interprétation permettra de fournir une méthode de calcul des indices de Gittins à horizon fini pour les bandits binaires, et de voir que contrairement au cadre actualisé, lorsqu'on fixe un horizon fini T , la politique d'indices associée aux indices de Gittins à horizon fini (appelée FH-Gittins pour Finite-Horizon Gittins algorithm), qui choisit à l'instant t A t = argmax a G(π t-1 a , Tt + 1), ne coïncide pas en général avec la solution bayésienne présentée plus haut. Toutefois, nous conjecturons que cette politique d'indices est une bonne approximation de la solution bayésienne. Cette conjecture est supportée par des expériences numériques montrant que pour des horizons courts où la solution optimale peut être calculée, le risque bayésien de l'algorithme FH-Gittins est très proche de celui de la stratégie optimale. Elle est aussi étayée par des approximations obtenues pour les indices de Gittins à horizon fini qui montrent que ceux-ci sont proches des indices utilisés par une variante de l'algorithme KL-UCB, que nous appelons KL-UCB-H + . KL-UCB-H + est la politique d'indices associée à u H,+ a (t) = sup q ≥ μa (t) ∶ N a (t)d (μ a (t), q) ≤ log

T N a (t) + c log log T N a (t) (7)
et [Lai, 1987] a prouvé, en fournissant une borne inférieure asymptotique sur le risque bayésien de tout algorithme, que cet algorithme constitue une bonne approximation de la solution bayésienne, pour des grandes valeurs de T . Aux chapitres 2 et 3 nous proposerons également des expériences numériques montrant les bonnes performances de l'algorithme FH-Gittins en terme de regret, mais nous n'avons pas pu obtenir de garanties théoriques pour justifier ce constat empirique. De plus, cet algorithme reste difficile à implémenter, et nous n'avons pu le tester que pour des horizons T ≤ 1000. Nous nous sommes donc focalisés dans la suite sur d'autres algorithmes bayésiens, plus facile d'implémentation, pour lesquels la propriété d'optimalité asymptotique vis-à-vis du regret sera établie.

Les algorithmes Bayes-UCB et Thompson Sampling

Nous introduisons dans cette section deux algorithmes bayésiens, pour lesquels nous proposons en particulier des analyses à temps fini montrant leur optimalité asymptotique dans des modèles de bandit binaires. Bayes-UCB, qui fait l'objet du chapitre 3, est une politique d'indices basée sur des quantiles bien choisi de la distribution a posteriori. L'échantillonnage de Thompson, ou Thompson Sampling, qui fait l'objet du chapitre 4, est un algorithme randomisé qui tire un bras selon sa probabilité a posteriori d'être optimal. Cette idée a été introduite par [Thompson, 1933] dans le premier article de bandit, mais les premiers résultats théoriques proposés pour cet algorithmes datent de la fin des années 2000, et la question de son optimalité asymptotique était encore ouverte.

BAYES-UCB

Bayes-UCB est une politique d'indices basée sur le principe d'optimisme qui a conduit à toute une famille de politiques d'indices fréquentistes. Ainsi l'algorithme UCB1 de [Auer et al., 2002a] (pour des bandits à support borné) choisit à l'instant t + 1 le bras

A t+1 = argmax a μa (t) + 2 log(t) N a (t) ,
et l'indice calculé peut être vu comme le sommet d'un intervalle de confiance obtenu avec l'inégalité de Hoeffding, alors que l'indice u a (t) utilisé par KL-UCB ( 5) est le sommet d'un intervalle de confiance construit avec l'inégalité de Chernoff (cf. chapitre 1). On parle de principe d'optimisme car pour chaque bras, un intervalle de confiance sur la moyenne inconnue est construit, et parmi tous les modèles statistiquement possibles, on agit optimalement de le meilleur des modèles possible (celui où les moyennes de tous les bras sont égales au sommet de leur intervalle de confiance). L'algorithme Bayes-UCB est basé sur ce même principe, mais les intervalles de confiance sont remplacés par des régions de confiance bayésiennes.

Soit ν = (ν θ 1 , . . . , ν θ K ) un modèle de bandit paramétrique. On suppose que les paramètres (θ a ) 1≤a'K sont tirés indépendamment selon des lois a priori (π 0 a ) 1≤a≤K . Soit π t a la loi a posteriori du paramètre θ a après t instants, et soit λ t a l'a posteriori sur la moyenne µ a . Si a l'instant t le bras A t = a est choisi, les distributions a posteriori sont mises à jour de la manière suivante :

π t a (θ) ∝ ν θ (X t ) π t-1 a (θ) , et pour tout i ≠ a, π t i = π t-1 i . (8)
L'algorithme Bayes-UCB dépend d'une famille de lois a priori Π 0 = (π 0 a ) 1≤a≤K et d'un paramètre réel c. Pour t = 1, . . . , K, l'algorithme tire les bras l'un après l'autre. Puis pour t ≥ K, le bras choisi à l'instant t + 1 est

A t+1 = argmax a=1...K Q 1 - 1 t(log t) c ; λ t a , où Q(α, π) désigne le quantile d'ordre α de la distribution π, défini par P X∼π (X ≤ Q(α, π)) = α.
Une illustration de l'algorithme est proposée sur la figure 1, pour des bandits binaires, où on le compare à KL-UCB. Dans les deux cas, on voit que le bras optimal est tiré la plupart de temps, conduisant à un intervalle de confiance resserré pour la moyenne de ce bras, ou à une loi a posteriori concentrée. Bayes-UCB peut être implémenté dans des classes de modèles de bandit exponentiels (définis par (1)) où les lois a posteriori sur les moyennes des bras ont des expressions explicites, sous réserve de choisir un a priori conjugué. Dans l'exemple des bandits binaires, paramétrés par leur moyenne, avec un a priori uniforme sur les moyenne, on a π t a = λ t a = Beta(S a (t) + 1, N a (t) -S a (t) + 1) et l'algorithme CONTENTS Bayes-UCB s'écrit

A t+1 = argmax a=1...K Q 1 - 1 t(log t) c ; Beta(S a (t) + 1, N a (t) -S a (t) + 1) .
Le résultat suivant, basé sur un encadrement précis de la queue d'une loi Beta, montre une connexion forte avec l'algorithme KL-UCB présenté en section 1.1. Bayes-UCB semble construire de manière automatique des intervalles de confiance basés sur la divergence de Kullback-Leibler.

Lemme 2. Soit d(x, y) la divergence de Kullback-Leibler entre deux lois de Bernoulli de paramètre x et y. La quantile d'a posteriori q a (t) utilisé par l'algorithme Bayes-UCB de paramètre c vérifie

ũa (t) ≤ q a (t) ≤ u a (t) , avec u a (t) = argmax x> Sa(t) Na(t) d S a (t) N a (t) , x ≤ log(t) + c log(log(t)) N a (t) , ũa (t) = argmax x> Sa(t) Na(t)+1 ⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ d S a (t) N a (t) + 1 , x ≤ log t Na(t)+2 + c log(log(t)) (N a (t) + 1) ⎫ ⎪ ⎪ ⎪ ⎬ ⎪ ⎪ ⎪ ⎭ .
Le chapitre 2 montre d'autres situations où Bayes-UCB présente des similarités avec des algorithmes fréquentistes existants. Par exemple, lorsque les distributions des bras sont gaussiennes de moyenne et de variance inconnues, on retrouve un algorithme proche de UCB1-norm proposé par [Auer et al., 2002a], et plus efficace en pratique.

Dans le cadre des bandits binaires, nous avons pu montrer le résultat suivant. Le fait qu'il soit vrai pour tout > 0 indique qu'on a bien lim sup

T →∞ E θ [N a (T )] log(T ) ≤ 1 d(µ 2 , µ 1 )
, et donc que la borne inférieure de Lai et Robbins est atteinte.

Théorème 3. L'algorithme Bayes-UCB avec un a priori uniforme sur les moyennes et un paramètre c = 5 vérifie, pour tout > 0 et tout T tel que

log T + 5 log log T ≥ d(µ 2 , µ 1 ) 1 + exp 8 (µ 1 (1 -µ 1 )) 2 (1 + ) 2 2 d(µ 2 , µ 1 ) 2 , E θ [N a (T )] ≤ 1 + d(µ a , µ 1 ) log(T ) + log T + 5 log log T 2π(1 + ) 3 d ′ (µ a , µ 1 ) 2 d(µ a , µ 1 ) 3 + 1 + d(µ a , µ 1 ) + 2e + 3 1 -µ 1 log log T + 27 + 2(1 + ) 2 d ′ (µ a , µ 1 ) d(µ a , µ 1 )
Ce théorème repose sur le lien avec des indices proches de ceux utilisés dans KL-UCB, ce qui permet d'adapter l'analyse en temps fini proposée par [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF]. Afin de comprendre quels outils mathématiques sont au coeur de la preuve, nous en présentons une esquisse.

Supposons sans perte de généralité que le bras 1 est un bras optimal, et soit a un bras sous-optimal. Du fait de la phase d'initialisation, nous avons

E θ [N a (T )] = 1 + E θ T -1 t=K 1 (A t+1 =a) .
L'événement (A t+1 = a) peut ensuite être décomposé de la manière suivante, en fonction de la position de la moyenne du bras optimal µ 1 par rapport à son indice q 1 (t) et en utilisant le fait que si a est tiré à l'instant t + 1, on a q a (t) > q 1 (t) :

(A t+1 = a) ⊆ (µ 1 ≥ q 1 (t), A t+1 ) ⋃ (µ 1 ≤ q 1 (t), A t+1 = a) ⊆ (µ 1 ≥ q 1 (t)) ⋃ (µ 1 ≤ q a (t), A t+1 = a) .
Cette décomposition est celle utilisée par [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF], mais pour l'analyse de Bayes-UCB nous remplacerons µ 1 par µ 1 -g t où g t = 2 log t. Nous obtenons

(A t+1 = a) ⊆ (µ 1 -g t ≥ q 1 (t)) ⋃ (µ 1 -g t ≤ q a (t), A t+1 = a) ⊆ (µ 1 -g t ≥ ũ1 (t)) ⋃ (µ 1 -g t ≤ u a (t), A t+1 = a) ,
en utilisant les notations et résultats du Lemme 2. Finalement, on obtient

E θ [N a (T )] ≤ 1 + T -1 t=K P θ (µ 1 -g t ≥ ũ1 (t)) A + T -1 t=K P θ (µ 1 -g t ≤ u a (t), A t+1 = a) B .
Le terme A est négligeable devant log(T ), car il est peu probable que l'indice ũ1 (t), qui est une borne supérieure sur µ 1 soit plus petit que µ 1 -g t . Pour montrer ceci, nous devons majorer la probabilité

P θ (µ 1 -g t ≥ ũ1 (t)) = P θ (N 1 (t) + 1)d + S 1 (t) N 1 (t) + 1 , µ 1 -g t ≥ log t N 1 (t) + 2 + 5 log log t , où d + (x, y) = d(x, y)1 (x<y)
. Pour cela nous devons établir une inégalité de déviation dit auto-normalisée car le nombre d'observations N 1 (t) est lui même une variable aléatoire. De plus les déviations sont mesurées à l'aide de la divergence de Kullback-Leibler. [START_REF] Garivier | The KL-UCB algorithm for bounded stochastic bandits and beyond[END_REF] proposent le premier résultat de ce type (pour l'analyse de KL-UCB) qui ne résulte pas d'une borne de l'union, et nous avons pu l'adapter (grâce au terme g t ) à la présence d'un biais et au taux d'exploration log(t (N 1 (t) + 2)) au lieu de log t.

Une réécriture classique (un peu astucieuse) du terme B permet de le débarrasser des quantités autonormalisées et en introduisant μa,s la moyenne empirique des s premières observations du bras a, nous avons

(B) ≤ T s=1 P θ (sd + (μ a,s , µ 1 -g s ) ≤ log T + 5 log log T ) .
Le nombre d'échantillons de a nécessaires pour que la moyenne µ 1 > µ a ne soit plus dans un intervalle autour de μa,s construit à l'aide de la divergence de KL est environ log(T ) d(µ a , µ 1 ) et on pourra montrer que (B) = (1 + ) log(T ) d(µ a , µ 1 ) + o(log(T )), ce qui justifie l'ordre de grandeur de la borne obtenue dans le théorème 3, dans laquelle nous tâchons d'expliciter les termes de second ordre.

CONTENTS THOMPSON SAMPLING

Soit toujours ν = (ν θ 1 , . . . , ν θ K ) un modèle de bandit paramétrique tel que les lois a priori sur chacun des paramètres, (π 0 a ) 1≤a≤K , sont indépendantes. On note µ(θ) la moyenne d'un bras paramétré par θ. L'échantillonnage de Thompson (ou Thompson Sampling) consiste à tirer à chaque instant un échantillon des loi a posteriori courantes π t a , et à choisir le bras ayant conduit à l'échantillon correspondant à la moyenne la plus élevée. Plus précisément, à l'instant t + 1,

∀a = 1 . . . K, θ a (t) ∼ π t a A t+1 = argmax a µ(θ a (t))
Cet algorithme peut toujours être interprété comme une politique d'indice, mais l'indice calculé pour chaque bras dépend des observations passées de ce bras et d'une randomisation externe. En particulier, l'indice obtenu n'est pas 'optimiste' au sens précédent, car ce n'est plus une borne de confiance supérieure pour la moyenne µ a : avec probabilité de l'ordre de un demi, il est même plus petit que la moyenne a posteriori. Thompson Sampling implémente un optimisme un peu différent, qui consiste a tirer un modèle selon l'a posteriori courant, et à agir de manière optimale dans ce modèle échantillonné, ce qui correspond à tirer les bras selon leur probabilité a posteriori d'être optimal. Ce principe simple peut aussi être implémenté dans des modèles de bandit plus complexes comme on le verra.

La première borne supérieure logarithmique sur le regret de cet algorithme est donnée par [START_REF] Agrawal | Analysis of Thompson Sampling for the multi-armed bandit problem[END_REF] pour les bandits binaires avec une loi a priori uniforme π U sur les moyennes. Mais leur résultat ne fait pas intervenir les divergences de Kullback-Leibler entre les bras d(µ a , µ 1 ) et ne permet pas de monter l'optimalité asymptotique de l'échantillonnage de Thompson. Dans le même contexte, le théorème suivant que nous prouvons au chapitre 3 permet de montrer que Thompson Sampling est asymptotiquement optimal au sens de la borne de Lai et Robbins.

E θ [N a (T )] ≤ (1 + ) log T d(µ a , µ 1 ) + log(T ) 2π(1 + ) 3 d ′ (µ a , µ 1 ) 2 d(µ a , µ 1 ) 3 + 2(1 + ) 2 d ′ (µ a , µ 1 ) d(µ a , µ 1 ) 2 + 5 + 2C b + N (b).
L'analyse à temps fini que nous proposons est assez proche des analyses que l'on peut proposer pour des politiques d'indices optimistes. Afin de pouvoir adapter de telles analyses, nous avons besoin du résultat suivant, qui montre que le bras optimal est souvent tiré par l'algorithme Thompson Sampling.

Proposition 5. Il existe des constantes b = b(µ 1 , µ 2 ) ∈ (0, 1) et C b < ∞ telles que ∞ t=1 P θ N 1 (t) ≤ t b ≤ C b .
Les éléments clés de notre analyse. Comme notre analyse de Bayes-UCB, celle proposée pour Thompson Sampling n'est valable que pour des bandit binaires puisqu'elle s'appuie sur une propriété spécifique Cette propriété avait déjà été utilisée pour établir un lien entre les quantiles d'a posteriori et les sommets d'intervalles de confiance basés sur la divergence de Kullback-Leibler (Lemme 2). Ce lien sera à nouveau être utile dans notre analyse de Thompson Sampling, où nous introduisons des quantiles d'a posteriori bien choisi, et utilisons le fait qu'avec forte probabilité, les échantillons θ a (t) utilisés par l'algorithme sont inférieurs à ces quantiles. En couplant cette remarque à une nouvelle décomposition de l'événement (A t+1 = a), on pourra avec l'aide de la Proposition 5 proposer une analyse proche de celle de Bayes-UCB. La preuve de cette dernière est plus complexe et utilise pleinement la nature randomisée de l'échantillonnage de Thompson.

Une généralisation. Au chapitre 3, nous présentons également une preuve de l'optimalité asymptotique de l'algorithme de Thompson pour des modèles de bandit exponentiels avec un choix particulier de distribution a priori : l'a priori de Jeffreys.

COMPARAISON NUM ÉRIQUE

Nous proposons au chapitre 3 des expériences numériques comparant les performances de Bayes-UCB et de Thompson Sampling à celles d'autres algorithmes 'fréquentistes' de l'état de l'art, qui sont introduits avec plus de précisions au chapitre 1. La figure 2 présente une estimation de la distributions du regret cumulé des différents algorithmes obtenu à l'aide de N = 50000 répétitions d'un jeu de bandit jusqu'à un horizon T = 20000, pour une modèle de bandit binaire à 10 bras de moyennes µ = [0.1 0.05 0.05 0.05 0.02 0.02 0.02 0.01 0.01 0.01]. Les quatre premiers algorithmes (dont le regret est représenté sur la même échelle) sont des variantes de l'algorithme UCB1 : UCB-Tuned ( [Auer et al., 2002a]) et UCB-V ( [START_REF] Audibert | Exploration-exploitation trade-off using variance estimates in multi-armed bandits[END_REF]) utilisent des intervalles de confiances incorporant la variance empirique des distributions des bras, alors que MOSS ( [Audibert and Bubeck, 2010]) remplace le log(t) dans l'indice UCB par log(t (KN a (t))). Ces quatre algorithmes ne sont pas asymptotiquement optimaux et on constate en effet un écart avec la borne inférieure de Lai et Robbins (en bleu). Les 6 algorithmes suivants (dont le regret est présenté sur la même échelle, différente de la précédente) possèdent au contraire la propriété d'optimalité asymptotique pour les bandits binaires. Des variantes de l'algorithme KL-UCB de [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF], KL-UCB-H + utilisant les indices (7), et KL-UCB+ où le log(T N a (t)) dans ( 7) est remplacé par log(t N a (t)), sont présentées, ainsi que l'algorithme DMED de [START_REF] Honda | An Asymptotically Optimal Bandit Algorithm for Bounded Support Models[END_REF]. On peut constater que Thompson Sampling atteint les même performances que les meilleurs algorithmes fréquentistes, et que Bayes-UCB a des performances similaires à celles de KL-UCB. Par ailleurs, l'utilisation de ces deux algorithmes bayésiens présente également un avantage computationnel, car il est plus facile de calculer un quantile ou de produire un échantillon d'une loi Beta que de calculer l'indice qui intervient dans KL-UCB.

Nous proposons aussi au chapitre 3 des expériences pour une horizon plus court où l'on compare ces algorithmes asymptotiquement optimaux à l'algorithme FH-Gittins, montrant que les performances de ce dernier (quoique grossièrement comparables) sont plus variables en fonctions des modèles de bandit choisis. D'autres expériences où l'on estime le risque bayésien des algorithmes montrent que Thompson Sampling et Bayes-UCB semblent aussi asymptotiquement optimaux vis-à-vis de ce dernier.

Des algorithmes bayésiens pour des modèles plus généraux

Nous avons proposé des garanties théoriques pour Bayes-UCB et Thompson Sampling dans les modèles de bandit les plus simples, où les bras sont notamment indépendants. Si nous revenons à l'exemple de la recommandation de contenu (par exemple de publicité), de l'information contextuelle est disponible, et les réponses d'un utilisateur à deux publicités similaires seront sans doute très corrélées. Pour tenir compte de cette structure des actions, on peut considérer des modèles de bandit contextuels, qui sont présentés plus en détails au chapitre 4. Nous montrons ici que Bayes-UCB et Thompson Sampling peuvent être facilement appliqués dans ces modèles plus généraux, et nous donnons des bornes supérieures sur le regret (sous le modèle bayésien) nouvelles par rapport à la littérature, pour un modèle linéaire.

Nous nous intéressons au modèle de bandit contextuel linéaire suivant, où la notion d'action est remplacée par celle de contexte (ou d'action contexualisée). A chaque instant t, un ensemble D t ⊆ R d de contextes est présenté à l'agent. Il doit choisir un contexte x t ∈ D t et rec ¸oit la récompense

y t = x T t θ + t ,
R θ (T, A) = T t=1 (x * t ) T θ -x T t θ , où x * t = argmax x∈Dt x T θ.
Comme précédemment, le paramètre de régression θ peut être vu comme un paramètre inconnu ou bien on peut faire l'hypothèse qu'il est tiré sous une loi a priori π 0 sur R d . Comme dans le cadre précédent, on peut définir des algorithmes utilisant des outils bayésiens et fréquentistes et s'intéresser aux performances de ces algorithmes sous le modèle fréquentiste (on notera P θ la probabilité associée) ou sous le modèle bayésien (on notera P la probabilité associée, qui dépend implicitement de l'a priori π 0 ). Le modèle bayésien que nous considérons ici fait l'hypothèse d'un a priori et d'un bruit gaussiens :

y t = θ T x t + t , avec θ ∼ N 0, κ 2 I d et t ∼ N 0, σ 2 . ( 9
)
Sous ces hypothèses, en introduisant

X t = ⎛ ⎜ ⎜ ⎜ ⎝ x T 1 x T 2 . x T t ⎞ ⎟ ⎟ ⎟ ⎠ ∈ M t,d (R), Y t = ⎛ ⎜ ⎜ ⎜ ⎝ y 1 y 2 . y t ⎞ ⎟ ⎟ ⎟ ⎠ ∈ R t , et E t = ⎛ ⎜ ⎜ ⎜ ⎝ 1 2 . t ⎞ ⎟ ⎟ ⎟ ⎠ ∈ R t ,
la loi a posteriori sur θ après t observations est gaussienne de moyenne θ(t) et de covariance Σ t , où

θ(t) = (B(t)) -1 X T t Y t avec B(t) = σ 2 κ 2 I d + X T t X t Σ t = σ 2 (B(t)) -1 .
Les algorithmes. L'algorithme Bayes-UCB tel que nous l'avons présenté pour les bandits à bras indépendants peut être utilisé ici, en calculant pour chaque contexte x ∈ D t un quantile de la loi a posteriori sur la moyenne associée, x T θ, qui a pour loi N x T θ(t), x Σt , où x A = √ x T Ax. Si l'algorithme original utilise un quantile d'ordre 1 -1 t, nous le définissons ici en fonction d'un taux d'exploration f (t, δ). Bayes-UCB choisit à l'instant t + 1 le contexte t+1,δ) ; N (0, 1) .

x t+1 = argmax x∈D t+1 x T θ(t) + x Σt Q 1 -e -f (
Les modèles de bandit linéaires tels que ceux que nous considérons ont été largement étudiés dans la littérature et le principe d'optimisme est appliqué de la manière suivante dans ces modèles. Etant donnée une région de confiance C t ⊆ R d pour le paramètre de régression, le contexte choisi est celui dont le produit scalaire avec un des paramètres de régression jugés possibles (dans C t ) est maximal :

x t+1 = argmax x∈D t+1 max θ ′ ∈Ct x T θ ′ .

CONTENTS

Dans les articles de [Auer, 2002, Dani et al., 2008, Rusmevichientong and Tsitsiklis, 2010, Chu et al., 2011, Abbasi-Yadkori et al., 2011], les régions de confiances qui sont construites sont de la forme

C t = θ ′ ∈ R d ∶ θ(t) -θ ′ Σ -1 t ≤ β(t + 1, δ)
. Ce sont des régions de confiance 'fréquentistes', c'està-dire vraies en forte probabilité sous P θ , pour tout paramètre θ. θ(t) ne s'interprète plus alors comme la moyenne de la loi loi a posteriori mais comme un estimateur des moindres carrés régularisé, où κ est un paramètre de régularisation. Les algorithmes associés se réécrivent

x t+1 = argmax x∈D t+1 θ(t) T x + x Σt β(t + 1, δ) . ( 10
)
On peut noter que Bayes-UCB prend cette forme. Il est possible de définir une variante de cet algorithme, qui implémente le principe d'optimisme pour les bandits linéaire tel qu'introduit ci-dessus de manière bayésienne. En effet, une région de confiance pour θ sous le modèle bayésien (9) peut être construite de la manière suivante :

P θ -θ(t) Σ -1 t ≤ Q 1 -e -f (t+1,δ) ; χ 2 d ≥ 1 -e -f (t+1,δ) , (11)
où un quantile d'une loi du chi-deux à d degrés de liberté apparait, car conditionnellement au t premières observations, θ -

θ(t) 2 Σ -1 t ∼ χ 2 d .
Nous définissons alors l'algorithme Bayes-LinUCB comme l'algorithme optimiste associé à cette région de confiance, qui choisit à l'instant t + 1 le contexte

x t+1 = argmax x∈D t+1 θ(t) T x + x Σt Q 1 -e -f (t+1,δ) ; χ 2 d .
L'échantillonnage de Thompson peut également être facilement implémenté dans des modèles de bandit contextuels linéaires. Les bonnes performances de Thompson Sampling dans des modèles plus complexes, les modèles logistiques, avaient d'ailleurs été constatées en pratique avant que des garanties théoriques n'émergent pour les bandits binaires ( [Scott, 2010, Chapelle andLi, 2011]). Dans notre modèle linéaire, à l'instant t + 1, un échantillon θ(t) de la loi a posteriori (gaussienne) sur θ est tiré et le contexte choisi est

x t+1 = argmax x∈D t+1 x T θ(t).
Nos résultats. Nous montrons le résultat suivant pour les algorithmes Bayes-UCB et Bayes-LinUCB.

Théorème 6. Avec le taux d'exploration f (t, δ) = log Kπ 2 t 2 3δ , si pour tout t, D t = K et si les contextes sont bornés par L, l'algorithme Bayes-UCB vérifie, sous le modèle (9),

P ⎛ ⎜ ⎝ ∀T ∈ N, R θ (T, A) ≤ √ T d 2C 1 log Kπ 2 T 2 6δ log 1 + T L 2 σ 2 dκ 2 ⎞ ⎟ ⎠ ≥ 1 -δ. Avec le taux d'exploration f (t, δ) = log π 2 t 2 6δ
, l'algorithme Bayes-LinUCB vérifie, sous le modèle (9) et avec des contextes bornés par L,

P ⎛ ⎜ ⎜ ⎝ ∀T ∈ N, R θ (T, A) ≤ d √ T C 1 log 1 + T L 2 σ 2 dκ 2 ⎛ ⎝ 1 + 2 d log π 2 T 2 6δ + 2 1 d log π 2 T 2 6δ ⎞ ⎠ ⎞ ⎟ ⎟ ⎠ ≥ 1 -δ. où C 1 ∶= 4L 2 κ 2 log(1+L 2 κ 2 σ -2 ) .

VERS DES ALGORITHMES FR ÉQUENTISTES OPTIMAUX POUR L'IDENTIFICATION DES

MEILLEURS BRAS

27

On a donc montré que le pseudo-regret de Bayes-UCB est en forte probabilité (sous le modèle bayésien) de l'ordre de Õ( dT log(K)) (où la notation Õ ignore les facteurs logarithmiques en T ) et celui de Bayes-LinUCB de l'ordre de Õ(d √ T ). Le pseudo-regret de l'algorithme optimiste qui utilise la région de confiance fréquentiste la plus fine, l'algorithme OFUL proposé par [START_REF] Abbasi-Yadkori | Improved Algorithms for Linear Stochastic Bandits[END_REF], est de l'ordre de O(d √ T ) en forte probabilité sous P θ , pour tout θ. En revanche, on ne trouve pas dans l'état de l'art fréquentiste un résultat spécifique dans le cas où le nombre de contexte est fini et la dépendance en dT log(K) obtenue pour Bayes-UCB dans ce cas est dans un certain sens optimale.

Pour Thompson Sampling, on peut déduire de l'analyse bayésienne de [Russo and Van Roy, 2014] que dans le cas où le nombre de contextes est fini, le risque bayésien, défini ici par BR π 0 (T, A) = E [R θ (T, A)] de Thompson Sampling est de l'ordre de Õ( dT log(K)). Nous donnons également une borne générale (indépendante du nombre de contextes) qui montre que le risque bayésien de Thompson Sampling est de l'ordre de Õ(d √ T ). Ainsi, en fonction de la relation entre la dimension d et le nombre d'actions K, une des deux variantes Bayes-UCB ou Bayes-Lin-UCB sera préférable ; celle qui conduit à la borne supérieure la plus petite sur le regret, ou aux meilleures performances pratiques. A l'inverse, l'échantillonnage de Thompson s'implémente d'une manière unique pour tous les modèles linéaires et son risque bayésien est majoré par la plus petite des deux bornes en Õ( dT log(K)) et Õ(d √ T ) obtenues.

3 Vers des algorithmes fréquentistes optimaux pour l'identification des meilleurs bras

On rappelle que contrairement à l'objectif de minimisation du regret, pour l'objectif d'identification des meilleurs bras, la notion d'algorithme (asymptotiquement) optimal n'existe pas dans la littérature. Nous avons proposé les deux notions de complexités suivantes pour un budget fixé (κ B (ν)) ou un niveau de confiance fixé (κ C (ν)) :

κ B (ν) = inf A consistant lim sup t→∞ - 1 t log p t (ν) -1 , κ C (ν) = inf A δ-PAC lim sup δ→0 E ν [τ ] log 1 δ .
Afin d'évaluer ces deux complexités et de pouvoir les calculer, deux types de résultats sont nécessaires. Nous donnons des bornes inférieures sur le nombre moyen d'observations nécessaires à un algorithme pour identifier les m meilleurs bras avec une probabilité plus grande que 1 -δ, ou sur la probabilité d'erreur d'un algorithme qui peut utiliser t observations des bras. Ensuite, nous présentons des algorithmes qui atteignent ces bornes inférieures. De tels algorithmes peuvent alors être qualifiés d'asymptotiquement optimaux. La section 3.1 présente nos outils pour obtenir des bornes inférieures, ainsi qu'une première borne inférieure sur κ C (ν), valable pour des distributions générales et pour toute valeurs de m ≥ 1 (ce qui n'existait pas dans la littérature). Nous donnons ensuite en section 3.2 deux algorithmes pour l'identification des meilleurs bras à niveau de confiance fixé, basés sur des intervalles de confiances construits à l'aide de la divergence de Kullback-Leibler. La borne supérieure obtenue sur la moyenne du nombre d'échantillons utilisés par KL-UCB fait aussi intervenir des quantités informationnelle, mais n'atteint pas exactement la borne inférieure précédente. Dans le cas particulier de l'identification du meilleur bras parmi deux, nous proposons en section 3.3 de nouvelles bornes inférieures ainsi que des algorithmes asymptotiquement optimaux.

CONTENTS

Une borne inférieure sur la complexité à niveau de confiance fixé

Toutes les bornes inférieure obtenues dans la littérature, que ce soit pour la minimisation du regret ( [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF]) ou l'identification des meilleurs bras ( [Mannor andTsitsiklis, 2004, Audibert et al., 2010]) reposent sur des changements de loi. Un changement de loi relie la probabilité du même événement sous deux modèles de bandit différents, ν et ν ′ . Le lemme ci-dessous présente une nouvelle formulation synthétique pour un changement de loi, sous la forme d'une inégalité qui fait directement intervenir l'espérance du nombre de tirages de chaque bras.

Lemme 7. Soient ν et ν ′ deux modèles de bandit tels que les distributions des tous les bras de ν et ν ′ soient absolument continues. Soit σ un temps d'arrêt par rapport à la filtration

(F t ) et soit A ∈ F σ tel que 0 < P ν (A) < 1. On a K a=1 E ν [N a (σ)]KL(ν a , ν ′ a ) ≥ d(P ν (A), P ν ′ (A)), où d(x, y) ∶= x log(x y) + (1 -x) log((1 -x) (1 -y)) désigne l'entropie relative binaire.
Soit A un algorithme δ-PAC. Pour minorer le nombre moyen d'observations E ν [τ ] pour un modèle de bandit ν fixé, on peut minorer E ν [N a ] pour chaque bras, où N a = N a (τ ) désigne le nombre total d'observations du bras a. Une telle minoration peut être obtenue en appliquant le Lemme 7 avec le temps d'arrêt τ , en choisissant pour événement A l'événement d'erreur sous ν et pour ν ′ un modèle de bandit qui diffère de ν par le bras a uniquement, et qui a un ensemble de bras optimaux différent de celui de ν. Ceci permet de montrer, sous certaines hypothèses sur la classe M m , vérifiées par exemple pour une classe de modèles de bandit exponentiels, le résultat suivant.

Théorème 8. Tout algorithme δ-PAC sur M m vérifie, pour δ ≤ 0.15,

E ν [τ ] ≥ ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ a∈S * m 1 KL(ν a , ν [m+1] ) + a∉S * m 1 KL(ν a , ν [m] ) ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ log 1 2δ .
Notons que le Lemme 7 sera introduit dès le chapitre 1, car il permet également de donner une preuve simple de la borne inférieure de [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF], qui généralise celle de Lai et Robbins.

Deux algorithmes : KL-LUCB et KL-Racing

Nous commenc ¸ons par présenter deux algorithmes génériques pour l'identification des m meilleurs bras parmi K, pour tout m ≥ 1, pour un niveau de confiance fixé. Ces algorithmes partagent l'utilisation d'intervalles de confiance, mais sont basés sur deux stratégies d'échantillonnage différentes. A chaque instant de jeu, indexé par t, ces algorithmes tirent entre 2 et K bras, et nous notons I a (t) = [L a (t), U a (t)] un intervalle de confiance construit pour le bras a à l'instant t, ainsi que N a (t) (resp. S a (t)) le nombre de tirages (resp. la somme des observations) du bras a entre les instants 1 et t, et μa (t) la moyenne empirique.

L'algorithme Racing est basé sur un échantillonnage uniforme couplé à des éliminations. A chaque instant t, les bras sont partitionnés en trois ensemble : R t est l'ensemble des bras restants en course, S t l'ensemble des bras sélectionnés et D t l'ensemble des bras éliminés. Chaque bras de R t est tiré une fois, d'où le terme échantillonnage uniforme (on pose R 1 = {1, . . . , K} si bien que dans les premiers instants on tire tous les bras). Les bras de R t sont ensuite triés par moyenne empirique décroissante, et si

S t+1 = S t ⋃{â}, R t+1 = R t {â} et D t+1 = D t .
Si â n'a pas été sélectionné, on se donne aussi la possibilité d'éliminer le moins bon bras empirique b si la borne borne supérieure U b(t) de son intervalle de confiance est inférieure aux bornes inférieures L a (t) des m -S t meilleurs bras empiriques :

S t+1 = S t , R t+1 = R t { b} et D t+1 = D t ⋃{ b}.
L'échantillonnage s'arrête au premier instant t tel que S t = m, et on recommande Ŝm = S t . Cet algorithme est inspiré des algorithmes de la littérature basés sur des éliminations (e.g. [START_REF] Jennison | Asymptotically optimal procedures for sequential adaptive selection of the best of several normal means[END_REF][START_REF] Even-Dar | Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems[END_REF], Heidrich-Meisner and Igel, 2009]), qui peuvent se réécrire comme l'algorithme Racing avec des intervalles de confiance particuliers. L'algorithme LUCB est basé sur un échantillonnage adaptatif, et ne tire que deux bras bien choisis à chaque instant t. Il a été proposé par [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF] avec des intervalles de confiances basés sur l'inégalité de Hoeffding, et sa formulation générique est la suivante. A chaque instant t, l'ensemble J(t) des m bras ayant les m plus grandes moyennes empiriques est formé. Les deux bras tirés sont celui parmi J(t) dont l'intervalle de confiance a la plus petit borne inférieure, et celui parmi J(t) c dont l'intervalle de confiance a la plus grande borne supérieure. Ces deux bras l t et u t ont en effet de plus grandes chances d'être mal classifiés dans J(t) et J(t) c . L'échantillonnage s'arrête au premier instant t tel que L lt > U ut (les intervalles de confiances des bras dans J(t) et J(t) c sont séparés) et l'ensemble J(t) est recommandé.

(t) = [l a (t)u a (t)] avec u a (t) ∶= max {q ∈ [μ a (t), 1] ∶ N a (t)d(μ a (t), q) ≤ β(t, δ)} , l a (t) ∶= min {q ∈ [0, μa (t)] ∶ N a (t)d(μ a (t), q) ≤ β(t, δ)} ,
où d(x, y) est la fonction de divergence associée à la famille exponentielle (voir ( 2)) et β(t, δ) est au taux d'exploration. Une illustration de ces deux algorithmes est proposée sur la figure 3 pour des modèles de bandit binaires, où l'on a d(x, y) = KL(B(x), B(y)). Une étude numérique sera présentée au chapitre 5 pour des bandit binaires, où l'on verra que comme dans le cadre de la minimisation du regret, l'utilisation de tels intervalles de confiance est préférable à celle d'intervalles proches de ceux utilisés par l'algorithme UCB1 et basés sur l'inégalité de Hoeffding. Par ailleurs, KL-LUCB semble nécessiter empiriquement moins d'observations que KL-Racing pour trouver les m meilleurs bras.

Analyse de KL-LUCB. Pour KL-LUCB nous avons obtenu une majoration de l'espérance du nombre de tirages de bras E ν [τ ] que nous présentons ici car elle conduit à une borne supérieure sur le terme de complexité κ C (ν).

La présentation de notre résultat nécessite l'introduction d'une nouvelle quantité informationnelle, l'information de Chernoff. L'information de Chernoff entre les distributions de Bernoulli B(x) et B(y), notée d * (x, y), est définie par

d * (x, y) = d(z * , x) = d(z * , y) où z * est l'unique z tel que d(z, x) = d(z, y). (12)
Cette définition peut être généralisée aux modèles de bandit exponentiels.

Théorème 9. L'algorithme KL-Racing avec

β(t, δ) = log k 1 Kt α δ pour α > 1 et k 1 > 1 + 1 α-1 recom- mande les m meilleurs bras avec une probabilité plus grande que 1 -δ. Soit c ∈ [µ [m+1] , µ [m]
]. Si de plus on choisit α > 2 dans l'expression ci-dessus du taux d'exploration, il existe une constante C α telle que

E ν [τ ] ≤ 4αH * c log ek 1 K(H * c ) α δ log k 1 K(H * c ) α δ + C α , avec H * c (ν) ∶= a∈{1,...,K} 1 d * (µ a , c) .
En utilisant les résultats des théorèmes 8 et 9, on obtient l'encadrement suivant sur la complexité pour un niveau de confiance fixé :

a∈S * m 1 d(µ a , µ [m+1] ) + a∉S * m 1 d(µ a , µ [m] ) ≤ κ C (ν) ≤ 8 min c∈[µ m+1 ,µm] K a=1 1 d * (µ a , c) .
Nous avons obtenu des bornes informationnelles sur le terme de complexité κ C (ν), mais un écart demeure entre la borne inférieure et la borne supérieure obtenues. Le paramètre c qui apparaît dans la borne supérieure semble être un avatar de notre preuve, et on s'attendrait plutôt à une borne supérieure de la forme

a∈S * m 1 d * (µ a , µ [m+1] ) + a∉S * m 1 d * (µ a , µ [m] )
Quant à la présence de l'information de Chernoff, elle semble aussi provenir de raisons techniques (voir chapitre 5). Mais nous allons voir que cette quantité informationnelle sera en effet caractéristique de la complexité d'un problème d'identification du meilleur bras dans un modèle de bandit à deux bras.

Caractérisation de la complexité pour des modèles de bandit à deux bras

Une partie du chapitre 5 est dédiée au calcul des complexités dans des modèles de bandit à deux bras. On peut tout d'abord souligner l'intérêt pratique de tels modèles de bandit, qui fournissent un cadre théorique pour l'A/B Testing séquentiel. L'A/B Testing est une procédure utilisée par exemple pour l'optimisation des contenus web : deux versions d'une page web sont comparées en étant présentées à des utilisateurs. On présente à chaque utilisateur une seule des deux versions, A t ∈ {1, 2}, et l'utilisateur fournit une réponse X t qui est un indice de la qualité de la page, modélisée comme une réalisation d'une loi de probabilité ν 1 ou ν 2 . Un objectif commun est de déterminer quelle version a le plus grand taux de conversion (probabilité qu'un utilisateur devienne un consommateur) en collectant des réponses binaires des utilisateurs.

Dans des modèles de bandit à deux bras, les deux algorithmes que nous avons présentés, KL-Racing et KL-LUCB se réduisent au même algorithme, qui tire les deux bras de manière uniforme et utilisent un critère d'arrêt basé sur la séparation d'intervalles de confiances. On peut en particulier se demander si cet échantillonnage uniforme fait sens.

Des bornes inférieures plus fines. En utilisant le même outil qu'en section 3.1, le Lemme 7, mais on considérant des changements de lois différents (où en particulier les deux bras du modèle de bandit alternatif ν ′ sont modifiés par rapport au modèle ν), on peut montrer le résultat suivant. Ce résultat est présenté dans un cadre plus général au chapitre 5, mais nous l'énonc ¸ons ici pour des modèles de bandit où les bras sont paramétrés continument par leurs moyennes. Cela inclut le cas des modèles de bandit gaussiens de variances connues et le cas des modèles de bandit exponentiels.

Théorème 10. Soit M 1 une classe de modèles de bandit à deux bras paramétrés continument par leurs moyennes. Soit ν = (ν 1 , ν 2 ) ∈ M 1 . On a les résultats suivants

Budget fixé

Niveau de confiance fixé Tout algorithme consistant sur M 1 vérifie Tout algorithme δ-PAC sur M 1 vérifie, pour δ ≤ 0.15,

lim sup t→∞ -1 t log p t (ν) ≤ KL * (ν 1 , ν 2 ) E ν [τ ] ≥ 1 KL * (ν 1 ,ν 2 ) log 1 2δ avec KL * (ν 1 , ν 2 ) ∶= KL(ν * , ν 1 ) = KL(ν * , ν 2 ). avec KL * (ν 1 , ν 2 ) ∶= KL(ν 1 , ν * ) = KL(ν 2 , ν * ). On en déduit κ B (ν) ≥ 1 KL * (ν 1 , ν 2 ) et κ C (ν) ≥ 1 KL * (ν 1 , ν 2 )
.

La quantité KL * (ν 1 , ν 2 ) dans la complexité à budget fixé peut être interprétée comme une information de Chernoff, alors que la quantité KL * (ν 1 , ν 2 ) apparait comme une quantité similaire, mais où les rôles des arguments sont inversés.

Algorithmes asymptotiquement optimaux. Dans la classe des bandits gaussiens, définie par

M 1 = {ν = N µ 1 , σ 2 1 , N µ 2 , σ 2 2 ∶ (µ 1 , µ 2 ) ∈ R 2 , µ 1 ≠ µ 2 }
où les variances σ 2 1 et σ 2 2 sont connues (mais potentiellement différentes), la divergence de Kullback-Leibler est symétrique, et les deux bornes inférieures obtenues dans le théorème 10 sont égales. On CONTENTS montre même que

κ B (ν) = κ C (ν) = 2(σ 1 + σ 2 ) 2 (µ 1 -µ 2 ) 2
en proposant des algorithmes asymptotiquement optimaux. Pour un budget fixé, un algorithme qui échantillonne les bras proportionnellement à leurs écarts types (et non leurs variances) et recommande le meilleur empirique vérifie en effet

lim inf t→∞ - 1 t log p t (ν) ≥ (µ 1 -µ 2 ) 2 2(σ 1 + σ 2 ) 2 ,
et atteint donc la borne inférieure du théorème 10. Pour un niveau de confiance fixé, en couplant un échantillonnage séquentiel qui maintient la proportion de tirages du bras 1, N 1 (t) t, proche du ratio des écarts types α = σ 1 (σ 1 + σ 2 ) (A t = 2 ssi ⌈αt⌉ = ⌈α(t -1)⌉) à une règle d'arrêt basée sur la différence des moyennes empiriques,

τ = inf t ∈ N ∶ μ1 (t) -μ2 (t) > 2σ 2 t (α) log(t δ) où σ 2 t (α) = σ 2 1 ⌈αt⌉ + σ 2 2 (t -⌈αt⌉) ,
on obtient un algorithme δ-PAC qui vérifie

E ν [τ ] ≤ (1 + ) 2(σ 1 + σ 2 ) 2 (µ 1 -µ 2 ) 2 log 1 δ + o δ→0 log 1 δ ,
et atteint donc la borne du théorème 10. Dans la classe des bandits binaires, définie par

M 1 = {ν = (B(µ 1 ), B(µ 2 )) ∶ (µ 1 , µ 2 ) ∈]0; 1[
κ B (ν) ≥ 1 d * (µ 1 , µ 2 ) et κ C (ν) ≥ 1 d * (µ 1 , µ 2 ) , et nous avons d * (µ 1 , µ 2 ) < d * (µ 1 , µ 2 ).
Pour un budget fixé, pour chaque ν, on peut montrer qu'il existe α(ν) tel qu'en allouant α(ν)t échantillon au bras 1, on atteint la borne du théorème 10. Ceci permet de montrer que

κ B (ν) = 1 d * (µ 1 , µ 2 ) et κ C (ν) > κ B (ν).
Par

Organisation du document

Dans cette thèse nous présentons des algorithmes pour deux problèmes d'allocation séquentielle de ressources : la maximisation des récompenses et l'identification des meilleurs bras dans des modèles de bandit. De bons algorithmes pour ces deux problèmes peuvent être obtenus en utilisant des intervalles de confiances basés sur la divergence de Kullback-Leibler. Nous avons transposé cette idée, proposée par [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF] pour l'objectif de maximisation des récompenses et donnant lieu à l'algorithme KL-UCB, à l'identification des meilleur bras, pour laquelle nous avons introduit l'algorithme KL-LUCB. Par ailleurs, pour l'objectif de maximisation des récompenses, nous avons proposé l'algorithme Bayes-UCB, basé sur une interprétation bayésienne du problème, et prouvé son optimalité asymptotique du point de vue du regret, une mesure de performance fréquentiste, dans des modèles de bandit binaires. Nous avons établi cette même propriété pour l'échantillonnage de Thompson, un autre algorithme bayésien, introduit en 1933 mais dont l'optimalité était encore une question ouverte. Comme on peut le constater sur la figure 4, les algorithmes pour les deux problèmes considérés sont différents. KL-UCB tire massivement le bras optimal, et très peu les autres bras, ce qui conduit à une très bonne estimation de la moyenne du meilleur bras, mais une mauvaise estimation de celles des autres bras. A l'inverse pour identifier le meilleur bras (donc pour m = 1), KL-LUCB va tirer beaucoup plus les bras sous-optimaux et a donc une meilleure estimation de leur moyenne.

Si les algorithmes pour ces deux problèmes sont différents, leur complexité l'est également. Nous avons vu que la complexité du problème de minimisation du regret est bien connue puisque nous avons, dans le cadre des bandits binaires par exemple,

inf A uniformément efficace lim sup T →∞ R ν (T, A) log(T ) = a∶µa<µ * (µ * -µ a ) d(µ a , µ a * ) ,
où des stratégies réalisant l'infimum sont par exemple KL-UCB, Bayes-UCB ou Thompson Sampling. Pour l'identification des meilleurs bras, avec un niveau de confiance 1 -δ fixé, nous avons montré que

inf A δ-PAC lim sup δ→0 E ν [τ ] log(1 δ) ≥ a∈S * m 1 d(µ a , µ [m+1] ) + a∉S * m 1 d(µ a , µ [m] )
.

CONTENTS

En revanche des résultats plus fins obtenus pour l'identification du meilleur parmi K = 2 bras montrent que cette borne inférieure ne peut pas être atteinte, et qu'en particulier des quantités informationnelles différentes, comme l'information de Chernoff, interviennent pour mesurer la complexité du problème.

CONTENU DE LA TH ÈSE L'organisation du manuscrit est la suivante : les chapitre 1 à 4 présentent nos contributions relatives à la maximisation des récompenses, et le chapitre 5 est consacré à l'identification des meilleurs bras.

Plus précisément, le chapitre 1 introduit les deux approches bayésienne et fréquentiste pour la maximisation des récompenses. Nous y présentons les mesures de performance associées, ainsi qu'un état de l'art des algorithmes fréquentistes et bayésiens. Nous y étudions notamment un algorithme basé sur les indices de Gittins à horizon fini.

Le chapitre 2 introduit l'algorithme Bayes-UCB. Nous y donnons une analyse à temps fini de l'algorithme pour les bandits binaires, et présentons également des applications dans d'autres contextes, illustrées de premières simulations numériques.

Le chapitre 3 introduit le Thompson Sampling. Nous donnons une analyse à temps fini de l'algorithme pour les bandits binaires, esquissons une analyse pour des modèles de bandit exponentiels et présentons une étude numérique comparant Bayes-UCB et Thompson Sampling aux différents algorithmes bayésiens et fréquentistes discutés au chapitre 1.

Le chapitre 4 présente l'utilisation de Bayes-UCB et Thompson Sampling dans des modèles de bandit contextuels linéaires. Nous y donnons notamment de nouveaux éléments d'analyse bayésienne.

Le chapitre 5 présente le cadre de l'identification des meilleurs bras, propose une analyse des algorithmes KL-LUCB et KL-Racing ainsi qu'une illustration numérique de leur performance. Nous y prouvons également de nouvelles bornes inférieures sur les termes de complexités des différents problèmes et proposons des algorithmes asymptotiquement optimaux dans des modèles de bandit à deux bras.

PUBLICATIONS ASSOCI ÉES

Les résultats présentés dans ce manuscrit sont le fruit de collaborations avec mes directeurs de thèse, Olivier Cappé, Aurélien Garivier et Rémi Munos, mais aussi avec Nathan Korda (ancien post-doctorant à l'INRIA de Lille avec Rémi Munos) ainsi que Shivaram Kalyanakrishnan (chercheur à Yahoo ! Labs à Bangalore). Ils ont fait l'objet des publications suivantes, dont certains des chapitres qui suivent (lorsque c'est indiqué) seront inspirés.

Articles de conférences avec actes :

• E. Chapter 1

Two probabilistic views on rewards maximization in bandit models

The first part of this thesis is dedicated to the design of strategies maximizing the sum of rewards in a bandit model. As discussed in the Introduction, this objective is motivated by many applications that range from clinical trials to the display of advertising. Different communities have worked on this bandit problem, and in the particular case of parametric bandit models, two different probabilistic points of view have been considered. This chapter aims at presenting these frequentist and Bayesian approaches. We introduce the two measures of performance associated to these two probabilistic modelings, namely the regret and the Bayes risk. We present state-of-the-art algorithms in each case. In particular, we discuss the use of a strategy based on Finite-Horizon Gittins indices.

Introduction

A stochastic multi-armed bandit model, denoted by ν = (ν 1 , . . . , ν K ), is a collection of K arms, where each arm ν a is a probability distribution with mean µ a . An agent can interact with a bandit model by choosing at each round t an arm A t to draw. This draw results in the observation of a realization X t from the associated distribution ν At . The bandit problem that we consider in the first four chapters of this thesis is the following. The samples (X t ) collected are perceived as rewards, and the agent has to choose the arms sequentially in order to maximize the sum of the rewards accumulated during his interaction with the bandit model. Of course, the bandit model ν is unknown to him, an so is the optimal arm a * , such that µ a * = max a µ a . The mean of the optimal is denoted by µ * .

In this thesis, we mostly consider parametric bandit models, of the form ν = ν θ = (ν θ 1 , . . . , ν θ K ). The distribution ν a of arm a depends on a parameter θ a ∈ Θ, and we let θ = (θ 1 , . . . , θ K ) ∈ Θ K denote the global parameter of the model. As in every parametric model, two different points of view can be adopted: the frequentist point of view, in which θ is seen as an unknown parameter, and the Bayesian point of view, in which θ is a random variable, drawn from some prior distribution. This leads to two different probabilistic frameworks, described in Table 1.1.

Frequentist bandit model

Bayesian bandit model θ ∈ Θ K is an unknown parameter -θ is drawn from Π 0 , a prior distribution on Θ K -∀a, (X a,t ) i.i.d. with distribution ν θa and mean µ a -∀a, conditionally to θ a , (X a,t ) is i.i.d. with distribution ν θa and mean µ a -(X a,t ) a,t is an independent family -conditionally to θ, (X a,t ) a,t is an independent family Table 1.1: Bayesian and frequentist bandit models

We denote by P θ (resp. E θ ) -or sometimes P ν (resp. E ν ) in a model that is not necessarily parametric-the probability (resp. expectation) under the frequentist modeling, and P Π 0 (resp.E Π 0 ) the probability (resp. expectation) under the Bayesian modeling. The subscripts might be omitted in some parts of this document when the associated probabilistic framework is clear.

In both settings, an agent interacts with the bandit model using a sampling strategy, sometimes called policy or bandit algorithm. This sampling strategy A = (A t ) t∈N is a sequence of choices of the arms based on previous outcomes. At time t, the agent draws the arm A t and receives as a reward an observation from arm A t , X t = X At,t ∼ ν At . Introducing F t the filtration defined by

F t = σ(A 1 , X 1 , . . . , A t , X t ),
in a deterministic strategy A t is assumed to be F t-1 -measurable and in a randomized strategy A t is drawn from a probability distribution p t on {1, . . . , K} such that the vector of probabilities p t is F t-1measurable. The bandit problem considered in the first part of this thesis is the following: the agent aims at building a strategy which minimizes the expected sum of rewards up to some horizon T .

In the frequentist view of the bandit problem, a strategy that maximizes the expected cumulated rewards equivalently minimizes the regret, defined for any strategy A and horizon T by

R θ (T, A) = E θ T µ * - T t=1 X t = T µ * -E θ T t=1 X t .

INTRODUCTION 37

This quantity depends on the parameter θ (fixed and unknown). The notion of regret was introduced by [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] in a frequentist, parametric setting. Regret represents the difference between the expected cumulative reward of the strategy drawing the (unknown) best arm at each round, and the expected cumulative reward obtained with the strategy A. It can be rewritten as a function of the number of draws of each sub-optimal arm. Let N a (t) denote the number of draws of arm a between the instants 1 and t. Using that E ν [X t F t-1 ] = µ At , one obtains the following useful expression of the regret, as a function of the expected number of draws of each arm (that is also defined for non-parametric bandit models, for which we use the subscript ν in place of θ):

R ν (T, A) = E ν T t=1 (µ * -µ At ) = K a=1 (µ * -µ a )E ν [N a (T )] .
(1.1)

In the Bayesian view of the bandit problem there is an equivalent notion, called Bayes risk (a denomination introduced by [Lai, 1987]), or sometimes Bayesian regret. A strategy that maximizes the expected cumulative rewards equivalently minimizes the Bayes risk, defined for a strategy A and horizon T by

BR Π 0 (T, A) = E Π 0 T µ * - T t=1 X t = E Π 0 E Π 0 T µ * - T t=1 X t θ = E Π 0 R θ (T, A) .
This quantity depends on the prior distribution Π 0 . An algorithm minimizing the Bayes risk is good in average on all bandit models with parameters in Θ K , whereas in the frequentist modeling we are looking for algorithms with small regret on every bandit model ν θ , for all θ ∈ Θ K .

The first bandit algorithm was introduced by [Thompson, 1933] in a Bayesian framework, and more generally the view adopted in the first works on bandit problems was mostly Bayesian. This could be explained by the fact that Bayes risk minimization has an exact solution (that can be obtained by dynamic programming, as explained in Section 1.3), whereas there exists no algorithm minimizing the regret on every bandit model. However the lower bound on the regret given by [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] allows to define the notion of asymptotic optimality in the frequentist setting and paves the way to a more abundant frequentist literature at the end of the 1980's.

The Bayesian and frequentist modelings of the bandit problem (and the dedicated performance criteria) can be dissociated from the tools related to these two frameworks. For example, algorithms from the regret minimization literature rely on maximum likelihood estimates of the unknown parameters or confidence intervals (that we call frequentist tools), whereas algorithms from the Bayesian literature choose the next arm based on the current posterior distribution. At the end of round t, the posterior distribution of the parameter θ is the conditional distribution of θ given the observation, denoted by

Π t (θ) = L(θ A 1 , X 1 , . . . , A t , X t ).
One can evaluate the Bayes risk of an algorithm that uses frequentist tools or conversely focus on the regret of a Bayesian algorithm that uses a prior distribution and the associated posterior distributions in its routine. The latter objective is at the heart of this thesis, in which we show that two Bayesian algorithms, Bayes-UCB and Thompson Sampling, are asymptotically optimal with respect to the regret.

Before studying the regret of Bayesian algorithms in the next chapters, we present in this chapter state-of-the-art Bayesian and frequentist algorithms and discuss the link between regret and Bayes risk. Section 1.2 is dedicated to the frequentist framework. We start by presenting the lower bound on the regret of [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF], and the subsequent definition of asymptotically optimal algorithms. We propose a new short proof for this result. Then we present the recent improvements in the frequentist literature that have led to the KL-UCB algorithm of [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF], that is asymptotically optimal with respect to Lai and Robbins' lower bound. We also present some tools for regret finite-time analysis that will be useful in the rest of this thesis.

In Section 1.3, we discuss the Bayesian optimal solution. It is often heard that Gittins ( [Gittins, 1979]) solved the Bayesian bandit problem by exhibiting an optimal index policy. Index policies are bandit algorithms in which, at each round, one index for each arm is computed, based on the history of this arm only, and then the arm with highest index is chosen. We explain here that when the goal is to maximize the expected cumulative rewards up to some finite horizon -and not the discounted sum of rewards, as in Gittins' original paper-the corresponding index policy is not optimal. Nevertheless, we conjecture that this Finite-Horizon Gittins algorithm closely approximates the Bayesian optimal solution and we present some numerical experiments supporting this claim. Approximations of the Finite-Horizon Gittins indices also indicate similarities with indices used by a frequentist index policy, KL-UCB-H + , that is proved to be asymptotically optimal in a Bayesian sense.

In a nutshell, this chapter introduces frequentist algorithms (asymptotically) optimal with respect to the regret, Bayesian algorithms optimal with respect to the Bayes risk, as well as a frequentist algorithm (asymptotically) optimal with respect to the Bayes risk. Table 1.2 summarizes the information of this chapter and can be completed with the contributions of this thesis relative to regret minimization (in bold), presented in Chapter 2 and Chapter 3.

Regret

Bayes risk Frequentist algorithm KL-UCB KL-UCB-H + Bayesian algorithm

Bayes-UCB Dynamic programming Thompson Sampling

FH-Gittins ?

Table 1.2: Optimal algorithms for each measure of performance Notation. We introduce here useful notation that enables us to define and later analyse bandit algorithms. Quantities that are functions of t depend on the history of the game up to the end of round t. We introduce -N a (t), the number of draws of arm a between instants 1 and t.

-S a (t) = ∑ t s=1 1 (As=a) X a,s , the sum of rewards obtained from arm a between instants 1 and t. -μa (t) = Sa(t) Na(t) , the empirical mean of the rewards obtained from arm a between instants 1 and t. We also introduce -(Y a,k ) k∈N * , the sequence of successive rewards obtained from arm a μa,s = 1 s ∑ s k=1 Y a,k the empirical mean of the first s rewards obtained from arm a If (A t = a), Y a,Na(t) = X a,t , where (X a,t ) is the i.i.d. sequence associated to arm a introduced in Table 1.1). One has S a (t) = ∑ Na(t) s=1 Y a,s and μa (t) = μa,Na(t) . The sequence (Y a,k ) k∈N * is i.i.d. with distribution ν a (conditionally to θ a in the Bayesian modeling).

The frequentist approach

For the sake of comparison with Bayesian algorithms, we mostly focus our presentation on parametric bandit models, in particular on models in which the rewards belong to an exponential family.

THE FREQUENTIST APPROACH 39

However, we will see that algorithms developed for parametric bandits can sometimes be generalized to some non-parametric bandit models (e.g. bandits with bounded rewards, that are often considered in the frequentist literature).

Lower bounds on the regret

In the frequentist framework, the bandit problem was first considered by [Robbins, 1952], who introduces two strategies for two-armed bandit models. For Bernoulli bandit models, he proposes a strategy that changes the arm that is drawn if and only if a zero is observed. For more general two-armed bandit models, Robbins proposes a second strategy that relies on two disjoint increasing sequences of integers

(a n ) and (b n ) fixed in advance, with a 1 = 1 and b 1 = 2. At time t, arm 1 is drawn if t ∈ (a n ) n∈N , arm 2 is drawn if t ∈ (b n ) n∈N .
Otherwise, the arm with highest empirical mean of past rewards μa (t -1) is chosen. This strategy modifies the 'greedy' strategy, that chooses A t = argmax a μa (t -1), in a way that forces exploration. If the sequences (a n ) and (b n ) are chosen such that the proportion of integers in {1, . . . , t} that belong to one of the two sequences goes to zero when t goes to infinity, [Robbins, 1952] shows that this strategy A R is such that, for every two-armed bandit model ν,

R ν (T, A R ) T → T →∞ 0.
The seminal paper of [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] gives, in simple parametric cases, a lower bound on the regret of strategies having the following stronger consistency property: for all θ ∈ Θ K such that in ν θ there is a unique optimal arm, for all α ∈]0, 1], R θ (T, A) = o(T α ).

Algorithms satisfying this property are called uniformly efficient. For bandits whose arms are parameterized by a single parameter (i.e. Θ ⊂ R), under some conditions on Θ, Lai and Robbins show that a uniformly efficient algorithm has to draw each sub-optimal arm at least in a logarithmic fashion. More precisely,

µ a < µ * ⇒ lim inf T →∞ E θ [N a (T )] log(T ) ≥ 1 KL(ν θa , ν θ * )
,

where KL(p, q) is the Kullback-Leibler divergence between the distributions p and q, defined by

KL(p, q) = ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ ∫ log dp dq (x) dp(x) if q ≪ p, +∞ otherwise.
From the regret decomposition (1.1), it follows that, for any uniformly efficient strategy A,

lim inf T →∞ R θ (T, A) log(T ) ≥ a∶µa<µ * (µ * -µ a ) KL(ν θa , ν θ * ) . (1.2)
This lower bound leads to the definition of asymptotic optimality. An algorithm is termed asymptotically optimal if it satisfies, for every θ ∈ Θ, sup

T →∞ R θ (T, A) log(T ) ≤ a∶µa<µ * (µ * -µ a ) KL(ν θa , ν θ * ) .
Lai and Robbins' lower bound was later generalized by [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF] to distributions that depend on multiple parameters. We give here a slightly more general result, that does not rely on parametric assumptions. Definition 1.1. A class M of bandit models is identifiable if it is of the form M = P 1 × ⋅ ⋅ ⋅ × P K , where P a is the set of possible distributions for arm a, and if for all a, P a is such that ∀p, q ∈ P a , p ≠ q ⇒ 0 < KL(p, q) < +∞.

Theorem 1.2. Let M be an identifiable class of bandit models. Let A be a bandit algorithm uniformly efficient on the class M: for all ν ∈ M with a unique optimal arm, for all α ∈]0, 1], R ν (T, A) = o(T α ).

Then, for all ν ∈ M,

µ a < µ * ⇒ lim inf T →∞ E ν [N a (T )] log(T ) ≥ 1 K a inf (ν a ; µ * ) , (1.3) with K a inf (p; µ) = inf {KL(p, q) ∶ q ∈ P a and E Y ∼q [Y ] > µ} .
All the distribution-dependent lower bounds derived in the bandit literature (e.g. [Lai andRobbins, 1985, Burnetas and[START_REF] Burnetas | [END_REF] but also [Mannor andTsitsiklis, 2004, Audibert et al., 2010] in the literature relative to best arm identification, that will be presented in Chapter 5) rely on changes of distribution, and so does Theorem 1.2. A change of distribution relates the probabilities of the same event under two different bandit models ν and ν ′ . Lemma 1.3 below provides a new, synthetic, inequality from which all the lower bounds presented in this thesis will be directly derived. In other words, this result, whose proof is postponed to Section 1.5.1, encapsulates the technical aspects of the change of distribution. Lemma 1.3 could also be used to give simple proofs for the lower bounds of [Graves andLai, 1997, Agrawal et al., 1989] in more general cases in which arms are not necessarily independent.

Lemma 1.3. Let ν and ν ′ be two bandit models such that the distributions of all arms in ν and ν ′ are mutually absolutely continuous. Let σ be a stopping time with respect to (F t ) such that (σ < +∞) a.s. under both models. Let A ∈ F σ be an event such that 0 < P ν (A) < 1. Then one has

K a=1 E ν [N a (σ)]KL(ν a , ν ′ a ) ≥ d(P ν (A), P ν ′ (A)),
where d(x, y) ∶= x log(x y) + (1 -x) log((1 -x) (1 -y)) is the binary relative entropy.

Proof of Theorem 1.2. Let ν = (ν 1 , . . . , ν K ) be a bandit model such that arm 1 is the unique optimal arm. Let a ≠ 1 be a suboptimal arm. Consider the alternative bandit model ν ′ such that ν

′ i = ν i for all i ≠ a and ν ′ a ∈ P a is such that E Y ∼ν ′ a [Y ] > µ 1 .
Arm 1 is thus the unique optimal arm in the bandit model ν, whereas arm a is the unique optimal arm in the bandit model ν ′ . For every integer T , let A T be the event defined by

A T = N 1 (T ) ≤ T - √ T .
Clearly, A T ∈ F T . From Lemma 1.3, applied to the stopping time σ = T a.s.,

E ν [N a (T )]KL(ν a , ν ′ a ) ≥ d(P ν (A T ), P ν ′ (A T )).
(1.4)

The event A T is not very likely to hold under the model ν, in which the optimal arm should be drawn of order T -O(log(T )) times, whereas it is very likely to happen under ν ′ , in which arm 1 is sub-optimal and thus only drawn little. More precisely, Markov inequality yields

P ν (A T ) = P ν (T -N 1 (T ) ≥ √ T ) ≤ ∑ i≠1 E ν [N i (T )] √ T P ν ′ (A c T ) = P ν ′ (N 1 (T ) ≥ T - √ T ) ≤ E ν ′ [N 1 (T )] T - √ T ≤ ∑ i≠a E ν ′ [N i (T )] T - √ T 1.2. THE FREQUENTIST APPROACH
From the formulation (1.1), every uniformly efficient algorithm satisfies

i≠1 E ν [N i (T )] = o(T α ) and i≠a E ν ′ [N i (T )] = o(T α ) for all α ∈]0, 1]. Hence P ν (A T ) → T →∞ 0 and P ν ′ (A T ) → T →∞
1. Therefore, we get

d(P ν (A T ), P ν ′ (A T )) log(T ) ∼ T →∞ 1 log(T ) log 1 P ν ′ (A c T ) ≥ 1 log(T ) log T - √ T ∑ i≠a E ν ′ [N i (T )]
.

The right hand side rewrites

1 + log 1 -1 √ T log(T ) - log (∑ i≠a E ν ′ [N i (T )]) log(T ) , → T →∞ 1
where we use the fact that

∑ i≠a E ν ′ [N i (T )] = o(T α ) for all α ∈]0, 1]. Finally, for every ν ′ a ∈ P a such that E Y ∼ν ′ a [Y ] > µ 1 one obtains, using inequality (1.4) lim inf T →∞ E[N a (T )] log(T ) ≥ 1 KL(ν a , ν ′ a )
.

For all ∈]0, 1[, ν ′ a can then be chosen such that K a inf (ν a , µ 1 ) ≤ KL(ν a , ν ′ a ) ≤ K a inf (ν a , µ 1 ) (1 -)
, and the conclusion follows when goes to zero.

◻

Distribution independent lower bounds on the regret. The lower bounds given in this section are distribution-dependent: for each bandit model ν and each algorithm A, the regret R ν (T, A) is lower bounded by a quantity that depends on the arms distributions. More precisely, there exists a constant C(ν) -given in (1.2) for simple classes of parametric bandits-such that R ν (T, A) ≥ C(ν) log(T ).

(1.5)

It is also possible to consider minimax performance bounds on the regret. Problem-independent upper and lower bounds on the regret have emerged in the literature on adversarial bandits (see [START_REF] Cesa-Bianchi | Prediction, Learning and Games[END_REF]), in which there is no stochastic assumptions on the arms. However, the distributionindependent lower bound first given by [START_REF] Cesa-Bianchi | Prediction, Learning and Games[END_REF] can also be formulated in our stochastic setting. Indeed, Theorem 3.5 of [Bubeck and Cesa-Bianchi, 2012] states that for every bandit algorithm A, there exists a stochastic bandit model ν such that ν a is supported in [0, 1] for all a and R ν (T, A) ≥ 1 20

√ KT . (1.6)
This worst-case result gives a regret lower bound for an algorithm belonging to argmin A max ν R ν (T, A).

In this thesis, we rather aim at finding algorithms that are optimal for every bandit model ν, with respect to the distribution-dependent lower bound (1.5).

As already noted by [START_REF] Bubeck | Prior-free and prior-dependent regret bounds for Thompson Sampling[END_REF], the proof of Theorem 3.5 of [Bubeck and Cesa-Bianchi, 2012] also yields a lower bound on the Bayes risk. Indeed, the authors lower bound the quantity

1 K K i=1 E ν (i) T t=1 (X i,t -X At,t ) ,
where ν (i) is the bandit model in which all arms are Bernoulli distributions with mean (1 -) 2, expect arm i which is Bernoulli with mean (1 + ) 2. Their result implies that for every bandit algorithm A, there exists a prior distribution Π (uniform over a finite set of bandit models {ν (1) , . . . , ν

(K) }) such that BR Π (T, A) ≥ 1 20 √ KT .
(1.7)

In Section 1.3 we will present the optimal strategy for any given prior distribution Π 0 . For some specific prior distribution, we provide in Section 1.3.5 a lower bound on the Bayes risk BR Π 0 (T, A) that depends on the prior, that can be qualified as prior-dependent.

Examples of bandit models and associated tools to build bandit algorithms

Classes of exponential bandit model, such that the arms distributions all belong to the same oneparameter canonical exponential family form important examples in which the Lai and Robbins' lower bound holds. In such a class M, there exists two functions A and b such that M = {ν = (ν θ 1 , . . . , ν θ K ) ∶ ∀a ∈ {1, . . . , K}, θ a ∈ Θ}, with the distribution ν θ having a density f (⋅; θ) with respect to some reference measure λ given by f

(x; θ) = A(x) exp(xθ -b(θ)), θ ∈ Θ ∈ R. (1.8)
The log-partition function b ∶ Θ → R is supposed to be twice differentiable. Under this assumption, it can be shown that the mean of the distribution ν θ is µ(θ) = ḃ(θ) and its variance is b(θ) (see e.g. [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF] for more details on exponential families). Thus, the mapping θ ↦ µ(θ) is increasing and distributions belonging to a one-parameter canonical exponential family can be either parameterized by their natural parameter θ or by their mean µ. Hence, each exponential family induces a divergence on ( ḃ(Θ)) 2 , defined by d(µ, µ ′ ) ∶= KL ν ḃ-1 (µ) , ν ḃ-1 (µ ′ ) .

(1.9)

Besides, the Kullback-Leibler divergence between two distributions in an exponential family parameterized by the natural parameters θ and θ ′ respectively will be denoted by K(θ, θ ′ ) ∶= KL(ν θ , ν θ ′ ).

Many classical families of parametric distributions form a one-parameter canonical exponential family, like Gaussian distributions with known variance (Gaussian bandit models), Bernoulli distributions (Bernoulli bandit models), Poisson and exponential distributions (see Table 1.3). Compared to the definition (1.8), one can also consider (following the definition of [START_REF] Bickel | Mathematical Statistics, Basic Ideas and Selected Topics[END_REF] for example) slightly more general families, for which the density of ν θ is given by A(x) exp(T (x)θ -b(θ)), with T (x) ≠ x. This generalization will be discussed in Chapter 3.

For a given class of exponential bandit models (for example Gaussian bandits or Bernoulli bandits), with d(µ, µ ′ ) the associated divergence defined in (1.9), every uniformly efficient bandit algorithm A satisfies lim inf

T →∞ R θ (T, A) log(T ) ≥ a∶µa<µ * (µ * -µ a ) d(µ a , µ * ) .
(1.10)

Deviation inequalities and confidence intervals for exponential families. In an exponential family, the relationship between the moment-generating function and Kullback-Leibler divergence given by Lemma 1.4 below is a crucial property that allows to build good confidence intervals. The logarithm of the moment-generating function of a random variable X and its Fenchel-Legendre transform are respectively defined by

φ X (λ) = log E[e λX ] and φ * X (x) = sup λ∈R {xλ -φ X (λ)} . 1.2. THE FREQUENTIST APPROACH Density θ b(θ) d(µ, µ ′ ) Gaussian distribution 1 √ 2πσ 2 e -(x-µ) 2 2σ 2 µ σ 2 σ 2 θ 2 2 (µ-µ ′ ) 2 2σ 2 N µ, σ 2 (known variance) Bernoulli distribution µ x (1 -µ) 1-x 1 {0,1} (x) log µ 1-µ log(1 + e θ ) µ log µ µ ′ + (1 -µ) log 1-µ 1-µ ′ B(µ) (mean µ) Poisson distribution λ x x! e -λ 1 N * (x) log(λ) e θ µ ′ -µ + µ log µ µ ′ P(λ) (mean λ) Exponential distribution λe -λx 1 R + (x) -λ -log(-θ) µ µ ′ -1 -log µ µ ′ E(λ) (mean 1 λ) Gamma distribution λ k Γ(k) x k-1 e -λx 1 R + (x) -λ -k log(-θ) k µ µ ′ -1 -log µ µ ′ Γ(k, λ) (mean k λ)
P (μ s ≥ x) ≤ e -sφ * X 1 (x) and lim s→∞ - 1 s log P (μ s ≥ x) = φ * X 1 (x).
Because of this second statement, the deviation inequality stated first is optimal with respect to the large deviation principle. In an exponential family, Lemma 1.4 gives a close form for the rate function φ * X 1 (x) that yields such an optimal deviation inequality for the empirical mean of i.i.d. samples.

Lemma 1.4. If X ∼ ν θ , then φ * X (x) = d(x, µ(θ)). Going a bit further, one can give a deviation inequality for the empirical mean of independent random variables 'dominated' by some distribution in an exponential family.

Lemma 1.5 (Chernoff inequality). Let (X i ) be a sequence of independent random variables such that ∀i

∈ N, φ X i (λ) ≤ φ Y (λ), (1.11)
where Y ∼ ν θ belong to an exponential family with associated divergence d(µ, µ ′ ).

Let µ = E[Y ]. Then if μs = 1 s ∑ s i=1 X i , one has for x > µ, P (μ s ≥ x) ≤ exp(-sd(x, µ)), for x < µ, P (μ s ≤ x) ≤ exp(-sd(x, µ)).
Proof. The result follows from the Cramer-Chernoff method (see e.g. [START_REF] Boucheron | Concentration inequalities. A non asymptotic theory of independence[END_REF]). Using Markov inequality, the independence of the X i and the upper bound on the φ X i (λ), one can write for any λ > 0,

P 1 s s i=1 X i ≥ x = P e λ ∑ s i=1 X i ≥ e λsx ≤ e -λsx s i=1 E e λX i = e -λsx s i=1 e φ X i (λ) ≤ e -λsx e φ Y (λ) s = e -s(λx-φ Y (λ)) .
Optimizing in λ ∈ R + to obtain the tightest possible inequality yields the result. Indeed, for x > µ it can be shown that sup

λ∈R + (λx -φ Y (λ)) = sup λ∈R (λx -φ Y (λ)) = φ * Y (x) = d(x, µ),
using Lemma 1.4. The proof for x < µ follows the same lines.

◻

Deviation inequalities for non-parametric distributions can be deduced from Lemma 1.5. The distribution of a random variable X is called σ 2 -subgaussian if φ X (λ) ≤ λ 2 σ 2 2, which is the momentgenerating function of the distribution N 0, σ 2 . The associated divergence is d(x, y) = (x -y) (2σ 2 ) and one obtains from Lemma 1.5 that the empirical mean of i.i.d. samples of a σ 2 subgaussian distribution satisfies

P (μ s ≥ x) = P (μ s ≤ -x) ≤ exp -s x 2 2σ 2 .
From Hoeffding's lemma (see [Hoeffding, 1963]) every centered distribution with bounded support in

[a, b] is (b-a) 2
4 -subgaussian. This remark yields Hoeffding's inequality.

Lemma 1.6 (Hoeffding's inequality). Let (X i ) be an i.i.d. sequence with mean µ that is supported in

[a, b]. Then P (μ s ≥ µ + x) = P (μ s ≤ µ -x) ≤ exp -s 2x 2 (b -a) 2 .
Moreover, when X 1 is supported in [0, 1] and has mean µ, another result from [Hoeffding, 1963] shows that φ X 1 (x) is upper bounded by φ B(µ) (x), the log moment-generating function of a Bernoulli distribution with same mean µ. Thus Lemma 1.5 also applies to bounded distributions with support in [0, 1], taking d(x, y) = KL (B(x), B(y)).

Refined confidence intervals for bounded and Bernoulli distributions. For Bernoulli distributions and for bounded distributions supported in [0, 1], both Hoeffding and Chernoff inequalities can be used to build confidence intervals on the mean. Let d(x, y) be the divergence associated to Bernoulli distributions. Let (X i ) be an i.i.d. sequence of Bernoulli (or bounded) random variables with mean µ and μs be the empirical mean of the first s observations. Hoeffding's inequality (Lemma 1.6) yields the following confidence interval for the mean µ:

P ⎛ ⎝ µ ∈ ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ μs - log(1 δ) 2s ; μs + log(1 δ) 2s ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ ⎞ ⎠ ≥ 1 -2δ,
(1.12) whereas using Chernoff inequality (Lemma 1.5) it can be shown that

P sd (μ s , µ) ≤ log (1 δ) ≥ 1 -2δ.
(1.13) Indeed, one has for example, letting x * be defined by x * < µ and sd(x * , µ) = log(1 δ), This second confidence region, (1.13), is implicitly defined through the function d. For example, the associated upper confidence bound can be written

P(µ ≥ μs , sd(μ s , µ) ≥ log(1 δ)) = P(μ s ≥ x * ) ≤ e -sd(x * ,µ) = δ.
u s (δ) = max {q > μs ∶ sd (μ s , q) ≤ log(1 δ)} .
The confidence region (1.12) also takes the form (1.13) if the function d is replaced by d(x, y) = 2(xy) 2 . From Pinsker's inequality, one has d(x, y) > 2(x -y) 2 . While the two confidence intervals have the same coverage probability, Figure 1.1 illustrates that the confidence interval of (1.13), represented in blue, is contained in that of (1.12), represented in black. The use of these refined confidence intervals based on the divergence d will be crucial to build asymptotically optimal bandit algorithms.

Asymptotically optimal algorithms

In addition to their lower bound on the regret, [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] provide the first asymptotically optimal bandit algorithms. These first optimal policies are index policies: at each round t, an index is computed for each arm (based on the past observations of this arm only), and the arm A t chosen is the one with highest index. Index policies are reminiscent of the policy based on Gittins indices introduced earlier by [Gittins, 1979] in a Bayesian framework, that will be presented in Section 1.3. The form of the indices proposed by [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] is however not very explicit, and the analysis proposed is asymptotic. [Agrawal, 1995] proposes simpler index policies and introduces the notion of UCB-type algorithm (for Upper Confidence Bound). Indeed, the indices used involve the empirical mean of past rewards and the number of draws of each arm and can be interpreted as upper confidence bound for the unknown mean of each arm. In the particular case of Gaussian bandits with known variance σ 2 , [START_REF] Katehakis | Sequential choice from several populations[END_REF] propose the following fully explicit index policy and prove its asymptotic optimality. After an initialization phase in which each arm is drawn once, their policy chooses at time t + 1

A t+1 = argmax a U a (t) with U a (t) = μa (t) + 2σ 2 log(t) N a (t) .
UCB-type algorithms were popularized at the beginning of the years 2000, with the introduction of the UCB1 algorithm by [Auer et al., 2002a], for which the authors provide the first finite-time analysis. UCB1 is designed for the class of bandit models whose arms distributions have bounded support in

CHAPTER 1. TWO PROBABILISTIC VIEWS ON REWARDS MAXIMIZATION IN BANDIT MODELS

[0, 1], that we refer to as bounded bandit models in the sequel. UCB1 chooses at time t + 1 the arm

A t+1 = argmax a U a (t) with U a (t) = μa (t) + β(t) 2N a (t) ,
with an exploration rate β(t) = 4 log(t). At round t, for arm a, the number of draws N a (t) being fixed, U a (t) can be interpreted as an Upper Confidence Bound obtained with Hoeffding inequality (see (1.12)) that holds with probability larger than 1 -1 t 4 . UCB-type algorithms use the principle of optimism in face of uncertainty: among all the statistically plausible models, the arm chosen is the optimal arm in the best possible model (in which the mean of each arm is equal to its upper confidence bound). The theoretical guarantees for UCB1 are the following.

Theorem 1.7 ( [Auer et al., 2002a],Theorem 1). Defining ∆ a ∶= µ * -µ a as the squared gap between the optimal arm and arm a, if ν is a bounded bandit model,

R ν (T, UCB1) ≤ 8 ⎛ ⎝ a∶µa<µ * 1 ∆ a ⎞ ⎠ log(T ) + 1 + π 2 3 K a=1 ∆ a .
[ Auer et al., 2002a] thus provide an efficient algorithm, together with a logarithmic upper bound on its regret, for the non-parametric class of bounded bandit models, which is interesting in itself. However, the class M of bounded bandit models is not identifiable (according to Definition 1.1), and Theorem 1.2 does not apply. There is no lower bound on the regret of algorithms that are uniformly efficient on this class M to which the result of Theorem 1.7 could be compared. But for Bernoulli bandit models, to which UCB1 can also be applied, this algorithm is sub-optimal with respect to the Lai and Robbins' lower bound. Indeed, Pinsker's inequality shows that d(µ a , µ * ) > 2∆ 2 a , hence the constant in front of log(T ) is at least sixteen times bigger than the optimal constant prescribed by (1.10).

Successive refinements in the proof of Theorem 1.7, like the use of a smaller exploration rate of the form β(t) = α log(t) with α > 1 (see e.g. [START_REF] Audibert | Exploration-exploitation trade-off using variance estimates in multi-armed bandits[END_REF], Bubeck, 2010]) lead to a smaller constant in front of the log(T ), yet still expressed in terms of the squared gaps ∆ a . The UCB-V algorithm of [START_REF] Audibert | Exploration-exploitation trade-off using variance estimates in multi-armed bandits[END_REF], still designed for bounded bandit models, uses for each arm a confidence interval built by taking into account the empirical variance of the arm (built using a empirical Bernstein bound). The regret bound obtained is no longer only a function of the means of the arms, but involves the quantities σ 2 a ∆ a for each suboptimal arm, where σ 2 a is the variance of arm a. The UCB-Tuned algorithm proposed by [Auer et al., 2002a] also uses variances estimates, but no theoretical guarantees for this algorithm are provided.

To obtain UCB-type algorithms matching Lai and Robbins' lower bound, one needs to use refined confidence intervals, based on Kullback-Leibler divergence. This idea, already suggested by [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF], Lai, 1987, Agrawal, 1995] reappeared with the DMED algorithm of [START_REF] Honda | An Asymptotically Optimal Bandit Algorithm for Bounded Support Models[END_REF], designed for bounded bandit models. [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF]] study a simpler index policy, that bears some similarities with DMED, called KL-UCB. While the general KL-UCB algorithm does not require any parametric assumption on the rewards distributions, the authors provide theoretical guarantees in two interesting parametric cases. They give a finite-time analysis for KL-UCB applied to exponential family bandit models (previously studied by [START_REF] Garivier | The KL-UCB algorithm for bounded stochastic bandits and beyond[END_REF]) and for bandit models whose arms distributions have a finite (known) support (previously studied by [START_REF] Maillard | A Finite-Time Analysis of Multi-armed Bandits Problems with Kullback-Leibler Divergences[END_REF]).

In an exponential bandit model, if

d(x, y) = KL(ν ḃ-1 (x) , ν ḃ-1 (x)
) is the associated divergence, the KL-UCB algorithm is the index policy associated to

u a (t) = sup{q ≥ μa (t) ∶ N a (t)d (μ a (t), q) ≤ β(t)}, 1.2. THE FREQUENTIST APPROACH 47 with β(t) = log(t) + 3 log log(t)
. This index appears as the upper confidence bound of a confidence interval built with Chernoff inequality (see (1.13)) that holds with probability 1 -1 (t log 3 (t)). This particular case of the algorithm is called kl-UCB by [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF]. In this document, the capital notation is kept to stand for the algorithm designed for rewards in an exponential family. One has the following result.

Theorem 1.8 ( [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF], Theorem 1). In an exponential bandit model with associated divergence d, the KL-UCB algorithm satisfies, for any suboptimal arm a,

E θ [N a (T )] ≤ 1 d(µ a , µ * ) log(T ) + 2 2πσ 2 a, * d ′ (µ a , µ * ) 2 d(µ a , µ * ) 3 log(T ) + 3 log log(T ) + 4e + 3 d(µ a , µ * ) log log(T ) + 8πσ 2 a, * d ′ (µ a , µ * ) d(µ a , µ * ) 2 + 6, where σ 2 a, * = max{Var[ν θ ] ∶ µ a ≤ E(ν θ ) ≤ µ * } and d ′ (x, y) = ∂d(x,y) ∂x .
It follows from Theorem 1.8 and the regret decomposition (1.1) that KL-UCB is asymptotically optimal in every class of exponential bandit models. As explained by [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF], KL-UCB can also be used as it is for bounded bandit models (with support in [0, 1]) with the divergence d associated to Bernoulli distributions, and the upper bound of Theorem 1.8 still holds true. This is (mostly) because the confidence region (1.13) also holds for this non-parametric class of bandit models. Similarly, it can be shown that KL-UCB with the divergence d(x, y) = (x -y) 2 (2σ 2 ) associated to Gaussian distributions with variance σ 2 can also be applied to bandit models with σ 2 -subgaussian rewards distributions, with the same theoretical guarantees. Thus, the analysis of [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF]] also yields a new upper bound on the regret of UCB1 in bounded bandit models: this algorithm can indeed be seen as KL-UCB with the divergence d(x, y) = 2(x -y) 2 associated to Gaussian distributions with variance 1/4. Sketch of a finite-time analysis. We do not provide the proof of Theorem 1.8, but we present the general structure of the finite-time analysis proposed by [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF], that can be applied to any optimistic index policy. We highlight the improvements proposed by the authors, some of which will be useful in the rest of this thesis.

Let ν = (ν 1 , . . . , ν K ) be a stochastic bandit model. An optimistic index policy proceeds in the following way. After an initialization phase in which each arm is drawn once, the arm chosen at time t + 1 maximizes some index U a (t) = U a,Na(t),t , chosen such that for all s ≤ t, P(U a,s,t < µ a ) ≤ e -β(t) for some exploration rate β(t). To upper bound the regret of such a strategy, using the decomposition (1.1), it is sufficient to upper bound, for every suboptimal a, the quantity

E ν [N a (T )] = E ν T t=1 1 (At=a) = 1 + E ν T -1 t=K 1 (A t+1 =a) .
The event (A t+1 = a) can then be decomposed in the following way, depending on whether the optimal arm µ 1 is or not under-estimated by its index U 1 (t). One also uses that if the suboptimal a is drawn at time t + 1, one has in particular U a (t) > U 1 (t). This leads to

(A t+1 = a) ⊆ (U 1 (t) < µ 1 ) ∪ (A t+1 = a, U 1 (t) > µ 1 ) ⊆ (U 1 (t) < µ 1 ) ∪ (A t+1 = a, U a (t) > µ 1 ). It follows that E ν [N a (T )] ≤ 1 + T t=K+1 P ν U 1,N 1 (t),t ≤ µ 1
Term A: related to the possible under estimation of optimal arm

+ T -1 t=K P ν A t+1 = a, U a,Na(t),t > µ 1
Term B: related to the possible over estimation of the suboptimal arm a

The 'under-estimation' term A is shown to be of order o(log(T )). To upper bound this term, one needs to control the probability of the event U 1,N 1 (t),t ≤ µ 1 , that involves the self-normalized quantity U 1,N 1 (t),t . This denomination comes from the fact that U 1,s,t is already random, and the number of observations N 1 (t) is also random. A first idea to control this quantity is to use a union bound:

P ν U 1,N 1 (t),t ≤ µ 1 ≤ P ν (∃s ∈ {1, . . . t} ∶ U 1,s,t ≤ µ 1 ) ≤ t s=1 P ν (U 1,s,t ≤ µ 1 ) ≤ te -β(t) .
[ [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF] advocate the use of more sophisticated self-normalized deviation inequalities to control this probability. To be able to analyse KL-UCB, the authors propose an informational selfnormalized deviation inequality, that is expressed with the divergence function d, stated in Lemma 1.9. [Garivier, 2013] presents more general self-normalized inequalities, and discuss different applications.

Lemma 1.9. Let (X i ) be a sequence of independent random variables such that φ X i (λ) ≤ φ Y (λ) where Y ∼ ν θ belongs to an exponential family with mean µ and associated divergence d. One has

P ∃s ∈ {1, . . . , t} ∶ s d + 1 s s i=1 X i , µ > γ ≤ e ⌈γ log(t)⌉ exp(-γ),
where

d + (x, y) = d(x, y)1 (x<y) .
From Lemma 1.5, introducing also d -(x, y) = d(x, y)1 (x>y) , it easily follows that

P(s d + (μ s , µ) ≥ γ) ≤ e -γ and P(s d -(μ s , µ) ≥ γ) ≤ e -γ (1.14)
Whereas a union bound, using (1.14) for all s, would upper bound the probability in Lemma 1.9 by te γ , the bound in Lemma 1.9 is of order log (t) ) )e -γ , and is thus significantly smaller. To obtain such a result, the union bound, summing over all the possible values of N 1 (t) ∈ {1, . . . , t} is replaced by a geometric 'peeling': one considers 'slices' on which

N 1 (t) ∈ [(1 + η) k-1 , (1 + η) k ]
for some parameter ν > 0. On each slice, a maximal inequality for a well-chosen martingale is applied. This 'peeling-trick' will also be used in the analysis of Bayes-UCB and Thompson Sampling in the next two chapters. Lemma 1.9 will also be used directly several times in this document. We thus provide its proof in Section A.2 of Appendix A, in which we also present other self-normalized deviation inequalities used in this thesis. The logarithmic factor in the regret comes from Term B. The upper bound proposed in Lemma 1.10 below allows to replace the self normalized quantity U a,Na(t),t by the quantity U a,s,T . This trick, introduced by [START_REF] Garivier | The KL-UCB algorithm for bounded stochastic bandits and beyond[END_REF], will be applied several times throughout this thesis, and leads to Lemma 1.10.

Lemma 1.10. If the index U a,s,t is nondecreasing in t,

T t=K P ν A t = a, U a,Na(t),t > µ 1 ≤ T s=1 P ν (U a,s,T > µ 1 ) ; 1.3. THE BAYESIAN APPROACH
Proof. The trick consists in upper bounding U a,s,t by U a,s,T and noting that only one term in the sum ∑ T t=s 1 (At=a,Na(t)=s) can be different from zero. This sum can then be upper bounded by 1. More precisely, one writes

T t=K P ν (A t = a, U a (t) > µ 1 ) = E ν T t=K t s=1 1 (At=a,Na(t)=s) 1 (Ua,s,t>µ 1 ) ≤ E ν ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ T s=1 1 (U a,s,T >µ 1 ) T t=s 1 (At=a,Na(t)=s) ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ ≤ T s=1 P ν (U a,s,T > µ 1 ) .

◻

It remains to upper bound the right hand side of the inequality in Lemma 1.10. As U a,s,T is an upper bound on the mean µ a < µ 1 , for s larger enough, we will be able to show (with a Chernoff-type inequality for the UCB1 or KL-UCB algorithms) that the probability P (U a,s,T > µ 1 ) is very small. The critical value of s is of order C 1,a log(T ), where C 1,a is a constant that depends on the distribution of arms 1 and a and on the choice of upper bounds.

The Bayesian approach

Bandit models were introduced in a Bayesian framework to model medical trials with two possible treatments. As this particular case was considered in an overwhelming majority of the first bandit papers, we start by presenting Bayesian Bernoulli bandit models. Treatment 1 (arm 1) has a probability of success p, and treatment 2 (arm 2) has a probability of success q. One assume that (p, q) is drawn from a prior distribution with density H(p, q) with respect to some reference measure. [Thompson, 1933] focuses on a particular case of product prior H(p, q) = F (p)G(q), in which F and G are Beta distributions. He proposes a randomized approach, in which each arm is drawn according to its posterior probability of being optimal. In the two papers [Thompson, 1933, Thompson, 1935], the author focuses on the explicit computation of these posterior probabilities, but no analysis or performance study of the algorithm is proposed. More than 70 years will go by without advances in the study of this first bandit algorithm. Chapter 3 is focused on Thompson Sampling. We will provide therein bibliographic details from recent studies of this strategy and above all our own proof that this algorithm is asymptotically optimal in a frequentist sense.

Bandit problems were considered later in the work of [Robbins, 1952], who state the bandit problem in a frequentist setting. [START_REF] Bradt | On sequential designs for maximizing the sum of n observations[END_REF] and [Bellman, 1956] study a particular Bayesian bandit model, in which the parameter of one of the arms is assumed to be known: q = q 0 is fixed, while p is drawn from a prior distribution F (p). The joint prior distribution is thus of the form H(p, q) = F (p)δ q 0 (q). For a fixed horizon T , [START_REF] Bradt | On sequential designs for maximizing the sum of n observations[END_REF] show that there exists a strategy minimizing the Bayes risk, that depends at time t on the remaining time to play n = Tt and on the current posterior distribution F on p. The unknown arm is played if and only is some index G(F, n) is larger than the mean q 0 of the known arm. [Bellman, 1956] considers the same Bayesian bandit model in which one arm is known, but rather studies a discounted bandit problem. For a fixed α ∈]0, 1[, called the discount factor, the goal is to maximize the expectation of the sum of discounted rewards, defined by

E H ∞ t=1 α t-1 X t .
He shows that the optimal strategy for this problem is solution of some dynamic programming equations. The theory of dynamic programming was at the time still under development (see [Bellman, 1954]). Quite similarly to the undiscounted case, the optimal strategy plays the unknown arm if and only if some index is larger that q 0 . In this case the index depends on the current posterior distribution F and on the discount factor α.

More general bandit problems were considered thereafter, with more than two arms, general rewards distributions, and even different objectives (e.g. maximizing the expected discounted rewards, where the sequence of discount is not necessary geometric). They are presented by [START_REF] Berry | Bandit Problems. Sequential allocation of experiments[END_REF], who give an overview of the bandit literature (mostly Bayesian) up to the 1980's. The two objectives considered above are notably presented: maximizing the expected sum of rewards up to some horizon T (i.e. minimizing the Bayes risk according to our definition) and maximizing the expected sum of discounted rewards (i.e. solving the discounted bandit problem). [START_REF] Berry | Bandit Problems. Sequential allocation of experiments[END_REF] explain that any such bandit problem appears as the solution of a dynamic programming equation. We present here this associated dynamic programming equation using elements from the theory of Markov Decision Processes (see e.g. [Puterman, 1994, Sigaud and[START_REF] Sigaud | [END_REF]), developed more recently. A Markov Decision Process (MDP) is a 4-tuple (X , A, P, R) where X is a state space, A is an action space, P ∶ X × A → M 1 (X ) is a transition kernel and R ∶ X × A → M 1 (R) is a reward kernel. When the agent is in state x ∈ X and chooses action a ∈ A, a transition occurs: the agent receives a reward r ∼ R(⋅ x, a) and his new state is y ∼ P(⋅ x, a).

The interaction of an agent with a bandit model in a Bayesian framework can be modeled by the following MDP. Let θ be drawn from a prior distribution Π 0 .

-the current state is the current posterior distribution Π t on θ: X ⊂ M 1 (Θ K ) -there are K actions, corresponding to the draw of each arm: A = {1, . . . , K} -when the agent chooses arm a in state Π, he observes a draw from arm a, x ∼ ν θa , receives the reward x, and computes the new posterior distribution Π ′ obtained by taking into account the new observation x. We let t x a be the operator such that Π ′ = t x a (Π). The agent's sampling strategy corresponds to a policy in this MDP. A deterministic policy g is a mapping that indicates which action is chosen in a state x t and at time t. If (X g t ) is a sequence of successive rewards obtained with g, one can consider the value function of this policy. Depending on the criterion considered (finite horizon or α-discounted rewards), value functions are defined by

V g (Π, T ) = E Π T t=1 X g t and V g α (Π) = E Π ∞ t=1 α t-1 X g t .
Bayes risk minimization is equivalent to solving the planning problem in the above MDP with a finitehorizon criterion, that is finding a policy maximizing V g (Π, T ). Similarly, the discounted bandit problem corresponds to the planning problem in the same MDP but with a discounted criterion.

From the the theory of MDPs, there exists an optimal policy g * such that its value function V * satisfies a dynamic programming equation. For the finite horizon criterion, for all k = 1, . . . , T -1, if Π a denotes the a th marginal distribution of Π, the optimal value function satisfies

V * (Π, k) = max a=1...K E θa∼Π a [µ(θ a )] + E W ∼f (⋅ θa) θa∼Π a V * (t W a (Π a ), k -1) (1.15) V * (Π, 0) = 0
The optimal value function can thus be computed by induction and the optimal policy in state Π if the remaining time to play is k chooses an action g * (Π, k) that realizes the maximum in (1.15). The optimal strategy for the Bayesian bandit problem with finite horizon is therefore

A t = g * (Π t-1 , T -t + 1).
For the discounted criterion, the optimal value function is solution of the following equation:

V * α (Π) = max a=1...K E θa∼Π a [µ(θ a )] + αE W ∼f (⋅ θa) θa∼Π a V * α (t W a (Π)) .
Due to the potentially very large state space, usual techniques to compute the optimal policy (value or policy iteration, induction when the horizon is finite) are often intractable. However, in some particular cases, a more practical description of the optimal policy can be obtained.

The solution of the Bayesian discounted or undiscounted bandit problem strongly depends on the prior distribution Π 0 . [Feldman, 1962] manages to exhibit this solution, for a finite horizon T , in a very special case of two-armed binary bandit model. The prior distribution he considers captures the fact that the means of the arms p 1 > p 2 are known:

P((p, q) = (p 1 , p 2 )) = ζ and P((p, q) = (p 2 , p 1 )) = 1 -ζ. If ζ t
denotes the posterior probability of the event {(p, q) = (p 1 , p 2 )} at the end of round t, [Feldman, 1962] shows that the optimal policy chooses arm 1 at time t if and only if ζ t-1 > 1 2. This myopic strategy chooses at each time the arm with highest posterior mean reward. It is not true that such a strategy is in general optimal.

In this particular example, the marginal distributions of each arm are very correlated. On the contrary, [Gittins, 1979] considers prior distributions with independent arms and shows that the solution of the discounted bandit problem reduces to an index policy. The computation of the Gittins index for each arm also resorts to dynamic programming but for a reduced state space. In Section 1.3.2, we define both the discounted and undiscounted Gittins indices (or Finite-Horizon Gittins indices) and we discuss the optimality of the associated index policies in Section 1. 3.3. Unlike what happens in the discounted case, the index policy associated to the Finite-Horizon Gittins indices does not coincide with the Bayesian solution of the bandit problem with a finite horizon. However we will show that the FH-Gittins algorithm performs well in practice, and that approximations of the FH-Gittins indices presented in Section 1. 3.4 show similarities with asymptotic approximations of the optimal strategy, discussed in Section 1.3.5.

Notation. Gittins indices are defined in Bayesian bandit models with independent arms, that we consider in the rest of this chapter. In this case, θ = (θ 1 , . . . , θ K ) is drawn from a product prior Π 0 = (π 0 1 , . . . , π 0 K ) such that π 0 a is the prior distribution on θ a , and the distributions (π 0 a ) a=1...K are independent. The posterior distribution on θ after t observations is thus a product prior,

Π t = (π t 1 , . . . , π t K ),
with the following notation:

π a,s is the posterior distributions on θ a after the first s observations of arm a:

π a,s = L(θ a Y a,1 , . . . , Y a,s ).
π t a = π a,Na(t) is the posterior distribution on θ a at the end of round t 1.3.1 Some examples of Bayesian bandit models.

In the particular case of exponential bandit models, introduced in Section 1.2.2, we now give examples of prior distributions that can be used, as well as a more explicit presentation of the associated Distribution Mean Prior distribution Posterior distribution on the mean on the mean after k observations whose sum is s

B(µ) µ Beta(a, b) Beta(a + s, b + k -s) N µ, σ 2 µ N µ 0 , m -1 0 N m 0 µ 0 +σ -2 s m 0 +kσ -2 , (m 0 + kσ -2 ) -1 P(λ) λ Γ(c, d) Γ(c + s, d + n) E(λ) 1 λ InvΓ(c, d) InvΓ(c + n, d + s) Γ(k, λ) k λ InvΓ(c, d) InvΓ(c + kn, d + ks)
Table 1.4: Conjugate prior on the mean and associated posterior distributions.

Markov Decision Process in the Bernoulli case. Recall that the distribution of arm a conditionally to θ a has a density of the form

f (x θ a ) = A(x) exp(θ a x -b(θ a )).
A nice property of this family of distributions is that if the prior distribution π 0 a has a density h 0 a , the density of the posterior distribution on θ a , π a,k takes the following simple parametric form:

p(θ a Y a,1 , . . . , Y a,k ) ∝ exp θ a k i=1 Y a,i -kb(θ a ) h 0 a (θ a ). (1.16)
This distribution can be parameterized by two sufficient statistics, k, s = ∑ k i=1 Y a,k : the number of observations and the sum of observations. The current posterior distribution on θ could therefore be described by a state S = {(k a , s a )} K a=1 ∈ (N × R) K . Distributions that form an exponential family are often more simply parameterized by their means than by their natural parameter θ. In many examples of practical interest, it is possible to choose a prior distribution on the mean (and no longer on the natural parameter, as it is the case above) that belongs to a family of conjugate priors. A family of conjugate priors is such that if the prior distribution belongs to that family, the same goes for all the associated posterior distributions. Table 1.4 gives examples of conjugate priors on the mean for several examples of distributions in an exponential families. The posterior distribution, that depends on the number of observations k and on the sum of observations s is also computed. Most of the densities of the different distributions involved are defined in Table 1

.1. The inverse Gamma distribution InvΓ(c, d) is the distribution of the random variable 1 X if X ∼ Γ(c, d).
In the Bernoulli case, a natural prior distribution on the mean is a Beta distribution, denoted by Beta (a, b). This distributions has a bounded support in [0, 1] and a density

f (a,b) (x) = Γ(a + b) Γ(a)Γ(b) x a-1 (1 -x) b-1 1 [0,1] (x).
In a Bernoulli bandit model with a Beta(a, b) prior on each mean, the posterior distribution on the mean of arm a at the end of round t is π t a = Beta(S a (t) + a, N a (t) + b). One often considers the uniform prior U([0, 1]) which correspond to Beta(1, 1). The history of a Bernoulli bandit game with Beta prior is summarized by a matrix of Beta posteriors, S = {(A a , B a )} K a=1 , that evolves as a state in the following Markov Decision Process:

-state

S = {(A a , B a )} K a=1 ∈ (N × N) K -actions {1...K} 1.3. THE BAYESIAN APPROACH 53 -if the current state is S = {(A a , B a )} K a=1
, and action A t = a is chosen, a binary reward X t = X a,t is drawn from B(µ a ) and the state (i.e. the posterior distribution) is updated in the following way:

A a ← A a + X t B a ← B a + (1 -X t ).
⎛ ⎜ ⎝ 1 2 5 1 0 2 ⎞ ⎟ ⎠ At=2 → ⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ ⎛ ⎜ ⎝ 1 2 6 1 0 2 ⎞ ⎟ ⎠ if X t,2 = 1 ⎛ ⎜ ⎝ 1 2 5 2 0 2 ⎞ ⎟ ⎠ if X t,2 = 0
V * ((a, b, c, d), n) = max a a + b + a a + b V * ((a + 1, b, c, d), n -1) + b a + b V * ((a, b + 1, c, d), n -1); c c + d + c c + d V * ((a, b, c + 1, d), n -1) + d c + d V * ((a, b, c, d + 1), n -1) (1.17)
and the optimal policy in state (a, b, c, d), with remaining time to play n, chooses arm 1 if the first argument is the maximum, arm 2 otherwise. For T not too large, the decisions of the optimal policy, g * ((a, b, c, d), n) ∈ {1, 2} can be computed by induction and stored, for elements ((a, b, c, d), n) such that a + b + c + d ≤ n + α + β and n ≤ T (α and β being the parameters of the prior). This permits to implement the optimal policy for small values of T . When there are more arms, the exact solution becomes even less tractable.

Discounted and Finite-Horizon Gittins indices

Gittins' theorem, stated in the seminal paper [Gittins, 1979], says that an index policy, based on the (later) so-called Gittins indices, is a solution to several sequential allocation problems, when the objective is to maximize the expected sum of discounted rewards. This larger class of problems can be interpreted as planning problems in Markov Decision Processes with particular structure and is presented in the book [START_REF] Gittins | Multi-armed bandit allocation indices[END_REF]]. Here we introduce Gittins' indices only in the context of (Bayesian) multi-armed bandit models with independent arms.

A possible definition of the Gittins' indices (like the one given by [START_REF] Berry | Bandit Problems. Sequential allocation of experiments[END_REF]) relies on the introduction of a calibration problem for each arm, that we call C λ . This calibration problem is sometimes called 'one-armed bandit problem'.

Let θ ∼ π and, conditionally to θ, let (X t ) be an i.i.d. sequence with distribution ν θ , that is a one-armed bandit. For λ ∈ R, one considers the following game, denoted by C λ . A each time t, an agent can choose between receiving a (known) reward λ or drawing the unknown arm and receiving a random reward drawn from ν θ . His goal is to maximize his rewards with respect to one of the criteria previously considered: either the sum of rewards up to horizon T , or the sum of discounted rewards with a discount factor α. This game can be naturally expressed as a planning problem in a MDP, and writing the associated dynamic programming equation, one can show that the optimal policy is a stopping policy: the unknown arm is played until some stopping time τ after which the reward λ is chosen until the end of the game. When the value of λ gets larger, the player has less incentive to play the unknown arm. There exists a critical value λ * such that for larger value of λ, the optimal strategy in C λ never draws the unknown arm (and always chooses λ). This critical value, which represents the price worth paying for playing the arm, is the Gittins index.

In the discounted one-armed bandit, the optimal policy plays the unknown arm as long as the current posterior distribution π on the parameter θ is such that G α (π) > λ, with G α (π) the Gittins index, defined in the following way.

Definition 1.11. The (discounted) Gittins index for the current posterior distribution π is

G α (π) = inf λ ∈ R ∶ sup τ ≥0 E π τ t=1 α t-1 X t + α τ λ 1 -α = λ 1 -α ,
where the supremum is taken over the set of stopping times τ , with the convention ∑ 0 t=1 = 0.

For Bernoulli bandit models, the calibration problem C λ was already solved by [Bellman, 1956] in the discounted case. In the case of a finite horizon T , one can also generalize the solution proposed by [START_REF] Bradt | On sequential designs for maximizing the sum of n observations[END_REF] in the binary case. In the finite-horizon one-armed bandit, the optimal policy plays the unknown arm as long as the current posterior distribution π and the remaining time to play n are such that G(π, n) > λ, where G(π, n) is the Finite-Horizon Gittins index, defined in the following way.

Definition 1.12. The Finite-Horizon Gittins index for a current posterior π and remaining time n is

G(π, n) = inf λ ∈ R ∶ sup 0≤τ ≤n E π τ t=1 X t + λ(n -τ ) = nλ
where the supremum is taken over the set of stopping times τ smaller than n almost surely, with the convention ∑ 0 t=1 = 0.

These Dynamic Allocation Indices, later called Gittins indices, were defined differently in Gittins' original paper (in the discounted case only). It is not difficult to show (see e.g. [START_REF] Gittins | Multi-armed bandit allocation indices[END_REF]) that the following two equalities hold, showing as a by product that Gittins' definition coincides with Definition 1.11:

G α (π) = sup τ >0 E π ∑ τ t=1 α τ -1 X t E π [∑ τ t=1 α τ -1 ] and G(π, n) = sup 0<τ ≤n E π [∑ τ t=1 X t ] E π [τ ]
.

(1.18)

In a multi-armed bandit problem, the Gittins index of an arm whose current posterior is π can be seen as the price worth paying to play this arm (the critical value of λ in the calibration problem for this arm), or the mean reward per unit of time it yields.

We mention here some properties of the FH-Gittins indices that easily follow from their definition and will be useful for the implementation of the associated index policy. First, from the expression

1.3. THE BAYESIAN APPROACH 55 (1.18), the index is lower bounded by E π [∑ τ t=1 X t ] E π [τ ] for any stopping time τ . For τ = 1 p.s., one obtains G(π, n) ≥ E X∼ν θ θ∼π [X], (1.19)
which allows to interpret the Gittins index as some upper confidence bound on the (posterior) mean, providing an analogy with UCB-like algorithms presented in Section 1.2. Moreover, one can show that, for all n ≥ 2, G(π, n) ≥ G(π, n -1).

(1.20)

To prove this, assume that there exists y such that G(π, n) < y < G(π, n -1) and consider the calibration problem C y with horizon n. As G(π, n) < y, the optimal policy starts by choosing y. But as y < G(π, n -1), the next optimal action is to draw the unknown arm, which contradicts the fact that the optimal policy is a stopping policy.

Computation of the Gittins indices. Chapter 8 of [START_REF] Gittins | Multi-armed bandit allocation indices[END_REF] addresses the computation of discounted Gittins indices in some examples of exponential family bandit models, whereas [Nino-Mora, 2011] discusses the computation of Finite-Horizon Gittins indices. Among the methods reviewed by the latter, the calibration method seems to perform well. This approaches solves the calibration problem C λ for a grid of values of λ, and identify an approximation of the critical value λ * above which the optimal policy in C λ * never plays the unknown arm. We now discuss our implementation of the calibration method for computing the indices G((a, b), n) in a Bernoulli bandit model with Beta prior ((a, b) are the parameters of the Beta posterior). One has

G((a, b), n) = inf{λ ∈ R ∶ V * λ ((a, b), b) = nλ} with V * λ ((a, b), n) = sup 0≤τ ≤n E (a,b) τ t=1 X t + λ(n -τ ) .
As already noted, the optimal policy in the calibration problem C λ is a stopping policies, and thus V * ((a, b), n) is the optimal value function in this calibration problem. It satisfies the following dynamic programming equation

V * λ ((a, b), 0) = 0 for all (a, b) V * λ ((a, b), n) = max λn; a a + b + a a + b V * λ ((a + 1, b), n -1) + b a + b V * λ ((a, b + 1), n -1) .
V * λ ((a, b), n) can thus be computed by induction in O(n 2 ) arithmetic operations. Rather than using a grid of values of λ or a dichotomic search to obtain an approximation of G((a, b), n), we suggest to apply the secant method to find the first zero on [0, 1] of the convex function

Z(λ; a, b, n) ∶= V * λ ((a, b), n) -nλ = sup 0≤τ ≤n E (a,b) τ t=1 (X t -λ) .
An illustration is proposed in Figure 1.3. To initialize the algorithm, one needs two lower bounds on the Gittins index: from (1.19), the posterior mean a (a + b) is a lower bound on G((a, b), n), and a second lower bound can be found using a dichotomic search.

Besides, for n = 1, 2, it is easy to obtain a close form expression for V * ((a, b), n) and thus an explicit expression of the Gittins indices. The results are the following (and where already given by [START_REF] Bradt | On sequential designs for maximizing the sum of n observations[END_REF]):

G((a, b), 1) = a a + b and G((a, b), 2) = a a + b × 1 + a+1 a+b+1 1 + a a+b .
(1.21) These expressions illustrate the fact mentioned above that the Finite Horizon Gittins indices are upper bounds on the mean of the posterior distribution, a (a + b). Moreover, they suggest that these indices take the form

G((a, b), k) = a a + b + B(a, b, k),
where the confidence bonus B(a, b, k) seems to decrease when remaining time k decreases, unlike what happens with UCB indices. We will see in Section 1.3.4 that approximations of the Gittins indices indeed feature a different exploration rate than that used by UCB-like algorithms.

Index policies using Gittins indices

Gittins' theorem was presented for the first time by [START_REF] Gittins | A dynamic allocation index for the sequential design of experiments[END_REF], but was popularized by the paper [Gittins, 1979]. This result, particularized to bandit problems, is given here as the first statement of Theorem 1.13. The second statement of this theorem highlights the fact that in the undiscounted case, the index policy using the Finite-Horizon Gittins indices is no longer optimal. This is known since [START_REF] Berry | Bandit Problems. Sequential allocation of experiments[END_REF], who prove that a geometric discounting sequence is necessary for Gittins' theorem to hold (see Chapter 6 therein).

Theorem 1.13. Let α > 1. With an independent prior distribution Π 0 , the strategy choosing at round t

A t = argmax a=1...K G α (π t-1 a ) maximizes E Π 0 ∑ ∞ t=1 α t-1 X t . Let T ∈ N * .
There exists bandit models and independent prior distributions such that the strategy choosing at time t

A t = argmax a=1...K G(π t-1 a , T -t + 1) does not maximize E Π 0 ∑ T t=1 X t .
Several proofs of Gittins' theorem have been proposed thereafter by different authors. Our own proof, provided in Section 1.5.2, relies on the prevailing charge argument introduced by [Weber, 1992]. More precisely, it is inspired by the version of this proof presented by [START_REF] Frostig | Four proofs of Gittins' multiarmed bandit theorem[END_REF]. After 1.3. THE BAYESIAN APPROACH 57 proving the first statement of Theorem 1.13, we highlight which parts of the proof cannot be adapted to the finite-horizon setting. To prove the second statement of the theorem, we show that for some choices of independent Beta prior on two-armed Bernoulli bandits, for the horizon T = 2, the dynamic programming solution and Gittins' policy, that can both be computed in this simple case, do not coincide. To prove that the FH-Gittins index policy is not optimal, [START_REF] Berry | Bandit Problems. Sequential allocation of experiments[END_REF]] also exhibit a counterexample in the class of Bernoulli bandit models, but with a prior distribution on each mean that takes two values.

From the second statement of Theorem 1.13, the Finite Horizon Gittins algorithm (FH-Gittins), that we define as the index policy using the Finite-Horizon Gittins indices, is not optimal. However, we conjecture that this algorithm is a good approximation of the Bayesian solution of the bandit problem with a finite horizon. To support this claim, we start by presenting a numerical comparison of FH-Gittins and the optimal solution, on a two-armed Bernoulli bandit problem, and for a small horizon T = 70 for which the optimal Dynamic Programming (DP) solution can indeed be computed. For the FH-Gittins algorithm and the Dynamic Programming solution designed for the horizon T = 70 and a uniform prior Π 0 , we present in Figure 1.4 the Bayes risk BR Π 0 (t, A) as a function of time t. To estimate the Bayes risk, we average over N = 10 6 draws of a bandit model from the prior distribution Π 0 on which each algorithm is played up to horizon T = 70. We see that indeed the Bayes-risk at horizon T is such that BR Π 0 (T, DP) ≤ BR Π 0 (T, FH-Gittins), but the difference is really small. Moreover, the Bayes risk of FH-Gittins is even smaller at round t < T . [START_REF] Ginebra | Small-sample performance of Bernoulli two-armed bandit Bayesian strategies[END_REF]] also propose a comparison of different algorithms with the Bayesian optimal solution for small horizon on which it can be computed. In particular, the authors similarly notice that the Bayes-risk at time T of the Finite-Horizon Gittins algorithm (that they call Λ-strategy) is very close to the optimal value, for various choices of prior and horizons.

Due to the costly implementation of the Bayesian optimal policy, it is important to develop good approximations of this strategy that are also efficient. FH-Gittins seems to perform well (based on numerical experiments) and might appears as an appealing alternative from a computational perspective, since the computation of each index require to solve (several) dynamic programming equations but on a much reduced state space. However, for large horizons, these repeated computations of indices also take time. To circumvent this issue, a first idea is to replace the FH-Gittins indices used in the algorithm by good approximations. In the next section, we present some approximations, that follow from the work of [START_REF] Chang | Optimal stopping and dynamic allocation[END_REF] and [START_REF] Burnetas | Asymptotic Bayes Analysis for the finite horizon one armed bandit problem[END_REF]], but only hold in an asymptotic regime in which the remaining time to play is very large. Interestingly, the Finite Horizon Gittins indices happen to be connected to the indices used by the KL-UCB algorithm, the (frequentist) asymptotically optimal algorithm discussed in Section 1.2.3. In Section 1.3.5 we will show that variants of KL-UCB, that are easier to implement than FH-Gittins, are also good approximations of the Bayesian optimal solution in an asymptotic sense.

Approximation of the FH-Gittins indices

[ [START_REF] Chang | Optimal stopping and dynamic allocation[END_REF] develop approximations of the classical (discounted) Gittins indices for Gaussian bandit models with a Gaussian prior distribution, when the discount factor α is close to 1. To do so, they approximate the solution of each calibration problem C λ by the solution of an optimal stopping problem for a Brownian motion with drift, with some Gaussian prior distribution on the drift. They mention that a similar approach can be adopted to approximate the Finite-Horizon Gittins indices in the Gaussian case, that we explicit here.

In a Gaussian bandit model with known variance σ 2 , with independent Gaussian prior N 0, κ 2 on each mean, the posterior distribution on the mean of arm a after t observations is

N S a (t) N a (t) + σ 2 κ 2 , σ 2 N a (t) + σ 2 κ 2 .
Thus each arm is characterized by the vector (u a , v a ) of mean and variance of its current posterior distribution. The Gittins indices can be considered as functions of these two parameters. One has

G((u, v), n) = inf λ ∈ R ∶ sup 0≤τ ≤n E (u,v) τ t=1 (X t -λ) = 0 = inf λ ∈ R ∶ sup 0≤τ ≤n E (u-λ,v) ∑ τ t=1 X t σ √ n = 0 ,
where the expectation E (u,v) is taken under the model in which the X t are i.i.d. with distribution N θ, σ 2 conditionally to θ and θ ∼ N (u, v). In order to introduce the discretization of a continuous optimal stopping problem, one introduces the notation

t ′ = t n , u ′ λ = (u -λ) √ n σ , v ′ = vn σ 2 , w n (t ′ ) = ∑ t s=1 X t σ √ n , τ ′ = τ n and µ = θ √ n σ .
One has

w n (t ′ ) µ ∼ N µt ′ , t ′ and µ ∼ N u ′ λ , v ′ (1.22) thus , letting E ′ (u ′ ,v ′ ) be the expectation under the model (1.22), sup 0≤τ ≤n E (u-λ,v) ∑ τ t=1 X t √ n = sup τ ′ ∈{0, 1 n ,..., n-1 n ,1} E ′ (u ′ λ ,v ′ ) w n (τ ′ )
Letting B t be a Brownian motion, and considering stopping times T with respect to B t , the Finite-Horizon Gittins index can be approximated, for large values of n, as

G((u, v), n) ≃ inf λ ∈ R ∶ sup 0≤T ≤1 E µ∼N u ′ λ ,v ′ [µT + B T ] = 0 .
[ [START_REF] Chang | Optimal stopping and dynamic allocation[END_REF] propose an asymptotic approximation of the solution of the continuous stopping problem involved that yields, when v ′ → ∞,

T * λ ≃ ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ inf t ∈ [0, T ] ∶ w(t) ≤ -u ′ λ v ′-1 -2(t + v ′-1 ) log 1 t + v ′-1 ⎫ ⎪ ⎪ ⎬ ⎪ ⎪ ⎭ .
Approximating the Gittins index by the value of λ such that T * λ = 0 and returning to the original values

(u, v) yields G((u, v), n) ≃ u + 2v(log(vn) + o(log(vn)).
In a bandit game, the index computed by the FH-Gittins algorithm for arm a at round t + 1 can therefore be approximated by

S a (t) N a (t) + σ 2 κ 2 + 2σ 2 log T -t Na(t)+σ 2 κ 2 N a (t) + σ 2 κ 2 . (1.23)
This approximation holds when the remaining time T -t is large and when the ratio

(T -t) (N a (t)+κ -2
) is large too. [START_REF] Chang | Optimal stopping and dynamic allocation[END_REF]] also propose approximations of the discounted Gittins indices in exponential families.

For one-parameter canonical exponential families, [START_REF] Burnetas | Asymptotic Bayes Analysis for the finite horizon one armed bandit problem[END_REF] propose a different approach to approximate the solution of the calibration problems C λ with finite horizon, that leads to an approximation of the Finite Horizon Gittins indices. In an exponential bandit model, the distribution of each arm can be parameterized by a natural parameter θ a such that, for every θ ∈ Θ, ν θ has a density

f (x θ) = A(x) exp(xθ -b(θ)), θ ⊆ Θ ⊆ R.
As explained in Section 1.3.1, given a product prior distribution Π 0 , the density of the posterior distribution on the parameter of each arm has a simple form (1.16) and can be parameterized by the number of observations k from the arm and the sum of these observations, s. We denote it H (k,s) . The FH-Gittins index of this arm also depends on these two parameters and can be written

G((k, s), n) = inf {λ ∈ T ∶ V * ((k, s), n) = λn} , with V * ((k, s), n) = sup 0≤τ ≤n E (k,s) τ t=1 X t + (n -τ )λ .
The expectation E (k,s) is taken under the model in which the X t are i.i.d. with distribution ν θ conditionally to θ and θ ∼ H (k,s) . V * ((k, s), n) is the value function in the MDP associated to C λ . The work of [START_REF] Burnetas | Asymptotic Bayes Analysis for the finite horizon one armed bandit problem[END_REF] relies on the following assumption on the prior distribution Π 0 .

Assumption 1. Π 0 is supported on A K , where A =]θ -, θ + [⊆ Θ is such that there exists m, M > 0: ∀ θ ∈ A m ≤ b(θ) ≤ M.
(1.24)

Under this assumption we let ]µ -, µ + [ be the corresponding interval on the means: ]µ -, µ + [= ḃ(A).

The approximation of the solution of C λ given by [START_REF] Burnetas | Asymptotic Bayes Analysis for the finite horizon one armed bandit problem[END_REF] relies on a careful rewriting of the dynamic programming equation for V * ((k, s), n) (see 1.15). They exhibit, for the 'continuation set' (on which the optimal action in C λ is to draw the unknown arm)

S λ n = {(k, s) ∶ V * ((k, s), n) > nλ} ,
two sets S λ n and S λ n such that S λ n ⊆ S λ n ⊆ S λ n . The definition of these two sets involve integral expressions for which one can obtain asymptotic equivalents when the remaining time n goes to infinity. A close examination of Theorem 4.2 of [START_REF] Burnetas | Asymptotic Bayes Analysis for the finite horizon one armed bandit problem[END_REF] shows that, when n is large,

(V * ((k, s), n) > nλ) ∼ ⇔ λ ≤ max q ∈ [µ -, µ + ] ∶ q ≥ s k , k dA s k , q ≤ log n k
with dA (x, y) the function defined below. It depends on the divergence d(x, y) associated to the exponential family and is defined on

( ḃ(Θ)) 2 by dA (x, y) = ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ d(x, y) if x > µ - f (x θ -) f (x ḃ-1 (y)) if x < µ -.
Thus, for large values of n, the Gittins index can be approximated as

G((k, s), n) ≃ max q ∈ [µ -, µ + ] ∶ q ≥ s k , k dA s k , q ≤ log n k
and the index computed by the FH-Gittins algorithm for arm a at time t + 1 is approximately, when Tt is large,

max q ∈ [µ -, µ + ] ∶ q ≥ S a (t) N a (t) , N a (t) dA S a (t) N a (t) , q ≤ log T -t N a (t) . (1.25)
This approximation is consistent with the one obtained in the Gaussian case (1.23), for which the divergence function is d(x, y) = (x -y) 2 (2σ 2 ). Both approximations of the index used by the FH-Gittins algorithm only hold when the remaining horizon is large. The indices (1.23) and (1.25) are reminiscent of the index used by the KL-UCB algorithm, but with a slightly different exploration rate: log t is replaced by log((T -t) N a (t)). A natural question is therefore: does this modified exploration rate lead to improvements? As elements of answer, a quite similar modified version of KL-UCB will be proved in the next section to be a good approximation of the Bayesian solution, at least asymptotically. Besides, we will see in the numerical experiments of Section 1.4 that the factor 1 N a (t) indeed leads to improvements.

Asymptotically optimal algorithms with respect to the Bayes risk

We saw that in the Bayesian framework, for a given prior distribution, the bandit problem with finite horizon T has an exact solution, that we denote by A DP (since it is solution of a Dynamic Programming equation). However, we did not obtain an expression of the Bayes risk of this optimal solution, BR Π 0 (T, A DP ). In the particular case of exponential bandit models, [Lai, 1987] provides a priordependent, asymptotic lower bound on the Bayes-risk of any bandit algorithm as well as an algorithm matching this bound. Hence, the Bayes risk of the Bayesian optimal strategy in such bandit models must grow at the rate log(T ) 2 specified by the lower bound of [Lai, 1987]. In this section, we see that the Bayes risk of the KL-UCB algorithm, discussed in Section 1.2.3 almost matches this lower bound (Proposition 15), and we discuss possible variants of this algorithm, one of them, KL-UCB-H + , being asymptotically optimal with respect to the Bayes risk.

Theorem 1.14 below is a rewriting of Theorem 3 of [Lai, 1987] in the particular case of a product prior. It holds under an extra assumption on the prior distribution: Assumption 1, already used in the previous section. The prior distribution on each arm must be supported on some interval A =]θ -, θ + [, on which the variance of the arms, b(θ), is bounded (see (1.24)). Recall that ]µ -, µ

+ [= ḃ (]θ -, θ + [).
For Gaussian distributions with known variance, one can choose A =]µ -, µ + [= R, whereas for Bernoulli distributions, this assumption is equivalent to considering that all the means belong to an interval of the form ]p, 1 -p[. Theorem 1.14 ( [Lai, 1987], Theorem 3). Let Π 0 be a distribution on A K with a density of the form h(θ 1 ) . . . h(θ K ). If h is such that ∫ A θ q(θ)dθ < ∞ and there exists ρ > 0 such that if h K-1 is the density of the random variable max 1≤i≤K-1 θ i ,

A sup λ∈]θ-ρ;θ]∩A h(λ) × h K-1 (θ)dθ < ∞.
(1.26)

Let H be the cumulative distribution function (c.d.f) associated to h. For any bandit algorithm A, Lai, 1987] also proposes an algorithm whose Bayes risk matches the lower bound of Theorem 1.14. In the sequel, we qualify such algorithms as Bayesian asymptotically optimal (or asymptotically optimal with respect to the Bayes risk). The algorithm proposed shares strong similarities with the KL-UCB algorithm presented in Section 1.2.3. Let d denote the divergence associated to the exponential family: d(x, y) = KL ν ḃ-1 (x) , ν ḃ-1 (y) . One introduce the application d A defined on ( ḃ(Θ)) 2 by, for all y,

lim inf T →∞ BR Π 0 (T, A) log 2 (T ) ≥ C(Π 0 , K) with C(Π 0 , K) = K 2 A K-1 h max 1≤i≤K-1 θ i h(θ 1 ) . . . h(θ K-1 )dθ 1 . . . dθ K-1 = K(K -1) 2 A h 2 (θ)(H(θ)) K-2 dθ [
d A (x, y) = ⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ d(x, y) if x ∈ A d(µ -, y) if x ≤ µ - d(µ + , y) if x ≥ µ + .
After an initialization phase in which each arm is drawn once, the algorithm presented by [Lai, 1987] chooses at time t + 1 the arm maximizing the index

U a (t) = sup q ∈ [µ -, µ + ] ∶ q ≥ S a (t) N a (t) , N a (t)d A S a (t) N a (t) , q ≤ g T N a (t) ,
for some function g satisfying g(t) ∼ log t when t → ∞ and g(t) ≥ log t + ξ log log t for some ξ > -3 2.

Apart from the fact that KL-UCB directly uses the divergence d(x, y) in place of d A (x, y), another difference between these two index policies is that the algorithm proposed by Lai uses an exploration rate log(T N a (t)) in place of the log(t) used by KL-UCB. This alternative exploration rate is also reminiscent of the MOSS algorithm of [Audibert and Bubeck, 2010], which is the index policy associated to

U a (t) = S a (t) N a (t) + log T KNa(t) N a (t)
.

In the same spirit, [START_REF] Garivier | The KL-UCB algorithm for bounded stochastic bandits and beyond[END_REF] propose the use of a variant of KL-UCB, called KL-UCB + , in which the exploration rate log(t) is replaced by log(t N a (t)). Another alternative that has been considered in the literature is the UCB-H algorithm (see [START_REF] Audibert | Exploration-exploitation trade-off using variance estimates in multi-armed bandits[END_REF]) in which the log(t) in UCB1 is replaced by log(T ). Inspired by all these different exploration rates, Definition 1.15 below gathers the corresponding variants of KL-UCB that can be considered, namely KL-UCB + , KL-UCB-H and KL-UCB-H + .

Definition 1.15. The KL-UCB, KL-UCB + , KL-UCB-H and KL-UCB-H + algorithms are index policies that choose at time t + 1 the arm with highest index

u a (t) = sup q > S a (t) N a (t) ∶ d S a (t) N a (t) , q ≤ β(t) ,
where the exploration rate β(t) is given by

Algorithm KL-UCB KL-UCB + KL-UCB-H KL-UCB-H + β(t) = f (t) f (t N a (t)) f (T ) f (T N a (t))
with f (t) = log(t) + c log log(t), for some parameter c.

The algorithm proposed by [Lai, 1987] is very similar to KL-UCB-H + and is asymptotically optimal with respect to the Bayes risk. From (1.23) and (1.25), the indices used by the Finite-Horizon Gittins algorithm can be approximated in some asymptotic regime by indices close to those of KL-UCB-H + . This gives heuristic arguments in favor of the Bayesian asymptotic optimality of FH-Gittins.

Interestingly, to prove the Bayesian asymptotic optimality of his algorithm, [Lai, 1987] starts by showing that it is asymptotically optimal in a frequentist sense, that is with respect to Lai and Robbins' lower bound on the regret (1.10). Then he integrates the asymptotic upper bound obtained over the prior distribution. Quite similarly, one can use the non-asymptotic upper bound on E θ [N a (T )] for KL-UCB given in Theorem 1.8 to obtain the following Bayes-risk bound for KL-UCB. The proof of this new result is given in Section 1.5.3.

Theorem 1.16. Let Π 0 be a product prior distribution satisfying the assumptions of Theorem 1.14. Then the Bayes risk of the KL-UCB satisfies lim sup

T →∞ BR Π 0 (T, KL-UCB) log(T ) 2 ≤ 2C(Π 0 , K),
with C(Π 0 , K) the constant defined in Theorem 1.14.

Theorem 1.16 shows that the KL-UCB algorithm is almost asymptotically optimal with respect to the Bayes risk, up to a multiplicative factor 2. The fact that KL-UCB, which is optimal in a frequentist sense, might not be optimal in a Bayesian sense will be illustrated on numerical experiments below and in Chapter 3. Conversely, KL-UCB-H + is asymptotically optimal in both settings. Its asymptotic optimality with respect to the regret is already established by Lai, and can also be established using elements from our analysis of Bayes-UCB that will be presented in Chapter 2. Indeed, Proposition 2.4 will give the (frequentist) asymptotic optimality of both KL-UCB-H + and KL-UCB + .

The notion of Bayesian asymptotic optimality introduced above, based on Theorem 1.14, might seem not very satisfying, since it applies to prior distributions that can be quite specific. For example, this notion is not defined in a Bernoulli bandit model with a uniform prior on the mean (whereas it is 1.4. NUMERICAL STUDY AND CONCLUSIONS 63 defined if the prior is uniform on [p, 1 -p] for example). Even if the right constant is still to be identified in this context, the Bayes risk of the optimal strategy must also grow as K log(T ) 2 . Indeed, an easy integration (see the precise computation in Section 1. 5.3) shows that any algorithm (like UCB1) that satisfies, for every suboptimal arm a,

E θ [N a (T )] ≤ C 1 (µ * -µ a ) 2 log(T ) + C 2
for some constants C 1 and C 2 , has its Bayes risk under the uniform prior on [0, 1] K upper bounded by

C 1 2 K(log(T )) 2 + C 1 K(log(T )) 3 2 + C 2 + 1 2 K.
(1.27)

The prior-independent worst-case lower bound on the Bayes risk given in (1.7) is therefore pessimistic in the case of Bernoulli bandit with independent prior distributions. Indeed, while there exists a prior distribution such that the Bayes risk is lower bounded by √ KT 20, for Bernoulli bandits with independent, uniform prior on the means, the Bayes-risk is rather of order K log(T ) 2 .

Numerical study and conclusions

In both the frequentist and Bayesian frameworks, state-of-the-art algorithms presented in the previous sections are mostly index policies. Our presentation was focused on exponential bandit models, for which, in the frequentist setting, the KL-UCB algorithm of [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF] is asymptotically optimal with respect to the regret. In the Bayesian framework, with independent arms, the Finite-Horizon Gittins algorithms (inspired by the first index policy proposed by [Gittins, 1979] for discounted rewards) is empirically close to the Bayesian optimal solution, albeit not optimal as in the discounted case. For large horizons, this index policy is however difficult to implement and a variant of KL-UCB, KL-UCB-H + has been shown by [Lai, 1987] to be asymptotically optimal with respect to the Bayes risk.

We start by illustrating numerically the performance of these different index policies in the Bayesian framework, in order to motivate the use of the FH-Gittins algorithm when the horizon is not too large. In Figure 1.5, we compare FH-Gittins with KL-UCB, KL-UCB + and KL-UCB-H + for the horizon T = 500. For each strategy, the cumulated Bayes risk is averaged over N = 5000 bandit games played up to horizon T . We see that FH-Gittins dramatically outperforms KL-UCB-H + , which is supposed to be a good approximation of the optimal strategy only for large values of T . Besides, KL-UCB-H + slightly outperform KL-UCB. While KL-UCB-H + requires the knowledge of the horizon T , KL-UCB + seems to be a good anytime (i.e. that does not use the horizon T ) approximation of this algorithm. We carry out similar experiments, reported in Figure 1.6 and in which the Bayes risk is averaged over N = 10000 bandit games, for an horizon T = 1000. The same trends can be observed, with an increased gap between KL-UCB and KL-UCB-H + .

By running numerical experiments, one can observe differences between the behavior of FH-Gittins and frequentist optimistic algorithms. FH-Gittins seems to explore much less than its frequentist counterpart: it ends up by playing always the same arm. From (1.19), Gittins indices can be seen as upper confidence bounds on the mean, but the 'confidence bonus' shrinks when getting closer to the horizon T . Besides, from (1.20), the FH-Gittins indices of arms that have not been played decrease, giving less incentive to exploration. Conversely, the KL-UCB indices of arms that have not been played increase in order to favor exploration. This property of FH-Gittins can be used in the implementation of the algorithm: if G denotes the value of the FH-Gittins index when a new arm starts to be played, the algorithm One of the objectives of this thesis is to present and analyse new efficient bandit algorithms, based on Bayesian ideas but whose performance is measured with (frequentist) regret. A first idea would be to investigate the performance of the Finite-Horizon Gittins algorithm, that is believed to be almost Bayesian optimal, in a frequentist setting. [START_REF] Ginebra | Small-sample frequentist properties of Bernoulli two-armed bandit Bayesian strategies[END_REF] present a first empirical study for two-armed Bernoulli bandits in which two Bayesian strategies are evaluated with respect to regret, for fixed values of the means: the myopic strategy (that chooses the arm with highest posterior mean) and FH-Gittins (under the name Λ-strategy). The authors show that, for small horizons, there exist values of the means such that the regret of the Bayesian optimal solution is larger than that of these strategies. However, they do not compare FH-Gittins to algorithms that are known to be asymptotically optimal with respect to the regret. In Chapter 2 and Chapter 3, we will propose a numerical comparison of FH-Gittins with KL-UCB, Bayes-UCB and Thompson Sampling on some fixed Bernoulli bandit models. For the fixed bandit models chosen in these experiments, FH-Gittins seems to perform quite well on the (rather) small horizons on which it can be implemented, but its asymptotic optimality with respect to the regret is still to be investigated.

Due to the hardness of implementation of FH-Gittins, we will rather focus in the next two chapters on two Bayesian algorithms that are easy to implement: Bayes-UCB and Thompson Sampling. Moreover, additionally to their good practical performance, we will be able to prove that these Bayesian algorithms are asymptotically optimal with respect to the regret.

Elements of proof

This section gathers the proofs of some important results presented in this chapter.

Changes of distribution: proof of Lemma 1.3

As all the arms in ν = (ν 1 , . . . , ν K ) and ν ′ = (ν ′ 1 , . . . , ν ′ K ) are mutually absolutely continuous, there exists a common measure λ such that for all a, ν a has a density f a with respect to λ and ν ′ a has a density f ′ a with respect to λ. One can introduce the log-likelihood ratio of the observations up to time t under a bandit algorithm algorithm A:

L t ∶= K a=1 t s=1 1 (As=a) log f a (X s ) f ′ a (X s )
.

The key element in a change of distribution is the following classical lemma that relates the probabilities of an event under P ν and P ν ′ through the log-likelihood ratio of the observations. Such a result is often used in the bandit literature for ν and ν ′ that differ just from one arm, for which the expression of the log-likelihood ratio is simpler. In the proof of Theorem 1.2, we indeed use this kind of change of distribution. However, we will consider in Chapter 5 changes of distributions where several arms are modified. A full proof of Lemma 1.17 in this more general setup can be found in the paper [START_REF] Kaufmann | On the Complexity of Best Arm Identification in Multi-Armed Bandit Models[END_REF].

Lemma 1.17. Let σ be any stopping time with respect to F t . For every event A ∈ F σ (i.e. A such that A ∩ (σ = t) ∈ F t ),

P ν ′ (A) = E ν [1 A exp(-L σ )]
Let σ be a stopping time with respect to (F t ). We start by showing that for all A ∈ F σ , P ν (A) = 0 if and only if P ν ′ (A) = 0. Thus, if 0 < P ν (A) < 1 one also has 0 < P ν ′ (A) < 1 and the quantity d(P ν (A), P ν ′ (A)) in Lemma 1.3 is well defined. Let A ∈ F σ . Lemma 1.17 yields

P ν ′ (A) = E ν [1 A exp(-L σ )]. Thus P ν ′ (A) = 0 implies 1 A exp(-L σ ) = 0 P ν -a.s. As P ν (σ < +∞) = 1, P ν (exp(L σ ) > 0) = 1 and P ν ′ (A) = 0 ⇒ P ν (A) = 0. A similar reasoning yields P ν (A) = 0 ⇒ P ν ′ (A) = 0.
Let A ∈ F σ be such that 0 < P ν (A) < 1 (then 0 < P ν ′ (A) < 1). Lemma 1.17 and the conditional Jensen inequality lead to

P ν ′ (A) = E ν [exp(-L σ )1 A ] = E ν [E ν [exp(-L σ ) 1 A ]1 A ] ≥ E ν [exp (-E ν [L σ 1 A ]) 1 A ] = E ν [exp (-E ν [L σ A]) 1 A ] = exp (-E ν [L σ A]) P ν (A),
Writing the same for the event

A yields P ν ′ (A) ≥ exp -E ν [L σ A] P ν (A), hence E ν [L σ A] ≥ log P ν (A) P ν ′ (A) and E ν [L σ A] ≥ log P ν (A) P ν ′ (A)
.

(1.28)

Therefore one can write

E ν [L σ ] = E ν [L σ A]P ν (A) + E ν [L σ A]P ν (A) ≥ P ν (A) log P ν (A) P ν ′ (A) + P ν (A) log P ν (A) P ν ′ (A) = d(P ν (A), P ν ′ (A)). (1.29)
Introducing (Y a,t ), the sequence of i.i.d. samples successively observed from arm a, the log-likelihood ratio L t can be rewritten

L t = K a=1 Na(t) t=1 log f a (Y a,t ) f ′ a (Y a,t )
; and

E ν log f a (Y a,t ) f ′ a (Y a,t ) = KL(ν a , ν ′ a ).
Applying Wald's Lemma (see e.g. [Siegmund, 1985]) to

L σ = ∑ K a=1 ∑ Na(σ) t=1 log fa(Ya,t) f ′ a (Ya,t) yields E ν [L σ ] = K a=1 E ν [N a (σ)]KL(ν a , ν ′ a ).
(1.30)

Combining this equality with inequality (1.29) gives Lemma 1.3.

1.5.2 On Gittins' theorem: proof of Theorem 1.13

Proof of the first statement (Gittins' theorem). Let α ∈]0, 1[ be the discount factor. For any bandit algorithm, one can introduce at round t, for each arm a -the fair charge g a (t) = G α (π t a ) as the Gittins index of arm a at the end round t (the denomination comes from its interpretation as the highest 'price' worth paying to play the arm), -the prevailing charge, g a (t) = min 1≤v≤t g a (v). The proof goes as follows: we upper bound the expected sum of discounted rewards for any bandit algorithm by a quantity that involves the prevailing charge process of each arm, and that does not depend on the algorithm. Then we show that for Gittins' policy, this inequality is actually an equality.

Lemma 1.18. For any bandit algorithm A,

E Π 0 ∞ t=1 α t-1 X t ≤ E Π 0 ∞ t=1 α t-1 g At (t -1)
and this inequality is an equality for Gittins policy.

Proof of Lemma 1.18. The result easily follows by summation if we prove that, for each arm a,

E Π 0 ∞ t=1 α t-1 X a,t 1 (At=a) ≤ E Π 0 ∞ t=1 α t-1 g a (t -1)1 (At=a) .
(1.31)

To show (1.31) holds, we fix an arm a and introduce a sequence of stopping times τ 0 = 0 and

τ k+1 = inf t > τ k g a (t -1) > g a (t) For t ∈ [τ k + 1; τ k+1 ],
the prevailing charge g a (t -1) is equal to the Gittins index g a (τ k ). Conditionally to F τ k , one can consider the calibration problem C ga(τ k ) , in which the optimal policy is either to play arm a until τ k+1 (i.e. as long as its Gittins index is larger that g a (τ k )) or not to play at all and receive reward g a (τ k ) at every time step: thus the expected cumulated reward is g a (τ k ) (1 -α). The inequality we write below follow from the fact that the policy that plays arm a when

(A t = a) and t ∈ [τ k + 1; τ k+1 ],
and receives rewards g a (τ k ) otherwise cannot be better than the optimal policy:

E Π 0 ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ τ k+1 t=τ k +1 α t-τ k -1 X a,t 1 (At=a) + g a (τ k )1 (At≠a) + ∞ t=τ k+1 +1 g a (τ k )α t-τ k -1 F τ k ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ ≤ g a (τ k ) 1 -α E Π 0 ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ τ k+1 t=τ k +1 α t-1 X a,t 1 (At=a) -g a (τ k )1 (At=a) + ∞ t=τ k +1 g a (τ k )α t-1 F τ k ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ ≤ α τ k g a (τ k ) 1 -α E Π 0 ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ τ k+1 t=τ k +1 α t-1 X a,t 1 (At=a) F τ k ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ -E Π 0 ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ τ k+1 t=τ k +1 α t-1 g a (t -1)1 (At=a) F τ k ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ ≤ 0
Summing over k and conditioning gives inequality (1.31). For every arm a, if the bandit algorithm we consider is Gittins strategy, (1.31) is an equality. Indeed, on every interval [τ k + 1, τ k ], if Gittins' policy plays arm a at time τ k + 1, all the Gittins' indices from the other arms are smaller than g a (τ k ), and arm a will therefore be played up to time τ k+1 , which coincides with an optimal policy in the calibration problem C ga(τ k ) .

◻

From Lemma 1.18, one can write, introducing the sequence g(t) = g At (t -1),

E Π 0 ∞ t=1 α t-1 X t ≤ E Π 0 ∞ t=1 α t-1 g(t) ≤ E Π 0 ∞ t=1 α t-1 g * (t) ,
where (g * (t)) is a nonincreasing rearrangement of (g(t)). Whereas the sequence (g(t)) t∈N * depends on the algorithm, as explained below (g * (t)) only depends on the sequences of successive rewards

CHAPTER 1. TWO PROBABILISTIC VIEWS ON REWARDS MAXIMIZATION IN BANDIT MODELS

obtained from each arm, and not on the algorithm itself any more. As Gittins policy is such that the sequence (g(t)) itself is already nonincreasing (since the arm chosen at time t maximizes g a (t)), one has

E Π 0 ∞ t=1 α t-1 X Git. t = E Π 0 ∞ t=1 α t-1 g * (t) ,
which show that Gittins' policy is optimal.

To understand why (g * (t)) does not depend on the policy, we introduce the following notation. For every t, g a (t) only depends on the N a (t) first observations gathered from arm a, Y a,1 , . . . , Y a,Na(t) . In other words, there exists a sequence of deterministic functions

(h s ), with h s ∶ {0, 1} s → R + , such that g a (t) = h Na(t) (Y a,1 , . . . , Y a,Na(t) ).
For each arm a, and each s ∈ N * , we define the stopping time τ a,s as the instant at which the s-th draw of arm a occurs (with the convention that τ a,s = +∞ if arm a has been drawn less than s times). Rearranging the sum, one obtains

∞ t=1 α t-1 g(t) = K a=1 ∞ s=1 α τa,s-1 h s (Y a,1 , . . . , Y a,s-1 ) ∶=ha,s
For each a, (h a,s ) is a nonincreasing sequence that does not depend on the algorithm (A t ). Building a nonincreasing rearrangement of (g(t)) is equivalent to sorting in nonincreasing order the (h a,s ) a=1,...,K,s∈N * . Letting τa,s be the ranking of (a, s) in this sorted list, one has

∞ t=1 α t-1 g(t) = K a=1 ∞ s=1 α τa,s-1 h s (Y a,1 , . . . , Y a,s-1 ) ∶=ha,s ≤ K a=1 ∞ s=1 α τa,s-1 h a,s = ∞ t=1 α t-1 g * (t).
As the sequence of stopping times τa,s only depends on the (h a,s ), and not on (A t ), the values of g * (t) are independent of the algorithm.

Why does this proof not work for Finite-Horizon Gittins indices? It is possible to generalize some of the arguments, by introducing g a (t) = G(π t a , Tt) as the fair charge at time t g a (t) = min 1≤v≤t g a (v) as the prevailing charge at time t, this time with finite-horizon Gittins indices. Referring to the same calibration problem as in the proof of Lemme 1.18 (but this time with a finite horizon), one can show an equivalent result:

E Π 0 T t=1 X t ≤ E Π 0 T t=1 g At (t -1) , (1.32)
with equality for the FH-Gittins policy. However, both the fact the the horizon is finite and that g a (t) depends on Y a,1 , . . . , Y a,Na(t) AND t do not permit to upper bound the quantity in (1.32) by something independent of the policy using some rearrangement argument.

Proof of the second statement. For Bernoulli bandit models with a Beta prior, the Finite-Horizon Gittins indices involved in the algorithm depend on the parameters (a, b) of the current posterior and the remaining time n. They are denotes by G((a, b), n) as in Section 1.3.2.

If the remaining time to play is T = 2, and the prior distribution is Beta(5, 8)⊗Beta (3,5), the explicit formula for Gittins indices available in this case (1.21) gives

G((5, 8), 2) = 5 13 × 1 + 6 14 1 + 5 13 = 0.3968 and G((3, 5), 2) = 3 8 × 1 + 4 9 1 + 3 8 = 0.3939,
thus the FH-Gittins algorithm chooses arm 1 (with highest FH-Gittins index).

With the same prior distribution, the dynamic programming equations (1.17) yield the following result for the optimal value function V * (which depends on (a, b, c, d), the 4-tuple representing the posterior distribution on each arm, and the remaining time n) and as V 1 < V 2 , the optimal policy (with value function V * ) chooses arm 2.

V * ((5, 8, 3, 5), 2) = max {V 1 , V 2 } with V 1 = 5 13 + 5

Proofs of Bayes risk bounds

PROOF OF THEOREM 1.16

The constant C(Π 0 , K) defined in Theorem 1.14 is such that

C(Π 0 , K) = K 2 I K with I K ∶= A K-1 h( max i=2...K θ i )h(θ 2 ) . . . h(θ K )dθ 2 . . . dθ K
The following decomposition of the Bayes risk can be deduced from the regret decomposition (1.1):

BR Π 0 (T, KLUCB) = K a=1 E Π 0 [(µ * -µ a )E θ [N a (T )]] = KE Π 0 [(µ * -µ 1 )E θ [N 1 (T )]] .
The last equality follow from the fact that the prior distribution is invariant under permutations of the arms. To compute the expectation

E Π 0 [(µ * -µ 1 )E θ [N 1 (T )]], we integrate (µ * -µ 1 )E θ [N 1 (T )
] on different regions, on which it can either be upper bounded trivially by (µ * -µ 1 )T or, if arm 1 is suboptimal, by the upper bound given of Theorem 1.8, that be rewritten as a function of the natural parameters in the following form:

E θ [N 1 (T )] ≤ 1 K(θ 1 , θ * ) log(T ) + 2 √ 2π √ M (θ * -θ 1 ) K(θ 1 , θ * ) 3 2 log(T ) + 3 log log(T ) + 4e + 3 K(θ 1 , θ * ) log log(T ) + 8πM (θ * -θ 1 ) 2 K(θ 1 , θ * ) 2 + 6, (1.33)
using additionally that the variance of the arms are bounded by sup b(θ) ≤ M . We recall that K(θ, θ ′ ) = KL(ν θ , ν θ ′ ) where the distributions are parameterized by their natural parameter. We also gather here some useful properties of exponential families. First, one has the two Taylor expansions

K(λ, θ) = b(θ) 2 (θ -λ) 2 + o((θ -λ) 2 ) (1.34) µ(θ) -µ(λ) = b(θ)(θ -λ) + o(θ -λ) (1.35)
Using Taylor-Lagrange formula together with the assumption that on A, m ≤ b(θ) ≤ M also yields the inequalities

µ(θ) -µ(λ) ≤ M θ -λ and K(λ, θ) ≥ m 2 (λ -θ) 2 (1.36)
Decomposition of an integral. Let α T , β T be two nonincreasing sequences of real numbers, to be chosen later, such that 0 < α T < β T and β T tends to zero as T tends to infinity. One has

E Π 0 [(µ * -µ 1 )E θ [N 1 (T )]] = 0<θ * -θ 1 <α T T (µ(θ * ) -µ(θ 1 ))dΠ 0 (θ) (term A) + α T <θ * -θ 1 <β T (µ(θ * ) -µ(θ 1 ))E θ [N 1 (T )]dΠ 0 (θ) (term B) + β T <θ * -θ 1 (µ(θ * ) -µ(θ 1 ))E θ [N 1 (T )]dΠ 0 (θ) (term C) Upper bound on term A. Let θ * 1 = max i≠1 θ i . Term A is upper bounded by M T 0<θ * -θ 1 <α T (θ * -θ 1 )h(θ 1 ) . . . h(θ K )dθ 1 . . . dθ K = M T A K-1 θ * 1 θ * 1 -α T (θ * 1 -θ 1 )h(θ 1 )dθ 1 h(θ 2 ) . . . h(θ K )dθ 2 . . . dθ K ≤ M T α 2 T 2 A K-1 sup λ∈[θ * 1 -α T ,θ * 1 ] h(λ)h(θ 2 ) . . . h(θ K )dθ 2 . . . dθ K ∼ T →∞ M T α 2 T 2 A K-1 h( max i=2...K θ i )h(θ 2 ) . . . h(θ K )dθ 2 . . . dθ K = M T α 2 T 2 I K .
The last equivalent follows from Assumption (1.26) -which allows to apply the dominated convergence theorem.

Upper bound on term B. To upper bound Term B, we use inequality (1.33). It then boils down to controlling the three integrals

I 1 (T ) = α T <θ * -θ 1 <β T µ(θ * ) -µ(θ 1 ) K(θ 1 , θ * ) dΠ 0 (θ), I 2 (T ) = α T <θ * -θ 1 <β T (µ(θ * ) -µ(θ 1 ))(θ * -θ 1 ) K(θ 1 , θ * ) 3 2 dΠ 0 (θ), I 3 (T ) = α T <θ * -θ 1 <β T (µ(θ * ) -µ(θ 1 )) 2 (θ * -θ 1 ) K(θ 1 , θ * ) 2 dΠ 0 (θ).
For I 1 (T ), we use as [Lai, 1987], that the following equivalent holds, uniformly in θ,

µ(θ) -µ(θ 1 ) K(θ 1 , θ) ∼ θ 1 →θ 2 θ -θ 1 .
The equivalent can be obtained using the Taylor expansions (1.34) and (1.35)). Therefore one can write

I 1 (T ) = A K-1 θ * 1 -α T θ * 1 -β T µ(θ * 1 ) -µ(θ 1 ) K(θ 1 , θ * 1 ) h(θ 1 )dθ 1 h(θ 2 ) . . . h(θ K )dθ 2 . . . dθ K ∼ T →∞ A K-1 θ * 1 -α T θ * 1 -β T 2h(θ 1 ) θ * 1 -θ 1 dθ 1 h(θ 2 ) . . . h(θ K )dθ 2 . . . dθ K ≤ A K-1 θ * 1 -α T θ * 1 -β T 2 θ * 1 -θ 1 dθ 1 sup λ∈[θ * 1 -β T ,θ * 1 -α T ] h(λ)h(θ 2 ) . . . h(θ K )dθ 2 . . . dθ K = 2 log β T α T A K-1 sup λ∈[θ * 1 -β T ,θ * 1 -α T ] h(λ)h(θ 2 ) . . . h(θ K )dθ 2 . . . dθ K ∼ T →∞ 2 log β T α T I K
Using inequalities (1.36), one can also show that

I 2 (T ) ≤ 2 2 3 M m 2 3 α T <θ * -θ 1 <β T 1 θ * 1 -θ 1 dΠ 0 (θ) ∼ T →∞ 2 2 3 M m 2 3 log β T α T I K , I 3 (T ) ≤ 4M 2 m 2 α T <θ * -θ 1 <β T 1 θ * 1 -θ 1 dΠ 0 (θ) ∼ T →∞ 4M 2 m 2 log β T α T I K .
These estimations of I 1 (T ), I 2 (T ) and I 3 (T ) together with the upper bound on E θ [N 1 (T )] given by (1.33) leads to

α T <θ * -θ 1 <β T (µ(θ * ) -µ(θ 1 ))E θ [N 1 (T )]dΠ 0 (θ) = 2 log β T α T log(T )I K + 0 log β T α T log(T )
Upper bound on term C. To upper bound Term C, using again inequality (1.33), we would need to upper bound the same three integrals but on a different region:

I 1 (T ) = θ * -θ 1 >β T µ(θ * ) -µ(θ 1 ) K(θ 1 , θ * ) dΠ 0 (θ), I 2 (T ) = θ * -θ 1 >β T (µ(θ * ) -µ(θ 1 ))(θ * -θ 1 ) K(θ 1 , θ * ) 3 2 dΠ 0 (θ), I 3 (T ) = θ * -θ 1 >β T (µ(θ * ) -µ(θ 1 )) 2 (θ * -θ 1 ) K(θ 1 , θ * ) 2 dΠ 0 (θ)
For I 1 (T ), one can write, using inequality (1.36),

I 1 (T ) ≤ M m A K-1 θ * 1 -α T θ * 1 -β T 2h(θ 1 ) θ * 1 -θ 1 dθ 1 h(θ 2 ) . . . h(θ K )dθ 2 . . . dθ K ≤ 2M m 1 β T
Similarly, one obtains

I 2 (T ) ≤ 2 3 2 M m 2 3 1 β T and I 3 (T ) ≤ 4M 2 m 2 1 β T . This shows that C is of order O log(T ) β T .
Choice of α T and β T and conclusion. The following tabular summarizes what we proved above.

term A term B term C O(T α 2 T ) 2 log β T α T log(T )I K + o log β T α T log(T ) O log(T ) β T Choosing α T = 1 √ T log(T ) and β T = 1 log(T )
, term B is the leading term and one obtains

E Π 0 [(µ * -µ a )E θ [N a (T )]] = log(T ) 2 I K + o(log(T ) 2 ). Thus BR Π 0 (T, KLUCB) = KI K log(T ) 2 + o(log(T ) 2 ) = 2C(Π 0 , K) log(T ) 2 + o(log(T ) 2 ),
which concludes the proof.

PROOF IF THE BAYES RISK BOUND (1.27)

Let A be an algorithm that satisfies, on every Bernoulli bandit model parameterized by θ, for every sub-optimal arm a,

E θ [N a (T )] ≤ C 1 (µ * -µ a ) 2 log(T ) + C 2 .
Let Π 0 be the uniform prior on the means. One has

BR Π 0 (T, A) = K a=1 E Π 0 [(µ * -µ a )E θ [N a (T )]]
and for every a, letting µ * a = max i≠a µ i , one can write

E Π 0 [(µ * -µ a )E θ [N a (T )]] = 0<µ * -µa< 1 √ T (µ * -µ a )T dµ 1 . dµ K + 1 √ T <µ * -µa< 1 log(T ) C 1 log(T ) (µ * -µ a ) dµ 1 . dµ K + µ * -µa> 1 log(T ) C 1 log(T ) (µ * -µ a ) dµ 1 . dµ K + C 2 = T [0,1] K-1 ⎛ ⎝ µ * a µ * a -1 √ T (µ * a -µ a )dµ a ⎞ ⎠ dµ 1 . dµ a-1 dµ a+1 . dµ K + [0,1] K-1 ⎛ ⎝ µ * a -1 √ T µ * a - 1 log(T ) C 1 log(T ) µ * a -µ a dµ a ⎞ ⎠ dµ 1 . dµ a-1 dµ a+1 . dµ K +C 1 log(T ) log(T ) + C 2 = T 1 2 1 √ T 2 + C 1 log(T ) log ⎛ ⎝ √ T log(T ) ⎞ ⎠ + C 1 log(T ) log(T ) + C 2 ≤ C 1 2 (log(T )) 2 + C 1 (log(T )) 3 2 + C 2 + 1 2 .

◻

Chapter 2

Bayes-UCB

In this chapter, we introduce and analyse the Bayes-UCB algorithm. We show that this algorithm can be applied in many contexts and shares strong similarities with existing frequentist algorithms. For Bernoulli bandits models, we propose a finite-time analysis proving its asymptotic optimality with respect to Lai and Robbins' lower bound on the regret. Bayes-UCB has been the object of a paper with Olivier Cappé and Aurélien Garivier for the conference AISTATS in 2012 ( [Kaufmann et al., 2012a]) of which the content of this chapter is largely inspired.

Compared to the original paper, the statement of the Bayes-UCB algorithm does no longer depend on the horizon T and the finite-time analysis has been slightly improved: following the last refinements of [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF] for KL-UCB, we are now able to give a fully explicit upper bound on the expected number of draws of each suboptimal arm. Moreover, we point out that our finite-time analysis of Bayes-UCB can be adapted to show that two variants of KL-UCB presented in Chapter 1, KL-UCB + and KL-UCB-H + , are asymptotically optimal.

Introduction

As discussed in the previous chapter, the literature on stochastic multi-armed bandit problems is separated in two distinct approaches. In the frequentist view, the expected mean rewards associated with each arm is considered as unknown deterministic quantity and the goal of the algorithm is to achieve the best parameter-dependent performance. In contrast, in the Bayesian view, each arm is characterized by a parameter which is endowed with a prior distribution. The Bayesian performance is then defined as the average performance over all possible problem instances weighted by the prior on the parameters. Here we argue that algorithms derived from the Bayesian perspective also prove efficient when evaluated using frequentist measures of performance.

Recall that in the classical (frequentist) parametric stochastic multi-armed bandit model, an agent faces K independent arms which depend on unknown parameters θ 1 , . . . , θ K ∈ Θ. The draw of arm a at time t results in a reward X t that is extracted from an i.i.d sequence (X a,t ) t≥1 marginally distributed under ν θa , whose expectation is denoted by µ a . The agent sequentially draws the arms according to a strategy A = (A t ) t≥1 , where A t denotes the arm chosen at round t, based on previous rewards X s = X As,s for 1 ≤ s ≤ t -1. The agent's goal is to find a strategy that maximizes the expected cumulated reward until time T , or equivalently minimizes the cumulated regret

R θ (T, A) = E θ T t=1 µ * -µ At = K a=1 (µ * -µ a )E θ [N a (T )] , (2.1)
where µ * = max{µ a ∶ 1 ≤ a ≤ K} and N a (t) denotes the number of draws of arm a up to time t. [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF], followed by [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF], have provided lower bounds on the number of suboptimal draws under any uniformly efficient strategy: for any arm a such that µ a < µ * , lim inf

T →∞ E θ [N a (T )] log(T ) ≥ 1 inf θ∈Θ∶µ(θ)>µ * KL(ν θ j , ν θ ) . (2.2)
For important classes of distributions, recent contributions have provided finite-time analysis of strategies that are asymptotically optimal in so far that they reach this lower bound. Among them, the KL-UCB algorithm of [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF], that uses confidence intervals based on Kullback Leibler divergence, has been proved optimal in one-parameter exponential family bandit models.

When considering the multi-armed bandit model from a Bayesian point of view, one assumes that the parameter θ = (θ 1 , ..., θ K ) is drawn from a prior distribution Π 0 . More precisely, we assume in the following that the parameters (θ a ) 1≤a≤K are drawn independently from prior distributions (π 0 a ) 1≤a≤K (usually chosen to be all equal), and that conditionally on (θ a ) 1≤a≤K , the sequences (X 1,t ) t≥1 , . . . , (X K,t ) t≥1 are jointly independent and i.i.d. with marginal distributions ν θ 1 , . . . , ν θ K .

In this Bayesian setting, the goal is to maximize E ∑ T t=1 X t , where the expectation is relative to the entire probabilistic model, including the randomization over θ. Bayesian optimality can equivalently be measured considering the Bayes risk, BR Π 0 (T, A) = E[R θ (T, A)], that averages the regret over the parameters. A major appeal of the Bayesian framework is the fact that a strategy with minimal Bayesian regret can be described: it appears as the solution of a planning problem in an associated Markov Decision Process.

To define a Bayesian strategy, let Π t denote the posterior distribution of θ after t rounds of game. Due to our choice of independent priors on (θ a ) 1≤a≤K , Π t is a product distribution which is equivalently defined by the marginal posterior distributions π t 1 , ..., π t K on θ 1 , . . . , θ K , after t rounds. If at round t one chooses arm A t = a and consequently observes X t = X a,t , the Bayesian update for arm a is

π t a (θ a ) ∝ ν θa (X t ) π t-1 a (θ a ) , (2.3)
whereas for i ≠ a, π t i = π t-1 i . A Bayesian algorithm is allowed to exploit the knowledge of the posterior Π t to determine the next action A t+1 .

We already presented in Chapter 1 two Bayesian algorithms: the index policies associated to the Gittins indices and Finite-Horizon Gittins indices. Recall that [Gittins, 1979] considers the infinitehorizon discounted problem in which one tries to maximize E ∑ ∞ t=1 α t X t , where 0 < α < 1 is a real discount parameter. He shows that the index policy associated to the Gittins indices (that depend on α) is optimal with respect to this alternative performance criterion. However, the model reduction argument he uses no longer holds when the horizon T is known and there is no discount. Therefore, the FH-Gittins algorithm does not coincide with the Bayesian optimal policy. This being said, we gave in Chapter 1 some arguments indicating that FH-Gittins should be very close to the Bayesian optimal solution. Here we go further and report in Section 2.4 some experiments on Bernoulli bandits that illustrate our finding that FH-Gittins outperforms its frequentist UCB-like competitors on their own ground, that is, when evaluated using the parameter-dependent (frequentist) regret. [Lai, 1987] shows that a variant of the frequentist optimal KL-UCB algorithm, KL-UCB-H + , is also asymptotically optimal with respect to the Bayes risk (see Theorem 1.14). Conversely, our finding that a (believed) close-to-optimal Bayesian strategy also achieves remarkable parameter-dependent performance for most (all?) value of the parameter θ is currently not supported by mathematical arguments. Furthermore, computing the finite-horizon variant of the Gittins indices is only feasible for moderate horizons due to the need to repeatedly perform (and store the results of) dynamic programming recursions on reduced models. Even for small horizons, the associated computational load and memory footprint are orders of magnitude larger than those of the UCB-like algorithms.

Our objective is thus to propose a generic bandit algorithm, termed Bayes-UCB, that is inspired by the Bayesian interpretation of the problem but retains the simplicity of UCB-like algorithms. Our hope is that this algorithm is simple enough to be effectively implemented and yet is able to reach the asymptotic lower bound of (2.2), including in cases that are currently not handled by UCB-like algorithms. In addition to promising simulation results reported in Section 2.4, we provide several significant elements that support our hopes. First, it is shown in Section 2.2 that instantiating the generic Bayes-UCB algorithm in different specific cases (one-parameter exponential families rewards, Gaussian-armed bandit with unknown means and variances, linear bandits, Gaussian process optimization) yields algorithms that share striking similarities with methods previously proposed in the literature. In the case of Bernoulli rewards, we provide in Section 2.3 a finite-time analysis of the Bayes-UCB algorithm that implies that it reaches the lower bound (2.2). The proof of this result is based on an interesting connection between Bayes-UCB and variants of the KL-UCB algorithm.

The Bayes-UCB algorithm

We start by presenting the rationale for the proposed algorithm before stating it more formally. First, being inspired by the Bayesian modeling of the bandit problem, the Bayes-UCB strategy is a function of the posteriors (π t a ) 1≤a≤K . Due to the nature of our performance measure, the relevant aspect of θ a is the expectation µ a . Hence, denoting by λ t a , for 1 ≤ a ≤ K, the posterior distribution of the mean µ a induced by π t a , the proposed strategy is a function of (λ t a ) 1≤a≤K only.

CHAPTER 2. BAYES-UCB

Algorithm 1 Bayes-UCB Require: Π 0 (initial prior on θ) c (parameters of the quantile)

1: for t = 0 to T -1 do 2:
for each arm a = 1, . . . , K do

3: compute q a (t) = Q 1 - 1 t(log t) c , λ t a (with the convention q a (0) = 1, q a (1) = 1) 4: end for 5:
draw an arm A t+1 ∈ arg max a=1...K q a (t)

6:
get reward X t+1 = X A t+1 ,t+1 and update Π t+1 according to (2.3) 7: end for The use of fixed-level quantiles of (λ t a ) 1≤a≤K as confidence indices appears in [START_REF] Pavlidis | Simulation studies of multi-armed bandits with covariates[END_REF]] as a special case of the Interval Estimation method. To be more specific, denote by Q(t, ρ) the quantile function associated to the distribution ρ, such that P ρ (X ≤ Q(t, ρ)) = t. [START_REF] Pavlidis | Simulation studies of multi-armed bandits with covariates[END_REF] use indices of the form Q(1 -α, λ t a ) for 1 ≤ a ≤ K, with α chosen to be equal to a few percents. In Bayes-UCB, we acknowledge the strong similarity between these posterior indices based on quantiles and the upper confidence bounds used in UCB and its variants: we consider indices of the form Q(1-α t , λ t a ), where α t is of order 1 t. As will be shown in Section 2.3 below for the case of binary rewards, this 1 t rate is deeply connected with the form of the upper confidence bounds used in variants of UCB that are known to reach the bound in (2.2). It is conjectured that no other rate can provide an algorithm that reaches the bound in (2.2) and that, furthermore, choices of the form 1 t β with β < 1 do not even guarantee a finite-time logarithmic control of the regret. As a more pragmatic comment, we also observed in experiments not reported here that, in the case of binary rewards, the empirical performance of the method were superior when using α t ≡ 1 t. We are now ready to state the generic version of the Bayes-UCB algorithm.

In Algorithm 1, the horizon-dependent term (log t) c is an artifact of the theoretical analysis that enables us, for c ≥ 5, to both guarantee finite-time logarithmic regret bounds and achieve asymptotic optimality with respect to (2.2) in Bernoulli bandit models. But in simulations, the choice c = 0 actually proved to be the most satisfying. In cases where the prior Π 0 is chosen to correspond to an improper prior (see, e.g., the Gaussian models below), q a (t) is not defined when t = 1. In those cases it suffices, as is commonly done in most bandit algorithms, to make sure that initially one gathers a sufficient number of observations to guarantee that the posterior Π t indeed becomes proper, for instance by drawing each arm a few times.

As such, Algorithm 1 corresponds to a general principle that does not even require the prior Π 0 to be chosen as a product distribution: Bayes-UCB can still compute one index for each arm, based on its marginal posterior distribution. In this case, the posterior update no longer reduces to (2.3) but is a global update on the joint distribution Π t . In fact, the GP-UCB algorithm for Gaussian processes [START_REF] Srinivas | Gaussian Process Optimization in the Bandit Setting : No Regret and Experimental Design[END_REF] can be seen as a variant of Bayes-UCB in which dependencies, in contrast, are of fundamental importance. This point will be later emphasized in Chapter 4, but here we mostly consider cases where the coordinates of θ are independent. Implementing Algorithm 1 may require additional tools from the Bayesian computational toolbox to perform (or approximate) the Bayesian update of Π t and/or to compute (or, again, approximate) the quantiles q a (t). We first discuss several important models for which Algorithm 1 corresponds to a a procedure that can be implemented exactly without the need to resort to numerical approximation (an example of the opposite situation will be considered in Section 2.4.3 below).

Bayes-UCB for one-parameter exponential family bandits. In the case where the reward distributions belong to a one-parameter canonical exponential family, that is ν θa (x) = A(x) exp(θ a xb(θ a )), with θ a ∈ R, as recalled in Section 1.3.1, it is well known that the prior distribution π 0 a can be chosen to belong to the conjugate family so that the posteriors π t a are all members of the same conjugate family, indexed by their sufficient statistics. As shown in Figure 1.4 of Chapter 1, in many cases of interest, the posterior distribution on the mean (λ t a ) then belong to a well-known parametric family of distribution, for which the quantile q a (t) is easy to compute. For Bernoulli rewards, for instance, using the prior Beta(a, b) for the probability of observing a non-zero reward, we have

π t a = Beta(a + S a (t), b + N a (t) -S a (t))
, where S a (t) = ∑ t i=1 1{A t = j}X t is the sum of rewards gathered up to time t. Likewise, for exponential rewards with a Γ(c, d) prior on the parameter, π t a = Γ(c + N a (t), d + S a (t)) and λ t a = InvΓ(c + N a (t), d + S a (t)). As will be proved below for binary rewards, the Bayes-UCB algorithm in that case is surprisingly related to the KL-UCB algorithm.

Bayes-UCB for Gaussian bandits with unknown means and variances. In general exponential family models, the Bayesian update is usually still computable explicitly (at least when using conjugate priors) but the relationship between the parameter θ a and the expectation µ a is less direct. A significant case where Bayes-UCB corresponds to a simple and efficient algorithm is when the rewards are assumed to be Gaussian, with both unknown mean µ a and unknown variance σ 2 a . For simplicity, we consider improper non-informative priors on each arm, that is, π 0 a (µ a , σ a ) = 1 σ 2 a . It is well known that the marginal posterior distribution of µ a at time t is then such that

µ a -S a (t) N a (t) S (2) a (t) N a (t) X 1 , ..., X t ∼ T (N a (t) -1) ,
where

S (2) a (t) = ∑ t s=1 1{A s = a}X 2 s -S 2 a (t) N a (t) N a (t) -1 ,
and T (k) denote the Student-t distribution with k degrees of freedom. Therefore Bayes-UCB is the index policy associated to upper confidence bound

q a (t) = S a (t) N a (t) + S (2)
a (t) N a (t) Q 1 - 1 t , T (N a (t) -1) ,
omitting the (log t) c factor for clarity. The Bayes-UCB index above is related to the index used in the UCB1-norm algorithm of [Auer et al., 2002a], where the quantile is replaced by 16 log(t -1), which is obtained as an upper bound of Q(1 -1 t 4 , T (N a (t) -1)). The practical performances of these two variants (Bayes-UCB and UCB1-norm) will be illustrated in Section 2.4 below.

Bayes-UCB for linear bandits. We end this section with the more elaborate case of linear bandits in which the arms can be numerous but share a strong common structure. We will consider the case of Gaussian rewards with a multivariate Gaussian prior for the parameter θ ∈ R d that defines the model. The arms are fixed vectors U 1 , ..., U K ∈ R d . In this model, the choice of arm A t = a at time t results in the reward y t = U T a θ + σ 2 t , where t is some centered noise and M T denotes the transpose of a matrix (or vector) M . In the sequel, we assume that t ∼ N (0, 1). Following [START_REF] Rusmevichientong | Linearly Parameterized Bandits[END_REF], our goal is to find algorithms A that minimize the frequentist regret

R θ (T, A) = E θ T t=1 max 1≤a≤K (U T a θ) -U T At θ .
Denoting by Y t = [y 1 , ..., y t ] T ∈ R d the vector of rewards and

X t = [U A 1 ...U At ] T ∈ M t,d (
R) the design matrix, the model rewrites:

Y t = X t θ + σ 2 E t , where E t ∼ N (0, σ 2 Id t ) .
The Bayesian modeling here consists in a Gaussian N 0, κ 2 Id d prior on θ, assuming the noise parameter σ 2 to be known. The posterior is

θ X t , Y t ∼ N (M t , Σ t ) , where M t = (X T t X t + (σ κ) 2 I d ) -1 X T t Y t and Σ t = σ 2 (X T t X t + (σ κ) 2 I d ) -1 . The posterior distribution λ t a on µ a = U ′ a θ is therefore N (U T a M t , U T a Σ t U a )
. Hence, using the notation where x A ∶= √ x T Ax, Bayes-UCB selects an arm that maximizes the index:

q a (t) = U T a M t + U a Σt Q 1 - 1 t , N (0, 1) .
[ [START_REF] Dani | Stochastic Linear Optimization under Bandit Feedback[END_REF], Rusmevichientong and Tsitsiklis, 2010, Abbasi-Yadkori et al., 2011] propose an optimistic approach for this problem based on a confidence ellipsoid located around the least-square estimate θt . This method is equivalent to choosing arm a such that U a θt + ρ(t) U a (X T t Xt) -1 is maximal. For an improper prior (κ = ∞), we have M t = θt and Σ t = σ 2 (X T t X t ) -1 . Thus, this approach can again be interpreted as a particular case of Bayes-UCB. In Section 2.4.3, we consider the case where θ is a sparse vector. It is not obvious how to design an UCB algorithm for this case. Yet, we show that Bayes-UCB can be implemented, using for example Gibbs sampling.

In Chapter 4, we will consider contextual linear bandits, a more general framework that encompasses the (static) linear bandit introduced here. We will provide some theoretical guarantees for Bayes-UCB in this more general framework.

Analysis of the Bayes-UCB algorithm for binary rewards

In this section, we focus on the case where the rewards have a Bernoulli distribution, and when the prior distribution on each arm is the Beta(1, 1); that is the uniform distribution U([0, 1]). In this case, recall the Bayes-UCB algorithm is the index policy associated to

q a (t) = Q 1 - 1 t(log(t)) c ; Beta(S a (t) + 1, N a (t) -S a (t) + 1 .
We show below that the Bayes-UCB algorithm is optimal, in the sense that it reaches the lower-bound (2.2) of Lai and Robbins.

ANALYSIS OF THE BAYES-UCB ALGORITHM FOR BINARY REWARDS 79

Asymptotic optimality and links with frequentist algorithms

Theorem 2.1. The Bayes-UCB algorithm with a uniform prior on the means and with the parameter c = 5 satisfies, for every > 0 and for T such that

log T + 5 log log T ≥ d(µ 2 , µ 1 ) 1 + exp 8 (µ 1 (1 -µ 1 )) 2 (1 + ) 2 2 d(µ 2 , µ 1 ) 2 , E θ [N a (T )] ≤ 1 + d(µ a , µ 1 ) log(T ) + log T + 5 log log T 2π(1 + ) 3 d ′ (µ a , µ 1 ) 2 d(µ a , µ 1 ) 3 + 1 + d(µ a , µ 1 ) + 2e + 3 1 -µ 1 log log T + 27 + 2(1 + ) 2 d ′ (µ a , µ 1 ) d(µ a , µ 1 ) 2 .
The analysis relies on the following tight bounds we have on the index q a (t).

Lemma 2.2. Denoting by d(x, y) the KL divergence between Bernoulli distributions with parameters x and y, the posterior quantile q a (t) used by the Bayes-UCB algorithm with parameter c = 5 satisfies

ũa (t) ≤ q a (t) ≤ u a (t) ,
where

u a (t) = argmax x> Sa(t) Na(t) d S a (t) N a (t) , x ≤ log(t) + 5 log(log(t)) N a (t) , ũa (t) = argmax x> Sa(t) Na(t)+1 ⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ d S a (t) N a (t) + 1 , x ≤ log t Na(t)+2 + 5 log(log(t)) (N a (t) + 1) ⎫ ⎪ ⎪ ⎪ ⎬ ⎪ ⎪ ⎪ ⎭ .
Surprisingly, the Bayesian quantiles match the upper confidence bounds based on Kullback-Leibler divergence used by variants of the KL-UCB algorithm of [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF]. The quantile q a (t) is exactly upper bounded by the KL-UCB index u a (t) using the exploration rate f (t) = log t + 5 log log t, whereas it is upper bounded by a biased version of the index used by the KL-UCB + variant, that uses f (t N a (t)) as an exploration rate (see Definition 1.15 in Chapter 1).

Remark 2.3. If Bayes-UCB were defined more generally depending on some exploration function f (t) as the algorithm choosing at time t + 1 the arm maximizing the index

q a (t) = Q 1 -e -f (t) , π t a ,
then following the proof of Lemma 2.2, one would have q a (t) ≤ u a (t) were u a (t) is the KL-UCB index with exploration function f (t), i.e.

u a (t) = sup{q ≥ μa (t) ∶ N a (t)d(μ a (t), q) ≤ f (t)}.
Thanks to this link with the KL-UCB indices, the finite-time analysis we provide share similarities with that of [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF]. The main difficulty is to deal with the bias and the alternative exploration rate that appear in the indices ũa (t), since no finite-time analysis of KL-UCB + existed in the literature. Following the same lines as our analysis of Bayes-UCB, it is now possible to give a finite-time analysis for the two variants KL-UCB + and KL-UCB-H + , leading to the following result.

Proposition 2.4. For bandits with rewards in an exponential family with associated divergence d(x, y) = KL(ν ḃ-1 (x) , ν ḃ-1 (y) ), the following instances of KL-UCB + and KL-UCB-H + , defined as the index policies respectively associated to the index u + a (t) and u H,+ a (t) given by

u + a (t) = sup q ≥ μa (t) ∶ N a (t)d (μ a (t), q) ≤ log t N a (t) + 5 log log t N a (t) u H,+ a (t) = sup q ≥ μa (t) ∶ N a (t)d (μ a (t), q) ≤ log T N a (t) + 5 log log T N a (t)
are asymptotically optimal with respect to Lai and Robbins' lower bound.

Bayes-UCB beyond Bernoulli distributions

Albeit being designed for Bernoulli distributions, as explained by [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF], the KL-UCB algorithm using the divergence associated to Bernoulli rewards can also be used for rewards distributions bounded in [0, 1], without any modification and with the same theoretical guarantees.

For Bayes-UCB, a slight modification of the original algorithm, introduced by [START_REF] Agrawal | Analysis of Thompson Sampling for the multi-armed bandit problem[END_REF] for Thompson Sampling (an other Bayesian algorithm studied in Chapter 3), yields a provably efficient algorithm for bandit models with bounded rewards. A bounded bandit model with means µ 1 , . . . , µ K can be 'transformed' into a Bernoulli bandit model with same means by introducing for each arm the reward process Xa,t = (U a,t ≤ X a,t ) where (X a,t ) is the original reward process of arm a and (U a,t ) t∈N * is an independent sequence of i.i.d. random variables uniform on [0, 1]. The modified algorithm proceeds as follows: if arm a is drawn at time t and reward X t = X a,t is observed, a transformed binary reward Xt = (U t ≤ X t ) is collected, where the (U t ) are i.i.d. uniform random variables. Letting Sa (t) = ∑ t s=1 1 (As=a) Xs , the arm chosen at time t + 1 maximizes the index

qa (t) = Q 1 - 1 t(log t) 5 ; Beta Sa (t) + 1, N a (t) -Sa (t) + 1 .
(2.4) Theorem 2.1 also holds for this algorithm, whereas there is no provable guarantee if the original Bayes-UCB algorithm designed for Bernoulli rewards is applied to bounded rewards. Indeed, the sum of 'true' rewards S a (t) is no longer an integer, and Lemma 2.2 on which our analysis is based would no longer hold (one can only control the tail -and thus the quantiles-of Beta distributions with integer coefficients).

The asymptotic optimality of Bayes-UCB is established only for Bernoulli distributions, but we believe this algorithm is also asymptotically optimal for other distributions in an exponential family. For Gaussian distributions with known variance σ 2 , Bayes-UCB with a normal (or improper) prior on the mean can also be proved to be asymptotically optimal, since tight bounds can be obtained for the quantiles of the (Gaussian) posterior distribution (using for example Theorem 1.2.3. of [Durrett, 2010]). For Thompson Sampling, we also propose a finite-time analysis for Bernoulli rewards, and extend it to rewards in an exponential family. This extension notably relies on an upper bound on the tail of each posterior distribution (see Lemma 3.11). Such a result could be used too in an analysis of Bayes-UCB, but one would also need either a lower bound of the tail of the posterior, or (as in the analysis of Thompson Sampling) a result showing separately that the optimal arm has to be drawn a lot. We leave the generalization of Bayes-UCB to exponential families as future work.

We now give the proof of Lemma 2.2 and Theorem 2.1, that hold for Bernoulli bandit models.

Finite-time analysis

We start by giving a sketch of the proof of Lemma 2.2. This proof is detailed in Section 2.5. Lemma 2.2 relies on two ingredients. The first is a connection between Beta and Binomial distributions: for any integers a, b, the distribution Beta(a, b) is the law of the a-th order statistic among a + b -1 uniform random variables, so that

P(X ≥ x) = P(S a+b-1,x ≤ a -1) = P(S a+b-1,1-x ≥ b) ,
where S n,x denotes a binomial distribution with parameters n and x. Bounding the beta quantiles then boils down to controlling the binomial tails, which is achieved by our second ingredient, Lemma 2.5 below. The upper bound in inequality (2.5) is easily obtained using Chernoff inequality, whereas the lower bound follows from a careful application of the method of types (adapting Lemma 2.1.9. of [START_REF] Dembo | Large Deviations Techniques and Applications[END_REF]).

Lemma 2.5. If S n,x is binomial with parameters n and x and k is an integer such that k ≥ nx,

e -nd k n ,x n + 1 ≤ P(S n,x ≥ k) ≤ e -nd k n ,x . (2.5)
We now prove Theorem 2.1, the proof of intermediate lemmas being postponed to Section 2.5. To ease the notation, we denote by P and E (in place of P θ and E θ ) the probability and expectation for a fixed Bernoulli bandit model parameterized by θ = (µ 1 , . . . , µ K ) . We recall some useful notations, already defined in the previous chapter: μa (t) = S a (t) N a (t) is the empirical mean of rewards obtained from arm a up to the end of round t (with the convention that it is set to zero when N a (t) = 0). (Y a,k ) is the sequence of successive rewards obtained from arm a. It is i.i.d. with Bernoulli distribution of mean µ a and we let μa,s = 1 s ∑ s k=1 Y a,k .

Proof of Theorem 2.1. Assume without loss of generality that arm 1 is optimal and let a ≠ 1 be a sub-optimal arm. Following the classic regret analysis for index policies discussed in Section 1.2.3 of Chapter 1, one would use the following decomposition of the event (A t+1 = a):

(A t+1 = a) ⊆ (µ 1 ≥ q 1 (t)) ⋃ (µ 1 ≤ q 1 (t), A t+1 = a) ⊆ (µ 1 ≥ q 1 (t)) ⋃ (µ 1 ≤ q a (t), A t+1 = a) ,
using that if a is drawn at time t+1, q a (t) > q 1 (t). However, for the purpose of our analysis, we introduce here a slightly different decomposition, that depends on the sequence

g t = 2 log(t)
.

Using the results and notation of Lemma 2.2, we write

(A t+1 = a) ⊆ (µ 1 -g t ≥ q 1 (t)) ⋃ (µ 1 -g t ≤ q a (t), A t+1 = a) ⊆ (µ 1 -g t ≥ ũ1 (t)) ⋃ (µ 1 -g t ≤ u a (t), A t+1 = a) . (2.6)
We analyse the Bayes-UCB algorithm with an initialization phase that draws each arm once, although no initialization is required in principle. This assumption is needed to make sure that when t ≥ K, all the indices u a (t), ũa (t) (that do depend on N a (t), unlike the quantile q a (t)), are well defined. One has

E[N a (T )] = 1 + E T -1 t=K 1 (A t+1 =a) ,

82

CHAPTER 2. BAYES-UCB which yields, using the decomposition (2.6),

E[N a (T )] ≤ 1 + T -1 t=K P (µ 1 -g t ≥ ũ1 (t)) A + T -1 t=K P (µ 1 -g t ≤ u a (t), A t+1 = a) B .
Term A will be small (o(log T )) since arm 1 is not likely to be 'under-estimated' by its index ũ1 (t) at each round. To show this, we have to adapt the proof of the self-normalized informational inequality introduced by [START_REF] Garivier | The KL-UCB algorithm for bounded stochastic bandits and beyond[END_REF], stated as Lemma 1.9 in the previous chapter, to the alternative exploration rate and the bias of index ũ1 (t). This is done in Lemma 2.6, and the reason why the extra term g t is needed to handle the alternative exploration rate appears in its proof.

Lemma 2.6.

T -1 t=K P (µ 1 -g t ≥ ũ1 (t)) ≤ 2e + 3 1 -µ 1 log log T + 26
To upper bound term B, we start by using that g Na(t) ≥ g t and x ↦ d + (μ a (t), x) is non-decreasing, where

d + (x, y) = d(x, y)1 (x<y) : T -1 t=K P (µ 1 -g t ≤ u a (t), A t+1 = a) ≤ T -1 t=K P (A t+1 = a, N a (t)d + (μ a (t), µ 1 -g t ) ≤ f (T )) ≤ T -1 t=K P A t+1 = a, N a (t)d + (μ a (t), µ 1 -g Na(t) ) ≤ f (T )
Summing over the possible values of N a (t) and interverting the sums to get rid of the self-normalized quantities (a technique already described in Lemma 1.10 in Chapter 1), yields

(B) ≤ T s=1 P (sd + (μ a,s , µ 1 -g s ) ≤ f (T )) .
Lemma 2.7 below permits to conclude the proof. A similar term is bounded in Appendix A.2. of [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF]: we adapt their proof to the presence of the extra term g s .

Lemma 2.7. Let N a ( ) = d(µ a , µ 1 ) 1 + exp 8 (µ 1 (1 -µ 1 )) 2 (1 + ) 2 2 d(µ a , µ 1 ) 2
For T such that f (T ) ≥ N a ( ), one has

T s=1 P (sd + (μ a,s , µ 1 -g s ) ≤ f (T )) ≤ (1 + ) f (T ) d(µ a , µ 1 ) + f (T ) 2π(1 + ) 3 d ′ (µ a , µ 1 ) 2 d(µ a , µ 1 ) 3 + 2(1 + ) 2 d ′ (µ a , µ 1 ) d(µ a , µ 1 ) 2 . ◻ 2

Numerical experiments 2.4.1 Binary bandits

Numerical experiments have been carried out in a frequentist setting for bandits with Bernoulli rewards: for a fixed parameter θ and an horizon T , N bandit games with Bernoulli rewards are repeated for a given strategy. The main purpose of these numerical experiments is to compare the performance of Bayes-UCB in terms of cumulated regret with those of UCB and KL-UCB. These are presented in Figure 2.1, where the regret is averaged over N = 5000 simulations for two different two-armed bandit problems with horizon T = 500. We also include in the comparison the Bayesian algorithm based on Finite-Horizon Gittins indices (FH-Gittins). Whereas the performance of FH-Gittins are more striking in the left situation (0.1/0.2) than in the right one (0.45/0.55), Bayes-UCB improves equally over KL-UCB in all scenarios.

The horizon T had to be chosen quite small because of the numerical complexity of the FH-Gittins algorithm. In the next chapter, we will display numerical results in the Bernoulli case for larger horizons, also including Thompson Sampling.

Gaussian rewards with unknown means and variances

For the bandit problem with Gaussian rewards with unknown mean and variance, few algorithms have been proposed. We compare Bayes-UCB with UCB1-norm and UCB-Tuned, two algorithms introduced by [Auer et al., 2002a]. UCB1-norm is designed for Gaussian rewards and a logarithmic upper bound on its regret is given, whereas UCB-Tuned relies on estimates of the variance of each arm and can be used for general distributions, but without any theoretical guarantee. Figure 2.2 presents the regret in a 4-arms problem, on a horizon T = 10000, averaged over N = 1000 simulations. UCB-Tuned seems unadapted to the problem, whereas UCB1-norm and Bayes-UCB achieve a regret proving that the asymptotic lower bound of Burnetas & Katehakis is pessimistic for such short horizons (see also [START_REF] Garivier | The KL-UCB algorithm for bounded stochastic bandits and beyond[END_REF]). Bayes-UCB outperforms UCB1-norm, mostly because of the more appropriate choice of a quantile of order 1 -1 t.

Sparse linear bandits

The linear bandit model presented in Section 2.2 relies on linear regression. Many recent works have highlighted the importance of sparsity issues in this context. We show that Bayes-UCB can address sparse linear bandit problems by using a prior that encourages sparsity of the parameter θ. This 'spikeand-slab' prior is defined as follows: the coordinates of θ are independent, with distribution

θ a ∼ δ 0 + (1 -)N (0, κ 2 ) .
Let C be the random vector in R d indicating the non-zero coordinates of θ: C a = 1 (θa≠0) . If J denotes a set of indices, let X t,J ∈ M t, J (R) be the submatrix of the design matrix X t with columns in J only and θ J ∈ R J the subvector with coordinates in J.

Given C and Y t , denote by J 1 the set of non-zero coordinates in C. The subvector θ J 1 is the solution of a Bayesian regression problem with prior N (0, κ 2 I J 1 ), hence

θ J 1 C, Y t ∼ N (X ′ t,J 1 X t,J 1 + (σ κ) 2 I J 1 ) -1 X ′ t,J 1 Y t ; σ 2 (X ′ t,J 1 X t,J 1 + (σ κ) 2 I J 1 ) -1 . The marginal distribution of C given Y is P (C Y ) ∝ J 0 (1 -) J 1 N Y t 0, κ 2 X t,J 1 X ′ t,J 1 + σ 2 I t .
The normalization term involves a sum over 2 d possible configurations of C. When d is small, the exact Bayes-UCB indices can be computed, as the dot-product U ′ a θ follows a mixture of Gaussian distributions. For higher dimensions, one can use Gibbs sampling to sample from C Y , and produce samples from θ Y that lead to approximated values of q a (t).

Numerical simulation have been carried out for a sparse problem in dimension d = 10 where θ only has two non-zero coordinates. In Figure 2.3 we compare the regret of Bayes-UCB for three different priors: the general multivariate Gaussian prior discussed in Section 2.2, an oracle Gaussian prior on the first two coordinates only (meaning that the sparsity pattern is known) and Bayes-UCB with a sparse prior. The 20 arms of the problem are chosen randomly on the unit sphere and the regret is averaged over N = 100 simulations for an horizon T = 1000. As expected, the use of a sparsity-inducing prior in this case results in an algorithm with greatly enhanced performance. Such experiments should now be carried out in larger dimension using more sophisticated MCMC algorithms, designed for sampling from sparsity-inducing priors in a regression model, like the STMALA algorithm of [START_REF] Schreck | A shrinkage-thresholding Metropolis adjusted Langevin algorithm for Bayesian variable selection[END_REF]. If X ∼ Beta(a, b), using Lemma 2.5 and the link between the tail of Beta and binomial distribution mentioned above, we get, for x > a-1 a+b-1 ,

e -(a+b-1)d( a-1 a+b-1 ,x) a + b ≤ P(X ≥ x) ≤ e -(a+b-1)d( a-1 a+b-1 ,x)
Let q 1-γ = Q(1 -γ, Beta(a, b)). Since :

(a + b -1)d a -1 a + b -1 , x ≥ log(1 γ) ⇒ x ≥ q 1-γ
we have that :

x * + = argmin x> a-1 a+b-1 {(a + b -1)d a -1 a + b -1 , x ≥ log(1 γ)} = argmax x> a-1 a+b-1 {(a + b -1)d a -1 a + b -1 , x ≤ log(1 γ)}
is still an upper bound for the quantile q 1-γ . The same reasoning shows q 1-γ is lower-bounded by

x * -= argmax x> a-1 a+b-1 (a + b -1)d a -1 a + b -1 , x ≤ log 1 γ(a + b)
Moreover we can easily show that

x * + ≤ argmax x> a-1 a+b-2 {(a + b -2)d a -1 a + b -2 , x ≤ log(1 γ)}
using mainly the fact that y ↦ d(y, x) is decreasing for y < x.We get the final result using a = S a (t) + 1,b = N a (t) -S a (t) + 1 and γ = 1 (t log(t) c ).

CHAPTER 2. BAYES-UCB

Proof of Lemma 2.6

To upper bound (A) = ∑ T -1 t=K P (µ 1 -g t ≥ ũ1 (t)) we start by splitting the sum according to the number of draws of the optimal arm.

(A) ≤ T -1 t=K P µ 1 -g t > ũ1 (t) , N 1 (t) + 2 ≤ log 2 (t) A 1 + T -1 t=K P µ 1 -g t > ũ1 (t) , N 1 (t) + 2 ≥ log 2 (t) A 2 .
To upper bound term A 1 , we use that log

t Na(t)+2 in ũ1 (t) is lower-bounded by log t log(t) 2 : µ 1 -g t > ũ1 (t), N 1 (t) + 2 ≤ (log t) 2 = (N 1 (t) + 1)d + S 1 (t) N 1 (t) + 1 , µ 1 -g t ≥ log t N 1 (t) + 2 + 5 log log t, N 1 (t) + 2 ≤ (log t) 2 ⊆ (N 1 (t) + 1)d + S 1 (t) N 1 (t) + 1 , µ 1 -g t ≥ log t log 2 t + 5 log(log t) ⊆ (N 1 (t) + 1)d + S 1 (t) N 1 (t) + 1 , µ 1 ≥ log t + 3 log(log t) .
Hence,

(A 1 ) ≤ T -1 t=K P ∃s ∈ {1, . . . t} ∶ (s + 1)d + S 1,s s + 1 , µ 1 ≥ log(t) + 3 log log(t) ,
where S 1,s = ∑ s k=1 Y 1,k is the sum of the s first rewards from arm 1. An adaptation of the proof of Lemma 1.9 yields the following self-normalized inequality, whose proof is given below. Lemma 2.8.

P ∃s ∈ {1, . . . , t} ∶ (s + 1)d + S 1,s s + 1 , µ 1 ≥ δ ≤ 1 1 -µ 1 (δ log(t) + 1) exp(-δ + 1).
Lemma 2.8 leads to the upper-bound

(A 1 ) ≤ e 1 -µ 1 T -1 t=K log 2 (t) + 3 log(t) log(log(t)) + 1 t(log(t)) 3 ≤ e 1 -µ 1 2 + 3 e T -1 t=K 1 t log(t) ≤ 2e + 3 1 -µ 1 log log T.
On the events involved in term A 2 , the optimal arm has been sufficiently drawn to be well estimated, so we use that

(µ 1 -g t > ũ1 (t)) ⊆ µ 1 -g t > S 1 (t) N 1 (t) + 1 ⊆ µ 1 -g t ≥ S 1 (t) N 1 (t) - 1 N 1 (t) + 1 .
Without the term g t , the probability of the above event wouldn't be small, but here one can write

(A 2 ) ≤ T -1 t=K P µ 1 -g t ≥ S 1 (t) N 1 (t) - 1 N 1 (t) + 1 ∩ N 1 (t) > log(t) 2 -2 ≤ 5 + T -1 t=6 P ∃s ∈ [log(t) 2 -2; t] ∶ µ 1 -g t ≥ ∑ s r=1 Y 1,r s - 1 s + 1 ≤ 5 + T -1 t=6 P ∃s ∈ [log(t) 2 -2; t] ∶ s r=1 (µ 1 -Y 1,r ) ≥ g t s -1
We start the sum at t = 6 since when t ≥ 6 one has simultaneously log(t) 2 -2 ≥ 1 and ∀s ≥ log(t) 2 -2, g t s -1 ≥ 0. For each t, noting t ′ = log(t) 2 -2, we use a peeling and split the interval [t ′ ; t] in smaller intervals of the form [2 k t ′ ; 2 k+1 t ′ [ on which we will use a maximal inequality.

P ∃s ∈ [t ′ ; t] ∶ s r=1 (µ 1 -Y 1,r ) ≥ g t s -1 = log(t t ′ ) log(2) k=0 P ∃s ∈ [2 k t ′ ; 2 k+1 t ′ [ ∶ s r=1 (µ 1 -Y 1,r ) ≥ g t s -1 ≤ ∞ k=0 P ∃s ∈ [2 k t ′ ; 2 k+1 t ′ [ ∶ s r=1 (µ 1 -Y 1,r ) ≥ g t 2 k t ′ -1 ≤ ∞ k=0 P ∃s ∈ [1; 2 k+1 t ′ [ ∶ s r=1 (µ 1 -Y 1,r ) ≥ g t 2 k t ′ -1 ≤ ∞ k=0 exp ⎛ ⎝ - 2 g t 2 k t ′ -1 2 2 k+1 t ′ ⎞ ⎠ ≤ ∞ k=0 exp ⎛ ⎝ -2 k (g t t ′ -1 2 k ) 2 t ′ ⎞ ⎠ ≤ ∞ k=0 e - g t t ′ -1 2 t ′ (k+1) ,
where we use that 2 k ≥ 1 + k. And, with the expressions g t and t ′ ,

∞ k=0 e - g t t ′ -1 2 t ′ (k+1) = 1 e (g t t ′ -1) 2 t ′ -1 = 1 exp ⎛ ⎝ 2 log 2 t-1 √ 2 √ log t-2 2 log 3 t-2 log t ⎞ ⎠ -1 ∼ t→∞ 1 t 2
This function of t is the general term of a convergent series. A numerical evaluation of the sum of this series finally yields (A 2 ) ≤ 26, which concludes the proof.

◻

Proof of Lemma 2.8. As in the proof of Lemma 1.9 (given in Appendix A.2), we start with a peeling argument, but on the values of s + 1. Let γ > 1.

P ∃s ∈ {1, . . . , t} ∶ (s + 1)d + S 1,s s + 1 , µ 1 ≥ δ (2.7) ≤ ⌈ log(t) log(γ) ⌉ k=1 P γ k-1 ≤ s + 1 < γ k , (s + 1)d + S 1,s s + 1 , µ 1 ≥ δ ≤ ⌈ log t log(γ) ⌉ k=1 P(E k ) with E k ∶= γ k-1 ≤ s + 1 < γ k , d + S 1,s s + 1 , µ 1 ≥ δ γ k 88 CHAPTER 2. BAYES-UCB If k is such that δ γ k ≥ d(0, µ 1 ) clearly P(E k ) = 0. Otherwise, one can introduce a unique z k such that S 1,s s + 1 < z k < µ 1 and d(z k , µ 1 ) = δ γ k .
As Bernoulli distribution belong to an exponential family, we have from Lemma 1.4 that

d(z k , µ 1 ) = max λ∈R {λz k -φ µ 1 (λ)} where φ µ 1 (λ) = E Y ∼B(µ 1 ) [e λY ].
The maximum is obtained for

λ k = log z k 1-z k -log µ 1 1-µ 1 . Thus there ex- ists λ k < 0 such that d(z k , µ 1 ) = λ k z k -φ µ 1 (λ k ) and one can check that φ µ 1 (λ k ) = log 1-µ 1 1-z k . Therefore, E k ⊂ γ k-1 ≤ s + 1, S 1,s s + 1 ≤ z k , λ k z k -φ µ 1 (λ k ) = δ γ k ⊂ S 1,s s + 1 ≤ z k , λ k z k -φ µ 1 (λ k ) ≥ δ γ(s + 1) ⊂ λ k S 1,s s + 1 -φ µ 1 (λ k ) ≥ δ γ(s + 1) ⊂ λ k S 1,s -(s + 1)φ µ 1 (λ k ) ≥ δ γ For every λ ∈ R, W λ s = exp (λS 1,s -(s + 1)φ µ 1 (λ)
) is a martingale. Thus,

P(E k ) ≤P W λ k t ≥ exp( δ γ ) ≤ Markov e -δ γ E W λ k t ≤ super-martingale e -δ γ E[W λ k 0 ] = e -δ γ e -φµ 1 (λ k ) = 1 -z k 1 -µ 1 e -δ γ ≤ 1 1 -µ 1 e -δ γ .
Summing over k and letting γ = δ δ-1 > 1, (2.7) is upper bounded by

1 1 -µ 1 log(t) log (δ (δ -1)) + 1 exp(-δ + 1) ≤ 1 1 -µ 1 (δ log(t) + 1) e -δ+1 ,
which concludes the proof.

Proof of Lemma 2.7

The quantity to be upper bounded is

(B) ′ ∶= T s=1 P (sd + (μ a,s , µ 1 -g s ) ≤ f (T )) .
The function g(q) = d + (μ a,s , q) is convex and differentiable and g ′ (q) = q-μa,s q(1-q) 1 (μa,s≤q) , thus

d + (μ a,s , µ 1 -g s ) ≥ d + (μ a,s , µ 1 ) -g s µ 1 -μa,s µ 1 (1 -µ 1 ) ≥ d + (μ a,s , µ 1 ) -g s 2 µ 1 (1 -µ 1 )
.

And

(B) ′ ≤ T s=1 P d + (μ a,s , µ 1 ) ≤ f (T ) s + 2g s µ 1 (1 -µ 1 )
.

Let > 0. Introducing K T ∶= ⌈ (1 + )f (T ) d(µ a , µ 1 ) ⌉
we can split the sum:

(B) ′ ≤ K T + T s=K T +1 P d + (μ a,s , µ 1 ) ≤ f (T ) s + b T , with b T = 2g K T µ 1 (1 -µ 1 )
.

If N a ( ) is chosen as in the statement of Lemma 2.7, one has

f (T ) ≥ N a ( ) ⇒ f (T ) s + b T ≤ d(µ a , µ 1 ) 1 + + b T ≤ d(µ a , µ 1 ).
For such values of T , for each

s ≥ K T +1 there exists µ * (s) ∈]µ a ; µ 1 [ such that d(µ * (s), µ 1 ) = f (T )
s +b T . Then, using Hoeffding inequality and comparing with an integral,

(B) ′ ≤ K T + ∞ K T exp(-2s(µ * (s) -µ a ) 2 )ds .
Using the convexity of the function x ↦ d(x, µ 1 ), a lower bound on µ * (s) -µ a can be obtained, as in Appendix 2 of [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF]:

µ * (s) -µ a ≥ d(µ a , µ 1 ) - f (T ) s + b T -d ′ (µ a , µ 1 )
[ [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF] also provide tight upper bound on the resulting integrals, and following their approach allows us to conclude the proof:

(B) ′ ≤ K T + ∞ K T exp ⎛ ⎝ - 2s d ′ (µ a , µ 1 ) 2 f (T ) s + b T -d(µ a , µ 1 ) 2 ⎞ ⎠ ds ≤ K T + f (T ) ∞ 1+ d(µa,µ 1 ) exp - 2uf (T ) d ′ (µ a , µ 1 ) 2 1 u + b T -d(µ a , µ 1 ) 2 du ≤ K T + f (T ) 2(1+ ) d(µa,µ 1 ) 1+ d(µa,µ 1 ) exp ⎛ ⎝ - 2(1 + ) 1 u + b T -d(µ a , µ 1 ) 2 d(µ a , µ 1 )d ′ (µ a , µ 1 ) 2 f (T ) ⎞ ⎠ du +f (T ) ∞ 2(1+ ) d(µa,µ 1 ) exp - 2uf (T ) d ′ (µ a , µ 1 ) 2 d(µ a , µ 1 ) 2 4(1 + ) 2 du ≤ K T + f (T ) 4(1 + ) 2 d(µ a , µ 1 ) 2 ∞ 0 exp -v 2 f (T ) 2(1 + ) d(µ a , µ 1 )d ′ (µ a , µ 1 ) 2 dv +2(1 + ) 2 d ′ (µ a , µ 1 ) d(µ a , µ 1 ) 2 ≤ K T + f (T ) 2π(1 + ) 3 d ′ (µ a , µ 1 ) 2 d(µ a , µ 1 ) 3 + 2(1 + ) 2 d ′ (µ a , µ 1 ) d(µ a , µ 1 )

Introduction

In 1933, Thompson proposed the first (Bayesian) bandit algorithm in the context of two-armed Bernoulli bandits, which model clinical trials with two possible treatments. The proposed algorithm chooses the next arm based on P , the posterior probability of arm 1 being better than arm 2: arm 1 is chosen at random with probability f (P ), arm 2 with probability 1 -f (P ), where f is some nondecreasing function. Thompson's work ([Thompson, 1933, Thompson, 1935]) was focused on the computation of the probability P when the posterior distribution on the mean of each arm is a Beta distribution. However, when f (P ) = P , that is when arms are sampled according to their posterior probabilities of being optimal, Thompson's algorithm can be implemented without computing the probability P . For example, when the arms are independent, drawing at time t+1 two independent samples θ 1 (t) and θ 2 (t) from each posterior distribution π t 1 and π t 2 and choosing the arm with highest sample (i.e. A t+1 = argmax a θ a (t)) is equivalent to Thompson's algorithm. Indeed, for a prior distribution Π 0 ,

P Π 0 (A t+1 = 1 F t ) = P Π 0 (θ 1 (t) > θ 2 (t) F t ) = P Π 0 (θ 1 > θ 2 F t ) = P t ,
since conditionally to F t , θ 1 and θ 2 are distributed according to the posterior distribution π t 1 and π t 2 , just like the samples θ 1 (t) and θ 2 (t).

This simple principle 'draw each arm according to its posterior probability of being optimal' is now referred to as 'Thompson Sampling' and can be easily generalized beyond two-armed Bernoulli bandit models. As explained above for two-armed bandits, one way to implement Thompson Sampling is to draw a model according to our current belief (i.e. posterior distribution) and act optimally in this sampled model. This is how we define Thompson Sampling for parametric bandit models with independent arms in Algorithm 2. Each bandit model depends on a parameter θ = (θ 1 , . . . , θ K ), and the mean of an arm parameterized by θ is given by µ(θ). As usual, π t a denotes the posterior distribution on θ a at the end of round t. When the arms are no longer independent, Thompson' heuristic samples a bandit model ν from the current posterior distribution over the joint distribution of the arms and chooses the optimal arm in this sampled model. This is how Thompson Sampling is implemented in contextual linear bandit, that will be studied in Chapter 4. Thompson Sampling using the prior distribution Π 0 will be denoted in the sequel by TS Π 0 .

Thompson Sampling for independent arms is still an index policy, but the index computed for each arm is longer an optimistic estimate of the mean -it can even be smaller than the posterior mean-, unlike the quantile used by Bayes-UCB. Bayes-UCB may appear as a 'regularized' version of Thompson Sampling, or conversely Thompson Sampling may be seen as a 'noisy' version of Bayes-UCB, in which the quantile is estimated using only one sample. Although Thompson Sampling was historically Algorithm 2 Thompson Sampling for parametric bandits with independent arms (TS Π 0 )

Require: Π 0 = (π 0 1 , . . . , π 0 K ) (initial prior on θ) 1: for t = 1 to T do 2:
for each arm a = 1, . . . , K do draw an arm A t ∈ arg max a=1...K µ (θ a (t -1)) 6: get reward X t = X At,t and update the posterior distribution Π t 7: end for 3.1. INTRODUCTION 93 introduced first, in this thesis we started by presenting Bayes-UCB, because this algorithm is inspired by the optimistic principle on which asymptotically optimal frequentist algorithms are based. Moreover, the finite-time analysis we propose for Thompson Sampling partly relies on the introduction of well-chosen quantiles of the posterior distribution, and thus theoretical elements from the analysis of Bayes-UCB will be useful in the analysis that we propose here for Thompson Sampling.

Despite its simplicity, Thompson Sampling has been more or less forgotten for decades: it was acknowledged as the first bandit algorithm (for example by [START_REF] Berry | Bandit Problems. Sequential allocation of experiments[END_REF]), but there has been no attempt to analyse its theoretical properties or empirical performance before the late 2000's. At this period, it was re-discovered (sometimes independently) by several authors, under different names. The Bayesian Learning Automaton proposed by [Granmo, 2010] is nothing but Thompson Sampling for two-armed Bernoulli bandits, and the author gives a first consistency result: the probability that the optimal arm is chosen at time t goes to one as t goes to infinity. [Scott, 2010] uses Thompson Sampling under the name randomized probability matching and proposes an empirical evaluation of this method in Bernoulli bandit models and in the generalized linear bandit model (see [START_REF] Filippi | Parametric Bandits : The Generalized Linear case[END_REF]). Similarly, [START_REF] Chapelle | An empirical evaluation of Thompson Sampling[END_REF] propose an empirical evaluation of Thompson Sampling in contextual bandit models. [START_REF] May | Optimistic Bayesian sampling in contextual bandit problems[END_REF]] study a slightly modified version of Thompson Sampling, called Optimistic Bayesian Sampling (OBS): if the sample θ a (t) is such that µ(θ a (t)) is smaller than the posterior mean E[µ(θ a ) F t ], it is replaced by the posterior mean. They prove that Thompson Sampling and OBS satisfy an 'average reward convergence criterion' for contextual bandits, which rewrites for classical bandits

∑ t s=1 µ As tµ * a.s → t→∞ 1.
All these works attracted a lot of interest in Thompson Sampling, because efficient algorithms for contextual bandit models can be used in add prediction systems (see Chapter 4). Besides [START_REF] Chapelle | An empirical evaluation of Thompson Sampling[END_REF] gave the first insight that Thompson Sampling might empirically outperform UCB-like algorithms in classical bandits. However, no theoretical results in terms of regret or Bayes risk could be extracted from these first works. The first logarithmic (finite-time) regret bound on the regret was given by [START_REF] Agrawal | Analysis of Thompson Sampling for the multi-armed bandit problem[END_REF]. More precisely, they prove the following theorem for Bernoulli bandit models. To simplify the presentation, assume that arm 1 is the unique optimal arm and let ∆ a = µ 1 -µ a . Theorem 3.1 ( [START_REF] Agrawal | Analysis of Thompson Sampling for the multi-armed bandit problem[END_REF],Theorem 2). Thompson Sampling using a uniform prior on the means, denoted by Π U , satisfies

R θ (T, TS Π U ) ≤ O ⎛ ⎝ K a=2 1 ∆ 2 a 2 log(T ) ⎞ ⎠
This result is optimal in the sense that the regret is logarithmic, as prescribed by Lai and Robbins' lower bound. However, its is not optimal in terms of the distribution-dependent constant multiplying log(T ), since Lai and Robbins lower bound states that the regret of an uniformly efficient algorithm A satisfies in this particular case

R θ (T, A) ≥ K a=2 ∆ a d(µ a , µ * ) log(T ) (3.1)
A subgaussian approximation of the distribution-dependent term (using Pinsker's inequality) yields the sum over the suboptimal arms of the quantity 1 ∆ a . Whereas Theorem 3.1 exhibits a worse dependency in the gaps ∆ a , [START_REF] Agrawal | Analysis of Thompson Sampling for the multi-armed bandit problem[END_REF] present a refined result for two-armed bandits showing that the regret is O(log(T ) ∆ 2 ), which matches the subgaussian approximation of (3.1) up to a constant factor.

Even in the Bernoulli case, whether Thompson Sampling is asymptotically optimal with respect to Lai and Robbins lower bound was not known, and in the paper [START_REF] Kaufmann | Thompson Sampling : an Asymptotically Optimal Finite-Time Analysis[END_REF] we answer this open question positively: we provide the first finite-time analysis showing that Thompson Sampling using a uniform prior is asymptotically optimal. The finite-time analysis for Bernoulli bandit models given in Section 3.2 closely follows the paper [START_REF] Kaufmann | Thompson Sampling : an Asymptotically Optimal Finite-Time Analysis[END_REF], with minor modifications leading in particular to more explicit constants. Later, [Agrawal and Goyal, 2013a] proposed a different finite-time analysis of Thompson Sampling that also proves its asymptotic optimality, still for Bernoulli bandits. Using elements from their analysis, we were later able in the paper [START_REF] Korda | Thompson Sampling for 1dimensional Exponential family bandits[END_REF] to prove the asymptotic optimality of Thompson Sampling when rewards belong to exponential families. Thompson Sampling for exponential family bandits is discussed in Section 3.3.

[ Agrawal and Goyal, 2013a] also present a distribution-independent upper bound, showing that the regret of Thompson Sampling for Bernoulli bandits satisfies R θ (T, TS Π U ) ≤ O( KT log(T )), and is thus optimal, up to a logarithmic factor in T , with respect to the distribution-independent worst case lower bound (1.6) given in Chapter 1. In the same spirit, [Russo and Van Roy, 2014] propose prior-independent upper bound on the Bayes risk of Thompson Sampling, for very general bandit models. Their analysis relies on strong connections with UCB-like algorithm and will be discussed further in Chapter 4. Their result for K-armed bandits has been improved by [START_REF] Bubeck | Prior-free and prior-dependent regret bounds for Thompson Sampling[END_REF] who show that, if the rewards are bounded in [0, 1], for any prior distribution Π 0 , one has BR Π 0 (T, TS Π 0 ) ≤ 14 √ KT . This bound essentially matches the worst-case lower bound (1.7) presented in Chapter 1. However, whether Thompson Sampling matches the prior-dependent lower bound on the regret given by [Lai, 1987] (see Theorem 1.14 in Chapter 1) is not known yet, and we propose some numerical experiments in Section 3.4 that investigate this question. As we prove the asymptotic optimality of Thompson Sampling in a frequentist sense, we also illustrate in this experimental section its good performance in terms of regret, when compared to the Bayesian and frequentist algorithms studied so far.

Finite-time analysis of Thompson Sampling for binary bandits

In this section, we fix a Bernoulli bandit model, parameterized by θ = (µ 1 , . . . , µ K ) where arm a is a Bernoulli distribution with mean µ a . We assume that there is a unique optimal arm. This is not a restrictive assumption since it can be shown that adding a second optimal arm can only improve the performance of Thompson Sampling (as explained in Appendix A of [START_REF] Agrawal | Analysis of Thompson Sampling for the multi-armed bandit problem[END_REF]). We moreover assume without loss of generality that the arms are ordered such that µ 1 > µ 2 ≥ ... ≥ µ K . To ease the notation, we denote by P and E (in place of P θ and E θ ) the probability and expectation under this bandit model.

As usual, we denote by S a (t) the number of successes observed from action a at the end of round t, and denote the empirical mean by: μa (t) ∶= S a (t) N a (t). With an uniform prior distribution over the means µ a of the arms, the posterior distribution on µ a at the end of round t is explicitly

π t a = Beta (S a (t) + 1, N a (t) -S a (t) + 1) .
Let F Beta a,b denote the cdf of a Beta(a, b) distribution and F B j,µ (resp f B j,µ ) the cdf (resp pdf) of a Binomial(j, µ) distribution. We recall an important link between Beta and Binomial distributions already used in the analysis of Bayes-UCB and in the analysis of Thompson Sampling by [START_REF] Agrawal | Analysis of Thompson Sampling for the multi-armed bandit problem[END_REF]:

F Beta a,b (y) = 1 -F B a+b-1,y (a -1)
We use this 'Beta-Binomial trick' at several stages of our analysis.

We denote by q a (t) a quantile of the posterior distribution that will be useful in the proof and by u a (t) the associated KL-UCB index. More precisely, letting Q(α, π) be the α-quantile of distribution π, these quantities are defined by

u a (t) ∶= argmax x> Sa(t) Na(t) d S a (t) N a (t) , x ≤ log(T ) N a (t) and q a (t) ∶ = Q 1 - 1 T , π t a .
Recall that from Lemma 2.2 in Chapter 2 (see Remark 2.3) q a (t) < u a (t).

Sketch of Analysis

Like the analysis of Bayes-UCB, our analysis of Thompson Sampling is inspired by standard analysis of frequentist index policies. At round t + 1, these policies compute an index U a (t) for each arm a, based on the sequence of observed rewards from this arm up to the end of round t, and choose A t+1 = argmax a U a (t). Such an analysis aims to bound the number of draws of a suboptimal arm, a, by considering two possible events that might lead to a play of this arm:

-the optimal arm (arm 1) is under-estimated, i.e. U 1 (t) < µ 1 ; -the optimal arm is not under-estimated and the suboptimal arm a is drawn at time t + 1. Taking these to be a good description of the event A t+1 = a leads to the decomposition

E[N a (T )] ≤ T -1 t=0 P (U 1 (t) < µ 1 ) + T -1 t=0 P ((U a (t) ≥ µ 1 ) ∩ (A t+1 = a)) .
As explained in Section 1.2.3 of Chapter 1, the analysis of an optimistic algorithm then proceeds by showing that the left term (the 'under-estimation' term) is o (log(T )) and the right term is of the form 1 d(µa,µ 1 ) log(T ) + o (log(T )) (or at worst 2 ∆ 2 a log(T ) + o (log(T )) as in the analysis of UCB1). This scheme of proof works for example for the analysis of UCB1 or KL-UCB (see [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF]).

However we cannot directly apply this method to analyse Thompson Sampling, as the sample θ a (t) is not an optimistic estimate of µ a . Indeed, even when π t 1 is well concentrated and therefore close to a Gaussian distribution centered in µ 1 , P (θ 1 (t) < µ 1 ) is close to 1 2 and the under-estimation term will not be small compared to log T . Hence we will not compare in our proof the sample θ 1 (t) to µ 1 , but to µ 1 -6 log(t) N 1 (t) (if N 1 (t) > 0) which is the lower bound of an UCB interval. We set the convention that if N 1 (t) = 0, 6 log(t) N 1 (t) = ∞. Similarly, when N a (t) = 0, the indices q a (t) and u a (t) previously defined are set to q a (t) = u a (t) = 1.

As observed by [START_REF] Agrawal | Analysis of Thompson Sampling for the multi-armed bandit problem[END_REF] the main difficulty in a regret analysis for Thompson Sampling is to control the number of draws of the optimal arm. We provide this control in the form of Proposition 3.2 whose proof, given in Section 3.2.3, explores in depth the randomized nature of Thompson Sampling.

Proposition 3.2. There exists constants

b = b(µ 1 , µ 2 ) ∈ (0, 1) and C b < ∞ such that ∞ t=1 P N 1 (t) ≤ t b ≤ C b . Remark 3.3.
In general, a result on the regret like E[N 1 (t)] ≥ t -K log(t) does not imply a deviation inequality for N 1 (t) (see [START_REF] Salomon | Deviations of stochastic bandit regret[END_REF]). Proposition 3.2 is therefore a strong result, that enables us to adapt the standard analysis mentioned above.

We can then reduce to analyzing the behavior of the algorithm once it has seen a reasonable number of draws from arm 1, and thus the posterior distribution is well concentrated. Using Proposition 3.

( , µ 1 , µ a ) such that for T ≥ N ( , µ 1 , µ a ), E[N a (T )] ≤ (1 + ) log T d(µ a , µ 1 ) + log(T ) 2π(1 + ) 3 d ′ (µ a , µ 1 ) 2 d(µ a , µ 1 ) 3 + 2(1 + ) 2 d ′ (µ a , µ 1 ) d(µ a , µ 1 ) 2 + 5 + 2C b + N (b).
The constants are made more explicit in the proofs of Proposition 3.2 and Theorem 3.4. The fact that Theorem 3.4 holds for every > 0 gives the asymptotic optimality of Thompson Sampling.

Proof of Theorem 3.4

Step 1: Decomposition. First we recall the modified decomposition mentioned above:

E[N a (T )] ≤ T -1 t=0 P θ 1 (t) ≤ µ 1 - 6 log t N 1 (t) + T -1 t=0 P θ a (t) > µ 1 - 6 log t N 1 (t) , A t+1 = a ≤ T -1 t=0 P θ 1 (t) ≤ µ 1 - 6 log t N 1 (t) + T -1 t=0 P θ a (t) > µ 1 - 6 log t N 1 (t) , A t+1 = a, θ a (t) < q a (t) + T -1 t=0 P (θ a (t) > q a (t)) .
The sample θ a (t) from the posterior is not very likely to exceed the quantile q a (t) introduced above:

T -1 t=0 P (θ a (t) > q a (t)) ≤ T -1 t=0 E [P (θ a (t) > q a (t) F t )] ≤ T -1 t=0 1 T = 1.
Finally, using that u a (t) ≥ q a (t) yields

E[N a (T )] ≤ T -1 t=0 P θ 1 (t) ≤ µ 1 - 6 log t N 1 (t) A + T -1 t=0 P u a (t) > µ 1 - 6 log t N 1 (t) , A t+1 = a B +1. (3.2)
Step 2: Bounding term A. Let b and C b be defined in Proposition 3.2.

To deal with term A we show a new self-normalized deviation inequality adapted to the randomization occurring at each round of Thompson Sampling.

N 0 (b) = inf t ∈ N ∶ log(t)t b ≥ ( √ 6 - √ 5) -2 .
One has

∞ t=1 P θ 1 (t) ≤ µ 1 - 6 log t N 1 (t) ≤ N 0 (b) + 3 + C b < ∞.
Proof Let (U t ) denote a sequence of i.i.d. uniform random variables on [0, 1], and let S 1,s = ∑ s k=1 Y 1,s be the sum of the first s rewards from arm 1. In the following, we make the first use of the link between Beta and Binomial distributions:

P θ 1 (t) ≤ µ 1 - 6 log t N 1 (t) = P U t ≤ F Beta S 1 (t)+1,N 1 (t)-S 1 (t)+1 µ 1 - 6 log t N 1 (t) = P ⎛ ⎝ ⎛ ⎝ U t ≤ 1 -F B N 1 (t)+1,µ 1 -6 log t N 1 (t) (S 1 (t)) ⎞ ⎠ ∩ N 1 (t) ≥ t b ⎞ ⎠ + P N 1 (t) ≤ t b = P ⎛ ⎝ ⎛ ⎝ F B N 1 (t)+1,µ 1 -6 log t N 1 (t) (S 1 (t)) ≤ U t ⎞ ⎠ ∩ N 1 (t) ≥ t b ⎞ ⎠ + P N 1 (t) ≤ t b ≤ P ∃s ∈ {t b ...t} ∶ F B s+1,µ 1 -6 log t s (S 1,s ) ≤ U t + P N 1 (t) ≤ t b = t s=⌈t b ⌉ P S 1,s ≤ (F B ) -1 s+1,µ 1 -6 log t s (U t ) + P N 1 (t) ≤ t b
The first term in the final line of this display now deals only with Binomial random variables with large numbers of trials (greater than t b ), and so we can draw on standard concentration techniques to bound this term. Proposition 3.2 takes care of the second term.

Let s be fixed. The random variable

(F B ) -1 s+1,µ 1 -6 log t s (U t ) ∼ Bin s + 1, µ 1 -6 log t s
is independent from S 1,s ∼ Bin (s, µ 1 ). One can write S 1,s = ∑ s l=1 Y 1,s , where (Y 1,l ) is an i.i.d sequence of Bernoulli random variables with mean µ 1 . Introducing ( Ỹ1,l ) l , a second i.i.d. sequence of Bernoulli random variables with mean µ 1 -6 log t s that is independent from (Y 1,l ), one has

P S 1,s ≤ (F B ) -1 s+1,µ 1 -6 log t s (U t ) = P s l=1 Y l,s ≤ s+1 l=1 Ỹl,s ≤ P ⎛ ⎝ s l=1 ⎛ ⎝ Y 1,l -Ỹ1,l - 6 log t s ⎞ ⎠ ≤ - 6s log t -1 ⎞ ⎠ .
Letting Z l ∶= µ 1 -6 log t s, (Z l ) is a sequence of i.i.d centered random variable with range 2, and Hoeffding's inequality can be used to bound the last sum. Moreover, for t ≥ N 0 (b) where

N 0 (b) ∶= inf t ∈ N ∶ log(t)t b ≥ ( √ 6 - √ 5) -2
CHAPTER 3. THOMPSON SAMPLING one has √ 6s log t -1 > √ 5s log t, for s ≥ t b , and one can write

P S 1,s < (F B ) -1 s+1,µ 1 -6 log t s (U t ) ≤ exp -2 ( √ 5s log t) 2 4s = e -5 2 log t = 1 t 5 2
.

We conclude that

∞ t=1 P θ 1 (t) < µ 1 - 6 log t N 1 (t) ≤ N 0 (b) + ∞ t=1 1 t 3 2 + C b ≤ N 0 (b) + 3 + C b .

◻

Step 3: Bounding Term B. We specifically show that Lemma 3.6. For T such that

log(T ) ≥ d(µ a , µ 1 ) 1 + exp 8 (µ 1 (1 -µ 1 )) 2 (1 + ) 2 2 d(µ a , µ 1 ) 2 , one has (B) ≤ (1 + ) log(T ) d(µ a , µ 1 ) + log(T ) 2π(1 + ) 3 d ′ (µ a , µ 1 ) 2 d(µ a , µ 1 ) 3 + 2(1 + ) 2 d ′ (µ a , µ 1 ) d(µ a , µ 1 ) 2 +1 + N 1 (b) + C b . with N 1 (b) ∶= inf t ≥ e 2 b ∶ 3(log t) 2 ≤ t b .
Proof Using Proposition 3.2, term B can be rewritten

(B) ≤ T -1 t=0 P u a (t) > µ 1 - 6 log t N 1 (t) , A t+1 = a, N 1 (t) ≥ t b + T -1 t=0 P N 1 (t) ≤ t b ≤ T -1 t=0 P ⎛ ⎝ u a (t) > µ 1 - 6 log t t b , A t+1 = a ⎞ ⎠ + C b
Let N 1 (b) be defined in the statement of Lemma 3.6. For t ≥ N 1 (b), one has 6 log t t b ≤ 2 log(t).

Letting g t ∶= 2 log t as in the finite-time analysis proposed for Bayes-UCB in the previous chapter, one can write, in a very similar way,

(B) ≤ T -1 t=N 1 (b)+1 P (u a (t) > µ 1 -g t , A t+1 = a) + N 1 (b) + C b = E ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ T -1 t=N 1 (b)+1 1 (Na(t)=0,A t+1 =a) + t s=1 1 (Na(t)=s,A t+1 =a) 1 (sd + (µa,s,µ 1 -gt)≤log(T )) ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ + N 1 (b) + C b ≤ 1 + E ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ T -1 t=N 1 (b)+1 t s=1 1 (Na(t)=s,A t+1 =a) 1 (sd + (µa,s,µ 1 -gs)≤log(T )) ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ + N 1 (b) + C b ≤ T -1 s=1 P (sd + (µ a,s , µ 1 -g s ) ≤ log(T ))) + 1 + N 1 (b) + C b .

FINITE-TIME ANALYSIS OF THOMPSON SAMPLING FOR BINARY BANDITS 99

We used as in the previous chapter that for all s, ∑ t∈N 1 (Na(t)=s) 1 (A t+1 =a) ≤ 1 and that when s ≤ t, g s ≤ g t . The sum in the right hand side can now exactly be upper bounded using Lemma 2.7 in Chapter 2 with f (T ) = log(T ), which concludes the proof. Since we focus on the number of draws of the optimal arm, let τ j be the occurrence of the j th play of the optimal arm (with τ 0 ∶= 0). Let ξ j ∶= (τ j+1 -1) -τ j : this random variable measures the number of time steps between the j th and the (j + 1) th play of the optimal arm, and so

∑ K a=2 N a (t) = ∑ N 1 (t)
j=0 ξ j . For each suboptimal arm, a relevant quantity is

C a = 32 (µ 1 -µ a ) 2 .
We let C = max a≠1 C a = 32 (µ 1 -µ 2 ) 2 and introduce δ a = (µ 1 -µ a ) 2 and δ = δ 2 .

Step 1: Initial Decomposition of Summands. First we use a union bound on the summands to extract the tails of the random variables ξ j :

P(N 1 (t) ≤ t b ) = P K a=2 N a (t) ≥ t -t b ≤ P ∃j ∈ 0, .., ⌊t b ⌋ ∶ ξ j ≥ t 1-b -1 ≤ ⌊t b ⌋ j=0 P(ξ j ≥ t 1-b -1) (3.3)
This means that there exists a time range of length t 1-b -1 during which only suboptimal arms are played.

In the case of two arms this implies that the (unique) suboptimal arm is played ⌈ t 1-b -1 2 ⌉ times during the first half of this time range. Thus its posterior becomes well concentrated around its mean with high probability, and we can use this fact to show that the probability that the suboptimal action is chosen a further ⌈ t 1-b -1 2 ⌉ times in a row is very small. In Figure 3.2.3 below, one indeed sees that the posterior of arm 2 -in blue-becomes concentrated, and so the samples θ 2 (s) used by Thompson Sampling on the second half of the time range are very likely to always fall below µ 2 + δ. Thus, as arm 1 is not drawn on this time range, the samples θ 1 (s) (that are i.i.d. samples from the current posterior on arm 1 -in red-, that does not change on the time range) have to always fall in the shaded region, which is not very likely.

To precise this heuristic argument and generalize it to more arms, we introduce a notion of a saturated, suboptimal action: Definition 3.7. Let t be fixed. For any a ≠ 1, an action a is said to be saturated at time s if it has been chosen at least C a ln(t) times. That is N a (s) ≥ C a ln(t). We shall say that it is unsaturated otherwise. Furthermore at any time we call a choice of an unsaturated, suboptimal action an interruption. We want to study the event E j = {ξ j ≥ t 1-b -1}. We introduce the interval I j = {τ j , τ j + ⌈t 1-b -1⌉} (included in {τ j , τ j+1 } on E j ) and begin by decomposing it into K subintervals:

I j,l ∶= τ j + ⌈ (l -1)(t 1-b -1) K ⌉ , τ j + ⌈ l(t 1-b -1) K ⌉ , l = 1, . . . , K.
Now for each interval I j,l , we introduce: -F j,l : the event that by the end of the interval I j,l at least l suboptimal actions are saturated; -n j,l : the number of interruptions during this interval. We use the following decomposition to bound the probability of the event E j :

P(E j ) = P(E j ∩ F j,K-1 ) + P(E j ∩ F c j,K-1 ) (3.4)
To bound both probabilities, we will need the fact, stated in Lemma 3.8, that the probability of θ 1 (s) being smaller than µ 2 + δ during a long subinterval of I j is small. This follows from the fact that the posterior on the optimal arm is always Beta(S 1 (τ j ) + 1, j -S 1 (τ j ) + 1) on I j : hence, when conditioned on S 1 (τ j ), θ 1 (s) is an i.i.d. sequence with non-zero support above µ 2 + δ, and thus is unlikely to remain below µ 2 + δ for a long time period. This idea is also important in the analysis of Thompson Sampling by [START_REF] Agrawal | Analysis of Thompson Sampling for the multi-armed bandit problem[END_REF]. Lemma 3.8. ∃λ 0 = λ 0 (µ 1 , µ 2 ) > 1 such that for λ ∈]1, λ 0 [, for every τ j -measurable interval J , such that J ≥ f (t) for some positive function f , one has

P (J ⊆ [τ j , τ j+1 [, ∀s ∈ J θ 1 (s) ≤ µ 2 + δ) ≤ (α µ 1 ,µ 2 ) f (t) + C λ,µ 1 ,µ 2 1 f (t) λ e -jd λ,µ 1 ,µ 2 where C λ,µ 1 ,µ 2 , d λ,µ 1 ,µ 2 > 0 and α µ 1 ,µ 2 = (1 2) 1-µ 2 -δ .

FINITE-TIME ANALYSIS OF THOMPSON SAMPLING FOR BINARY BANDITS 101

The proof of this important lemma will be postponed to Section 3.5.1 and all the constants are explicitly defined there. Another key point in the proof is the fact that a sample from a saturated suboptimal arm cannot fall too far from its true mean. The following lemma is very close to Lemma 7 of [START_REF] Agrawal | Analysis of Thompson Sampling for the multi-armed bandit problem[END_REF]. We propose a proof in Section 3.5.2. Lemma 3.9.

P (∃s ≤ t, ∃a ≠ 1 ∶ θ a (s) > µ a + δ a , N a (s) > C a ln(t)) ≤ 2(K -1) t 2 .
Step 2: Bounding P(E j ∩ F j,K-1 ). On the event E j ∩ F j,K-1 , only saturated suboptimal arms are drawn on the interval I j,K . Using the concentration results for samples of these arms in Lemma 3.9, we get

P(E j ∩ F j,K-1 ) ≤P({∃s ∈ I j,K , a ≠ 1 ∶ θ a (s) > µ a + δ} ∩ E j ∩ F j,K-1 ) + P({∀s ∈ I j,K , a ≠ 1 ∶ θ a (s) ≤ µ a + δ a } ∩ E j ∩ F j,K-1 ) ≤P(∃s ≤ t, a ≠ 1 ∶ θ a (s) > µ a + δ a , N a (t) > C a ln(t)) + P({∀s ∈ I j,K , a ≠ 1 ∶ θ a (s) ≤ µ 2 + δ} ∩ E j ∩ F j,K-1 ) ≤ 2(K -1) t 2 + P(I j,K ⊆ [τ j , τ j+1 [, ∀s ∈ I j,K θ 1 (s) ≤ µ 2 + δ).
The last inequality comes from the fact that if arm 1 is not drawn, the sample θ 1 (s) must be smaller than some sample θ a (s) and therefore smaller than µ 2 + δ. Since I j,K is a τ j -measurable interval of size ⌈ t 1-b -1 K ⌉ we get using Lemma 3.8, for some fixed λ ∈]1, λ 0 [,

P(I j,K ⊆ [τ j , τ j+1 [, ∀s ∈ I j,K θ 1 (s) ≤ µ 2 + δ) ≤ (α µ 1 ,µ 2 ) t 1-b -1 K + C λ,µ 1 ,µ 2 1 t 1-b -1 K λ e -jd λ,µ 1 ,µ 2 =∶ g(µ 1 , µ 2 , b, j, t). (3.5)
Hence we have show that

P(E j ∩ F j,K-1 ) ≤ 2(K -1) t 2 + g(µ 1 , µ 2 , b, j, t), (3.6)
and choosing b such that b < 1 -1 λ , the following hypothesis on g holds:

t≥1 j≤t b g(µ 1 , µ 2 , b, j, t) < +∞.
Step 3: Bounding P(E j ∩ F c j,K-1 ). We show through an induction that for all 2 ≤ l ≤ K, if t is larger than some deterministic constant N µ 1 ,µ 2 ,b specified in the base case,

P(E j ∩ F c j,l-1 ) ≤ (l -2) 2(K -1) t 2 + f (µ 1 , µ 2 , b, j, t)
for some function f such that ∑ t≥1 ∑ 1≤j≤t b f (µ 1 , µ 2 , b, j, t) < ∞. For l = K we get

P(E j ∩ F c j,K-1 ) ≤ (K -2) 2(K -1) t 2 + f (µ 1 , µ 2 , b, j, t) . (3.7)
Step 4: The Base Case of the induction. Note that on the event E j only suboptimal arms are played during I j,1 . Hence at least one suboptimal arm must be played ⌈ t 1-b -1 K 2 ⌉ times. There exists some deterministic constant

N µ 1 ,µ 2 ,b such that for t ≥ N µ 1 ,µ 2 ,b , ⌈ t 1-b -1 K 2 ⌉ ≥ C ln(t)
(the constant depends only on µ 1 and µ 2 because C = C 2 ). So when t ≥ N µ 1 ,µ 2 ,b , at least one suboptimal arm must be saturated by the end of I j,1 . Hence, for t

≥ N µ 1 ,µ 2 ,b P(E j ∩ F c j,1 ) = 0.
This concludes the base case.

Step 5: The Induction. As an inductive hypothesis we assume that for some

2 ≤ l ≤ K -1 if t ≥ N µ 1 ,µ 2 ,b then P(E j ∩ F c j,l-1 ) ≤ (l -2) 2(K -1) t 2 + f (µ 1 , µ 2 , b, j, t) .
Then, making use of the inductive hypothesis,

P(E j ∩ F c j,l ) ≤ P(E j ∩ F c j,l-1 ) + P(E j ∩ F c j,l ∩ F j,l-1 ) ≤ (l -2) 2(K -1) t 2 + f (µ 1 , µ 2 , b, j, t) + P(E j ∩ F c j,l ∩ F j,l-1 ).
To complete the induction we therefore need to show that:

P(E j ∩ F c j,l ∩ F j,l-1 ) ≤ 2(K -1) t 2 + f (µ 1 , µ 2 , b, j, t). (3.8)
On the event (E j ∩ F c j,l ∩ F j,l-1 ), there are exactly l -1 saturated arms at the beginning of interval I j,l and no new arm is saturated during this interval. As a result there cannot be more than KC ln(t) interruptions during this interval, and so we have

P(E j ∩ F c j,l ∩ F j,l-1 ) ≤ P(E j ∩ F j,l-1 ∩ {n j,l ≤ KC ln(t)}).
Let S l denote the set of saturated arms at the end of I j,l and introduce the following decomposition:

P(E j ∩ F j,l-1 ∩ {n j,l ≤ KC ln(t)}) ≤ P({∃s ∈ I j,l , a ∈ S l-1 ∶ θ a (s) > µ a + δ a } ∩ E j ∩ F j,l-1 ) A + P({∀s ∈ I j,l , a ∈ S l-1 ∶ θ a (s) ≤ µ a + δ a } ∩ E j ∩ F j,l-1 ∩ {n j,l ≤ KC ln(t)}) B .
Clearly, using Lemma 3.9:

(A) ≤ P (∃s ≤ t, ∃a ≠ 1 ∶ θ a (s) > µ a + δ a , N a (s) > C a ln(t)) ≤ 2(K -1) t 2 .
To deal with term (B), we introduce for k in {0, . . . , n j,l -1} the random intervals J k as the time range between the k th and (k+1) st interruption in I j,l . For k ≥ n j,l we set J k = ∅. Note that on the event in the probability (B) there is a subinterval of I j,l of length ⌈ t 1-b -1 CK 2 ln(t) ⌉ during which there are no interruptions.

THOMPSON SAMPLING FOR EXPONENTIAL FAMILIES 103

Moreover on this subinterval of I j,l , for all a ≠ 1, θ a (s) ≤ µ 2 + δ 2 . (This holds for unsaturated arms as well as for saturated arms since their samples are smaller than the maximum sample of a saturated arm.) Therefore,

(B) ≤ P( ∃k ∈ {0, ..., n j,l } ∶ J k ≥ (t 1-b -1) (CK 2 ln(t)) ∩ ∀s ∈ I j,l , a ∈ S l-1 ∶ θ a (s) ≤ µ 2 + δ ∩ E j ∩ F j,l-1 ) ≤ KC ln(t) k=1 P J k ≥ t 1-b -1 CK 2 ln(t) ∩ {∀s ∈ J k , a ≠ 1 ∶ θ a (s) ≤ µ 2 + δ} ∩ E j ≤ KC ln(t) k=1 P J k ≥ t 1-b -1 CK 2 ln(t) ∩ {∀s ∈ J k , θ 1 (s) ≤ µ 2 + δ} (3.9)
Now, we have to bound the probability that θ 1 (s) ≤ µ 2 + δ for all s in an interval of size t 1-b -1 CK 2 ln(t) in I j . So we apply Lemma 3.8 to get:

(B) ≤ CK ln(t)(α µ1,µ2 ) t 1-b -1 CK 2 ln(t) + C λ,µ1,µ2
CK ln(t)

t 1-b -1 CK 2 ln(t) λ e -jd λ,µ 1 ,µ 2 ∶= f (µ 1 , µ 2 , b, j, t).
Choosing the same b as in (3.5), we get that ∑ t≥1 ∑ 1≤j≤t b f (µ 1 , µ 2 , b, j, t) < +∞. It follows that for this value of b, (3.8) holds and the induction is complete.

Step 8: Conclusion. Let b be the constant chosen in Step 2. From the decomposition (3.4) and the two upper bounds (3.6) and (3.7), we get, for t ≥ N µ 1 ,µ 2 ,b :

P(E j ) ≤ (K -2) 2(K -1) t 2 + f (µ 1 , µ 2 , b, j, t)) + 2(K -1) t 2 + g(µ 1 , µ 2 , b, j, t).
Recalling (3.3), summing over the possible values of j and t we obtain:

t≥1 P(N 1 (t) ≤ t b ) ≤N µ 1 ,µ 2 ,b + 2(K -1) 2 t≥1 1 t 2-b + t≥1 t b j=1 [Kf (µ 1 , µ 2 , b, j, t) + g(µ 1 , µ 2 , b, j, t)] < C µ 1 ,µ 2 ,b for some constant C µ 1 ,µ 2 ,b < ∞.

Thompson Sampling for Exponential families

Just like the analysis of Bayes-UCB, the finite-time analysis we gave in the previous section for Thompson Sampling strongly relies on the specific properties of Beta posterior with integer coefficients, and can therefore only be applied to Bernoulli rewards and uniform prior. In the paper [START_REF] Korda | Thompson Sampling for 1dimensional Exponential family bandits[END_REF] we propose a different analysis, that is suited for exponential family bandit models, with the use of the Jeffreys' prior. While providing the full paper in Appendix B, we present in this Section the outline of this new analysis, trying to highlight the similarities and differences with our previous work. The notation used in this Section is consistent with the notation used in all the thesis, whereas that used in Appendix B are slightly different.

Thompson Sampling with Jeffreys' prior for general one-parameter canonical exponential families

In this work, we consider bandit models in which each arm belong to one-parameter canonical exponential family, for which the density of ν θ is given by

f (x θ) = A(x) exp(T (x)θ -b(θ)).
(3.10) This is the general definition of one-parameter canonical exponential families given for example by [START_REF] Bickel | Mathematical Statistics, Basic Ideas and Selected Topics[END_REF], even if so far we only considered particular cases in which the sufficient statistic T is such that

T (x) = x.
ḃ(θ) = E X∼ν θ [T (X)] and b(θ) = Var X∼ν θ [T (X)].
Whereas these distributions can still be parameterized by their means µ(θ), a natural alternative parametrization consists in using the mean of the sufficient statistic, μ(θ) = E X∼ν θ [T (X)]. For a given exponential family, one can introduce the divergence associated as a function of this new parameter:

d(μ, μ′ ) = KL ν ḃ-1 (μ) , ν ḃ-1 (μ ′ ) .
When T (x) = x, μ(θ) = µ(θ), so this definition of the divergence function d coincides with the definition given in Chapter 1 in this particular case. For X ∼ ν θ with mean of sufficient statistic μ, introducing φX (λ) = log E e λT (X) and φ * X (x) its Fenchel-Legendre transform (convex conjugate function), one has d(x, μ) = φ * X (x). This important property is given in Lemma 1.4 of Chapter 1 for exponential families such that T (x) = x, for which it allows to build confidence interval based on KL-divergence. Similarly, for more general exponential families, it is possible to build KL-confidence intervals for the mean of the sufficient statistic μ, and to define and analyse the associated KL-UCB algorithm as the index policy associated to

u a (t) = sup ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ q ≥ 1 N a (t) Na(t) i=1 T (Y a,i ) ∶ N a (t)d ⎛ ⎝ 1 N a (t) Na(t) i=1 T (Y a,i ), q⎞ ⎠ ≤ f (t) ⎫ ⎪ ⎪ ⎬ ⎪ ⎪ ⎭ .
For exponential families defined by (3.10), it is also easy to implement the Thompson Sampling algorithm, since the posterior distribution still has an explicit form. If the prior distribution on the parameter θ has density h 0 (θ), the posterior distribution after n observations y 1 , . . . , y n is given by

p(θ y 1 , . . . , y n ) ∝ h 0 (θ) exp θ n i=1 T (y i ) -nb(θ) .
We consider in the sequel the particular implementation of Thompson Sampling that uses the Jeffreys' prior. This prior, introduced by [Jeffreys, 1946] is non-informative in the sense that it is invariant under re-parametrization of the parameter space. It can be shown to be proportional to the square root of the Fischer information I(θ), which is equal to b(θ) in our particular case. Thompson Sampling thus draws for each arm a a sample

θ a (t) ∼ π t a ∝ b(θ) exp ⎛ ⎝ θ Na(t) i=1 T (Y a,i ) -N a (t)b(θ) ⎞ ⎠

THOMPSON SAMPLING FOR EXPONENTIAL FAMILIES

105 and chooses at time t + 1 arm A t+1 = argmax a µ(θ a (t)).

We give in Appendix B examples of implementation for classical distributions in an exponential family with alternative parametrization. For example for Bernoulli bandits, the Jeffreys' prior on the mean is Beta(1 2, 1 2) and does not coincide with the uniform prior always considered so far. This version of Thompson Sampling thus draws samples from Beta(1 2 + S a (t), 1 2 + N a (t) -S a (t)) and cannot be analysed with our previous tools. In many other cases, Jeffreys' prior turns out to be an improper prior ( ∫ Θ b(θ)dθ = +∞), but after one observation, the resulting posterior becomes a probability distribution, thus Thompson Sampling can be implemented with an initialization phase drawing each arm once.

Main result and sketch of the proof

Theorem 3.10 is the main result proved in the paper [START_REF] Korda | Thompson Sampling for 1dimensional Exponential family bandits[END_REF]. It implies that Thompson Sampling using the Jeffreys' prior is asymptotically optimal when the rewards distributions belong to an exponential family defined by (3.10). We use the shorthand K(θ, θ ′ ) to refer to the Kullback-Leibler divergence between the distributions ν θ and ν θ ′ . Theorem 3.10. Assume that µ 1 > µ a for all a ≠ 1, the prior distribution Π J is such that for all a π a,0 is taken to be the Jeffreys' prior over Θ. Then for every > 0 there exists a constant C( , P) depending on and on the problem P such that the regret of Thompson Sampling using the Jeffreys' prior satisfies

R θ (T, TS π J ) ≤ 1 + 1 - K a=2 (µ 1 -µ a ) K(θ a , θ 1 ) ln(T ) + C( , P).
Our analysis relies on three main ingredients: a new decomposition inspired by the one proposed by [Agrawal and Goyal, 2013a], a non-asymptotic upper bound on the tail of the posterior distribution provided that the sufficient statistics are well concentrated (Theorem B.4 in Appendix B) and a result akin to Lemma 3.2 that controls the number of draws of the optimal arm. When Jeffreys' prior is improper, the statement of the posterior concentration result is quite involved, and requires to introduce events (denoted by Ẽa,t in Appendix B) with complicated expression. For the sake of clarity, we present below the outline of our finite-time analysis in cases where Jeffreys' prior is proper. The general case is dealt with in Appendix B.

A new decomposition. [Agrawal and Goyal, 2013a] propose an alternative finite-time analysis that leads to the asymptotic optimality of Thompson Sampling with a uniform prior in Bernoulli bandit models. The decomposition we use here is close to the one they introduce.

For all a = 1 . . . K let δ a > 0 be fixed and let E a (t)(= E a (t, δ a )) be the event defined by

E a (t) = ⎛ ⎝ N a (t) ≠ 0 ⇒ 1 N a (t) Na(t) s=1 T (Y a,s ) -b ′ (θ a ) ≤ δ a ⎞ ⎠ .
For a ≠ 1, let ∆ a < µ 1 -µ a be fixed and let E θ a (t)(= E θ a (t, ∆ a )) be the event

E θ a (t) = (µ(θ a (t)) ≤ µ a + ∆ a )
.

On E a (t) the empirical sufficient statistic is well concentrated around its mean and on E θ a (t) the sample from π t a used in the algorithm does not lead to an over-estimation of the true mean µ a . The conjunction of these two events holds with high probability. Now we introduce the following decomposition:

E [N a (T )] = T -1 t=0 P A t+1 = a, E a (t), E θ a (t) (A) + T -1 t=0 P A t+1 = a, E a (t), (E θ a (t)) c (B) + T -1 t=0 P (A t+1 = a, (E a (t)) c ) (C)
.

Term (C) only concerns the concentration of empirical sufficient statistics around their means and standard concentration techniques show that it is bounded. To deal with term (B), one needs to introduce the posterior concentration result below.

Lemma 3.11. Let π(n, s) denote the posterior distribution under Jeffreys' prior and n observations such that the sum of sufficient statistics is s = ∑ n i=1 T (y i ). If Jeffreys' prior is proper and s k -ḃ(θ a ) ≤ δ , there exists three constants N a , C 1,a and C 2,a = C 2,a (∆) such that for k ≥ N a ,

P θ∼π(k,s) (µ(θ) > µ a + ∆) ≤ C 1,a k exp(-(k -1)(1 -δC 2,a )K(θ a , µ -1 (µ a + ∆))) P θ∼π(k,s) (µ(θ) < µ a -∆) ≤ C 1,a k exp(-(k -1)(1 -δC 2,a )K(θ a , µ -1 (µ a + ∆)))
Using Lemma 3.11, it can be proved (see Section B.8) in Appendix B) that for all > 0, there exists a constant C 1 ( , θ, δ, ∆ a ) such that

(B) ≤ log(T ) (1 -)(1 -δ a C 2,a )K(θ a , µ -1 (µ a + ∆ a )) + C 1 ( , θ, δ a , ∆ a ).
Term (A) is the most delicate to control: when the sample from arm a used by the algorithm does not over-estimate the mean µ a , it should be explained why arm a should not be drawn to much. To do so, we show that this implies some event regarding the optimal arm that hold with small probability since this arm has been drawn a lot. Indeed, one can show a deviation result similar to Proposition 3.2, stated as Proposition 3.12. The proof of this result is essentially the same as that of Proposition 3.2, except that we replace Lemma 3.8 (in which constants are computed explicitly) by an asymptotic argument based on the posterior concentration phenomenon.

Proposition 3.12. For all b ∈]0, 1[, ∑ ∞ t=1 P(N 1 (t) ≤ t b ) ≤ +∞.
Term (A) can be upper bounded in the following way. As Proposition 3.12 holds for any value of b we apply it for example for b

= 1 2. Let ∆ ′ a = µ 1 -µ a -∆ a . (A) ≤ T -1 t=0 P(A t+1 = a, E θ a (t), N 1 (t) > √ t) + C 1 2 ≤ T -1 t=0 P(µ(θ 1 (t)) ≤ µ 1 -∆ ′ a , N 1 (t) > √ t) + C 1 2 ≤ T -1 t=0 P(µ(θ 1 (t)) ≤ µ 1 -∆ ′ a , E 1 (t), N 1 (t) > √ t) + T -1 t=0 P((E 1 (t)) c , N 1 (t) > √ t) + C 1 2
Whereas the second sum in the last display is upper bounded using classical concentration techniques, for the first term, we need the second statement of Lemma 3.11 to show

P(µ(θ 1 (t)) ≤ µ 1 -∆ ′ a F t )1 E 1 (t) ≤ C 1,1 e -(N 1 (t)-1)(1-δ 1 C 2,1 )K(θ 1 ,µ -1 (µ 1 -∆ ′ a ))+log(N 1 (t))
and finally lower bound term (A) by some constant plus the series

∞ t=1 C 1,1 e -( √ t-1)(1-δ 1 C 2,1 )K(θ 1 ,µ -1 (µ 1 -∆ ′ a ))+(1 2) log(t) < +∞.
To conclude, one has shown that there exists a constant C( , θ, δ a , ∆ a ) such that

E[N a (T )] ≤ ln(T ) (1 -δ a C 2,a )K(θ a , µ -1 (µ a + ∆ a ))(1 -) + C( , θ, δ a , ∆ a )
The constant is of course increasing (dramatically) when δ a goes to zero, ∆ a to µ 1 -µ a , or to zero. But one can choose ∆ a close enough to µ 1 -µ a and δ a small enough, such that

(1 -C 2,a (∆ a )δ a )K(θ a , µ -1 (µ a + ∆ a )) ≥ K(θ a , θ 1 ) (1 + ) ,
and this choice leads to

E[N a (T )] ≤ 1 + 1 - ln(T ) K(θ a , θ 1 ) + C( , θ, δ a , ∆ a ).

Numerical experiments and discussion

We illustrate here the performance of Thompson Sampling on numerical experiments with Bernoulli rewards. We compare both the regret and Bayes risk of Thompson Sampling to those of state-of-the-art frequentist algorithms and Bayes-UCB.

Regret of Thompson Sampling

We start by comparing, on several two-armed bandit models and for a quite small horizon (T = 1000), the regret of Thompson Sampling, Bayes-UCB, KL-UCB and its two variants KL-UCB + and KL-UCB-H + and the FH-Gittins algorithms. Preliminary experiments in Chapter 2 showed that FH-Gittins, that we believe to be a good approximation of the Bayesian optimal solution, displays good performance for several fixed bandit models. Here we investigate this trend further, adding also a comparison with Thompson Sampling. Results are reported in Figure 3.2. On the four different two-armed bandit considered with small (left) and high mean rewards (right), we see that FH-Gittins compares well to asymptotically optimal algorithms, even if it does not always outperform them. In particular, the performance of FH-Gittins seems to deteriorate on problems with high mean rewards. On this small scale, we also see that Thompson Sampling and Bayes-UCB do not always outperform KL-UCB, and that Bayes-UCB performs better than Thompson Sampling. This trend seems to be reversed when we consider larger horizons.

In our second experiment, we study regret up to some larger horizon T = 20000 for a 10-armed bandit problem, already studied by [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF], with means given by µ = (0.1, 0.05, 0.05, 0.05, 0.02, 0.02, 0.02, 0.01, 0.01, 0.01).

(3.11) The first four algorithms are variants of UCB, displayed using the same scale, that are not known to be optimal. Of these, the UCB-V algorithm of [START_REF] Audibert | Exploration-exploitation trade-off using variance estimates in multi-armed bandits[END_REF] is close to the index policy to which Thompson Sampling is compared in [START_REF] Chapelle | An empirical evaluation of Thompson Sampling[END_REF] in the Bernoulli setting. This algorithm incorporates an estimation of the variance of the rewards in the index which is defined to be, for an arm that have produced k rewards in n draws,

k n + 2 log(t) n k n 1 - k n + 3 log(t) n
UCB-Tuned is an heuristic proposed by [Auer et al., 2002a] that also uses estimates of the variance, whereas MOSS ( [Audibert and Bubeck, 2010]) is a variant of UCB using an alternative exploration rate that is reminiscent of KL-UCB-H + : the log(t) in UCB is replaced by log(T (KN a (t))).

The other six algorithms displayed in Figure 3.3, on a different scale, have a mean regret closer to (sometimes smaller than) the lower bound of Lai and Robbins, displayed in blue, which we recall is only asymptotic. All these algorithms are provably asymptotically optimal. Among them, Thompson Sampling outperforms Bayes-UCB and on this specific bandit model its performance is comparable to the best frequentist algorithms, KL-UCB + and KL-UCB-H + . It is also the easiest optimal policy to implement, since at each round the indices computed by KL-UCB (and its variants) and even the quantiles computed by Bayes-UCB are more costly than the production of posterior samples.

In Section 3.2 and 3.3 we proved the asymptotic optimality of two versions of Thompson Sampling: the first using the uniform prior (Beta(1, 1)) and the second the Jeffreys' prior, which corresponds to a Beta (1 2, 1 2) prior distribution over each mean. In Figure 3.4, we compare the two resulting algorithms. While for the 10-armed bandit studied before using Jeffreys' prior appears to reduce the regret for both Bayes-UCB and Thompson Sampling, one can find other problems (like the two-armed bandit with means 0.8 and 0.9) for which it does not. Besides, the performance of the two variants is quite close on both problems presented in Figure 3.4. Thus we may conjecture that for distributions depending on a single parameter, the prior distribution chosen has little influence on the asymptotic optimality of Thompson Sampling. However, [START_REF] Honda | Optimality of Thompson Sampling for Gaussian Bandits depends on priors[END_REF] show that this conjecture in not true for the particular (two-parameter) case of Gaussian distributions with unknown mean and variance. Indeed, the authors prove that for a choice of prior π 0 a (µ a , σ a ) ∼ (σ a ) -1-2α with α < 0, Thompson Sampling is asymptotically optimal (i.e. its regret matches the lower bound of Theorem 1.2), whereas for α ≥ 0, the regret is not asymptotically logarithmic.

Bayes risk of Thompson Sampling

As discussed in Chapter 1, KL-UCB-H + is asymptotically optimal with respect to the Bayes risk, in the sense of the lower bound of [Lai, 1987]. Theorem 1.16 also shows that KL-UCB is close to optimal since its Bayes risk is within a multiplicative factor 2 of the lower bound. Compared to the experiments in Chapter 1, we propose here experiments for a larger horizon, including also Bayes-UCB and Thompson Sampling using the uniform prior. This large horizon does not allow for a comparison with FH-Gittins, that would be extremely heavy to implement.

For K = 5 arms (left plot in Figure 3.5) and K = 10 arms (right plot in Figure 3.5), for each algorithm we approximate the Bayes risk up to horizon T = 20000 by sampling N = 50000 bandit models with K arms from the prior distribution and playing the algorithm on each bandit model up to horizon T . As already observed on smaller horizons, KL-UCB + appears as a good anytime approximation of KL-UCB-H + . Meanwhile, the gap between KL-UCB and KL-UCB-H + increases, indicating KL-UCB might not be asymptotically optimal. As for the Bayesian algorithms, Thompson Sampling outperforms KL-UCB-H + and Bayes-UCB, despite its slightly larger Bayes risk, still appears as a fair competitor to KL-UCB-H + and KL-UCB + . We therefore conjecture that Bayes-UCB and Thompson Sampling are good approximations of the Bayesian optimal policy, at least with a uniform prior. Recently, [START_REF] Guha | Stochastic Regret Minimization via Thompson Sampling[END_REF] have investigated the Bayesian optimality of Thompson Sampling, showing that for two-armed bandits, with arbitrary prior Π 0 , for all T ,

E Π 0 N TS b (T ) ≤ 2 × min A E Π 0 N A b (T ) ,
where N A a (t) denotes the number of draws of arm a by the algorithm A up to time t and b denotes the (random) suboptimal arm. As the Bayes risk of an algorithm in that case is

E Π 0 [(µ * -µ b )N A b (T )]
, this result does not exactly state that the Bayes risk of Thompson Sampling is within a multiplicative factor 2 of the Bayes-risk of the Bayesian optimal solution, but it still provides good performance guarantees for Thompson Sampling in the Bayesian framework On the two bandit problems displayed in Figure 3.5, one can check that the prior-independent upper bound on the Bayes risk given by [START_REF] Bubeck | Prior-free and prior-dependent regret bounds for Thompson Sampling[END_REF] is quite pessimistic: 14 √ T K is more than one hundred times the actual regret obtained when there are ten arms. However, this upper bound holds for any bandit model with a finite number of arms and any prior distribution π 0 on reward distributions bounded in [0, 1], which includes more general situations with possibly correlated arms.

Thompson Sampling in more general frameworks

In the Bernoulli case, we have seen that Thompson Sampling is the easiest to implement asymptotically optimal policy. This computational advantage will be even stronger in more complex models, in which it might not be possible to design a UCB-like algorithm, or for which a complicated prior distribution is used, such that the associated posterior distributions can only be sampled from using MCMC simulation. In the latter case, an (approximate) implementation of Bayes-UCB needs several samples from the posterior distribution to estimate the quantiles, whereas Thompson Sampling only needs to produce one sample per round.

The performance of Thompson Sampling beyond one-parameter exponential family bandits has been recently investigated. [START_REF] Honda | Optimality of Thompson Sampling for Gaussian Bandits depends on priors[END_REF] propose the first analysis of Thompson Sampling in the particular case of Gaussian bandit models with unknown means and variances. They prove that this algorithm is asymptotically optimal for some choices of independent prior distributions (but not for all the possible choices of independent priors, as explained above). [START_REF] Bubeck | Prior-free and prior-dependent regret bounds for Thompson Sampling[END_REF] study Thompson Sampling for Gaussian bandits with known variance but for a special form of non-independent prior. For two-armed bandit models such that the means µ 1 , µ 2 are known up to a permutation, they show that Thompson Sampling has a finite regret. This regret has an optimal dependency in µ 1 -µ 2 , in a sense specified by [Bubeck et al., 2013a], who study the particular setting in which the mean of the best arm and (a lower bound on) the gap between the best and second best means are known. Recently, [START_REF] Gopalan | Thompson Sampling for Complex Online Problems[END_REF] have proposed the first analysis for Thompson Sampling that holds for quite general bandit problems. The authors derive a logarithmic upper bound on the regret that involves a (non-explicit) constant that captures correlations between arms. However, this bound holds for finitely supported arms and prior distribution, and it would be interesting to investigate whether these assumptions could be relaxed to recover the upper bound obtained in the Bernoulli case with independent arms. Thompson Sampling has also been successfully used for bandit problems with switching environments (see [START_REF] Mellor | Thompson Sampling in Switching Environments with Bayesian Online Change Point Detection[END_REF]), yet without theoretical guarantees.

As already mentioned in the Introduction, the good empirical performance of Thompson Sampling in contextual bandit models was known before any theoretical guarantee even in the Bernoulli case was available. Chapter 4 is dedicated to the presentation of contextual bandit models. We will notably review (and prove new) regret and Bayes risk upper bounds for Thompson Sampling in this setting. Thompson Sampling has also been successfully used in the more general framework of reinforcement learning. In model-based reinforcement learning, under some assumptions on the transition and reward functions in a Markov Decision Process, the goal is to design algorithms using estimates of these functions that act (almost) optimally in the MDP. Optimistic approaches (building set of statistically plausible MDPs and acting as in the best possible MDP) have been considered, leading to the UCRL2 ( [START_REF] Jaksch | Near-Optimal regret bounds for reinforcement learning[END_REF]) or KL-UCRL ( [Filippi et al., 2010a]) algorithms. [Strens, 2000] introduces the following algorithm, inspired by Thompson Sampling, that consists in several episodes. A prior distribution over the rewards and transitions is maintained. At the beginning of each episode a MDP is drawn from the current posterior distribution, the optimal policy for this sampled MDP is computed and played until the end of the episode. Recently [START_REF] Osband | More) Efficient Reinforcement Learning Via Posterior Sampling[END_REF] have provided the first (Bayesian) regret guarantees for this algorithm, while illustrating its practical performance. Other algorithms, sampling several MDPs at each round (or at the beginning of each episode) have been considered. [START_REF] Asmuth | A Bayesian sampling approach to exploration in reinforcement learning[END_REF] merge the samples obtained into a mixed MDP, whereas [START_REF] Fonteneau | An optimistic posterior sampling strategy for Bayesian reinforcement learning[END_REF] suggest to combine Thompson Sampling with the optimism principle: at each round, the best action in the best possible sampled MDP is chosen. 1 is fixed and the (θ 1 (s)) are then, when conditioned on S 1 (τ j ), an i.i.d. sequence with common distribution Beta(S 1 (τ j ) + 1, j -S 1 (τ j ) + 1). Thus, one can write

Elements of proof

P ((J ⊆ [τ j , τ j+1 [) ∩ (∀s ∈ J , θ 1 (s) ≤ µ 2 + δ)) = E E 1 (J ⊆[τ j ,τ j+1 [) 1 (∀s∈J ,θ 1 (s)≤µ 2 +δ) τ j , S 1 (τ j ) ≤ E E 1 (∀s∈J , θ1 (s)≤µ 2 +δ) τ j , S 1 (τ j )
where θ1 (s) is an i.i.d. sequence conditionally to S 1 (τ j ) with distribution Beta(S 1 (τ j )+1, j-S 1 (τ j )+1). Using that J is τ j -measurable,

P ((J ⊆ [τ j , τ j+1 [) ∩ (∀s ∈ J , θ 1 (s) ≤ µ 2 + δ)) ≤ E F Beta (S 1 (τ j )+1,j-S 1 (τ j )+1) (µ 2 + δ) J ≤ E 1 -F B (j+1,µ 2 +δ) (S 1 (τ j )) f (t)
,

where we use the link between the tail of Beta and Bernoulli distribution mentioned above and the fact that J ≥ f (t). It remains to upper bound this last expectation. An exact computation yields

E (1 -F B (j+1,µ 2 +δ) (S 1 (τ j ))) f (t) = j s=0 (1 -F B (j+1,µ 2 +δ) (s)) f (t) f B j,µ 1 (s)
To simplify notation, from now on let y = µ 2 + δ. Using, as [START_REF] Agrawal | Analysis of Thompson Sampling for the multi-armed bandit problem[END_REF], that

F B j+1,y (s) = (1 -y)F B j,y (s) + yF B j,y (s -1) ≥ (1 -y)F B j,y (s),
we get:

(1 -F B (j+1,y) (s)) f (t) ≤ exp -f (t)F B (j+1,y) (s) ≤ exp -f (t)(1 -y)F B (j,y) (s)
Therefore,

E (1 -F B (j+1,µ 2 +δ) (S 1 (τ j ))) f (t) ≤ j s=0 exp -f (t)(1 -y)F B (j,y) (s) f B j,µ 1 (s)
Using the fact that for s ≥ ⌈yj⌉, F B j,y (s) ≥ 1 2 (since the median of a binomial distribution with parameters j and y is ⌈yj⌉ or ⌊yj⌋), we get

E (1 -F B (j+1,µ 2 +δ) (S 1 (τ j ))) f (t) ≤ ⌊jy⌋ s=0 exp -f (t)(1 -y)F B (j,y) (s) f B j,µ 1 (s) + j s=⌈jy⌉ 1 2 (1-y)f (t) f B j,µ 1 (s) ≤ ⌊jy⌋ s=0 exp -f (t)(1 -y)F B (j,y) (s) f B j,µ 1 (s) E + 1 2 (1-y)f (t)
.

It is easy to show that for every λ > 1,∀x > 0, x λ exp(-x) ≤ λ e λ This allows us to upper-bound the exponential for all λ > 1, using C λ = λ e λ ,by:

(E) ≤ C λ (f (t)(1 -y)) λ ⌊jy⌋ s=0 f B j,µ 1 (s) F B (j,y) (s) λ ≤ C λ (f (t)(1 -y)) λ ⌊jy⌋ s=0 f B j,µ 1 (s) f B (j,y) (s) λ 114 CHAPTER 3. THOMPSON SAMPLING
Now, inspired by Agrawal and Goyal's work (proof of Lemma 3) we compute:

f B j,µ 1 (s) f B (j,y) (s) λ = j s µ s 1 (1 -µ 1 ) j-s j s λ (y λ ) s ((1 -y) λ ) j-s ≤ µ s 1 (1 -µ 1 ) j-s (y λ ) s ((1 -y) λ ) j-s = 1 -µ 1 (1 -y) λ j µ 1 (1 -y) λ y λ (1 -µ 1 ) s Let R λ (µ 1 , y) = µ 1 (1-y) λ y λ (1-µ 1 )
. There exists some

λ 1 > 1 such that, if λ < λ 1 , R λ > 1. More precisely, R λ > 1 ⇔ µ 1 1 -µ 1 > y 1 -y λ ⇔ ln µ 1 1 -µ 1 > λ ln y 1 -y and so λ 1 (µ 1 , y) = ⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ ln µ 1 1-µ 1 ln y 1-y if y > 1 2 +∞ if y < 1 2 For 1 < λ < λ 1 : ⌊jy⌋ s=0 f B j,µ 1 (s) (f B (j,µ 2 +δ) (s)) λ ≤ 1 -µ 1 (1 -y) λ j ⌊jy⌋ s=0 R s λ = 1 -µ 1 (1 -y) λ j R ⌊jy⌋+1 λ -1 R λ -1 ≤ 1 -µ 1 (1 -y) λ j R λ R λ -1 R jy λ = R λ R λ -1 1 -µ 1 (1 -y) λ j-jy µ 1 y λ jy = R λ R λ -1 e -jd λ (y,µ 1 )
where d λ (y, µ 1 ) = y ln y λ µ 1 + (1 -y) ln

(1-y) λ

1-µ 1 . Rearranging we can write

d λ (y, µ 1 ) = λ [y ln(y) + (1 -y) ln(1 -y)] -[y ln(µ 1 ) + (1 -y) ln(1 -µ 1 )]
which is an affine function of λ with negative slope (y ln(y) + (1 -y) ln(1 -y) < 0 for all y ∈ (0, 1)) and d 1 (y, µ 1 ) = K (y, µ 1 ) > 0. Hence, for fixed 0 < y < µ 1 ≤ 1 this function is positive whenever

λ < y ln(µ 1 ) + (1 -y) ln(1 -µ 1 ) y ln(y) + (1 -y) ln(1 -y) =∶ λ 2 (µ 1 , y).
Clearly, λ 2 (µ 1 , y) > 1 and we choose λ 0 = min(λ 1 , λ 2 ). After some calculation one can show that λ 2 ≤ λ 1 , and therefore that

λ 0 (µ 1 , µ 2 ) = λ 2 (µ 1 , µ 2 + δ) = 1 + K(µ 2 + δ, µ 1 ) (µ 2 + δ) ln 1 µ 2 +δ + (1 -µ 2 -δ) ln 1 1-µ 2 -δ
.

To obtain the constants used in the statement of the lemma we define

d λ,µ 1 ,µ 2 ∶= d λ (y, µ 1 ) C λ,µ 1 ,µ 2 ∶= C λ 0 (1 -µ 2 -δ) -λ R λ 1 -R λ .
This concludes the proof.

Proof of Lemma 3.9

Let a ∈ {2, . . . K}. Recall that C a = 32 ∆ 2 a and δ a = ∆a 2 .

P (∃s ≤ t ∶ θ a (s) > µ a + δ a , N a (s) > C a log t) ≤ P ∃s ≤ t ∶ S a (t) N a (t) > µ a + δ a 2 , N a (s) > C a log t A + P ∃s ≤ t ∶ θ a (s) > S a (t) N a (t) + δ a 2 , N a (s) > C a log t B
Term A is easily upper bounded using a union bound and Hoeffding inequality:

(A) ≤ t s=⌈Ca log t⌉ P ∑ s k=1 Y a,s s > µ a + δ a 2 ≤ t s=⌈Ca log t⌉ exp - 2sδ 2 a 4 ≤ t exp - C a log t∆ 2 a 8 = 1 t 3 ≤ 1 t 2 .
Term B is upper bounded as follows:

(B) ≤ t s=⌈Ca log t⌉ s r=1 P θ a (s) > r s + δ a 2 S a (s) = r, N a (s) = s P(S a (t) = r, N a (t) = s) = t s=⌈Ca log t⌉ s r=1 1 -F Beta r+1,s-r+1 r s + δ a 2 P(S a (t) = r, N a (t) = s) = t s=⌈Ca log t⌉ s r=1 F Bin s+1, r s + δa 2 (r)P(S a (t) = r, N a (t) = s)
Introducing a sequence (Z i ) of i.i.d Bernoulli random variables with mean r s + δ a , one has

F Bin s+1, r s + δa 2 (r) = P s+1 i=1 Z i < r ≤ P s+1 i=1 Z i - r s - δ a 2 ≤ - δ a 2 (s + 1) ≤ exp -2C a log t δ 2 a 4 ≤ 1 t 3 .
Hence,

(B) ≤ t s=⌈Ca log t⌉ s r=1 1 t 3 P(S a (s) = r, N a (s) = s) = t s=⌈Ca log t⌉ 1 t 3 ≤ 1 t 2 .
Finally a union bound over the arm yields Lemma 3.9.

y t = f (x t ) + t , (4.1)
where f ∶ R d → R is a (unknown) real-valued function and t some centered noise. The agent aims at maximizing the sum of his rewards.

The context set D t can be seen as a set of structured actions that evolves over time. A more general contextual bandit model involves both actions (in an action set A) and contexts (in a context set C). At each time t, a context c t ∈ C is revealed, the agent chooses an action a t ∈ A an receives a reward

y t = g(c t , a t ) + t .
In the online advertisement application, the context c t could be a feature vector relative to the t th user of the website, while action a corresponds to some advertisement that could be shown to him (a could be itself a feature vector for this add). It is reasonable to assume that some feature vector for the pair user/add x a t = φ(c t , a), is built for each add a available and that there exists f such that g(c, a) = f (φ(c, a)). In that case, we are in the model (4.1) with D t = (x a t ) a∈A . For example, [START_REF] Chapelle | Simple and scalable response prediction for display advertising[END_REF] suggest that such coupled features are used in some add prediction systems. Besides, if the users can be clustered into different categories, one can assume that in each category, the model (4.1) holds, with D t the set of feature vectors of adds available for display at time t. In the rest of this chapter, we will study only the model (4.1).

Several assumptions on the function f have been considered in the literature. If f is a linear function, that is f (x) = x T θ with θ ∈ R d some unknown vector we are in a linear contextual bandit model, first considered by [Auer, 2002] (under the name 'associative reinforcement learning with linear value function'). [START_REF] Filippi | Parametric Bandits : The Generalized Linear case[END_REF] consider the richer generalized linear bandit model for which f (x) = µ(x T θ), where µ is some link function and θ ∈ R d some unknown parameter. [START_REF] Valko | Finite-time analysis of kernelized contextual bandits[END_REF] consider kernelized contextual bandit models for which f (x) = φ(x) T θ, where x belongs to some set X , φ ∶ X → H is a mapping to some Hilbert space H, and θ ∈ H. In the rest of this chapter, we will consider only the linear case, and present -as for classical bandit models in previous chaptersoptimistic approaches and Bayesian alternatives for the linear contextual bandit problem.

Contextual linear bandits. The contextual model mostly considered in this chapter is the following. Let θ be a parameter in R d . At time t, the agent chooses a context x t ∈ D t based on past observations, according to his strategy (or bandit algorithm) A, and receives a reward

y t = x T t θ + t .
The σ-field H t = σ(D 1 , x 1 , y 1 , . . . , D t , x t , y t , D t+1 ) represents the information available at the end of round t: contexts chosen and rewards observed up to the end of round t, as well as the new set of contexts

x * t = argmax x∈Dt x T θ
and the agent aims at minimizing the following random quantity, called pseudo-regret:

1 R θ (T, A) = T t=1 r t where r t = (x * t ) T θ -x T t θ.
In the literature, this model has been often introduced in the static case in which ∀t, D t = D (as in the papers by [Dani et al., 2008, Rusmevichientong andTsitsiklis, 2010] for example). In Chapter 2, Bayes-UCB was also applied to a linear contextual bandit problem with a static, finite set of contexts D = {U 1 , . . . , U K }. We call this problem a 'linear bandit problem', reserving the adjective 'contextual' to changing sets of contexts.

Depending on whether the regression parameter θ is regarded as an unknown parameter or is assumed to be drawn from a prior distribution π 0 on R d , one may consider two different probabilistic frameworks. We denote by P,E the probability and expectation under the Bayesian model (with an implicit dependency on the prior distribution π 0 ) and by P θ , E θ the probability and expectation in the frequentist framework (that is, conditionally to θ). As for classical bandits, one defines Bayesian algorithms to be algorithms using at round t the posterior distribution on θ to make a decision. Independently, for any algorithm, Bayesian or 'frequentist' (i.e. that does not use a prior in its routine) one can carry out two types of analyses:

-a frequentist analysis bounds the pseudo-regret of an algorithm either in probability or in expectation, conditionally to θ. For example one can bound the regret (or expected regret) defined by

R θ (T, A) = E θ [R θ (T, A)]
-a Bayesian analysis bounds the pseudo-regret of an algorithm in the Bayesian modeling (including an average over the prior π 0 ). For example one can bound the Bayes risk, defined by

BR π 0 (T, A) = E[R θ (T, A)] = E[R θ (T, A)].
The goal of this chapter is to present the implementation of Bayes-UCB and Thompson Sampling in linear contextual bandit models, as well as Bayesian and frequentist analyses of these two algorithms. We make the following classical assumptions:

Assumption 1. The contexts are bounded: there exists

L > 0 such that ∀t ∈ N, ∀x ∈ D t , x 2 ≤ L. Assumption 2.
The noise is centered and σ 2 -subgaussian :

E θ [η t H t-1 ] = 0 and ∀λ > 0, E θ [e ληt H t ] = E[e ληt θ, H t ] ≤ e λ 2 σ 2 2 .
The Bayesian analyses presented here hold under an additional assumption presented below: Gaussian prior and Gaussian noise.

1. In some papers, pseudo-regret may be called regret, and what we call regret should be called 'expected regret'. We chose these denominations to be consistent with what we call regret in previous chapters: for classical bandits, R θ (T, A) is the expectation of the pseudo-regret ∑ t t=1 (µ *µ A t ).

CHAPTER 4. BAYESIAN ALGORITHMS FOR LINEAR CONTEXTUAL BANDITS

Contextual linear bandit with Gaussian prior. A natural prior distribution on the parameter θ is a Gaussian prior with covariance κ 2 I d ; that is θ ∼ N 0, κ 2 I d . Indeed, under this prior distribution, if one assumes that the noise is Gaussian with known variance, the posterior distribution is explicitly computable. Introducing for every t ≥ 1 the matrix and vectors

X t = ⎛ ⎜ ⎜ ⎜ ⎝ x T 1 x T 2 . x T t ⎞ ⎟ ⎟ ⎟ ⎠ ∈ M t,d (R), Y t = ⎛ ⎜ ⎜ ⎜ ⎝ y 1 y 2 . y t ⎞ ⎟ ⎟ ⎟ ⎠ ∈ R t , and E t = ⎛ ⎜ ⎜ ⎜ ⎝ 1 2 . t ⎞ ⎟ ⎟ ⎟ ⎠ ∈ R t , one has Y t = X t θ + E t .
If the noise is such that t ∼ N 0, σ 2 , the posterior distribution on θ at the beginning of round

t + 1 is p(θ H t ) = N θ(t), Σ t where θ(t) = (B(t)) -1 X T t Y t with B(t) = σ 2 κ 2 I d + X T t X t Σ t = σ 2 (B(t)) -1 .
The posterior mean θ(t) is the regularized least-square estimator of θ with regularization parameter λ = σ 2 κ 2 .

The Bayes-UCB, Bayes-LinUCB and Thompson Sampling algorithms presented in Sections 4.3 and 4.4 use the Gaussian prior distribution defined above. It is sometimes possible to implement (approximations of) these algorithms using a more general prior distribution π 0 by resorting to MCMC simulation when there is no close form for the posterior distribution, as illustrated in Chapter 2 for Bayes-UCB with a sparsity-inducing prior. For our Bayesian analyses, Assumption 2 above will be replaced by the assumption of a Gaussian noise with variance σ 2 :

y t = θ T x t + t , with θ ∼ N 0, κ 2 I d and t ∼ N 0, σ 2 . (4.2)
Optimistic approaches and related works. Optimistic algorithms for linear contextual bandits build a confidence region C t in R d for the unknown parameter θ and choose

x t+1 = argmax x∈D t+1 max θ ′ ∈Ct x T θ ′ . (4.3)
This optimism-in-face-of-uncertainty principle is implemented in the algorithms proposed by [Auer, 2002, Dani et al., 2008, Rusmevichientong and Tsitsiklis, 2010, Chu et al., 2011, Abbasi-Yadkori et al., 2011], with successive refinement on how to build the confidence region C t . Some of the confidence regions used are built using a (potentially regularized) least-square estimate of θ. Keeping the notation B(t), Σ t and θ(t) of the previous section (with κ = +∞ allowed when there is no regularization), some confidence region used are of the form

C t = θ ′ ∈ R d ∶ θ(t) -θ ′ Σ -1 t ≤ β(t + 1, δ) ,
where we recall that v A = √ v T Av is the L 2 -norm associated to the matrix A. For C t of the following form, the maximum in (4.3) can be computed explicitly and the algorithm rewrites

x t+1 = argmax x∈D t+1 θ(t) T x + x Σt β(t + 1, δ) .

BAYESIAN AND FREQUENTIST CONFIDENCE REGIONS 121

When the number of contexts is finite, the above algorithm appears as an index policy. It can also be implemented when D t is infinite and convex as the maximization of a convex differentiable function. When D t is a polytope, [START_REF] Dani | Stochastic Linear Optimization under Bandit Feedback[END_REF] suggest to use L 1 confidence regions.

Among theses optimistic algorithms, the one using the tightest confidence region is the OFUL algorithm of [START_REF] Abbasi-Yadkori | Improved Algorithms for Linear Stochastic Bandits[END_REF], which picks at time t

x t+1 = argmax x∈D t+1 ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ θ(t) T x + x Σt ⎛ ⎜ ⎝ 2 log 1 δ + d log 1 + (t + 1) L 2 κ 2 dσ 2 + 1 κ θ ⎞ ⎟ ⎠ ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦
.

OFUL was introduced in a frequentist setting, in which κ is no longer seen as the parameter of a prior distribution but is a parameter of the algorithm such that σ 2 κ 2 is the regularization coefficient in the regularized least square estimate θ(t).

[ [START_REF] Abbasi-Yadkori | Improved Algorithms for Linear Stochastic Bandits[END_REF] propose a frequentist analysis of OFUL under Assumption 1 and 2 and the additional assumption that for all t ∈ N * , for all x ∈ D t , x T θ ≤ 1. They show that, for a fixed parameter θ, the pseudo-regret of OFUL is of order Õ(d √ T ) with high probability. The Õ notation means that we ignore logarithmic factors in T . In the static case, that is when D t = D, there exists some (worst-case) lower bounds on the regret and Bayes risk, under specific assumptions on the set D. For D the unit sphere in R d , [START_REF] Rusmevichientong | Linearly Parameterized Bandits[END_REF] show that there exists a prior distribution π 0 (namely a Gaussian prior with covariance matrix 1 d I d ) such that every bandit algorithm A satisfies BR π 0 (T, A) ≥ 0.006d √ T . Consequently, for every algorithm A, there exists [START_REF] Dani | The Price of Bandit Information in Online Optimization[END_REF] show that for any bandit algorithm A, there exists a vector θ and a centered noise t such that R θ (T, A) is at least of order d √ T . In this example, d √ T = O( dT log(K)), and more generally when the number of contexts is finite, one may expect to have an upper bound on the regret or Bayes risk that scales in dT log(K) in place of d √ T . This has been shown to be possible in a more general, adversarial, setting. In an adversarial linear bandit problem (rather called bandit linear optimization in the literature), at each time t the agent chooses a context x t ∈ D while an 'adversary' simultaneously chooses a vector l t ∈ R d . The agent receives the rewards x T t l t and wants to minimize its pseudo-regret, defined in this setting by

θ ∈ R d such that in the associated linear bandit model R θ (T, A) ≥ 0.006d √ T . When D is the hypercube {0, 1} d , of cardinal K = 2 d ,
R(T, A) = max x∈D E T t=1 x T l t -E T t=1 x T t l t .
No stochastic assumptions are made on l t unlike in the (stochastic) linear bandit problem considered so far. When the number of contexts in D is finite, algorithms for online linear optimization mostly consist in adapting the EXP3 algorithm of [Auer et al., 2002b] suited for classical adversarial bandits (see e.g. [Bubeck and Cesa-Bianchi, 2012]) by incorporating estimates of l t at each step. Among them, EXP2 with John exploration, proposed by [Bubeck et al., 2012] can be applied to the stochastic linear contextual bandit problem with a static, finite set of context D and is such that R θ (T, A) ≤ 2 √ 3 dT log(K). However, this algorithm is difficult to implement. We will show below that the pseudo-regret of Bayes-UCB and Thompson Sampling also scales in Õ(d √ T ) or Õ( dT log(K)), whichever is smaller.

Bayesian and frequentist confidence regions

Confidence regions on θ in R d but also confidence intervals (in R) on the mean of each context x T θ are necessary to define and analyse optimistic algorithms, like the algorithms mentioned above or the two variants of Bayes-UCB presented in the next section. In Lemma 4.1, we recall a confidence region given by [START_REF] Abbasi-Yadkori | Improved Algorithms for Linear Stochastic Bandits[END_REF]. This confidence region was obtained using the so-called 'method of mixtures', that we discuss in Appendix A. One can find in Section A.3 therein the deviation inequality for vector-valued martingales proved by [START_REF] Abbasi-Yadkori | Improved Algorithms for Linear Stochastic Bandits[END_REF] that allows to show the following lemma.

Lemma 4.1. Under Assumptions 1 and 2, for any algorithm A,

P θ ⎛ ⎜ ⎝ ∀t ∈ N, θ -θ(t) Σ -1 t ≤ ⎛ ⎜ ⎝ 2 log 1 δ + d log 1 + (t + 1) L 2 κ 2 dσ 2 + 1 κ θ ⎞ ⎟ ⎠ ⎞ ⎟ ⎠ ≥ 1 -δ.
Proof. Introducing the martingale S t = X T t E t = ∑ t s=1 s x s , and letting V t = B(t), one can write

θ(t) -θ = V -1 t X T t Y t -θ = V -1 t X T t X t θ + V -1 t X T t E t -θ = - σ 2 κ 2 V -1 t θ + V -1 t S t θ(t) -θ Σ -1 t = σ -1 θ(t) -θ Vt ≤ σ -1 S t V -1 t + σ κ 2 θ V -1 t . S t = ∑ t s=1 s
x s is such that the x s are F s-1 measurable and s is centered and σ-subgaussian. Thus the deviation inequality of Lemma A.8 in Appendix A can be applied with V = σ 2 κ 2 I d . With probability larger than 1 -δ, for all t ∈ N,

θ(t) -θ Σ -1 t ≤ 2 log 1 δ + log det(V t ) det σ 2 κ 2 I d + 1 κ θ .
Following Lemma 10 of [START_REF] Abbasi-Yadkori | Improved Algorithms for Linear Stochastic Bandits[END_REF], the determinant of V t is bounded as

det(V t ) ≤ σ 2 κ 2 + t L 2 d d , (4.4)
which concludes the proof.

◻

Lemma 4.1 also leads to a confidence intervals on x T θ since from the Cauchy-Schwarz inequality

θ T x -θ(t) T x ≤ θ -θ(t) Σ -1 t x Σt .
This confidence region on θ given in Lemma 4.1 holds conditionally to θ and is therefore a 'frequentist' confidence region. It is used to define and analyse the OFUL algorithm [START_REF] Abbasi-Yadkori | Improved Algorithms for Linear Stochastic Bandits[END_REF]) as well as in the frequentist analysis of a version of Thompson Sampling proposed by [START_REF] Agrawal | Thompson Sampling for Contextual Bandits with Linear Payoffs[END_REF]. Integrating over the prior yields a Bayesian confidence region, that is used by [Russo and Van Roy, 2014] in a general Bayesian analysis of Thompson Sampling.

The Bayesian analysis we present here for Thompson Sampling and Bayes-UCB are based on the Bayesian confidence regions we introduce here in a model with Gaussian noise and Gaussian prior.

BAYESIAN AND FREQUENTIST CONFIDENCE REGIONS 123

Lemma 4.2. Under the Bayesian model (4.2), for any algorithm A,

P θ -θ(t) Σ -1 t ≤ Q 1 -δ; χ 2 d ≥ 1 -δ. (4.5)
Moreover, if at each round D t = K, for any algorithm A, for all t ≥ 1,

P ∀x ∈ D t+1 , x T θ -x T θ(t) ≤ x Σt Q 1 - δ 2K ; N (0, 1) ≥ 1 -δ. (4.6)
where Q(α, π) is the quantile of order α of the distribution π.

Proof. We first write

θ -θ(t) 2 Σ -1 t = (θ -θ(t)) T Σ -1 t (θ -θ(t)) = [(θ -θ(t)) T Σ -1 2 t ]Σ -1 2 t (θ -θ(t)) = Σ -1 2 t (θ -θ(t)) 2 . Given H t , θ has distribution N θ(t), Σ t , hence Σ -1 2 t (θ -θ(t)) ∼ N (0, I d ) and Σ -1 2 t (θ -θ(t)
) 2 follows a chi-square distribution with d degrees of freedom. We have shown that

θ -θ(t) 2 Σ -1 t H t ∼ χ 2 d .
Hence, we have, by definition of the quantile, that

P θ -θ(t) Σ -1 t ≤ Q 1 -δ; χ 2 d H t = P θ -θ(t) 2 Σ -1 t ≤ Q 1 -δ; χ 2 d H t ≥ 1 -δ
and inequality (4.5) follows in conditioning.

In the particular case where D t = K for all t ∈ N, we can introduce an (arbitrary) ordering of the contexts and write D t = (b 1 (t), . . . , b K (t)). Then, for each context, we use that conditionally to

H t , b i (t + 1) T θ ∼ N b i (t + 1) T θ(t), b i (t + 1) Σt . Thus P ∃x ∈ D t+1 , x T θ -x T θ(t) > x Σt Q 1 - δ 2K ; N (0, 1) ≤ K i=1 P b i (t + 1) T θ -b i (t + 1) T θ(t) > b i (t + 1) Σt Q 1 - δ 2K ; N (0, 1) = K i=1 E P b i (t + 1) T θ -b i (t + 1) T θ(t) > b i (t + 1) Σt Q 1 - δ 2K ; N (0, 1) H t = K i=1 δ K = δ
which yields inequality (4.6).

◻ 4.3 The Bayes-UCB algorithm and a generalization

The algorithms

The Bayes-UCB algorithm presented in Chapter 2 picks at time t the arm whose quantile of order 1 -1 t of the marginal posterior distribution on the mean is maximal. Ignoring dependencies among the arms, we explained in Chapter 2 how this algorithm could be applied in a (static) linear bandit model. Bayes-UCB can also naturally be extended to a linear contextual bandit (with changing contexts). To obtain theoretical guarantees, however, we need to consider a different level of confidence, and 1 t is replaced in the definition below by e -f (t,δ) , for some exploration rate f (t, δ). The Bayes-UCB algorithm using the Bayesian model (4.2) chooses at time t + 1 t+1,δ) ; N (0, 1) , where f (t, δ) is some exploration rate. Indeed, the posterior distribution on the mean x T θ of arm x at the beginning of round t + 1 is N x T θ(t), x Σt .

x t+1 = argmax x∈D t+1 Q 1 -e -f (t+1,δ) ; N x T θ(t), x Σt , x t+1 = argmax x∈D t+1 x T θ(t) + x Σt Q 1 -e -f (
Bayes-UCB can be seen as a variant of a UCB-type algorithm that uses Bayesian confidence regions on the mean of each arm. When we go from classical bandit to linear bandits, the optimism-in-face-ofuncertainty principle is used differently : one rather picks the context for which there exists a 'possible' regression parameter θ (in the sense that it lies in some confidence region) for which the associated mean x T θ is the highest among all contexts and all 'possible' regression parameters (see (4.3)). A natural Bayesian optimistic algorithm for the linear contextual bandit problem consists therefore in applying this principle with a Bayesian confidence region. It follows from inequality (4.5) that t+1,δ) .

P θ -θ(t) Σ -1 t ≤ Q 1 -e -f (t+1,δ) ; χ 2 d ≥ 1 -e -f (
We define the Bayes-LinUCB algorithm to be the algorithm of the form (4.3) using the Bayesian confidence region

C t = θ ′ ∶ θ ′ -θ(t) Σ -1 t ≤ Q 1 -e -f (t+1,δ) ; χ 2 d .
As explained above, using a bit of algebra one can show Bayes-LinUCB picks at time t + 1 the context

x t+1 = argmax x∈D t+1 θ(t) T x + x Σt Q 1 -e -f (t+1,δ) ; χ 2 d .

Bayesian analysis of Bayes-UCB and Bayes-LinUCB

Bayes-UCB and Bayes-LinUCB are algorithms of the form

x t+1 = argmax x∈D t+1 θ(t) T x + x Σt β(t + 1, δ) .
(4.7)

One of the first analyses proposed for an algorithm of this form was that of ConfidenceBall given by [START_REF] Dani | Stochastic Linear Optimization under Bandit Feedback[END_REF]. A closer examination of their analysis shows that its cornerstone can be summarized in Lemma 4.3 below, whose proof is given in Section 4.6.1. This Lemma is also implicitly used in the analysis of the Uncertainty Ellipsoid algorithm of [START_REF] Rusmevichientong | Linearly Parameterized Bandits[END_REF] and of the OFUL algorithm of [START_REF] Abbasi-Yadkori | Improved Algorithms for Linear Stochastic Bandits[END_REF].

THE BAYES-UCB ALGORITHM AND A GENERALIZATION 125

Lemma 4.3. For an algorithm A that picks at time t + 1 the context x t+1 according to (4.7), on the event

E = ⋂ t∈N ∀x ∈ D t+1 , x T θ -x T θ(t) ≤ x Σt β(t + 1, δ)
the pseudo-regret of A satisfies, under Assumption 1,

∀T ∈ N * , R θ (A, T ) ≤ β(T, δ) √ T d C 1 log 1 + T L 2 κ 2 σ 2 d , with C 1 ∶= 4L 2 κ 2 log(1+L 2 κ 2 σ -2 ) .
This result states that on some event E, the pseudo-regret R θ (T, A) is upper bounded by a deterministic quantity. For the OFUL algorithm, Lemma 4.1 shows that the event E in Lemma 4.3 holds with probability at least 1-δ, leading to a high-probability upper bound on the pseudo-regret in the frequentist framework. Upper bounding the pseudo-regret of Bayes-UCB or Bayes-LinUCB with high probability, then boils down to choosing the exploration rate f (t, δ) in such a way that the probability of the event E associated is larger than 1 -δ. Using the Bayesian confidence regions given in Lemma 4.2, and upper bounds on the quantiles of the normal and chi-square distribution leads to PAC-Bayesian bounds on the pseudo-regret stated in Theorem 4.4.

Up to logarithmic factor in T , the pseudo-regret of Bayes-UCB when there is a finite number of contexts K is of order Õ( dT log(K)), in high probability under the Bayesian model (4.2). For Bayes-LinUCB, suited for the general case (with a potentially infinite set of context D t ), the pseudo-regret is of order Õ(d √ T ), also with high probability.

Theorem 4.4. Choosing f (t, δ) = log Kπ 2 t 2 3δ , when the number of context is finite and D t ≤ K, the Bayes-UCB algorithm satisfies, under Assumption 1 and the Bayesian model (4.2)

P ⎛ ⎜ ⎝ ∀T ∈ N, R θ (T, A) ≤ √ T d 2C 1 log Kπ 2 T 2 6δ log 1 + T L 2 σ 2 dκ 2 ⎞ ⎟ ⎠ ≥ 1 -δ.
With the exploration rate f (t, δ) = log π 2 t 2 6δ , the Bayes-LinUCB algorithm satisfies

P ⎛ ⎜ ⎜ ⎝ ∀T ∈ N, R θ (T, A) ≤ d √ T C 1 log 1 + T L 2 σ 2 dκ 2 ⎛ ⎝ 1 + 2 d log π 2 T 2 6δ + 2 1 d log π 2 T 2 6δ ⎞ ⎠ ⎞ ⎟ ⎟ ⎠ ≥ 1 -δ.
where C 1 is the constant introduced in Lemma 4.3.

Proof of Theorem 4.4 Assume that ∀t ∈ N, D t = K. With f (t, δ) = log Kπ 2 t 2 3δ
, Bayes-UCB picks at time t + 1 the context x t+1 according to the Equation (4.7) with β(t, δ) = Q 1 -3δ Kπ 2 t 2 ; N (0, 1) . Let E be the event

E = ⋂ t∈N ∀x ∈ D t+1 , x T θ -x T θ(t) ≤ x Σt Q 1 - 3δ Kπ 2 (t + 1) 2 ; N (0, 1) . Inequality (4.6) yields, for all t ∈ N * P ∀x ∈ D t , x T θ -x T θ(t -1) ≤ x Σ t-1 Q 1 - 6δ 2Kπ 2 t 2 ; N (0, 1) ≥ 1 - 6δ π 2 t 2
and an union bound on t ≥ 1 yields P(E) ≥ 1 -δ. On E, from Lemma 4.3 the pseudo-regret is upper bounded for all T as follows:

R θ (T, A) ≤ Q 1 - 3δ Kπ 2 T 2 ; N (0, 1) √ T d C 1 log 1 + T L 2 κ 2 σ 2 d ≤ √ T d 2 log Kπ 2 T 2 6δ C 1 log 1 + T L 2 κ 2 σ 2 d ,
using a classic upper bound on the tail of a Gaussian distribution: if X ∼ N (0, 1) it can be shown that

P(X > c) ≤ (1 2)e -c 2 2 .
Bayes-LinUCB using the exploration rate stated above picks at time t + 1 the context x t+1 according to (4.7) with

β(t, δ) = Q 1 -6δ π 2 t 2 ; χ d 2 . Introducing E = ⋂ t∈N ⎛ ⎝ ∀x ∈ D t+1 , x T θ -x T θ(t) ≤ x Σt Q 1 - 6δ π 2 t 2 ; χ d 2 ⎞ ⎠ ,
inequality (4.5) and a union bounds also yields P(E) ≥ 1 -δ. By Lemma 4.3, on E one has

∀T ∈ N * , R θ (T, A) ≤ Q 1 - 6δ π 2 T 2 ; χ 2 d √ T d C 1 log 1 + T L 2 κ 2 σ 2 d .
Inequality (4.3) in [START_REF] Laurent | Adaptive estimation of a quadratic functional by model selection[END_REF] provides an upper bound of the quantile of a chi-square distribution, namely

Q 1 -α; χ 2 d ≤ d + 2 log 1 α + 2 d log 1 α . (4.8)
Using this inequality, on event E,

R θ (T, A) ≤ d + log π 2 T 2 6δ + 2 d log π 2 t 2 6δ , √ T d C 1 log 1 + T L 2 κ 2 σ 2 d
which concludes the proof.

◻

Comparison with other optimistic algorithms

A linear bandit model with Gaussian prior can also be seen as the simplest example of Gaussian process bandit models as presented by [START_REF] Srinivas | Gaussian Process Optimization in the Bandit Setting : No Regret and Experimental Design[END_REF], Srinivas et al., 2012]. Indeed, the model y t = x T t θ + t with the assumption that θ ∼ N 0, κ 2 I d is equivalent to the model y t = f (x t ) + t with the assumption that f is sampled from a Gaussian process ∼ GP (0, k(x, x ′ )) where k is a linear kernel on D such that k(x, x ′ ) = κ 2 x T x ′ . [START_REF] Srinivas | Gaussian Process Optimization in the Bandit Setting : No Regret and Experimental Design[END_REF]] also make the assumption that the noise is Gaussian with variance σ 2 and introduce the GP-UCB algorithm, that chooses at time t + 1, when D t+1 = K,

x t+1 = argmax x∈D t+1 ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ θ(t) + x Σt 2 log K(t + 1) 2 π 2 6δ ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ . Algorithm Value of β(t, δ) Bayes-UCB * Q 1 -3δ Kπ 2 t 2 ; N (0, 1) Bayes-LinUCB Q 1 -6δ π 2 t 2 ; χ 2 d GP-UCB * 2 log Kt 2 π 2 6δ OFUL 2 log 1 δ + d log 1 + t L 2 κ 2 dσ 2 + 1 κ θ
Table 4.1: Index for various policies, some of them * suited for D t = K.

While the original version of GP-UCB was suited for a static context set D, the authors later proposed a contextualized version ( [START_REF] Krause | Contextual Gaussian Process Bandit Optimization[END_REF]). With this particular linear kernel, GP-UCB is very close to Bayes-UCB: more precisely, since 2 log (Kπ 2 t 2 (6δ)) is a tight upper bound on the quantile Q 1 -3δ (Kπ 2 t 2 ); N (0, 1) used by the provably efficient version of Bayes-UCB given in Theorem 4.4. [START_REF] Srinivas | Gaussian Process Optimization in the Bandit Setting : No Regret and Experimental Design[END_REF] also propose an algorithm in the case when D is a compact set, based on a discretization argument and on regularity properties of the Gaussian process. Bayes-LinUCB on the contrary can be implemented for any set D and is based on the simple Bayesian confidence region given by (4.5).

Optimistic algorithms for linear contextual bandits presented so far appear as generalized index policies, of the form

x t+1 = argmax x∈D t+1 θ(t) T x + x Σt β(t + 1, δ) ,
with different exploration rates β(t, δ), that we recall in Table 4.1. From the analysis we gave, based on Lemma 4.3, the smaller β(t, δ), the smaller the upper bound on the regret obtained. However, the way the different values of β(t, δ) in Table 4.1 compare is not obvious. As just mentioned, the exploration rate of Bayes-UCB is always smaller than that of GP-UCB. Among the other three algorithms, one would expect the exploration of Bayes-UCB to be smaller than that of Bayes-LinUCB on bandit models with finite number of contexts such that log(K) ≤ d. Figure 4.1 illustrates this tendency: for a bandit model with log(K) ≃ 7, we show that the exploration rate of Bayes-UCB is smaller than that of Bayes-LinUCB when d = 5, and larger when d = 10. In these cases, both exploration rates are smaller than that of OFUL. The original Bayes-UCB algorithm (see Chapter 2) is proved optimal for classical bandits with Bernoulli rewards in a frequentist sense: the algorithm matches the problem-dependent lower bound on the regret given by [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] for every Bernoulli bandit model. In the same spirit, we could expect to be able to give a frequentist analysis of Bayes-UCB (or Bayes-LinUCB) in the more general linear contextual bandit problem studied here. As already explained, most of analyses of algorithms of the form (4.7) rely on Lemma 4.3. For example, with a fixed number of contexts K, frequentist guarantees for Bayes-UCB would rely on an upper bound for the probability

P θ ∃t ∈ N * , ∃x ∈ D t+1 , x T θ -x T θ(t) > x Σt Q 1 - π 2 δ 6Kt 2 ; N (0, 1) .
However, the best deviation inequality currently available in the frequentist setting is the one deduced from Lemma 4.1:

P θ ⎛ ⎜ ⎝ ∃t ∈ N, ∃x ∈ D t+1 ∶ θ T x -θ(t) T x > x Σt ⎛ ⎜ ⎝ 2 log 1 δ + d log 1 + (t + 1) L 2 κ 2 dσ 2 + 1 κ θ ⎞ ⎟ ⎠ ⎞ ⎟ ⎠ ≤ δ.
Even for a fixed context x, it is not known whether one can prove a deviation inequality of the form

P θ ∃t ∈ N * θ T x -θ(t) T x > x Σt β(t + 1, δ) ≤ δ (4.9)
with an exploration rate β(t, δ) independent on the dimension d.

Thompson Sampling

The algorithm

Thompson Sampling in linear contextual bandit models is easy to implement. Given a Bayesian model such that samples from the posterior distributions on θ at each round can be computed, the algorithm draws at round t + 1 a sample θ(t) from the posterior distribution at the end of round t and picks the context x t+1 according to

x t+1 = argmax x∈D t+1 x T θ(t).
This ensures that the context chosen has the highest posterior probability of being the best context. Whereas optimistic algorithms reduce to solving the optimization problem (4.7), Thompson Sampling only requires to solve a linear optimization problem, which is in general easier.

Recently, Bayesian and frequentist guarantees for Thompson Sampling in linear contextual bandits have emerged in the literature. On the one hand [Russo and Van Roy, 2014] propose a general Bayesian analysis of Thompson Sampling that can be applied to linear contextual bandits when the prior distribution is such that θ remains bounded. On the other hand, [START_REF] Agrawal | Thompson Sampling for Contextual Bandits with Linear Payoffs[END_REF] propose the first frequentist analysis of Thompson Sampling with a Gaussian prior. They give a high-probability upper bound on the pseudo-regret of Thompson Sampling for a specific choice of prior (and likelihood), that depends on δ and on assumptions on the noise.

As in the rest of the chapter, we focus here on Thompson Sampling with Gaussian prior. We review existing results from [Russo and Van Roy, 2014] and [START_REF] Agrawal | Thompson Sampling for Contextual Bandits with Linear Payoffs[END_REF] and additionally propose a new Bayesian analysis suited for a potentially infinite set of contexts D t , a case not completely covered by the work of [Russo and Van Roy, 2014]. Russo and Van Roy, 2014] propose a Bayesian analysis of Thompson Sampling for a wide class of models, including linear contextual bandits. Their analysis holds for quite general prior distributions π 0 and uses the fact that conditionally to the history H t , θ and the sample θ(t) used in the algorithm have the same distribution, and so do x t+1 and x * t+1 . This remark leads to a Bayes risk decomposition extensively used by [Russo and Van Roy, 2014], that rewrites for linear bandits

A Bayesian analysis of Thompson Sampling

[

BR π 0 (T, TS π 0 ) = E[R θ (T, TS π 0 )] ≤ E T t=1 [U t-1 (x t ) -θ T x t ] + E T t=1 [θ T x * t -U t-1 (x * t )], (4.10)
where (U t ) is any sequence of confidence bounds. The Bayes-risk bound sketched in [Russo and Van Roy, 2014] for linear contextual bandit uses classic upper bounds of the form U t (x) = θ(t) T x+ x Σt β t+1 and holds for prior distribution satisfying P( θ 2 ≤ C) = 1 for some constant C. This assumption does not hold for a Gaussian prior distribution. However, as already pointed out in Section 4.3.3, a linear bandit model with Gaussian prior and Gaussian noise is equivalent to a particular case of Gaussian process bandit, for which [Russo and Van Roy, 2014] also provide a Bayes risk bound. This bound is based on the decomposition (4.10) and holds when the total number of contexts D t is finite and static. Their argument can be easily extended to a set of changing contexts, leading to the following theorem.

Theorem 4.5 (adapted from [Russo and Van Roy, 2014], Proposition 5). In the Bayesian model (4.2), if for all t, D t = K, under Assumption 1, the Bayes-risk of Thompson sampling using a Gaussian prior π 0 = N 0, κ 2 I d is upper bounded as

BR π 0 (T, TS π 0 ) ≤ κ 2 L 2 √ 2π + 2dT log KT 2 π 2 6 C 1 log 1 + T L 2 κ 2 σ 2 d ,
with C 1 defined as in Lemma 4.3.

The Bayes risk bound we now provide in Theorem 4.6 holds without the assumption that the context set D t is finite. It is based on techniques similar to those of [Russo and Van Roy, 2014] (even if the decomposition (4.10) is not used explicitly) and on the fact that conditionally to H t , θ-θ(t) Σ -1 t follows a chi-square distribution with d degrees of freedom, as already used to obtain the Bayesian confidence region (4.5) in Lemma 4.2. The proof of this theorem is given in Section 4.6.2.

Theorem 4.6. Under Assumption 1 and the Bayesian model (4.2), the Bayes risk of Thompson Sampling using a Gaussian prior π 0 = N 0, κ 2 I d is upper bounded in the following way:

BR π 0 (T, TS π 0 ) ≤ d √ T C 1 log 1 + T L 2 κ 2 σ 2 d 3 + 8 log T + 2 log(2T 4 ) + π 2 Lκ √ d 3σ , with C 1 = 4L 2 κ 2 log(1+L 2 κ 2 σ -2 )
. In a linear contextual bandit problem with a finite number of contexts presented at each round, we saw in the previous section that either Bayes-UCB or Bayes-LinUCB could be used, with a high probability upper bound on the pseudo-regret of order Õ( dT log(K)) or Õ(d √ T ) respectively. Thus, depending on the relation between the dimension d an the number of contexts K, one of these two variants could be preferred based on these theoretical bounds or on the observation that one version performs better than the other in practice. Conversely, for Thompson Sampling, no adaptation of the algorithm is needed to obtain the same guarantees. That is, the Bayes risk of Thompson Sampling is upper bounded by whichever is smaller between the two bounds given in Theorem 4.5 and Theorem 4.6.

A frequentist analysis of Thompson Sampling

The only frequentist guarantees available to date for Thompson Sampling in contextual bandits with linear payoff follow from the work of [START_REF] Agrawal | Thompson Sampling for Contextual Bandits with Linear Payoffs[END_REF]. For a fixed value of θ, under Assumptions 1 and 2, the authors give a high-probability upper bound on the pseudo-regret of Thompson Sampling, but with a specific choice of prior and likelihood.

If the noise is assumed to be σ 2 -subgaussian (cf. Assumption 2), letting

κ = v = σ 9d log T δ , (4.11)
[ [START_REF] Agrawal | Thompson Sampling for Contextual Bandits with Linear Payoffs[END_REF] analyse the version of Thompson Sampling assuming θ ∼ N 0, κ 2 and t ∼ N 0, v 2 . This algorithm, denoted by TS ′ chooses at round t + 1 the context maximizing the dot product with the sample

θ(t) ∼ N θ(t), v 2 B(t) -1 with B(t) = I d + X T t X t .
The parameters κ and v have to be chosen as a function of the horizon T . To circumvent this issue, the authors propose and analyse the algorithm drawing at each time a sample from a normal distribution with mean θ(t) and covariance v 2 t B(t) -1 , with v t = σ 9d log t δ . This distribution can be regarded as a posterior distribution if the prior distribution and the distribution of the noise are N 0, v 2 t . Due to this varying prior distribution, this horizon-free variant cannot really be interpreted as Thompson Sampling. For TS ′ and its horizon-free variant, [START_REF] Agrawal | Thompson Sampling for Contextual Bandits with Linear Payoffs[END_REF] prove the following theorem.

Theorem 4.7 ( [START_REF] Agrawal | Thompson Sampling for Contextual Bandits with Linear Payoffs[END_REF], Theorem 1). For a fixed value of θ, under Assumptions 1 and 2, the algorithm TS ′ described above satisfies, with probability larger than 1 -δ,

R θ (T, TS ′ ) = O ⎛ ⎝ min d 3 2 √ T ; d T log(K) ⎛ ⎝ log(T ) + log(T ) log 1 δ ⎞ ⎠ ⎞ ⎠ .
The proof of Theorem 4.7 given by [START_REF] Agrawal | Thompson Sampling for Contextual Bandits with Linear Payoffs[END_REF]] is interesting since it departs from frequentist analyses presented so far. Indeed, it does not reduce to a deterministic upper bound of the pseudo-regret on some event whose probability can be shown to be larger than 1 -δ (as is the case when using Lemma 4.3). The cornerstone of this new analysis lies in an upper bound on the conditional expectation E θ [r t F t-1 ] which takes the form

E θ r t F t-1 ≤ Cg t E θ x t B(t-1) -1 F t-1 + Dg t t 2
, C, D being two real constants and g t some function growing logarithmically.

Theorem 4.7 leaves interesting open questions regarding the analysis of Thompson Sampling from a frequentist perspective. First, this bound displays an extra factor √ d when compared to the bounds obtained in the Bayesian framework, whereas we will see in the experimental section to follow that in practice, the regret of Thompson Sampling is smaller. Second, if the noise is assumed to be σ 2subgaussian, it is natural to try to obtain performance guarantees for the version of Thompson Sampling that assumes a normal distribution for the noise with variance σ 2 , and not v 2 given by (4.11)).

Numerical experiments

Experiments assessing the performance of Thompson Sampling beyond the Bernoulli case are mostly carried out for a particular instance of generalized linear bandit model, based on logistic regression (see [Scott, 2010, Chapelle and Li, 2011, May et al., 2012, Chapelle et al., 2014]). The rewards are assumed to be binary, such that y t ∈ {-1, +1}, modeling for example click or no click from the users, and

P(y t = 1 x t , θ) = 1 1 + exp(-θ T x t )
.

If θ is assumed to be drawn from a Gaussian prior distribution N 0, 1 λ I d , under this model the posterior distribution is no longer exactly computable. However, using a Laplace approximation (see e.g. [Bishop, 2006]), it can be approximated by a Gaussian distribution:

p(θ x 1 , y 1 , . . . , x t , y t ) ∼ N m, Diag(q -1 i ) with m = argmin w∈R d λ 2 w 2 + n i=1 log(1 + exp(-y i w T x i )) q i = n j=1 x 2 j,i p j (1 -p j ) with p j = (1 + exp(-m T x j )) -1 .
Thompson Sampling in this model then boils down to drawing at each round a sample θ(t) from the (approximate) posterior distribution at the end of round t and then choosing the context

x t+1 = argmax x∈D t+1 1 1 + exp(-θ(t) T x) = argmax x∈D t+1 θ(t) T x.
Hence this version of Thompson Sampling is quite similar to Thompson Sampling as presented for linear contextual bandits, but with a different posterior distribution motivated by the logistic model. In the experiments below, we focus on the linear model.

CHAPTER 4. BAYESIAN ALGORITHMS FOR LINEAR CONTEXTUAL BANDITS

Even if most of the results of this chapter are obtained under the Bayesian model (4.2), we propose here frequentist experiments. Indeed, for a fixed value of the regression parameter θ ∈ R d , we repeat several bandit games up to some horizon T and estimate the regret (i.e. the expectation of the pseudoregret, given this fixed value of θ). In each bandit game, the reward after choosing context x is drawn from N x T θ, σ 2 , with σ = 0.25. In order to simulate the changing contexts, we first choose (at random) a large set of K m = 100000 contexts, and at each round t, a subset of K = 2000 contexts is chosen at random from this large set and forms the set D t . For this value of K, one has log(K) ≃ 7.6. We propose experiments in two different bandit models in which the dimension d is either smaller or larger than this value. Results are reported in Figure 4.2.

In the first model, d = 5 (therefore d < log(K)). Regret curves (averaged over N = 1000 bandit games) are displayed on the left of Figure 4.2 for OFUL and the three Bayesian strategies studied in this chapter: Bayes-UCB, Bayes-LinUCB and Thompson Sampling. In this bandit model, Bayes-LinUCB outperforms Bayes-UCB, as was expected from Figure 4.1, but OFUL outperforms both algorithms (maybe because for small horizons, the exploration rate used by OFUL is smaller than that used by both algorithms, as displayed in Figure 4.1). The regret of Thompson Sampling is significantly smaller than that of all other algorithms. In the second model, for which regret curves, averaged over N = 500 bandit games, are displayed on the right of Figure 4.2, one has d = 10 (therefore d > log(K)). In this model, Bayes-UCB outperforms Bayes-LinUCB and performs almost as well as OFUL, whereas Thompson Sampling still outperforms all his competitors by a large margin.

4.6 Elements of proof 4.6.1 Proof of Lemma 4.3

Let U t (x) ∶= θ(t) T x + x Σ -1 t β(t + 1, δ)
and A be the algorithm picking at round t

x t = argmax x∈Dt U t-1 (x).
We start by upper bounding the quantity r t = θ T x * t -θ T x t on the event E. By definition, the algorithm is such that U t-1 (x t ) ≥ U t-1 (x) for all x ∈ D t , thus one has in particular U t-1 (x * t ) ≤ U t-1 (x t ). On the event E, one also has θ

T x * t ≤ U t-1 (x * t ). Therefore, if E holds, r t ≤ U t-1 (x * t ) -θ T x t ≤ U t-1 (x t ) -θ T x t = θ(t -1) T x t + x t Σ t-1 β(t, δ) -θ T x t = ( θ(t -1) -θ) T x t + x t Σ t-1 β(t, δ) ≤ 2β(t, δ) x t Σ t-1 .
To obtain the last inequality, we use that on E, ( θ(t -1) -θ) T x t ≤ x t Σ t-1 β(t, δ). Technical lemma 4.8 stated below can now be used to upper bound deterministically the sum of the norms x t Σ t-1 . Using additionally the Cauchy-Schwarz (C.S.) inequality, one can write, if event E holds,

R θ (T, A) = T t=1 r t ≤ 2 T t=1 β(t, δ) x t Σ t-1 ≤ 2β(T, δ) T t=1 x t Σ t-1 C.S. ≤ 2β(T, δ) √ T T t=1 x t 2 Σ t-1 Lemma 4.8 ≤ 2β(T, δ) √ T L 2 κ 2 log(1 + L 2 κ 2 σ -2 ) d log 1 + T L 2 κ 2 σ 2 d = β(T, δ) √ T d C 1 log 1 + T L 2 κ 2 σ 2 d
which concludes the proof.

◻ Lemma 4.8. T t=1 x t 2 Σ t-1 ≤ L 2 κ 2 log(1 + L 2 κ 2 σ -2 ) d log 1 + T L 2 κ 2 σ 2 d Proof of Lemma 4.8. First x t 2 Σ t-1 = σ 2 x t 2 B(t-1) -1 . One has x t 2 B(t-1) -1 = x T t B(t -1) -1 x t ≤ CS x t B(t -1) -1 x t ≤ κ 2 σ 2 x t 2 ≤ κ 2 σ 2 L 2 , (4.12)
where we use that B(t) -1 ≤ B(t) -1 ≤ (σ2 κ 2 ) -1 , since the eigenvalues of matrix B(t) are lower bounded by σ 2 κ 2 . For every a > 0 it can be easily shown that

∀x ∈ [0, a], x ≤ a log(1 + a) log(1 + x).
Applying this inequality to the value a = κ 2 L 2 σ -2 yields

T t=1 x t 2 B(t-1) -1 ≤ κ 2 L 2 σ -2 log(1 + κ 2 L 2 σ -2 ) T t=1 log 1 + x t 2 B(t-1) -1 .
(4.13) Now, following for example [START_REF] Dani | Stochastic Linear Optimization under Bandit Feedback[END_REF], one can show that

T t=1 log 1 + x t 2 B(t-1) -1 = log ⎛ ⎜ ⎝ det(B(T )) det σ 2 κ 2 I d ⎞ ⎟ ⎠ . (4.14) Indeed, det(B(t + 1)) = det σ 2 κ 2 I d + t+1 s=1 x s x T s = det B(t) + x t+1 x T t+1 = det(B(t))det I d + B(t) -1 2 x t+1 x T t+1 B(t) -1 2 = det(B(t))det I d + (B(t) -1 2 x t+1 )(B(t) -1 2 x t+1 ) T = det(B(t)) 1 + B(t) -1 2 x t+1 2 = det(B(t)) 1 + x t+1
To obtain the last equality, we used that for every

v ∈ R d , det(I d + vv T ) = (1 + v T v)
. By induction, it easily follows that det(B(T )) = ∏ T t=1 (1 + x t 2 B(t-1) -1 ) det σ 2 κ 2 I d which proves (4.14). Using the bound (4.4) already used for the determinant of B(T ), one has

det(B(T )) ≤ σ 2 κ 2 + T L 2 d d .
This leads to an upper bound on (4.14), that yields, combined with Equation (4.13)

T t=1 x t 2 Σ t-1 ≤ κ 2 L 2 log(1 + κ 2 L 2 σ -2 ) d log 1 + T L 2 κ 2 dσ 2

◻

Remark 4.9. Introducing the notation σ 2 t (x) = x 2 Σ t-1 , the quantity introduced in (4.13) can be written

1 2 T t=1 log 1 + x t 2 B(t-1) -1 = 1 2 T t=1 log 1 + σ -2 σ 2 t (x t ) .
This last quantity is explicitly bounded by [START_REF] Srinivas | Gaussian Process Optimization in the Bandit Setting : No Regret and Experimental Design[END_REF] in the more general context of Gaussian Process bandits. It is upper bounded by the maximum information gain called γ T . Here we rather use techniques from [START_REF] Dani | Stochastic Linear Optimization under Bandit Feedback[END_REF] to upper bound this quantity in the simpler case of a linear kernel.

Proof of Theorem 4.6

As already used in [Russo and Van Roy, 2014], x t+1 and x * t+1 have the same distribution conditionally to H t . Thus, for any function f (x, y), if X is a H t measurable random variable, the random variables f (X, x t+1 ) and f (X, x * t+1 ) have the same conditional expectation with respect to H t . [Russo and Van Roy, 2014]

use that E[U t (x t+1 ) H t ] = E[U t (x * t+1 ) H t ]
for any confidence bound introduced in the analysis. Here we use similar equalities:

E[ x t+1 Σt H t ] = E[ x * t+1 Σt H t ] (4.15) E[ θ(t) T x t+1 H t ] = E[ θ(t) T x * t+1 H t ] (4.16)
We start with the following decomposition:

r t+1 = θ T x * t+1 -θ T x t+1 = θ T x * t+1 -θ(t) T x * t+1 + θ(t) T x * t+1 -θ T x t+1
Conditioning by H t and using equality (4.16) yields, for any sequence β t introduced below,

E[r t+1 ] ≤ E (θ -θ(t)) T x * t+1 + E ( θ(t) -θ) T x t+1 ≤ E θ -θ(t) Σ -1 t x * t+1 Σt + E θ -θ(t) Σ -1 t x t+1 Σt ≤ E ⎡ ⎢ ⎢ ⎢ ⎣ θ -θ(t) Σ -1 t ( x * t+1 Σt + x t+1 Σt ) 1 θ-θ(t) Σ -1 t ≤β t+1 ⎤ ⎥ ⎥ ⎥ ⎦ +E ⎡ ⎢ ⎢ ⎢ ⎣ θ -θ(t) Σ -1 t ( x * t+1 Σt + x t+1 Σt ) 1 θ-θ(t) Σ -1 t >β t+1 ⎤ ⎥ ⎥ ⎥ ⎦ ≤ 2β t+1 E [ x t+1 Σt ] + 2Lκ E ⎡ ⎢ ⎢ ⎢ ⎣ θ -θ(t) Σ -1 t 1 θ-θ(t) Σ -1 t >β t+1 ⎤ ⎥ ⎥ ⎥ ⎦ ,
where the last inequality is obtained using (4.15) together with the upper bound x Σt ≤ Lκ for any x ∈ D t+1 (deduced from inequality (4.12) in Section 4.6). One obtains, by summing these terms and using the Cauchy-Schwarz inequality,

BR π 0 (T, TS π 0 ) ≤ T t=1 2β t E [ x t Σ t-1 ] + 2Lκ T t=1 E ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ θ -θ(t -1) Σ -1 t-1 1 θ-θ(t-1) Σ -1 t-1 >βt ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ ≤ 2β T √ T E ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ T t=1 x t 2 Σ t-1 ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ + 2Lκ T -1 t=0 E ⎡ ⎢ ⎢ ⎢ ⎣ θ -θ(t) Σ -1 t 1 θ-θ(t) Σ -1 t >β t+1 ⎤ ⎥ ⎥ ⎥ ⎦
The sum ∑ T t=1 x t 2 Σ t-1 that appears in the first term is upper bounded deterministically by Lemma 4.8, given in Section 4.6. To bound the second term, we use again that θθ(t) 2 Σ -1 t follows a chi-square distribution with d degrees of freedom. Jensen inequality gives

⎛ ⎝ E ⎡ ⎢ ⎢ ⎢ ⎣ θ -θ(t) Σ -1 t 1 θ-θ(t) Σ -1 t >β t+1 ⎤ ⎥ ⎥ ⎥ ⎦ ⎞ ⎠ 2 ≤ E ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ θ -θ(t) 2 Σ -1 t 1 θ-θ(t) 2 Σ -1 t >β 2 t+1 ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ = ∞ β 2 t+1 x 2 d 2 Γ d 2 x d 2 -1 e -x 2 dx = d ∞ β 2 t+1 1 2 d+2 2 Γ d+2 2 x d+2 2 -1 e -x 2 dx = d(1 -F χ 2 d+2 (β 2 t+1 )).
where

F χ 2 d+2
denotes the cdf of a chi-square distribution with d + 2 degrees of freedom. Thus,

T -1 t=0 E ⎡ ⎢ ⎢ ⎢ ⎣ θ -θ(t) Σ -1 t 1 θ-θ(t) Σ -1 t >β t+1 ⎤ ⎥ ⎥ ⎥ ⎦ ≤ √ d T t=1 1 -F χ 2 d+2 (β 2 t ) (4.17)
Hence, choosing

β t = Q 1 - 1 t 4 ; χ 2 d+2 ≤ d + 2 + 8 log t + 2 (d + 2) log 2t 4 ,
inequality (4.17) and Lemma 4.8 yield

BR π 0 (T, TS π 0 ) ≤ 2β T √ T L 2 κ 2 log(1 + L 2 κ 2 σ -2 ) d log 1 + T L 2 κ 2 σ 2 d + 2Lκ √ d π 2 6 ≤ d √ T C 1 log 1 + T L 2 κ 2 σ 2 d 3 + 8 log T + 2 log(2T 4 ) + π 2 Lκ √ d 3 with C 1 = 4L 2 κ 2 log(1+L 2 κ 2 σ -2 ) .
Chapter 5

Refined frequentist tools for best arm identification

In the first chapters of this thesis, we studied optimal algorithms for the objective of maximizing rewards in bandit models. A different objective can be considered, in which samples collected from the arms are not perceived as rewards, and the goal is to optimally explore the environment so as to identify the best arm(s) (without an incentive to exploration). This chapter gathers our contributions to this objective, called best arms identification (or pure-exploration) in bandit models.

Unlike that of regret minimization, the complexity of best arms identification is not so well understood, and our goal is to identify optimal algorithms in this framework. In this chapter, we introduce a dedicated notion of complexity in two different settings that have been considered in the literature: the fixed-budget and the fixed-confidence settings. We propose new lower bounds on these complexities, that involve information-theoretic quantities (like the Lai and Robbins' lower bound on the regret), as well as improved algorithms. Some of these algorithms, inspired by KL-UCB, are based on refined confidence intervals using Kullback-Leibler divergence. Whereas we were not able to close the gap between informational upper and lower bounds in the general case, we do identify the complexities for particular instances of two-armed bandits.

This chapter is based on two publications: a joint work with Shivaram Kalyanakrishnan ([Kaufmann and [START_REF] Kaufmann | [END_REF]), presented at the COLT conference in 2013, in which we study algorithms for m best arms identification, and a joint work with Olivier Cappé and Aurélien Garivier in which we introduce the complexities and notably present lower bounds ( [START_REF] Kaufmann | On the Complexity of Best Arm Identification in Multi-Armed Bandit Models[END_REF], submitted to JMLR). A shorter version of this last paper, [Kaufmann et al., 2014a], focused on two-armed bandits, was also presented at COLT in 2014.

Introduction

Recall a bandit model is a collection of K arms, where each arm ν a (1 ≤ a ≤ K) is a probability distribution on R with expectation µ a . At each time t = 1, 2, . . . , an agent chooses an option A t ∈ {1, . . . , K} and receives an independent draw X t of the corresponding arm ν At . We denote by P ν (resp. E ν ) the probability law (resp. expectation) of the corresponding process (X t ). The agent's goal is now to identify the m best arms, that is, the set S * m of indices of the m arms with highest expectation. Letting (µ [1] , . . . , µ [K] ) be the K-uple of the expectations (µ 1 , . . . , µ K ) sorted in decreasing order, we assume that the bandit model ν belong to a class M m such that for every

ν ∈ M m , µ [m] > µ [m+1]
, in which S * m is unambiguously defined. This last assumption is not necessary any more in an -relaxation of the problem that is sometimes considered in the literature: for some tolerance parameter ≥ 0 the agent has to ensure that Ŝm is included in the set of ( , m)-optimal arms S * m, = {a ∶ µ a ≥ µ [m] -}. In order to identify S * m , the agent must use a strategy defining which arms to sample from, but also when to stop sampling, and which set Ŝm to choose. The sampling rule determines how, at time t, the arm A t is chosen based on the past observations; in other words, A t is F t-1 -measurable, with F t = σ(A 1 , Z 1 , . . . , A t , X t ). The stopping rule τ is a stopping time with respect to (F t ) t∈N . The recommendation rule is a F τ -measurable random subset Ŝm of {1, . . . , K} of size m. This triple ((A t ), τ, Ŝm ) entirely determines the strategy, which we denote in the sequel by A.

In the bandit literature, two different settings have been considered. In the fixed-confidence setting, a risk parameter δ is fixed.

A strategy A is called δ-PAC if, for every choice of ν ∈ M m , P ν ( Ŝm = S * m ) ≥ 1 -δ, or in the -relaxation described before, if P ν ( Ŝm ⊂ S * m, ) ≥ 1 -δ.
The goal is, among the δ-PAC strategies, to minimize the expected number of draws E ν [τ ] (sometimes called sample complexity). In the fixed-budget setting, the number of draws τ is fixed in advance (τ = t almost surely) and the goal is to choose the sampling and recommendation rules so as to minimize p t (ν) ∶= P ν ( Ŝm ≠ S * m ). In the fixed-budget setting, a strategy A is called consistent if, for every choice of ν ∈ M m , p t (ν) goes to zero when t goes to infinity.

Recall that the complexity of regret minimization, the alternative objective considered so far, is well understood for parametric bandits. Indeed, [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] define a dedicated notion of consistency and prove that, in generic one-parameter models,

inf A consistent lim inf t→∞ E ν [R ν (T, A)] log t ≥ a∶µa<µ [1] (µ [1] -µ a ) KL(ν a , ν [1]
) .

INTRODUCTION 139

We saw that there exists strategies whose regret attain this bound, including the two Bayesian strategies discussed in Chapter 2 and 3 and the KL-UCB algorithm of [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF] that uses informational upper-bounds. Similarly, one would expect in the fixed-confidence setting a lower bound on the sample complexity E ν [τ ] of any δ-PAC algorithm (resp. in the fixed-budget setting a lower bound on the probability of error p t (ν) of any consistent algorithm) that also features information-theoretic quantities, as well as algorithms whose sample complexity (resp. probability of error) matches the lower bound, called matching algorithms. In order to unify the two settings, we define the complexity κ C (ν) (resp. κ B (ν)) of best arms identification in the fixed-confidence (resp. fixed-budget) setting, as follows:

κ C (ν) = inf A δ-PAC lim sup δ→0 E ν [τ ] log 1 δ , κ B (ν) = inf A consistent lim sup t→∞ - 1 t log p t (ν) -1 . (5.1)
Heuristically, for a given bandit model ν and a small enough value of δ, a fixed-confidence optimal strategy needs an average number of samples of order κ C (ν) log 1 δ to identify the m best arms, whereas a fixed-budget optimal strategy requires approximately t = κ B (ν) log 1 δ draws in order to ensure a probability of error of order δ. Our goal in this chapter is to evaluate and compare these two complexities. We will do so by providing new lower bounds on κ C (ν) and κ B (ν) that feature information-theoretic quantities and by analyzing refined strategies, some of which matching the lower bounds in particular instances of two-armed bandit problems.

Most of the existing performance bounds for pure-exploration (a denomination that encompasses the fixed-budget and fixed-confidence settings) can be expressed using the two complexity measures defined above, as we see now.

The problem of best arms identification has been studied since the 1950s under the name 'ranking and identification problems'. The first advances on this topic are summarized in the monograph by [START_REF] Bechhofer | Sequential identification and ranking procedures[END_REF] who consider the fixed-confidence setting. More recently, in the same setting [START_REF] Even-Dar | Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems[END_REF] propose algorithms for (single) best arm identification in bounded bandit models, in which each arm ν a is a probability distribution on [0, 1]. m best arms identification with m > 1 was considered for example by [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF] (under the name Explore-m), who propose the LUCB (for Lower and Upper Confidence Bounds) algorithm, still for bounded bandit models. Bounded distributions are particular examples of subgaussian distributions, to which the proposed algorithms can be easily generalized. A relevant quantity introduced in the analysis of algorithms for bounded (or subgaussian) bandit models is the 'complexity term'

H(ν) = a∈{1,2,...K} 1 ∆ 2 a with ∆ a = ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ µ a -µ [m+1] for a ∈ S * m , µ [m] -µ a for a ∈ (S * m ) c .
(5.

2)

The upper bound on the sample complexity of the LUCB algorithm of [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF] implies in particular that κ C (ν) ≤ 292H(ν). Some of the existing works on the fixed-confidence setting do not bound τ in expectation but rather show that P ν ( Ŝm = S * m , τ = O (H(ν))) ≥ 1 -δ. These results are not directly comparable with the complexity κ C (ν), although no significant gap is to be observed yet.

For m = 1, the work of [START_REF] Mannor | The Sample Complexity of Exploration in the Multi-Armed Bandit Problem[END_REF] provides a lower bound on κ C (ν), who address Bernoulli bandit models with the -relaxation described before. The authors show that if an algorithm is δ-PAC, then in the bandit ν = (B(µ 1 ), . . . , B(µ K )) such that ∀a, µ a ∈ [0, α] for some α ∈]0, 1[, there 140 CHAPTER 5. REFINED FREQUENTIST TOOLS FOR BEST ARM IDENTIFICATION exists two sets M α (ν) ⊂ S * 1 and N α (ν) ⊂ {1, . . . , K} S * 1 and a positive constant C α such that

E ν [τ ] ≥ C α ⎛ ⎝ a∈Mα(ν) 1 2 + a∈Nα(ν) 1 (µ [1] -µ a ) 2 ⎞ ⎠ log 1 8δ .
This bound is non asymptotic (as emphasized by the authors), although not completely explicit. In particular, the subset M α and N α do not always form a partition of the arms (it can happen that M α ∪ N α ≠ {1, . . . K}), hence the complexity term does not involve a sum over all the arms. For m > 1, the only lower bound available in the literature is the worst-case result of [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF]. It states that for every δ-PAC algorithm there exists a bandit model ν such that

E ν [τ ] ≥ K (18375 2 ) log (m 8δ
). This yields, however, no lower bound on the complexity κ C (ν).

The fixed-budget setting has been studied by [Audibert et al., 2010, Bubeck et al., 2011] for single best-arm identification in bounded bandit models. For multiple arm identification (m > 1), still in bounded bandit models, [START_REF] Bubeck | Multiple Identifications in multi-armed bandits[END_REF] introduce the SAR (for Successive Accepts and Rejects) algorithm. An upper bound on the failure probability of the SAR algorithm yields κ B (ν) ≤ 8 log(K)H (ν).

For m = 1, [Audibert et al., 2010] prove an asymptotic lower bound on the probability of error for Bernoulli bandit models. They state that for every algorithm and every bandit problem ν such that ∀a, µ 1 ∈ [α, 1 -α], there exists a permutation of the arms ν ′ such that 1)). This result does not imply a lower bound on κ B (ν) and for m > 1, no such lower bound exists either.

p t (ν ′ ) ≥ exp(-t C α H 2 (ν ′ ))), with H 2 (ν) = max i∶µ [i] <µ [1] i µ [1] -µ [i] 2 and C α = α(1 -α) (5 + o(
The gap between lower and upper bounds known so far does not permit to identify exactly the complexity terms κ B (ν) and κ C (ν) defined in (5.1). Not only do they involve imprecise multiplicative constants but by analogy with the Lai and Robbins' bound for the regret, the quantities H(ν) or H 2 (ν) presented above are only expected to be relevant in the Gaussian case. Moreover, when m > 1, no lower bounds on the complexities are available.

Our contributions are the following. In the fixed-confidence setting, we first propose in Section 5.2 two algorithms, KL-LUCB and KL-Racing that use informational upper and lower bounds, transposing the improvements of KL-UCB from regret minimization to pure-exploration. The analysis of these algorithms leads to the first informational upper bound on κ C (ν). We then propose in Section 5.3 a lower bound on κ C (ν), that holds for general classes of bandit models, also when m > 1. This bound takes the form of a sum over all arms of an individual complexity term involving Kullback-Leibler divergence so that the quantity H(ν) appears as a subgaussian approximation. However, this lower bound is not attained by the KL-LUCB algorithm. For specific families of two-armed bandits (Gaussian bandits with known -but possibly different-variances, Bernoulli bandits) a refined lower bound as well as improved algorithms then lead to an exact expression of some complexity terms. Interestingly, we prove that for Gaussian bandit models κ C (ν) = κ B (ν), whereas for Bernoulli bandit models κ C (ν) > κ B (ν), showing that the two complexities are not equal in general. The particular case of two-armed bandits is studied in Section 5.4.

Algorithms: KL-LUCB and KL-Racing

In this Section, we present general algorithms for m best arms identification among K arms. [START_REF] Bubeck | Pure Exploration in Finitely Armed and Continuous Armed Bandits[END_REF] show that in the fixed-budget setting, for single best arm identification, any sampling strategy designed to minimize regret performs poorly with respect to the simple regret r t ∶= µ * -µ Ŝ1 , a quantity closely related to the probability p t (ν) of recommending the wrong arm. Therefore, good strategies for best arms identification are expected to be quite different from UCB-like strategies.

In Section 5.2, we present a review of existing algorithms for the fixed-confidence and fixed-budget settings, identifying two main classes of algorithms: those based on uniform sampling and eliminations and those based on adaptive sampling. We introduce, for the fixed-confidence setting, two generic algorithms based on confidence intervals (using both upper and lower confidence bounds) that belong to each of these classes and analyse in Section 5.2.2 two particular instances, KL-Racing and KL-LUCB. These algorithms are shown to perform well in practice (experiments are reported in Section 5.2.3) and the upper bounds obtained on their sample complexity are the first upper bounds featuring informationtheoretic quantities.

To ease the presentation, we will restrict our attention to Bernoulli bandit models, but KL-LUCB and KL-Racing (and our results) can be extended to exponential bandits, using the appropriate d function (see Section 1.2.2 in Chapter 1). Recall a Bernoulli bandit model, of the form ν = (B(µ 1 ), . . . , B(µ K )), can be either regarded as a bounded bandit model or as an exponential bandit model. In the regret minimization framework, this allowed to use algorithms based on upper confidence bounds built either with Hoeffding's inequality (UCB1, designed for any bandit bounded bandit model) or on Chernoff inequality, that lead to Bernoulli-specific confidence regions based on KL-divergence (KL-UCB). In this Section, we introduce similar improved algorithms for the problem of m-best arms identification in Bernoulli bandit models in the fixed-confidence setting.

We consider in this section the -relaxation of the fixed-confidence setting described above, and assume to ease the notation that the arms are ordered such that µ 1 ≥ ⋅ ⋅ ⋅ ≥ µ m ≥ µ m+1 ≥ ⋅ ⋅ ⋅ ≥ µ K .

Two classes of algorithms based on confidence intervals

Virtually all the algorithms proposed to date for pure-exploration problems can be classified according to their sampling strategy: algorithms using uniform sampling and eliminations maintain a set of remaining arms, and sample all these remaining arms at each round, whereas algorithms using adaptive sampling sample at each round one or two well-chosen arms.

Just as upper confidence bounds have been used successfully in the regret setting, most existing algorithms for the fixed-confidence setting have used both upper and lower confidence bounds on the means of the arms. We state here a generic version of an algorithm using uniform sampling and eliminations, Racing, and a generic version of an adaptive sampling algorithm, LUCB. To describe these contrasting heuristics, we use generic confidence intervals, denoted by I a (t) = [L a (t), U a (t)], where t is the round of the algorithm, L a (t) and U a (t) are the lower and upper confidence bounds on the mean of arm a. Let N a (t) denote the number of draws, and S a (t) the sum of the rewards gathered from arm a up to time t. Let μa (t) = Sa(t) Na(t) be the corresponding empirical mean reward, and let μa,u be the empirical mean of the first u i.i.d. samples from arm a. Additionally, let J(t) be the set of m arms with the highest empirical means at time t (for the Racing algorithm, J(t) only includes m ′ ≤ m arms if mm ′ have already been selected). Also, l t and u t are two 'critical' arms from J(t) and J(t) c that are likely to be misclassified:

u t = argmax b∉J(t) U b (t) and l t = argmin a∈J(t)
L a (t).

( 5.3) rejects based on confidence intervals are introduced by [START_REF] Heidrich-Meisner | Hoeffding and Bernstein Races for Selecting Policies in Evolutionary Direct Policy Search[END_REF] and applied within the context of reinforcement learning. The authors do not formally analyse the algorithm's sample complexity, as we do here. The Racing algorithm, stated precisely as Algorithm 3, samples at each round t all the remaining arms, and updates the confidence bounds. Then a decision is made to possibly select the empirical best arm if its lower confidence bound (LCB) is larger than the upper confidence bounds (UCBs) of all arms in J(t) c , or to discard the empirical worst arm if its UCB is smaller than the LCBs of all arms in J(t). The successive elimination algorithm [START_REF] Even-Dar | Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems[END_REF]) for finding the single best arm can be regarded as a specification of Algorithm 3 using Hoeffding bounds.

The LUCB algorithm. A general version of the LUCB algorithm proposed by [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF] is stated in Algorithm 4, using generic confidence bounds U and L, while the original LUCB uses Hoeffding confidence regions. Unlike Racing, this algorithm does not sample the arms uniformly; rather, it draws at each round the two critical arms u t and l t . This sampling strategy is associated with the natural stopping criterion (B(t) < ) where B(t) ∶= U ut (t) -L lt (t). That is the algorithm stops when the confidence intervals for the means of the arms in J(t) and those for the means of the arms in J(t) c are well separeted. An illustration of the LUCB algorithm can be found in Figure 5.1 The UGapEc algorithm of [START_REF] Gabillon | Best Arm Identification: A Unified Approach to Fixed Budget and Fixed Confidence[END_REF]] also uses adaptive sampling and is very close to LUCB: it uses an alternative definition of J(t) using confidence bounds on the simple regret, and a correspondingly different stopping criterion B(t). But as LUCB, it also samples the corresponding critical arms u t or l t .

KL-Racing and KL-LUCB. The two algorithms mentioned above both use generic upper and lower confidence bounds on the mean of each arm, and one has the intuition that the smaller these confidence regions are, the smaller the sample complexity of these algorithms will be. Most of the previous algorithms use Hoeffding bounds, of the form

U a (t) = μa (t) + β(t, δ) 2N a (t)
and L a (t) = μa (t) -β(t, δ) 2N a (t) (5.4) for some exploration rate β(t, δ). Previous work [START_REF] Mnih | Empirical Bernstein stopping[END_REF], Heidrich-Meisner and Igel, 2009, Gabillon et al., 2012] has also considered the use of empirical Bernstein bounds, that can be tighter.

Here, we introduce the use of confidence regions based on KL-divergence for Explore-m, inspired by recent improvements in the regret setting [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF]. We define, for some exploration rate β(t, δ) (that is a function of t and δ, and not only t as in the regret minimization framework), u a (t) ∶= max {q ∈ [μ a (t), 1] ∶ N a (t)d(μ a (t), q) ≤ β(t, δ)} , and (5.5) l a (t) ∶= min {q ∈ [0, μa (t)] ∶ N a (t)d(μ a (t), q) ≤ β(t, δ)} .

(5.6)

As already noted in Chapter 1 (Section 1.2.2) , for Bernoulli distributions KL-confidence regions are always smaller than those obtained with Hoeffding bounds, while sharing the same coverage probability:

μa (t) - β(t, δ) 2N a (t) ≤ l a (t) and u a (t) ≤ μa (t) + β(t, δ) 2N a (t) .
(5.7)

We define, for a given function β, the KL-Racing and KL-LUCB algorithms with exploration rate β as the instances of Racing and LUCB, respectively, that use u a (t) and l a (t) as confidence bounds. Section 5.2.2 provides conditions on β for both algorithms to be δ-PAC and sample complexity bounds under these conditions. In our theoretical and experimental analysis to follow, we address the "KL versus Hoeffding" and "uniform versus adaptive sampling" questions.

Other algorithms for the fixed-confidence and fixed-budget setting. Among the family of algorithms using uniform sampling and eliminations, the algorithms presented so far, including (KL)-Racing, consider the possibility of eliminating one arm at each round. Algorithms inspired by the Median Elimination algorithm of [START_REF] Even-Dar | Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems[END_REF] for m = 1 -extended to Halving for m > 1 by [START_REF] Kalyanakrishnan | Efficient Selection in Multiple Bandit Arms: Theory and Practice[END_REF] -are quite different. The basic Median Elimination algorithm consists in phases and at the end of each phase, the empirical worst half of the arms is discarded, based on the samples gathered in this phase only. For Bernoulli bandits, the Exponential-Gap Elimination of [START_REF] Karnin | Almost optimal Exploration in multi-armed bandits[END_REF] uses the Median Elimination algorithm as a subroutine. For this δ-PAC algorithm, there exists a constant C such that, with high probability, the number of samples needed to identify the best arm is upper bounded by

C K a=2 1 ∆ 2 a log 1 δ log 1 ∆ a (5.8)
This is an improvement when compared to most existing algorithm for which τ is upper bounded by some constants multiplied by H(ν) log (H(ν) δ): the relatively small factor log 1 ∆ a compared to H(ν) can be a significant improvement, especially when the number of arms is large. Moreover, in the regime where δ is fixed and ∆ a goes to zero, [START_REF] Jamieson | lil'UCB: an Optimal Exploration Algorithm for Multi-Armed Bandits[END_REF] show that the term ∆ -2 a log log ∆ -2 a is optimal, and propose an upper bound similar to (5.8) for the LIL-UCB algorithm. In both cases the constant C is quite large, which does not lead to an improved upper bound on the complexity term κ C (ν).

For the KL-Racing and KL-LUCB algorithms, we therefore propose in the next Section upper bound on τ with explicit constants, featuring moreover informational quantities, and no longer the squared gaps ∆ a .

In the fixed-budget setting, [START_REF] Bubeck | Multiple Identifications in multi-armed bandits[END_REF] propose the Successive Accepts and Rejects (SAR) algorithm for the objective of finding the m best arms, generalizing the Successive Reject algorithm of [Audibert et al., 2010] for m = 1. This algorithm samples uniformly the arms in each of the K -1 phases with predetermined length, and at the end of each phase exactly one arm is eliminated. The empirical best arm is selected or the empirical worst discarded, according to its empirical gap with J(t) c or J(t) respectively (a criterion than cannot be formulated with confidence intervals). A variant of this algorithm in which at the end of each phase half of the remaining arms is eliminated has been recently proposed by [START_REF] Karnin | Almost optimal Exploration in multi-armed bandits[END_REF], showing improvements on the theoretical probability of error compared to that of SAR, but which does not improve the resulting upper bound on the (asymptotic) complexity κ B (ν). Some adaptive sampling algorithms do exist for the fixed-budget setting too, namely UCB-E of [Audibert et al., 2010] for m = 1, or UGapEb of [START_REF] Gabillon | Best Arm Identification: A Unified Approach to Fixed Budget and Fixed Confidence[END_REF]. These algorithm are not efficient in practice since they share the need to know the complexity term H(ν). In the paper [START_REF] Kaufmann | Information complexity in bandit subset selection[END_REF] we propose an other adaptive algorithm for the fixed-budget setting, KL-LUCB-E, derived from KL-LUCB by choosing the exploration rate β as a function of n, but suffering from the same weakness as its existing counterpart. In the practical experiments of Section 5.2.3, we will discuss further the interest of using adaptive sampling in the fixed-budget setting.

Analysis of KL-Racing and KL-LUCB

Theorem 5.1 gives a choice of β for which KL-Racing and KL-LUCB are correct with probability at least δ (δ-PAC). Note that these choices of β lead to the same guarantees for their Hoeffding counterpart, (Hoeffding)-Racing and LUCB. This definition can be generalized to distributions in an exponential family. We will discuss later the interpretation we can propose for the relevance of Chernoff information as a complexity measure for pure-exploration, here we first explain why Chernoff information in necessary in the proof of Theorem 5.6. In Theorem 5.5, an other quantity is involved, related to Chernoff information:

d * * (x, y) = d * (z * * , x) = d * (z * * , y) where z * * is the unique z such that d * (z, x) = d * (z, y).
A tight concentration result involving Chernoff information. In our analysis of KL-LUCB, we need to bound the probability that some constant c belongs to the interval I a (t) after this arm has already been sufficiently sampled. Deriving such a result for intervals based on KL-divergence brings up Chernoff information: Lemma 5.2. Let T ≥ 1 be an integer. Let γ > 0 and c ∈]0, 1[ be such that µ a ≠ c. This result is a corollary of the important Lemma 5.4 below, proved in Section 5.6.2, that is in some sense an optimal deviation result involving KL-divergence. Some functions based on KL-divergence need to be defined in order to state this more general result.

F C 1 (y, c) = C 1 d(s C 1 (y, c), y) d(y, c) . Lemma 5.4. Let C 1 > 1, γ > 0 and c ∈]0, 1[ such that µ a ≠ c. For any integer T , T u=⌈ C 1 γ d(µa,c) ⌉+1 P (ud(μ a,u , c) ≤ γ) ≤ exp (-F C 1 (µ a , c)γ) d (s C 1 (µ a , c), µ a ) .
(5.9)

The sum in Lemma 5.4 is bounded tightly in the recent analysis of KL-UCB by [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF] for the value C 1 = 1. However, the related bound shows no exponential decay in γ, unlike the one we prove for C 1 > 1 in Section 5.2.4. Whereas it was used to bound an expectation for KL-UCB, in the proof of Theorem 5.6 we will use it to bound a probability and thus need this exponential decay. This technical difference ushers in the bifurcation between Chernoff information and KL-divergence. Indeed, F C 1 (µ a , c), that is the optimal rate in the exponential (see Section 5.2.4), depends on the problem and to be able to later choose an exploration rate that does not, we have to choose C 1 such that F C 1 (µ a , c) = 1. As we can see below, there is a unique constant C 1 (µ a , c) satisfying F C 1 (µa,c) (µ a , c) = 1 and it is related to Chernoff information: Sample complexity results and discussion. Theorem 5.5 gives an upper bound on the number of samples used by KL-Racing that holds with high probability, when = 0 (i.e. when µ m > µ m+1 and no relaxation is considered). It involves the quantity d * * defined above and related to Chernoff information. We do not provide an upper bound on the expectation of τ for this algorithm, and the proof of Theorem 5.5 share a common structure with the analysis of many algorithms for which only a high-probability bound on τ is provided (e.g. [START_REF] Even-Dar | Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems[END_REF], Gabillon et al., 2012, Karnin et al., 2013]): on some event W on which the algorithm outputs the right subset, the number of draws of each arm is upper bounded deterministically (see details in Section 5.2.4).

F C 1 (µ a , c) = 1 ⇔ d(s C 1 (µ a , c), µ a ) = d(µ a , c) C 1 ⇔ d(s C 1 (µ a , c), µ a ) = d(s C 1 (µ a , c), c) ⇔ s C 1 (µ a ,
Theorem 5.5. Let β(t, δ) = log k 1 Kt α δ , with α > 1 and k 1 > 1 + 1 α-1 . The number of samples τ used in KL-Racing with = 0 is such that ,c),2 2) .

P ν τ ≤ α m a=1 1 d * * (µ a , µ m+1 ) + K a=m+1 1 d * * (µ a , µ m ) log 1 δ + o log 1 δ , Ŝm = S * m ≥ 1 -2δ.
The proof of Theorem 5.6 follow from an upper bound on P(τ ≥ T ) for any deterministic time T and is given in Section 5.2.4. In the sequel, the parameter c ∈ [µ m+1 , µ m ] that we introduce in the analysis will be assumed to be in ]0, 1[, excluding the case µ m = µ m+1 = 0 or 1.

Theorem 5.6. Let c ∈ [µ m , µ m+1 ], ≥ 0. Let β(t, δ) = log k 1 Kt α δ with k 1 > 1 + 1 α-1 . Then for α > 1, P ν τ ≤ 2αH * ,c log ek 1 K(H * ,c ) α δ log k 1 K(H * ,c ) α δ , Ŝm ⊆ S * m, ≥ 1 -2δ.
Moreover, for α > 2,

E ν [τ ] ≤ 4αH * ,c log ek 1 K(H * ,c ) α δ log k 1 K(H * ,c ) α δ + C α , with C α = 2 α-1 δ k 1 ∑ ∞ t=1 log(k 1 Kt α δ)+1 t α-1
.

Theorem 5.5 and Theorem 5.6 provide upper bounds on τ for the KL-Racing and KL-LUCB algorithms that involve informational quantities and explicit constants. For KL-LUCB, we believe that the finite-time upper bound on the sample complexity is the first of its kind involving KL-divergence (through Chernoff information). This results yields (for = 0) an upper bound on κ C (ν), the complexity term introduced in (5.1):

κ C (ν) ≤ 8H * 0 (ν), with H * 0 (ν) ∶= min c∈[µ m+1 ,µm] K a=1 1 d * (µ a , c) .
Pinsker's inequality shows that d * (x, y) ≥ (x -y) 2 2 and d * * (x, y) ≥ (x -y) 2 8, which gives a relationship with the complexity term H(ν) that involves squared gaps defined in (5.2) : H * 0 (ν) ≤ 8H(ν). Although H * 0 (ν) cannot be shown to be strictly smaller than H(ν) on every problem (this will be the case for example when the parameters of the arms are small), the explicit bound in Theorem 5.6 still improves over that of [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF], which implies that κ C (ν) ≤ 192H(ν).

We believe that the constant c that appears in Theorem 5.6 is an artifact of our proof, but we are currently unable to eliminate it. We would therefore conjecture that it is possible to give an upper bound on κ C (ν) that rather involves the quantity

m a=1 1 d * (µ a , µ m+1 ) + K a=m+1 1 d * (µ a , µ m )
.

We can propose an interpretation for this quantity, using that Chernoff information is a relevant quantity in testing problems ( [START_REF] Cover | Elements of Information Theory (2nd Edition)[END_REF]). Let X 1 , X 2 , . . . , X n be n i.i.d. samples and H 1 ∶ X i ∼ B(x) against H 2 ∶ X i ∼ B(y) be two alternative hypotheses. For a test φ, let α n (φ) = P 1 (φ = 2) and β n = P 2 (φ = 1) be respectively the type I and type II error. Chernoff's Theorem states that when the objective is to minimize both type I and type II error, the best achievable exponent is algorithm with KL-LUCB-E, discussed in the paper [START_REF] Kaufmann | Information complexity in bandit subset selection[END_REF], which has a provably-optimal exploration rate involving the problem complexity (H * ≈ 13659). Quite surprisingly, we find that KL-LUCB-log(t) significantly outdoes KL-LUCB-E for every setting of m from 1 to 14. KL-LUCB-log(t) also outperforms the SAR algorithm of [START_REF] Bubeck | Multiple Identifications in multi-armed bandits[END_REF], yielding yet another result in favor of adaptive sampling. A tuned version of KL-LUCB-E (using an exploration rate of n 2×180 ) performs virtually identical to KL-LUCB-log(t), and is not shown in the figure.

d * (x, y) = lim n→∞ - 1 n log min φ max(α n (φ), β n (φ)).

Proofs of the theorems of Section 5.2

Before giving the proofs of Theorems 5.1, 5.5 and 5.6, we introduce the following notation, already used in Chapter 1, that will be useful in the proof:

d + (x, y) = d(x, y)1 (x<y) and d -(x, y) = d(x, y)1 (x>y) .
(5.11)

Proof of Theorem 5.1. We first introduce the following lemma.

Lemma 5.7. KL-LUCB and KL-Racing are such that Ŝm ⊆ S * m, on the event

W = ⋂ t∈N ⋂ a∈S * m (U a (t) > µ a ) ⋂ b∈(S * m ) c (L b (t) < µ b ).
(5.12)

where U and L denote the generic confidence bounds used by these two algorithms.

Proof for Racing. If Racing is not correct, there exists some first round t on which either an arm in (S * m, ) c is selected (first situation), or an arm in S * m is dismissed (second situation). Before t, all the arms in the set of selected arms S are in S * m, , and all the arms in set of discarded arms D are in (S * ) c . In the first situation, let b be the arm in (S * m, ) c selected : for all arms a in J(t) c , one has U a (t) -L b (t) < . Among these arms, at least one must be in S * m . So there exists a ∈ S * m and b ∈ (S * ) c such that U a (t) < L b (t) + . The second situation leads to the same conclusion. Hence if the algorithm fails, the following event holds:

⋃ t∈N (∃a ∈ S * m , ∃b ∈ (S * m, ) c ∶ U a (t) -L b (t) < ) ⊂ ⋃ t∈N ∃a ∈ S * m , ∃b ∈ (S * m, ) c ∶ (U a (t) < µ a ) ∪ (L b (t) > µ a -> µ b ) ⊂ ⋃ t∈N ⋃ a∈S * m (U a (t) < µ a ) ⋃ b∈(S * m, ) c (L b (t) > µ b ) ⊂ W c .
Proof for LUCB. If LUCB is not correct, there exists some stopping time τ , arm a in S * m and an arm b in (S * m, ) c such that a ∈ J(τ ) and b ∈ J(τ ) c . As the stopping condition holds, on has U a (t)-L b (t) < . Using the same reasoning as above, if the algorithm fails, the following event holds:

⋃ t∈N ⋃ a∈S * m (U a (t) < µ a ) ⋃ b∈(S * m, ) c (L b (t) > µ b ) ⊂ W c .

◻

The probability of error of both algorithms is upper bounded as

P( Ŝm ⊈ S * m, ) ≤ a∈S * m P (∃t ∈ N * ∶ u a (t) < µ a ) + b∈(S * m ) c P (∃t ∈ N * ∶ l b (t) > µ b )
And for a ∈ S *

P( Ŝm ⊈ S * m, ) ≤ K ∞ s=1 exp(-β(s, δ)) ≤ ∞ s=1 δ C 1 t α ≤ δ,
which proves Theorem 5.1.

Proof of Theorem 5.5. Let W be the event (5.12) defined in Lemma 5.7, on which the algorithm KL-Racing is correct. We upper bound deterministically the number of samples t a from each arm a used by the algorithm assuming that this event hold.

On W , at every round of the algorithm the set A of accepted arms (resp. D of discarded arms) contains only arms from S * m (resp.

(S * m ) c ). Let a ∈ S * m . If l a (t) is larger than u b (t) for all b in (S * m ) c ∩ R, as the set (S * m ) c ∩ R contains K -m -D arms, l a (t)
is also larger that u b (t) for all b in (J(t)) c , and is thus accepted (and no longer drawn). Indeed, (J(t)) c also contains Km -D arms and if any two arms still in the race are such that μi (t) > μj (t), then u i (t) > u j (t) (this holds because the arms have been drawn the same number of time). This shows that, for a ∈ S * m ,

t a ≤ inf {t ∈ N ∶ ∀b ∈ (S * m ) c ∩ R, l a (t) > u b (t)} . Similarly, for b ∈ (S * m ) c , t b ≤ inf {t ∈ N ∶ ∀a ∈ (S * m ) ∩ R, l a (t) > u b (t)} .
The following lemma then allows to further upper bound t a and t b . Assume that t ≥ T * a,b and W holds. We prove that l a (t) > c and u b (t) < c, which leads to the result. We give here only the proof that l a (t) > c, the proof that u b (t) < c is similar.

We start by showing that μa,t > la . As µ a < u a (t), one has td + (μ a,t , µ a ) ≤ β(t, δ). For t ≥ T * a,b ,

β(t, δ) ≤ td * * (µ a , µ b ) = td( la , µ a ) = td + ( la , µ a ).
Hence, the facts that d + (μ a,t , µ a ) ≤ d + ( la , µ a ) and that x ↦ d + (x, µ a ) is non-increasing (cf. Figure 5.3) implies that μa,t > la . By definition, td -(μ a,t , l a (t)) = β(t, δ). On the one hand the fact that the mapping x ↦ d -(x, l a (t)) is non-decreasing (Figure 5.3) and that, as proved above, μa,t > la , yields td -(μ a,t , l a (t)) > td -( la , l a (t)). On the other hand, as t ≥ T * a,b , one has

β(t, δ) ≤ td * * (µ a , µ b ) = td( la , c) = td -( la , c).
Putting everything together gives d -( la , l a (t)) ≤ d -( la , c). This together with the fact that x ↦ d -( la , x) is non-increasing (Figure 5.3) implies that l a (t) > c. To conclude the proof, it remains to give an upper bound on T * a,m+1 and T * m,b , and Lemma 5.22, given in Appendix 5.6.1, permits to show that

T * a,b ≤ α d * * (µ a , µ b ) log k 1 K δd * * ((µ a , µ b )) α + log log k 1 K δd * * ((µ a , µ b )) α + 1
which gives Theorem 5.5.

Proof of Theorem 5.6. At each round of the KL-LUCB algorithm, exactly two arms are drawn. The total number of samples used τ is therefore such that τ = 2σ where σ is the random number of rounds of the algorithm. Theorem 5.6 easily follows from these two inequalities that holds for any exploration rate:

for α > 1, T ≥ T * 1 , P (σ ≥ T ) ≤ H * ,c e -β(T,δ) + ∞ t=1 (β(t, δ) log(t) + 1)e -β(t,δ) (5.13) for α > 2, T ≥ T * 2 , P (σ ≥ T ) ≤ H * ,c e -β(T,δ) + KT 2 (β(T, δ) log(T ) + 1)e -β(T 2,δ) , (5.14)
with

T * 1 = min{T 0 ∶ ∀T ≥ T 0 , H * c, β(T, δ) < T } and T * 2 = min{T 0 ∶ ∀T ≥ T 0 , 2H * c, β(T, δ) < T },
and from upper bounds on T * 1 and T * 2 obtained from Lemma 5.22. We now prove (5.14). For c ∈ [µ m+1 , µ m ], if the algorithm hasn't stopped at time t, then one of the two intervals I ut (t) or I lt (t) is quite large and contains the parameter c. This simple idea is expressed in Proposition 5.7, which is proved below. To state it, we need to define the event

W t = ⋂ a∈S * m (u a (t) > µ a ) ⋂ b∈(S * m ) c (l b (t) < µ b ) .
Proposition 5.9. If U ut -L lt > and W t holds, then there exists a ∈ {l t , u t } such that c ∈ I a (t) and βa (t) > 2 ,

where we define βa (t) ∶=

β(t,δ) 2Na(t) .
The remainder of this proof borrows from Lemma 5 of [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF]]. Let T be some fixed time and σ the random number of rounds of the algorithm. Our goal is to find an event on which min(σ, T ) < T ; that is, the algorithm must have stopped after T rounds. Writing T = ⌈ T 2 ⌉, we upper bound min(σ, T ):

min(σ, T ) = T + T t=T 1 (σ≥t) = T + T t=T 1 (Uu t -L l t > ) ≤ T + T t=T 1 (Uu t -L l t > ) 1 Wt + T t=T 1 W c t ≤ T + T t=T 1 ∃a∈{ut,lt}∶c∈Ia(t)∩ βa(t)> 2 1 Wt + T t=T 1 W c t ≤ T + K a=1 T t=T 1 (a∈{ut,lt}) 1 (c∈Ia(t)) 1 ( βa(t)> 2 ) (Ba) + T t=T 1 W c t
where the first inequality comes from Proposition 5.7. Let A ∶= a ∈ {1, 2, . . . , K} ∶ d * (µ a , c) < 2 2 . For a ∈ A , the term (B a ) is upper bounded as

(B a ) ≤ T t=T 1 (a∈{ut,lt}) 1 ( βa(t)> 2 ) = T t=T 1 (a∈{ut,lt}) 1 Na(t)< β(t,δ) 2 2 ≤ β(T, δ) 2 2
whereas for a ∉ A , (B a ) is upper bounded as Let A T and B T be the two events

(B a ) ≤ T t=T 1 (a∈{ut,lt}) 1 (c∈Ia(t)) ≤ T t=T 1 (a∈{ut,lt}) 1 Na(t)≤ β(T,
A T = ⋂ a∈A c ⋂ s≥⌈ β(T,δ) d * (µa,c) ⌉+1
(sd(μ a,s , c) ≥ β(T, δ)) and

B T = T ⋂ T W t .
On A T ∩ B T , one has min(σ, T ) ≤ T + H * β(T, δ), thus (σ ≤ T ) for all T ≥ T * 2 , with

T * 2 = min{T 0 ∶ ∀T ≥ T 0 , 2H * c, β(T, δ) < T } Hence for T ≥ T * 2 , P(σ ≥ T ) ≤ P(A c T ) + P(B c T ). From Lemma 1.9, P(W c t ) ≤ a∈S * m P(∃s ≤ t ∶ sd + (µ a,s , µ a ) ≥ β(t, δ)) + a∉S * m P(∃s ≤ t ∶ sd -(µ a,s , µ a ) ≥ β(t, δ)) ≤ Keβ(t, δ)(log t + 1) exp(-β(t, δ))
Using moreover Lemma 5.2, one obtains

P(A c T ) ≤ a∈A c ∞ s=⌈ β(T,δ) d * (µa,c) ⌉+1 P (sd(μ a,s , c) ≤ β(T, δ)) ≤ a∈A c e -β(T,δ) d * (µ a , c) ≤ H * ,c exp (-β(T, δ)) , P(B c T ) ≤ K T t=T eβ(t, δ)(log t + 1) exp(-β(t, δ)) ≤ K T 2 β(T, δ)(log T + 1) exp (-β (T 2, δ)) .
This proves (5.14). The proof of (5.13) follows along the same lines, except we do not introduce T and replace it by zero in the above equations. The introduction of T to show (5.13) is necessary to be able to upper-bound P(σ ≥ T ) by the general term of a convergent series. We now give the proof of Proposition 5.9 .

Proof of Proposition 5.9. We first show that at time t, if the stopping condition does not hold (U ut -L lt > ) and the event W t holds, then either c ∈ I ut (t) or c ∈ I lt (t). This comes from a straightforward adaptation of the beginning of the proof of Lemma 2 from [START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF]. Then we also observe that if U ut -L lt > , the two intervals I ut (t) and I lt (t) cannot be too small simultaneously. Indeed, Pinsker's inequality (5.7) This together with βlt (t) + βut (t) > leads to βut (t) > 2 and proves statement (5.16). The proof of statement (5.18) use identical arguments.

The proof of statement (5.17) goes as follows :

(U ut (t) -L lt (t) > ) ∩ (L ut < c) ∩ (c < L lt (t)) ⇒ (U ut (t) > c + ) ∩ (L ut < c) ⇒ (p ut (t) + βut (t) > c + ) ∩ (p ut (t) -βut (t) < c) ⇒ 2 βut (t) > .
And the proof of statement (5.19) is similar. The lower bound we provide here is not restricted to Bernoulli bandit models discussed in the previous section. Rather, we focus on identifiable classes of bandit models. A class M m of bandit models is called identifiable is there exists a set et probability measures P satisfying ∀p, q ∈ P, p ≠ q ⇒ 0 < KL(p, q) < +∞, such that for all ν = (ν 1 , . . . , ν K ) ∈ M m and for a ∈ {1, . . . , K}, ν a ∈ P.

All the lower bounds we propose in this chapter rely on the powerful technical Lemma 5.10, that was already introduced in Chapter 1 as Lemma 1.3 to prove the lower bound on the regret given in Theorem 1.2. This new, simple expression of a change of distribution was first presented in the paper [Kaufmann et al., 2014a].

Lemma 5.10. Let ν and ν ′ be two bandit models. Let F t = σ(A 1 , Z 1 , . . . , A t , X t ) be the filtration associated to a sampling strategy (A t ). If σ is a stopping time with respect to F t , for any A ∈ F σ such that 0 < P ν (A) < 1, one has 5.20) where d(x, y) ∶= KL(B(x), B(y

K a=1 E ν [N a (σ)]KL(ν a , ν ′ a ) ≥ d(P ν (A), P ν ′ (A)), (
)) = x log(x y) + (1 -x) log((1 -x) (1 -y)).
We now propose a non asymptotic lower bound on the expected number of samples needed to identify the m best arms in the fixed-confidence setting, which straightforwardly yields a lower bound on κ C (ν). Theorem 5.11 holds for an identifiable class of bandit models of the form: 5.21) such that the set of probability measures P satisfies assumption 1 below.

M m = {ν = (ν 1 , . . . , ν K ) ∶ ν i ∈ P, µ [m] > µ [m+1] } (
Assumption 1. For all ν, ν ′ ∈ P 2 such that ν ≠ ν ′ , for all α > 0, there exists ν 1 ∈ P: KL(ν, ν ′ ) < KL(ν, ν 1 ) < KL(ν, ν ′ ) + α and

E X∼ν 1 [X] > E X∼ν ′ [X],
there exists ν 2 ∈ P:

KL(ν, ν ′ ) < KL(ν, ν 2 ) < KL(ν, ν ′ ) + α and E X∼ν 2 [X] < E X∼ν ′ [X].
These conditions are reminiscent of assumptions made by [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF]; they include simple classes of parametric bandits continuously parameterized by their means.

Theorem 5.11. Let ν ∈ M m , where M m is defined by (5.21), and assume that P satisfies Assumption 1; any algorithm that is δ-PAC on M m satisfies, for δ ≤ 0.15,

E ν [τ ] ≥ ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ a∈S * m 1 KL(ν a , ν [m+1] ) + a∉S * m 1 KL(ν a , ν [m] ) ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ log 1 2δ .
Proof Without loss of generality, one may assume that the arms are ordered such that µ 1 ≥ ⋅ ⋅ ⋅ ≥ µ K . Thus S * m = {1, ..., m}. Let A = ((A t ), τ, Ŝm ) be a δ-PAC algorithm and fix α > 0. For all a ∈ {1, . . . , K}, from Assumption 1 there exists an alternative model

ν ′ = (ν 1 , . . . , ν a-1 , ν ′ a , ν a+1 , . . . , ν K )
in which the only arm modified is arm a, and ν ′ a is such that: CHAPTER 5. REFINED FREQUENTIST TOOLS FOR BEST ARM IDENTIFICATION -KL(ν a , ν m+1 ) < KL(ν a , ν ′ a ) < KL(ν a , ν m+1 ) + α and µ ′ a < µ m+1 if a ∈ {1, . . . , m}, -KL(ν a , ν m ) < KL(ν a , ν ′ a ) < KL(ν a , ν m ) + α and µ ′ a > µ m if a ∈ {m + 1, . . . , K}. In particular, in the bandit model ν ′ , the set of optimal arms is no longer {1, . . . , m}. Thus, introducing the event A = ( Ŝm = {1, . . . , m}) ∈ F τ , any δ-PAC algorithm satisfies P ν (A) ≥ 1 -δ and P ν ′ (A) ≤ δ. Lemma 5.10 applied to the stopping time τ (such that N a (τ ) = N a is the total number of draws of arm a) and the monotonicity properties of d(x, y) (x ↦ d(x, y) is increasing when x > y and decreasing when

x < y) yield KL(ν a , ν ′ )E ν [N a ] ≥ d(1 -δ, δ).
From the definition of the alternative model, one obtains for a ∈ {1, . . . , m} or b ∈ {m + 1, . . . , K} respectively, for every α > 0,

E ν [N a ] ≥ d(1 -δ, δ) KL(ν a , ν m+1 ) + α and E ν [N b ] ≥ d(1 -δ, δ) KL(ν b , ν m ) + α .
For δ ≤ 0.15, it can be shown that d(1 -δ, δ) ≥ log(1 (2δ)). Thus, letting α tend to zero and summing over the arms leads to the lower bound on

E ν [τ ] = ∑ K a=1 E ν [N a ]. ◻
Remark 5.12. Lemma 5.10 can also be used to improve the result of [START_REF] Mannor | The Sample Complexity of Exploration in the Multi-Armed Bandit Problem[END_REF]] that holds for m = 1 under the -relaxation described before. Combining the changes of distribution of this paper with Lemma 5.10 yields, for every > 0 and δ ≤ 0.15,

E ν [τ ] ≥ ⎛ ⎝ {a ∶ µ a ≥ µ [1] -} -1 KL B(µ [1] ), B(µ [1] -) + a∶µa≤µ [1] - 1 KL B(µ a ), B(µ [1] + ) ⎞ ⎠ log 1 2δ ,
where X denotes the cardinal of the set X and B(µ) the Bernoulli distribution of mean µ.

A class of exponential bandit models, such that

M m = ν = (ν θ 1 , . . . , ν θ K ) ∶ (θ 1 , . . . , θ K ) ∈ Θ K , θ [m] > θ [m+1] ,
where ν θ belongs to a canonical one-parameter exponential family and has a density with respect to some reference measure given by

f θ (x) = A(x) exp(θx -b(θ)), for θ ∈ Θ ⊂ R (5.22)
is an example of class satisfying Assumption 1. Using the shorthand K(θ, θ ′ ) = KL(ν θ , ν θ ′ ) for (θ, θ ′ ) ∈ Θ 2 , the lower bound of Theorem 5.11 together of the upper bound on E[τ ] for KL-LUCB (that can be generalized to exponential families) yield 5.23) where K * (θ 1 , θ 2 ) is the Chernoff information between the distributions ν θ 1 , ν θ 2 , defined, as a function of the natural parameters, as

m a=1 1 K(θ a , θ m+1 ) + K a=m 1 K(θ a , θ m ) ≤ κ C (ν) ≤ 8 min θ∈[θ m+1 ,θm] K a=1 1 K * (θ a , θ) , (
K * (θ 1 , θ 2 ) = K(θ * , θ 1 ), where K(θ * , θ 1 ) = K(θ * , θ 2 ).
A gap remains between the upper and lower bounds in (5.23), even when K = 2. For two armedbandits we propose in the next Section a refined lower bounds on both κ C (ν) and κ B (ν) along with matching algorithms.

The complexity of A/B Testing

In this section, we present our contributions related to the complexity of best arm identification in two-armed bandit models, following closely the paper [Kaufmann et al., 2014a]. Two armed-bandits are of particular interest as they offer a theoretical framework for sequential A/B Testing. A/B Testing is a popular procedure used, for instance, for website optimization: two versions of a webpage, say A and B, are empirically compared by being presented to users. Each user is shown only one version A t ∈ {1, 2} and provides a real-valued index of the quality of the page, X t , which is modeled as a sample of a probability distribution ν 1 or ν 2 . For example, a standard objective is to determine which webpage has the highest conversion rate (probability that a user actually becomes a customer) by receiving binary feedback from the users.

In standard A/B Testing algorithms, the two versions are presented equally often. It is thus of particular interest to investigate whether an algorithm using a (pure) uniform sampling strategy, such that the arms are sampled in a round-robin fashion, can be efficient in two-armed bandit model. We saw in Section 5.2 that when there are more than K > 2 uniform sampling in the above sense is not desirable (it should at least be coupled with eliminations). However, when there are two arms, KL-LUCB and KL-Racing reduce to the same algorithm, sampling both arms at each round. An algorithm using uniform sampling can be regarded as a statistical test of the hypothesis

H 0 ∶ (µ 1 ≤ µ 2 ) against H 1 ∶ (µ 1 > µ 2 )
based on paired samples (X s , Y s ) of ν 1 , ν 2 ; namely a test based on a fixed number of samples in the fixedbudget setting, and, a sequential test in the fixed-confidence setting, in which a randomized stopping rule determines when the experiment is to be terminated.

In two-armed bandit models, classical sequential testing theory provides a first element of comparison between the fixed-budget and fixed-confidence settings, in the simpler case of fully specified alternatives. Consider for instance the case where ν 1 and ν 2 are Gaussian laws with the same known variance σ 2 , the means µ 1 and µ 2 known up to a permutation. Denoting by P the joint distribution of the paired samples (X s , Y s ), one must choose between the hypotheses H 0 ∶ P = N µ 1 , σ 2 ⊗ N µ 2 , σ 2 and H 1 ∶ P = N µ 2 , σ 2 ⊗ N µ 1 , σ 2 . It is known since [Wald, 1945] that among the sequential tests such that type I and type II error probabilities are both smaller than δ, the Sequential Probability Ratio Test (SPRT) minimizes the expected number of required samples, and is such that

E ν [τ ] = 2σ 2 (µ 1 -µ 2 ) 2 log(1 δ).
However, the batch test that minimizes both probabilities of error is the Likelihood Ratio test; it can be shown to require a sample size of order 8σ 2 (µ 1 -µ 2 ) 2 log(1 δ) in order to ensure that both type I and type II error probabilities are smaller than δ. Thus, when the sampling strategy is uniform and the parameters are known, there is a clear gain in using randomized stopping strategies. We will show below that this conclusion is not valid anymore when the values of µ 1 and µ 2 are not assumed to be known. Indeed, for two-armed Gaussian bandit models we show that κ B (ν) = κ C (ν) and for two-armed Bernoulli bandit models we show that κ C (ν) > κ B (ν).

To prove this, we start by giving in Section 5.4.1 a refined lower bound on κ C (ν), based on a different change of distribution, as well as a lower bound on κ B (ν). We then provide in two particular cases, Gaussian bandits with known variances (Section 5.4.2) and Bernoulli bandits (Section 5.4.3) efficient algorithms (almost) matching these bounds. In particular, we show that for Bernoulli bandits only little can be gained by departing from uniform sampling, and propose an algorithm for the fixed-confidence setting based on a non-trivial stopping criterion that is reminiscent of KL-LUCB.

Lower bounds on the two complexities

For M 1 = M an identifiable class of two-armed bandit models, Theorem 5.13 provides lower bounds on κ B (ν) and κ C (ν) for every ν ∈ M. Theorem 5.13. Let ν = (ν 1 , ν 2 ) be a two-armed bandit model such that µ 1 > µ 2 . In the fixed-budget setting, any consistent algorithm satisfies

lim sup t→∞ - 1 t log p t (ν) ≤ c * (ν), where c * (ν) ∶= inf (ν ′ 1 ,ν ′ 2 )∈M∶µ ′ 1 <µ ′ 2 max KL(ν ′ 1 , ν 1 ), KL(ν ′ 2 , ν 2 ) .
In the fixed-confidence setting any algorithm that is δ-PAC on M satisfies, when δ ≤ 0.15,

E ν [τ ] ≥ 1 c * (ν) log 1 2δ
, where c * (ν) ∶= inf

(ν ′ 1 ,ν ′ 2 )∈M∶µ ′ 1 <µ ′ 2 max KL(ν 1 , ν ′ 1 ), KL(ν 2 , ν ′ 2 ) .
In particular, Theorem 5.13 implies that κ B (ν) ≥ 1 c * (ν) and κ C (ν) ≥ 1 c * (ν). Proceeding similarly, one can obtain lower bounds for the algorithms that use uniform sampling of both arms. The proof of both results is provided below.

Theorem 5.14. Let ν = (ν 1 , ν 2 ) be a two-armed bandit model such that µ 1 > µ 2 . In the fixed-budget setting, any consistent algorithm using a uniform sampling strategy satisfies

lim sup t→∞ - 1 t log p t (ν) ≤ I * (ν) where I * (ν) ∶= inf (ν ′ 1 ,ν ′ 2 )∈M∶µ ′ 1 <µ ′ 2 KL (ν ′ 1 , ν 1 ) + KL (ν ′ 2 , ν 2 ) 2 .
In the fixed-confidence setting, any algorithm that is δ-PAC on M and uses a uniform sampling strategy satisfies, for δ ≤ 0.15,

E ν [τ ] ≥ 1 I * (ν) log 1 2δ where I * (ν) ∶= inf (ν ′ 1 ,ν ′ 2 )∈M∶µ ′ 1 <µ ′ 2 KL (ν 1 , ν ′ 1 ) + KL (ν 2 , ν ′ 2 ) 2 .
Obviously, one always has I * (ν) ≤ c * (ν) and I * (ν) ≤ c * (ν) suggesting that uniform sampling can be sub-optimal. It is possible to give explicit expressions for the quantities c * (ν), c * (ν) and I * (ν), I * (ν) for specific classes of parametric bandit models that will be studied in the next Sections. In the case of two-armed Gaussian bandits with known variance (see Section 5.4.2): 5.24) using that

M = {ν = N µ 1 , σ 2 1 , N µ 2 , σ 2 2 ∶ (µ 1 , µ 2 ) ∈ R 2 , µ 1 ≠ µ 2 }, (
KL(N (µ 1 , σ 1 ) , N (µ 2 , σ 2 )) = (µ 1 -µ 2 ) 2 2σ 2 2 + 1 2 σ 2 1 σ 2 2 -1 -log σ 2 1 σ 2 2 , (5.25) one obtains c * (ν) = c * (ν) = (µ 1 -µ 2 ) 2 2(σ 1 + σ 2 ) 2 and I * (ν) = I * (ν) = (µ 1 -µ 2 ) 2 4(σ 2 1 + σ 2 2 )
.

Hence, the lower bounds of Theorem 5.13 are equal in this case, and we provide in Section 5.4.2 matching upper bounds confirming that indeed κ B (ν) = κ C (ν). In addition, the observation that, when the variances are different c * (ν) > I * (ν), will be shown to imply that strategies based on uniform sampling are sub-optimal. The values of c * (ν) and c * (ν) can also be computed in the class of two-armed exponential bandit models,

M = {ν = (ν θ 1 , ν θ 2 ) ∶ (θ 1 , θ 2 ) ∈ Θ 2 , θ 1 ≠ θ 2 }
where ν θa has density f θa given by (5.22). One can show that

c * (ν) = inf θ∈Θ max (K(θ, θ 1 ), K(θ, θ 2 )) = K(θ * , θ 1 ), where K(θ * , θ 1 ) = K(θ * , θ 2 ), c * (ν) = inf θ∈Θ max (K(θ 1 , θ), K(θ 2 , θ)) = K(θ 1 , θ * ), where K(θ 1 , θ * ) = K(θ 2 , θ * ).
The coefficient c * (ν) is equal to the Chernoff information K * (θ 1 , θ 2 ) between the arms, already introduced in Section 5.2, whereas c * (ν) corresponds to a quantity close to the Chernoff information but with 'reversed' roles for the arguments. By analogy, we denote this quantity by

K * (θ 1 , θ 2 ) = K(θ 1 , θ * ).
For exponential bandits the quantities c * (ν) and c * (ν) are not equal in general, although it can be shown that it is the case when the log-partition function b(θ) is (Fenchel) self-conjugate (e.g., for Gaussian and exponential variables). In Section 5.4.3, we will focus on the case of Bernoulli models for which c * (ν) > c * (ν). By exhibiting a matching strategy in the fixed-budget setting, we will show that this implies that κ C (ν) > κ B (ν) in this case.

On the changes of distribution used. Theorem 5.11 applied to a two-armed exponential bandit model ν

= (ν θ 1 , ν θ 2 ) yields κ C (ν) ≥ 1 K(θ 1 , θ 2 ) + 1 K(θ 2 , θ 1 ) , (5.26)
while the lower bound given in Theorem 5.13 is .27) which can be shown to be always tighter than (5.26). Interestingly, the changes of distribution used to derive the two results are not the same. On the one hand, for inequality (5.26), the changes of distribution involved modify a single arm at a time: one of the arms is moved just below (or just above) the other (see Figure 5.4,left). This is the idea also used, for example, to obtain the lower bound of [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] on the cumulative regret (see the proof of Theorem 1.2 in Chapter 1). On the other hand, for inequality (5.27), both arms are modified at the same time: they are moved close to the common intermediate value θ * but with a reversed ordering (see Figure 5.4, right). In the fixed-budget setting, the changes of distribution used to obtain the lower bound on κ B (ν) that follow from Theorem 5.13 moves both arms close to the value θ * . Proof of Theorem 5.13 and 5.14. Without loss of generality, assume that the bandit model ν = (ν 1 , ν 2 ) is such that a * = 1. Consider any alternative bandit model ν ′ = (ν ′ 1 , ν ′ 2 ) in which a * = 2. For a given strategy A, let A be the event A = ( Ŝ1 = 1), which belongs to F τ , for τ the stopping rule of A.

κ C (ν) ≥ 1 K * (θ 1 , θ 2 ) . ( 5
θ 2 θ 1 θ 1 +α θ 2 θ 1 θ * θ * +α
Fixed-budget setting Assume the strategy A is consistent. For every t ∈ N, if the budget is τ = t a.s., Lemma 5.10 applied to the stopping time σ = t and the event A = ( Ŝ1 = 1) ∈ F t defined above yields

E ν ′ [N 1 (t)]KL(ν ′ 1 , ν 1 ) + E ν ′ [N 2 (t)]KL(ν ′ 2 , ν 2 ) ≥ d(P ν ′ (A), P ν (A)
). One has p t (ν) = 1 -P ν (A) and p t (ν ′ ) = P ν ′ (A). As A is consistent, for every > 0 there exists t 0 ( ) such that for all t ≥ t 0 ( ), P ν ′ (A) ≤ ≤ P ν (A). For t ≥ t 0 ( ),

E ν ′ [N 1 (t)]KL(ν ′ 1 , ν 1 ) + E ν ′ [N 2 (t)]KL(ν ′ 2 , ν 2 ) ≥ d( , 1 -p t (ν)) ≥ (1 -) log 1 - p t (ν) + log .
Taking the limsup and letting go to zero, one can show that

lim sup t→∞ - 1 t log p t (ν) ≤ lim sup t→∞ E ν ′ [N 1 (t)] t KL(ν ′ 1 , ν 1 ) + E ν ′ [N 2 (t)] t KL(ν ′ 2 , ν 2 ) ≤ max a=1,2 KL(ν ′ a , ν a ).
The first statement of Theorem 5.13 follows by optimizing over the possible model ν

′ satisfying µ ′ 1 < µ ′ 2
to make the right hand side of the inequality as small as possible.

If the sampling strategy of A is uniform, using that E

ν [N 1 ] = E[N 2 ] = E[τ ] 2, lim sup -1 t log p t (ν) is upper bounded by (KL(ν ′ 1 , ν 1 ) + KL(ν ′ 2 , ν 2
)) 2, and similarly optimizing over the choice of ν ′ gives the first statement of Theorem 5.14.

Fixed-confidence setting Assume the strategy A is δ-PAC. It therefore satisfies P ν (A) ≥ 1 -δ and P ν ′ (A) ≤ δ. Applying Lemma 5.10 (with the stopping rule τ ) and using again the monotonicity properties of d(x, y), one obtains that

E ν [N 1 ]KL(ν 1 , ν ′ 1 ) + E ν [N 2 ]KL(ν 2 , ν ′ 2 ) ≥ d(δ, 1 -δ).
(5.28)

For δ ≤ 0.15, as already used in the proof of Theorem 5.11, one has d(δ, 1 -δ) ≥ log(1 (2δ)). Using moreover that τ = N 1 + N 2 , one has

E ν [τ ] ≥ 1 max a KL(ν a , ν ′ a ) log 1 2δ .
The second statement of Theorem 5.13 follows by optimizing over the possible model ν ′ satisfying µ ′ 1 < µ ′ 2 to make the right hand side of the inequality as large as possible. If A uses a uniform sampling strategy, using the fact that E (5.28) similarly gives the second statement of Theorem 5.14.

ν [N 1 ] = E[N 2 ] = E[τ ] 2 in Equa- tion

The Gaussian Case

We study in this Section the class of two-armed Gaussian bandit models with known variances defined by (5.24), where σ 1 and σ 2 are fixed. In this case, we observed above that the lower bounds of Theorem 5.13 are similar, because c * (ν) = c * (ν). We prove in this section that indeed

κ C (ν) = κ B (ν) = 2(σ 1 + σ 2 ) 2 (µ 1 -µ 2 ) 2
by exhibiting strategies that reach these performance bounds. These strategies are based on the simple recommendation of the empirical best arm but use non-uniform sampling in cases where σ 1 and σ 2 differ. When σ 1 = σ 2 we provide in Theorem 5.15 an improved stopping rule that is δ-PAC but results in a significant reduction of the running time of fixed-confidence algorithms.

FIXED-BUDGET SETTING

We consider the simple family of static strategies that draw n 1 samples from arm 1 followed by n 2 = tn 1 samples of arm 2, and then choose arm 1 if μ1,n 1 < μ2,n 2 , where μi,n i denotes the empirical mean of the n i samples from arm i. Assume for instance that µ

1 > µ 2 . Since μ1,n 1 -μ2,n 2 -µ 1 + µ 2 ∼ N 0, σ 2 1 n 1 + σ 2 2 n 2
, the probability of error of such a strategy is easily upper bounded as:

P (μ 1,n 1 < μ2,n 2 ) ≤ exp ⎛ ⎝ - σ 2 1 n 1 + σ 2 2 n 2 -1 (µ 1 -µ 2 ) 2 2 ⎞ ⎠ .
The right hand side is minimized when n

1 (n 1 + n 2 ) = σ 1 (σ 1 + σ 2 )
, and the static strategy drawing

n 1 = ⌈σ 1 t (σ 1 + σ 2 )⌉ times arm 1 is such that lim inf t→∞ - 1 t log p t (ν) ≥ (µ 1 -µ 2 ) 2 2(σ 1 + σ 2 ) 2 ,
which matches the bound of Theorem 5.13 for Gaussian bandit models.

FIXED-CONFIDENCE SETTING

Equal Variances. We start with the simpler case σ 1 = σ 2 = σ. Thus, the quantity I * (ν) introduced in Theorem 5.14 coincides with c * (ν), which suggests that uniform sampling could be optimal. A uniform sampling strategy equivalently collects paired samples (X s , Y s ) from both arms. The difference X s -Y s is normally distributed with mean µ = µ 1 -µ 2 and a δ-PAC algorithm is equivalent to a sequential test of H 0 ∶ (µ < 0) versus H 1 ∶ (µ > 0) such that both type I and type II error probabilities are bounded by δ. [Robbins, 1970] proposes the stopping rule

τ = inf ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ t ∈ 2N * ∶ t 2 s=1 (X s -Y s ) > 2σ 2 tβ(t, δ) ⎫ ⎪ ⎪ ⎬ ⎪ ⎪ ⎭ , with β(t, δ) = t + 1 t log t + 1 2δ . ( 5

.29)

The recommendation rule chooses the empirically best arm at time τ . This procedure can be seen as an elimination strategy, in the sense of [START_REF] Jennison | Asymptotically optimal procedures for sequential adaptive selection of the best of several normal means[END_REF]. The authors of this paper derive a lower bound on the sample complexity of any δ-PAC elimination strategy (whereas our lower bound applies to any δ-PAC algorithm) which is matched by Robbins' algorithm: the above stopping rule τ satisfies

lim δ→0 E ν [τ ] log(1 δ) = 8σ 2 (µ 1 -µ 2 ) 2 .
This value coincide with the lower bound on κ C (ν) of Theorem 5.13 in the case of two-armed Gaussian distributions with similar known variance σ 2 . This proves that in this case, Robbins' rule (5.29) is not only optimal among the class of elimination strategies, but also among the class of δ-PAC algorithm.

Any δ-PAC elimination strategy that uses a threshold function (or exploration rate) β(t, δ) smaller than Robbins' also matches our asymptotic lower bound, while stopping earlier than the latter. From a practical point of view, it is therefore interesting to exhibit smaller exploration rates that preserve the δ-PAC property. The failure probability of such an algorithm is upper bounded, for example when µ 1 < µ 2 , by

P ν ∃k ∈ N ∶ k s=1 X s -Y s -(µ 1 -µ 2 ) √ 2σ 2 > 2kβ(2k, δ) = P ∃k ∈ N ∶ S k > 2kβ(2k, δ) (5.30)
where S k is a sum of k i.i.d. variables of distribution N (0, 1). [Robbins, 1970] obtains a nonexplicit confidence region of risk at most δ by choosing β(2k, δ) = log (log(k) δ) + o(log log(k)).

The dependency in k is in some sense optimal, because the Law of Iterated Logarithm (LIL) states that lim sup k→∞ S k 2k log log(k) = 1 almost surely. Recently, [START_REF] Jamieson | lil'UCB: an Optimal Exploration Algorithm for Multi-Armed Bandits[END_REF] proposed an explicit confidence region inspired by the LIL. However, Lemma 1 of [START_REF] Jamieson | lil'UCB: an Optimal Exploration Algorithm for Multi-Armed Bandits[END_REF] cannot be used to upper bound (5.30) by δ and we provide in Appendix A a result derived independently (Lemma A.1 therein) that achieves this goal and can be used to obtain the following result (proved in Section 5.4.5).

Theorem 5.15. For δ small enough, the elimination strategy with threshold g

(t, δ) = 2σ 2 tβ(t, δ) is δ-PAC with β(t, δ) = log 1 δ + 3 4 log log 1 δ + 3 2 log(1 + log(t 2)).
(5.31)

We refer to Section 5.4.4 for numerical simulations that illustrate the significant savings (in the average number of samples needed to reaching a decision) resulting from the use of the less conservative exploration rate allowed by Theorem 5.15.

Mismatched Variances. In the case where σ 1 ≠ σ 2 , we rely on the α-Elimination strategy, described in Algorithm 5 below. For a = 1, 2, μa (t) denotes the empirical mean of the samples gathered from arm a up to time t. The algorithm is based on a non-uniform sampling strategy governed by the parameter α ∈ (0, 1) which ensures that, at the end of every round t, N 1 (t) = ⌈αt⌉, N 2 (t) = t -⌈αt⌉ and μ1 (t) -μ2 (t) ∼ N µ 1 -µ 2 , σ 2 t (α) (where σ 2 t (α) is defined at line 6 of Algorithm 5). The sampling schedule used here is thus deterministic.

Algorithm 5 α-Elimination

Require: Exploration function β(t, δ), parameter α.

1: Initialization: μ1 (0) = μ2 (0) = 0, σ 2 0 (α) = 1, t = 0 2: while μ1 (t) -μ2 (t) ≤ 2σ Theorem 5.16 shows that the σ 1 (σ 1 + σ 2 )-elimination algorithm, with a suitable exploration rate, is δ-PAC and matches the lower bound on E ν [τ ], at least asymptotically when δ → 0. Its proof can be found in Section 5.4.5.

Theorem 5.16. If α = σ 1 (σ 1 + σ 2 ), the α-elimination strategy using the exploration rate β(t, δ) = log t δ + 2 log log(6t) is δ-PAC on M and satisfies, for every ν ∈ M, for every > 0,

E ν [τ ] ≤ (1 + ) 2(σ 1 + σ 2 ) 2 (µ 1 -µ 2 ) 2 log 1 δ + o δ→0 log 1 δ .
Remark 5.17. When σ 1 = σ 2 , 1 2-elimination reduces, up to rounding effects, to the elimination procedure described in the previous paragraph, for which Theorem 5.15 suggests an exploration rate of order log(log(t) δ). As the feasibility of this exploration rate when σ 1 ≠ σ 2 is yet to be established, we focus on Gaussian bandits with equal variances in the numerical experiments of Section 5.4.4.

The Bernoulli Case

We consider in this section the class of Bernoulli bandit models defined by

M = {ν = (B(µ 1 ), B(µ 2 )) ∶ (µ 1 , µ 2 ) ∈]0; 1[ 2 , µ 1 ≠ µ 2 },
where each arm can be equivalently parameterized by the natural parameter of the exponential family, θ a = log(µ a (1 -µ a )). Recall the Kullback-Leibler divergence between two Bernoulli distributions can be indifferently considered as a function of the means, d(µ 1 , µ 2 ) = KL(B(µ 1 ), B(µ 2 )), or of the natural parameters, K(θ 1 , θ 2 ).

In this Section, we prove that κ C (ν) > κ B (ν) for Bernoulli bandit models (Proposition 5.19). To do so, we first introduce a static strategy matching the lower bound of Theorem 5.13 in the fixed-budget setting (Proposition 5.18). This strategy is reminiscent of the algorithm exhibited for Gaussian bandits in Section 5.4.2 and uses parameter-dependent non uniform sampling. This strategy is not directly helpful in practice but we show that it can be closely approximated by an algorithm using uniform sampling. In the fixed-confidence setting we similarly conjecture that little can be gained from using a non-uniform sampling strategy and propose an algorithm based on a non-trivial stopping strategy that is believed to match the bound of Theorem 5.14.

FIXED-BUDGET SETTING

By carefully upper bounding the probability of error of a static strategy in the Bernoulli case, one can show the following result, proved in Section 5.6.3. Proposition 5.18. Let α(θ 1 , θ 2 ) be defined by

α(θ 1 , θ 2 ) = θ * -θ 1 θ 2 -θ 1 where K(θ * , θ 1 ) = K(θ * , θ 2 ).
For all t, the static strategy that allocates ⌈α(θ 1 , θ 2 )t⌉ samples to arm 1, and recommends the empirical best arm, satisfies p t (ν) ≤ exp(-tK * (θ 1 , θ 2 )).

This shows in particular that for every ν ∈ M there exists a consistent static strategy such that

lim inf t→∞ - 1 t log p t ≥ K * (θ 1 , θ 2 ), and hence that κ B (ν) = 1 K * (θ 1 , θ 2 )
.

By combining this observation with Theorem 5.13 and the fact that for Bernoulli distributions it can be shown that K * (θ 1 , θ 2 ) < K * (θ 1 , θ 2 ), one obtains the following inequality.

Proposition 5.19. For all ν ∈ M, κ C (ν) > κ B (ν).

Note that we have determined the complexity of the fixed-budget setting by exhibiting and algorithm that is of limited practical interest for Bernoulli bandit models. Indeed, the optimal static strategy defined in Proposition 5.18 requires the knowledge of the quantity α(θ 1 , θ 2 ), that depends in general on the unknown means of the arms. So far, it is not known whether there exists a universal strategy, that would satisfy p t (ν) ≤ exp(-K * (θ 1 , θ 2 )t) on every Bernoulli bandit model. However, Lemma 5.23 in Section 5.6.3 shows that the strategy using uniform sampling and recommending the empirical best arm satisfies p t (ν) ≤ exp(-I * (ν)t), where I * (ν) is the quantity defined in Theorem 5.14 whose expression for Bernoulli distributions is

I * (ν) = I * (θ 1 , θ 2 ) = K θ 1 +θ 2 2 , θ 1 + K θ 1 +θ 2 2 , θ 2 2 .
Hence this simple strategy matches the bound of Theorem 5.14 in the fixed-budget setting (see Remark TC). It can be moreover observed that I * (ν) is very close to c * (ν) = KL * (θ 1 , θ 2 ), and thus the problem-dependent optimal strategy described in Proposition 5.18 can be approximated by a very simple, universal algorithm. This fact is illustrated in

FIXED-CONFIDENCE SETTING

As illustrated in Figure 5.5, c * (ν) and I * (ν) are also very close, thus there is a strong incentive to use uniform sampling in the fixed-confidence setting as well. Finding an algorithm sampling the arms uniformly and matching the bound of Theorem 5.14 is therefore a crucial matter. This boils down to determining a proper stopping rule. In all the algorithms studied so far, the stopping rule was based on the difference of the empirical means of the arms. For Bernoulli arms the 1/2-Elimination procedure described in Algorithm 5 can be used, as each distribution ν a is bounded and therefore 1/4-subgaussian. More precisely, with β(t, δ) as in Theorem 5.15, the algorithm stopping at the first time t such that μ1 (t) -μ2 (t) > 2β(t, δ) t has its sample complexity bounded by 2 (µ 1 -µ 2 ) 2 log(1 δ) + o (log(1 δ)). The expressions of I * (ν) as a function of the means of the arms is given by

I * (ν) = I * (µ 1 , µ 2 ) = d µ 1 , µ 1 +µ 2 2 + d µ 2 , µ 1 +µ 2 2 2 .
Pinsker's inequality implies that I * (µ 1 , µ 2 ) > (µ 1 -µ 2 ) 2 2 and this algorithm does not match the lower bound of Theorem 5.14 relative to the fixed-confidence setting. The approximation

I * (µ 1 , µ 2 ) = (µ 1 - Algorithm 6 Sequential Generalized Likelihood Ratio Test (SGLRT) Require: Exploration function β(t, δ). 1: Initialization: μ1 (0) = μ2 (0) = 0. t = 0. 2: while (tI * (μ 1 (t), μ2 (t)) ≤ β(t, δ)) ⋃(t = 1[2]) do 3: t = t + 1. A t = t[2]. 4:
Observe X t ∼ ν At and compute the empirical means μ1 (t) and μ2 (t). 5: end while 6: return a = argmax a=1,2 μa (t).

µ 2 ) 2 (8µ 1 (1 -µ 1 )) + o (µ 1 -µ 2 )
2 suggests that the loss with respect to the optimal error exponent is particularly significant when both means are close to 0 or 1.

To circumvent this drawback, we propose the SGLRT (for Sequential Generalized Likelihood Ratio Test) algorithm, described in Algorithm 6. The stopping rule is based on the distance between the empirical means of the arms, measured with the function I * , and is related to the generalized likelihood ratio statistic for testing the equality of two Bernoulli proportions. To test

H 0 ∶ (µ 1 = µ 2 ) against H 1 ∶ (µ 1 ≠ µ 2 ) based on t 2 paired samples of the arms W s = (X s , Y s ), the Generalized Likelihood Ratio Test (GLRT) rejects H 0 when exp(-tI * (μ 1,t 2 , μ2,t 2 )) = max µ 1 ,µ 2 ∶µ 1 =µ 2 L(W 1 , . . . , W t 2 ; µ 1 , µ 2 ) max µ 1 ,µ 2 L(W 1 , . . . , W t 2 ; µ 1 , µ 2 ) < z δ ,
where L(W 1 , . . . , W t 2 ; µ 1 , µ 2 ) denote the likelihood of the observations given parameters µ 1 and µ 2 .

The equality in the previous display is a consequence of the rewriting

I * (x, y) = H x + y 2 - 1 2 H x 2 + H y 2 ,
where H(x) = -x log(x) -(1 -x) log(1 -x) denotes the binary entropy function. Hence, Algorithm (6) can be interpreted as a sequential version of the GLRT with (varying) threshold z t,δ = exp(-β(t, δ)).

µ 1,t µ 2,t l 1 (t) u 2 (t) β(t,δ)/t d * (µ 1,t ,µ 2,t ) x -> d(µ 1,t ,x) x -> d(µ 2,t ,x)
(t, δ) t is below d * (μ 1,t , μ2,t )
Elements of analysis of the SGLRT. The SGLRT algorithm is also related to the KL-LUCB or KL-Racing algorithms described in Section 5.2. Indeed, for two armed-bandits, both algorithm use uniform sampling and stop at round t (having used 2t samples of the arms), when l 1 (t) > u 2 (t) or l 2 (t) > u 1 (t). Figure 5.6 can help convince oneself that this stopping condition is equivalent to stopping when d * (μ 1,t , μ2,t ) > β(t, δ) t, where β(t, δ) is the exploration rate used by KL-LUCB.

From Figure 5.5, I * (x, y) mostly coincides with d * (x, y) and more precisely I * (x, y) < d * (x, y). Using all this, one can upper bound the probability of error of the SGLRT, for example when µ 1 < µ 2 :

P ν ∃t ∈ 2N * ∶ μ1,t 2 > μ2,t 2 , tI * (μ 1,t 2 , μ2,t 2 ) > β(t, δ) ≤ P ν ∃t ∈ 2N * ∶ μ1,t 2 > μ2,t 2 , (t 2)d * (μ 1,t 2 , μ2,t 2 ) > (β(t, δ) 2) = P ν (∃s ∈ N * ∶ μ1,s > μ2,s , sd * (μ 1,s , μ2,s ) > (β(2s, δ) 2))
This is an upper bound of the probability of error of KL-LUCB using the exploration rate β(t, δ) = β(2t, δ) 2 and proceeding as in Theorem 5.1, one can prove the following result.

Lemma 5.20. With the exploration rate

β(t, δ) = 2 log t(log(3t)) 2 δ the SGLRT algorithm is δ-PAC.
For this exploration rate, we were able to obtain the following asymptotic guarantee on the stopping time τ of Algorithm 6, using Lemma 5.21 below (proved in Section 5.6.4):

∀α > 0, lim sup δ→∞ τ log(1 δ) ≤ 2(1 + α) I * (µ 1 , µ 2 ) a.s.
Still from Lemma 5.21, it follows that with an exploration rate β(t, δ) = log((log(t)+1) δ) -for which the SGLRT algorithm is not provably δ-PAC-, one has ∀α > 0, lim sup

δ→∞ τ log(1 δ) ≤ (1 + α) I * (µ 1 , µ 2 )
a.s..

No upper bound on E ν [τ ] log(1 δ) can be deduced from this result, but it provides an intuition on which stopping rule to use. By analogy with the result of Theorem 5.15 we conjecture that the use of and exploration rate of order log(log(t) δ) should also lead to a δ-PAC algorithm. This conjecture is supported by the numerical experiments reported in Section 5.4.4 below.

Lemma 5.21. Let f and g be two continuous function such that f (µ 1 , µ 2 ) ≠ 0 and g(t) = o(t r ) for all r > 1. For every α > 0 the strategy using uniform sampling and the stopping rule

τ = inf t ∈ 2N * ∶ t f (μ 1,t 2 , μ2,t 2 ) ≥ log g(t) δ satisfies P ν lim sup δ→0 τ log(1 δ) ≤ 1 + α f (µ 1 , µ 2 ) = 1.

Numerical experiments

The goal of this Section is twofold: to compare results obtained in the fixed-budget and fixedconfidence settings and to illustrate the improvement resulting from the adoption of the reduced exploration rate of Theorem 5.15.

In Figure 5.7, we consider two Gaussian bandit models with known common variance: the 'easy' one is {N (0.5, 0.25) , N (0, 0.25)}, corresponding to κ C = κ B = κ = 8, on the left; and the 'difficult' one is {N (0.01, 0.25) , N (0, 0.25)}, that is κ = 2 × 10 4 , on the right. In the fixed-budget setting, stars ('*') report the probability of error p n (ν) as a function of n. In the fixed-confidence setting, we plot both the empirical probability of error by circles ('O') and the specified maximal error probability δ by crosses ('X') as a function of the empirical average of the running times. Note the logarithmic scale used for the probabilities on the y-axis. All results are averaged on N = 10 6 independent Monte Carlo replications. For comparison purposes, a plain line represents the theoretical rate x ↦ exp(-x(1 κ)) which is a straight line on the log scale.

In the fixed-confidence setting, we report results for algorithms of the form (5.29) with g(t, δ) = 2σ 2 tβ(t, δ) for three different exploration rates β(t, δ). The exploration rate we consider are: the provably-PAC rate of Robbins' algorithm log(t δ) (large blue symbols), the conjectured 'optimal' exploration rate log((log(t) + 1) δ), almost provably δ-PAC according to Theorem 5.15 (bold green symbols), and the rate log(1 δ), which would be appropriate if we were to perform the stopping test only at a single pre-specified time (orange symbols). For each algorithm, the log probability of error is approximately a linear function of the number of samples, with a slope close to -1 κ, where κ is the complexity. We can visualize the gain in sample complexity achieved by smaller exploration rates, but while the rate log((log(t) + 1) δ) appears to guarantee the desired probability of error across all problems, the use of log(1 δ) seems too risky, as one can see that the probability of error becomes larger than δ on difficult problems. To illustrate the gain in sample complexity when the means of the arms are known, we add in red the SPRT algorithm mentioned in the introduction of Section 5.4 along with the theoretical relation between the probability of error and the expected number of samples, materialized as a dashed line. The SPRT stops for t such that (µ 1 -µ 2 )(S 1,t 2 -S 2,t 2 ) > log(1 δ).

Robbins' algorithm is δ-PAC and matches the complexity (which is illustrated by the slope of the measures), though in practice the use of the exploration rate log((log(t) + 1) δ) leads to huge gain in terms of number of samples used. It is important to keep in mind that running times play the same role as error exponents and hence the threefold increase of average running times observed on the rightmost plot of Figure 5.7 when using β(t, δ) = log(t δ) is really prohibitive.

In Figure 5.8, we compare on two Bernoulli bandit models the performance of the SGLRT algorithm (Algorithm 6) using two different exploration rates, log(1 δ) and log((log(t) + 1) δ), to the algorithm that stops when the difference of empirical means exceeds the threshold 2β(t, δ) t (for the same exploration rates), that we refer to as 'Elimination'. Plain lines also materialize the theoretical optimal rate x ↦ exp(-x κ C (ν)) and the rate attained by the Elimination algorithm x ↦ exp(-x κ ′ ), where κ ′ = 2 (µ 1 -µ 2 ) 2 . On the bandit model 0.51 -0.5 (right) theses two rates are very close and SGLRT mostly coincides with Elimination, but on the bandit model 0.2 -0.1 (left) the practical gain of the use of a more sophisticated stopping strategy is well illustrated. Besides, our experiments show that SGLRT using log((log(t) + 1) δ) is δ-PAC on both the (relatively) easy and difficult problems we consider, unlike the other algorithms considered.

If one compares on each problem the results for the fixed-budget setting to those for the best δ-PAC algorithm (or conjectured δ-PAC for the SGLRT algorithm for Bernoulli bandits), in green, one can see that to obtain the same probability of error, the fixed-confidence algorithm needs an average number of samples of order at least twice larger than the deterministic number of samples required by the fixed-budget setting algorithm. This remark should be related to the fact that a δ-PAC algorithm is designed to be uniformly good across all problems, whereas consistency is a weak requirement in the fixed-budget setting: any strategy that draws both arm infinitely often and recommends the empirical best is consistent. Figure 5.7 shows that when the values of µ 1 and µ 2 are unknown, the sequential version of the test is no more preferable to its batch counterpart and can even become much worse if the exploration rate β(t, δ) is chosen too conservatively. This observation should be mitigated by the fact that the sequential (or fixed-confidence) approach is adaptive with respect to the difficulty of the problem whereas it is impossible to predict the efficiency of a batch (or fixed-budget) experiment without some prior knowledge regarding the problem under consideration. Theorem 5.15 and Theorem 5.16 Proof of Theorem 5.15. According to (5.30) it boils down to finding an exploration rate such that P(∃t ∈ N * ∶ S t > 2σ 2 tβ(t, δ)) ≤ δ, where S t = X 1 + ⋅ ⋅ ⋅ + X t is a sum of i.i.d. normal random variable.

Proof of

Let β(t, δ) be of the form β(t, δ) = log 1 δ + c log log 1 δ + d log log(et), for some constants c > 0 and d > 1. Lemma A.1 given in Appendix A yields

P ∃t ∈ N ∶ S t > 2σ 2 tβ(t, δ) ≤ ζ d 1 - 1 2(z + c log z) √ e (2 √ 2) d ( √ z + c log z + √ 8) d z c δ,
where z ∶= log 1 δ > 0. To upper bound the above probability by δ, at least for large values of z (which corresponds to small values of δ), it suffices to choose the parameters c and d such that

√ e ζ d 1 - 1 2(z + c log z) 1 (2 √ 2) d ( √ z + c log z + 2 √ 2) d z c ≤ 1. For c = d 2, the left hand side tends to √ eζ (d) (2 √
2) d when z goes to infinity, which is smaller than 1 for d ≥ 1.47. Thus, for δ small enough, the desired inequality holds for d = 3 2 and c = 3 4, which corresponds to the exploration rate of Theorem 5.15.

Proof of Theorem 5.16. Let α = σ 1 (σ 1 + σ 2 ). We first prove that with the exploration rate β(t, δ) = log(t δ) + 2 log log(6t) the algorithm is δ-PAC. Assume that µ 1 > µ 2 and recall τ

= inf{t ∈ N ∶ d t > 2σ 2 t (α)β(t, δ)}.
The probability of error of the α-elimination strategy is upper bounded by

P ν d τ ≤ -2σ 2 τ (α)β(τ, δ) ≤ P ν d τ -(µ 1 -µ 2 ) ≤ -2σ 2 τ (α)β(τ, δ) ≤ P ν ∃t ∈ N * ∶ d t -(µ 1 -µ 2 ) < -2σ 2 t (α)β(t, δ) ≤ ∞ t=1 exp (-β(t, δ)) ,
by an union bound and Chernoff bound applied to d t -(µ 1 -µ 2 ) ∼ N 0, σ 2 t (α) . The choice of β(t, δ) mentioned above ensures that the series in the right hand side is upper bounded by δ, which shows the algorithm is δ-PAC:

∞ t=1 e -β(t,δ) ≤ δ ∞ t=1 1 t(log(6t)) 2 ≤ δ 1 (log 6) 2 + ∞ 1 dt t(log(6t)) 2 = δ 1 (log 6) 2 + 1 log(6) ≤ δ.
To upper bound the expected sample complexity, we start by upper bounding the probability that τ exceeds some deterministic time T :

P ν (τ ≥ T ) ≤ P ν ∀t = 1 . . . T, d t ≤ 2σ 2 t (α)β(t, δ) ≤ P ν d T ≤ 2σ 2 T (α)β(T, δ) = P ν d T -(µ 1 -µ 2 ) ≤ -(µ 1 -µ 2 ) -2σ 2 T (α)β(T, δ) ≤ exp - 1 2σ 2 T (α) (µ 1 -µ 2 ) -2σ 2 T (α)β(T, δ) 2 .
The last inequality follows from Chernoff bound and holds for T such that (µ

1 -µ 2 ) > 2σ 2 T (α)β(T, δ). Now, for γ ∈]0, 1[ we introduce T * γ ∶= inf t 0 ∈ N ∶ ∀t ≥ t 0 , (µ 1 -µ 2 ) -2σ 2 t (α)β(t, δ) > γ(µ 1 -µ 2 ) .

CHAPTER 5. REFINED FREQUENTIST TOOLS FOR BEST ARM IDENTIFICATION

This quantity is well defined as σ 2 t (α)β(t, δ) go to zero when t goes to infinity. Then,

E ν [τ ] ≤ T * γ + T =T * γ +1 P (τ ≥ T ) ≤ T * γ + T =T * γ +1 exp - 1 2σ 2 T (α) (µ 1 -µ 2 ) -2σ 2 T (α)β(T, δ) 2 ≤ T * γ + ∞ T =T * γ +1 exp - 1 2σ 2 T (α) γ 2 (µ 1 -µ 2 ) 2 .
For all t ∈ N * , it is easy to show that the following upper bound on σ 2 t (α) holds: .32) Using the bound (5.32), one has

∀t ∈ N, σ 2 t (α) ≤ (σ 1 + σ 2 ) 2 t × t -σ 1 σ 2 t -σ 1 σ 2 -1 . ( 5
E ν [τ ] ≤ T * γ + ∞ 0 exp ⎛ ⎝ - t 2(σ 1 + σ 2 ) 2 t -σ 1 σ 2 -1 t -σ 1 σ 2 γ 2 (µ 1 -µ 2 ) 2 ⎞ ⎠ dt ≤ T * γ + 2(σ 1 + σ 2 ) 2 γ 2 (µ 1 -µ 2 ) 2 exp γ 2 (µ 1 -µ 2 ) 2 2(σ 1 + σ 2 ) 2 .
We now give an upper bound on T * γ . Let r ∈ [0, e 2 -1]. There exists N 0 (r) such that for t ≥ N 0 (r), β(t, δ) ≤ log(t 1+r δ). Using also (5.32), one gets T * γ = max(N 0 (t), Tγ ), where

Tγ = inf ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ t 0 ∈ N ∶ ∀t ≥ t 0 , (µ 1 -µ 2 ) 2 2(σ 1 + σ 2 ) 2 (1 -γ) 2 t > t -σ 1 σ 2 -1 t -σ 1 σ 2 log t 1+r δ ⎫ ⎪ ⎪ ⎬ ⎪ ⎪ ⎭ . If t > (1 + γ σ 1 σ 2 ) γ one has (t -σ 1 σ 2 -1) (t -σ 1 σ 2 ) ≤ (1 -γ) -1 . Thus Tγ = max((1 + γ σ 1 σ 2 ) γ, T ′ γ ), with T ′ γ = inf t 0 ∈ N ∶ ∀t ≥ t 0 , exp (µ 1 -µ 2 ) 2 2(σ 1 + σ 2 ) 2 (1 -γ) 3 t ≥ t 1+r δ .
Applying Lemma 5.22 with η = δ, s = 1 + r and β

= (1 -γ) 3 (µ 1 -µ 2 ) 2 (2(σ 1 + σ 2 ) 2 ) leads to T ′ γ ≤ (1 + r) (1 -γ) 3 × 2(σ 1 + σ 2 ) 2 (µ 1 -µ 2 ) 2 log 1 δ + log log 1 δ + R(µ 1 , µ 2 , σ 1 , σ 2 , γ, r), with R(µ 1 , µ 2 , σ 1 , σ 2 , γ, r) = 1 + r (1 -γ) 3 2(σ 1 + σ 2 ) 2 (µ 1 -µ 2 ) 2 1 + (1 + r) log 2(σ 1 + σ 2 ) 2 (1 -γ) 3 (µ 1 -µ 2 ) 2 .
Now for > 0 fixed, choosing r and γ small enough leads to

E ν [τ ] ≤ (1 + ) 2(σ 1 + σ 2 ) 2 (µ 1 -µ 2 ) 2 log 1 δ + log log 1 δ + C(µ 1 , µ 2 , σ 1 , σ 2 , ),
where C is a constant independent of δ. This concludes the proof.

CONCLUSIONS AND FUTURE WORK

Conclusions and future work

In this chapter, we made progress towards understanding the complexity of best arms identification in bandit models. For two-armed bandits, we obtained complete results, identifying the complexity of both the fixed-budget and fixed-confidence settings in important parametric families of distributions. With the example of Bernoulli bandits, we especially show that these two complexities are not always equal.

In the fixed-confidence setting, we also provided in the general case (m best arm identification among K > 2 arms) new lower bounds as well as improved algorithms. The KL-LUCB and KL-Racing algorithms perform well in practice, but there is a small gap between the (informational) upper bound obtained for KL-UCB and the (informational) lower bound derived, that should be investigated further. As future work we also plan to examine whether the refined exploration rate that can be used in elimination algorithm for two-armed (sub)gaussian bandits could be incorporated to the KL-LUCB or KL-Racing algorithms, and if better theoretical guarantees for the SGLRT algorithm can be obtained.

We acknowledge that we were not able to characterize optimal algorithms in the fixed budget setting when there are more than two arms. We presented a first attempt to derive new lower bounds in the paper [START_REF] Kaufmann | On the Complexity of Best Arm Identification in Multi-Armed Bandit Models[END_REF], but there is still room for improvements in this direction.

Elements of proof

A useful technical lemma

The following lemma is useful in Sections 5.2 and 5.4 to obtain upper bounds on the number of sample used by algorithms for the fixed-confidence setting. Proof Lemma 5.22 easily follows from the fact that for any s, η > 0,

x 0 = s log ⎛ ⎜ ⎝ e log 1 η η ⎞ ⎟ ⎠ ⇒ ∀x ≥ x 0 , e x ≥ x s η
Indeed, it suffices to apply this statement to x = xβ and η = ηβ s . The mapping x ↦ e xx s η is increasing when x ≥ s. As x 0 ≥ s, it suffices to prove that x 0 defined above satisfies e x 0 ≥ x s 0 η. This sum also arises in the analysis of the KL-UCB algorithm and is precisely upper-bounded by [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF] in Appendix A.2, for the choice C 1 = 1. However, in order to obtain an exponential decay in γ, we have to adapt their method to the choice C 1 > 1. Introducing This asymptotic equivalent shows that, starting from (5.33), we cannot improve the constant F C 1 (µ a , c) in the exponential with a bigger (and maybe non problem-dependent) one. If C 1 = 1 the same reasoning holds, but the Laplace equivalent is different and leads to:

m γ ∼ γ→∞ √ γ π -2φ ′′ 1 d(µa,c)
, which does not exhibit an exponential decay.

An 'optimal' bound of the probability. We now give a non-asymptotic upper bound of (5.33) involving the optimal rate F C 1 (µ a , c) in the exponential. If v ≥ C 1 d(µa,c) , s * Bounding the probability of error of a static strategy using n 1 samples from arm 1 and n 2 samples from arm 2 relies on the following lemma.

Lemma 5.23. Let (X 1,t ) t∈N and (X 2,t ) t∈N be two independent i.i.d sequences, such that X 1,1 ∼ ν θ 1 and X 2,1 ∼ ν θ 2 belong to an exponential family. Assume that µ(θ 1 ) > µ(θ 2 ). Then

P 1 n 1 n 1 t=1 X 1,t < 1 n 2 n 2 t=1
X 2,t ≤ exp(-(n 1 + n 2 )g α (θ 1 , θ 2 )),

where α = n 1 n 1 +n 2 and g α (θ 1 , θ 2 ) ∶= αK(αθ 1 + (1 -α)θ 2 , θ 1 ) + (1 -α)K(αθ 1 + (1 -α)θ 2 , θ 2 ). The function α ↦ g α (θ 1 , θ 2 ), can be maximized analytically, and the value α * that realizes the maximum is given by

K(α * θ 1 + (1 -α * )θ 2 , θ 1 ) = K(α * θ 1 + (1 -α * )θ 2 , θ 2 ) α * θ 1 + (1 -α * )θ 1 = θ * α * = θ * -θ 2 θ 1 -θ 2
where θ * is defined by K(θ * , θ 1 ) = K(θ * , θ 2 ) = K * (θ 1 , θ 2 ). More interestingly, the associated rate is such that g α * (θ 1 , θ 2 ) = α * K(θ * , θ 1 ) + (1 -α * )K(θ * , θ 2 ) = K * (θ 1 , θ 2 ), which leads to Proposition 5.18. which shows that the strategy based on uniform sampling that recommends the empirical best arm matches the lower bound of Theorem 5.13 for the fixed-budget setting.

Proof of Lemma 5.23. The i.i.d. sequences (X 1,t ) t∈N and (X 2,t ) t∈N have respective densities f θ 1 and f θ 2 where f θ (x) = exp(θx-b(θ)) and µ(θ 1 ) = µ 1 , µ(θ 2 ) = µ 2 . α is such that n 1 = αn and n 2 = (1-α)n.

One can write

P 1 n 1 n 1 t=1 X 1,t - 1 n 2 n 2 t=1 X 2,t < 0 = P α n 2 t=1 X 2,t -(1 -α) n 1 t=1 X 1,t ≥ 0 .
For every λ > 0, multiplying by λ, taking the exponential of the two sides and using Markov's inequality (this technique is often referred to as Chernoff's method), one gets Using the expression of the KL-divergence between ν θ 1 and ν θ 2 as a function of the natural parameters: K(θ 1 , θ 2 ) = µ(θ 1 )(θ 1 -θ 2 ) -b(θ 1 ) + b(θ 2 ), one can also show that

P 1 n 1 n 1 t=1 X 1,t - 1 n 2 n 2 t=1 X 2,t < 0 ≤ E ν [e λαX
αK(αθ 1 + (1 -α)θ 2 , θ 1 ) = -α(1 -α)µ(αθ 1 + (1 -α)θ 2 )(θ 1 -θ 2 ) + α[-b(αθ 1 + (1 -α)θ 2 ) + b(θ 1 )] (1 -α)K(αθ 1 + (1 -α)θ 2 , θ 2 ) = α(1 -α)µ(αθ 1 + (1 -α)θ 2 )(θ 1 -θ 2 ) + (1 -α)[-b(αθ 1 + (1 -α)θ 2 ) + b(θ 2 )]
Summing these two equalities leads to G(λ * ) = -[αK(αθ 1 + (1 -α)θ 2 , θ 1 ) + (1 -α)K(αθ 1 + (1 -α)θ 2 , θ 2 )] = -g α (θ 1 , θ 2 ).

Hence the inequality P 1

n 1 ∑ n 1 t=1 X 1,t < 1 n 2 ∑ n 2
t=1 X 2,t ≤ exp(nG(λ * )) concludes the proof.

5.6.4 Proof of Lemma 5.21.

We fix α > 0 and introduce

σ = max t ∈ 2N * ∶ f (μ 1,t 2 , μ2,t 2 ) ≤ f (µ 1 , µ 2 ) 1 + α 2 .
By the law of large numbers, P(σ < +∞) = 1. Hence, lim n→∞ P(σ ≤ n) = 1 and for every α ∈]0, 1[ there exists N (α, α, µ 1 , µ 2 ) such that P(σ ≤ N (α, α, µ 1 , µ 2 )) ≥ 1 -α. Therefore, introducing the event

E α = ∀t ≥ N (α, α, µ 1 , µ 2 ), f (μ 1,t 2 , μ2,t 2 ) > f (µ 1 , µ 2 ) 1 + α 2
, one has P(E α ) ≥ 1 -α.

On the event E α ,

τ ≤ max N (α, α, µ 1 , µ 2 ); inf t ∈ N ∶ t f (µ 1 , µ 2 ) 1 + α 2 ≥ log g(t) δ τ ≤ N (α, α, µ 1 , µ 2 ) + inf t ∈ N ∶ t f (µ 1 , µ 2 ) 1 + α 2 ≥ log g(t) δ
We can use Lemma 5.22 to bound the right term in the right hand side, which shows that there exists a constant C(α, µ 1 , µ 2 ) independent of δ such that

τ ≤ N (α, α, µ 1 , µ 2 ) + 1 + α f (µ 1 , µ 2 ) log 1 δ + log log 1 δ + C(α, µ 1 , µ 2 )
Thus we proved that for all α > 0,

P lim sup δ→0 τ log(1 δ) ≤ 1 + α f (µ 1 , µ 2 ) ≥ 1 -α.
This concludes the proof.

Conclusion and perspectives

In this thesis, two different bandit problems have been studied: reward maximization and best arm(s) identification in bandit models. For the former, whose complexity is well-known, we have proposed and/or analysed algorithms based on Bayesian ideas that are optimal with respect to the (frequentist) regret. For the latter, we have introduced two complexity notions in the fixed-budget and fixed-confidence settings. We have provided new lower bounds on these complexities as well as improved algorithms matching these lower bounds in particular cases of two-armed bandits.

In both frameworks, we have been focused on obtaining distribution-dependent performance guarantees that feature information-theoretic quantities. A first comment is that the information quantities that appears in the complexity of regret minimization (Kullback-Leibler divergence) and best arm identification (Chernoff information) differ. A possible interpretation of this fact is that the error events in these two problems are different. Assume there are two arms and arm 1 is the best. Algorithms for regret minimization are such that arm 1 has been drawn a lot, so its mean is well estimated: one can consider that the s th draw of arm 2 occurs when (μ 2,s > µ 1 ). The probability of this event involves Kullback-Leibler divergence. On the other hand, in best arm identification, the arms are drawn in a more comparable way, and a typical error occurs when the empirical means μ1,s and μ2,s are somewhere in the middle of the interval [µ 2 , µ 1 ] and in a reversed order. The probability of this event involves Chernoff information. In the fixed confidence setting, the lower bound derived in Chapter 5 when there are two arms involves another information-theoretic quantity, K * (θ 1 , θ 2 ), a Chernoff information in which the role of the arguments are reversed. It would be interesting to be able to interpret this quantity.

The general algorithms proposed for m best arms identification transpose recent improvements, related to the use of confidence intervals based on the Kullback-Leibler divergence, from regret minimization to the pure-exploration framework. As we demonstrated in this thesis the interest of using Bayesian algorithms for regret minimization, a natural question is: could Bayesian algorithms also be used for best arm(s) identification? Finding an heuristic like Thompson Sampling adapted for this different objective is not obvious. But for Bayes-UCB, I suspect that the Racing and LUCB algorithms using Bayesian confidence regions of the form [l a (t), u a (t)] with l a (t) = Q( δ Ct , π t-1 a ) and u a (t) = Q(1 -δ Ct , π t-1 a ) can be shown to be δ-PAC for some constant C. At least, such an algorithm can be easily shown to be δ-PAC under the Bayesian modeling. However, it is not clear how to give an upper bound on the sample complexity E[τ ] in this Bayesian framework (that is, when the expectation includes an average over a prior distribution). In the same way we considered the Bayesian optimal strategy for the reward maximization objective, it would also be interesting to describe a Bayesian optimal strategy for best arm(s) identification. The papers [START_REF] Naghshvar | [END_REF]Javidi, 2013, Chandrasekaran andKarp, 2014] consider particular cases with specific prior distributions.

Bandit models with correlated arms, like linear (contextual) bandit models have also only been studied in the rewards maximization framework. [START_REF] Hoffman | On correlation and budget constraints in model-based bandit optimization with application to automatic machine learning[END_REF] are the first to study (single) best arm identification in a linear bandit model, and to propose a Bayesian algorithm for this task. More precisely, each arm a (among K) is normally distributed with mean x T a θ and known variance σ 2 , where x a ∈ R d is some features vector for am a and θ ∈ R d is an unknown parameter shared by all arms. Assuming a Gaussian prior distribution N 0, η 2 I d on θ, the proposed algorithm uses upper and lower confidence bounds of the form µ a (t) ± βσ a (t), where µ a (t) and σ a (t) are respectively the mean and variance of the posterior distribution on µ a ∶= x T a θ at time t. The algorithm is an adaptation of the UGapE algorithm of [START_REF] Gabillon | Best Arm Identification: A Unified Approach to Fixed Budget and Fixed Confidence[END_REF] using Bayesian confidence regions, for which the authors provide an upper bound on the probability of error in the fixed-budget setting. A first possible extension would be to consider the fixed-confidence setting, an using the LUCB algorithm with the Bayesian confidence regions described above should work. However, this algorithm would probably share the same shortcoming as the algorithms proposed by [START_REF] Hoffman | On correlation and budget constraints in model-based bandit optimization with application to automatic machine learning[END_REF] for the fixed-budget setting, whose complexity term features a sum over all arms of an inverse squared gap (µ * -µ a ) 2 . In other words, the algorithm does not seem to take advantage of the correlation between arms, as it achieves the same performance as if the arms were regarded as independent. Thus, the right complexity of best arm identification in a linear bandit model is still to be investigated. [START_REF] Hoffman | On correlation and budget constraints in model-based bandit optimization with application to automatic machine learning[END_REF] also mention that their algorithm can be more generally applied to Gaussian Process optimization with a discretized space. In Gaussian Process Optimization, we have to find the maximum of a function f ∶ X → R, where f is assumed to be drawn from a Gaussian process. This optimization task naturally generalizes best arm identification in a (Bayesian) multi-armed bandit model. However, despite the optimization objective, the proposed algorithms are often analysed in terms of regret and not optimization error, or simple regret. Besides, in the benchmark of Bayesian optimization, we find the GP-UCB algorithm of [START_REF] Srinivas | Gaussian Process Optimization in the Bandit Setting : No Regret and Experimental Design[END_REF], which is inspired by algorithms minimizing regret in classical bandits (and not algorithms for pure-exploration). It would be interesting to see whether transposing ideas from the best arm identification literature, like the use of upper and lower confidence bounds, could yield improved algorithms for Gaussian Process optimization. The algorithm proposed by [START_REF] Hoffman | On correlation and budget constraints in model-based bandit optimization with application to automatic machine learning[END_REF] indeed seems to improve over GP-UCB. Other authors, like [START_REF] Contal | Parallel Gaussian Process Optimization with Upper Confidence Bounds and Pure Exploration[END_REF] also start to consider the use of pure-exploration tools for Gaussian Process optimization.

Interesting bandit problems that have not been studied in this thesis, neither from the rewards maximization nor from the pure-exploration perspective, are combinatorial bandit problems. Combinatorial bandit problems have been studied by [START_REF] Cesa-Bianchi | Combinatorial Bandits[END_REF] in an adversarial setting. In such problems, arms are edges on a graph and a set of configurations M in this graph is available (e.g. subgraphs with m edges, spanning trees, matchings...). When an agent chooses a configuration, he observes some function of the rewards of each edge in this configuration (for example the sum of rewards, or the reward of each edge). Combinatorial bandit problems have seldom been considered in a stochastic setting, in which each arm produces i.i.d. rewards. It has been studied for example by [START_REF] Lelarge | Spectrum Bandit Optimization[END_REF] with an application to spectrum allocation in a wireless network. Still in a stochastic setting, the identification of the best configuration (without considering regret) is also a challenging task. For example it is not obvious how to design an algorithm based on eliminations (that is, a generalization of the Racing algorithm), as each individual arm belongs to several configurations and eliminating the worst arm in the best configuration would be problematic. Finally, it could also be investigated whether adopting a Bayesian approach is possible for these more complex combinatorial bandit problems. The peeling trick consists in partitioning the interval on which we want to control the process S t √ 2tσ 2 into 'slices' of exponentially growing size, and using inequality (A.2) for a well chosen value of λ on each slice. This idea is inspired by a proof of the Law of Iterated Logarithm given by [Neveu, 1972] and has been used for example by [START_REF] Garivier | On Upper-Confidence Bound Policies for Switching Bandit Problems[END_REF] and [START_REF] Garivier | The KL-UCB algorithm for bounded stochastic bandits and beyond[END_REF] to obtain deviation inequalities. It yields the following result.

Lemma A.1. Let S t = X 1 + ⋅ ⋅ ⋅ + X t be a sum of independent, σ 2 -subgaussian increments. Let ζ(u) = ∑ k≥1 k -u . For all β > 1 and x ≥ 8 (e-1) 2 ,

P ∃t ∈ N * ∶ S t √ 2σ 2 t > x + β log log(et) ≤ √ e ζ β 1 - 1 2x √ x 2 √ 2 + 1 β exp(-x).
Proof of Lemma A.1 We start by stating three technical lemmas, whose proofs are partly omitted.

Lemma A.2. For every η > 0, every positive integer k, and every integer t such that

(1 + η) k-1 ≤ t ≤ (1 + η) k , (1 + η) k-1 2 t + t (1 + η) k-1 2 ≤ (1 + η) 1 4 + (1 + η) -1 4 .
Lemma A.3. For every η > 0, A(η) ∶= 4

(1 + η) 1 4 + (1 + η) -1 4 2 ≥ 1 -η 2 16 .

Lemma A.4. Let t be such that (1 + η) k-1 ≤ t ≤ (1 + η) k . Then, if λ = σ -1 2zA(η)

(1 + η) k-1 2 , σ √ 2z ≥ A(η)z λ √ t + λσ 2 √ t 2 .
Proof of Lemma A.4:

A(η)z λ √ t + λσ 2 √ t 2 = σ 2zA(η) 2 ⎛ ⎝ (1 + η) k-1 2 t + t (1 + η) k-1 2 ⎞ ⎠ ≤ σ √ 2z
according to Lemma A.2.

◻

Let η ∈]0, e -1] to be defined later, and let

T η k = N ∩ (1 + η) k-1 , (1 + η) k . P ⋃ t≥1 S t σ √ 2t > x + β log log(et) ≤ ∞ k=1 P ⎛ ⎜ ⎝ ⋃ t∈T η k S t σ √ 2t > x + β log log(et) ⎞ ⎟ ⎠ ≤ ∞ k=1 P ⎛ ⎜ ⎝ ⋃ t∈T η k S t σ √ 2t > x + β log (k log(1 + η)) ⎞ ⎟ ⎠
.

We use that η ≤ e -1 to obtain the last inequality since this condition implies log(log(e(1 + η) k-1 ) ≥ log(k log(1 + η)).

For a positive integer k, let z k = x+β log (k log(1 + η)) and λ k = σ -1 2z k A(η) (1 + η) k-1 2 . Lemma A.4 shows that for every t ∈ T η k ,

S t σ √ 2t > √ z k ⊂ S t √ t > A(η)z k λ k √ t + σ 2 λ k √ t 2 .
Thus, using inequality (A.2), η) .

P ⎛ ⎜ ⎝ ⋃ t∈T η k S t σ √ 2t > √ z k ⎞ ⎟ ⎠ ≤ P ⎛ ⎜ ⎝ ⋃ t∈T η k S t √ t > A(η)z k λ k √ t + σ 2 λ k √ t 2 ⎞ ⎟ ⎠ = P ⎛ ⎜ ⎝ ⋃ t∈T η k λ k S t - σ 2 λ 2 k t 2 > A(η)z k ⎞ ⎟ ⎠ ≤ exp (-A(η)z k ) = exp(-A(η)x) (k log(1 + η)) βA(
For x such that x ≥ 8 (e-1) 2 , one chooses η 2 = 8 x (which ensures η ≤ e -1). Using Lemma A.3, one obtains that exp(-A(η)x) ≤ √ e exp(-x). Moreover,

1 log(1 + η) ≤ 1 + η η = √ x 2 √ 2 + 1 .
Thus,

P ⎛ ⎜ ⎝ ⋃ t∈T η k S t σ √ 2t > √ z k ⎞ ⎟ ⎠ ≤ √ e k βA(η) √ x 2 √ 2 + 1 βA(η) exp(-x) ≤ √ e k βA(η) √ x 2 √ 2 + 1 β exp(-x)
and hence,

P ⋃ t≥1 S t σ √ 2t > x + β log log(et) ≤ √ eζ (βA(η)) √ x 2 √ 2 + 1 βA(η) exp (-x) ≤ √ eζ β 1 - 1 2x √ x 2 √ 2 + 1 β exp (-x) ,
using the lower bound on A(η) given in Lemma A.3 and the fact that A(η) is upper bounded by 1.

◻

A.1.2 The mixtures method for martingales with subgausian increments

The so-called 'method of mixtures', introduced by [START_REF] De | [END_REF], does not originally rely on the fact that W λ t defined in (A.1) is a super-martingale, but rather on the fact (which is a consequence of the super-martingale property in our case) that for all λ ∈ R, E exp λS t -λ 2 σ 2 2 t ≤ 1.

The method then consists in averaging this inequality over a Gaussian prior distribution. More precisely, assuming that λ ∼ N 0, y -2 , one stil has under this new probabilistic model

E exp λS t - λ 2 σ 2 2 t = E E exp λS t - λ 2 σ 2 2 t λ ≤ 1.
Besides, a direct integration (over λ) gives a close form for the random variable

E exp λS t - λ 2 σ 2 2 t F t ,
whose expectation is then smaller than 1. Finally, the use of Markov inequality yields a deviation inequality. Corollary 12.5 of [De La Pena et al., 2009] could be applied directly to obtain a deviation inequality. However, in order to obtain an inequality that holds for all t ∈ N, we follow [START_REF] Abbasi-Yadkori | Improved Algorithms for Linear Stochastic Bandits[END_REF] and consider randomly stopped super-martingales. This leads to the following result.

Lemma A.5. Let S t = X 1 + ⋅ ⋅ ⋅ + X t be a sum of independent, σ 2 -subgaussian increments. For all y > 0,

P ∃t ∈ N * ∶ S t √ 2σ 2 t > 1 + y t x + 1 2 log 1 + t y ≤ e -x
Proof of Lemma A.5 For every λ ∈ R, the super-martingale W λ t satisfies E[W λ t ] ≤ 1. Let τ be a stopping time with respect to F t . Using the same arguments as in Lemma 8 of [START_REF] Abbasi-Yadkori | Improved Algorithms for Linear Stochastic Bandits[END_REF], W λ τ is well defined and satisfies E[W λ τ ] ≤ 1. Let y > 0 and assume that λ ∼ N 0, y -2 . One still has E

[W λ τ ] = E[E[W λ τ λ]] ≤ 1. Besides, E[W λ τ F τ ] = R exp λS τ - λ 2 σ 2 2 τ y √ 2π e -y 2 2 λ 2 dλ = y √ 2π R exp - y 2 + σ 2 τ 2 λ - S τ y 2 + σ 2 τ 2 - S 2
≤ √ e ζ β 1 - 1 2(x + c log x) √ x + c log x 2 √ 2 + 1 β 1 x c F β,c (x)
exp(-x).

With a choice c = β 2 + 1 and β > 1, for x large enough, one has F β,c (x) ≤ 1. Hence, for every > 0, for large values of x, one can write the following two inequalities, following from Lemma A.1 and Lemma A.5 respectively: The dependency in x seems a bit worse in inequality (A.3) compared to (A.4). However, when considering the dependency in t, the deviation inequality (A.3) improves over (A.4) and is in some sense 'optimal' with respect to the Law of Iterated Logarithm in the Gaussian case, that states that lim sup t→∞ S t √ 2σ 2 t log log(t) = 1 a.s.

P ∃t ∈ N * ∶ S t √
It would be interesting to be able to compare the deviation inequalities obtained with these two methods in different settings. However to prove the deviation inequality presented in Section A.3 and Section A.2 that were useful in this thesis in different contexts, we were not able to use both methods. Indeed, the method of mixtures can be generalized to deal with vector-valued martingales, as shown Moreover, it can be checked that, as z k < µ, λ k < 0. Therefore, on A k , for all s ∈ T η k ,

λ k S s s -φ Y (λ k ) > λ k z k -φ Y (λ k ) λ k S s s -φ Y (λ k ) > γ (1 + η) k λ k S s -φ Y (λ k )s > γs (1 + η) k λ k S s -φ Y (λ k )s > γ (1 + η)
.

One can now upper bound P(A k ) using (A.6):

P(A k ) ≤ P ∃t ∈ T η k ∶ λ k S s -φ Y (λ k )s > γ 1 + η ≤ exp - γ 1 + η .
Lemma A.8. Under the above assumptions, for any matrix V ∈ S ++ n (R),

P ⎛ ⎜ ⎝ ∃t ∈ N ∶ S t (V +Vt) -1 > 2σ 2 x + 1 2 log det(V + V t ) det(V ) ⎞ ⎟ ⎠ ≤ e -x
Proof of Lemma A.8 Let τ be a stopping time. For all λ ∈ R d , E[W λ τ ] ≤ 1. Assuming that λ ∼ N 0, σ 2 V , one still has E[W λ τ ] ≤ 1. Moreover, a direct integration and a bit of linear algebra (see [START_REF] Abbasi-Yadkori | Improved Algorithms for Linear Stochastic Bandits[END_REF]) yields

E W λ τ F τ = det(V ) det(V + V t ) exp 1 2σ 2 S t (V +Vt) -1 . Thus E ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ det(V ) det(V + V t ) exp 1 2σ 2 S t (V +Vt) -1 ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ ≤ 1.
We conclude by using Markov inequality and choosing the stopping time τ as in the proof of Lemma A.5.

◻

Remark A.9. It is no known yet how to use a peeling-trick to obtain a result analogous to Lemma A.8. As W λ t is a super-martingale, for all λ ∈ R d P ∀t ∈ N * , λ T S t -σ 2 2 λ 2 Vt > u ≤ e -u , but it is not clear how this inequality could be used (which 'slices' should be considered in the peeling, and how to choose interesting values of λ ∈ R d ).

With the notation of this section, one could also use that, for any x ∈ R d , for any λ ∈ R,

W λ t = exp λx T S t - σ 2 λ 2 2 x 2
Vt is a super martingale. As explained in Section A.1.3, both the peeling-trick and mixtures method can be applied with A t = x T S t and B t = σ 2 x 2 Vt in (A.5), leading to a deviation inequality of the form P ∃t ∈ N ∶ x T S t > x Vt β(t, δ) ≤ δ. (A.9)

One can note that this inequality does not yield a deviation inequality of the form (4.9) for linear contextual bandits (see Section 4.3.3 in Chapter 4). Indeed, an inequality of the form (4.9) would follow from

P ∃t ∈ N ∶ x T V -1 t S t > x V -1 t β(t, δ) ≤ δ,
which cannot be obtained using inequality (A.9).

Related Work

Another line of recent work has focused on distribution-independent bounds for Thompson Sampling. [START_REF] Agrawal | Thompson Sampling for Contextual Bandits with Linear Payoffs[END_REF] establishes that R(A π U , T ) = O( KT ln(T )) for Thompson Sampling for bounded rewards (with the classic uniform prior π U on the underlying Bernoulli parameter). [Russo and Van Roy, 2014] go beyond the Bernoulli model, and give an upper bound on the Bayes risk (i.e. the regret averaged over the prior) independent of the prior distribution. For the parametric multi-armed bandit with K arms described above, their result states that the regret of Thompson Sampling using a prior π 0 is not too big when averaged over this same prior:

E θ∼π ⊗K 0 [R(A π 0 , T )(θ)] ≤ 4 + K + 4 KT log(T ).
Building on the same ideas, [START_REF] Bubeck | Prior-free and prior-dependent regret bounds for Thompson Sampling[END_REF] have improved this upper bound to 14 √ KT . In our paper, we rather see the prior used by Thompson Sampling as a tool, and we want therefore to derive regret bounds for any given problem parametrized by θ that depend on this parameter. [Russo and Van Roy, 2014] also use Thompson Sampling in more general models, like the linear bandit model. Their result is a bound on the Bayes risk that does not depend on the prior, whereas [START_REF] Agrawal | Thompson Sampling for Contextual Bandits with Linear Payoffs[END_REF] gives a first bound on the regret in this model. Linear bandits consider a possibly infinite number of arms whose mean rewards are linearly related by a single, unknown coefficient vector. Once again, the analysis in [START_REF] Agrawal | Thompson Sampling for Contextual Bandits with Linear Payoffs[END_REF] encounters the problem of describing the concentration of posterior distributions. However by using a conjugate normal prior, they can employ explicit concentration bounds available for Normal distributions to complete their argument.

Paper Structure In Section B.2 we describe important features of the one-dimensional canonical exponential families we consider, including closed-form expression for KL-divergences and the Jeffreys' prior. Section B.3 gives statements of the main results, and provides the proof of the regret bound. Section B.4 proves the posterior concentration result used in the proof of the regret bound.

B.2 Exponential Families and the Jeffreys Prior

A distribution is said to belong to a one-dimensional canonical exponential family if it has a density with respect to some reference measure ν of the form: p(x θ) = A(x) exp(T (x)θ -F (θ)), (B.2) where θ ∈ Θ ⊂ R. T and A are some fixed functions that characterize the exponential family and F (θ) = log ( ∫ A(x) exp [T (x)θ] dν(x)). Θ is called the parameter space, T (x) the sufficient statistic, and F (θ) the normalisation function. We make the classic assumption that F is twice differentiable with a continuous second derivative. It is well known [Wasserman, 2010] that:

E X θ (T (X)) = F ′ (θ) and Var X θ [T (X)] = F ′′ (θ)
showing in particular that F is strictly convex. The mean function µ is differentiable and stricly increasing, since we can show that µ ′ (θ) = Cov X θ (X, T (X)) > 0.

In particular, this shows that µ is one-to-one in θ.

B.3 Results and Proof of Regret Bound

An exponential family K-armed bandit is a K-armed bandit for which the reward distributions p a are known to be elements of an exponential family of distributions P(Θ). We denote by p θa the distribution of arm a and its mean by µ a = µ(θ a ). Theorem B.2 (Regret Bound). Assume that µ 1 > µ a for all a ≠ 1, and that π a,0 is taken to be the Jeffreys prior over Θ. Then for every > 0 there exists a constant C( , P) depending on and on the problem P such that the regret of Thompson Sampling using the Jeffreys prior satisfies

R(A π J , T ) ≤ 1 + 1 - K a=2
(µ 1 -µ a ) KL(θ a , θ 1 ) ln(T ) + C( , P).

Proof: We give here the main argument of the proof of the regret bound, which proceed by bounding the expected number of draws of any suboptimal arm. Along the way we shall state concentration results whose proofs are postponed to later sections.

Step 0: Notation We denote by y a,s the s-th observation of arm a and by N a,t the number of times arm a is chosen up to time t. (y a,s ) s≥1 is i.i.d. with distribution p θa . Let Y u a ∶= (y a,s ) 1≤s≤u be the vector of first u observations from arm a. Y a,t ∶= Y Na,t a is therefore the vector of observations from arm a available at the beginning of round t. Recall that π a,t , respectively π a,0 , is the posterior, respectively the prior, on θ a at round t of the algorithm.

We define L(θ) to be such that P Y ∼p( θ) (p(Y θ) ≥ L(θ)) ≥ 1 2 . Observations from arm a such that p(y a,s θ) ≥ L(θ a ) can therefore be seen as likely observations. For any δ a > 0, we introduce the event Ẽa,t = Ẽa,t (δ a ): For all a ≠ 1 and ∆ a such that µ a < µ a + ∆ a < µ 1 , we introduce E θ a,t = E θ a,t (∆ a ) ∶= µ (θ a,t ) ≤ µ a + ∆ a .

Ẽa,t = ⎛ ⎜ ⎝ ∃1 ≤ s ′ ≤ N a,
On Ẽa,t , the empirical sufficient statistic of arm a at round t is well concentrated around its mean and a 'likely' realization of arm a has been observed. On E θ a,t , the mean of the distribution with parameter θ a,t does not exceed by much the true mean, µ a . δ a and ∆ a will be carefully chosen at the end of the proof.

Step 1: Decomposition The idea of the proof is to decompose the probability of playing a suboptimal arm using the events given in Step 0, and that E[N a,T ] = ∑

.

where E c denotes the complement of event E. Term (C) is controlled by the concentration of the empirical sufficient statistic, and (B) is controlled by the tail probabilities of the posterior distribution. We give the needed concentration results in Step 2. When conditioned on the event that the optimal arm is played at least polynomially often, term (A) can be decomposed further, and then controled by the results from

Step 2.

Step 3 proves that the optimal arm is played this many times.

Step 2: Concentration Results We state here the two concentration results that are necessary to evaluate the probability of the above events. [T (y s ) -F ′ (θ)] ≥ δ ≤ 2e -u K(θ,δ) , where K(θ, δ) = min(K(θ+g(δ), θ), K(θ-h(δ), θ)), with g(δ) > 0 defined by F ′ (θ+g(δ)) = F ′ (θ)+δ and h(δ) > 0 defined by F ′ (θ -h(δ)) = F ′ (θ) -δ.

The two following inequalities that will be useful in the sequel can easily be deduced from Lemma B. The second result tells us that concentration of the empirical sufficient statistic around its mean implies concentration of the posterior distribution around the true parameter:

Theorem B.4 (Posterior Concentration). Let π a,0 be the Jeffreys prior. There exists constants C 1,a = C 1 (F, θ a ) > 0, C 2,a = C 2 (F, θ a , ∆ a ) > 0, and N (θ a , F ) s.t., ∀N a,t ≥ N (θ a , F ),

1 Ẽa,t P µ(θ a,t ) > µ(θ a ) + ∆ a Y a,t ≤ C 1,a e -(Na,t-1)(1-δaC 2,a )KL(θa,µ -1 (µa+∆a))+ln (Na,t) whenever δ a < 1 and ∆ a are such that 1 -δ a C 2,a (∆ a ) > 0.

Step 3: Lower Bound the Number of Optimal Arm Plays with High Probability The main difficulty adressed in previous regret analyses for Thompson Sampling is the control of the number of draws of the optimal arm. We provide this control in the form of Proposition B.5 which is adapted from Proposition 1 in [START_REF] Kaufmann | Thompson Sampling : an Asymptotically Optimal Finite-Time Analysis[END_REF]. The proof of this result, an outline of which is given in Appendix B.9, explores in depth the randomised nature of Thompson Sampling. In particular, we show that the proof in [START_REF] Kaufmann | Thompson Sampling : an Asymptotically Optimal Finite-Time Analysis[END_REF]] can be significantly simplified, but at the expense of no longer being able to describe the constant C b explicitly:

Proposition B.5. ∀b ∈ (0, 1), ∃C b (π, µ 1 , µ 2 , K) < ∞ such that ∑ ∞ t=1 P N 1,t ≤ t b ≤ C b .

Step 4: Bounding the Terms of the Decomposition Now we bound the terms of the decomposition as discussed in Step 1: An upper bound on term (C) is given in (B.6), whereas a bound on term (B) follows from Lemma B.6 below. Although the proof of this lemma is standard, and bears a strong similarity to Lemma 3 of [START_REF] Agrawal | Thompson Sampling for Contextual Bandits with Linear Payoffs[END_REF], we provide it in Appendix B.8 for the sake of completeness.

Lemma B.6. For all actions a and for all > 0, ∃ N = N (δ a , ∆ a , θ a ) > 0 such that (B) ≤ [(1 -)(1 -δ a C 2,a )KL(θ a , µ -1 (µ a + ∆ a ))] -1 ln(T ) + max{N , N (θ a , F )} + 1.

where N = N (δ a , ∆ a , θ a ) is the smallest integer such that for all n ≥ N (n -1) -1 ln(C 1,a n) < (1 -δ a C 2,a )KL(θ a , µ -1 (µ a + ∆ a )),

and N (θ a , F ) is the constant from Theorem B.4.

When we have seen enough observations on the optimal arm, term (A) also becomes a result about the concentration of the posterior and the empirical sufficient statistic, but this time for the optimal arm:

(A) ≤ T t=1 P a t = a, Ẽa,t , E θ a,t , N 1,t > t b + C b ≤ T t=1 P µ(θ 1,t ) ≤ µ 1 -∆ ′ a , N 1,t > t b + C b ≤ T t=1 P µ(θ 1,t ) ≤ µ 1 -∆ ′ a , Ẽ1,t (δ 1 ), N 1,t > t b B ′ + T t=1 P Ẽc 1,t (δ 1 ) ∩ N 1,t > t b C ′ +C b (B.8)
where ∆ ′ a = µ 1 -µ a -∆ a and δ 1 > 0 remains to be chosen. The first inequality comes from Proposition B.5, and the second inequality comes from the following fact: if arm 1 is not chosen and arm a is such that µ(θ a,t ) ≤ µ a + ∆ a , then µ(θ 1,t ) ≤ µ a + ∆ a . A bound on term (C') is given in (B.7) for a = 1 and δ 1 . We assume that the event Ẽu holds, s ′ ≤ u. So, on this event we have

One then has

P (µ(θ u ) ≥ µ + ∆ Y u ) = ∫ θ ′ ∈Θ θ

  µ a )E ν [N a (T )] .

  µ a )E θ [N a (T )] .

FIGURE 1 -

 1 FIGURE 1 -Les intervalles de confiance utilisés par KL-LUCB (à gauche) et les lois a posteriori sur les moyennes utilisées par Bayes-UCB (à droite) après T = 500 instants, pour un modèle de bandit binaire à 5 bras (les losanges rouges représentent leurs moyennes).

Théorème 4 .

 4 Soit > 0 et soient b et C b les constantes définies dans la Proposition 5 ci-dessous. Pour tout bras sous-optimal a, il existe des constantes N (b) et N ( , µ 1 , µ a ) telles que pour T ≥ N ( , µ 1 , µ a ),

  des lois Beta à coefficients entiers. En effet, si on note F Beta a,b la fonction de répartition d'une loi Beta(a, b) et F Bin j,µ la fonction de répartition d'une loi binomiale de paramètres j et µ, on a F Beta a,b (y) = 1 -F Bin a+b-1,y (a -1). Ce résultat peut être établi en remarquant que la a-ème statistique d'ordre parmi a + b -1 variables aléatoires uniformes suit une loi Beta(a, b).

FIGURE 2 -

 2 FIGURE 2 -Regret cumulé des divers algorithmes en fonction du temps. Sur chaque graphique, la courbe en bleu présente la borne inférieure, la courbe en gras présente le regret moyen, et les zones grisées claires et sombres correspondent respectivement aux 0.05% supérieurs et aux 99% centraux

FIGURE 3 -

 3 FIGURE 3 -Illustration des algorithmes pour K = 6, m = 3. Les losanges noirs représentent les moyennes inconnues. Les m meilleurs empiriques (resp. Km moins bons) sont en rouge (resp. en bleu). Les bras en gras sont ceux tirés à l'instant courant : à gauche KL-LUCB tire tous les bras, et le meilleur bras empirique va être éliminé. A droite KL-LUCB tire uniquement les bras l t et u t .

FIGURE 4 -

 4 FIGURE 4 -KL-UCB (à gauche) et KL-LUCB pour l'identification du meilleur bras (à droite) après 1500 observations des bras environ.

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 1.2 The frequentist approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 1.2.1 Lower bounds on the regret . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 1.2.2 Examples of bandit models and associated tools to build bandit algorithms . . 42 1.2.3 Asymptotically optimal algorithms . . . . . . . . . . . . . . . . . . . . . . . . . 45 1.3 The Bayesian approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 1.3.1 Some examples of Bayesian bandit models. . . . . . . . . . . . . . . . . . . . . 51 1.3.2 Discounted and Finite-Horizon Gittins indices . . . . . . . . . . . . . . . . . . 53 1.3.3 Index policies using Gittins indices . . . . . . . . . . . . . . . . . . . . . . . . . 56 1.3.4 Approximation of the FH-Gittins indices . . . . . . . . . . . . . . . . . . . . . . 58 1.3.5 Asymptotically optimal algorithms with respect to the Bayes risk . . . . . . . . 60 1.4 Numerical study and conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 1.5 Elements of proof . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 1.5.1 Changes of distribution: proof of Lemma 1.3 . . . . . . . . . . . . . . . . . . . 65 1.5.2 On Gittins' theorem: proof of Theorem 1.13 . . . . . . . . . . . . . . . . . . . . 66 1.5.3 Proofs of Bayes risk bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

2 Figure 1 . 1 :

 211 Figure 1.1: Two confidence intervals with the same coverage probability.

Figure 1 .

 1 Figure 1.2 gives an illustration of a transition in this MDP when there are three arms.

Figure 1 . 2 :

 12 Figure 1.2: Transition in the MDP associated to a Bernoulli bandit model with Beta prior

Figure 1 . 3 :

 13 Figure 1.3: Illustration for a = 15, b = 10 and n = 30. The secant (in black) intersects the X axis at x 2 , which is a new lower bound x 2 on the Gittins index G((a, b), n).

Figure 1

 1 Figure 1.4: Bayes risk of the optimal strategy versus Bayes risk of FH-Gittins

Figure 1 . 5 :Figure 1

 151 Figure 1.5: Bayes risk of the different strategies under a uniform prior distribution on the means, for K=5 arms (left) and K=10 arms (right)

Figure 2 . 1 :

 21 Figure 2.1: Cumulated regret for the two armed-bandit problem with µ 1 = 0.1, µ 2 = 0.2 (left) and µ 1 = 0.45, µ 2 = 0.55 (right).

Figure 2 . 2 :

 22 Figure 2.2: Regret in a 4-arms problem with parameters µ = [1.8 2 1.5 2.2], σ = [0.5 0.7 0.5 0.3].

Figure 2 . 3 :

 23 Figure 2.3: Cumulated regret in a 20 arms problem for Bayes-UCB with different prior distributions.

2. 5

 5 Elements of proof 2.5.1 Proof of Lemma 2.2

  2 and the new decomposition yields: Theorem 3.4. Consider > 0 and b and C b as in Proposition 3.2. For every suboptimal arm a, there exists constants N (b) and N

Lemma 3 . 5 .

 35 Let b and C b be defined as in Proposition 3.2 and define

◻Conclusion:

  Theorem 3.4 follows fromLemmas 3.5, 3.6 and inequality (3.2), letting N (b) ∶= N 0 (b) + N 1 (b).

3. 2 . 3

 23 Proof of Proposition 3.2: Exploiting the randomized nature of Thompson Sampling.

Figure 3

 3 Figure 3.1: Thompson Sampling draws a lot the optimal arm: an heuristic explanation for two-armed bandits

Figure 3 . 2 :Figure 3 . 3 :Figure 3 . 4 :

 323334 Figure 3.2: Regret of several algorithms including FH-Gittins on four different two-armed bandit problems: 0.05-0.15 (top, left) 0.85-0.95 (top, right) and two more difficult problems, 0.2-0.25 (bottom, left) and 0.75-0.8 (bottom, right). Regret is estimated based on N = 10000 simulations.

Figure 3 .

 3 Figure 3.3 displays for several algorithms an estimation of the distribution of the cumulative regret based on N = 50000 trials. The first four algorithms are variants of UCB, displayed using the same scale, that are not known to be optimal. Of these, the UCB-V algorithm of[START_REF] Audibert | Exploration-exploitation trade-off using variance estimates in multi-armed bandits[END_REF] is close to the index policy to which Thompson Sampling is compared in[START_REF] Chapelle | An empirical evaluation of Thompson Sampling[END_REF] in the Bernoulli setting. This algorithm incorporates an estimation of the variance of the rewards in the index which is defined to be, for an arm that have produced k rewards in n draws,

Figure 3 . 5 :

 35 Figure 3.5: Bayes risk of several algorithm with a uniform prior distribution on the means of the 5 arms (left) or 10 arms (right).

3. 5 . 1

 51 Proof ofLemma 3.8 On the event {J ⊆ [τ j , τ j+1 [}, for all s ∈ J the posterior distribution π s 1 = π τ j

  Figure 4.1: Exploration rates β(t, δ) used by the different algorithms as a function of t for δ = 0.05, on a bandit model for which K = 2000 and d = 5 (left) and K = 2000 and d = 10 (right)

Figure 4 . 2 :

 42 Figure 4.2: Regret curves for several algorithms in two different linear contextual models, in which d = 5 and K = 2000 (left) or d = 10 and K = 2000 (right).

Contents 5 .

 5 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 5.2 Algorithms: KL-LUCB and KL-Racing . . . . . . . . . . . . . . . . . . . . . . . . . . 141 5.2.1 Two classes of algorithms based on confidence intervals . . . . . . . . . . . . . 141 5.2.2 Analysis of KL-Racing and KL-LUCB . . . . . . . . . . . . . . . . . . . . . . . 145 5.2.3 Numerical experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 5.2.4 Proofs of the theorems of Section 5.2 . . . . . . . . . . . . . . . . . . . . . . . . 149 5.3 Generic lower bound on the complexity in the fixed-confidence setting . . . . . . . 155 5.4 The complexity of A/B Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

Figure 5

 5 Figure 5.1: KL-LUCB for finding the m = 3 best arms among 6. In red (resp. blue) arms in J(t) (resp. J(t) c ) and their confidence intervals. Arms in bold are l t and u t . Red circles represent the current number of draws of each arm.

Theorem 5 . 1 .

 51 The KL-Racing and KL-LUCB algorithms using β(t, δ) = log k 1 Kt α δ as an exploration rate, with α > 1 and k 1 > 1 + 1 α-1 , are correct with probability at least 1 -δ. In order to introduce our results on the sample complexity of KL-Racing and KL-LUCB, one needs to introduce a new informational quantity: Chernoff information. The Chernoff information between two Bernoulli distributions B(x) and B(y), denoted by d * (x, y), is defined by d * (x, y) = d(z * , x) = d(z * , y) where z * is the unique z such that d(z, x) = d(z, y).

P

  (sd (μ a,s , c) ≤ γ) ≤ exp (-γ) d * (µ a , c).

Definition 5 . 3 .

 53 Let C 1 > 1, (y, c) ∈]0, 1[ 2 , y ≠ c. Let s C 1 (y, c) be the function implicitly defined by d(s C 1 (y, c), c) = d(y, c) C 1 and s C 1 (y, c) ∈ y, c ,where (y, c) denotes the interval [y, c] if y < c, and [c, y] otherwise. We define F C 1 as:

  c) is the unique z satisfying d(z, µ a ) = d(z, c). Hence, C 1 (µ a , c) can be rephrased using Chernoff information which is precisely defined for two Bernoulli by d * (µ a , c) = d(z * , c) = d(z * , µ a ). One gets C 1 (µ a , c) = d(µ a , c) d * (µ a , c) (5.10) and invoking Lemma 5.4 with this particular value of C 1 leads to Lemma 5.2.

For

  KL-LUCB, we provide in Theorem 5.6 an upper bound on E[τ ], in the -relaxation framework, for ≥ 0. It involves, for c ∈ [µ m+1 , µ m ], the quantity H * ,c (ν) ∶= a∈{1,...,K} 1 max(d * (µ a

Figure 5 . 2 :

 52 Figure 5.2: Experimental results (descriptions in text).

mP

  (∃t ∈ N * ∶ u a (t) < µ a ) = P (∃t ∈ N * ∶ N a (t)d(μ a (t), µ a ) ≥ β(t, δ), μa (t) < µ a ) = P (∃t ∈ N * ∶ N a (t)d(μ a (t), µ a ) ≥ β(N a (t), δ), μa (t) < µ a ) ≤ P (∃s ∈ N * ∶ sd + (μ a,s , µ a ) ≥ β(s, δ)) ≤ ∞ s=1 P (sd + (μ a,s , µ a ) ≥ β(s, δ)) ≤ ∞ s=1 exp(-β(s, δ))from inequality (1.14). The probability P (∃t ∈ N * ∶ l b (t) > µ b ) is similarly upper bounded for b ∈ (S * m ) c , thus we get

Lemma 5 . 8 .

 58 Let a ∈ S * m , b ∈ (S * m ) c and T * a,b be the unique solution of td * * (µ a , µ b ) = β(t, δ). On W , if t ≥ T * a,b and a and b are still in the race, then l a (t) > u b (t) . Proof of Lemma 5.8. Let c be such that d * (c, µ a ) = d * (c, µ b ) = d * * (µ a , µ b ). One also introduce la and ũb such that d * (c, µ a ) = d( la , µ a ) = d( la , c) and d * (c, µ b ) = d(ũ b , µ b ) = d(ũ b , c).

Figure 5 . 3 :◻

 53 Figure 5.3: Functions based on KL-divergence used in the proof ofLemma 5.8

1

  (sd(μa,s,c)≤β(T,δ))

5. 3 . 5 . 3

 353 GENERIC LOWER BOUND ON THE COMPLEXITY IN THE FIXED-CONFIDENCE SETTING155 Generic lower bound on the complexity in the fixed-confidence setting

Figure 5 . 4 :

 54 Figure 5.4: Alternative bandit models considered to obtain the lower bounds of Theorem 5.11 (left) and Theorem 5.13 (right), in the fixed-confidence setting.

Figure 5 . 5 :

 55 Figure 5.5: Comparison of different informational quantities for Bernoulli bandit models.

Figure 5 . 5 ,

 55 on which we represent the different informational function c * ,I * ,c * and I * (defined in Theorem 5.13 and 5.14) when the mean µ 1 varies, for two fixed values of µ 2 . In can be observed that c * (ν) and c * (ν) are almost indistinguishable from I * (ν) and I * (ν) respectively, while there is a gap between c * (ν) and c * (ν).

Figure 5 .

 5 Figure 5.6: The KL-confidence intervals used by KL-LUCB are separated if and only if the threshold β(t, δ) t is below d * (μ 1,t , μ2,t )

Figure 5 . 7 :Figure 5 . 8 :

 5758 Figure 5.7: Experimental results for Gaussian bandit models

Lemma 5 .

 5 22. For every β, η > 0 and s ∈ [0, e 2], the following implication is true:x 0 = s β log e log (1 (β s η)) β s η ⇒ ∀x ≥ x 0 , e βx ≥ x s η .

  that for all y, log(y) ≤ 1 e y. Thenlog x s 0 η ≤ s log(s) + log(2) + log log 1The quantity we have to bound in order to prove Lemma 5.4 isA ∶= T u=⌈C 1 γ d(µa,c)⌉+1P (ud(μ a,u , c) ≤ γ) .

P

  d + (x, c) = d(x, c)1 (x<c) and d -(x, c) = d(x, c)1 (x>c) , (ud + (μ a,u , c) ≤ γ) for µ a < c, andA ≤ T u=n 1 (a,c,γ)+1 P (ud -(μ a,u , c) ≤ γ) for µ a > c,with n 1 (a, c, γ) = ⌈ C 1 γ d(µa,c) ⌉. We now introduce notation that will be useful in the rest of the proof. The two mappingsd + ∶ [0, c] → [0, d(0, c)] d -∶ [c, 1] → [0, d(1, c)] x ↦ d(x, c) x ↦ d(x, c)are bijective and monotone. Then, for α ∈ [0, d(µ a , c)], the quantity s * α (µ a , c) is well-defined by:d(s * α (µ a , c), c) = α and s * α (µ a , c) ∈ (µ a , c).With this new notation, one has, for a ∈ (S * m ) c :P (ud + (μ a,u , c) ≤ γ)) = P d + (μ a,u , c) ≤ γ u = P μa,u ≥ s * γ u (µ a , c) .And for a ∈ S * m :P (ud -(μ a,u , c) ≤ γ)) = P μa,u ≤ s * γ u (µ a , c) .Using Chernoff's concentration inequality and a comparison with an integral yields in both cases:A ≤ T u=n 1 (a,c,γ)+1 exp -ud s * γ u (µ a , c), µ a ≤ ∞ n 1 (a,c,γ) exp -ud s * γ u(µ a , c), µ a du.

  of variable u = γv, one has: An asymptotic equivalent. This last integral takes the form∞ C 1 d(µa,c) exp (-γφ(v)) with φ(v) = vd s * 1 v (µ a , c), µ aand φ is increasing. We can use the Laplace method for approximating the integral when γ goes to infinity.φ ′ (v) = d s * 1 v (µ a , c), µ a -,c) = 0 iff C 1 = 1. If C 1 > 1 the following equivalent holds: , c) = s C 1 (µ a , c), we get m γ ∼ γ→∞ exp(-γF C 1 (µ a , c)) φ ′ C 1 d(µa,c) with F C 1 (µ a , c) = C 1 d (s C 1 (µ a , c), µ a ) d(µ a , c) . a , c) = d(µ a , c) C 1 F C 1 (µ a , c) -d ′ (s C 1 (µ a , c), µ a ) d ′ (s C 1 (µ a , c), c) .

  ,c) exp (-ud (s C 1 (µ a , c), µ a )) du = exp (-F C 1 (µ a , c)γ) d (s C 1 (µ a , c), µ a ).

Remark 5 . 2 + K θ 2 ,

 522 24. When µ 1 > µ 2 , applyingLemma 5.23 withn 1 = n 2 = t 2 yields P μ1,t 2 < µ 2,t 2 ≤ exp ⎛ ⎝ -K θ 1 , θ 1 +θ 2 -I * (ν)t ,

=

  exp n (1 -α)φ X 2,1 (λα) + αφ X 1,1 (-(1 -α)λ) Gα(λ) with φ X (λ) = log E ν [e λX ] for any random variable X. If X ∼ f θ a direct computation gives φ X (λ) = b(λ + θ) -b(θ). Therefore the function G α (λ) introduced above rewrites G α (λ) = (1 -α)(b(λα + θ 2 ) -b(θ 2 )) + α(b(θ 1 -(1 -α)λ) -b(θ 1 )).Using that b ′ (x) = µ(x), we can compute the derivative of G and see that this function as a unique minimum in λ * given byµ(θ 1 -(1 -α)λ * ) = µ(θ 2 + αλ * ) ⇔ θ 1 -(1 -α)λ * = θ 2 + αλ * ⇔ λ * = θ 1 -θ 2 ,using that θ ↦ µ(θ) is one-to-one. One can also show thatG(λ * ) = (1 -α)[b(αθ 1 + (1 -α)θ 2 ) -b(θ 2 )] + α[b(αθ 1 + (1 -α)θ 2 ) -b(θ 1 )].

  It follows thatP(∃t ∈ N ∶ W λ t > e u ) = P(N < +∞) = lim n→∞ P(N ≤ n) ≤ e -u .

  Figure A.1: Illustration of z k in the Bernoulli case

Γ

  Figure B.1: The posterior distribution after observations y 1 , . . . , y n depends on n and s = ∑ n i=1 T (y i )

  t ∶ p(y a,s ′ θ a ) ≥ L(θ a ), ∑ Na,t s=1,s≠s ′ T (y a,s ) N a,t -1 -F ′ (θ a ) ≤ δ a

Lemma B. 3 .

 3 Let (y s ) be an i.i.d sequence of distribution p(⋅ θ) and δ > 0.

3 .2t 2 e

 32 Their proof is gathered in Appendix B.6 with that of Lemma B.3. For any arm a, for any b ∈]0, 1[,T t=1 P(a t = a, ( Ẽa,t (δ a )) c ) ≤ Ẽa,t (δ a )) c ∩ N a,t > t b ) ≤ -(t b -1) K(θa,δa) , (B.7)

P1t 2

 2 (p(y a,1 θ a ) ≤ L(θ a )) s + y a,k ) -F ′ (θ a )) ≥ δ a s-1) K(θa,δa) ,where we use that the definition of L(θ) gives P (p(y a,1 θ a ) ≤ L(θ a )) ≤ 1 2 . This leads to inequality (B.6). To proof (B.7), we write:T t=1 P( Ẽa,t (δ a ) c ∩ N a,t > t b ) ≤ E Na,t=s (1 Ba,s + 1 Da,s ) y a,1 θ a ) ≤ L(θ a )) s y a,k ) -F ′ (θ a )) ≥ δ a exp(-t b K(θ a , δ)).B.7 Extracting the KL-divergence: Proof of Lemma B.7

  s θ ′ )p(y s ′ θ ′ )π(θ ′ )dθ ′ ∫ θ ′ ∈Θ u ∏ s=1,s≠s ′ p(y s θ ′ )p(y s ′ θ ′ )π(θ ′ )dθ ′ = ∫ θ ′ ∈Θ θ,∆ u ∏ s=1,s≠s ′ p(ys θ ′ ) p(ys θ) p(y s ′ θ ′ )π(θ ′ )dθ ′ ∫ θ ′ ∈Θ u ∏ s=1,s≠s ′ p(ys θ ′ ) p(ys θ) p(y s ′ θ ′ )π(θ ′ )dθ ′ = ∫ θ ′ ∈Θ θ,∆ e -(u-1)K[Y ′u ,θ,θ ′ ] π(θ ′ y s ′ )dθ ′ ∫ θ ′ ∈Θ e -(u-1)K[Y ′u ,θ,θ ′ ] π(θ ′ y s ′ )dθ ′where π(θ y s ′ ) denotes the posterior distribution on θ after observation y s ′ andK[Y u s ′ , θ, θ ′ ] ∶= 1 u -1u s=1,s≠s ′ ln p(y s θ) p(y s θ ′ )

  où t est un bruit centré, θ ∈ R d est un paramètre de régression et x T désigne la transposée du vecteur x. Revenant à l'exemple de la publicité en ligne, on peut imaginer que pour chaque utilisateur un vecteur de caractéristiques est disponible (provenant de son historique de navigation et des données disponibles sur lui). De même, on dispose de caractéristiques pour chaque publicité qu'on pourrait lui présenter. présenter et le modèle suppose une dépendance linéaire entre cette réponse et le contexte associé. L'hypothèse d'un ensemble D t de contextes changeant à chaque instant correspond donc au fait qu'un utilisateur différent arrive à chaque fois, et aussi au fait que différentes publicités peuvent entrer ou sortir de la campagne. Une autre exploitation possible du modèle est la suivante : on suppose que les utilisateurs qui arrivent sont tous de même type (si une classification a été effectuée au préalable). Dans ce cas D t est l'ensemble des vecteurs de caractéristiques des différentes publicités, et le vecteur θ indique les préférences de ce type d'utilisateur.L'objectif de l'agent est de maximiser ses récompenses. De manière (quasiment) équivalente, il peut chercher à minimiser la quantité suivante appelée pseudo-regret (car contrairement au regret, qui est une espérance, cette quantité est aléatoire)

Pour chaque publicité, un vecteur de caractéristiques conjointes de la paire (utilisateur/publicité) peut être formé, ce qui donne l'ensemble de contextes disponibles. L'agent (le gestionnaire du site) choisit alors la publicité à

  3. ALGORITHMES ET COMPLEXIT É DE L'IDENTIFICATION DES MEILLEURS BRAS29 le meilleur empirique â est tel que la borne inférieure L â(t) de son intervalle de confiance est supérieure aux bornes supérieures U b (t) des Km -D t moins bons bras empiriques, on sélectionne â comme faisant partie des m meilleurs :

Table 1

 1

	.3: Examples of exponential families and associated divergence
	From Cramér's Theorem (Theorem 2.2.3 of [Dembo and Zeitouni, 2010]), this last quantity can be
	regarded as the optimal rate at which the empirical mean of i.i.d. samples concentrates around the true
	mean. More precisely, if (X i ) is an i.i.d. sequence with expectation µ, and if μs = 1 s ∑ s i=1 X i denotes the empirical mean of the first s observations, for every x > µ,

  Most of the examples of practical interest actually belong to this sub-class, but in Appendix B we give two examples with more general sufficient statistics: Pareto and Weibull distributions.The properties of these exponential families include

  4.5 Numerical experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 4.6 Elements of proof . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 4.6.1 Proof of Lemma 4.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 4.6.2 Proof of Theorem 4.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 4.1 Introduction Let D t ⊂ R d be a set of contexts (or 'contextualized actions') available at time t. In a contextual bandit model, at time t an agent chooses a context x t ∈ D t and receives a reward

  t+1 from which the agent has to choose from at round t + 1. The noise t satisfies E[ t H t-1 ] = 0. In a deterministic strategy, x t is assumed to be H t-1 -measurable, whereas in a randomized strategy, x t is drawn from some distribution p t on D t , such that p t is F t-1 -measurable.The best context (or arm) at time t, i.e. the one with highest mean, is

	4.1. INTRODUCTION	119
	D

  and the fact that put (t) < plt (t) leads to Hence either βut (t) > 2 or βut (t) > 2 . It remains to show that one of k = l t and k = u t such that c ∈ I k (t) also satisfies this second condition. This part is the Proof uses properties of KL-divergence, and cannot directly be adapted from[START_REF] Kalyanakrishnan | PAC subset selection in stochastic multi-armed bandits[END_REF].It remains to show that if U ut (t) -L lt (t) > , then the four statements below hold. ∈ I ut (t) and c > U lt (t), one hasd(p ut (t), c) ≤ 2 βut (t) 2 and d(p lt (t), c) ≥ 2 βlt (t) 2 .Moreover, as c > U lt , c > plt (t) > put (t) holds, and therefore d(p lt (t), c) ≤ d(p ut (t), c). Hence, 2 βlt (t) 2 ≤ d(p lt (t), c) ≤ d(p ut (t), c) ≤ 2 βut (t) 2 and βlt (t) ≤ βut (t)

	βut (t) + βlt (t) >	with βa (t) ∶=	β(t, δ) 2N a (t)	.	(5.15)
	c ∈ I ut (t) and c > U lt (t) ⇒ βut (t) >	2	.	(5.16)
	c ∈ I ut (t) and c < L lt (t) ⇒ βut (t) >	2	.	(5.17)
	c ∈ I lt (t) and c > U ut (t) ⇒ βlt (t) >	2	.	(5.18)
	c ∈ I lt (t) and c < L ut (t) ⇒ βlt (t) >	2	.	(5.19)
	To prove (5.16), note that if c

  At first sight, it may be difficult to compare Lemma A.1 and Lemma A.5. Writing Lemma A.1 with x replaced by x + c log(x) yields P ∃t ∈ N

	A.1. PEELING TRICK VERSUS MIXTURES METHOD: THE SUBGAUSSIAN CASE	183
	Now if τ is chosen to be the stopping time
			τ = inf	⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ t ∈ N ∶	√	S t 2σ 2 t	> 1 +	y 2 σ 2 t	x +	1 2	log 1 +	σ 2 t y 2	⎫ ⎪ ⎪ ⎪ ⎬ ⎪ ⎪ ⎪ ⎭	,
	the probability in Lemma A.5 is upper bounded by
		P(τ < +∞) ≤ P	⎛ ⎜ ⎝	S τ 2σ 2 τ √	> 1 +	y 2 σ 2 τ	x +	1 2	log 1 +	σ 2 τ y 2	⎞ ⎟ ⎠	≤ e -x ,
	which concludes the proof.
													◻
	A.1.3 Comparison and generalization
				=	y y 2 + σ 2 τ	exp	S 2 τ 2(y 2 + σ 2 τ )	.
	Hence, E	y y 2 +σ 2 τ	exp	S 2 τ 2(y 2 +σ 2 τ )	≤ 1 and Markov inequality yields
						P		⎛ ⎝		y y 2 + σ 2 τ	exp	S 2 τ 2(y 2 + σ 2 τ )	> e x ⎞ ⎠	≤ e -x
			P	⎛ ⎜ ⎝	S τ 2σ 2 τ √	> 1 +	y 2 σ 2 τ	x +	1 2	log 1 +	σ 2 τ y 2	⎞ ⎟ ⎠	≤ e -x .

τ (y 2 + σ 2 τ ) 2 dλ * ∶ S t √ 2σ 2 t > x + c log(x) + β log log(et)

Quelques définitions et notations. Dans cette thèse nous allons considérer des classes de modèles de bandit à K bras, notées en général M, pour lesquelles nous voudrons trouver des algorithmes efficaces pour l'ensemble des modèles de bandit de la classe M. Par exemple on s'intéressera à la classe des modèles de bandit binaires, où le bras a est une distribution de Bernoulli de moyenne µ a , B(µ a ), qui permet de modéliser de nombreuses applications pratiques, comme on l'a vu plus haut.Plus généralement, on considèrera les classes de modèles de bandit paramétriques, où la distribution du bras a dépend d'un paramètre θ a : ν a = ν θa , avec θ a ∈ Θ. Un cas particulier important est celui des modèles de bandit exponentiels. Une classe M de modèles de bandit exponentiels est telle qu'il existe des fonctions A et b telle que pour tout ν ∈ M, la distribution ν θa du bras a admet pour densitéf (x; θ a ) = A(x) exp(θ a xb(θ)).(1)En d'autres termes, les distributions des bras appartiennent à une famille exponentielle canonique à un paramètre. De telles distributions peuvent également être paramétrées par leur moyenne µ(θ) = ḃ(θ), ce qui permet d'introduire la fonction de divergence suivante, associée à une famille exponentielle donnée, qui correspond à la divergence de Kullback-Leibler entre deux distributions de cette famille, exprimée

une distribution νa est dite σ 2 sous-gaussienne si ∀λ ∈ R, E X∼νa [e λ(X-E X]) ] ≤ exp λ 2 σ 2 2

KL(B(µa), B(µ ′ a )) > 2(µaµ ′ a ) 2

B(t) -1 .

Using Ẽa,t in place of Ea,t from[START_REF] Kaufmann | Thompson Sampling : an Asymptotically Optimal Finite-Time Analysis[END_REF] only changes slightly the constant Ca.

Remerciements

Chapter 3 Thompson Sampling

In this chapter, we propose a finite-time analysis of Thompson Sampling that proves its asymptotic optimality in the context of regret minimization, when the rewards are Bernoulli distributed and the algorithm uses a uniform prior on the means. This work is a joint work with Nathaniel Korda and Rémi Munos, and was presented to the conference ALT in 2012 ( [START_REF] Kaufmann | Thompson Sampling : an Asymptotically Optimal Finite-Time Analysis[END_REF]). In a follow-up work [START_REF] Korda | Thompson Sampling for 1dimensional Exponential family bandits[END_REF]), we also proved that Thompson Sampling with a specific prior, the Jeffreys' prior is asymptotically optimal when the rewards belong to an exponential family.

A large part of this chapter is devoted to the presentation of our analysis for Bernoulli bandits (with only minor changes compared to that proposed in the paper [START_REF] Kaufmann | Thompson Sampling : an Asymptotically Optimal Finite-Time Analysis[END_REF]). We highlight the links with the Bayes-UCB analysis presented in the previous chapter. Our results relative to Thompson Sampling for exponential families are also be presented, with only a sketch of proof, the whole paper ( [START_REF] Korda | Thompson Sampling for 1dimensional Exponential family bandits[END_REF]) being provided in Appendix B. This chapter also includes a numerical study that illustrates the performance of Thompson Sampling both in terms of regret and Bayes risk, compared to that of Bayes-UCB and other bandit algorithms discussed in previous chapters.

Chapter 4

Bayesian algorithms for linear contextual bandits

Recall that the initial motivation for studying the stochastic multi-armed bandit problem with Bernoulli rewards was sequential allocation of medical treatments: the response of patients to each treatment are assumed to be i.i.d. binary random variables (indicating whether the patient is cured or not). This assumption is however oversimplified, since the doctor has information about both the patient and the treatments, and the optimal treatment for two patients might actually differ. One way to incorporate this side information is to consider contextual bandit problems. In a more recent motivation for bandit problems, online advertisement, contexts are also of utmost importance, since a lot of information on the add, user or webpage is available. For all these reasons, we chose to dedicate a chapter of this thesis to the presentation of these more general bandit models of practical interest. We especially focus on linear contextual bandit problems and we argue that Bayes-UCB and Thompson Sampling can also be used in this more general framework.

The contributions of this chapter are the following. If d is the dimension of the context space, we give upper bounds on the Bayes risk of Bayes-UCB and Thompson Sampling using a Gaussian prior on the regression parameter that scale in O(d √ T ) and O( dT log(K)) in the particular case of a finite number of contexts K. For Bayes-UCB, we give a high-probability result that is inspired by standard analysis of optimistic algorithms for linear bandits, whereas for Thompson Sampling, we use ideas introduced by [Russo and Van Roy, 2014] and provide upper bounds in expectation. Both analyses rely on the use of Bayesian confidence regions instead of frequentist confidence regions used in previous work.

The Racing algorithm. The first algorithms for best arms identification (see [START_REF] Bechhofer | Sequential identification and ranking procedures[END_REF]) used pure uniform sampling (sometimes called vector-at-a-time sampling): they proceed in rounds, where all the arms are drawn, and stop when a global stopping criterion is met. [Paulson, 1964] introduces the idea of coupling eliminations to uniform sampling to reduce the number of sample used. His goal was to find the (single) best arm in a Gaussian bandit model. [START_REF] Jennison | Asymptotically optimal procedures for sequential adaptive selection of the best of several normal means[END_REF] in the same setup define general elimination procedures. An elimination procedure depends on some function g, samples all the remaining arms in R at each round t and eliminate arm b if there exists a ∈ R such that S a (t) -S b (t) > g(t). This criterion can be rephrased as some lower confidence bound for µ a being larger than some upper confidence bound for µ b : this is the idea of Racing as introduced by [START_REF] Maron | The Racing algorithm: Model selection for Lazy learners[END_REF] in the context of model selection.

For finding the m best arms with m > 1, [START_REF] Levin | On a Conjecture of Bechhofer, Kiefer, and Sobel for the Levin-Robbins-Leu Binomial Subset Selection Procedures[END_REF]] modify an existing procedure by introducing eliminations: both accepts and rejects (called eliminations and recruitments). Accepts and Algorithm 4 LUCB Require: ≥ 0 (tolerance level), U, L (confidence bounds) t = 1 (number of stage of the algorithm), B(1) = ∞ (stopping index) for a=1...K do Sample arm a, compute confidence bounds U a (1), L a (1) end for while B(t) > do Draw arm u t and l t . t = t + 1. Update confidence bounds, set J(t) and arms u t , l t B(t) = U ut (t) -L lt (t) end while return J(t).

CHAPTER 5. REFINED FREQUENTIST TOOLS FOR BEST ARM IDENTIFICATION

Hence, for small δ,

represents the minimal number of samples needed to discriminate between arm a and arm m (resp. arm a and arm m + 1) with both error probabilities smaller than δ.

However, the general lower bound on κ C (ν) we propose in the next Section does not involve Chernoff information but Kullback-Leibler divergence, so it is still to be determined whether KL-LUCB is optimal with respect to the complexity κ C (ν). Nevertheless, Chernoff information will turn out to be a relevant measure of complexity for two-armed bandits, in the fixed-budget setting.

Numerical experiments

On the basis of our theoretical analysis from the previous section, could we expect the "KL-ized" versions of our algorithms to perform better in practice? Does being "fully sequential" make our adaptive sampling algorithms more efficient than uniform sampling algorithms in practice? In this section, we present numerical experiments that answer both these questions in the affirmative.

In our experiments, in addition to (KL-)LUCB and (KL-)Racing, we include (KL-)LSC, an adaptive sampling algorithm akin to (KL)-LUCB. This algorithm uses the same stopping criterion as (KL)-LUCB, but rather than sample arms u t and l t at stage t, (KL)-LSC samples the least-sampled arm from J(t) (or J(t) c ) that collides (overlaps by at least ) with some arm in J(t) c (J(t)). To ensure that all algorithms are provably PAC, we run them with the parameters α = 1.1, k 1 = 11.1 justified by Theorem 5.1. Results are summarized in Figure 5.2.

As a first order of business, we consider bandit instances with K = 10, 20, . . . , 60 arms; we generate 1000 random instances for each setting of K, with each arm's mean drawn uniformly at random from [0, 1]. We set m = K 5 , = 0.1, δ = 0.1. The expected sample complexity of each algorithm on the bandit instances for each K are plotted in Figure 5.2(a). Indeed we observe for each K that (1) the KL-ized version of each algorithm enjoys a lower sample complexity, and (2) (KL)-LUCB outperforms (KL-)LSC, which outperforms (KL-)Racing.

These trends, aggregated from multiple bandit instances, indeed hold for nearly every individual bandit instance therein. In fact, we find that KL-izing has a more pronounced effect on bandit instances with means close to 0 or 1. For illustration, consider instance B 1 (K = 15; µ 1 = 1 2 ; µ a = 1 2 -a 40 for a = 2, 3, . . . , K), an instance used by [START_REF] Bubeck | Multiple Identifications in multi-armed bandits[END_REF] (see Experiment 5). Figure 5.2(b) compares the runs of LUCB and KL-LUCB both on B 1 (with m = 3, = 0.04, δ = 0.1), and a "scaled-down" version B 2 (with m = 3, = 0.02, δ = 0.1) in which each arm's mean is half that of the corresponding arm's in B 1 (and thus closer to 0). While LUCB and KL-LUCB both incur a higher sample complexity on the harder B 2 , the latter's relative economy is clearly visible in the graph.

How conservative are the stopping criteria of our PAC algorithms? In our third experiment, we halt these algorithms at intervals of 1000 samples, and at each stage record the probability that the set J(t) of m empirical best arms that would be returned at that stage is non-optimal. Results from this experiment, again on B 1 (with m = 3, = 0.04, δ = 0.1), are plotted in Figure 5.2(c). Notice that (KL)-LUCB indeed drives down the mistake probability much faster than its competitors. Yet, even if all the algorithms have an empirical mistake probability smaller than δ after 5,000 samples, they only stop after at least 20,000 episodes, leaving us to conclude that our formal bounds are rather conservative. On the low-reward instance B 2 (with m = 3, = 0.02, δ = 0.1), we observe that KL-LUCB indeed reduces the mistake probability more quickly than LUCB, indicating a superior sampling strategy. This difference is between LUCB and KL-LUCB is not apparent on B 1 in Figure 5.2(c).

We test KL-LUCB-log(t), a version of KL-LUCB with an exploration rate of log(t) (which yields no provable guarantees) as a candidate for the fixed-budget setting. On B 1 (with n = 4000), we compare this Appendix A

Self normalized deviation inequalities

In several places in this thesis, we needed deviation inequalities for 'self normalized quantities' of the form A t B t , where both A t and B t are random. For example we need to control the empirical mean of rewards collected from an arm up to a given time t, S a (t) N a (t), where both the number of draws up to time t, N a (t) and the sum of observations , S a (t), are random. Such deviation inequalities can be obtained by controlling the quantity the quantity S a,s s uniformly for s ∈ {1, . . . , t}, or for s ∈ N * . We present deviation inequalities for similar quantities in this section.

A.1 Peeling trick versus mixtures method: the subgaussian case

In Chapter 5, in the proof of Theorem 5.15, a tight deviation inequality, uniform in t, is needed for the process S t √ 2tσ 2 , where S t is a sum of i.i.d random variables with distribution N 0, σ 2 . We consider here the more general framework in which S t = X 1 + ⋅ ⋅ ⋅ + X t is a sum of independent increments that are σ 2 -subgaussian, i.e. that satisfy, for every λ ∈ R,

A deviation inequality for S t √ 2tσ 2 can either be obtained using a so-called 'peeling-trick', as shown in Section A.1.1, or by the 'method of mixtures' that we present in Section A.1.2. Both methods rely in this case on the family of super-martingale ((W λ t ) λ∈R ), indexed by λ ∈ R, defined by

W λ t is a super-martingale with respect to the filtration F t = σ(X 1 , . . . , X t ).

A.1.1 An 'optimal' confidence region obtained with the peeling-trick

For each λ ∈ R, for each u > 0, the maximal inequality for super-martingales yields

Indeed, introducing the stopping time N = inf{t ∈ N ∶ W λ t > e u }, a maximal inequality yields

by [START_REF] Abbasi-Yadkori | Improved Algorithms for Linear Stochastic Bandits[END_REF] with applications to linear contextual bandits, but the peeling-trick does not. Conversely, using a peeling trick, [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF] give an informational self-normalized deviation inequality useful in the analysis of KL-UCB, Bayes-UCB and Thompson Sampling, but it is not known yet how the method of mixtures could be applied in this setting.

Both methods can however be applied in a slightly more general setting to control a self-normalized process of the form A 2 t 2B t , where A t and B t ≥ 0 are such that, for all λ,

is a super-martingale. A straightforward adaptation of the proof of Lemma A.1 (using the slices

and Lemma A.5 yields, for x small enough,

A.2 An informational deviation inequality

Let (X i ) be an i.i.d. sequence of random variables, whose log-moment generating function satisfy

where the distribution of Y , ν θ , belong to an exponential family and has mean µ. Then, if

and the maximal inequality yields, for every λ ∈ R, u ∈ R,

Lemma A.6 below (which corresponds to Lemma 1.9 in Chapter 1) can be obtained with a peelingtrick using the maximal inequality (A.6). Another key ingredient is the relationship between the logmoment generating function and the divergence d associated to the exponential family that follow from Lemma 1.4:

where Y ∼ ν θ belongs to an exponential family with mean µ and associated divergence d.

where d + (x, y) = d(x, y)1 (x<y) .

A.2. AN INFORMATIONAL DEVIATION INEQUALITY

From (A.8), one gets

γ-1 and using that log(γ (1 -γ)) ≥ 1 γ yields the result.

◻

Remark A.7. It is not known whether the method of mixtures could be applied were, using the supermartingale W λ t . Indeed, assuming that λ ∼ N 0, y -1 , there is no close form for

Maybe a different 'prior distribution' for λ should be considered.

A.3 Deviation inequalities for vector-valued martingales

In Chapter 4, we present in Lemma 4.1 a confidence region for the parameter θ of a linear contextual bandit model. This confidence region follows from a deviation inequality for vector-valued martingales proposed by [START_REF] Abbasi-Yadkori | Improved Algorithms for Linear Stochastic Bandits[END_REF], that applies in the following more general framework.

For (F t ) a filtration, let (X t ), (η t ) be two sequences of random variables such that

Then S t is a martingale in R d and for every λ ∈ R d , the sequence (W λ t ) t∈N * , with

Applying the method of mixtures to this super-martingale, with a (vector-valued) Gaussian prior distribution for λ yields the following result.

A.3. DEVIATION INEQUALITIES FOR VECTOR-VALUED MARTINGALES

Appendix B

Thompson Sampling for One-Dimensional Exponential Family Bandits Abstract. Thompson Sampling has been demonstrated in many complex bandit models, however the theoretical guarantees available for the parametric multi-armed bandit are still limited to the Bernoulli case. Here we extend them by proving asymptotic optimality of the algorithm using the Jeffreys prior for 1-dimensional exponential family bandits. Our proof builds on previous work, but also makes extensive use of closed forms for Kullback-Leibler divergence and Fisher information (through the Jeffreys prior) available in an exponential family. This allow us to give a finite time exponential concentration inequality for posterior distributions on exponential families that may be of interest in its own right. Moreover our analysis covers some distributions for which no optimistic algorithm has yet been proposed, including heavy-tailed exponential families.

B.1 Introduction

K-armed bandit problems provide an elementary model for exploration-exploitation tradeoffs found at the heart of many online learning problems. In such problems, an agent is presented with K distributions (also called arms, or actions) {p a } K a=1 , from which she draws samples interpreted as rewards she wants to maximize. This objective induces a trade-off between choosing to sample a distribution that has already yielded high rewards, and choosing to sample a relatively unexplored distribution at the risk of loosing rewards in the short term. Here we make the assumption that the distributions, p a , belong to a parametric family of distributions P = {p(⋅ θ), θ ∈ Θ} where Θ ⊂ R. The bandit model is described by a parameter θ 0 = (θ 1 , . . . , θ K ) such that p a = p(⋅ θ a ). We introduce the mean function

, and the optimal arm θ * = θ a * where a * = argmax a µ(θ a ).

An algorithm, A, for a K-armed bandit problem is a (possibly randomised) method for choosing which arm a t to sample from at time t, given a history of previous arm choices and obtained rewards, H t-1 ∶= ((a s , x s )) t-1 s=1 : each reward x s is drawn from the distribution p as . The agent's goal is to design an algorithm with low regret:

This quantity measures the expected performance of algorithm A compared to the expected performance of an optimal algorithm given knowledge of the reward distributions, i.e. sampling always from the distribution with the highest expectation. Since the early 2000s the "optimisim in the face of uncertainty" heuristic has been a popular approach to this problem, providing both simplicity of implementation and finite-time upper bounds on the regret (e.g. [Auer et al., 2002a, Cappé et al., 2013]). However in the last two years there has been renewed interest in the Thompson Sampling heuristic (TS). While this heuristic was first put forward to solve bandit problems eighty years ago in [Thompson, 1933], it was not until recently that theoretical analyses of its performance were achieved [START_REF] Agrawal | Analysis of Thompson Sampling for the multi-armed bandit problem[END_REF][START_REF] Agrawal | Thompson Sampling for Contextual Bandits with Linear Payoffs[END_REF][START_REF] Kaufmann | Thompson Sampling : an Asymptotically Optimal Finite-Time Analysis[END_REF], May et al., 2012]. In this paper we take a major step towards generalising these analyses to the same level of generality already achieved for "optimistic" algorithms.

Thompson Sampling Unlike optimistic algorithms which are often based on confidence intervals, the Thompson Sampling algorithm, denoted by A π 0 uses Bayesian tools and puts a prior distribution π a,0 = π 0 on each parameter θ a . A posterior distribution, π a,t , is then maintained according to the rewards observed in H t-1 . At each time a sample θ a,t is drawn from each posterior π a,t and then the algorithm chooses to sample a t = arg max a∈{1,...,K} {µ(θ a,t )}. Note that actions are sampled according to their posterior probabilities of being optimal.

Our contributions TS has proved to have impressive empirical performances, very close to those of state of the art algorithms such as DMED and KL-UCB [START_REF] Kaufmann | Thompson Sampling : an Asymptotically Optimal Finite-Time Analysis[END_REF], Honda and Takemura, 2010, Cappé et al., 2013]. Furthermore recent works [Kaufmann et al., 2012b, Agrawal and[START_REF] Agrawal | Thompson Sampling for Contextual Bandits with Linear Payoffs[END_REF] have shown that in the special case where each p a is a Bernoulli distribution B(θ a ), TS using a uniform prior over the arms is asymptotically optimal in the sense that it achieves the asymptotic lower bound on the regret provided by Lai and Robbins in [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] (that holds for univariate parametric bandits). As explained in [Agrawal andGoyal, 2012, Agrawal and[START_REF] Agrawal | Thompson Sampling for Contextual Bandits with Linear Payoffs[END_REF]], Thompson Sampling with uniform prior for Bernoulli rewards can be slightly adapted to deal with bounded rewards. However, there is no notion of asymptotic optimality for this non-parametric family of rewards. In this paper, we extend the optimality property that holds for Bernoulli distributions to more general families of parametric rewards, namely 1-dimensional exponential families if the algorithm uses the Jeffreys prior: Theorem B.1. Suppose that the reward distributions belong to a 1-dimensional canonical exponential family and let π J denote the associated Jeffreys prior. Then,

where KL(θ, θ ′ ) ∶= KL(p θ , p ′ θ ) is the Kullback-Leibler divergence between p θ and p ′ θ .

This theorem follows directly from Theorem B.2. In the proof of this result we provide in Theorem B.4 a finite-time, exponential concentration bound for posterior distributions of exponential family random variables, something that to the best of our knowledge is new to the literature and of interest in its own right. Our proof also exploits the connection between the Jeffreys prior, Fisher information and the Kullback-Leibler divergence in exponential families.

B.2. EXPONENTIAL FAMILIES AND THE JEFFREYS PRIOR

KL-divergence in Exponential Families

In an exponential family, a direct computation shows that the Kullback-Leibler divergence can be expressed as a Bregman divergence of the normalisation function, F:

Jeffreys prior in Exponential Families In the Bayesian literature, a special "non-informative" prior, introduced by Jeffreys in [Jeffreys, 1946], is sometimes considered. This prior, called the Jeffreys prior, is invariant under re-parametrisation of the parameter space, and it can be shown to be proportional to the square-root of the Fisher information I(θ). In the special case of the canonical exponential family, the Fisher information takes the form I(θ) = F ′′ (θ), hence the Jeffreys prior for the model (B.2) is

Under the Jeffreys prior, the posterior on θ after n observations is given by

When ∫ Θ F ′′ (θ)dθ < +∞, the prior is called proper. However, stasticians often use priors which are not proper: the prior is called improper if ∫ Θ F ′′ (θ)dθ = +∞ and any observation makes the corresponding posterior (B.4) integrable.

Some Intuition for choosing the Jeffreys Prior In the proof of our concentration result for posterior distributions (Theorem B.4) it will be crucial to lower bound the prior probability of an -sized KLdivergence ball around each of the parameters θ a . Since the Fisher information

, choosing a prior proportional to F ′′ (θ) ensures that the prior measure of such balls are Ω( √ ).

Examples and Pseudocode Algorithm 7 presents pseudocode for Thompson Sampling with the Jeffreys prior for distributions parametrized by their natural parameter θ. But as the Jeffreys prior is invariant under reparametrization, if a distribution is parametrised by some parameter λ ≡ θ, the algorithm can use the Jeffreys prior ∝ I(λ) on λ, drawing samples from the posterior on λ. Note that the posterior sampling step (in bold) is always tractable using, for example, a Hastings-Metropolis algorithm. Some examples of common exponential family models are given in Figure B.1, together with the posterior distributions on the parameter λ that is used by TS with the Jeffreys prior. In addition to examples already studied in [START_REF] Cappé | Kullback-Leibler upper confidence bounds for optimal sequential allocation[END_REF] for which T (x) = x, we also give two examples of more general canonical exponential families, namely the Pareto distribution with known min value and unknown tail index λ, Pareto(x m , λ), for which T (x) = log(x), and the Weibul distribution with known shape and unknown rate parameter, Weibull(k, λ), for which T (x) = x k . These last two distributions are not covered even by the work in [START_REF] Garivier | The KL-UCB algorithm for bounded stochastic bandits and beyond[END_REF], and belong to the family of heavy-tailed distributions.

For the Bernoulli model, we note futher that the use of the Jeffreys prior is not covered by the previous analyses. These analyses make an extensive use of the uniform prior, through the fact that the coefficient of the Beta posteriors they consider have to be integers.

In Theorem B.4, we bound the conditional probability that µ(θ a,t ) exceed the true mean. Following the same lines, we can also show that

For any ∆ ′ a > 0, one can choose δ 1 such that 1 -δ 1 C 1,1 > 0. Then, with N = N (P) such that the function u ↦ e -(u-1)(1-δ 1 C 2,1 )KL(θ 1 ,µ -1 (µ 1 -∆ ′ a ))+ln u is decreasing for u ≥ N , (B ′ ) is bounded by

Step 4: Choosing the Values δ a and a So far, we have shown that for any > 0 and for any choice of δ a > 0 and 0 < ∆ a < µ 1 -µ a such that 1 -δ a C 2,a > 0, there exists a constant C(δ a , ∆ a , , P) such that

The constant is of course increasing (dramatically) when δ a goes to zero, ∆ a to µ 1 -µ a , or to zero. But one can choose ∆ a close enough to µ 1 -µ a and δ a small enough, such that

and this choice leads to For ease of notation, we drop the subscript a and let (y s ) be an i.i.d. sequence of distribution p θ , with mean µ = µ(θ). Furthermore, by conditioning on the value of N s , it is enough to bound

Step 1: Extracting a Kullback-Leibler Rate The argument rests on the following Lemma, whose proof can be found in Appendix B.7

Lemma B.7. Let Ẽu be the event defined by (B.5), and introduce

The following inequality holds:

B.4. POSTERIOR CONCENTRATION: PROOF OF THEOREM B.4 197

Step 2: Upper bounding the numerator of (B.9) We first note that on Θ θ,∆ the leading term in the exponential is K(θ, θ ′ ). Indeed, from (B.3) we know that

which, by strict convexity of F , is strictly increasing in θθ ′ for any fixed θ. Now since µ is one-to-one and continuous, Θ c θ,∆ is an interval whose interior contains θ, and hence, on Θ θ,∆ ,

So for δ such that 1 -δC 2 > 0 we can bound the numerator of (B.9) by:

where we have used that π(⋅ y s ′ ) is a probability distribution, and that, since µ is increasing, KL(θ, µ -1 (µ+

Step 3: Lower bounding the denominator of (B.9) To lower bound the denominator, we reduce the integral on the whole space Θ to a KL-ball, and use the structure of the prior to lower bound the measure of that KL-ball under the posterior obtained with the well-chosen observation y s ′ . We introduce the following notation for KL balls: for any x ∈ Θ, > 0, we define

We have

Using this inequality we can then bound the denominator of (B.9) whenever u ≥ N 1 (θ, F ) and δ < 1:

Finally we turn our attention to the quantity

Now since the KL divergence is convex in the second argument, we can write B 1 u 2 (θ) = (a, b). So, from the convexity of F we deduce that

As p(y θ) → 0 as y → ±∞, the set C(θ) = {y ∶ p(y θ) ≥ L(θ)} is compact. The map y ↦ ∫ Θ p(y θ ′ ) F ′′ (θ ′ )dθ ′ < ∞ is continuous on the compact C(θ). Thus, it follows that

is an upper bound on the denominator of (B.12). Now by the continuity of F ′′ , and the continuity of (y, θ) ↦ p(y θ) in both coordinates, there exists an N 2 (θ, F ) such that for all u ≥ N 2 (θ, F )

Finally, for u ≥ N 2 (θ, F ), we have a lower bound on the numerator of (B.12):

Puting everything together, we get that there exist constants C 2 = C 2 (F, θ, ∆) and N (θ, F ) = max{N 1 , N 2 } such that for every δ < 1 satisfying 1 -δC 2 > 0, and for every u ≥ N , one has

Remark B.8. Note that when the prior is proper we do not need to introduce the observation y s ′ , which significantly simplifies the argument. Indeed in this case, in (B.10) we can use π 0 in place of π(⋅ y s ′ ) which is already a probability distribution. In particular, the quantity (B.12) is replaced by π 0 B 1 u 2 (θ) , and so the constants L and L ′ are not needed.

B.5 Conclusion

We have shown that choosing to use the Jeffreys prior in Thompson Sampling leads to an asymptotically optimal algorithm for bandit models whose rewards belong to a 1-dimensional canonical exponential family. The cornerstone of our proof is a finite time concentration bound for posterior distributions in exponential families, which, to the best of our knowledge, is new to the literature. With this result we built on previous analyses and avoided Bernoulli-specific arguments. Thompson Sampling with Jeffreys prior is now a provably competitive alternative to KL-UCB for exponential family bandits. Moreover our proof holds for slightly more general problems than those for which KL-UCB is provably optimal, including some heavy-tailed exponential family bandits.

Our arguments are potentially generalisable. Notably generalising to n-dimensional exponential family bandits requires only generalising Lemma B.3 and Step 3 in the proof of Theorem B.4. Our result is asymptotic, but the only stage where the constants are not explicitly derivable from knowledge of F , T , and θ 0 is in Lemma B.9. Future work will investigate these open problems. Another possible future direction lies the optimal choice of prior distribution. Our theoretical guarantees only hold for Jeffreys' prior, but a careful examination of our proof shows that the important property is to have, for every θ a , ln

which could hold for prior distributions other than the Jeffreys prior. (see [START_REF] Boucheron | Concentration inequalities. A non asymptotic theory of independence[END_REF]). For any λ > 0.

where we have used the Markov inequality, and where

Now we optimize in λ by choosing λ > 0 that maximizes

(Note that λ * > 0 since F ′ is increasing). Finally, we get

A ≤ e -u((δ+F ′ (θ))λ * -F (θ+λ * )+F (θ)) =e -u(F ′ (θ+λ * )λ * -F (θ+λ * )+F (θ)) = e -uK(θ+λ * ,θ) .

The same reasoning leads to the upper bound θ) ,

◻

For the proof of inequalities (B.6) and (B.7), we intoduce the notation Y u a,s ′ = Y s a {y a,s } (the first u observations of arms a exept observation y a,s ′ ). First note that we have Ẽc a,t ⊆ B a,Na,t ⋃ D a,Na,t , with

we can rewrite

and one also recalls that

The inequality (B.13) leads to the result, using that on Ẽu ,

From Theorem B.4 we know that, for N a,t ≥ N (θ a , F ), ,t-1)(1-δaC 2,a )KL(θa,µ -1 (µa+∆a))+ln Na,t ≤ e -(Na,t-1) (1-δaC 2,a )KL(θa,µ -1 (µa+∆a))-ln(C 1,a Na,t) (Na,t-1)

Let N = N (δ a , ∆ a , θ a ) be the smallest integer such that for all n ≥ N

we have that for all t and T such that N a,t -1 ≥ max(L T , N , N (θ a , F )),

τ is a stopping time with respect to F t . Then,

B.9 Controling the Number of Optimal Plays: Outline Proof of Proposition B.5

The proof of this proposition is quite detailed, and essentially the same as the proof given for Proposition 1 in [START_REF] Kaufmann | Thompson Sampling : an Asymptotically Optimal Finite-Time Analysis[END_REF], which we will sometimes refer to. However, in generalising to the case of exponential family bandits we show how to avoid the need to explicity calculate posterior probabilities that lead to Lemma 4 in [START_REF] Kaufmann | Thompson Sampling : an Asymptotically Optimal Finite-Time Analysis[END_REF]. While simplifying the proof we loose the ability to specify the constants explicitly, and so the analysis becomes asymptotic, but holds for every b ∈]0, 1[.

Sketch of the proof and key results

Let τ j be the occurrence of the j th play of the optimal arm (with τ 0 ∶= 0). Let ξ j ∶= (τ j+1 -1) -τ j : this random variable measures the number of time steps between the j th and the (j + 1) th play of the optimal arm, and so ∑ K a=2 N a,t = ∑ N 1,t j=0 ξ j . We then upper bound P(N 1,t ≤ t b ) as in [START_REF] Kaufmann | Thompson Sampling : an Asymptotically Optimal Finite-Time Analysis[END_REF]:

We introduce the interval I j = {τ j , τ j + ⌈t 1-b -1⌉}: on the event E j , I j is included in {τ j , τ j+1 } and no draw of arm 1 occurs on I. We also introduce for each arm a ≠ 1 d a ∶= µ 1 -µa 2 .

B.9. CONTROLING THE NUMBER OF OPTIMAL PLAYS: OUTLINE PROOF OF PROPOSITION B.5203

The idea of the rest of the analysis is based on the following remark. If on a subinterval I ⊆ [τ j , τ j+1 [ of size f (t) arm 1 is not drawn and all the samples of the suboptimal arms fall below µ 2 + d 2 < µ 1 , then for all s ∈ I, µ(θ 1,s ) ≤ µ 2 + d 2 . On I, the sequence (θ 1,s ) is i.i.d. with distribution π 1,τ j , and hence,

At this point, an asymptotic result, telling that the posterior on θ 1 concentrates to a Dirac in θ 1 (the Bernstein-Von-Mises theorem, see [Van der Vaart, 1998]) , leads to

Assuming that ∀j, P(µ(θ 1,τ j ) ≤ µ 2 + δ 2 ) ≠ 1, we have shown the following Lemma, which plays the role of an asymptotic counterpart for Lemma 3 in [START_REF] Kaufmann | Thompson Sampling : an Asymptotically Optimal Finite-Time Analysis[END_REF].

Lemma B.9. There exists a constant C = C(π 0 ) < 1, such that for every (random) interval I included in I j and for every positive function f , one has

Another key lemma is the following which generalizes Lemma 4 in [START_REF] Kaufmann | Thompson Sampling : an Asymptotically Optimal Finite-Time Analysis[END_REF]. The proof of this lemma is standard: it proceeds by conditioning on the event Ẽa,t 1 and applying Theorem B.4, and Lemma B.3.

Lemma B.10. For every a ∈ A, δ > 0, there exist constants C a = C a (µ a , δ, F ) and N such that for t ≥ N ,

The rest of the proof proceeds by finding a subinterval of I j on which all the samples of all the suboptimal arms indeed fall below the corresponding thresholds µ a + d a . This is done exactly as in [START_REF] Kaufmann | Thompson Sampling : an Asymptotically Optimal Finite-Time Analysis[END_REF] and we recall the main steps of the proof below. Before that, we need to introduce the notion of saturated, suboptimal action. Definition B.11. Let t be fixed. For any a ≠ 1, an action a is said to be saturated at time s if it has been chosen at least C a ln(t) times, i.e. N a,t ≥ C a ln(t). We shall say that it is unsaturated otherwise. Furthermore at any time we call a choice of an unsaturated, suboptimal action an interruption.

Step 1: Decomposition of I j We want to study the process of saturation on the event E j = {ξ j ≥ t 1-b -1}. We start by decomposing the interval I j = {τ j , τ j + ⌈t 1-b -1⌉} into K subintervals:

Now for each interval I j,l , we introduce: -F j,l : the event that by the end of the interval I j,l at least l suboptimal actions are saturated; -n j,l : the number of interruptions during this interval. We use the following decomposition to bound the probability of the event E j :

Note that the quantities E j , I j,l , F j,l and n j,l all depend on t, however we suppress this dependency for notational convenience. However, we keep in mind that we bound the different probabilities for t ≥ N , so that Lemma B.10 applies.

Step 2: Bounding P(E j ∩F j,K-1 ) On the event E j ∩F j,K-1 , only saturated suboptimal arms are drawn on the interval I j,K . Using Lemma B.10, we get

for 0 < C < 1 as in Lemma B.9. The second last inequality comes from the fact that if arm 1 is not drawn, the sample θ 1,s must be smaller than some sample θ a,s and therefore smaller than µ 2 + d 2 .

Step 3: Bounding P(E j ∩ F c j,K-1 ) A similar argument to that employed in Step 2 can be used in an induction to show that for all 2 ≤ l ≤ K, if t is larger than some deterministic constant N µ 1 ,µ 2 ,b specified in the base case,

We refer the reader to [START_REF] Kaufmann | Thompson Sampling : an Asymptotically Optimal Finite-Time Analysis[END_REF] for a precise description of the induction. For l = K we then get

.

(B.17)

Step 4: Conclusion Putting Steps 2 and 3 together we obtain that for t ≥ N 0 ∶= max(N, N µ 1 ,µ 2 ,b ),

where we use B.15. It then follows that

Analyse de stratégies bayésiennes et fréquentistes pour l'allocation séquentielle de ressources

Emilie KAUFMANN RESUME : Dans cette thèse, nous étudions des stratégies d'allocation séquentielle de ressources. Le modèle statistique adopté dans ce cadre est celui du bandit stochastique à plusieurs bras. Dans ce modèle, lorsqu'un agent tire un bras du bandit, il reçoit pour récompense une réalisation d'une distribution de probabilité associée au bras. Nous nous intéressons à deux problèmes de bandit différents : la maximisation de la somme des récompenses et l'identification des meilleurs bras (où l'agent cherche à identifier le ou les bras conduisant à la meilleure récompense moyenne, sans subir de perte lorsqu'il tire un «mauvais» bras). Nous nous attachons à proposer pour ces deux objectifs des stratégies de tirage des bras, aussi appelées algorithmes de bandit, que l'on peut qualifier d'optimales.

La maximisation des récompenses est équivalente à la minimisation d'une quantité appelée regret. Grâce à une borne inférieure asymptotique sur le regret d'une stratégie uniformément efficace établie par Lai et Robbins, on peut définir la notion d'algorithme asymptotiquement optimal comme un algorithme dont le regret atteint cette borne inférieure. Dans cette thèse, nous proposons pour deux algorithmes d'inspiration bayésienne, Bayes-UCB et Thompson Sampling, une analyse à temps fini dans le cadre des modèles de bandit à récompenses binaires, c'est-à-dire une majoration non asymptotique de leur regret. Cette majoration permet d'établir l'optimalité asymptotique des deux algorithmes.

Dans le cadre de l'identification des meilleurs bras, on peut chercher à déterminer le nombre total d'échantillons des bras nécessaires pour identifier, avec forte probabilité, le ou les meilleurs bras, sans la contrainte de maximiser la somme des observations. Nous définissons deux termes de complexité pour l'identification des meilleurs bras dans deux cadres considérés dans la littérature, qui correspondent à un budget fixé ou à un niveau de confiance fixé. Nous proposons de nouvelles bornes inférieures sur ces complexités, et nous analysons de nouveaux algorithmes, dont certains atteignent les bornes inférieures dans des cas particuliers de modèles de bandit à deux bras, et peuvent donc être qualifiés d'optimaux.

MOTS-CLEFS:

Modèles de bandit, minimisation du regret, identification des meilleurs bras.

ABSTRACT:

In this thesis, we study strategies for sequential resource allocation, under the so-called stochastic multi-armed bandit model. In this model, when an agent draws an arm, he receives as a reward a realization from a probability distribution associated to the arm. In this document, we consider two different bandit problems. In the reward maximization objective, the agent aims at maximizing the sum of rewards obtained during his interaction with the bandit, whereas in the best arm identification objective, his goal is to find the set of m best arms (i.e. arms with highest mean reward), without suffering a loss when drawing 'bad' arms. For these two objectives, we propose strategies, also called bandit algorithms, that are optimal (or close to optimal), in a sense precised below.

Maximizing the sum of rewards is equivalent to minimizing a quantity called regret. Thanks to an asymptotic lower bound on the regret of any uniformly efficient algorithm given by Lai and Robbins, one can define asymptotically optimal algorithms as algorithms whose regret reaches this lower bound. In this thesis, we propose, for two Bayesian algorithms, Bayes-UCB and Thompson Sampling, a finite-time analysis, that is a non-asymptotic upper bound on their regret, in the particular case of bandits with binary rewards. This upper bound allows to establish the asymptotic optimality of both algorithms.

In the best arm identification framework, a possible goal is to determine the number of samples of the arms needed to identify, with high probability, the set of m best arms. We define a notion of complexity for best arm identification in two different settings considered in the literature: the fixed-budget and fixed-confidence settings. We provide new lower bounds on these complexity terms and we analyse new algorithms, some of which reach the lower bound in particular cases of two-armed bandit models and are therefore optimal. KEY-WORDS: Bandit models, regret minimization, best arm identification