Pierre Perrault

Efficient Learning in Stochastic Combinatorial Semi-Bandits

Combinatorial stochastic semi-bandits appear naturally in many contexts where the exploration/exploitation dilemma arises, such as web content optimization (recommendation/online advertising) or shortest path routing methods. This problem is formulated as follows: an agent sequentially optimizes an unknown and noisy objective function, defined on a power set P([n]). For each set A tried out, the agent suffers a loss equal to the expected deviation from the optimal solution while obtaining observations to reduce its uncertainty on the coordinates from A. Our objective is to study the efficiency of policies for this problem, focusing in particular on the following two aspects: statistical efficiency, where the criterion considered is the regret suffered by the policy (the cumulative loss) that measures learning performance; and computational efficiency. It is sometimes difficult to combine these two aspects in a single policy. In this thesis, we propose different directions for improving statistical efficiency, while trying to maintain the computational efficiency of policies. In particular, we have improved optimistic methods by developing approximation algorithms and refining the confidence regions used. We also explored an alternative to the optimistic methods, namely randomized methods, and found them to be a serious candidate for combining the two types of efficiency.

Avant tout, je voudrais exprimer ma profonde et sincère gratitude à Michal et Vianney, pour leur accompagnement et leur support pendant ces trois années. Travailler avec vous a été un réel plaisir. Vous avez su me faire découvrir le monde de la recherche avec beaucoup de bienveillance et d'engagement, le tout dans une ambiance de travail conviviale et ludique. Je me suis vraiment senti libre concernant le choix des directions de recherche, et vous m'avez à chaque fois encouragé à privilégier celles que j'aimais, sans rien imposer, mais toujours avec de précieux conseils. Je vous en suis très reconnaissant car je pense que c'est l'ingrédient essentiel d'une thèse épanouissante.

Michal, I especially wanted to thank you for your investment in the success of my thesis. You have always opened me to the best opportunities, for example by allowing me to co-teach your Graphs in Machine Learning course, or by helping me find my internship at Adobe in San-José! You also have contributed a lot to my social life outside of research, I am thinking of the squash, swimming pool and beer sessions (by the way, thanks again for the racquet you offered me when my first paper got accepted, we absolutely have to test it together after the Covid!).

Vianney, ton côté très détendu en cherchant toujours à s'attaquer aux problèmes les plus "marrants" me restera longtemps en mémoire. Je pense en particulier aux longues discussions avec Emmanuel et Thomas devant ton écran tactile géant, à essayer de caractériser l'ordre optimal dans le "problème du secrétaire" (je pense également à Mathilde, qui a été à l'origine de ces discussions de mon point de vue), ou encore à mon stage du MVA, où le but était de "trouver des gens dans des arbres". J'approuve totalement cette façon de considérer la recherche comme un jeu, ce qui n'empêche pas d'obtenir des résultats intéressants ! Je tiens aussi à remercier tout particulièrement Marc, Alexandre, Claire, Richard et Raphaël d'avoir accepté d'évaluer ma thèse. Ce fut un honneur et un privilège de vous avoir en tant que membres du jury, tant vos travaux sur les bandits ont grandement inspiré mes recherches. J'ai également beaucoup apprécié vos nombreux retours, ainsi que l'intérêt que vous avez porté à mon manuscrit et à ma présentation.

Jennifer and Zheng, thank you so much for hosting me and teaching me a lot during my internship at Adobe! It was a very rewarding experience for me, both on a scientific level, with our double research project, and on a human level, with our trips to San-Francisco or our rodeo on the mechanical bull with Georgios! Hope to see you all very soon! Je voulais également remercier tous les chercheurs et doctorants de l'équipe Sequel/Scool et de l'équipe du CMLA/Centre Borelli. J'ai apprécié nos discussions de groupe, surtout lorsqu'il s'agissait de questions mathématiques épineuses (dédicace à Valentin, qui maîtrise vraiment cet art !). Merci à toi Étienne pour la qualité de tes réflexions, tant lors de notre collaboration que pendant les "group meetings". Mes pensées vont aussi à Xavier, mon "binôme" de thèse. J'ai vraiment apprécié notre coopération. Je me souviendrai longtemps des bons moments passés au quotidien avec toi, Tina, Jérémie, Mariano et Roger, dans notre bonne vieille "salle des doctorants", où l'ambiance n'était pas toujours des plus sérieuses 1 ! Je veux aussi saluer ceux qui me sont chers, à savoir ma famille et mes amis. Leurs attentions et leurs encouragements m'ont accompagné tout le long de cette aventure. Vous m'avez également permis de décompresser et de faire la fête ! Pour tous ces bons moments, je vous dis merci ! Mes pensées vont en particulier à mes parents, mes grands-parents et mes frères, qui se sont toujours intéressés à mes travaux, et ont fait leur possible pour créer un environnement propice à ma réussite. Enfin, un petit mot pour Julia, ma petite nièce adorée : j'ai hâte de pouvoir t'expliquer ma thèse quand tu seras assez grande ! Par-dessus tout, je tiens à témoigner ma reconnaissance à ma Mégane ! Tu as su être ma principale source d'inspiration et de soutien dans mes recherches. C'est à toi que je veux dédier ce travail, car tu y as largement contribué, notamment en m'apportant chaque jour un peu plus de bonheur.

Chapter 1 Introduction

Notations

We present here the general notation used in this thesis. We will also add more specific notations when the corresponding concepts are introduced.

For two real numbers x and y, we use the notation x ∨ y max{x, y} and x ∧ y min{x, y}. We also define sign(x) to be 0 if x = 0 and x/|x| otherwise. We denote by |A| the cardinality of a set A, and by A c the complement of A when the ground set is clear from the context. For any integer n ∈ N * , the set of integers between 1 and n is denoted by [n] {1, . . . , n}, and the associated power set is denoted P([n])

{A, A ⊂ [n]}. We denote the Minkowski sum of two sets Z, Z ⊂ R n as Z + Z {z + z , z ∈ Z, z ∈ Z }, and z + Z {z} + Z . We typeset vectors and matrices in bold and indicate components with indices, e.g., for some set I, a = (a i ) i∈I ∈ R I is a vector on I. When there is no ambiguity on the index set, we simply use the notation (a i ). We denote by a b (a i b i ) the Hadamard product of two vectors a and b. The Hadamard product of two matrices is simply the entrywise product, producing another matrix of the same dimension as the operands. We let e n,i be the i th canonical unit vector of R n . The incidence vector of any subset A ∈ P ([n]) is e n,A i∈A e n,i . When the dimension is clear from the context, we omit the n and use the notation e A and e i . If x, y are two real-valued vectors, we write x ≥ y if xy has components in R + , and use x ∨ y (x i ∨ y i ) i (resp. x ∧ y (x i ∧ y i ) i ). For a vector a, and p ≥ 1, we define the p -norm of a as a p ( i |a i | p ) 1/p , and the ∞ -norm of a as a ∞ sup i |a i |. We also define a 0 |{i, a i = 0}|. We denote by diag(a) the diagonal matrix with the elements a = (a i ) on the diagonal. We write I n diag e n, [n] , and only I when the dimension is clear from the context. We use A B to indicate the Löwner ordering of two n × n matrices [START_REF] Horn | Matrix analysis[END_REF], i.e., that A -B is a positive semi-definite (PSD) matrix. Furthermore, A + B means λ T (A -B)λ ≥ 0 for all λ ∈ R n + , and A A B (resp. A +A B) means λ T (A -B)λ ≥ 0 for all λ ∈ R n (resp. λ ∈ R n + ), with λ i = 0 for i / ∈ A. Obviously, we also use the notations y ≤ x, B , + , A , +A A. For any n × n matrix A 0, we define the euclidean seminorm associated to A as x A √ x T Ax, where x is a vector of R n . Events are indicated using the fraktur font A. I{A} is the {0, 1} indicator of the event A, i.e.,

I{A} =

1 if A holds 0 otherwise.

We denote by ¬A the complementary event of A. The following definitions are stated for a random variable X ∈ R but can also be extended for a random vector X ∈ R n . We denote by P X the probability distribution of X. For another random variable Y , we write Y ∼ P X (or Y ∼ X) to indicate that P Y = P X . For a sequence of random variables (X t ), σ(X 1 , . . . , X t ) is the sigma-algebra generated by random variables X 1 , . . . , X t . We write (X t ) ∼ P X to indicate that P Xt = P X for all t ∈ N * , and (X t )

iid ∼ P X to indicate that P (X 1 ,X 2 ,...,Xt) = P ⊗t X for all t ∈ N * . For two familly of distributions D 1 , D 2 , we use the notation D 1 ⊗ D 2 {P 1 ⊗ P 2 , P 1 ∈ D 1 and P 2 ∈ D 2 }.

For any finite set X , and any p ∈ [0, 1] X , such that x∈X p x = 1, the distribution x∈X p x δ x is a discrete probability measure on X . For example, δ x is the Dirac distribution at x, and for some p ∈ [0, 1], Bernoulli(p) = pδ 1 + (1 -p)δ 0 . For x ∈ [0, 1] n with i x i ≤ 1, we define the multinoulli distribution (also called categorical distribution) as Multinoulli(x) = i x i δ e i + (1i x i )δ 0 R n .

Présentation du contenu de la thèse (en Français)

Cette thèse étudie les problèmes d'optimisation combinatoire séquentielle dans un environnement stochastique, avec des observations que l'on qualifie de "semi-bandits". La terminologie "semi-bandits" fait reférence au fameux problème du bandit manchot stochastique (aussi appelé bandit stochastique à plusieurs bras). Les problèmes de bandit forment une classe importante de problèmes d'optimisation séquentielle, et ils ont été largement étudiés dans le domaine des statistiques et de l'apprentissage automatique. A l'origine, ils ont été introduits dans l'article fondateur de [START_REF] Robbins | Some aspects of the sequential design of experiments[END_REF] pour étudier les plans d'expériences séquentielles. La version classique du problème est formulée comme un système de n bras (ou machines). Un agent doit sélectionner à plusieurs reprises un bras parmi les n disponibles. Chaque bras i a une loi de probabilité inconnue P X i , de moyenne inconnue µ * i , où la variable X i encode une récompense. Après avoir sélectionné un bras, l'agent observe une réalisation indépendante de la distribution de récompense correspondante. Chaque décision est basée sur les décisions passées et les récompenses observées. La tâche pour l'agent consiste à jouer tour à tour les bras de manière à maximiser l'espérance de la récompense cumulée. L'agent doit jouer en équilibrant l'exploitation et l'exploration. En effet, les bras dont les récompenses observées sont les plus élevées doivent être sélectionnés souvent, tandis que tous les bras doivent être explorés pour connaître leurs récompenses moyennes. De manière équivalente, la performance de l'agent (ou de la politique) peut être évaluée par son regret R T , définie comme l'espérance de l'écart sur T tours (T ∈ N * étant appelé l'horizon) entre la récompense accumulée par la politique et celle accumulée par une politique oracle sélectionnant toujours le meilleur bras. Le regret peut se réécrire comme

R T i∈[n] E[N i,T ]∆ i ,
où ∆ i max j µ * j -µ * i est l'écart des moyennes entre le bras optimal et le bras i, et où N i,t est le nombre de fois où le bras i a été tiré jusqu'à l'instant t ∈ N * . La notion de regret quantifie ainsi la perte due à la nécessité d'apprendre les récompenses moyennes des différents bras.

Le problème défini ci-dessus est un exemple de problème d'apprentissage par renforcement. En effet, un problème de bandit manchot peut être vu comme un processus de décision markovien avec un seul état. Mentionnons au passage que le nom du problème vient du fait d'imaginer un joueur face à une rangée de machines à sous, devant décider quelles machines jouer, combien de fois jouer à chaque machine et dans quel ordre les jouer. Bien entendu, il ne s'agit que d'une modélisation, et ce problème aborde en pratique tout type de situation où un agent tente simultanément d'acquérir de nouvelles connaissances (exploration) et d'optimiser ses décisions sur la base des connaissances existantes (exploitation). Il existe de nombreuses applications pratiques du modèle du bandit (manchot), par exemple :

• Les essais cliniques qui étudient les effets de différents traitements expérimentaux tout en minimisant les vies humaines perdues [START_REF] Thompson | On the likelihood that one unknown probability exceeds another in view of the evidence of two samples[END_REF][START_REF] Gittins | Multi-armed Bandit Allocation Indices[END_REF][START_REF] Berry | Bandit Problems: Sequential Allocation of Experiments[END_REF].

• La conception de portefeuilles financiers [START_REF] Hoffman | Portfolio Allocation for Bayesian Optimization[END_REF].

• La prise de décision pour l'accès dynamique au spectre dans le contexte des radios cognitives [START_REF] Lai | Medium access in cognitive radio networks: A competitive multi-armed bandit framework[END_REF].

Le problème des bandits a notamment été étudié par [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF]. Ils ont prouvé une borne inférieure asymptotique sur le regret R T . Cette borne indique que toute politique "raisonnable" (en un certain sens) doit subir au moins un regret logarithmique par rapport à l'horizon T . Elle fournit ainsi une limite de performance fondamentale qu'aucune politique "raisonnable" ne peut battre. La constante dans la borne inférieure est linéaire en n, le nombre de bras. Plus précisement, elle vaut la somme sur les bras sous-optimaux i de l'écart ∆ i fois l'inverse de la divergence de Kullback-Leibler (KL) [START_REF] Kullback | On information and sufficiency[END_REF]) entre la distribution du bras i et celle du bras optimal. Grossièrement, log(T ) fois l'inverse de la KL représente le nombre de tours nécessaires pour distinguer le bras i du bras optimal, donnant ainsi une interprétation claire de la borne inférieure. Dans le cas de distributions Gaussiennes, l'inverse de la KL est minoré (à une constante multiplicative près) par la variance σ 2 i du bras i divisée par ∆ 2 i . En simplifiant, la borne inférieure devient donc

Ω   log(T ) i∈[n], ∆ i >0 σ 2 i ∆ i   .
(1.1)

Il n'est pas surprenant que la variance apparaisse, car elle peut être considérée comme une mesure de l'incertitude des échantillons : plus la variance est élevée, plus l'estimation est difficile, et donc plus le regret doit être élevé. [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] ont également élaboré des politiques pour certaines distributions de récompense et ont montré qu'elles atteignent asymptotiquement la borne inférieure. En effet, ces politiques ont une borne supérieure sur leur regret qui est logarithmique en T , avec une constante du même ordre que dans la borne inférieure. Dans le cas où les bras ont des distributions σ 2 i -sous-Gaussiennes, la stratégie ucb (pour Upper Confidence Bound [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF]) atteint asymptotiquement la borne (1.1). Elle consiste à choisir, au tour t, le bras i t maximisant la borne supérieure de l'intervalle de confiance µ i,t-1 + 2 log(t)/N i,t-1 , où µ i,t-1 est la moyenne empirique des récompenses reçues en ayant tiré le bras i. Il s'agit d'une stratégie dite optimiste dans l'incertain, car elle choisit le bras qui serait le meilleur si les moyennes vallaient leurs estimations optimistes.

Bandit stochastique combinatoire Pour revenir au sujet de cette thèse, à savoir les problèmes d'optimisation séquentielle combinatoire, nous pouvons étendre le cadre mentionné ci-dessus : l'agent peut désormais, lors d'un tour, choisir plusieurs bras au lieu d'un seul (avec éventuellement quelques contraintes sur les ensembles de bras possibles). La récompense obtenue est fonction de la réalisation individuelle de chaque bras choisi. Ceci est bien adapté aux applications où les actions dont dispose l'agent appartiennent à un espace combinatoire et sont construites à partir de plusieurs petites actions (les bras). Mentionnons par exemple le problème du plus court chemin entre deux points d'un réseau (qui a été étudié notamment par [START_REF] Liu | Adaptive shortest-path routing under unknown and stochastically varying link states[END_REF], [START_REF] Talebi | Stochastic Online Shortest Path Routing: The Value of Feedback[END_REF]). Chaque arête du réseau est un bras auquel est associé une loi de probabilité, modélisant par exemple la fluidité du trafic sur cette arête. L'agent choisit à chaque tour un chemin reliant les deux points, et obtient en récompense une réalisation indépendante de la fluidité globale du chemin choisi. Cette fluidité globale est définie selon le contexte, et peut être par exemple la somme des fluidités des arêtes du chemin (voir Figure 1.1). Ce cas particulier où la récompense est la somme des réalisations des bras choisis correspond à un problème d'optimisation combinatoire séquentiel avec une fonction objectif linéaire, et il s'agit du scénario le plus répandu parmi ces types de problèmes. Il est étudié, par exemple, par [START_REF] Abernethy | Competing in the dark: An efficient algorithm for bandit linear optimization[END_REF], [START_REF] Dani | The Price of Bandit Information for Online Optimization[END_REF], ou encore Bubeck, Cesa-Bianchi, et [START_REF] Bubeck | Towards minimax policies for online linear optimization with bandit feedback[END_REF]. De manière générale, l'extension combinatoire du problème de bandit est connue dans la littérature sous le nom de bandit stochastique combinatoire. Elle implique (au moins) deux cadres distincts concernant les observations dont dispose l'agent sur les réalisations des bras à chaque tour :

• Observation "semi-bandit" : toutes les réalisations des bras choisis sont observées par l'agent.

• Observation "bandit" : seule la récompense (qui est fonction des réalisations des bras choisis) est observée.

Comme nous l'avons déjà mentionné, nous nous intéressons ici au premier cadre, communément appelé semi-bandits stochastiques combinatoires (que nous abrégerons en CMAB dans cette thèse, pour combinatorial multi-armed bandits, le "semi" étant omis par souci de concision). Puisque l'agent a plus d'observations, cette hypothèse peut être plus difficile à satisfaire dans la pratique, mais peut conduire à des politiques plus performantes. Notons aussi que le second cadre est en fait un cas particulier du problème de bandit introduit plus haut, où le nombre de bras est certes combinatoire, mais où une certaine structure est spécifiée entre les récompenses.

Observation "semi-bandit"

Nous formalisons ici le problème CMAB. La réalisation du bras i ∈ [n] au tour t est notée X i,t ∈ R. Les vecteurs aléatoires (X t ) t≥1 de R n sont i.i.d., avec une distribution inconnue P X , de moyenne µ * ∈ R n . Néanmoins, sauf mention contraire, X i et X j pour deux bras distincts i = j peuvent être arbitrairement corrélées. L'action (aussi appelée super-bras) sélectionnée par l'agent au tour t (i.e., l'ensemble des bras choisis) est notée A t . L'ensemble des super-bras possibles est appelé espace d'action, et est noté A. C'est un sous-ensemble fixé de P([n]), tel que chacun de ses éléments A est un sous-ensemble d'au plus m bras. Après avoir sélectionné un bras A t au tour t, l'agent reçoit comme retour d'information e At X t . Bien que nous considérons un cadre plus général dans la thèse, afin de simplifier les explications, nous nous limitons ici au cas d'une fonction de récompense linéaire : la récompense au tour t est donc e T At X t . Nous supposons aussi que A est tel que l'optimisation de fonctions linéaires peut être fait de manière efficace (en temps polynomial en n).

L'objectif est d'identifier une politique qui maximise la récompense cumulée espérée sur l'ensemble des T tours. L'espérance est prise sur l'aléa des récompenses et sur l'éventuel aléa de la politique suivie par l'agent (une action peut être choisie au hasard). De manière équivalente, nous visons à concevoir une politique π qui minimise le regret, défini par :

R T (π) max A∈A E T t=1 (e A -e At ) T X t .
Contrairement au problème de bandit classique, nous pouvons remarquer qu'il y a une notion de similarité entre les actions. En effet, deux actions "proches", c'està-dire ayant beaucoup de bras en commun, ont tendance à avoir une récompense "proche". Cette structure peut être exploitée dans l'apprentissage, afin de contrebalancer la taille combinatoire de l'espace d'action. Il en résulte la conception de politiques efficaces de sélection des super-bras, basée sur le principe d'optimisme dans l'incertain, dont le regret est de l'ordre de O(n • m log(T )/∆) (Kveton, Wen, Ashkan, and Szepesvari, 2015b), où ∆ > 0 est la différence minimale entre la moyenne d'un super-bras optimal et la moyenne d'un super-bras sous-optimal. Ces politiques atteignent asymptotiquement une borne inférieure : en considérant l'espace d'action A = {A 1 , . . . , A n/m } avec A k = {(k -1)m + 1, . . . , km}, et en posant pour tout k, X (k-1)m+1 = • • • = X km , tirée selon une Gaussienne de variance 1, on réduit le problème à un problème de bandit classique, permettant d'appliquer la borne (1.1), avec σ 2 i = m 2 . La politique la plus connue de ce type est cucb (Combinatorial Upper Confidence Bound), et est fortement inspirée de ucb. En effet, elle joue au tour t A t ∈ arg max A∈A e T A µ i,t-1 + 2 log(t) N i,t-1 i , en d'autres termes, les estimations optimistes sont les mêmes que pour ucb. Notons que dans les problèmes CMAB, la distribution conjointe tout entière du vecteur des réalisations X a de l'importance, contrairement aux bandits classiques où seulement les marginales sont suffisantes pour caractériser une instance d'un problème. Ainsi, lorsqu'il est question d'estimer le vecteur des moyennes µ * , nous sommes davantage Chapter 1. Introduction

µ t-1 • Figure 1
.2: La région de confiance de type hypercube (en noir) correspond à un a priori Gaussien pour chaque marginale. La corrélation entre ces marginales peut cependant être arbitraire : voici plusieurs exemples, de très négativement corrélées (en orange) à très positivement corrélées (en rouge), en passant par indépendantes (en vert).

On voit donc que tout l'hypercube est susceptible de contenir la vraie moyenne µ * tant que l'on ne spécifie pas (et que l'on n'apprend pas) les corrélations entre les marginales.

intéressés par une région de confiance que par la simple collection de n intervalles de confiance. Une autre manière de voir cucb est donc de considérer la région de confiance C t qui lui est associée : il s'agit ni plus ni moins que du produit cartésien des intervalles de confiance qui étaient considérés par ucb. Voici une façon de décrire différemment cucb en terme de région de confiance : , on parle également de bonus de type 1 ), soit regarder la région de confiance autour du vecteur des moyennes empiriques µ t-1 (ici, la région est un hypercube, on parle également de région de type ∞ ). Nous pouvons remarquer que le problème d'optimisation ci-dessus peut être résolu efficacement (c'est un problème d'optimisation linéaire sur A) : cucb est donc efficiente sur le plan statistique (parce qu'elle atteint la borne inférieure) et sur le plan computationnel.

A t ∈
Notons que la distribution P X choisie pour établir la borne inférieure ci-dessus est extrême : elle représente une situation où les bras appartenant à une même action sont parfaitement corrélés. Ce comportement extrême est également mis en évidence lorsque l'on regarde la région de confiance considérée par cucb : on ne s'attend pas vraiment à avoir une zone de confiance ayant la forme d'un hypercube, mais plutôt ayant la forme d'une ellipsoïde, en prenant comme a priori une distribution Gaussienne multivariée. L'hypercube vient du fait que l'orientation et l'excentricité de l'ellipsoïde peuvent être arbitraire (voir Figure 1.2). Si, par exemple, nous nous limitons à la classe des distributions satisfaisant P X = ⊗ i∈[n] P X i (ce qui signifie que les distributions des bras sont mutuellement indépendantes), alors la borne inférieure devient Ω(n log(T )/∆) (car σ 2 i = m dans ce cas). Cela signifie que cucb n'est plus efficiente sur le plan statistique, et que nous pouvons potentiellement gagner un facteur m dans la borne supérieure du regret. [START_REF] Combes | Combinatorial Bandits Revisited[END_REF] ont étudié ce problème, et ont donné une politique qui s'appuie effectivement sur l'indépendance des réalisations pour réduire la région de confiance utilisée. Plus précisément, ils proposent une politique, appelée escb (pour Efficient Sampling for Combinatorial Bandit), qui jouent au tour t l'action

A t ∈ arg max A∈A e T A µ t-1 + i∈A f (t) N i,t-1
, où f (t) est de l'ordre de O(log(t)). On remarque cette fois-ci qu'on a affaire à un bonus de type 2 , correspondant à la région de confiance de type 2 (ici, l'ellipsoïde de confiance) suivante :

C t µ t-1 +    ξ ∈ R n , i∈[n] N i,t-1 ξ 2 i ≤ f (t)    .
Remarquons que l'ellipsoïde ci-dessus est alignée par rapport aux axes (ce qui est cohérent avec l'hypothèse d'indépendance). Degenne et Perchet (2016b) sont allés plus loin en considérant la classe des distributions P X qui sont C-sous-Gaussiennes (multivariées), pour une matrice C 0, i.e., telles que ∀λ ∈ R n , E e λ T (X-µ * ) ≤ e λ T Cλ/2 .

Dans ce cas, ils ont construit une politique, ols-ucb, nécessitant la connaissance d'une autre matrice Γ 0 à coefficients positifs telle que Γ + C. Cette politique considère une ellipsoïde de confiance basée sur la matrice Γ (i.e., non alignée par rapport aux axes quand Γ n'est pas diagonale). Elle se réduit essentiellement à la politique escb dans le cas spécifique où la matrice Γ est diagonale. ols-ucb a une borne sur le regret de l'ordre de

O   log T ∆ i∈[n] Γ ii (1 -γ) log 2 (m) + γm   , où γ max A∈A max (i,j)∈A 2 ,i =j
Γ ij / Γ ii Γ jj . On voit donc que escb, dans le cas de distributions mutuellement indépendantes, est efficiente sur le plan statistique (d'où son nom). En effet, on a alors γ = 0 dans la borne ci-dessus, donnant un regret pour escb de l'ordre de O(log 2 (m)n log(T )/∆), qui est serré à un facteur polylogarithmique en m près. Degenne et Perchet (2016b) ont aussi montré que leur politique ols-ucb était efficiente sur le plan statistique pour la classe des distributions sous-Gaussiennes multivariées considérées : ils ont en effet prouvé une borne inférieure de l'ordre de Ω n log T ∆ ((1 -γ) + γm) . On voit en particulier apparaître une interpolation entre deux régimes : pour γ proche de 0, on retrouve la même borne inférieure que dans le cas où les bras ont des distributions mutuellement indépendantes, et pour γ proche de 1, on retrouve la borne inférieure qui correpond au cas où les bras sont parfaitement corrélés.

Nous pouvons enfin remarquer que les politiques escb et ols-ucb ne sont pas efficientes sur le plan computationnel en général (car le problème combinatoire à résoudre à chaque tour n'est plus linéaire, et est même NP-difficile en général [START_REF] Atamtürk | Maximizing a class of utility functions over the vertices of a polytope[END_REF]).

Un bref aperçu sur les récompenses non linéaires et les contraintes de budget

Généralement, on suppose que la récompense espérée lorsque l'agent choisit une action A est de la forme r(A; µ * ). Si telle est le cas, la fonction r est appellée fonction de récompense. Elle est dite linéaire lorsque r(A; µ * ) = e T A µ * . On suppose ici, pour simplifier, que pour un vecteur µ fixé, la fonction A → r(A; µ) peut être maximisé exactement et efficacement sur A. Nous avons déjà mentionné que cette thèse ne se concentrait pas sur les fonctions de récompense linéaires. Néanmoins, la plupart des fonctions de récompense que nous rencontrerons ont un comportement semblable au cas des récompenses linéaires. Pour être plus précis, nous verrons qu'il existe fréquemment un contrôle en norme 1 sur la déviation de la récompense lorsque le paramètre que l'on désire apprendre (ici la moyenne) varie. Par exemple, une hypothèse usuelle, considérée par [START_REF] Wang | Improving regret bounds for combinatorial semi-bandits with probabilistically triggered arms and its applications[END_REF], est r(A; µ) -r(A; µ ) ≤ e A (µµ ) 1 .

(1.2)

Lorsque cette hypothèse est vérifiée, les approches optimistes ci-dessus restent valables (c'est-à-dire que les bornes sur le regret sont du même ordre) en considérant, dans les politiques, le problème d'optimisation max A∈A, µ∈Ct r(A; µ) afin de choisir A t . Une difficulté des récompenses non linéaires dans les approches optimistes est que la quantité max µ∈Ct r(A; µ) peut ne pas être simple à calculer (et encore moins à optimiser sur A ∈ A). Néanmoins, il existe des cas où c'est possible. Par exemple, si nous avons une propriété de croissance (composante par composante) de la fonction µ → r(A; µ), alors, pour une région de confiance de type ∞ , on peut directement appliquer le vecteur des estimations optimistes au lieu de la vraie moyenne afin de sélectionner l'action A t à prendre [START_REF] Chen | Combinatorial Multi-Armed Bandit: General Framework and Applications[END_REF], 2016), il en résulte donc que cette politique est efficiente sur le plan computationnel. Un cas particulièrement intéressant de fonction de récompense non linéaire apparaît lorsque nous considérons un problème de bandit combinatoire à budget [START_REF] Xia | Budgeted multi-armed bandits with multiple plays[END_REF]. Pour expliquer brièvement ce cadre, il n'y a plus d'horizon T , mais plutôt un budget B, et chaque action qui est prise lors d'un tour est coûteuse. Nous verrons que ce cadre peut être essentiellement résolu de la même manière que le problème standard. Une différence notable est que la fonction objectif à optimiser au sein de chaque tour n'est plus une fonction de récompense, mais un rapport entre une fonction de récompense et une fonction de coût. Ce type de problème est donc à l'origine de nombreuses fonctions objectifs non linéaires dans les problèmes CMAB, même si les fonctions de récompense et de coût étaient initialement linéaires.

Bras à déclenchement probabiliste

Dans le cadre typique du problème CMAB, l'ensemble des bras déclenchés et l'action joué par l'agent sont confondus. Plus précisément, l'agent sélectionne une action à jouer à chaque tour, déclenchant un ensemble de bras, et les réalisations de ces bras sont alors observées. Une généralisation intéressante, appelé bras à déclenchement probabiliste (que nous abrégeons en CMAB-T dans cette thèse), a été introduite par Chen, Wang, et Yuan (2016) et [START_REF] Wang | Improving regret bounds for combinatorial semi-bandits with probabilistically triggered arms and its applications[END_REF]. L'idée est que l'action sélectionnée par l'agent n'est plus réduite à un ensemble de bras observés, mais appartient à un espace extérieur. Après qu'une action soit sélectionnée, certains bras sont déclenchés de manière probabiliste. Ce cadre a notamment été utilisé afin d'aborder des problèmes comme les bandits en cascade (portant sur la sélection des pages web à afficher pour une requête sur un moteur de recherche, [START_REF] Kveton | Cascading Bandits: Learning to Rank in the Cascade Model[END_REF] et [START_REF] Kveton | Combinatorial Cascading Bandits[END_REF]) ou encore la maximisation d'influence incrémentielle (portant sur la sélection d'influenceurs dans un réseau social, Chen, Wang, et Yuan (2016), [START_REF] Wen | Online influence maximization under independent cascade model with semi-bandit feedback[END_REF] et [START_REF] Wang | Improving regret bounds for combinatorial semi-bandits with probabilistically triggered arms and its applications[END_REF]).

Une hypothèse très pratique pour conserver la borne sur le regret obtenue plus haut pour les régions de type ∞ est de considérer la même relation de régularité que (1.2), mais en pondérant chaque terme (correspondant à un bras) dans la norme par la probabilité d'observer le bras correspondant en choisissant l'action. Cette hypothèse est vérifiée notamment dans les problèmes de bandit en cascade et de maximisation d'influence incrémentielle [START_REF] Wang | Improving regret bounds for combinatorial semi-bandits with probabilistically triggered arms and its applications[END_REF]).

Problématiques de la thèse et contributions

Après l'état des lieux qui précède, de nombreuses questions se posent, et chercher à y répondre constitue les objectifs de cette thèse. De manière générale, on s'intéressera à améliorer l'efficience des politiques, tant du point de vue computationnel que du point de vue statistique. Plus précisément, nous porterons un intérêt particulier pour les questions suivantes :

• Pouvons-nous espérer implémenter une version efficiente des politiques escb et ols-ucb ? Existe-t-il des alternatives qui sont efficientes tant sur le plan computationnel que sur le plan statistique (dans le cas de distributions indépendantes, ou sous-Gaussiennes multivariées) ?

• Est-il possible d'associer l'analyse 2 avec le cadre des contraintes de budget ou des bras à déclenchement probabiliste ? Quel genre de borne sur le regret pouvons-nous obtenir ? Une telle analyse peut-elle contribuer à améliorer les politiques existantes (et le regret qu'elles suscitent) pour des problèmes connus, comme la maximisation d'influence incrémentielle ?

• Pouvons-nous optimiser l'analyse de Degenne et Perchet (2016b) ? Par exemple, pouvons-nous assouplir l'hypothèse selon laquelle la matrice de sous-Gaussianité Γ est connue ? Existe-t-il une alternative à la famille des variables aléatoires sous-Gaussiennes multivariées ? Est-il possible d'expliciter le comportement d'interpolation dans le regret, par exemple en remplaçant Γ ii (1 -γ + γm) par la quantité plus faible max A∈A: i∈A j∈A Γ ij ?

Nous allons maintenant présenter le contenu de la thèse, chapitre par chapitre, en mettant l'accent sur les aspects liés aux questions soulevées ci-dessus.

Chapitre 2, Bandit Manchot Stochastique Nous passons d'abord en revue les résultats de base du problème classique des bandits à plusieurs bras. Certains d'entre eux seront utiles plus tard dans le contexte combinatoire.

Chapitre 3, Cadre Général pour les Observations Semi-Bandits Nous formalisons ici le cadre des semi-bandits stochastiques combinatoires avec bras à déclenchement probabiliste (qui, on le rappelle, englobe le cadre habituel). Nous donnons ensuite une multitude d'applications de ce cadre que l'on trouve dans la littérature. Enfin, nous fournissons des résultats techniques généraux qui seront utiles tout au long de la thèse pour prouver des bornes supérieures sur le regret des politiques. Plus précisément, les théorèmes énoncés dépendent du type d'erreur que l'on veut contrôler (qui n'est rien d'autre que le type de bonus que l'on utilise). Pour l'analyse ∞ (i.e., basée sur une région de confiance de type ∞ , ou d'une manière équivalente sur un bonus de type 1 ), nous avons un théorème qui améliore légèrement (et surtout simplifie grandement) l'analyse ∞ de [START_REF] Wang | Improving regret bounds for combinatorial semi-bandits with probabilistically triggered arms and its applications[END_REF]. Pour ce qui est de l'analyse 2 , nous fournissons plusieurs nouveaux résultats, étendant le travail de Degenne et Perchet (2016b), et surpassant l'analyse ∞ (gagnant un facteur m, à un facteur polylogarithmique près). Ce faisant, nous parvenons à associer l'analyse 2 avec le cadre des bras à déclenchement probabiliste. Ce chapitre contient quelques résultats non publiés, et quelques améliorations de résultats publiés dans :

• Pierre Perrault, Jennifer Healey, Zheng Wen, Michal Valko (2020a) "Budgeted Online Influence Maximization", dans 37th International Conference on Machine Learning (ICML).

• Pierre Perrault, Vianney Perchet, Michal Valko (2020) "Covariance-adapting algorithm for semi-bandits with application to sparse rewards", dans 33rd Conference on Learning Theory (COLT).

• Pierre Perrault, Etienne Boursier, Vianney Perchet, Michal Valko (2020) "Statistical Efficiency of Thompson Sampling for Combinatorial Semi-Bandits", dans 34th Conference on Neural Information Processing Systems (NeurIPS).

Chapitre 4, Un Exemple de Problème de type CMAB-T : Rechercheet-Arrêt Séquentiels Nous présentons ici un nouvel exemple de problème qui se situe à la fois dans le cadre du bras à déclenchement probabiliste et dans le cadre à budget. Nous appelons ce problème recherche-et-arrêt séquentiels. Il est décrit de la manière suivante : on considère un graphe acyclique dirigé, où chaque noeud est un bras associé à un coût. Il y a un objet qui est caché au hasard dans l'un des noeuds (selon une distribution inconnue à apprendre). Dans un tour, l'agent peut inspecter des noeuds, choisis un par un, avec la contrainte qu'un noeud est accessible si ses voisins entrants ont déjà été inspectés. Il peut décider à tout moment d'arrêter sa recherche, et ainsi passer à une nouvelle instance indépendante (où l'objet est replacé au hasard). L'agent dispose d'un budget initial, et chaque fois qu'un noeud est inspecté, son prix est payé. L'objectif de l'agent est de trouver un nombre maximal d'objet (en espérance). Nous fournissons une politique basée sur une approche ∞ , en décrivant notamment un stratégie quasi-optimale et efficace pour le problème "horsligne" (i.e., lorsque la distribution est connue par l'agent, qui doit alors se focaliser sur l'exploitation). Ce chapitre est adapté de la publication suivante :

• Pierre Perrault, Vianney Perchet, Michal Valko (2019b) "Finding the bandit in a graph: Sequential search-and-stop", dans 22nd International Conference on Artificial Intelligence and Statistics (AIStats).

Chapitre 5, La Structure de l'Incertitude Nous commençons par rappeler les deux principales méthodes pour obtenir des zones de confiance 2 : la méthode de Laplace, et l'argument de la couverture. Nous étudions ensuite plus en profondeur la structure des bonus 2 , et nous prouvons qu'en tant que fonctions sur P([n]), elles sont sous-modulaires. Nous nous appuyons ensuite sur cette structure pour proposer plusieurs algorithmes d'approximation afin de maximiser la somme d'une fonction linéaire (A → e T A µ t-1 ) et d'une fonction sous-modulaire (le bonus d'exploration). Ceci nous permet d'obtenir des approximations de escb pour des contraintes de matroïde (un certain type d'espace d'action A généralisant la contrainte sur le cardinal). Ces approximations sont efficientes sur le plan computationnel et ne dégradent pas l'ordre du regret. Nous fournissons également une extension de cette méthode au cadre à budget. Ce chapitre est basé sur la publication suivante :

• Pierre Perrault, Vianney Perchet, Michal Valko (2019a) "Exploiting structure of uncertainty for efficient matroid semi-bandits", dans 36th International Conference on Machine Learning (ICML).

Chapitre 6, Maximisation d'Influence Incrémentielle Budgétisée Ce chapitre vise à approfondir l'étude d'un autre exemple bien connu de problème impliquant des bras à déclenchement probabiliste, à savoir la maximisation d'influence incrémentielle. Pour rappel, dans ce problème, un agent apprend activement à connaître un réseau social en interagissant avec lui de manière répétée, en essayant de trouver les meilleurs ensembles d'influenceurs. Nous incorporons un cadre à budget à ce problème, en considérant le coût total d'une campagne publicitaire au lieu de la contrainte (par tour) de cardinalité. Notre approche modélise ainsi mieux le contexte du monde réel où le coût des influenceurs varie et où les publicitaires veulent trouver le meilleur rapport qualité-prix pour leur budget global. Nous proposons des politiques de types ∞ et 2 , en utilisant la sous-modularité d'une borne supérieure sur les bonus 2 . Ce chapitre est adapté de la publication et de la prépublication suivantes :

• Pierre Perrault, Jennifer Healey, Zheng Wen, Michal Valko (2020a) "Budgeted Online Influence Maximization", dans 37th International Conference on Machine Learning (ICML).

• Pierre Perrault, Jennifer Healey, Zheng Wen, Michal Valko (2020b) "On the Approximation Relationship between Optimizing Ratio of Sub-modular (RS) and Difference of Submodular (DS) Functions", en cours de révision.

Chapitre 7, Politique Adaptative à la Covariance Le but ici est d'offrir une solution plus pratique, et plus précise pour construire une région de confiance 2 . En effet, celles que nous avons vues jusqu'à présent sont soit basées sur l'indépendance mutuelle des réalisations, soit dues à la connaissance irréaliste d'une matrice de sous-Gaussianité Γ. Nous considérons une nouvelle famille générale de distributions sousexponentielles paramétrée par la matrice de covariance, supposée inconnue. Cette famille contient les distributions à support borné et les Gaussiennes. Nous prouvons une nouvelle borne inférieure sur le regret de cette famille, qui est paramétrée par la matrice de covariance, et non plus la matrice de sous-Gaussianité. Nous construisons ensuite un algorithme qui utilise des estimations de la covariance, et fournissons une analyse asymptotique du regret, atteignant la borne inférieure. Enfin, nous appliquons et étendons nos résultats à la famille des réalisations parcimonieuses, qui a des applications dans de nombreux systèmes de recommandation. Ce chapitre est adapté de la publication suivante :

• Pierre Perrault, Vianney Perchet, Michal Valko (2020) "Covariance-adapting algorithm for semi-bandits with application to sparse rewards", dans 33rd Conference on Learning Theory (COLT).

Chapitre 8, Efficience Statistique et Computationnelle de l'Echantillonnage de Thompson Dans ce chapitre, nous étudions une alternative intéressante aux méthodes optimistes envisagées jusqu'à présent, à savoir l'échantillonnage de Thompson. Outre sa supériorité empirique [START_REF] Chapelle | An empirical evaluation of Thompson sampling[END_REF], l'échantillonnage de Thompson est intéressant dans le contexte des bandits combinatoires car l'action à jouer est facile à calculer : il ne faut considérer ni bonus, ni maximum sur une région de confiance, car le paramètre à utiliser dans la fonction objectif est simplement pris au hasard, selon un a priori bien choisi. L'échantillonnage de Thompson pourrait donc répondre à la question de l'existence d'une politique efficace (computationnellement) avec un regret asymptotique optimal (à un facteur polylogarithmique en m près), qui est encore ouverte pour de nombreuses familles de distributions, incluant les réalisations mutuellement indépendantes, et plus généralement la famille des réalisations sous-Gaussiennes multivariées. Nous proposons de répondre à la question ci-dessus pour ces deux familles en analysant des variantes de la politique Combinatorial Thompson Sampling (cts). Pour des réalisations mutuellement indépendantes dans [0, 1], nous proposons une borne serrée pour le regret de cts, en utilisant la loi bêta comme a priori. Nous examinons ensuite le cadre plus général des réalisations sous-Gaussiennes multivariées et proposons une borne serrée pour le regret de cts à l'aide d'un a priori Gaussien. Ce dernier résultat nous donne une alternative efficiente à la politique escb. Ce chapitre est adapté de la publication suivante :

• Pierre Perrault, Etienne Boursier, Vianney Perchet, Michal Valko (2020) "Statistical Efficiency of Thompson Sampling for Combinatorial Semi-Bandits", dans 34th Conference on Neural Information Processing Systems (NeurIPS).

Chapitre 9, Conclusions et Perspectives Ce chapitre tire les conclusions de la thèse et fournit quelques orientations pour les futurs travaux.

Presentation of the thesis content (in English)

This thesis studies sequential combinatorial optimization problems in a stochastic environment, with observations that we call "semi-bandit". The terminology "semibandit" refers to the famous stochastic multi-armed bandit problem. Bandit problems are an important class of sequential optimization problems and have been widely studied within the field of statistics and machine learning. They were originally introduced in the seminal paper of [START_REF] Robbins | Some aspects of the sequential design of experiments[END_REF] to study sequential experimental designs. The classical version of the problem is formulated as a system of n arms (or machines). An agent must repeatedly select an arm from the n. Each arm i has an unknown probability distribution P X i , of unknown mean µ * i , where the variable X i encodes a reward. After selecting an arm, the agent observes an independent realization (also called outcome) of the corresponding reward distribution. Each decision is based on past decisions and observed rewards. The agent's task is to sequentially play the arms in order to maximize the expectation of the cumulative reward. The agent must play by balancing exploitation and exploration. Indeed, the arms with the highest observed rewards must be selected often, while all arms must be explored to know their mean rewards. Equivalently, the performance of the agent (or policy) can be evaluated by its regret R T , defined as the expectation of the gap on T rounds (T ∈ N * is called the horizon) between the reward accumulated by the policy and that accumulated by a policy that always selects the best arm. Regret can be rewritten as

R T i∈[n] E[N i,T ]∆ i ,
where ∆ i max j µ * j -µ * i is the difference in expectation between the optimal arm and the arm i, and where N i,t is the number of times the arm i has been pulled until the round t ∈ N * . The notion of regret thus quantifies the loss due to the need to learn the mean rewards of the different arms.

The problem defined above is an example of a reinforcement learning problem. Indeed, a multi-armed bandit problem can be seen as a Markov decision process with a single state. Let's mention in passing that the name of the problem comes from imagining a player facing a row of slot machines, having to decide which machines to play, how many times to play each machine, and in what order to play them. Of course, this is only a model, and in practice this problem addresses any kind of situation where an agent simultaneously tries to acquire new knowledge (exploration) and to optimize its decisions on the basis of existing knowledge (exploitation). There are many practical applications of the bandit model, for example:

• Clinical trials that study the effects of different experimental treatments while minimizing the loss of human life [START_REF] Thompson | On the likelihood that one unknown probability exceeds another in view of the evidence of two samples[END_REF][START_REF] Gittins | Multi-armed Bandit Allocation Indices[END_REF][START_REF] Berry | Bandit Problems: Sequential Allocation of Experiments[END_REF].

• The design of financial portfolios [START_REF] Hoffman | Portfolio Allocation for Bayesian Optimization[END_REF].

• Decision-making for dynamic spectrum access in the context of cognitive radios [START_REF] Lai | Medium access in cognitive radio networks: A competitive multi-armed bandit framework[END_REF].

Multi-armed bandit has been studied by [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF]. They proved an asymptotic lower bound on the regret R T . This bound indicates that any "reasonable" policy (in a certain sense) must have at least a logarithmic regret with respect to the horizon T . It thus provides a fundamental performance limit that no "reasonable" policy can beat. The constant in the lower bound is linear in n, the number of arms. More precisely, it is worth the sum on the sub-optimal arms i of the difference ∆ i times the inverse of the Kullback-Leibler (KL) divergence [START_REF] Kullback | On information and sufficiency[END_REF] between the distribution of the arm i and that of the optimal arm. Roughly, log(T ) times the inverse of the KL represents the number of rounds needed to distinguish the arm i from the optimal one, giving a clear interpretation of the lower bound. In the case of Gaussian distributions, the inverse of the KL is lower bounded (up to a multiplicative constant) by the variance σ 2 i of the arm i divided by ∆ 2 i . To simplify, the lower bound thus becomes

Ω   log(T ) i∈[n], ∆ i >0 σ 2 i ∆ i   .
(1.3)

It is not surprising that the variance appears, as it can be seen as a measure of the uncertainty we have in our samples: the higher the variance, the more difficult the estimation, and therefore the higher the regret. [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] also developed policies for some reward distributions and showed that they asymptotically reach the lower bound. Indeed, these policies have an upper bound on their regret which is logarithmic in T , with a constant of the same order as in the lower bound.

In the case where the arms have distributions σ 2 i -sub-Gaussian, the strategy ucb (Upper Confidence Bound [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF]) asymptotically reaches the bound (1.3). It consists in choosing, in round t, the arm i t maximizing the upper confidence bound µ i,t-1 + 2 log(t)/N i,t-1 , where µ i,t-1 is the empirical average of the rewards of the arm i. This is a strategy based on the optimistic in face of uncertainty principle, because it chooses the arm that would be best if the means were worth their optimistic estimates. (graph) where each edge is an arm. The node in blue represents the origin, and the node in red represents the destination. The edges represented by bold lines form a path between the origin and the destination: it is therefore an action that the agent can decide to play during a round. Doing so, each arm of the path produces one outcome, and the reward can then be the sum of the outcomes of the path.

Stochastic combinatorial bandit

To come back to the subject of this thesis, i.e., sequential combinatorial optimization problems, we can extend the above-mentioned framework: the agent can now, during a round, choose several arms instead of one (with possibly some constraints on the possible sets of arms). The reward obtained depends on the individual outcome of each chosen arm. This is well adapted to applications where the actions available to the agent belong to a combinatorial space and are built from several small actions (the arms). Let us mention for example the problem of the shortest path between two points of a network (which has been studied in particular by [START_REF] Liu | Adaptive shortest-path routing under unknown and stochastically varying link states[END_REF] and [START_REF] Talebi | Stochastic Online Shortest Path Routing: The Value of Feedback[END_REF]). Each edge of the network is an arm associated to a probability distribution, modelling for example the traffic flow on this edge. The agent chooses at each round a path connecting the two points, and is rewarded with an independent realization of the overall traffic flow of the chosen path. This overall traffic flow is defined according to the context, and can be for example the sum of the path's edge flows (see Figure 1.3). This particular case where the reward is the sum of the outcomes of the chosen arms corresponds to a sequential combinatorial optimization problem with a linear objective function, and it is the most common scenario among these types of problems. It is studied, for example, by [START_REF] Abernethy | Competing in the dark: An efficient algorithm for bandit linear optimization[END_REF], [START_REF] Dani | The Price of Bandit Information for Online Optimization[END_REF][START_REF] Dani | The Price of Bandit Information for Online Optimization[END_REF]Bubeck, Cesa-Bianchi, and[START_REF] Bubeck | Towards minimax policies for online linear optimization with bandit feedback[END_REF]. In general, the combinatorial extension of the bandit problem is known in the literature as stochastic combinatorial bandit. It implies (at least) two distinct frameworks concerning the observations available to the agent on the outcomes of the arms at each round:

• Semi-bandit feedback: all the outcomes of the selected arms are observed by the agent.

• Bandit feedback: only the reward (which is a function of the outcomes of the chosen arms) is observed.

As we have already mentioned, we are interested here in the first framework, commonly called stochastic combinatorial semi-bandits (which we will abbreviate to CMAB in this thesis, for combinatorial multi-armed bandits, the "semi" being omitted for sake of brevity). Since the agent has more observations, this assumption may be more difficult to satisfy in practice, but may lead to better policies. Note also that the second framework is in fact a special case of the bandit problem introduced above, where the number of arms is combinatorial, but where a certain structure is specified between rewards.

Semi-bandit feedback

Here we formalize the CMAB problem. The outcome of the arm i ∈ [n] in round t is denoted as X i,t ∈ R. The random vectors (X t ) t≥1 of R n are i.i.d., with an unknown distribution P X , of mean µ * ∈ R n . Nevertheless, unless otherwise stated, X i and X j for two distinct arms i = j can be arbitrarily correlated. The action (also called super-arm) selected by the agent at round t (i.e., the set of arms selected) is denoted A t . The set of possible super-arms is called action space, and is noted A. It is a fixed subset of P([n]), such that each of its elements A is a subset of at most m arms. After selecting an arm A t in round t, the agent receives as feedback e At X t . Although we consider a more general framework in the thesis, in order to simplify the explanations, we limit ourselves here to the case of a linear reward function: the reward at round t is therefore e T At X t . We also assume that A is such that the optimization of linear functions can be done efficiently (in time polynomial in n).

The objective is to identify a policy that maximizes the expected cumulative reward over the T rounds. The expectation is taken on the randomness of the rewards and on the possible randomness of the policy followed by the agent (an action can be chosen at random). In an equivalent way, we aim at designing a policy π that minimizes the regret, defined as:

R T (π) max A∈A E T t=1 (e A -e At ) T X t .
Contrary to the classic bandit problem, there is a notion of similarity between actions. Indeed, two "close" actions, i.e., having many arms in common, tend to have a "close" reward. This structure can be exploited in the learning process, in order to counterbalance the combinatorial size of the action space. This results in the design of efficient super-arm selection policies, based on the principle of optimism in face of the uncertainty, whose regret is of the order of O(n • m log(T )/∆) (Kveton, Wen, Ashkan, and Szepesvari, 2015b), where ∆ > 0 is the minimum difference between the mean of an optimal super-arm and the mean of a sub-optimal super-arm. These policies asymptotically reach a lower bound: considering the action space A = {A 1 , . . . , A n/m } with A k = {(k -1)m + 1, . . . , km}, and posing for all k, X (k-1)m+1 = • • • = X km , drawn according to a Gaussian of variance 1, we reduce the problem to a classical bandit problem, allowing us to apply the bound (1.3), with σ 2 i = m 2 . The best-known policy of this kind is cucb (Combinatorial Upper Confidence Bound), which is heavily influenced by ucb. Indeed, it plays at round t the action

A t ∈ arg max A∈A e T A µ i,t-1 + 2 log(t) N i,t-1 i , Chapter 1. Introduction µ t-1 • Figure 1.4:
The hypercube type confidence region (in black) corresponds to a Gaussian prior for each marginal. However, the correlation between these marginals can be arbitrary: here are several examples, from very negatively correlated (in orange) to very positively correlated (in red), through independent (in green). We can see that the whole hypercube is likely to contain the true mean µ * as long as we do not specify (and learn) the correlations between the marginals.

in other words, the optimistic estimates are the same as for ucb. Note that in CMAB problems, the entire joint distribution of the random vector X matters, unlike in classical bandits where only the marginals are sufficient to characterize a problem instance. Thus, when estimating the vector of means µ * , we are more interested in a confidence region than just a collection of n confidence intervals. Another way to look at cucb is therefore to consider the confidence region C t associated with it: it is no more than the Cartesian product of the confidence intervals that were considered by ucb. Here is a way of rewriting cucb in terms of confidence region:

A t ∈ arg max A∈A max µ∈Ct e T A µ.
We thus have two view points in the optimistic methods: We can either look at the exploration bonus that is added to the empirical estimate (here, the bonus is linear and is worth e

T A 2 log(t) N i,t-1 i
, we also say the bonus is of type 1 ), or look at the confidence region around the vector of empirical means µ t-1 (here, the region is a hypercube, we also say the region is of type ∞ ). We can notice that the above optimization problem can be solved efficiently (it is a linear optimization problem on A): cucb is therefore statistically efficient (because it reaches the lower bound) and computationally efficient.

Note that the distribution P X chosen to establish the above lower bound is extreme: it represents a situation where the arms belonging to the same action are perfectly correlated. This extreme behavior is also highlighted when we look at the confidence region considered by cucb: we do not really expect to have a confidence zone having the shape of a hypercube, but rather having the shape of an ellipsoid, taking as prior a multivariate Gaussian distribution. The hypercube comes from the fact that the orientation and eccentricity of the ellipsoid can be arbitrary (see Figure 1.4). If, for example, we limit ourselves to the class of distributions that satisfy P X = ⊗ i∈[n] P X i (which means that the distributions of the arms are mutually independent), then the lower bound becomes Ω(n log(T )/∆) (because σ 2 i = m in this case). This means that cucb is no longer statistically efficient, and that we can potentially gain a factor m in the regret upper bound. [START_REF] Combes | Combinatorial Bandits Revisited[END_REF] have studied this problem, and have come up with a policy that indeed relies on the independence of the outcomes to reduce the confidence region used. Specifically, they proposed a policy, called escb (for Efficient Sampling for Combinatorial Bandit), which plays at round t the action

A t ∈ arg max A∈A e T A µ t-1 + i∈A f (t) N i,t-1 ,
where f (t) is of order O(log(t)). This time we notice that we're dealing with an 2 bonus, corresponding to the following 2 confidence region (here, a confidence ellipsoid):

C t µ t-1 +    ξ ∈ R n , i∈[n] N i,t-1 ξ 2 i ≤ f (t)    .
Note that the above ellipsoid is aligned with respect to the axes (which is consistent with the independence assumption). Degenne and Perchet (2016b) went further by considering the class of distributions P X which are C-sub-Gaussian (multivariate), for a matrix C 0, i.e., such that

∀λ ∈ R n , E e λ T (X-µ * ) ≤ e λ T Cλ/2 .
In this case, they provided a policy, ols-ucb, requiring knowledge of another matrix Γ 0 with non-negative coefficients such that Γ + C. This policy considers a confidence ellipsoid based on the matrix Γ (i.e., not aligned with respect to the axes when Γ is not diagonal). It essentially boils down to the policy escb in the specific case where the matrix Γ is diagonal. ols-ucb has a regret bound of order

O   log T ∆ i∈[n] Γ ii (1 -γ) log 2 (m) + γm   ,
where γ max

A∈A max (i,j)∈A 2 ,i =j Γ ij / Γ ii Γ jj .
So we can see that escb, in the case of mutually independent distributions, is statistically efficient (hence its name). Indeed, we then have γ = 0 in the above bound, giving a regret for escb of the order of O(log 2 (m)n log(T )/∆), which is tight up to a polylogarithmic factor in m. Degenne and Perchet (2016b) also showed that their policy was statistically efficient for the class of multivariate sub-Gaussian distributions under consideration. They proved a lower bound on the order of Ω n log T ∆ ((1 -γ) + γm) . In particular, an interpolation between two regimes appears: for γ close to 0, we find the same lower bound as in the case where the arms have mutually independent distributions, and for γ close to 1, we find the lower bound corresponding to the case where the arms are perfectly correlated.

We can finally notice that the policies escb and ols-ucb are not computationally efficient in general (because the combinatorial problem to be solved at each round is no longer linear, and is even NP-hard in general [START_REF] Atamtürk | Maximizing a class of utility functions over the vertices of a polytope[END_REF]).

A brief look at non-linear rewards and budget constraints

Generally, it is assumed that the expected reward when the agent chooses an action A is in the form r(A; µ * ). If this is the case, the function r is called reward function. It is linear when r(A; µ * ) = e T A µ * . It is assumed here, for simplicity, that for a fixed vector µ, the function A → r(A; µ) can be maximized exactly and efficiently on A. We have already mentioned that this thesis did not focus solely on linear reward functions. Nevertheless, most of the reward functions we will encounter behave similarly to linear rewards. To be more precise, we will see that there is frequently an 1 -norm control on the deviation of the reward when the parameter we want to learn (here the mean) varies. For example, a common assumption considered by [START_REF] Wang | Improving regret bounds for combinatorial semi-bandits with probabilistically triggered arms and its applications[END_REF] is

r(A; µ) -r(A; µ ) ≤ e A (µ -µ ) 1 .
(1.4) Under this assumption, the optimistic approaches above remain valid (i.e. the bounds on the regret are of the same order) by considering, in policies, the optimization problem max

A∈A, µ∈Ct r(A; µ)
in order to select A t . One difficulty with non-linear rewards in optimistic approaches is that the quantity max µ∈Ct r(A; µ) may not be simple to compute (and even less to optimize over A ∈ A). Nevertheless, there are cases where this is possible. For example, if we have a monotonicity property (component-wise) of the function µ → r(A; µ), then, for a confidence region of type ∞ , we can directly plug the vector of optimistic estimates instead of the true mean in order to select the action A t [START_REF] Chen | Combinatorial Multi-Armed Bandit: General Framework and Applications[END_REF]Chen, Wang, and Yuan, 2016), the result is that this policy is computationally efficient. A particularly interesting case of a non-linear reward function arises when we consider a budgeted combinatorial bandit problem [START_REF] Xia | Budgeted multi-armed bandits with multiple plays[END_REF]. To briefly explain this framework, there is no longer a time horizon T , but rather a budget B, and every action that is taken in a round is costly. We will see that this framework can be essentially solved in the same way as the standard problem. A notable difference is that the objective function to be optimized within each round is no longer a reward function, but a ratio between a reward function and a cost function. This type of problem is therefore at the origin of many non-linear objective functions in CMAB problems, even if the reward and cost functions were initially linear.

Probabilistically triggered arms

In the typical CMAB problem, the set of triggered arms and the action played by the agent coincide. Specifically, at each round, the agent selects an action to play, which triggers a set of arms, and the outcomes of these arms are then observed. An interesting generalization, called probabilistically triggered arms (which we abbreviate to CMAB-T in this thesis), was introduced by Chen, Wang, and Yuan (2016) and [START_REF] Wang | Improving regret bounds for combinatorial semi-bandits with probabilistically triggered arms and its applications[END_REF]. The idea is that the action selected by the agent is no longer reduced to a set of observed arms, but belongs to an external space. After an action is selected, some arms are triggered probabilistically. In particular, this framework has been used to address problems such as Cascading bandits (about the selection of web pages to be displayed for a search engine query [START_REF] Kveton | Cascading Bandits: Learning to Rank in the Cascade Model[END_REF][START_REF] Kveton | Combinatorial Cascading Bandits[END_REF]) or online influence maximization (about selecting influencers in a social network (Chen, Wang, and Yuan, 2016;[START_REF] Wen | Online influence maximization under independent cascade model with semi-bandit feedback[END_REF][START_REF] Wang | Improving regret bounds for combinatorial semi-bandits with probabilistically triggered arms and its applications[END_REF]). A very practical assumption so that the above regret bound still (for ∞ type regions) holds is to consider the same smoothness relation as (1.4), but by weighting each term (corresponding to an arm) in the norm by the probability of observing the corresponding arm when choosing the action. This assumption is verified in particular in the problems of cascading bandit and online influence maximization [START_REF] Wang | Improving regret bounds for combinatorial semi-bandits with probabilistically triggered arms and its applications[END_REF].

Challenges of the thesis and contributions

After the above overview, several questions can be asked, and trying to answer them constitutes the objectives of this thesis. In general, the focus will be on improving the efficiency of policies, both from a computational and statistical point of view. Specifically, we will be particularly interested in the following questions:

• Can we hope to implement an efficient version of the escb and ols-ucb policies?

Are there alternatives that are computationally as well as statistically efficient (in the case of independent, or multivariate sub-Gaussian distributions)?

• Is it possible to associate the 2 analysis with the budgeted or the probabilistically triggered arms frameworks? What kind of bound on the regret can we get? Can such an analysis help improve existing policies (and the regret they have) for known problems, such as online influence maximization?

• Can we optimize the analysis from Degenne and Perchet (2016b)? For example, can we relax the assumption that the sub-Gaussianity matrix Γ is known? Is there an alternative to the family of multivariate sub-Gaussian random variables? Is it possible to explicit the interpolation behavior in the regret, for example by replacing Γ ii (1 -γ + γm) by the smaller quantity max A∈A: i∈A j∈A Γ ij ?

We will now present the contents of the thesis, chapter by chapter, focusing on aspects related to the issues raised above.

Chapter 2, Multi-Armed Bandits We first review the basic results of the classical multi-armed bandit problem. Some of them will be useful later in the combinatorial context.

Chapter 3, General Framework for Semi-Bandit Feedback We formalize here the framework of combinatorial stochastic semi-bandits with probabilistically triggered arms (which, we recalled, encompasses the usual framework). We then give a multitude of applications of this framework that can be found in the literature. Finally, we provide general technical results that will be useful throughout the thesis to prove upper bounds on policy regrets. More precisely, the theorems stated depend on the type of error we want to control (which is nothing else than the type of bonus we use). For the ∞ analysis (i.e., based on an ∞ type confidence region, or in an equivalent way on an 1 type bonus), we have a theorem that slightly improves (and also simplifies) the ∞ analysis of [START_REF] Wang | Improving regret bounds for combinatorial semi-bandits with probabilistically triggered arms and its applications[END_REF]. As for the 2 analysis, we provide several new results, extending the work of Degenne and Perchet (2016b), and surpassing the ∞ analysis (gaining a factor m, up to a polylogarithmic factor).

In doing so, we manage to associate the 2 analysis with the probabilistically triggered arms framework. This chapter contains some unpublished results, and some improvements of results published in:

• Pierre Perrault, Jennifer Healey, Zheng Wen, Michal Valko (2020a) "Budgeted Online Influence Maximization", in 37th International Conference on Machine Learning (ICML).

• Pierre Perrault, Vianney Perchet, Michal Valko (2020) "Covariance-adapting algorithm for semi-bandits with application to sparse rewards", in 33rd Conference on Learning Theory (COLT). Here we present a new example of a problem that falls within both the probabilistically triggered arms and the budget frameworks. We call this problem sequential searchand-stop. It is described as follows: we consider a directed acyclic graph, where each node is an arm associated with a cost. There is an object that is randomly hidden in one of the nodes (according to an unknown distribution to be learned). In a round, the agent can inspect nodes, chosen one by one, with the constraint that a node is accessible if its in-neighbors have already been inspected. It can decide at any time to stop its search, and thus move to a new independent instance (where the object is randomly placed again). The agent has an initial budget, and each time a node is inspected, its price is paid. The objective of the agent is to find a maximum number of objects (in expectation). We provide a policy based on an ∞ approach, describing in particular a quasi-optimal and efficient strategy for the "offline" problem (i.e., when the distribution is known to the agent, who must then focus on exploitation). This chapter is adapted from the following publication:

• Pierre Perrault, Vianney Perchet, Michal Valko (2019b) "Finding the bandit in a graph: Sequential search-and-stop", in 22nd International Conference on Artificial Intelligence and Statistics (AIStats).

Chapter 5, The Structure of Uncertainty We begin by recalling the two main methods for obtaining 2 confidence regions: the Laplace method, and the covering argument. We then look more deeply into the structure of the 2 bonuses, and discover that as functions on P([n]), they are submodular. We then use this structure to propose several approximation algorithms to maximize the sum of a linear function (A → e T A µ t-1 ) and a submodular function (the exploration bonus). This allows us to obtain approximations of escb for matroid constraints (a certain type of action space A generalizing the constraint on the cardinality). These approximations are computationally efficient and do not degrade the regret rate. We also provide an extension of this method to the budgeted framework. This chapter is based on the following publication:

• Pierre Perrault, Vianney Perchet, Michal Valko (2019a) "Exploiting structure of uncertainty for efficient matroid semi-bandits", in 36th International Conference on Machine Learning (ICML).

Chapter 6, Budgeted Online Influence Maximization The purpose of this chapter is to further investigate another well-known example of a problem involving probabilistically triggered arms, namely online influence maximization. As a reminder, in this problem, an agent actively learn a social network by interacting with it repeatedly, trying to find the best sets of influencers. We incorporate a budgeted framework to this problem, considering the total cost of an advertising campaign instead of the (per round) cardinality constraint. Our approach thus better models the real-world context where the cost of influencers varies and where advertisers want to find the best value for their overall budget. We propose ∞ and 2 policies, using the submodularity of an upper bound on 2 bonuses. This chapter is adapted from the following publication and prepublication:

• Pierre Perrault, Jennifer Healey, Zheng Wen, Michal Valko (2020a) "Budgeted Online Influence Maximization", in 37th International Conference on Machine Learning (ICML).

• Pierre Perrault, Jennifer Healey, Zheng Wen, Michal Valko (2020b) "On the Approximation Relationship between Optimizing Ratio of Sub-modular (RS) and Difference of Submodular (DS) Functions", submitted.

Chapter 7, Covariance-Adapting Policy The goal here is to offer a more practical, and more accurate solution to build an 2 confidence region. Indeed, the ones we have seen so far are either based on mutual independence of outcomes, or due to the unrealistic knowledge of a sub-Gaussianity matrix Γ. We consider a new general family of sub-exponential distributions parameterized by the covariance matrix, assumed to be unknown. This family contains the bounded support distributions and the Gaussian distributions. We prove a new lower bound on the regret of this family, which is parameterized by the covariance matrix, instead of the sub-Gaussianity matrix. We then construct an algorithm that uses covariance estimates, and provide an asymptotic analysis of the regret, reaching the lower bound. Finally, we apply and extend our results to the family of sparse outcomes, which has applications in many recommender systems. This chapter is adapted from the following publication:

• Pierre Perrault, Vianney Perchet, Michal Valko (2020) "Covariance-adapting algorithm for semi-bandits with application to sparse rewards", in 33rd Conference on Learning Theory (COLT).

Chapter 8, Statistical and Computational Efficiency of Thompson Sampling

In this chapter, we consider an interesting alternative to the optimistic methods considered so far, namely Thompson sampling. In addition to its empirical superiority [START_REF] Chapelle | An empirical evaluation of Thompson sampling[END_REF], Thompson sampling is interesting in the context of combinatorial bandits because the action to be played is easy to compute: one should consider neither bonus nor maximum on a confidence region, because the parameter to be used in the objective function is simply taken at random, according to a well-chosen prior. Thompson sampling could thus answer the question of the existence of a (computationally) efficient policy with optimal asymptotic regret (up to a polylogarithmic factor in m), which is still open for many families of distributions, including mutually independent outcomes, and more generally the family of multivariate sub-Gaussian outcomes. We propose to answer the above question for these two families by analyzing variants of the Combinatorial Thompson Sampling policy (cts). For mutually independent outcomes in [0, 1], we propose a tight bound for the regret of cts, using a beta prior. We then examine the more general framework of multivariate sub-Gaussian outcomes and propose a tight bound for the regret of cts using a Gaussian prior. This latter result gives us an efficient alternative to the escb policy. This chapter is adapted from the following publication:

Multi-Armed Bandits

This chapter is an introduction to the theory of stochastic multi-armed bandits (MAB), that is the simplest setting of reinforcement learning [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. It is organized as follows: we first formally introduce the MAB problem. Next, we present different examples of motivation. Then we provide the basic theoretical results and introduce some MAB policies by giving guarantees on their performance. At the end of this chapter, we examine possible extensions of the classical MAB problem that are of particular interest in this thesis.

The stochastic multi-armed bandits problem

We consider a statistical model with incomplete observations. An automated agent partially observes a certain random phenomenon, described by a random vector X ∈ R n , for a fixed integer n ∈ N * . We assume for the moment that the agent has no prior information on the distribution P X , except that the mean µ * E[X] exists but is unknown. As usual in statistics, the phenomenon is accessible through a random process (X t )

iid ∼ P X . The observations are incomplete in the sense that the vector X t is never observed entirely by the agent at any round t, but only one of its components, X it,t , where the index i t is selected by the agent at the beginning of round t. The quantity X it,t also represents a gain, or reward for the agent.

The objective is not directly to correctly estimate the distribution P X (so it is not really a statistical problem in the traditional sense), but rather to maximize the expected cumulative reward

E T t=1 X it,t
over the sequence of index choices (i t ), for a fixed unknown time horizon T . This is a learning problem called stochastic multi-armed bandits (MAB), which was first introduced in [START_REF] Robbins | Some aspects of the sequential design of experiments[END_REF]. It takes its name from a casino slot machine. Indeed, it is worth remembering that a synonym for "slot machine" is "one-armed bandit": a slot machine has only one arm. Another view on the MAB problem is the following: we consider that the agent is facing n machines, or, more simply, an unusual machine, with n arms (numbered from 1 to n), which is why we talk about multi-armed bandits. Each of the n arms i, as soon as it is pulled, draws a reward from P X i at random. Distributions P X 1 , . . . , P Xn are unknown and may be different from each other. In particular, some arm might be better for the agent, that is, they might return a higher reward. The machine is without memory: this ensures that the payment vectors X t are effectively iid when t varies. Imagine now that the agent wants to play T times on the casino slot machines. The MAB problem raises the following natural question for the agent: what strategy to adopt for these T draws in order to maximize the expected total gain? Chapter 2. Multi-Armed Bandits

We will see that a smart allocation strategy for the agent has to trade-off between two behaviors:

• Exploring to collect information on the distribution P X , and in particular, to estimate µ * effectively.

• Exploiting the arm that seems to be better given the rewards already observed.

Unlike the usual statistical problems, the agent is not interested in the proper description of the underlying stochastic phenomenon, but rather in the consequences of this good modeling in terms of reward.

Remark 1. Although we use the notation P X , we must keep in mind that an MAB problem is defined only by the marginals P X 1 , . . . , P Xn . Indeed, we saw above tow ways of simulating the sequence (i 1 , X i 1 ,1 , i 2 , X i 2 ,2 , . . . ): at each round, the first uses P X , whereas the second only uses a single marginal. As a consequence, for any distribution P X , the two MAB instances defined by reward distributions P X and P X 1 ⊗ • • • ⊗ P Xn are the same. In particular, we can assume that P X = P X 1 ⊗ • • • ⊗ P Xn in this chapter. We'll see that this is no longer the case in stochastic combinatorial semibandits.

The feedback

In our description of the MAB problem, the agent observes only the reward for the selected arm i t , and nothing else. In particular, it does not observe rewards for other actions that could have been selected. This type of feedback is called bandit feedback. Another well-known type of feedback is called full information, where the whole vector X t is observed by the agent at every round t (the observation is thus no longer partial), allowing it to focus on exploitation only. In this chapter, we will always assume that the agent has bandit feedback, that is a more realistic scenario (yet more challenging).

Mathematical definition of a policy

We mentioned the term "strategy" above, without really defining it formally. First of all, we will rather use the term policy, to emphasize that it gives the agent's way of behaving at a given time. We already have a pretty good intuition about what a policy is allowed to do, namely that the arm i t is chosen in the round t based only on the information available at the beginning of the round, which is formed by past actions and gains. The arm i t can also be chosen using an extra source of randomness. As a result, the agent chooses i t ∈ [n] according to a probability distribution P it built with the past information. By universality of the standard uniform distribution, we can model the extra source of randomness needed to sample from P it by the random variable U t , where (U t )

iid ∼ U (0, 1), independently from (X t ). Here is the same definition of a policy, but formulated in a rigorous mathematical way.

Definition 1 (Policy).

A policy π is a sequence of random variables π = (i t ) ∈ [n] N * such that for all t ≥ 1, i t is F t -measurable, where (F t ) is the filtration defined by

F t = σ(U 1 ) if t = 1 σ(U 1 , X i 1 ,1 , . . . , U t-1 , X i t-1 ,t-1 , U t ) if t ≥ 2.
When U t is actually used in the choice of i t , the policy is referred to as randomized.

Remark 2. In addition to being measurable with respect to F t , the choice i t can also depend on n and t. However, since T is unknown, i t can't depend on it.

A performance metric: the regret

The expected cumulative reward provides only an absolute assessment of the agent's total gain, but does not directly measure the relative deviation of this gain from the optimal total gain. From Definition 1, we can rewrite the expected cumulative reward of a policy π = (i t ) as

E T t=1 X it,t = E T t=1 E[ X it,t |F t ] = E T t=1 µ * it .
We see that the optimal policy for the MAB problem is π *

(i * , i * , . . . ), where i * ∈ arg max i∈[n] µ * i .
In other words, when the distribution P X is known, the best choice is to pull at each round the arm with the largest mean. But since the agent first has to explore the environment to get information about the reward distribution of each arm, unavoidably some sub-optimal arms will be pulled. Thus, to evaluate the performance of a given policy π, we consider the regret of the agent for not playing optimally:

Definition 2 (Expected cumulative regret). The expected cumulative regret (that we shall simply call regret) for a policy π = (i t ) is

R T (π) T µ * i * -E T t=1 X it,t = E T t=1 ∆ it ,
where for any i ∈ [n], we define the gap between means of arm i * and arm i as

∆ i µ * i * -µ * i .
The goal for the agent is to play according to a policy π such that R T (π) is as small as possible.

Remark 3. The regret (for a policy π = (i t )) can be rewritten as π

= (i t ) is R T (π) = i∈[n] ∆ i E[N i,T ],
where for any round t ∈ [T ], and any arm i ∈ [n], N i,t t t =1 I{i t = i} is the counter of the number of times arm i was chosen by π during the t first rounds. This expression follows from Wald's identity.

A simple example: the Bernoulli bandit problem

As we will see in the next section, perhaps the simplest and most widely used reward distribution is the Bernoulli distribution, when the reward of each arm is either 0 or 1 ("failure or success", "heads or tails", etc.). This reward distribution is fully specified by the mean reward µ * . We present in Figure 2.1 a very simple example of MAB problem with n = 4 arms and T = 50, with a naive policy π rand that selects the arm i t uniformly at random.

t 0 5 t t =1 ∆ i t t t =1 (µ * 2 -X i t ,t )
Figure 2.1: A Bernoulli bandit toy example. At the top, the points represent the choices i t . When they are "full", the corresponding reward is 1, and when they are "empty", it is 0. At the bottom, we have plotted two types of cumulative regret curves that are both worth R t (π rand ) in expectation.

Some real world applications

This section presents some of the applications found in the MAB literature. This list (far from exhaustive) is intended to illustrate the great diversity of applications of the MAB framework and to motivate its study in this thesis.

Clinical trials

MAB models are historically guided by clinical trials [START_REF] Thompson | On the likelihood that one unknown probability exceeds another in view of the evidence of two samples[END_REF]. Optimal clinical trial design is indeed a typical motivating application [START_REF] Villar | Multi-armed bandit models for the optimal design of clinical trials: benefits and challenges[END_REF][START_REF] Aziz | On Multi-Armed Bandit Designs for Phase I Clinical Trials[END_REF], although very little of the resulting theory has actually been used in clinical trial design and analysis. On the contrary, MAB models are now widely studied with completely different applications in mind (as we will see in the next subsections). In a clinical trial, a set of n different drugs are tested to treat a disease. The most typical application is the choice between an old and a new drug (i.e., n = 2). It must be determined as soon as possible whether the new drug should be adopted or the old one maintained. Any error would result in lost human lives (or, at least in people suffering from disorders resulting from either incomplete treatment or excessive side effects). During the test phase, patients come sequentially to be administered one drug. It is assumed that the success of a drug on a patient is a random variable whose distribution depends on the drug in question. This variable can, for instance, be drawn from a Bernoulli distribution, and be worth 1 if the patient survives and 0 otherwise. The objective is then to maximize the total number of patients cured, or more precisely to achieve results that are almost as good as the best drug among the two. In particular, it is important to avoid using the sub-optimal drug too often, but rather to focus as soon as possible on the best one. Thus, a trade-off appears between collecting information about the two drugs to estimate their performance and using the information obtained so far to be efficient.

Web document ranking

MABs can also be used to rank web documents [START_REF] Radlinski | Learning diverse rankings with multi-armed bandits[END_REF] and more specifically web pages. The objective is to improve the performance of a Web search engine by increasing the probability that users will click on the highestranked results. The problem can be formalized as follows. We consider a fixed set of n documents and a fixed search query. Suppose we have a population of users, where each user has a personal set of documents considered as relevant with regard to the query, as well as a tolerance threshold giving the maximum number of documents the user is willing to check. Intuitively, users with different interpretations of the query have different relevant sets. At any round t, a (random) user comes hoping to find a document related to the query. An ordered set of m documents (among the n) is then proposed to the user. Results are consulted in order, and the goal is to avoid as much as possible the abandonment phenomenon, where no relevant document is presented before the user has reached its tolerance on the number of documents seen. The reward gained at round t is 1 if the user clicks on some document proposed, and 0 otherwise.

A first approach would be to consider each possible m-ordered set as an arm. However, [START_REF] Radlinski | Learning diverse rankings with multi-armed bandits[END_REF] rather proposed a more efficient method that runs m MAB instances in parallel. Each instance i ∈ [m] has n arms and is in charge of choosing the document presented at rank i. If an MAB instance selects a document that has already been selected, the choice is saved, but another arbitrary unselected document is proposed instead. When the user clicks on document i, all the MAB instances that had selected the arm i as their first choice receive a reward of 1, and the others receive a reward of 0.

We will see in subsection 3.3.1 that this problem can be formulated as a stochastic combinatorial semi-bandit called "Cascading bandits".

Algorithm selection and black-box stochastic optimization

MABs have been used for algorithm selection in [START_REF] Gagliolo | Algorithm selection as a bandit problem with unbounded losses[END_REF]. We give a description of the setting in the following. Let's consider a problem for which one can easily evaluate the quality of an algorithm to solve it. For example, the quality can be assessed by the processing time to deliver a solution, or by the value that an objective function takes at that solution. Suppose we have n algorithms (which may differ only by an extra parameter to tune), candidates to solve the problem, and that algorithms have some internal stochasticity, or that they can only be evaluated with additional noise (this is often the case for the running time). We can consider this setting as an MAB problem, where arms are the n algorithms, and rewards are (for instance) minus the time needed for the selected algorithm to run on the problem. MAB policies thus translate into strategies to automatically select the best algorithm among the n.

Web-based recommendation of products (films, music, articles, etc.) is an example where this method might be used. Indeed, in order to design a recommender system on their platform, companies collect enough data on their customers' preferences, as well as contextual metadata, and use collaborative or content-based filtering methods to deliver a recommendation. Now, if n recommender systems are designed in different ways, a company might want to know which one is the best to use, which is nothing more than an algorithm selection problem.

Online advertising

MABs represent a natural framework for expressing problems related to advertisement placement on a web page. This type of application has been widely and intensively studied over the last decade [START_REF] Pandey | Handling Advertisements of Unknown Quality in Search Advertising[END_REF][START_REF] Babaioff | Characterizing truthful multi-armed bandit mechanisms[END_REF][START_REF] Li | A contextual-bandit approach to personalized news article recommendation[END_REF][START_REF] Li | Collaborative Filtering Bandits[END_REF], stimulated by e-commerce and the need for many companies to provide personalized content. The problem can be described as follows. Given a fixed search query, there are n advertisers who wish to advertise on the search engine result for this query. The search engine wants to know which ad is the best to display, i.e., which one generates the most clicks from users. Specifically, when a user arrives hoping to find a web page related to the query, the search engine may display one of the n ads, hoping that the user will click on it. The objective in selecting ads is to maximize the total income of the search engine (each click on an advertisement proposed by the search engine yields a certain income).

It is sometimes simpler to break down the rounds not according to users (because the search engine cannot necessarily afford to change the advertising decision to each user, mainly for computing time reasons), but according to time (e.g., a round equals a day). Thus, for a given day t, the engine will choose the ad it will display throughout the day. A standard measure of the displayed ad performance at the end of the day is the click-through rate (CTR), defined as the ratio between the number of clicks and the number of users who saw the ad.

Cognitive radios

Opportunistic access to spectrum is the most common application context for cognitive radios. In telecommunications, certain frequency bands are reserved for certain uses, such as mobile communications, television or military communications. These uses are called primary uses and require a license. Although most of the spectrum that can be used for communications today is already allocated, it is left free most of the time by the primary users. It is to compensate for this under-use of spectrum that opportunistic access to spectrum has been considered. It allows secondary users to use licensed bands left free by primary users. Typically, a secondary user has a set of n frequency channels in which to transmit. Each of these channels is more or less used by primary users. The secondary user must then use the most vacant channel for its communications. At each round t, the secondary user chooses to transmit through one of the n channels, and receives a reward of 1 if the selected transmission channel is vacant (not used by a primary user, the transmission is effective), and 0 if that channel is busy (no communication is possible through the selected channel). This setting can be seen as an MAB problem when the use of a channel by primary users is uncertain, and the reward received can, therefore, be modeled as a Bernoulli random variable.

The applications of MABs to cognitive radios have been studied in a more general context of multiplayer bandits, where several secondary users want to use the telecommunication network simultaneously [START_REF] Rosenski | Multi-player banditsa musical chairs approach[END_REF][START_REF] Besson | Multi-player bandits revisited[END_REF][START_REF] Lugosi | Multiplayer bandits without observing collision information[END_REF][START_REF] Boursier | SIC-MMAB: synchronisation involves communication in multiplayer multi-armed bandits[END_REF]. Opportunistic access to spectrum is a major issue for wireless technologies, in particular for 5G technology [START_REF] Wang | Intelligent Cognitive Radio in 5G: AI-Based Hierarchical Cognitive Cellular Networks[END_REF].

Algorithm 1 A MCTS algorithm

Initially, the tree has a single node (the root r, the current state of the game).

The following steps are repeated T times, and ultimately, the most visited child of r is returned. Selection: From r, use successively the MAB algorithm assigned to the current node to select a next child node until a leaf is reached. Expansion: Unless is terminal, create a child node c in the tree that is any valid move from the game position defined by . Simulation: Complete a random play-out from c by choosing uniform random moves until the game is decided. Back-propagation: Use the result of the play-out as a reward for each MAB algorithm on the path from c to r.

Monte-Carlo tree search

Monte Carlo tree search (MCTS) is a heuristic search algorithm used in some kind of decision processes. It is notably used in games (see [START_REF] Browne | A survey of Monte Carlo tree search methods[END_REF] and the references therein for different variants of MCTS and applications to games and other search, optimization, and control problems). We can mention its implementation in recent computer programs of Go, such as Crazy-Stone [START_REF] Coulom | Efficient selectivity and backup operators in Monte-Carlo tree search[END_REF], MoGo [START_REF] Gelly | Modification of UCT with patterns in Monte-Carlo Go[END_REF][START_REF] Wang | Modifications of UCT and sequence-like simulations for Monte-Carlo Go[END_REF], AlphaGo [START_REF] Silver | Mastering the game of Go with deep neural networks and tree search[END_REF], and AlphaGoZero [START_REF] Silver | Mastering the game of go without human knowledge[END_REF]. Given the current state of the game, the goal is to determine the next action to be selected by the player. MCTS algorithms explore the tree of possibilities in a non-symmetric way. The root is the current configuration of the game, and each node is a configuration and its children are the following possible configurations. The MAB setting has been used to guide exploration in the tree structure, allowing to perform efficient tree search by assigning an MAB algorithm to each node and following an optimistic search strategy that explores in priority the most promising branches (see Algorithm 1). The intuition of Algorithm 1 is that at a given node there are some possible choices, i.e., arms, corresponding to the child nodes, and the use of an MAB algorithm should enable the selection of the best arm given noisy rewards samples. The reason why this algorithm has shown good performance in practice in several large tree search problems is that it prioritizes the most promising branches according to previously observed sample rewards. This is very useful in situations where the reward function has some smoothness property (i.e., the initial random reward samples provide information on where the search should focus). Theoretically, however, since the reward samples obtained from any node are not iid, the algorithm does not have nice finite-time performance guarantees (see [START_REF] Coquelin | Bandit algorithms for tree search[END_REF]).

One of the best known algorithms extends the Upper Confidence Bound algorithm (ucb) of [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF] (which we will explore in more detail in section 2.4) to tree-structured search, defining the Upper Confidence bound applied to Trees algorithm (uct) [START_REF] Kocsis | Bandit-based Monte-Carlo planning[END_REF]. [START_REF] Madhawa | A multi-armed bandit approach for exploring partially observed networks[END_REF] studied the problem of exploring a large unweighted and undirected graph G = (V , E). This has applications in collecting information about user profiles and their friends in a social network G. In their setting, G cannot be fully observed, and only a subgraph G t = (V t , E t ) is available to the agent at time t. Initially, G 0 is only composed of a single node. At each round t, the agent selects a node i t ∈ V t to probe in order to expand G t by adding the neighbors of i t . More precisely, when i t is selected, G t+1 is obtained from G t as follows.

Network exploration

V t+1 = V t ∪ N it , where N it {j ∈ V , {i t , j} ∈ E} is the neighbourhood of i t in G
, and E t+1 = E t ∪ {{i t , j}, j ∈ N it }. An example of the graph G t is given in Figure 2.2. The goal for the agent is to discover as many nodes as possible within a limited time budget T , i.e., |V T | has to be maximized.

As for MCTS, MAB algorithms might be useful heuristics, although the problem watched is not strictly speaking an MAB problem. In order to do so, [START_REF] Madhawa | A multi-armed bandit approach for exploring partially observed networks[END_REF] considered that each node i is associated with a certain set of features F i (e.g., interests, liked pages, etc.), and that the neighborhood size of i (which is a reward for the agent if it probes i) could be represented as a sample according to an unknown distribution P F i depending on F i . In other words, nodes with similar features in the observed network will result in similar rewards. The trade-off between exploration and exploitation thus comes naturally. Indeed, for exploitation, the agent can consider choosing i t = i based on the a priori quality of F i , which can be estimated by looking at the past performance of nodes j, with F j close to F i . For exploration, the criterion to be taken into account is the amount of information brought by a new reward sampled from P F i , or equivalently how much information the agent has already collected on features close to F i .

Regret lower bounds

Before giving examples of MAB policies, it is important to know to what extent the agent can hope to minimize the expected cumulative regret. Indeed, we already saw that the agent, not knowing which arm is the best, must inevitably suffer some regret. A lower bound on the regret would, therefore, provide information on what the agent can aim for. However if one considers, as a performance measure, R T (π) for a single reward distribution P X without restricting the class of admissible policies π, then it is clear that we can achieve R T (π) = 0. Indeed, policies that always chose the same arm are admissible, and one of them has a 0 regret (it is obvious that these policies are of no interest because they do not learn, and although they are very good on a specific problem, they are very bad on the others). Thus, the regret must be considered either in more than one instance P X at the same time (Definition 3), or by limiting the class of policies allowed (Definition 4). This gives two kinds of regret lower bound.

Definition 3 (Distribution-free lower bound). A distribution-free lower bound is a lower bound on inf

π sup P X ∈D R T (π),
where D is a family of reward distributions.

Definition 4 (Distribution-dependent lower bound). A distribution-dependent lower bound is a lower bound on R T (π) that holds for any distribution P X ∈ D and any policy π ∈ Π, where Π is a family of admissible policies and D is a family of reward distributions.

For distribution-free lower bounds, the performance is measured considering the worst possible reward distribution P X (specific to the policy and the horizon T ). This kind of lower bound doesn't depend on P X . On the other hand, distributiondependent lower bounds hold for a fixed distribution P X and depend on it. Only a restricted class of policies Π is considered in order to avoid the consideration of policies of little interest that gives a small regret for the distribution P X , but perform badly on other instances.

We will see in the following that these two measures may have very different behaviors in T . [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] proved a lower bound of the expected cumulative regret of order log(T ) in a particular parametric framework. This work has then been extended by [START_REF] Agrawal | Sample mean based index policies by o (log n) regret for the multi-armed bandit problem[END_REF] and [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF]. These papers deal with the class of consistent policies, defined as follows.

Distribution-dependent lower bound

Definition 5 (Consistent policies). A consistent policy π (with respect to D = D 1 ⊗ . . . ⊗ D n ) is such that for all a > 0 and all reward distribution P X ∈ D, the expected cumulative regret satisfies

R T (π) = T →∞ o(T a ).
Consistent policies offer good performance for all arm distribution in D. The authors were interested in the price to be paid for a policy to be consistent. In other words, they ask the question What is the minimum regret that a consistent policy must suffer? The logarithmic bound of [START_REF] Agrawal | Sample mean based index policies by o (log n) regret for the multi-armed bandit problem[END_REF] and [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF] answers this question. It uses the Kullback-Leibler divergence (KL) [START_REF] Kullback | On information and sufficiency[END_REF], defined in Definition 6.

Definition 6 (Kullback-Leibler divergence (KL)). The Kullback-Leibler divergence between two probability distributions P and Q is defined as

KL(P Q) log dP dQ dP , if P is absolutely continuous with respect to Q +∞ otherwise,
where dP dQ is the Radon-Nikodym derivative of P with respect to Q. The KL is a measure of how one probability distribution is different from a second one. Intuitively, it measures the rate (in terms of the number of samples) at which the two distributions can be distinguished based on the samples from the first distribution. The smaller the KL divergence, the more difficult it is to distinguish the two distributions. The KL divergence is non-negative, and the value zero is reached if and only if the distributions are identical. it is generally asymmetric in the two distributions.

We are now ready to state the lower bound of [START_REF] Agrawal | Sample mean based index policies by o (log n) regret for the multi-armed bandit problem[END_REF] and [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF], in Theorem 1.

Theorem 1 [START_REF] Agrawal | Sample mean based index policies by o (log n) regret for the multi-armed bandit problem[END_REF] and [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF]). Let π be a consistent policy with respect to a distribution family D = D 1 ⊗ . . . ⊗ D n . Then we have, for all P X ∈ D,

R T (π) ≥ i∈[n], ∆ i >0 log(T )(1 -o(1))∆ i inf P ∈D i (µ * i * ) KL(P X i P )
, where D i (µ)

P X i ∈ D i , E[X i ] > µ .
Proof. We fix P X ∈ D and i ∈ [n] such that ∆ i > 0. In the proof, we use the notation

d i = inf P ∈D i (µ * i * ) KL(P X i P ).
To get the desired result, it is sufficient to prove the following lower bound (and then sum over i, weighting by ∆ i ) lim inf

T →∞ E[N i,T ] log(T ) ≥ d i -1 .
Notice that it is equivalent to prove that ∀ε ∈ (0, 1), lim inf

T →∞ E[N i,T ] log(T ) ≥ (1 -ε)d i -1 .
We thus fix ε ∈ (0, 1). From Markov's inequality we get

E[N i,T ] log(T ) ≥ (1 -ε)P N i,T /log(T ) ≥ (1 -ε)d i -1 d i .
Thus, it suffices to show that lim

T →∞ P N i,T log(T ) ≥ 1 -ε d i = 1,
or equivalently that lim

T →∞ P N i,T log(T ) < 1 -ε d i = 0. (2.1) Let δ > 0 be such that (1 -δ)/(1 + δ) > 1 -ε. By definition of d i , there exists P ∈ D i (µ * i * ) such that d i < KL(P X i P ) < (1 + δ)d i .
(2.2) Define the events

A T N i,T log(T ) < 1 -δ KL(P X i P ) , C T L N i,T ≤ (1 -δ/2) log(T ) ,
where

L t t t =1 log dP X i dP (X i,t ) .
We write P , E the probability and expectation when samples from arm i are drawn from P instead of P X i (i is thus the best arm in this new environment). Notice, since D = D 1 ⊗ . . . ⊗ D n , and P ∈ D i , we have that

P X 1 ⊗ • • • ⊗ P X i-1 ⊗ P ⊗ P X i+1 ⊗ • • • ⊗ P Xn (2.3) ∈ D.
We now show that

P[A T ] = P[A T ∩ C T ] + P[A T \C T ] → T →∞ 0
, which together with (2.2) proves (2.1). On the one hand, we have

P[A T ∩ C T ] ≤ e (1-δ/2) log(T ) P [A T ∩ C T ] (2.4) ≤ T 1-δ/2 P [A T ] = T 1-δ/2 P T -N i,T > T - 1 -δ KL(P X i P ) log(T ) ≤ T 1-δ/2 E [T -N i,T ] T - 1-δ KL(P X i P ) log(T ) (2.5) = T -δ/2 j =i E [N j,T ] 1 - 1-δ KL(P X i P ) log(T )/T → T →∞ 0.
(2.6)

(2.4) is from a change of measure: the random variable I{C T ∩ {N i,T = t}} is measurable with respect to X i,1 , . . . , X i,t and X j,1 , . . . , X j,T (j = i) (and a possible extra source of randomness). Its partial integral against t t =1 dP X i (x i,t ) is lower than the one against e (1-δ/2) log(T ) t t =1 dP (x i,t ), by virtue of the event C T . (2.5) is a consequence of Markov's inequality, and the limit in (2.6) is by consistency of π and by (2.3).

On the other hand, letting b T log(T )(1 -δ)/KL(P X i P ), we have

P[A T \C T ] ≤ P max t<b T L t > (1 -δ/2) log(T ) = P b -1 T max t<b T L t > 1 -δ/2 1 -δ KL(P X i P ) .
This last term tends to zero, as a consequence of the law of large numbers.

Example 1 (The Bernoulli bandit case). As we already saw in the previous section, a well studied particular case of MAB setting is where D is the family of distributions P X such that X ∈ {0, 1} n , i.e, X i follows a Bernoulli distribution for all i ∈ [n].

In this case, Theorem 1 is reformulated as follows: Let π be a consistent policy with respect to a distribution family D. Then we have, for all P X ∈ D,

R T (π) ≥ i∈[n], ∆ i >0 log(T )(1 -o(1))∆ i kl(µ * i , µ * i * )
,

where kl(x, y) KL(Bernoulli(x) Bernoulli(y)) = x log x y + (1 -x) log 1-x 1-y , x, y ∈ [0, 1].
In Example 1, the result can be interpreted as follows: To achieve good performance for all arm distributions, each sub-optimal arm i needs to be explored, asymptotically, no fewer than log(T )/kl(µ * i , µ * i * ) times, which grows in a logarithmic order with T and is inversely proportional to the distribution divergence kl(µ * i , µ * i * ) between this sub-optimal arm i and the optimal arm i * . Another classical example is given by Gaussian distributions.

Example 2 (The Gaussian bandit case). Consider the MAB setting where D is the family of distributions such that X i follows a Gaussian distribution of variance σ 2 i for all i ∈ [n]. In this case, Theorem 1 is reformulated as follows: Let π be a consistent policy with respect to a distribution family D. Then we have, for all P X ∈ D,

R T (π) ≥ i∈[n], ∆ i >0 2σ 2 i log(T )(1 -o(1)) ∆ i .
In case the distribution family D has not a product form, one can still obtain lower bounds, where the change of distribution argument is done in several arms instead of one. The important thing is to change the distribution while remaining within the family under consideration. We give in the following an example of such lower bound result in the Bernoulli bandit problem.

Theorem 2 [START_REF] Graves | Asymptotically efficient adaptive choice of control laws incontrolled markov chains[END_REF]). Let π be a consistent policy with respect to a distribution family D such that for all P X ∈ D, X ∈ {0, 1} n . Then we have, for all P X ∈ D,

lim inf T →∞ R T (π) log(T ) ≥ inf c∈C i∈[n], ∆ i >0 c i ∆ i ,
where

C (c i ) i ∈ R n + , ∀P X ∈ B(µ * ), i c i kl(µ * i , E[X i ]) ≥ 1 , B(µ * ) P X ∈ D, s.t. max i E[X i ] > µ * i * and E[X i * ] = µ * i * .

Distribution-free lower bound

We are now stating a distribution-free lower bound for the expected cumulative regret, in Theorem 3. This result dates back to [START_REF] Vogel | An asymptotic minimax theorem for the two armed bandit problem[END_REF], and was more recently improved by Bubeck and Cesa-Bianchi (2012).

Theorem 3 (Bubeck and Cesa-Bianchi (2012)). Assume that n ≥ 2. Let D be the family of distributions P X such that P X i is a Bernoulli distribution for all i ∈ [n].

Then we have inf

π sup P X ∈D R T (π) ≥ 1 20 √ nT .
Proof. For i ∈ {0, . . . , n}, we consider the bandit problem P i ∈ D where all arm rewards are sampled from a Bernoulli distribution of mean 1/2, except for one arm i that follows a Bernoulli distribution of mean 1/2 + ε, for ε > 0 to be stated later (thus, for i = 0, all arms have the same distribution). We denote (respectively) P (i) , E (i) and R (i)

T the probability, the expectation, and the regret when rewards are sampled from the distribution P i . We have for all policy π sup

P X ∈D R T (π) ≥ sup i∈[n] R (i) T (π) ≥ 1 n i∈[n] R (i) T (π) = ε n i∈[n]
T -

E (i) [N i,T ] . (2.7) Notice that inf π 1 n i∈[n] R (i) T (π) = inf π deterministic 1 n i∈[n] R (i) T (π),
since a randomized policy is a convex combination of deterministic ones. We thus assume that π is deterministic in the following. We can bound the difference between

E (i) [N i,T ] -E (0) [N i,T ] for all i ∈ [n] as E (i) [N i,T ] -E (0) [N i,T ] = T t=1 P (i) [i t = i] -P (0) [i t = i] ≤ T P (i) -P (0) TV ≤ T KL P (0) P (i) /2,
where, for two probability distributions P and Q on a measurable space (Ω, A), P -Q TV sup A∈A |P (A) -Q(A)| is the total variation distance between P and Q, and where the last inequality is from Pinsker's inequality (see Proposition 1). Since π is a deterministic policy, we get from the chain rule for the KL that KL P (0) P (i) = E (0) [N i,T ]KL(P

0 P i ) = E (0) [N i,T ]KL(Bernoulli(1/2) Bernoulli(1/2 + ε)) = E (0) [N i,T ] log(1/(1 -4ε 2 ))/2
Substituting the obtained upper bound on E (i) [N i,T ] in (2.7) gives sup

P X ∈D R T (π) ≥ εT - ε n i∈[n] E (0) [N i,T ] - εT 2n log 1 1 -4ε 2 i∈[n] E (0) [N i,T ] ≥ εT - ε n T - εT 2n log 1 1 -4ε 2 √ nT ,
where this last inequality uses i∈[n] E (0) [N i,T ] = T and the Cauchy-Schwarz type

inequality i∈[n] E (0) [N i,T ] ≤ n i∈[n] E (0) [N i,T ]. Setting ε = √ n/
T /4 and usinglog(1 -y) ≤ 4 log(4/3)y for y ∈ [0, 1/4] gives the final result.

Proposition 1 (Pinsker's inequality, see e.g. [START_REF] Tsybakov | Springer Series in Statistics[END_REF]). Let P and Q be two distributions over some measurable space (Ω, A), then

KL(P Q) ≥ 2 P -Q 2 TV .
Proof. Without loss of generality, we can assume that P is absolutely continuous with respect to Q, since otherwise KL(

P Q) = ∞. Recall that P -Q TV = sup A∈A |P (A) -Q(A)| = 1 2 dP dQ -1 dQ. We thus have, with r = dP /dQ -1, f = |r|/g and g = √ 1 + r/3, 2 P -Q 2 TV = 1 2 dP dQ -1 dQ 2 = 1 2 f g dQ 2 ≤ 1 2 f 2 dQ g 2 dQ (2.8) = 1 2 f 2 dQ = r 2 2(1 + r/3) dQ ≤ ((1 + r) log(1 + r) -r) dQ = KL(P Q), (2.9)
where (2.8) is from Cauchy-Schwarz inequality, and (2.9) is from

0 ≤ h(x) = -x 2 2(1 + x/3) + (1 + x) log(1 + x) -x ∀x ≥ -1, indeed, h (x) = log(1 + x) -3x(x + 6)/(2(x + 3) 2 ) and h (x) = x 2 (x + 9)/((x +
3) 3 (x + 1)) ≥ 0 (and equals 0 only for x = 0), so h is increasing and h (x) ≥ h (0) = 0 if and only if x ≥ 0, so min x≥-1 h(x) = h(0) = 0.

The minimax nature of the formulation implies that the lower bound in Theorem 3 holds for all distribution families that include those with {0, 1} support, for example distributions with bounded support on [0, 1] n or more generally distributions whose marginals are 1/4-sub-Gaussian (see Definition 7 for the definition of a κ 2 -sub-Gaussian distribution, and Proposition 2 for the fact that a random variable in [0, 1] is 1/4-sub-Gaussian). The question is in the achievability of this lower bound for these more general sets of distributions.

Definition 7 (κ 2 -sub-Gaussian distribution, [START_REF] Buldygin | Sub-Gaussian random variables[END_REF]). We say that a random variable

X ∈ R (or a distribution P X ) is κ 2 -sub-Gaussian if ∀λ ∈ R, E e λ(X-E[X]) ≤ e λ 2 κ 2 /2 .
Proposition 2 (Hoeffding's Lemma, Hoeffding (1963)). Let X be a random variable almost surely in [a, b] Evaluating this inequality in X and taking the expectation, we get

, with a < b ∈ R. Then, X is (b -a) 2 /4-sub-Gaussian.
E e λX ≤ e ψ(λ(b-a)) = b b -a e λa - a b -a e λb , with ψ(u) = -pu + log(1 -p + pe u ), p = -a b -a .
We see that ψ(0) = ψ (0) = 0. In addition,

ψ (u) = (1 -p)pe u (1 -p + pe u ) 2 ≤ 1 4 .
Finally, thank to Taylor's theorem, there is θ ∈ (0, 1) such that

ψ(u) = ψ(0) + ψ (0)u + ψ (θu) u 2 2 ≤ u 2 8 .
Remark 4. In Theorem 3, when instead D is the family of distributions P X such that X is in {a, b} n almost surely, for a < b ∈ R, we have the lower bound

inf π sup P X ∈D R T (π) ≥ b -a 20 √ nT .
Indeed, we can trivially reduce to the case X a.s.

∈ {0, 1} n by translating and scaling up all rewards X i as

X i -a b -a a.s. ∈ {0, 1}.
This simply has the effect of dividing the regret by b -a, and can be seen as a simple convention taken on the unit of the rewards.

Policies

In this section, we review some examples of well known MAB policies. We will see in particular that the proposed policies have one thing in common, namely that they all make the decision i t by maximizing a certain function defined on [n] that depends on the agent observation history up to the round t -1. We refer to this kind of policy as index-based (the function value is the index that guides the decision).

Index-based MAB policies

A large majority of existing MAB policies are based on a scoring index maintained across rounds, which gives for each arm i and each round t a numerical value representing the potential quality of arm i after t -1 rounds. The score of the arm i can be computed from n, t, the history and a possible source of randomness, i.e., is measurable with respect to F t . These policies are described in the Algorithm 2 and work as follows. During the first n rounds, they sequentially play arms 1, 2, ..., n to perform the initialization. In all subsequent rounds, they compute for each arm the corresponding score and select the arm with the greatest one (ties are broken at random). The scoring index of arm i at round t can generally be seen as a proxy or estimates of the true mean µ * i . Indeed, ultimately, when the agent have collected enough knowledge to know the mean vector µ * , the policy should behave as the optimal policy π * , that is nothing else than Algorithm 2 with µ * i as the score of arm i. This is the reason why we sometimes use the notation µ i,t for the score of arm i at round t.

Examples

We give in Table 2.1 some well-known examples of index-based MAB policies. The indexes provided are valid when the reward vector X is in [0, 1] n almost surely (for the sake of simplicity, we limit ourselves to this case here). There are several quantities that we have to define in order to be able to read Table 2.1. First, the agent can estimate the mean µ * i of every arm i with their corresponding empirical averages defined as

µ i,t-1 t ∈[t-1] I{i = i t }X i,t N i,t-1 , for t ≥ 1, where N i,t-1 t ∈[t-1] I{i = i t }
is the number of time arm i have been drawn for the first t -1 rounds, as mentioned above in Remark 3. Another quantity that can be used is the empirical variance of an arm i, defined as

σ 2 i,t-1 t ∈[t-1] I{i = i t } X i,t -µ i,t-1 2 N i,t-1 ,
for t ≥ 1. Notice that another expression for the empirical variance (that is more practical to maintain for a policy) is

σ 2 i,t-1 = t ∈[t-1] I{i = i t }X 2 i,t N i,t-1 -µ 2 i,t-1 .
The advantage of maintaining such a quantity is that it allows the agent to measure how uncertain the rewards obtained are, and can therefore be useful in the control of the estimation error. For example, when the rewards of an arm i are deterministic, the corresponding empirical variance is zero, allowing the agent to limit its exploration on this arm. We begin by describing ε-greedy (see [START_REF] Watkins | Learning from Delayed Rewards[END_REF], that is probably one of the simplest MAB policy. Then, we describe policies which can be qualified as optimistic, since they rely on the optimism in face of uncertainty principle (OFU principle). We then provide a Bayesian randomized policy, thompson sampling (ts), that maintain an estimated payoff distribution for each arm. Finally, we notice that these previous policies are rather destined to satisfy the obtimality criterion given in Theorem 1, and give moss, an example of a policy satisfying the second criterion of optimality, given by Theorem 3.

ε-greedy

ε-greedy is easy to understand, since it simply chooses the arm uniformly at random with probability , and choose the leader arm (the one with the highest empirical average) with probability 1 -ε. The exploration/exploitation phases are thus clearly separated. A typical parameter value might be ε = 0.1, but, actually, any fixed value of ε that is independent of T gives an expected cumulative regret that scales

Name of the policy

The score µ i,t ε-greedy µ

i,t-1 I{U t ≥ ε} ucb 1 ∧ µ i,t-1 + 0.5δ(t) N i,t-1 ucb-v 1 ∧   µ i,t-1 + 2σ 2 i,t-1 δ(t) N i,t-1 + 3δ(t) N i,t-1    ucb-kl The largest x ∈ [µ i,t-1 , 1] such that N i,t-1 kl µ i,t-1 , x ≤ δ(t)
thompson sampling linearly with T . Thus, since T is unknown, a value of ε decreasing as the experiment progresses is more judicious [START_REF] Cesa-Bianchi | Finite-Time Regret Bounds for the Multiarmed Bandit Problem[END_REF][START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF], leading to very exploratory behavior at the beginning and very exploitative behavior at the end. We can also comment on why ε = 0 is a policy that will also have a linear regret in T . Let's consider to illustrate this fact the example of the Bernoulli bandit problem, with means in (0, 1). With a positive probability, after the initialization phase, the empirical average of the best arm will be 0 and that of all the others will be 1. From then on, the agent will never choose the best arm again, because there will always be another sub-optimal arm with a positive empirical average.

An independent sample from Beta(α, N i,t-1 -α), where α = N i,t-1 µ i,t-1 moss 1 ∧ µ i,t-1 + 0 ∨ log(T /(nN i,t-1 )) N i,t-1

Optimistic policies

ucb, ucb-v and ucb-kl belongs to the family of optimistic policies, which proceed by maintaining a confidence region for each arm expected payoff. For an arm i, we construct the score µ i,t as the supremum of the corresponding region. Thus defined, the arms score can also be interpreted as the largest statistically plausible mean value of the arm, given the currently available observation. Key ingredients in the construction of these algorithms (and also in their analysis) are concentration inequalities (see subsection 2.5.1), used to upper bound the estimation error with high probability (which is equivalent to constructing the confidence region). The terminology "optimistic" comes from the fact that for each arm i, the score µ i,t is an upper confidence bound (UCB) on the true mean µ * i at round t, i.e., it is a slight overestimate of µ * i , with high probability. This overestimation is actually intended to counterbalance the fact that the empirical mean is negatively biased as an estimator for the true mean [START_REF] Nie | Why adaptively collected data have negative bias and how to correct for it[END_REF]. This principle thus suggests to follow what seems to be the best arm, based on the optimistically constructed arm-scores. In other words, the agent chooses the arm i t by considering the most favorable environment among those that seem likely, i.e., chooses in an optimistic way.

Upper Confidence Bound (ucb) Introduced by [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF], the ucb algorithm relies essentially on the Hoeffding's inequality (see Theorem 5) to derive an upper bound on the regret (see Theorem 10). It derives for each arm a confidence interval around the empirical mean, and chooses at each round the arm with the highest UCB. Hoeffding's inequality allows a close form for the UCB of each arm i. More precisely, if we do not take into account that the final score is capped at 1 (which is a simple consequence of the assumption that the variables are in [0, 1]), it is the sum of two terms: the empirical mean µ i,t-1 and the exploration bonus 0.5δ(t)/N i,t-1 , that is a high probability upper bound on the estimation error of

µ i,t-1 -µ * i .
The empirical mean is intended to guide the exploitation, measuring which arm is the best given the observed history. The exploration bonus is intended to guide exploration by indicating which arm has not been sampled much in the past. Indeed, the less an arm i has been sampled, the lower its counter N i,t-1 will be and the bigger the bonus will be.

We will see later that the ucb algorithm can be analyzed relatively easily, and that its theoretical performance is very close to the lower bound given in Theorem 1. This makes ucb an ideal candidate for designing policies in many generalizations of MABs, such as stochastic combinatorial semi-bandits.

Upper Confidence Bound with Variance estimates (ucb-v) [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF] were the first to have the idea of exploiting the empirical variance of the arm in order to design the exploration bonus. Informally, as we have seen, an arm with a large variance should be explored more often than an arm with a small variance. Thus, the ucb policy that we presented in the previous paragraph explores too much, in the sense that the exploration bonus can be reduced in order to improve policy performance. The formal study of variance estimation in ucb type policies has been encouraged by the empirical superiority of such an approach [START_REF] Audibert | Explorationexploitation trade-off using variance estimates in multi-armed bandits[END_REF]. The resulting policy, known as the Upper Confidence Bound with Variance estimates (ucb-v), is based on the Bernstein's concentration inequality, sharper than that of Hoeffding (see Theorem 6). Since an approach considering a variance-dependent exploration bonus is not possible to construct for the agent (because the true variances are unknown), confidence regions around the empirical variances are also considered, giving in sum an approach based on an empirical Bernstein inequality, where the exploration bonus depends on the empirical variance (see Theorem 8). The gain in the exploration bonus compared to the ucb policy is immediately reflected on the regret bound (see Theorem 10).

Upper Confidence Bound using the Kullback-Leibler divergence (ucb-kl)

The ucb-kl policy (also called kl-ucb in the literature [START_REF] Garivier | The {KL}-{UCB} algorithm for bounded stochastic bandits and beyond[END_REF]) uses an approach other than variance estimation in order to improve ucb, and is based on a KL confidence region for bounded random variables in [0, 1] (see Corollary 2). This policy and its analysis simultaneously appeared in [START_REF] Garivier | The {KL}-{UCB} algorithm for bounded stochastic bandits and beyond[END_REF] and [START_REF] Maillard | Finite-Time Analysis of Multi-armed Bandits Problems with Kullback-Leibler Divergences[END_REF], and was latter unified by [START_REF] Cappé | Kullback-leibler upper confidence bounds for optimal sequential allocation[END_REF]. It is inspired by the seminal papers of [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] and [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF]. As [START_REF] Maillard | Finite-Time Analysis of Multi-armed Bandits Problems with Kullback-Leibler Divergences[END_REF] indicate, it aims to achieve optimality guarantees in the sense of [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF], i.e., to match the asymptotic regret lower bound by deriving an upper bound of the form

R T (π) ≤ i∈[n], ∆ i >0 log(T )(1 + o(1))∆ i kl(µ * i , µ * i * )
.

To give an intuition about the algorithm, ucb-kl treats the problem as if it were a Bernoulli bandit one, and uses a confidence region specially adapted to this case.

Noticing that binary rewards is a kind of worst case, the method extends to any variable in [0, 1]. More concretely, as we will see in subsection 2.5.1, the crucial step for deriving a concentration inequality relies on a bound on the Moment-generating function (MGF) λ → E e λX . This bound is of the same type as the one given for sub-Gaussian random variables (Definition 7). Thus, from random variables in [0, 1], we can go back to binary variables thanks to the following Lemma.

Lemma 1. If X is a random variable in [0, 1] and Y ∼ Bernoulli(E[X]), then, for all λ ∈ R, E e λX ≤ E e λY .
Proof.

If λ ≥ 0, we have from X ∈ [0, 1], E e λX = k≥0 λ k E X k k! ≤ 1 + k≥1 λ k E[X] k! = 1 + E[X] e λ -1 = E e λY .
In the case λ < 0, we first write E e λX = E e -λ(1-X) e λ . Using the above result with 1 -X ∈ [0, 1] and -λ > 0, we have E e -λ(1-X) ≤ E e -λ(1-Y ) . Thus,

E e -λ(1-X) e λ ≤ E e -λ(1-Y ) e λ = E e λY .
We can easily compare ucb-kl to ucb, since we can go from the first to the second by considering the quadratic divergence 2(p -q) 2 instead of the Kullback-Leibler divergence kl(p, q). From Pinsker's inequality (giving kl(p, q) ≥ 2(p -q) 2 in this context), the KL confidence region is tighter, so the policy explores less, resulting in a better bound on the regret (see Theorem 10, one can also see [START_REF] Cappé | Kullback-leibler upper confidence bounds for optimal sequential allocation[END_REF] for an improved regret upper bound). Comparing ucb-kl to ucb-v is not straightforward, because ucb-kl treats the problem as if rewards were binary, leading to an overestimation of the variance. For instance, if rewards are deterministic in (0, 1), then ucb-v is better. On the other hand, if rewards are binary, then ucb-kl is better.

Thompson sampling

thompson sampling (ts) is an example of Bayesian randomized policy, introduced by [START_REF] Thompson | On the likelihood that one unknown probability exceeds another in view of the evidence of two samples[END_REF] (see also Thompson (1935)), simultaneously with the MAB problem. They have since been widely studied. [START_REF] Chapelle | An empirical evaluation of Thompson sampling[END_REF] provided a thorough empirical evaluation of the policy, highlighting its advantages. [START_REF] Agrawal | Analysis of Thompson sampling for the multi-armed bandit problem[END_REF] gave an important first theoretical result, establishing a logarithmic regret bound. Latter, [START_REF] Kaufmann | Thompson Sampling: An Asymptotically Optimal Finite Time Analysis[END_REF] proved that ts is actually matching the Lai and Robbins lower bound.

ts policy follows the Bayesian inference framework. The unknown distributions are parameterized with an assumed prior distribution. ts uses the prior distribution in each round with two phases: first it uses the prior distribution to sample a parameter, which is used to determine the action to play in the current round; second it uses the feedback obtained in the current step to update the prior distribution to posterior distribution according to the Bayes' rule.

The use of Beta prior distribution is convenient as it is a conjugate distribution of the Bernoulli distribution. This allows easy computations of the posterior distribution after the observation of a Bernoulli realization. As for ucb-kl, we can extend the algorithm for non-binary rewards. The trick here is to update the model using another binary sample having the same mean as the true reward sample received. Indeed, in terms of expected regret, suffering the binary sample instead of the true reward is identical. Note that ts defines a family of algorithms as the choice of the priors is left to the user.

Policy for the distribution-free case

Let us mention here that there are also policies being optimal in the minimax sense, i.e., uniformly on all problems (they are matching the lower-bound from Theorem 3). The most well-known such policy is moss (Minimax Optimal Strategy in the Stochastic case) [START_REF] Audibert | Regret Bounds and Minimax Policies under Partial Monitoring[END_REF]Degenne and Perchet, 2016a). This policy is very similar to ucb, involving a modified exploration function for each arm.

Low/high probability events

In designing the above policies (particularly those that are optimistic), we have referred to events that have a high probability of occurring. We give here a more precise meaning of this. For a round t, saying that an event A t holds with low probability means that P[A t ] = o(1/t) (and high-probability events are those whose complement holds with a low probability). We justify this now: By taking out events occurring with low probability, we wish to restrict the scope of possibilities to a high probability event in order to better guide the choice i t . Specifically, the choice i t will be relevant only under the high probability event ¬A t , and will be meaningless otherwise. So if we decompose the regret as follows

E T t=1 ∆ it = E T t=1 ∆ it I{A t } + E T t=1 ∆ it I{¬A t } , then the first term can be bounded by max i∈[n] ∆ i T t=1 P[A t ]. We thus want T t=1 P[A t ] = o(log(T )),
in order to be certain that it will be negligible compared to the second term, which we have seen must grow at least logarithmically (Theorem 1). One way to ensure this is to require P[A t ] = o(1/t).

Regret upper bound analysis

There are two types of upper bounds we can provide, exactly as for lower bounds. The first one is referred to as gap-dependent, because it depends explicitly on the underlying gaps ∆ i , i ∈ [n]. The second type is independent of the gaps, and is referred to as gap-independent, or gap-free. A gap-free bound gives more general guarantees as it would apply to several problem instances. However, the bound is then the worst case among these instances. On the other hand, a gap-dependent bound depends on the problem instance and, in particular, helps to understand how fast the policy can learn in the easier problem instances (i.e., those with large gaps).

We will only give an analysis of the regret of the optimistic policies we presented above, for brevity's sake. As already mentioned, this kind of analysis is simpler to implement in the generalizations of the MAB setting to semi-bandit feedback and is therefore of primary interest for our presentation. We will also focus on the gap-dependent bounds. Indeed, we have just seen that gap-free bounds were not informative about the difficulty of the problem at hand, so we have chosen not to give priority to this kind of bound. Of course, when we have the opportunity to do so, we will indicate the extent to which policies behave from a minimax point of view. We refer the reader to the references given above for each policy for a case-by-case analysis.

Before going into more details about regret upper bounds, we first give an overview of the essential results allowing these bounds, namely concentration inequalities.

A tour of concentration inequalities

In this subsection, we prove the main results on the concentration of the empirical mean towards the true mean. Concentration is a phenomenon concerning the behavior of the distribution of the empirical mean. In order to quantify how the empirical mean converges to the true mean, we look at a region around the empirical mean that contains the true mean with high probability. The smaller this region is, the more the empirical mean has concentrated towards the true mean. In the MAB context, these regions are confidence intervals. They will not only serve to prove regret upper bounds, but are also useful in understanding how policies are designed. Indeed, for an arm i, the policy will choose the score µ i,t as the UCB of the confidence interval around µ i,t-1 . Thus, the challenge is to build a confidence interval -containing the true mean with high probability -which is as tight as possible, because it would determine how the policy will explore. A region that is too big leads to a policy exploring too much. If the region is too small, the probability that the true mean does not belong to the region becomes non-negligible, and gives rise to poor performance as well.

In the whole subsection, we fix i ∈ [n], t ∈ N * , and assume that N i,t-1 ≥ 1. We start with two definitions. The first defines the cumulant-generating function of the random variable X i as the logarithm of the MGF. The second defines the Legendre-Fenchel conjugate of this function. We will use the latter, evaluated as the empirical mean, to quantify the deviation from the true mean.

Definition 8 (Cumulant-generating function). The cumulant-generating function of

X i is φ i (λ) log E e λX i ,
defined for all λ ∈ R where it is finite, which includes at least λ = 0. If it exists on (λ 1 , λ 2 ) containing 0, then φ i is infinitely differentiable and is strictly convex on this interval (if X i is not deterministic).

Definition 9 (Convex conjugate of φ i ). If φ i is defined on an open interval (λ 1 , λ 2 ) containing 0, then φ * i : R → R ∪ {∞} is the convex conjugate of φ i , defined by

φ * i (z) sup λ∈(λ 1 ,λ 2 ) (λz -φ i (λ)) ≥ 0.
It is strictly convex on {φ * < +∞}, and φ i (µ * i ) = 0 (by Jensen's inequality). In particular, it is continuous and decreasing (resp. increasing) on the interval (-∞,

µ * i ) ∩ {φ * i < ∞} (resp. (µ * i , +∞) ∩ {φ * i < ∞}).
Before moving on to the central result of this subsection, it is useful to notice a link between the sign of z -µ * i and and the side where the maximizer λ should be taken.

Lemma 2. If z < µ * i , then φ * i (z) = sup λ∈(λ 1 ,0), λz-φ i (λ)>0 (λz -φ i (λ)),
and if z > µ * i , then

φ * i (z) = sup λ∈(0,λ 2 ), λz-φ i (λ)>0
(λz -φ i (λ)).

Proof. We only prove the first part, since the second uses symmetric arguments. The function to maximize ψ z : λ → λz -φ i (λ) is strictly concave. If there is a maximizer λ ∈ (λ 1 , λ 2 ), then the first order condition ψ z (λ) = 0 gives φ i (λ) = z < µ * i = φ i (0). Since φ i is increasing, this means that λ < 0. By strict concavity, we necessarily have ψ z (λ) > 0, proving the lemma in this case. If no maximizer exists, since ψ z (0) = z -µ * i < 0, we have by continuity that ψ z < 0 on (λ 1 , λ 2 ), so ψ z is decreasing and

φ * i (z) = lim λ→λ 1 (λz -φ i (λ)). Since ψ z (0) = 0, all the values λ ∈ (λ 1 , 0) are such that ψ z (λ) > 0, while values λ ∈ [0, λ 2 ) are such that ψ z (λ) ≤ 0,
so the lemma is proved in this case as well.

We now state the following theorem, which gives a bound on the probability that the gap between the empirical mean and the true mean is greater than a certain quantity. As we announced, the gap is measured with the function φ * i . The preceding Lemma 2 allows us to distinguish two cases, depending on whether the empirical mean is smaller or larger than the true mean. As usual, we can group these two cases together and end up with a probability multiplied by 2. The proof uses several ingredients, including a peeling to deal with the random counters N i,t-1 .

Theorem 4. Assume that φ i is finite on an open interval (λ 1 , λ 2 ) containing 0. Then, for δ > 1, the event

A t µ * i > µ i,t-1 and N i,t-1 φ * i µ i,t-1 ≥ δ
holds with probability lower than δ log(t) e 1-δ . In a same way,

A t µ * i < µ i,t-1 and N i,t-1 φ * i µ i,t-1 ≥ δ
holds with probability lower than δ log(t) e 1-δ .

Proof. As for Lemma 2, we only prove the first bound. We use a peeling argument in order to deal with the random counter N i,t-1 . For this, let γ = δ/(δ -1) > 1, and consider the partition given by

B d,t N i,t-1 ∈ (γ d-1 , γ d ] ,
for d ∈ 1, . . . , log γ (t) . We now want to bound the probability of the event A t ∩ B d,t and then use an union bound on d ∈ 1, . . . , log γ (t) . Since φ * i is decreasing and continuous on the interval (-∞,

µ * i ) ∩ {φ * i < ∞}, we can define x i (N ) for N ≥ 1 as x i (N ) = -∞ if φ * i < δ/N
µ i,t-1 ≥ γ d δ/N i,t-1 ≥ δ, that µ i,t-1 ≤ x γ d . Taking some λ ∈ (λ 1 , 0) such that λx γ d > φ i (λ) thus gives exp λN i,t-1 µ i,t-1 -λN i,t-1 x γ d ≥ 1, i.e., exp N i,t-1 λµ i,t-1 -φ i (λ) ≥ exp N i,t-1 λx γ d -φ i (λ) ≥ exp γ d-1 λx γ d -φ i (λ) .
We can thus bound the probability of the event A t ∩ B d,t by the probability that the above inequality holds. The definition of φ i implies that

E exp N i,t-1 λµ i,t-1 -φ i (λ) = E t-1 u=1 exp(I{i u = i}(λX i,u -φ i (λ))) = E E t-1 u=1 exp(I{i u = i}(λX i,u -φ i (λ))) F t-1 = E    t-2 u=1 exp(I{i u = i}(λX i,u -φ i (λ))) E[ exp(I{i t-1 = i}(λX i,t-1 -φ i (λ)))|F t-1 ] =1    = • • • = 1.
where we used that I{i t = i} ∈ F t , and X i,t is independent of F t . So from Markov inequality,

P[A t ∩ B d,t ] ≤ P exp N i,t-1 λµ i,t-1 -φ i (λ) ≥ exp γ d-1 λx γ d -φ i (λ) ≤ exp -γ d-1 λx γ d -φ i (λ) .
Notice that we can take the supremum of λx γ d -φ i (λ) over λ ∈ (λ 1 , 0) such that λx γ d > φ i (λ), thus giving using Lemma 2 with x γ d < µ * i that

P[A t ∩ B d,t ] ≤ exp -γ d-1 φ * i x γ d = e -δ/γ .
Putting everything together with an union bound, we have using log(δ/(δ -1)) ≥ 1/δ, that

P[A t ] ≤ log γ (t) d=1 P[A t ∩ B d,t ] ≤ log γ (t) e -δ/γ = δ log(t) e 1-δ .
As we've seen, the event A t defined in the previous theorem must be such that P[A t ] = o(1/t). This determines which choice for δ > 1 we're going to take. Indeed, taking δ = δ(t) = log(t) + 3 log log(t), for t ≥ 3, with δ(1) = δ(2) = δ(3), gives for t ≥ 3 δ log(t) e 1-δ = e log 2 (t) + 3 log(t) log log(t) t -1 log -3 (t) = o(1/t).

Notice also that δ = δ(t) = ζ log(t) for ζ > 1 also gives

δ log(t) e 1-δ = e ζ log 2 (t) t -ζ = o(1/t).
One last ingredient needed in order to effectively use A t in policy design is the actual construction of φ * i . This is where the hypothesis on the family of random variables under consideration has an impact. In fact, we are not going to use φ * i directly, but rather another function also measuring the gap to the true mean and which bounds inferiorly φ * i . Indeed, φ * i can be directly replaced by this lower bound in the event A t , as this can't increase the probability of A t . One way to build a lower bound on φ * i is to build an upper bound on φ i , which coroborates the discussion before Lemma 1, where we mention the importance of upper bounding the MGF in order to have a concentration inequality. The simpler example of such upper bound is the one given in Definition 7, leading to the following well-known theorem, the Hoeffding's inequality.

Theorem 5 (Hoeffding's inequality, Hoeffding (1963)).

If X i is κ 2 -sub-Gaussian, then P µ * i -µ i,t-1 ≥ 2κ 2 δ(t) N i,t-1 ≤ δ(t) log(t) e 1-δ(t) , P µ i,t-1 -µ * i ≥ 2κ 2 δ(t) N i,t-1 ≤ δ(t) log(t) e 1-δ(t) ,
where δ(t) > 1.

Proof. We prove the first inequality, the second is obtained symmetrically. The event can be rewritten as

A t µ * i > µ i,t-1 , 0.5N i,t-1 µ * i -µ i,t-1 2 /κ 2 ≥ δ(t) .
Thus, from Theorem 4, it is sufficient to prove that φ * i is lower bounded by z → 0.5(µ * i -z) 2 /κ 2 . For this, we use the upper bound ψ(λ) = λµ * i + κ 2 λ 2 /2 on φ i (λ), valid for all λ ∈ R, thanks to the sub-Gaussianity assumption. Since we have

ψ * (z) ≤ φ * i (z), it is sufficient to prove that ψ * (z) = 0.5(µ * i -z) 2 /κ 2 . This is a direct consequence of ψ * (z) = sup λ∈R λz -λµ * i -κ 2 λ 2 /2
, where the maximizer is λ = (z -µ * i )/κ 2 . Remark 5. Note that our definition of the κ 2 -sub-Gaussianity gives a bound on the MGF of the centered version of X i . However, Theorem 5 also holds when for all λ ∈ R, E e λX i ≤ e λ 2 κ 2 /2 , with the same arguments.

Using Hoefding's lemma (Proposition 2), we can state the following corollary.

Corollary 1. If X i ∈ [0, 1], then P µ * i -µ i,t-1 ≥ 0.5δ(t) N i,t-1 ≤ δ(t) log(t) e 1-δ(t) , P µ i,t-1 -µ * i ≥ 0.5δ(t) N i,t-1 ≤ δ(t) log(t) e 1-δ(t) ,
where δ(t) > 1.

Due to its generality, Hoeffding's inequality has been widely used in MAB scenarios (the archetype is ucb). As mentioned before, a drawback of the bound is that it does not scale with the variance of X i . If the variance is known, Bernstein's inequality can be used instead, which can yield significant improvements when the variance is small (as we will see in Chapter 4). In a more concrete way, in the case of bounded random variables in [0, 1], there exist better bounds on the MGF than the one given by sub-Gaussianity, and thus better concentration inequalities. Indeed, we provide the following lemma, giving that bounded random variables satisfies an MGF bound called Bernstein's condition, which will be useful in the derivation of the Bernstein's inequality.

Lemma 3 (Bernstein's condition for bounded random variables, see [START_REF] Vershynin | A note on sums of independent random matrices after Ahlswede-Winter[END_REF]).

If X is a random variable such that |X -E[X]| ≤ c, for some c > 0, then for all λ ∈ (-3/c, 3/c), E e λ(X-E[X]) ≤ e λ 2 V[X]/2 1-|λ|c/3 . Proof. Let |λ| < 3/c, then log E e λ(X-E[X]) = log   1 + k≥2 λ k k! E (X -E[X]) k   ≤ k≥2 λ k k! E (X -E[X]) k log(x) ≤ x -1 ∀x > 0, ≤ λ 2 V[X] k≥2 |cλ| k-2 k! |X -E[X]| ≤ c ≤ 0.5λ 2 V[X] k≥2 c|λ| 3 k-2 k!/2 ≥ 3 k-2 , ∀k ≥ 2 = λ 2 V[X]/2 1 -|λ|c/3 .
We see that making the second order moment appear helps us to bound the MGF more precisely. We can also give a bound depending on the non-centered second order moment.

Lemma 4 (Non-centered one-sided Bernstein's condition). If X is a random variable such that X ≤ c a.s., for some c ≥ 0, then for all λ ∈ (0, 3/c),

E e λ(X-E[X]) ≤ e λ 2 E [ X 2 ] /2 1-λc/3 . Proof. Notice that g(x) = x -2 (e x -1 -x) is increasing on R. We have e λX = 1 + λX + λ 2 X 2 g(λX) ≤ 1 + λX + λ 2 X 2 g(λc)
. Then, taking the expectation and using log(x) ≤ x -1 ∀x > 0, we have

log E e λX ≤ λE[X] + λ 2 E X 2 g(λc).
Now, in the same way as previously, using

k!/2 ≥ 3 k-2 , ∀k ≥ 2, we have g(λc) = k≥2 (λc) k-2 /k! ≤ 1/(2(1 -λc/3
)), giving the result.

Remark 6. Notice that we can have c = 0 in Lemma 4, in which case the result hold for any positive λ. This will be particularly useful in the proof of empirical Bernstein bound, using it with variables -(X i -µ * i ) 2 ≤ c = 0. We are now ready to state Bernstein's inequality. We also give a non-centered one-sided version. In a same way as for the Hoeffding's inequality, we prove the following Theorem 6 using the MGF bound given in Lemma 3.

Theorem 6 (Bernstein's inequality, Bernstein (1924)).

If |X i -µ * i | ≤ c, then P µ i,t-1 -µ * i ≥ cδ(t) 3N i,t-1 + 2V[X i ]δ(t) N i,t-1 ≤ δ(t) log(t) e 1-δ(t) , P µ * i -µ i,t-1 ≥ cδ(t) 3N i,t-1 + 2V[X i ]δ(t) N i,t-1 ≤ δ(t) log(t) e 1-δ(t) ,
where δ(t) > 1.

Proof. As previously, we only prove the first inequality (in fact, the second inequality can be found from the first replacing X i by -X i , and noticing that the upper bound given in the previous Lemma 3 remains valid, as

|-X i -E[-X i ]| ≤ c). We assume that V[X i ] > 0, since otherwise µ i,t-1 = µ * i a.s.
, and the result trivially holds. We can derive the following lower bound on the function

φ * i (z), z > µ * i , ψ * (z) = sup λ∈(0,3/c) (z -µ * i )λ - λ 2 V[X i ]/2 1 -|λ|c/3 = V[X i ] sup λ∈(0,3/c) 3uλ/c - λ 2 /2 1 -λc/3 , where u = c(z -µ * i )/(3V[X i ]
). The objective is concave in (0, 3/c), so it is sufficient to find a critical point in (0, 3/c). The critical points are given by 3u

/c - λ 1 -λc/3 - λ 2 /6 (1 -λc/3) 2 = 0, i.e., λ 2 - 6λ c + 18u c 2 (1 + 2u) = 0.
Solving the two degree polynomial gives λ = 3c -1 1 ± 1/(1 + 2u) , so there is a single critical point in (0, 3/c), given by λ = 3c -1 1 -1/(1 + 2u) . Injecting this λ into the objective gives

ψ * (z) = 9c -2 V[X i ]h c(z -µ * i ) 3V[X i ] ,
where

h(x) 1 + x - √ 1 + 2x for x ≥ 0. Notice that h : R + → R + is an increasing function, and h -1 (x) = x + √ 2x, x ≥ 0.
Applying Theorem 4, we get that

A t µ * i < µ i,t-1 and N i,t-1 ψ * µ i,t-1 ≥ δ(t)
holds with probability lower than δ(t) log(t) e 1-δ(t) . We can rewrite the condition

N i,t-1 ψ * µ i,t-1 ≥ δ(t), using h -1 , as c(µ i,t-1 -µ * i ) 3V[X i ] ≥ h -1 c 2 δ(t) 9V[X i ]N i,t-1 , i.e., µ i,t-1 -µ * i ≥ cδ(t) 3N i,t-1 + 2V[X i ]δ(t) N i,t-1 > 0.
Theorem 7 (Non-centered one-sided Bernstein's inequality).

If X i ≤ c ∈ [0, ∞), then P µ i,t-1 -µ * i ≥ cδ(t) 3N i,t-1 + 2E[X 2 i ]δ(t) N i,t-1 ≤ δ(t) log(t) e 1-δ(t) ,
where δ(t) > 1.

Proof. We assume that E X 2 i > 0, since otherwise µ i,t-1 = µ * i a.s., and the result trivially holds. We can derive the same lower bound as previously on the function φ * i (z), z > µ * i , except that the variance is replaced by the second order moment E X 2 i using Lemma 4.

ψ * (z) = sup λ∈(0,3/c) (z -µ * i )λ - λ 2 E X 2 i /2 1 -λc/3 = E X 2 i sup λ∈(0,3/c) 3uλ/c - λ 2 /2 1 -λc/3 , where u = c(z -µ * i )/(3E X 2 i ).
All the remaining of the proof is the same as in Theorem 6, giving the desired result.

Since a priori knowledge of each arm variance is rarely available, this approach is not practical. A more suitable approach for MAB scenarios is to apply the previous Bernstein's inequality also to the empirical variance σ 2 i,t-1 . This results in a bound, which we will call the empirical Bernstein bound [START_REF] Audibert | Explorationexploitation trade-off using variance estimates in multi-armed bandits[END_REF] that we now state.

Theorem 8 (Empirical Bernstein inequality). Assume X i ∈ [0, 1] and let δ(t) > 1.

With probability 1 -3 δ(t) log(t) e 1-δ(t) , we have

µ * i -µ i,t-1 ≤ 3δ(t) N i,t-1 + 2σ 2 i,t-1 δ(t) N i,t-1 .
Proof. We do not directly apply Bernstein's inequality to the empirical variance, because the latter looks at the deviations from the empirical mean, whereas we wish to have the deviations from the true mean, in order to be able to apply the previous result replacing

X i by -(X i -µ * i ) 2 . Indeed, this is possible since -(X i -µ * i ) 2 ≤ c = 0. We thus get the two following inequalities, with probability 1 -3 δ(t) log(t) e 1-δ(t) , µ * i -µ i,t-1 ≤ δ(t) 3N i,t-1 + 2V[X i ]δ(t) N i,t-1 (2.10) -σ 2 i,t-1 -(-V[X i ]) ≤ 2E[(X i -µ * i ) 4 ]δ(t) N i,t-1 ≤ 2V[X i ]δ(t) N i,t-1 , (2.11)
where

σ 2 i,t-1 = (1/N i,t-1 ) t t =1 I{i t = i}(X i,t -µ * i ) 2 . We can notice that σ 2 i,t-1 -σ 2 i,t-1 = µ * i -µ i,t-1 2 , so, using (2.11), V[X i ] -σ 2 i,t-1 = V[X i ] -σ 2 i,t-1 + µ * i -µ i,t-1 2 ≤ 2V[X i ]δ(t) N i,t-1 + µ * i -µ i,t-1 2 .
(2.12)

Letting = δ(t)/N i,t-1 , we now prove that V[X i ] ≤ σ 2 i,t-1 + 1.8 √ . (2.13) Since variables X i are in [0, 1], we have V[X i ] ≤ 1/2. If > 1/(3.6) 2 , then 1.8 √ ≥ 1/2 ≥ V[X i ]
and the inequality (2.13) holds. If ≤ 1/(3.6) 2 , then, plugging (2.10) into (2.12) gives

V[X i ] -σ 2 i,t-1 -2V[X i ] ≤ 3 + 2V[X i ] 2 = 2 9 + 2 2V[X i ] 3 + 2V[X i ] ≤ 9 • (3.6) 2 + 2 2V[X i ] 3 • (3.6) 2 + V[X i ] 3.6 ≤ 100 + 1.77 V[X i ].
This gives a second order polynomial inequality in V[X i ], thus giving the bound

V[X i ] ≤ 0.5 • 1.77 √ + σ 2 i,t-1 + 0.8 ≤ σ 2 i,t-1 + 1.8
√ , finishing the proof of (2.13). We finally plug (2.13) into (2.10) to get:

µ * i -µ i,t-1 ≤ 2σ 2 i,t-1 + 1.8 √ 2 + 1/3 < 2σ 2 i,t-1 + 3 .
This proves the inequality of Theorem 8.

Note that in the previous Theorem 8, we manage to quantify the error committed during the empirical estimation of the true mean, using an empirical quantity itself (the empirical variance). This is essential because it allows the agent to efficiently calculate and use the exploration bonus. However, once the choice i t has been made using this empirical bonus, we wish to express a bound on the regret that can account for the advantage of using variance estimation. It may then be useful to further upper bound the empirical variance to bring out the true variance. This is the purpose of the next Proposition 3

Proposition 3 (Empirical variance concentration). Assume X i ∈ [0, 1] and let δ(t) > 1. With probability 1 -δ(t) log(t) e 1-δ(t) , we have σ 2 i,t-1 ≤ V[X i ] + 0.5δ(t) N i,t-1 .
Proof. We use the same notations than in the proof of Theorem 8 concerning σ 2 i,t-1

and . Using Theorem 6 with X i replaced by

(X i -µ * i ) 2 ∈ [0, 1],
we can consider the following event, which holds with probability 1 -δ(t) log(t) e 1-δ(t) .

σ 2 i,t-1 -V[X i ] ≤ 3 + 2V[X i ] .
The above inequality is obtained noticing that V (X i -

µ * i ) 2 ≤ V[X i ].
We assume that this event holds. Since

σ 2 i,t-1 -σ 2 i,t-1 = µ * i -µ i,t-1 2
, we can write

σ 2 i,t-1 -V[X i ] ≤ σ 2 i,t-1 -V[X i ] + µ * i -µ i,t-1 2 ≤ 3 + 2V[X i ] .
This is again a second order polynomial inequality in

V[X i ], giving V[X i ] ≥ - √ 2 + 2 /3 + 4 σ 2 i,t-1 2 or V[X i ] ≤ - √ 2 -2 /3 + 4 σ 2 i,t-1 2 .
Since the second inequality is impossible, we have

V[X i ] ≥ - √ 2 + 2 /3 + 4 σ 2 i,t-1 2 ≥ σ 2 i,t-1 - √ 2 2 .
The inequalities of Hoeffding and Bernstein are, as we have seen, obtained from a bound on the MGF. The tighter the bound, the more precise the inequality obtained. It is then legitimate to wonder if it is really necessary to bound this function, and if it is not possible to express it exactly. For some random variables (e.g., Gaussian random variables), the MGF is easy to express:

if X ∼ N (0, V[X]), E e λX = e λ 2 V[X]/2 .
However, we often wish we could consider a family that is as general as possible. Actually, in the vast majority of MAB problems, we only know that the variables are bounded. Hoeffding's idea is then to look at what this boundedness can give as an upper bound on the MGF. He discovered (see Hoeffding's lemma, Proposition 2) that bounded variables MGF are bounded by that of a Gaussian, i.e., that they were sub-Gaussian. In Lemma 1, we have seen that the MGF of a random variable in [0, 1] is bounded by that of a Bernoulli distribution. This seems better than Hoeffding's result because a Gaussian random variable is no longer bounded, whereas a Bernoulli one is. So the question is: can we simply express the MGF of a Bernoulli random variable? The answer is yes, and we'll even see that we can calculate φ * i easily in this case.

Proposition 4. If X i is a Bernoulli distribution, then φ * i (z) = kl(z, µ * i ).
Proof. We want to prove that for all z ∈ R,

sup λ λz -log E e λX i = kl(z, µ * i ). First order condition implies z = E X i e λX i E[e λX i ] = µ * i e λ µ * i e λ + (1 -µ * i ) . Thus, kl(z, µ * i ) = z log z µ * i + (1 -z) log 1 -z 1 -µ * i = z log e λ µ * i e λ + (1 -µ * i ) + (1 -z) log 1 µ * i e λ + (1 -µ * i ) = zλ -log µ * i e λ + (1 -µ * i ) .
We can thus adapt Theorem 4 together with the previous Proposition 4 and Lemma 1 to get the following Corollary 2.

Corollary 2. If X i ∈ [0, 1], then the event A t µ * i > µ i,t-1 and N i,t-1 kl µ i,t-1 , µ * i ≥ δ(t)
holds with probability lower than δ(t) log(t) e 1-δ(t) . In a same way,

A t µ * i < µ i,t-1 and N i,t-1 kl µ i,t-1 , µ * i ≥ δ(t)
holds with probability lower than δ(t) log(t) e 1-δ(t) , with δ(t) > 1.

Regret upper bounds

In this section, we focus on proving gap dependent regret bounds for the optimistic policies we previously introduced in section 2.4. To do so, we will make extensive use of the previously introduced concentration inequalities. The scheme of the proof is quite similar for the three algorithms, so we will unify the method used. More concretely, each algorithm considers, for each arm i a confidence region, which we will note C i,t . The score µ i,t of the i arm is then the sup of the region C i,t . We recall in the Table 2.2 the definition of C i,t for each of the three algorithms. The tighter the confidence region around the empirical average will be, the tighter the bound will be. Since confidence intervals are only used on one side (because the policy is characterized by the sup of the n intervals), we need a second ingredient to control the behavior of the empirical mean on the left side of the true mean. This is the purpose of Theorem 9.

Theorem 9. Let i be a sub-optimal arm. Let f be a non negative, decreasing and continuous function defined on the interval

[µ * i , µ * i * ], with f (µ * i * ) = 0. Then for all t ≤ t, ε > 0, the event A t,t N i,t-1 = t , f (µ i,t-1 )I µ i,t-1 < µ * i * < f (µ * i )/(1 + ε) holds with probability lower than exp(-t r(ε, µ * i * , µ * i )), where r(ε, µ * i * , µ * i ) is a positive constant dependent of ε, µ * i * , µ * i .
In particular, we have that =

E   t∈[T ] I i t = i, f (µ i,t-1 )I µ i,t-1 < µ * i * < f (µ * i )/(1 + ε)   = E t ∈ [T ], i t = i, f (µ i,t-1 )I µ i,t-1 < µ * i * < f (µ * i )/(1 + ε) Name of the policy The region C i,t ucb x ∈ [0, 1], 2N i,t-1 x -µ i,t-1 2 ≤ δ(t) ucb-v      x ∈ [0, 1], x -µ i,t-1 ≤ 2σ 2 i,t-1 δ(t) N i,t-1 + 3δ(t) N i,t-1      ucb-kl x ∈ [0, 1], N i,t-1 kl µ i,t-1 , x ≤ δ(t)
E[ t ∈ [T ], ∃t ≥ t , A t,t ] ≤ t ∈[T ] exp(-t r(ε, µ * i * , µ * i )) ≤ 1/(1 -e -r(ε,µ * i * ,µ * i ) ) is bounded by a constant dependent of ε, µ * i * , µ * i . Proof. We can define x(ε) as the unique solution x ∈ (µ * i , µ * i * ) to the equation f (x) = f (µ * i )/(1 + ε). Under the event A t,t , we necessarily have µ i,t-1 > x(ε), and so exp N i,t-1 (λµ i,t-1 -φ i (λ)) ≥ exp(t (λx(ε) -φ i (λ)
)) for all λ ≥ 0. By Markov inequality, the probability is thus bounded by exp(-t (λx(ε) -φ i (λ))), since the expectation of the LHS is 1:

E exp N i,t-1 (λµ i,t-1 -φ i (λ)) = E t-1 u=1 exp(I{i u = i}(λX i,u -φ i (λ))) = E E t-1 u=1 exp(I{i u = i}(λX i,u -φ i (λ))) F t-1 = E    t-2 u=1 exp(I{i u = i}(λX i,u -φ i (λ))) E[ exp(I{i t-1 = i}(λX i,t-1 -φ i (λ)))|F t-1 ] =1    = • • • = 1.
Using the fact that x(ε) > µ * i in Lemma 2, we can take the supremum of λx(ε) -φ i (λ) over λ, which gives a probability bound of exp(-t φ * i (x(ε))).

Theorem 10. Assume that rewards are bounded in

[0, 1]. Let δ(t) = log(t) + 3 log log(t), for t ≥ 3, with δ(1) = δ(2) = δ(3), and ε > 0. If π is the policy described by ucb, then R T (π) ≤ (1 + ε) i ,∆ i >0 log(T )(1 + o(1)) 2∆ i .
If π is the policy described by ucb-v, then there exists a constant c such that

R T (π) ≤ c i ,∆ i >0 log(T )(V[X i ] + ∆ i + o(1)) ∆ i .
If π is the policy described by ucb-kl, then

R T (π) ≤ (1 + ε) i ,∆ i >0 ∆ i log(T )(1 + o(1)) kl(µ * i , µ * i * )
.

Proof. Fixing any sub-optimal arm i, we can write

N i,T = 1 + T t=n+1 I{i t = i}.
The goal is to bound the expectation of this sum. Indeed, this then induces a bound on the regret using the expression given in Remark 3. Let t ≥ n + 1 be some round where i t = i. The confidence region of the optimal arm i * satisfies

P[µ * i * ∈ C i * ,t ] ≥ 1 -c δ(t) log(t) e 1-δ(t) ,
with c being a universal constant, so we assume that this event holds at round t (the regret under its complementary is o(log(T ))). In addition, we can also assume that the event

f (µ i,t-1 )I µ i,t-1 < µ * i * ≥ f (µ * i )/(1 + ε) (2.14)
holds at round t, since the regret under its complementary is bounded by a constant thanks to Theorem 9. We will chose f to be either

f (x) = kl(x, µ * i * ) or f (x) = (x -µ * i * )
2 depending if we consider the policy ucb-kl or ucb/ucb-v respectively. Using µ * i * ∈ C i * ,t and the definition of µ t , the following holds at round t

µ i,t = sup j∈[n] sup C j,t ≥ sup C i * ,t ≥ µ * i * .
Using that µ * i * ≥ µ i,t-1 (thanks to the event (2.14)), this rewrite as

kl µ i,t-1 , µ i,t ≥ kl µ i,t-1 , µ * i * , for ucb-kl and µ i,t-1 -µ i,t 2 ≥ µ i,t-1 -µ * i * 2
, for ucb/ucb-v.

The event (2.14) also gives a lower bound on the RHS of the two inequalities above:

kl µ i,t-1 , µ i,t ≥ kl(µ * i , µ * i * )/(1 + ε), for ucb-kl and µ i,t-1 -µ i,t 2 ≥ ∆ 2 i /(1 + ε), for ucb/ucb-v. If Λ f ,ε,i
denotes the square root of the RHS in the above inequalities, we can use the definition of the confidence intervals to further upper bound Λ f ,ε,i by some quantity D i,t . When the policy is ucb-v, we first use the high probability bound on the empirical variance given by Proposition 3. We thus obtain the bound with

D i,t = 2V[X i ]δ(t)/N i,t-1 + 4δ(t)/N i,t-1 . To sum up, it is sufficient to bound T t=n+1 I{i t = i, D i,t ≥ Λ f ,ε,i }.
If the case of ucb-v, we use that

I{i t = i, D i,t ≥ Λ f ,ε,i } is upper bounded by I i t = i, 2V[X i ]δ(t)/N i,t-1 ≥ Λ f ,ε,i /2 + I{i t = i, 4δ(t)/N i,t-1 ≥ Λ f ,ε,i /2},
and treat each case separately. So, let's note that all that's left to do is to bound a term of the form

T t=n+1 I i t = i, aδ(t)/Λ α f ,ε,i ≥ N i,t-1 ,
with a ≥ 0 and α ∈ {1, 2}. Noticing that i t = i implies that N i,t = N i,t-1 + 1, by bounding δ(t) by δ(T ), we get that the term above is lower than the number of integers in [1, aδ(T )/Λ α f ,ε,i ], so is bounded by aδ(T )/Λ α f ,ε,i .

We can notice that the asymptotic bounds above give a logarithmic rate of regret when T is large. One question is what happens when T is relatively small compared to the terms 1/∆ i . The minimax performance measure is then in this case more desirable. Results as Theorem 9 are not useful in this case, because although they allow to bound the regret under the corresponding event by a constant, the latter depends on the gaps ∆ i , and we rather wish to have a universal constant, even if it means having a greater multiplicative factor in front of the regret. We illustrate here how a logarithmic but not asymptotic bound on ucb can be converted into a minimax upper bound. This kind of conversion can also be derived for ucb-v and ucb-kl.

Theorem 11 (Minimax upper bound for ucb). Assume that rewards are bounded in

[0, 1]. Let δ(t) = ζ log(t), with ζ > 1. If π is the policy described by ucb, then R T (π) ≤ c + c i ,∆ i >0 log(T ) ∆ i ,
where c, c are two universal constants.

Letting η > 0, we can thus write

R T (π) = E   t∈[T ] I{∆ it ≤ η}∆ it   + E   t∈[T ] I{∆ it > η}∆ it   ≤ T η + c + c (n -1) log(T ) η . Taking η = (c (n -1) log(T )) 1/2 T -1/2 gives R T (π) ≤ c + 2(c (n -1) log(T )T ) 1/2 .
Proof. The proof is similar to the one given for Theorem 10. For a round t, we can write

P[∀i ∈ [n], µ * i ∈ C i,t ] ≥ 1 -2n δ(t) log(t) e 1-δ(t)
. We now assume that this high probability event holds at round t (the regret under the complementary is bounded by a constant independent of ∆ i ), and use it to have

µ i,t ≥ µ * i * , i.e., 2δ(t)/N i,t-1 = diam C i,t ≥ µ i,t -µ * i ≥ ∆ i , since both µ i,t and µ * i belongs to C i,t .
Exactly as for Theorem 10, we obtain the main term rate.

Some extensions

This section presents two specific extensions of the MAB problem found in the literature, namely, stochastic linear bandits and budgeted multi-armed bandit. We chose to focus on these two extensions because they find applications in stochastic combinatorial semi-bandits, as we will see in upcoming chapters.

Stochastic linear bandits

An interesting variant of MAB is the stochastic linear bandit problem, introduced by [START_REF] Auer | Using confidence bounds for exploitation-exploration trade-offs[END_REF]. In this setting, the set of arms is no longer [n] but a subset X of R n . The set X is fixed and revealed to the agent. Pulling an arm x ∈ X , the agent observes a noisy reward whose expected value is the dot product between x and an unknown parameter

θ * ∈ R n , X xt,t = x T t θ * + η t ,
where x t is the arm pulled at round t and η t is a centered noise affecting the reward observation at round step t. Usually, more details about the noise models are specified in each problem definition.

The above setting is intended to impose a linear structure on the reward received by the agent. It becomes particularly interesting in applications where the number of arms is very important compared to n. Indeed, we can see that the rewards are linearly parameterized in a fixed n-dimensional space that does not depend on the number of arms, contrary to classical MABs. On the other hand, in linear bandits, pulling one arm gives information on the parameter θ * and thus indirectly, on the reward value of the other arms. Therefore, the estimate of the average reward in MAB is replaced here by the estimate of the components of θ * . To put it plainly, by modelling such a structure, the agent will be able to take advantage not only of the dimension reduction imposed by the latent parameter, but also of the assumption of linearity of the reward.

In this context, to guide the sample allocation strategy, the agent has to rely on the estimation of θ * obtained from the observed rewards. More precisely, after observing a sequence of rewards X x 1 ,1 , . . . , X xt,t , an ordinary least squares (OLS) estimator can be defined for the sample parameter θ * as

θt V -1 t   t ∈[t] x t X x t ,t   ,
where the matrix V t t ∈[t] x t x t T is called the design matrix, and is an agglomeration of the pulling strategy up to round t. The OLS estimate enjoys a series of interesting properties, such as concentration inequalities, that we are not going to detail here, but that can be found for instance in [START_REF] Abbasi-Yadkori | Improved algorithms for linear stochastic bandits[END_REF]. It is by using the concentration inequalities on θt-1 that the bandit policies can be designed, and select which arm x t to play. Specifically, we can construct a confidence ellipsoid whose center is the empirical estimate θt-1 and which containing the true parameter θ * with high probability. The confidence ellipsoid is usually with respect to the euclidean norm associated to the design matrix V t-1 , i.e., of the form

C t θ ∈ R n , θ -θt-1 2 V t-1 ≤ δ(t) .
In fact, since the uncertainty here comes from the n components of θ * , the width of the ellipsoid in a certain direction is determined by the precision of the estimates corresponding to that direction. Thus, the goal is to be able to shrink as fast as possible the volume of the confidence ellipsoid as the agent observes more and more samples.

Similarly to the MAB setting, the policies for cumulative regret minimization can use a UCB strategy: they construct upper-bounds on the arms values using the margins of the confidence ellipsoids. Then, according to the OFU principle, the policies pull the arms with the largest index. This is a direct adaptation of the ucb policy, that start from the empirical estimate on the arm value x T θt-1 , to which an exploration bonus is added, based on the uncertainty of estimating the reward of x. Specifically, here the uncertainty is captured in the width of the confidence ellipsoid in direction x. In other word, the UCB of the arm x is

x T θt-1 + δ(t) x V -1 t-1
. This arm index construction has been introduced in [START_REF] Auer | Using confidence bounds for exploitation-exploration trade-offs[END_REF]. The good empirical performance of an index-based policy following this type of construction was shown in [START_REF] Li | A contextual-bandit approach to personalized news article recommendation[END_REF].

The construction of the arm indices can also be seen as a bilinear optimization problem max

x∈X , θ∈Ct x T θ = max x∈X x T θt-1 + δ(t) x V -1 t-1
.

We can finish by noticing that when X is large, the above optimization problem is usually difficult to solve (it is in general NP-Hard, see [START_REF] Atamtürk | Maximizing a class of utility functions over the vertices of a polytope[END_REF]). Thus, methods of the type ts may be preferred [START_REF] Agrawal | Analysis of Thompson sampling for the multi-armed bandit problem[END_REF][START_REF] Abeille | Linear Thompson sampling revisited[END_REF], because, in this context, the above optimization problem becomes a linear program over X , since instead of maximizing over θ, we rather sample this vector from a prior belief, and then simply optimize max x∈X x T θ.

Budgeted Multi-armed Bandit

In budgeted MABs, to play an arm, the agent needs to pay a cost while receiving the reward. In this new context, the agent targets at maximizing the cumulative reward under a budget constraint for the total costs. This setting has been first studied by [START_REF] Tran-Thanh | Knapsack Based Optimal Policies for Budget-Limited Multi-Armed Bandits[END_REF], [START_REF] Ding | Multi-armed bandit with budget constraint and variable costs[END_REF], [START_REF] Vanchinathan | Discovering valuable items from massive data[END_REF], and [START_REF] Xia | Budgeted bandit problems with continuous random costs[END_REF]. The budgeted MAB problem can be formally defined as follows. At round t, the agent pulls an arm i t ∈ [n], receives a random reward X it,t , and pays a random cost C it,t . Usually, we assume that both X i,t and C i,t are supported in [0, 1] for all arm i ∈ [n], and have mean µ * i and c * i respectively. The agent can keep pulling until the budget, B, runs out. B is a positive number and does not need to be known to the agent in advance. Note that we do not assume that the rewards of an arm are independent of its costs. The agent wants to follow a policy π that minimizes the regret, which is usually defined as the differences between F * B , the maximum expect cumulative reward that a pulling policy can obtain within a budget B when the reward/cost distributions of all the arms are known, and the expected reward that the policy π obtains under the budget constraint B. Formally,

R B (π) F * B -F B (π),
where, for a policy π selecting arm i t at round t,

F B (π) E τ B -1 t=1 X it,t ,
τ B being the random round at which the remaining budget becomes negative: if

B t B -t ≤t C i t ,t , then B τ B -1 ≥ 0 and B τ B < 0.
It is hard to find the optimal policy for the above setting. Even for the offline setting in which the reward and cost of each arm are deterministic and known, the problem is an unbounded knapsack problem, which is NP-hard [START_REF] Lueker | Two NP-complete problems in nonnegative integer programming[END_REF]. We can nevertheless define a quasi-optimal policy, in the sense that its regret is bounded by a constant. The quasi-optimal policy can simply be described as an index policy: as long as the budget is not exhausted, the agent pulls the arm i = arg max µ * i /c * i . As in standard MAB, this offline policy can inspire online policies. Indeed, in a learning context, the constant due to the non-optimality of the offline policy will be relatively small compared to the exploration/exploitation error that appears when B is sufficiently large (more precisely, this error is logarithmic in B). Thus, to be more specific, a ucb type policy can be used: the agent chooses in each round t the arm maximizing the ratio µ i,t /c i,t , where µ i,t is a UCB on the true mean µ * i , and c i,t is a lower confidence bound (LCB) on the true expected cost c * i . Note here that the OFU principle results in an overestimation of the gains, but in an underestimation of the costs.

Chapter 3

General Framework for Semi-Bandit Feedback

This chapter is an introduction to the framework of stochastic combinatorial multiarmed bandits with semi-bandit feedback (CMAB), called more concisely stochastic (combinatorial) semi-bandits. We organize it in the following way: first of all, we formulate the CMAB setting in a general way, thus allowing a large number of problem to be expressed. Then we give concrete examples of real-life use to motivate the framework introduced. Finally, we present some general results that will be useful to obtain regret upper bound for CMAB.

The stochastic combinatorial semi-bandits problem

The extension of MAB to the CMAB framework can be described in a few words: the agent can now choose more than one arm in the same round. We call superarm or action a set of arm that the agent is allowed to play. The set of actions is denote by A ⊂ P([n]) and is called the action space. This makes it possible to model more complex situations, where the actions given to the agent are complex and can be broken down into several small actions (the arms). In addition to the trade-off between exploration and exploitation, CMAB must also deal with the exponential explosion of possible actions that makes the exploration of all actions unfeasible. The feedback received by the agent at round t is composed of the individual feedback of each arm in the chosen super-arm A t ∈ A, i.e., the agent observes e At X t = (X i,t I{i ∈ A t }) i∈[n] (it observes only the outcomes of played arms in one round of play). The reward obtained is a function of the individual outcomes. In the majority of cases, the most natural reward to consider is linear with respect to the chosen action [START_REF] Kveton | Combinatorial Cascading Bandits[END_REF], i.e., it is the sum of the individual outcomes:

e T At X t = i∈At X i,t .
Remark 7. We can note that the distribution P X is no longer reduced to the collection of the marginals. Indeed, since several arms are played together in one round, the entire joint distribution of rewards plays a role. In particular, the correlations between the arms matter. We will come back to this point in Chapter 7.

In the literature, the CMAB setting was first considered through some specific instances of the problem. A number of studies considered simultaneous plays of any subset of m arms among the n arms that are available [START_REF] Anantharam | Asymptotically efficient allocation rules for the multiarmed bandit problem with multiple plays-Part I: IID rewards[END_REF][START_REF] Caro | Dynamic assortment with demand learning for seasonal consumer goods[END_REF][START_REF] Liu | Logarithmic weak regret of non-Bayesian restless multi-armed bandit[END_REF]. More complex scenarios include:

• The matching bandit problem [START_REF] Gai | Learning multiuser channel allocations in cognitive radio networks: A combinatorial multi-armed bandit formulation[END_REF], where the set of arms is the set of edges in a fixed bipartite graph, and A is the set of matchings in this bipartite graph (we recall that a matching is a set of edges without common vertices).

• The online shortest path problem [START_REF] Liu | Adaptive shortest-path routing under unknown and stochastically varying link states[END_REF][START_REF] Talebi | Stochastic Online Shortest Path Routing: The Value of Feedback[END_REF], where the set of arms is the set of directed edges in a fixed directed acyclic graph with a given source and destination, and A is the set of path from the source to the destination.

We will give in sections 3.2 more scenarios of application for the CMAB setting. In the following subsection we formalize more precisely the CMAB problem, incorporating a generalization that will allow us to consider more application cases: we will assume that the feedback obtained by the agent is random. This semi-bandit setting is known as Semi-Bandits with Probabilistically Triggered Arms (Chen, Wang, and Yuan, 2016;[START_REF] Wang | Improving regret bounds for combinatorial semi-bandits with probabilistically triggered arms and its applications[END_REF], and is denoted CMAB-T.

Probabilistically triggered arms

We consider the setting in which actions may trigger arms probabilistically. In this context, we denote the action space S, which is no longer necessarily a subset of P([n]), but rather an external set that we allow to be infinite. At round t, the agent selects an action S t ∈ S, based on the feedback history from the previous rounds and a possible extra source of randomness. Then, the environment draws an independent sample

X t ∼ P X , X t ∈ R n . Then, a random subset of arms A t ⊂ [n] are triggered.
The set A t depends on S t and X t , but may have additional randomness. In other words, we assume that A t is drawn independently from a distribution D trig (S t , X t ).

The outcomes of each triggered arm is observed as the feedback to the agent, i.e., e At X t is observed. The agent finally obtains a reward ρ(S t , A t , X t ). Quantities S, D trig , ρ are known to the agent. This setting notably captures the CMAB framework (without probabilistically triggered arms) described at the beginning of the section, taking S = A and D trig the Dirac measure at S t . More generally, it also encodes randomized policies, taking S equal to the set of probability measures on A, and D trig (S t , X t ) = S t . It is sometimes convenient to consider randomization as "undergone" by the agent, meaning that it is encoded in A t whereas S t is deterministic. This is the case when the good properties of the policy (such as being a solution to a certain optimization problem) are carried by the distribution S t used by the agent to generate A t rather than by A t itself. Conversely, it is sometimes more practical to consider randomization as "controlled" by the agent, meaning that it is encoded in the action S t , as in the previous chapter. This is the case when the random action S t bears the good properties of the policy (i.e., S t satisfies some criterion, such as being solution to some optimization problem, but is generated with a distribution that is not easy to describe otherwise). In any case, we can extend the definition of the filtration (F t ) as

F t σ(U 1 ) if t = 1 σ U 1 , A 1 , e A 1 X 1 , . . . , U t-1 , A t-1 , e A t-1 X t-1 , U t if t ≥ 2.
When we do not want to consider the randomization in the filtration, we use H t instead, that we call history:

H t σ(∅) if t = 1 σ U 1 , A 1 , e A 1 X 1 , . . . , U t-1 , A t-1 , e A t-1 X t-1 if t ≥ 2.
We will give in section 3.3 more examples of application for CMAB-T, where D trig also depends on the sample X t .

Remark 8. In the following, we will use the notation A t to designate the set of arms that is sampled in round t, and S t the action that is chosen by the agent. In the classical CMAB setting (without probabilistically triggered arms), we can notice that A t and S t coincides. In this case, we will often use the notation A t only, for the sake of simplicity. Nevertheless, in some contexts it is convenient to distinguish the set A t (the coordinates of the vector X t which are revealed to the agent) from the action S t (the action chosen by the agent). This is notably the case when the action S t can be described more succinctly than by the set A t of arms associated with it.

The (approximation) regret

The performance of a policy is measured by its regret, which is the difference in expected cumulative reward between always playing the best action and playing actions selected by the policy. Note, however, that even when the agent can compute exactly the expected reward E[ρ(S, A, X)] for any S ∈ S, i.e., even when the learning process of the distribution P X is completed, maximizing the quantity E[ρ(S, A, X)] for S ∈ S can be hopeless. In some cases, there exist efficient approximation algorithms, that we call oracles, outputting S ∈ S such that

E[ρ(S, A, X)] ≥ α sup S ∈S E[ρ(S , A , X)],
where α ∈ (0, 1] is an approximation ratio, and A ∼ D trig (S, X), A ∼ D trig (S , X). Under an α-approximation oracle, the benchmark cumulative reward should be the α fraction of the optimal reward. Thus to evaluate the performance of a learning policy π, we use the notion of approximation regret [START_REF] Kakade | Playing games with approximation algorithms[END_REF][START_REF] Streeter | An online algorithm for maximizing submodular functions[END_REF]Chen, Wang, and Yuan, 2016), defined as follows.

Definition 10 (Approximation regret). The T -round α-approximation regret of a learning policy π that selects action S t ∈ S at round t is defined as

R T ,α (π) E   t∈[T ] ∆(S t )   ,
where the approximation gap is defined as

∆(S) 0 ∨ α sup S ∈S E[ρ(S , A , X)] -E[ρ(S, A, X)] .
We also use the shortcut ∆ t ∆(S t ). When α = 1, meaning that one can efficiently maximize E[ρ(S, A, X)], we denote the regret as R T = R T ,1 .

Definition 11 (Reward function). Generally, the expected reward E[ρ(S, A, X)] is a function of some unknown parameter vector w * related to the distribution P X , in which case we use the notation r(S; w * ) E[ρ(S, A, X)]. The agent thus aims to minimize the (approximation) regret by learning this unknown vector. For instance, the mean vector µ * can be this parameter vector when

P X = ⊗ i∈[n] Bernoulli(µ * i ).

The function r is called the reward function. A first example (that does not involves probabilistically triggered arms) is when the reward is linear: we have r(A; µ) e T A µ, because E[e T

A X] = e T A µ * .

Other types of feedback

There exists other type of feedback in combinatorial bandits [START_REF] Audibert | Minimax Policies for Combinatorial Prediction Games[END_REF], that include: full information, in which the player observes the outcomes of all arms, and bandit, in which the player only observes the final reward but no outcome of any individual arm. More complicated feedback dependences are also considered in [START_REF] Mannor | From bandits to experts: On the value of side-observations[END_REF]. In the whole thesis, we focus on the semi-bandit feedback we introduced above.

Real world applications of CMAB

CMAB problems are driven by a wide range of real-world situations. Although many models are quite simplistic to fully solve real world scenarios, they provide a very good benchmark for providing effective solutions to these problems. In this section, we will describe some examples of applications of the CMAB framework. We recall that the feedback is semi-bandit: the agent observes the outcomes of the arms belonging to the chosen action S t = A t . The last two applications (about maximum coverage) are examples where it is convenient to have S t = A t , although they do not involves probabilistically triggered arms.

In each of the following settings (except for the last two), note that we have the choice to take a linear reward function depending on the need and the context. When we choose a linear reward function, we take α = 1 in the definition of the regret, because there is an offline algorithm that can optimize exactly the reward function in a time polynomial in n. For example, the maximum weighted bipartite matching problem (also called the assignment problem) can be solved in polynomial time using the Hungarian algorithm [START_REF] Kuhn | The Hungarian method for the assignment problem[END_REF]. For the shortest path problem, many efficient algorithms exists, and the most important are the Dijkstra's algorithm [START_REF] Dijkstra | A note on two problems in connexion with graphs[END_REF] (with non-negative edge weight) and the Bellman-Ford algorithm (see e.g., [START_REF] Jukna | On the optimality of Bellman-Ford-Moore shortest path algorithm[END_REF]).

More on the matching bandit problem

There is considerable interest in the development of mechanisms to access the digital spectrum for more efficient use (as we saw earlier for the application of MAB to cognitive radios). Indeed, cognitive radio networks, characterised by higher levels of autonomy, intelligence and learning, are expected to play an important role in this area. The matching bandit setting described in the previous section can model a type of problem that can occur when optimizing the use of the communication network. More precisely, the bipartite graph might represent, on the one hand, the users, and on the other hand, the channels in which these users want to transmit. At each round, a central instance (the agent) must assign each user to a channel, so that there is no collision (i.e., if an user is assigned to some channel, there is no other conflicting user on that channel). Notice that we thus implicitly assume that there are more channels than users. Each edge of the graph is associated to a random outcome, and the goal for the agent is to choose its matching so as to maximize the sum over the edges belonging to the matching of the corresponding outcomes (notice, the reward function is thus linear here). These outcomes represent the quality of the transmission made by the user. The difference between the outcomes encodes the geographical dispersion of the users: the channels are more or less accessible depending on many factor related to the user behaviour. We give in Figure 3.1 an example of a bipartite graph encoding an instance of the setting just described. Another problem falling into the same type of combinatorial optimization is the following: the agent is an online dating site, and it is earning at each round an uncertain gain for each match made in that round. In addition to the dating market, we can imagine other scenarios where finding a maximum weight match is a goal for the agent.

More on the online shortest path problem

Routing is the mechanism by which paths are selected in a network to route data from a sender to one or more recipients. Broadly, routing is performed in many types of networks, including circuit-switched networks, such as the public switched telephone network (PSTN), and computer networks, such as the Internet. Its performance is important in decentralized networks, i.e., where information is not distributed by a single source, but exchanged between independent nodes. Here, we consider, as introduced in the previous section, a multi-hop communication network represented by a directed graph. Assume that we have to send a stream of packets from the source node to the destination node. At each round, a packet is sent along a chosen route connecting the source to the destination. Depending on the network traffic (and other factors, such as signal strength in wireless networks), each edge in the network may have a different delay (the random outcome is usually minus the delay, to transform the minimization problem into a maximization one), and the total delay the packet suffers on the chosen route is the sum of delays of the edges composing the route (i.e., again, the reward function is linear).

The setting can also benefit when searching for the shortest route in a road network. With increasing urbanisation and the growth of cities with several million inhabitants, road networks are under great pressure. The problem of traffic allocation concerns the allocation of routes on a network that seeks to minimise a cost defined by the instance of the problem (normally travel time is used). In order to assign the shortest route to a user, the route planner (the agent) can use the outputs of the previous recommendations (such as the speed of the vehicle along the route) to optimize the proposed itinerary.

Dynamic assortment

We consider a price-taking retailer (the agent) that has n products to sale. We assume that each product has a fixed known marginal profit resulting from selling it. For instance, the marginal profit can be the price at which the product is sold minus the marginal cost paid by the agent for offering the product. At each round, a customer arrives, with some unknown random valuation vector over products. Then, the agent offers any subset of m products (due to display space constraints, the retailer can offer at most m products simultaneously), and the customer buys an offered product if and only if its valuation is greater than its price. The agent is interested in maximizing the total profit (revenue minus cost) from sales over T rounds, where T denotes the total number of customers that arrive during the selling season. This setting have been studied for instance in [START_REF] Sauré | Optimal dynamic assortment planning with demand learning[END_REF].

Web page optimization

Building web pages that match user preferences is a major issue in a variety of situations: online advertising is an important application, as is the customization of home pages or search engine results [START_REF] Lagrée | Multiple-play bandits in the position-based model[END_REF]. Learning how to place items in multi-position displays or lists is a task that can be integrated into the semi-banded framework. To be more precise, our agent here is the designer of the web page in question. At the beginning of each round (a round can be for instance an hour, a day, or simply a user reaching the page), the agent chooses a design for the web page: it chooses which elements (clickable) and where to place them. At the end of the round, the agent observes the outcomes corresponding to the number of clicks (or the click-through-rate) on each element placed, and can update the design for a new round. The agent's goal is to maximize the total number of clicks. In this setting, arms are pairs (item, position), and super-arms are a set of m arms with pairwise disjoint positions and items. Depending on the setting considered, several types of reward functions can be used: we can for example associate to each pair (item, position) an outcome, in which case we have a linear reward function. Note that this setting is similar to the matching bandit setting, where the items form one part of the bipartite graph, and the positions form the second part.

Crowdsourcing

Crowdsourcing has received considerable attention in recent years and many applications have been successful, such as gathering information quickly during a disaster, performing tasks that are difficult to automate and need to be solved by human workers, conducting large-scale surveys or contributing to scientific projects. There are at least two types of crowdsourced tasks: micro-tasks, uniformly priced at a few cents, and expert tasks, where the employer (the agent) has much more control over the selection of individual workers, and must also take into account the potentially very heterogeneous and higher costs (workers often charge 10 to 50 per hour). To address the specific challenges of crowdsourcing experts, the agent can see workers as arms. This set of workers is usually determined by an open call for participation by the employer, to which qualified and available workers respond. Assigning a single task to a worker can be regarded as pulling the arm. This incurs a cost that is set by the worker, and the assignment produces an outcome. The agent has a total budget of B to spend on crowdsourcing the tasks, and wishes to maximize the overall sum of the outcomes (notice, we thus here also fall into the budgeted bandit setting described in the end of the previous chapter). Last but not least, the agent would also like to have completed the tasks in a minimum amount of time. Thus, at each round, the agent pays an additional fixed cost, modeling the time. In this context, the agent should therefore assign tasks to several workers in a single round, in order to go faster, while taking into account the efficiency of each agent. A similar setting have been considered in [START_REF] Tran-Thanh | Efficient crowdsourcing of unknown experts using bounded multi-armed bandits[END_REF].

Real-time strategy games

We have already seen that MCTS approaches such as uct are useful for exploring game trees. These techniques have been successful for games with a moderate branching factor. For real-time strategy (RTS) games, where the magnitude of the branching factor is combinatorial due to the fact that multiple units can be issued actions simultaneously, a CMAB approach may be preferable. Indeed, RTS games are adversarial games, generally simulating battles or complex interactions between a large number of units (workers, military units, ...), that pose a significant challenge to both human and artificial intelligence (because of the enormous action and state space, and because they are real time). In addition, some RTS games are also partially observable and non-deterministic, which further motivates the use of CMABs. More specifically, the use of MABs in MCTS alorithms was useful to explore and evaluate the quality of the actions available. As in the context of RTS games, the available actions are broken down into several local actions, the MAB approach naturally translates into a CMAB approach: the agent benefits from the estimation of a local action (i.e., an arm) in each super-arm containing it. The internal structure of the setting is thus exploited. The above setting have been considered in [START_REF] Ontanón | The combinatorial multi-armed bandit problem and its application to real-time strategy games[END_REF].

It should be noted, however, that the reward function may be difficult to capture in many RTS games, and that it is generally not linear, as specific combinations of local actions may lead to specific rewards. On the other hand, the assumption of semi-bandit feedback is also debatable. Indeed, the outcome of a played arm is not always observed individually, especially when other arms interfere with the perceived outcome (which can be, depending on the situation, a gain of resources, the destruction of strategic buildings, ...). Nevertheless, it can be argued that only the current state of the game (which does not change during MC simulations) influences the outcome that a certain arm produces.

Probabilistic maximum coverage bandit

The probabilistic maximum coverage (PMC) bandit problem is the bandit version of the maximum coverage problem (Hochbaum, 1997). As stated previously, it is an example where it is practical to distinguish the chosen action S t from the set of arms A t that is associated to that action. The problem can be described as follows: consider a bipartite graph G = (L, R, E), with E ⊂ L × R, where each edge ij is an arm and has an unknown probability µ * ij ∈ [0, 1] associated to it. The action space S is the collection of subsets of L of size m. The random vector

X ∈ R E is sampled from ⊗ ij∈E Bernoulli µ * ij . Each time an action S is chosen, the set of arms A = E ∩ (S × R) = {ij ∈ E : i ∈ S} is observed. The reward is the cardinality of the set {j : ij ∈ A, X ij = 1}, i.e.
, is the number of right hand vertex that are covered by a node from the action S (see Figure 3.2). Notice thus that the reward function (i.e., the expected reward) is Edges ij represented with bold lines are those with X ij = 1, and the others, represented with dash lines, are such that X ij = 0. A bold edge adjacent to a red node makes the other node of that edge blue. Thus, blue nodes are those covered by the red ones. The reward is the number of blue nodes.

r(S; µ) j∈R   1 - i∈S: ij∈E (1 -µ ij )   .
As a function of S, the reward function r is monotone, meaning that r(S; µ) ≥ r(S ; µ) for S ⊂ S, and submodular [START_REF] Fujishige | Submodular functions and optimization[END_REF], meaning that for any pair of sets S and S , we have r(S; µ) + r(S ; µ) ≥ r(S ∪ S ; µ) + r(S ∩ S ; µ).

Submodularity will be further studied in chapters 5 and 6. We can say here that the problem of maximizing r, for a certain parameter µ, over subsets of L of size m, is inapproximable within a factor better than 1 -1/e, unless P = NP [START_REF] Feige | A threshold of ln n for approximating set cover[END_REF]. The greedy algorithm for maximum coverage chooses sets according to one rule: at each stage (and until the solution is of size m), add the L node to the current solution which covers the largest number of uncovered elements. It can be shown that this algorithm achieves an approximation ratio of 1 -1/e [START_REF] Hochbaum | Approximating covering and packing problems: set cover, vertex cover, independent set, and related problems[END_REF]. In conclusion, in the above Definition 10 of the approximation regret, we take α = 1 -1/e for the PMC bandit problem.

PMC bandit have been studied by [START_REF] Chen | Combinatorial Multi-Armed Bandit: General Framework and Applications[END_REF], and more recently by [START_REF] Merlis | Batch-Size Independent Regret Bounds for the Combinatorial Multi-Armed Bandit Problem[END_REF]. Several applications of PMC exist [START_REF] Mcgregor | Better streaming algorithms for the maximum coverage problem[END_REF], such as sensor allocation (Krause and Guestrin, 2007), information retrieval [START_REF] Anagnostopoulos | Stochastic query covering for fast approximate document retrieval[END_REF], ad placement [START_REF] Radlinski | Learning diverse rankings with multi-armed bandits[END_REF], influencer marketing [START_REF] Kempe | Maximizing the spread of influence through a social network[END_REF] or blog monitoring [START_REF] Saha | On maximum coverage in the streaming model & application to multi-topic blog-watch[END_REF]. We give here one real-world application of the PMC framework introduced above.

Content delivery network A content delivery network (CDN) [START_REF] Pathan | Content delivery networks: State of the art, insights, and imperatives[END_REF]) is a collaborative collection of network elements spanning the Internet and arranged for more effective delivery of the content. In a CDN, content is replicated over several mirrored Web servers in order to perform transparent and effective delivery of content to the end users. The mirrored servers are strategically placed at various locations to deal with sudden spikes in the requests. A critical aspect in the operation of a CDN is how contents are placed on surrogate servers.

Ideally, content is placed on a set of surrogate servers that should be chosen to maximize the total number of requests served. For each server hosting the content, the list of users who have made a request to access the content from that server is known, and may change on the fly. The same user can send a request to several servers when possible. The PMC bandit setting can be used to sequentially select the set of servers (L nodes). The rounds represent time steps, and in each round t, the agent selects a set of m servers (m stands for a constraint on the maximum number of servers that can be used). Then, for each selected server, the list of users (R nodes) submitting a request to that server within the current round is received (modeled by X ij,t ). The goal is to maximize the total number of users covered (summing over all rounds).

Weighted maximum coverage bandit

For some applications, in the PMC bandit problem introduced above, the sets of R nodes covered by a selected L node is not random. On the other hand, the gain associated with each node can be random (no longer equal to 1). This kind of scenario can happen for example in the above CDN example: let's imagine that surrogate servers are connected to several infrastructures (deterministically), but that the number of requests linked to these infrastructures is random, and does not depend on the type of server connected to the infrastructure. In this kind of context, the arms are no longer the edges, but the R nodes, and the outcomes are the random gain (i.e., the number of requests) associated to them. It is thus easier to learn how to optimize the coverage because the number of parameters to learn is no longer |L| × |R|, but only |R|. Moreover, the set R can in principle be much smaller than before, because users are grouped into infrastructures (one can imagine that the total number of infrastructures can potentially be much smaller than the total number of users). If S is a set of m nodes from L, and if w ∈ R R + is a parameter vector standing for the mean of the R node outcomes, then the reward function is defined as

r(S; w) j∈R w j I{S is connected to j}.
This function is still monotone and submodular in S, and for the same reasons as for the PMC bandit, we take α = 1 -1/e in Definition 10.

Real world applications of CMAB-T

In this section, we give some concrete examples of use of the CMAB-T setting. We recall that S t and A t are always different in this case: A t is an element of P([n]), while S t belongs to an action space S that may not be a subset of P([n]).

Cascading bandits

Cascading bandits have been studied in [START_REF] Kveton | Cascading Bandits: Learning to Rank in the Cascade Model[END_REF], Kveton, Wen, Ashkan, and[START_REF] Kveton | Combinatorial Cascading Bandits[END_REF][START_REF] Li | Contextual Combinatorial Cascading Bandits[END_REF], and can be introduced as follows. There are n arms, associated with independent Bernoulli outcomes. An action is any ordered sequence of arms of cardinality m. Playing an action means that the outcomes of the arms are revealed one by one following the sequence order until certain stopping condition is satisfied. The feedback is the outcomes of revealed arms and the reward is a function of these arms. There are two main forms for this problem:

• The disjunctive form: the agent stops when the first 1 is revealed and gains reward of 1, or it reaches the end and gains reward 0. The reward function is thus of the form r(S;

µ) 1 - i∈S (1 -µ i ).
The triggered set of arms A is a prefix set of the action S = (s 1 , . . . , s m ):

A = {s 1 , . . . , s k }, such that k is the first in S with X s k = 1.
• The conjunctive form: the agent stops when the first 0 is revealed (and receives reward 0) or it reaches the end with all 1 outcomes (and receives reward 1). The reward function is thus of the form r(S; µ)

i∈S µ i .
The triggered set of arms

A is {s 1 , . . . , s k }, such that k is the first in S with X s k = 0.
Cascading bandits can be used to model online recommendation and advertising. In this context, the agent recommend a list of m items to a user (such as web pages).

The user examines the recommended list from the first item to the last, and selects the first attractive item (i.e., click on it for web search), or don't select any item if none is attractive. This is an example of the disjunctive form with an outcome 1 for each attractive item, and a reward 1 if a click occurs. Another application is network routing reliability. In this context, the action S represents of a path to route data from a sender to a recipients. The goal for the agent is to be able to route the data using a path that does not contain a defective arm (i.e., a broken edge). This is an example of the conjunctive form with an outcome 0 for each broken arm, and a reward of 1 when the data was successfully transmitted (i.e., no broken edge were present in the selected path).

Let us note finally that the functions r described here are maximized exactly by taking S containing the m arms having the largest means (we can thus take α = 1 in the regret).

Weighted probabilistic maximum coverage bandit

We can mix the two maximum coverage bandit problems introduced in the previous section. In this new setting, we have random weights associated with R nodes on the top of the PMC bandit setting. Even if both problems were instances of CMAB, their association becomes a CMAB-T problem, because the random weights associated with R nodes are only observed if there is a selected L node covering it: there is therefore randomness in the feedback the agent receives. More precisely, our set of arms here is E, but it should be noted that each arm ij ∈ E has two outcomes: a Bernoulli variable X ij ∈ {0, 1} encoding if the node i covers the node j, and another variable W j ∈ R + encoding the weight associated with the node j. We assume that P (W,X) = P W ⊗ ⊗ ij∈E P X ij , i.e., weights are assume to be independent from X (note, however, that the weights are not assumed to be mutually independent). When an action S t ⊂ L is played at round t, the feedback received is

(X ij,t I{i ∈ S}), (W j,t I{∃i ∈ S, X ij,t = 1}).
We can thus see that the feedback set associated with X t is not random (it is the same as for the PMC bandit problem), whereas the feedback set associated with W t is: the randomness is given by X t itself.

Using the independence assumption we can see that the reward function is

r(S; µ, w) j∈R w j   1 - i∈S: ij∈E (1 -µ ij )   .
As before, r is submudular, and we can define the approximation regret with α = 1 -1/e. The types of scenarios in which this setting can occur are the same as those in the PMC bandit problem, but where not every node has the same importance to be covered. In the context of CDN, this can happen, for example, when users are heterogeneous (they do not have the same type of subscription, they are willing to pay more for access to content, ...).

Online influence maximization

Influence maximization (IM) is the problem of finding a small set of most influential nodes in a social network so that their aggregated influence in the network is maximized. Online influence maximization (OIM) is the bandit version of this problem (Chen, Wang, and Yuan, 2016;[START_REF] Wen | Online influence maximization under independent cascade model with semi-bandit feedback[END_REF]. We refer the reader to Chapter 6 for further details about OIM. In IM [START_REF] Kempe | Maximizing the spread of influence through a social network[END_REF], we are given a directed graph G = (V , E), where V and E are sets of vertices and edges respectively. Each edge ij represents a connections between the two users i and j (e.g., i follows j in the social network). An underlying diffusion model D governs how information spreads in G. More precisely, D is a probability distribution on subgraphs 1 G of G. Several types of distributions D exist in the literature, and the two best known are as follows:

• Independent cascade model (IC): the random subgraph G = G W is G W (V , {ij ∈ E : W ij = 1}), where P W ⊗ ij∈E Bernoulli w * ij . • Linear threshold model (LT): the random subgraph G is G W , with P W ⊗ j∈V Multinoulli w * ij i: ij∈E
, i.e., for every j ∈ V , select at most one of its incoming edges at random, such that edge ij is selected with probability w * ij , and no edge is selected with probability 1

-i: ij∈E w * ij ≥ 0.
Given some seed set S, the spread σ(S; w) is defined as the expected number of Sreachable nodes in G ∼ D (i.e., the number of nodes in G that are reachable from some node in S), where D is a distribution parameterized by w as in the IC and LT models. IM aims to find S that is a solution to max |S|=m σ(S; w).

(3.1)

Although IM is NP-hard under standard diffusion models -i.e., IC and LTσ is a monotone submodular function of S. In particular, Kempe, Kleinberg, and Tardos

1 A subgraph of a graph G is obtained by removing some edges from G.

(2015) provide a greedy algorithm with an approximation ratio of α = 1 -1/e -ε for all ε > 0.

For OIM, the distribution D is unknown and is parameterized by an unknown vector w * . D must be learned over time by repeated influence maximization tasks: at each round t, m seed nodes S t are selected, the influence propagation of S t is observed and the reward is the number of nodes activated in that round. The agent wants to repeat this process to accumulate as much reward as possible. More precisely, the set of arms is the set of edges E, the outcomes are W t and the feedback set is

A t = ij ∈ E : S t Wt i ,
= G Wt . The reward ρ(S, A, W)
is the number of nodes that is reached from S through G , and the expected reward is exactly the influence spread σ(S; w * ).

General technical results: toward proving regret upper bounds

In this section, now that we have reviewed many applications of the CMAB and CMAB-T settings, we will begin to see how an upper bound on the regret of a policy can be proven. In all the thesis, we are only interested in the rate that regret has up to a multiplicative universal factor. In other words, we are interested in proving upper bound of the form

R T (π) = O(f (T )),
with f being a function that may depend on the input of the problem. By O, we mean that the ratio R T (π)/f (T ) must be bounded by a universal constant for all T ∈ N * . It should be noted that this section contains some fairly technical derivations, and that the reader who wants to go through the thesis quickly can skip the rest of the chapter to go to the next Chapter 4. The results that are presented here will find applications throughout the thesis, and are stated in a general enough way so that they can be used in a large number of applications, that may go beyond the scope of the thesis. In this section, for the sake of generality, and to cover also the budgeted bandit settings, we assume that the horizon T may be random. Let's recall the definition of the counters:

∀i ∈ [n], ∀t ≥ 1, N i,t-1 = t-1 t =1 I{i ∈ A t }.
There are also some other definitions that will be useful for the analysis and the regret bound expression.

Definition 12 (Triggering probabilities). For an arm i ∈ [n] and an action S ∈ S, we define the probability that action S triggers arm i as

p i (S) P[i ∈ A],
where A ∼ D trig (S, X). In particular, since S t ∈ F t , we have

p i (S t ) = P[ i ∈ A t |F t ].
Several other quantities can be defined from the gaps and p i (S):

∀i ∈ [n], ∆ i,min min S∈S: p i (S)>0, ∆(S)>0 ∆(S), ∀i ∈ [n], ∆ i,max max S∈S, p i (S)>0, ∆(S)>0 ∆(S).
As a convention, if there is no action S such that p i (S) > 0 and ∆(S) > 0, we define ∆ i,min = +∞ and ∆ i,max = 0. We also define ∆ min min i∈[n] ∆ i,min and ∆ max max i∈[n] ∆ i,max . We define the maximum number of arms that can be triggered together with some arm i as

m i max S∈S, p i (S)>0 j∈[n]
I{p j (S) > 0}.

Finally, we let b i (S t ) ≥ 0 be some quantity depending on i and S t , which we can choose according to our needs.

How to prove regret bounds?

As we saw in the previous chapter, the regret analysis of an MAB policy is based on high probability events. It will be the same for CMAB-T policies. To recall, at a round t, the possibility space for this round is divided into several mutually exclusive and exhaustive events, and the regret is filtered against each of these events. We thus obtain a sum of several event-filtered regrets. The goal is to upper bound each of these terms. One term dominates the others when T is large. It is called the leading term and it summarizes the regret upper bound rate. In a sense, therefore, the other events aim to restrict the possibilities -without their filtered regret being too large -in order to gather the right conditions to prove the leading term upper bound. Generally, the leading term is obtained by bounding ∆ t by an error term that is easier to apprehend when summed over t ∈ [T ]. The way to bound ∆ t by an error term depends on the policy that is under consideration. As for index-based MAB policies, there is a usual form for CMAB-T policies, that can be described as follows:

• Consider an estimate µ t of the true mean at round t.

• Plug µ t into an oracle that α-approximately maximizes S → r(S; µ t ).

Under the right conditions, the estimate µ t is "close enough" to µ * so that we can bound ∆ t by α sup

S ∈S r(S ; µ t ) -r(S t ; µ * ).
Using the definition of the oracle, this is further bounded by

r(S t ; µ t ) -r(S t ; µ * ). (3.2)
The type of error term that we obtain in the end therefore depends on how "close enough" µ t is to µ * . More precisely, it is necessary to arbitrate between a µ t fairly optimistic so as to provide the first bound on ∆ t , and a µ t not too optimistic so as to reduce the gap (3.2) as much as possible. If, to simplify, we leave optimism aside, the choice of the empirical average µ t = µ t-1 seems obvious. The gap (3.2) is thus controlled by the concentration of µ t-1 to µ * . Therefore, we generally refer to the error term as a bonus, even if it is not explicitly used in the policy (or if the policy is not even optimistic). The purpose of this section is to see what kind of regret bound we can get depending on the type of bonus we have.

An example of bonus building

There are many ways to proceed in order to build a bonus. Generally speaking, there are two basic ingredients at work. First, a smoothness relation to link the variation of the objective function r(S; µ) to the variation of the parameter µ. This smoothness relation provides a bound on (3.2). Then, a concentration inequality, in order to limit the variation of the parameter. In the context of CMAB-T, a widespread smoothness relation allowing to treat objective functions resembling to linear functions is the following 1 -norm triggering probability modulated condition [START_REF] Wang | Improving regret bounds for combinatorial semi-bandits with probabilistically triggered arms and its applications[END_REF]:

∀S ∈ S, ∀µ, µ ∈ R n , r(S; µ) -r(S; µ ) ≤ B i p i (S) µ i -µ i ,
where B is a constant. For the right choice of B, this relation holds in all the CMAB-T problems we considered previously. In the following, we give an example of bonus built from such relation for the reward function. To do so, we use a confidence region on the parameter vector together with the following Hölder's inequality for p ≥ 1:

i p i (S t ) µ i,t-1 -µ * i ≤ p i (S t ) N 1/2 i,t-1 i p N 1/2 i,t-1 µ i,t-1 -µ * i i p p-1 (3.3)
.

Here, the confidence region is defined as a bound on (3.3). For simplicity's sake, let's assume that outcomes received are Gaussian of unit variance, and that N i,t-1 is not random (thus, the vector inside the p/(p -1)-norm have N (0, 1) components). Furthermore, we consider two cases: arbitrary correlated and independent outcomes. For arbitrary correlated outcomes, we take p = 1 and get that (3.3) =

O log(t) with high probability.2 For independent outcomes, we take p = 2 so that (3.3) follows a χ distribution. Again, with high probability, we have [START_REF] Laurent | Adaptive estimation of a quadratic functional by model selection[END_REF]. In any case, (3.3) is bounded by O log(t) with high probability. To sum up, the bonus is an p -norm of some error vector of the form "probability of observing i" (here, p i (S t )) times "error associated with i" (here, c log(t)/N i,t-1 for some constant c). In what follows we consider a more general form for the error vector.

(3.3) ≤ n + 2 n log(t) + 2 log(t) = O log(t)

Several types of bonuses, several types of regret rates

In this subsection, we bound the regret for various types of bonuses. We rely on some results that we will state later in the "appendix" subsection 3.4.3.

Initialization As we will notice, in the propositions from subsection 3.4.3, the counters start at 1. There are several ways to deal with the case where some counters are 0. A first possibility is to use a few rounds in order to properly initialize all the counters, for example, for t = i, by choosing S t such that p i (S t ) = 1, we are assured that after n rounds, all the counters are at least equal to 1. Thus, such an initialization only adds the additive term i∈[n] ∆ i,max to the regret. Another possibility is when an a priori bound on ∆ t of the form i∈[n] p i (S t )K i is known. This is the purpose of the following proposition.

Proposition 5 (Initialization). For all i ∈ [n], let K i ∈ R + . For all t ≥ 1, consider the event

A t    ∆ t ≤ i∈[n], N i,t-1 =0 p i (S t )K i    . Then, if {t ≤ T } ∈ F t , the event-filtered regret E T t=1 ∆ t I{A t } is upper bounded by i∈[n] K i . Proof. Since I{t ≤ T } ∈ F t and by definition of p i (S t ) = P[ i ∈ A t |F t ], we have E T t=1 I{A t }∆ t ≤ E   T t=1 E   i∈[n] I{N i,t-1 = 0, i ∈ A t } F t   K i   .
Since the counter N i,t-1 is updated as soon as i ∈ A t , we have that the event can occur for at most one round, giving the upper bound i∈[n] K i .

Analysis for 1 -bonuses We give here an analysis in the case where the bonus is built from the 1 -norm of some error vector. As we will see in Chapter 4, bonuses based on the 1 -norm are obtained with a hypercube-type confidence region. This kind of confidence region is always possible to establish under the right concentration conditions for the marginals. Our formulation here is general enough to include many types of error vectors, and it will be easy to adapt each parameter to our needs. In the proof of the following Theorem 12, we use a technique called reverse amortisation [START_REF] Wang | Improving regret bounds for combinatorial semi-bandits with probabilistically triggered arms and its applications[END_REF], which replaces the analysis from (Kveton, Wen, Ashkan, and Szepesvari, 2015b). Reverse amortisation is used to transform a bound on ∆ t by considering only the error components that are large enough. It is based on the following observation: for a random variable Z such that E[Z] ≥ 0, we have

E[Z] ≤ E[2ZI{2Z ≥ E[Z]}]. This is tight, since taking Z = 1/2 -ε with probability 1 -ε ε -1 (1 -(1/2 -ε)(1 -ε)) with probability ε, we have E[Z] = 1 and E[2ZI{2Z ≥ E[Z]}] = 1 + O(ε).
The fast reader can skip the proof of the following theorem to directly go to Theorem 13.

Theorem 12 (Regret bound for

1 -bonus). For all i ∈ [n], let (α i , β i,T ) ∈ (0, 1] × R + . For all t ≥ 1, consider the event A t    ∆ t ≤ i∈[n], N i,t-1 >0 p i (S t )b i (S t )β α i i,T e i N α i i,t-1 1    . Then, if {t ≤ T } ∈ F t , the event-filtered regret E T t=1 ∆ t I{A t } is upper bounded by i∈[n] E[β i,T ]γ i I{α i = 1}2 2 + log ∆ i,max γ i δ i,min + I{α i < 1} 2 1/α i 1 -α i δ i,min 1-1/α i ,
where

γ i = max S∈S, p i (S)>0 b i (S), δ i,min = min S∈S, p i (S)>0 ∆(S) j∈[n] p j (S)b j (S) , δ i,max = max S∈S, p i (S)>0 ∆(S) j∈[n] p j (S)b j (S)
.

Proof. Let t ≥ 1. The first step is the reverse amortisation technique, that allows us to modify the upper bound on ∆ t in such a way that indices i such that N i,t-1 is high enough are removed. Assuming that A t holds, we get

∆ t ≤ -∆ t + i∈[n], N i,t-1 >0 2p i (S t )b i (S t )β α i i,T e i N α i i,t-1 1 (3.4) = - i∈[n] ∆ t p i (S t )b i (S t ) i∈[n] p i (S t )b i (S t ) + i∈[n], N i,t-1 >0 2p i (S t )b i (S t )β α i i,T N α i i,t-1 ≤ i∈[n] p i (S t )b i (S t )0 ∨ 2I{N i,t-1 > 0}β α i i,T N α i i,t-1 - ∆ t j∈[n] p j (S t )b j (S t ) (3.5) ≤ i∈[n], N i,t-1 >0 I 2β α i i,T N α i i,t-1 ≥ ∆ t j∈[n] p j (S t )b j (S t ) 2p i (S t )γ i β α i i,T N α i i,t-1 (3.6)
where (3.4) uses the event A t , (3.5) uses the sub-additivity of x → 0 ∨ x, (3.6) uses the fact that 0

∨ (x -y) ≤ xI{x ≥ y} if y ≥ 0 and x ∈ R. Now, by definition of p i (S t ) = P[ i ∈ A t |F t ], we can get from (3.6) that ∆ t is upper bounded by i∈[n] E I i ∈ A t , p i (S t ), ∆ t , N i,t-1 > 0, 2β α i i,T N α i i,t-1 ≥ ∆ t j∈[n] p j (S t )b j (S t ) 2γ i β α i i,T N α i i,t-1 F t .
Letting f i (x) = β i,T 2 1/α i x -1/α i , the previous upper bound rewrites in the following way, with δ t = ∆ t / j∈[n] p j (S t )b j (S t ):

∆ t ≤ i∈[n] E γ i I{i ∈ A t , p i (S t ), ∆ t , N i,t-1 > 0, N i,t-1 ≤ f i (δ t )}f -1 i (N i,t-1 ) F t .
Now, we want to apply Proposition 8 and Proposition 9 from the next subsection 3.4.3.

To do so, we distinguish two cases:

α i = 1 and α i < 1. i∈[n] γ i I{α i = 1, i ∈ A t , p i (S t ), ∆ t , N i,t-1 > 0, N i,t-1 ≤ f i (δ t )}f -1 i (N i,t-1 ) (3.7)t + i∈[n] γ i I{α i < 1, i ∈ A t , p i (S t ), ∆ t , N i,t-1 > 0, N i,t-1 ≤ f i (δ t )}f -1 i (N i,t-1 ) (3.8)t
.

We consider the event

B t ∀i ∈ A t such that α i = 1, p i (S t ) > 0, we have N i,t-1 > f i ∆ i,max γ i
that allows us to write the following upper bound

I{A t }∆ t ≤ E[ (3.8) t + I{B t }(3.7) t + I{¬B t }∆ t |F t ].
Since I{t ≤ T } ∈ F t , we have

E T t=1 I{A t }∆ t ≤ E T t=1 E[ (3.8) t + I{B t }(3.7) t + I{¬B t }∆ t |F t ] = E T t=1 (3.8) t + I{B t }(3.7) t + I{¬B t }∆ t .
Using Proposition 9, we handle the sum over t of the first term:

E T t=1 (3.8) t ≤ i∈[n] γ i I{α i < 1} 2 1/α i E[β i,T ] 1 -α i δ i,min 1-1/α i .
Using Proposition 8, we can handle the sum over t of the second term:

E T t=1 (3.7) t I{B t } ≤ i∈[n] I{α i = 1}2γ i E[β i,T ] 1 + log ∆ i,max γ i δ i,min .
The choice ∆ i,max /γ i is justified when we handle T t=1 ∆ t I{¬B t }:

∆ t I{¬B t } ≤ i∈At,α i =1 I p i (S t ) > 0, 0 < N i,t-1 ≤ f i 1 γ i ∆ i,max ∆ t ≤ i∈At, α i =1 I p i (S t ) > 0, 0 < N i,t-1 ≤ f i 1 γ i ∆ i,max ∆ i,max .
So, by summing over t ∈ [T ], we get that T t=1 ∆ t I{¬B t } is upper bounded by

i∈[n], α i =1 ∆ i,max T t=1 I i ∈ A t , 0 < N i,t-1 ≤ f i 1 γ i ∆ i,max ≤ i∈[n] ∆ i,max f i 1 γ i ∆ i,max I{α i = 1} = i∈[n] 2γ i β i,T I{α i = 1}.
In summary, we have that

E T t=1 I{A t }∆ t is upper bounded by i∈[n] E[β i,T ]γ i I{α i = 1}2 2 + log ∆ i,max γ i δ i,min + I{α i < 1} 2 1/α i 1 -α i δ i,min 1-1/α i .
Analysis for 2 -bonuses We now give the analysis in the case where the bonus is based on the 2 -norm of the error vector. We will see in Chapter 5 that this kind of bonus appears naturally when the confidence region is ellipsoidal. In this case, there are several possibilities concerning the form of the bonus in question. Indeed, contrary to the 1 -norm, the 2 -norm does not commute with the expectation (on the randomness of A t ). Here we give three different results according to the "position" of the expectation in the bonus. Each of the three results will find applications within the thesis. The first result concerns the case where the expectation is within the norm. It generalizes the result provided in Degenne and Perchet (2016b) by incorporating it in the CMAB-T framework (their result was only stated within the CMAB setting). In addition, we provide here a simpler proof, like the simplification that Wang and Chen (2017) had proposed on the Kveton, Wen, Ashkan, and Szepesvari (2015b) method for 1 type bonuses.

Theorem 13 (Regret bound for 2 -bonus, with expectation inside the norm). For all

i ∈ [n], let (α i , β i,T ) ∈ [1/2, 1] × R + . For t ≥ 1, consider the event A t    ∆ t ≤ i∈[n], N i,t-1 >0 p i (S t )b i (S t )β α i i,T e i N α i i,t-1 2    . Then, if {t ≤ T } ∈ F t , we have E T t=1 I{A t }∆ t ≤ i∈[n] 4 log 2 (4 √ m i )E[β i,T ] max S∈S, p i (S)>0 b i (S)η i ,
where

η i            8 log 2 (4 √ m i )δ -1 i,min if α i = 1/2 2 -1 α i -2 -2 (1 -α i )δ 1-α i α i i,min -1 if 1/2 < α i < 1 4 1 + log δ i,max δ i,min if α i = 1, δ i,min min S∈S, p i (S)>0 ∆(S) p i (S)b i (S) , δ i,max max S∈S, p i (S)>0 ∆(S) p i (S)b i (S)
.

Looking at the two previous theorems, we can see that there is a gain in using 2 type bonuses over 1 type bonuses. Indeed, in Theorem 13, note the disappearance of the term j∈[n] p j (S)b j (S), which could potentially be quite large. For example, in the case where A t = S t is of maximum cardinality m, b i (S t ) = 1 and α i = 1/2, the use of a bonus of 2 improves the regret by a factor of m log -2 (m). Again, the following proof can be skipped to directly go to Theorem 14.

Proof of Theorem 13. Let t ≥ 1, and define

Λ t i∈[n] N i,t-1 >0 p i (S t )b i (S t )β α i i,T e i N α i i,t-1 2
, such that the event A t rewrites as {∆ t ≤ Λ t }. We will only use this event at the end of the proof, and are focused on working with Λ t for now. Our goal is first of all to bound Λ t by an 2 -norm where the components are controlled from the bottom and from the top. This then makes it possible to handle the 2 -norm using a peeling argument by grouping the components according to their range. We start by a simple lower bound on Λ t , holding for any j ∈ [n] such that p j (S t ) > 0:

Λ t ≥ p j (S t )b j (S t )β α j j,T e j N α j j,t 2 = p j (S t )b j (S t )β α j j,T N α j j,t
.

(3.9)

We then use a similar reverse amortisation technique than in the proof of Theorem 12, but this time with the 2 -norm. For this, we define

m(S t ) i∈[n] I{p i (S t ) > 0}
being the cardinality of the set of candidates to be in the triggering set A t . We have that Λ t is equal to

-Λ t + 2 i∈[n] N i,t-1 >0 p i (S t )b i (S t )β α i i,T e i N α i i,t-1 2 = - i∈[n] Λ t I{p i (S t ) > 0}e i m(S t ) 2 + i∈[n] N i,t-1 >0 2p i (S t )b i (S t )β α i i,T e i N α i i,t-1 2 ≤ i∈[n] I{p i (S t ), N i,t-1 > 0}0 ∨   2p i (S t )b i (S t )β α i i,T N α i i,t-1 - Λ t m(S t )   e i 2 = i∈[n] 0 ∨   2p i (S t )b i (S t )β α i i,T N α i i,t-1 - Λ t m(S t )   I{B i,t }e i 2 ≤ i∈[n] I    B i,t , 2p i (S t )b i (S t )β α i i,T N α i i,t-1 ≥ Λ t m(S t )    2p i (S t )b i (S t )β α i i,T e i N α i i,t-1 2
, where

B i,t p i (S t ), N i,t-1 > 0, Λ t ≥ p i (S t )b i (S t )β α i i,T N α i i,t-1
, and where the penultimate relation uses (3.9). We now decompose the interval [1/ m(S t ), 2] using a peeling:

[1/ m(S t ), 2] ⊂ log 2 √ m(St) k=0 [2 -k , 2 1-k ].
This induces a partition of the set of indices:

I    i ∈ [n], p i (S t ), N i,t-1 > 0, 2Λ t ≥ 2p i (S t )b i (S t )β α i i,T N α i i,t-1 ≥ Λ t m(S t )    ⊂ log 2 √ m(St) k=0 J k,t ,
where for all integer 1

≤ k ≤ log 2 m(S t ) , J k,t i ∈ [n], p i (S t ), N i,t-1 > 0, 2 1-k Λ t ≥ 2p i (S t )b i (S t )β α i i,T N α i i,t-1 ≥ 2 -k Λ t .
We can thus upper bound Λ 2 t using this decomposition:

Λ 2 t ≤ i∈[n] I    B i,t , 2p i (S t )b i (S t )β α i i,T N α i i,t-1 ≥ Λ t m(S t )    2p i (S t )b i (S t )β α i i,T e i N α i i,t-1 2 2 ≤ log 2 √ m(St) k=0 i∈J k,t 2p i (S t )b i (S t )β α i i,T e i N α i i,t-1 2 2 ≤ log 2 √ m(St) k=0 2 2-2k Λ 2 t e J k,t 2 2 .
This last inequality implies that there must exist one integer

k t such that |J kt,t | = e J k t ,t 2 2 ≥ 2 2kt-2 1 + log 2 m(S t ) -1
. We now use this integer to get the fol-

lowing upper bound on T t=1 I{A t }∆ t T t=1 log 2 √ m(St) k=0 I{k t = k, A t }∆ t ≤ T t=1 log 2 √ m(St) k=0 I{k t = k, A t } i∈[n] I{i ∈ J k,t }2 2-2k log 2 m(S t ) + 1 ∆ t = i∈[n] T t=1 log 2 √ m(St) k=0 I{k t = k, A t , i ∈ J k,t } (3.10) i,k,t 2 2-2k log 2 m(S t ) + 1 ∆ t .
(3.11)

Using the event A t and that i ∈ J k,t , we have

(3.10) i,k,t ≤ I p i (S t ), N i,t-1 > 0, N α i i,t-1 ≤ 2 k+1 p i (S t )b i (S t )β α i i,T ∆ t = I p i (S t ), N i,t-1 > 0, N α i i,t-1 ≤ 2 k+1 β α i i,T δ t (3.12) i,k,t , with δ t = ∆ t /(p i (S t )b i (S t )).
We now want to apply Proposition 7 from the next subsection 3.4.3. For this, we first need to add the event {i ∈ A t } to the previous indicator. As in Theorem 12, this will be done using p i (S t ). Then, we want to invert the sum over t and the one over k. We thus have that (3.11) is bounded by

i∈[n] T t=1 log 2 √ m(St) k=0 p i (S t )(3.12) i,k,t 2 2-2k log 2 m(S t ) + 1 b i (S t )δ t (3.13) ≤ i∈[n] log 2 ( √ m i ) k=0 T t=1 p i (S t )(3.12) i,k,t 2 2-2k ( log 2 ( √ m i ) + 1) max S∈S, p i (S)>0 b i (S)δ t = i∈[n] log 2 ( √ m i ) k=0 2 2-2k ( log 2 ( √ m i ) + 1) max S∈S, p i (S)>0 b i (S) T t=1 p i (S t )(3.12) i,k,t δ t (3.14) i,k,t
.

Using the definition of p i (S t ) and that {t ≤ T } ∈ F t , we can write

E (3.14) i,k,t = E T t=1 p i (S t )I p i (S t ), N i,t-1 > 0, N α i i,t-1 ≤ 2 k+1 β α i i,T δ t δ t = E T t=1 E I i ∈ A t , p i (S t ), N i,t-1 > 0, N α i i,t-1 ≤ 2 k+1 β α i i,T δ t δ t F t = E T t=1 I i ∈ A t , p i (S t ), N i,t-1 > 0, N α i i,t-1 ≤ 2 k+1 β α i i,T δ t δ t (3.15) i,k
Now, Proposition 7 gives

(3.15) i,k ≤ E[β i,T ]   I{α i < 1} 2 k+1 α i 1 -α i δ 1-1/α i i,min + I{α i = 1}2 k+1 1 + log δ i,max δ i,min   . So using log 2 ( √ m i ) + 1 ≤ log 2 (4 √ m i ), we get E T t=1 I{A t }∆ t ≤ i∈[n] 4 log 2 (4 √ m i )E[β i,T ] max S∈S, p i (S)>0 b i (S)η i ,
where

η i =            8 log 2 (4 √ m i )δ -1 i,min if α i = 1/2 2 1 α i 1 -2 1 α i -2 (1 -α i )δ 1-α i α i i,min -1 if 1/2 < α i < 1 4 1 + log δ i,max δ i,min if α i = 1.
The second result we propose starts from the previous bonus (the one provided in Theorem 13) and uses Jensen's inequality in order to bound it by placing the expectation outside the norm. One can thus see that the result given in the following Theorem 14 can be used instead of the previous one. We can nevertheless notice that the result provided is a little less good. For example, when α i = 1/2, and b i (S t ) = b, p i (S t ) = p, ∆(S t ) = ∆, we see that the factor p ∈ [0, 1] disappears in the bound of Theorem 14, compared to the one from Theorem 13. We can skip the proof of the following theorem to go to Theorem 15.

Theorem 14 (Regret bound for 2 -bonus, with expectation outside the norm). For all

i ∈ [n], let (α i , β i,T ) ∈ [1/2, 1) × R + . For t ≥ 1, consider the event A t    ∆ t ≤ E   i∈At,N i,t-1 >0 b i (S t )β α i i,T e i N α i i,t-1 2 F t      . Then, if {t ≤ T } ∈ F t , we have E T t=1 I{A t }∆ t ≤ i∈[n] 4 log 2 (4 √ m i ) max S∈S, p i (S)>0 b i (S) 1 α i E[β i,T ]η i ,
where

η i =      32 log 2 (4 √ m i )∆ -1 i,min if α i = 1/2 2 2 α i 1 -2 1 α i -2 (1 -α i )∆ 1-α i α i i,min -1 if 1/2 < α i < 1.
Proof. Let t ≥ 1. With a first reverse amortisation, we start by restricting the set of possibles for A t by only taking those whose error is at least twice as large as ∆ t : assuming that A t holds, we have

∆ t ≤ E   2 i∈At,N i,t-1 >0 b i (S t )β α i i,T e i N α i i,t-1 2 -∆ t F t   ≤ E   I    i∈At,N i,t-1 >0 2b i (S t )β α i i,T e i N α i i,t-1 2 ≥ ∆ t    i∈At,N i,t-1 >0 2b i (S t )β α i i,T e i N α i i,t-1 2 F t  
We now define

Λ(A t ) i∈At,N i,t-1 >0 2b i (S t )β α i i,T e i N α i i,t-1 2
, and have for any j ∈ A t that

Λ(A t ) ≥ 2b j (S t )β α j j,T N α j j,t . (3.16)
Then, a similar technique as in Theorem 13 gives that

Λ(A t ) equals -Λ(A t ) + i∈At,N i,t-1 >0 4b i (S t )β α i i,T e i N α i i,t-1 2 = - i∈At Λ(A t )e i e At 2 2 + i∈At,N i,t-1 >0 4b i (S t )β α i i,T e i N α i i,t-1 2 ≤ i∈At,N i,t-1 >0 0 ∨ 4b i (S t )β α i i,T N α i i,t-1 - Λ(A t ) e At 2 e i = i∈At,N i,t-1 >0 0 ∨ 4b i (S t )β α i i,T N α i i,t-1 - Λ(A t ) e At 2 I Λ(A t ) ≥ 2b i (S t )β α i i,T N α i i,t-1 e i 2 (3.16) ≤ i∈At,N i,t-1 >0 I 2Λ(A t ) ≥ 4b i (S t )β α i i,T N α i i,t-1 ≥ Λ(A t ) e At 2 4b i (S t )β α i i,T e i N α i i,t-1 2
.

We now consider the following partition of the set of indices:

I i ∈ A t , N i,t-1 > 0, 2Λ(A t ) ≥ 4b i (S t )β α i i,T N α i i,t-1 ≥ Λ(A t ) e At 2 ⊂ log 2 ( e A t 2 ) k=0 J k,t ,
where for all integer 1

≤ k ≤ log 2 ( e At 2 ) , J k,t i ∈ A t , N i,t-1 > 0, 2 1-k Λ(A t ) ≥ 4b i (S t )β α i i,T N α i i,t-1 ≥ 2 -k Λ(A t ) .
We bound Λ(A t ) 2 as

Λ(A t ) 2 ≤ i∈At,N i,t-1 >0 I 2Λ(A t ) ≥ 4b i (S t )β α i i,T N α i i,t-1 ≥ Λ(A t ) e At 2 4b i (S t )β α i i,T e i N α i i,t-1 2 2 = log 2 ( e A t 2 ) k=0 i∈J k,t 4b i (S t )β α i i,T e i N α i i,t-1 2 2 ≤ log 2 ( e A t 2 ) k=0 2 2-2k Λ(A t ) 2 e J k,t 2 2 . So there is an integer k t such that |J kt,t | = e J k t ,t 2 2 ≥ 2 2kt-2 (1 + log 2 ( e At 2 ) ) -1 . T t=1 I{A t }∆ t ≤ T t=1 E[ I{Λ(A t ) ≥ ∆ t }Λ(A t )|F t ] ≤ T t=1 E    log 2 ( e A t 2 ) k=0 I{k t = k, Λ(A t ) ≥ ∆ t }Λ(A t ) F t    ≤ T t=1 E    log 2 ( e A t 2 ) k=0 I{k t = k, Λ(A t ) ≥ ∆ t } i∈[n] I{i ∈ J k,t } 2 2k-2 (1+ log 2 ( e At 2 ) ) -1 Λ(A t ) F t    ≤ T t=1 n i=1 E     log 2 ( e A t 2 ) k=0 I i ∈ A t , 0 < N α i i,t-1 ≤ 2 k+2 b i (St)β α i i,T Λ(At) , Λ(A t ) ≥ ∆ t 2 2k-2 (1 + log 2 ( e At 2 ) ) -1 Λ(A t ) F t     .
Taking the expectation of the above, and using {t ≤ T } ∈ F t , we have the bound

E T t=1 I{A t }∆ t ≤ n i=1 E     T t=1 log 2 ( e A t 2 ) k=0 I i ∈ A t , 0 < N α i i,t-1 ≤ 2 k+2 b i (St)β α i i,T Λ(At) , Λ(A t ) ≥ ∆ t 2 2k-2 (1 + log 2 ( e At 2 ) ) -1 Λ(A t )     ≤ n i=1 log 2 ( √ m i ) k=0 E 1+ log 2 ( √ m i ) 2 2k-2 (3.17) i,k , where (3.17) i,k T t=1 I i ∈ A t , 0 < N α i i,t-1 ≤ 2 k+2 b i (S t )β α i i,T Λ(A t ) , Λ(A t ) ≥ ∆ t Λ(A t ).
Applying Proposition 7 from the next subsection 3.4.3 gives

(3.17) i,k ≤ max S∈S, p i (S)>0 b i (S) 1 α i β i,T 2 k+2 α i 1 -α i ∆ 1-1/α i i,min , So using log 2 ( √ m i ) + 1 ≤ log 2 (4 √ m i ), we get E T t=1 I{A t }∆ t ≤ i∈[n] 4 log 2 (4 √ m i ) max S∈S, p i (S)>0 b i (S) 1 α i E[β i,T ]η i ,
where

η i =      32 log 2 (4 √ m i )∆ -1 i,min if α i = 1/2 2 2 α i 1 -2 1 α i -2 (1 -α i )∆ 1-α i α i i,min -1 if 1/2 < α i < 1.
Finally, the last result re-uses Jensen's inequality in the bonus from Theorem 14, with the concavity of the square root. Again, the result loses power compared to the Theorems 13, 14. In fact, we see that by linearity of the 2 -norm squared, we are reduced to a particular form of Theorem 13 where, in the error vector, the probability p i (S t ) is changed to the square root of this probability.

Theorem 15 (Regret bound for 2 -bonus, with expectation between the root and the squared norm).

For all i ∈ [n], let β i,T ∈ R + . For t ≥ 1, consider the event A t        ∆ t ≤ E    i∈At,N i,t-1 >0 b i (S t )β 1/2 i,T e i N 1/2 i,t-1 2 2 F t           . Then, if {t ≤ T } ∈ F t , we have that E T t=1 I{A t }∆ t is bounded by i∈[n]
16 log 2 2 (4

√ m i )E[β i,T ] max S∈S, p i (S)>0 b i (S) 2 ∆(S) (1 + log(η i )),
where

η i max S∈S, p i (S)>0 ∆(S) 2 / p i (S)b i (S) 2 min S∈S, p i (S)>0 ∆(S) 2 /(p i (S)b i (S) 2 ) .
Proof. Let t ≥ 1 and notice that

E    i∈At,N i,t-1 >0 b i (S t )β 1/2 i,T e i N 1/2 i,t-1 2 2 F t    = i∈[n],N i,t-1 >0 p i (S t )b i (S t )β 1/2 i,T e i N 1/2 i,t-1 2
.

So, we can use Theorem 13 with α i = 1/2 and another b i function, that we denote b i , and that is defined as

b i (S) = b i (S)   1 p i (S) I{p i (S) > 0} + I{p i (S) = 0}   .
We however do not use the result directly, but rather rework the end of the proof in order to get a logarithmic dependence in p i . More precisely, the proof remains the same until the relation (3.13). In particular, we have the definition

δ t = ∆ t /(p i (S t ) b i (S t )).
Then, in (3.13), we bound

I{p i (S t ) > 0} b i (S t )δ t = I{p i (S t ) > 0} ∆ t p i (S t ) by max S∈S, p i (S)>0 b i (S) 2 ∆(S) I{p i (S t ) > 0} ∆ 2 t p i (S t )b i (S t ) 2 = max S∈S, p i (S)>0 b i (S) 2 ∆(S) I{p i (S t ) > 0}δ 2 t .
The end of the proof is the same, except from the application of Proposition 7, where having δ 2 t instead of δ t gives a behavior as if α i = 1, giving the final bound

i∈[n]
16 log 2 2 (4

√ m i )E[β i,T ] max S∈S, p i (S)>0 b i (S) 2 ∆(S) 1 + log δ 2 i,max δ 2 i,min
.

The regret bound given in Theorem 15 has a logarithmic dependence in p i . A question that arises then is: can we express a bound that does not depend at all on p i ? The answer is yes, and it is done using Proposition 9 instead of Proposition 7 at the end of the proof. This is the purpose of the following Theorem 16.

Theorem 16. Under the same assumptions as in Theorem 15, we have the following upper bound on E

T t=1 I{A t }∆ t , i∈[n]
16 log 2 2 (4

√ m i )E[β i,T ]η i 1 + E log 16β i,T m i min S∈S, p i (S)>0 ∆(S) 2 /b i (S) 2 ,
where

η i = max S∈S, p i (S)>0 b i (S) 2 ∆(S) .
Proof. We use the same proof as in Theorem 15 until before the application of Proposition 7. In particular, we thus follow the beginning of the proof of Theorem 13. Before applying Proposition 9 instead of Proposition 7, we first upper bound δ 2 t by

2 2k+2 β i,T N i,t-1 ,
Thanks to (3.12) i,k,t . We then use Proposition 9 from the next subsection 3.4.3 with α i = 1, giving the logarithmic factor

1 + log 2 2k+2 β i,T δ 2 i,min instead of 1 + log δ 2 i,max δ 2 i,min
.

Finally, the desired result is obtained by bounding k by log 2 (

√ m i ) ≤ log 2 (2 √ m i ),
and by bounding δ -2 i,min by max

S∈S, p i (S)>0 b i (S) 2 /∆(S) 2 .
When we derive an upper bound on ∆ t , it may happen that we do not fall into one of the cases mentioned in the above theorems. Specifically, it may be that the bonus is composed of several terms, each of which has the proper form to be treated as above. In addition, when initialization is handled with Proposition 5, this adds an additional term (corresponding to zero counters), so it is useful to know how to handle it. We will see in the following Proposition 6 that previous results are sufficient to treat a composed bonus.

Proposition 6 (Regret bound for a composed bonus). Let K ∈ N * . For all t ≥ 1, consider the event

A t    ∆ t ≤ k∈[K] B k,t    , for some B k,t ≥ 0. Then, the event-filtered regret E T t=1 ∆ t I{A t } is upper bounded by k∈[K] E   t∈[T ] ∆ t I{∆ t ≤ KB k,t }   . Proof. From A t , there is a k such that ∆ t ≤ KB k,t . So 1 ≤ k∈[K] I{∆ t ≤ KB k,t }, i.e., ∆ t ≤ k∈[K] ∆ t I{∆ t ≤ KB k,t }.
We can therefore see that we can use Proposition 6 coupled with the above theorems in order to treat the composed bonuses. Doing so, we only lose a factor K in the previous analyses. Typically, K is a small universal constant, so we do not lose in the regret rate.

Appendix

As we have already seen in Theorem 10 for MAB, once we had managed to bound the gap ∆ t by the bonus, a rearrangement of the inequality made it possible to have that the counter of the chosen arm is bounded by a function of the minimal gap, leading to the final bound by summing over t. We will see here that the same principle also applies in the CMAB-T context. As we saw, this is due to the fact that the bonuses are each time expressed using the counters N i,t-1 for i ∈ A t . We give in this subsection a series of results that, for an arm i, bound the regret filtered by the event where i ∈ A t and an upper bound on a counter N i,t-1 is known. These results are extending those given in Chen, Wang, and Yuan (2013), Chen, Wang, and Yuan (2016), and Wang and Chen (2017).

Proposition 7. Let i ∈ [n] and f i : R + → R + be a non increasing function, integrable on an interval [δ i,min , δ i,max ] ⊂ R * + . Then for any sequence of real numbers

(δ t ) ∈ ([δ i,min , δ i,max ] ∪ {0}) T , T t=1 I{i ∈ A t , 1 ≤ N i,t-1 ≤ f i (δ t )}δ t ≤ f i (δ i,min )δ i,min + δ i,max δ i,min f i (x)dx.
In particular,

• If f i (x) = β i,T x -1/α i , α i ∈ (0, 1) and β i,T ≥ 0, then T t=1 I{i ∈ A t , 1 ≤ N i,t-1 ≤ f i (δ t )}δ t ≤ δ 1-1/α i i,min β i,T 1 -α i -δ 1-1/α i i,max α i β i,T 1 -α i ≤ δ 1-1/α i i,min β i,T 1 -α i . • If f i (x) = β i,T x -1 , β i,T ≥ 0, then T t=1 I{i ∈ A t , 1 ≤ N i,t-1 ≤ f i (δ t )}δ t ≤ β i,T 1 + log δ i,max δ i,min . Proof. Consider δ i,max = δ i,1 ≥ δ i,2 ≥ • • • ≥ δ i,K i = δ i,
min being all possible values for δ t when δ t = 0. We define a dummy gap δ i,0 = ∞ and let f i (δ i,0 ) = 0. In (3.18), we look at times t where δ t = 0 and first break the range (0, f i (δ t )] of the counter N i,t-1 into sub intervals:

(0, f i (δ t )] = (f i (δ i,0 ), f i (δ i,1 )] ∪ • • • ∪ (f i (δ i,kt-1 ), f i (δ i,kt )],
where k t is the index such that δ i,kt = δ t . This index k t exists by assumption that the subdivision contains all possible values for δ t when δ t = 0. Notice that in (3.18), we do not explicitly use k t , but instead sum over all k ∈ [K i ] and filter against the

event {δ i,k ≥ δ t }, which is equivalent to summing over k ∈ [k t ]. T t=1 I{i ∈ A t , N i,t-1 ≤ f i (δ t )}δ t = T t=1 K i k=1 I{i ∈ A t , f i (δ i,k-1 ) < N i,t-1 ≤ f i (δ i,k ), δ i,k ≥ δ t }δ t . (3.18)
Over each event that N i,t-1 belongs to the interval (f i (δ i,k-1 ), f i (δ i,k )], we upper bound the gap δ t by δ i,k .

(3.18)

≤ T t=1 K i k=1 I{i ∈ A t , f i (δ i,k-1 ) < N i,t-1 ≤ f i (δ i,k ), δ i,k ≥ δ t }δ i,k . (3.19)
Then, we further upper bound the summation by adding events that N i,t-1 belongs to the remaining intervals (f i (δ i,k-1 ), f i (δ i,k )] for k t < k ≤ K i , associating them to a suffered gap δ i,k . This is equivalent to removing the filtering against the event {δ i,k ≥ δ t }.

(3.19) ≤ T t=1 K i k=1 I{i ∈ A t , f i (δ i,k-1 ) < N i,t-1 ≤ f i (δ i,k )}δ i,k . (3.20)
Now, we invert the summation over t and the one over k.

(3.20) = K i k=1 T t=1 I{i ∈ A t , f i (δ i,k-1 ) < N i,t-1 ≤ f i (δ i,k )}δ i,k . (3.21) For each k ∈ [K i ], the number of times t ∈ [T ] that the counter N i,t-1 belongs to (f i (δ i,k-1 ), f i (δ i,k )
] can be upper bounded by the number of integers in this interval. This is due to the event {i ∈ A t }, imposing that N i,t-1 is incremented, so N i,t-1 cannot be worth the same integer for two different times t satisfying i ∈ A t . We use the fact that for all x, y ∈ R, x ≤ y, the number of integers in the interval (x, y] is exactly y -x .

(3.21)

≤ K i k=1 ( f i (δ i,k ) -f i (δ i,k-1 ) )δ i,k . (3.22)
We then simply expand the summation, and some terms are cancelled (remember that f i (δ i,0 ) = 0).

(3.22) = f i (δ i,K i ) δ i,K i + K i -1 k=1 f i (δ i,k ) (δ i,k -δ i,k+1 ) (3.23)
We use x ≤ x for all x ∈ R. Finally, we recognize a right Riemann sum, and use the fact that f i is non increasing to upper bound each

f i (δ i,k )(δ i,k -δ i,k+1 ) by δ i,k δ i,k+1 f i (x)dx, for all k ∈ [K i -1]. (3.23) ≤ f i (δ i,K i )δ i,K i + K i -1 k=1 f i (δ i,k )(δ i,k -δ i,k+1 ) (3.24) ≤ f i (δ i,K i )δ i,K i + δ i,1 δ i,K i f i (x)dx. ( 3
(δ t ) ∈ ([δ i,min , δ i,max ] ∪ {0}) T , T t=1 I{i ∈ A t , δ t = 0, f i (δ i,max ) < N i,t-1 ≤ f i (δ t )}f -1 i (N i,t-1 ) ≤ f i (δ i,min )δ i,min + δ i,max δ i,min f i (x)dx.
Proof. The proof is very similar to the one for Proposition 7. The key difference will be that instead of a right Riemann sum, this is a left Riemann sum that appears at the end of the proof. Therefore, we can't bound this Riemann sum directly by the integral. However, it can be made as close as desired to the integral by making the subdivision fine enough. This is possible since we never use any assumption on the subdivision of [δ i,min , δ i,max ] we consider (except that it contains all possible values for δ t when i ∈ A t , which can only reinforce its refinement). In addition, at the end, the subdivision only appears within the Riemann sum. We thus consider a general subdivision

δ i,max = δ i,1 ≥ δ i,2 ≥ . . . δ i,K i = δ i,min
, that contains all possible values for δ t when δ t = 0. Using this decomposition, we can write as previously 26) Now, we can upper bound f -1 i (N i,t-1 ) by δ i,k-1 to recover a summation similar to (3.19) in the proof of Proposition 7. Using the same derivations as in (3.19)-(3.22), we get the upper bound

T t=1 I{i ∈ A t , δ t = 0, f i (δ i,max ) < N i,t-1 ≤ f i (δ t )}f -1 i (N i,t-1 ) = T t=1 K i k=2 I{i ∈ A t , f i (δ i,k-1 ) < N i,t-1 ≤ f i (δ i,k ), δ i,k ≥ δ t > 0}f -1 i (N i,t-1 ). (3.
(3.26) ≤ K i k=2 ( f i (δ i,k ) -f i (δ i,k-1 ) )δ i,k-1 = f i (δ i,K i ) δ i,K i -f i (δ i,1 ) δ i,1 + K i k=2 f i (δ i,k ) (δ i,k-1 -δ i,k ) ≤ f i (δ i,K i )δ i,K i + K i k=2 f i (δ i,k )(δ i,k-1 -δ i,k ),
where in the last inequality, we first upper bound the term -f i (δ i,1 ) δ i,1 by 0, and then use x ≤ x for all x ∈ R. Finally, we recognize a left Riemann sum. We get the desired result taking the infimum over all possible subdivisions containing all possible values for δ t when δ t = 0. Remark 9. Proposition 8 does not exactly generalizes Proposition 7 since we have to filter against the event {f i (δ i,max ) < N i,t-1 }. However, for f i (x) = β i,T x -1/α i , β i,T ≥ 0 and α i ∈ (0, 1), the upper bound is

δ 1-1/α i i,min β i,T 1 -α i -δ 1-1/α i i,max α i β i,T 1 -α i ≤ δ 1-1/α i i,min β i,T 1 -α i ,
so δ i,max can be taken sufficiently large without altering the desired upper bound. In particular, for δ i,max such that f i (δ i,max ) < 1, the event {f i (δ i,max ) < N i,t-1 } always holds (when the counter is non-zero). We will see another (simpler) proof of this fact in the next Proposition 9.

Proposition 9. Let i ∈ [n] and f i (x) = β i,T x -1/α i , α i ∈ (0, 1] and β i,T ≥ 0. Then T t=1 I{i ∈ A t , δ t = 0, 1 ≤ N i,t-1 ≤ f i (δ t )}f -1 i (N i,t-1 ) ≤ δ 1-1/α i i,min β i,T 1 -α i I{α i < 1} +I{α i = 1}β i,T 1 + log β i,T δ i,min .
Proof. We upper bound f i (δ t ) by f i (δ i,min ) directly in the event, and then simply count the number of integers in (0, f i (δ i,min )]. For each such integer s, the regret suffered is f -1 i (s). We then upper bound the sum by an integral (using the fact that f -1 i is decreasing), to get the final result.

T t=1 I{i ∈ A t , δ t = 0, 1 ≤ N i,t-1 ≤ f i (δ t )}f -1 i (N i,t-1 ) ≤ T t=1 I{i ∈ A t , 1 ≤ N i,t-1 ≤ f i (δ i,min )}f -1 i (N i,t-1 ) ≤ f i (δ i,min ) s=1 f -1 i (s) ≤ f -1 i (1) + f i (δ i,min ) 1 f -1 i (s)ds = β α i i,T + β i,T δ -1/α i i,min 1 β α i i,T s -α i ds ≤ I{α i < 1}δ 1-1/α i i,min β i,T 1 -α i + I{α i = 1}β i,T 1 + log β i,T δ i,min .
Chapter 4

An Example of CMAB-T Problem: Sequential Search-and-Stop

This chapter is based on Perrault, Perchet, and Valko (2019b), and aims to introduce a new example of a problem falling within the CMAB-T framework.

We consider here the problem where the agent wants to find a hidden object that is randomly located in some vertex of a directed acyclic graph (DAG) according to a fixed but possibly unknown distribution. The agent can only examine vertices whose in-neighbors have already been examined. We address a learning setting where we allow the agent to stop before having found the object and restart searching on a new independent instance of the same problem. Our goal is to maximize the total number of hidden objects found given a time budget. The agent can thus skip an instance after realizing that it would spend too much time on it. Our contributions are both to the search theory and multi-armed bandits. If the distribution is known, we provide a quasi-optimal and efficient stationary strategy. If the distribution is unknown, we additionally show how to sequentially approximate it and, at the same time, act near-optimally in order to collect as many hidden objects as possible.

Problem formulation and motivation

We study the setting where an object, called hider, is randomly located in one vertex of a directed acyclic graph (DAG), and where an agent wants to find it by sequentially selecting vertices one by one, and examining them at a (possibly random) cost. The agent has a strong constraint: its search must respect precedence constraints imposed by the DAG, i.e., a vertex can be examined only if all its in-neighbors have already been examined. The goal of the agent is to minimize the expected total search cost incurred before finding the hider. This setting is a type of single machine scheduling problem [START_REF] Lín | Scheduling results applicable to decision-theoretic troubleshooting[END_REF], where a set of n jobs [n] have to be processed on a single machine that can process at most one job at a time. Once a job processing is started, it must continue without interruption until the processing is complete. Each job j has a cost c j representing its processing time, and a weight w j representing its importance. In our context, w j is the probability that j contains the hider. The aim is to find a schedule (i.e., a permutation of jobs) that minimizes the total weighted completion time while respecting precedence constraints. 1 The setting was already shown to be NP-hard [START_REF] Lawler | Sequencing jobs to minimize total weighted completion time subject to precedence constraints[END_REF][START_REF] Lenstra | Complexity of Scheduling under Precedence Constraints[END_REF]. On the positive side, several polynomial-time α-approximations exist, depending on the assumption we take on the DAG (see e.g., the recent survey of [START_REF] Prot | A survey on how the structure of precedence constraints may change the complexity class of scheduling problems[END_REF].

For instance, the case of α = 2 can be dealt without any additional assumption. On the other hand, there is an exact O(n log n)-time algorithm when the partially ordered set (poset) defined by the DAG is a series-parallel order [START_REF] Lawler | Sequencing jobs to minimize total weighted completion time subject to precedence constraints[END_REF]. More generally, when the poset has fractional dimension of at most f , there is a polynomial-time approximation with α = 2 -2/f [START_REF] Ambühl | On the Approximability of Single-Machine Scheduling with Precedence Constraints[END_REF]. In this work, we assume the DAG is such that an exact polynomial-time algorithm is available. 2 We denote this algorithm as scheduling. For example, this is true for two-dimensional posets [START_REF] Ambühl | Single machine precedence constrained scheduling is a vertex cover problem[END_REF].

The problem is also well known in search theory [START_REF] Stone | Theory of optimal search[END_REF][START_REF] Fokkink | On Submodular Search and Machine Scheduling[END_REF], one of the disciplines originating from operations research. Since in our case, the search space is a DAG, we fall within the network search setting [START_REF] Kikuta | Initial point search on weighted trees[END_REF][START_REF] Gal | On the optimality of a simple strategy for searching graphs[END_REF][START_REF] Evans | Static search games played over graphs and general metric spaces[END_REF]. When the DAG is an out-tree, the problem reduces to the expanding search problem introduced by [START_REF] Alpern | Mining coal or finding terrorists: The expanding search paradigm[END_REF].

The case of unknown distribution of the hider is usually studied within the field of search games, i.e., a zero-sum game where the agent picks the search and plays against the hider with search cost as payoff [START_REF] Alpern | The theory of search games and rendezvous[END_REF][START_REF] Alpern | Search theory[END_REF][START_REF] Hohzaki | Search games: Literature and survey[END_REF]. In our work, we deal with an unknown hider distribution by extending the stochastic setting to the sequential case, where at each round t, the agent faces a new, independent instance of the problem. The challenge is the need to learn the distribution through repeated interactions with the environment. Each instance, the agent has to perform a search based on the instances observed during the previous rounds. Furthermore, contrary to the typical search setting, the agent can additionally decide whether it wishes to abandon the search on the current instance and start a new one in the next round, even if the hider was not found. The goal of the agent is to collect as many hiders as possible, using a fixed budget B. This may be particularly useful, when the remaining vertices have large costs and it would not be cost-effective to examine them.

As a result, the hider may not be found in each round and the agent has to make a trade-off between exhaustive searches, which lead to a good estimation (exploration) and efficient searches, which leads to a good benefit/cost ratio (exploitation). The sequential exploration-exploitation trade-off is well studied in multi-armed bandits (Cesa-Bianchi and [START_REF] Cesa-Bianchi | Prediction, learning, and games[END_REF][START_REF] Lattimore | Bandit algorithms[END_REF] and has been applied (as we saw in Chapter 2) to many fields including mechanism design [START_REF] Mohri | Optimal regret minimization in postedprice auctions with strategic buyers[END_REF], search advertising [START_REF] Tran-Thanh | Efficient crowdsourcing of unknown experts using bounded multi-armed bandits[END_REF] and personalized recommendation [START_REF] Li | A contextual-bandit approach to personalized news article recommendation[END_REF]. Since several vertices can be visited within each round, our setting can be seen as an instance of stochastic combinatorial semi-bandits. For this reason, we refer to a vertex j ∈ [n] as an arm. We shall see, however, that this specific semi-bandit problem is challenging. In particular, the agent pays a non-linear search cost at each round (with respect to the selected combinatorial action), that additionally depends on the ordering. Moreover, due to the budget constraint, it is also an instance of budgeted bandits, also known as bandits with knapsacks [START_REF] Badanidiyuru | Bandits with knapsacks[END_REF], in the case of single resource and infinite horizon. We thus evaluate the performance of a learning policy with the (common) notion of budgeted regret. It measures the expected difference, in terms of cumulative reward collected within the budget constraint B, between the learning policy and an oracle policy that knows a priori the exact parameters of the problem. Budgeted combinatorial semi-bandits have been already studied by [START_REF] Sankararaman | Combinatorial Semi-Bandits with Knapsacks[END_REF] for several resources, but with a finite horizon. Moreover, their algorithm is efficient only for some specific combinatorial structures (such as matroids). The structure of constraints in sequential search-and-stop is in general more complex.

Motivation There are several motivations behind this setting. One example is the decision-theoretic troubleshooting problem of giving a diagnosis for several devices having a malfunctioning component and arriving sequentially to the agent. In many troubleshooting applications, we additionally face precedence constraints. These restrictions are imposed to the agent as the ordering of component tests, see e.g., [START_REF] Jensen | The SACSO methodology for troubleshooting complex systems[END_REF]. Moreover, allowing the agent to stop gives a new alternative to the socalled service call [START_REF] Heckerman | Decision-theoretic troubleshooting[END_REF][START_REF] Jensen | The SACSO methodology for troubleshooting complex systems[END_REF] in order to deal with non-cost-effective vertices: Instead of giving a high cost to an extra action that will automatically find the fault in the device, we give it a zero cost, but do not reward such diagnostic failure. This way, we do not need to estimate any call-service cost. This alternative is used, for example, when a new device is sent to the user if the diagnostic fails, with a cost that depends on a disutility for the user: loss of personal data, device reconfiguration, etc. Maximizing the number of hiders found is then analogous to maximizing the number of successful diagnoses.

Another example comes from online advertisement. There are several different actions that might generate a conversion from a user, such as sending one or several emails, displaying one or several ads on a website, buying keywords on search engines, etc. We assume that some precedence constraints are imposed between actions and that a conversion will occur if some sequence of actions is made, for instance, first, display an ad, then send the first email, and finally the second one. As a consequence, the conversion is "hidden", the precedence constraints restrict our access to it, and the agent aims at finding it. However, for some users, finding the correct sequence might be too expensive and it might be more interesting to abandon that specific user to focus on more promising ones.

Related settings

Finally, there are several settings related to ours. One of them is stochastic probing [START_REF] Gupta | A stochastic probing problem with applications[END_REF], which differs in the fact that each arm can contain a hider, independently from each other. Another one is the machine learning framework of optimal discovery [START_REF] Bubeck | Optimal discovery with probabilistic expert advice: finite time analysis and macroscopic optimality[END_REF].

Our contributions One of our main contributions is a stationary offline policy (i.e., an algorithm that solves the problem when the distribution is known), for which we prove the approximation guarantees and adapt it in order to fit the online problem. In particular, we prove that it is quasi-optimal and use scheduling to prove its computational efficiency. Next, we provide a solution when the distribution is unknown to the agent, based on the combinatorial upper confidence bounds (cucb) algorithm from Chen, Wang, and Yuan (2016), and ucb-variance (ucb-v) of Audibert, Munos, and Szepesvári (2009a). Dealing with variance estimates allows us to sharp the bound on the expected regret, improving the overall dependence on the dimension n compared to the simple use of cucb. We also propose a new method (that can be of independent interest) to avoid the typical c -2 min term in the expected regret bound [START_REF] Tran-Thanh | Knapsack Based Optimal Policies for Budget-Limited Multi-Armed Bandits[END_REF][START_REF] Ding | Multi-Armed Bandit with Budget Constraint and Variable Costs[END_REF][START_REF] Xia | Budgeted bandit problems with continuous random costs[END_REF][START_REF] Xia | Budgeted multi-armed bandits with multiple plays[END_REF][START_REF] Watanabe | KL-UCB-Based Policy for Budgeted Multi-Armed Bandits with Stochastic Action Costs[END_REF], where c min is the minimal expected search cost paid over a single round.

Background

We formalize in this subsection the setting we consider. We denote a finite DAG by G ([n], E), where [n] is its set of vertices, or arms, and E is its set of directed edges. For more generality, we assume arm costs are random and mutually independent. We denote C j ∈ [0, 1], with expectation c * j E[C j ] > 0, the cost of arm j. We thus have c * = E[C] ∈ (0, 1] n . We also assume that one specific vertex, called hider, is chosen at random, independently from C, accordingly to some fixed categorical distribution (or Multinoulli) parameterized by vector w * satisfying3 n i=1 w * i = 1 and

w * i ∈ [0, 1]. Notice that W ∼ Multinoulli(w * ) if, given i ∈ [n] and with probability w * i , W i = 1 and W j = 0 for all j = i.
Let G A be the sub-DAG in G induced by A, i.e., the DAG with A as vertex set, and with (i, j) an edge in G A if and only if (i, j) ∈ E. We call support of an ordered arm set S = (s 1 , . . . , s k ) the corresponding non-ordered set. For two disjoint ordered arm sets S and S , we let SS = (s 1 , s 2 , . . . , s |S| , s 1 , s 2 , . . . , s |S | ) be the concatenation of S and S .

We assume that G allows a polynomial-time algorithm (w.r.t. n), that takes some parameters w, c ∈ R n + , and outputs S = scheduling(w, c, G) minimizing

d(S; w, c) |S| i=1 w s i i j=1 c s j
over linear extensions4 S = (s 1 , . . . , s n ) of the poset defined by G (that we call G- linear extensions). Notice that d(S; w * , c * ) represents the expected cost to pay for finding the hider with the G-linear extension S, i.e., by searching arm s 1 first and paying C s 1 , then s 2 by paying C s 2 in case W s 1 = 0, and so on until the hider is found, i.e., the last arm i searched is such that W i = 1. We define a search in G as an ordering S = (s 1 , . . . , s k ) of different arms such that for all i ∈ [k], predecessors of s i in G are included in {s 1 , . . . , s i-1 }, i.e., a search is a prefix of a G-linear extension. We denote by S the set of searches in G. Search supports are called initial sets.

Protocol

Our search setting is sequential. We consider an agent, also called a learning algorithm or a policy that knows G but that does not know P (C,W) . At each round t, an independent sample (C t , W t ) is drawn from P (C,W) . The aim of the agent is to search the hider (i.e., the arm i such that W i,t = 1) by constructing a search on G. Since the hider may be located at some arm that does not belong to the search, it is not necessarily found over each round.

The search to be used by the agent can be chosen based on all its previous observations, i.e., all the costs of explored vertices (and only those) and all the locations where the hider has been found or not. Obviously, the search cannot use the nonobserved quantities. For example, the agent may estimate w * and c * in order to choose the search accordingly. Each time an arm j is searched, the feedback W j,t and C j,t is given to the agent. The agent can keep searching round after round until its global budget, B, runs out. B is a positive number and does not need to be known to the agent in advance. The agent wants to maximize the overall number of hiders found under the budget constraint.

The setting described above allows the agent to modify its behavior depending on the feedback it received during the current round. However, the only feedback on W i susceptible to modify the search the agent chose at the beginning of a round t is the observation of W i,t = 1 for some arm i. Even if nothing prevents the agent from continuing "searching" some arms after having seen such an event, it would not increase the number of hiders found (there is no more hider to find), while this would still decrease the remaining budget, and therefore it would have a pure exploratory purpose. Knowing this, for the case where costs are deterministic, an oracle policy that knows exactly P W thus selects a search S t at the beginning of round t, and then performs the search that follows S t until either W i,t = 1 is observed or S t is exhausted (i.e., no arms are left in S t ). Therefore, the performed search is composed of the set of arms

A t = {s t,1 , . . . , s t,j },
where j is such that W s j ,t = 1, and j = |S t | if no such node exists. In the general case of random costs, when we will design a policy, we choose to restrict ourselves to agents that select a search S t at the beginning of each round t and then performs A t over this round. As a consequence, the selected search is computed based on observations collected during previous rounds t -1, t -2, . . . , denoted H t , that we refer to as history.

Remark 10. Since a single arm i is associated to a pair of outcomes (C i , W i ), it must be kept in mind that the above set A t is the feedback set corresponding to the cost C only (it is on this feedback that the above setting is of the CMAB-T type).

Concerning the feedback for W, we remark that e St W t is observed at round t in the above setting, i.e., the feedback set coincides with the action selected (i.e., this feedback is only of the CMAB type).

Following [START_REF] Stone | Theory of optimal search[END_REF], we refer to our problem as sequential search-and-stop. We now detail the overall objective for this problem: The agent wants to follow a policy π, that performs a search A t at round t, while maximizing the expected overall reward

F B (π) E   τ B -1 t=1 i∈At W i,t   ,
where τ B is the random round at which the remaining budget becomes negative: In particular, we have that if

B t B -t t =1 e T A t C t , then B τ B -1 ≥ 0 and B τ B < 0.
We evaluate the performance of a policy using the expected (budgeted) regret with respect to F * B , the maximum value of F B (among all possible oracle policies that know P (C,W) and B), defined as

R B (π) F * B -F B (π).
Example 3. One may wonder if there exist cases where it is interesting for the agent to stop the search earlier (i.e. to select S with |S| < n). Consider for instance the simplest non-trivial case with two arms and no precedence constraint. The costs are deterministically chosen to be ε and 1 and the location of the hider is chosen uniformly at random. An optimal search will always first sample the arm with ε < 1 cost. If it also samples the other one, then the hider will be found with an expected cost of ε + 1/2. However, if the agent always stops the search after the first arm, and reinitializes on a new instance by doing the same, the overall cost to find one hider is

∞ t=1 1 2 t tε = 2ε < ε + 1 2 , for ε < 1 2 .
Therefore, stopping searches, even if the location of the hider is known, may be better than always trying to find it.

Offline oracle

In this section, we provide an algorithm for sequential search-and-stop when parameters w * and c * are given to the agent. We show that a simple stationary policy (i.e., the same search S * is selected at each round) can obtain almost the same expected overall reward as F * B . We will denote by Oracle an algorithm that takes w * , c * , and G as input and outputs S * . This offline oracle will eventually be used by the agent for the online problem, i.e., when parameters are unknown. Indeed, at round t, the agent can approximate S * by the output S t of Oracle(w t , c t , G), where w t , c t can be any guesses/estimates of the true parameters. Importantly, depending on the estimation followed by the agent, w t may not stay in the simplex anymore. We will thus build Oracle such that an "acceptable" output is given for any input (w, c) ∈ (R n + ) 2 .

Objective design

A standard paradigm for designing a stationary approximation of the offline problem in budgeted multi-armed bandits is the following: S * has to minimize the ratio between the expected cost paid and the expected reward gain, over a single round, selecting S * . We thus define, for S ∈ S,

J (S; w * , c * ) E[e T A C]E[e T S W] -1 = d(S; w * , c * ) + (1 -e T S w * )e T S c * e T S w * = |S| i=1 c * s i 1 -e T {s 1 ,...,s i-1 } w * e T S w * .
Notice that we allow J to be equal to +∞ (when e T S w * = 0). We use the convention J (∅; w * , c * ) = +∞, because there is no interest in choosing an empty search for a round. We define the optimal values of J on S as

J * min S∈S J (S; w * , c * ), S * arg min S∈S J (S; w * , c * ).
We now provide guarantees for this stationary policy.

Proposition 10. If π * is the offline policy selecting S * ∈ S * at each round t, then

B -n J * ≤ F B (π * ) ≤ F * B ≤ B + n J * .
Proof. The proof follows the one provided for Lemma 1 of [START_REF] Xia | Budgeted multi-armed bandits with multiple plays[END_REF] If we let B 0 = B, then for any offline policy π, if we denote by A t , S t the search selected, resp. performed by π at round t (we saw that an optimal policy is such that S t does not depend on W t ), and if we let

B t = B -t t =1 e T A t C t be the remaining budget at time t, F B (π) = ∞ t=1 E   i∈St I{B t ≥ 0, W i,t = 1}   (4.1) ≤ ∞ t=1 E   i∈St I{B t-1 ≥ 0, W i,t = 1}   (4.2) = ∞ t=1 E   i∈St I{B t-1 ≥ 0}w * i   (4.3) = ∞ t=1 E I{B t-1 ≥ 0}w * T e St = ∞ t=1 E I{B t-1 ≥ 0} d(S t ; w * , c * ) + (1 -w * T e St )c * T e St J (S t ; w * , c * ) ≤ ∞ t=1 E I{B t-1 ≥ 0} d(S t ; w * , c * ) + (1 -w * T e St )c * T e St J * = 1 J * E τ B t=1 (d(S t ; w * , c * ) + (1 -w * T e St )c * T e St ) (4.4) = 1 J * E τ B t=1 c * T e At = 1 J * E τ B -1 t=1 C T t e At + C T τ B e Aτ B ≤ B + n J * , (4.5)
where (4.2) uses B t ≥ 0 ⇒ B t-1 ≥ 0, (4.3) is obtained by conditioning on previously sampled arms, (4.4) uses the random round τ B such that B τ B -1 ≥ 0 and B τ B < 0, and (4.5) uses the definition of B τ B -1 and C i,t ≤ 1. Now, for the lower bound, we have that

F B (π * ) ≥ ∞ t=1 E i∈S * I{B t-1 ≥ n, W i,t = 1} (4.6) = ∞ t=1 E[I{B t-1 ≥ n}e T S * w * ] (4.7) = 1 J * E τ t=1 C T t e A * t (4.8) ≥ B -n J * , (4.9)
where (4.6) uses B t-1 ≥ n ⇒ B t ≥ 0, (4.7) uses the same derivation as previously, (4.8) uses τ , the random round such that B τ -1 ≥ n and B τ < n, and (4.9) is by definition of B τ . In (4.8), A * t = s * 1 , . . . , s * j , where j is such that W s j ,t = 1, and j = |S * | if no such node exists.

Intuitively, Proposition 10 states that the optimal overall expected reward that can be gained (i.e., the maximum expected number of hiders found) is approximately B/J * (we assume that B n). This is quite intuitive, since this quantity is actually the ratio between the overall budget and the minimum expected cost paid to find a single hider. Indeed, one can consider the related problem of minimizing the overall expected cost paid, over several rounds, to find a single hider. It can be expressed as an infinite-time horizon Markov decision process (MDP) with action space S and two states: whether the hider is found (which is the terminal state) or not. The goal is to choose a strategy S 1 , S 2 , . . . , S t , . . ., minimizing

J (S 1 , S 2 , . . .) E τ t=1 e T At C t = ∞ t=1 e T St w * t-1 u=1 e T Su c * + d(S t ; w * , c * ) t-1 u=1 1 -e T Su w * ,
where the stopping time τ is the first round at which the hider is found. The Bellman equation is

J (S 1 , S 2 , . . .) = d(S 1 ; w * , c * ) + 1 -e T S 1 w * e T S 1 c * + J (S 2 , . . .) ,
from which we deduce there exists an optimal stationary strategy [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF] such that S t = S for all t ∈ N * . Therefore, we can minimize J (S, S, . . . ) = J (S; w * , c * ) that gives the optimal value of J * . As we already mentioned, Oracle aims at taking inputs (w, c) ∈ (R n + ) 2 . The first straightforward way to do is to consider

J (S; w, c) |S| i=1 c s i 1 -e T {s 1 ,...,s i-1 } w e T S w
.

However, notice that with the definition above, J ( • ; w, c) could output negative values (if e T

[n] w > 1), which is not desired, because the agent would then be enticed to search arms with a high cost. We thus need to design a non-negative extension of J to (w, c) ∈ (R n + ) 2 . One way is to replace 1 -e T {s 1 ,...,s i-1 } w by e T {s 1 ,...,s i-1 } c w, another is to consider 0 ∨ J (S; w, c). There is a significant advantage of considering the second way, even if it is less natural than the first one, which is that for (w, c) ∈ (R n + ) 2 , 0 ∨ J (S; w, c) ≤ J (S; w * , c * ),

if w ≥ w * and c ≤ c * . This property5 is known to be useful for analysis of many stochastic combinatorial semi-bandit algorithms (see e.g., Chen, Wang, and Yuan, 2016). Thus, we choose for Oracle the minimization of the surrogate 0 ∨ J ( • ; w, c).

Algorithm and guarantees

We now provide Oracle in Algorithm 3 and claim in Theorem 17 that it minimizes 0 ∨ J ( • ; w, c) over S. Notice that Oracle needs to call the polynomial-time algorithm scheduling(w, c, G), that minimizes the objective function d(S; w, c) over G-linear extensions S. Then, Algorithm 3 only computes the maximum value index of a list of size n that takes linear time. To give an intuition, S * follows the ordering given by scheduling(w, c, G), and stops at some point when it becomes more interesting to start a fresh new instance.

Algorithm 3 Oracle

Input: w, c and G. S = {s 1 , . . . , s n } scheduling(w, c, G). i * arg min i∈[n] 0 ∨ J ({s 1 , . . . , s i }; w, c) (ties may be broken arbitrarily). Output: the search S * {s 1 , . . . , s i * }.

Theorem 17. For every (w, c) ∈ (R n + ) 2 , Algorithm 3 outputs a search minimizing 0 ∨ J ( • ; w, c) over S.

The proof of Theorem 17 mixes known concepts of scheduling theory, such as Sidney decomposition [START_REF] Sidney | Decomposition Algorithms for Single-Machine Sequencing with Precedence Relations and Deferral Costs[END_REF], with some new results for our objective function.

Until the end of this subsection, we might abbreviate 0 ∨ J ( • ; w, c) into J + , J ( • ; w, c) into J and d( • ; w, c) into d, keeping in mind that our results will be valid for all (w, c) ∈ (R n + ) 2 . To prove Theorem 17 we first define the concept of density, well know in scheduling and search theory.

Definition 13 (Density). The density is the function defined on A ∈ P([n]) by λ(A) e T

A w/e T A c, and λ(∅) = 0. Density of A ⊂ [n] can be understood as the quality/price ratio of that set of arms: the quality is the overall probability of finding the hider in it, while the price is the total cost to fully explore it. Without precedence constraint, the so-called Smith's rule of ratio [START_REF] Smith | Various optimizers for single-stage production[END_REF] gives that S minimizes d over linear orders (i.e., permutations of [START_REF] Sidney | Decomposition Algorithms for Single-Machine Sequencing with Precedence Relations and Deferral Costs[END_REF] generalized this principle to any precedence constraint with the concept of Sidney decomposition. Recall that an initial set is the support of a search. Definition 14 (Sidney decomposition). A Sidney decomposition (Z 1 , Z 2 , . . . , Z k ) is an ordered partition of [n] such that for all i ∈

[n]) if and only if λ(s 1 ) ≥ • • • ≥ λ(s n ). 6
[k], Z i is an initial set of maximum density in G Z i • • • Z k .
Notice that the Sidney decomposition defines a poset on [n], with the constraint that an element of Z i must be processed before those of Z j for i < j. Any G-linear extension that is also a linear extension of this poset is said to be consistent with the Sidney decomposition. The following theorem was proved by [START_REF] Sidney | Decomposition Algorithms for Single-Machine Sequencing with Precedence Relations and Deferral Costs[END_REF]:

Theorem 18 [START_REF] Sidney | Decomposition Algorithms for Single-Machine Sequencing with Precedence Relations and Deferral Costs[END_REF]. Every minimizer of d over G-linear extensions is consistent with some Sidney decomposition. Moreover, for every Sidney decomposition (Z 1 , . . . , Z k ), there is a minimizer of d over G-linear extensions that is consistent with (Z 1 , . . . , Z k ).

Notice that Theorem 18 does not provide a full characterization of minimizers of d over G-linear extensions, but only a necessary condition. Nothing is stated about how to chose the ordering inside each Z i 's, and this highly depends on the structure of G [START_REF] Lawler | Sequencing jobs to minimize total weighted completion time subject to precedence constraints[END_REF][START_REF] Ambühl | Single machine precedence constrained scheduling is a vertex cover problem[END_REF][START_REF] Ambühl | On the Approximability of Single-Machine Scheduling with Precedence Constraints[END_REF]. We are now ready to prove Theorem 17, thanks to Lemma 5.

Lemma 5. For any Sidney decomposition (Z 1 , . . . , Z k ), there exists i ≤ k and a search with support

Z 1 • • • Z i that minimizes J + .
Proof of Theorem 17. We know from first statement of Theorem 18 that S scheduling(w, c, G)

given in Algorithm 3 is consistent with some Sidney decomposition (Z 1 , . . . , Z k ). Let i ≤ k and S * minimizing J + of support Z 1 • • • Z i given by Lemma 5. Consider the following decomposition of S: S = S S with S being the restriction of S to

Z 1 • • • Z i (and thus S is its restriction to Z i+1 • • • Z k ).
Let's prove that S is also a minimizer of J + by showing J + (S ) ≤ J + (S * ), thereby concluding the proof. By definition of S, we have 0

≤ d(S * S ) -d(S S ) = d(S * ) -d(S ), so d(S ) + (1 -w T e Z 1 ••• Z i )c T e Z 1 ••• Z i w T e Z 1 ••• Z i ≤ d(S * ) + (1 -w T e Z 1 ••• Z i )c T e Z 1 ••• Z i w T e Z 1 ••• Z i ,
i.e., J (S ) ≤ J (S * ), and because x → 0 ∨ x is non-deacreasing on R, we have J + (S ) ≤ J + (S * ).

The proof of Lemma 5 also uses Sidney's Theorem 18, but this time the second statement. However, although it provides a crucial analysis, with fixed support, concerning the order to choose for minimizing d and therefore J + , nothing is said about the support to choose. Thus, to prove Lemma 5, we also need the following Proposition 11, that gives the key support property satisfied by J + .

Proposition 11 (Support property). If SS , SS S ∈ S with λ(S ) ≥ λ(S ), then

J + (SS ) ≥ J + (S) ∧ J + (SS S ).
(4.10)

Proof. If J (SS S ) < 0, then J + (SS S ) = 0 ≤ J + (SS ) and (4.10) is true. We thus assume J (SS S ) ≥ 0. Since J (S )

≤ 1 λ(S ) , 0 ≤ J (SS S ) = J (SS )
λ(z 1,1 ) = • • • = λ(z 1,j 1 ) ≥ • • • ≥ λ(z k,1 ) = • • • = λ(z k,j k ).
Let S * be a maximum-size minimizer of J + of support A * . Assume A * is not of the form given by Lemma 5, and let x be the first, for the order

(z 1,1 , . . . , z 1,j 1 , . . . , z k,1 , . . . , z k,j k ), in some Z i \A * while A * ∩ (Z i • • • Z k ) = ∅.

Proof of Lemma 5

Before proving Lemma 5, we state some preliminaries about initial sets of the DAG G.

Fact 1. A is an initial set in G if and only if for all a ∈ A, the predecessors of a in G are also in A.

Let us recall that

L ⊂ P([n]) is a lattice if A, A ∈ L ⇒ (A ∩ A , A ∪ A ∈ L).
Proposition 12. The set of initial sets in G is a lattice.

Proof. Let A and A be two initial sets in G. If a ∈ A ∪ A (respectively a ∈ A ∩ A ), then the predecessors of a are included in predecessors of A or (respectively and) the predecessors of A , i.e., in A or (respectively and) A , so in A ∪ A (respectively A ∩ A ).

Even if we do not use the following proposition,7 we provide it nonetheless, since it illustrates how to handle density λ. A ∩ A and A ∪ A are initial sets, so by maximality of density of A,

w T e A∪A c T e A∪A ∨ w T e A∩A c T e A∩A ≤ w T e A c T e A .
Therefore, the equality holds, and it needs to be the case that

w T e A∪A c T e A∪A = w T e A∩A c T e A∩A = w T e A c T e A ,
so both A ∩ A and A ∪ A have maximum density.

We now provide a proof of Lemma 5.

Proof of Lemma 5. Let j be the largest integer such that there is a search minimizing

J + of the form S = S 1 • • • S j S with S i of support Z i for all i ∈ [j]. Let S = S 1 • • • S j S be
S j+1 S of d S 1 • • • S j • over G Z j+1 ∪ A -linear extensions, with S j+1 of support Z j+1 . Z j+1 ∩ A is an initial set (as an intersection of two initial sets) of G Z j+1 • • • Z k . Therefore, by maximiality of the density of Z j+1 in G Z j+1 • • • Z k , we have λ(Z j+1 ∩ A ) ≤ λ(Z j+1 ).
This translates, by property of the density, into

λ(Z j+1 ) = λ((Z j+1 ∩ A ) (Z j+1 \A )) ≤ λ(Z j+1 \A ), and thus λ(A ) ≤ λ(Z j+1 ) ≤ λ(Z j+1 \A ). If we let S be a search of G (Z j+1 \A ) Z j+2 • • • Z k with support Z j+1
\A (one can take the order induced by Z j+1 by removing the arms of A ), then by Proposition 11, associated with

d S 1 • • • S j S j+1 S ≤ d S 1 • • • S j S S , we have that J + (S) ≥ J + (S 1 • • • S j ) ∧ J + (S 1 • • • S j S S ) ≥ J + (S 1 • • • S j ) ∧ J + (S 1 • • • S j S j+1 S ),
contradicting either the definition of j or the minimality of |S|.

Online search-and-stop

In this section, we consider an additional challenge where the distribution P (C,W) is unknown and the agent must deal with it, while minimizing R B (π) over sampling policies π, where B is a fixed budget. Recall that a policy π selects a search S t = s 1,t , . . . , s |St|,t at the beginning of round t, using previous observations H t , and then performs the search that choose the set of arms A t = {s 1,t , . . . , s j,t }, where W j,t = 1 or j = |S t |. We treat the setting as a variant of stochastic combinatorial semi-bandits (Gai, Krishnamachari, and Jain, 2012). The feedback received by an agent at round t is random, because it depends on W t , and thus it is not measurable w.r.t. H t . More precisely, (W i,t , C i,t ) is observed only for arms i ∈ A t . Notice that since W t is a one-hot vector, the agent can always deduce the value of W i,t for all i ∈ S t . As a consequence, we will maintain two types of counters for all arms i ∈ [n] and all t ≥ 1,

N ⊕ i,t-1 t-1 t =1 I{i ∈ S t }, N i,t-1 t-1 t =1 I{i ∈ A t }. (4.13) w i,t-1 t-1 t =1 I{i ∈ S t }W i,t N ⊕ i,t-1 , c i,t-1 t-1 t =1 I{i ∈ A t }C i,t N i,t-1 . (4.14)
We propose an approach similar to ucb-v of Audibert, Munos, and Szepesvári (2009a), based on cucb of Chen, Wang, and Yuan (2016), called cucb-v, that uses a variance estimation of w * in addition to the empirical average. Notice that the variance of W i for an arm i is σ 2 i w * i (1 -w * i ). Furthermore, since W i is binary, the empirical variance of W i after t rounds is w i,t (1 -w i,t ). For every round t and every edge i ∈ [n], with the previously defined empirical averages, we use the confidence bounds 8 defined as

c i,t 0 ∨ c i,t-1 - 0.5ζ log t N i,t-1 , w i,t w i,t-1 + 2ζw i,t-1 (1 -w i,t-1 ) log t N ⊕ i,t-1 + 3ζ log t N ⊕ i,t-1 ∧ 1,
where we choose the exploration factor to be ζ 1.2. Notice that we could take any ζ > 1 as shown by Audibert, Munos, and Szepesvári (2009a). We provide the policy π cucb-v that we consider in Algorithm 4.

Analysis

Notice that since an arm i ∈ S t is pulled (and thus C i,t is revealed to the agent) with probability 1 -e T {s 1,t ,...,s i-1,t } w * over round t, we fall, as anticipated in Remark 10, into the setting of probabilistically triggered arms w.r.t. costs, described by Chen, Wang, and Yuan (2016) and [START_REF] Wang | Improving regret bounds for combinatorial semi-bandits with probabilistically triggered arms and its applications[END_REF]. Thus we could rely on these prior results. However, the main difficulty in our setting is that we also need to deal with probabilities W i,t , that the agent actually observes for every arm i in S t , either because it actually pulls arm i, or because it deduces the value (that is thus 0) from other pulls of round t. In particular, if we follow the analysis of Chen, Wang, and Yuan (2016) and [START_REF] Wang | Improving regret bounds for combinatorial semi-bandits with probabilistically triggered arms and its applications[END_REF], the double sum in the definition of J leads to expected regret bound that is quite large. Indeed, assuming that all costs are deterministically equal to 1, if we suffer an error of δ when approximating each w * i , then the global error can be as large as n O(nδ) for the approximation error w.r.t. costs, that is more common in combinatorial semi-bandits. Thus, we rather combine their work with the variance estimates of w * i . Often, this does not provide a significant improvement over ucb in terms of expected regret (otherwise we could do the same for the costs), but since in our case, the variance is of order 1/n, the gain is non-negligible. 9 We let c min > 0 be any deterministic lower bound on the set e T At c * , t ≥ 1 . Furthermore, we let T B 2B/c min and for any search S, we define the gap of S as

∆(S) e T S w * J (S; w * , c * ) J * -1 = 1 J * |S| i=1 c * s i   1 - i-1 j=1 w * s j   - |S| i=1 w * s i ≥ 0,
that represents the local regret of selecting a sub-optimal search S at some round. In addition, for each arm i ∈ [n], we define

∆ i,min inf S / ∈S * : i∈S ∆(S) > 0.
We provide bounds for the expected regret of π cucb-v in Theorem 19. The first bound is gap-dependent, and is characterized by c min , J * , and

σ 2 i , ∆ i,min , i ∈ [n].
Its main term scales logarithmically w.r.t. B. The second bound is true for any sequential search-and-stop problem instance having a fixed value of c min > 0 and J * > 0.

Theorem 19. The expected regret of cucb-v satisfies

R B (π cucb-v ) = O   n log T B i∈[n] 1 + (J * + n) 2 σ 2 i J * 2 ∆ i,min + (J * + n) J * n log n∆ i,max ∆ i,min   .
In addition,

sup R B (π cucb-v ) = O √ n 1 + n J * T B log T B ,
where the sup is taken over all possible sequential search-and-stop problems with fixed c min and J * .

In the proof (that can be found in subsection 4.5.1), the main challenge comes from the estimation of w * and not from c * . We recall that our analysis does not use triggering probability groups of Wang and Chen, 2017 for dealing with costs, since we succeed in dealing with the triggering probability by seeing it as the expectation of the event where the corresponding counter is updated (see the end of Chapter 3). For hider probabilities, there is no triggering probability (they are equal to 1). 10 Notice, the analysis of Wang and Chen, 2017 only considers a deterministic horizon. In our case, we need to deal with a random-time horizon. For that, notice that the regret upper bounds that hold in expectation are obtained by splitting the expectation into two parts. The first part is filtered with a high-probability event on which the regret grows as the logarithm of the random horizon and the second one is filtered with a low-probability event, on which we bound the regret by a constant. Since the log function is concave, we can upper bound the expected regret by a term growing as the logarithm of the expectation of the random horizon, with Jensen's inequality. Finally, we upper bound the expectation of the random horizon to get the rate of log T B .

Tightness of our regret bounds

Since we succeeded in reducing the dependence on n in the expected regret with confidence bounds based on variance estimates, we can now ask whether this dependence in Theorem 19 is tight. We stress that our solution to sequential search-and-stop is computationally efficient. In particular, both the offline oracle optimization and the computation of the optimistic search S t in the online part are tractable.

Whenever rewards are not arbitrary correlated (as is the case in our setting), we can potentially exploit these correlations in order to reduce the gap-dependent regret's dependence on n even further. This could be done by choosing a tighter confidence region such as a confidence ellipsoid (Degenne and Perchet, 2016b), or a KL-confidence ball [START_REF] Combes | Combinatorial Bandits Revisited[END_REF] instead of coordinate-wise confidence intervals. Unfortunately, these do not lead to computationally efficient algorithms. It also seems from our Theorem 20 that there is an extra √ n factor in our gap-free bound. It is an open question whether a better gap-free regret bound exists.

To show that we are only a √ n factor away, in the following theorem we provide a class of sequential search-and-stop problems (parameterized by n and B) on which the regret bound provided in Theorem 19 is tight up to a √ n factor (and a logarithmic one).

Theorem 20. For simplicity, let us assume that n is even and that B is a multiple of n. For any optimal online policy π, there is a sequential search-and-stop problem with n arms and budget B such that

-4 + 1 28 B n ≤ R B (π) = O B log B n .
For the proof (given in subsection 4.5.2), we consider a DAG composed of two disjoint paths (Figure 4.1), with all costs deterministically set to 1 and with the hider located either at s n/2 or s n/2 . This information is given to the agent. We then reduced this setting to a two-arm bandit over at least B/n rounds. Notice that bounds provided in Theorem 20 decrease with n. This is because, in the sequential search-and-stop problem, the increasing dependence on n is counterbalanced by the fact that the number of rounds is of order B/n, and that J * is of order n.

s 1 s 2 s 3 s n 2 -1 s n 2 s 1 s 2 s 3 s n 2 -1 s n 2 path S path S

Algorithm Definition of w

i,t cucb w i,t-1 + 0.5ζ log t N ⊕ i,t-1 ∧ 1 cucb-kl
The unique solution x to

N ⊕ i,t-1 kl(w i,t-1 , x) = ζ log t such that x ∈ [w i,t-1 , 1] cts An independent sample from Beta(α, N ⊕ i,t-1 -α), where α = N ⊕ i,t-1 w i,t-1

Experiments and discussion

In this section, we present an experiment for sequential search-and-stop. We compare our cucb-v with three other online algorithms, which are same as cucb-v except for the estimator w t to be plugged in Oracle. We give corresponding definitions of w t in Table 4.1, where we take ζ 1.2, and where kl(p, q) p log p q + (1 -p) log 1 -p 1 -q is the Kullback-Leibler divergence between two Bernoulli distributions of parameters p, q ∈ [0, 1] respectively. In this table, cts stands for combinatorial Thompson sampling [START_REF] Wang | Thompson Sampling for Combinatorial Semi-Bandits[END_REF]). We run simulations for all the algorithms with n = 100 and without precedence constraints, i.e., when the DAG is an edgeless graph. Notice that in this case, a search can be any ordered subset of arms (thus, the set of possible searches is of cardinality n k=0 n!/k! ≤ en!). This restriction does not remove complexity from the online problem, but rather from the offline one, so even in that case, the online problem is challenging. We take parameter w * defined as

                 w * i = 1 2 i for i ∈ [m -1] w * m = 1 2 + ε w * m-1 w * i = 1 2 -ε w * m-1 n -m for i ∈ {m + 1, . . . , n},
where we chose m 40. For ε ∈ (0, 1/2), and for all expected cost being equal to each other, one can see that S * = {[m]}. Intuitively, w * i models the proportion of users answering i to some fixed request:11 When ε = 0, half of the population answers 1, a quarter answers 2, . . . , until m, and remaining users answer uniformly on remaining arms {m + 1, . . . , n}. We chose ε = 0.1, c * i = 1/2 for all i ∈ [n] and take C i ∼ Bernoulli(c * i ). In Figure 4.2, for each algorithm considered, we plot the quantity

B J * - τ B -1 t=1 e T St[Wt] W t ,
with respect to budget B, averaged over 100 simulations. As shown in Proposition 10, the curves obtained this way provide good approximations to the true regret curves. We notice that cucb-kl, cucb-v, and cts are significantly better than cucb, since the latter explores too much. In addition, the regret curves of cucb-kl, cucb-v and cts are quite similar. In particular, their asymptotic slopes seem equal, which hints that regret rates are comparable on this instance. Finally, let us note that it is not surprising that cucb-kl outperforms cucb-v, as we have binary variables here (cf. Chapter 2). More precisely, a cucb-kl analysis would have given a slight improvement in the regret (using the kl function), but that the dependency in the dimension n does not change.

Discussion and future work

We presented sequential search-and-stop problem and provided a stationary offline solution. We gave theoretical guarantees on its optimality and proved that it is computationally efficient. We also considered the learning extension of the problem where the distribution of the hider and the cost are not known. We provided cucb-v, an upper-confidence bound approach, tailored to our case and gave expected regret guarantees with respect to the optimal policy.

We now discuss several possible extensions of our work. We could consider several hiders rather than just one. Another would be to explore the cts approach [START_REF] Chapelle | An empirical evaluation of Thompson sampling[END_REF]Agrawal and Goyal, 2012a;[START_REF] Komiyama | Optimal Regret Analysis of Thompson Sampling in Stochastic Multi-armed Bandit Problem with Multiple Plays[END_REF][START_REF] Wang | Thompson Sampling for Combinatorial Semi-Bandits[END_REF] further in the learning case by considering a Dirichlet prior on the whole arm set. The Dirichlet seems appropriate because a sample w from this prior is in the simplex. The main drawback however is the difficulty of efficiently updating such prior to get the posterior, because in the case when the hider is not found, the one-hot vector is not received entirely.

Missing proofs

Proof of Theorem 19

Proof of Theorem 19. We start with showing a lower bound on the expected reward of any policy π,

F B (π) ≥ t≥1 E I{B t-1 ≥ n}e T St w * (4.15) = t≥1 E I{B t-1 ≥ n, S t ∈ S * }e T St w * + t≥1 E I{B t-1 ≥ n, S t / ∈ S * }e T St w * = 1 J * t≥1 E I{B t-1 ≥ n, S t ∈ S * } d(S t ; w * , c * ) + (1 -e T St w * )e T St c * + 1 J * t≥1 E I{B t-1 ≥ n, S t / ∈ S * } d(S t ; w * , c * ) + (1 -e T St w * )e T St c * - t≥1 E[I{B t-1 ≥ n, S t / ∈ S * }∆(S t )] = 1 J * t≥1 E I{B t-1 ≥ n} d(S t ; w * , c * ) + (1 -e T St w * )e T St c * - t≥1 E[I{B t-1 ≥ n, S t / ∈ S * }∆(S t )] ≥ B -n J * - t≥1 E[I{B t-1 ≥ n, S t / ∈ S * }∆(S t )], (4.16)
with (4.15) obtained as (4.6) and (4.7), and (4.16) as (4.9). Therefore, since F * B ≤ (B + n)/J * by Proposition 10, we have that

R B (π) - 2n J * ≤ t≥1 E[I{B t-1 ≥ n, S t / ∈ S * }∆(S t )] ≤ t≥1 E[I{B t ≥ 0}∆(S t )] = E τ B -1 t=1 ∆(S t ) .
It is thus sufficient to bound this last expectation, which has the form of the classical regret (the non-budgeted one), and is thus easier to handle (note that we find here a random horizon, as anticipated in the end of the previous chapter).

For some round t ≥ 1, recall that S t minimizes 0 ∨ J ( • ; w t , c t ). In particular, 0 ∨ J (S t ; w t , c t ) ≤ 0 ∨ J (S * ; w t , c t ).

We define the event

M t {w t ≥ w * , c t ≤ c * },
under which 0 ∨ J (S * ; w t , c t ) ≤ J * : indeed, we can first use c t ≤ c * to write J (S * ; w * , c t ) ≤ J (S * ; w * , c * ) = J * because w * belongs to the simplex (thus, 1 -e T {s 1 ,...,s i-1 } w * ≥ 0 for all i). Then, using c t ≥ 0 with w t ≥ w * , we can write J (S * ; w t , c t ) ≤ J (S * ; w * , c t ). The result follows since x → x ∨ 0 is non-deacreasing on R. As a consequence, under the event M t , we have 0 ∨ J (S t ; w t , c t ) ≤ J * .

We thus have that ∆(S t ) equals

1 J *   |St| i=1 c * s i,t 1 -w * T e {s t,1 ,...,s t,i-1 } -J * w * T e St   = 1 J *   |St| i=1 c * s i,t 1 -w * T e {s t,1 ,...,s t,i-1 } -J * w T t e St   + (w t -w * ) T e St ≤ 1 J *   |St| i=1 c * s i,t 1 -w * T e {s t,1 ,...,s t,i-1 } -0 ∨ J (S t ; w t , c t )w T t e St   + (w t -w * ) T e St ≤ 1 J *   |St| i=1 c * s i,t 1 -w * T e {s t,1 ,...,s t,i-1 } -J (S t ; w t , c t )w T t e St   + (w t -w * ) T e St = 1 J * |St| i=1 c * s i,t 1 -w * T e {s t,1 ,...,s t,i-1 } -c s i,t ,t 1 -w T t e {s t,1 ,...,s t,i-1 } + (w t -w * ) T e St = 1 J *   |St| i=1 c * s i,t -c s i,t ,t 1 -w * T e {s t,1 ,...,s t,i-1 } + |St| i=1 c s i,t ,t (w t -w * ) T e {s t,1 ,...,s t,i-1 }   + (w t -w * ) T e St ≤ 1 J *   |St| i=1 c * s i,t -c s i,t ,t 1 -w * T e {s t,1 ,...,s t,i-1 } + (n + J * )(w t -w * ) T e St   .
We now define some more events:

N 1,t ∀i ∈ S t , c * i -c i,t ≤ 1 ∧ 2ζ log t N i,t-1 • N 2,t ∀i ∈ S t , w i,t -w * i ≤ 1 ∧ 8ζσ 2 i log t N ⊕ i,t-1 + 13 3 ζ log t N ⊕ i,t-1 • N t N 1,t ∧ N 2,t
Under N t , we further have

∆(S t ) ≤ 1 J * |St| i=1 1 -w * T e {s t,1 ,...,s t,i-1 } P[ s i ∈At|Ft] 1 ∧ 2ζ log t N s i ,t-1 + n + J * J * i∈St 1 ∧ 8ζσ 2 i log t N ⊕ i,t-1 + 13 3 ζ log t N ⊕ i,t-1 .
On can notice that the factor 1 -w * T e {s t,1 ,...,s t,i-1 } is actually the probability of observing the cost of s i , which is P[

s i ∈ A t |F t ] = P[ s i ∈ A t |H t ]
(since the policy is not randomized). If we summarize from that point on, we used the events M t and N t to prove the above upper bound on ∆(S t ). So we can break down the regret into two pieces, whether or not M t ∧ N t occurs. Let's first treat the case where it occurs (first piece): We want to use Theorem 12 from Chapter 3. First of all, see that τ B -1 is such that {t ≤ τ B -1} ∈ F t+1 , so we use the horizon τ B instead of τ B -1 in this theorem. We can see in the above bound that we have a composed bonus (with 3 terms). In addition to that, we can take apart the null counters to have an initialization term of the form of the one in Proposition 5. We can thus use Proposition 6 to treat each of this four term separately, each one giving a final term in the regret. From Proposition 5, the initialization term induces the regret term i∈[n] 2 J * . Theorem 12 can be used for the three other terms (with b i (S t ) being a constant):

• with β i,τ B = log(τ B )/J * 2 , α i = 1/2 and p s j (S t ) = 1 -w * T e {s t,1 ,...,s t,i-1 } , the first term induces a regret term of order

i∈[n] nE[log(τ B )] J * 2 ∆ i,min .
• with β i,τ B = (n + J * ) 2 σ 2 i log(τ B )/J * 2 , α i = 1/2 and p s j (S t ) = 1, the second term induces a regret term of order

i∈[n] n(n + J * ) 2 σ 2 i E[log(τ B )] J * 2 ∆ i,min
.

• with β i,τ B = (n + J * ) log(τ B )/J * , α i = 1 and p s j (S t ) = 1, the last term induces a regret term of order i∈ [n] (n + J * )E[log(τ B )] J * log n∆ i,max ∆ i,min .

We will now turn to the second piece. For this, we are going to use several Theorems from Chapter 2. In particular, we can use Corollary 1 (Hoeffding's inequality) and Theorem 8 (empirical Bernstein inequality) to get that M t holds with probability at least 1 -4n ζ log 2 (t) t -ζ . On the other hand, using the other inequality from Corollary 1, Theorem 6 (Bernstein's inequality) and Proposition 3, we get that N t holds with probability at least 1 -3n ζ log 2 (t) t -ζ . Therefore, the event M t ∧ N t holds with probability at least 1 -7n ζ log 2 (t) t -ζ and the regret under the complementary event can be bounded by

7n∆ max t≥1 ζ log 2 (t) t -ζ < ∞.
This terminates the analysis for the gap-dependent regret bound (we note that there's still E[log(τ B )] to be bounded, which we'll do right after we deal with the gap-free regret bound).

Gap-free bound

For the gap-free bound, we consider the event that the gap ∆(S t ) is greater than

(n + J * ) n i∈[n] σ 2 i log(τ B ) J * τ 1/2 B .
(4.17)

Under this event, we have the same analysis as above, but with ∆ i,min replaced by the quantity (4.17). The obtained rate is the same under the complementary event, since the regret is thus trivially bounded by τ B times the quantity (4.17). The result thus follows noticing that i∈[n] σ 2 i ≤ 1.

Control on the random time horizon

We note that in both the gap-free and gap-dependent bound above, one must control the expectation of a concave increasing function of τ B . Thus, with Jensen's inequality, we have to control E[τ B ]. This is the purpose of the following.

E[τ

B ] = 1 + E   t≥1 I{B t ≥ 0}   = 1 + t≥1 P B -tc min + tc min ≥ t t =1 C T t e A t ≤ T B + 1 + t≥T B +1 exp -2(B -tc min ) 2 t (4.18) ≤ T B + 1 + t≥T B +1 exp -c 2 min t 2 (4.19) ≤ T B + 1 + 2 c 2 min exp c 2 min 2 - c 2 min (T B + 1) 2 (4.20) ≤ T B + 1 + 2 c 2 min exp(-c min B),
where (4.18) makes use of Fact 2, (4.19) is obtained because 2(B -tc min ) 2 ≥ c 2 min t 2 /2 for t ≥ 2B/c min and we get (4.20) since 1/(1 -e -c 2 min /2 ) ≤ 2e c 2 min /2 /c 2 min . Fact 2 [START_REF] Hoeffding | Probability inequalities for sums of bounded random variables[END_REF][START_REF] Flajolet | Logarithmic regret bounds for Bandits with Knapsacks[END_REF]. Let X 1 , . . . , X t be the random variables with common support [0, 1] and such that there exists a ∈ R with ∀u ∈

[t], E[X u |X 1 , . . . , X u-1 ] ≥ a. Let x t 1 t (X 1 + • • • + X t ), then ∀ε ≥ 0 , P[x t -a ≤ -ε] ≤ e -2ε 2 t .
The above proved upper bound on the expected horizon is sufficient for our purpose, as we are interested in the asymptotic rate of regret when B is large. We can nevertheless note that in the regime where c min is small, our gap dependent bound has only a logarithmic dependence in c -1 min .

Proof of Theorem 20

Proof of Theorem 20. Let 0 < ε < 1/4. We consider a DAG composed of two disjoint paths (Figure 4.1), both with n/2 nodes. We denote the two paths by S and S . We deterministically set all the costs to 1,

w i = 0 for i / ∈ s n 2 , s n 2
. All this information is given to the agent. Notice that this does not make the problem harder. Now consider two distributions D 1 and D 2 defined by

D 1 : w s n 2 1 2 + ε, w s n 2 1 2 -ε and D 2 : w s n 2 1 2 -ε, w s n 2 1 2 + ε.
Notice that an optimal online policy does not modify its behavior during a round t, since after having seen W i,t = 1, continuing searching would only give information about cost distribution which is known by the problem definition, and no additional information about the rewards. Therefore, there is an optimal online policy that selects some search S t and perform A t over round t. Observe that S * = SS for D 1 and S * = S S for D 2 . We have J * = 3 4 n -εn ≥ 1 2 n for both D 1 and D 2 . We now show that we can restrict ourselves to policies that take searches in {SS , S S}.

• First, an optimal online policy does not select a search that would not include at least one of the leaves s n 2 , s n 2 for a round. Therefore, it has a full information on W . Indeed, a search of support included in s n 2 , s n 2 c is noninformative and does not bring any reward while having a cost. within round t, π gains 0 and pays at least n/2, whereas π gains 1 and pays n within round t. Thus, the budget of π compared to π is augmented by at most n/2, with which it can increase its reward by at most 1.

The overall reward of π remains higher than that of π for both cases.

A direct consequence of the restriction to {SS , S S} is that c min = n/2, giving the upper bound in Theorem 20 by invoking the result of Theorem 19. Now for a policy π using searches from {SS , S S}, we have

F B (π) = ∞ t=1 E   i∈St I{B t ≥ 0, W i,t = 1}   ≤ ∞ t=1 E   i∈St I{B t-1 ≥ 0, W i,t = 1}   = t≥1 E I{B t-1 ≥ 0, S t ∈ S * }e T St w * + t≥1 E I{B t-1 ≥ 0, S t / ∈ S * }e T St w * = 1 J * t≥1 E I{B t-1 ≥ 0, S t ∈ S * } d(S t ; w * , c * ) + (1 -e T St w * )e T St c * + 1 J * t≥1 E I{B t-1 ≥ 0, S t / ∈ S * } d(S t ; w * , c * ) + (1 -e T St w * )e T St c * - t≥1 E[I{B t-1 ≥ 0, S t / ∈ S * }∆(S t )] = 1 J * t≥1 E I{B t-1 ≥ 0} d(S t ; w * , c * ) + (1 -e T St w * )e T St c * - t≥1 E[I{B t-1 ≥ 0, S t / ∈ S * }∆(S t )] ≤ B + n J * - t≥1 E[I{B t-1 ≥ 0, S t / ∈ S * }∆(S t )].
As a result we get

B B (π) = F * B -F B (π) ≥ B -n J * - B + n J * + t≥1 E[I{B t-1 ≥ 0, S t / ∈ S * }∆(S t )] = - 2n J * + t≥1 E[I{B t-1 ≥ 0, S t / ∈ S * }∆(S t )]•
Since we restrict π to take a search in {SS , S S}, we have a single gap (the same for D 1 and D 2 ):

∆ = n 2 1 2 -ε + n 1 2 + ε n 2 1 2 + ε + n 1 2 -ε -1 = 1.5 + ε 1.5 -ε -1 = 2ε 1.5 -ε ≥ 4ε 3 • (4.21)
Furthermore we can bound the number of rounds from below by B/n. To proceed we use high-probability Pinsker inequality Tsybakov, 2009, Lemma 2.6.

Fact 3 (high-probability Pinsker inequality). Let P and Q be probability measures on the same measurable space, and let A be an event. Then

P (A) + Q(¬A) ≥ 1 2 exp(-KL(P Q)),
where KL is the Kullback-Leibler divergence.

We let R 1,B (π) be the regret of π for distribution D 1 and similarly, R 2,B (π) for D 2 . If P 1 and P 2 denote the probability when random variable are samples from D 1 and D 2 respectively, we have

max{R 1,B (π), R 2,B (π)} ≥ R 1,B (π) + R 2,B (π) 2 (4.22) ≥ - 2n J * + ∆ 2 B/n t=1 (P 1 [B t-1 ≥ 0, S t = S S] + P 2 [B t-1 ≥ 0, S t = SS ]) ≥ - 2n J * + ε 3 B/n t=1 exp -KL D ⊗t 1 D ⊗t 2 (4.23) = - 2n J * + ε 3 B/n t=1 exp(-tKL(D 1 D 2 )),
where (4.23) is due to Fact 3 and (4.21). Then,

KL(D 1 D 2 ) = 1 2 + ε log 1 2 + ε 1 2 -ε + 1 2 -ε log 1 2 -ε 1 2 + ε ≤ 2ε 1 2 + ε 1 2 -ε -2ε 1 2 -ε 1 2 + ε = 4ε 2 1 4 -ε 2 ≤ 64 3 ε 2 log(x) ≤ x -1.
Thus, with J * ≥ n 2 , we have

max{R 1,B (π), R 2,B (π)} ≥ -4 + ε 3 B/n t=1 exp - 64 3 tε 2 ≥ -4 + ε 1 -exp -64 3 Bε 2 n 3 exp 64 3 ε 2 -1 ≥ -4 + 1 -exp -64 3 Bε 2 n 64ε ≥ -4 + min 1 128ε , εB 6n .
Taking ε = (6n)/(128B), the lower bound becomes

max{R 1,B (π), R 2,B (π)} ≥ -4 + B 768n ≥ -4 + 1 28 B n • Chapter 5
The

Structure of Uncertainty

This chapter aims to introduce some tools necessary to the construction of more sophisticated confidence regions than those we saw in the previous chapter. We will then see that although the use of these confidence regions can lead to improved regret bounds, they also can lead to efficiency issue in the algorithms, and that in some cases, more complex optimization methods have to be implemented. The beginning of the chapter is mainly based on known structured bandit results. More precisely, we will rely on Degenne and Perchet (2016b) and on [START_REF] Magureanu | Lipschitz Bandits: Regret Lower Bound and Optimal Algorithms[END_REF]. The end of the chapter is based on our article Perrault, Perchet, and Valko (2019a).

About confidence regions

Here we give a more precise idea of the type of confidence region we are interested in. In the previous chapter, we could see that the optimistic estimates of the means were constructed from either the Hoeffding inequality or the Bernstein's one (in the end, we also mentioned the use of kl-based confidence intervals). In other words, if we look at the confidence regions from the vectorial point of view, they all have the shape of a hypercube (i.e., a Cartesian product of confidence intervals). Note that these confidence regions are all from Chapter 2, i.e., they are all already used for the standard MAB setting. Here we want to investigate other types of confidence regions, which will be based more on the 2 -norm (ellipsoid) rather than the ∞ -norm (hypercube). Indeed, we saw in the previous chapter that a region based on the ∞ -norm induces the use of bonuses of type 1 . We'll see that, naturally, the use of regions based on the 2 -norm will induce the use of 2 -type bonuses. This is interesting because, as we saw in Chapter 3, there is a gain in using the 2 -bonuses compared to the 1 -bonuses. For example, in the case where A t = S t is of maximum cardinality m, b i (S t ) = 1 and α i = 1/2 (note that these settings are the most standard to consider), the use of an 2 -bonus improves the regret by a factor m log -2 (m) (more precisely, the linear dependence of the regret in the batch size m can be replaced by a polylogarithmic dependence log 2 (m)). In the following, we will see two methods for building ellipsoidal confidence regions. Of course, these constructions (and the resulting gain in the regret bound) are not free. Specifically, the 2 -regions are generally based on an assumption about the outcomes X. As a reminder, the assumptions for the ∞ -regions only concern the marginals P X i (like the κ 2 -sub-Gaussianity, the boundedness, ...), and nothing is said about the type of dependence that exists between these marginals. So, by providing additional information about these dependencies, we get more structure about the uncertainty associated with the outcomes X. This structure of uncertainty can be exploited in order to improve the performance of the policies. The most natural way to proceed, which we will follow here for sub-Gaussianity, is to extend the structures used for the marginals to the multivariate case.

For the two methods we look at, we make the following assumption on the vector of outcomes X.

Assumption 1 (Subgaussian outcomes).

There is a matrix C 0 and a known matrix Γ + C with Γ ij ∈ R + for all i, j ∈ [n], such that E e λ T (X-µ * ) ≤ e λ T Cλ .

The motivation for sub-Gaussian outcomes is the following: In the same way as boundedness generalizes to sub-Gaussianity in 1d, we have that if X is a.s. in a compact K, it is C-sub-Gaussian, with C built from the John's ellipsoid of K.

Here, we focus on the linear reward case, for sake of simplicity, with an optimal action denoted A * (so that ∆ t = µ * T (e A * -e At )). This construction will inspire us to generalize it to a more complex reward function. A common statistic computed from the feedback received up to the beginning of round t is the empirical average of each arm i ∈ [n], defined as

µ i,0 = 0, ∀t ≥ 2, µ i,t-1 = t ∈[t-1] I{i ∈ A t }X i,t N i,t-1 where ∀t ≥ 1, N i,t-1 t ∈[t-1] I{i ∈ A t }.

Laplace's method

We review here the most used method for ellipsoidal confidence region for µ * : the Laplace's method. A very detailed explanation of Laplace's method is given in Lattimore and Szepesvári (2019) and in Abbasi-Yadkori, Pál, and Szepesvári (2011). Here, we are going to focus on its use in our context of semi-bandit feedback. The difficulty of designing confidence sets for µ * stems from the fact that the actions are correlated to the outcomes (because the agent is adaptive).

Since we are considering a linear reward function, we want to compare µ * and µ t-1 in some fixed direction e A , where A ∈ A. Indeed, we can first obtain the following proposition in the same way as Theorem 9.

Proposition 14. The regret under the event (µ t-1 -µ * ) T e At > ∆ t /2 is bounded by

∆ max 8nm 2 max i C ii /∆ 2 min . Proof. Using Assumption 1, T t=1 ∆ t P (µ t-1 -µ * ) T e At > ∆ t /2 ≤ T t=1 ∆ t i∈At P µ i,t-1 -µ * i > ∆ min /(2m) ≤ T t=1 ∆ max i∈[n] P i ∈ A t , µ i,t-1 -µ * i > ∆ min /(2m) ≤ ∆ max i∈[n] t∈[T ] e -∆ 2 min t/(8m 2 C ii ) ≤ ∆ max 8nm 2 max i C ii ∆ 2 min .
We can thus safely assume that (µ t-1 -µ * ) T e At ≤ ∆ t /2. Then, bounding e T A * (µ * -µ t-1 ) by some bonus t (A * ) with high probability, and using a policy that chooses A t maximizing A → µ T t-1 e A + bonus t (A), we have

∆ t = µ * T (e A * -e At ) ≤ µ T t-1 e A * + bonus t (A * ) -µ * T e At ≤ µ T t-1 e At + bonus t (A t ) -µ * T e At ≤ bonus t (A t ) + ∆ t /2,
This provides the bound ∆ t ≤ 2bonus t (A t ), which is what we want to apply theorems from the end of Chapter 3. The bound on e T A * (µ * -µ t-1 ) is obtained with the following Theorem 21.

Theorem 21 (Laplace's method). For any A ∈ A, with probability at least 1 -

1 t log 2 (t) ∧ 1, we have that e T A (µ * -µ t-1 ) is upper bounded by 2δ(t) e T A diag t-1 t =1 e A t -1 diag ηΓ ii t-1 t =1 I{i∈A t } i + t-1 t =1 Γ e A t e T A t
, for some real parameter η > 0, and where

δ(t) = log(t) + (2 + m) log log(t) + m log(1 + e/η)/2.
Proof. We can write the following using the Cauchy-Schwarz inequality.

e T A (µ * -µ t-1 ) = e T A diag t-1 t =1 e A t ∩A -1 t-1 t =1 e A t ∩A (µ * -X t ) ≤ e T A diag t-1 t =1 e A t ∩A -1 C t-1 t-1 t =1 e A t ∩A (µ * -X t ) C -1 t-1
, where the matrix C t-1 is D + t-1 t =1 C e A t ∩A e T A t ∩A , with D being a positive definite diagonal matrix to be specified latter. Our goal is to bound the right hand term with high probability. This is an interesting quantity for the following reason:

1 2 t-1 t =1 e A t ∩A (µ * -X t ) 2 C -1 t-1 = max λ∈R n - 1 2 λ T C t-1 λ + λ T t-1 t =1 e A t ∩A (µ * -X t ),
and composing by the exponential, we begin to see a form similar to the sub-Gaussianity assumption mentioned above. More precisely, for a fixed λ, the expectation of the exponential of the RHS is lower than 1. However, we see here that there is a maximum over λ. The Laplace's method (also called method of mixtures) is precisely a way to get rid of this maximum. Intuitively, it provides an approximation of the maximum by integrating the exponential against a multivariate normal centered at

C -1 t-1 t-1 t =1 e A t ∩A (µ * -X t )
, where the maximum is attained. The integrals over λ and the expectation can then be swapped by Fubini's theorem, to get an approximation of the expectation of the maximum using an integral of the expectations: Let f be the density of a multivariate normal random variable independent of all other variables with mean 0 and covariance D -1 , we have

det(D) det(C t-1 ) exp    1 2 t-1 t =1 e A t ∩A (µ * -X t ) 2 C -1 t-1    = λ∈R n t-1 t =1 exp - 1 2 λ T C e A t ∩A e T A t ∩A λ + λ T e A t ∩A (µ * -X t ) f (λ)dλ.
From Assumption 1, we see that the product inside the integral is a supermartingale (with respect to F t-1 ). Thus, taking the expectation, we have

E    det(D) det(C t-1 ) exp    1 2 t-1 t =1 e A t ∩A (µ * -X t ) 2 C -1 t-1       ≤ 1.
To deal with det(D) det(C t-1 ) , we use a peeling argument: we consider the event under which for all i ∈ A, e a i ≤ N i,t-1 ≤ e a i +1 . The number of a to consider to cover all the possibilities is log(t) m . With the choice

D i = I{i ∈ A}ηe a i C ii + I{i / ∈ A}, we get the lower bound det(D) det(C t-1 ) ≥ (1 + e/η) m .
In summary, we have using Markov's inequality that

P    t-1 t =1 e A t ∩A (µ * -X t ) C -1 t-1 ≥ δ(t)    ≤ 1 t log 2 (t) ∧ 1,
with δ(t) = log(t) + (2 + m) log log(t) + m log(1 + e/η)/2. The proof is concluded using the assumption that C + Γ.

The Laplace's method was originally used for linear bandits (Abbasi-Yadkori, Pál, and Szepesvári, 2011). The above proof in taken from Degenne and Perchet (2016b), who adapted this method to the semi-bandit feedback setting. Bounds like those given in Theorem 21 are called self-normalized bounds [START_REF] Peña | Self-normalized processes: Limit theory and Statistical Applications[END_REF], because the control is with respect to the norm associated with the inverse of a regularized design matrix. We notice that this matrix is non-diagonal if Γ is, and that we do not have in our arsenal a theorem dealing with bonus of this form. To the best of our knowledge, there is no such work that treats directly this kind of bonus for semi-bandit feedback, which represents an interesting open question.

A first way to proceed is to further bound this bonus by an 2 -type one, i.e., corresponding to a diagonal matrix (the associated confidence region is then ellipsoidal, but is aligned with respect to the axes). More precisely, if we define the non-diagonal matrix M = t-1 t =1 Γ e A t e T A t , we can rely on the inequality

e T A diag t-1 t =1 e A t ∩A -1 2 M = i,j∈A N ij,t-1 Γ ij N i,t-1 N j,t-1 ≤ i∈A 1 N i,t-1 j∈A Γ ij ≤ i∈A 1 N i,t-1 max A ∈A, i∈A j∈A Γ ij ,
where N ij,t-1 = t-1 t =1 I{i, j ∈ A t }. A second method, which has been used by Degenne and Perchet (2016b), is to decompose the bonus as

e T A diag t-1 t =1 e A t ∩A -1 2 M ≤ (1 -γ) i∈A Γ ii N i,t-1 + γ i∈A Γ ii N i,t-1 2 , (5.1)
where γ max (i,j)∈A 2 ,i =j Γ ij / Γ ii Γ jj . We thus get this way a decomposed bonus (with an 1 term and an 2 one).

We want to insist on the fact that the original bonus can be used in the algorithm, and that these two processes for changing the bonus are only used in the analysis.

Covering argument

An alternative to the method previously proposed is to use a covering argument 1 . We saw that the main issue is that the max and the expectation couldn't be swapped. Put it another way, the issue is that λ is random when we want to use the sub-Gaussianity inequality. We have a clear expression for λ, and the randomness comes mainly from the empirical mean (also from counters, but these are treated separately using a peeling). A covering argument is another approach that removes the randomness in λ by replacing, in its definition, the empirical mean by a deterministic quantity that is provably close to it. A positive point of the covering method is that it can be easily extended to a more general setting than the sub-Gaussian one, as we will see in Chapter 7. A negative point is that it is tricky to use when the matrix Γ is not diagonal. Thus, in this section, we take Γ diagonal in Assumption 1 (notice, this happens for instance when outcomes are mutually independent and component-wise sub-Gaussian).

The following Theorem 22 is adapted from Magureanu, Combes, and Proutiere (2014). In particular, under the complementary event considered in the theorem, we can use the Cauchy-Schwarz inequality to get a high probability bound on e T A (µ * -µ t-1 ) ≤ e T A (µ * -µ t-1 ) ∨ 0 that has the form of an 2 -bonus.

Theorem 22 (Covering argument). In the case Γ is diagonal, we have

P    i∈A I µ * i ≥ µ i,t-1 N i,t-1 µ * i -µ i,t-1 2 2Γ ii ≥ δ    ≤ e m+1 (δ -1) log(t) m m e -δ ,
for some δ ≥ m + 1.

Proof. We fixe some δ ≥ m + 1, and define the following events:

A t      i∈A I µ * i ≥ µ i,t-1 N i,t-1 µ * i -µ i,t-1 2 2Γ ii ≥ δ      ∀d ∈ (N * ) A , B d,t i∈A δ δ -1 d i -1 ≤ N i,t-1 < δ δ -1 d i .
Since each number of pulls N i,t-1 for i ∈ A is bounded by t, the number of possible d ∈ (N * ) A such that P[B d,t ] > 0 is bounded by (log(t)/ log(δ/(δ -1))) m . Thanks to the following Lemma 6, and an union bound on those possible d ∈ (N * ) A , we get

P[A t ] ≤ e m+1 (δ -1) log(t) m log(δ/(δ -1)) m e -δ . Lemma 6. Let d ∈ (N * ) A . Then, P[A t ∩ B d,t ] ≤ (δ-1)e m m e 1-δ .
Proof. The idea is to get rid of randomness by replacing the empirical mean µ i,t-1 by some non-random value

x i . Let ζ ∈ R A + . For i ∈ A, we define x i (N ) = µ * i -2ζ i Γ ii N .
Under the events B d,t and

A t i∈A      N i,t-1 0 ∨ µ * i -µ i,t-1 2 2Γ ii > ζ i      , we have i∈A µ i,t-1 ≤ x i δ δ -1 d i .
(5.2)

With ε i µ * i -x i δ δ-1 d i and λ i ε i Γ ii , i ∈ A, this further implies: δ -1 δ i∈A ζ i = i∈A δ δ -1 d i -1 ε 2 i 2Γ ii ≤ i∈A N i,t-1 ε 2 i 2Γ ii B d,t = i∈A N i,t-1 λ i ε i - i∈A N i,t-1 Γ ii λ 2 i /2 ≤ i∈A N i,t-1 λ i µ * i -µ i,t-1 - i∈A N i,t-1 Γ ii λ 2 i /2 using (5.2).
This last quantity can be rewritten as

t ∈[t-1] λ e A t ∩A T (µ * -X t ) -λ e A t ∩A T Γ λ e A t ∩A /2 .
Since λ ≥ 0, we have using Assumption 1

λ e A t ∩A T C λ e A t ∩A ≤ λ e A t ∩A T Γ λ e A t ∩A ,
Thus, still using Assumption 1, we have

δ -1 δ i∈A ζ i ≤ t ∈[t-1] λ e A t ∩A T (µ * -X t ) -λ e A t ∩A T C λ e A t ∩A /2 5.3. Efficiency of algorithms 123 ≤ t ∈[t-1] λ e A t ∩A T (µ * -X t ) -log E e (λ e A t ∩A ) T (X-µ * )
.

The proof now follows the usual path, and we get using Markov's inequality that

P    i∈A      I B d,t N i,t-1 0 ∨ µ * i -µ i,t-1 2 2Γ ii > ζ i         ≤ e -i∈A ζ i δ-1 δ .
By Lemma 7 (Lemma 8 of Magureanu, Combes, and Proutiere ( 2014)), since δ ≥ m + 1, we have

P[B d,t ∩ A t ] ≤ (δ -1)e m m e 1-δ . Lemma 7. For K ≥ 2, a > 0, Z ∈ R K a random vector and ζ ∈ R K + a vector, if P[Z ≥ ζ] ≤ e -a i ζ i , then for δ ≥ K/a, P i Z i ≥ δ ≤ aδe K K e -aδ .
Proof. We have with λ = a -K/δ ≥ 0

P i Z i ≥ δ ≤ E i e λZ i e -λδ = R K + P e λZ > x dxe -λδ = R K + P[Z > log(x)/λ]dxe -λδ ≤ 1 + ∞ 1 e -a log(x)/λ dx K e -λδ = a a -λ K e -λδ = aδe K K e -aδ .

Efficiency of algorithms

In the two previous sections, we have seen how concentration inequalities could be derived to build 2 -bonuses. A (µ * -µ t-1 ) we considered in the previous sections.

Class of possible outcome distributions

Gap-dependent lower bound

Gap-dependent upper bound (i) + (iii) In what follows, we give a more precise overview of the improvement induced by the use of 2 -bonuses compared to 1 -bonuses, stressing that this improvement is unavoidable since the two approaches are matching two different lower bounds. Next, we identify the inefficiency of the 2 -bonus policies by noting that for many situations that are considered in CMAB, the above combinatorial optimization problem (5.3) is not tractable. Finally, we propose an efficient implementation in the case A has a matroid structure.

⇒(i) + (v) ⇒(iv) Ω n log T ∆ 2 -bonus: O n log 2 (m) log T ∆ (ii) + (iii) ⇒(ii) + (v) Ω nm log T ∆ 1 -bonus: O nm log T ∆

Lower bounds

In an instance of a CMAB problem, we can consider different properties satisfied by the outcomes X. For example, some properties that are commonly assumed are:

(i) X 1 , . . . , X n ∈ R are mutually independent, (ii) X 1 , . . . , X n ∈ R are arbitrary correlated, (iii) X ∈ [-1, 1] n , (iv) X ∈ R n is multivariate sub-Gaussian, i.e., E e λ T (X-µ * ) ≤ e λ 2 2 /2 , ∀λ ∈ R n , (v) X ∈ R n is component-wise sub-Gaussian, i.e., E e λ i (Xi-µ * i ) ≤ e λ 2 i /2 , ∀i ∈ [n], ∀λ ∈ R n .
Combining some of the above properties, we consider different classes of possible distributions for X. In the following, we consider five examples of such combination (classifying them by implication):

• (i) + (iii) ⇒ (i) + (v) ⇒ (iv), • (ii) + (iii) ⇒ (ii) + (v).
Notice, the above classes are special cases of Assumption 1. Specifically, for the first point, we can take Γ = I, which allows the agent to use either the Laplace's method or a covering argument to build an 2 -bonus. For the second point, even if an 2 -bonus can still be possible, the resulting bonus will be less tight than the 1 one. Thus, we can see a difference (by a factor of m log -2 (m), as mentioned before) between these two classes in terms of regret upper bound that can be reached by the agent. We will see here that this difference is inherent to the classes, and that the two approaches ( 1 and 2 ) are matching the respective lower bounds of the classes. In Table 5.1, we show two existing gap-dependent lower bounds on R T that depend on the respective class. They are valid for at least one combinatorial structure A ⊂ P ([n]) such that m = max A∈A |A|, one distribution P X having all gaps equal to ∆ and for any consistent policy [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF], for which the regret on any problem of the class verifies R T = o(T a ) as T → ∞ for all a > 0. We also show in Table 5.1 the regret upper bound reached by the optimistic approaches, showing that they are essentially tight. In Table 5.1, the first lower bound is proved in [START_REF] Combes | Combinatorial Bandits Revisited[END_REF], and the second one is proved in Kveton, Wen, Ashkan, and Szepesvari (2015b). Each time, they use an action space A that is reducing to classical MAB, allowing to use the [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] lower bound (Theorem 1).

Let us note finally that in their paper, Degenne and Perchet (2016b) have unified the two above lower bounds by interpolating between the two regimes thanks to a parameter related to the matrix Γ. More precisely, their lower bound is of order

Ω i Γ ii log(T ) ∆ (1 + γ(m -1))
, where we already defined γ after (5.1). Their policy, ols-ucb, uses the bonus from Laplace's method and has a regret upper bound matching this lower bound (up to a polylogarithmic factor in m): using the upper bound (5.1), we can apply Theorem 12 and 13 to obtain the regret bound

O   log T ∆ i∈[n] Γ ii (1 -γ) log 2 (m) + γm   .
(5.4)

Submodular maximization

In this subsection, we discuss the efficiency of existing algorithms matching the lower bounds in Table 5.1. We consider that an algorithm is efficient as soon as the time and space complexity for each round t is polynomial in n and polylogarithmic2 in t.

Notice that the per-round complexity depends substantially on A. We assume A is such that linear optimization problems on A -of the form max A∈A e T A ξ for some ξ ∈ R n -can be solved efficiently. As a consequence, an agent knowing µ * can efficiently compute A * . Assuming efficient linear maximization is crucial (cf. [START_REF] Neu | An efficient algorithm for learning with semi-bandit feedback[END_REF][START_REF] Combes | Combinatorial Bandits Revisited[END_REF]Kveton, Wen, Ashkan, and Szepesvari, 2015b;Degenne and Perchet, 2016b). Without this assumption, e.g., for A being dominating sets in a graph, even the offline problem cannot be solved efficiently, and we would have to consider the notion of approximation regret instead, as was done by [START_REF] Chen | Combinatorial Multi-Armed Bandit: General Framework and Applications[END_REF].

Many combinatorial semi-bandit algorithms can be seen as a special case of Algorithm 5 for different confidence regions C t around µ t-1 .

Algorithm 5 Generic confidence-region-based algorithm.

At each round t : Find a confidence region C t ⊂ R n . Solve the bilinear program

(µ t , A t ) ∈ arg max µ∈Ct,A∈A e T A µ . Play A t .
In particular, cucb (class (ii) + (v)), cucb-v and cucb-kl (class (ii) + (iii)) are such examples where C t is a hypercube. escb-kl [START_REF] Combes | Combinatorial Bandits Revisited[END_REF] (class (i) + (iii)) is defined by the following kl-ball (that resembles to an ellipsoid):

C t    ξ ∈ [0, 1] n , i∈[n] kl µ i,t-1 , ξ i ≤ δ(t)    ,
where δ(t) = log(t) + 4m log log(t). We saw that escb and ols-ucb are defined by Algorithm 5 where C t is an ellipsoid.

In what follows, we will focus on regions that are axis aligned. In particular, all of the policies mentioned above have such a region, except ols-ucb when Γ is nondiagonal (we can still use the upper bound (5.1) to get an axis aligned region). More precisely, we assume that C t is defined through some parameters p, r ∈ {1, ∞}, and some functions

g i,t , i ∈ [n] by C t [-r, r] n ∩ µ t-1 + ξ ∈ R n , (g i,t (ξ i )) i p ≤ 1 ,
where g i,t = 0 if N i,t-1 = 0 and, otherwise, is convex, strictly decreasing on [-rµ i,t-1 , 0] and strictly increasing on [0, r -µ i,t-1 ] such that g i,t (0) = 0. Typically, r = 1 under assumption (iii) and r = ∞ otherwise.

In Algorithm 5, only A t needs to be computed. It is a maximizer over A of the set function

P([n]) → R A → max µ∈Ct e T A µ .
(5.5)

We can easily evaluate the function (5.5) above for some set A ∈ P([n]), since it only requires solving a linear optimization problem on the convex3 set C t . In Proposition 15, we show that in some cases, the evaluation can be even simpler. However, maximizing the function (5.5) over a combinatorial set A is not straightforward. Yet, we can already say that when p = ∞, this function is linear in A, and therefore can be maximized efficiently. This attests to the efficiency of the cucb-type approaches. When p = 1, we have that the maximization problem is NP-Hard in general [START_REF] Atamtürk | Maximizing a class of utility functions over the vertices of a polytope[END_REF]. We will nevertheless extract some interesting properties from this function, in order to be able to maximize it in particular cases. Before studying this function more closely, Definition 15 recalls some well-known properties that can be satisfied by a set function

F : P([n]) → R. Definition 15. A set function F is: • normalized, if F (∅) = 0, • linear (or modular) if F (A) = e T A ξ + b, for some ξ ∈ R n , b ∈ R, • non-decreasing if F (A) ≤ F (B) ∀ A ⊂ B ⊂ [n], • submodular if for all A, B ⊂ [n] , F (A ∪ B) + F (A ∩ B) ≤ F (A) + F (B). Equivalently, F is submodular if for all A ⊂ B ⊂ [n], and i / ∈ B, F (A ∪ {i}) - F (A) ≥ F (B ∪ {i}) -F (B).
The function (5.5) is clearly normalized, and it can be decomposed into two set functions in the following way,

∀A ⊂ [n], max µ∈Ct e T A µ = e T A µ t-1 + max ξ∈Ct-µ t-1 e T A ξ.
The linear part A → e T A µ t-1 is efficiently maximized alone, we thus focus on the other part, A → max ξ∈Ct-µ t-1 e T A ξ (the exploration bonus). It aims to compensate for the negative selection bias of the first term. We define

C + t [-r, r] n ∩ µ t-1 + ξ ∈ R n + , (g i,t (ξ i )) i p ≤ 1 = C t ∩ µ ∈ R n , µ ≥ µ t-1 and rewrite A → max ξ∈Ct-µ t-1 e T
A ξ through Lemma 8.

Lemma 8. For all

A ∈ P([n]), max ξ∈Ct-µ t-1 e T A ξ = max ξ∈C + t -µ t-1 e T A ξ.
The lemma holds as (0

∨ ξ i ) i , ξ ∈ C t -µ t-1 ⊂ C t -µ t-1 .
As a corollary, this set function is non-negative, and non-decreasing. It can be written in closed form under additional assumptions, see Proposition 15 and Example 5.

Proposition 15. Let A ∈ P([n]), t ∈ N * , p = 1. Assume that for all i ∈ A, g i,t has a strictly increasing, continuous derivative g i,t defined on [0, r -µ i,t-1 ]. For i ∈ A, let f i (λ) g -1 i,t (1/λ) if 1/λ < g i,t (r -µ i,t-1 ), r -µ i,t-1 otherwise, defined for λ ≥ 0. Then, the smallest λ * satisfying e T A (g i,t (f i (λ * ))) i ≤ 1 is such that (ξ * i ) i (I{i ∈ A}f i (λ * )) i ∈ arg max ξ∈C + t -µ t-1 e T A ξ.
Proof. It suffices to maximize on the coordinates of ξ belonging to A (the others being zero). For all i ∈ A, we let

η * i 1 -λ * g i,t (r -µ i,t-1 ) I ξ * i = r -µ i,t-1 γ * i λ * g i,t (0) -1 I{ξ * i = 0} = -I{ξ * i = 0}.
For all i ∈ A, the function

f i is continuous, non-increasing on R + , hence so is λ → e T A (g i,t (f i (λ))) i . If e T A (g i,t (f i (λ * ))) i < 1, then necessarily λ * = 0.
Thus, the following KKT conditions are satisfied:

λ * i∈A g i,t (ξ * i ) -1 = 0, and ∀i ∈ A, λ * g i,t (ξ * i ) + η * i -γ * i = 1, η * i (ξ * i -r + µ i,t-1 ) = 0, -γ * i ξ * i = 0,
which concludes the proof by the convexity of the constraints and the objective function.

An important use-case example of Proposition 15 is the following

Example 5. Let A ∈ P([n]), t ∈ N * . If for all i ∈ [n], g i,t = (•) 2 α i,t
for some α i,t > 0, and r = ∞, p = 1, then

max ξ∈C + t -µ t-1 e T A ξ = e T A 1 α i,t i .
Indeed, since the maximizer ξ * lies at the boundary,

max ξ∈C + t -µ t-1 e T A ξ = max ξ∈R n + , i α i,t ξ 2 i =1 e T A ξ,
and from the first-order optimality condition we deduce that

e A = 2λ * (α i,t ξ * i ) i , i.e., ξ * i = I{i ∈ A}/2λ * α i,t
, where λ * is necessarily 1 2 e T A (1/α i,t ) i . We thus recover the escb's exploration bonus for α i,t = N i,t-1 /δ(t). [START_REF] Combes | Combinatorial Bandits Revisited[END_REF] for developing the computation of the escb-kl exploration bonus.

Remark 11. The proof of Proposition 15 follows the same technique as the proof of Theorem 4 by

Example 5 is a specific case where the exploration bonus A → max ξ∈C + t -µ t-1 e T A ξ has a particularly simple form: It is the square root of a non-decreasing linear set function. Such a set function is known to be submodular [START_REF] Stobbe | Efficient Minimization of Decomposable Submodular Functions[END_REF]. This interesting property helps for maximizing the function (5.5). In Theorem 23, we prove that A → max ξ∈C + t -µ t-1 e T A ξ is in fact always submodular.

Theorem 23. A → max ξ∈C + t -µ t-1 e T A ξ is submodular.

Proof. Let t ∈ N * . We consider here the restriction of g i,t to [0, r -µ i,t-1 ], that we still denote as g i,t . Notice that for all i ∈ [n], g i,t is either 0 or a bijection on [0, r -µ i,t-1 ] by assumption. For p = ∞, we have that max

ξ∈C + t -µ t-1 e T A ξ = e T A g -1 i,t (1) ∧ r -µ i,t-1 I{N i,t-1 ≥ 1} + rI{N i,t-1 = 0} i
is a linear set function of A. Assume now that p = 1. To show the submodularity of A → max ξ∈C + t -µ t-1 e T A ξ in this case, we will use the notion of polymatroid. Definition 16 (Polymatroid). A polymatroid is a polytope of the forme

ξ ∈ R n + , e T A ξ ≤ F (A), ∀A ⊂ [n] ,
where F is a non-decreasing submodular function.

Fact 4 (Theorem 3 of [START_REF] He | Polymatroid Optimization, Submodularity, and Joint Replenishment Games[END_REF]. Let P be a polymatroid, and let h 1 , . . . , h n be concave functions.

Then A → max ξ ∈P e T A (h i (ξ i )) i is submodular. Notice that g -1 i,t ({0}) = [0, r -µ i,t-1 ]
when N i,t-1 = 0, and that g -1 i,t (•) is a strictly increasing concave function on [0, g i,t (r -µ i,t-1 )], as the inverse function of a strictly increasing convex function when N i,t-1 ≥ 1. So we can rewrite C + t -µ t-1 as an union of product sets:

C + t -µ t-1 =    ξ ∈ i∈[n] [0, r -µ i,t-1 ], i∈[n] g i,t (ξ i ) ≤ 1    µ t-1 Polymatroid defined by A → max µ∈C t -µ t-1 e T A µ C t µ t-1 Polymatroid defined by A → max µ∈C t -µ t-1 e T A µ C t µ t-1 C t µ t-1 C t Figure 5
.1: Illustration of Theorem 23: The confidence region C t and the polymatroid defined by the submodular function

A → max µ∈C t -µ t-1 e T A µ. = ξ ∈ i∈[n] [0,g i,t (r-µ i,t-1 )], i∈[n] ξ i ≤1 i∈[n] g -1 i,t ( ξ i ).
We can thus rewrite our function as

max ξ∈C + t -µ t-1 e T A ξ = max ξ ∈ i∈[n] [0,g i,t (r-µ i,t-1 )], i∈[n] ξ i ≤1 e T A g -1 i,t (ξ i ) i , with the convention g -1 i,t (0) = r -µ i,t-1 when N i,t-1 = 0. The constraints' set ξ ∈ i∈[n] [0, g i,t (r -µ i,t-1 )], i∈[n] ξ i ≤ 1 is equal to the intersection between i∈[n] [0, g i,t (r -µ i,t-1 )] and the polymatroid ξ ∈ R n + , e T A ξ ≤ I{A = ∅}, ∀A ⊂ [n] .
This intersection is itself equal to the polymatroid

ξ ∈ R n + , e T A ξ ≤ min B⊂A I{B = A} + e T B g i,t (r -µ i,t-1 ) i , ∀A ⊂ [n] .
Thus, max ξ∈C + t -µ t-1 e T A ξ is the optimal objective value on a polymatroid of a separable concave function, as a function of the index set A. Now, using Fact 4, it is submodular.

Theorem 23 yields that when the outcome class is strengthen to target the tighter lower bound n log(T )/∆, Algorithm 5 reduces to maximizing a submodular set function over A (the sum of a linear and a submodular function is submodular). Submodular maximization problems have been applied in machine learning before (see e.g., [START_REF] Krause | Submodular function maximization[END_REF][START_REF] Bach | Learning with Submodular Functions: A Convex Optimization Perspective[END_REF], however, maximizing a submodular function F , even for A = {A ∈ P([n]), |A| ≤ m} and F non-decreasing, is NP-Hard in general [START_REF] Schrijver | Combinatorial Optimization: Polyhedra and Efficiency[END_REF], with an approximation factor of 1 -1/e by the greedy algorithm [START_REF] Nemhauser | An analysis of approximations for maximizing submodular set functions-I[END_REF].

In the next subsection, with a stronger assumption on A (but for which submodular maximization is still NP-Hard), we show that both parts of the objective can have different approximation factors. More precisely, we show how to approximate the linear part with factor 1, and the submodular part with a constant factor. This is enough for our purpose, since it simply means that the event {∆ t ≤ 2bonus t (A t )} is replaced by {∆ t ≤ c • bonus t (A t )}, where c is a constant.

Efficient algorithms for matroid constraints

In this subsection, we will consider additional structure on A, using the notion of matroid, recalled below. Definition 17. A matroid is a pair ([n], I), where I is a family of subsets of [n], called the independent sets, with the following properties:

• The empty set is independent, i.e., ∅ ∈ I.

• Every subset of an independent set is independent, i.e., for all A ∈ I, if A ⊂ A, then A ∈ I.

• If A and B are two independent sets, and |A| > |B|, then there exists x ∈ A\B such that B ∪ {x} ∈ I.

Matroids generalize the notion of linear independence. A maximal (for the inclusion) independent set is called basis and all bases have the same cardinality m, which is called the rank of the matroid [START_REF] Whitney | On the abstract properties of linear dependence[END_REF]. Many combinatorial problems such as building a spanning tree for network routing [START_REF] Oliveira | A survey of combinatorial optimization problems in multicast routing[END_REF] can be expressed as a linear optimization on a matroid (see [START_REF] Edmonds | Matroids and the Greedy Algorithm[END_REF]Fulkerson, 1965 or Perfect, 1968, for other examples). Let I ∈ P([n]) be such that ([n], I) forms a matroid. Let B ⊂ I be the set of bases of the matroid ([n], I). In the following, we may assume that A is either I or B. We also assume that an independence oracle is available, i.e., given an input A ⊂ [n], it returns true if A ∈ I and false otherwise. Maximizing a linear set function L on A is efficient, and it can be done as follows [START_REF] Edmonds | Matroids and the Greedy Algorithm[END_REF]): Let σ be the permutation of [n] and j the integer such that j = m in case A = B and otherwise, j satisfies

1 ≥ • • • ≥ j ≥ 0 ≥ j+1 ≥ • • • ≥ n , where i = L({σ(i)}) ∀i ∈ [n]
. The optimal S is built greedily starting from S = ∅, and for i ∈ [j], σ(i) is added to S only if S ∪ {σ(i)} ∈ I .

Matroid bandits with A = B has been studied by [START_REF] Kveton | Matroid bandits: Fast combinatorial optimization with learning[END_REF] and [START_REF] Talebi | An Optimal Algorithm for Stochastic Matroid Bandit Optimization[END_REF]. In this case, the two lower bounds in Table 5.1 coincide to Ω(n log(T )/∆) (with ∆ being the minimal positive gap) and cucb reaches it. Yet, the efficient implementation of escb-style policies are still interesting in this context, for at least two reasons:

• escb is, as we'll see, better in practice than cucb.

• It may provide ideas for efficient implementation of escb for other action spaces where cucb is not known to reach the lower bound.

In the rest of this subsection, we provide efficient approximation routines to maximize the function (5.5) on A = I and B within a factor 1 for the linear part, and a constant factor for the bonus. Therefore, using these routines in Algorithm 5 do not alter its regret upper bound rate.

Let L be a normalized, linear set function, that will correspond to the linear part A → e T A µ t-1 ; and let F denote a normalized, non-decreasing, submodular function, that will correspond to the submodular part A → max ξ∈C + t -µ t-1 e T A ξ. Unless stated otherwise, we further assume that F is positive (for A = ∅). This is a mild assumption as it holds for A → max ξ∈C + t -µ t-1 e T A ξ in the unbounded case, i.e., if (iii) is not assumed and r = ∞. If (iii) is true, then adding an extra e T A 1 N i t-1 2 i term will Algorithm 6 LocalSearch for maximizing L + F on I.

Input: L, F , I, m, ε > 0.

Initialization: S init ∈ arg max A∈I L(A). if S init = ∅ then if ∃{x} ∈ I such that (L + F )({x}) > 0 then S 0 ∈ arg max {x}∈I, (L+F )({x})>0 L({x}). else Output ∅ end if else S 0 ← S init end if S ← S 0 .
Repeatedly perform one of the following local improvements while possible:

• Delete an element: if ∃x ∈ S such that (L + F )(S\{x}) > (L + F )(S) + ε m F (S), then S ← S\{x}. end if • Add an element: if ∃y ∈ [n]\S, S ∪ {y} ∈ I, such that (L + F )(S ∪ {y}) > (L + F )(S) + ε m F (S), then S ← S ∪ {y}. end if • Swap a pair of elements: if ∃(x, y) ∈ S × [n]\S, S\{x} ∪ {y} ∈ I, such that (L + F )(S\{x} ∪ {y}) > (L + F )(S) + ε m F (S) then S ← S\{x} ∪ {y} end if end while Output: S.
recover positivity and increase the regret upper bound by only an additive constant. In the following subsections, we will provide algorithms that efficiently outputs S such that (5.6) with some appropriate approximation factor κ ≥ 1. It is possible to efficiently output S 1 and S 2 such that we get L(S 1 )

L(S) + κF (S) ≥ L(O) + F (O), ∀O ∈ A,
≥ L(O 1 ) and κF (S 2 ) ≥ F (O 2 ) for any O 1 , O 2 ∈ A.
Although we can take O 1 = O 2 , S 1 and S 2 are not necessarily equal, and (5.6) is not straightforward.

Local Search Algorithm

Here, we assume that A = I. In Algorithm 6, we provide a specific instance of LocalSearch that we tailored to our needs to approximately maximize L + F . It starts from the greedy solution S init ∈ arg max A∈I L(A). Then, Algorithm 6 repeatedly tries three basic operations in order to improve the current solution. Since every S ∈ A can potentially be visited, only significative improvements are considered, i.e., improvements greater than ε m F (S) for some input parameter ε > 0. The smaller ε is, the higher complexity will be. Notice the improvement threshold ε m F (S) does not depend on L. In fact, this is crucial to ensure that the approximation factor of L is 1. However, this can increase the time complexity. To prevent this increase, the second essential ingredient is the initialization, where only L plays a role. In Theorem 24, we state the approximation guarantees for Algorithm 6 and its time complexity. For C t given by any algorithm previously considered, F (A) = max ξ∈C + t -µ t-1 e T A ξ, and ε = 1, the time complexity is bounded by O m 2 n log(mt) , and the algorithm is thus efficient. Theorem 24 gives a parameter κ arbitrary close to 2 in (5.6). 4Theorem 24. Algorithm 6 outputs S ∈ I such that

L(S) + 2(1 + ε)F (S) ≥ L(O) + F (O), ∀O ∈ I. Its complexity is O mn log max A∈I F (A) F (S 0 ) /log 1 + ε m .
Before proving Theorem 24, we state some well known results about submodular optimization on a matroid.

Proposition 16. Let A, B ⊂ [n]. If F is submodular, then b∈B\A (F (B) -F (B\{b})) ≤ F (B) -F (A ∩ B), a∈A\B (F (B ∪ {a}) -F (B)) ≥ F (A ∪ B) -F (B). Proof. Let (b 1 , . . . , b |B\A| ) be an ordering of B\A. Then, by submodularity of F , |B\A| i=1 (F (B) -F (B\{b i })) ≤ |B\A| i=1 (F (B\{b 1 , . . . , b i-1 }) -F (B\{b 1 , . . . , b i })) = F (B) -F (A ∩ B).
In the same way, let (a 1 , . . . , a |A\B| ) be an ordering of A\B. Then, by submodularity of F ,

|A\B| i=1 (F (B ∪ {a i }) -F (B)) ≥ |A\B| i=1 (F (B ∪ {a 1 , . . . , a i }) -F (B ∪ {a 1 , . . . , a i-1 })) = F (A ∪ B) -F (B).
Fact 5 (Theorem 1 of [START_REF] Lee | Maximizing Nonmonotone Submodular Functions under Matroid or Knapsack Constraints[END_REF]. Let A, B ∈ A. Then, there exists a mapping α : B\A → A\B ∪ {∅} such that 4 We could design a different version of Algorithm 6, based on Non-ObliviousLocalSearch [START_REF] Filmus | A tight combinatorial algorithm for submodular maximization subject to a matroid constraint[END_REF] , in order to get κ arbitrary close to 1 + 1/(e -1), but with a much worst time complexity. Actually, [START_REF] Sviridenko | Optimal approximation for submodular and supermodular optimization with bounded curvature[END_REF] proposed such an approach, with an approximation factor for L arbitrary close to 1, but not equal, so we would get back the undesirable term, which would require a complexity polynomial in t to control.

• ∀b ∈ B\A, A\{α(b)} ∪ b ∈ A • ∀a ∈ A\B, α -1 (a) ≤ 1.
Proposition 17. Let A, B ∈ A. Let F be a submodular function and α : B\A → A\B ∪ {∅} be the mapping given in Fact 5. Then, b∈B\A

(F (A) -F (A\{α(b)} ∪ {b})) + a∈A\B, α -1 (a)=∅ (F (A) -F (A\{a})) ≤ 2F (A) -F (A ∪ B) -F (A ∩ B). Proof. We decompose b∈B\A (F (A) -F (A\{α(b)} ∪ {b})) into sum of two terms, b∈B\A (F (A) -F (A\{α(b)})) + b∈B\A (F (A\{α(b)}) -F (A\{α(b)} ∪ {b})).
Remark that the first part is equal to

a∈α(B\A) (F (A) -F (A\{a})) = a∈A\B, α -1 (a) =∅ (F (A) -F (A\{a})).
Thus, together with a∈A\B, α -1 (a)=∅ (F (A) -F (A\{a})), we get that b∈B\A

(F (A) -F (A\{α(b)} ∪ {b})) + a∈A\B, α -1 (a)=∅ (F (A) -F (A\{a})) is equal to a∈A\B (F (A) -F (A\{a})) + b∈B\A (F (A\{α(b)}) -F (A\{α(b)} ∪ {b})).
Finally, we upper bound the first term by F (A) -F (A ∩ B) using first inequality of Proposition 16, and the second term by F (A) -F (A ∪ B) using first, the submodularity of F to remove α(b) in the summands, and then the second inequality of Proposition 16.

Proof of Theorem 24. The proof is divided into two parts:

Approximation guarantee If Algorithm 6 outputs ∅ before entering in the while loop, then by submodularity, for any S ∈ I,

(L + F )(S) ≤ x∈S (L + F )({x}) ≤ 0. Thus, ∅ is a maximizer of L + F .
Otherwise, the output S of Algorithm 6 satisfies the local optimality of the while loop. We apply Proposition 17 with A = S and B = O for L and F separately,

b∈O\S (L(S) -L(S\{α(b)} ∪ {b})) + a∈S\O, α -1 (a)=∅ (L(S) -L(S\{a})) ≤ 2L(S) -L(S ∪ O) -L(S ∩ O), b∈O\S (F (S) -F (S\{α(b)} ∪ {b})) + a∈S\O, α -1 (a)=∅ (F (S) -F (S\{a})) ≤ 2F (S) -F (S ∪ O) -F (S ∩ O).
Then, we sum these two inequalities, b∈O\S

((L + F )(S) -(L + F )(S\{α(b)} ∪ {b})) + a∈S\O, α -1 (a)=∅ ((L + F )(S) -(L + F )(S\{a})) ≤ 2(L + F )(S) -(L + F )(S ∪ O) -(L + F )(S ∩ O) = 2F (S) -F (S ∪ O) -F (S ∩ O) + L(S) -L(O),
where the last equality uses linearity of L. Since F is increasing and non-negative,

F (S ∪ O) + F (S ∩ O) ≥ F (O), and we get b∈O\S ((L + F )(S) -(L + F )(S\{α(b)} ∪ {b})) + a∈S\O, α -1 (a)=∅ ((L + F )(S) -(L + F )(S\{a})) ≤ 2F (S) -F (O) + L(S) -L(O).
From the local optimality of S, the left hand term in this inequality is lower bounded by

b∈O\S -ε m F (S) + a∈S\O, α -1 (a)=∅ -ε m F (S) ≥ -2εF (S).
The last statement finishes the proof for the approximation inequality.

Time complexity

Computing S 0 has a negligible complexity compared to the while loop. The following lemma gives a characterization of S 0 .

Lemma 9. S 0 ∈ arg max{L(A), A ∈ I, (F + L)(A) > 0}.

Proof. From Algorithm 6, if S init = ∅, then S 0 = S init and L(S 0 ) = max A∈I L(A) ≥ 0. Thus, F (S 0 ) > 0 by assumption on F , giving (F + L)(S 0 ) > 0, which ends the proof.

If S init = ∅, then L(S 0 ) = max{L({x}), {x} ∈ I, (L + F )({x}) > 0}. Let A ∈ arg max{L(A), A ∈ I, (F + L)(A) > 0}.
A is clearly non-empty, and by submodularity of F + L, there exists x ∈ A such that (F + L)({x}) > 0. L is nonincreasing from S init = ∅, so we get L({x}) ≥ L(A), which means there is a singleton {x} in arg max{L(A), A ∈ I, (F + L)(A) > 0}, so

S 0 ∈ arg max{L(A), A ∈ I, (F + L)(A) > 0},
which finishes the proof.

From this lemma, necessarily L(S 0 ) ≥ L(S ) for every iterations ≥ 1, since the sequence (L(S ) + F (S )) is increasing, and thus (F + L)(S ) > 0, ∀ ≥ 1. At each iteration ≥ 1, Algorithm 6 constructs S such that

F (S ) > 1 + ε m F (S -1 ) + L(S -1 ) -L(S ).
Thus, we must have

F (S ) -1 + ε m F (S 0 )
≥ j=1 1 + ε m -j (L(S j-1 ) -L(S j )) = L(S 0 ) 1 + ε m -1 - ε m -1 j=1 L(S j ) 1 + ε m -j-1 -L(S ) ≥ L(S 0 ) 1 + ε m -1 - ε m -1 j=1 L(S 0 ) 1 + ε m -j-1 -L(S 0 ) = 0,
where the last inequality uses L(S 0 ) ≥ L(S ), ∀ ≥ 1. This gives the following upper bound on the number of iteration :

≤ log F (S ) F (S 0 ) log 1 + ε m ≤ log max A∈A F (A) F (S 0 ) log 1 + ε m .
Finally, the result follows remarking that time complexity per iteration is O(mn).

Greedy Algorithm

Here, we assume that A = B. This situation happens, for instance, under a nonnegativity assumption on L, i.e., if we consider non-negative outcomes X i . We show that the standard greedy algorithm (Algorithm 7) improves over Algorithm 6 by exploiting this extra constraint, both in terms of the running time and the approximation factor. We state the result in Theorem 25. Notice that another advantage is that we do not need to assume F (A) > 0 for A = ∅ here.

Theorem 25. Algorithm 7 outputs S ∈ B such that

L(S) + 2F (S) ≥ L(O) + F (O), ∀O ∈ B.

Its complexity is O(mn).

As we did previously, before starting the proof of Theorem 25, we state some useful results.

Fact 6 (Brualdi's lemma). Let A, B ∈ B. Then, there exists a bijection β :

A → B such that ∀a ∈ A, A\{a} ∪ {β(a)} ∈ B.
Furthermore, β is the identity on A ∩ B.

Proof. A proof is given by [START_REF] Brualdi | Comments on bases in dependence structures[END_REF] and is also proved by [START_REF] Schrijver | Combinatorial Optimization: Polyhedra and Efficiency[END_REF], as Corollary 39.12a.

Proposition 18. Let A, B ∈ B. Let F be a submodular function and β : A → B be the mapping given in Fact 6. Let a 1 , . . . , a k be elements of A, and

A i = {a 1 , . . . , a i }.
Then,

i∈[k] (F (A i ) -F (A i-1 ∪ {β(a i )})) ≤ 2F (A) -F (A ∪ B) -F (∅). Proof. We can split i∈[k] (F (A i-1 ∪ {a i }) -F (A i-1 ∪ {β(a i )})) into two terms, k i=1 (F (A i-1 ∪ {a i }) -F (A i-1 )) + k i=1 (F (A i-1 ) -F (A i-1 ∪ {β(a i )})).
The first term is equal to F (A k ) -F (∅). Using submodularity of F , the second term is upper bounded by

k i=1 (F (A m ) -F (A m ∪ {β(a i )})),
which is upper bounded by F (A k ) -F (A k ∪ B) thanks to Proposition 16 and its second inequality.

Proof of Theorem 25. The time complexity proof is trivial. Let S i {s 1 , . . . , s i } be the set maintained in Algorithm 7 after i iterations. Instatianting Proposition 18 with A i = S i and B = O, we have

i∈[k] (F (S i ) -F (S i-1 ∪ {β(s i )})) ≤ 2F (S) -F (S ∪ O) -F (∅).
(5.7)

Furthermore, we also have, by linearity of L, and bijectivity of β,

i∈[k] (L(S i ) -L(S i-1 ∪ {β(s i )})) = i∈[k] (L({s i }) -L({β(s i )}))) = L(S) -L(O).
(5.8) Thus, we can sum up (5.7) and (5.8) to get

i∈[k] ((L + F )(S i ) -(L + F )(S i-1 ∪ {β(s i )})) ≤ 2F (S) -F (S ∪ O) -F (∅) + L(S) -L(O) ≤ 2F (S) -F (O) + L(S) -L(O),
where the last inequality uses the fact that F is increasing and F (∅) = 0. We finish the proof remarking that by definition of Algorithm 7,

(L + F )(S i ) -(L + F )(S i-1 ∪ {β(s i )}) ≥ 0.
t-free complexity for I-constraint using greedy The greedy algorithm can also be used in the case that A = I. Indeed, see that greedy computes an increasing sequence

∅ = S 0 ⊂ • • • ⊂ S m = S.
Instead of outputting S m , we can output S i that maximizes L + 2F . This way, we can write

L(S i ) + 2F (S i ) ≥ L(S |O| ) + 2F (S |O| ) ≥ L(O) + F (O), ∀O ∈ I.
The second inequality is by using the property of greedy (Theorem 25) on the independent set I O I ∩ {A ∈ P([n]), |A| ≤ |O|}, noticing that both O and S |O| are bases of the matroid ([n], I O ). This leads to a complexity of order nm, improving over the m 2 n log(mt) that we got previously.

The budgeted setting

In this section, we extend results of the two previous subsections to budgeted matroid semi-bandits. In budgeted bandits with single resource and infinite horizon [START_REF] Ding | Multi-Armed Bandit with Budget Constraint and Variable Costs[END_REF][START_REF] Xia | Budgeted bandit problems with continuous random costs[END_REF], each arm is associated with both a reward and a cost. The agent aims at maximizing the cumulative reward under a budget constraint for the cumulative costs. [START_REF] Xia | Budgeted multi-armed bandits with multiple plays[END_REF] studied budgeted bandits with multiple play, where an m-subset A of arms is selected at each round. An optimal (up to a constant term) offline algorithm chooses the same action A * within each round, where A * is the minimizer of the ratio "expected cost paid choosing A" over "expected reward gained choosing A". In the setting of [START_REF] Xia | Budgeted multi-armed bandits with multiple plays[END_REF], the agent observes the partial random cost and reward of each arm in A (i.e., semi-bandit feedback), pays the sum of partial costs of A and gains the sum of partial rewards of A. A * can be computed efficiently, and a [START_REF] Xia | Budgeted multi-armed bandits with multiple plays[END_REF] give an algorithm based on cucb. It minimizes the ratio where the averages are replaced by UCBs. We extend this setting to matroid constraints. We assume that total costs/rewards are non-negative linear set functions of the chosen action A. The objective is to minimize a ratio of linear set functions. As previously, two kinds of constraint can be considered for the minimization: either A = I or A = B. Theorem 23 implies that an optimistic estimation of this ratio is of the form L 1 -F 1 L 2 +F 2 , where for i ∈ {1, 2}, F i are positive (except for ∅), normalized, non-decreasing, submodular; and L i are non-negative and linear. L 1 -F 1 is a high-probability lower bound on the expected cost paid, and L 2 + F 2 is a high-probability upper bound on the expected reward gained. Notice that the numerator, L 1 -F 1 , can be negative, which can be an incitement to take arms with a high cost/low rewards. Therefore, we consider the minimization of the surrogate 0

∨ L 1 -F 1 L 2 +F 2 . Indeed, (L 1 -F 1 )/(L 2 + F 2
) is a high probability lower bound on the ratio of expectation, so by monotonicity of x → 0 ∨ x on R, 0 ∨ (L 1 -F 1 )/(L 2 + F 2 ) is also a high-probability lower bound. We assume L 2 is normalized, but not necessarily L 1 . L 1 (∅) can be seen as an entry price for a round.

Remark 12. Notice, If A = I, and L 1 is normalized, then there is an optimal solution of the form {s} ∈ I (assuming ∅ is not feasible): If L 1 -F 1 is negative for some S = {s} ⊂ I, then such S is a minimizer. Otherwise, by submodularity (and thus subadditivity, since we consider normalized functions), L 1 -F 1 is non-negative, and we have

L 1 (S) -F 1 (S) L 2 (S) + F 2 (S) ≥ s∈S L 1 ({s}) -F 1 ({s}) s∈S L 2 ({s}) + F 2 ({s}) ≥ min s∈S L 1 ({s}) -F 1 ({s}) L 2 ({s}) + F 2 ({s}) .
This minimization problem is at least as difficult as previous submodular maximization problems, taking L 1 = 1 and F 1 = 0. In order to use our approximation algorithms, we consider the Lagrangian function associated to the problem (see e.g., [START_REF] Fujishige | Submodular functions and optimization[END_REF],

L(λ, S) L 1 (S) -F 1 (S) -λ(L 2 (S) + F 2 (S)),
for λ ≥ 0 and S ⊂ [n]. For a fixed λ ≥ 0, -L(λ, •) is a sum of a linear and a submodular function, and both Algorithms 6 and 7 can be used. However, the first step is to find λ sufficiently close to the optimal ratio

λ * = min A∈A L 1 (A) -F 1 (A) L 2 (A) + F 2 (A) ∨ 0.
Remark 13. For some λ ≥ 0,

min A∈A L(λ, A) ≥ 0 ⇒ λ ≤ λ * , min A∈A L(λ, A) ≤ 0 ⇒      λ ≥ λ * , or min A∈A L 1 (A) -F 1 (A) ≤ 0, which further gives λ * = 0.
From Remark 13, if it was possible to compute min A∈A L(λ, A) exactly, then a binary search algorithm would find λ * . This dichotomy method can be extended to κ-approximation algorithms by defining the approximation Lagrangian as

L κ (λ, S) L 1 (S) -κF 1 (S) -λ(L 2 (S) + κF 2 (S)), for λ ≥ 0 and S ⊂ [n].
The idea is to use the following approximation guarantee for a κ-approximation algorithms outputing S (with objective function -L),

min A∈A L κ (λ, A) ≤ L κ (λ, S) ≤ min A∈A L(λ, A).
Thus, for a given λ, either the l.h.s is strictly negative or the r.h.s is non-negative, depending on the sign of L κ (λ, S). Therefore, from Remark 13, a lower bound λ 1 on λ * , and an upper bound λ 2 on min A∈A 0 ∨ L 1 (A)-κF 1 (A)

L 2 (A)+κF 2 (A) can be computed. The detailed method is given in Algorithm 8. Notice that it takes as input some Algo κ , that can be either Algorithm 6 or Algorithm 7, depending on the assumption on the constraint (either A = I or A = B). We denote the output as Algo κ (L + F ), for some linear set function L and some submodular set function F , for maximizing the objective L + F on A, so that

S = Algo κ (L + F ) satisfies L(S) + κF (S) ≥ max A∈A L(A) + F (A), i.e., κ = 2(1 + ε) if Algo κ = Algorithm 6, A = I and κ = 2 if Algo κ = Algorithm 7, A = B.
In Theorem 26, we state the result for the output of Algorithm 8.

Theorem 26. Algorithm 8 outputs

A such that 0 ∨ L 1 (A) -(κ + η)F 1 (A) L 2 (A) + κF 2 (A) ≤ λ * , where λ * is the minimum of 0 ∨ L 1 -F 1 L 2 +F 2 over I if Algo κ = Algorithm 6
, and over B if Algo κ = Algorithm 7. For C t given by any policy considered before, F (A) = max ξ∈C + t -µ t-1 e T A ξ, the complexity is of order log(mt/η) times the complexity of Algo κ .

Proof. Let A be the output of Algorithm 8 and let

L κ 1 ,κ 2 (λ, S) L 1 (S) -κ 1 F 1 (S) -λ(L 2 (S) + κ 2 F 2 (S)).
Algorithm 8 Binary search for minimizing the ratio 0 ∨ (L 1 -F 1 )/(L 2 + F 2 ).

Input: L 1 , L 2 , F 1 , F 2 , Algo κ , η > 0. δ ← η min {s}∈A F 1 ({s}) L 2 (B)+κ 2 F 2 (B) with B = Algo κ (L 2 + κF 2 ). A ← A 0 ∈ A\{∅} arbitrary. if L κ (0, A) > 0 then λ 1 ← 0, λ 2 ← L 1 (A)-F 1 (A) L 2 (A)+F 2 (A) . while λ 2 -λ 1 ≥ δ do λ ← λ 1 +λ 2 2 . S ← Algo κ (-L(λ, •)). if L κ (λ, S) ≥ 0 then λ 1 ← λ. else λ 2 ← λ. A ← S. end if end while end if Output: A. Recall that L κ = L κ,κ . Algorithm 8 satisfies either L κ (0, A 0 ) ≤ 0 -in which case Theorem 26 is trivial since 0 ∨ L 1 (A) -(κ + η)F 1 (A) L 2 (A) + κF 2 (A) = λ * = 0 -or L κ (0, A 0 ) > 0, in which case we have 0 > L κ (λ 2 , A) ≥ L κ+η,κ (λ 2 -δ, A) ≥ L κ+η,κ (λ 1 , A) ≥ L κ+η,κ (λ * , A).
(5.9)

The first inequality is comes from the update of λ 2 : Notice that before the while loop, we have

λ 2 = L 1 (A 0 ) -F 1 (A 0 ) L 2 (A 0 ) + F 2 (A 0 ) > L 1 (A 0 ) -κF 1 (A 0 ) L 2 (A 0 ) + κF 2 (A 0 ) > 0, since F 2 (A 0 ) > 0, so 0 > L κ (λ 2 , A 0 ) multiplying by L 2 (A 0 ) + F 2 (A 0 ) on both sides.
Notice that in particular, this inequality gives that A = ∅.

The second inequality follows from

δ = η min {s}∈A F 1 ({s}) L 2 (B) + κ 2 F 2 (B) ≤ ηF 1 (A) L 2 (A) + κF 2 (A) since A = ∅ and L 2 (B) + κ 2 F 2 (B) ≥ L 2 (A) + κF 2 (A). Thus, multiplying by L 2 (A) + κF 2 (A) > 0, and adding L 1 (A) -κF 1 (A) -λ 2 (L 2 (A) + κF 2 (A)) gives L 1 (A) -(κ + η)F 1 (A) -(λ 2 -δ)(L 2 (A) + κF 2 (A)) ≤ L 1 (A) -κF 1 (A) -λ 2 (L 2 (A) + κF 2 (A)), i.e., L κ+η,κ (λ 2 -δ, A) ≤ L κ (λ 2 , A).
The third inequality uses λ 2 -λ 1 ≤ δ, and the last inequality uses λ 1 ≤ λ * . Indeed, since L κ (λ 1 , S) ≥ 0, the approximation relation given by Algo κ ,

L κ (λ 1 , S) ≤ L(λ 1 , O),
where O is the minimizer of 0 ∨ L 1 -F 1 L 2 +F 2 (for the constraints considered by Algo κ ), gives 0 ≤ L(λ 1 , O). Thus,

L + (λ 1 , O) 0 ∨ (L 1 (O) -F 1 (O)) -λ 1 (L 2 (O) + F 2 (O)) ≥ L(λ 1 , O) ≥ 0. Finally, since L 2 (O) + F 2 (O) > 0 (O = ∅), we have λ 1 ≤ λ * .
In (5.9), since A = ∅, we have

L 1 (A)-(κ+η)F 1 (A) L 2 (A)+κF 2 (A)
≤ λ * and therefore,

0 ∨ L 1 (A) -(κ + η)F 1 (A) L 2 (A) + κF 2 (A) ≤ λ * .
The time complexity for the binary search is O(log(1/δ)) ≤ O(log(mt/η)) for C t given by any policy previously considered, and F (A) = max ξ∈C + t -µ t-1 e T A ξ.

Experiments and discussion

We provide experiments for a graphic matroid, on a five nodes complete graph, as did [START_REF] Combes | Combinatorial Bandits Revisited[END_REF]. We thus have n = 10, m = 4. We consider two experiments.

In the first one we use

A = B, µ * i = 1 + ∆I{i ≤ m}, for all i ∈ [n]
, and in the second, A = I, where we set

µ * i = ∆(2I{i ≤ m -1} -1), ∀i ∈ [n].
We take ∆ = 0.1, with outcomes drawn from independent unit variance Gaussian distributions. Figure 5.2 illustrates the comparison between cucb and our implementations of escb [START_REF] Combes | Combinatorial Bandits Revisited[END_REF] using Algorithm 7 (left) and 6 (right, with ε = 0.1), showing the behavior of the regret vs. time horizon. We observe that although we are approximating the confidence region within a factor at least 2 (and thus force the exploration), our efficient implementation outperforms cucb. Indeed, we take advantage of the previously inefficient algorithm (that we made efficient) to use outcome independence, which the more conservative cucb is not able to. The latter algorithm has still a better per round-time complexity of O(n log m) and may be more practical on larger instances.

Discussion

We gave several approximation schemes for the confidence regions and applied them to combinatorial semi-bandits with matroid constraints and their budgeted version. We believe our approximation methods can be extended to approximation regret for non-linear objective functions (e.g., for influence maximization, [START_REF] Wang | Thompson Sampling for Combinatorial Semi-Bandits[END_REF], if the maximization algorithm keeps the same approximation factor for the objective, either with or without the bonus.

Chapter 6

Budgeted Online Influence Maximization

This chapter aims to further study another well-known example of CMAB-T problem, which is online influence maximization (OIM). It is based on our papers Perrault, Healey, et al. (2020a) and Perrault, Healey, et al. (2020b). In particular, we will introduce a new budgeted framework for OIM, considering the total cost of an advertising campaign instead of the common cardinality constraint on a chosen influencer set. Our approach models better the real-world setting where the cost of influencers varies and advertizers want to find the best value for their overall social advertising budget. We propose an algorithm assuming an independent cascade diffusion model and edge-level semi-bandit feedback, and provide both theoretical and experimental results. Our analysis is also valid for the cardinality-constraint setting and improves the state of the art regret bound in this case.

Problem formulation

Viral marketing through online social networks now represents a significant part of many digital advertising budgets. In this form of marketing, companies incentivize chosen influencers in social networks (e.g., Facebook, Twitter, YouTube) to feature a product in hopes that their followers will adopt the product and repost the recommendation to their own network of followers. The effectiveness of the chosen set of influencers can be measured by the expected number of users that adopt the product due to their initial recommendation, called the spread. Influence maximization (IM, [START_REF] Kempe | Maximizing the spread of influence through a social network[END_REF]) is the problem of choosing the optimal set of influencers to maximize the spread under a cardinality constraint on the chosen set.

In order to define the spread, we need to specify a diffusion process such as independent cascade (IC) or linear threshold (LT) [START_REF] Kempe | Maximizing the spread of influence through a social network[END_REF]. The parameters of these models are usually unknown. Different methods exist to estimate the parameters of the diffusion model from historical data, however historical data is often difficult to obtain. Another possibility is to consider online influence maximization (OIM) [START_REF] Vaswani | Influence maximization with bandits[END_REF][START_REF] Wen | Online influence maximization under independent cascade model with semi-bandit feedback[END_REF] where an agent actively learns about the network by interacting with it repeatedly, trying to find the best seed influencers. The agent thus faces the dilemma of exploration versus exploitation, allowing us to see it as multiarmed bandits problem [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF]. More precisely, the agent faces IM over T rounds. Each round, it selects m seeds (based on feedback from prior rounds) and diffusion occurs; then it gains a reward equal to the spread and receives some feedback on the diffusion.

IM and OIM optimize with the constraint of a fixed number of seeds. This reflects a fixed seed cost model, for example, where influencers are incentivized by being given an identical free product. In reality, however, many influencers demand different levels of compensation. Those with a high out-degree (e.g., number of followers) are usually more expensive. Due to these cost variations, marketers usually wish to optimize their seed sets S under a budget c(S) ≤ b rather than a cardinality constraint |S| ≤ m. Optimizing a seed set under a budget has been studied in the offline case by [START_REF] Nguyen | On budgeted influence maximization in social networks[END_REF]. In the online case, [START_REF] Wang | Fast Thompson Sampling Algorithm with Cumulative Oversampling: Application to Budgeted Influence Maximization[END_REF] considered the relaxed constraint E[c(S)] ≤ b, where the expectation is over the possible randomness of S. 1We believe however that the constraint of a fixed, equal budget c(S) ≤ b at each round does not sufficiently model the willingness to choose a cost-efficient seed set. Indeed, we see that the choice of b is crucial: a b too large translates into a waste of budget (some seeds that are too expensive will be chosen) and a b too small translates into a waste of time (a whole round is used to influence only a few users). To circumvent this issue, instead of a budget per round, in our framework, we allow the agent to choose seed sets of any cost at each round, under an overall budget constraint (equal to B = bT for instance). In summary, we incorporate the OIM framework into a budgeted bandit setting. Our setting is more flexible for the agent, and better meets real-world needs.

Related work on IM

We recall that IM can be formally defined as follows. A social network is modeled as a directed graph G = (V , E), with nodes V representing users and edges E representing connections. An underlying diffusion model D governs how information spreads in G. More precisely, D is a probability distribution on subgraphs G of G, and given some seed set S, the spread σ(S) is defined as the expected number of S-reachable 2 nodes in G ∼ D. IM aims to find S that is a solution to max |S|=m σ(S).

(6.1)

Although IM is NP-hard under standard diffusion models -i.e., IC and LTσ is a monotone submodular 3 function [START_REF] Fujishige | Submodular functions and optimization[END_REF], and given a value oracle access to σ, the standard greedy algorithm solves (6.1) within a 1 -1/e approximation factor [START_REF] Nemhauser | An analysis of approximations for maximizing submodular set functions-I[END_REF]. There have been multiple lines of work for IM, including the development of heuristics, approximation algorithms, as well as alternative diffusion models [START_REF] Leskovec | Cost-effective Outbreak Detection in Networks[END_REF][START_REF] Goyal | SIMPATH: An Efficient Algorithm for Influence Maximization under the Linear Threshold Model[END_REF][START_REF] Tang | Influence maximization: Nearoptimal time complexity meets practical efficiency[END_REF][START_REF] Tang | Influence Maximization in Near-Linear Time: A Martingale Approach[END_REF]. Additionally, there are also results on learning D from data in the case it is not known [START_REF] Saito | Prediction of Information Diffusion Probabilities for Independent Cascade Model[END_REF][START_REF] Goyal | Learning Influence Probabilities in Social Networks[END_REF][START_REF] Gomez-Rodriguez | Inferring Networks of Diffusion and Influence[END_REF][START_REF] Netrapalli | Learning the Graph of Epidemic Cascades[END_REF].

Related work on OIM

Prior work in OIM has mainly considered either node level semi-bandit feedback [START_REF] Vaswani | Influence maximization with bandits[END_REF], where the agent observes all the S-reachable nodes in G , or edge level semi-bandit feedback [START_REF] Wen | Online influence maximization under independent cascade model with semi-bandit feedback[END_REF], where the agent observes the whole S-reachable subgraph (i.e., the subgraph of G induced by S-reachable nodes). Other, weaker, feedback settings have also been studied including: pairwise influence feedback, where all nodes that would be influenced by a seed set are observed but not the edges connecting them, i.e., ({i}-reachable nodes) i∈S is observed (Vaswani, Kveton, et al., 2017); local feedback, where the agent observes a set of out-neighbors of S [START_REF] Carpentier | Revealing graph bandits for maximizing local influence[END_REF] and immediate neighbor observation where the agent only observes the out-degree of S [START_REF] Lugosi | Online Influence Maximization with Local Observations[END_REF].

Our contributions

We define the budgeted OIM paradigm and propose a performance metric for an online policy on this problem using the notion of approximation regret [START_REF] Chen | Combinatorial Multi-Armed Bandit: General Framework and Applications[END_REF]. To the best of our knowledge, the both of contributions are new. We then focus our study on the IC model with edge level semi-bandit feedback. We design a cucb-style algorithm and prove logarithmic regret bounds. We also propose some modifications of this algorithm with improving the regret rates. These gains apply to the non-budgeted setting, giving an improvement over the state-of-the-art analysis of the standard cucb-approach (Wang and Chen, 2017). Our proof incorporates an approximation guarantee of greedy for ratio of submodular and modular functions, which may also be of independent interest.

Problem definition

In this subsection, we formulate the problem of budgeted OIM and give a regret definition for evaluating policies in that setting. We also justify our choice for this notion of regret. We consider a fixed directed network G = (V , E), known to the agent, with V {1, . . . , |V |}. We denote by ij ∈ E the directed edge from node i to j in G. We assume that G doesn't have self-loops, i.e., for all ij ∈ E, i = j. For a node i ∈ V , a subset S ⊂ V , and a vector w ∈ {0, 1} E , the predicate S w i holds if, in the graph defined by G w (V , {ij ∈ E, w ij = 1}), there is a forward path from a node in S to the node i. If it holds, we say that i is influenced by S under w. We define p i (S; w) I S w i and the spread as σ(S; w) i ∈ V , S w i . Our diffusion process is defined by the random vector W ∈ {0, 1} E , and our cost is defined by the random 4 vector C ∈ [0, 1] V ∪{0} where the added component C 0 represents any fixed costs5 . Notice, random costs are neither assumed to be mutually independent nor independent from W. We will see that components of W might however be mutually independent (e.g., for the IC model).

Budgeted online influence maximization

The agent interacts with the diffusion process across several rounds, using a learning policy. At each round t ≥ 1, the agent first selects a seed set S t ⊂ V , based on its past observations. Then, the random vectors for both the diffusion process W t ∼ P W and the costs C t ∼ P C are sampled independently from previous rounds. Then, the agent observes some feedback from both the diffusion process and the costs. We provide in (6.2) the expected cumulative rewards F B defined for some total budget B > 0. The goal for the agent is to follow a learning policy π maximizing F B . In (6.2), recall that S t is the seed set selected by π at round t.

F B (π) E τ B -1 t=1 σ(S t ; W t ) . (6.2)
τ B is the random round at which the remaining budget becomes negative: if B t Bt ≤t e T S t C t + C 0,t , then B τ B -1 ≥ 0 and B τ B < 0. Notice, quantities B t and τ B are usual in budgeted multi-armed bandits [START_REF] Xia | Budgeted multi-armed bandits with multiple plays[END_REF][START_REF] Ding | Multi-armed bandit with budget constraint and variable costs[END_REF].

Performance metric

We restrict ourselves to efficient policies, i.e., we consider a complexity constraint on the policy the agent can follow: For a round t, the space and time complexity for computing S t has to be polynomial in |V |, and polylogarithmic in t. To evaluate the performance of a learning policy π, we use the notion of approximation regret [START_REF] Kakade | Playing games with approximation algorithms[END_REF][START_REF] Streeter | An online algorithm for maximizing submodular functions[END_REF]Chen, Wang, and Yuan, 2016). The agent wants to follow a learning policy π which minimizes

R B,ε (π) (1 -1/e -ε)F * B -F B (π),
where F * B is the best possible value of F B over all policies (thus leveraging on the knowledge of P W and P C ), and where ε > 0 is some parameter the agent can control to determine the tradeoff between computation and accuracy.

Remark 14. This OIM with a total budget B is different from OIM in previous work, such as [START_REF] Wang | Improving regret bounds for combinatorial semi-bandits with probabilistically triggered arms and its applications[END_REF], even when we set all costs to be equal. In our setting, there is only one total budget for all rounds, and the policy is free to choose seed sets of different cost in each round, whereas in the previous work, each round had a fixed budget for the number/cost of seeds selected. Our setting thus avoid the use of a budget per round, which is in practice more difficult to establish than a global budget B. Nevertheless, as we will see in subsection 6.3.2, both types of constraints (global and per round) can be considered simultaneously when the true costs are known.

Justification for the approximation regret

In the non-budgeted OIM problem with a cardinality constraint given by m ∈ [|V |], let us recall that the approximation regret

R T ,ε (π) t≤T max S⊂V , |S|=m E[(1 -1/e -ε)σ(S; W) -σ(S t ; W)]
is standard [START_REF] Wen | Online influence maximization under independent cascade model with semi-bandit feedback[END_REF][START_REF] Wang | Improving regret bounds for combinatorial semi-bandits with probabilistically triggered arms and its applications[END_REF]. In this notion of regret, the factor (1 -1/e -ε) (Feige, 1998; Chen, Wang, and Wang, 2010) reflects the difficulty of approximating the following NP-Hard [START_REF] Kempe | Maximizing the spread of influence through a social network[END_REF] problem in the case the distribution P W is described by IC or LT, and is known to the agent:

max S⊂V , |S|=m E[σ(S; W)].
(6.3)

For our budgeted setting, at first sight, it is not straightforward to know which approximation factor to choose. Indeed, since the random horizon may be different in F B (π) and in F * B , the expected regret R B,ε (π) is not expressed as the expectation of a sum of approximation gaps, so we can't directly reduce the regret level approximability to the gap level approximability. We thus consider a quantity provably close to R B,ε (π) and easier to handle.

Proposition 19. Define

λ * max S⊂V E[σ(S; W)] E e T S∪{0} C
.

For all S ⊂ V , define the gap corresponding to S as

∆(S) (1 -1/e -ε)λ * E e T S∪{0} C -E[σ(S; W)].
Then, for any policy π selecting S t at round t,

R B,ε (π) -E τ B -1 t=1 ∆(S t ) ≤ 2|V | + 2λ * (1 + |V |).
From Proposition 19 (which is proven in subsection 6.5.1), R B,ε (π) and E τ B -1 t=1 ∆(S t ) are equivalent in term of regret upper bound rate. Therefore, the factor (1 -1/e -ε) should reflect the approximability of

max S⊂V E[σ(S; W)]/E e T S∪{0} C = max S⊂V f (S)/c(S). (6.4)
Considering the specific problem where the cost function is of the form c(S) = c 1 |S| + c 0 , for some (c 0 , c 1 ) ∈ [0, 1] 2 , we can reduce6 the approximability of (6.4) to the approximability of the following problem considered in [START_REF] Wang | Fast Thompson Sampling Algorithm with Cumulative Oversampling: Application to Budgeted Influence Maximization[END_REF]: (6.5) for some given integer m, where the expectations are with respect to a randomization in the approximation algorithm. [START_REF] Wang | Fast Thompson Sampling Algorithm with Cumulative Oversampling: Application to Budgeted Influence Maximization[END_REF] proved that this problem is NP-hard by reducing to the set cover problem. We show here that an approximation ratio α better that 1 -1/e yields a high probability approximation for set cover within [START_REF] Feige | A threshold of ln n for approximating set cover[END_REF]. Consider the graph where the collection of closed out-neighborhoods is exactly the collection of sets in the set cover instance. First, trying out all possible values of m, we concentrate on the case in which the optimal m for set cover is tried out. As in [START_REF] Feige | A threshold of ln n for approximating set cover[END_REF], for k ∈ N * , we repeatedly apply the algorithm that α-approximate (6.5). It outputs a set S k (that can be associated with a set of neighborhoods) and after each application the nodes already covered by previous applications are removed from the graph, giving a sequence of objective functions (f k ) with f 1 = f . We thus obtain

max S⊂V E[f (S)] s.t. E[|S|] = m,
(1 -δ) log(|V |), δ > 0, which is impossible unless NP ⊂ BPTIME n O(log log(|V |)) (
E[f k (S k )|S 1 , . . . , S k-1 ] ≥ α |V | - k-1 k =1 f k (S k ) . Noticing that E[f (S 1 ∪ • • • ∪ S k )] = k k =1 E[f k (S k )], we get E[f (S 1 ∪ • • • ∪ S k )] ≥ 1 -(1 -α) k |V |. After = log(1/|V |)/ log(1 -α) < (1 -δ) log(|V |) iterations, we obtain that S = S 1 ∪ • • • ∪ S is a cover, i.e., f (S) = |V |.
The result follows noticing that in expectation (and so with probability at least 1/( m)), we have |S| ≤ m.

1 -based approach

We are now considering algorithms for IC with edge level semi-bandit feedback.

Setting

For w ∈ [0, 1] V , we recall that we can define an IC model by taking

P W = ⊗ ij∈E Bernoulli(w ij ).
We can extend the two previous functions p i and σ to w taking values in [0, 1] V as follows: Let W ∼ ⊗ ij∈E Bernoulli(w ij ). We define the probability that i is influ- enced by S under W as p i (S; w) P S W i , and we let the spread be σ(S; w)

E i ∈ V , S W i . Another expression for the spread is σ(S; w) = i∈V p i (S; w).
We fix a weight vector on E,

w * w * ij ij∈E ∈ [0, 1] E , a cost vector on V ∪ {0}, c * (c * i ) i∈V ∪{0} ∈ [0, 1] V ∪{0}
, with c * 0 > 0. These quantities are initially unknown to the agent. We assume from now that

P W ⊗ ij∈E Bernoulli w * ij ,
and that

E[C] = c * .
We also define S * ∈ arg max S⊂V σ(S; w * )/e T S∪{0} c * . We assume that the feedback received by the agent at round t is

W ij,t , ij ∈ E, S t Wt i .
The agent also receives semi-bandit feedback from the costs, i.e.,

{C i,t , i ∈ V , i ∈ S t ∪ {0}}
is observed.

Algorithm design

In this subsection, we present boim-cucb, cucb for Budgeted OIM problem as Algorithm 9. As we saw in Proposition 19, the policy that, at each round, (1 The budget is exhausted: leave the for loop. end if end for has a bounded regret. Thus, boim-cucb shall be based on this objective. Not only there are some estimation concerns due to the unknown parameters w * , c * , but in addition to that, we also need to evaluate/optimize our estimates of (6.6).

We begin by introducing some notations. We define the empirical means for t ≥ 1 as: For all i ∈ V ∪ {0},

c i,t-1 t ∈[t-1] I{i ∈ S t ∪ {0}}C i,t N i,t-1 ,
and for all ij ∈ E,

w ij,t-1 t ∈[t-1] I S t W t i W ij,t N ⊕ i,t-1 ,
where

N i,t-1 t-1 t =1 I{i ∈ S t }, N ⊕ i,t-1 t-1 t =1 I S t W t i . Using concentra-
tion inequalities, we get confidence intervals for the above estimates. We are then able to use an upper-confidence-bound (UCB) strategy [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF]. More precisely, in the case costs are unknown, we first build the lower confidence bound (LCB) on c * i as follows

c i,t 0 ∨ c i,t-1 - 1.5 log(t) N i,t-1 .
We can also define UCBs for w * ij :

w ij,t 1 ∧ w ij,t-1 + 1.5 log(t) N ⊕ i,t-1 .
We use w ij,t = 1 (and c i,t = 0) when the corresponding counter is equal to 0. Our boim-cucb approach chooses at each round t the seed set S t given by Algorithm 10 which, as we shall see, approximately maximize S → σ(S; w t )/ e T S∪{0} c t . Indeed, with high probability, this set function is an upper bound on the true ratio (6.6) (using that σ is non decreasing w.r.t. w). Notice that this approach is followed by [START_REF] Wang | Improving regret bounds for combinatorial semi-bandits with probabilistically triggered arms and its applications[END_REF] for the non budgeted setting, i.e., they choose S t , |S t | ≤ m that approximately maximize S → σ(S; w t ). To complete the description of our algorithm, we need to describe Algorithm 10. This is the purpose of the following.

Greedy for ratio maximization

In boim-cucb, one has to approximately maximize the ratio S → σ(S; w t )/e T S∪{0} c t , that is a ratio of submodular over modular function. A greedy technique can be used (see Algorithm 10). Indeed, instead of maximizing the marginal contribution at each time step, as the standard greedy algorithm do, the approach is to maximize the so called bang-per-buck, i.e., the marginal contribution divided by the marginal cost. This builds a sequence of increasing subsets, and the final output is the one that maximizes the ratio. The following Proposition 20 gives an approximation factor of 1 -1/e for Algorithm 10.

Proposition 20. Algorithm 10 with input σ, c is guaranteed to obtain a solution S such that:

1

-e -1 σ(S * ) e T S * ∪{0} c ≤ σ(S) e T S∪{0} c
.

Proof. In the proof, we use the notation

σ(i|S) σ({i} ∪ S) -σ(S). For any k ∈ [|V |], σ(S * ) -σ(S k-1 ) ≤ i∈S * \S k-1 σ(i|S k-1 ) Submodularity, monotonicity of σ ≤ σ(i k |S k-1 ) c i k i∈S * \S k-1 c i Algorithm 10 ≤ σ(i k |S k-1 ) c i k i∈S * c i . i.e., for all k ∈ [|V |] such that σ(S * ) -σ(S k-1 ) ≥ 0, c i k e T S * c ≤ σ(i k |S k-1 ) σ(S * ) -σ(S k-1 )
. (6.7)

There must be an index ∈ {0, 1, . . .

, |V | -1} such that e T S c ≤ e T S * c ≤ e T S +1 c. Let β ∈ [0, 1] be such that e T S * c = (1 -β)e T S c + βe T S +1 c. (6.8) If σ(S * ) -(1 -β)σ(S ) -βσ(S +1 ) ≤ 0, then we have 1 -e -1 σ(S * ) e T S * c ≤ σ(S * ) e T S * c ≤ (1 -β)σ(S ) + βσ(S +1 ) (1 -β)e T S c + βe T S +1 c . Else, σ(S * ) -(1 -β)σ(S ) -βσ(S +1 ) > 0 and σ(S * ) -σ(S k ) > 0 for all k ∈ [ ],
(6.9) so we can write the following,

σ(S * ) -(1 -β)σ(S ) -βσ(S +1 ) σ(S * ) ≤ σ(S * ) -(1 -β)σ(S ) -βσ(S +1 ) σ(S * |∅) = σ(S * ) -σ(S ) -βσ(i +1 |S ) σ(S * ) -σ(S ) k∈[ ] σ(S * ) -σ(S k ) σ(S * ) -σ(S k-1 ) = 1 - βσ(i +1 |S ) σ(S * ) -σ(S ) k∈[ ] 1 - σ(i k |S k-1 ) σ(S * ) -σ(S k-1 ) ≤ 1 - βc i +1 e T S * c k∈[ ] 1 - c i k e T S * c
(6.7) and (6.9)

≤ exp - βc i +1 + k∈[ ] c(i k ) e T S * c 1 -x ≤ e -x = exp - (1 -β)e T S c + βe T S +1 c e T S * c = e -1 .
(6.8)

Rearranging the inequality, we obtain 1

-e -1 σ(S * ) ≤ (1 -β)σ(S ) + βσ(S +1 ), i.e., 1 -e -1 σ(S * ) e T S * ∪{0} c ≤ (1 -β)σ(S ) + βσ(S +1 ) (1 -β)e T S ∪{0} c + βe T S +1 ∪{0} c
.

The output S of Algorithm 10 maximizes the ratio of σ(S k )/e T S k ∪{0} c over k. Thus,

max k≤ +1 σ(S k ) e T S k ∪{0} c ≤ σ(S) e T S∪{0} c
.

We end the proof remarking that max

k∈{ , +1} σ(S k ) e T S k ∪{0} c ≥ (1 -β)σ(S ) + βσ(S +1 ) (1 -β)e T S ∪{0} c + βe T S +1 ∪{0} c
.

Notice, a similar result as Proposition 20 is stated in Theorem 3.2 of [START_REF] Bai | Algorithms for optimizing the ratio of submodular functions[END_REF]. However, their proof doesn't hold in our case, since their inequality (16) would be true only for a normalized cost (i.e. c 0 = 0). Actually, c 0 = 0 implies that S * is a singleton, from subadditivity of σ.

For more efficiency, we use a greedy algorithm with lazy evaluations [START_REF] Minoux | Accelerated greedy algorithms for maximizing submodular set functions[END_REF][START_REF] Leskovec | Cost-effective Outbreak Detection in Networks[END_REF], leveraging on the submodularity of σ. More precisely, in Algorithm 10, instead of taking the arg max in the step

S k ← S k-1 ∪ arg max i∈V \S k-1 σ({i} ∪ S k-1 ) -σ(S k-1 ) c i ,
we maintain an upper bound ρ (initially ∞) on hhe marginal gain, sorted in decreasing order. In each iteration k, we evaluates the element on top of the list, say i, and updates its upper bound with the marginal gain at S k-1 . If after the update the upper bound is greater than the others, submodularity guarantees that i is the element with the largest marginal gain. Algorithm 10 (and the approximation factor) can't be used directly in the OIM context, since computing the exact spread σ is #P hard [START_REF] Chen | Scalable influence maximization for prevalent viral marketing in large-scale social networks[END_REF]. However, with Monte Carlo (MC) simulations, it can efficiently reach an Algorithm 10 greedy for ratio, Lazy implementation Input: σ that is an increasing submodular function,

c ∈ [0, 1] V ∪{0} . S 0 ← ∅. ρ ← [(∞, i)] i∈V . for k ∈ [|V |] do checked ← S k-1 . ( * ) Remove the first element ρ[0] = (∼, i) from ρ. if i / ∈ checked then Insert ((σ({i} ∪ S k-1 ) -σ(S k-1 ))/c i , i) in ρ, such that ρ[:][0] is sorted in de- creasing order. Add i to checked and go back to ( * ). else S k ← S k-1 ∪ {i}. end if end for k ← arg max k∈{0,...,|V |} σ(S k )/e T S k ∪{0} c. Output: S k .
arbitrarily close ratio of α = 1 -1/e -ε, with a high probability 1 -1/ t log 2 (t) [START_REF] Kempe | Maximizing the spread of influence through a social network[END_REF].

Regret bound for Algorithm 9

We provide a gap dependent upper bound on the regret of boim-cucb in Theorem 27. For this, we define, for i ∈ V , the gap ∆ i,min min S⊂V , p i (S;w * )>0, ∆(S)>0 ∆(S).

We also define, with d k being the out-degree of node k,

p i,max max S⊂V , p i (S;w * )>0 k∈V d k p k (S; w * ).
We also state the following smoothness property of the spread.

Fact 7 (Smoothness property of the spread). for all S ⊂ V , and all w, w ∈

[0, 1] E , ∀k ∈ V , p k (S; w) -p k (S; w ) ≤ ij∈E p i (S; w) w ij -w ij .
In particular,

σ(S; w) -σ(S; w ) ≤ |V | ij∈E p i (S; w) w ij -w ij .
We refer the reader to Proposition 22 for a proof of the above fact, where we provide a more general statement.

Theorem 27. If π is the policy described in Algorithm 9, then

R B,ε (π) = O log B i∈V |V | λ * + d i p i,max |V | ∆ i,min .
In addition, if true costs are known, then

R B,ε (π) = O log B i∈V d i p i,max |V | 2 ∆ i,min .
Proof. Let α = 1 -1/e -ε, and t ≥ 1. From Proposition 19, we have to upper bound

E τ B -1 t=1 ∆(S t ) .
Fix t ≥ 1. We consider the following events:

W t ∀ij ∈ E, 0 ≤ w * ij -w ij,t ≤ 2 1.5 log(t) N ⊕ ij,t-1 , C t ∀i ∈ V ∪ {0}, 0 ≤ c * i -c i,t ≤ 2 1.5 log(t) N i,t-1 .
We also consider the event A t under which the α-approximation in Algorithm 9 holds. We already saw that

P[¬A t ] ≤ 1 t log 2 (t)
.

From Hoeffding's inequality, C t holds with probability 1 -2(|V | + 1)/t 2 , and W t holds with probability 1 -2|E|/t 2 . Thus, under the event that either C t , W t or A t doesn't hold, then the regret is bounded by a constant (since we have a convergent series).

We thus now assume that C t , W t and A t hold. In particular, using C t and W t , we can write

λ * = σ(S * ; w * ) e T S * ∪{0} c * ≤ σ(S * ; w t ) e T S * ∪{0} c t .
We can use this relation to write

∆(S t ) = λ * α e T St c * + c * 0 -σ(S t ; w * ) = λ * α e T St c * + c * 0 - σ(S t ; w t ) αλ * + (σ(S t ; w t ) -σ(S t ; w * )) ≤ λ * α e T St c * + c * 0 - σ(S t ; w t ) ασ(S * ; w t ) (e T S * c t + c 0,t ) + (σ(S t ; w t ) -σ(S t ; w * )) ≤ λ * α e T St c * + c * 0 -e T St c t + c 0,t + (σ(S t ; w t ) -σ(S t ; w * )),
where the last inequality is from A t . Notice here that in the case the costs are known, the first term in this bound disappears, and we can then safely take λ * =0, explaining why in the final bound the term in front of λ * disappears. We now use Fact 7, and then C t , W t to further get the bound

∆(S t ) ≤ λ * α i∈St 1 ∧ 2 1.5 log(t) N i,t-1 (6.10) + |V | ij∈E p i (S t ; w * ) 1 ∧ 2 1.5 log(t) N ⊕ ij,t-1 (6.11)
.

Then, necessarily either ∆(S t ) ≤ 2 • (6.10) or ∆(S t ) ≤ 2 • (6.11) is true. The first event can be handle with Theorem 12 (with no probabilistically triggered arms), using the upper bound on the expectation of the random horizon given in the proof of Theorem 19. This allows us to get a term of order

λ * log(B/c * 0 ) i∈V |V | ∆ i,min ,
in the regret upper bound. Theorem 12 with probabilistically triggered arms takes care of the second event to get a bound of order log(B/c * 0 )

i∈V d i |V | 2 max S⊂V , p i (S;w * )>0 k∈V d k p k (S; w * ) ∆ i,min .
Problem-independent bound The problem-independent bound of

O   |V | B log B i∈V d i p i,max  
is an immediate consequence of our problem-dependent bound, decomposing, classically, the regret in two terms by filtering by whether or not ∆(S t ) ≤ δ, and then taking the worst regime for δ.

Notice that the analysis can be easily used for the non budgeted setting. In this case, it reduces to the state-of-the-art analysis of [START_REF] Wang | Improving regret bounds for combinatorial semi-bandits with probabilistically triggered arms and its applications[END_REF], except that we slightly simplify and improve the analysis to replace the factor max S⊂V k∈V

d k I{p k (S; w * ) > 0}
by a potentially much lower quantity p i,max . In the case this last quantity is still large, we can further improve it by considering slight modifications to the original Algorithm 9. This is the purpose of the next section.

2 -based approach

We observe that the factor p i,max in Theorem 27 can be as large as |E| in the worst case. In other word, if ∆ = min i ∆ i,min , the rate can be as large as

O log B λ * |V | 2 + |E| 2 |V | 2 ∆ .
We argue here that we can replace

|E| 2 |V | 2 by |E||V | 3 log 2 (|V |))
. Indeed, leveraging on the mutual independence of random variables W ij , we can hope to get a tighter confidence region for w * , and thus a provably tighter regret upper bound [START_REF] Magureanu | Lipschitz Bandits: Regret Lower Bound and Optimal Algorithms[END_REF][START_REF] Combes | Combinatorial Bandits Revisited[END_REF]Degenne and Perchet, 2016b). We consider the following 2 confidence region:

Fact 8 (Confidence ellipsoid for weights, Degenne and Perchet (2016b)). For all t ≥ 2, with probability at least 1 -

1/ t log 2 (t) , ij∈E N ⊕ i,t-1 w * ij -w ij,t-1 2 ≤ δ(t),
where δ(t) 2 log(t) + 2(|E| + 2) log log(t) + 1.

For OIM (both budgeted and non budgeted), there is a large potential gain in the analysis using the confidence region given by Fact 8 compared to simply using an Hoeffding based one, like in boim-cucb. More precisely, for classical combinatorial semi bandits, Degenne and Perchet (2016b) reduced the gap dependent regret upper bound by a factor / log 2 ( ), where in our case can be as large as |E|. However, there is also a drawback in practice with such confidence region: computing the optimistic spread might be inefficient, even if an oracle for evaluating the spread is available. Indeed, for a fixed S ⊂ V , the problem of maximizing w → σ(S; w) over w belonging to some ellipsoid might be hard, since the objective is not necessarily concave. We can overcome this issue using the Fact 7. For S ⊂ V and w ∈ R E , we define the confidence "bonus" as follows:

bonus(S; w) |V | δ(t) i∈V ,N ⊕ i,t-1 >0 d i p i (S; w) 2 N ⊕ i,t-1 .
Notice, we don't sum on vertices with a zero counter. We compensate this by using the convention w ij,t-1 = 1 when N ⊕ i,t-1 = 0. We can successively use Fact 7, Cauchy-Schwartz inequality, and Fact 8 to get, with probability at least 1 -1/ t log 2 (t) , σ(S; w * ) ≤ σ(S; w t-1 ) + bonus(S; w t-1 ). (6.12)

In the same way, with probability at least 1 -1/ t log 2 (t) , we also have (6.13).

σ(S; w * ) ≤ σ(S; w t-1 ) + bonus(S; w * ). (6.13) Contrary to (6.12), this "optimistic spread" can't be used directly by the agent since w * is not known.

Although the optimistic spread defined in (6.12) is now much easier to compute, there is still a major drawback that remains: As a function of S ⊂ V , bonus(S; w t-1 ) is not necessarily submodular, so the optimistic spread is itself no longer submodular. This is an issue because submodularity is a crucial property for reaching the approximation ratio 1 -1/e -ε. We propose here several submodular upper bound to bonus, defined for S ⊂ V and w ∈ R E :

• bonus 1 is actually modular, and simply uses the subadditivity (w.r.t. S) of bonus:

bonus 1 (S; w) |V | j∈S δ(t) i,N ⊕ i,t-1 >0 d i p i ({j}; w) 2 N ⊕ i,t-1 .
• bonus 2 uses the subadditivity of the square root:

bonus 2 (S; w) |V | i,N ⊕ i,t-1 >0 p i (S; w) δ(t)d i N ⊕ i,t-1 .
• bonus 3 uses p i (S; w) 2 ≤ p i (S; w), and is submodular as the composition be- tween a non decreasing concave function (the square root) and a monotone submodular function:

bonus 3 (S; w) |V | δ(t) i,N ⊕ i,t-1 >0 d i p i (S; w) N ⊕ i,t-1 .
• bonus 4 uses Jensen's inequality, and is submodular as the expectation of the square root of a submodular function.

bonus 4 (S; w)

E     |V | i∈V ,N ⊕ i,t-1 >0,S W i δ(t)d i N ⊕ i,t-1     ,
where W ∼ ⊗ ij∈E Bernoulli(w ij ).

We can write the following approximation guarantees for the two first bonus: bonus(S; w) ≤ bonus 1 (S; w) ≤ |S|bonus(S; w), (6.14) bonus(S; w) ≤ bonus 2 (S; w) ≤ |V |bonus(S; w).

Notice, another approach to get a submodular bonus is to approximate p i (S; w t-1 ) by the square root of a modular function [START_REF] Goemans | Approximating submodular functions everywhere[END_REF]. However, not only this bonus would be much more computationally costly to build than ours, but also, we would get only a |V | log|V | approximation factor, which is worst than the one with our bonus 2 . Since increasing the bonus by a factor α ≥ 1 increases the gap dependent regret upper bound by a factor α 2 , we only loose a factor |V | for bonus 2 , compared to the use of bonus, which is still better than the cucb approach. bonus 1 can also be interesting to use when we have some upper bound guarantee on the cardinality of seed sets used. An approximation factor for bonus 3 or bonus 4 doesn't seem interesting, because it would involve the inverse of triggering probabilities. We can, however, further upper bound bonus 3 (S; w * ) as follows:

bonus 3 (S; w * ) = |V | δ(t) i,N ⊕ i,t-1 >0 d i p i (S; w * ) N ⊕ i,t-1 ≤ |V | j∈S i∈V , N ⊕ i,t-1 >0 d i δ(t)p i ({j}; w * ) N ⊕ i,t-1 ≤ |V | δ(t) j∈S |E| 8 N j,t-1 ∧ 1 ,
where the last inequality only holds under some high probability event, given by the following Proposition 21 (which is proven in subsection 6.5.2), involving counters on the costs and counters on the weights.

Proposition 21. Consider the event defined by

P t {∀i ∈ V , N ⊕ i,t-1 ≥ δ(t)}.
Then, for all i, j ∈ V ,

P P t and δ(t)p i ({j}; w * ) N ⊕ i,t-1 > 8δ(t) N j,t-1 ≤ 1/t 2 .
We thus define for S ⊂ V , bonus 5 (S) |V | δ(t) j∈S

|E| 8 N j,t-1 ∧ 1 .
This bonus is much more convenient since it does not depend anymore on w * , and can thus be computed by the agent. Indeed, although the first four bonuses are likely to be tighter than this last submodular bonus 5 , their dependence in w forces us to use them for w = w t-1 . Even if this doesn't pose any problem in practice, this is more difficult to handle in theory since it would involve optimistic estimates on p i ( • ; w t-1 ) itself (see the next section for further details). Actually, we will see that the analysis with bonus 5 is slightly better than the one we would get with bonus 2 , since we loose a factor |V | log 2 (|V |)/ log 2 (|E|) compared to the use of bonus. In addition, it allows a much more interesting constant term in the regret upper bound thanks to the suppression of the dependence in w * .

Although we can improve the analysis based on Fact 8 and 7, the inequality in this last fact may be be less rough in practice (we confirm this in section 6.4). In that case, we suffer from this roughness, since we actually use Fact 7 to design our bonus. In contrast, boim-cucb only uses it in the analysis, and can therefore adapt to a better smoothness inequality. Thus, we consider boim-cucb 5 , where we first compute S t using the boim-cucb approach, and accept it only if

σ(S t ; w t ) ≤ σ(S t ; w t-1 ) + bonus 5 (S t ), (6.15)
otherwise, we chose S t maximizing σ(S; w t-1 ) + bonus 5 (S). For technical reason (due to Proposition 21), we replace S t by S t ∪ {j} if it exists a j ∈ V , such that N ⊕ j,t-1 < δ(t). (6.16)

We thus both enjoy the theoretical advantages of bonus 5 and the practical advantages of boim-cucb. We give the following regret bounds for this approach.

Theorem 28. If π is the policy following boim-cucb 5 , then

R B,ε (π) = O log B i∈V |V | λ * + |V ||E| log 2 |V | ∆ i,min + λ * |V | 2 .
Proof. Let α = 1 -1/e -ε, and t ≥ 1. From Proposition 19, we have to upper bound

E τ B -1 t=1 ∆(S t ) .
In the proof, in addition to P t {∀i ∈ V , N ⊕ i,t-1 ≥ δ(t)}, we consider the following events:

W t    ij∈E N ⊕ i,t-1 w * ij -w ij,t-1 2 ≤ 2δ(t)    , C t ∀i ∈ V ∪ {0}, 0 ≤ c * i -c i,t ≤ 1 ∧ 2 1.5 log(t) N i,t-1 . B t ∀i, j ∈ V , δ(t)p i ({j}; w * ) N ⊕ i,t-1 ≤ 8δ(t) N j,t-1 .
For each node i ∈ V , if N ⊕ i,t-1 ≥ δ(τ B -1), then i will not be intentionally added to the seed set in boim-cucb 5 . Then, each node is intentionally added for at most δ(τ B -1) + 1 times. Thus, we can write

E τ B -1 t=1 ∆(S t )I{¬P t } ≤ E[(δ(τ B -1) + 1)|V |λ * (|V | + 1)] ≤ δ (2B/c * 0 + 1) 2 + 1 |V |λ * (|V | + 1).
We can therefore assume that P t holds. In this case, we have by Proposition 21 that B t doesn't hold with probability bounded by |V | 2 /t 2 . On the other hand, from Fact 8, W t doesn't hold with probability bounded by 1/ t log 2 (t) , and from Hoeffding inequality, C t doesn't hold with probability bounded by 2(|V | + 1)/t 2 . We can consider the event A t under which the α-approximation in boim-cucb 5 holds.

We already saw that

P[¬A t ] ≤ 1 t log 2 (t) .
The regret in the case one of the events ¬A t , ¬B t , ¬W t , ¬C t holds is thus bounded by a constant depending on |V | and λ * . It thus remains to upper bound

E τ B -1 t=1 ∆(S t )I{A t , B t , W t , C t } .
For this, notice that from A t , S t which is the seed set chosen by our policy at round t, is an α-approximate maximizer of A → f (A)/(e T A c t + c 0,t ), where f is one of the optimistic spreads considered in boim-cucb 5 . We thus have

f (S t ) e T St c t + c 0,t ≥ α f (S * ) e T S * c t + c 0,t
, where S * ∈ arg max S⊂V σ(S;w * ) e T S c * +c * 0 . Since under W t , f (S * ) ≥ σ(S * ; w * ), we can derive the following upper bound on the gap:

∆(S t ) = λ * α e T St c * + c * 0 -σ(S t ; w * ) = λ * α e T St c * + c * 0 - f (S t ) αλ * + (f (S t ) -σ(S t ; w * )) ≤ λ * α e T St c * + c * 0 - f (S t ) αf (S * ) (e T S * c t + c 0,t ) + (f (S t ) -σ(S t ; w * )) W t , C t ≤ λ * α e T St c * + c * 0 -e T St c t + c 0,t + (f (S t ) -σ(S t ; w * )). A t
From this point, we can use the condition satisfied by f in boim-cucb 5 :

f (S t ) ≤ σ(S t ; w t-1 ) + bonus 5 (S t ).

Using Fact 7 with Fact 8, we can further have with Cauchy-Schwartz inequality σ(S t ; w t-1 ) -σ(S t ; w * ) ≤ bonus 5 (S t ).

This allows us to get, using C t ,

∆(S t ) ≤ λ * α i∈St∪{0} 1 ∧ 2 1.5 log(t) N i,t-1 + 2bonus 5 (S t ).
(6.17)

The first part in (6.17) can be handle using Theorem 12 to get a term of order

λ * log(B/c * 0 ) i∈V |V | ∆ i,min ,
in the regret upper bound. We can use Theorem 13 to deal with the second part, to get a term of order

δ(B/c * 0 )|V | 2 |E| i∈V log 2 (|V |) ∆ i,min ,
in the regret upper bound. We thus get the desired result.

Such analysis also holds in the non-budgeted setting, and maximizing the spread only instead of the ratio, we can build a policy π satisfying the following (with the standard definition of the non-budgeted gaps):

R T ,ε (π) = O log T i∈V |V | 2 |E| log 2 |V | ∆ i,min .
The regret rate is thus better than the one from Wang and Chen (2017), gaining a factor |E|/ |V | log 2 (|V |) .

Improvements using bonus 1 and bonus 4

In this subsection, we show that the use of bonus 1 and bonus 4 leads to a better regret leading term, at the cost of a large second order term. In the following, we propose boim-cucb 1 (resp. boim-cucb 4 ), that are the same approach as boimcucb 5 with bonus 1 ( • ; w t-1 ) (resp. bonus 4 ( • ; w t-1 )) instead of bonus 5 , and where condition (6.16) is replaced by

∃j ∈ V , N ⊕ j,t-1 ≤ |E|δ(t).
bonus 1 for low cardinality seed sets

In many real world scenarios, maximal cardinality of seed set is small compared to |V |. Indeed, in the non-budgeted setting, it is limited by m, and it is usually assumed that m is much smaller than |V |. In the budgeted setting, we will see in subsection 6.3.2 how to limit the cost of the chosen seeds, and this is likely to also induce a limit on the cardinality of seeds. Using bonus 1 is more appropriate in this situation, according to the approximation factor (6.14). We state in Theorem 29 the regret bound for boim-cucb 1 .

Theorem 29. If π is the policy boim-cucb 1 , and if all seeds selected have a cardinality bounded by m, then we have

R B,ε (π) = O log B i∈V m λ * + m|V | 2 d i log 2 (|E|) ∆ i,min + λ * |V | 2 |E| .
Proof. Let α = 1 -1/e -ε, and t ≥ 1. The beginning of the proof is the same as in Theorem 28, except we no longer consider the event B t , and we consider a new event:

R t {∀i ∈ V , N ⊕ i,t-1 ≥ |E|δ(t)}.
As for Theorem 28:

• The regret in the case R t doesn't hold can be bounded by a term of order

λ * |V | 2 |E| log(B).
• When all the events hold, the same analysis gives

∆(S t ) ≤ 2λ * α i∈St 1 ∧ 2 1.5 log(t) N i,t-1 + 4bonus 1 (S t ; w t-1 ),
and the first term can be handled in the same way.

The second term can be analyzed in the following way: After bounding it by 4mbonus(S t ; w t-1 ),

see that using Fact 7 on the quantity p i (S; w t-1 ) present in this bonus, we get

p i (S t ; w t-1 ) ≤ p i (S t ; w * ) + 1 |V | bonus(S t ; w * ).
By subadditivity, and from R t , we have 4mbonus(S t ; w t-1 ) ≤ 4mbonus(S t ; w * ) + 4m|V | δ(t)

i∈V d i bonus(S t ; w * ) 2 |V | 2 N ⊕ i,t-1 ≤ 4mbonus(S t ; w * ) + 4m|V | i∈V d i bonus(S t ; w * ) 2 |V | 2 |E| = 8mbonus(S t ; w * ).
We can now use Theorem 13 to get a bound of order

δ(B/c * 0 )m 2 |V | 2 i∈V d i log 2 (|E|) ∆ i,min .
As previously, we can state the following non-budgeted version, with seed set cardinality constrained by m:

R T ,ε (π) = O log T i∈V m 2 |V | 2 d i log 2 (|E|) ∆ i,min + |E||V | 2 .
Notice, for both settings, there is an improvement in the main term (the gap dependent one), in the case m ≤ |V |. However, there is also a higher gap independent term that appears.

bonus 4 : the same performance as bonus?

We show here that the regret with bonus 4 is of the same order as what we would have had with bonus (which is not submodular). However, bonus 4 does not have the calculation guarantees of the other bonuses. We state in Theorem 30 the regret bound for the policy boim-cucb 4 . Notice that we obtain a bound whose leading term improves by a factor |E|/ log 2 |E| that of boim-cucb.

Theorem 30. If π is the policy boim-cucb 4 , then we have

R B,ε (π) = O log B i∈V |V |λ * + |V | 2 d i log 2 (|E|) ∆ i,min + λ * |V | 2 |E| .
Before proving this result, we provide some preliminaries. If we let p S S (w)

P i ∈ V , S W i = S , then another expression is bonus 4 (S; w) = S ⊃S p S S (w) |V | δ(t) i∈S d i N ⊕ i,t-1 g(S ) = k≥0 (g(S k+1 ) -g(S k )) S / ∈{S 0 ,...,S k } p S S (w).
where g(S) = g(S 1 ) ≤ g(S 2 ) ≤ . . . and S 0 = ∅.

Since this bonus shall be used with w t-1 , we need a smoothness inequality to link p S S (w t-1 ) to p S S (w * ). We prove here the following such inequality. Proposition 22. For all S ⊂ V , all w, w ∈ [0, 1] E and all collection of subsets of vertices S, we have

S ∈S (p S S (w) -p S S (w )) ≤ ij∈E p i (S; w) w ij -w ij .
Proof. We assume w.l.o.g. that w ≥ w. We consider the random graph

G W = (V , {ij ∈ E, W ij = 1}), where W ∼ ⊗ ij∈E Bernoulli(w ij ). We build G W from G W
by adding edges ij independently with probability

w ij -w ij 1-w ij for each ij that is not an edge in G W . Now, see that S ∈S p S S (w) = P S W S - S / ∈S p S S (w),
where S W S means S W i for all i ∈ S . Thus,

0 ≤ p S S (w ) -p S S (w) = P S W S -P S W S -   S / ∈S p S S (w ) - S / ∈S p S S (w)   ≤ P S W S -P S W S = P S W S but not S W S ≤ P ∃ij ∈ E s.t. S W i, W ij = 0, and W ij = 1 .
The last inequality is by noticing that if S W S but not S W S , then there must be a edge ij accessible from S in G W such that W ij = 0 and W ij = 1. Taking the first such edge ij (watching the contagion spread from S step by step), we see that ij must be accessible from S in G W as well (since otherwise there's a previous accessible edge k that verifies W k = 0 and W k = 1). We have that

S W i is independent from W ij , W ij . Since P W ij = 0, and W ij = 1 = (1 -w ij ) w ij -w ij 1 -w ij = w ij -w ij ,
we have

P ∃ij ∈ E s.t. S W i, W ij = 0, and W ij = 1 ≤ ij∈E p i (S; w) w ij -w ij .
Proof of Theorem 30. We apply a similar analysis as above. When all the events hold, the same analysis gives

∆(S t ) ≤ 2λ * α i∈St 1 ∧ 2 1.5 log(t) N i,t-1 + 4bonus 4 (S t ; w t-1 ),
and the first term can be handled in the same way. The second term can be analyzed in the following way: Using Proposition 22 with S = {S ⊂ V , S / ∈ {S 0 , . . . , S k }}, we get 4bonus 4 (S t ; w t-1 ) ≤ 4bonus 4 (S t ; w * ) +

k≥0 (g(S k+1 ) -g(S k )) 1 |V | bonus 4 (S t ; w * ) =   4 + δ(t) i∈V d i N ⊕ i,t-1   bonus 4 (S t ; w * ) ≤ 5bonus 4 (S t ; w * ),
where the last inequality uses the event

R t {∀i ∈ V , N ⊕ i,t-1 ≥ |E|δ(t)}.
Relying on Theorem 14, we can deal with this last term and obtain a term of order

δB/c * 0 i∈V |V | 2 d i log 2 (|E|) ∆ i,min .
As previously, we can state the following non-budgeted version. Notice that the cardinality constraint does not appear in the bound.

R T ,ε (π) = O log T i∈V |V | 2 d i log 2 (|E|) ∆ i,min + |E||V | 2 .
In spite of the superiority in terms of regret of the use of bonus 4 , we must point out that, in the worst case, the calculation of this bonus may require a number of sample (and thus a time complexity) polynomial in t, which does not meet the criterion of efficiency that we set ourselves at the beginning of the chapter.

Knapsack constraint for known costs

In their setting, [START_REF] Wang | Fast Thompson Sampling Algorithm with Cumulative Oversampling: Application to Budgeted Influence Maximization[END_REF] considered the relaxed constraint E e S∪{0} c * ≤ b, (6.18) instead of ratio maximization, where the expectation is over the possible randomness of S. When true costs are known to the agent, we can actually combine the two settings: a seed set S can be chosen only if it satisfies (6.18). In this section, we describe modifications this new setting implies. First of all, the regret definition is impacted, and F * B is now maximal for policies respecting the constraint (6.18) within each round. Naturally, the definitions of λ * and S * are also modified accordingly. Otherwise, apart from Algorithm 10, there is conceptually no change in the approaches that have been described in this paper. We now described the modification needed to make Algorithm 10 works in this setting. The same sequence of set S k is considered, but instead of choosing the set that maximizes the ratio over all k ∈ {0, . . . , |V |}, we restrict the maximization to k ∈ {0, . . . , j}, where j is the first index such that e T S j ∪{0} c * > b. If this maximizer is not S j , then it satisfies the constraint and is output. Else, we output S j with probability (be T S j-1 ∪{0} c * )/c * j and S j-1 with probability 1 -(be T S j-1 ∪{0} c * )/c * j . This way, the expected cost of the output is b. The following Proposition 23 gives an approximation factor of 1 -1/e for the above modification of Algorithm 10.

Proposition 23. The solution S obtained by the modified Algorithm 10 is such that:

1 -e -1 E[σ(S * )] E e T S * ∪{0} c * ≤ E[σ(S)] E e T S∪{0} c * ,
where the expectation is over the possible randomness of S, S * .

Proof. There are two possibilities for S * : either E e T S * ∪{0} c * < b, or E e T S * ∪{0} c * = b. In the first case, we know that S * is not random. Indeed, if it is not the case, then

E[σ(S * )] E e T S * ∪{0} c * is a convex combination of some σ(S)
e T S∪{0} c * for S in the support of the distribution of S * . Necessarily, the maximizer (over S in the support) of the ratio is such that e T S∪{0} c * > b, since otherwise this maximizer contradicts the definition of S * . Therefore, increasing the coefficient of this maximizer in the convex combination increases E e T S * ∪{0} c * , which can thus be set to b. Since this also increases

E[σ(S * )] E e T S * ∪{0} c *
, we get a contradiction since we improved the solution S * while still satisfying the constraint.

• Consider the first case. We have as for the proof of Proposition 20, that

1 -e -1 σ(S * ) e T S * ∪{0} c * ≤ (1 -β)σ(S ) + βσ(S +1 ) (1 -β)e T S ∪{0} c * + βe T S +1 ∪{0} c * ,
where ∈ {0, 1, . . . , |V | -1} is such that e T S c ≤ e T S * c ≤ e T S +1 c, and e T S * ∪{0} c * = (1 -β)e T S ∪{0} c * + βe T S +1 ∪{0} c * . In the case S has a greater ratio than S +1 , it is chosen by our algorithm and has the desired approximation. In the case S +1 has the better ratio, it is chosen if its cost is lower than b. If its cost is greater than b, then + 1 = j and the algorithm chooses S with some probability 1 -β and S +1 with probability β . The goal is to show that the coefficient β we use for S +1 is greater than β. This must be the case since (1

-β )e T S ∪{0} c * + β e T S +1 ∪{0} c * = b > e T S * ∪{0} c * = (1 -β)e T S ∪{0} c * + βe T S +1 ∪{0} c * .
• For the second case, we let S be the output of the Algorithm 1 considered by [START_REF] Wang | Fast Thompson Sampling Algorithm with Cumulative Oversampling: Application to Budgeted Influence Maximization[END_REF]. We thus have from their Theorem 1 that

1 -e -1 E[σ(S * )] ≤ E[σ(S)].
Since E e T S∪{0} c * = E e T S * ∪{0} c * = b, we have

1 -e -1 E[σ(S * )] E e T S * ∪{0} c * ≤ E[σ(S)] E e T S∪{0} c * •
If the expected cost of the output S of our algorithm is b, then both algorithms coincides and we have the desired result. Else, we have that S maximizes the ratio over {S 0 , . . . , S j }, which contains the support of S (that is {S j-1 , S j }), so the ratio evaluated at S is greater than

E[σ(S)] E e T S∪{0} c *
, giving again the desired result.

Experiments and discussion

In this section, we present an experiment for Budgeted OIM. In Figure 6.1, we plot E τ B -1 t=1 ∆(S t ) with respect to the budget B used, running over up to T = 10000 rounds. This quantity is a good approximations to the true regret according to Proposition 19. Plotting the true regret would require to compute F * B , which is NP-Hard to do. We consider a subgraph of Facebook network [START_REF] Leskovec | {SNAP Datasets}: {Stanford} Large Network Dataset Collection[END_REF], with |V | = 333 and |E| = 5038, as in [START_REF] Wen | Online influence maximization under independent cascade model with semi-bandit feedback[END_REF]. We take w * ∼ U (0, 0.1) ⊗E and take deterministic, known costs with c * 0 = 1, and c * i = d i / max j∈V d j . boim-cucb + is the same approach as boim-cucb 5 with bonus( • ; w t-1 ) instead of bonus 5 , ignoring that bonus( • ; w t-1 ) is not submodular (it is only sub-additive).

We observed that in boim-cucb 1 , boim-cucb 4 , boim-cucb 5 , Condition 6.15 (with the correct bonus instead of bonus 5 ) always holds, meaning that those algorithms coincide with boim-cucb in practice, and that the gain only appears through the analysis. We thus plot a single curve for these 4 algorithms in Figure 6.1. On the other hand, we observe only a slight gain of boim-cucb+ compared to boim-cucb.

Our experiments confirm that Fact 7 is less rough in practice, as we already anticipated. Indeed, our submodular bonuses are not tight enough to compete with boim-cucb, although we gain in the analysis. The slight gain that we have for boim-cucb+ suggests that the issue is not only about the tightness of a submodular upper bound, but rather about the tightness of Fact 7. This is supported by the following observation we made: for the Facebook subnetwork, for 1000 random draws of a seed set and vector pairs in [0, 0.1] E , the ratio of the RHS and the LHS in Fact 7 is each time greater than 0.4|V |.

In the following, we conducted further experiments on a synthetic graph comparing boim-cucb to boim-cucb-regularized, which greedily maximizes the regularized spread S → σ(S; w t ) -λe S c t , where λ is a parameter to set. We observed that for an appropriate choice of λ, a performance similar to boim-cucb can be obtained. The experiments were conducted on a complete 10 nodes graph, with known costs c * 0 = 1, and for all i ∈ V , c * i ∼ U (0, 1). We also chose w * ∼ U (0, 0.1) ⊗E , as previously. We compare the boim-cucb algorithm to boim-cucb-regularized, another algorithm that might challenge boim-cucb in our setting. boim-cucb-regularized is exactly as boim-cucb except that the objective that is optimized is S → σ(S; w t ) -λe S c t , for λ being an input parameter to the algorithm. We can see that as for boim-cucb, this algorithm have the willingness to maximize the function σ while minimizing the cost function. The fundamental difference is on the importance given to one or the other function, controlled by λ. We use a greedy maximization in boimcucb-regularized. A greedy optimization of the objective S → σ(S; w t ) -λe S c t is a heuristic which, although not supported in theory, performs well in practice. We run experiments over up to T = 10000 rounds, on five different draws for w * and c * , and 3 different values λ = 2, 3, 4. Results are shown in Figure 6.2. We observe that boim-cucb is in general better than boim-cucb-regularized. If the variable λ is properly chosen, performances similar to boim-cucb can be obtained. This is not surprising since boim-cucb aims (

Discussion and Future work

We introduced a new Budgeted OIM problem, taking both the costs of influencers and fixed costs into account in the seed selection, instead of the usual cardinality constraint. This better represents the current challenges in viral marketing, since top influencers tend to be more and more costly. Our fixed cost can also be seen as the time that a round takes: A null fixed cost would mean that reloading the network to get a new independent instance is free and instantaneous.7 Obviously, this is not realistic. We also provided an algorithm for Budgeted OIM under the IC model and the edge level semi-bandit feedback setting.

Interesting future directions of research would be to explore other kinds of feedback or diffusion models for Budgeted OIM. For practical scalability, it would also be good to investigate the incorporation of the linear generalization framework [START_REF] Wen | Online influence maximization under independent cascade model with semi-bandit feedback[END_REF] into Budgeted OIM. Notice, this extension is not straightforward if we want to keep our tighter confidence region. More precisely, we believe that a linear semi-bandit approach that is aware of independence between edge observations should be developed (the linear generalization approach of Wen, [START_REF] Wen | Online influence maximization under independent cascade model with semi-bandit feedback[END_REF] treats each edge observation as arbitrary correlated).

In addition to this, exploring how the use of Fact 7 in the Algorithm might be avoided while still using confidence region given by Fact 8 would surely improve the algorithms. One possible way would be to use a Thompson Sampling (TS) approach [START_REF] Wang | Thompson Sampling for Combinatorial Semi-Bandits[END_REF][START_REF] Perrault | Statistical Efficiency of Thompson Sampling for Combinatorial Semi-Bandits[END_REF], where the prior takes into account the mutual independence of weights. However, [START_REF] Wang | Thompson Sampling for Combinatorial Semi-Bandits[END_REF] proved in their Theorem 2 that TS gives linear approximation regret for some special approximation algorithms. Thus, we would have to use some specific property of the greedy approximation algorithm we use.

Missing proofs

Proof of Proposition 19

Proof of Proposition 19. Let α = 1 -1/e -ε. In the proof, we shall consider several policies π one after the other. In each case, we will denote by S t the seed selected by π at round t, and τ B the random round where π has exhausted its budget.

Consider first the policy π that selects

S t = S * ∈ arg max S⊂V E[σ(S; W)]E e T S∪{0} C -1
at each round t ≥ 1. We can write

F * B + |V | ≥ F * B + E[σ(S * ; W)] ≥ F B (π) + E[σ(S * ; W)] definition of F * B = t≥1 E[σ(S * ; W t )I{B t-1 ≥ 0}] = t≥1 E[E[σ(S * ; W)]I{B t-1 ≥ 0}] conditioning on H t = λ * t≥1 E E e T S * ∪{0} C I{B t-1 ≥ 0} = λ * t≥1 E[(C 0,t + e T S * C t )I{B t-1 ≥ 0}] conditioning on H t ≥ λ * B definition of τ B .
We can use the inequality

F * B + |V | ≥ λ * B (6.19)
with any policy π in the following ways:

• First, we can bound the cost part in the cumulative gap:

αλ * E τ B -1 t=1 E e T St∪{0} C ≤ αλ * t≥1 E E e T St∪{0} C I{B t-1 ≥ 0} B t ≥ 0 ⇒ B t-1 ≥ 0 = αλ * t≥1 E e T St C t + C 0,t I{B t-1 ≥ 0} conditioning on H t ≤ αλ * B + αλ * (1 + |V |) definition of τ B , ≤ αF * B + α|V | + αλ * (1 + |V |) inequality (6.19).
• Next, we can bound the reward part:

E τ B -1 t=1 E[σ(S t ; W)] ≥ E τ B t=1 E[σ(S t ; W)] -|V | = t≥1 E[E[σ(S t ; W)]I{B t-1 ≥ 0}] -|V | = t≥1 E[σ(S t ; W t )I{B t-1 ≥ 0}] -|V | conditioning on H t ≥ t≥1 E[σ(S t ; W t )I{B t ≥ 0}] -|V | B t ≥ 0 ⇒ B t-1 ≥ 0 = F B (π) -|V |.
Adding these two inequalities, we get the following upper bound on the cumulative gap:

E τ B -1 t=1 ∆(S t ) = αλ * E τ B -1 t=1 E e T St∪{0} C -E τ B -1 t=1 E[σ(S t ; W)] ≤ R B,ε (π) + (α + 1)|V | + αλ * (1 + |V |).
In the same way, we can derive a lower bound on E τ B -1 t=1 ∆(S t ) , considering first the policy π such that F B (π) = F * B :

F * B = t≥1 E[σ(S t ; W t )I{B t ≥ 0}] ≤ t≥1 E[σ(S t ; W t )I{B t-1 ≥ 0}] B t ≥ 0 ⇒ B t-1 ≥ 0 = t≥1 E[E[σ(S t ; W)]I{B t-1 ≥ 0}] conditioning on H t ≤ t≥1 E λ * E e T St∪{0} C I{B t-1 ≥ 0} definition of λ * = λ * E τ B t=1 C 0,t + e T St C t conditioning on H t ≤ λ * (B + 1 + |V |) definition of τ B .
i.e.,

F * B -λ * (1 + |V |) ≤ λ * B. (6.20)
Considering any policy π:

αλ * E τ B -1 t=1 E e T St∪{0} C ≥ αλ * t≥1 E E e T St∪{0} C I{B t-1 ≥ 1 + |V |} B t-1 ≥ 1 + |V | ⇒ B t ≥ 0 = αλ * t≥1 E e T St C t + C 0,t I{B t-1 ≥ 1 + |V |} conditioning on H t ≥ αλ * B -αλ * (1 + |V |) definition of τ B . ≥ αF * B -2αλ * (1 + |V |) inequality (6.20),
and

E τ B -1 t=1 E[σ(S t ; W)] ≤ E τ B t=1 E[σ(S t ; W)] = t≥1 E[E[σ(S t ; W)]I{B t-1 ≥ 0}] = t≥1 E[σ(S t ; W t )I{B t-1 ≥ 0}] conditioning on H t ≤ t≥1 E[σ(S t ; W t )I{B t ≥ 0}] + |V | = F B (π) + |V |.
Again adding these two inequalities, we get the desired lower bound:

E τ B -1 t=1 ∆(S t ) = αλ * E τ B -1 t=1 E e T St∪{0} C -E τ B -1 t=1 E[σ(S t ; W)] ≥ R B,ε (π) -2αλ * (1 + |V |) -|V |.

Proof of Proposition 21

Proof of Proposition 21. First, notice that we trivially have

P P t and p i ({j}; w * ) ≤ 8δ(t) N j,t-1 and δ(t)p i ({j}; w * ) N ⊕ i,t-1 > 8δ(t) N j,t-1 = 0.
Thus, let's prove that P P t and p i ({j}; w * ) > 8δ(t) N j,t-1 and δ(t)p i ({j}; w * )

N ⊕ i,t-1 > 8δ(t) N j,t-1 ≤ 1/t 2 .
We define another counter for (i, j) ∈ V 2 as follows:

N i,j,t-1 t-1 t =1 I j ∈ S t , {j} W t i .
Note that we have N i,j,t-1 ≤ (N ⊕ i,t-1 ∧ N j,t-1 ). 2017)). Consider X 1 , . . . , X t be Bernoulli random variables, of parameter µ, then for Y = X 1 + • • • + X t , we have with δ ∈ (0, 1),

P[Y ≤ (1 -δ)tµ] ≤ e -δ 2 tµ/2 .
(2012). As we already mentioned, in this setting, the whole joint distribution of the vector of outcomes is relevant, contrary to standard bandit problems where only the n marginals are sufficient to characterize the difficulty of the instance. If we define X (X 1 , . . . , X n ), the objective is to design a learning algorithm adaptive to the distribution P X . This is more challenging than in standard bandits, where adaptivity is only with respect to ⊗ i∈[n] P X i .

In a first approach, Degenne and Perchet (2016b) considered the general family of C-sub-Gaussian probability distributions, with C 0 (i.e., C is positive semidefinite). Formally, those distributions P X of mean µ * satisfy ∀λ ∈ R n , E e λ T (X-µ * ) ≤ e λ T Cλ/2 .

(7.1) Degenne and Perchet (2016b) devised an algorithm with a regret bound depending on the components of another matrix Γ 0, satisfying Γ + C (i.e., λ T (Γ -C)λ ≥ 0 for all λ ∈ R n + ) and Γ ij ≥ 0 for all i, j. The major downside is that this algorithm requires the knowledge of Γ. Their upper bound is of order

log T ∆ i∈[n] Γ ii (1 -γ) log 2 (m) + γm , (7.2)
where γ max A∈A max (i,j)∈A 2 ,i =j Γ ij / Γ ii Γ jj is the the maximal off-diagonal correlation coefficient, ∆ is the minimal positive gap between expected total reward of two actions, and m max{|A|, A ∈ A}. Interestingly, their regret upper bound highlights regimes interpolating between worst case correlation between outcomes (corresponding to γ = 1) and mutually independent outcomes (where γ = 0). In particular, the learning rate is much faster in the latter case. The main drawback however, is that their approach is not adaptive since the correlation structure of the arms needs to be given to the agent (through the matrix Γ).

Our main objective is to alleviate this issue, and to strive to obtain fast rates for combinatorial semi-bandits, as Degenne and Perchet (2016b) in the case where there is a favorable covariance structure, but without knowing it beforehand. Therefore, algorithms should be able to capture the covariance structure given by Γ from the data processed and adapt to it. We actually go further by asking whether the matrix Γ is the relevant parameter to characterize the difficulty of a problem. We argue that the covariance matrix Σ * E (X -µ * )(X -µ * ) T is more pertinent, as it allows to better differentiate complex problems from the easy ones. One can indeed already argue in favor of a Σ * dependence rather than a Γ one, based on the relation Σ * + Γ. Indeed, this results from the fact that Σ * C, which can be proved as follows: Fix x ∈ R n . For any λ ∈ R, E e λx T (X-µ * ) ≤ e λ 2 x T Cx/2 . The second order Taylor expansion in λ gives

λ 2 2 E (x T (X -µ * )) 2 + o λ 2 ≤ λ 2 2 x T Cx + o λ 2 .
Dividing the inequality by λ 2 , and letting λ → 0 yields E (x T (X -µ * )) 2 ≤ x T Cx, i.e., Σ * C.

Results and limitations of the results of Degenne and Perchet (2016b)

Below, we list the main limitations of the approach of Degenne and Perchet (2016b):

(i) The matrix Γ needs to be known. This requires specific knowledge about the outcome structure, which is often not precise, as it is usually only known that outcomes are bounded, or at most that there exists some constant κ such that κ 2 ≥ C ii for all i ∈ [n]. The latter is equivalent1 to Γ ij = κ2 for all i, j ∈ [n] and corresponds to the worst case correlation between outcomes (γ = 1) in the regret bound (5.4).

(ii) The value γ can be 1, even when outcomes are only weakly correlated: For instance, if n is even, Γ can be a block-diagonal matrix with n/2 blocks of size 2 × 2 containing only ones. This scenario can actually occur in many examples; we provide two types below:

• Arms are nodes on a given graph, with some small communities on which outcome tends to be constant [START_REF] Cesa-Bianchi | A gang of bandits[END_REF][START_REF] Valko | Spectral bandits for smooth graph functions[END_REF][START_REF] Gentile | Online clustering of bandits[END_REF][START_REF] Valko | Bandits on graphs and structures[END_REF].

• Arms are market-basket-like items, with some highly correlated pairs of items (e.g., people buying from category "books" tend to also buy from category "CDs", [START_REF] Zhang | Finding Highly Correlated Pairs Efficiently with Powerful Pruning[END_REF][START_REF] He | Mining top-k strongly correlated item pairs without minimum correlation threshold[END_REF].

(iii) The value Γ ii can be high, even for low-variance outcomes, while intuitively, low variance outcomes should be easy to work with. For example, if X is a binary 1-sparse random variable -as in some recommender systems, where a single item is desired by the user -then

X i ∼ Bernoulli(µ * i ) with n i=1 µ * i = 1, and Γ ii ≥ C ii ≥ (µ * i -1/2)/(log(µ * i ) -log(1 -µ * i )
) (and this is tight, see, e.g., [START_REF] Buldygin | The sub-Gaussian norm of a binary random variable[END_REF]. For µ * i of order 1/n, Γ ii is thus at least of order 1/(2 log n) for n large, whereas V(X i ) is of order 1/n.

To sum up the arguments above, we claim that (1) knowing a good upper bound on the sub-Gaussianity matrix C + Γ is not realistic and (2) even this upper bound is not a good proxy for the complexity of the instance at hand.

Contributions

We address the three aforementioned criticisms (i), (ii), and (iii).

As a consequence, we do not assume that a good upper bound Γ on the sub-Gaussianity matrix C is known, but only that the agent knows that each marginal P X i is κ 2 -sub-Gaussian. We compensate this relaxation by restricting the distribution family considered through a sub-exponential-type assumption involving the covariance matrix Σ * . We argue that this restriction is mild and satisfied by many outcome distributions, including bounded and Gaussian.

We characterize the difficulty of the problem with Σ * ; specifically, we provide a new lower bound, with a dependence on Σ * , more precise than Degenne and Perchet (2016b). We also design a new algorithm with matching asymptotic regret upper bound, improving over the state-of-the-art results. One of the key techniques is to build an online adapted estimation of the matrix Σ * .

Our main contribution is in the analysis of this approach, that is not based on the usual Laplace's method, which works in the sub-Gaussian framework, but does not handle well our sub-exponential-type assumption. Thus, our analysis is rather based on a covering-argument [START_REF] Magureanu | Lipschitz Bandits: Regret Lower Bound and Optimal Algorithms[END_REF]. An important part of our proof is based on the transformation of the axis-unaligned ellipsoidal confidence region associated to a given action A ∈ A into an axis-aligned region, using the following relation

Σ * ij ij∈A + diag j∈A 0 ∨ Σ * ij i∈A
. This allows us to conduct the same type of proof than for the independent outcome case (where confidence regions are always axis-aligned), but with a Bernstein-type analysis. 2We also consider an application of our approach to the family of sparse bounded outcomes: we provide a lower bound on the regret, with an algorithm having a matching asymptotic regret upper bound.

Prior work on stochastic semi-bandits We review algorithms for stochastic semi-bandits, coming with the analysis that depends on the family of probability distributions to which P X belongs. To begin, Kveton, Wen, Ashkan, and Szepesvari (2015b) and Chen, Wang, and Yuan (2016) studied the general family of distributions having sub-Gaussian or bounded marginals. Their algorithms are not adaptive to P X and regret bounds depend on parameters characterizing the family, that need to be known (such as the sub-Gaussian constant or a bound on X ∞ ). On the other hand, many algorithms are only adaptive to marginals of P X , either with variance estimates (Perrault, Perchet, and Valko, 2019b;[START_REF] Merlis | Batch-Size Independent Regret Bounds for the Combinatorial Multi-Armed Bandit Problem[END_REF], or using Kullback-Leibler divergence. These approaches are agnostic to possible correlation between marginals since the confidence region used in their algorithm are always a Cartesian product of confidence intervals (so they are always n-dimensional hypercubes). As a consequence, this translates into guarantees w.r.t. the worst-case correlations quantity possible. Notice that these algorithms are actually almost direct applications of corresponding classical multi-arm bandits algorithms to the semi-bandit setting. In particular, confidence regions considered are the same in both settings.

Another line of works restricts the probability distributions family of P X , so that the dependence existing between arms is controlled. This conveniently induce better confidence regions valid for distributions in the family, and leads to the development of algorithms based on these regions, having sharper regret upper bounds. For instance, [START_REF] Combes | Combinatorial Bandits Revisited[END_REF] assumed that P X = ⊗ i∈[n] P X i . Confidence regions resemble to axis-align ellipsoid in this specific case. They designed UCB (resp. Thompson sampling) based algorithms, leveraging on such tighter ellipsoidal confidence region. The key difference between the above case is that this time, marginals do characterize the problem, by assumption on the probability distributions family.

Remark that Degenne and Perchet (2016b) provided a regret bound which adapts to the probability distribution family at hand through the matrix Γ, although their algorithm is not fully adaptive. The confidence region used by their algorithm is also ellipsoidal, and depends on the matrix Γ. This matrix gives the control on the correlations between arms. The confidence ellipsoid is not axis aligned unless Γ is diagonal. To the best of our knowledge, their work is the main competitor in terms of regret bound.

Sparse bandits Independently to combinatorial bandits, there exists a different setting actually dealing with correlated outcomes in online learning known as sparse bandits [START_REF] Kwon | Sparse Stochastic Bandits[END_REF][START_REF] Kwon | Gains and Losses are Fundamentally Different in Regret Minimization: The Sparse Case[END_REF][START_REF] Bubeck | Sparsity, variance and curvature in multi-armed bandits[END_REF][START_REF] Abbasi-Yadkori | Onlineto-Confidence-Set Conversions and Application to Sparse Stochastic Bandits[END_REF][START_REF] Carpentier | Bandit Theory meets Compressed Sensing for high dimensional Stochastic Linear Bandit[END_REF][START_REF] Gerchinovitz | Sparsity regret bounds for individual sequences in online linear regression[END_REF]. The overall idea is to introduce by now a standard sparsity assumption (some parameter vector has only s out of its n components that are non zero) into sequential decision making. As usual, the objective is to replace the linear/polynomial dependence in the dimension n by a linear/polynomial dependence in s. Quite interestingly, the sparsity assumption has been studied in two different directions. The fist one assumes that the vector µ * is s-sparse, typically in (linear) stochastic bandits [START_REF] Kwon | Sparse Stochastic Bandits[END_REF][START_REF] Abbasi-Yadkori | Onlineto-Confidence-Set Conversions and Application to Sparse Stochastic Bandits[END_REF][START_REF] Carpentier | Bandit Theory meets Compressed Sensing for high dimensional Stochastic Linear Bandit[END_REF][START_REF] Gerchinovitz | Sparsity regret bounds for individual sequences in online linear regression[END_REF]. The second one assumes that the realized vector X t is s-sparse, usually in adversarial bandits [START_REF] Kwon | Gains and Losses are Fundamentally Different in Regret Minimization: The Sparse Case[END_REF][START_REF] Bubeck | Sparsity, variance and curvature in multi-armed bandits[END_REF].

Sparsity in realized outcomes naturally induces negative correlation; this is not necessarily true for sparsity in expectation. More generally both concepts are complementary, since µ * can be sparse with non-sparse realization (for instance, if all X i are i.i.d., equal to ±1 with probability 1/2) and reciprocally (if X is a canonical unit vector at random, then its expectation has full support). Surprisingly, the sparse outcomes setting has not been investigated in stochastic bandits, even if it lies at the junction of several notions of correlations between outcomes.

A multivariate sub-exponential distribution

Let us recall that the goal for the agent is to minimize the regret, that is defined as follows, with

A * ∈ arg max A∈A e T A µ * , ∀T ≥ 1, R T E T t=1 (e A * -e At ) T X t .
For any action A ∈ A, we define its gap as the difference ∆(A) (e A * -e A ) T µ * . We then rewrite the regret as R T = E T t=1 ∆(A t ) . We start by stating the assumptions satisfied by P X .

Assumption 2 (κ 2 -sub-Gaussian marginals). There is a constant κ > 0 (known to the agent) such that ∀i ∈

[n], ∀λ ∈ R, E e λ(X i -µ * i ) ≤ e κ 2 λ 2 /2 .
Assumption 2 is not difficult to satisfy, and does not require any precision on the correlations between outcomes. In particular, Assumption 2 includes Gaussian outcomes (with variance lower than κ 2 ) and bounded outcomes (with X ∞ ≤ κ). We also assume that X satisfies the following.

Assumption 3 ( • 1 -sub-exponential distribution). ∀λ ∈ R n such that λ 1 ≤ 1/(2κ), we have E e λ T (X-µ * ) ≤ e λ T Σ * λ , where Σ * E (X -µ * )(X -µ * ) T is the covariance matrix of X.
Importantly, the agent does not know the covariance matrix Σ * . Remark that Assumption 3 trivially holds for X ∼ N (µ * , Σ * ), where ∀λ ∈ R n , E e λ T (X-µ * ) = e λ T Σ * λ/2 . The following proposition states that it also holds for bounded outcomes.

Proposition 24. If X ∞ ≤ κ, then both Assumption 2 and 3 hold.

Proof. Assumption 2 is a direct consequence of Hoeffding's Lemma. For Assumption 3, we have Xµ * ∞ ≤ 2κ. For λ 1 ≤ 1/(2κ), we have:

log E e λ T (X-µ * ) = log   1 + k≥2 E (λ T (X -µ * )) k k!   ≤ k≥2 E (λ T (X -µ * )) k k! log(x) ≤ x -1 ∀x > 0, = k≥2 E (λ T (X -µ * )) k-2 (λ T (X -µ * )) 2 k! ≤ k≥2 E ( λ 1 X -µ * ∞ ) k-2 (λ T (X -µ * )) 2 k! ≤ k≥2 E (λ T (X -µ * )) 2 k! = (e -2)λ T Σ * λ ≤ λ T Σ * λ.
Notice, up to a re-normalization of the regret, we assume w.l.o.g. that κ = 1.

Covariance-dependent regret (lower) bound

Lower bound

We start by proving in Theorem 31 a new gap-dependent lower bound on R T , valid for any covariance matrix Σ * 0, for some P X satisfying Assumptions 2 and 3, some action space A, and for any consistent algorithm [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF], for which the regret on any problem verifies R T = o(T a ) as T → ∞, for all a > 0. This lower bound demonstrates the link between Σ * and the difficulty of the problem. It also indicates, in anticipation, that we have to examine a subclass of action sets to hope to improve the upper bound we will provide in Theorem 32.

Theorem 31. For any n, m ∈ N * such that n/m ≥ 2 is an integer, any n × n matrix Σ * 0, any ∆ > 0, and any consistent policy, there exists an instance with n arms -characterized by some action space A, with m = max{|A|, A ∈ A}, some outcome distribution P X satisfying Assumptions 2 and 3 with all gaps equal to ∆ and covariance matrix Σ * -on which the regret satisfies

lim inf T →∞ ∆ log(T ) R T ≥ 2 i∈[n], i/ ∈A * max A∈A, i∈A j∈A Σ * ij . The proof considers A containing n/m disjoint actions A 1 , . . . , A n/m composed of m arms, with A k = {(k -1)m + 1, . . . , km}, and X ∼ N (-∆/m(I{i / ∈ A 1 }) i , Σ *
). The idea is to make a reduction to some standard bandit problems with n/m arms, and to compute the number of rounds t needed to distinguish between A k and A 1 . Roughly speaking, t is at least equal to the inverse of the KL between outcome distributions of A k and its centered version, and in the case of Gaussian distributions, we get t

≥ 2V( i∈A k X i )/∆ 2 = 2e T A k Σ * e A k /∆ 2 .
It is not surprising that the variance appears, since this can be seen as a measure of the uncertainty we have in our samples: The higher the variance, the harder the estimation, and therefore the higher the round t must be. Notice that Theorem 31 is a refinement of Theorem 1 from Degenne and Perchet (2016b), in which they consider the same action space A but a specific choice for the matrix Σ * : it is a block-diagonal matrix with n/m blocks, where each block (corresponding to an action A) is equal to σ 2 ((1 -γ)diag(e A ) + γe A e T A ), i.e., they take the worst case correlation under the controls given by σ 2 and γ, and knowing that the problem given by A is agnostic to the correlations between the arms of two different blocks.

Proof of Theorem 31. Consider A containing n/m disjoint actions A 1 , . . . , A n/m composed of m arms, with A k = {(k -1)m + 1, . . . , km}, and

X ∼ N (-∆/m(I{i / ∈ A 1 }) i , Σ * ).
This problem reduces to a standard bandit problem with n/m arms. We use a result from [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF], a generalization of [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF], that states that lim inf

T →∞ R T log(T ) ≥ n/m k=2 ∆ inf Y , E[Y ]=0 KL P i∈A k X i P Y .
As we can write inf

Y , E[Y ]=0 KL P i∈A k X i P Y ≤ KL N -∆, e T A k Σ * e A k N 0, e T A k Σ * e A k = ∆ 2 /2 e T A k Σ * e A k , it holds that lim inf T →∞ R T log(T ) ≥ 2 n/m k=2 e T A k Σ * e A k ∆ = 2 i∈[n], i/ ∈A 1 max A∈A, i∈A j∈A Σ * ij ∆ ,
where we used the fact that {A ∈ A, i ∈ A} is a singleton.

In the next subsection, we describe our algorithm escb-c (Algorithm 11) and provide an upper bound on its regret in Theorem 32, where the expression max A∈A,i∈A j∈A 0 ∨ Σ * ij appears. Notice that this is very close to the expression given in Theorem 31. In fact, both expressions coincide when Σ * has only non-negative entries.

Main algorithm and the guarantees

In this section, we present an algorithm for the setting introduced in Section 4.1.1. The method is stated as Algorithm 11. To find the action with the highest mean, the agent estimates the mean µ * i of every arm i with their corresponding empirical averages defined as µ i,t-1

u∈[t-1] I{i∈Au}X i,u N i,t-1 , for t ≥ 1, where N i,t-1 u∈[t-1] I{i ∈ A u }
is the number of time arm i have been drawn for the first t -1 rounds. As mentioned above, the agent also estimates the covariance

Σ * ij = E[X i X j ] -µ * i µ * j of each pair (i, j) ∈ [n]
2 . This will be done with the following estimate

Σ ij,t-1 u∈[t-1] I{i, j ∈ A u } X i,u -µ i,t-1 X j,u -µ j,t-1 N ij,t-1 = u∈[t-1] I{i, j ∈ A u } X i,u X j,u -µ i,t-1 X j,u -µ j,t-1 X i,u N ij,t-1 + µ i,t-1 µ j,t-1 ,
where N ij,t-1 u∈[t-1] I{i, j ∈ A u } is the number of times where arm i and j have been drawn together for the first t -1 rounds. Notice that in order to efficiently update Σ ij,t-1 , in addition to µ i,t-1 and µ i,t-1 , we only have to maintain the three quantities,

u∈[t-1] I{i, j ∈ A u }X i,u X j,u N ij,t-1 , u∈[t-1] I{i, j ∈ A u }X i,u N ij,t-1
, and

u∈[t-1] I{i, j ∈ A u }X j,u N ij,t-1 .
Using concentration inequalities, we get confidence intervals for the above estimates.

We are then able to use an upper-confidence-bound strategy [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF]. More precisely, we first build the upper confidence bound on Σ * ij using the fact that X i • X j is a sub-exponential random variable, since both X i and X j are sub-Gaussian by virtue of Assumption 7.1. The result is stated in the following proposition.

Proposition 25. With probability 1 -10t -2 , we have

Σ * ij -Σ ij,t-1 ≤ g ij (t) 16 3 log(t) N ij,t-1 ∨ 3 log(t) N ij,t-1 + 48 log 2 (t) N ij,t-1 N i,t-1 + 36 log 2 (t) N ij,t-1 N j,t-1 .
In particular, defining the upper confidence bound

Σ ij,t Σ ij,t-1 + g ij (t), it holds that 0 ≤ Σ ij,t -Σ * ij ≤ 2g ij (t) with probability 1 -10t -2 . Proof. We define Σ ij,t-1 u∈[t-1] I{i,j∈Au}(X i,u -µ * i )(Xj,u-µ * j ) N ij,t-1
and for k ∈ {i, j},

µ k,t-1 1 N ij,t-1 u∈[t-1] I{i, j ∈ A u }X k,u .
Notice that the following relation holds

Σ ij,t-1 = Σ ij,t-1 + µ * i -µ i,t-1 µ j,t-1 -µ j,t-1 + µ * j -µ j,t-1 ( µ i,t-1 -µ * i ).
We now state Lemma 10 giving sub-exponential parameters for a product of sub-Gaussian random variables. A proof comes from [START_REF] Honorio | Tight Bounds for the Expected Risk of Linear Classifiers and PAC-Bayes Finite-Sample Guarantees[END_REF].

Lemma 10. If Y ,Z are 1-sub-Gaussian random variables, then ∀|λ| ≤ 1/8,

E e λ(Y Z-E[Y Z]) ≤ e 64λ 2 .
We apply Lemma 10 with a Chernoff argument and an union bound (to avoid the randomness of counters) in order to get the following Bernstein inequality

P Σ * ij -Σ ij,t-1 ≥ 16 3 log(t) N ij,t-1 ∨ 3 log(t) N ij,t-1 ≤ 2t -2 .
In the same way, Hoeffding's inequality gives directly that with probability 1 -8t -2 , we have simultaneously

                       µ * i -µ i,t-1 ≤ 6 log(t) N i,t-1 µ j,t-1 -µ j,t-1 ≤ 8 log(t) N ij,t-1 µ * j -µ j,t-1 ≤ 6 log(t) N j,t-1 | µ i,t-1 -µ * i | ≤ 6 log(t) N ij,t-1 ,
which is enough to conclude the proof. Notice that for the second inequality above, we take the union bound for two counters. When they are not random,

N ij,t-1 µ j,t-1 -µ j,t-1 ,
that is equal to

u∈[t-1] I{i, j ∈ A u }X j,u 1 - N ij,t-1 N j,t-1 - u∈[t-1] I{j ∈ A u , i / ∈ A u }X j,u N ij,t-1 N j,t-1 , is a sum of N j,t-1 independent random variables, N ij,t-1 of which are 1 - N ij,t-1 N j,t-1 2 -
sub-Gaussian and the remaining ones are

N 2 ij,t-1 N 2 j,t-1 -sub-Gaussian. Therefore, this ran- dom variable is N ij,t-1 1 - N ij,t-1
N j,t-1 -sub-Gaussian, and in particular N ij,t-1 -sub-Gaussian.

To build estimates well concentrated around µ * , we will use the matrix Σ t defined above to design the following high probability confidence region for all

A ∈ A C t (A) µ t-1 + ξ ∈ R n , i∈A N i,t-1 ξ 2 i |A||ξ i |+ j∈A 0 ∨ Σ ij,t
≤ 8(log t + log log t) + 4em .

(7.

3)

The intuition behind this confidence region is similar to the one for empirical Bernstein confidence intervals, but the term j∈A 0 ∨ Σ ij,t in the denominator replaces the usual empirical variance. To compare our confidence region with the one of Degenne and Perchet, 2016b, notice first that their algorithm uses the matrix Γ to build a confidence ellipsoid. They provide an analysis for this confidence ellipsoid using the Laplace's method and the matrix relation C + Γ. In contrast, our confidence region is based on the covariance matrix Σ * . Our analysis is also different, as we use a coveringargument analysis. This is because the covariance estimation and Assumption 3 are both hard to handle with Laplace's method, that is more appropriate for sub-Gaussian random variables. Indeed, all calculations can be explicit and it is easy to construct a conjugate prior. This is not the case for sub-exponential random variables. Covering arguments are much more easier to use together with a diagonal matrix, so axisaligned confidence region are desirable. We use an axis-realignment technique based on the matrix relation

Σ * ij ij∈A + diag j∈A 0 ∨ Σ * ij i∈A
. The upside is to avoid dealing with off-diagonal terms by transforming them into diagonal ones. From all these previous observations, we can say that the confidence ellipsoid of Degenne and Perchet, 2016b is tighter as it does not require any axis realignment; however, not e T A µ.

only the matrix Γ is generally looser than Σ * but also axis realignment does not alter the analysis, so that our new approach outperforms theirs in terms of asymptotic regret upper bound.

As common in bandits, the major challenge in the analysis is to prove that with high probability, µ * ∈ C t (A) for any action A ∈ A. The covering argument together with the conversion from an axis-unaligned confidence region into an axis-aligned confidence region allows us to achieve this result (see Lemma 12). Therefore, an optimistic estimate µ t of the true mean µ * can be found using an upper-confidencebound approach: if A t , µ t are defined as in Algorithm 11, then, since µ * ∈ C t (A * ), we have e T At µ t ≥ e T A * µ * . The regret bound for escb-c is stated in Theorem 32, and proven in subsection 7.5.2.

Theorem 32. Assume that the outcome distribution P X satisfies Assumptions 7.1 and 3, and define ∆ min A∈A, ∆(A)>0 ∆(A), ∆ max max A∈A, ∆(A)>0 ∆(A). If ∆ is small enough, i.e., there exists a universal constant c such that

∆ ∨ ∆+∆ log ∆ max ∆ 3/2 ≤ c log(m + 1) i∈[n] max A∈A,i∈A j∈A 0 ∨ Σ * ij n 2 3/2
, then the regret of Algorithm 11 is upper bounded as lim sup

T →∞ ∆ log T R T ≤ c log 2 (m + 1) i∈[n] max A∈A,i∈A j∈A 0 ∨ Σ * ij ,
where c is a universal constant.

Notice that the bound in Theorem 32 is tight, up to a poly-logarithmic factor in m, with respect to the lower bound in Theorem 31, in the case where Σ * has non-negative entries. Moreover, we focus on the asymptotic behavior of the regret (w.r.t. T ) when ∆ is small, i.e., when the problem becomes very difficult. While the quantity c log 2 (m + 1) i∈ [n] max A∈A,i∈A j∈A 0 ∨ Σ * ij log(T )/∆ presented in Theorem 32 highlights the main dependence on both ∆ and T , we prove a more precise non-asymptotic upper bound in (7.12), which holds for all ∆ > 0. Indeed, as for ucb-v, the errors from estimating Σ * generate an extra term in the upper bound. However, since these errors are multiplied with estimation errors on the means, their impact is of second order. In particular, for ∆ small enough, this extra term becomes negligible compared to the main term. Therefore, the term from covariance estimation errors is not present in Theorem 32, but appears when ∆ is far from 0. Finally, remark that when the covariance Σ * is known, then one can consider the confidence region where Σ ij,t is replaced by Σ * ij . This avoids covariance estimation errors, and gives the upper bound of Theorem 32 when ∆+∆ log(∆ max /∆) is smaller than

i∈[n] max A∈A,i∈A j∈A 0 ∨ Σ * ij n • m .
Remark 15. Considering the intersection of the region from Algorithm 11 with the one of cucb-v, we can replace log 2 (m + 1) j∈A 0 ∨ Σ * ij by

(mΣ * ii ) ∧   log 2 (m + 1) j∈A 0 ∨ Σ * ij  
in Theorem 32.

Sparse outcomes

In this section, we shall consider an additional structural assumption on the vector X, namely that it is s-sparse in the sense that

X 0 ≤ s,
i.e., the number of nonzero components of X is smaller than s, where s is a fixed known parameter.3 Importantly, the set of components which are nonzero is not fixed nor known, and may change over time. It should be noted, however, that there is a significant difference between the stochastic and the adversarial cases: In the later, the set of components which are nonzero change arbitrarily over time, whereas in the former, this set is sampled i.i.d. Notice, this sparse stochastic setting is different than the usual stochastic sparse bandit, where µ * is assumed to be sparse; see e.g., [START_REF] Kwon | Sparse Stochastic Bandits[END_REF] for the classical MAB setting, and Abbasi-Yadkori, Pal, and Szepesvari (2012) and [START_REF] Carpentier | Bandit Theory meets Compressed Sensing for high dimensional Stochastic Linear Bandit[END_REF] for the linear bandit setting. For simplicity, we further assume that X ∞ ≤ 1. As we already saw in Proposition 24, this implies Assumption 2 and 3. The difficulty of this setting is that both the approach of Degenne and Perchet (2016b) and standard methods such as cucb-v would not reach the lower bound for the regime s ≤ m, as we will see. The reason is that a correlation exists between the components, because of sparsity, and must be taken into account.

Why sparsity in semi-bandits? Sparsity is nowadays a very standard assumption in learning theory (that potentially does not need any further motivations). There are many examples of online learning scenarios naturally involving some sparse structure. For instance, in the celebrated click-through-rate optimization, it is safe to assume that users would only click on a few of the different ads that can be displayed (those that can catch their eyes for any reason, say). Similarly, in recommender systems, it is safe to assume that a user will browse/buy items from a specific category and not the other (for instance, a segment of the population in e-shops only buy bottles of wines and others only video-games or clothes).

Other examples involve settings where outcomes are usually zero except on very rare occasions: In the online routing, the packets are sent in a network and are lost if a server of that network has a failure. Because of failsafe procedures, failures are desynchronized and typically only one (or at most a few) of them can happen simultaneously. In all of these examples, the decision maker has some combinatorial problem to solve: select an admissible path, select a diverse bundle of object/ads to display, etc., and only a few of the base items will generate non-zero outcome.

Lower bound

To start our study of sparse outcomes, we state a new lower bound in Theorem 33, that is valid for the setting described above. This lower bound is built on the same ideas as Theorem 31, with a notable variation: when reducing to an MAB problem, we do not obtain the necessary conditions for the application of [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF], because of the linear dependence between the µ * i 's. Thus, we use instead the the lower bound from [START_REF] Graves | Asymptotically efficient adaptive choice of control laws incontrolled markov chains[END_REF]. More precisely, we consider the same action space A, and incorporate the sparsity assumption as an extra constraint for defining a worst case distribution.

Theorem 33. For any n, m, s ∈ N * such that n/m, n/s, 1 ∨ (s/m) are integers, n/m, n/s ≥ 2, any ∆ ∈ (0, ms 2(n-m) ] and any consistent policy, there is a problem with n arms -characterized by some action space A with m = max{|A|, A ∈ A} and some vector of outcomes X with all gaps equal to ∆ satisfying X ∞ ≤ 1, X 0 ≤ s -on which the regret satisfies

lim inf T →∞ ∆ log(T ) R T ≥ s(s ∧ m)(1 -2m/n) 4 .
To give an idea of the proof, contrary to Theorem 31, we have more freedom in the covariance, and X can be chosen to maximize V( i∈A X i ) for each action A, up to the constraints X ∞ ≤ 1, X 0 = s. The maximal value of i∈A X i is thus (s ∧ m). Now consider for simplicity the softer constraint E X 0 = s. If X is chosen so that i∈A X i /(s ∧ m) is Bernoulli of parameter p, then the optimal p is equal to (s ∨ m)/n. The variance is about p(s ∧ m) 2 = ms(s ∧ m)/n. Multiplying this by n/m (the number of actions) and dividing by the gap ∆ gives the order of the lower bound.

Proof of Theorem 33. Consider A containing n/m disjoint actions A 1 , . . . , A n/m composed of m arms. X is constructed as follows: (1 ∨ s/m) different actions are ran- domly chosen among A, with equal probability, except the one for action A 1 , that have an offset of δ. From

(1 ∨ s/m) = E A∈A I{A is chosen} = (n/m -1)(P[A 1 is chosen] -δ) + P[A 1 is chosen], we have P[A 1 is chosen] = (1 ∨ s/m)m/n + δ(1 -m/n).
We pose X i = 1 for i spanning the (s ∧ m) first arm of each chosen action (the other components are set to 0). Remark that X is s-sparse with this construction.

This problem reduces to a standard bandit problem with n/m Bernoulli arms. However, we have an additional piece of information, namely that the sum of the means is s. Thus, we can't apply the lower bound from [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF], since the distribution family has not a product form (changing the mean of one arm, we have to make sure that the sum of the means doesn't change, so we have to change at least another mean). Instead, we use the lower bound result from [START_REF] Graves | Asymptotically efficient adaptive choice of control laws incontrolled markov chains[END_REF] (Theorem 2), where we can increase the mean of one arm i while decreasing the mean of the others. Scaling the regret by (s ∧ m) -1 , we want to upper bound KL P 1 (s∧m) i∈A k

X i P 1 (s∧m) i∈A 1 X i = kl m n ∨ s n -δ m n , m n ∨ s n + δ 1 - m n ,
which corresponds to an arm i that becomes a best arm for the new distribution. We also want to upper bound

kl m n ∨ s n -δ m n - δ n m -2 , m n ∨ s n -δ m n ,
which corresponds to the decrease of the mean of each sub-optimal arm k different from i (so that the sum of the mean remain constant). We are going to use the inequality kl(x, y) ≤ (x-y) 2 y(1-y) for all x, y ∈ (0, 1). Since ms

2(n-m) ≥ ∆ = (s ∧ m)δ, we have δ m n ≤ δ(1 -m n ) ≤ m n ∨ s n /2 ≤ 1/4
, and thus

m n ∨ s n -δ m n 1 - m n ∨ s n + δ m n ≥ m n ∨ s n /4, m n ∨ s n + δ 1 - m n 1 - m n ∨ s n -δ 1 - m n ≥ m n ∨ s n /4.
Thus, we get the upper bounds

kl m n ∨ s n -δ m n , m n ∨ s n + δ 1 - m n ≤ 4δ 2 m n ∨ s n (7.4) kl m n ∨ s n -δ m n - δ n m -2 , m n ∨ s n -δ m n ≤ 4δ 2 n m -2 2 m n ∨ s n . (7.5)
From [START_REF] Graves | Asymptotically efficient adaptive choice of control laws incontrolled markov chains[END_REF], we have the lower bound lim inf

T →∞ R T log(T ) ≥ (s ∧ m) inf c n/m k=2 δc k ,
where the above infimum is over all c 2 , . . . , c n/m in R + such that for all i ∈ {2, . . . , n/m},

c i kl m n ∨ s n -δ m n , m n ∨ s n +δ 1- m n + n/m k=2,k =i c k kl m n ∨ s n -δ m n - δ n m -2 , m n ∨ s n -δ m n ≥ 1.
Using the bounds (7.4) and (7.5), we can relax the above constraint as

∀i ∈ {2, . . . , n/m}, c i 4δ 2 m n ∨ s n + n/m k=2,k =i c k 4δ 2 n m -2 2 m n ∨ s n ≥ 1.
By symmetry of the constraint with respect to c i , and by linearity of the objective, there is a maximizer c that satisfies

c 1 = • • • = c n/m = c, with 4cδ 2 1 m n ∨ s n + 1 n m -2 m n ∨ s n = 1.
Thus, since ∆ = (s ∧ m)δ, we get lim inf

T →∞ R T log(T ) ≥ s(s ∧ m)(1 -2m/n) 4∆ .
Notice that we recover the full information case (with a lower bound that equals 0) when n/m = 2, as expected.

Our approach for sparse semi-bandits

In this subsection, we adapt our techniques to the sparse semi-bandit setting. Since X ∞ ≤ 1, the 0 -inequality X 0 ≤ s immediately implies the 1 -inequality X 1 ≤ s. As we will actually only use sparsity through the latter inequality, we can relax our assumption on the model into X 1 ≤ s, for more generality. Let

ν * i E[|X i |],
and ν i,t-1 the corresponding empirical average estimate:

ν i,t-1 u∈[t-1] I{i∈Au}|X i,u | N i t-1 . Our approach is based on replacing j∈A 0 ∨ Σ * ij by ν * i (s ∧ m) (see Lemma 11). Using this, it is possible to estimate ν * i instead of each Σ * ij . Lemma 11. j∈A 0 ∨ Σ * ij ≤ 2ν * i (s ∧ m).
Proof. We use the fact that j∈A |X j | ≤ (s ∧ m). This gives

j∈A 0 ∨ Σ * ij = j∈A 0 ∨ E X i X j -µ * i µ * j ≤ j∈A E[|X i X j |] + µ * i µ * j = E   |X i | j∈A |X j |   + |µ * i | j∈A µ * j ≤ 2E[|X i |](s ∧ m).
We can therefore use the same algorithm (Algorithm 11), but with a confidence region C t independent of A, since summing over A or [n] on the main sum doesn't change the algorithm and the second sum j∈A 0 ∨ Σ ij,t is replaced by an estimates of the upper bound given in Lemma 11.

C t µ t-1 +    ξ ∈ R n , i∈[n] N i t -1ξ 2 i m|ξ i |+2(s∧m)ν i,t ≤ 8(log(t)+log(log(t)))+4em    , (7.6)
where the upper bound estimate ν i,t ν i,t-1 + 1.5 log(t) N i t-1 of ν * i is a simple consequence of Heoffding's inequality, using that |X i,u | is 1/4-sub-Gaussian. Our algorithm is stated in Algorithm 12. As a byproduct of Theorem 32, we provide an upper bound for the regret in the sparse semi-bandit setting in Corollary 3 (see (7.7), for a more

ν * i = E X 1 ≤ s.
Corollary 3. Assume that the outcome distribution P X satisfies X ∞ ≤ 1 and X 1 ≤ s, and that

(∆(s ∧ m)) 2/3 ∨ (m∆ + m∆ log(∆ max /∆)) ≤ c log(m + 1) i∈[n] ν * i (s ∧ m) n ,
for some universal constant c. Then the regret of Algorithm 12 is upper bounded as

lim sup T →∞ ∆ log(T ) R T ≤ c log 2 (m + 1) i∈[n] ν * i (s ∧ m) ≤ c log 2 (m + 1)(s ∧ m)s,
where c is a universal constant.

The corollary is obtained in the same way as Theorem 32. We can underline the difference that we don't have to construct n 2 covariance estimates, but only n (only the ν * i 's). In particular, as these estimates uses 1/4-sub-Gaussian variables, we don't use the sub-exponential concentration of Lemma 10, which removes one term from the previous result. The obtained bound is

R T ≤ ∆ max n + 8nm 2 ∆ 2 + c + c log(m + 1)δ(T )   log(m + 1) i∈[n] ν * i (s ∧ m) ∆ i,min + i∈[n] m 1 + log ∆ i,max ∆ i,min + i∈[n] (s ∧ m) 2/3 ∆ 1/3 i,min   , (7.7)
where c and c are two constants. Notice that to make the first term dominates the others, we must have

n(s ∧ m) 2/3 /∆ 1/3 ∨ (nm(1 + log(∆ max /∆))) ≤ c log(m + 1) i∈[n] ν * i (s ∧ m) ∆ ,
for some constant c, which gives our condition in Corollary 3.

Remark 16. It should be noticed that semi-bandits algorithms as cucb-v or cucbkl (that are variant of the classical cucb (Kveton, Wen, Ashkan, and Szepesvari, 2015b), where the confidence region is a Cartesian product of confidence intervals, with Bernstein and kl-base confidence intervals respectively) also reach the lower bound of Theorem 33 for the regime s ≥ m, since V(X i ) ≤ 2ν * i (thanks to Lemma 11). However, in the regime where s ≤ m, these algorithms are not able to reach it, while escb-c is. In the following, we describe the two algorithms cucb-v and cucb-kl, and comment further on the tightness difference between confidence regions.

Confidence regions comparison

We give here the two algorithms cucb-v and cucb-kl, which, as we have seen, also matches the lower bound given to the Theorem 33, in the specific regime where s ≥ m. Both the two algorithms rely on the same optimization A t = arg max A∈A e T A µ t , where the vector µ t is defined for cucb-v as

∀i ∈ [n], µ i,t 1 ∧   µ i,t-1 + 2ζσ 2 i,t-1 log(t) N i,t-1 + 3ζ log(t) N i,t-1   ,
where

σ 2 i,t-1 t ∈[t-1] I{i = i t } X i,t -µ i,t-1 2 N i,t-1 ,
and for cucb-kl as ∀i ∈ [n], µ i,t is the unique solution x to

N i,t-1 kl µ i,t-1 , x = ζ log(t) such that x ∈ [µ i,t-1 , 1].
We take ζ = 1.2 (although all ζ > 1 are valid). The algorithms above can also be seen as a bilinear maximization where µ t is maximized over a confidence region that is a Cartesian product one 1-demendional confidence intervals. We illustrate in Figure 7.1 the difference between the confidence region considered in escb-c (when the correlation is low) and cucb-kl. The red points represent µ t for each region. It can be seen that the Cartesian product confidence region greatly overestimates the risk in directions that are not close to the axes, giving rise to over-exploration. It is important to note however that this price to pay can be interesting in practice, because the corresponding algorithms are then very efficient (LP over A, supposed possible4 ). As we noted in Remark 15, considering the intersection between the two confidence regions gives rise to an even tighter region, and therefore a better regret bound.

Experiments and discussion

We consider the following dynamic assortment problem. An agent has n products to sale, with fixed known prices. At each round, a customer arrives, with some unknown random valuation vector over products. Then, the agent offers any subset of products, by paying a fixed known cost for each offered product (e.g. transportation and display cost), and the customer buys an offered product if and only if its valuation is greater than its price. The agent is interested in maximizing the total profit (revenue minus cost) from sales over T rounds. We use the n = 120 products from the Kaggle (2013) dataset containing 7500 grocery store transactions. At each round, valuations are determined by sampling a random transaction from this dataset. The choice of such data is motivated by correlations that exist between arms, as illustrated in Figure 7.2 -left, representing the correlation matrix. We ran 36 independent simulations with T = 10 4 , and with a common product price and cost respectively equal to 1.5 and 0.1. We compared cucb-v and cucb-kl with the Lovász extension implementation of escb-c (see subsection 7.5.1) and results are plotted in log-scale (Figure 7.2right); error bars represent the sample standard deviation over simulations. There is less volatility in the regret of cucb-v and cucb-kl; this is due to the fact that their confidence regions overestimate the risk, and the "bad" event where the regret deviates is almost negligible. Nevertheless, we clearly observe that escb-c outperforms the two other approaches in terms of the average regret. Finally, let us point out that we did not empirically compare to the ols-ucb algorithm of Degenne and Perchet (2016b) since it is inefficient to implement (the combinatorial problem to be solved within each round is NP-Hard in general [START_REF] Atamtürk | Maximizing a class of utility functions over the vertices of a polytope[END_REF]. We noticed that for the choice of sub-Gaussianity matrix where all the correlation coefficients equals 1, ols-ucb (if it could be implemented) would return a solution very close to cucb-v.

Discussion

We improved the analysis of combinatorial semi-bandits in multiple ways. First, we brought new perspectives by considering a fairly large family of sub-exponential probability distributions, that crucially do not depend on parameters difficult to obtain in real situations. We have built an algorithm for this family, based on the estimation of the covariance matrix. We have therefore already significantly improved existing approaches by adapting not only to the variance of the arms, but also to the correlation between them. A tight analysis of our proposed method gives a new stateof-the-art upper bound on the regret. Our new bound is also more intuitive, and is more relevant to reflect the complexity of the instance at hand (through correlations between arms). Finally, we applied our approach to a setting not yet studied before, that assumes sparsity of the outcome vector. We gave a lower bound, as well as a matching algorithm that leverages the sparsity assumption.

Appendix

Implementation using the Lovász extension

We now discuss the computational efficiency of our approaches. First, Algorithm 11 (and both those of [START_REF] Combes | Combinatorial Bandits Revisited[END_REF] and Degenne and Perchet (2016b)) is not efficient for arbitrary combinatorial space A. However, the evaluation of F : A → max µ∈Ct(A) e T A µ, can be done efficiently as it is an LP over a convex set. In practice, when A allows it, greedy 5 [START_REF] Nemhauser | An analysis of approximations for maximizing submodular set functions-I[END_REF] can be used to maximize F . In general, it is unknown if this alters the regret rate. On the one hand, it does not when A is given by a matroid, and C t is as in Algorithm 12. This is because F is submodular and the following approximation guaranty holds for the output A t of greedy (see Chapter 5): 2 F (A t ) -e T At µ t-1 + e T At µ t-1 ≥ F (A * ), where the l.h.s. is simply F where C t is scaled by a factor 2 from its center µ t-1 . On the other hand, when C t (A) is as in Algorithm 11, a concave extension of A → F (A) can be considered, and can thus be maximized efficiently. Notice, when considering the intersection of the two confidence regions as in Remark 15, this implementation is still tractable since the minimum of two concave functions is still concave. Since the obtained solution might not be fractional, we use a randomized rounding to obtain a feasible set A t ∈ A = {0, 1} n . We provide in the following further details and prove that this method scales the regret by a factor 1 + log m log(T ) ∆ 2 , an acceptable price for efficiency.

From the step 1 of the proof of Theorem 32, we have that

e T At µ t ≤ e T At µ t-1 + 2 δ(t) i∈At j∈At 0 ∨ Σ ij,t N i t -1 + 4mδ(T ) i∈At 1 N 2 i,t-1 .
5 Starting from A = ∅, we sequentially add (when possible) the best possible i to the current A if F (A ∪ {i}) > F (A).

Since the final bound of Theorem 32 relies on the above upper bound, in Algorithm 11, instead of maximizing A → max µ∈Ct(A) e T A µ, we can maximize

A → e T A µ t-1 + 2 δ(t) i∈A j∈A 0 ∨ Σ ij,t N i t -1 + 4mδ(T ) i∈A 1 N 2 i,t-1
• Our goal here is to provide a continuous extension of the above set function that is concave on [0, 1] n , and thus efficient to maximize. The linear term trivially extends to the linear function x → x T µ t-1 . The last two term can be extended relying on the Lovász extension [START_REF] Lovász | Submodular functions and convexity[END_REF]. We recall that the Lovász extension of a set function f is defined as f

L (x) E[f ({i ∈ [n], x i ≥ U })],
where the expectation is over U ∼ U [0, 1]. The Lovász extension is concave if and only if f is a supermodular function [START_REF] Lovász | Submodular functions and convexity[END_REF], i.e.,

f (A) + f (B) ≤ f (A ∪ B) + f (A ∩ B) ∀A, B ⊂ [n].
It is easy to check that a function G : A → i∈A j∈A a ij is supermodular for a ij ≥ 0, so its Lovász extension is concave. Composing by the square root, we thus have a concave extension of the second and last term.

After the maximization of the extension, a continuous maximizer x t is returned, and the agent plays

A t = {i ∈ [n], x i,t ≥ U } where U ∼ U [0, 1]. Let σ t be a per- mutation such that x σt(1),t ≥ . . . , ≥ x σt(n),t .
Then, the set S j = {σ t (1), . . . , σ t (j)} is chosen with probability p j,t = x σt(j) -x σt(j+1) (with the convention x σt(n+1) = 0). The continuous extension evaluated at x t is of the form j p j,t e T S j µ t-1 + j p j,t G 1 (S j ) + j p j,t G 2 (S j ),

where G 1 and G 2 are the supermodular functions corresponding to the second and last term respectively. Then, using an 2 upper bound on G i , i ∈ {1, 2} the probabilities p j can be aggregated to build triggering probabilities. Since the triggering probabilities are inside the square root in the above bonus, we can apply Theorem 16. This gives a regret bound with an extra factor of 1 + log m log(T ) ∆ 2

. Notice, an application of Theorem 15 is also possible, but the extra factor would depend on p and would be difficult to bound.

Remark 17. We addressed the efficiency by using randomization. Notice, this randomization is an example of "undergone randomization", since the selected action S is actually x, that well describe the distribution used to generate the played set of arms. Saying it differently, notice that the initial setting was into the CMAB framework, whereas we used triggering probabilities to solve it. The fact that randomization has been transformed into triggering probabilities illustrates that the action of interest is not the random set played but the vector x which is the result of the continuous optimization of the objective considered above.

In Chapter 8, we will see an example where the opposite happens: the randomization is controlled by the agent. More precisely, this policy will be based on Thompson sampling, which is an example where the chosen action is the one of interest, because it is the solution of a maximization problem where the parameters are random.

Proof of Theorem 32

Proof of Theorem 32. Let t ≥ 1, and δ(t) 2(log(t) + log(log(t))) + em. Through initialization, we can assume N ij,t-1 ≥ 1 for all i, j ∈ [n] (as this only adds n(n -1)∆ max /2 to the regret bound). We will decompose contributions to regret by considering the following events:

C t µ * ∨ µ t-1 ∈ C t (A * ) , D t e T At (µ t-1 -µ * ) ≤ ∆(A t )/2 , S t ∀i, j ∈ [n], 0 ≤ Σ ij,t -Σ * ij ≤ 2g ij (t) . We also define g ij (t) 16   3 log(t) N 2 ij,t-1 ∨ 3 log(t) N 3 ij,t-1   + 48 log 2 (t) N 4 ij,t-1 + 36 log 2 (t) N 4 ij,t-1 , ∀i ∈ [n], ∆ i,min min A∈A, i∈A, ∆(A)>0 ∆(A), ∆ i,max max A∈A, i∈A, ∆(A)>0 ∆(A), and ∀i, j ∈ [n], ∆ ij,min min A∈A, i,j∈A, ∆(A)>0 ∆(A), ∆ ij,max max A∈A, i,j∈A, ∆(A)>0 ∆(A).
Step 1: If C t , D t and S t hold We have

∆(A t ) = (e A * -e At ) T µ * ≤ e T A * µ * ∨ µ t-1 -e T At µ t + e T At (µ t -µ * ) ≤ e T At (µ t -µ * ) C t ≤ ∆(A t )/2 + e T At (µ t -µ t-1 ) D t
i.e., using Cauchy-Schwarz and µ t ∈ C t (A t ),

∆(A t ) ≤ 2e T At (µ t -µ t-1 ) ≤ 2 i∈At 4 j∈At 0 ∨ Σ ij,t + m µ i,t -µ i,t-1 δ(t) N i,t-1 ≤ 4 δ(t) i∈At   j∈At 0 ∨ Σ ij,t N i,t-1 + m µ i,t -µ i,t-1 min j∈At N j,t-1   .
Solving the corresponding quadratic inequation in the variable x = e T At (µ t -µ t-1 ), we get

∆(A t ) ≤ 2e T At (µ t -µ t-1 ) ≤ 4   δ(t) 2 m 2 min i∈At N 2 i,t-1 + i∈At δ(t) j∈At 0 ∨ Σ ij,t N i,t-1 + mδ(t) min j∈At N j,t-1   ≤ 4 δ(t) i∈At j∈At 0 ∨ Σ ij,t N i,t-1 + 8mδ(t) min j∈At N j,t-1 ≤ 4 δ(t) i∈At j∈At 0 ∨ Σ * ij + 2g ij (t) N i,t-1 + 8mδ(t) min j∈At N j,t-1 S t ≤ 4 δ(t) i∈At j∈At 0 ∨ Σ * ij N i,t-1 + 4 δ(t) i∈At j∈At 2g ij (t) N i,t-1 + 8mδ(t) min j∈At N j,t-1 ≤ 4 δ(T ) i∈At max A∈A,i∈A j∈A 0 ∨ Σ * ij N i,t-1 (7.8) + 4 δ(T ) i,j∈At g ij (T ) (7.9) + 8mδ(T ) min j∈At N j,t-1 (7.10) .
Where the last inequality uses that N

i,t-1 ∧ N j,t-1 ≥ N ij,t-1 ∀i, j ∈ [n].
From this point, we treat each term separately, using the relation I{∆(A t ) ≤ (7.8) + (7.9) + (7.10)} ≤ I{∆(A t )/3 ≤ (7.8)} + I{∆(A t )/3 ≤ (7.9)} + I{∆(A t )/3 ≤ (7.10)}.

We apply Theorem 13, that is helpful to bound the regret on each of this 3 events. Indeed, for the first term, applying it with β i,T = 12 2 δ(T ) max A∈A,i∈A j∈A 0 ∨ Σ * ij and α i = 1/2 gives the bound

T t=1 I{∆(A t )/3 ≤ (7.8)}∆(A t ) ≤ 4608 log 2 2 (4 √ m) i∈[n] δ(T ) max A∈A,i∈A j∈A 0 ∨ Σ * ij ∆ i,min .
The second term can be itself decomposed into two terms, bounding the max by the sum and using log(T ) ≤ δ(T ).

(7.9) ≤ 4δ(T )

i,j∈At 54 + √ 48 N -2 ij,t-1 + 4δ(T ) 0.75 16 √ 3 i,j∈At N -1.5 ij,t-1 .
Thus, again, it is sufficient to treat each term separately. We also apply Theorem 13, but with [n] 2 as the set of arms, and with A 2 t as the set of played arms, taking respectively α i = 1, β i,T = 24 54 + √ 48δ(T ) and α i = 0.75, β i,T = 192 • 6 2/3 δ(T ) for each term. This gives

T t=1 I{∆(A t )/3 ≤ (7.9)}∆(A t ) ≤ 1152 √ 6 log 2 (4m) i,j∈[n] δ(T ) 1 + log ∆ ij,max ∆ ij,min + 12288 • 6 2/3 4 1/3 -1 -1 log 2 (4m) i,j∈[n] δ(T )∆ -1/3 ij,min .
The last term can be analyzed in the same way by first upper bounding it as

(7.10) ≤ 8mδ(T ) i∈At 1 N 2 i,t-1 . Then, taking α i = 1, β i,T = 24mδ(T ) in Theorem 13 gives T t=1 I{∆(A t )/3 ≤ (7.10)}∆(A t ) ≤ 1152 log 2 (4 √ m) i∈[n] mδ(T ) 1 + log ∆ i,max ∆ i,min .
This concludes step 1; notice that all subsequent steps will aim to bound the regret by a term independent of T , over a certain event. Thus, we can see that the bounds above are the actual contributions to the rate of the regret. To show Theorem 32, we must therefore choose the regime for ∆ ≤ ∆ i,min so that the first term prevails over the others. In other words, we want to have

n 2 ∆ -1/3 ∨ (1 + log(∆ max /∆)) ≤ c log(m + 1) i∈[n] max A∈A,i∈A j∈A 0 ∨ Σ * ij ∆ ,
where c is a constant. This gives exactly our condition in Theorem 32.

Step

2: If S t ,¬C t hold Let σ 2 i j∈A * 0 ∨ Σ * ij for all arms i ∈ [n].
We fixe some δ ≥ e • m, and define the following events:

A t      i∈A * I µ * i ≥ µ i,t-1 N i,t-1 µ * i -µ i,t-1 2 4 σ 2 i + |A * | µ * i -µ i,t-1 ≥ δ      ∀d ∈ (N * ) A * , B d,t i∈A * δ δ -1 d i -1 ≤ N i,t-1 < δ δ -1 d i .
Notice that S t , ¬C t implies A t for δ = δ(t). Since each number of pulls N i,t-1 for i ∈ A * is bounded by t, the number of possible d ∈ (N * ) A * such that P[B d,t ] > 0 is bounded by (log(t)/ log(δ/(δ -1))) m . Thanks to the following Lemma 12, and an union bound on those possible d ∈ (N * ) A * , we get

P[A t ] ≤ e m+1 (δ -1) log(t) m log(δ/(δ -1)) m e -δ ,
so the regret under this event is bounded by a universal constant, since the upper bound above is the term of a convergent series for δ = δ(t). Indeed, it rewrites as

t -2 e m+1-em       2 -log -1 (t) m ≤2/m + 2 log(log(t)) log(t) + e log -1 (t) ≤2e e/2-1       m , that is bounded by t -2 e • e 1-e • 2e e/2-1 m ≤1 e 1-e/2 m + 1 m ≤e e 1-e/2
. Lemma 12 (Covering-argument). Let d ∈ (N * ) A * . Then,

P[A t ∩ B d,t ] ≤ (δ -1)e m m e 1-δ .
Proof. We rely on a covering argument. The idea is to get rid of randomness by replacing the empirical mean µ i,t-1 by some non-random value

x i . Let ζ ∈ R A * + . For i ∈ A * , we define x i (N ) for N ∈ R + as the unique solution x ∈ (-∞, µ * i ] of the equation N (µ * i -x) 2 4(σ 2 i +|A * |(µ * i -x)) = ζ i . Notice that for all i ∈ A * , x i is non-decreasing since x → (µ * i -x) 2 4(σ 2 i +|A * |(µ * i -x)) is decreasing on (-∞, µ * i ]. The event i∈A *      N i,t-1 0 ∨ µ * i -µ i,t-1 2 4 σ 2 i + |A * | µ * i -µ i,t-1 > ζ i      implies i∈A * µ i,t-1 ≤ x i (N i,t-1 ) .
Under the event B d,t , this implies

i∈A * µ i,t-1 ≤ x i δ δ -1 d i . (7.11) With i µ * i -x i δ δ-1 d i and λ i i 2(σ 2 i +|A * |ε i ) , i ∈ A * , this further implies: δ δ -1 -1 i∈A * ζ i = i∈A * δ δ -1 d i -1 2 i 4(σ 2 i + |A * |ε i ) x i (e d i ) > -∞, ≤ i∈A * N i,t-1 2 i 4(σ 2 i + |A * |ε i ) B d,t = i∈A * N i,t-1 2 i 2(σ 2 i + |A * |ε i ) - i∈A * N i,t-1 2 i 4(σ 2 i + |A * |ε i ) ≤ i∈A * N i,t-1 2 i 2(σ 2 i + |A * |ε i ) - i∈A * N i,t-1 σ 2 i ε 2 i 4(σ 2 i + |A * |ε i ) 2 σ 2 i σ 2 i + |A * |ε i ≤ 1, = i∈A * N i,t-1 λ i ε i - i∈A * N i,t-1 σ 2 i λ 2 i ≤ i∈A * N i,t-1 λ i µ * i -µ i,t-1 - i∈A * N i,t-1 σ 2 i λ 2 i using (7.11),
and this last quantity is equal to

u∈[t-1] (λ e Au∩A * ) T (µ * -X u ) -(λ e Au∩A * ) T D(λ e Au∩A * ) ,
where D is the diagonal matrix with D ii = σ 2 i for all i ∈ [n]. For all u ∈ [t -1], since λ ≥ 0, we can write the following axis-realignment inequality

(λ e Au∩A * ) T Σ * (λ e Au∩A * ) = i∈Au∩A * j∈Au∩A * Σ * ij λ i λ j ≤ i∈Au∩A * j∈Au∩A * 0 ∨ Σ * ij 2 λ 2 i + λ 2 j = i∈Au∩A *   j∈Au∩A * 0 ∨ Σ * ij   λ 2 i ≤ (λ e Au∩A * ) T D(λ e Au∩A * ).
Thus, we have

δ δ -1 -1 i∈A * ζ i ≤ u∈[t-1] (λ e Au∩A * ) T (µ * -X u ) -(λ e Au∩A * ) T Σ * (λ e Au∩A * ) ≤ u∈[t-1] (λ e Au∩A * ) T (µ * -X u ) -log E e (λ e Au∩A * ) T (X-µ * )
, where the last inequality uses Assumption 3 and λ e Au∩A * 1 ≤ 1/2. Now, notice that

E   exp   u∈[t-1] (λ e Au∩A * ) T (µ * -X u ) -log E e (λ e Au∩A * ) T (X-µ * )     equals E     u∈[t-1] e (λ e Au∩A * ) T (µ * -Xu) E e (λ e Au∩A * ) T (X-µ * )     = u∈[t-1] E     e (λ e Au∩A * ) T (µ * -Xu) E e (λ e Au∩A * ) T (X-µ * )     = 1,
so from Markov inequality, we get the following bound:

P   u∈[t-1] (λ e Au∩A * ) T (µ * -X u ) -log E e (λ e Au∩A * ) T (X-µ * ) ≥ e -1 i∈A * ζ i   ≤ e -i∈A * ζ i ( δ δ-1 ) -1
, thus, we showed that

P   B d,t ∩ i∈A *      N i,t-1 0 ∨ µ * i -µ i,t-1 2 4 σ 2 i + |A * | µ * i -µ i,t-1 > ζ i         ≤ e -i∈A * ζ i ( δ δ-1 ) -1 , i.e. P    i∈A *      I B d,t N i,t-1 0 ∨ µ * i -µ i,t-1 2 4 σ 2 i + |A * | µ * i -µ i,t-1 > ζ i         ≤ e -i∈A * ζ i e -1 ,
By Lemma 8 of [START_REF] Magureanu | Lipschitz Bandits: Regret Lower Bound and Optimal Algorithms[END_REF], since δ ≥ em, we have

P[B d,t ∩ A t ] = P   B d,t ∩      i∈A * N i,t-1 0 ∨ µ * i -µ i,t-1 2 4 σ 2 i + |A * | µ * i -µ i,t-1 ≥ δ         = P    i∈A * I B d,t N i,t-1 0 ∨ µ * i -µ i,t-1 2 4 σ 2 i + |A * | µ * i -µ i,t-1 ≥ δ    ≤ (δ -1)e m m e 1-δ .
Step 3: If ¬D t hold The regret under this event can be bounded by 8nm 2 ∆ max /∆ 2 using Proposition 14.

Step 4: If ¬S t hold From Proposition 25, the regret under this event is bounded by a universal constant.

Putting it all together Finally, we have shown that there exists two universal constant c, c satisfying the following (we display the scaled back (by κ) version of the regret bound to get the dependence into κ) the expected reward between an optimal super arm and any non-optimal super arm, and where m max A∈A |A|.

R T ≤ ∆ max n(n -1) 2 + 8nm 2 ∆ 2 + c + c log(m + 1)δ(T )   log(m + 1) i∈[n] max A∈A,i∈A j∈A 0 ∨ Σ * ij ∆ i,min + i,j∈[n] κ 1 + log ∆ ij,max ∆ ij,min + i∈[n] mκ 1 + log ∆ i,max ∆ i,min + i,j∈[n] κ 4/3 ∆ 1/3 ij,min   . (7.12) (i) (ii) (iii) cucb m m m escb * log 2 (m) m log 2 (m) cts-beta log 2 (m) - - cts-gaussian log 2 (m) mlog 2 (m) log 2 (m) clip cts-gaussian log 2 (m) m log 2 (m) Lower bound Ω(1) Ω(m) Ω(1)
Many CMAB policies are based on the Upper Confidence Bound (UCB) approach, extending the classical ucb policy [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF] from MAB to CMAB. This type of approach uses an optimistic estimate µ t of µ * (i.e., for which the reward function is overestimated), lying in a well-chosen confidence region. For setting (ii), there exist UCB-style policies that match the lower bound mentioned above. An example of such policy is Combinatorial Upper Confidence Bound (cucb) [START_REF] Chen | Combinatorial Multi-Armed Bandit: General Framework and Applications[END_REF]Kveton, Wen, Ashkan, and Szepesvari, 2015b), that uses a Cartesian product of the individual confidence intervals of each arm as a confidence region. For setting (i), [START_REF] Combes | Combinatorial Bandits Revisited[END_REF] provided the UCB-style policy Efficient Sampling for Combinatorial Bandit (escb), that uses the assumption of mutual independence between arm distributions in order to build a tighter ellipsoidal confidence region around the empirical mean, which helps to better restrict the exploration. Degenne and Perchet (2016b) gave the following generalization of setting (i):

(iii) The joint probability P X is C-sub-Gaussian, for a positive semi-definite matrix C 0, i.e., E e λ T (X-µ * ) ≤ e λ T Cλ/2 , for all λ ∈ R n .

In this case, they provided the policy ols-ucb, leveraging this additional assumption and such that it essentially reduces to escb in the specific case of diagonal matrix C with a regret bound of O log 2 (m)n log(T )/∆ ) (so it matches the above lower bound up to a polylogarithmic factor in m). We refer the reader to Table 8.1 for an overview of the above regret (lower) bounds.

In some CMAB problems, the action space A and the reward function are simple enough for the existence of an exact oracle that takes as input a vector µ ∈ R n and outputs the solution of the combinatorial problem (associated to the mean vector µ), with a polynomial time complexity O(poly(n)). Under this assumption (referred to as Assumption 4), cucb, that plays the action A t = Oracle(µ t ) at round t, is efficient to implement, and has a O(poly(n)) time complexity per round. In that case, the setting (ii) is therefore essentially solved. On the other hand, this is not true for the settings (i) and (iii), as escb needs to solve a difficult combinatorial problem in each round (NP-Hard in general [START_REF] Atamtürk | Maximizing a class of utility functions over the vertices of a polytope[END_REF]).

The inefficiency of escb triggered some attempts to implement an efficient version: Perrault, Perchet, and Valko (2019a) proposed an efficient approximation method for implementing escb in the case the action space has a matroid structure: they prove a time complexity of O(poly(n)) while keeping the same regret rate. However, this improvement is mitigated by the fact that cucb reaches the optimal regret rate O(n log(T )/∆) for the special case of matroid semi-bandits [START_REF] Anantharam | Asymptotically efficient allocation rules for the multiarmed bandit problem with multiple plays-Part I: IID rewards[END_REF][START_REF] Kveton | Matroid bandits: Fast combinatorial optimization with learning[END_REF][START_REF] Talebi | An Optimal Algorithm for Stochastic Matroid Bandit Optimization[END_REF]. Recently, [START_REF] Cuvelier | Statistically Efficient, Polynomial Time Algorithms for Combinatorial Semi Bandits[END_REF] provided another approach for approximating escb for a wide variety of action spaces, including the matching bandit setting [START_REF] Gai | Learning multiuser channel allocations in cognitive radio networks: A combinatorial multi-armed bandit formulation[END_REF] and the online shortest path problem [START_REF] Liu | Adaptive shortest-path routing under unknown and stochastically varying link states[END_REF], where cucb is not known to be better than escb. However, their policies are still computationally expensive when T is large, since the time complexity at round t is of order O(t • poly(n)).

Another line of research is to find an efficient alternative to escb. One of the most promising candidate is Thompson Sampling (ts). Although introduced much earlier by [START_REF] Thompson | On the likelihood that one unknown probability exceeds another in view of the evidence of two samples[END_REF], the theoretical analysis of ts for frequentist MAB is quite recent: [START_REF] Kaufmann | Thompson Sampling: An Asymptotically Optimal Finite Time Analysis[END_REF] and [START_REF] Agrawal | Analysis of Thompson sampling for the multi-armed bandit problem[END_REF] gave a regret bound matching the ucb policy theoretically. Moreover, ts often performs better than ucb in practice, making ts an attractive policy for further investigations. For CMAB, ts extends to Combinatorial Thompson Sampling (cts). In cts, the unknown mean µ * is associated with a belief (a prior distribution) updated to a posterior with the Bayes'rule, each time a feedback is received. In order to choose an action at round t, cts draws a sample θ t from the current belief, and plays the action given by Oracle(θ t ). cts is an attractive policy because its time complexity is O(poly(n)) under Assumption 4. Recently, for the setting (i) with bounded outcomes, [START_REF] Wang | Thompson Sampling for Combinatorial Semi-Bandits[END_REF] proposed an analysis of cts-beta, which is cts where the prior distribution is chosen to be a product of n Beta distributions. They proved two regret upper bounds depending on the class of reward functions: Although the aforementioned upper bound in the linear reward case outperforms the one of cucb, it doesn't match the one of escb. To summarize, and despite many efforts, the existence of a policy that is both optimal (up to a polylogarithmic factor in m) and efficient in the setting (i) or (iii) is still an open problem, which we tackle here.

Further related work

We refer the reader to [START_REF] Wang | Thompson Sampling for Combinatorial Semi-Bandits[END_REF] for further related work on ts for combinatorial bandits, and particularly for [START_REF] Gopalan | Thompson sampling for complex bandit problems[END_REF], that provided a frequentist high-probability regret bounds for ts with a general action space and a general feedback model - [START_REF] Komiyama | Optimal Regret Analysis of Thompson Sampling in Stochastic Multi-armed Bandit Problem with Multiple Plays[END_REF], that investigated ts for the m-sets action space -Wen, Kveton, and Ashkan (2015), that studied ts for contextual CMAB problems, using the Bayesian regret metric (see also [START_REF] Russo | An information-theoretic analysis of thompson sampling[END_REF]).

Contributions

We first improve the result of [START_REF] Wang | Thompson Sampling for Combinatorial Semi-Bandits[END_REF] by providing the regret upper bound O log 2 (m)n log(T )/∆ for cts-beta in the setting (i) with bounded outcomes. This bound is valid even for non linear reward functions. Our main contribution is a regret bound for the setting (iii). We propose an efficient policy called cts-gaussian, that is cts where the prior distribution is chosen to be a multivariate Gaussian. An analysis of cts-gaussian allows us to obtain a regret bound reducing to O log 2 (m)n log(T )/∆ for a diagonal sub-Gaussianity matrix. When the reward function is linear, we generalize the setting (iii) assuming only λ ∈ R n + . This allows us to get rid of negative correlations between the outcomes, and focus on positive correlations. We propose in this setting the policy clip cts-gaussian, where the score is truncated from below with the empirical mean, and from above with the UCB. Truncations from above are not necessary, but can limit optimism, especially when positive correlations are significant. We obtain an improved regret bound for clip cts-gaussian, where negative correlations no longer appear in the regret bound and where, in setting (ii), the extra log 2 (m) factor present in the regret bound of cts-gaussian disappears. All these results are summarized and compared to other state-of-the-art policies in Table 8.1.

Model

The CMAB problem we consider is formally introduced as follows. At each round t ∈ [T ], the agent chooses a super arm (or action) A t ∈ A ⊂ P([n]) based on the history of observations H t σ X 1 e A 1 , . . . , X t-1 e A t-1 and a possible extra source of randomness (as usual, we denote by F t the filtration containing H t and the extra randomness of round t -in particular, A t ∈ F t ). The feedback received is then X t e At and the associated expected reward of the agent at that stage is r(A t ; µ * ), for some known function r. The objective of the agent is to minimize the regret, defined for a policy π as

∀T ≥ 1, R T (π) E T t=1 ∆ t , where ∆ t ∆(A t ) r(A * ; µ * ) -r(A t ; µ * ) with A * ∈ arg max A ∈A r(A ; µ * ).
As stated in the introduction, we will assume the following: Assumption 4. The agent has access to an oracle with a time complexity O(poly(n)) such that for any mean vector µ, Oracle(µ) ∈ arg max A∈A r(A; µ).

As in Chen, Wang, and Yuan (2016), we assume that the function r satisfies the following smoothness property. Assumption 5. There exists a constant B, such that for every super arm A ∈ A and every pair of mean vectors µ and µ , |r(A; µ) -r(A; µ )| ≤ B e A (µ -µ ) 1 .

For an arm i ∈ [n], we define the number of time i has been chosen at the beginning of round t as N

i,t-1 t ∈[t-1] I{i ∈ A t }.
We also define the minimum size of an optimal action m * min A∈arg max A ∈A e T A µ * |A|.

The independent case: Thompson sampling with beta prior

In this section, we consider the following assumption on top of the CMAB setting from section 8.1.2. Assumption 6. The outcomes X i are bounded (in [0, 1], w.l.o.g.), and are mutually independent (we are thus in a special case of (i)).

i ← b i + 1 -Y i,t .
For this problem, we consider cts-beta in Algorithm 13, which is described as follows. The prior is set to be a product of n beta distributions (being thus uniform over [0, 1] initially). Notice, this prior is conjugate to a product of Bernoulli distributions. After the agent get an observation X i,t , it first binarizes it by sampling Y i,t ∼ Bernoulli(X i,t ) (the regret of the problem defined by the observations Y i,t is the same because E[Y i,t ] = µ * i ). Then the prior is updated using Bayes' rule with each sample Y i,t . When choosing a super arm at round t, the agent draws θ t from the beta belief, and then plugged it into the oracle, which outputs the super arm A t to play.

The main result of this section is Theorem 34, that improves the regret bound of Wang and Chen (2018) for cts-beta.

Theorem 34. The policy π described in Algorithm 13 has regret R T (π) of order

O   i∈[n] B 2 log 2 (m) log(T ) ∆ i,min   .
The proof of Theorem 34, as well as the complete non-asymptotic upper-bound is postponed to subsection 8.5.1. Our analysis incorporates two novelties that we detail in the two following paragraphs.

An improved leading term (cf.

Step 3 of the proof of Theorem 34 in subsection 8.5.1) We define the empirical average of each arm i ∈ [n] at the beginning of round t as µ

i,t-1 t ∈[t-1] I{i∈A t }Y i,t N i,t-1
. Notice that this empirical average definition differs from the one that is classically used in CMAB, since samples Y i,t are used rather than X i,t . The improved dependence in m in the leading term of Theorem 34 (compared to Equation 8.1) is a consequence of two ingredients. The first is the following concentration inequality (see Lemma 14), which improves that of [START_REF] Wang | Thompson Sampling for Combinatorial Semi-Bandits[END_REF] by extending it to the case of non-linear reward. Indeed, we rather control the 1 norm in this case, instead of the ∞ -norm, which leads to a tighter bound.

P   e At (θ t -µ t-1 ) 1 ≥ 1 2 log(|A|2 m T ) i∈At 1 N i,t-1 H t   ≤ 1/T . (8.2)
The second ingredient is a more careful handling of the square-root term in the above probability, based on a method similar to the one in Degenne and Perchet (2016b).

T -independent term (cf.

Step 4 of the proof of Theorem 34 in subsection 8.5.1) Similarly to [START_REF] Wang | Thompson Sampling for Combinatorial Semi-Bandits[END_REF], our regret bound also contains an exponential term that is constant in T . Note, however that the term of Wang and Chen ( 2018) is of order O(ε -2m * -2 ), whereas ours is of order O ε -4m * -2 , where ε ∈ (0, 1) is of order ∆ min /(m * ) 2 . This discrepancy is due to the correction of a minor negligence inaccuracy in their Lemma 7, where they assume, at the end of the proof, that one could decorrelate the counters from the outcomes received. We manage to circumvent this issue by doing a careful union bound over the counters. It is this union bound that brings a larger dependence in this constant term.

The general case: Thompson sampling with Gaussian prior

Algorithm 14 cts-gaussian Input: The vector D, and a parameter β > 1.

Initialization: Play each arm once (if the agent knows that µ * ∈ [a, b] n , this might be skipped) For every subsequent round t:

Draw θ t ∼ ⊗ i∈[n] N µ i,t-1 , N -1 i,t-1 βD i (θ i,t ∼ U [a, b] if N i,t-1 = 0). Play A t = Oracle(θ t ).
Get the observation X t e At , and update µ t-1 and counters accordingly.

In this section, we consider the setting from section 8.1.2, with a more general sub-Gaussian family for X ∈ R n . More precisely, we make the following similar assumption as in Degenne and Perchet (2016b). Proposition 26 gives two examples included in this assumption. Proof. Assumption 7 encompasses κ 2 i -sub Gaussian outcomes with D i = κ 2 i m for all i ∈ [n]. Indeed, let λ = λ e A for some action A and observe that E e λ T (X-µ * ) is bounded by

E i |κ i λ i | κ λ 1 e κ λ 1 sign(λ i ) X i -µ * i κ i ≤ e κ λ 2 1 /2 ≤ e κ λ 2 2 |A|/2 ≤ e κ λ 2 2 m/2 .
The case of C-sub-Gaussian outcomes with a known sub-Gaussianity matrix C (i.e., E e λ T (X-µ * ) ≤ e λ T Cλ/2 for all λ ∈ R n ) is also captured, taking 2

D i = max A∈A, i∈A j∈A |C ij |.
Indeed, for an action A, i,j∈A

λ i λ j C ij ≤ i,j∈A λ 2 i + λ 2 j 2 |C ij | = i∈A λ 2 i j∈A |C ij | ≤ i∈n λ 2 i max A∈A, i∈A j∈A |C ij |.
For the above setting, we provide cts-gaussian in Algorithm 14, where we define the empirical mean of arm i at round t ≥ 1 as µ

i,t-1 t ∈[t-1] I{i∈A t }X i,t N i,t-1
. This algorithm is comparable to Algorithm 13 but considers a Gaussian prior for each arm. Notice, the Gaussian family is self-conjugate, so except in the Gaussian-outcomes case, we do not rely on exact conjugated prior here. Although this is not surprising -since it is known that ts can work without exact conjugate prior with respect to the outcomes -obtaining an upper bound on the regret of the policy cts-gaussian is non-trivial and constitutes our main contribution. We state our main result in Theorem 35.

Theorem 35. The policy π described in Algorithm 14 has regret R T (π) of order

O   i∈[n] B 2 D i log 2 (m) log(T ) ∆ i,min   •
The proof of Theorem 35, as well as the complete non-asymptotic upper-bound is postponed to subsection 8.5.4. Nonetheless, in the following paragraphs, we provide some insights and highlight the novelty of our analysis. Notice, β > 1 is an artefact of the analysis and can in practice be taken equal to 1 (as we will do in our experiments).

Main proof challenges

In the setting of the previous section, the outcomes are independent in [0, 1] and an important step in Algorithm 13 was to transform the outcomes into binary variables in order to be consistent with the posterior. Here, outcomes are no longer independent. In addition to that, we cannot transform the outcomes into Gaussian variables in the same way as in Algorithm 13. These two points are the main technical challenges to address in our analysis.

Stochastic dominance Before providing details on how we deal with the above challenges, first recall that the standard analysis (in the case of a factorized prior, that we have here 3 ) consists in bounding the expected number of rounds needed for the sample θ t to be close to the true mean µ * on a certain set Z ⊂ A * , i.e., for the event (µ * -θ t ) e Z ∞ > ε to happens. We let T t (Z) denote the complementary event. As for the proof of Theorem 34, we can condition on the history to rewrite this expected number of rounds and then upper bound it as

E   t≥1 (t -1)P[ ¬T t (Z)|H t ] t-1 j=1 P[ T j (Z)|H j ]   ≤ E sup t≥1 1 P[ ¬T t (Z)|H t ] -1 ≤ Z ⊂Z, Z =∅ E   sup t≥1 i∈Z 1 P[ |θ i,t -µ * i | ≤ ε|H t ] -1   .
3 In practice, for C-sub Gaussian outcomes, the choice

N µ t-1 , C ij N ij,t-1 N -1 i,t-1 N -1 j,t-1 ij
for the prior where N ij,t-1 t ∈[t-1] I{i ∈ A t }I{j ∈ A t } may be preferred.

Now, using the fact that the conditional distribution of θ i,t -µ i,t-1 is symmetric and depends only on the counter N i,t-1 , we obtain that the probability P

[ |θ i,t -µ * i | ≤ ε|H t ] is a monotonic function of the deviation µ i,t-1 -µ * i .
Let us emphasize that this property of the Gaussian prior used is crucial and that it is not obvious to transfer the same technique to a beta prior. To sum up, we have to control a term of the form E sup t≥1 i∈Z g i µ i,t-1 -µ * i , where g i are non-negative increasing functions. Our approach is to prove that µ i,t-1 -µ * i i is weakly stochastically dominated by

βD i N i,t-1 |η i | i
, where η ∼ ⊗ i N (0, 1), which is the same vector but where the empirical mean is built with independent Gaussian outcomes instead. Notice, independence is crucial to be able to factorize the expectation E[ i∈Z g i ], in the same way as in the proof of Theorem 34. We recall two equivalent definitions of U is weakly stochastically dominated by V, see [START_REF] Shaked | Stochastic orders[END_REF] for more details and properties of dominances,

• For all non-negative, non-increasing functions

f i , E[ i f i (U i )] ≤ E[ i f i (V i )].
• For any vector x, it holds P

[U ≥ x] ≤ P[V ≥ x].
The first point applied to g i 's (and up to the supremum over t) is a simple way to obtain the aforementioned wanted control. Thus, it's enough to prove the second point, which is a consequence of the sub-Gaussianity of outcomes given by Assumption 7 and some concentration inequality. Finally, we circumvent the supremum over t ≥ 1 issue thanks to Doob's optional sampling theorem for non-negative super-martingales (see [START_REF] Durrett | Probability: theory and examples[END_REF], Theorem 5.7.6).

Importance of using a factorized prior in our analysis Note that in Algorithm 14, the samples θ i,t are independent, while the outcomes are not necessarily independent. This independence is in fact crucial in order to be able to start the analysis in the same way as in the proof of Theorem 34 (recall that Algorithm 13 also uses a factorized prior). More precisely, a factorized prior allows us to link the filtered regret against the event S t (Z) ∧ T t (Z) to the expected number of rounds needed for ¬T t (Z) to occur (see (8.3) in Step 4 of the proof of Theorem 34 in subsection 8.5.1 for a definition of S t (Z)). Indeed, without the factorized prior, the two events S t (Z), T t (Z) would no longer be independent conditionally to the history, and the term 1/P[ ¬T t (Z)|H t ] obtained in the previous paragraph would then be replaced by 1/P[ ¬T t (Z)|S t (Z), H t ], which is much more difficult to deal with. To the best of our knowledge, it is unknown how to get the desired bound when S t (Z) and T t (Z) are not independent conditionally to the history.

clip cts-gaussian for the linear reward case

In this subsection, we make the following assumptions on top of section 8.1.2.

Assumption 8. The reward function is linear, defined as r(A, µ) e T

A µ.

Assumption 9. The agent knows a matrix Γ 0 s.t. ∀λ ∈ R n + , E e λ(X-µ * ) ≤ e λ T Γλ/2 . Notice that Assumption 9 slightly generalises the setting from Degenne and Perchet (2016b). Requiring λ ∈ R n + allows us to take D i = max A∈A, i∈A j∈A (0 ∨ Γ ij ), so that negative correlations are no longer harmful. D i can still be too large (and thus θ t might be over-sampled), so we cap θ t with the score µ t used by cucb. The resulting policy is clip cts-gaussian, where the score θ t is replaced by µ t-1 ∨ θ t ∧ µ t before we plug it into Oracle, where µ i,t = µ i,t-1 + Γ ii 2(log(t)+4 log log(t))

N i,t-1

. clip ctsgaussian enjoys the following regret bound.

Theorem 36. The policy clip cts-gaussian has regret of order

O   i∈[n]
D i log 2 (m) ∧ mΓ ii log(T )

∆ i,min   .
Not only D i is improved through the above relaxation, but also, the leading term is never worst than the one of cucb. The proof and the complete non-asymptotic upperbound is delayed to subsection 36. We note that we rely heavily on reward linearity to analyse this clip version, not only using monotony to restrict the controls to the R n + directions (and thus to cap from bellow the sample by the empirical mean), but also using the oracle's invariance property Oracle(µ) = Oracle µ + δ e Oracle(µ) , with δ ≥ 0, to cap the sample from above by the UCB.

Comparison with the ols-ucb analysis of Degenne and Perchet (2016b)

The leading term in the regret bound given from Theorem 36 is comparable to the one for ols-ucb from Degenne and Perchet (2016b). Indeed, we recall that they obtained a factor of order Γ ii (1 -γ) log 2 (m) + γm , with γ as defined in the introduction of the thesis (i.e., γ = max A∈A max (i,j)∈A 2 ,i =j (0 ∨ Γ ij )/ Γ ii Γ jj ), where we have D i log 2 (m) ∧ mΓ ii . When γ ∈ {0, 1} (this is the case when we are in the settings (i) and (ii) respectively), these two terms coincide. When γ ∈ (0, 1), they are incomparable in general. We can still see that our variance term D i is always lower than their Γ ii ((1 -γ) + γm), i.e., that our bound rate is lower than log 2 (m) times theirs.

Experiments and discussion

Before describing the experiments carried out, notice that in the cts-gaussian policies, β > 1 is an artefact of the analysis and can in practice be taken equal to 1. This is what we did in our experiments.

The shortest path problem We compare our cts policies to cucb and cucb-kl, for the shortest path problem on the road chesapeake network [START_REF] Rossi | The Network Data Repository with Interactive Graph Analytics and Visualization[END_REF]. This network contains 39 nodes and n = 170 edges. A is the set of paths from an origin to a destination in the network. We choose a linear reward, so that an efficient Oracle exists for this problem. We choose µ * uniformly in [-1, 0] n and then normalize its sum so that i µ * i = -s, where s is unknown to the agent. The parameter s stands for the global network traffic (e.g., the total number of vehicles in the network). We run two experiments, one with -X ∼ ⊗ i Bernoulli(-µ * i ) and another with -X ∼ ⊗ i Bernoulli(-µ * i ) conditionally on i X i = -s. They are presented in Figure 8.1. Since the outcomes are not mutually independent in this last experiment, we use (clip) cts-gaussian rather than cts-beta, where we take D i = 1/4, using that for any λ ∈ R n + , E e λ T X ≤ i∈[n] E e λ i X i (see e.g., Borcea, Brändén, and Liggett (2009), corollary 4.18). It is clear from the experiments that cts policies outperform both cucb and cucb-kl. In the second experiment, we see that clip cts-gaussian and cts-gaussian are very similar -which is not surprising because D i is not large here (unlike in the next experiment) -and that for a small s, cucbkl becomes competitive, since the kl is much larger than the quadratic divergence in that case.

Comparison to escb for the matching problem

We consider here a comparison between (clip) cts-gaussian, cucb and escb (we refer the reader to [START_REF] Wang | Thompson Sampling for Combinatorial Semi-Bandits[END_REF] for a comparison between cts-beta and escb). Since escb is computationally intractable, we limit ourselves to a toy matching problem on the complete bipartite graphs K 4,4 , with X ∼ N (µ * , (cI{i = j} + I{i = j}) ij ), where this covariance is known to the agent. Our results are shown in Figure 8.2, where we observe that clip cts-gaussian (resp. escb) is slightly better for c small (resp. large), thus reaching the best of both worlds. This is because a large c forces clip cts-gaussian to oversample (as evidenced by cts-gaussian whose performance is even worse than cucb for c = 1). We also recorded the computation time for larger instances (see

Correlated vs independent prior in practice

We briefly discussed the use of a correlated prior in footnote 3, with covariance

C ij N ij,t-1 N -1 i,t-1 N -1 j,t-1 ij
, mentioning that the policy would perform better than using an independent prior. We ran additional empirical comparisons to assess this, plotting the results in Figure 8.3 where we also compared with a common prior policy approach [START_REF] Agrawal | Thompson sampling for the mnl-bandit[END_REF], i.e., with covariance N -1/2 i,t-1 N -1/2 j,t-1 ij

. 4 As expected, the correlated prior policy is better than the independent one (when outcomes are correlated). This motivates the theoretical study of such policy for future work. The common prior approach is comparable to the correlated prior one on the matching problem, but it is outperformed in the worst-case scenario of a separate action space A = {km + 1, . . . , (k + 1)m} | k ∈ 0, . . . , n m -1 with independent outcomes. This is because such problem reduces to a classical MAB problem with a covariance scaled up by a factor m, whereas the common prior approach has a variance scaled up by a factor m 2 .

Conclusion and future work

In this work, we have provided the first efficient policies having an optimal regret bound for a wide spectrum of problems instances for CMAB with semi-bandit feedback. Our approach also answers the question of finding an analysis for cts under correlated arm distributions. There are several possible extensions that could be considered as future work. For example, it would be interesting to have an analysis of cts with a correlated (Gaussian) prior. Indeed, apart from the empirical gain, this would open up the possibility of estimating the covariance matrix for use in the prior distribution, as we did in the previous chapter. Further relevant results would be an analysis of cts-beta without the mutual independence of outcomes, or also an improved concentration bound for a sum of independent betas, relying on the kl rather than using the sub-Gaussianity. This latter result would thus show that cts-beta dominates cucb-kl, which is empirically observed.

Missing proofs

Proof of Theorem 34

Proof of Theorem 34. We first restate the complete non-asymptotic upper-bound as follows. 4 We also tried the policy (without displaying the results, for the sake of clarity) with covariance

C ij N -1/2 i,t-1 N -1/2 j,t-1 ij
, and observed about the same performance as the correlated prior approach. Theorem.

Preliminary lemmas

In order to prove Theorem 34, we modify two lemmas from Wang and Chen, 2018: first, in their Lemma 3, we replace ε by ∆ min /(2B) -(m * 2 + 1)ε > 0, which gives the following Lemma 13 (that is proved in the same way as Proposition 14).

Lemma 13. In Algorithm 13, for any arm i, we have

E t ∈ [T ], i ∈ A t , |A t | • µ i,t-1 -µ * i > ∆ min 2B -(m * 2 + 1)ε ≤ 1 + ∆ min 2mB - (m * 2 + 1)ε m -2
.

Then, we modify Lemma 4 from Wang and Chen, 2018 as follows, leveraging on the mutual independence of θ 1,t , . . . , θ n,t to get a tighter confidence region for the sample θ t . Lemma 14. In Algorithm 13, for all round t, we have

P   e At (θ t -µ t-1 ) 1 ≥ 1 2 log(|A|2 m T ) i∈At 1 N i,t-1 H t   ≤ 1/T .
Proof. From [START_REF] Marchal | On the sub-Gaussianity of the Beta and Dirichlet distributions[END_REF], the Beta random variable from θ i,t is sub-Gaussian with variance 1/(4N i,t-1 ). i∈A E e λt(A)(θ i,t -µ i,t-1 ) + e λt(A)(µ i,t-1 -θ i,t ) H t ≤ A∈A 2 |A| e -λt(A)αt(A) e λt(A) 2 i∈A 1/(8N i,t-1 ) ≤ 1/T .

Main proof

With the two lemmas from the previous subsection, we are ready to demonstrate Theorem 34. We consider the following events.

• Z t {∆ t > 0} • B t ∃i ∈ A t , |A t | • µ i,t-1 -µ * i > ∆ min /(2B) -(m * 2 + 1)ε • C t e At (θ t -µ * ) 1 > ∆ t /B -m * 2 + 1 ε
• D t e At (θ t -µ t-1 ) 1 ≥ 0.5 • log(|A|2 m T ) i∈At 1/N i,t-1 .

We break down our analysis into 4 steps. The main novelties are in the last two steps:

Step 3 gives us the tighter dependence in m, and Step 4, that contains the main difficulties, gives the new exponential constant term.

Step 1: bound under Z t ∧ B t By Lemma 13,

t∈[T ] E[∆ t I{Z t ∧ B t }] ≤ ∆ max i∈[n] E t ∈ [T ], i ∈ A t , |A t | • µ i,t-1 -µ * i > ∆ min /(2B) -(m * 2 + 1)ε ≤ n∆ max   1 + ∆ min 2mB - (m * 2 + 1)ε m -2   .
Step 2: bound under Z t ∧ ¬B t ∧ C t ∧ D t By Lemma 14,

t∈[T ] E[∆(A t )I{Z t ∧ ¬B t ∧ C t ∧ D t }] ≤ ∆ max t∈[T ] E[P[ D t |H t ]] ≤ ∆ max t∈[T ]
1/T = ∆ max .

Step So we have that the following event holds

A t    ∆ t ≤ B 2log(|A|2 m T ) i∈At 1 N i,t-1    .
We can thus apply Theorem 13 to get the bound

t∈[T ] E[∆ t I{Z t , ¬B t , C t , ¬D t }] ≤ t∈[T ] E[∆ t I{A t }] ≤ 32B 2 log 2 2 (4 √ m) i∈[n]
∆ -1 i,min 2log(|A|2 m T ).

Step 4: bound under Z t ∧ ¬C t We consider the following events for a subset Z ⊂ T t (Z) (µ * -θ t ) e Z ∞ > ε .

We can state the three following lemmas. Note that Lemma 15 is exactly the Lemma 1 from [START_REF] Wang | Thompson Sampling for Combinatorial Semi-Bandits[END_REF]. The other two replace their Lemma 7.

Lemma 15. In Algorithm 13, for all round t, we have Z t , ¬C t ⇒ ∃Z ⊂ A * , Z = ∅ s.t. the event S t (Z) ∧ T t (Z) holds.

Lemma 16. Given Z ⊂ A * , Z = ∅, let τ q be the round at which S t (Z) ∧ ¬T t (Z) occurs for the q-th time, and let τ 0 = 0. Then, in Algorithm 13, we have

E   τ q+1 t=τq +1 I{S t (Z), T t (Z)}   ≤ E sup τ ≥τq +1 i∈Z 1 P[ |θ i,τ -µ * i | ≤ ε|H τ ]
Lemma 17. In Algorithm 13, we have

E sup τ ≥τq +1 i∈Z 1 P[ |θ i,τ -µ * i | ≤ ε|H τ ]
-1 ≤ cε -4 |Z| for every q ≥ 0 e -ε 2 q/8 c ε -4 |Z| if q > 8/ε 2 , where c and c are two universal constants.

These lemmas allow us to get a constant regret under the event Z t ∧ ¬C t . Indeed, we have from Lemma 15 that

t∈[T ] E[∆ t I{Z t ∧ ¬C t }] ≤ ∆ max Z⊂A * , Z =∅ E   t∈[T ] I{S t (Z) ∧ T t (Z)}   = ∆ max Z⊂A * , Z =∅ q≥0 E   τ q+1 t=τq +1 I{S t (Z), T t (Z)}   .
Lemma 16 and 17 gives that the above is further upper bounded by

∆ max Z⊂A * , Z =∅    8/ε 2 -1 q=0 cε -4 |Z| + q≥ 8/ε 2 e -ε 2 q/8 c ε -4 |Z|   
which is bounded by

∆ max C ε 2 C ε 4 m * ,
where C and C are two universal constants. This concludes the proof of the theorem.

Proof of Lemma 16. Since S t (Z), T t (Z) are independent conditioned on the history H t , the LHS is

E   k≥1
(k -1)P ¬T t k,q (Z) H t k,q k-1 j=1 P T t j,q (Z) H t j,q   , where t k,q is the round t where S t (Z) holds for the k-th time since the beginning of the round τ q + 1. Within the expectation, one can recognize the expectation of a time-varying geometric distribution, where the success probability of the k-th trial is P ¬T t k,q (Z) H t k,q . We can upper bound this inner expectation by the expectation of a geometric distribution whose success probability inf

τ ≥τq +1 P[ ¬T τ (Z)|H τ ] = inf τ ≥τq +1 i∈Z P[ |θ i,τ -µ * i | ≤ ε|H τ ]
is lower than all the success probabilities of the time-varying geometric distribution. This gives the result by monotonicity of the expectation, and rewriting the expectation of the geometric distribution.

Proof of Lemma 17. For any arm i ∈ [n], k i ∈ N, we define p i,k i as the probability of θ i,k i -µ * i ≤ ε, where θ i,k i is a sample from the posterior of arm i when there are k i observations of arm i (i.e., p i,k i is a random variable measurable with respect to those k i independent draws of arm i). From Lemma 5,6 in [START_REF] Wang | Thompson Sampling for Combinatorial Semi-Bandits[END_REF], we know that

E 1 p i,k i ≤ 4/ε 2 for every k i ≥ 0 1 + 6c • e -ε 2 k i /2 ε -2 + 2 e ε 2 k i /8 -2 if k i > 8/ε 2 ,
for some universal constant c . Since S t (Z) ∧ ¬T t (Z) implies that Z ⊂ A t , we know that for τ ≥ τ q + 1, N i,τ -1 ≥ q for all i ∈ Z. Using the mutual independence of outcomes, and the fact that the distribution of θ i,τ depends only on the history of arm i, we have

E sup τ ≥τq +1 i∈Z 1 P[ |θ i,τ -µ * i | ≤ ε|H τ ] -1 = E   sup τ ≥τq +1 Z ⊂Z, Z =∅ i∈Z 1 P[ |θ i,τ -µ * i | ≤ ε|H τ ] -1   ≤ Z ⊂Z, Z =∅ E   i∈Z sup τ ≥τq +1 1 P[ |θ i,τ -µ * i | ≤ ε|H τ ] -1   ≤ Z ⊂Z, Z =∅ E   i∈Z k i ≥q 1 p i,k i -1   , = Z ⊂Z, Z =∅ i∈Z E   k i ≥q 1 p i,k i -1   .
From this point, there are two cases: If q > 8/ε 2 , ≤ Z ⊂Z, Z =∅ i∈Z k i ≥q 6c • e -ε 2 k/2 ε -2 + 2e -ε 2 k/8 1 -2e -ε 2 k/8 -1 ≤ e -ε 2 q/8 c ε -4 |Z| , and if q ≤ 8/ε 2 ,

≤ Z ⊂Z, Z =∅ i∈Z    8/ε 2 k=q 4/ε 2 -1 + ∞ k≥ 8/ε 2 +1 6c • e -ε 2 k/2 ε -2 + 2e -ε 2 k/8 1 -2e -ε 2 k/8    ≤ cε -4 |Z| ,
where c, c are two universal constant.

Proof of Theorem 35

Proof of Theorem 35. We beginning by stating the complete version of Theorem 35.

Theorem.

+ nm 2 ∆ max ∆ min 2B -(m * 2 + 1)ε 2 + ∆ max Cε -2 β max i D i C √ β -1 ε -4 β 3 max i D 2 i m *
, where C, C are two universal constants, and ε ∈ (0, 1) is such that ∆ min /(2B) -(m * 2 + 1)ε > 0.

For the proof of Theorem 35, we consider the same events as in the proof of Theorem 34, except for the event D t , that becomes

D t    e At (θ t -µ t-1 ) 1 ≥ 2log(|A|2 m T ) i∈At βD i /N i,t-1    .
Step 1 is unchanged. Step 2 and Step 3 are modified only through the event D t , using the following modification of Lemma 14.

Lemma 18. In Algorithm 14, for all round t, we have that P[ D t |H t ] ≤ 1/T . Proof. We rely on the fact that conditionally on the history, the sample θ t is Gaussian of mean µ t-1 and of diagonal covariance given by βD i N -1 i,t-1 . We thus define the functions i∈A E e λt(A)(θ i,t -µ i,t-1 ) + e λt(A)(µ i,t-1 -θ i,t ) H t ≤ A∈A 2 |A| e -λt(A)αt(A) e λt(A) 2 i∈A βD i /(2N i,t-1 ) ≤ 1/T .

α t (A) 2log(|A|2 m T ) i∈A βD i N i,t
The final bound on the regret in Step 3 is obtained using the same derivation as in Theorem 34, which gives the following leading term:

256 log 2 2 (4 √ m) i∈[n]
B 2 βD i log(2 m |A|T ) ∆ i,min .

In the following, we consider the last step, consisting in bounding the regret under the event Z t and ¬C t . From the initialization phase, we also assume that the event

M t {∀i ∈ [n], N i,t-1 ≥ 1}
holds (the regret under the complementary event is clearly bounded by n∆ max ). If there is no initialization, we can have q = 0 in the following, noticing that when θ i,t is uniform on [a, b], then the probability P[ |θ i,t -µ * i | ≤ ε|H t ] is equal to 2ε/(b -a).

Step 4: bound under M t ∧ Z t ∧ ¬C t We use the independence of the prior, as for Theorem 34, to obtain the following upper bound, using M t to be able to start from q = 1.

t∈[T ] E[∆(A t )I{M t ∧ Z t ∧ ¬C t }] ≤ Z⊂A * , Z =∅ q≥1 E   sup τ ≥τq +1 Z ⊂Z, Z =∅ i∈Z 1 P[ |θ i,τ -µ * i | ≤ ε|H τ ] -1   ≤ Z⊂A * , Z =∅ q≥1 Z ⊂Z, Z =∅ E   sup τ ≥τq +1 i∈Z 1 P[ |θ i,τ -µ * i | ≤ ε|H τ ] -1   (8.4)
.

However, the expectation can't be put inside the product since outcomes are not mutually independent. We can still take a union bound on counters: One can notice that for all i ∈ Z , all k i ≥ q, I{N i,τ -1

= k i } 1 P[ |θi,τ -µ * i |≤ε|Hτ ]
-1 is of the form I{N i,τ -1 = k i }g i µ i,τ -1 -µ * i , with g i being an increasing function on R + . Indeed, we see that the conditional distribution of θ i,τ -µ i,τ -1 is N 0, βD i N -1 i,τ -1 , which is symmetric, so we have

P[ |θ i,τ -µ * i | ≤ ε|H τ ] = P θ i,τ -µ i,τ -1 + µ i,τ -1 -µ * i ≤ ε H τ .
In addition, under I{N i,τ -1 = k i }, the conditional distribution of θ i,τ -µ i,τ -1 does not depend on the history, but only on k i . Therefore, the above probability is a function of µ i,τ -1 -µ * i and so the function g i exists. It is increasing on R + because for any fixed σ > 0,

∂ ∂x x+ε x-ε 1 √ 2πσ 2 e -u 2 2σ 2 du = 1 √ 2πσ 2 e -(x+ε) 2 2σ 2 -e -(x-ε) 2 2σ 2
< 0 for x > 0.

In particular, we can consider the inverse function g -1 i . We now want to use a stochastic dominance argument in order to treat the outcomes as if they were Gaussian: we have for any k ∈ [q..∞) Z , (8.6) where τ * is the first time τ such that the event

E   sup τ ≥τq +1 i∈Z I{N i,τ -1 = k i }g i µ i,τ -1 -µ * i   = E   sup τ ≥τq +1 i∈Z I{N i,τ -1 = k i } ∞ 0 I g i µ i,τ -1 -µ * i ≥ u i du i   ≤ u∈R Z + E   sup τ ≥τq +1 i∈Z I{N i,τ -1 = k i }I g i µ i,τ -1 -µ * i ≥ u i   du = u∈R Z + E   i∈Z I{N i,τ * -1 = k i }I g i µ i,τ * -1 -µ * i ≥ u i   du,
I ∀i ∈ Z , N i,τ -1 = k i and g i µ i,τ -1 -µ * i ≥ u i
holds, and is ∞ if it never holds.

(8.6) =

u∈R Z + E   i∈Z I{N i,τ * -1 = k i }I g i µ i,τ * -1 -µ * i ≥ u i ∨ g i (0)   du = u∈R Z + E   i∈Z I{N i,τ * -1 = k i }I µ i,τ * -1 -µ * i ≥ g -1 i (u i ∨ g i (0))   du = u∈R Z + s∈{
s i g -1 i (u i ∨g i (0)) D i µ i,τ * -1 -µ * i - (g -1 i (u i ∨g i (0))) 2 2D i exp i∈Z (g -1 i (u i ∨g i (0))) 2 k i 2D i ≥ 1       ≤ E exp i∈Z N i,τ * -1 s i g -1 i (u i ∨g i (0)) D i µ i,τ * -1 -µ * i - (g -1 i (u i ∨g i (0))) 2 2D i exp i∈Z (g -1 i (u i ∨g i (0))) 2 k i 2D i = E exp τ * -1 t=1 i∈Z ∩At s i g -1 i (u i ∨g i (0)) D i (X i,t -µ * i ) - (g -1 i (u i ∨g i (0))) 2 2D i exp i∈Z (g -1 i (u i ∨g i (0))) 2 k i 2D i .
From Assumption 7, we have that

M τ = exp    τ -1 t=1 i∈Z ∩At    s i g -1 i (u i ∨ g i (0)) D i (X i,t -µ * i ) - g -1 i (u i ∨ g i (0)) 2 2D i       is a supermartingale, because E[ M τ |F τ -1 ]/M τ -1 is equal to E   exp    i∈Z ∩A τ -1    s i g -1 i (u i ∨ g i (0)) D i (X i,τ -1 -µ * i ) - g -1 i (u i ∨ g i (0)) 2 2D i       F τ -1    ≤ 1.
Since τ * is a stopping time with respect to F τ , we have from Doob's optional sampling theorem for non-negative supermartingales5 that E[M τ * ] ≤ 1. Therefore, (8.7) ≤ exp

  - i∈Z g -1 i (u i ∨ g i (0)) 2 k i 2D i   .
Now, we want to use the following fact (see [START_REF] Chang | Chernoff-type bounds for the Gaussian error function[END_REF]): if η ∼ N (0, 1), then with β > 1,

2e π √ β -1 β e -βx 2 /2 ≤ P[|η| ≥ x].
Indeed, this gives 2e π

√ β -1 β exp   - g -1 i (u i ∨ g i (0)) 2 k i 2D i    ≤ P |η i | ≥ g -1 i (u i ∨ g i (0)) k i βD i ,
where η ∼ N (0, 1) ⊗Z . Thus,

(8.6) ≤ π 2e 2β √ β -1 |Z | u∈R Z + i∈Z P βD i k i |η i | ≥ g -1 i (u i ∨ g i (0)) du = π 2e 2β √ β -1 |Z | u∈R Z + i∈Z P g i βD i k i |η i | ≥ u i ∨ g i (0) du = π 2e 2β √ β -1 |Z | u∈R Z + i∈Z P g i βD i k i |η i | ≥ u i du = π 2e 2β √ β -1 |Z | i∈Z ∞ 0 P g i βD i k i |η i | ≥ u i du i = π 2e 2β √ β -1 |Z | i∈Z E g i βD i k i |η i | .
We now want to bound E g i βD i k i |η i | . We define α = 2 -√ 2, the unique solution in (1/2, 1) of α -1/2 = (α -1) 2 /2. Notice that α -1/2 ≥ 1/12. Define ε i ε k i βD i . By definition, we have .

E g i βD i k i |η i | = +∞ -∞ e -x
We first bound A 1 . With the change of variable u = y -x, we get:

A 1 = 2 +∞ αε i 1 ε i -ε i e -u 2 /2-ux du dx ≤ 2 +∞ αε i 1 0
-ε i e -u 2 /2-ux du dx Note that for x ≥ αε i and u ∈ [-ε i , 0], -u 2 /2 -ux ≥ -(1 -1 2α )ux and thus:

A 1 ≤ 2 +∞ αε i 1 0 -ε i e -(1-1 2α )ux du dx = 2 +∞ αε i (1 -1 2α )x e (1-1 2α )ε i x -1 dx.
(8.9)

We distinguish two regimes. First, if ε 2 i ≥ 12, then (8.9) ≤ 2e (α-1 2 )ε 2 i e (α-1 2 )ε 2 i -1

+∞ αε i 1 - 1 2α xe -(1-1 2α )ε i x dx = 2e (α-1 2 )ε 2 i e (α-1 2 )ε 2 i -1 1 (1 -1 2α )ε 2 i +∞ (α-1 2 )ε 2 i xe -x dx = 2e (α-1 2 )ε 2 i e (α-1 2 )ε 2 i -1 1 (1 -1 2α )ε 2 i -(x + 1)e -x ∞ (α-1 2 )ε 2 i = 2e (α-1 2 )ε 2 i e (α-1 2 )ε 2 i -1 1 (1 -1 2α )ε 2 i α - 1 2 ε 2 i + 1 e -(α-1 2 )ε 2 i = 2 e (α-1 2 )ε 2 i -1 α + α (α -1 2 )ε 2 i ≤ 4e -ε 2 i /12 .
Otherwise, we have

(8.9) = 2(1 -1 2α ) ε 2 i ∞ αε 2 i u e (1-1 2α )u -1 du ≤ 2(1 -1 2α ) ε 2 i ∞ 0 u e (1-1 2α )u -1 du = 2(1 -1 2α ) ε 2 i π 2 6 1 -1 2α 2 ≤ 24βD i ε 2 .
We now bound A 2 . As x ∈ [-αε i , αε i ], it comes that [-(

1 -α)ε i , (1 -α)ε i ] ⊂ [x -ε i , x + ε i ]. This implies that A 2 ≤ αε i -αε i e -x 2 /2 dx (1-α)ε i -(1-α)ε i e -x 2 /2 dx -1 = 2 αε i (1-α)ε i e -x 2 /2 dx (1-α)ε i -(1-α)ε i e -x 2 /2 dx ≤ 2 ∞ (1-α)ε i e -x 2 /2 dx (1-α)ε i -(1-α)ε i e -x 2 /2 dx ≤ e -(1-α) 2 ε 2 i /2 1 -e -(1-α) 2 ε 2 i /2 ≤ 1 + 12 ε 2 i e -ε 2 i /12 .
The penultimate inequality relies on ∞ x e -u 2 /2 du≤ π 2 e -x 2 /2 (see [START_REF] Jacobs | Principles of communication engineering[END_REF], eq. (2.122)). We obtain again two regimes: 2e -ε 2 i /12 if ε 2 i ≥ 12, and 1 + 12βD i ε 2 otherwise. To summarize, we proved that (8.6) is bounded by

π 2e 2β √ β -1 |Z | i∈Z I ε 2 k i βD i < 12 1 + 36 βD i ε 2 + I ε 2 k i βD i ≥ 12 6e -ε 2 k i 12βD i .
After the summation on k, on Z , on q, and on Z, we obtain that there exists two constants C, C such that

Z⊂A * , Z =∅ q≥1 Z ⊂Z, Z =∅ k∈[q..∞) Z (8.6) ≤ Cε -2 β max i D i C β √ β -1 ε -4 β 2 max i D 2 i m * so, t∈[T ] E[∆(A t )I{M t ∧ Z t ∧ ¬C t }] ≤ ∆ max Cε -2 β max i D i C β √ β -1 ε -4 β 2 max i D 2 i m * .

Proof of Theorem 36

Proof of Theorem 36. The regret bound of clip cts-gaussian is stated completely as follows.

Theorem.

+ ∆ max Cε -2 β max i D i C √ β -1 ε -4 β 3 max i D 2 i m *
, where C, C are two universal constants, and ε ∈ (0, 1) is such that ∆ min /(2B) -(m * 2 + 1)ε > 0.

More precisely, notice that the modification on the sample θ t has an impact only in two places in the analysis: in the concentration bound and in the event controlling optimism. We detail these two points in the following.

Concentration bound

In this subsection, we provide the concentration bound of clip cts-gaussian. Our strategy here is to either use the concentration from µ t or from θ t , depending on which regime is the best for each arm. Thus, we define We have the following lemma.

Lemma 19.

P   e T At∩S (µ t-1 ∨ θ t ∧ µ t -µ t-1 ) ≥ 2log(|A|2 m T ) i∈At∩S βD i /N i,t-1 H t   ≤ 1/T .
Proof. We define the functions Γ ii 2(log(t) + 4 log log(t)) N i,t-1 .

To conclude, we have the following event

A t    ∆ t ≤ 8log(|A|2 m T ) i∈At∩S βD i /N i,t-1 + i∈At∩S c
Γ ii 8(log(t) + 4 log log(t))

N i,t-1    .
Using Proposition 6, we have

t∈[T ] E[∆ t I{A t }] ≤ t∈[T ] E   ∆ t I    ∆ t ≤ 2 8log(|A|2 m T ) i∈At∩S βD i /N i,t-1      + t∈[T ] E   ∆ t I    ∆ t ≤ 2 i∈At∩S c
Γ ii 8(log(t) + 4 log log(t))

N i,t-1      .
We can thus apply Theorem 12 and Theorem 13 to get the bound 512 log 2 2 (4

√ m) i∈S ∆ -1 i,min βD i log(|A|2 m T )+128m i∈S c
∆ -1 i,min Γ ii (log(T ) + 4 log log(T )).

Optimism

In this subsection, we examine the theoretical impact of considering clip cts-gaussian on the optimism-controlling event (event ¬C t ), in the case of linear rewards. For this purpose, we modify the beginning of Step 4 in the analysis by considering the following events.

• Z t {∆ t > 0}

• C t e T At θ t > e T A * µ * -(m * (m * + 1)/2 + 1)ε

• S t (Z) ∀θ s.t. 0 ≤ (µ * -θ ) e Z ≤ εe Z , R(θ e Z + θ t e Z c , Z) holds

• T t (Z) ∃i ∈ Z, µ * i -µ * i ∧ θ i,t > ε . • J t {∀i ∈ [n], µ * i ≤ µ i,t },
where R(θ , Z) is the event

∀A ∈ arg max

A ∈A e T A (θ ) : Z ⊂ A, e T Oracle(θ ) θ > e T A * µ * -(m * (m * + 1)/2 + 1)ε .

In the above events, θ t is µ t ∧ θ t ∨ µ t . The last event J t holds with probability at least 1 -n/(t log 2 (t)) from Hoeffding's inequality [START_REF] Hoeffding | Probability inequalities for sums of bounded random variables[END_REF]. We thus assume that this event hods in the following, since the regret under the complementary event is bounded by 3.2n∆ max . We first state the following lemma.

Lemma 20.

Z t , ¬C t ⇒ ∃Z ⊂ A * , Z = ∅ s.t. the event S t (Z) ∧ T t (Z) holds.

This allows us to consider the success probability P[ ¬T t (Z)|H t ] in the analy- sis. Notice however that Z ⊂ Oracle µ * ∧ θ t e Z + θ t e Z c , that is guaranteed when S t (Z) ∧ ¬T t (Z) holds, does not necessarily implies that Z ⊂ Oracle θ t . How- ever, it turns out that the following (true) predicate + such that for all A ∈ arg max A ∈A e T A η, we have Z ⊂ A. Then, for all A ∈ arg max A ∈A e T A (η + δ e Z ), we have Z ⊂ A. It now remains to explain how to handle the probability P[ ¬T t (Z)|H t ] in the analysis. Notice that from the high probability event J t , it suffices to treat the case θ t = θ t ∨ µ t . We provide here the places where the analysis differs, the rest of the proof remains unchanged.

Z ⊂ A,
• We use that

P[ ¬T t (Z)|H t ] = P ∀i ∈ Z, ε ∨ µ * i -µ i,t-1 -0 ∨ θ i,t -µ i,t-1 ≤ ε H t ,
is a product of functions that are decreasing with respect to ε ∨ µ * i -µ i,t-1 .

• We use that ε ∨ µ * i -µ i,t-1 ≥ g -1 i (u i ∨ g i (ε)) is equivalent to µ * i -µ i,t-1 ≥ g -1 i (u i ∨ g i (ε)). A θ e Z 2 + θ t e Z 2 c , where the last inequality is obtained in the same way as in inequalities from (8.24) to (8.25).

We could repeat the above argument and each time the size Z i is decreased by at least 1. Thus, after at most m * -1 steps, since m * + (m * -1) + (m * -2) + • • • + 1 = m * (m * + 1)/2 is still less than m * (m * + 1) 2 /2 + 1, we could reach the end and find a Z i = ∅ such that S t (Z i ) holds. A (η + δ e Z ).

• The consideration of a non-optimistic policy. We also considered randomized policies in Chapter 8, using a Thomspon sampling approach. We could see in particular that the regret rate of those policies were essentially the same as the ones for optimistic 2 policies. More precisely, we proposed cts policies in two different settings, first with a beta prior for independent outcomes (which was already existing), and then with a Gaussian prior for sub-Gaussian outcomes. For the two cases, we gave a new tight regret analysis. We also noted that the empirical performance of cts was generally superior compared to the optimistic 1 and 2 approaches.

Computational point of view

Regarding the computational efficiency of policies, we can once again detail three aspects:

• Exploitation of the uncertainty structure. Although 2 approaches are more powerful, they pose some computational efficiency issues when used with optimistic policies. In Chapter 5, for CMAB with linear reward function, we were able to develop (for matroid-based action spaces) some methods to remedy the 2 inefficiency. More precisely, we gave several approximation algorithms for the maximization of a set function that is a sum of a linear one and a submodular one. This allowed us to give efficient implementations of the escb policy with matroid constraints. We also studied the budgeted setting, and gave an approximation scheme to adapt the above approach.

• Finding an efficient surrogate for exploration bonuses. In Chapter 6, we circumvented the inefficiency of the 2 methods by relaxing the exploration bonus under consideration, the goal being that the new bonus, although statistically less efficient, would be computationally efficient to use in the polities. More precisely, in the problems we looked at in this chapter (i.e., OIM and BOIM), the oracle (greedy) leverages the submodularity of the reward function to provide the 1 -1/e approximation factor. We thus saved the computational efficiency by constructing submodular surrogate bonuses, since submodularity is closed under non-negative linear combinations.

• Computational efficiency of the cts policy. Faced with the difficulty of making optimistic 2 policies efficient, we proposed in Chapter 8 a nonoptimistic alternative, namely cts. As we have already seen, cts obtained essentially the same performance as the 2 methods, while being computationally efficient to implement. Therefore, this approach is surely the best answer, in this thesis, to our initial challenge about the trade-off between the two efficiencies: it then seems that such a trade-off is, in fact, unnecessary. As we will see, we think future efforts should be made to try to transfer systematically any optimistic 2 approach to a cts approach of the same performance.

Miscellaneous results in combinatorial optimization

In this thesis, we also had the opportunity to study different combinatorial optimization problems. Indeed, in Chapter 4, we introduced a new sequential search-and-stop problem, that falls into the CMAB-T setting and the budgeted setting. The offline sequential problem led us to the formulation of an interesting new combinatorial optimization problem, which is a variant of the classical scheduling problem 1|prec| w j C j where the agent can, whenever it wishes, restart its search all over again by re-shuffling the not found hidden object. We proposed an exact and efficient solution that makes an elegant link to the exact solution of the original 1|prec| w j C j problem. The BOIM problem (9.2) may raise some issues. Indeed, the errors generated by the uncertainty that we have on the covariance estimates then concern non-observed coordinates (because of the maximum taken on A ∈ A). Having a bonus depending on the counters whose coordinates do not get feedback is redibitory for the analysis. In contrast, a prior of the form (9.1) doesn't have that inconvenience, and seems more promising to use for covariance estimation.

Approximation regret for cts Another unresolved difficulty for cts is its use for approximation regrets. Although [START_REF] Wang | Thompson Sampling for Combinatorial Semi-Bandits[END_REF] have shown that this is generally impossible, exploiting the special design of some approximation oracles to prove that cts works with them seems an interesting direction. Title: Efficient Learning in Stochastic Combinatorial Semi-Bandits Keywords: Combinatorial bandit, confidence regions, computational efficiency Abstract: Combinatorial stochastic semibandits appear naturally in many contexts where the exploration/exploitation dilemma arises, such as web content optimization (recommendation/online advertising) or shortest path routing methods. This problem is formulated as follows: an agent sequentially optimizes an unknown and noisy objective function, defined on a power set P([n]). For each set A tried out, the agent suffers a loss equal to the expected deviation from the optimal solution while obtaining observations to reduce its uncertainty on the coordinates from A. Our objective is to study the efficiency of policies for this problem, focusing in particular on the following two aspects: sta-tistical efficiency, where the criterion considered is the regret suffered by the policy (the cumulative loss) that measures learning performance; and computational efficiency. It is sometimes difficult to combine these two aspects in a single policy. In this thesis, we propose different directions for improving statistical efficiency, while trying to maintain the computational efficiency of policies. In particular, we have improved optimistic methods by developing approximation algorithms and refining the confidence regions used. We also explored an alternative to the optimistic methods, namely randomized methods, and found them to be a serious candidate for combining the two types of efficiency.

Université Paris-Saclay

Espace Technologique / Immeuble Discovery Route de l'Orme aux Merisiers RD 128 / 91190 Saint-Aubin, France

Figure 1 . 1 :

 11 Figure 1.1: Exemple d'un réseau (graphe) où chaque arête est un bras. Le noeud en bleu représente l'origine, et le noeud en rouge la destination. Les arêtes représentées par des lignes en gras forment un chemin entre l'origine et la destination : il s'agit donc d'une action que l'agent peut décider de jouer lors d'un tour. Se faisant, chaque bras du chemin produit une réalisation, et la récompense peut alors être la somme des réalisations du chemin.

Figure 1 . 3 :

 13 Figure 1.3: Example of a network(graph) where each edge is an arm. The node in blue represents the origin, and the node in red represents the destination. The edges represented by bold lines form a path between the origin and the destination: it is therefore an action that the agent can decide to play during a round. Doing so, each arm of the path produces one outcome, and the reward can then be the sum of the outcomes of the path.

Figure

  Figure 2.2: Example of an exploration graph G t . Red nodes are probed, blue nodes are only observed, grey nodes exist in the original graph G and are yet to be observed. Vertices of the graph G t are red and blue nodes, and edges of G t are represented with bold lines. Dash lines are used to denote unobserved links at the given moment t. To build G t+1 , the agent has to choose a blue node. Then, it becomes red and its neighborhood in G becomes blue.

Proof.

  Let λ ∈ R. We can assume that E[X] = 0 by replacing a by a -E[X] and b by b -E[X]. By convexity of the function x → e λx , we have for all x ∈ [a, b], e λx ≤ b -x b -a e λa + x -a b -a e λb .

Figure 3 . 1 :

 31 Figure 3.1: Example of a bipartite graph considered in the matching bandit problem. Red nodes are users, blue nodes are channels. Edges represents the base arms.When an edge between a user and a channel is present in the graph, it means that the user can transmit through that channel. An example of matching is formed by the bold edges.

Figure 3 . 2 :

 32 Figure 3.2: Example of a bipartite graph considered in the PMC bandit problem. Red nodes are those in the action S.Edges ij represented with bold lines are those with X ij = 1, and the others, represented with dash lines, are such that X ij = 0. A bold edge adjacent to a red node makes the other node of that edge blue. Thus, blue nodes are those covered by the red ones. The reward is the number of blue nodes.

Proposition 13 .

 13 The set of initial sets of maximum density in G is a lattice. Proof. We use the fact that for a, b ≥ 0 and a , b > 0, a+b a +b ≤ a a ∨ b b , with equality if and only if a a = b b . Indeed, if A and A are two initial sets of maximum density in G, then w T e A c T e A = w T (e A + e A ) c T (e A + e A ) = w T (e A∪A + e A∩A ) c T (e A∪A + e A∩A ) ≤ w T e A∪A c T e A∪A ∨ w T e A∩A c T e A∩A .

  δ = O(n 2 δ), contrary to just 8 With the convention x/0 = +∞, ∀x ≥ 0.

Figure 4 . 1 :

 41 Figure 4.1: The DAG considered in Theorem 20.

  Figure 4.2: Cumulative regret for sequential search-and-stop, with B up to 10 5 , averaged over 100 independent simulations.

Figure 5 . 2 :

 52 Figure 5.2: Cumulative regret for the minimum spanning tree setting in up to 10 5 rounds, averaged over 100 independent simulations. Left: for A = B. Right: for A = I.

Figure 6 . 1 :

 61 Figure 6.1: Regret curves with respect to the budget B (expectation computed by averaging over 10 independent simulations).

Figure 7 . 2 :Figure 7 . 1 :

 7271 Figure 7.2: Left: Correlation matrix of the dataset, right: Cumulative regret, averaged over 36 independent simulations

Assumption 7 .Proposition 26 .

 726 There exists a vector D (D 1 , . . . , D n ) ∈ R n + known to the agent such that ∀A ∈ A, ∀λ ∈ R n s.t. λ = λ e A , E e λ T (X-µ * ) ≤ e λ T D λ/2 . Assumption 7 encompasses the κ 2 i -sub Gaussian outcomes with worst case dependencies between the arm distributions, taking D i = κ 2 i m. It also captures C-sub-Gaussian outcomes with a known sub-Gaussianity matrix C (setting (iii)), taking D i = max A∈A, i∈A j∈A |C ij |.

Figure 8 . 1 :Figure 8 . 2 :

 8182 Figure 8.1: Cumulative regret (averaged over 50 simulations) for the shortest path problem. Top: with mutually independent outcomes, taking the opposite sum of means being s = 70, 90, 110, 130 respectively. Bottom: with correlated outcomes, taking the opposite sum of outcomes being s = 70, 90, 110, 130 respectively.

  Figure 8.3: Comparison with correlated prior sampling and common prior sampling (averaged over 50 simulations).The first 4: for the K 4,4 matching problem, with Gaussian outcomes, taking c = 0, 0.2, 0.5, 1.The last:for A = {km + 1, . . . , (k + 1)m} | k ∈ 0, . . . , n m -1 , c = 0.

  [n] R(θ , Z) Z ⊂ Oracle(θ ), e Oracle(θ ) (θ -µ * ) 1 > ∆(Oracle(θ )) -(k * 2 +1)ε S t (Z) ∀θ s.t. (µ * -θ ) e Z ∞ ≤ ε, R(θ e Z + θ t e Z c , Z) holds (8.3)

I

  ∀i ∈ Z , N i,τ -1 = k i i∈Z 1 P[ |θ i,τ -µ * i | ≤ ε|H τ ]

  Si∈ [n], Γ ii m(log(T ) + 4 log log(T )) ≥ 4 log 2 2 (4 √ m)βD i log(|A|2 m T ) .

  Proof of Lemma 21. Let's prove that arg maxA ∈A e T A (η + δ e Z ) ⊂ arg max A ∈A e T A η. Consider any action A ∈ arg max A ∈A e T A (η + δ e Z ). If A / ∈ arg max A ∈A e T A η, then there exists B ∈ arg max A ∈A e T A η such that e T A η < e T B η.Furthermore, since Z ⊂ B and δ ≥ 0, we also havee T A (δ e Z ) ≤ e T B (δ e Z ),so we finally have e T A (η + δ e Z ) < e T B (η + δ e Z ), contradicting that A ∈ arg max A ∈A e T

An Example of CMAB-T Problem: Sequential Search-and-Stop

  • PierrePerrault, Etienne Boursier, Vianney Perchet, Michal Valko (2020) "Statistical Efficiency of Thompson Sampling for Combinatorial Semi-Bandits", in 34th Conference on Neural Information Processing Systems (NeurIPS).

	Chapter 4,

  2.2: Example of an exploration graph G t . Red nodes are probed, blue nodes are only observed, grey nodes exist in the original graph G and are yet to be observed. Vertices of the graph G t are red and blue nodes, and edges of G t are represented with bold lines. Dash lines are used to denote unobserved links at the given moment t. To build G t+1 , the agent has to choose a blue node. Then, it becomes red and its neighborhood in G becomes blue.

Algorithm 2

  Generic index-based MAB policy Initialization: Play each arm once. for t ∈ {n + 1, . . . , T } do For all i ∈ [n], compute µ i,t , the score of arm i at round t. Play the arm i t arg max i∈[n] µ i,t , ties are broken at random. end for

Table 2

 2

	.1: Some examples of popular index score for rewards in
	[0, 1]. δ(t) is an exploration function, that can be of the form δ(t) =
	ζ log(t), with ζ > 1, or δ(t) = log(t) + 3 log log(t), for t ≥ 3, with
	δ(1) = δ(2) = δ(3). For ts, µ i,t-1 can be µ i,t-1 in practice. In
	theory, when rewards are not binary, it is maintained replacing each
	received reward X i,t by a sample Y i,t ∼ Bernoulli X i,t .

  on this interval, and as the unique solution x ∈ (-∞, µ * i ) to the equation N φ * i (x) = δ otherwise. Consider now that A t ∩ B d,t holds. We have from γ d φ * i

Table 2

 2

.2: Confidence intervals for optimistic policies.

  where G Wt ∼ D, and where the predicate S t Wt i holds if, in the graph G Wt , there is a forward path from a node in S t to the node i. In other word, A t is the set of edges reachable from S t in G

  + (SS S ) ≤ J + (SS ) and (4.10) is true. Thus, we suppose that 1 -w T e SS ≥ 0. If J (S) ≤ J (SS ), then J + (S) ≤ J + (SS ) and (4.10) is true. Else, SS S + w T e S (1 -w T e SS ) λ(S )w T e SS S + (SS S ) ≤ J + (SS ) and (4.10) is true. Now, as a preview, we can actually derive easily the proof of Lemma 5 when there is no precedence constraints, the idea in the general case being very similar. Let (Z 1 , . . . , Z k ) be a Sidney decomposition. Then, if z i,1 , . . . , z i,j i are arms of Z i , we have

	≤	w T e S w T e SS S	-(1 -w T e SS ) λ(S )	+	1 -w T e SS λ(S )
	Example 4.
		J (SS ) ≥	1 w T e S	(J (SS )w T e SS -J (S)w T e S ) ≥	1 -w T e SS λ(S )	.	(4.12)
	Thus, we have

w T e SS w T e SS S + w T e S J (S ) -w T e SS c T e S w T e SS S ≤ J (SS )w T e SS w T e SS S + w T e S (1 -w T e SS ) λ(S )w T e SS S . (4.11) If 1 -w T e SS ≤ 0, by (4.11), we have that 0 ≤ J (SS S ) ≤ J (SS )w T e SS w T e SS S ≤ J (SS ), so J J (SS S ) -J (SS ) ≤ J (SS )w T e SS w T e SS S + w T e S (1 -w T e SS ) λ(S )w T e SS S -J (SS ) (4.11)

= -w T e S J (SS ) w T e ≤ 0 (4.12) and λ(S ) ≥ λ(S ).

So, J

  such search, with |S| being the smallest possible (i.e., |S | being the smallest possible). Let A be the support of S . By contradiction, assume A = ∅. By Theorem 18, there exists a minimizer of the form

Table 4 . 1 :

 41 Comparison algorithms in the experiment.

  • Second, for a policy π that does not select a search in {SS , S S} for some round t, we construct π that acts like π except for this round t where it selects SS if π would see the leaf s n 2 first, and S S otherwise, i.e., if π would first see the leaf s n 2 . Now compare both policies on the same realization of W 1 , W 2 , . . . . We claim that the global reward of π is never smaller than that of π. By symmetry, assume that π sees s n 2 first within round t and thus π selects SS .

-If W s n 2 ,t = 1 or W s n 2 ,t = 1 and π visits s n 2 within round t , both policies obtain the same reward of 1 within round t, but π pays less than π. -If W s n 2 ,t = 1 and π does not visit s n 2

  Algorithm 7 greedy for maximizing L + F on B.

	Input: L, F , I, m.
	Initialization: S ← ∅.
	for i ∈ [m] do
	x ∈ arg max x/ ∈S, S∪{x}∈I (L + F )(S ∪ {x}).
	S ← S ∪ {x}.
	end for
	Output: S.

  Algorithm 9 boim-cucb Input: ε > 0, B 0 = B > 0. for each round t ≥ 1 do If true costs are known, then c t ← c * . Compute S t given by Algorithm 10 with input S → σ(S; w t ), c t . Select seed set S t , and pay e T St∪{0} C t (i.e., remove this cost from B t-1 to get the new budget B t ). if B t ≥ 0, then Get the reward σ(S t ; W t ), get the feedback, and update corresponding quantities accordingly. else

-1/eε)-approximately maximize S → σ(S; w * ) e T S c * + c * 0 (6.6)

  but only approximately) to select S * t ∈ arg max S⊂V σ(S; w t )/e S∪{0} . If λ = σ(S * t ; w t )/e S * t ∪{0} c t , then we also have that boim-cucb-regularized aims at choosing S * t , since one can notice that S * t ∈ arg max S⊂V σ(S; w t ) -λe S c t .

	1e4					1e4						1e4
					1.6						1.75
	0.8				1.4						1.50
					1.2						1.25
	0.4 0.6 t = 1 ∆(S -1 τB t )			0.6 t = 1 0.8 1.0 ∆(S -1 τB t )					0.75 t = 1 1.00 ∆(S -1 τB t )
											0.50
	0.2	BOIM-CUCB-regularizedλ=2 BOIM-CUCB-regularizedλ=3 BOIM-CUCB-regularizedλ=4	0.2 0.4			BOIM-CUCB-regularizedλ=2 BOIM-CUCB-regularizedλ=3 BOIM-CUCB-regularizedλ=4	0.25		BOIM-CUCB-regularizedλ=2 BOIM-CUCB-regularizedλ=3 BOIM-CUCB-regularizedλ=4
	0.0	BOIM-CUCB			0.0			BOIM-CUCB			0.00		BOIM-CUCB
	0	2000 4000 6000 8000 10000 12000 14000 B		0	2000 4000 6000 8000 10000 12000 14000 B		0	2000	4000	6000 B	8000	10000 12000
		1e4						1e4
		1.75 2.00						1.50 1.75	BOIM-CUCB-regularizedλ=2 BOIM-CUCB-regularizedλ=3 BOIM-CUCB-regularizedλ=4
									BOIM-CUCB
		1.50
								1.25
		0.75 t = 1 1.00 1.25 ∆(S -1 τB t )					0.75 t = 1 1.00 ∆(S -1 τB t )
		0.25 0.50			BOIM-CUCB-regularizedλ=4 BOIM-CUCB-regularizedλ=3 BOIM-CUCB-regularizedλ=2	0.25 0.50
		0.00			BOIM-CUCB		0.00
		0	2000	4000	6000 B	8000	10000 12000	0	2000	4000	6000 B	8000	10000 12000
		Figure 6.2: Regret curves on five different problem instances, with
		respect to the budget B (expectation computed by averaging over 10
					independent simulations).

  We can thus remove P t and replace N ⊕ i,t-1 by N i,j,t-1 , since this can only increases the probability. By an union bound we have, Since the random variables I {j} W t i are bernouillies of mean p i ({j}; w * ), we can apply the Fact 9 to get P N j,t-1 = t , t p i ({j}; w * ) 8 > N i,j,t-1 ≤ exp -(7/8) 2 8δ(t)/2 < 1/t 3 . By taking t over {0, . . . , t -1}, the proposition holds.

	Fact 9 (Multiplicative Chernoff Bound Mitzenmacher and Upfal (
	P p i ({j}; w * ) >	8δ(t) N j,t-1	and	p i ({j}; w * ) N i,j,t-1	>	8 N j,t-1
	≤	t >	p i ({j};w * ) 8δ(t)	P N j,t-1 = t ,	t p i ({j}; w * ) 8	> N i,j,t-1 .

  Play A 1 = [n], or at least a sequence A 1 , A 2 , . . . , (no more than n(n -1)/2) such that for any i, j ∈ [n], one of these A t 's contains {i, j}. We thus have N ij,t-1 ≥ 1 for all i, j ∈ [n].

	Algorithm 11 escb-c (Efficient Sampling for Combinatorial Bandits with Covari-
	ance estimate)
	Initialization:
	For all subsequent rounds t:
	Solve the following bilinear program to get A t , with C t (A) defined by (7.3), and
	play A t ,
	(A t , µ t ) ∈ arg max
	A∈A, µ∈Ct(A)

  Algorithm 12 escb-c modified for the case of • 1 -constrained outcomes Initialization:Play A 1 = [n],or at least a sequence A 1 , A 2 , . . . , (no more than n) such that all arm have been sampled once. We thus have N i t -1 ≥ 1 for every arm i ∈ [n].

	For all subsequent rounds t:
	Solve the following bilinear program to get A t , with C t define by (7.6), and play
	A t .
	(A t , µ t ) ∈ arg max A∈A, µ∈Ct	e T A µ.
	precise bound). Again, notice we are reaching the lower bound of Theorem 33, using
	the relation i

Table 8 .

 8 1: Factor in front of n log(T )/∆ in the regret bound (O(•) for upper bounds), computationally inefficient policies are printed with a subscript * , setting (iii) is for C diagonal, clip cts-gaussian is for linear reward functions, and with only λ ∈ R n + in (iii). Our results are printed in bold, see Theorem 34, Theorem 35, Theorem 36 related to cts-beta, cts-gaussian, clip cts-gaussian respectively.

  Algorithm 13 cts-beta Initialization: For each arm i, let a i = b i = 1. For all t ≥ 1: Draw θ t ∼ ⊗ i∈[n] Beta(a i , b i ), and play A t = Oracle(θ t ). Get the observation X t e At , and draw Y t ∼ ⊗ i∈At Bernoulli(X i,t ). For all i ∈ A t update a i ← a i + Y i,t and b

  Table 8.2), and observe the efficiency of cucb and clip cts-gaussian compared to escb. K 3,3 K 4,4 K 5,5 K 6,6 K 7,7 K 8,8 cucb 0.39 0.64 1.23 1.65 2.45 3.88 clip cts-gaussian 0.50 0.80 1.75 1.79 3.30 5.42 escb 0.45 1.93 10.3 75.6 541 4694 Table 8.2: Computation time per round (ms), with c = 0.3, T = 100, averaged over 5 simulations.

  The policy π described in Algorithm 13 has regret R T (π) bounded by

	16 log 2 2 (16m)	i∈[n]	B 2 log(2 m |A|T ) ∆ i,min
	+∆ max (1 + n)+	nm 2 ∆ max 2B -(m * 2 + 1)ε ∆ min	2 +∆ max	ε 2 C	ε 4 C	m

*

, where C, C are two universal constants, and ε ∈ (0, 1) is such that ∆ min /(2B) -(m * 2 + 1)ε > 0.

  At (θ t -µ t-1 ) 1 ≥ α t (A t ) H t ≤ A (θ t -µ t-1 ) 1 ≥ α t (A) H t ≤ A∈A e -λt(A)αt(A) E e λt(A) e A (θt-µ t-1 ) 1 H t

	≤	e -λt(A)αt(A)	E e λt(A)|θ i,t -µ i,t-1 | H t
	A∈A			i∈A
	≤	e -λt(A)αt(A)
	A∈A
				Thus, defining the functions
	α t (A)	1 2	log(|A|2 m T )

i∈A 1 N i,t-1

, and λ t (A) 4α t (A) i∈A 1/N i,t-1 , we have P e A∈A P e

  3: bound under Z t ∧ ¬B t ∧ C t ∧ ¬D t ∆ t /B ≤ e At (θ t -µ * ) 1 + m * 2 + 1 ε C t ≤ e At (θ t -µ t-1 ) 1 + e At (µ t-1 -µ * ) 1 + m * 2 + 1 ε ≤ e At (θ t -µ t-1 ) 1 + ∆ min /(2B) -m * 2 + 1 ε + m * 2 + 1 ε ¬B t ≤ e At (θ t -µ t-1 ) 1 + ∆ t /(2B)

							Z t
	≤	1 2	log(|A|2 m T )	i∈At	1 N i,t-1	+ ∆ t /(2B).	¬D t

  The policy π described in Algorithm 14 has regret R T (π) bounded by

	256 log 2 2 (4 √	m)
		i∈[n]

B 2 βD i log(2 m |A|T ) ∆ i,min + ∆ max (1 + 2n)

  The policy clip cts-gaussian has regret bounded by ∆ max (1 + 5.2n) + nm 2 ∆ max ∆ min 2B -(m * (m * + 1)/2 + 1)ε

	2
	128 4 log 2 2 (4
	i∈[n]

√

m)βD i log(2 m |A|T ) ∧ mΓ ii (log(T ) + 4 log log(T ))

∆ i,min +

  -λt(A∩S)αt(A∩S) E e λt(A∩S) e A∩S (0∨(θt-µ t-1 )) 1 H t We now use the definition of µ t to have e T At∩S c (µ t-1 ∨ θ t ∧ µ t -µ t-1 ) ≤ e T At∩S c (µ t -µ t-1 )

					=
					i∈At∩S c
	α t (A)	2log(|A|2 m T )	i∈A	βD i N i,t-1	, and λ t (A)	α t (A)
	≤	e -λt(A∩S)αt(A∩S)		E e λt(A∩S)(0∨(θ i,t -µ i,t-1 )) H t
		A∈A		i∈A∩S
	≤	e -λt(A∩S)αt(A∩S)		E 1 + e λt(A∩S)(θ i,t -µ i,t-1 ) H t
		A∈A		i∈A∩S
	≤	e -λt(A∩S)αt(A∩S)		E 2e λt(A∩S)(θ i,t -µ i,t-1 ) H t
		A∈A		i∈A∩S
	≤

i∈A βD i /N i,t-1 , we have P e T At∩S (µ t-1 ∨ θ t ∧ µ t -µ t-1 ) ≥ α t (A t ∩ S) H t ≤ A∈A P[ e T A∩S (µ t-1 ∨ θ t -µ t-1 ) ≥ α t (A ∩ S) H t ] ≤ A∈A e A∈A

2 |A∩S| e -λt(A∩S)αt(A∩S) e λt(A∩S) 2 i∈A∩S βD i /(2N i,t-1 )

≤ 1/T .

  ∀A ∈ arg max ∧ θ t e Z + θ t e Z c implies Z ⊂ A, ∀A ∈ arg max This fact is from Lemma 21, with η = µ * ∧ θ t e Z + θ t e Z c and δ = θ t -µ * ∧ θ t e Z . Let η ∈ R n , δ ∈ R n

	e T
	A ∈A
	A ∈A	e T A θ t .
	Lemma 21.

A µ *

  Thus, we don't sum on s, and can use Assumption 7 with λ ∈ R n + . Proof of Lemma 20. It is sufficient to prove that Z t , ¬C t ⇒ ∃Z ⊂ A * , Z = ∅ s.t. S t (Z) holds, (8.10) because ¬C t and S t (Z) together imply T t (Z). Indeed, see that from ¬T t (Z), we can plug θ = µ * ∧ θ t into S t (Z) to get . To prove (8.10), we first consider the choice Z = Z 1 = A * . Two cases can be distinguished: 1a) ∀θ s.t. 0 ≤ (µ * -θ ) e A * ≤ εe A * , we have A * ⊂ A for any action A ∈ arg max A ∈A e T A θ e A * + θ t e A * c . 1b) ∃θ s.t. 0 ≤ (µ * -θ ) e A * ≤ εe A * such that A * ⊂ A for some action A ∈ arg max A ∈A e T A θ e A * + θ t e A * c . 1a) For the first case, consider any vector θ such that 0 ≤ (µ * -θ ) e A * Oracle θ e A * + θ t e A * c . We can write e T A θ e A * + θ t e A * c (8.13)≥ e T A * θ e A * + θ t e A * c -m * ε,where (8.13) is from (8.12), and (8.14) is from (8.11). This rewrites ase T A θ e A * + θ t e A * c ≥ e T A * µ * -m * ε > e T A * µ * -(m * (m * + 1)/2 + 1)ε, so R t (θ e A * + θ t e A * c, A * ) holds. Therefore, we have proved that S t (A * ) holds.1b) For the second case, we have some vector θ such that 0 , and some actionA ∈ arg max A ∈A e T A θ e A * + θ t e A * c such that A * ⊂ A. We consider Z 2 = A * ∩ A. We first prove that Z 2 = ∅ by showing that if an action S is such that S ∩ A * (8.17) = ∅, then A = S : e T S θe A * + θ t e A * c + θ t e A * c , where (8.18) is from (8.17), (8.19) is from the definition of A t , (8.20) is from ¬C t and (8.21) is from (8.16). Now, we again distinguish two cases: 2a) ∀θ s.t. 0 ≤(µ * -θ ) e Z 2 ≤ εe Z 2 , we have Z 2 ⊂ B for any action B ∈ arg max A ∈A e T A θ e Z 2 + θ t e Z 2 c . 2b) ∃θ s.t. 0 ≤ (µ * -θ ) e Z 2 ≤ εe Z 2 such that Z 2 ⊂ B for some action B ∈ arg max A ∈A e T A θ e Z 2 + θ t e Z 2 c . Notice that when 0 ≤ (µ * -θ ) e Z 2 + θ t e Z 2 c ≥ e T A θ e A * + θ t e A * c -(m * -1)ε. -1) + 0,where we used (8.22), (8.15) and that Z 2 is strictly included in A * .2a) For the first case, considering any vector θ such that 0 ≤ (µ * -θ ) e Z 2 ≤ εe Z 2 , we have with B = Oracle θ e Z 2 + θ t e Z 2 c that e T B θ e Z 2 + θ t e Z 2 c ≥ e T A θ e Z 2 + θ t e Z 2 c (8.24) ≥ e T A θ e A * + θ t e A * c -(m * -1)ε ≥ e T A * θ e A * + θ t e A * c -(m * -1)ε -1)ε, where (8.24) uses (8.23) and (8.25) uses (8.16). This rewrites as e T B θ e Z 2 + θ t e Z 2 c ≥ e T A * µ * -(m * (m * + 1)/2 + 1)ε, so R t (θ e Z 2 + θ t e Z 2 c , Z 2 ) holds, and thus we proved that S t (Z 2 ) holds. 2b) For the second case, we have a vector θ such that 0 ≤ (µ * -θ ) e Z 2 ≤ εe Z 2 and an action B ∈ arg max A ∈A e T A θ e Z 2 + θ t e Z 2 c such that Z 2 ⊂ B. We consider Z 3 = Z 2 ∩ B. Again, Z 3 = ∅ because for any S such that S ∩ Z 2 = ∅, we have S = Oracle θ e Z 2 + θ t e Z 2 c :

			(8.11)
			≤ εe A *
	and let A A e T (8.12) = (8.14) ≥ e T A (8.15) (8.25) ≥ e T
		(8.18) = e T S θ t	(8.19) ≤ e T At θ t
		(8.20) ≤ e T A (8.21) ≤ e T A (8.22) ≤ εe Z 2 , then
	e T A θ	e T At θ t = max A∈A e Z 2 (8.23) e T A θ t
	≥ max Indeed, (8.23) is a consequence of
	e T

A∈A e T

A θ e Z + θ t e Z c = e

T

Oracle θ e

Z + θt e Z c θ e Z + θ t e Z c > e T A * µ * -(m * (m * + 1)/2 + 1)ε, giving C t * µ * ≤ (µ * -θ ) e A * (8.16) ≤ εe A * * µ * -(m * (m * + 1)/2 + 1)ε < e T A * µ * -m * ε * θ e A * A θ e Z 2 + θ t e Z 2 c -θ e A * -θ t e A * c = e T Z 2 (θ -θ ) = e T Z 2 (θ -µ * ) + e T Z 2 (µ * -θ ) ≥ -ε(m * * µ * -m * ε -(m * S θ e Z 2 + θ t e Z 2 c = e T S θ t ≤ e T At θ t ≤ e T A * µ * -(m * (m * + 1)/2 + 1)ε < e T A * µ * -(m * + (m * -1))ε ≤ e T

  Titre: Apprentissage Efficient dans les Problèmes de Semi-Bandits Stochastiques Combinatoires Mots clés: Bandit combinatoire, régions de confiance, efficience computationnelle Résumé: Les problèmes de semi-bandits stochastiques combinatoires se présentent naturellement dans de nombreux contextes où le dilemme exploration/exploitation se pose, tels que l'optimisation de contenu web (recommandation/publicité en ligne) ou encore les méthodes de routage à trajet minimal. Ce problème est formulé de la manière suivante : un agent optimise séquentiellement une fonction objectif inconnue et bruitée, définie sur un ensemble puissance P([n]). Pour chaque ensemble A testé, l'agent subit une perte égale à l'écart espéré par rapport à la solution optimale tout en obtenant des observations lui permettant de réduire son incertitude sur les coordonnées de A. Notre objectif est d'étudier l'efficience des politiques pour ce problème, en nous intéressant notamment aux deux aspects suivants : l'efficience statistique, où le critère considéré est le regret subi par la politique (la perte cumulée) qui mesure la performance d'apprentissage ; et l'efficience computationnelle (i.e., de calcul). Il est parfois difficile de réunir ces deux aspects dans une seule politique. Dans cette thèse, nous proposons différentes directions pour améliorer l'efficience statistique, tout en essayant de maintenir l'efficience computationnelle des politiques. Nous avons notamment amélioré les méthodes optimistes en développant des algorithmes d'approximation et en affinant les régions de confiance utilisées. Nous avons également exploré une alternative aux méthodes optimistes, à savoir les méthodes randomisées, et avons constaté qu'elles constituent un candidat sérieux pour pouvoir réunir les deux types d'efficience.

Batailles de claque-doigts, tests des fréquences de résonance des salles, jeux de société, les anecdotes ne manquent pas !

Since each component is O( log(t)), the ∞-norm is so.

The standard scheduling notation[START_REF] Graham | Optimization and approximation in deterministic sequencing and scheduling: A survey[END_REF] denotes this setting as 1|prec| w j C j .

One can notice that extending this work to the general case can be possible considering an α-approximation regret.

i.e., w * belongs to the simplex of R n

A linear extension of a poset is a total ordering consistent with the poset, i.e., if a is before b in the poset, then the same has to be true for its linear extension.

Notice this is not exactly a monotonicity property, because we compare to a single point (w * , c * ).

One can see that {i,j}∈I(σ), i<j cs i cs j (λ(s i ) -λ(s j )) is the variation of d when swapping a linear order S by a permutation σ, where I(σ) the set of inversions in σ.

Theorem 18 does need this proposition.

The error δ is thus scaled by the standard deviation, of order 1/ √ n, giving a global error of O(n 1.5 δ). We therefore recover the factor n 1.5 given in Theorem 19.

When we select search S, all feedback W i , i ∈ S is received with probability 1, so triggering probabilities are not useful.

For recommender systems or search engines, w * i can thus be seen as the probability that a user aims to find i when entering the request.

Although the method presented is quite different from the classical "covering argument" used in linear bandits, we will still call this method so, in order to distinguish it from the Laplace method.

In streaming settings with near real-time requirements, it is imperative to have algorithms that can run with a complexity that stay almost constant across rounds.

C t is convex since functions g i,t are convex.

This relaxation is to avoid a computationally costly partial enumeration(Krause and Guestrin,

[START_REF] Fujishige | Submodular functions and optimization[END_REF][START_REF] Khuller | The budgeted maximum coverage problem[END_REF] 2 nodes that are reachable from some node in S.3 f is submodular if f (A ∪ {i}) -f (A) is non-increaing with A.

Although costs are usually deterministic, we assume randomness for more generality (influencer campaigns may have uncertain surcharges for example).

We provide a toy example where C 0 models a concrete quantity: Assume you want to fill your restaurant. You may pay some seeds and ask them to advertise/influence people. C 0 represents the cost of the food, the staff, the rent, the taxes, ...

We can first easily reduce to one variable, denoted c. The corresponding ratio is denoted rc(S) and its approximate maximizer is denoted Sc. For a fixed c, with a linear search, we can consider the algorithm outputting Sc = arg max S c rc(S c ). Then, c → Sc is monotone, so for a given m, we can find c s.t. either Sc = m or S c -< m < S c + . In this last case, we can build a randomized set of expected cardinality m.

In this case, using |S| rounds choosing each time a single different influencer i ∈ S is better than choosing the whole S in a single round.

Indeed, C + Γ ⇒ C ii ≤ κ

for all i ∈ [n] ⇒ C ij ≤ C ii C jj ≤ κ 2 for all i, j ∈ [n] ⇒ i,j C ij |λ i ||λ i | ≤ κ 2 i |λ i | 2 for all λ ∈ R n ⇒ C + Γ.

Remark that contrary to previous work on variance based confidence region, our method can't be easily generalized to Kullback-Leibler divergence based confidence region, since this would require control on higher moments of X.

For example, the Dirichlet-multinomial distribution with s trials is s-sparse.

Otherwise an approximation regret would be a more appropriate performance measure to consider.

We recall here the fact that in MAB, whether the horizon T is known or not is not really relevant as algorithms can be easily adaptedDegenne and Perchet,

2016a.

This D i can be computed whenever linear maximization on A is efficient: for x high enough, we have max A∈A, i∈A j∈AC ij = C ii -x + max A∈A j∈A C ij I{j = i} + xI{j = i} .

We use the version that relies on Fatou's lemma[START_REF] Durrett | Probability: theory and examples[END_REF], Theorem 5.7.6), so that it is not needed to have any additional condition on the stopping time τ * .

Acknowledgements -Remerciements

Chapter 7

Covariance-Adapting Policy

This chapter is based on our paper Perrault, Perchet, and Valko (2020). Its aim is to offer a more practical, and more accurate solution for building elliptical confidence region. Indeed, the ones we have seen so far are either based on the mutual independence of outcomes, or due to the unrealistic knowledge of a sub-Gaussianity matrix Γ. We alleviate this issue by instead considering a new general family of sub-exponential distributions, which contains bounded and Gaussian ones. We prove a new lower bound on the regret on this family, that is parameterized by the unknown covariance matrix, a tighter quantity than the sub-Gaussianity matrix. We then construct an algorithm that uses covariance estimates, and provide a tight asymptotic analysis of the regret. Finally, we apply and extend our results to the family of sparse outcomes, which has applications in many recommender systems.

An alternative to sub-Gaussian outcomes

Complete automatic adaptation of algorithms to the processed data, as opposed to the requirement of prior knowledge on underlying structure or to some manual tuning of parameters, is one of the fundamental challenges in machine learning. We address this challenge for stochastic (combinatorial) semi-bandits, and provide an algorithm adaptive to the correlation structure of the data, leading to provably faster learning in a sequential setting with limited feedback.

In MAB, there exist sophisticated learners adaptive to the environment, in the sense that their performance guarantees improve (or stated otherwise, their regret upper bounds decrease) when the problem instance is "simpler" for some appropriate notions of complexity. For instance, [START_REF] Audibert | Explorationexploitation trade-off using variance estimates in multi-armed bandits[END_REF] and [START_REF] Mukherjee | Efficient-UCBV: An Almost Optimal Algorithm using Variance Estimates[END_REF] proposed to estimate the variance of each arm to construct adaptive confidence intervals for each mean µ * i , based on Bernstein's inequality. This leads to an algorithm having variance-dependent regret bounds. [START_REF] Garivier | The {KL}-{UCB} algorithm for bounded stochastic bandits and beyond[END_REF] went beyond variance estimation and proposed a Kullback-Leibler divergence based confidence region, and provided a tighter regret upper bound. Thompson sampling can also offer such adaptive regret upper bounds [START_REF] Kaufmann | Thompson Sampling: An Asymptotically Optimal Finite Time Analysis[END_REF]. Our objective is to attain such adaptivity, but for the challenging semi-bandits setting.

In all this chapter, we consider a linear reward function. To recall, at each round t, the agent chooses some action A t ∈ A, receives the total reward associated to the selected actions A t , assumed to be i∈At X i,t , and observes the outcome of each base arm of A t , i.e., the vector (X i,t I{i ∈ A t }) i∈[n] . The action space A depends on the combinatorial problem at hand. For example, actions in A could be a path from an origin to a destination in a network [START_REF] György | The on-line shortest path problem under partial monitoring[END_REF][START_REF] Talebi | Stochastic Online Shortest Path Routing: The Value of Feedback[END_REF] or a subset of items to recommend to a customer [START_REF] Wang | Thompson Sampling for Contextual Combinatorial Bandits[END_REF]. Many other examples and applications are given by Cesa-Bianchi and Lugosi Chapter 8

Statistical and Computational Efficiency of Thompson Sampling

In this chapter, which is based on our article [START_REF] Perrault | Statistical Efficiency of Thompson Sampling for Combinatorial Semi-Bandits[END_REF], we study an interesting alternative to the optimistic methods considered so far, namely Thompson sampling (TS). Apart from its empirical superiority [START_REF] Chapelle | An empirical evaluation of Thompson sampling[END_REF], TS is interesting in the context of CMAB because the played action is easy to compute: there isn't an additive bonus to the main objective, contrary to the 2based approaches. TS could thus answer the question of the existence of an efficient policy with an optimal asymptotic regret (up to a factor poly-logarithmic with the action size), which is still open for many families of distributions, including mutually independent outcomes, and more generally the multivariate sub-Gaussian family. We propose to answer the above question for these two families by analyzing variants of the Combinatorial Thompson Sampling policy (cts). For mutually independent outcomes in [0, 1], we propose a tight analysis of cts using Beta priors. We then look at the more general setting of multivariate sub-Gaussian outcomes and propose a tight analysis of cts using Gaussian priors. This last result gives us an alternative to the Efficient Sampling for Combinatorial Bandit policy (escb), which, although optimal, is not computationally efficient.

A trade-off between optimality and computational efficiency?

In this chapter, we assume that the reward, given the choice of A t , is a function of µ * e At . As we already saw, the following two extreme problem instances are distinct within the CMAB framework:

(i) Each P X i is sub-Gaussian and the arm distributions are mutually independent, i.e., P X = ⊗ i∈[n] P X i .

(ii) Each P X i is sub-Gaussian but the stochastic dependencies between the arm distributions are "worst case": the performance metric is the supremum of the regret over all possible dependencies between the marginals.

Those two settings are indeed different as two different lower bounds on the asymptotic 1 (in T ) regret can be derived. In particular, the regret scales as Ω(n log(T )/∆) for the setting (i), and as Ω(mn log(T )/∆) for (ii), where ∆ is the minimum gap in Chapter 9

Conclusion and Perspectives

In this chapter, we conclude the thesis. In particular, we state the answers to the main research questions we raised, summarize the main contributions we have made and propose some directions for future research.

Conclusion

Let's remember that our main interest was to improve the efficiency of the policies, both from a statistical and computational point of view. We will therefore first look at how the thesis addressed this point.

Statistical point of view

The statistical improvement of policies has been made on several levels in the thesis, which can be summarized in the following three aspects:

• The improvement and extension of the regret analysis. This point has been particularly addressed in Chapter 3, where we gave several results widely usable in the CMAB-T setting in order to prove regret upper bounds. Going further, a take-home message can be drawn from this chapter, namely that CMAB-T problems are essentially no more difficult than CMAB ones, because we can convert the probability p i (S t ) of observing the arm i into I{i ∈ A t } using conditioning in the expectation of the regret. More specifically, in this chapter, we provided a simpler state-of-the-art 1 analysis under the 1 -norm triggering probability modulated condition. In addition, under the same condition, we also provided several 2 analyses that are the first of this kind in CMAB-T. These last results opened the way to the use of 2 bonuses in CMAB-T problems, such as the OIM problem, that we considered in Chapter 6.

• The improvement of confidence regions used in optimistic methods.

Apart from the improvement due to the shift from ∞ to 2 regions, the improvement of confidence regions was mainly explored in Chapters 4 and 7, where the use of auxiliary estimates helped to speed up the learning process of the mean. More precisely, in Chapter 4 for ∞ analysis, the use of the cucb-v method improved the regret rate by a factor n, leveraging on the 1-sparsity of the outcomes. We extended this type of improvement in Chapter 7 for 2 analysis, using the estimation of the entire covariance matrix to refine the confidence regions. We notably proved a tight covariance-dependent regret bound for the resulting escb-c policy, outperforming over ols-ucb. It should be noted in passing that the assumption we considered on the outcomes are more practical than the sub-Gaussian one. Finally, we were able to develop a tight approach to deal with the case of s-sparse outcomes.

considered in Chapter 6 also led us to formulate a new combinatorial optimization problem (in the same way, considering the budgeted version of an already known problem). This problem is the maximization of a ratio between a sub-modular function and a modular one. We contributed by improving the analysis of the greedy algorithm for this ratio maximization problem.

Perspectives

We have already mentioned some possible future works at the end of the corresponding chapters. Here, we detail those which seem to us the most relevant and promising with regard to our subject matter, i.e. those related to the cts policy.

Correlated Gaussian prior for cts In our approach, even though we have treated the general case of correlated sub-Gaussian outcomes, the prior we used in the cts policy is not (i.e., the components are independent Gaussian random variables). This independence has been very useful in the analysis, and an interesting open question is whether we can use a correlated prior. More specifically, under the sub-Gaussian assumption, we would like to use the prior

which is tighter than the following that we used

Apart from the gain in the empirical performance of the policy, there are some advantages to being able to conduct an analysis with the correlated prior (9.1). Indeed, it then allows us to conduct the Degenne and Perchet (2016b) analysis. This is a good point for the following reasons

• Admittedly, as we have seen, the bound obtained in the end would depend on the factor γ and is not as explicit as the one we have obtained (which depends on the whole covariance), but in cases close to the "arbitrarily correlated" case, the extra log 2 (m) is removed (by the way, an interesting question is how to associate both a dependence in the whole covariance and a suppression of this log 2 (m) factor when we approach the "arbitrarily correlated" case).

• We can still conduct the explicit analysis from (9.1), while returning to the Degenne and Perchet (2016b) analysis from (9.2) does not seem possible.

• This avoids having to calculate max A ∈A, i∈A j∈A

N i,t-1 , that may not be feasible efficiently. Actually, even if this is feasible, an extension that replaces Γ with a covariance estimation may raise some issues, as we will see in the next paragraph.

Covariance estimation for cts Another interesting question is whether an approach similar to the one presented in Chapter 7 can be used for cts. There may be more than one way of doing this (e.g., with a prior or with an optimistic estimator on the covariance), but we argue here that the use of an independent prior as in