A Multi-Armed Bandits (MAB) is a learning problem where an agent sequentially chooses an action among a given set of candidates, collects a reward, and implements a strategy in order to maximize her sum of reward. Motivated by a case study in agriculture, we tackle in this thesis several problems that are relevant towards real-world applications of MAB.

The first central question that we considered in this thesis is about the assumptions made on the distributions of rewards. While in theory it is usually convenient to consider simple parametric assumptions (e.g gaussian distributions), the practitioner may have some difficulty to find the right model fitting their problem. For this reason, we analyze two families of nonparametric algorithms, in the sense that they do not require strong parametric assumptions on the distributions for their implementation. We show that these two approaches can achieve strong theoretical guarantees in the standard bandit setting, improving what should be known in advance by the learner compared with previous algorithms.

Then, we analyze some extensions of these algorithms that make them more suitable for some real-world applications. A second focus of our work is to consider alternative performance metrics, that may be more suitable than the expected sum of rewards for the practitioner. We propose a risk-aware algorithm for a bandit problem where the learner wants to find a safe arm according to a risk metric: the Conditional-Value-at-Risk. We also propose efficient algorithms for a problem analogous to the limit case of this setting, known as Extreme Bandits. Finally, we also adapt some of our approaches for standard variant of MAB, including one with non-stationary rewards and one with feedback grouped into batches of observations. vi Characteristic 2: The outcome (reward) when applying a fixed crop-management policy is uncertain, and follows a probability distribution.

 can give us some intuitions. Harnessing more than 30 years of expert knowledge, this simulator is calibrated on historical field data (soil measurements, genetics, planting date...) and generates realistic crop yields. Such simulations can be used to explore crop management policies in silico before implementing them in the real world, where collecting enough data would take several years. Recently, Gautron et al. (2022b) implemented a Gym environment compatible with DSSAT, making its access easier for the ML community.

. The main remark that comes from this figure is that the distributions hardly fit the usual parametric models that are generally used in machine learning (e.g the distributions are gaussian): they are typically right-skewed, multimodal and exhibit a peak at zero corresponding to years of poor harvest. Hence, the question of the right assumptions we can make on the reward distributions is crucial to design our learning algorithm.

 we may prefer the distribution in salmon (last one) with an average yield of 3504 kg/ha and a relatively small probability of harvesting 0 kg/ha than the cyan distribution (3 rd on top) with an expected

Ce manuscrit vient clôturer trois années de doctorat inoubliables. Je remercie d'abord chaleureusement mes directeurs de thèse, Emilie et Odalric, pour cette expérience. Emilie, je te remercie de m'avoir fait confiance au tout début de cette aventure. J'ai appris énormément à tes côtés, et j'ai autant apprécié nos longs échanges sur tableau blanc que les moments de convivialité que nous avons été amenés à partager. Ta gentillesse, ta rigueur mathématique et ton sens de l'organisation sont un véritable modèle pour le jeune chercheur que je suis. Odalric, je te remercie également pour tout ce que tu m'as apporté, et pour m'avoir montré la voie des mathématiques artistiques. Tes connaissances encyclopédiques et ta créativité continueront également de m'inspirer pour longtemps. Je réalise la chance immense que j'ai eu d'être dirigé par deux brillants chercheurs, avec des visions très complémentaires de la recherche.

Au fil de mes travaux, j'ai également eu également la chance de pouvoir travailler avec d'excellents collaborateurs. Je remercie en premier lieu Yoan, que j'ai embarqué dans l'aventure SDA, et qui en plus d'être un ami cher s'avéra être un collègue brillant. Merci également à Romain pour m'avoir fait découvrir DSSAT lors d'un footing, et d'avoir conséquemment suscité un bon nombre des questions qui ont alimenté cette thèse. Merci enfin à tous mes collaborateurs pour votre talent et votre professionalisme: Marquinhos, Olivier, Patrick, Rémy et Rianne.

I also would like to thank a lot the members of Shimodaira lab in Kyoto University, for hosting me for these fantastic three months. Thanks a lot to Junya Honda for the warm welcome, and a very nice collaboration. I also want to address special thanks to Charles Riou and Taira Tsuchiya, who made me discover many aspects of the Japanese culture and with whom I shared great experiences and memories.

Je tiens évidemment à remercier l'ensemble de l'équipe ScooL (SequeL au début de ma thèse), pour m'avoir accueilli pendant ces trois années. Loin de l'image des chercheurs solitaires, j'ai découvert à ScooL une équipe dynamique et très soudée, ce qui semble avoir toujours fait l'identité de cette équipe. Je pense d'abord aux membres de la période pré-covid: merci de m'avoir intégré à l'équipe et d'avoir guidé mes premiers pas d'apprenti doctorant. Je n'oublierai jamais les discussions enflammées à la pause-café, le club ciné, les verres à la Capsule ou au Beerchoppe, ou encore les intenses parties d'Among Us. Je remercie particulièrement Mathieu d'être resté à Lille pendant le covid: notre amitié, notre cinéphilie, et le niveau de nos jam en sont sortis grandis. Cette période rendit également possible une expérience sociale inédite: la délocalisation de l'équipe ScooL à Varangeville sur mer en Normandie, lorsque des doctorants exténués par les confinements à répétition décidèrent d'aller vivre en communauté aux alentours de Dieppe. Je remercie chaleureusement Edouard pour nous avoir accueilli, et Ariane, Mathieu, Moulmoul, Nathan pour ces souvenirs inoubliables. L'année 2022 permit un retour à la vie normale, et apporta son lot de nouvelles personalités/activités au labo. Ainsi, la pétanque devint l'activité phare du groupe, avec notamment Marquinhos en star et Achraf en chaotique dégommeur, et une pratique tellement assidue que même l'hiver ne parvint pas à l'interrompre. Merci enfin à Hector pour nos discussions très sérieuses autour du foot ou du Muay Thaï.

Finalement, je tiens à remercier ma famille et mes amis, sans qui je n'aurais jamais pu accomplir tout cela. Merci à ma famille pour m'avoir inculqué le goût du savoir et du dépassement de soi. Je suis très heureux d'avoir pu vous rendre fier par mon parcours. Merci à ma soeur, Mélanie, d'avoir toujours été là pour moi. Merci également à mes amis, pour avoir fait partie de ma vie tout ce temps: William, chez qui une partie de ce rapport fut écrit entre deux virées à la plage; Filito et Filita, je remercie le destin de nous avoir réuni à Lille pendant ces 3 ans; Maxime, qui a contribué à m'inculquer l'amour du RC Lens; Abeba et Renan, les meilleurs colloc-zinhos de l'univers; et Estelle pour son brin de folie. Merci également aux coachs et adhérents de Team Naja de m'avoir rendu accroc à ce délicat sport qu'est le Muay Thaï; et à Seko Fofana et Florian Sotoca pour m'avoir fait vivre des émotions incomparables à Bollaert. Merci enfin à Noémie pour l'amour qu'elle m'a prodigué au quotidien durant toutes ces années. Vivre avec un chercheur n'est pas facile, et tu t'en es accommodé avec brio.

Résumé

Un bandit est un problème d'apprentissage dans lequel un agent choisit séquentiellement de tester une action parmi un ensemble de candidats fixé, collecte une récompense, et met en place une stratégie dans le but de maximiser son gain cumulé. Motivés par une étude de cas dans le domaine de l'agriculture, nous abordons dans cette thèse plusieurs problématiques pertinentes pour les applications réelles des bandits.

La première question que nous considérons concerne les hypothèses faites sur les distribution des récompenses. Alors qu'en théorie il est généralement commode de considérer des hypothèses paramétriques simples (par exemples, des distributions gaussiennes), le practicien peut avoir des difficultés à trouver un modèle adapté à son problème. Pour cette raison, nous étudions deux familles d'algorithmes non-paramétriques, dans la mesure où ils ne nécessitent pas d'hypothèses paramétriques fortes sur les distributions pour leur implémentation. Nous montrons que ces deux approches peuvent obtenir de bonnes garanties théoriques pour le problème de bandits usuel, tout en utilisant moins d'hypothèses que les méthodes précédemment proposées.

Nous proposons ensuite différentes extensions de ces algorithmes afin de faciliter leur mise en pratique. La deuxième question principalement étudiée dans nos travaux concerne la prise en compte de critères de performance alternatifs à l'espérance des récompenses cumulées, qui pourraient potentiellement mieux refléter les préférences réelles du practicien. Nous proposons notamment des algorithmes sensibles au risque, pour des problèmes dans lesquels l'objectif est d'identifier un bras peu risqué selon une mesure: la Conditional-Value-at-Risk. Nous proposons également des algorithmes efficaces pour un problème analogue au cas limite du précédent, appelé Bandits Extrêmes. Enfin, nous adaptons nos méthodes pour traiter des variantes usuelles du problème de bandit, avec notamment le cas de récompenses non-stationnaires et un exemple où les données sont collectées dans des groupes d'observations et non dans un cadre purement séquentiel.

Foreword From Machine Learning to Bandits

The works presented in this thesis are part of a vast branch of computer science called Machine Learning, that includes any algorithm that is able to improve on a task by analyzing and drawing inferences from patterns in data. We refer to [START_REF] Hastie | The Elements of Statistical Learning[END_REF] for a complete introduction of this field. In general the algorithm is presented some data, that have been collected beforehand, and then tries to learn to solve a task by training on this database. However, in some applications the data are collected during the training, and the algorithm needs to continuously learn from the new data feeding the database. It may even be directly responsible for the data collection process. In Reinforcement Learning (RL), the algorithm (called agent) interacts with their environment and learns to collect and maximize rewards. A complete introduction to RL can be found in [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF], some examples of applications include interactive speaker recognition [START_REF] Seurin | A Machine of Few Words Interactive Speaker Recognition with Reinforcement Learning[END_REF], or self-driving cars [START_REF] Leurent | Safe and Efficient Reinforcement Learning for Behavioural Planning in Autonomous Driving[END_REF].

In RL, the agent navigates in its environment, performs actions and observe a feedback (reward) associated with this action. The objective is to find the best action to do in every situation provided by the environment. When the agent repeatedly faces the same situation the problem becomes simpler, as it is reduced to the evaluation of each action: this is Multi-Armed Bandits (MAB). The name comes from slot machines (one-armed bandits): the agent pulls an arm, and observe a reward. Hence, each action is called an arm. The usual formulation of this problem can be traced back as far as [START_REF] Thompson | On the likelihood that one unknown probability exceeds another in view of the evidence of two samples[END_REF], and a very complete introduction to Bandits can be found in [START_REF] Lattimore | Bandit algorithms[END_REF]. While this setting is relatively simple, it is powerful to model a variety of problems. Hence, Multi-Armed Bandits are still a very active research field. All the works presented in this thesis fit into this theoretical framework, and in the next section we detail a case-study that motivated our research.

Motivating application : a recommendation algorithm for agriculture

While the works presented in this thesis are mostly theoretical, many questions that we considered during these three years of research came after discussing with my supervisors and my fellow PhD student Romain Gautron about the applications of bandits for a recommendation problem in agriculture. Detailing this problem is a natural introduction for this manuscript for several reasons: it is in our opinion an interesting illustrative example of the potential applications of bandits in the real world, it is relatively easy to understand for people who are not familiar with bandits, and it allows us to introduce the main research directions that we considered in this thesis. As this manuscript is written from a mathematician's perspective we only introduce the aspects of the problem that raise the theoretical questions considered in this thesis. For an agronomic point of view on this problem we refer to (Gautron et al., 2022a).

Experimental set-up Farmers have been reported to primarily seek advice that reduces uncertainty in highly uncertain decision making environment [START_REF] Mccown | Changing systems for supporting farmers' decisions: problems, paradigms, and prospects[END_REF][START_REF] Hochman | Emerging consensus on desirable characteristics of tools to support farmers' management of climate risk in australia[END_REF][START_REF] Evans | From data to decisions: helping crop producers build their actionable knowledge[END_REF]. Let us consider a group of farmers who would like to collectively learn to improve their crop management practices for a rain-fed crop. In the context of increasing global food demand, the experiment is aimed toward small farmers under challenging weather and soil conditions, such as maize farmers in Sub-Saharian Africa. The global objective of the experiment is to help farmers find good crop-management practices, while not putting their own food security at risk. We assume that some experts can provide a learning algorithm that would meet these objectives, and that during several consecutive seasons some volunteers in the group are willing to follow the suggestions of this learning system. In the following, we try to define some of the most important aspects that need to be considered to design such algorithm, and leading to the questions that we tried to tackle during this thesis.

We can first elaborate on what defines a crop management practice. In this experiment the crop species and type of soils are fixed for the group of farmers, which still lets several factors to optimize: the planting date, that impacts the weather during crop growth, or the fertilization policy (quantity, planning, . . . ). Furthermore, all these elements can be defined as a set of rules to follow, allowing some adaptation to external events (e.g weather conditions). A crop management policy is then simply defined as the combination of all these choices (or rules). All the parameters have a direct influence on the efficiency of the policy, with very intricate effects. Hence, trying to optimize all of them at the same time is a very complex task since there is a large number of possibilities. For this reason, we propose to simplify this problem by leveraging existing expert knowledge on the field, by asking experts to design a set of reasonable crop management policies to try. For each policy the rules are set in advance, and the farmers will follow them each time the policy is tried. From the point of view of the learning system, which will be our point of view for the rest of this manuscript, all crop-management policies are in fact considered as black-boxes. We assume that the joint impact of each part of the process may be very hard to model accurately, and that the quantity of data we could collect in real life may not be enough to learn a complex dynamic. We instead assume that we are given a set of policies to try, and that we do not have enough knowledge to model their relationship. We can then define more precisely the objective of the algorithm. Ultimately, it would be to find the policy that would be the best according to the farmers' needs. However, experiments are costly: each bad trial is potentially deleterious to the farmer at all steps of the learning process. Hence, we would like to make farmers try "good" policies most of the time. Summarizing the properties we established in this first part, we obtain the first important characteristic that should be satisfied by a learning system.

Characteristic 1:

The learning algorithm's objective is to recommend most often the best policy among a finite set of policies provided by experts, during the whole learning phase.

Evaluation of the policies

In order to characterize our problem the next step is to define how to compare policies and determine which one is the best. We first need to consider the output obtained after a farmer applies a recommendation: at the end of the season, the crops are harvested and the farmer can observe a realized yield for this season. The yield is simply the ratio of the quantity of crop harvested divided by the field surface (e.g in kg/ha). More sophisticated criterion can be used, for instance by taking into account the economic and environmental performance of the fertilization policy. In any case, we consider that each recommendation followed by the farmer leads to an observable numeric output at the end of the season, that we call a reward. We also assume that the higher a reward is, the better it is for the farmer. The main challenge for the learning strategy is that this output depends on many factors that are external to crop-management: weather conditions during the crop growing process (rainfalls, temperature,. . . ), potential diseases, pests, extreme climate events, crops genetics . . . making the outcome very uncertain. Hence, for a fixed policy we can assume that the rewards follow a probability distribution, and thus each policy needs to be tried many times as they can all provide both good and bad outcomes.

Chapter 1

Introduction to some Bandit Problems

In this chapter we introduce the theoretical formalism associated with the different questions raised by the recommendation problem in agriculture that we introduced in the preliminary part of this thesis. After defining the proper mathematical formalism, we provide an overview of existing works in each of these domains. We also explain how our contributions fit into the context of existing research.

Introduction to some Bandit Problems

Stochastic Multi-Armed Bandits

A Multi-Armed Bandit (MAB) is a sequential decision-making problem in which a learner (or bandit algorithm) sequentially samples from K unknown distributions called arms. In each successive round the learner chooses an arm A t ∈ {1, . . . , K} and obtains a random reward X t drawn from the distribution of the chosen arm. The choice of arm A t depends on the strategy of the learner, that is based on the past observations H t = (A 1 , y 1 , . . . , A t-1 , X t-1 ). In the standard formulation of the bandit problem this strategy aims at maximizing the expected sum of rewards obtained after a time horizon T . This is equivalent to minimizing the regret, defined as the difference between the expected total reward of an oracle strategy always selecting the arm with largest mean and the expected total reward of our strategy. In the rest of this manuscript we denote by (ν k ) k∈{1,...,K} the distributions of the arms, (µ 1 , . . . , µ K ) their means, and assume that the arms belong to a family of distributions F. Definition 1. 1 (Regret). Consider a policy π and a bandit problem ν = (ν 1 , . . . , ν K ). Using the notation µ ⋆ = max k∈{1,...,K} µ k , the regret after T rounds is

R ν (T, π) = µ ⋆ T -E ν,π T t=1 X t = E ν,π T t=1 (µ ⋆ -µ At ) .
From that definition, it is clear that a policy that minimizes the regret needs to sample as often as possible arms with means that are close to µ ⋆ . An arm k with mean µ k < µ ⋆ is said to be sub-optimal, and we call the quantity ∆ k = µ ⋆ -µ k the sub-optimality gap (often referred to as gap for simplicity). We further define the number of pulls of any arm k ∈ {1, . . . , K} at time T as N k (T ) = T t=1 1(A t = k), which is simply the number of rounds at which arm k has been selected by the algorithm and pulled. Using these two quantities, we can conveniently rewrite the regret as

R ν (T, π) = E ν,π T t=1 (µ ⋆ - K k=1 µ k 1(A t = k)) = K k=1 ∆ k E ν,π [N k (T )] .
( 1.1) Thanks to this equation, it appears clearly that an algorithm with small regret has to minimize the expected number of pulls of each sub-optimal arm. To do that, it needs to balance exploration (gaining information about arms that have not been sampled a lot) and exploitation (select arms that look promising based on the available information). Before elaborating on the many approaches that have been proposed to solve this problem, it is interesting to have a look on the theoretical guarantees that are achievable by a bandit algorithm. To reformulate, we want to know what is the smallest expected number of pulls of a sub-optimal arm an 1.1 Stochastic Multi-Armed Bandits algorithm can obtain for a specific class of problem F, that we also call family of distributions.

In the standard bandit setting the answer to this question has been known for a long time already, with a first lower bound provided by [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] for parametric families of distributions. Before stating their result, we recall the definition of the Kullback-Leibler divergence between two distributions ν 1 and ν 2 , where ν 2 is absolutely continuous with respect to ν 1 :

KL(ν 1 , ν 2 ) = E ν 1 log dν 1 dν 2
Lemma 1.2 (Lai & Robbins lower bound [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF]). Assume that the distributions ν = (ν 1 , . . . , ν K ) ⊂ F K are continuously parameterized by their means. Then under any uniformly efficient strategy π, that is any π satisfying R ν (T, π) = o(T α ) for any α > 0 and any ν, the number of pulls of any sub-optimal arm k satisfies lim inf .2) where kl(µ, µ ′ ) is the Kullback-Leibler divergence between the distribution of mean µ and that of mean µ ′ in the considered family of distributions.

T →∞ E ν,π [N k (T )] log(T ) ≥ 1 kl(µ k , µ ⋆ ) , ( 1
This result applies to several families of distributions that are widely used in practice, such as Gaussian (with shared variance), Bernoulli, Poisson or Exponential distributions. More generally, the result holds if the distributions come from a single-parameter exponential family of distributions (SPEF). A family F is a SPEF if there exists a parameter set Θ and some functions A : R → R and b : Θ → R such that for any distribution ν ∈ F, there exists a parameter θ ∈ Θ such that ν (that we denote then by ν θ ) satisfies

dν θ dη (x) = exp(θx -b(θ)) ,
for any x, and where η is a reference measure. Some properties of these families of distributions can be found in [START_REF] Cappé | Kullback-leibler upper confidence bounds for optimal sequential allocation[END_REF], among many other resources. For instance, the mean of ν is then equal to b ′ (θ) and the KL divergence between the distribution of parameter θ 1 and the one of parameter θ 2 is equal to

kl(b ′ (θ 1 ), b ′ (θ 2 )) = (θ 1 -θ 2 )b ′ (θ 1 ) + b(θ 2 ) -b(θ 1 ) .
Back to bandit algorithms, the lower bound in Equation ( 1.2) teaches us that a reasonable strategy can expect to pull the sub-optimal arms a logarithmic number of times, and that it is impossible to obtain a better constant before the logarithm than the one in Equation 1. 2. In the first case we simply say that the algorithm has a logarithmic regret, while an algorithm matching Introduction to some Bandit Problems this lower bound 1 is said to be asymptotically optimal. Lemma 1.2 has later been extended by [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF] for any (possibly non-parametric) family of distribution F. Lemma 1.3 (Burnetas & Katehakis lower bound [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF]). Consider a bandit ν = (ν 1 , . . . , ν K ) ⊂ F K . Under any uniformly efficient algorithm the number of pulls of any sub-optimal arm k must satisfy lim inf .3) where

T →∞ E ν,π [N k (T )] log(T ) ≥ 1 K F inf (ν k , µ ⋆ ) , ( 1
K F inf (ν k , µ ⋆ ) = inf G∈F {KL(ν k , G) : E G (X) > µ ⋆ }.
Again, any algorithm matching this lower bound is said to be asymptotically optimal for the family F. A very common example of such non-parametric family is the set of distributions with supports admitting a known upper bound B. In that case, the K F inf has been studied in depth by [START_REF] Honda | An Asymptotically Optimal Bandit Algorithm for Bounded Support Models[END_REF], who provide its dual form and use it to derive an asymptotically optimal strategy. Remark 1. 4 (Other notions of optimality). In this thesis we study the optimality of bandit algorithms in terms of problem-dependent guarantees. Other kind of results are studied in the literature, such as minimax (worst-case) optimality (see for instance Chapters 9 and 15 of [START_REF] Lattimore | Bandit algorithms[END_REF]). These bounds are typically in O( √ KT ), and the constant does not depend on the instance of the problem.

Notably, a line of work on best-of both worlds consists in deriving algorithms that would achieve simultaneously optimal problem-dependent regret for stochastic bandits and optimal regret for adversarial bandits [START_REF] Zimmert | Tsallis-inf: An optimal algorithm for stochastic and adversarial bandits[END_REF][START_REF] Ito | Adversarially robust multi-armed bandit algorithm with variancedependent regret bounds[END_REF], but with a weaker notion of problem-dependent optimality in the stochastic case. These work are out of the scope of this thesis.

A non-exhaustive overview of multi-armed bandit algorithms

Multi-Armed Bandits have been a very active research field in the past years, and summarizing all contributions to this domain would be a tremendous task. For this reason we detail in this section a selection of bandit algorithms, with a special focus on the ones that are asymptotically optimal. We refer the reader to e.g [START_REF] Lattimore | Bandit algorithms[END_REF] for a broader survey of this research area. Motivated by the third characteristic that we raised in the use-case considered in the introduction we are specifically interested by the following questions: what type of 1 i.e that satisfies lim infT →∞ Eν,π [N k (T )] log(T ) ≤ 1 kl(µ k ,µ⋆)

Stochastic Multi-Armed Bandits

approaches can lead to optimal algorithms? For what type of distributions do they achieve optimality? What kind of prior knowledge on the distribution do they require to be optimal? Before detailing the main families of algorithms that can be found in the literature we define an index policy, which is a generic name for a bandit algorithm that (1) computes a quantity (index) for each arm using past observations for this arm only, and (2) pulls the arm with the largest index. We detail this principle in Algorithm 1.1, that will be useful in the next paragraphs since many bandit algorithms are index policies.

Introduction to some Bandit Problems

Introduction to some Bandit Problems

Historically, TS was first studied for Bernoulli distributions since the application considered in [START_REF] Thompson | On the likelihood that one unknown probability exceeds another in view of the evidence of two samples[END_REF] was clinical trials. In this setting TS using a Beta-Bernoulli conjugate prior is asymptotically optimal, which was proved much later by (Agrawal and Goyal, 2013a;[START_REF] Kaufmann | Thompson sampling: An asymptotically optimal finite-time analysis[END_REF]. This result was then extended to general SPEF in [START_REF] Korda | Thompson Sampling for 1-dimensional Exponential family bandits[END_REF], using Jeffreys prior. Intuitively, the algorithm works because: (1) in the first rounds the prior distribution allows to explore each arm a sufficient number of times, and (2) when arms are sampled enough the posteriors are concentrated around the true means so that the algorithm will mostly exploit the best arm.

Recently, the principle of TS has been extended to obtain an optimal algorithm for bounded distributions with known upper bound: Non-Parametric Thompson Sampling (NPTS) [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF]. The idea of the authors is to build on the Beta-Bernoulli Thompson sampling. They first extend it for Multinomial distributions, where the Beta-Bernoulli prior/posteriors are naturally replaced by Dirichlet distributions (see Appendix 5.7 for a short summary of the properties of this distribution). The resulting Multinomial TS algorithm is asymptotically optimal when the arms have a known finite support. Then, applying the same mechanism for general bounded distributions leads to using a sample from a Dirichlet distribution of parameter (1, . . . , 1) to re-weight the observations. We denote this distribution by D n for ndimensional weights. This procedure is actually known as the Bayesian Bootstrap, introduced by [START_REF] Rubin | The bayesian bootstrap[END_REF]. To ensure sufficient exploration the known upper bound of the support is added to the set of observations at the beginning of the experiment. We summarize NPTS in Algorithm 1. 6. [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF] The fact that NPTS is asymptotically optimal is noteworthy, because contrarily to KL-UCB it does not require to explicitly compute K F inf . This function naturally appears in the concentration inequalities related to the Dirichlet distribution. This remarkable feature made us consider this algorithm more thoroughly, and Part II of this thesis is dedicated to extensions of NPTS.

Re-Sampling algorithms

In the past years, there has been a surge of interest for the design of non-parametric algorithms that would perturb the empirical distribution of the data instead of trying to fit it in an already defined model, and are therefore good candidates for the problem in agriculture that we are considering. A first line of works explored re-sampling schemes [START_REF] Efron | An introduction to the bootstrap[END_REF] to balance exploration and exploitation [START_REF] Osband | Bootstrapped thompson sampling and deep exploration[END_REF]Kveton 1.1 Stochastic Multi-Armed Bandits et al., 2019a,b;[START_REF] Wang | Residual bootstrap exploration for bandit algorithms[END_REF]. The idea is to compute the mean of a noisy version of the empirical distribution by, for instance, drawing random weights for each of their observations. In (Kveton et al., 2019b) the authors propose the term of General Randomized Exploration for index policies satisfying this principle.

In fact, Thompson Sampling algorithms are part of the GRE framework, but this family contains also non-Bayesian algorithms such as the non-parametric bootstrap (sampling with replacement) presented in Algorithm 1.7.

1 Input: Data Y = (y 1 , . . . , y n ) 2 Return: I B = 1 n n i=1 z i , where ∀i, z i is drawn uniformly at random in Y Algorithm 1.7: Index based on sampling with replacement Unfortunately, re-sampling in itself is not enough: an index policy based on Algorithm 1.7 trivially fails to achieve sub-linear regret in some cases. The question is then to determine both what kind of re-sampling procedure and/or what kind of modifications to the empirical data are needed to make this method work. In (Kveton et al., 2019a,b), the authors propose to perturb the empirical mean by adding fake rewards in the history (by either directly adding them or sampling them at each step). They prove that when appropriately tuned these algorithms can achieve logarithmic regret for bounded distributions. Interestingly, for the gaussian/subgaussian case [START_REF] Wang | Residual bootstrap exploration for bandit algorithms[END_REF] proposed an algorithm with gaussian weights, that also achieves logarithmic regret. So far NPTS [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF] is the only algorithm using this principle that achieve optimality in its setting, using the Bayesian Bootstrap. An interesting open question is to determine if non-Bayesian algorithms based on bootstrapping could achieve optimal regret.

Intuitively, the reason for the failure of the non-parametric bootstrap is the potential underestimation of a good arm due to bad rewards in the first draws. To circumvent this issue, TS uses a convenient prior distribution while the re-sampling based algorithms introduce additional data points to improve exploration (e.g fake good samples). Interestingly, another class of algorithms consider an alternative by instead penalizing the arms that have been pulled the most so far, and hence do not require any exploration bonus: these algorithms are based on sub-sampling. This idea was introduced in [START_REF] Baransi | Sub-sampling for multi-armed bandits[END_REF] with Best Empirical Sample Average (BESA). It relies on pairwise comparisons between arms: the arm was has been less pulled (challenger) uses its empirical mean, while the other arm uses the empirical mean of a sub-sample of its history of the same size as the challenger's. The core idea is that if the two means are computed with the same number of samples the comparison is "fair". In BESA, the sub-sample is drawn using sampling without replacement. We summarize this comparison step in Algorithm 1.8.

Introduction to some Bandit Problems

1 Input: Arm 1 with X = (x 1 , . . . , x [START_REF] Baransi | Sub-sampling for multi-armed bandits[END_REF] When there are only 2 arms, at each round the arm winning the comparison is pulled. With more arms, [START_REF] Baransi | Sub-sampling for multi-armed bandits[END_REF] propose to organize a tournament: in successive rounds half of the arms is eliminated until only 1 arm remains. Unfortunately, the tournament is hard to analyze and [START_REF] Baransi | Sub-sampling for multi-armed bandits[END_REF] prove logarithmic regret only for K = 2. Furthermore, some of the assumptions they required may not be valid in general. However, the general idea of comparing sub-samples of the same size is interesting and differs from all existing approaches. More recently, Chan (2020) further explored this idea by proposing the Sub-Sample Mean Comparison (SSMC) algorithm. To analyze the algorithm for K > 2 arms, the author propose a more convenient leader vs challenger approach: the arm that has been pulled the most so far is defined as leader, and then competes with every other arm (called challengers) in pairwise comparisons. Then, any winning challenger or the leader (if none) is pulled. Another difference with BESA is that this time the sub-sample used is not random: it is the worst sequence of successive observations (in order of collection). Hence, the mean of this subsample is intuitively an empirical lower bound of the true mean of the leader. We summarize the comparison step of SSMC in Algorithm 1.9 1 Input: Arm 1 with X = (x 1 , . . . , x n ), arm 2 with Y = (y 1 , . . . , y m ), n ≥ m 2 Compute µ y = 1 m m i=1 y i 3 Compute µ x = min j∈{1,. ..,n-m+1} 1 m m-1 i=0 x j+i 4 Return: argmax x,y {µ x , µ y } ; ▷ Return winning arm Algorithm 1.9: pairwise comparison in SSMC (Chan, 2020) The strength of these algorithms is that they do not use any information on the arm's distributions. While the performance of BESA is not clear, in (Chan, 2020) the authors prove that SSMC is asymptotically optimal when arms come from the same SPEF. This is actually a very strong result, since the identity of the SPEF does not have to be specified by the learner. In the works presented in this thesis we aim at extending these results and bridging the gap between BESA and SSMC.

Introduction to some Bandit Problems

the SPEF to be optimal. Hence, we naturally studied extensions of these two algorithms to consider the question:

How can we design algorithms with the best theoretical guarantees with a minimum knowledge on the distributions?.

The first part of this thesis is dedicated to sub-sampling algorithms, inspired by SSMC and BESA, while in the second part we study some generalizations of NPTS. Regarding the standard regret minimization problem, in Chapter 2 we propose a family of algorithms called Sub-Sampling Dueling Algorithm that is optimal for SPEF, but also achieves logarithmic regret on a broader class of distributions that we further characterize. In particular, we analyze the assumptions that are needed to obtain at least a logarithmic regret in a more general setting than the ones with fully-parametric assumption. Then, in Chapter 6 we study a generalization of Non-Parametric Thompson Sampling outside the family of bounded distributions with a known upper bound, from alternative families of bounded distributions to general light-tailed distributions. For the Dirichlet Sampling algorithm that we propose, we found a trade-off between the theoretical guarantees (optimal, logarithmic, or superlogarithmic regret) and the generality of the family of distributions considered, that can be resolved by the practitioner depending on the knowledge available on the distributions.

Bandits with alternative performance metrics

Over the past few years, a number of works have focused on adapting multi-armed bandit strategies to optimize another criterion than the expected cumulative reward. Indeed, in a large number of application domains (healthcare, agriculture, marketing,...), one needs to take into account personalized preferences of the practitioner that are not captured by the expected reward. For example, in the preamble of this thesis we introduced a crop-management policy recommendation problem: small farmers are typically risk-averse as their harvest is necessary to ensure the subsistence of their household.

Assume that the learner decides to evaluate a bandit algorithm with a metric U . Then, consider two datasets of n collected points X = (X 1 , . . . , X n ) and Y = (Y 1 , . . . , Y n ) and assume that U can return a numeric value for U (X ) and U (Y) for any value of n. We say that the trajectory X is better than Y with respect to U if U (X ) > U (Y). This is sufficient to propose a natural adaptation of the expected regret for an alternative performance metric U .

Bandits with alternative performance metrics

Definition 1.6 (U-regret). Consider a metric U that can return a numeric value for any trajectory of observations Y T = (Y 1 , . . . , Y T ) for any time horizon T , and that larger values of U are preferred by the learner. Then, the U -regret can be defined as

R U ν (T, π) = E ν,π ⋆ [U (Y T )] -E ν,π [U (Y T )] ,
where π ⋆ is the oracle optimal policy, π

⋆ = argmax π E ν,π [U (Y T )].
Sometimes the performance metric can only be defined for a sample X 1 , . . . , X n (e.g extreme values, range), but in the majority of cases it can be defined more generally for a probability distribution. Being able to compute the value of a performance metric for general distributions is interesting because it allows us to evaluate the arms of a bandit problem of distributions (ν 1 , . . . , ν K ). We choose to refer to the distributions through their cumulative distribution functions (cdf) F 1 , . . . , F K , and to use the notation U (F ) (where F is a cdf). Then, we can compare the quality of the arms by simply computing U (F 1 ), . . . , U (F K ). This allows us to define the sub-optimality gap of an arm in an analogous way as for the standard expected regret. Definition 1.7 (Sub-optimality gaps in terms of metric U). Consider a bandit ν = (ν 1 , . . . , ν K ) ⊂ F K with respective cdf F 1 , . . . , F K . Consider a metric U : F → R, then the U-gap of any arm k ∈ {1, . . . , K} is defined as

∆ U k = max j∈{1,...,K} U (F j ) -U (F k )
Interestingly, when U is not linear (as the expectation) the U-regret of Definition 1.6 may not be convenient to obtain theoretical analysis of bandit strategies. In that case, it may be relevant to consider an alternative regret definition inspired by Equation (1.1) and involving the sub-optimality gaps, called proxy regrey.

Definition 1.8 (U-proxy regret). Consider a bandit

ν = (ν 1 , . . . , ν K ) ⊂ F K with respective cdf F 1 , . . . , F K . Denote their respective U-gaps by ∆ U 1 , . . . , ∆ U k , then the U -proxy regret is defined as R U,proxy ν (T, π) = T t=1 ∆ U k E ν,π [N k (T )]
Introduction to some Bandit Problems Remark 1.9. This definition of the proxy regret is different from the one in [START_REF] Cassel | A general approach to multi-armed bandits under risk criteria[END_REF], that consider another definition of sub-optimality gaps. However, as regret proofs for the proxy regret consist in upper bounding the expected number of pulls of sub-optimal arms the guarantees obtained with one definition easily translate to the other.

Risk Metrics

In statistics, a natural way to analyze a distribution is to compute its moments. The standard bandit theory is based on evaluating the expectation of the arms, but one could want to consider moments of higher order. For example some criteria aim at penalizing arms with high variance, such as the mean-variance [START_REF] Markowitz | Portfolio selection*[END_REF] and the sharpe ratio [START_REF] Sharpe | The sharpe ratio[END_REF]. The first one is a linear combination of the two measures, while the second is the ratio of mean and standard deviation. Both are widely used in finance due to their simplicity. However, if the distributions have a complex shape the mean and variance may not capture well their risk profile. It may be interesting to consider instead a quantile, or Value-at-Risk (VaR). Hence, we denote by VaR α the quantile of order α ∈ [0, 1]. In order to better capture the behavior on the tail of the distribution, one can further consider the Conditional-Value-at-Risk [START_REF] Artzner | Coherent measures of risk[END_REF][START_REF] Rockafellar | Optimization of conditional value-at-risk[END_REF]. Several definitions of the CVaR exist in the literature, depending on whether the samples are considered as losses or as rewards. We consider the reward version: given a level α ∈ (0, 1], the CVaR α is easily interpretable as the expected reward in the worst α-fraction of the outcomes. It can hence capture different preferences: α = 1 simply provides the expectation of the distribution, which expresses risk-neutrality, while making α very small corresponds to maximizing the expected reward in the worst-case scenarios. We illustrate this in Figure 1.3. CVaR is further a coherent spectral measure in the sense of [START_REF] Rockafellar | Optimization of conditional value-at-risk[END_REF], see [START_REF] Acerbi | On the coherence of expected shortfall[END_REF]). This definition entails some properties that are generally considered as desirable for a risk metric. Other general families of risk measures have been defined, such as spectral risk measures (SRM) [START_REF] Acerbi | On the coherence of expected shortfall[END_REF], and cumulative prospect theory (CPT) [START_REF] Tversky | Advances in prospect theory: Cumulative representation of uncertainty[END_REF]. Finally, Entropic Risk is also a way to penalize distributions with heavy tails. We summarize in Table 1.2 some of the measures introduced in this paragraph with their proper definition (for a distribution of CDF F ) and parameters.

We see that the risk metrics generally require to choose a parameter, that needs to be fixed in advance by the learner to model its level of risk-aversion. In the next paragraph we introduce the literature considering risk metrics to evaluate the performance of bandit algorithms.

Risk-aware bandits At a high level, the multi-armed bandit literature considering risk metrics is largely based on adapting the popular Upper Confidence Bounds (UCB) algorithms (Auer et al. (2002a)), and is hence mainly focused on deriving appropriate concentration tools for the risk metric under consideration. The first line of works on this topic followed this principle, and considered variations of the mean-variance criterion [START_REF] Sani | Risk-aversion in multi-armed bandits[END_REF][START_REF] Vakili | Mean-variance and value at risk in multi-armed bandit problems[END_REF][START_REF] Vakili | Risk-averse multi-armed bandit problems under mean-variance measure[END_REF][START_REF] Zimin | Generalized risk-aversion in stochastic multi-armed bandits[END_REF]. [START_REF] Szorenyi | Qualitative multi-armed bandits: A quantilebased approach[END_REF] study algorithms for the quantile (Value-at-Risk) criterion in both the regret minimization and the pure exploration setting, while [START_REF] David | Pure exploration for max-quantile bandits[END_REF][START_REF] Zhang | Quantile bandits for best arms identification[END_REF] investigate the second problem only. Interestingly, the algorithm proposed in [START_REF] Szorenyi | Qualitative multi-armed bandits: A quantilebased approach[END_REF] for regret minimization implements optimism for VaR by comparing the arms using rank statistics of larger order than the one corresponding to the target quantile. [START_REF] Maillard | Robust risk-averse stochastic multi-armed bandits[END_REF] focuses on the Entropic Risk and extend the KL-UCB algorithm of [START_REF] Cappé | Kullback-leibler upper confidence bounds for optimal sequential allocation[END_REF] for this risk metric under the assumption that the distributions are bounded with a known upper bound.

Bandits with alternative performance metrics

More recently the Conditional Value at Risk (CVaR) have received specific attention from the bandit community [START_REF] Galichet | Exploration vs exploitation vs safety: Risk-aware multi-armed bandits[END_REF][START_REF] Galichet | Contributions to multi-armed bandits: Risk-awareness and sub-sampling for linear contextual bandits[END_REF][START_REF] Tamkin | Distributionally-aware exploration for cvar bandits[END_REF][START_REF] Prashanth | Concentration bounds for cvar estimation: The cases of light-tailed and heavy-tailed distributions[END_REF] to cite a few). These works focus on proving or refining concentration inequalities for CVaR under different assumptions on the distributions (e.g as those of [START_REF] Brown | Large deviations bounds for estimating conditional value-at-risk[END_REF]; [START_REF] Thomas | Concentration inequalities for conditional value at risk[END_REF]; [START_REF] Prashanth | Concentration bounds for cvar estimation: The cases of light-tailed and heavy-tailed distributions[END_REF]; [START_REF] Holland | Learning with cvar-based feedback under potentially heavy tails[END_REF]; Bhat and L.A. (2019a)), and analyzing the corresponding bandit algorithm. Interestingly, [START_REF] Tamkin | Distributionally-aware exploration for cvar bandits[END_REF] exhibits two possible approaches to implement optimism for CVaR bandits: adding directly an exploration bonus to the empirical CVaR as in MaRaB [START_REF] Galichet | Exploration vs exploitation vs safety: Risk-aware multi-armed bandits[END_REF][START_REF] Galichet | Contributions to multi-armed bandits: Risk-awareness and sub-sampling for linear contextual bandits[END_REF], U-UCB [START_REF] Cassel | A general approach to multi-armed bandits under risk criteria[END_REF] or Brown-UCB [START_REF] Brown | Large deviations bounds for estimating conditional value-at-risk[END_REF][START_REF] Tamkin | Distributionally-aware exploration for cvar bandits[END_REF]; or exploiting the link between the CVaR and the CDF to build an optimistic CDF as in CVaR-UCB [START_REF] Tamkin | Distributionally-aware exploration for cvar bandits[END_REF], resorting to the celebrated Dvoretzky-Kiefer-Wolfowitz (DKW) concentration inequality (see [START_REF] Massart | The tight constant in the dvoretzky-kiefer-wolfowitz inequality[END_REF]). While the two approaches are equivalent for α = 1 (mean setting), the

Introduction to some Bandit Problems

Metric

Definition Parameters

Expectation E F [X] = xdF Variance V F [X] = σ F (X) = (X -E F [X]) 2 dF Mean-Variance MV ρ = E F [X] -ρV F [X] scaling ρ Sharpe ratio SR r 0 = E F [X]-r 0 σ F (X) reference value r 0 Value-at-Risk VaR α = sup{x ∈ R : F (x) ≤ α} quantile level α Conditional Value-at-Risk CVaR α = sup x∈R x -1 α (x -X) + dF quantile level α Entropic Risk ER θ = -1 θ log ( exp(-θx)dF ) risk level θ Spectral Risk SRM ϕ = 1 0 ϕ(β)F -1 (β)dβ risk spectrum ϕ
empirical results from [START_REF] Tamkin | Distributionally-aware exploration for cvar bandits[END_REF] suggest that the second method should be preferred for α < 1. Our empirical results in Chapter 5 confirm these findings.

Notably, [START_REF] Cassel | A general approach to multi-armed bandits under risk criteria[END_REF] provided a unified UCB1-like algorithm for various risk metrics called U-UCB. They show that if the risk-metric is quasi-convex2 and is a strongly stable performance metric the regret of an appropriately tuned U-UCB is logarithmic. The tuning depends on the metric U , but also on the class of distributions considered, and scales in log(T )/N k (t) in the case where the metric is Lipschitz. Strong stability is defined by the two following properties (from [START_REF] Cassel | A general approach to multi-armed bandits under risk criteria[END_REF]).

Definition 1.10 (Strongly stable performance metric [START_REF] Cassel | A general approach to multi-armed bandits under risk criteria[END_REF]). A metric is said to be strongly stable if: 1. There exists b > 0, q > 1 and a seminorm ||.|| such that on F extended to the set of empirical distributions it holds that

|U (F ) -U (G)| ≤ b × (||F -G|| + ||F -G|| q ) .
2. There exists a > 0 such that for any F ∈ F and x > 0

P (||F n -F || ≥ x) ≤ 2 exp(-anx 2 ) ,
where F n denotes an empirical distribution corresponding to n samples drawn from F .

Bandits with alternative performance metrics

For appropriate sets of distributions (e.g bounded), all the risk metrics we mentioned in this section can satisfy this requirement. Hence, the analysis of U-UCB is very general. Investigating if other families of algorithms could achieve this level of generality is an interesting perspective. However, investigating other methods that would be tailored for a specific problem is also still interesting since (1) the generality of U-UCB may have a cost on the empirical performance (the confidence bound may not be tight), and (2) U-UCB adapts UCB1, that is known to perform worse in practice than asymptotically optimal algorithms like TS or KL-UCB for the standard bandit problem.

Introduction to some Bandit Problems

bound of [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF] for CVaR bandits in this setting, establishing the guarantees that should be achieve by asymptotically optimal algorithms. Then, we extended the Non-Parametric Thompson Sampling algorithm under the two assumptions considered: multinomial distributions (M-CVTS) and bounded distributions with known support (B-CVTS). We prove that both algorithms are asymptotically optimal for the settings they consider, which is the first result of this type for risk-averse bandits. Finally we show empirically the benefits of the TS approaches over UCB-based algorithms in practice, on problems using both synthetic data and the DSSAT simulator.

Extreme Bandits

In the previous paragraphs we introduced the vast literature on risk-aware bandits, and in particular CVaR bandits that is the topic of Chapter 5. We explained that the CVaR relies on a quantile level α ∈ (0, 1], where a small α allows to model risk-averse preferences. The question we can ask is: what happens if we actually want to set α = 0? Intuitively, this would correspond to an extremely risk-averse learner, wanting to avoid as much as possible worst case scenarii. While the CVaR is not defined for α = 0, we can instead turn to extreme statistics, by trying for instance to maximize the expected minimum value collected by the bandit algorithm during a trajectory. A similar problem has been introduced in the literature by [START_REF] Cicirello | The max k-armed bandit: A new model of exploration applied to search heuristic selection[END_REF], and is known as Extreme Bandits. In this setting, the learner's objective is simply to collect the largest possible reward. While the objectives are rather different, the theoretical problems faced when trying to maximize the expected minimum or maximum are very similar. Hence, the algorithms proposed in the literature for Extreme Bandits could also be applied in our "extremely risk-averse" setting. For this reason in the following we introduce the literature associated with the Extreme Bandit problem, even if the risk-averse objective actually suits better our ideas of applications.

Letting X k,t be the reward obtained from arm k at time t, a bandit algorithm selects an arm I t using past observations and receives the reward X It,t . The rewards stream (X k,t ) is drawn i.i.d. from ν k and independently from other rewards streams. The case where distributions have bounded support is studied by [START_REF] Nishihara | No regret bound for extreme bandits[END_REF], so in the following we assume that the distributions have unbounded supports. To evaluate an extreme bandit algorithm, [START_REF] Carpentier | Extreme bandits[END_REF] propose an adaptation of the regret called extreme regret that fits Definition 1.6 for a performance metric U that returns the maximum of a set of observations, .4)

R π T = max k≤K E[max t≤T X k,t ] -E π [max t≤T X It,t ] . ( 1

Bandits with alternative performance metrics

This problem is hard because we are only interested in the asymptotic behavior of the right tail of the distribution: an arm could be the best for extreme bandits while providing very bad reward 99% of the time, so intuitively most observations may be useless (or even misleading in some sense) to the algorithm. Furthermore, the optimal policy may change according to the time horizon. Consider a simple example with two gaussian distributions N (1, 1) and N (0, 1.7): for short time horizons the first arm should be preferred, for example with T = 1 the arm with largest expectation should be preferred. On the other hand, for large time horizons the arm with the largest variance will provide the best expected maximum. We illustrate this in Figure 1.4 below. Expected maximum with (1,1) Expected maximum with (0, 1.7) In this simple example there exists some time horizon T 0 for which the optimal policy changes. For this reason, we will consider asymptotic performance guarantees with the assumption that one arm asymptotically dominates the others (i.e the optimal policy can only change a finite number of times). Two types of performance guarantees have been derived in previous works. Using the terminology of [START_REF] Bhatt | Extreme bandits using robust statistics[END_REF], we introduce these two definitions below. .5) and π has a vanishing regret in the strong sense if .6)

Definition 1.11. An Extreme Bandit algorithm π has a vanishing regret in the weak sense if

R π T = o T →∞ max k≤K E[max t≤T X k,t ] (1
lim T →∞ R π T = 0 . ( 1

Introduction to some Bandit Problems

It is now understood that the peculiarities of the Extreme Bandits setting make the algorithms designed for the K-arm setting suboptimal. For this reason a line of works have designed algorithms specifically for this setting. Furthermore, the question of the assumptions that are made on the distributions needs to be considered. Existing algorithms for this problem can be divided into three categories:

1. Fully-parametric approaches [START_REF] Cicirello | The max k-armed bandit: A new model of exploration applied to search heuristic selection[END_REF]Streeter and Smith, 2006a) where the family of distributions is assumed to be known (e.g Frechet, Gumbel).

2. Semi-parametric approaches: [START_REF] Carpentier | Extreme bandits[END_REF][START_REF] Achab | Max k-armed bandit: On the extremehunter algorithm and beyond[END_REF] consider a setting where distributions satisfy a second-order Pareto assumption.

3. Distribution-free approaches (Streeter and Smith, 2006b;[START_REF] Bhatt | Extreme bandits using robust statistics[END_REF], that do not leverage any assumption on the reward distributions. Assumptions are only required for the analysis of the algorithms.

The fully-parametric setting is not so different from the standard bandit, since the problem is reduced to using the estimated parameters to balance exploration and exploitation. The semi-parametric setting allows more flexibility. To illustrate this, we detail the second-order Pareto assumption used in [START_REF] Carpentier | Extreme bandits[END_REF][START_REF] Achab | Max k-armed bandit: On the extremehunter algorithm and beyond[END_REF]. Definition 1.12 (Second order Pareto (definition 2 in [START_REF] Carpentier | Extreme bandits[END_REF]

)). A distribution F is se(α, β, C, C ′ )-second order Pareto if for x ≥ 0: |P F (X ≥ x) -Cx -α | ≤ C ′ x -α(1+β) , which implies that P F (X ≥ x) = Cx -α + O(x -α(1+β) ).
With this definition the tail of the distribution is asymptotically very close to the tail of a Pareto distribution, and the parameter β controls the deviation wrt this asymptotic equivalent. In [START_REF] Carpentier | Extreme bandits[END_REF], weakly vanishing regret is obtained under this assumption, further assuming that a lower bound on a parameter of the distribution is known to the algorithm. [START_REF] Achab | Max k-armed bandit: On the extremehunter algorithm and beyond[END_REF] refined this analysis and obtained strongly vanishing regret when this lower-bound is large enough. However, their approach still relies on the estimation of the parameters. The variant compared with the fully-parametric approaches is that a decreasing fraction of the samples (depending on the lower bound on β) is used to perform this estimation.

Regarding non-parametric algorithms, a first algorithm inspired by UCB1, ThresholdAscent, was proposed in (Streeter and Smith, 2006b). The principle is to compute the mean of a fixed number of the largest samples obtain for an arm, with an exploration bonus. Unfortunately, this very simple algorithm do not have theoretical guarantees, and its performance is very

Non-stationary Bandits

sensitive to the number of observations kept for the computation. [START_REF] Bhatt | Extreme bandits using robust statistics[END_REF] recently proposed Max-Median, an algorithm based on robust statistics that can be employed for any kind of distribution. Max-Median is proved to have weak vanishing regret for polynomial-like arms and strongly vanishing regret for exponential-like arms.

Contribution

The recent work of [START_REF] Bhatt | Extreme bandits using robust statistics[END_REF] suggests that non-parametric approaches could be an efficient way to tackle the Extreme Bandit problem. This can be performed by finding a right way to compare the arms instead of individually estimating some parameters. In Chapter 4, we propose a novel algorithm for Extreme Bandits based on sub-sampling. This algorithm extends the Sub-Sampling Dueling Algorithm introduced in Chapter 2, comparing arms with a robust estimator. We establish weakly vanishing regret for non-parametric distributions, assuming only that one tail dominates the others (we properly define this notion in the chapter), and refine these results for polynomial and exponential arms. We believe these guarantees to be the most general obtained so far for this problem.

Non-stationary Bandits

In the previous sections we considered bandit problems where the arms do not change during the experiment. However, in many practical application they are likely to evolve: for example, if we consider again the crop-management problem we can assume that if the experience lasts for several years then climate change may have an impact on the reward distribution associated with each policy.

In the following we get back to the standard setting where the learner aims at maximizing her expected sum of rewards, as in Section 1.1, and introduce the different algorithms that have been developed in the literature to handle non-stationary rewards. Before that we define the dynamic regret, that naturally adapts Definition 1.1.

Definition 1.13 (Dynamic regret).

For a policy π, the dynamic regret after T rounds is defined as

R ν (T, π) = T t=1 max k∈{1,...,K} µ k,t -E ν,π T t=1 X t ,
where ν = (ν k,t ) k∈{1, ...,K},t∈{1,...,T } denotes the set of distributions corresponding to all arms at each time step, and µ k,t denotes the mean of ν k,t (arm k at time t).

Introduction to some Bandit Problems

The question is then : can a bandit algorithm learn to minimize the dynamic regret? If the distributions can change completely arbitrarily, this task seems clearly impossible: it is necessary to make some assumptions on the non stationarity. A natural idea consists in fitting a stochastic model on the dynamic, as for instance [START_REF] Whittle | Restless bandits: Activity allocation in a changing world[END_REF] that consider a markovian dynamic for the arms. Under this assumption, changes can then be predicted by learning the model. However, in many applications the learner may not have access to such knowledge and have to rely on simpler assumptions. One of the most famous is assuming an abruptly changing (or piece-wise stationary) environment: the distributions are assumed to be stationary between breakpoints, which are the time instants when they can change. This number has to be limited to allow the strategy to learn between the changes, and hence the number of breakpoints (denoted by Γ T ) is assumed to be sub-linear in the time-horizon.

Definition 1.14 (Number of breakpoints). The number of break-points in a piece-wise stationary model is formally defined as

Γ T = T -1 t=1 1(∃k ∈ {1, . . . , K} : µ k,t ̸ = µ k,t+1 ) ,
where the times

T = (t 1 , t 2 , . . . , t Γ T ) := {t ∈ {1, . . . , T } : 1(∃k ∈ {1, . . . , K} : µ k,t ̸ = µ k,t+1 )}
are called the breakpoints and a time interval

[t i + 1, t i+1 ] is called a stationary phase.
Intuitively, the existence of these breakpoints forces the bandit algorithms to explore much more than in the standard stochastic setting. Indeed, a strategy with a regret in O(log(T )) only marginally explores arms that have been observed as sub-optimal after a sufficient number of pulls. As breakpoints can occur at any time, an algorithm evolving in a piecewise-stationary environment has to try regularly all arms to check if they have changed or not. This intuition is formalized by a lower bound on the dynamic regret, first proved in [START_REF] Garivier | On upper-confidence bound policies for switching bandit problems[END_REF].

Theorem 1.15 (Theorem 31.2 in Lattimore and Szepesvári (2020)). Let k = 2, and fix ∆ ∈ (0, 1) and a policy π. Let µ be so that µ i,t = µ i is constant for both arms and

∆ = µ 1 -µ 2 > 0. If the expected regret R ν (T, π) of policy π on bandit µ satisfies R ν (T, π) = o(T )
, then for all sufficiently large T , there exists a non-stationary bandit ν ′ with at most two change points and

min t∈[T ] |µ ′ 1,t -µ ′ 2,t | ≥ ∆ such that R ν (T, π) ≥ T 22R ν ′ (T, π)
.

Non-stationary Bandits

This result shows that no strategy can hope to obtain a better regret than Ω( √ T ) for this class of problems. In particular, it shows that the standard bandit strategies with O(log(T )) regret in the stationary case will fail to achieve a sub-linear regret on all instances. The lower bound was later refined by [START_REF] Seznec | A single algorithm for both restless and rested rotting bandits[END_REF] who included the dependency in the number of arms and number of breakpoints.

Theorem 1.16 (Proposition 4 in Seznec et al. (2020)). In a piece-wise stationary environment with at most Γ T breakpoints, there exists an environment such that

R(T, π) ≥ KT Γ T .
Hence, the best achievable worst-case guarantee that can be expected from a bandit algorithm in an abruptly changing environment is of order O( √ KT Γ T )

An alternative way to model non-stationarity is to consider possibly smoother changes by introducing a variation budget [START_REF] Besbes | Stochastic multi-armed-bandit problem with non-stationary rewards[END_REF] that controls the total amplitude of arms' changes.

Definition 1.17. Define a bandit ν = (µ k,t ) k∈{1,. ..,K},t∈{1,...,T } , its variation budget is defined as

B T := T -1 t=1 max k∈{1,...,K} |µ k,t+1 -µ k,t | .
Contrarily to the piece-wise stationary model, this setting can allow for instance small changes at any time step. A special case of slowly drifting environment, where the amplitude of changes is controlled, was introduced recently in (Krishnamurthy and Gopalan, 2021). [START_REF] Besbes | Stochastic multi-armed-bandit problem with non-stationary rewards[END_REF] also proved a lower bound for the regret for the variation budget setting. 2014)). Assume that rewards have a Bernoulli distribution. Then, there is some absolute constant C > 0 such that for any policy π and for any

T ≥ 1, K ≥ 2 and B T ∈ [1/K, T /K], the regret satisfies R ν (T, π) ≥ C(KB T ) 1/3 T 2/3 .
This result shows that a sub-linear regret is possible only if the variation budget is sublinear. We can also remark that the regret is in Ω(T 2/3 ) instead of Ω( √ T ) for the piece-wise Introduction to some Bandit Problems stationary model. Now that we introduced those two main assumptions we can detail the different families of algorithms that have been proposed in the literature.

Algorithms for non-stationary bandits

During the past ten years, several works have considered non-stationary variants of the multiarmed bandit model, proposing methods that can be grouped into two main categories: they passively forget past information [START_REF] Garivier | On upper-confidence bound policies for switching bandit problems[END_REF][START_REF] Raj | Taming non-stationary bandits: A bayesian approach[END_REF][START_REF] Trovo | Sliding-window thompson sampling for non-stationary settings[END_REF], or actively try to detect modifications in the distribution of the arms with changepoint detection algorithms [START_REF] Liu | A change-detection based framework for piecewise-stationary multi-armed bandit problem[END_REF][START_REF] Cao | Nearly optimal adaptive procedure with change detection for piecewise-stationary bandit[END_REF][START_REF] Auer | Adaptively tracking the best bandit arm with an unknown number of distribution changes[END_REF][START_REF] Chen | A new algorithm for non-stationary contextual bandits: Efficient, optimal and parameter-free[END_REF][START_REF] Besson | Efficient change-point detection for tackling piecewise-stationary bandits[END_REF].

Passively forgetting strategy A natural idea is to consider adaptations of standard bandit algorithms with simple mechanism to forget past data. The two standard idea for that are the use of a sliding window or discounted rewards. In the first case all the algorithm works with only the most recent collected rewards: if the window size is τ and rewards collected are denoted by X 1 , . . . , X T then only rewards X T -τ +1 , . . . , X T are used. In the second case all rewards are kept but the oldest rewards are discounted in order to have a reduced impact: using a fixed discount rate γ, at time T the reward collected at time t is associated with a discount γ T -t . In [START_REF] Garivier | On upper-confidence bound policies for switching bandit problems[END_REF] the authors analyze two variants of UCB1: SW-UCB and D-UCB, respectively implementing UCB1 with a sliding window and discounted rewards. The algorithms are proved to achieve a O( KT Γ T log(T )) dynamic regret in abruptly changing environment, with an appropriate tuning of the window/discount factor requiring the knowledge of the order of Γ T . Similarly the celebrated Thompson Sampling algorithm [START_REF] Thompson | On the likelihood that one unknown probability exceeds another in view of the evidence of two samples[END_REF] has also been adapted to include these mechanisms, with the Discounted Thompson Sampling (DTS) [START_REF] Raj | Taming non-stationary bandits: A bayesian approach[END_REF] and the Sliding Window Thompson Sampling (SW-TS) [START_REF] Trovo | Sliding-window thompson sampling for non-stationary settings[END_REF], though theoretical guarantees have been obtained only for SW-TS. In Algorithm 1.11 and 1.12, we respectively provide one time step of the generic adaptation of any bandit algorithm for stationary rewards with a sliding window and a discount factor.

For example, we could naturally adapt KL-UCB [START_REF] Cappé | Kullback-leibler upper confidence bounds for optimal sequential allocation[END_REF] and IMED [START_REF] Honda | Non-asymptotic analysis of a new bandit algorithm for semi-bounded rewards[END_REF] in the discounted computing the divergence function of an arm k with n rewards y 1 , . . . , y n with the discounted empirical distribution

F k (t) : x → n i=1 ρ i 1(y i ≤
x), where the weights ρ 1 , . . . , ρ n are computed following Algorithm 1.12 (and normalized) according to the time where all rewards have been collected.

Another line of work inspired by adversarial bandits have been studied for non stationary settings. In short, in adversarial bandits the rewards are not drawn from a probability distributions but can be any arbitrary sequence: in this harder setting an adversary can decide the next rewards, and hence try to confuse the algorithm. However, a usual assumption is that

Non-stationary Bandits

1 Input: Sliding window τ , rewards X = (X 1 , . the rewards are chosen independently of the bandit algorithm: the adversary is said to be non-adaptive. In that case, the objective of the learner is to sample most often the action with the best trajectory of rewards for the time horizon T considered: the best arm in hindsight. To make it possible to solve this problem rewards are usually assumed to be bounded in [0, 1]. As the adversarial setting is very general the algorithms proposed for this problem can be used to tackle non-stationary rewards, even if their guarantees are defined according to the best arm in hindsight and not the dynamic regret. For instance, the EXP3.S algorithm (Auer et al., 2002b) can be used to benchmark non-stationary bandit algorithms, and even has guarantees in the piecewise stationary setting. Furthermore, [START_REF] Besbes | Stochastic multi-armed-bandit problem with non-stationary rewards[END_REF] proposed the Rexp3 strategy, which is simply a combination of EXP3 (Auer et al., 2002b) and scheduled restarts of the algorithm. Interestingly, Rexp3 obtains an optimal worst-case regret for the variation budget setting when an the budget B T are known.

Change-point detection algorithms

The second main category of non-stationary bandit algorithms consist in combining a stochastic bandit algorithm with a change-point detector (CPD).

As the name states, a CPD scans the history of rewards collected to determine if a change of distributions occurred. If this is the case, the algorithm simply erases the history and restarts. Two changepoint detection algorithms, CUSUM [START_REF] Liu | A change-detection based framework for piecewise-stationary multi-armed bandit problem[END_REF] and M-UCB [START_REF] Cao | Nearly optimal adaptive procedure with change detection for piecewise-stationary bandit[END_REF], have been proposed using the standard UCB algorithm. They obtain optimal guarantees for respectively Bernoulli and bounded rewards in the piece-wise stationary environment, with some assumptions on the detectability of the changes and the knowledge of the number Introduction to some Bandit Problems of changes. Interestingly, a number of works have proposed algorithms that get rid of this assumption and are instead fully adaptative [START_REF] Chen | A new algorithm for non-stationary contextual bandits: Efficient, optimal and parameter-free[END_REF][START_REF] Auer | Adaptively tracking the best bandit arm with an unknown number of distribution changes[END_REF][START_REF] Besson | Efficient change-point detection for tackling piecewise-stationary bandits[END_REF]. For example, [START_REF] Besson | Efficient change-point detection for tackling piecewise-stationary bandits[END_REF] consider a GLR test combined with the KL-UCB algorithm (GLR-KL-UCB) and an exploration scheme depending on the number of changes already detected. GLR-KL-UCB achieves optimal guarantees in the abruptly changing setting under a detectability assumption. On the other hand, [START_REF] Chen | A new algorithm for non-stationary contextual bandits: Efficient, optimal and parameter-free[END_REF][START_REF] Auer | Adaptively tracking the best bandit arm with an unknown number of distribution changes[END_REF] propose algorithms based on elimination rules of empirically sub-optimal arms and scheduled replay phases for the eliminated arms. ADSWITCH [START_REF] Auer | Adaptively tracking the best bandit arm with an unknown number of distribution changes[END_REF] is optimal in abruptly changing environment, while ADA-ILTCB + is optimal in both abruptly changing and variation budget settings.

Recently, some works have tried to improve the practical performance of CPD-based algorithm by considering most significant changes instead of restarting every time a change is detected. Indeed, intuitively if the distributions change but the best arm remains the same we do not want our algorithm to restart completely. For this reason, considering a number of breakpoints or a variation budget may be too conservative with respect to the true complexity of the problem. For instance, [START_REF] Manegueu | Generalized non-stationary bandits[END_REF] proposed a change point algorithm based on the empirical gaps between the arms. [START_REF] Suk | Tracking most significant arm switches in bandits[END_REF] further extended this idea by quantifying and trying to detect significant shifts at each step of the algorithm. They also remarkably avoid using knowledge on the non-stationarity by using an elaborate method to re-explore sub-optimal arms often enough, and obtain optimal guarantees for both piecewise-stationary and variation budget assumptions. Notably, an independent and parallel work of [START_REF] Abbasi-Yadkori | A new look at dynamic regret for non-stationary stochastic bandits[END_REF] also obtained comparable (but slightly weaker) guarantees in the abruptly changing setting with similar ideas.

Contribution

The family of non-stationary bandit algorithms is now very dense, and under the assumption that rewards are bounded the practitioner can choose between many options. This choice can depend on the complexity of the algorithms (the ones with the best guarantees are unfortunately difficult to implement), their theoretical guarantees, and the assumptions they need on the non-stationarity structure. To the best of our knowledge, no work have tried to combine a mechanism for non-stationarity and a non-parametric algorithm, able to tackle more structural changes. We study this in Chapter 3, with the combination of a simple sliding window mechanism and the Last-Block Sub-sampling Dueling Algorithm (LB-SDA) introduced in Chapter 2. We show that LB-SDA can achieve optimal guarantees in the abruptly changing environment when the order of the number of breakpoints is known. The main advantage of the resulting SW-LB-SDA algorithm is that these guarantees allow more general changes of distributions than previous approaches.

Batch Bandits

Batch Bandits

The formulation of batch bandits can be traced back to [START_REF] Perchet | Batched bandit problems[END_REF], and this problem have been studied for instance in [START_REF] Gao | Batched multi-armed bandits problem[END_REF][START_REF] Esfandiari | Regret bounds for batched bandits[END_REF][START_REF] Jin | Almost optimal anytime algorithm for batched multi-armed bandits[END_REF]). Outside of the example that we introduced in agriculture, clinical trials is a typical example where this setting applies: patients come in cohorts, and the practitioner analyzes the results for each cohort before considering the next one. The main questions considered in this literature are to determine the number of batches, and the size of each batch necessary to obtain theoretical guarantees that are as close as possible as the ones obtained in the purely sequential setting. For example, [START_REF] Jin | Almost optimal anytime algorithm for batched multi-armed bandits[END_REF] consider grids of exponentially increasing size and obtain a logarithmic regret (in the total number of trials) with a batch size of order Ω(log log(T )). [START_REF] Kalkanli | Batched thompson sampling[END_REF] analyze Thompson Sampling for Gaussian rewards coupled with an algorithm that defines the size of each batch accordingly to the previous plays of TS. They prove a logarithmic regret and an expected number of batches of O(log log T ). While this line of work is interesting it does not exactly correspond to the experiment introduced in the preamble of this thesis. Indeed, in our crop-management problem the number of farmers is fixed at the beginning of the experiment and we cannot decide how many farmers will participate at each season.

We propose in Theorem 5.11 an analysis of the B-CVTS algorithm introduced in Chapter 5 in the batch setting, showing no performance loss compared with the purely sequential setting. This analysis shows that this algorithm is suitable for the experiment in agriculture introduced at the beginning of this thesis.

Introduction to some Bandit Problems 1.5 Outline and Contributions

Thesis organization

After this introductory chapter, the thesis is divided in two parts corresponding to the two families of algorithms that we studied. Each part provides an initial version of the algorithm for the standard MAB setting, and then details their extensions to some of the variants of MAB introduced in the previous sections.

The first part introduces the family of Sub-Sampling Dueling Algorithms (SDA), which is a novel family of non-parametric bandit algorithms relying on fair pairwise comparisons between arms using sub-samples to penalize the arms that have been explored the most.

• In Chapter 2 we introduce SDA in the standard MAB setting. We present both randomized (RB-SDA, WR-SDA) and deterministic (LB-SDA) sub-sampling mechanisms. We prove logarithmic regret for both RB-SDA and LB-SDA under some conditions on the arms' distributions. In particular, we prove that for standard parametric assumptions (e.g Bernoulli, Gaussian with shared variance) these algorithms are even asymptotically optimal. This is interesting because these guarantees hold without the algorithm using any prior knowledge of what distributions it is facing. We further discuss the merits of each algorithm from a practical point of view.

• In Chapter 3 we extend the LB-SDA algorithm to tackle non-stationary rewards using a sliding window and some additional mechanisms, which yields the SW-LB-SDA algorithm. We prove that in abruptly changing environments the window can be tuned efficiently to make the algorithm optimal, which is on par with similar strategies in the literature. However, the non-parametric nature of SW-LB-SDA allows to obtain these guarantees while allowing a potentially broader class of possible distribution changes.

• In Chapter 4 we prove that combining LB-SDA with a robust estimator based on the upper tail of distributions can lead to efficient algorithms for the Extreme Bandit problem. This setting is an example of bandits with alternative performance metric, as in that case the learner wants to collect the largest possible reward. We illustrate the interest of non-parametric approaches in this setting, where the learner would like to make as little assumptions as possible on the tails of the distributions.

In the second part of this thesis we study algorithms inspired by the Non-Parametric Thompson Sampling (NPTS) algorithm of [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF].

• In Chapter 5 we propose an extension of NPTS for CVaR bandit under the name CVaR Thompson Sampling (CVTS), with one algorithm for multinomial and one for bounded distributions. After extending the notion of asymptotic optimality for CVaR bandits, we

Outline and Contributions

prove that the resulting M-CVTS and B-CVTS algorithms are the first algorithms to be asymptotically optimal in a risk-aware setting. We further propose an empirical evaluation of B-CVTS using the DSSAT simulator, that allows to emulate the recommendation problem in agriculture introduced in the foreword of this thesis.

List of publications

This section references the works that have been published during my PhD thesis. They are the fruit of research projects that were conducted with several collaborators, including my supervisors, other PhD students, and several researchers. They are ordered according to their date of publication, from the oldest to the most recent.

Publications in international conferences with proceedings

Sub-Sampling Dueling Algorithms

In this chapter we propose a new family of bandit algorithms based on sub-sampling, called

Sub-sampling Dueling Algorithms (SDA). Unlike most existing approaches these algorithms do not use any knowledge on the arms' distributions. Still, we prove that some instances SDA can achieve strong theoretical guarantees under some assumptions that we detail in this chapter.

In particular, they are asymptotically optimal when distributions come from the same Single-Parameter Exponential Family, including some of the most common distributions encountered in practice. After introducing this novel family of algorithms, we analyze some instances of SDA and highlight the core properties that make this strategy work. Finally, we perform an experimental study assessing the flexibility and robustness of this promising novel approach for exploration in bandit models. This chapter unifies the results that were published in [START_REF] Baudry | Sub-sampling for efficient non-parametric bandit exploration[END_REF] and [START_REF] Baudry | On Limited-Memory Subsampling Strategies for Bandits[END_REF], respectively for the RB-SDA and LB-SDA algorithms, and provides additional intuitions on this family of algorithms.

Sub-Sampling Dueling Algorithms

Introduction

In this chapter we consider the standard Multi-Armed Bandit problem introduced in Chapter 1 (Section 1.1). We recall that a K-armed bandit is a sequential decision-making problem in which a learner sequentially samples from K unknown distributions, called arms. In each round the learner chooses an arm A t ∈ {1, . . . , K} and obtains a random reward X t drawn from the distribution of the chosen arm, that has mean µ At . The learner should adjust her sequential sampling strategy π (or bandit algorithm) in order to maximize the expected sum of rewards obtained after T selections. This is equivalent to minimizing the regret, defined as

R ν (T, π) = max k∈{1,...,K} µ k T -E T t=1 X t = E T t=1 max k∈{1,...,K} µ k -µ At ; ,
An algorithm with small regret needs to balance exploration (gain information about arms that have not been sampled a lot) and exploitation (select arms that look promising based on the available information). In Chapter 1 we introduced the standard approaches used in bandits. We also detailed the lower bounds of [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF][START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF], that set the target performance that we would like to obtain with a bandit algorithm. We further discussed that most existing algorithms require knowledge on the arms' distribution, e.g to calibrate confidence bounds or use appropriate conjugate prior/posteriors with Bayesian methods to reach these guarantees. The main question we study in this chapter is:

Under which conditions can we achieve strong theoretical guarantees with an algorithm that do not use any knowledge on the arms' distributions?

A recent family of algorithms based on sub-sampling [START_REF] Baransi | Sub-sampling for multi-armed bandits[END_REF]Chan, 2020), introduced in Section 1.1, seems promising to tackle this question. The Sub-sampling Dueling Algorithms (SDA) that we propose in this chapter combine an algorithm structure based on pairwise comparisons (that we call duels), inspired by SSMC, and generic sub-sampling schemes. Our objective is to bridge the gap between BESA [START_REF] Baransi | Sub-sampling for multi-armed bandits[END_REF] and SSMC (Chan, 2020) by allowing both randomized and deterministic sub-sampling schemes in the same framework.

The rest of the chapter is structured as follows. In Section 2.2 we introduce the generic principle of SDA and present different instances corresponding to possible choices for the sub-sampling algorithms (that we also call samplers). In Section 2.3 we present general theoretical results for SDA. We provide a first regret upper bound for a category of sub-sampling algorithms that we call Block Samplers, only assuming that the empirical means concentrate with an exponential decay for each arm. This bound exhibits two terms: a term in O(log(T )), and the sum of the probabilities that the optimal arm is under-sampled (i.e sampled less than some quantity). We then further study this second term, exhibiting some sufficient conditions

Sub-sampling Dueling Algorithms

on the sub-sampling algorithm and the family of distributions to obtain a logarithmic regret. In Section 2.4 we first show that the condition on the sampler is satisfied by at least two instances: RB-SDA and LB-SDA. We then propose a condition on the family of distributions ensuring that SDA is sufficient to avoid under-exploring the best arm. Finally, in Section 2.5 we present the results of an empirical study comparing several instances of SDA to asymptotically optimal parametric algorithms, and other algorithms based on re-sampling or sub-sampling. These experiments reveal the robustness of the SDA approaches, which match the performance of Thompson Sampling, without exploiting the knowledge of the distribution.

Sub-sampling Dueling Algorithms

In the following we define Sub-sampling Duelling Algorithms (SDA). We first introduce a few notation: for every integer n, we let [n] = {1, . . . , n}. We denote by (Y k,s ) s∈N the i.i.d. sequence of successive rewards from arm k, drawn from a distribution ν k with mean µ k . For every finite subset S of N, we denote by Ȳk,S the empirical mean of the observations of arm k indexed by S:

if |S| > 1, Ȳk,S := 1 |S| i∈S Y k,i .
We also let Ȳk,n as a shorthand notation for Ȳk,[n] .

A round-based algorithm Unlike index policies, SDA relies on rounds, in which several arms can be played (at most once). In each round r the learner selects a subset of arms A r ⊆ {1, . . . , K}, and receives Y k,N k (r) for each arm k ∈ A r , where N k (r) := r s=1 1(k ∈ A s ) denotes the number of times arm k was selected up to round r. Letting rT ≤ T be the (random) number of rounds used by the algorithm before the T -th arm selection, the regret of a roundbased algorithm can be upper bounded as follows.

Proposition 2.1 (Regret of a round-based algorithm).

R T (A) = E T t=1 (µ ⋆ -µ At ) ≤ E rT s=1 K k=1 (µ ⋆ -µ k )1(k ∈ A s ) ≤ E T s=1 K k=1 (µ ⋆ -µ k )1(k ∈ A s ) = K k=1 (µ ⋆ -µ k )E [N k (T )] .
(2.1)

Hence, upper bounding the number of pulls after T rounds E[N k (T )] for each sub-optimal arm provides a regret upper bound (in the usual purely sequential sense).

Proposition 2.1 shows that adopting a round-based approach where several arms can potentially be pulled per round do not really change the way to start analyzing the algorithms.

Sub-Sampling Dueling Algorithms

Sub-sampling Duelling Algorithms SDA takes as input a sub-sampling algorithm SP(m, n, r) that depends on three parameters: two integers n ≥ m and a round r. A call to SP(m, n, r) at round r produces a subset of [n] that has size m, modeled as a random variable that is further assumed to be independent of the rewards generated from the arms, (Y k,s ) k∈[K],s∈N * . We also call samplers such sub-sampling algorithms.

In the first round, a SDA selects A 1 = [K] in order to initialize the history of all arms. Given a sampler SP, we refer to the corresponding SDA as SP-SDA. For any r ≥ 1, at round r + 1 SP-SDA first defines the leader as one of the arms that have been selected the most in the first r round, namely ℓ(r) ∈ argmax k N k (r). Ties are broken in favor of the arm with the largest mean, and if several arms are still candidate the algorithm chooses one of them at random. The set A r+1 is then initialized to the empty set and K -1 duels are performed. For each "challenger" r), and arm k wins the duels if its empirical mean is larger than the empirical mean of the sub-sampled history of the leader. We can write that for any challenger k,

arm k ̸ = ℓ(r), a subset S r k of [N ℓ(r) (r)] of size N k (r) is obtained from SP(N k (r), N ℓ(r) (r),
Ȳk,N k (r) ≥ Ȳℓ(r),S r k =⇒ A r+1 = A r+1 ∪ {k} .
If the leader wins all the duels, that is if A r+1 is still empty after the K -1 duels, we set A r+1 = {ℓ(r)}. Arms in A r+1 are then selected by the learner in a random order and are pulled. The pseudo-code of SP-SDA is given in Algorithm 2.1, for an horizon of T rounds.

Sub-sampling Dueling Algorithms

To properly define the random variable Sr k used in the algorithm, we introduce the following probabilistic modeling: for each round r, each arm k, we define a family (S r k (m, n)) m≤n of independent random variables such that S r k (m, n) ∼ SP(m, n, r). In words, S r k (m, n) is the subset of the leader history used should arm k be a challenger drawn m times up to round r dueling against a leader that has been drawn n times. With this notation, for each arm k ̸ = ℓ(r) one has Sr k = S r k N k (r), N ℓ(r) (r) . We recall that in the SDA framework, we require those random variables to be independent from the reward streams (Y k,s ) of all arms k. We call such sub-sampling algorithms independent samplers.

Definition 2.2 (Independent Sampler). A independent sampler satisfies

∀(k, k ′ ) ∈ [K], ∀r, m ≥ n : S r k (m, n) ⊥ ⊥ Y k ′ ,s s∈N .
Some instances of SDA We now present some sub-sampling algorithms that we believe are interesting to use within the SDA framework. Intuitively, these algorithms should ensure enough diversity in the output subsets when called in different rounds, so that the leader cannot always look good, and challengers may win and be explored from time to time. [START_REF] Drmota | Sequences, discrepancies and applications[END_REF][START_REF] Halton | Algorithm 247: Radical-inverse quasi-random point sequence[END_REF][START_REF] Sobol | On the distribution of points in a cube and the approximate evaluation of integrals[END_REF]. We summarize these sub-sampling algorithms in the following, considering two sample sizes m ≤ n and a round r:

• Sampling Without Replacement (WR) returns a subset of size m selected uniformly at random without replacement in [n] (i.e each element can be drawn only once).

• Random Block Sampling (RB) draws an integer n 0 uniformly at random in [n -m] and returns {n 0 , ..., n 0 + m -1}. This is faster than WR as only one element is drawn.

• Last Block Sampling (LB) simply returns the last observations {n -m + 1, . . . , n}.

• Low Discrepancy Sampling (LDS) is a deterministic version of Random Block Sampling where n 0 is equal to u r at round r, for some low discrepancy sequence (u r ).

We propose in Figure 2.1 below an example of sub-samples that could be returned for a given duel by several of the algorithms that we propose. We proposed WR and RB samplers in hope that their randomness will naturally introduce sufficient diversity in the sub-samples. Similarly, LDS should ensure by construction the exploration of different part of the history of the leader during successive rounds. On the other hand, the reason why Last Block Sampling should work in producing diverse samples is less intuitive: after a sufficient number of rounds we expect the leader to be pulled most of the time, making the sub-sample move almost at each round in a sliding window fashion. We will prove later in this chapter that this simple mechanism is actually enough to obtain nice theoretical guarantees. Finally, we can remark that except the WR sampler most of the proposed algorithms return a set of consecutive observations, which is generally faster from a computational perspective. We call such algorithm a block sampler.

Definition 2.3 (Block Sampler).

A Block Sampler is a sub-sampling algorithm that only returns sub-sets of observations that have been collected successively in the history of an arm. Given

(m, n) ∈ N 2 , S r k (m, n) = {j, j + 1, . . . , j + m -1} for some j ≤ n -m.
In section 2.3 we will show that Block Samplers have convenient properties for the regret analysis of their associated SDA algorithm.

Sub-sampling Dueling Algorithms

Links with existing algorithms The BESA algorithm [START_REF] Baransi | Sub-sampling for multi-armed bandits[END_REF] with K = 2 coincides with WR-SDA. However beyond K > 2, [START_REF] Baransi | Sub-sampling for multi-armed bandits[END_REF] rather suggest a tournament approach, without giving a regret analysis. WR-SDA can therefore be seen as an alternative generalization of BESA beyond 2 arms, that may perform better than the tournament, as can be seen in experiments of Section 2.5. While the structure of SDA is close to that of SSMC (Chan, 2020), SSMC is not a SP-SDA algorithm. Indeed, its sub-sampling algorithm heavily relies on the rewards and is therefore not an independent sampler. It actually outputs the set S = {n 0 + 1, . . . , n 0 + n} for which Ȳℓ(r),S is the smallest, which is hence the worst possible sub-sample that can be returned by block samplers. For this reason SSMC may be more conservative than SDA algorithms, that may able to "forget" faster a sequence of bad observations that could have been drawn successively (or at least using them less often). Furthermore, our intuition is that LB-SDA could actually been seen as a simpler and possibly faster version of SSMC, as the two seem to perform very comparably in practice. Indeed, when an arm has been leader for some time the challengers can be pulled only if the newly arriving data allows to get a new sub-sample that is worse than the previous "worst sub-sample". Otherwise the comparison used by SSMC is the same as the one used in the previous round, where the challenger lost. Hence, the SDA framework is somehow unifying the ideas of BESA and SSMC as two of its instances are close, and maybe improved, variants of these algorithms.

On the use of forced exploration In (Chan, 2020), SSMC additionally implements some forced exploration: at a round r, each arm k such that N k (r) is smaller than some value f r is added to A r+1 . SSMC is proved to be asymptotically optimal for SPEF provided that f r = o(log r) and log log r = o(f r ). In the next sections, we show that SDA does not need forced exploration for some family of distributions: Bernoulli, Gaussian and Poisson. However, we will show that forced exploration allows to obtain more general theoretical guarantees and is not harmful in practice. Indeed, the exploration in log(r) suggested by Chan (2020) only forces to draw each arm 4 times for a time horizon T = 10 7 .

Finally, it is interesting to mention that duel-based algorithms were an early alternative to index policies. As already mentioned by Chan (2020), the very first asymptotically optimal algorithm proposed in [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] already performed pairwise comparison between the empirical mean of a leader (defined therein as the empirical best arm) and an upper confidence bound of the challenger, UCB k (t), thus helping the challengers instead of disadvantaging the leader.

Sub-Sampling Dueling Algorithms

Generic Regret Analysis

In this section we present our core contributions for this chapter, which are the theoretical guarantees for the SDA family of algorithms. We proceed step by step by sequentially including new assumptions on the family of distributions and the sub-sampling algorithm, and analyzing what can be obtained at each step. These guarantees come by deriving upper bounds on the expected number of selections of each sub-optimal arm k, E [N k (T )], which directly yields an upper bound on the regret via Equation (2.1). To ease the presentation, we assume that there is a unique optimal arm1 , that we define as arm 1 for simplicity.

Concentration of means and sub-sample means

A first natural question is to consider whether computing empirical means is a good idea or not for the family of distributions we consider. This is the case only if these empirical estimators concentrate around the true means at a sufficient speed, that we formalize as follow.

Assumption 2.4 (Concentration of empirical means).

For each arm k, the distributions ν k (of mean µ k ) admits a good rate function

I k : R → R + , that is ∀x > µ k , P Ȳk,n ≥ x ≤ e -nI k (x) , ∀x < µ k , P Ȳk,n ≤ x ≤ e -nI k (x) ,
where I k is continuous and

I k (x) = 0 if and only if x = µ k .
This assumption is satisfied by many usual families of distributions (SPEF, bounded, . . . ). More generally it is satisfied by any light-tailed distribution, that we define as any distribution ν for which

∃λ 0 > 0 : ∀|λ| < λ 0 , E X∼ν [exp(λX)] < +∞ .
If Assumption 2.4 is satisfied it makes sense to compare sub-sample means, and we can then start considering their concentration. Considering two arms ℓ and k and some x < µ ℓ , we want to upper bound the probability P( Ȳℓ,S r k ≤ x, N k (r) ≥ n 0 ) at some round r and for some integer n 0 . We now exhibit a convenient property of Block Samplers, introduced in definition 2.3.

Generic Regret Analysis

Lemma 2.5 (Concentration of sub-samples with Block Samplers). Let SP be a Block Sampler. Consider a round r and n 0 ∈ N, then it holds that

∀x > µ k , P Ȳk,S r k (N k (r),N ℓ (r)) ≥ x, N k (r) ≥ n 0 ≤ r 2 r n=n 0 P Ȳk,n ≥ x , ∀x < µ k , P Ȳk,S r k (N k (r),N ℓ (r)) ≤ x, N k (r) ≥ n 0 ≤ r 2 r n=n 0 P Ȳk,n ≤ x ,
Proof. Consider the first inequality. We use a union bound on the possible values of N ℓ (r), N k (r) and of the first element of the block, which provides the result by further remarking that each block of observation satisfy the same concentration properties as the first block.

P Ȳk,S r k (N k (r),N ℓ (r)) ≥ x, N k (r) ≥ n 0 ≤ r n ℓ =1 r n k =1 r i=1 P Ȳk,i:i+n k -1 ≥ x ≤ r 2 r n k =n 0 P Ȳk,n k ≥ x .
The exact same steps with x ≤ µ ℓ gives the second inequality.

To obtain this result we used that the number of blocks is at most linear in r. For a sampler that is not using blocks this number can be much larger. For instance, with sampling without replacement we would obtain n ℓ n k sub-samples for each couple (n k , n ℓ ), making the upper bound vacuous. Hence, samplers that do not use blocks require novel concentration tools. For that reason, the results presented in the rest of this chapter are only valid for Block Slampers.

Unfortunately, the upper bound presented in lemma 2.5 is not sufficient in some parts of the upcoming proofs. However, it can be improved under the additional event that the arm k by summing on the consecutive rounds s = 1, . . . , r.

Lemma 2.6 (Concentration of sub-samples inside a trajectory of a bandit algorithm). Let SP be a Block Sampler. Consider a round r and n 0 ∈ N, then it holds that

∀x > µ k , r s=1 P Ȳk,S s k (N k (s),N ℓ (s)) ≥ x, N k (s) ≥ n 0 , k ∈ A s+1 ≤ r r n=n 0 P Ȳk,n ≥ x , ∀x < µ k , r s=1 P Ȳk,S s k (N k (s),N ℓ (s)) ≤ x, N k (s) ≥ n 0 , k ∈ A s+1 ≤ r r n=n 0 P Ȳk,n ≤ x ,

Sub-Sampling Dueling Algorithms

The obtained bound is surprisingly better than when considering an individual sub-sample.

Proof. We first write that

(A) := r s=1 P Ȳk,S s k (N k (s),N ℓ (s)) ≥ x, N k (s) ≥ n 0 , k ∈ A s+1 =E r s=1 1 Ȳk,S s k (N k (s),N ℓ (s)) ≥ x, N k (s) ≥ n 0 , k ∈ A s+1 .
Now let's consider the sum inside of the expectation. It represents the count of the number of times a block of ℓ of size N k (r) has led to arm k being pulled. The rest of the proof is based on the remark that each possible block of observations (for any size and any location in the history of ℓ) can only have a cost of 1 in this sum: if k wins the duel its number of pulls will increase. Hence, the sum is upper bounded by the total number of blocks with a sub-sample mean satisfying the inequality, and we directly have

(A) ≤E r s=1 r n k =n 0 1( Ȳk,s:s+n k -1 ≥ x) ≤r r n k =n 0 P Ȳk,n k ≥ x
The results presented so far are sufficient to obtain a first upper bound on the number of pulls of each sub-optimal arm, for any SDA equipped with a block sampler.

Lemma 2.7 (First upper bound). Consider a block sampler SP and a bandit ν = (ν 1 , . . . , ν k ) ∈ F K where F is a family of distributions satisfying Assumption 2. 4. Then, for any suboptimal arm k ̸ = 1, the expected number of pulls of each sub-optimal arm k under SP-SDA is upper bounded by

E[N k (T )] ≤ 1 + ε I 1 (µ k ) log(T ) + C k (ν, ε) + 9 T r=1 P (N 1 (r) ≤ C 1 log(r)) ,
where C k (ν, ε) and C 1 are both problem-dependent constants.

This result is interesting because it shows that Assumption 2.4 along with elementary properties of block samplers are enough to exhibit the logarithmic term that we hope to be dominant in the regret upper bound.

The proof is inspired from the analysis of SSMC (Chan, 2020), using the concentration inequality of Lemma 2. 6. The details can be found in Appendix D of [START_REF] Baudry | Sub-sampling for efficient non-parametric bandit exploration[END_REF]. We

Generic Regret Analysis

refer the interested reader to Chapter 3 where we provide the proof of this result for a variant of LB-SDA using a limited memory size, that is valid for Lemma 2.7 by setting the limit to +∞.

The rest of the analysis consists in further upper bounding the quantity P(N 1 (r) ≤ C 1 log(r)) for a given round r. This probability corresponds to the case where arm 1 have been underexplored. In the next part we examine the conditions that should be met to make this probability small under an instance of SDA.

Avoiding under-exploration of the best arm

It is clear that whether the best arm is sure to be explored or not depends jointly on the subsampling algorithm and the properties of arms' distributions. First, we remark that if the number of pulls of arm 1 stays below C 1 log(r) then it has necessarily lost a lot of duels. We will further prove that it has even lost a large number of duels against the same leader, and while being stuck with some fixed sample the entire time. The question is then: has it been given a fair chance and lost a lot of diverse duels, or has it been provided too few different duels? We introduce the definition of non-overlapping sub-samples as a way to formalize this question.

Definition 2.8 (Non-overlapping sets). Two sets of integers

I 1 ⊂ N and I 2 ⊂ N are non- overlapping if I 1 ∩ I 2 = ∅. Furthermore, the sets in J = (I 1 , I 2 , . . . , I M ) are said to be pairwise non-overlapping if for any pair (I k , I k ′ ) ∈ J 2 it holds that I k ∩ I k ′ = ∅.
Definition 2.8 allows to quantify more precisely what kind of diversity we care about: we want sub-sampling algorithms to provide enough pairwise non-overlapping sub-samples during a run of SDA. Note that we use the expressions non overlapping sets or non-overlapping subsamples indifferently due to the isometry between a sub-sample and the positions of its items in the history of the arms.

Let us now consider the case for which the sampler was able to provide a large number of pairwise non-overlapping sub-samples. If arm 1 has not been drawn, its sample mean was smaller than the mean of each sub-sample from the leader. If the samples were non-overlapping, this event can be written using the balance function, introduced by [START_REF] Baransi | Sub-sampling for multi-armed bandits[END_REF] and that we recall here. Definition 2.9 (Balance function). Letting ν k,j denote the distribution of the sum of j independent variables drawn from ν k , and F ν k,j its corresponding CDF, the balance function of arm k is

α k (M, j) = E X∼ν 1,j 1 -F ν k,j (X) M .

Sub-Sampling Dueling Algorithms

For a fixed M and j, α k (M, j) corresponds exactly to the probability that arm 1 loses M independent duels with its j first observations. Then, using the result of Lemma 2.7, Definition 2.8 and Definition 2.9 we can formulate the following definition of sufficient diversity for a given sampler SP.

Definition 2.10 (Sufficient Diversity). Consider a sampler SP, a forced exploration f r , and a bandit problem ν = (ν 1 , . . . , ν K ). We say that SP introduces sufficient diversity in the sub-samples if for any round r large enough SP-SDA satisfies

P (N 1 (r) ≤ C 1 log(r)) = K k=2 r j=fr α k (M j,r , j) + o(r -2 ) ,
for some sequence M j,r that is linear in r up to poly-log terms.

In other words, SP provides enough pairwise non-overlapping sub-samples so that if arm 1 is not pulled enough it is (with high probability) because it has lost a large number of diverse duels.

An interesting remark from this definition is that, if SP provides sufficient diversity, then the guarantees of SP-SDA depends only on the family of distributions of the arms through their balance function. Hence, we disentangled in the analysis the impact of the sampler and of the family of distributions in the performance of an instance of SDA.

Furthermore, Definition 2.10 allows to naturally establish the condition we want to check for a family of distributions to determine whether SP-SDA can work or not for this family.

Definition 2.11 (Balance condition).

The balance condition is respected for a family F and a forced exploration f r if for any sequence M j,r satisfying M j,r = Ω r (log r) a for a large enough it holds that

T r=1 C 1 log(r) j=fr α k (M j,r ) = o(log(T )) .
Note that the level of forced exploration at round r is included in these definitions, hence some distributions may satisfy the balance condition for some values of f r but not for others. We can now provide the main theorem of this chapter, that is using Lemma 2.7 and all the definitions presented in this section. The statement remains as general as possible, and we refer to the next section for its application to specific instances of SDA and families of distributions.

Theoretical guarantees for RB-SDA and LB-SDA

Theorem 2.12 (Logarithmic Regret for SP-SDA). Consider a bandit ν = (ν 1 , . . . , ν K ) ∈ F K , where F is a family of distributions satisfying Assumption 2.4 (concentration of the means). Let SP be a block sampler providing sufficient diversity. Further assume that all the arms drawn from F satisfy the balance condition. Then for any ε > 0 the regret of SP-SDA is upper bounded by

R ν (T, π SP-SDA ) ≤ K k=2 1 + ε I k (µ 1 ) log(T ) + o ε (log T ) .
Proof. This result is a direct combination of Lemma 2.7, and the definition of sufficient diversity and the balance condition.

The challenge is now to determine which samplers and family of distributions satisfy these properties. However, these two investigations can be made separately. Before that, to provide the reader further intuitions let us consider some simple examples of failure for SDA.

Example 1 (bad sampler): Consider a sampler returning the first successive observations of the leader. Then, the regret of SDA is linear. → The worst rewards are more likely for the best arm than for the other, in that case the sub-sampling is not enough to recover from them.

In the following we formalize some sufficient conditions to avoid these cases.

Theoretical guarantees for RB-SDA and LB-SDA

In this part we instantiate the results from previous section for some instances of samplers and some families of distributions. We will first prove that both Random Block and Last Block samplers ensures sufficient diversity. Then we will provide a general technique to upper bound balance functions and some examples of distributions satisfying the balance condition.

Theoretical guarantees for RB-SDA and LB-SDA

history, and we have to skip j blocks for 1 observation to avoid overlaps. This result directly yields the first statement of the lemma.

We now consider X H,H,1 . In this case the distribution of the points collected is the one of sampling with replacement in a set of size H. It has been well studied in the literature, that allows to obtain the following upper bound.

Lemma 2.15. Combining the exact expression of P(X m,H,1 = k) from [START_REF] Mendelson | What is the distribution of the number of unique original items in a bootstrap sample?[END_REF] and an upper bound on Stirling numbers of the second kind from [START_REF] Rennie | On stirling numbers of the second kind[END_REF], we obtain that for any k ≤ H

P(X H,H,1 = k) ≤ 1 2 k H H-k H k .
As k should be small compared to

H it is then natural to use H k ≤ H k k! .
We then bound 1/k! by its equivalent given by Stirling formula and introduce a multiplicative constant c to pay for the approximation,

H k ≤ c H k √ 2πk × k k e k .
Refactoring and assuming that k ≤ H -2k we obtain

P(X H,H,1 = k) ≤ c 2 e k √ 2πk k H H-2k ≤ c 2 √ 2π ke H H-2k
.

To obtain the upper bound of Lemma 2.14 we need to consider this for k ≤ αH. We assume that α < 1/e, so we can finally obtain

P(X H,H,1 ≤ αH) ≤ c 2 √ 2π ⌊αH⌋ k=0 (αe) H-2k = c 2 √ 2π (αe) H (αe) -2⌊αH⌋ -1 (αe) -2 -1 = O (αe) H(1-2α) ,
which is exponentially decreasing in H for small enough α, and hence gives the result.

Sub-Sampling Dueling Algorithms

The complete proof of Lemma 2.13, that relies on Lemma 2.14, can be found in Appendix 2. 6. It remains to prove that if arm 1 is not pulled then it has been stuck with its first j observations for an interval of size Ω(r/ log(r)) in which the leader already had at least Ω(r/ log(r)) observations (corresponding to m and H here).

Hence, the proof of sufficient diversity with Random Block relied on probabilistic arguments due to the randomized sampling. For Last Block, on the contrary, all arguments will be purely deterministic.

Lemma 2.16 (Last Block Sampler). LB-SDA ensures sufficient diversity in the sub-samples.

Proof. The main trick to prove this result is to show that for any trajectory of the bandit algorithm the leader (whatever arm it is) will be played many times, making sure that the sub-samples presented to the challengers will be diverse. To measure this, we define the number of rounds for which the current leader was pulled up to a round r,

W r = 1 + r-1 s=1 1(A s+1 = {ℓ(s)}) ,
where we added 1 for the first round where every arm (including the arbitrary leader) is pulled once. For any trajectory, we show that this quantity is equal to the number of pulls of the leader, that we know to be linear in r.

Proposition 2.17. For any round r ≥ 2, under LB-SDA it holds that

W r = N ℓ(r) (r) ≥ r/K .
This proposition, formally proved in [START_REF] Baudry | On Limited-Memory Subsampling Strategies for Bandits[END_REF], comes from considering each phase between changes in leadership, and that any time the leader gets ahead in terms of number of pulls this arm must be drawn while being leader. We want to use this results on the rounds for which arm 1 has already "enough" samples thanks to the sampling obligation. We refer to Appendix 2. 6.1, where we prove that if f r = (log r) 1/k for k > 1 there exists a round

a r = o(r) for which N 1 (r) ≥ f r -1.
For the rest of the proof we consider the number of duels lost by arm 1 after a r against unique sub-samples of a sub-optimal leader. The number of duels won by the leader between the rounds a r and r is equal to W r -W ar . Out of those duels, at most C 1 log(r) can be won by the optimal arm. Consequently, there is at least W r -W ar -C 1 log(r) duels won by a suboptimal leader between rounds a r and r. Using Lemma 2.17 and W ar ≤ a r

Theoretical guarantees for RB-SDA and LB-SDA

we can then prove that for any β ∈ (0, 1), there exists a round r β,K from which

W r -W ar -C 1 log(r) ≥ β r K . (2.2)
Under N 1 (r) ≤ C 1 log(r) we are sure that there exists some j ∈ {1, ..., ⌊C 1 log(r)⌋} such that a fraction 1/(C 1 log(r)) of the duels counted above have been played with N 1 (r) = j. Let us denote W r = W r -W ar -C 1 log(r) and show this by contradiction. Out of those duels, we denote W r,j the number of duels played with N 1 (r) = j. If we assume that for all j ≤ ⌊C 1 log(r)⌋, there is strictly less than

β C 1 log(r) r K duels played with N 1 (r) = j.
The following would hold,

W r := W r -W ar -C 1 log(r) = ⌊C 1 log(r)⌋ j=1 W r,j < ⌊C 1 log(r)⌋ j=1 β C 1 log(r) r K < β r K .
There is a contradiction with Equation ( 2.2) and so there is necessarily some j ≤ ⌊C 1 log(r)⌋ and βr/(C 1 log(r)K) duels such that arm 1 competes using exactly its first j samples.

Furthermore, with the same argument we are sure that a fraction 1/(K -1) of these duels is played against the same leader k ∈ {2, . . . , K}. We would now like to obtain duels with non-overlapping blocks. Even if the blocks are all consecutive, waiting for j steps is enough to ensure that they are not overlapping. Hence, taking a fraction 1/j of the duels from the previous subsets is a conservative estimate of the true number of non-overlapping blocks they contain. Finally, we conclude that for any β ∈ (0, 1) there exists a constant r β,K such that for any round r > r β,K , under the event {N 1 (r) ≤ C 1 log r} there exists some k ∈ {2, . . . , K} and some j ∈ {⌊f r -1⌋, ⌊C 1 log r⌋} such as arm 1 lost at least β r K(K-1)(C 1 log r)j duels against non-overlapping blocks of arm k while k is the leader and 1 has exactly j observations. This term correspond exactly to the balance function α k (M, j) from Definition 2.9, with M = β r K(K-1)(C 1 log(r))j . This concludes the proof, as we obtain that LB-SDA satisfies

P(N 1 (r) ≤ C 1 log(r)) ≤ r β,K + K k=2 C 1 log(r) j=1 α k β r K(K -1)(C 1 log r)j , j ,
which satisfies Definition 2.10 of sufficient diversity.

To conclude this section on sub-sampling algorithms we explain why the tournament structure of BESA [START_REF] Baransi | Sub-sampling for multi-armed bandits[END_REF] may not work due to problems related to the diversity of sub-samples drawn. With the following example, we explain that the tournament (for K ≥ 4) may fail because an under-sampled best arm will be required to win against other arms that have not been sampled a lot too, potentially repeatedly playing the same unsuccessful duels.

Sub-Sampling Dueling Algorithms

Example 2.18 (Failure of the tournament structure of [START_REF] Baransi | Sub-sampling for multi-armed bandits[END_REF]). Let us consider a 4-armed bandit with in Lemma 2.7), andthat Ȳ1,j ≤ µ 4 -ε for some ε > 0

µ 1 > µ 2 > µ 3 = µ 4 . Assume that N 1 (l) = C log(r) for some 0 < C < C 1 (the constant
. If all other arms are well estimated (| Ȳk,N k (r) -µ k | ≤ ε for k ≥ 2)
, then 2 should be sampled most of the time (be the leader) and3 and4 have O(log(r)) samples. Further assume that 3 and4 have roughly C log(r) samples too. Then, the best arm may sometimes win a duel against arm 2 thanks to diversity of sub-samples and Assumption 2.20. However, after that it will most likely lose its duel against arm 3 or 4, and hence take an unreasonably long time to be pulled because of its duels against other challenger arms.

A deeper look at the balance condition

In this section we consider properties of families of distributions, that are independent of the run of SDA and the sampler it uses. We first exhibit a general property that ensures that the balance condition is satisfied with some amount of forced exploration, and then show that several families including SPEF actually satisfy it. We finally prove that several families of distributions (Bernoulli, Gaussian, Poisson) satisfy the balance condition without forced exploration, and on the other hand that some amount of forced exploration is necessary for exponential distributions.

We first propose a general way to upper bound the balance function. We use the notation F k,j for the cdf of the distribution of the sum of j i.i.d samples drawn from any arm k, and G k,j = 1 -F k,j . Then, the intuition is rather simple: take a reference value u, if arm 1 lost the M duels then either (1) the sum of all the samples collected from arm 1 is smaller than u, or (2) the sum of each of the M sub-samples drawn from the history of arm k is larger than u.

Proposition 2.19. α k (M, j) ≤ F 1,j (u) + G k,j (u) M , ∀u ∈ R . (2.
3)

The idea is then to consider values of u of the form

u = G -1 2,j (1-(log M ) a M
), that allows to upper bound the second term by M -a . We then need to control

F 1,j F -1 2,j (log M ) a M
. Interestingly, only having F 1,j (x) ≤ F 2,j (x) for x small enough is not sufficient as log(M ) a M is not small enough for the balance condition to hold. Interestingly, it is however enough to ensure a sub-linear regret, more precisely a poly-logarithmic regret. In the following we show that a slightly stronger condition is sufficient to ensure a logarithmic regret.

Theoretical guarantees for RB-SDA and LB-SDA

A general family satisfying the balance condition

In this section we will prove that assuming the best distribution to have a "better" lower tail than the others is enough to prove the balance condition. We formalize this as follow.

Assumption 2.20 (Dominant left tail).

∀k ≥ 2 :

∃y k ∈ R, c k ∈ (0, 1) : ∀x ≤ y k , dP ν 1 dP ν k (x) ≤ c k .
Under this assumption the worst rewards have to be more likely for bad arms than for good arms, to avoid the best arm getting stuck for a long time with a "bad" sample. The parameter c ≤ (0, 1) is added to ensure that the two tails are distinguishable enough for our proof to work.

In the following we discuss this condition and show that it is satisfied by many families of distributions. We now prove that it is enough to make the balance condition hold, thanks to the following result.

Proposition 2.21. If assumption 2.20 is satisfied with some parameters

(c k , y k ) k≥2 , then for all k ≥ 2, j ≥ 1 it holds that for all x ≤ y k : F 1,j (x) ≤ c j k F k,j (x) .
Proof. For any x ≤ y k The computation of F 1,j (x) directly provides the result,

F 1,j (x) = x -∞ x-x 1 -∞ • • • x-x 1 -•••-x j-1 -∞ dP ν 1 (x 1 ) . . . dP ν 1 (x j ) ≤ x -∞ x-x 1 -∞ • • • x-x 1 -•••-x j-1 -∞ c j k dP ν k (x 1 ) . . . dP ν k (x j ) = c j k F k,j (x)
This result allows to use Equation 2.3 to provide an upper bound on the balance function.

α k (M, j) ≤ γ × c j k log(M ) M + 1 M γ ,

Sub-Sampling Dueling Algorithms

for any M ∈ N satisfying log(M )/M ≤ y k , any γ > 1, and j ∈ N. Furthermore, the balance condition holds for any forced exploration f r satisfying fr log log(r) → +∞.

Proof. Using Proposition 2.21 and Equation 2.3, for any x ≤ y k the balance function of any arm k ≥ 2 can be upper bounded by

α k (M, j) ≤ c j k F k,j (x) + G k,j (x) M = c j k F k,j (x) + (1 -F k,j (x)) M
Interestingly, to provide the smallest possible upper bound on α k (M, j) we can simply consider minimizing the function z → c j k z + (1 -z) M . For simplicity we use that (1 -z) M ≤ exp(-M z), and then consider the value z M = γ log(M ) M for some γ > 1 (which is not exactly the minimizer but of the same order). This provides the first statement for M large enough such that z M ≤ y k . Now, consider a sequence M j,r = (log r) a r for some a > 1 (we omit potential constants for simplicity of notations). We consider the T r=1 C 1 log(r) j=fr α k (M j,r ). We use the upper bound we obtained on α k (M j,r ), and first remark that for any γ > 1 it holds that

T r=1 C 1 log(r) j=fr 1 M γ j,r = T r=1 C 1 log(r) j=fr (log(r)) aγ r γ = T r=1 C 1 (log(r)) aγ+1 r γ = O(1) ,
for any value of f r . Hence, the difficulty comes from upper bounding the term resulting from the first part of the upper bound of α k (M, j),

T r=1 C 1 log(r) j=fr c j k log(M j,r ) M j,r = T r=1 C 1 log(r) j=fr c j k log(r/(log(r) a )) r/(log(r) a ) ≤ T r=1 C 1 log(r) j=fr c j k log(r) a+1 r ≤ T r=1 log(r) a+1 r +∞ j=fr c j k = T r=1 log(r) a+1 r c fr k 1 -c k = 1 1 -c k T r=1 log(r) a+1 c fr k r .

Theoretical guarantees for RB-SDA and LB-SDA

We now investigate the possible values of f r to make this sum converge. For any ε > 0, this is the case if

log(r) a+1 c fr k = exp((a + 1) log log(r) -f r log(1/c k )) ≤ log(r) -(1+ε) , which holds if f r ≥ a+2+ε log(1/c k ) log log(r).
Summarizing these results, we obtain that the balance condition holds for f r satisfying fr log log(r) → +∞.

Examples

We provide a few examples of problems for which Assumption 2.20 holds. The first one is Single Parameter Exponential Families (SPEF), that contain some usual families of distributions.

Proposition 2.23 (Any SPEF satisfies Assumption 2.20). Assume that for a family of distribution

F there exists some functions g, ψ and a set Θ such that for any ν ∈ F there exists θ ∈ Θ satisfying

dP ν dη = e g(θ)y-ψ(θ) ,
for a reference measure η and with g(θ) that is increasing with the mean of ν. Then any bandit problem with arms in F satisfy Assumption 2.20.

Proof. According to the definition and properties of SPEF for any x ≤ µ k it holds that

dP ν 1 dP ν k (x) = exp ((g(θ 1 ) -g(θ k ))x -Ψ(θ 1 ) + Ψ(θ k )) ≤ exp ((g(θ 1 ) -g(θ k ))µ k -Ψ(θ 1 ) + Ψ(θ k )) ≤ exp (-kl(µ k , µ 1 )) , with kl(µ k , µ 1 ) > 0 as µ k < µ 1 .
This is exactly what we need for Assumption 2.20 with

y k = µ k and c k = exp(-kl(µ k , µ 1 ))
This result combined with Lemma 2.22 show that SPEF are well adapted to sub-sampling based algorithms, but we can find other type of assumptions on the distributions that would make Assumption 2.20 hold. For instance, a quite common assumption in bandits is to consider that rewards are generated by adding a noise to their mean, i.e any reward X k received from arm k can be written as

X k = µ k + η ,
where µ k is the mean of arm k and η is a noise, drawn from some fixed probability distribution of mean zero. For instance, η can be a be a sub-gaussian or sub-exponential noise. In that case, if the noise admits a density f then the densities of arm 1 and arm k in any x are respectively

f (x -µ 1 ) and f (x -µ k ). Hence, Assumption 2.20 is valid if there exists c k ∈ (0, 1) such that for x small enough f (x -∆ k ) f (x) ≤ c k .
If the noise is supported on R this depends of the asymptotic behavior of f . This will be typically true if f has an exponential or gaussian asymptotic equivalent.

Another question is to determined whether or not forced exploration is necessary. Indeed, the upper bound on the balance function may not be tight and it may be possible to improve it for specific distributions. In Appendix 2.7 we actually prove that forced exploration is not necessary for Gaussian (with shared variance), Poisson and Bernoulli distributions. On the other hand, we discuss that the balance condition cannot be satisfied without at least some forced exploration for exponential distributions. Finding other characterizations of families of distributions that can satisfy the balance condition and the level of forced exploration they need is an interesting future research direction.

Summary of our results

We provide a short summary of the results we obtained in previous sections:

• With Theorem 2.12, we proved that SP-SDA has a regret of order

K k=2 I 1 (µ k ) -1 log(T )
if the family of distribution F satisfies Assumption 2.4 (concentration of means with exponential decay) for some rate functions (I k ) k∈{1,...,K} , if SP is a block sampler bringing sufficient diversity, and if any bandit problem from F satisfies the balance condition.

• Lemma 2.13 and Lemma 2.16 show that both the Random Block and Last Block samplers bring a sufficient diversity of sub-samples to their respective algorithms.

• Lemma 2.22 shows that the balance condition is satisfied if F satisfies Assumption 2.20 with forced exploration f r satisfying f r / log log(r) → +∞. Furthermore, Proposition 2.23 shows that this assumption holds if F is a SPEF.

Hence, we can conclude this part with two results that showcase the potential of the algorithms based on sub-sampling.

Corollary 2.24 (of Theorem 2.12, Lemma 2.13,2.16 and Lemma 2.22). Let ν = (ν 1 , . . . , ν K ) ∈ F K be a bandit problem satisfying assumption 2.20 (dominant tail for the best arm), with arms satisfying assumption 2.4 (concentration of empirical means) with rate functions I 1 , . . . , I K . Then,

Theoretical guarantees for RB-SDA and LB-SDA

if the forced exploration f r satisfies fr log log(r) → +∞ and f r = o(log(r)) the number of pulls of each sub-optimal arm k > 2 for LB-SDA and RB-SDA can be upper bounded by

E[N k (T )] ≤ 1 + ε I 1 (µ k ) log(T ) + O ε (1) ,
for any ε > 0.

As the LB and RB samplers are proved to be sufficient, the learner only needs to check that the family of distributions satisfy the two assumptions we consider. We recall that assumption 2.4 is always valid if the distributions are light-tailed, which is in general a knowledge that is affordable in practice. The other assumption may be more difficult to check but is easy to re-formulate as:

worst-case scenarios need to be less probable for better arms. We showed that this is true for SPEF (e.g Bernoulli, Gaussian with shared variance, Poisson, . . . ), that are very common distributions. Furthermore, Corollary 2.24 can be further refined for SPEF by remarking that these families of distributions satisfy I 1 (µ k ) = kl(µ k , µ 1 ) (the proof simply relies on Chernoff inequality), so that the regret upper bound actually matches the lower bound of [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF].

Corollary 2.25 (Asymptotic optimality for SPEF

). Let ν = (ν 1 , . . . , ν K ) ∈ F K be a bandit problem,
where F is a SPEF satisfying Assumption 2. 4. Then, if the forced exploration f r satisfies fr log log(r) → +∞ and f r = o(log(r)) the number of pulls of each sub-optimal arm k > 2 for LB-SDA and RB-SDA can be upper bounded by

E[N k (T )] ≤ 1 + ε kl(µ k , µ 1 ) log(T ) + O ε (1) ,
for any ε > 0. Both RB-SDA and LB-SDA are then asymptotically optimal.

Again, a striking result is that these theoretical guarantees are simultaneously achieved for very different examples of distribution: some are bounded (Bernoulli), un-bounded (Gaussian, Poisson, Exponential), discrete or continuous, . . . and SDA works without using any of those information in its implementation. The only algorithm with the same guarantees is SSMC (Chan, 2020), that inspired SDA, and we think that we demonstrated further interesting properties of algorithms based on sub-sampling. First, we bridged the gap between BESA and SSMC by proposing algorithms that are their close variants (WR-SDA and LB-SDA respectively) in the same framework, with LB-SDA being simpler and more computationally efficient than SSMC. Then, we showed that both randomized and deterministic samplers can achieve strong theoretical guarantees, for families of distributions satisfying a generic property that we defined. Both approaches can have their interest according to additional constraints faced by the learner.

Sub-Sampling Dueling Algorithms

Finally, we also showed that forced exploration is not necessary for some distributions (Bernoulli, Gaussian, Poisson), but it seems necessary to some extent for others (Exponential).

We further remark that RB-SDA and LB-SDA being asymptotically optimal for binomial distributions is noteworthy. Indeed, this ensures that it is possible to build on these algorithms to propose a bandit algorithm that has logarithmic regret for distributions that are bounded in a known support. To do so, we can use the binarization trick already proposed by Agrawal and Goyal (2013a) for Thompson Sampling, and run RB-SDA or LB-SDA on top of a binarized history

H ′ k for each arm k in which a reward Y k,s is replaced by a binary pseudo-reward is Y ′ k,s
generated from a Bernoulli distribution with mean Y k,s . The resulting algorithm inherits the regret guarantees of these algorithms applied to Bernoulli distributions.

An interesting future work consists in finding a variant of SDA that could work when Assumption 2.20 is not satisfied. For bounded distributions this can be done with the binarization trick introduced in previous paragraph. However, the right way to achieve this goal is much less clear for unbounded distributions.

Experiments

In this section, we perform experiments on simulated data in order to illustrate the good performance of the four instances of SDA algorithms introduced in Section 2.2 for various distributions. The Python code used to perform these experiments is available on Github.

Bernoulli and Gaussian arms First, in order to illustrate Corollary 2.25, we investigate the performance of SDA for both Bernoulli and Gaussian distributions (with known variance 1). Our first objective is to check that for a finite horizon the regret of SDA is comparable with the regret of Thompson Sampling (with respectively a beta and improper uniform prior), which efficiently use the knowledge of the distribution. Our second objective is to empirically compare different variants of SDA to other non-parametric approaches based on sub-sampling (BESA, SSMC) or on re-sampling. For Bernoulli and Gaussian distribution, Non-Parameteric TS coincides with Thompson Sampling, so we focus our study on algorithms based on history perturbation. We experiment with PHE (Kveton et al., 2019a) for Bernoulli bandits and ReBoot [START_REF] Wang | Residual bootstrap exploration for bandit algorithms[END_REF] for Gaussian bandits, as those two algorithms are guaranteed to have logarithmic regret in each of these settings. As advised by the authors, we use a parameter a = 1.1 for PHE and σ = 1.5 for ReBoot. As results in Appendix 2.7 show that SDA do not require forced exploration with Bernoulli and Gaussian arms we set f r = 1.

We ran experiments on 4 different Bernoulli bandit models, that we present in Table 2. 1. The objective of the experiments is to test different cases that can happen with Bernoulli distributions: large mean, high variance (µ around 0.5), low mean, and many similar arms. µ 2:4 = 0.01,µ 5:7 = 0.03,µ 8:10 = 0.05 xp 4B K = 8 µ = {0.9, 0.85, . . . , 0.85}

Experiments

} xp 1B K = 2 µ = {0.8, 0.9} xp 2B K = 2 µ = {0.5, 0.6} xp 3B K = 10 µ 1 = 0.1,
We then propose 3 experiments with gaussian arms N (µ k , 1), that we present in Table 2.2.

} xp 1G K = 2 µ = {0.5, 0} xp 2G K = 4 µ = {0.5, 0, 0, 0} xp 3G K = 4 µ = {1.5, 1, 0.5, 0}
For each experiment with Bernoulli and Gaussian arms, Table 2.3 andTable 2.4 report an estimate of the regret at time T = 20000 based on 5000 independent runs, as well as standard deviation across al trajectories. The best performing algorithms are highlighted in bold. In Figure 2.2 we further plot the regret of several algorithms as a function of time (in logarithmic scale) for t ∈ [15000; 20000] for one Bernoulli and one Gaussian experiment respectively. We also add the Lai and Robbins lower bound [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF]. These figures aim at showing that the first order term of the empirical regret matches the lower bound for SDA (and other) algorithms. In all of these experiments, we notice that SDA algorithms are indeed strong competitors to Thompson Sampling (with appropriate prior) for both Bernoulli and Gaussian bandits. The y axis is in logarithmic scale, and the x axis starts at T = 15×104 to illustrate the "asymptotic" regime of the algorithms (parallel straight lines correspond to a logarithmic regret with the same constant before the log).

on two instances, just like SSMC, which can be seen from the parallel straight lines with the x axis in log scale. The fact that the lower bound is above shows that it is really asymptotic and only captures the right first order term. The same observation was made for all experiments, but is not reported due to space limitation. Even if we only established the asymptotic optimality of RB-SDA and LB-SDA, these results suggest that the other SDA algorithms considered in this chapter may also be asymptotically optimal. Compared to SDA, re-sampling algorithms based on history perturbation seem to be much less robust. Indeed, in the Bernoulli case, PHE performs very well for experiment 2, but is significantly worse than Thompson Sampling on the three other instances. In the Gaussian case, ReBoot always performs significantly worse than other algorithms. Finally, we notice that the standard deviations are comparable for SDA algorithms and TS. However, for gaussian arms some of them may experience large standard deviations for some experiments. This may be due to the fact that the constant upper bounding the balance function is larger than for bernoulli distributions, and may be reduced by the asymptotically negligible forced exploration in f r = log(r) (which would correspond to only

Experiments

Turning our attention to algorithms based on sub-sampling, we first notice that WR-SDA seems to be a better generalization of BESA with 2 arms than the tournament approach currently proposed, as in experiments with K > 2, WR-SDA often performs significantly better than BESA. Then we observe that SSMC and SDA algorithms have similar performance. Looking a bit closer, we see that the performance of SSMC is very close to that of LB-SDA as we intuited in previous sections, whereas SDA algorithms based on "randomized" (or pseudo-randomized for LDS-SDA) samplers tend to perform slightly better on average, at the cost of larger variance in the results.

Truncated Gaussian Theorem 2.24 suggests that RB-SDA and LB-SDA may attain logarithmic regret beyond exponential families. As an illustration, we present the results of experiments performed with Truncated Gaussian distributions (in which the distribution of arm k is that of

Y k = 0 ∨ (X k ∧ 1) where X k ∼ N (µ k , σ 2 ))
. These distributions trivially satisfy Assumption 2.20 since the probability of obtaining 0 is strictly smaller for the best arm than for the other arms. We present in 2.5 the 4 problems we considered in this setting, and report in Table 2.6 the regret at time T = 20000 (estimated over 5000 runs) of various algorithms on four different problem instances:

xp 1TG K = 2 µ = {0.5, 0.6}, σ = 0.1 xp 2TG K = 2 µ = {0, 0.2}, σ = 0.3 xp 3TG K = 2 µ = {1.5, 2}, σ = 1 xp 4TG K = 4 µ = {0.4, 0.5, 0.6, 0.7}, σ = 1
We include Non-Parametric TS which is known to be asymptotically optimal in this setting (while TS which uses a Beta prior and a binarization trick is not), PHE, and all algorithms based on sub-sampling. We again observe the good performance of SSMC and SDA algorithms across all experiments. They even outperform NPTS in some experiments, which suggests SDA algorithms may be also asymptotically optimal for this class of distributions.

Exponential Arms

In Appendix 2.7, we show that exponential actually requires some level of forced exploration. In this section we propose to check the performance of our algorithms without and with forced exploration to see if it seems really necessary in practice or if forced exploration is just a proof artifact. We performed 6 experiments, with mean parameters reported in Table 2.7.

It is interesting to remark that the standard deviation of an exponential distribution is equal to its mean, so with similar gaps problems are harder when the means are high. We first report in Table 2.8 our result without forced exploration.

Sub-Sampling Dueling Algorithms

xp 1E K = 2 µ = {1.5, 1} xp 2E K = 2 µ = {0.2, 0.1} xp 3E K = 2 µ = {11, 10} xp 4E K = 4 µ = {4, 3, 2, 1} xp 5E K = 4 µ = {0.4, 0.3, 0.2, 0.1} xp 6E K = 4 µ = {5, 4, 4, 4}
First, we notice that the performance of the SDA in terms of the average regret are reasonable, although less impressive than with the other distributions we tested. IMED is almost always the best algorithm in these experiments, and SSMC performs pretty well on many examples (note that we left f r = log(r) for SSMC). We remark that there is much more variability in the results of RB-SDA, WR-SDA and LDS-SDA than before, where they performed quite similarly. For instance, we notice that on example 3, LDS-SDA and RB-SDA are much worse than WR-SDA. A look at the quantile table for this experiment (Table 2.9), which displays the empirical quantiles of the regret estimated over 5000 runs, shows that this is due to a small number of "bad" trajectories for these algorithms: We see that up to the 80% quantile, RB-SDA and LDS-SDA are even significantly better than IMED. This is very different when we look at the 95% and 99% quantiles, which are much greater for our 2 algorithms (even 2.5 times greater for the 99% quantile).

Experiments

We believe that this very high variability prevents SDA to have a logarithmic regret for exponential arms. Still, the regret is not as bad as being linear, and a closer look at the balance function allows to prove a O((log(T )) 2 ) regret, with the term P(N 1 (r) < C 1 log(r)) becoming the first order term of the regret. Hence, the experiments seem to confirm that the asymptotically negligible forced exploration is actually necessary for exponential arms. In Table 2.10 we run the same experiments but choose f r = log(r) as suggested in (Chan, 2020) We remark that adding forced exploration results in a noticeable improvement for SDA algorithms, with RB-SDA, WR-SDA and LDS-SDA becoming competitive with IMED on most examples. Furthermore, LB-SDA becomes again comparable with SSMC. Considering all these results and the fact that a forced exploration of f r = log(r) is not harmful in practice we recommend to always run SDA with this level of exploration.

Sub-Sampling Dueling Algorithms

Bayesian Experiments So far we tried our algorithms on specific instances of the distributions we considered. It is also interesting to check the robustness of the algorithms when the means of the arms are drawn at random according to some distribution. In this section we consider two examples: Bernoulli bandits where the arms are drawn uniformly at random in [0, 1], and Gaussian distributions with the mean parameter of each arm itself drawn from a gaussian distribution µ k ∼ N (0, 1). In both cases we draw 10000 random problems with K = 10 arms and run the algorithms for a time horizon T = 20000. We experiment with TS, SSMC, RB-SDA and WR-SDA and IMED. We do not add LDS-SDA and LB-SDA as they are similar to RB-SDA and SSMC, respectively. In the Bernoulli case, we also run the PHE algorithm, which fails to compete with the other algorithms. This is not in contradiction with the results of Kveton et al. (2019a) as in the Bayesian experiments of this paper, arms are drawn uniformly in [0.25, 0.75] instead of [0, 1]. Actually, we noticed that PHE with parameter a = 1.1 has some difficulties when several arms are close to 1. Results reported in Tables 2.11 and 2.12 show that RB-SDA and WR-SDA are strong competitors to TS and IMED for both Bernoulli and Gaussian bandits. Recall that these algorithm operate without the need for a specific tuning for each distribution, unlike TS and IMED. Moreover, observe that in the Bernoulli case, TS further uses the same prior as that from which the means are drawn.

Experiments

Computational aspects

To choose a sub-sampling based algorithm, numerical consideration can be taken into account. First, compared to algorithms like UCB1 or Thompson Sampling the main drawback of sub-sampling based algorithms is that they require to store the full history of rewards. This motivated our study in Chapter 3 about a variant of LB-SDA with limited memory. On the other hand, the computation cost of sub-sampling varies across algorithms: block samplers are generally more efficient than WR-SDA as the latter requires to draw a random subset while the formers only need at most to draw the random integer starting the block and compute the sub-samples' mean. However, for distributions with finite supports WR-SDA can be made as efficient as TS using multivariate geometric distributions.

LDS-SDA could be preferred to RB-SDA to avoid randomization, as it uses a deterministic sequence. Finally, LB-SDA has the smallest computational cost in the SDA family and while its performance is very close to that of SSMC, it can avoid the cost of scanning all the sub-sample means in this algorithm. The computational cost of these two algorithms depends on what happened during the round: their update can be made very efficient when the leader does not change and is pulled. Indeed, in that case one only needs to update K -1 means by replacing the oldest observation by the last collected. However, the alternative case is costly for SSMC as it requires to perform a complete scan of the leaders' history. Under such scenario LB-SDA is much more efficient, and does not cost more than a step of RB-SDA. We complete this discussion in next chapter.

Sub-Sampling Dueling Algorithms

Appendix A: Sufficient diversity for RB-SDA (Lemma 2.13)

We start with a decomposition that follows the steps of [START_REF] Baransi | Sub-sampling for multi-armed bandits[END_REF] for BESA with 2 arms that we generalize for K arms. Furthermore, in the following we first write the proof for a forced exploration f r = 1, as the notations are already heavy, and we detail in the next section 2.6.1 why introducing an asymptotically negligible forced exploration does not change the proof.

We first denote by r j the round of the j th play of arm 1 with r 0 = 0 and let τ j = r j+1 -r j . We notice that τ 0 ≤ K as all arms are initialized once. Then:

P (N 1 (r) ≤ C 1 log(r)) ≤ P (∃j ∈ {1, ..., C 1 log(r)} : τ j ≥ r/C 1 log(r) -1) ≤ C 1 log(r) j=1 P (τ j ≥ r/C 1 log(r) -1) Indeed, if we assume that ∀j τ j ≤ r/C 1 log(r) -1 then t C 1 log(r) = C 1 log r j=0 τ j < r, which yields N ℓ (r) > C 1 log r + 1.
We now fix j ≤ C 1 log(r) and upper bound the probability of the event

E j := {τ j ≥ r/C 1 log(r) -1} .
On this event arm 1 lost at least r/C 1 log(r) consecutive duels between r j + 1 and r j+1 (either as a challenger of as the leader) which yields P(E j ) ≤ P(∀s ∈ {r j + 1, ..., r j + ⌊r/C 1 log(r) -1⌋} :

{ Ȳ1,j ≤ Ȳℓ(s),S s 1 (N ℓ(s) (s),j) , N 1 (s) = j, N ℓ(s) (s) ≥ j} ∪ {ℓ(s) = 1, N 1 (s) = j})
The important change compared to the proof of [START_REF] Baransi | Sub-sampling for multi-armed bandits[END_REF] is that with K > 2, we don't know the identity of the leader and the leader is not necessarily pulled if it wins its duel against 1. We then notice that when r is large, the time range considered in each E j includes a large number of rounds. By looking at the second half of this time range only, we can ensure that the leader has been drawn a large number of times. More precisely, introducing the two intervals

M 1 j,r = r j + 1, r j + r/(C 1 log(r)) -1 2 M 2 j,r = r j + r/(C 1 log(r)) -1 2 , r j + ⌊r/(C 1 log(r))⌋ -1

Appendix A: Sufficient diversity for RB-SDA (Lemma 2.13)

it holds that

P(E j ) ≤ P(∀s ∈ M 2 j,r :{ Ȳ1,j ≤ Ȳℓ(s),S s 1 (N ℓ(s) (s),j) , N 1 (s) = j, N ℓ(s) (s) ≥ j} ∪ {ℓ(s) = 1, N 1 (s) = j}) .
But we know that on M 2 j,r the leader must has been selected at least

1 K j + r/(C 1 log(r))-1 2 times. Let r K be the first integer such that C 1 log(r) < 1 K-1 r/(C 1 log(r))-1 2
, for every r ≥ r K , as j ≤ C 1 log(r), the leader has been selected strictly more than j times, which prevents arm 1 from being the leader for any round in M 2 j,r . Hence, for r ≥ r K , for all j ≤ C 1 log(r),

P(E j ) ≤ P ∀s ∈ M 2 j,r : { Ȳ1,j ≤ Ȳℓ(s),S s 1 (N ℓ(s) (s),j) , N 1 (s) = j, N ℓ(s) (s) ≥ j} .
To remove the problem of the identity of the leader we would like to find a way to fix our attention on one arm. To this extent, we notice that during an interval of length |M 2 j,r |, if there are only K -1 candidates for the leader then one of them must have been leader at least m r := |M 2 j,r |/(K -1) -1 times during this range. We also know that at any round in M 2 j,r , the leader satisfies

N ℓ(s) (s) ≥ (t j +⌊ r/C 1 log(r)-1 2 ⌋)/K -1 ≥ (⌊ r/C 1 log(r)-1 2 ⌋)/K -1 = |M 1 j,r | K -1 := c r .
Observe that m r > c r . Finally, we introduce the notation

I k j,r = {s ∈ M 2 j,r : ℓ(s) = k}
for the set of rounds in M 2 j,r in which a particular arm k is leader. From the above discussion, we know that there exists an arm k such that |I k j,r | ≥ m r . To ease the notation, we introduce the event

W k s,j = Ȳ1,j < Ȳk,S s 1 (N k (s),j) , N k (s) ≥ c r , N 1 (s) = j
and write

P(E j ) ≤ P s∈M 2 j,r K k=2 {ℓ(s) = k, 1 / ∈ A s )} ≤ P K k=2 s∈I k j,r W k s,j ≤ P    K k=2      |I k j,r | > m r , s∈I k j,r W k s,j         ≤ K k=2 P   |I k j,r | > m r , s∈I k j,r W k s,j    .

Sub-Sampling Dueling Algorithms

Finally, we define for any integer M the event that we can find M pairwise non-overlapping sub-samples in the set of the sub-samples of arm k drawn in rounds s ∈ I k j,r :

F k,r j,M = ∃i 1 , ..., i M ∈ I k j,r : ∀m < m ′ ∈ [M ], S im 1 (N k (i m ), j) ∩ S i m ′ 1 (N k (i m ′ ), j) = ∅ Introducing H k j,r = min s∈I k j,r
N k (s), the minimal size of the history of arm k during rounds in I k j,r (which is known to be larger than c r as k is leader in these rounds), one has

P(E j ) ≤ K k=2 P |I k j,r | > m r , ∩ s∈I k j,r W s,j ∩ {F k,r j,M ∪ Fk,r j,M } ≤ K k=2 P |I k j,r | ≥ m r , H k j,r ≥ c r , Fk,r j,M + K k=2 P |I k j,r | > m r , ∩ s∈I k j,r W s,j ∩ F k,r j,M (2.4)
Upper bound on the first term in (2.4) The probability P |I k j,r | ≥ m r , H k j,r ≥ c r , Fk,r j,M can be upper bounded by

P # pairwise non-overlapping subsets in (S s 1 (N k (s), j)) s∈I k j,r < M |I k j,r | > m r , H k j,r ≥ c r .
This probability can be related to some intrinsic properties of the sampler SP(H, j). To formalize this, we introduce the following definition.

Definition 2.26. For every integers N, H, j such that H > j, X N,H,j is a random variable which counts the maximum number of non-overlapping subsets among N i.i.d. samples from SP(H, j).

Letting H 1 , . . . , H mr be integers that are all larger than c r , and letting S 1 , . . . , S mr be independent subsets such that S i ∼ SP(H i , j), the above probability is upper bounded by

P (# {pairwise non-overlapping subsets in (S i ) mr i=1 } < M )
which is itself upper bounded by P (X mr,cr,j < M ).

This last inequality is quite intuitive: if one draws subsets of size j from histories that may be larger than c r , there is more "room" for non-overlapping subsets than if we always draw them from the same history of size c r . For Random Block sampling, where the drawn subset is fully determined by the random position of its first element, to formalize this intuition it is sufficient to prove that if X i , Y i are two sequences of random variables such that

X i is uniform in [H i -j] and Y i is uniform in [H -j],
where H i ≥ H, the random variable that counts the maximal number of elements in the sequence (Y i ) whose pairwise distance are larger than j is stochastically dominated by that the same random variable but for the sequence (X i ).

Appendix A: Sufficient diversity for RB-SDA (Lemma 2.13)

Upper bound on the second term in (2.4) On the event

|I k j,r | > m r , ∩ s∈I k j,r
W s,j ∩ F k,r j,M , one can define ĩ1 , . . . , ĩM the first M rounds in I k j,r for which the subsets Sm := S ĩm (N k (ĩ m ), j) are pairwise non-overlapping and we get

P |I k j,r | > m r , ∩ s∈I k j,r W s,j ∩ F k,r j,M ≤ P ∀m ∈ [M ], Ȳ1,j ≤ Ȳk, Sm .
By definition the subsets Sm are pairwise non-overlapping, hence the sub-samples Ȳk, Sm are independent. We prove that this probability can be in fact upper bound by the balance function we defined in section 2. 3.

Indeed, introducing X ∼ ν 1,j and an independent i.i.d. sequence Z i ∼ ν k,j , one can write

P |I k j,r | > m r , ∩ s∈I k j,r W s,j ∩ F k,r j,M ≤ P(X < min i∈[M ] Z i ) = E X∼ν 1,j Z∼ν ⊗j k,j M i=1 1 X≤Z i = E X∼ν 1,j E Z∼ν ⊗j k,j i 1 X≤Z i X = E X∼ν 1,j (1 -F k,j (X)) M = α k (M, j).

Conclusion

Putting things together, we have proved that

P(E j ) ≤ (K -1)P (X mr,cr,j < M ) + K k=2 α k (M, j),
where X N,H,j and α k (M, j) are introduced in Definition 2.26 and 2.9 respectively. If we replace M by the sequence β r,j we have

T r=1 P(N 1 (r) ≤ C 1 log(r)) ≤ r K + T r=r K C 1 log(r) j=1 (K -1)P (X mr,cr,j < β r,j ) + K k=2 α k (β r,j , j) ≤ r K + T r=r K C 1 log(r) j=1 (K -1)P (X cr,cr,j < β r,j ) + K k=2 α k (β r,j , j) ,
as c r ≤ m r . We then conclude by using Lemma 2.14, introduced earlier in this chapter. The quantity P (X cr,cr,j < β r,j ) admits an exponential upper bound in c r , which ensures that

T r=r K C 1 log(r) j=1 (K -1)P (X cr,cr,j < β r,j ) = O(1) ,

Sub-Sampling Dueling Algorithms

which concludes the proof that RB-SDA provides sufficient diversity according to definition 2.10.

Adding Forced Exploration in the proof

The idea is to use the same proof sketch as without forced exploration. We consider any sequence f r of the form f r = (log r) 1 k for some k > 1 for simplicity. The following proof sketch can be adapted to other sequences. Let us denote for simplicity f

-1 (x) = inf{r ∈ N : f r ≥ x} for any x ∈ R.
Let us consider the round a r = f -1 (f r -1). At this round, the value of exploration function is f r -1 = log(r) 1/k -1, which means that the number of pulls of arm 1 is at least ⌊(log r) 1/k -1⌋.

Now we aim at proving that the number of rounds in the interval r -a r is very close to r when r is large. To do that, we use that

a r = f -1 (f r -1) = exp(((log r) 1 k -1) k ) .
We then compare the exponent with log r. For η ∈ (0, 1) and r large enough it holds that log(r) -((log r) r), and we can conclude that for any γ ∈ (0, 1), there exists some round r γ such that for r ≥ r γ , r -a r ≥ γr .

1 k -1) k = log(r) 1 -1 - 1 (log(r)) 1/k k ≥ log(r) 1 -exp - k (log r) 1/k ≥ log(r) 1 -1 -η k (log r) 1/k = ηk log(r) 1-1 k -→ +∞ , so a r = o(
This means that after the round a r arm 1 faces a linear amount of duels, and has an history of at least j = ⌊(log r) 1/k -1⌋ samples. Introducing b r the random variable giving the first time when N 1 (b r ) = ⌊(log r) 1/k -1⌋, we necessarily have b r ≤ a r .

Then, using the exact proof as in the previous section we finally obtain a result of the form

Appendix B: Further Analysis of the Balance Function of some distributions

T r=1 P(N 1 (r) ≤ C 1 log(r)) ≤ r ′ K + T r=r ′ K (log r) 2 j=⌊fr⌋-1 (K -1)P(X cr,cr,j < M ′ r,j ) + K k=2 α k (M ′ r,j , j) (2.5)
for a new sequence M r,j smaller than the previous one but of the same order in (r, j), and a new constant r ′ K . This result concludes this part.

Appendix B: Further Analysis of the Balance Function of some distributions

In this section we detail the upper bound on the balance function that allows to show that gaussian distributions do not require forced exploration. This proof can be found in Appendix G of [START_REF] Baudry | Sub-sampling for efficient non-parametric bandit exploration[END_REF], where we also prove similar results for the Bernoulli and Poisson distributions. We choose to provide the results for Gaussian distributions as an example of how such bound can be derived. Then, we provide a simple result that shows that exponential distributions require forced exploration to some extent.

In the next parts we use the notation

G(x) = 1 -F (x)
where F if the CDF of the distribution considered. For some arm distribution ν i the distribution of the sum of j independent observations drawn from ν i is denoted by ν i,j . With this notation, for two arms 1 and 2 we write

α(M, j) = E Z∼ν 1,j [G 2,j (Z) M ]; .

Gaussian Distribution

For the Gaussian distribution we leverage the fact that both the PDF and CDF can be expressed with the PDF and CDF of the standard normal distribution. We use the notations f and F for the PDF and CDF of the N (0, 1) distribution, write G = 1 -F , ∆ for the gap between the two arms, and compute the expectation

α(M, j) = +∞ -∞ f 1,j (x)G 2,j (x) M dx ≤ z -∞ f 1,j (x)G 2,j (x) M dx + G 2,j (z) M , ∀z ∈ R ≤ z -∞ f j(x -µ 1 ) G j(x -µ 2 ) M dx + G 2,j (z) M ≤ 1 √ j √ j(z-µ 2 ) -∞ f y -j∆ G(y) M dy + G 2,j (z) M .

Sub-Sampling Dueling Algorithms

At this step we use that f (x -a) = e -a 2 /2+ax f (x) for all a, x, and that the function h : x →

(M + 1)f (x)G(x) M is a probability distribution of CDF x → 1 -G(x) M +1 . These properties allow to write α(M, j) ≤ 1 √ j e -j ∆ 2 2 M + 1 √ j(z-µ 2 ) -∞ e √ j∆y h(y)dy + G 2,j (z) M ≤ 1 √ j e -j ∆ 2 2 M + 1 e j∆(z-µ 2 ) 1 -G j(z -µ 2 M +1 + G j(z -µ 2 ) M ≤ 1 √ j e -j ∆ 2 2 -∆(z-µ 2 ) M + 1 + G j(z -µ 2 ) M .
As the inequality is true for all z ∈ R, it holds that

∀y ∈ R, α(M, j) ≤ e -j ∆ 2 2 M + 1 e √ j∆y + G (y) M . Now let y M be such as G(y M ) = 1 -1 √ M .
This value ensures that the second term satisfies

G(y M ) M ≤ e - √ M = o(M -2 ). Observe that y M = F -1 ( 1 √ M +1 ).
Using the following equivalent of the quantile function of the normal distribution when the quantile is small (see for instance [START_REF] Ledford | Modelling dependence within joint tail regions[END_REF]):

F -1 (p) = -log 1 p 2 -log log 1 p + log 2π + o p→0 (1) , there exists a constant C ∈ R such that y M ≤ -C √ log M -log log M + log 4π. This yields α(M, j) ≤ e -j ∆ 2 2 M + 1 e -C √ j∆ √ log M -log log M +log 4π + e - √ M .
We then remark that for all k ∈ N * ,

k log log M = o(C j∆ log M -log log M + log 4π)
and as a consequence that

α(M, j) = o   e -j ∆ 2 2 (M + 1)(log M ) k   ,

Appendix B: Further Analysis of the Balance Function of some distributions

for any k ∈ N * . This is sufficient to prove that the Gaussian distribution is balanced. Indeed, with M = O(r/(C 1 log r)) the series in r is convergent (O 1 r(log r) k for some k > 1).

Exponential Distribution

For j = 1, a direct calculation yields

α(M, 1) = 1 1 + µ 1 µ 2 M .
Using this and M = β r log r we then obtain

T r=1 ⌊C 1 log(r)⌋ j=1 α k (⌊βr/ log r⌋ , j) ≥ T t=1 α k (⌊βr/ log r⌋ , 1) = T t=2 1 1 + µ 1 µ k ⌊βr/logr⌋ ≥ T t=2 1 1 + µ 1 µ k βr/ log r ≥ C T r=2 log(r) r = Ω(log(T ) 2 ),
where C is some small enough constant that depend on µ 1 , µ k and β.

Introduction

In this chapter we still consider the Multi-Armed Bandit problem introduced in Chapter 1, as well as the algorithms based on sub-sampling that we introduced in Chapter 2. More specifically, we build on the Last-Block Sub-sampling Dueling Algorithm (LB-SDA), that we introduced and analyzed, and that is particularly attractive because of its simplicity and computational efficiency compared to other instances of SDA. Our first contribution in this chapter is to show that the theoretical guarantees of LB-SDA still hold, without additional changes, for a variant of the algorithm using a limited memory of the observations of each arm. We prove in particular that storing a poly-logarithmic amount of observations (instead of linear) in the number of rounds played is sufficient to maintain the theoretical guarantees, making the algorithm more tractable for larger time horizons. This is interesting since the main drawback of the algorithms analyzed in Chapter 2 is the requirement to store all T rewards.

Furthermore, building a sub-sampling algorithm based on the most recent observations makes it an ideal candidate for a passively forgetting policy in a non-stationary environment. We presented a short introduction to the vast literature on non-stationary bandits in Section 1.3 of Chapter 1, detailing some known theoretical results in this setting along with common approaches to tackle this problem. Our second contribution is to propose a natural extension of the LB-SDA strategy to non-stationary environments by using a sliding window. By limiting the extent of the time window in which sub-sampling is allowed to occur, one obtains a passively forgetting non-parametric bandit algorithm, which we refer to as Sliding Window Last Block Sub-sampling Duelling Algorithm (SW-LB-SDA). To analyze the performance of this algorithm, we assume an abruptly changing environment in which the reward distributions change at unknown time instants called breakpoints. We show that SW-LB-SDA guarantees a regret of order O( Γ T T log(T )) for any abruptly changing environment with at most Γ T breakpoints, thus matching the lower bounds from [START_REF] Garivier | On upper-confidence bound policies for switching bandit problems[END_REF], up to logarithmic factors. The only required assumption is that, during each stationary phase, the reward distributions satisfy Assumption 2.4 and 2.20, introduced in the previous chapter.

Due to its non-parametric nature, this algorithm can thus be used in many scenarios of interest beyond the standard bounded-rewards / change-in-the-mean framework. We discuss some of these scenarios in Section 3.5, where we validate numerically the potential of the approach by comparing it with a variety of state-of-the-art algorithms for non-stationary bandits. Hence, our contribution is not about providing novel insights on how non-stationarity can be handled by a bandit algorithm, but about analyzing the adaptation of a non-parametric algorithm to include a well-known passively forgetting strategy, that can tackle settings that are potentially not covered by existing approaches in terms of the family of distributions and changes that are allowed.

Preliminaries

Preliminaries

Just as in Chapter 2, the algorithms to be presented below are designed for the stochastic Karmed bandit problem. We briefly re-introduce the two variants of this basic model that will be considered in the chapter: stationary and abruptly changing environments.

Stationary environments

When the environment is stationary, we recall that the K arms are characterized by the reward distributions (ν k ) k≤K and their associated means (µ k ) k≤K , with µ ⋆ = max k∈{1,...,K} µ k denoting the highest expected reward. We denote by (Y k,s ) s∈N the i.i.d. sequence of rewards from arm k. We further recall that LB-SDA operates in successive rounds, whose length varies between 1 and K time steps. At each round r, the leader denoted ℓ(r) is defined and (K -1) duels with the remaining arms called challengers are performed. Denoting by N k (r) the number of pulls of arm k up to the round r, the leader is the arm that has been most pulled: ℓ(r) ∈ argmax k∈{1,...,K} N k (r). When several arms are candidate, the one with the largest sum of rewards is chosen. If this is still not sufficient to obtain a unique arm, the leader is chosen at random among the arms maximizing both criteria. At round r, a subset A r ⊂ {1, ..., K} is selected by the learner based on the outcomes of the duels against ℓ(r). Next, all arms in A r are drawn, yielding

Y k,N k (r) for k ∈ A r , where N k (r) = r s=1 1(k ∈ A s ).
We refer the reader to definition to Section 1.1 for the definition of the regret and the lower bounds of [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] for parametric families of distributions.

Abruptly changing environments In Section 3.4, we consider abruptly changing environments that we introduced in Section 1. 3. We recall that the number of breakpoints up to time T , denoted Γ T , is defined by

Γ T = T -1 t=1 1{∃k, ν k,t ̸ = ν k,t+1 },
and that the time instants (t 1 , ..., t Γ T ) associated to these breakpoints define Γ T + 1 stationary phases where the reward distributions are fixed. In such environments, letting µ ⋆ t denote the best arm at time t, the performance of a policy is measured through the dynamic regret defined as

R T = E T t=1 (µ ⋆ t -µ At ) .
In the non-stationary case, the lower bound for the regret takes a different form: for any strategy, there exists an abruptly changing instance such that R T = Ω( √ T Γ T ) [START_REF] Garivier | On upper-confidence bound policies for switching bandit problems[END_REF][START_REF] Seznec | A single algorithm for both restless and rested rotting bandits[END_REF]. In this chapter, we only consider this type of non-stationary environments, and hence target this theoretical performance.

LB-SDA with Limited-Memory

LB-SDA with Limited Memory in Stationary Environments

In this section we recall the general principle of LB-SDA and analyze a variant with limited memory, that we call LB-SDA-LM.

Last Block Sampling

We recall the general principle of SDA: at each round r, the algorithm (1) selects a leader ℓ(r), (2) makes this leader compete against every other arm (called challenger) in duels (pairwise comparisons), and ( 3) at the end of the round each winning challenger (if any) is pulled, otherwise the leader is pulled. This general principle is detailed in Algorithm 2.1, and in the following Algorithm 3.1 we simply recall the duel step of LB-SDA. In the rest of this chapter we assume that LB-SDA uses a forced exploration f r = log(r), as suggested in Chapter 2.

1 Input: 2 arms ℓ (leader) and k (challenger), History

H ℓ = (Y ℓ,1 , . . . , Y ℓ,N ℓ ) and H k = (Y k,1 , . . . , Y k,N k ), set of arms to pull A. 2 Define Ȳk,N k := 1 N k N k i=1 Y k,i and Ȳℓ,N ℓ -N k +1:N ℓ := 1 N k N k j=1 Y ℓ,N ℓ -N k +j 3 if N k ≤ log(r) or Ȳk,N k ≥ Ȳℓ,N ℓ -N k +1:N ℓ ;
▷ Using last block for the leader We finally recall that LB-SDA works because it provides a sufficient diversity of sub-samples so that the challenger has a fair chance of winning. This is due to the fact that the leader will be pulled a linear number of times during a run of the algorithm. According to Lemma 2.16 and Definition 2.10, the performance of LB-SDA depends only on the family of distributions of the arms, that needs to satisfy the balance condition (Definition 2.11) to ensure a logarithmic regret. For example, Single Parameter Exponential Families (SPEF) satisfy this condition with f r = log(r), and LB-SDA is even asymptotically optimal in that case as the upper bound of the regret matches the lower bound of [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF].

In the rest of this chapter we present our results assuming that the arms come from the same SPEF, as our focus is on the mechanisms to ensure the performance of LB-SDA in two settings with limited memory. We refer the interested reader to Chapter 2 for discussions on the assumptions that can be made on the arms' distributions to ensure the performance of LB-SDA. All the results we are going to present are valid for distributions satisfying the more general Assumptions 2.4 and 2.20.

LB-SDA with Limited Memory in Stationary Environments

Memory-Limited LB-SDA

One of our main motivations for further studying LB-SDA is its simplicity and efficiency. Yet, all existing subsampling algorithms [START_REF] Baransi | Sub-sampling for multi-armed bandits[END_REF]Chan, 2020;[START_REF] Baudry | Sub-sampling for efficient non-parametric bandit exploration[END_REF], including the vanilla version of LB-SDA (see Chapter 2), have to store the entire history of rewards for all the arms. In this section, we explain how to modify LB-SDA to reduce the storage cost while preserving the theoretical guarantees.

Consider a family of distributions for which LB-SDA has logarithmic regret. When T is large, the arm with the largest mean is the leader with high probability, and all the challengers should have a number of pulls that is of order O(log T ) only. With duels based on the last block, this would mean in particular that only the last O(log T ) observations from the optimal arm should be stored and that very old observations will never be used again in practice. Based on this intuition, one might think that keeping only log(T )/(µ ⋆ -µ k ) 2 observations is enough for LB-SDA. However, this could only be done with the knowledge of the true gaps.

We propose instead to limit the storage memory of each arm at round r to a quantity m r = Ω log(r) 1+γ , for some γ > 0. Following the definition of Agrawal and Goyal (2012b), we then define the set of saturated arms at a round r as

S r = {k ∈ {1, . . . , K} : N k (r) ≥ m r } .
The only modification of LB-SDA is the following: at each round r, if a saturated arm is pulled then the newly collected observation replaces the oldest observation in its history. The following result shows that LB-SDA-LM keeps the same asymptotical performance as LB-SDA for m r satisfying m r / log(r) → +∞. We stated this result for SPEF for simplicity, but the guarantees under Assumption 2.20 translate similarly. Θ where P Θ is any single parameter exponential family of distributions satisfying Assumption 2.4, if m r / log(r) → ∞, the regret of memory-limited LB-SDA satisfies, for all ε > 0,

R T ≤ k:µ k <µ ⋆ 1 + ε kl(µ k , µ ⋆ ) log(T ) + C ′ (ν, ε, M) ,
where M = (m 1 , m 2 , . . . , m T ) denotes the sequence (m r ) r∈N and C ′ (ν, ε, M) is a problemdependent constant.

LB-SDA with Limited-Memory

We detail the proof of this result, making explicit all the constant terms. Furthermore, we highlight that it also works for Lemma 2.7 for LB-SDA by setting m r = +∞ or m r = r in some parts of the proof (that we will point out).

Proof. We introduce a sequence m r of allowed memory for each arm at a round r. In the beginning of the proof we do not make any assumption on the sequence (m r ) r≥1 except that m r / log(r) → +∞, which is required in the statement of Theorem 3.1. We further assume that m r is an integer for any round r, which does not change anything for the algorithm but simplifies the notation for the proof. Without loss of generality, we assume that the arm 1 is the unique optimal arm, µ 1 = max k∈[K] µ k . We also recall that the arms are assumed to come from the same SPEF for simplicity, so Assumption 2.4 is satisfied for some rate functions K} . In terms of notation, we remark that if N k (r) ≥ m r and ℓ(r) ̸ = k then the duel between k and ℓ(r) is the comparison between Ȳk,N k (r)-mr+1:N k (r) and Ȳℓ(r),N ℓ(r) (r)-mr+1:N ℓ(r) (r) . Otherwise, if N k (r) ≤ m r and ℓ(r) ̸ = k then the duel is the comparison between Ȳk,N k (r) and Ȳℓ(r),N ℓ(r) (r)-N k (r)+1:N ℓ(r) (r) , which is the same as for the vanilla LB-SDA.

(I k ) k∈{1,...,
We recall that the set of saturated arms at round r is defined as S r = {k : N k (r) ≥ m r }.

To simplify the notation for each arm k we define the real number

x k = µ 1 +µ k 2 ∈ (µ k , µ 1 ), and write ω k = min(I 1 (x k ), I k (x k )).
Hence, we will write most of our results using concentration with this value ω k for arm k.

Our first step decomposes the number of pulls of arm k depending of if arm 1 is the leader or not, and if it is the case whether k is saturated or not, which gives

E[N k (T )] ≤ 1 + E T -1 r=1 1(ℓ(r) ̸ = 1) Zr + E T -1 r=1 1(k ∈ A r+1 , k / ∈ S r , ℓ(r) = 1) Gr + E T -1 r=1 1(k ∈ A r+1 , k ∈ S r , ℓ(r) = 1) Ḡr .
We first study Ḡr , and start the sum on the rounds at 2m 1 because two arms cannot be saturated before this round is reached. We upper bound Ḡr by taking a union bound on the sample size of each arm, and that if two random variables (X, Y ) satisfy X > Y then either X ≥ ξ or Y ≤ ξ for any real value ξ. We then obtain

Ḡr ≤ T -1 r=2m 1 P (ℓ(r) = 1, k ∈ A r+1 , N k (r) ≥ m r , N 1 (r) ≥ m r )

LB-SDA with Limited Memory in Stationary Environments

≤ T -1 r=2m 1 P N 1 (r) ≥ N k (r) ≥ m r , Ȳk,N k (r)-mr+1:N k (r) ≥ Ȳ1,N 1 (r)-mr+1:N 1 (r) ≤ T -1 r=2m 1 r n k =mr P Ȳk,n k -mr+1:n k ≥ x k , N k (r) = n k + T -1 r=2m 1 r n 1 =mr P Ȳ1,n 1 -mr+1:n 1 ≤ x k , N 1 (r) = n 1 ≤2 T -1 r=2m 1 re -mrω k ,
where we used the concentration inequality of Assumption 2.4, that is satisfied for SPEF.

We then consider G r and distinguish two cases, whenever N k (r) ≤ n 0 (T ) or not at each round, for some n 0 (T ) that will be specified later. First, we have that

G r ≤ n 0 (T ) + E T -1 r=1 1(k ∈ A r+1 , k / ∈ S r , ℓ(r) = 1, N k (r) ≥ n 0 (T )) .
We then use that on the event k / ∈ S r the duels played between k and 1 will be the classical duel with the last block: k will compete with its empirical mean and 1 with the mean of its last block of size N k (r). We define some η k ∈ (µ k , µ 1 ) and write

G r ≤ n 0 (T ) + E T -1 r=1 1(k ∈ A r+1 , k / ∈ S r , ℓ(r) = 1, N k (r) ≥ n 0 (T )) ≤ n 0 (T ) + E T -1 r=1 1(k ∈ A r+1 , Ȳk,N k (r) ≥ Ȳ1,N 1 (r)-N k (r)+1:N 1 (r) , N k (r) ≥ n 0 (T )) .
We then use the same trick as for Ḡr to separate the means of the two arms, before taking a union bound on the sample size for arm k, and use Lemma 2.6 for the leader. These steps give

G r ≤ n 0 (T ) + T -1 n k =n 0 (T ) P Ȳk,n k ≥ η k + T -1 n k =n 0 (T ) T -1 n 1 =n 0 (T ) P Ȳ1,n 1 -n k +1:n 1 ≤ η k .
for any η k ∈ R. We remark that the sum on r disappeared for arm k. Indeed, the fact that arm

k is pulled ensures that T -1 r=1 1(k ∈ A r+1 , N k (r) = n k ) ≤ 1.
We finally obtain

LB-SDA with Limited-Memory

G r ≤ n 0 (T ) + e -n 0 (T )I k (η k ) 1 -e -I k (η k ) + T e -n 0 (T )I 1 (η k ) 1 -e -I 1 (η k ) .
We then calibrate n 0 (T ) and η k in order to makes these terms converge properly. We define ε > 0 and state n 0 (T ) = 1+ε I 1 (µ k ) log T . We then use the continuity of the rate functions on (µ k , µ 1 ) to state that for any ε > 0 there exists

η k ∈ (µ k , µ 1 ) satisfying I 1 (η k ) ≥ I 1 (µ k ) 1+ε .
That translates in our result to T e -n 0 (T )I 1 (η k ) ≤ 1, and the remaining term is upper bounded by a constant that depends on ε (typically, in O(ε -2 )). Hence, for any ε > 0 it holds that

G r ≤ 1 + ε I 1 (µ k ) log T + C k,ε ,
where C k,ε is a constant. Combining these results we can write a first upper bound on

E[N k (T )] as E[N k (T )] ≤ 1 + 1 + ε I 1 (µ k ) log T + 2 T -1 r=2m 1 re -mrω k + C k,ε + T -1 r=2m 1 P(ℓ(r) ̸ = 1) . ( 3.1)
We remark that without the memory limit (equivalently m r = +∞) the first sum vanishes. Indeed this expression provides an explicit dependence in m r , that justifies the condition in Theorem 3.1 for m r ( namely, m r /(log r) → +∞) to make the sum converge.

We then work on upper bounding Z r . As in the proof of Chan (2020) this part is the most technically challenging. In the next steps we will consider the same events as in the original proof, but the storage limitation will add some complexity to the task. We first recall that at each round the leader satisfies

ℓ(r) = k ⇒ N k (r) ≥ r K .
However, adding the storage constraint we have that for any r satisfying r ≥ Km r the leader has necessarily more than m r observations. For this reason, its history will be always truncated to the m r last observations. For r is reasonably large, m r is still large enough to guarantee a good concentration of the empirical mean of the saturated arms. We define a r = r 4 , and write

P (ℓ(r) ̸ = 1) = P ({ℓ(r) ̸ = 1} ∩ D r ) + P {ℓ(r) ̸ = 1} ∩ Dr . (3.2)
We define D r the event under which the optimal arm has been leader at least once in [a r , r]. r] such that ℓ(u) = 1}.

D r = {∃u ∈ [a r ,

LB-SDA with Limited Memory in Stationary Environments

We then upper bound the term in the left hand side of Equation (3.2). As Chan (2020), we use that if arm 1 is not the leader and D r holds, a leadership takeover happened (arm 1 was leader, arm k becomes the next leader). In that case, some arm k ′ had at some point a better empirical average as the leader while having the same sample size. We denote this event by D k ′ r . For any r 0 ∈ N it holds that T -1

r=r 0 P D k ′ r ≤ E T -1 r=r 0 r u=ar 1( Ȳ1,N 1 (u) ≤ Ȳk ′ ,N k ′ (r) , N 1 (u) = N k ′ (u), N 1 (u) ≤ m u ) + E T -1 r=r 0 r u=ar 1( Ȳ1,N 1 (u)-mr:N 1 (r) ≤ Ȳk ′ ,N k ′ (r)-mr:N k ′ (r) , N 1 (u) = N k ′ (u), N 1 (u) ≥ m u ) .
We remark that if r 0 is large enough such that for r ≥ r 0 it holds that

a r K -1 ≤ m ar ,
then the first term is equal to 0, and the leader is necessarily saturated. It remains to upper bound the second term. Similarly as for the upper bound of Ḡr a union bound on the sample sizes provides T -1

r=r 0 P D k ′ r ≤ E T -1 r=r 0 r u=ar 1 N 1 (u) = N k ′ (u), Ȳ1,N 1 (r)-mr+1:N 1 (r) ≤ Ȳk ′ ,N k ′ (r)-mr+1:N k ′ (r) ≤ r 0 + T -1 r=r 0 r u=max(ar,2m 1 ) 2ue -muω k ′ ≤ r 0 + 2 T -1 r=r 0 r 2 e -ma r ω k ′ .
We first use this result without commenting its dependence in the sequence (m r ) r≥1 . Summing on all suboptimal arms k ′ we obtain T -1 .3) Again, the constraint m r / log(r) → +∞ is sufficient to ensure a proper convergence of this sum to a constant with the same arguments as before, because a r is still linear in r.

r=r 0 P ({ℓ(r) ̸ = 1} ∩ D r ) ≤ 2 K k ′ =2 T -1 r=r 0 r 2 e -ma r ω k ′ . ( 3

LB-SDA with Limited-Memory

Remark 3.2. In that case we can again set m r = +∞ to obtain the proof without limited memory, so this last term would be 0. However, our argument to discard the "unsaturated leader" case is not valid anymore. Deriving an upper bound for this term would give

2 K k ′ =2 T -1 r=1 e -brω k ′ instead, which is smaller.
Hence, the limited memory makes sub-optimal leadership takeovers more likely (which is intuitive).

We then consider the event for which arm 1 has never been the leader between a r and r. The idea in this part is to leverage the fact that if the optimal arm is not leader between ⌊r/4⌋ and r, then it has necessarily lost a lot of duels against the current leader at each round. We then use the fact that when the leader has been drawn "enough", concentration prevents this situation with large probability. We introduce

L r = r u=ar 1(C u ) , with C u defined as C u = {∃k ̸ = 1, ℓ(u) = k, 1 / ∈ A u+1 }.
The following holds

P(ℓ(r) ̸ = 1 ∩ D r ) ≤ P(L r ≥ r/4) . ( 3.4)
This result comes from (Chan, 2020), along with the direct use of the Markov inequality to provide the upper bound .5) We further decompose P(C u ) in two parts according to the number of selections of arm 1. For a constant C > 0, we write T -1

P(L r ≥ r/4) ≤ E(L r ) r/4 = 4 r r u=ar P(C u ) . ( 3
r=r 0 P({ℓ(r) ̸ = 1} ∩ D r ) ≤ T -1 r=r 0 4 r r u=ar P N 1 (u) ≤ C 4 log(u) D1 + T -1 r=r 0 4 r r u=ar P C u , N 1 (u) ≥ C 4 log(u) D2 .
We consider D2 . Again, we decompose it according to if the optimal arm is saturated or not. We also introduce K}. We first upper bound

C u k = {ℓ(u) = k, 1 / ∈ A u+1 } for any k ∈ {2, . . . ,

LB-SDA with Limited Memory in Stationary Environments

Dk,1 :

= T -1 r=r 0 4 r r u=ar P C u k , N 1 (u) ≥ C 4 log(u), 1 ∈ S u ≤ T -1 r=r 0 8 r r u=ar ue -ma u ω k ≤ 8 T -1 r=r 0 r u=ar e -ma u ω k ≤ 8 T -1 r=r 0 re -ma r ω k .
Then, we consider the alternative

Dk,2 := T -1 r=r 0 4 r r u=ar P C u k , N 1 (u) ≥ C 4 log(u), 1 / ∈ S u ≤ T -1 r=r 0 4 r r u=ar P Ȳk,N k (u)-N 1 (u)+1:N k (u) > Ȳ1,N 1 (u) , N k (u ) ≥ N 1 (u) ≥ C 4 log(u), 1 / ∈ S u ≤ T -1 r=r 0 4 r 1 1 -e -ω k e -C 4 log(ar)ω k + r 1 -e -ω k e -C 4 log(ar)ω k ≤ T -1 r=r 0 4(r + 1) r(1 -e -ω k ) e -C 4 log(ar)ω k ≤ T -1 r=r 0 6 1 -e -ω k e -C 4 log(ar)ω k .
So finally,

D2 ≤ K k=2 8 T -1 r=r 0 re -ma r ω k + T -1 r=r 0 6 1 -e -ω k e -C 4 log(ar)ω k .
At this step we need to choose the constant C large enough in order to make this sum converge, which is possible since C is only a parameter of the analysis. Furthermore, the first sum vanishes if we set m r = +∞, and again converges if m r / log(r) → +∞.

We then consider the term D1 . We transform the double sum in a simple sum by simply counting the number of times each term is included.,

D1 = T r=r 0 4 r r u=ar P N 1 (u) ≤ C 4 log(u) = T r=r 0   r t=ar 0 4 t 1(t ∈ [r, 4r])   P N 1 (r) ≤ C 4 log(r) .

LB-SDA with Limited-Memory

If we remark that r Hence, the storage limit may introduce larger constant terms in the proof, but asymptotically the dominant terms are the same as in the proof of the vanilla LB-SDA algorithm.

Remark 3.3. This step concludes the proof of Lemma 2.7 for LB-SDA by setting m r = +∞ when the memory limit is used. Note that the proof only requires slight adaptations for any block sampler. Details can be found in [START_REF] Baudry | Sub-sampling for efficient non-parametric bandit exploration[END_REF].

The last step is to show that we can upper bound the remaining term with the same results as the ones we used in Chapter 2. We only need to prove that if r 0 is large enough and {N 1 (r) ≤ C/4 log(r)}, then the arm 1 has not been saturated for a long time. Indeed, in that case the saturation would have no impact in the way to upper bound this term. Defining m -1 (x) = inf{r : m r ≥ x}, and knowing that if C and r ≥ r 0 (we can increase r 0 ) are large enough m r ≥ C log r and so m -1 (x) ≤ exp(x/C), we have m -1 (C/4 log r) ≤ exp(C/4 log(r)C -1 ) = r 1/4 . Hence, after the round r 0 we are sure that arm 1 has never been saturated since the round r 1/4 , but also that the sub-sample required by LB-SDA-LM at each step will not be altered by the memory limit too. In this scenario LB-SDA-LM does exactly the same as LB-SDA. As LB-SDA ensures sufficient diversity (Lemma 2.16) we conclude that if the arms come from the same SPEF,

T r=r 0 P N 1 (r) ≤ C 4 log(r) = O(1) .

Storage and Computational Cost

To the best of our knowledge, LB-SDA-LM is the only bandit algorithm based on the idea of sub-sampling that does not require to store the full history of rewards. In Table 3.1 we compare estimates of the computational cost of LB-SDA-LM and the other algorithms based on sub-sampling. We consider best and worst cases since SSMC and LB-SDA can be updated se-

LB-SDA with Limited Memory in Stationary Environments

quentially when the leader does not change and is pulled. The estimates we provide correspond to the cost of a single step of the algorithm at time T .

O(T ) O(1)-O(log T ) LB-SDA-LM (this chapter) O((log T ) 2 ) O(1)-O(log T )
Efficient updates for LB-SDA Let us detail the possible scenarios for a given leader and challenger. Assume that at round r the leader was using the sample mean ȲN-n+1:N for some

(n, N ) ∈ N 2 .
If it is pulled then Y N +1 is collected, and at next round it will use ȲN-n+2:N+1 against the same challenger. An efficient update consists in computing

ȲN-n+2:N+1 = ȲN-n+1:N + 1 n (Y N +1 -Y N -n+1 ) ,
which comes at almost no cost. Furthermore, an efficient update can also be performed if the challenger is pulled. In that case, the leader is not pulled and needs to use ȲN-n:N , that can be computed as ȲN-n:N = n n+1 ȲN-n+1:N + 1 n+1 Y n . Hence, the most costly scenario is when the leader changes, and new sub-sample means have to be computed. However, our proof shows that the number of changes of leadership is expected to be finite if there a single best arm.

Updating SSMC We can compare these results to possible updates of SSMC. We denote by Y - r = min j∈[N -n+1] Ȳj:j+n-1 the sub-sample mean used at a given round against some challenger. In the first scenario we described (when the leader is pulled), we need to compute

Y - r+1 = min{Y - r , ȲN-n+2:N+1 } at next round.
If the value of Y - r and of the sub-sample mean of the last block (as for LB-SDA) are kept in memory, then the update costs the same as for LB-SDA with Limited-Memory LB-SDA: only the last block can change the outcome of the duel. In the second and third scenario, however, the leader needs to perform a screening of its entire history. This comes at a linear cost in terms of the sample size of the leader, which is itself linear in the round r. Contrarily to the number of leadership changes, the number of rounds when a challenger is pulled is expected to be logarithmic in the time horizon.

Details for other algorithms

The computational cost can be broken into two parts: (a) the sub-sampling cost and (b) the computation of the means of the samples. We assume that drawing a sample of size n without replacement has O(n) cost (independently of the size of the set) and that computing the mean of this sub-sample costs another O(n). Furthermore, at round T , each challenger to the best arm has about O(log T ) samples. This gives an estimated cost of O (log T ) 2 for BESA [START_REF] Baransi | Sub-sampling for multi-armed bandits[END_REF]. For RB-SDA (see Chapter 2) the estimated cost is O(log(T )), because the sampling cost for random block sampling is O(1) but a sub-sample mean is recomputed at each round.

For the three deterministic algorithms (namely SSMC (Chan, 2020), LB-SDA, LB-SDA-LM), when the leader arm wins all its duels, we assume that the sequential update costs O(1). This is the best case in terms of computational cost. However, we detailed that in some cases SSMC requires a full screening of the leader's history, with O(T ) cost, while LB-SDA and LB-SDA-LM need at most the computation of the mean of the last O(log T ) samples from the leader.

LB-SDA in Non-Stationary Environments

In stationary environments, LB-SDA achieves optimal regret rates, even when its decisions are constrained to use at most (for instance) O((log T ) 2 ) observations. One might think that this argument itself is sufficient to address non-stationary scenarios as the duels are performed mostly using recent observations. For instance, if the distribution of the best arm changes we can hope that LB-SDA will "adapt" to this change relatively fast. However, there are other cases where LB-SDA is not sufficient. For instance, if an arm has been bad for a long period of time and suddenly becomes the best arm, adapting to the change would still be prohibitively slow. For this reason, LB-SDA has to be equipped with an additional mechanism to perform well in non-stationary environments.

SW-LB-SA: LB-SDA with a Sliding-Window

We keep a round-based structure for the algorithm, where, at each round r, duels between arms are performed and the algorithm subsequently selects the subset of arms A r that will be pulled. In contrast to Section 3. 3.2, where we introduced a constraint on the number of observations

LB-SDA in Non-Stationary Environments

kept in memory for each arm, we propose here to use a sliding window of length τ to limit the historical data available to the algorithm to that of the last τ rounds. We highlight the fundamental difference between the two approaches: the sliding window will make LB-SDA forget old observations for all arms, whether they have been sampled a lot or not.

Modified leader definition

The introduction of a sliding window requires a new definition for the leader. By analogy with the stationary case, the leader could be defined as the arm that has been pulled the most during the τ last rounds. However, with the inclusion of the sliding window, a new phenomenon, which we call passive leadership takeover, can occur. Let us define

N τ k (r) = r-1 s=r-τ 1 (k ∈ A s+1
), the number of times arm k has been pulled during the last τ rounds and consider a situation with 3 arms {1, 2, 3}. Assume that the leader is arm 1 and at a round (r -1) we have N τ 1 (r -1) = N τ 2 (r -1). If the leader has been pulled τ rounds away and wins its duel against arm 2 but looses against arm 3, only arm 3 will be pulled at round r. Consequently, at round r, arm 2 will have a strictly larger number of pulls than arm 1 without having actually defeated the leader. This situation, illustrated on Figure 3.1, is not desirable. We fix this by imposing that any arm has to defeat the current leader to become the leader itself. Let us define

B r = {k ∈ A r+1 ∩ {N τ k (r + 1) ≥ min(r, τ )/K}} .
Then for any r ∈ N, we propose a new definition of the leader as round r + 1 as

ℓ τ (r + 1) =    argmax k∈{1,...,K} N τ k (r + 1) , if N τ ℓ τ (r) (r + 1) < min(r, τ )/(2K) . argmax k∈Br∪{ℓ τ (r)} N τ k (r + 1) otherwise.
This modified definition of the leader ensures that an arm can become the leader only after earning at least τ /K samples and winning a duel against the current leader, or if the leader loses so many duels that its number of samples falls under a fixed threshold. Thanks to this definition it always holds that N τ ℓ τ (r) (r) ≥ min(r, τ )/(2K).

Additional diversity flags

As in the vanilla LB-SDA, we use a sampling obligation to ensure that each arm has a minimal number of samples. However, in contrast to the stationary case, this very limited number of forced samples may not be sufficient to guarantee an adequate variety of duels, due to the forgetting window. To this end, the sampling obligation is coupled with a diversity flag. We define it as a binary random variable D τ k (r), satisfying D τ k (r) = 1 only when, for the last ⌈(K -1)(log τ ) 2 ⌉ rounds the three following conditions are satisfied: 1) some arm k ′ ̸ = k has been leader during all these rounds, 2) k ′ has not been pulled, and 3) k has not been pulled and satisfy N τ k (r) ≤ (log τ ) 2 . In practice, there is a very low probability that these conditions are met simultaneously but this additional mechanism is required for the theoretical analysis. Note that the diversity flags have no impact on the computational cost A blue square is added when an arm has an observation for the corresponding round and the red square correspond to the information that will be lost at the end of the round due to the sliding window.

LB-SDA with Limited-Memory r -τ r -τ r -τ r -1 r -1 r -1 l o o s e w in ℓ(r -1) ℓ(r -1) ℓ(r -1) ℓ(r) ℓ(r) ℓ(r) round r r r r -τ + 1 r -τ + 1 r -τ + 1 r r r
of the algorithm as they require only to store the number of rounds since the last draw of the different arms (which can be updated recursively) as well as the last leader takeover. Arms that raise their diversity flag are automatically added to the set of pulled arms.

Bringing these parts together and keeping a forced exploration f τ = log(τ ) gives the pseudo-code of SW-LB-SDA in Algorithm 3.2.

Regret Analysis in Abruptly Changing Environments

In this section we aim at upper bounding the dynamic regret in abruptly changing environments, as defined in Section 3.2. Our main result is the proof that the regret of SW-LB-SDA matches the asymptotic lower bound of [START_REF] Garivier | On upper-confidence bound policies for switching bandit problems[END_REF].

R T = O T Γ T log T ,
if Assumptions 2.4 and 2.20 are satisfied during each stationary phase.

To prove this result we need to assume that, during each stationary period, the rewards satisfy Assumptions 2.4 (concentration of the means with exponential decay) and 2.20 (dominant left tail for the best arm). In contrast, current state-of-the-art algorithms for non-stationary bandits typically require the assumption that the rewards are bounded to obtain similar guarantees.

LB-SDA in Non-Stationary Environments

Hence, this result is of particular interest for tasks involving unbounded reward distributions. SW-LB-SDA can also be used for general bounded rewards with the same guarantees by using the binarization trick (Agrawal and Goyal, 2013b). However, the knowledge of the horizon T and the estimated number of change point Γ T is still required to obtain optimal rates, and removing this assumption is an interesting direction for future works, for instance inspired by [START_REF] Auer | Adaptively tracking the best bandit arm with an unknown number of distribution changes[END_REF][START_REF] Besson | Efficient change-point detection for tackling piecewise-stationary bandits[END_REF]. In the following we provide a high-level outline of the analysis of Theorem 3.4. Our objective is to provide an intuition on the dominant terms of the regret, and on the interest of the additional mechanisms introduced compared to LB-SDA.

Proof sketch For the Γ T + 1 stationary phases [t ϕ , t ϕ+1 -1] with ϕ ∈ {1, . . . , Γ T }, we define r ϕ as the first round where an observation from the phase ϕ was pulled. Introducing the gaps

∆ ϕ k = µ * t ϕ -µ t ϕ ,k
and denoting the optimal arm k ⋆ ϕ , we can rewrite the regret as

R T = E    Γ T ϕ=1 r ϕ+1 -2 r=r ϕ -1 k̸ =k ⋆ ϕ 1 (k ∈ A r+1 ) ∆ ϕ k    = Γ T ϕ=1 k̸ =k * ϕ E[N ϕ k ]∆ ϕ k ,
where we define

N ϕ k = r ϕ+1 -2 r=r ϕ -1 1(k ∈ A r+1
) as the number of pulls of arm k during a phase ϕ when it is suboptimal. We highlight that the sequence (r ϕ ) ϕ≥1 is a random variable that depends on the trajectory of the algorithm, but this causes no additional difficulty for upper bounding the regret. We introduce δ ϕ = t ϕ+1 -t ϕ the length of a phase ϕ. Using Lemma 25 from [START_REF] Garivier | On upper-confidence bound policies for switching bandit problems[END_REF], we obtain the following result.

Proposition 3.5. Consider a phase ϕ and defineA ϕ,τ k = b ϕ k log(τ ) for some constant b ϕ k > 0. The expected number of pulls of a sub-optimal arm k can be upper bounded as

E[N ϕ k ] ≤ 2τ + δ ϕ A ϕ,τ k τ + c ϕ,τ k,1 + c ϕ,τ k,2 + c ϕ,τ k,3 ,
where denoting by D τ k (r) ∈ {0, 1} the variable stating if the diversity flag is raised or not we defined

c ϕ,τ k,1 = E   r ϕ+1 -2 r=r ϕ +2τ -2 1 k ∈ A r+1 , ℓ τ (r) = k * ϕ , N τ k (r) ≥ A ϕ,τ k , D τ k (r) = 0   , c ϕ,τ k,2 = E   r ϕ+1 -2 r=r ϕ +2τ -2 1 ℓ τ (r) = k * ϕ , D τ k (r) = 1   , c ϕ,τ k,3 = E   r ϕ+1 -2 r=r ϕ +2τ -2 1 ℓ τ (r) ̸ = k * ϕ   .

LB-SDA with Limited-Memory

Though the notation is more complicated these three terms are actually quite similar to the decomposition in the stationary case. Furthermore, our objective will be to prove that they are actually not dominant in the regret bound. Re-defining the notion of saturated arm of previous section as each arm k that have been sampled more than the threshold A ϕ,τ k it holds that • c ϕ,τ k,1 is an upper bound on the expected number of times a saturated sub-optimal arm can defeat k * ϕ while k * ϕ is the current leader.

• c ϕ,τ k,2 is an upper bound on the regret that can be caused by the diversity flag.

• c ϕ,τ k,3 is, as

E T -1 r=1 1(Z r )
in the previous section an upper bound of the regret that can be caused by the leader being a sub-optimal arm.

The three terms have hence intuitive interpretation and summarize well the technical contributions behind Theorem 3. 4. We now introduce the novel concentration result for SW-LB-SDA that will allow us to upper bound these terms.

Lemma 3.6. We consider a stationary phase ϕ and a MAB (ν ϕ 1 , . . . , ν ϕ K ). Let k * ϕ denote the arm with the largest mean. We assume that each arm ν ϕ k satisfies Assumption 2.4 for some rate function

I ϕ k . Then, for any constant n ∈ N satisfying n ≥ f τ = √ log τ , by letting ñ = min(n, ⌊τ /(2K)⌋) under SW-LB-SDA it holds that E   r ϕ+1 -2 r=r ϕ +2τ -2 1 k ∈ A r+1 , ℓ τ (r) = k * ϕ , N τ k (r) ≥ n, D τ k (r) = 0   ≤ δ ϕ (τ + 1) e -ñω ϕ k 1 -e -ω ϕ k , ( 3.7)
where we defined

ω ϕ k = min I ϕ k 1 2 (µ ϕ k + µ ϕ k * ϕ ) , I ϕ k * ϕ 1 2 (µ ϕ k + µ ϕ k * ϕ
) , and δ ϕ is the length of the phase and τ the size of the sliding window. Similarly, For the sake of simplicity we choose

E   r ϕ+1 -2 r=r ϕ +τ -2 1 k * ϕ / ∈ A r+1 , ℓ τ (r) = k, N τ k * ϕ (r) ≥ n   ≤ δ ϕ (τ + 1) e -ñω ϕ k 1 -e -ω ϕ k . ( 3
ξ k = 1 2 (µ ϕ k + µ ϕ k * ϕ ).
We then write

A := E   r ϕ+1 -2 r=r ϕ +2τ -2 1 k ∈ A r+1 , ℓ τ (r) = k * ϕ , N τ k (r) ≥ n, D τ k (r) = 0   ≤ E   r ϕ+1 -2 r=r ϕ +2τ -2 1 k ∈ A r+1 , µ τ k * ϕ ,k (r) ≤ ξ k , N τ k * ϕ (r) ≥ τ /(2K), N τ k (r) ≥ n   + E   r ϕ+1 -2 r=r ϕ +2τ -2 1 k ∈ A r+1 , µ τ k (r) ≥ ξ k , N τ k * ϕ (r) ≥ τ /(2K), N τ k (r) ≥ n   .
Assuming rewards are sequentially collected in a stream

Y ϕ k,1 , . . . , Y ϕ k, ∆ ϕ
for a given arm k, all possible blocks of observations are uniquely described by two quantities: N ϕ k (r) (number of data collected since the beginning of phase ϕ) and N τ k (r) (number of data collected in the last τ rounds). We will use this property to bound the two sums.

We start by the term featuring the arm k, and introduce

S n,m k (r) = {k ∈ A r+1 , µ τ k (r) ≥ ξ k , N ϕ k (r) = m + n -1, N τ k (r) = n} ,
that allows to use that

{k ∈ A r+1 , µ τ k (r) ≥ ξ k , N τ k (r) ≥ n} ⊂ δϕ n k =n δϕ m k =1 S n k ,m k k (r) . (3.9)
Furthermore, the same block (same value for both n and m) can not be used for upcoming rounds because the total count will be incremented. More specifically, for the arm k for any possible block there is at most one round for which the indicator function can be 1., i.e.

δϕ

n k =n δϕ m k =1 r ϕ+1 -2 r=r ϕ +2τ -2 1 S n k ,m k k (r) ≤ δϕ n k =n δϕ m k =1 1 Ȳk,m k :m k +n k -1 ≥ ξ k .
Similarly, we denote Y k ⋆ ϕ ,1 , ..., Y k ⋆ ϕ , δϕ the set of possible rewards for the arm k ⋆ ϕ and let

S n,m k ⋆ ϕ (r) = {k ∈ A r+1 , µ τ k ⋆ ϕ ,k (r) ≤ ξ k , N ϕ k * ϕ (r) = m + n -1, N τ k * ϕ (r) = n} .
We also have .10)

{k ∈ A r+1 , µ τ k ⋆ ϕ ,k (r) ≤ ξ k , N τ k ⋆ ϕ (r) ≥ n ′ } ⊂ δϕ n ⋆ =n ′ δϕ m ⋆ =1 S n ⋆ ,m ⋆ k ⋆ ϕ (r) . ( 3

LB-SDA with Limited-Memory

The main difference here is that several rounds can use the same block of observations of k ⋆ ϕ . This can be explained because when the indicator function equals 1 the arm k is drawn instead of k ⋆ ϕ and the previous argument do not hold anymore. Yet, N τ k * ϕ (r) can not remain unchanged for more than τ steps because of the sliding window. This implies in particular,

δϕ n ⋆ =n ′ δϕ m ⋆ =1 r ϕ+1 -2 r=r ϕ +2τ -2 1(S n ⋆ ,m ⋆ k ⋆ ϕ (r)) ≤ τ δϕ n ⋆ =n ′ δϕ m ⋆ =1 1 Ȳk * ϕ ,m ⋆ :m ⋆ +n ⋆ -1 ≤ ξ k .
Bringing things together and applying the previous inequality with n ′ = ⌊τ /(2K)⌋ we obtain

A ≤ E    δϕ m ⋆ =1 δϕ n ⋆ =n ′ τ 1 Ȳk * ϕ ,m ⋆ :m ⋆ +n ⋆ -1 ≤ ξ k + δϕ m k =1 δϕ n k =n 1 Ȳk,m k :m k +n k -1 ≥ ξ k    .
We then have to handle carefully the fact that δ ϕ is a random variable depending on the bandit algorithm. Indeed, as several arms can be pulled at each round we don't know what will be the length of a phase in terms of rounds. However, this quantity is upper bounded by the length of the phase in terms of arms pulled δ ϕ .

Thus, using the concentration inequality corresponding to the family of distributions for an appropriate rate function we can write

A ≤ δ ϕ m ⋆ =n δ ϕ n ⋆ =n ′ τ P Ȳk * ϕ ,m ⋆ :m ⋆ +n ⋆ -1 ≤ ξ k + δ ϕ m k =1 δ ϕ n k =n P Ȳk,m k :m k +n k -1 ≥ ξ k ≤ δ ϕ m ⋆ =1 δ ϕ n ⋆ =n ′ τ e -n ⋆ I k * ϕ (ξ k ) + δ ϕ m k =n δ ϕ n k =n e -n k I k (ξ k ) ≤ δ ϕ   τ e -n ′ I k * ϕ (ξ k ) 1 -e -I k * ϕ (ξ k ) + e -nI k (ξ k ) 1 -e -I k (ξ k )   ≤ δ ϕ (τ + 1) e -nω k 1 -e -ω k ,
where in the last inequality we have introduced ñ = min(n, n ′ ) = min(n, ⌊τ /(2K)⌋).

Finally, the proof of the second statement is a direct adaptation of this proof by inverting k and k * ϕ . We don't need the event D ϕ k (r) = 0 because if k * ϕ is not drawn it has necessarily lost its duel against the leader k.

LB-SDA in Non-Stationary Environments

Upper bounding c ϕ,τ k,1 Equation (3.7) in Lemma 3.6 is enough to upper bound c ϕ,τ k,1 , by replacing n by A ϕ,τ k . Assuming that A ϕ,τ k ≤ τ /(2K) it holds that .11)

E[c ϕ,τ k,1 ] ≤ δ ϕ (τ + 1) e -A ϕ,τ k ω k 1 -e -ω k . ( 3
Upper bounding c ϕ,τ k,2
If the diversity flag is activated while k * ϕ is leader, then k * ϕ has lost at least ⌈(K -1)(log τ ) 2 ⌉ successive duels while being leader. Hence, for at least one of them the sub-optimal arm has at least (log τ ) 2 observations, and by definition the diversity flag was not activated during this round. Hence, we can apply again Lemma 3.6 with n = (log τ ) 2 , obtaining .12)

E[c ϕ,τ k,2 ] ≤ k ′ ̸ =k * ϕ δ ϕ (τ + 1) e -(log τ ) 2 ω k ′ 1 -e -ω k ′ . ( 3
Upper bounding c ϕ,τ k,3
As in the stationary case, this term is the most difficult to handle. The main challenge is to upper bound the probability that the optimal arm is not saturated after a large number of rounds.

The remaining parts of the proof are similar to the stationary case. We first consider the case when the optimal arm has already been leader during the last τ rounds. The additional mechanisms in the non stationary case (new definition of leader, diversity flag) adds new possible scenario for a leadership takeover, and details can be found in [START_REF] Baudry | On Limited-Memory Subsampling Strategies for Bandits[END_REF]. We only report that the contribution of this term to the upper bound of c ϕ,τ k,3 is of the form

D ϕ := 3(K -1)δ ϕ (τ + 1) 3 k̸ =k * ϕ e - τ 2K(K-1) ω k 1 -e -ω k ,
which comes from identifying the relevant events for leadership takeovers and using Lemma 3. 6.

For large enough τ this term is actually smaller than the previous ones we derived due as it decays exponentially with τ (up to polynomial terms).

We then consider the case when k * ϕ has never been leader during the last τ rounds, and use Markov inequality to relate this event to the number of duels lost by arm k * ϕ and obtain an upper bound

Dϕ := E   r ϕ+1 -2 r=r ϕ +2τ -1 2 τ r-1 s=r-τ 1 k * ϕ / ∈ A s+1 , ℓ τ (s) ̸ = k * ϕ   .
We then consider whether

N τ k * ϕ (r) ≥ A ϕ,τ k * ϕ or not.
If this is the case we can use Lemma 3.6 to derive a first upper bound

LB-SDA with Limited-Memory

Dϕ 1 ≤ 2E   r ϕ+1 -2 r=r ϕ +2τ -2 1 k * ϕ / ∈ A r+1 , ℓ τ (r) ̸ = k * ϕ , N τ k * ϕ (r) ≥ A ϕ,τ k * ϕ   ≤ 2δ ϕ (τ + 1) k̸ =k * ϕ e -A ϕ,τ k * ϕ ω k 1 -e -ω k .
The rest of the proof requires additional work compared with the stationary case. Indeed, if k * ϕ has been pulled a lot in the previous windows its index may change a lot. To avoid this we further consider whether

N τ k * ϕ (r -τ ) ≥ A ϕ,τ k * ϕ or not.
In the first case, k * ϕ necessarily lost a duel with exactly A ϕ,τ k * ϕ observations at some point. Using a union bound and Lemma 3.6 this event contributes to our upper bound up to the following factor

Dϕ 2 := δ ϕ τ (τ + 1)e -A ϕ,τ k * ϕ ω k .
In the last remaining case both

N τ k * ϕ (r -τ ) and N τ k * ϕ (r) are smaller than A ϕ,τ k * ϕ .
In that case, we can finally use the arguments introduced in Chapter 2 to conclude, evaluating the diversity of sub-samples and using the balance condition. Thanks to these properties we know that during the interval [r -τ, r -1], k * ϕ lost at least τ -A ϕ,τ k * ϕ duels and that a fraction 1/(2A ϕ,τ k * ϕ ) of them occurred while the index of k * ϕ remained the same. Applying the same methodology as in the stationary case we can identify that there exists some β ∈ (0, 1) such that for any value of τ large enough k * ϕ lost at least a number of duels M τ against non-overlapping blocks of some challenger k, with a fixed sub-sample of size larger than f τ (the forced exploration), with

M τ =     βτ 2(K -1) 2 (log τ ) 2 (A ϕ,τ k * ϕ ) 2     .
These observations allow to obtain a final contribution to our upper bound with the term

Dϕ 3 := 2δ ϕ A ϕ,τ k * ϕ k̸ =k * ϕ A ϕ,τ k * ϕ j= √ log τ α ϕ k (M τ , j) ,

LB-SDA in Non-Stationary Environments

where we assumed for simplicity that A ϕ,τ k * ϕ is integer. Here α ϕ k are balance functions, as defined in Definition 2.9, for arm k and phase ϕ.

We recall (Proposition 2.23) that for SPEF the balance function satisfies

α k (M τ , j) ≤ e -jω ϕ k u + (1 -u) M τ .
for some constant ω ϕ k . We choose the value u = 3 log τ M τ , which leads to

(1 -u) M τ = exp (M τ log(1 -u)) = exp M τ log 1 - 3 log τ M τ ≤ exp (-3 log τ ) ≤ 1 τ 3 .
If we plug this expression to upper bound the sums we obtain

Dϕ 3 ≤ 2δ ϕ A ϕ,τ k * ϕ (K -1)   e - √ log τ ω ϕ 1 -e -ω ϕ 3 log τ M τ + A ϕ,τ k * ϕ τ 3   ,
where ω ϕ = min k̸ =k * ϕ ω ϕ k . Even if these terms look impressive we explain in the next section that they are not first order terms in the regret analysis.

Summary Due to the large number of terms introduced in the analysis we provide in this section a clarification of the final upper bound we obtained for the regret. We proved that for any given phase ϕ,

E[N ϕ k ] ≤ 2τ + δ ϕ A ϕ,τ k τ + E[c ϕ,τ k,1 ] + E[c ϕ,τ k,2 ] + D ϕ + Dϕ 1 + Dϕ 2 + Dϕ 3 ,
where we provided explicit upper bounds for all the terms, summing as

E[N ϕ k ] ≤ 2τ + δ ϕ A ϕ,τ k τ + δ ϕ (τ + 1) e -A ϕ,τ k ω k 1 -e -ω k + δ ϕ (τ + 1) k ′ ̸ =k * ϕ e -(log τ ) 2 ω k ′ 1 -e -ω k ′ + 3(K -1)δ ϕ (τ + 1) 3 k̸ =k * ϕ e - τ 2K(K-1) ω k 1 -e -ω k + 2δ ϕ (τ + 1) k̸ =k * ϕ e -A ϕ,τ k * ϕ ω k 1 -e -ω k

LB-SDA with Limited-Memory

+ δ ϕ τ (τ + 1)e -A ϕ,τ k * ϕ ω k + 2δ ϕ A ϕ,τ k * ϕ (K -1)   e - √ log τ ω ϕ 1 -e -ω ϕ 3 log τ M τ + A ϕ,τ k * ϕ τ 3   .
While this bound is relatively scary, we can now tune A ϕ,τ k and τ to obtain the desired order of magnitude. First, we can tune A ϕ,τ k = O(log(τ )) large enough in order to make all the terms with A ϕ,τ k in their exponent as o δ ϕ τ . We then remark that this condition is also satisfied by the terms with exponents in Ω((log τ ) 2 ) and Ω(τ ). The most challenging term is the first term of the upper bound of Dϕ 3 . Thankfully, the exponent in log(τ ) ensures an upper bound in e.g

O   δ ϕ A ϕ,τ k * ϕ τ log(τ )   .
Putting these results together we conclude that

E[N ϕ k ] ≤ 2τ + O δ ϕ log(τ ) τ + o δ ϕ log(τ ) τ .
We can finally tune τ by considering the first two terms of the upper bound and summing on the phases, which provides the optimal tuning and guarantees of Theorem 3.4.

Experiments

In this section we test empirically the algorithms presented in this chapter. We first check that the performance of LB-SDA-LM is indeed close to the one of LB-SDA in stationary environments, and then implement some experiments with non-stationary arms.

Limiting the storage in stationary environments. In our first experiment1 reported on Figure 3.2, we compare LB-SDA and LB-SDA-LM on a stationary instance with K = 2 arms with Bernoulli distributions for a horizon T = 10000. We add natural competitors (Thompson Sampling [START_REF] Thompson | On the likelihood that one unknown probability exceeds another in view of the evidence of two samples[END_REF], kl-UCB [START_REF] Cappé | Kullback-leibler upper confidence bounds for optimal sequential allocation[END_REF]), that know ahead of the experiment that the reward distributions are Bernoulli and are tuned accordingly. The arms satisfy (µ 1 , µ 2 ) = (0.05, 0.15) with a gap ∆ = 0.1. We run LB-SDA-LM with a memory limit m r = log(r) 2 + 50, which gives a storage ranging from 50 to 150 samples for each arm (much smaller than the horizon T = 10000). The regret are averaged on 2000 independent replications and the upper and lower quartiles are reported. In this setup LB-SDA-LM performs similarly to KL-UCB, and the impact of limiting the memory is mild, when compared to LB-SDA. This illustrates that even with relatively small gaps (here 0.1), a substantial reduction of the storage can be done with only minor loss of performance with LB-SDA-LM. [START_REF] Raj | Taming non-stationary bandits: A bayesian approach[END_REF] and SW-TS [START_REF] Trovo | Sliding-window thompson sampling for non-stationary settings[END_REF]. We also add EXP3S (Auer et al., 2002a) designed for adversarial bandits and our SW-LB-SDA algorithm for the comparison. The different algorithms make use of the knowledge of T and Γ T .

Experiments

Experiments

evolution of the arm's means is pictured on the right of Figure 3.3 and Figure 3.5 displays the performance of the algorithms. CUSUM and M-UCB can not be applied in this setting because they both consider bounded distributions. Even if no theoretical guarantees have been proved for Thompson sampling with a sliding window or discount factors when the distribution are Gaussian, we add them as competitors. The analysis of SW-UCB and D-UCB was done under the bounded reward assumption but the algorithms can be adapted to the Gaussian case. Yet, the tuning of the discount factor and the sliding window had to be adapted to obtain reasonable performance, using τ = 2(1 + 2σ) T log(T )/Γ T for D-UCB and γ = 1 -1/(4(1 + 2σ)) Γ T /T for SW-UCB (considering that, practically, most of the rewards lie under 1 + 2σ). For reference, Figure 3.5 also displays the performance of the UCB1 algorithm that ignores the non-stationary structure. Clearly, SW-LB-SDA, in addition of being the only algorithm analyzed in this setting with unbounded rewards, also has the best empirical performance. Changes affecting the variance. The last experiment features the same Gaussian means but with different standard errors. The standard error takes the values 0.5, 0.25, 1 and 0.25, respectively, in the four stationary phases. The algorithms based on upper confidence bound are given the maximum standard error σ = 1, whereas SW-LB-SDA is not provided with any information of this sort. Figure 3.6 shows that the non-parametric nature of SW-LB-SDA is effective, with a significant improvement over state-of-the-art methods in such settings. Update

LB-SDA with Limited-Memory

N k = N k + 1, N τ k = N τ k + 1, t = t + 1 end for k ∈ {1, ..., K} do if k ∈ A r-τ +1 then 28 N τ k = N τ k -1 end end r = r + 1 end Algorithm 3.2: SW-LB-SDA

Introduction

In this thesis we consider the question of adapting Multi-Armed Bandits algorithms to work with alternative performance metric to the expected reward, that can be of practical relevance.

In Section 1.2.2 of Chapter 1 we introduced Extreme Bandit, that is a variant of MAB where the learner seeks to collect the largest possible reward. We discussed that the algorithms that have been developed in this setting could also be used for an alternative problem where the learner would like to maximize the expected minimum, or equivalently sample most often the arm with the lightest left tail (asymptotically, if it exists). While this variant is maybe more relevant for the case-study in agriculture that we introduced by being a way to model extreme risk-aversion of the farmers, we consider in this chapter following the version of this problem with the maximum.

Letting X k,t be the reward that would be obtained from sampling arm k at time t, a bandit algorithm (or policy) selects an arm I t using past observations and receives the reward X It,t . The rewards stream (X k,t ) is drawn i.i.d. from ν k and independently from other rewards streams. In this work, we assume that all arms have an unbounded support (the finite support case is studied by [START_REF] Nishihara | No regret bound for extreme bandits[END_REF]), and define the extreme regret of a policy In this context, [START_REF] Carpentier | Extreme bandits[END_REF] define the extreme regret of a policy as

R π T = max k≤K E[max t≤T X k,t ] -E π [max t≤T X It,t ] . ( 4.1)
We recall from Section 1.2.2 that we expect an extreme bandit algorithm to achieve a vanishing regret, in the weak sense (R π T = o (max k≤K E[max t≤T X k,t ])) or in the strong sense (if lim T →∞ R π T = 0). To obtain these guarantees, algorithms need to use available information on the tails of distributions. However, precise knowledge (e.g a parametric or semi-parametric model) may not be accessible to the learner in many realistic cases.

For this reason, we revisit the extreme bandit problem with the idea of designing algorithms based on pairwise comparisons of tails with provable guarantees under minimal assumptions on the arms. The motivation clearly stems from the study of bandit algorithms based on sub-sampling in Chapter 2 and 3, that perform "fair" pairwise comparisons of empirical means based on an equal sample size, and attain good performance for several types of distributions without changing the algorithm.

In Section 4.2, we highlight the limitation of comparing directly the maxima of n i.i.d. samples and introduce the Quantile of Maxima (QoMax) estimator. Instead of computing the maximum of n samples, the learner separates the collected data into batches of equal size and compute the quantile of order q of the maxima over the different batches. QoMax is inspired by the Median of Means estimator [START_REF] Alon | The space complexity of approximating the frequency moments[END_REF] that was used for heavy-tail bandits [START_REF] Bubeck | Bandits with heavy tail[END_REF]. We derive upper bounds on the probability that one QoMax exceeds Sub-sampling for Extreme Bandits we introduced in Chapter 1 (Definiton 1.12). We now define a general notion that allows to compare two (arbitrary) tails.

Definition 4.3 (Dominating tail).

Let G 1 and G 2 be the survival functions of two distributions ν 1 and ν 2 . We say that the tail of ν 1 dominates the tail of ν 2 (we write ν 1 ≻ ν 2 ) if the ratio of their survival functions is larger than a fixed constant for large enough values, that is

ν 1 ≻ ν 2 ⇐⇒ ∃C > 1, ∃x ∈ R : ∀y > x , G 1 (y) > CG 2 (y) .
In the rest of the chapter we will consider a bandit model that has a dominating arm, denoted by 1 without loss of generality: we assume that ν 1 ≻ ν k for all k ̸ = 1. Under this assumption, arm 1 is optimal in the sense that for T large enough an oracle strategy would select this arm only. To the best of our knowledge, this is the weakest assumption introduced so far for extreme bandits.

Direct Comparison of Maxima

Let ν X and ν Y be two distributions from which we observe n i.i.d. samples denoted by X 1 , . . . , X n and Y 1 , . . . , Y n respectively. A natural idea to compare their tails is to use the samples' maxima, that we denote by X + n and Y + n respectively. For these estimators to serve as a proxy for comparing the tails, we need the probability P(X + n < Y + n ) to decay fast enough when ν X ≻ ν Y , ideally exponentially with the sample size. Unfortunately, the following result shows that this is not possible even under semi-parametric assumptions.

Lemma 4.4 (Lower bound). Assume that both ν X and ν Y have either polynomial or exponential tails, with respective parameters

(C X , λ X ) and (C Y , λ Y ), with λ X < λ Y (so that ν X ≻ ν Y ). Then, P X + n ≤ Y + n = Ω n - λ Y λ X .
Proof. Let F X and F Y be the respective cdf of the two distributions. For any sequence (m n ) n∈N we lower bound the probability of interest as follows:

P(X + n ≤ Y + n ) ≥ P(Y 1 ≥ max 1≤i≤n X i ) = E Y 1 [F X (Y 1 ) n ] = R F X (x) n dF Y (x)

Comparing Tails of Distributions with Quantiles of Maxima

≥ +∞ mn F X (x) n dF Y (x) ≥ F X (m n ) n (1 -F Y (m n )) .
For exponential tails we can choose m n = 1 λ X log(n) and obtain an asymptotic equivalent of this term of e

-C X C Y n λ Y λ X
. Furthermore, choosing m n = n 1/λ X for polynomial tails provides the same result. This concludes the proof. Lemma 4.4 proves that direct comparison of maxima is not satisfying to obtain an exponential decay (in n) of P(X + n < Y + n ). However, it is also interesting to determine if there is a decay at all. We hence propose an upper bound of this probability. To obtain it, we use a trick that we already used in previous chapters: for any sequence (x n ), it holds that

P(X + n < Y + n ) ≤ P(X + n ≤ x n ) + P(Y + n > x n ) .
Using first that P(X

+ n ≤ x) = (1 -G X (x)) n ≤ exp(-nG X (x)) and then that P(Y + n > x) ≤ n i=1 P(Y i > x) = nG Y (x)
, and optimizing for x n yields the following result, that will be useful in the next section.

Lemma 4.5 (Comparison of Maxima under semi-parametric assumptions).

Assume that both ν X and ν Y have either polynomial or exponential tails, with respective second parameter λ X and λ Y , with λ X < λ Y (so that ν X ≻ ν Y ). Define δ = λ X λ Y -1 > 0, then there exists a sequence (x n ) and an integer n X,Y such that for all n ≥ n X,Y ,

max{P(X + n ≤ x n ), P(Y + n ≥ x n )} = O (log n) δ+1 n δ .
Proof. The key of the proof is to consider

x n "slightly" below G -1 X (1/n). Consider the expo- nential tails first, for which G X (x) ∼ C X exp(-λ 1 x) and G Y (x) ∼ C Y exp(-λ Y x).
We prove without loss of generality the result by continuing the proof as if the survival functions were exactly equal to their asymptotic equivalents. We choose

x n = 1 λ X (log n + log(C X ) -log(δ log n)) ,
and then compute

G X (x n ) and G Y (x n ). First, G X (x n ) = C X exp(-(log n + log C X -log(δ log n)))

Sub-sampling for Extreme Bandits

= δ(log n) n .
Then,

G Y (x n ) = C Y exp (-(δ + 1) (log n + log C X -log(δ log n)) = 1 n δ+1 × (δ log n) δ+1 × C Y C δ+1 X So finally, we obtain P Y + n ≥ x n = O (log n) δ+1 n δ and P X + n ≤ x n ≤ 1 n δ ,
which gives the first part of the result.

For polynomial tails we define the sequence

x n = (C X n) 1 λ X × (δ log n) -1 λ X .
We obtain exp(-nG X (x n )) = n -δ , and

nG 2 (x n ) = O (log n) δ+1 n δ
, completing the proof.

Lemma 4.5 shows that we can upper bound the decay rate of the probability that one maximum exceeds another. However, this rate δ is problem-dependent and can be arbitrarily small. As pointed out by [START_REF] Carpentier | Extreme bandits[END_REF] it can actually be seen as the Extreme Bandits equivalent of the gap in bandits, we therefore call δ the tail gap. With this notation the lower bound of Lemma 4.4 is of order Ω(n -(1+δ) ).

Interestingly, we can also obtain a result when the two distributions only satisfy ν X ≻ ν Y without further assumption. Lemma 4.6. Assume that ν X ≻ ν Y . Then, for any q ∈ (0, 1) there exists n ν X ,ν Y ,q ∈ N, a sequence (x n ) n∈N and some ε > 0 such that for all n ≥ n ν 1 ,ν 2 ,q ,

P(X + n ≤ x n ) ≤ q -ε , and P(Y + n ≤ x n ) ≥ q + ε .
Proof. Let q ∈ (0, 1) and ε ∈ (0, q). We define the sequence (x n ) by

G X (x n ) = 1 -(q -ε) 1 n , so that P(X + n ≤ x n ) = q -ε. As ν X ≻ ν Y , there exists a constant C > 1 such that G X (x) ≥ CG Y (x) for x large enough. Hence, as x n → +∞ it holds that G 1 (x n ) > CG 2 (x n ) for n large
P(Y + n ≤ x n ) = (1 -G Y (x n )) n ≥ (1 - 1 C G X (x n )) n = 1 - 1 C 1 -(q -ε) 1 n n .
We then use that for 0

≤ x ≤ 1, log(1 -x) ≥ -x √ 1-x to get P(Y + n ≤ x n ) ≥ exp   n C (q -ε) 1 n -1 × 1 1 -1 C (1 -(q -ε) 1 n )   ≥ exp   log(q -ε) C 1 1 -1 C (1 -(q -ε) 1 n )  
and then state that for n large enough this lower bound can be arbitrarily close to (q -ε)

1 C , which can be made strictly larger than q + ε if ε is small enough, as C is fixed. Hence, for an appropriate choice of ε we found a sequence x n satisfying the statement of the lemma.

Quantile of Maxima (QoMax)

Results similar to those of Section 4.2.1 have been previously encountered in the bandit literature. In [START_REF] Bubeck | Bandits with heavy tail[END_REF], the authors study the problem of bandit with heavy tails, prove a concentration inequality in n -δ for some δ > 0 and use this result to build several estimators with faster convergence. Among them, they consider the Median-of-Means (MoM) introduced by [START_REF] Alon | The space complexity of approximating the frequency moments[END_REF]. Building on the results of previous section, we consider a natural variant of MoM, that we call Quantile of Maxima (QoMax). The principle of QoMax is simple: the learner chooses a quantile q, and has access to N = b × n data Y = (Y m,i ) m≤n,i≤b . It then allocates the data in b batches of size n and: (1) finds the maximum of each batch, (2) computes the empirical quantile of order q of the b maxima. We summarize QoMax in Algorithm 4.1.

1 Input: quantile q, b batches of size n, table of observations (Y m,i ) m≤n,i≤b 2 for i = 1, . . . , b do 3 Compute (Y + n ) (i) = max{Y 1,i , . . . , Y n,i } 4 end 5 Return: quantile of order q of {(Y + n ) (1) , . . . , (Y + n ) (b) } Algorithm 4.1: Quantile of Maxima (QoMax)

Sub-sampling for Extreme Bandits

For a finite set of size b, the quantile q is simply the observation of rank ⌈bq⌉ in the list of sorted data (in increasing order). In the sequel we denote by Xq n,b and Ȳ q n,b the QoMax of order q computed from two datasets (X m,i ) m≤n,i≤b and (Y m,i ) m≤n,i≤b .

We are now ready to state the crucial property of QoMax estimators that will be used in the analyses of our algorithms for Extreme Bandits.

Theorem 4.7 (Comparison of QoMax). Let ν X and ν Y be two distributions satisfying ν X ≻ ν Y and q ∈ (0, 1). Then, if b is large enough there exists a sequence x n , a constant c > 0, and an integer n ν 1 ,ν 2 ,q such that for n

≥ n ν 1 ,ν 2 ,q , max P( Xq n,b ≤ x n ), P( Ȳ q n,b ≥ x n ) ≤ exp(-cb) .
If the tails are furthermore either polynomial or exponential with a positive tail gap, then the result holds for any c > 0 and n larger than some n c,ν 1 ,ν 2 ,q .

Proof. We let kl(x, y) = x log(x/y) + (1 -x) log((1 -x)/(1 -y)) denote the binary relative entropy. Just like for the analysis of Median-of-Means, the starting point is to relate deviations inequalities for a QoMax to deviation inequalities for binomial distributions. Letting

(X + n ) (i) (resp. (Y + n ) (i)
) denote the maximum over the i-th batch of observations from ν X (resp. ν Y ),

P( Xq n,b ≤ x) ≤ P b i=1 1 (X + n ) (i) ≤ x ≥ bq ≤ exp(-b × kl(q, P(X + n ≤ x))) .
The last step applies the Chernoff inequality to a binomial distribution with parameters b and p = P(X + n ≤ x), and holds whenever P(X + n ≤ x) ≤ q. Similarly, if P(Y + n ≥ x) ≤ 1 -q -1/b, we have

P( Ȳ q n,b ≥ x) ≤ P b i=1 1 (Y + n ) (i) ≥ x ≥ b -bq -1 ≤ exp(-bkl(1 -q -1/b, P(Y + n ≥ x)))
For exponential and polynomial tails, thanks to Lemma 4.5 there exists a sequence (x n ) such that both P(X + n ≤ x n ) and P(Y + n ≥ x n ) satisfy the desired property, and the result follows easily. With the notation of Lemma 4.6, Theorem 4.7 then holds for c = min (kl(q, q -ε), kl(1 -q -ε/2, 1 -q -ε)) , provided that the batch size is larger than 2/ε.

QoMax-ETC

It follows from Theorem 4.7 that P( Xq n,b ≤ Ȳ q n,b ) ≤ 2 exp(-cb) for n large enough. Strikingly, this result tells us that, under the simple assumption that one tail dominates, the comparison of QoMax computed with the same parameters will not be in favor of the dominating arm with a probability that decreases exponentially with the batch size. Remark 4.8. In general QoMax is not an estimate of the expectation of the maximum. We will use it only for the purpose of comparing two tails, in order to find the heaviest. Remark 4.9 (Choice of quantile level q). Note that Theorem 4.7 holds for any value of q ∈ (0, 1), but the impact of q is materialized in the (problem-dependent) sample size n ν 1 ,ν 2 ,q needed for the inequality to hold. For the practitioner, we think that in most cases choosing q = 1/2 is appropriate. Still, in Section 6.6 we exhibit a difficult setting where a choice of q close to 1 is helpful.

QoMax-ETC

In this section, we propose QoMax-ETC, a simple Explore-Then-Commit algorithm using QoMax estimators. The algorithm is reported in Algorithm 4.2 and works as follows. First, the learner selects a quantile q, and given the time horizon T picks a batch size b T and a sample size n T . Then, the exploration phase starts where every arm is pulled N T = b T × n T times allocated in b T batches of size n T . At the end of this step, the learner computes a q-QoMax estimator from the history of each arm using the different batches. Next comes the exploitation phase where the algorithm pulls the arm I T with the largest QoMax until time T . We remark that an ETC algorithm has already been proposed by [START_REF] Achab | Max k-armed bandit: On the extremehunter algorithm and beyond[END_REF] for extreme bandits. Their algorithm differs from ours by the choice of the arm I T drawn in Sub-sampling for Extreme Bandits the exploitation phase: they build an upper confidence bound on the maximum under the assumption that the distributions are second-order Pareto (Definition 1.12) and select I T as the arm with largest upper confidence bound. In contrast, QoMax-ETC does not assume anything about the arms distributions. In our case, Theorem 4.7 is the main motivation for building an ETC strategy: with a large enough batch size b T and sample size n T strong concentration of QoMax estimates can be obtained. We now analyze QoMax-ETC under a bandit model

ν = (ν 1 , . . . , ν K ) such that ν 1 ≻ ν k for all k ̸ = 1.
R π T ≤ E max t≤T Y 1,t -E max t≤T -KN T Y 1,t Exploration cost + P(I T ̸ = 1)E max t≤T Y 1,t
Cost of picking a wrong arm .

Proof. We recall that N T = b T × n T is the number of pulls of each arm during the exploration phase of the ETC algorithm (see Algorithm 4.2) and that X k,t corresponds to the observation of arm k at time t (if any). We also denote by Y k,n the n-th observation collected from arm k.

The ETC simplifies a lot the study of the extremal regret, as we can separate the explore and commit phase in the analysis. First, an exact decomposition of the expected value of the policy is

E max t≤T X It,t = E max max k max t≤KN T X k,t , max t=[KN T +1,T ] X I T ,t .
We obtain the lower bound by ignoring the exploration phase and using that the rewards collected during the exploitation phase are conditionally independent of the outcome of the exploration phase

E max t≤T X It,t ≥ E max t=[KN T +1,T ] X It,t = E max t=[KN T +1,T ] X I T ,t = E max t=[KN T +1,T ] X I T ,t K k=1 1(I T = k) = K k=1 E max t=[KN T +1,T ] X I T ,t 1(I T = k) = K k=1 P(I T = k)E max t=[KN T +1,T ] Y k,t ≥ P(I T = 1)E max t=[1,T -KN T ] Y 1,t = (1 -P(I T ̸ = 1))E max t=[1,T -KN T ] Y 1,t ≥ E max t≤T -KN T Y 1,t -P(I T ̸ = 1)E max t≤T Y 1,t .

QoMax-ETC

We also used that if the distributions are supported on R the expectation of their maximum is positive for T large enough. This concludes the proof.

This proposition shows that the regret of the ETC algorithm can be properly controlled by two factors:

1. the probability of picking a wrong arm for the exploitation phase.

the gap between the growth rate of the maximum over T or T -KN T observations of

the dominant arm, that we call "exploration cost" as it is fully determined by the length of the exploration phase and the arms' distributions.

In the rest of the chapter we will assume that the distribution of the dominant arm satisfies the following assumption.

N T = o(T γ ) then E Y + T -E Y + T -N T -----→ T →+∞ 0 .
This condition is satisfied for nearly all distributions encountered in practice (e.g polynomial, exponential or gaussian tails). The following results support this claim by providing a generic way to upper bound the exploration cost and an application to semi-parametric tails.

Proposition 4.12 (Universal upper bound on the exploration cost). For any distribution of survival function G, for any constant B > 0 it holds that

E[Y + T ] -E[Y + T -N T ] ≤ N T B T + ∞ B G(x)dx Proof.
We write

E[Y + T ] -E[Y + T -N T ] = E Y + T -N T +1:T 1 Y + T ̸ = Y + T -N T +1:T ≤ E Y + T -N T +1:T 1 Y + T ̸ = Y + T -N T +1:T 1 Y + T -N T +1:T ≤ B + E Y + T -N T +1:T 1 Y + T -N T +1:T > B ≤ BP Y + T ̸ = Y + T -N T +1:T + ∞ B P Y + T -N T +1:T > x dx
Sub-sampling for Extreme Bandits

= B N T T + ∞ B P Y + N T > x dx ≤ B N T T + N T ∞ B P(Y 1 > x)dx ≤ N T B T + ∞ B G(x)dx .
where we have used the fact that that maximum has the same probability to be attained in each element and the union bound

P Y + N T > x ≤ N T i=1 P(Y i > x).
To prove that Assumption 4.11 is satisfied for exponential and polynomial tails, we then exhibit a value of B such that the resulting upper bound in Proposition 4.12 tends to 0. Lemma 4.13. Assumption 4.11 is satisfied for exponential and polynomial tails.

Proof. If the tail is exponential there exists λ > 0 and for any choice of B there exists a constant

C B such that G(x) ≤ C B e -λx if x ≥ B, so E[Y + T ] -E[Y + T -N T ] ≤ N T B T + C B ∞ B e -λx dx = N T B T + C B λ e -λB .
If there exists γ ∈ (0, 1) such that N T = o(T γ ), choosing B = log(T ) λ yields the result.

If the tail is polynomial, there exists similarly a constant λ and for any B a constant

C B such that G(x) ≤ C B x -λ for x ≥ B, so E[Y + T ] -E[Y + T -N T ] ≤ N T B T + C B ∞ B 1 x λ dx = N T B T + C B λ B 1-λ Choosing B = T 1/λ yields E[Y + T ] -E[Y + T -N T ] ≤ 1 + C B λ N T T 1-1 λ If for all γ ∈ (0, 1), N T = o(T γ ) then in particular N T = o(T 1-1 λ ) and lim T →∞ E[Y + T ] - E[Y + T -N T ] = 0.

QoMax-ETC

Remark 4.14. In both cases we chose B = G -1 (1/T ), which should work for other cases as well.

We now state our main theoretical claim for QoMax-ETC. Proof. From Theorem 4.7, there exists constants c k for k ≥ 2 such that for T large enough (such that n T becomes larger than n ν 1 ,ν k ,q ), it holds that Even if Theorem 4.15 is stated in an asymptotic way, we emphasize that its proof provides a finite-time upper bound on the probability of picking a wrong arm, P(I T ̸ = 1), that is valid provided that T is larger than some (problem-dependent) constant. In particular, T needs to be large enough so that n T ≥ max k̸ =1 n ν 1 ,ν k ,q where n ν 1 ,ν k ,q is the number of samples needed in Theorem 4.7 for the concentration of QoMax. This number is not always large. For example if we have two Pareto distributions with parameters λ 1 = 1.5 and λ 2 = 2, n T = 3 is enough. Using our regret decomposition, this result would lead to a finite-time upper bound on the extremal regret for distributions for which a finite-time bound on the exploration cost is available.

P(I T ̸ = 1) ≤ K k=2 P( Xq k,n T ,b T > Xq 1,n T ,b T ) ≤ K k=2 e -c k b T It follows that P(I T ̸ = 1) = o(T -1 ) if b T / log(T ) → ∞
To satisfy the theoretical requirements while obtaining good empirical performance, we recommend using b T = (log(T )) 2 and n T = log(T ) when running the algorithm. All the experiments reported in Section 4.5 use these values. QoMax-ETC is computationally appealing and has strong asymptotic guarantees. However in practice we found that its performance can vary significantly depending on the choices of b T and n T , which should in particular use a reasonable guess for the horizon T . For this reason, in the next section we propose QoMax-SDA, which is still based on QoMax comparisons but is anytime (i.e. independent on T ) and requires less parameter tuning.

Sub-sampling for Extreme Bandits

QoMax-SDA

In this section we present QoMax-SDA, an algorithm using a sub-sampling mechanism based on LB-SDA, that we studied extensively in Chapters 2 and 3. We detail the key principles of the algorithm and propose a theoretical analysis.

Algorithm and Implementation

From a high level QoMax-SDA follows the structure of the Sub-sampling Dueling Algorithms (SDA) introduced in Chapter 2. The algorithm operates in successive rounds composed of ( 1) the selection of a leader, (2) the different duels between the leader and the challengers and (3) a data collection phase. We develop each of those steps in the sequel.

Data and leader selection

At the beginning of a round r, the learner has access to the history of the different arms denoted by Y r k . For the needs of the QoMax, the collected rewards for arm

k are gathered within b k (r) batches of equal size n k (r) such that |Y r k | = b k (r)n k (r). n k (r)
is called the number of queries and corresponds to the number of times the arm k has been selected by the learner at the end of round r. The leader at round r, denoted by ℓ(r), is the arm that has been queried the most up to round r, that is ℓ(r) ∈ argmax k≤K n k (r). The K -1 remaining arms are called challengers. In case of equality, ties are broken first in favor of the arm with the largest QoMax, then at random.

Duels

Once the leader is selected, it plays a duel against each challenger. As in previous chapters we denote by A r+1 the set of arms that will be pulled at the end of round r. An arm k is added to A r+1 in two cases: (1) if it wins its duel or (2) if its number of queries is too small: n k (r) ≤ f (r) for a fixed function f (r) representing the forced exploration. As for standard SDA the leader is pulled only if no challenger is added to A r+1 . We now detail the duel procedure that is reported in Algorithm 4. 3. We assume that an infinite stream of rewards is available for each arm, in the form of an array with an infinite number of rows and columns, so that we denote the rewards of arm k by (Y k,n,b ) n∈N,b∈N , where Y k,n,b corresponds to the n-th sample of b-th batch from arm k. We further assume that the number of batches available for an arm k depends only on its number of queries n k (r) so that b k (r) = ⌈B(n k (r))⌉ for some function B. Following the principle of SDA, the duel is a comparison of the QoMax of the challenger using its entire history and the QoMax of the leader on a sub-sample of its history.

Our sub-sampling mechanism is inspired by LB-SDA, but has to consider two dimensions: we keep the rewards collected from n k (r) the last queries of the b k (r) first batches from ℓ(r). This way the QoMax from the leader and the challenger are computed using the same amount for m ∈ {k, ℓ} 4.1) ; ▷ QoMax of the challenger of data, but the diversity in the sub-samples (see Chapter 2 for intuitions on this topic) comes from the "query" dimension. We explain why in next paragraph.

2 Compute I k = QoMax(q, b k , n k , Y k ) (Alg.
3 Collect Y ℓ = (Y ℓ,i,j ) i∈[n ℓ -n k +1,n ℓ ],j∈[1,b k ] ; ▷ Leader's sub-sample 4 Compute I ℓ = QoMax(q, b k , n k , Y ℓ ) (Alg.

Data Collection

We now detail the data collection procedure that is used by QoMax-SDA and illustrated on Figure 4. 1. If we query an arm k at round r, we will ( 1) add 1 observation to each existing batch, and ( 2) collect enough data to build a new batch. We formalize this for one arm in Algorithm 6. Implementation of QoMax-SDA The combination of the leader selection, the duel step, and the data collection gives QoMax-SDA, reported in Algorithm 4. 5. While we wrote the algorithm with this function for simplicity, we actually do not recommend to use Algorithm 6 for the data collection step. Indeed, our algorithm can enjoy a significant reduction of storage with two different tricks:

1. The maxima can be stored efficiently : when a new value x is added inside a given batch, all stored values smaller than x are deleted as the algorithm will never need them again.

However we need to store the round at which the observation was collected in case we have to sub-sample this arm. where (M T ) T ∈N is a fixed sequence. Then, for T ≥ KM T , for any constant x T ∈ R, it holds that,

R π T ≤ E max t≤T Y 1,t -E max t≤T -KM T Y 1,t Exploration cost + x T P(ξ T ) + E max t≤T Y 1,t 1 max t≤T Y 1,t ≥ x T Cost incurred by ξ T .
Proof. Performing the exact same steps as for the proof of Proposition 4.10 one can first obtain

E max t≤T X It,t ≥ E max t≤T -KM T Y 1,t -E max t≤T Y 1,t 1(ξ T ) .
However, contrarily to the ETC strategy for SDA the variable 1(ξ T ) and the rewards are not conditionally independent. Additional steps are required to analyze the two quantities separately. Using the notation Y + T = max t≤T Y 1,t for simplicity, and then considering a constant x T ∈ R we can write

E max t≤T Y 1,t 1(ξ T ) = E[Y + T 1(ξ T )] ≤ E[Y + T 1(ξ T )1(Y + T ≤ x T )] + E[Y + T 1(ξ T )1(Y + T ≥ x T )] ≤ x T P(ξ T ) + E[Y + T 1(Y + T ≥ x T )] .

Sub-sampling for Extreme Bandits

This concludes the proof.

Remark 4.17 (Comparison with Proposition 4.10). The expression we obtained can be compared with the result for ETC strategies. The first part (exploration cost) is the same. However, the second term is more complicated as we could simply write P(ξ T )E[max t≤T X 1,t ] for the ETC strategy. If we did this we would now obtain P(ξ

T ) × E Y + T |ξ c T ,
where the second term is unfortunately not equal to E X + T . We cannot proceed further with this upper bound as the conditional expectation may be very intricate to compute. However, we conjecture that the upper bound

E Y + T |ξ c T ≤ E Y + T should hold. Indeed, ξ c
T and the maximum should be positively correlated, as this event corresponds to arm 1 being pulled a lot, and hence performing quite well. This seems however difficult to prove.

Another interesting observation is that, even if the "cost incurred by ξ T " features two terms, interestingly only the first term depends on the algorithm. In fact, just as with Proposition 4.10 our analysis again depends mostly on upper bounding the probability of a "bad" event. We propose the following result for QoMax-SDA.

= (ν 1 , . . . , ν K ) satisfying ν 1 ≻ ν k , ∀k ̸ = 1.
For any q ∈ (0, 1), any M T and any γ > 0, under QoMax-SDA with parameters B(n) = n γ and f (r) = (log r) 1+γ and Markov inequality we obtain

1 γ , P(ξ T ) = O (log T ) 1 γ M -1 1+γ T . Moreover, for all k ̸ = 1, E[n k (T )] = O((log T ) 1/γ ).

Sketch of proof. We first use that

P(ξ T ) ≤ K k=2 P(N k (T ) ≥ M T ). Using that N k (T ) = b k (T ) × n k (T ) = n k (T )
P(ξ T ) ≤ M -1 1+γ T K k=2 E[n k (T )] .
It remains to study the expected number of queries of sub-optimal arms k ≥ 2. This can be done following the outline of the proof of Theorem 2.12 in Chapter 2 for SDA in the standard setting while using the deviation inequalities from Theorem 4.7 instead of the ones from Assumption 2. 4. The second term in the "cost incurred by ξ T " only depends on the distribution of the optimal arm and can be further upper bounded assuming exponential and polynomial tails, leading to the following result.

Sub-sampling for Extreme Bandits

X ⌈q + T ⌉ , as an estimator of their expected maximum. This allows to compute what we call Proxy Empirical Regret (PER), that we define below. Definition 4.20 (Proxy Empirical Regret (PER)). Assume that we perform N runs of the algorithm up to horizon T . Further assume that E[Y + 1,T ] is known, and that the distribution of Y + 1,T is also known and is denoted by ν + T . Let q + T be the quantile of

ν + satisfying VaR q + T (ν + T ) = E[Y + 1,T ].

Then, the Proxy Empirical Regret is computed by replacing the empirical estimate of E[X +

T ] by the empirical estimate of q + T in the regret definition, namely

R proxy T,π = E[Y + 1,T ] -X + ⌈q + T ⌉ E[Y + 1,T ]
,

where X + 1 , . . . , X + N are the maxima obtained in each of the N runs. The normalization aims at facilitating the check of a weakly vanishing regret.

More precisely, by definition q

+ T satisfies q + T = P(X + T ≤ E[X + T ]) ≈ exp(-T G(E[X + T ])).
In the experiments we plug the equivalents of E[X + T ] in each setting: for Pareto distribution we obtain q + T ≈ exp -1 Γ(1-1/λ) λ , while for exponential distributions we obtain q + T ≈ e -1 . We are able to compute the PER to approximate (I) for experiments 1-6. When (I) is not available we recommend looking at (IV) with the same quantiles as for (III).

Experiments

We describe the settings of a selection of experiments from [START_REF] Baudry | Efficient algorithms for extreme bandits[END_REF] that we present in this section. We will then refer to them by their number (e.g exp.1), keeping the same numbering as in the paper.

• Experiment 1 (exp.1 in [START_REF] Bhatt | Extreme bandits using robust statistics[END_REF]): K = 5 Pareto distributions with tail parameters 1, 2.3, 1.3, 1.1, 1.9].

λ k ∈ [2.
• Experiment 3 (exp.3 in (Bhatt et al., 2021)) K = 10 Exponential arms with a survival function .1, 2.4, 1.9, 1.3, 1.1, 2.9, 1.5, 2.2, 2.6, 1.4].

G k (x) = e -λ k x with parameters λ k = [2
• Experiment 6 (exp.2 in [START_REF] Carpentier | Extreme bandits[END_REF])) K = 3 arms, including 2 Pareto distributions with λ k ∈ [ 1.5, 3], and arm 3 is a mixture Dirac/Pareto: pulls 0 with 80% probability, reward from a Pareto distribution with λ = 1.1 with 20% probability. Hence, the last arm dominates asymptotically.

• Experiment 7: we consider K = 5 log-normal arms with parameters µ k ∈ [1, 1.5, 2, 3, 3.5] and σ k ∈ [4, 3, 2, 1, 0.5]. When T is large enough the parameter σ determines which arm dominates (arm 1 in our case). We recall that if X follows a log-normal distribution with parameters (µ, σ) then log(X) ∼ N (µ, σ)

Practical performance

The code to reproduce the experiments is available on Github.

Objective of each experiment Before reporting the results, we explain why each experiment is interesting for the empirical evaluation of Extreme Bandits algorithms. Experiment 1 is quite difficult because the tail gap between arm 3 and arm 4 is relatively small. All algorithms are supposed to have guarantees in this setting so their comparison is fair. Experiments 3 allows to test the different algorithms with exponential tails, showing the performance of the algorithms when the tails are not polynomial. Experiment 6 will be interesting for discussing the limits of parameter-free approaches, as the dominant tail provides low rewards with relatively high probability. Finally, experiment 7 allows to try the algorithms on heavy-tail distributions that do not have a polynomial tail.

Parameters

We recall the parameters used for the different experiments. For each experiment, we run N = 10 4 independent trajectories for 10 time horizons T ∈ [10 3 , 2.5 × 10 3 , 5 × 10 3 , 7.5 ×

10 3 , 9×10 3 , 10 4 , 1.5×10 4 , 2×10 4 , 3×10 4 , 5×10 4 ].
This methodology allows for a fair comparison between ETC and more adaptive strategies, and can stabilize the results as an extreme trajectory can introduce bias for only one time horizon. The parameters we used for each algorithm are the following:

• ThresholdAscent: s = 100, δ = 0.1, as suggested in (Streeter and Smith, 2006b).

• ExtremeETC/ExtremeHunter: b = 1, as in [START_REF] Carpentier | Extreme bandits[END_REF]. As the authors, we use δ = 0.1 for the experiments instead of the theoretical value that is too large for the time horizons considered, and D = E = 10 -3 for the UCB. Other theoreticallymotivated parameters are r = T -1/(2b+1) (fraction of samples used for the tail estimation),

N = (log T ) 2b+1 b
(length of the initial exploration phase). δ = exp(-log 2 (T ))/(2T K) in the paper but set to 0.1 here.

• MaxMedian: The exploration probability is set to ε t = 1/(1 + t) as suggested in [START_REF] Bhatt | Extreme bandits using robust statistics[END_REF].

• QoMax-ETC: We test q = 1/2 and q = 0.9, b T = (log T ) 2 and n T = log T to match both the theoretical requirements of Section 4.3 and the length of the exploration phase of ExtremeETC for a fair comparison.

• QoMax-SDA: f (r) = (log r)

1 γ and B(n) = n γ for γ = 2/3, which works well across all the experiments. The quantile is either equal to q = 1/2 or q = 0.9.

Results

For each experiment, we report the results according to the criteria (I)-(IV) that are defined above. The criteria (I)-(II) are reported side by side for each experiment in Figures 4.2-4.8, except for Exp.7 for which (I) cannot be computed.

associated with (III)

Sub-sampling for Extreme Bandits report the result for the statistics on the number of pulls of the best arm on all trajectories at T = 5 × 10 4 . Finally, Tables 4.4-4.10 related to (IV) report the results for the statistics on the empirical distribution of the maxima on all trajectories at T = 5 × 10 4 .

We summarize our key observations on the results with the following points:

• Non-robustness of reporting the average maximum collected. Several examples can serve to illustrate this point. For experiment 1 (Table 4.4) if we look at the average maximum only, we would conclude that QoMax-SDA with q = 1/2 is by far the best algorithm with an average of 1.8 × 10 5 (1.1 × 10 5 for the second). However, we see that the quantiles of the maxima distributions are almost identical to those of other QoMax algorithms. Hence, even if 99% of their distribution matches, QoMax-SDA with q = 1/2 has a nearly 70% better average caused by less than 1% of the trajectories. The same thing seems to happen on different problems: the 10 4 and 8.5 × 10 3 of 1/2-QoMax-ETC and ExtremeETC are clearly over-estimated means in experiment 2 considering that they both have the same quantiles as 1/2-QoMax-SDA (even a bit worse), which has an average of 7.5 × 10 3 , and MaxMedian with 7.9 × 10 3 . Without surprise, this phenomenon is more present when the tails are heavier. Hence looking at the average maxima is meaningful only with the statistics from Experiments 3 and 4 with lighter tails.

• QoMax Performance. QoMax algorithms clearly outperform their competitors in Experiments 1, 3, 7 according to all criteria. As those experiments include polynomial, exponential and log-normal tails with different number of arms, this shows the generality and efficiency of the QoMax approach. QoMax-SDA seems to work better than QoMax-ETC, in particular it is competitive even for small time horizons (T < 5 × 10 3 ) in most experiments. However, we see that QoMax-ETC almost matches the performance of QoMax-SDA for T = 5 × 10 4 . For a practitioner who would be interested in larger time horizons QoMax-ETC seems to be a perfectly suitable choice.

• On the contrary, ExtremeHunter performs significantly better than ExtremeETC for larger horizons: the probability of mistake of the latter is still quite large, and the ability of ExtremeHunter to recover from a mistake is valuable, but we recall that the time complexity of ExtremeHunter is detrimental for the practitioner. Results from Experiment 3 show that the two algorithms are not able to handle exponential tails.

• ThresholdAscent is never the best algorithm but has the advantage of being consistently better than the uniform strategy (according to (II)), as it always pulls the best arm at a frequency larger that 1/K. It is the most stable baseline in terms of (III) (it always has the narrowest range for the statistics we consider), but this is detrimental to its capacity to collect large values.

Practical performance

• We tested MaxMedian on larger time horizons than in the original paper, which explains the difference in some results. Indeed, we observe that in Experiments 1, 3, 5, MaxMedian is quite competitive for shorter time horizons (T ≤ 10 4 ), but almost stops improving at this step. This suggests that the algorithm does not explore enough, which is confirmed by a closer look at (III): the number of pulls of the best arm are either very close to 0% or to 100% in most of the cases, which is a behavior specific to this algorithm and that we would like to avoid in practice. This behavior also has an impact on the statistics on the maxima distributions (IV). The exploration function may be responsible for this, and results may be better with a larger forced exploration.

• Experiment 6 shows that in some examples parametric algorithms can perform much better than non-parametric approaches. Indeed, the distribution of arm 3 enters in the second-order Pareto family, and the parameter b = 1 makes ExtremeHunter calibrate its parameters with the ≈ 5% best samples of each arm. This is enough for the algorithm to "detect" the Pareto tail of the mixture and sample it most often. Most of the other algorithms fail, including QoMax. However, two important remarks on QoMax can be made based on this experiment: the 0.9-QoMax-SDA performs much better than the others, showing that when the tails are harder to detect choosing a larger quantile can be valuable. Furthermore, we tested another experiment imposing at least 100 samples in each batch. This time, 0.9-QoMax-SDA was able to pull the best arm 60% of the time.

Hence, the practitioner has the ability to increase the exploration and the quantile q if very difficult tails are expected (typically where the value x of Definition 4.3 is large), which depends on the characteristics of the problem at hand.

Conclusion of the experiments

Overall, QoMax-based algorithms seem to be solid choices for the practitioner, as demonstrated in a variety of examples. Their strong theoretical guarantees and implementation tricks reducing the time and space complexities make them an efficient solution for the Extreme Bandits problem. They work well on most examples that we tried with the same parameters (avoiding painful tuning), including settings with different kind of tails (polynomial, exponential, log-normal) with different number of arms, and both easy and hard instances. We explained however with experiment 6 the limits of a distribution-free approach if we consider a hard problem. It also showed that in this case augmenting the quantile q (and/or the forced exploration function f for QoMax-SDA) used in QoMax algorithms can be beneficial. Furthermore, we can recommend to use QoMax-ETC when the time horizon is expected to be very large (larger than 5 × 10 4 for instance) and QoMax-SDA for smaller time horizons, as it seems to learn faster on all examples but it is more computationally demanding. Proof. We instantiate the decomposition of Proposition 4.16 using the value of P(ξ T ) obtained in Lemma 4.18. Plugging all of these values and using Proposition 4.12, we write for π being any instance of QoMax-SDA with parameter γ,

R π T ≤ E max t≤T Y 1,t -E max t≤T -KM T Y 1,t Exploration cost + x T P(ξ T ) + E max t≤T Y 1,t 1 max t≤T Y 1,t ≥ x T Cost incurred by ξ T ≤ KM T B T T + +∞ B T G 1 (x)dx A 1 + x T C(log T ) 1 γ M 1 1+γ T A 2 + T +∞ x T G 1 (x)dx A 3
, for any values of x T , B T , M T , that we now specify for each of the two families considered.

Exponential tails

We recall that if G 1 (x) = O(exp(-λx)), then for any y ∈ R we have

+∞ y G 1 (x)dx = O (exp(-λy)) . First, if we choose B T = 1 λ log(T ) then A 1 vanishes for any choice of M T = T α with 0 < α < 1. Similarly, choosing x T = 2 λ log T ensures that A 3 = O(1/T ). Then, A 2 is O (log T ) 1+ 1 γ M 1 1+γ T
, which is vanishing for any choice of M T = T α , α ∈ (0, 1). We conclude that for exponential tails, lim T →∞ R π T = 0.

Polynomial tails Consider again M T = T α , for some α ∈ (0, 1). This time,

+∞ y G 1 (x)dx = O 1 y λ-1 .

Sub-sampling for Extreme Bandits

Plugging into A 3 , we get a term of order O(T ×x 1-λ T ). Let's take x T = T β for some β ∈ (0, 1), we then have (1-λ) ) . Now consider A 2 , omitting the polylog terms we obtain

A 3 = O(T 1+β
A 2 = O(T β-α 1+γ ) .
Consider finally A 1 . Choosing B T = T 1 λ (as in Appendix ??) we obtain the tightest upper bound on the exploration cost:

A 1 = O M T T 1-1 λ = O(T α-1+ 1 λ ) .
To get the smallest order with this proof technique we want to equalize all these three exponents, which gives

α -1 + 1 λ = β - α 1 + γ = 1 + β(1 -λ) .
For simplicity we write β = 1 λ + η and try to find η instead. Re-writing the the three equalities yields

α + 1 λ -1 = 1 λ + η - α 1 + γ = 1 λ -(λ -1)η .
This can be further simplified in

α -1 = η - α 1 + γ = -(λ -1)η .
This gives in particular a system of two equations with two unknowns η and α. By substituing α we get

η - 1 -(λ -1)η 1 + γ = -(λ -1)η ⇔ η [1 + γ + λ -1 + (λ -1)(1 + γ)] = 1 , which gives η = 1 λ(2+γ)-1 and α = λ(1+γ) λ(2+γ)-1 .

Appendix B: Implementation Tricks for QoMax-SDA

Plugging in these values, we obtain that A 1 , A 2 and

A 3 are all in O T 1 λ -λ-1 λ(2+γ)-1 = o(T 1/λ ).
Plugging the rate of growth of the maximum for polynomial tails we get that

R π T = o T →∞ E max t≤T Y 1,t

Appendix B: Implementation Tricks for QoMax-SDA

In this section we provide both empirical and theoretical evidences on the memory gains obtained with the implementation trick we introduced for the storage of maxima in QoMax-SDA. We first recall this procedure in Algorithm 4.6

1 Input: List of indices I = {i 1 , . . . , i L }, sorted list X = {X 1 , . . . , X L }, X 1 > X 2 > • • • > X L , new index i, new data X 2 Find the largest j ∈ {1, . . . , L} satisfying X j > X ; ▷ Binary search 3 Set X = {X 1 , X 2 , .
a standard normal distribution. We report in Figure 4.12 the average number of data stored by the algorithm for each sample sizes, along with the empirical 5% and 95% quantiles on the 1000 simulations and the curve y = log(N ) for comparison. We observe that: (1) The results do not depend on the distribution. ( 2) All 4 curves are very close to exactly N n=1 1 n , which is as small as ≈ 10 for a sample size of 5 × 10 4 . (3) 90% of the simulations have no more than 17

Appendix B: Implementation Tricks for QoMax-SDA

Then, we remark that if I N = j, all observations of higher order X j+1 , . . . , X N are removed from the history. Hence, it only remains to count the average amount of data considering only X 1 , . . . , X j-1 , which is equal to E[C j-1 ] and add 1 for the last observation. Using that

E[C 1 ] = 1, E[C N ] = 1 N N j=1 E[C N |I N = j] = 1 N N j=1 (1 + E[C j-1 ]) ⇒ (N + 1)E[C N +1 ] -N E[C N ] = N +1 j=1 (1 + E[C j ]) - N j=1 (1 + E[C j ]) = 1 + E[C N ] ⇒ (N + 1)(E[C N +1 ] -E[C N ]) = 1 ⇒ E[C N +1 ] = E[C N ] + 1 N + 1 ⇒ E[C N ] = N n=1 1 n .

Introduction

In Section 1.2 of Chapter 1 we presented the literature on bandit algorithms with alternative performance metric, with a focus on risk-aware bandits, and more specifically on bandit algorithms evaluating the arms' distributions through their Conditional Value at Risk.

The Conditional Value at Risk (CVaR) at level α ∈ [0, 1] (see [START_REF] Mandelbrot | The variation of certain speculative prices[END_REF][START_REF] Artzner | Coherent measures of risk[END_REF]) is easily interpretable as the expected reward in the worst α-fraction of the outcomes, and hence captures different preferences, from being neutral to the shape of the distribution (α = 1, mean criterion) to trying to maximize the reward in the worst-case scenarios (α close to 0, typically in finance or insurance). Several definitions of the CVaR exist in the literature, depending on whether the samples are considered as losses or as rewards. [START_REF] Brown | Large deviations bounds for estimating conditional value-at-risk[END_REF][START_REF] Thomas | Concentration inequalities for conditional value at risk[END_REF]Agrawal et al., 2021b) consider the loss version of CVaR. We here follow [START_REF] Galichet | Exploration vs exploitation vs safety: Risk-aware multi-armed bandits[END_REF] and [START_REF] Tamkin | Distributionally-aware exploration for cvar bandits[END_REF] who use the reward version, defined for arm k with distribution ν k as

CVaR α (ν k ) = sup x∈R x - 1 α E X∼ν k (x -X) + . ( 5.1)
This implies that the best arm is the one with the largest CVaR. To simplify the notation we write c α k = CVaR α (ν k ) in the sequel. Following e.g. [START_REF] Tamkin | Distributionally-aware exploration for cvar bandits[END_REF], for unknown arm distributions ν = (ν 1 , . . . , ν K ) we measure the CVaR regret at time T for some risk-level α of a sequential sampling strategy A = (A t ) t∈N as

R α ν (T ) = E ν T t=1 max k c α k -c α At = K k=1 ∆ α k E ν [N k (T )] , (5.2)
where

∆ α k = max k ′ c α k ′ -c α k
is the gap in CVaR between arm k and the best arm, and

N k (t) = t s=1 1(A s = k)
is the number of selections of arm k up to round t. This corresponds to the proxy regret of Definition 1.8 using the CVaR α as performance metric. However, for the problem we consider (CVaR bandits with bounded distributions) it will be easier to analyze than the regret of definition 1.6, furthermore [START_REF] Cassel | A general approach to multi-armed bandits under risk criteria[END_REF] proved that upper bounds on the proxy regret translates into an upper bound on the true regret. For this reason, we will actually call CVaR regret the proxy regret defined in Equation ( 5.2) for the rest of this chapter.

Some basic CVaR properties

Before continuing this chapter we explain some well-known properties of the CVaR in order to improve the intuitions of the un-familiar reader. First, the definition of the CVaR as the solution of an optimization problem was first introduced by [START_REF] Rockafellar | Optimization of conditional value-at-risk[END_REF], to formalize previous heuristic definitions of the CVaR as an average over a certain part of the distribution. The definition (5.1) is indeed appealing as it applies to any distribution for which E[(x -X) + ] is defined, including both discrete and continuous

Introduction

distributions.

To understand the CVaR it is particularly useful to look at its expression in these two particular cases. First, for any continuous distribution ν of CDF F ν it can be shown (see, e.g. [START_REF] Acerbi | On the coherence of expected shortfall[END_REF]) that

CVaR α (ν) = E X∼ν X|X ≤ F -1 ν (α) .
This expression provides a good intuition on what the CVaR represents, as the expectation of the distribution including only the worst scenarios, representing the fraction α of the total mass. A similar definition exists for real-valued distributions ν with discrete support X = (x 1 , x 2 , . . . ) (either finite or infinite). Assuming that the sequence (x i ) is increasing and letting .3) Indeed, the function to maximize in (5.1) is piece-wise linear, so the maximum is necessarily achieved in a point of discontinuity. In particular, we can easily prove that if n α is the first index satisfying nα i=1 p i ≥ α, then the supremum is achieved in n α and

p i = P X∼ν (X = x i ), one has CVaR α (ν) = sup xn∈X x n - 1 α n-1 i=1 p i (x n -x i ) . ( 5
CVaR α (ν) = x nα - 1 α nα-1 i=1 p i (x nα -x i ) = 1 α nα-1 i=1 p i x i + α - nα-1 i=1 p i x nα .
Hence in that case the CVaR can also be seen as an average when we consider the lower part of the distribution before reaching a total mass α.

From the general definition (5.1), one can also observe that for α = 1, CVaR α (ν) = E X∼ν (X). Moreover, the mapping α → CVaR α (ν) is continuous on (0, 1]. Thus, considering CVaR bandits allows to smoothly interpolate between standard bandits (that correspond to α = 1) and risk-averse problems. Finally, the CVaR is also conveniently Lipschitz w.r.t some well-chosen distances between distributions when they are bounded. This property is useful to prove the results presented in this chapter.

Contributions

Our objective is to find algorithms minimizing the CVaR regret (Eq. 5.2) considering either distributions with discrete, finite support, or with continuous and bounded support, as we believe this is a relevant problem, motivated by the case-study in agriculture introduced at the beginning of this thesis. More precisely, we target first-order asymptotic optimality for these (sometimes called "non-parametric") families and first derive in Theorem 5.3 a lower-bound on the CVaR regret, adapting that of [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF]; Burnetas and

Non-Parametric Thompson Sampling for CVaR Bandits

Notation We introduce the notation C α (X , p) for the CVaR of a discrete distribution of support X and probability p ∈ P |X | , where P n denotes the probability simplex of size n. For a multinomial arm k we denote its known support by X k = (x 1 k , . . . , x M k k ) for some M k ∈ N, and its true probability vector by p k . We also define N i k (t) as the number of times the algorithm has observed x i k for arm k before the time t. For general bounded distributions we denote by ν k the distribution of arm k and introduce X k,t the set of its observed rewards before time t, augmented with a known upper bound B k for the support of ν k . We further introduce D n as the uniform distribution on the simplex P n , corresponding to the Dirichlet distribution Dir((1, ..., 1)) (with n ones). We provide some properties of the Dirichlet distribution in Appendix 5.7.

M-CVTS

For multinomial distributions M-CVTS (Multinomial-CVaR-Thompson-Sampling), described in Algorithm 5.1, directly follows the Thompson Sampling principle introduced in Chapter 1. For each arm k, the probability p k is assumed to be drawn from D M k , the uniform prior on P M k . The posterior distribution at a time t is Dir(β k,t ), with

β k,t = (N i k (t)+1) i∈{1,...,M k } . At time t, M-CVTS draws a sample w k,t ∼ Dir(β k,t ) for each arm k and computes c α k,t = C α (X k , w k,t ). Then, it selects A t = argmax k c α k,t .
For α = 1, this algorithm coincides with the Multinomial Thompson Sampling algorithm of [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF]. for general bounded distributions. B-CVTS, stated as Algorithm 5.2, bears some similarity with a Thompson Sampling algorithm, although it does not explicitly use a prior distribution. The algorithm retains the idea of using a noisy version of ν k , obtained by a random re-weighting of the previous observations. Hence, at a time t the index used by the algorithm for an arm k is

c k,t = C α (X k,t , w k,t ), where w k,t ∼ D N k (t)
is drawn uniformly at random in the simplex P |X k,t | .

Asymptotic optimality of the CVTS algorithms

Definition 5.2. Let F be a family of distributions, α ∈ (0, 1], and KL(ν, ν ′ ) be the KL-divergence between ν ∈ F and ν ′ ∈ F. For any ν ∈ F and c ∈ R, we define

K α,F inf (ν, c) := inf ν ′ ∈F KL(ν, ν ′ ) : CVaR α (ν ′ ) ≥ c .
This definition generalizes the K F inf function considered in Chapter 1 for the expectation setting. We now state our result.

Proposition 5.3 (Regret Lower Bound in CVaR bandits). Let α ∈ (0, 1]. Let F = F 1 ו • •×F K be a set of bandit models ν = (ν 1 , . . . , ν K ) where each ν k belongs to the class of distribution F k . Let A be a strategy satisfying R α ν (A, T ) = o(T β
) for any β > 0 and ν ∈ F. Then for any ν ∈ D, for any sub-optimal arm k, under the strategy A it holds that lim

T →+∞ E ν [N k (T )] log T ≥ 1 K α,F k inf (ν k , c ⋆ )
,

where c ⋆ = max i∈[K] CVaR α (ν i ).
Using Equation ( 5.2), this result directly yields an asymptotic lower bound on the regret. The proof of Proposition 5.3 follows from a classical change-of-distribution argument, as that of any lower bound proof in the bandit literature. It follows from the proof of Theorem 1 in [START_REF] Garivier | Explore first, exploit next: The true shape of regret in bandit problems[END_REF] originally stated for α = 1, and the details can be found in Appendix D.1 of (Baudry et al., 2021a).

It is well known in the bandit literature that this lower bound can lead to the desired scaling of the regret in terms of the gaps. Indeed, for α = 1, Pinsker inequality provides that for a distribution ν ∈ F and µ

∈ R satisfying µ -E ν [X] = ∆ > 0, K 1,F inf (ν, µ) ≥ ∆ 2 2 .
This scaling in the gap is convenient to analyze algorithm that do not target asymptotic optimality, and we say that an algorithm achieving E[N k (T )] = O log(T )/∆ 2 k achieves order optimality (which is stronger than simply having a logarithmic regret).

We can now discuss how the lower bound in Proposition 5.3 yields a weaker regret bound expressed in terms of the CVaR gaps (by Pinsker inequality). Using Lemma A.2 of Tamkin et al. (2020), we can show that for any distributions ν F and ν G with respective CDFs F and G that are supported in [0, 1],

|CVaR α (F ) -CVaR α (G)| ≤ 1 α ||F -G|| ∞ .

Asymptotic optimality of the CVTS algorithms

The two parts of this theorem are proved in (Baudry et al., 2021a). In this manuscript we choose to only focus on the B-CVTS algorithm as the proofs of the two statements share many similarities. In the following we highlight the key steps of the analysis. First, using Equation ( 5.2) it is sufficient to upper bound E[N k (T )] for each sub-optimal arm k. As in previous chapters we assume that arm 1 is optimal to ease the notation. Then our analysis follows the general outline of that of [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF], but some steps require a careful adaptation to CVaR bandits. First, the proof leverages some properties of the function K α inf for the sets of distributions we consider. Secondly, it requires novel boundary crossing bounds for Dirichlet distributions that we detail in Section 5. 4.

Proof Sketch

The first step of the analysis consists in upper bounding the number of selections of a sub-optimal arm by a post-convergence term (Post-CV) and a pre-convergence term (Pre-CV). The first term controls the probability that a sub-optimal arm over-performs when its empirical distribution is "close" to the true distribution of the arm, while the second term considers the possibility that the CVaR of arm 1 could be under-estimated. To measure how close two distributions are we use the Levy distance, that is defined as follow for two distributions of respective cdf F and G,

D L (F, G) = inf {ε > 0 : ∀x ∈ [0, B], F (x -ε) -ε ≤ G(x) ≤ F (x + ε) + ε} .
The following result summarizes this decomposition in two terms.

E[N k (T )] ≤ n 0 (T ) + T t=1 E ν k ,π 1 (G k (t)) × P w∼D N k (t) (C α (Y k (t), w) ≥ c α 1 -ε 1 ) post-convergence term (Post-CV) + T n=1 E Yn∼ν n 1 P w∼Dn (C α (Y 1,n , w) ≤ c α 1 -ε 1 ) 1 -P w∼Dn (C α (Y 1,n , w) ≤ c α 1 -ε 1 )
pre-convergence term (Pre-CV)

+O (1) ,

where G k (t) = {N k (t -1) ≥ n 0 (T ), D L (F k (t), F k ) ≤ ε 2 }
is the event corresponding to arm k being sampled "enough" and "close" to its true distribution, Y 1,n is the set of n first data collected from arm 1 and Y k (t) is the set of data collected from arm k up to time t. The empirical distribution F k (t) includes the upper bound B added to the set of observations.

Non-Parametric Thompson Sampling for CVaR bandits

The proof of proposition 5.5 uses standard arguments in bandits, and is a direct adaptation of the proof of NPTS from [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF]. It decomposes the number of pulls of each sub-optimal arm between a constant n 0 (T ), a post-convergence, and a pre-convergence terms. Interestingly, the former depends only on arm k, and corresponds to this arm being sampled while having a sufficient number of samples and an empirical cdf that accurately reflect its true cdf. On the other hand, the later is the cost of under-estimating the best arm in the regret. Finally, the additional constant term corresponds to T t=1 P(G k (t) c , A t = k). [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF] provide a method to derive a constant upper bound, that can be refined by using DKW inequality instead [START_REF] Massart | The tight constant in the dvoretzky-kiefer-wolfowitz inequality[END_REF]. We omit the development of this part, that can be found in the paper.

Further upper bounding (Post-CV) and (Pre-CV) requires to provide upper and lower bound on boundary crossing probabilities for Dirichlet random variables, that we define as follows.

Definition 5.6 (Boundary Crossing Probabilities (BCP)). A BCP is a probability of the form

P w∼Dir(β) (C α (Y, w) ≥ c) ,
for some known support Y = (y 1 , . . . , y n ), parameter β ∈ R n + of the Dirichlet distribution, and some real value c that are defined in context.

Finally, proving the asymptotic optimality of B-CVTS consists in showing that both the pre-convergence and post-convergence terms can be further upper bounded by constants, while choosing

n 0 (T ) = 1 K α,C B k inf (ν k , c α ⋆ ) -ε log(T ) ,
for any constant ε > 0. In the next section we present the technical tools that allow to draw these conclusions.

Further upper bounds on Pre-CV and Post-CV

In the analysis of NPTS, [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF] provide several results to upper and lower bound the BCP for α = 1. However, replacing the linear expectation by the CVaR, that is non-linear, causes several technical challenges that makes the adaptation not direct. In this section we provide the technical results that allow to complete the proof of Theorem 5.4 from Proposition 5.5: the dual form of the K α,D inf function, and upper and lower bounds on the BCP.

Further upper bounds on Pre-CV and Post-CV

Technical tools

Dual of K α,D inf Our first result is the derivation of the dual formulation of the K α,D inf functional, providing an extension of the result of [START_REF] Honda | An Asymptotically Optimal Bandit Algorithm for Bounded Support Models[END_REF] when α = 1.

Lemma 5.7. If a discrete distribution F supported on Y satisfies E F (y-c)α (y-X) + < 1, then for any c > CVaR α (F ) it holds that K α inf (F, c) = inf y∈Y max λ∈ 0, 1 α(y-c) g(y, λ, X) , with g(y, λ, X) = E F log(1 -λ((y -c)α) -(y -X) + ) . If E F (y-c)α (y-X) + ≥ 1, then for any c > CVaR α (F ) K α inf (F, c) = inf y∈Y E F (y -X) + (y -c)α .
Proof. We let P M denote he simplex of dimension M . We rewrite the optimization problem, defined for any p ∈ P M , α ∈ (0, 1] and c ∈ [0, 1] as

K α,D inf (p, c) = inf q∈P M {KL(p, q) : C α (Y, q) ≥ c} .
First of all, we recall that C α (Y, q) = sup x∈D x -1 α E X∼q (x -X) + . We then introduce the set

P M y,α,c = q ∈ P M : y - 1 α E X∼q ((y -X) + ) ≥ c = q ∈ P M : E X∼q ((y -X) + ) ≤ (y -c)α .
Thanks to this definition we can rewrite the problem as

K α,D inf (p, c) = min y∈D inf q∈P M KL(p, q) : y - 1 α E X∼q ((y -X) + ) ≥ c ,
where we used that {q :

C α (Y, q) ≥ c} = ∪ y∈D q ∈ P M : y -1 α E X∼q (X -y) + ≥ c .

Non-Parametric Thompson Sampling for CVaR bandits

of a continuous function on a compact set. We consider the set

S 2 = {w ∈ P M +1 : w i ∈ [0, p * i ], ∀j ≤ M -1, w M ≥ p * M }. Let us remark that ∀p ∈ S 2 , C α (Y, p) ≥ C α (Y, p * ) ≥ c.
Indeed, if we transfer some of the mass from some items of the support to largest items we can only increase the CVaR. It then holds that

P w∼Dir(β) (C α (Y, w) ≥ c) ≥ P w∼Dir(β) (w ∈ S 2 ) = Γ(n) M i=1 Γ(β i ) x∈S 2 M i=1 y β i -1 i dx ≥ Γ(n) M i=1 Γ(β i ) (p * M ) β M -1 M -1 j=1 p * j y j =0 y β j -1 j dy j = Γ(n) M i=1 Γ(β i ) (p * M ) β M -1 M -1 j=1 (p * j ) β j β j = Γ(n) M i=1 Γ(β i ) β M p ⋆ M M j=1 (p * j ) β j β j ,
which proves the first inequality. We then exhibit the KL-divergence between two multinomial distributions using

P w∼Dir(β) (w ∈ S 2 ) ≥ Γ(n) M i=1 Γ(β i ) M j=1 (p * j ) β j β j = Γ(n) M i=1 Γ(β i ) M j=1 p * j β j β j × M j=1 β β j -1 j = Γ(n) M i=1 Γ(β i + 1) M j=1 p * j β j /n β j × M j=1 β j n β j = Γ(n) M i=1 Γ(β i + 1) M j=1 β j n β j exp -nKL β n , p ⋆ .
This corresponds to the second inequality in the lemma, as we can remark that the terms before the exponential correspond to the desired multinomial distributions, up to a factor 1/n. We finally provide a lower bound of this quantity using Stirling formula (similarly to the proof scheme of Riou and Honda (2020)),

√ 2πn n e n ≤ n! ≤ √ 2πn n e n (1 + C(n)) ,

Further upper bounds on Pre-CV and Post-CV

with

C(n) = 1 12n + 1 288n 2 .
We then obtain that

Γ(n) M i=1 Γ(β i + 1) M j=1 β j n β j = 1 n n! n n M j=1 β β j j β j ! ≥ 2π n e -n × M j=1 1 1 + C(β j ) e β j 2πβ j = 2π n M j=1 1 (1 + C(β j )) 2πβ j ≥ 2π n e -M 12 M j=1 1 (1 + C(β j )) 2πβ j ≥ 2π n e -M 12
1 √ 2π M × M n M 2 ≥ e -M 12 M M 2 √ 2π M -1 n -M +1 2 ≥ √ 2π M 2.13 M 2 × n -M +1 2 = C M n -M +1 2 ,
where we used that C(n) is maximum when n = 1 and that 1/12 ≥ log(25/288), and on the other hand that M j=1 β j is minimized when all β j are equal to n/M (if we allow continuous values).

The results presented in this section contains most of the difficulty induced by the replacement of the expectation by the CVaR in the proofs. Extending these results to other criterion is an interesting future work and may help generalize the Non Parametric Thompson Sampling algorithms to broader settings. In the next section we use these results to complete the proof of the asymptotic optimality of B-CVTS for bounded distributions.

Application to derive upper bounds on Post-CV and Pre-CV

We now use the results from previous section in order to prove that both Post-CV and Pre-CV are upper bounded by constants.

Upper bounding (Post-CV)

We start with this term because the result is direct using Lemma 5.8 and the continuity of K α,B k inf . The continuity ensures that for any ε 0 > 0 there exists

ε 1 > 0 such that if D L (F k (t), F k ) ≤ ε ⇒

Further upper bounds on Pre-CV and Post-CV

Upper bound on A 2 In order to control the term A 2 we can upper bound the numerator by 1 and use Lemma 5.9 to obtain

A 2 ≤ T n=1 E y 1 ,...,yn 1 (c α 1 -ε 1 ≤ C α (Y n ) ≤ c α 1 -ε 1 /2) 25n 3 1(Y 1 < c α 1 -ε 1 ) Y ⌈nα⌉ -Y 1 .
Here, we have introduced Y 1 , . . . , Y n to denote the ordered list of (y 1 , . . . , y n

) (i.e Y 1 ≤ Y 2 ≤ • • • ≤ Y n ). We also added the indicator 1(Y 1 ≤ c α 1 -ε 1 ) because if Y 1 ≥ c α 1 -ε 1 then C α (Y n , w) ≥ c α 1 -ε 1 for any w ∈ P n .
Furthermore, under the events we consider it also holds that

Y ⌈nα⌉ ≥ C α (Y n ) ≥ c α 1 -ε 1 .
Note that it is impossible to conclude at this step in general because the variable Y ⌈nα⌉ -Y 1 may be arbitrarily small in case all the n observations are very concentrated. However, if n is large and the distribution is continuous this event can only happen with a very low probability. This is a place in the proof where continuity is crucial. To do so, we upper bound the rest of the terms with a peeling argument on the values of Y 1 . This is done using the closed-form formulas for the distribution of the minimum of n random variable that are independent and identically distributed. Indeed, if f 1 denotes the density of arms 1, and we write the cdf and pdf of the minimum of n independent observations of ν 1 respectively L n and l n , then it holds that ∀x ∈ [0, B] 1 . The next step consists in defining a strictly decreasing sequence (a k ) k≥0 , and to look at the intervals

L n (x) = 1 -(1 -F 1 (x)) n . Now, since ν 1 is continuous it follows that in each point the density is l n (x) = nf 1 (x)(1 - F 1 (x)) n-
S k = [c α 1 -a k -ε 1 , c α 1 -a k+1 -ε 1 ].
On each of these intervals we obtain by construction that Y ⌈nα⌉ ≥ c α 1 -ε 1 ≥ Y 1 + a k+1 , and thus

E Yn 25n 3 Y ⌈nα⌉ -Y 1 1 (Y 1 ∈ S k ) ≤ 25n 3 a k+1 × P (Y 1 ∈ S k ) .
Using the properties of the density l n it holds that

P (y 1 ∈ S k ) = c α 1 -ε 1 -a k+1 c α 1 -ε 1 -a k nf 1 (x)(1 -F 1 (x)) n-1 dx ≤ sup x∈[0,B] f 1 (x) c α 1 -ε 1 -a k+1 c α 1 -ε 1 -a k n(1 -F 1 (x)) n-1 dx ≤ sup x∈[0,B] f 1 (x)(a k -a k+1 )n(1 -F 1 (c α 1 -ε 1 -a k )) n-1 .

Non-Parametric Thompson Sampling for CVaR bandits

With these results we can now upper bound A 2 by writing

A 2 ≤ T n=1 E y 1 ,...,yn 1 (c α 1 -ε 1 ≤ C α (Y n ) ≤ c α 1 -ε 1 /2) 25n 3 Y ⌈nα⌉ -Y 1 1(y 1 ≤ c α 1 -ε 1 -a 0 ) A 21 + T n=1 E y 1 ,...,yn 1 (c α 1 -ε 1 ≤ C α (Y n ) ≤ c α 1 -ε 1 /2) 25n 3 Y ⌈nα⌉ -Y 1 1(y 1 ≥ c α 1 -ε 1 -a 0 ) A 22 .
The left-hand side term can be handled thanks to Brown's inequality [START_REF] Brown | Large deviations bounds for estimating conditional value-at-risk[END_REF], which is the equivalent of Hoeffding's inequality for CVaR. Using that Y ⌈nα⌉ -Y 1 ≥ a 0 on the considered interval, we obtain

A 21 ≤ T n=1 25n 3 a 0 e -2n α(a 0 +ε 1 ) B k 2 = O(1) .
Regarding the second term A 22 we have

A 22 ≤ T n=1 sup x∈[0,B] nf 1 (x) × +∞ k=0 a k -a k+1 a k+1 (1 -F 1 (c α 1 -ε 1 -a k )) n-1 .
We first use that the cdf is increasing, which enables to upper bound

(1-F 1 (c α 1 -ε 1 -a k )) n-1 by the quantity (1 -F 1 (c α 1 -ε 1 -a 0 )) n-1 .
It remains to choose the sequence (a k ) in order to make the sum +∞ k=0 a k -a k+1 a k+1 converge. We define recursively the sequence as

a k+1 = 2 k 2 k +1 a k , starting from a 0 = c α 1 -ε 1 2 . This way, +∞ k=0 a k -a k+1 a k+1 = +∞ k=0 1 2 k = 2. This shows that A 22 ≤ 50 T n=1 n 4 sup x∈[0,B] f 1 (x) exp (-n log(1 -F 1 (c α 1 -ε 1 ))) = O(1) .
Hence, both terms A 21 and A 22 are upper bounded by constants, so A 2 = O(1).

Upper bound on A 3

To upper bound A 3 we use the same discretization arguments as in [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF]. More precisely, we introduce a number of bins M that is specified later in the proof, and for any i ∈ {1, . . . , n + 1} we define y i = ⌊M y i ⌋ M and Y n the corresponding set of truncated observations. Thanks to these definitions we can upper bound Ā3 as

Further upper bounds on Pre-CV and Post-CV

A 3 ≤ T n=1 E Yn 1 C α Y n , w < c α 1 -ε 1 1 P w∼Dn (C α ( Ỹn , w) ≥ c α 1 -ε 1 -1/M ) ≤ T n=1 E Yn 1 C α Y n , w < c α 1 -ε 1 - 1 M 1 P w∼Dn (C α ( Ỹn , w) ≥ c α 1 -ε 1 -1/M )
. Now, we use the first result of Lemma 5.10 that provides with the additional observation on the last item

P w∼Dn (C α ( Ỹn , w) ≥ c α 1 -ε 1 -1/M ) ≥ 1 n n! M i=1 β i M j=1 (p ⋆ j ) β j ,
where p ⋆ ∈ P M is a well chosen weight vector. We choose ε 1 , M small enough so that the order of the CVaRs of the arms is preserved with the discretization. Writing

ε 1 +1/M = ε ′ 1 .
Using with a slight abuse of notations M the set of allocations considered in the expectation, we can further write

A 3 ≤ T n=1 β∈M n n! M i=1 β i ! M j=1 ( p i ) β i n! M i=1 β i ! M j=1 (p ⋆ i ) β i = T n=1 n β∈M M j=1 p i p ⋆ i β i = T n=1 n β∈M exp -n KL β n , p -KL β n , p ⋆
Interestingly, the discretization allows to consider for each possible allocation of n data the ratio between the probability of pulling arm 1 in this situation, and the probability of drawing this allocation. This quantity is easy to interpret, and can be upper bounded by two terms involving the KL between the empirical frequency and some probability vector.

We can now choose p ⋆ in order to have

KL β n , p ⋆ = K α,Y inf β n , c α 1 -ε 1 -1 M . Furthermore, it holds by definition that KL β n , p ≥ K α, Ỹ inf β n , c α 1 -1 M . We can hence write KL β n , p -KL β n , p ⋆ ≥ K α, Ỹ inf β n , c α 1 - 1 M -K α,Y inf β n , c α 1 -ε 1 - 1 M ,
which is strictly positive using basic properties of K inf on the set of bounded distributions. This is however not sufficient to conclude, but we can further introduce

δ = inf p∈P M :Cα(Y,p)≤c α 1 -ε 1 -1 M K α, Ỹ inf p, c α 1 - 1 M -K α, Ỹ inf p, c α 1 -ε 1 - 1 M ,

Additional theoretical results of practical interest

Theorem 5.11 (CVaR Regret of Batched-B-CVTS). Consider a bandit problem (F 1 , . . . , F K ) ∈ F K , with respective CVaR α denoted by (c 1 , . . . , c K ) with c 1 = argmax k=1,...,K c k . Assume that the algorithm runs for T seasons, and that at each season the size of the batch is n T ≤ M ∈ N. Then, for any ε > 0 small enough there exists some ε 1 > 0, ε 2 > 0 such that the regret of Batched-B-CVTS satisfies

R α T ≤ K k=2 ∆ α k m k T + M + 2M e -2m k T ε 2 1 1 -e -2 ε 1 1 + C α 1,ε 2
,

where m k T = log(T )+log(M) K α inf (F k ,c 1 )-ε and C 1,ε 2 is a constant depending only on the distribution F 1 , the family F and ε 2 .
We see that if M is indeed a constant (i.e does not depend on the time horizon) when T is large enough it has not impact on the scaling of the regret. Before proving this result we explain the intuition behind the new terms depending on M . First, the dominant term in

m k T = O(log(T M )
) is expected since it corresponds to what we would obtained by playing at most T M times in the purely sequential setting. The main impact of the batch setting lies instead in the additive M , and corresponds to the sub-optimal arm being sampled for the whole season just before being sufficiently sampled to be identified as sub-optimal. Finally, the multiplicative M term corresponds to the empirical distribution of arm k begin mis-estimated and "costing" a full batch.

Proof. We decompose the expected number of pulls of each sub-optimal arm inside the cohort in a similar fashion as in the proof of Theorem 5. 4. First, the expected number of pulls of arm k during the total duration of the experiment is

E[N k (T )] = E   T t=1 nt f =1 1(A t,f = k)   ,
where A t,f denotes the recommendation to farmer f at season t. For any m T ∈ R we decompose the expectation according to whether N k (t -1) ≤ m T or not. We handle the corresponding first term by considering the random variable τ = {sup t≤T : N k (t -1) ≤ m T }. By construction, τ is the last season for which the total number of observations for arm k is smaller than m T . We obtain that

E T,1 := T t=1 nt f =1 1(A t,f = k, N k (t -1) ≤ m T ) ≤ τ t=1 nt f =1 1(A t,f = k, N k (t -1) ≤ m T ) + T t=τ +1 nt f =1 1(A t,f = k, N k (t -1) ≤ m T )

Non-Parametric Thompson Sampling for CVaR bandits

≤ N k (τ ) + n τ +1 f =1 1(A τ +1,f = k) ≤ m T + M ,
which is fully deterministic and gives the first two terms of the theorem. We now consider the case N k (t -1) ≥ m T , that we further analyze according to the same events as for Theorem 5.4 , i.e using that

1(F k (t -1) / ∈ B ε 1 (F k )) + 1(F k (t -1) ∈ B ε 1 (F k ), c k,t,f ≥ c 1 -ε 2 ) + 1( c 1,t,f ≤ c 1 -ε 2 ) ≥ 1 , where B ε 1 (F k ) is an ε 1 -Levy ball around F k ,
T,2 = E T t=1 nt f =1 1(A t,f = k, N k (t -1) ≥ m T ) this leads to E T,2 ≤ E   T t=1 nt f =1 1(A t,f = k, N k (t -1) ≥ m T , F k (t -1) / ∈ B ε 1 (F k ))   e 1 + E   T t=1 nt f =1 1(A t,f = k, N k (t -1) ≥ m T , F k (t -1) ∈ B ε 1 (F k ), c k,t,f ≥ c 1 -ε 2 )   e 2 + E   T t=1 nt f =1 1(A t,f = k, N k (t -1) ≥ m T , c 1,t,f ≤ c 1 -ε 2 )   e 3
.

Upper bounding e 1 Denoting by F k,n the empirical distribution of arm k after a total number of pulls n (instead of after season t), we obtain

e 1 := E   T t=1 nt f =1 1(A t,f = k, N k (t -1) ≥ m T , F k (t -1) / ∈ B ε 1 (F k ))   ≤ E   T t=1 1(N k (t -1) ≥ m T , F k (t -1) / ∈ B ε 1 (F k )) nt f =1 1(A t,f = k)   ≤ E   T t=1 T n=m T 1(N k (t -1) = n, F k,n / ∈ B ε 1 (F k )) nt f =1 1(A t,f = k)   ,

Additional theoretical results of practical interest

with a union bound on the number of pulls. Under N k (t -1) = n it holds that F k (t -1) = F k,n , and so we can further write that

e 1 ≤ E   T t=1 T n=m T 1(N k (t -1) = n, F k,n / ∈ B ε 1 (F k )) nt f =1 1(A t,f = k)   ≤ E   T n=m T 1(F k,n / ∈ B ε 1 (F k )) T t=1 nt f =1 1(A t,f = k, N k (t -1) = n)   ≤ M E T n=m T 1(F k,n / ∈ B ε 1 (F k )) = M +∞ n=m T P(F k,n / ∈ B ε 1 (F k ))
Finally, using the Dvoretzky-Kiefer-Wolfowitz inequality [START_REF] Massart | The tight constant in the dvoretzky-kiefer-wolfowitz inequality[END_REF] we obtain

e 1 ≤ M +∞ n=m T 2e -2n ε 2 1 ≤ 2M e -2m T ε 2 1 1 -e -2 ε 2 1 .
This upper bound holds for any choice of m T , ε 1 , and we remark that if m T → +∞ then e 1 → 0.

Upper bounding e 2

The term e 2 can be upper bounded with the exact same steps as in the purely sequential setting. Further using the continuity of K α,D inf for any ε > 0 we can choose ε 1 , ε 2 small enough such that the proof leads to

e 2 ≤ M × T × e -m T (K α,D inf (F k ,c 1 )-ε) . Choosing m T = log(T M ) K α,D inf (F k ,c 1 )-ε
then ensures that e 2 ≤ 1 and m T becomes the dominant term in the regret bound.

Non-Parametric Thompson Sampling for CVaR bandits

Upper bounding e 3 The final term is the one leading to the most complicated part of the analysis of B-CVTS. Fortunately, the batch setting have no impact on this part, so we can directly reuse the upper bound of e 3 in the proof of Theorem 5. 4. Indeed, we can re-write e 3 to make it equivalent to the corresponding term in the purely sequential problem:

e 3 = E   T t=1 nt f =1 1( c 1,t,f ≤ c 1 -ε 2 )   = E   S T r=1 1( c 1 (r) ≤ c 1 -ε 2 )   ,
where c 1,t,f denotes the noisy CVaR computed at season t for farmer f , while ( c 1 (r)) r≥1 represent the same quantities but assigning CVaRs computed in the same batch an arbitrary order. We used the notation S T = T t=1 n t ≤ M T . We further remark that contrarily to the previous terms, the upper bound of A 3 does not depend on M at all since the upper bound is a convergent series.

Logarithmic regret with B = +∞

This result comes from a discussion with Junya Honda, after talking about the experimental results of the next section reporting that the performance of B-CVTS does not seem to be altered by setting a conservative upper bound. The question is then to determine how far this observation can hold, and pushing it to the limit what guarantees we would obtain by setting B = +∞. The striking result that we obtained is that if α < 1 then B-CVTS still attains logarithmic regret, and the upper bound of the expected number of pulls still matches the lower bound in some "easy enough" cases. However, this comes at the price of losing asymptotic optimality in general.

Before stating the result we explain how B = +∞ can be allowed in B-CVTS. First, we recall that for a probability distribution defined by a probability w = (w 1 , . . . , w n+1 ) and a set X = (X 1 < • • • < X n+1 ) the CVaR can be expressed using n α (w) = inf{j ∈ [n + 1] : j i=1 w i ≥ α} as

C α (X , w) = 1 α nα-1 i=1 w i X i + α - nα-1 i=1 w i X nα if n α (w) ≥ 2, otherwise C α (X , w) = X 1 .
Hence, we remark that if n α ≤ n + 1 (or equivalently, w n+1 > 1 -α) then the value of exact X n+1 does not matter in the computation of C α (X , w), it just needs to be larger than X n . This allows us to adapt B-CVTS as follows: for a given arm, draw w ∼ D n+1 and compute n α (w). If n α (w) = n + 1 the bonus is included in the computation, we then set the noisy CVaR of this arm to +∞. Otherwise, we use the above formula to compute the perturbed CVaR by replacing (X 1 , . . . , X nα(w) ) by their corresponding values in the history of the arm. These two steps are actually at the core of the proof of the following Theorem 5.12.

Non-Parametric Thompson Sampling for CVaR bandits

upper bound on the BCP and follow the same steps as for Theorem 5.4 for the post-convergence term.

We believe this result is somehow surprising, or at least quite original in bandits as it is known that in the standard setting (α = 1) it is impossible to obtain logarithmic regret without knowledge of the upper bound [START_REF] Hadiji | Adaptation to the range in k-armed bandits[END_REF]. A very interesting remaining question is to determine whether this algorithm could have guarantees for a broader family of distributions (e.g light-tailed?). This would require to derive new techniques at different steps of the proof. A related question is to analyze if the function K α,+∞ inf would be actually well defined, continuous in all the arguments and if the upper bound we provide matches it, or could be improved to match it.

Experiments

In this section we report the results of experiments with the algorithms presented in the previous sections, first on synthetic examples, and then on a use-case study in agriculture based on the DSSAT agriculture simulator. With this second set of experiments we aim at showing the potential of B-CVTS to tackle the realistic problem introduced in the preamble of this thesis.

The experiments presented here are a selection of those displayed in (Baudry et al., 2021a), and we chose to focus on the B-CVTS algorithms as in the rest of this chapter. The conclusions made in this section are however still valid for M-CVTS.

Preliminary Experiments on Synthetic Examples

We first performed various experiments on synthetic data in order to check the good practical performance of B-CVTS on settings that are simple to implement and are good illustrative examples of the performance of the algorithms.

Truncated Gaussian Mixtures (TGM)

In this section we consider bounded multi-modal distributions, built by truncated Gaussian Mixture models in [0, 1]. We call these distributions Truncated Gaussian Mixtures (TGM for short). We first remark that these distributions are not continuous because they can have a positive mass in 0 and 1, but it is still a good illustrative example to check the performance of B-CVTS. Indeed, as a sanity check we performed all the experiments presented in this part by making the distributions continuous (instead of truncating, we re-sampled observations until they were in (0, 1)) and the results were deemed to be exactly the same. 154.3 (11.9) 119.5 (11.7) .18, 0.13, 0.15, 0.10, 0.13, 0.08]. The results are reported in Table 5.5, in which we observe a very large performance gap between B-CVTS and UCB algorithms. This is particularly interesting because it shows that the UCB algorithms are not really able to learn for very small values of α (indeed α = 1% is very small when drawing only a total number of 10 4 observations) before the horizon becomes extremely large. We already observed this behavior for CVaR-UCB in previous experiments, but this time we can see as well that its average regret is even higher than the one of U-UCB, and its variance spiked. On the other hand, B-CVTS seems to learn smoothly even for α = 1%, as its average regret only doubles between T = 1000 and T = 5000, and increases even less between T = 5000 and T = 10000.

Non-Parametric Thompson Sampling for CVaR bandits

Random Problems with more modes and more arms Finally, we further check the robustness of B-CVTS to more arms and more diverse distribution profiles by increasing the number of possible modes. To do so, we implement an experiment with K = 30 arms, with TGM distributions with 10 modes exhibiting different means and variances, which covers a large variety of shapes of distributions. All of those parameters are drawn uniformly at random, and we summarize their distributions as (µ, σ) ∼ U([0.25, 1] 10 × [0, 0.1] 10 ), and p ∼ D 10 (uniform distribution on the simplex, presented in Section 5.3). We name this setting TGM Experiment 6. The results of this experiment are reported in Table 5.6 for a parameter α = 0.05 averaged over 400 random instances. Again, we choose a smaller value for α than in the previous extensive sets of experiments because problems with small α seem to be more challenging. The results highlight that best performances are obtained by B-CVTS. Summary We preliminary evaluated the CVaR bandit algorithms on synthetic problems before testing them on a realistic-world bandit environment in the next section. These experiments seem to highlight a greater robustness of B-CVTS to many different settings regarding several parameters: the risk-level α, the number of arms K and the different possible shapes of the distributions (materialized by the number of modes and variances in our synthetic experiments). In particular, B-CVTS is the only algorithm that has not shown to be affected by the value of α, as the two UCB algorithms had their respective performances degraded to some extent depending on α values.

Experiments

Experiments with DSSAT crop-model

Setting In this section we study a simplified version of the problem in agriculture we introduced in the preamble of this thesis. We consider the choice of the best planting date for a maize crop in conditions emulating the types of soil and climate that can be found in Southern

Non-Parametric Thompson Sampling for CVaR bandits

Mali. We use the DSSAT simulator, to test our algorithms on this problem in silico decision. We specifically address maize planting date decision, as maize is a crucial crop for global food security [START_REF] Shiferaw | Crops that feed the world 6. past successes and future challenges to the role played by maize in global food security[END_REF]. Each simulation is assumed to be realistic, and starts from the same field initial conditions as ground measured. The simulator takes as input historical weather data, field soil measures, crop specific genetic parameters and a given crop management plan. Modeling is based on simulations of atmospheric, soil and plants compartments and their interactions. In the considered experiments, after a decision is made on planting date in the simulator, daily stochastic meteorologic features are generated according to historical data [START_REF] Richardson | Wgen: A model for generating daily weather variables[END_REF] and injected in the complex crop model. At the end of crop cycle, a maize grain yield is measured to evaluate decision-making. We parameterized the crop-model under challenging rainfed conditions on shallow sandy soils, i.e. with poor water retention and fertility. Such experiment intends to be representative of realistic conditions faced by small-holder farmers under heavy environmental constraints, such as in Sub-Saharan Africa. Thus, this setting can help picturing how CVaR bandits may perform in real-world conditions. Furthermore, we recall that that risk-aware bandits are particularly relevant for this experiment since bad trials are deleterious to the farmers, that rely on their harvest for the subsistence of their household. Depending on her profile, a farmer may be more or less risk averse, and the Conditional Value at Risk can be used to personalize her level of risk-aversion. For instance, a small-holder farmer looking for food security may seek to avoid very poor yields compromising auto-consumption (e.g α ≤ 20%), while a market-oriented farmer may be more prone to risky choices in order to increase her profit but still not risk neutral (e.g α = 80%). Furthermore, yield distributions are supposed to be bounded. Indeed, a finite yield potential can be defined under non-stressing conditions for a given crop and environment [START_REF] Evans | Yield potential: its definition, measurement, and significance[END_REF][START_REF] Tollenaar | Yield potential, yield stability and stress tolerance in maize[END_REF]. Observed yields can be modeled as following Von Liebig's law of minimum [START_REF] Paris | The return of von liebig's "law of the minimum[END_REF]: limiting factors will determine how much of the yield potential can be expressed. Hence, this experiment fits the theoretical setting we consider in this chapter. Finally, we built a bandit-oriented Python wrapper to DSSAT that we made available1 to the bandit community for reproducibility.

Experiment with 4 planting dates

We test bandit performances on the 4 armed DSSAT environment described in Table 5.7. To illustrate the non-parametric nature of these distributions, we report in Figure 5.2 estimations of their density obtained with Monte-Carlo simulations, as well as of their CVaRs. The resulting distributions are typically multi-modal, with one of their mode very close to zero (years of bad harvest), and with upper tails that cannot be properly characterized. However the practitioner can realistically assume that the distributions are upper-bounded, due to the physical constraints of crop-farming. The yield upper-bound is set to 10 t/ha thanks to expert knowledge for the considered conditions. The presented DSSAT environment advocates for the use of algorithms specifically designed for CVaR bandits, as the optimal arm can change depending on the value of the parameter α. Our experiment consists in running 64 trajectories for three algorithms U-UCB, CVaR-UCB and B-CVTS defined in Section 5. 2. Experiments are carried out with an horizon of 10 4 time steps, and we compare the results for each algorithm for α ∈ {5%, 20%, 80%} to see how the parameter impacts their performance. Indeed we want a strategy to perform well on all α choices, allowing to freely model any farmer's risk aversion level. As shown in Figure 5.3 and Table 5.8, B-CVTS appears to be consistently better than its UCB counterparts in DSSAT environment for all tested α values, which is encouraging for real-life applications.

Experiments

Experiment with 7 planting dates

We consider a bandit instance that consists of 7 arms, each arm corresponds to a planting date spaced of 15 days from the previous one. An illustration of the underlying distributions is given in Figure 5. 4. In this case, the best arm is consistent with all values of α, as shown in Table 5.9. Nevertheless, arms exhibit different gaps when considering different values of α. This experiment intends to evaluate B-CVTS performance with a greater number of real-world alike arms with a diversity of reward distribution shapes. The results are reported in Table 5. [START_REF] Cassel | [END_REF]. Furthermore, the regret curves for the three algorithms, with α ∈ {20%, 80%} parameter are illustrated in Figure 5. 5. In this experiment, by exhibiting superior performances B-CVTS appears to be more robust than the algorithms based on UCB for CVaR bandits when we increase the number of arms. In practice for the planting-date problem, a global, few months planting-window is known but needs further refinements e.g. to identify the best two-week time slot for planting. That is to say, the number of arms is unlikely to be greater that what has been tested in this experiment, making B-CVTS a particularly fit-for-purpose candidate in this setup.

Impact of support upper bound over-estimation

In this experiment we get back to the 4armed problem, but here we largely over-estimate the yield upper-bound to 30 t/ha, when a close to reality yield upper bound is about 10 t/ha. From an agronomic point of view, this yield value is a very unlikely over-estimation in the given conditions. This experiment intends to empirically evaluate how a rough arms' upper-bound estimation affects algorithms' performances, when little expert knowledge is available. An illustration of the underlying distributions and how the upper-bound estimation is exaggerated is given in Figure 5. 4.

We provide the results of this experiment in Table 5.11, and display the regret curves in Figures 5.7. This experiment addresses one possible concern for practitioners: the prerequisite of rewards' support upper bound. We empirically demonstrate that with realistic simulations, when a highly over-estimated, unrealistic support upper-bound is given to all algorithms -triple of expert's estimation -, B-CVTS keeps outperforming UCB-like CVaR bandit algorithms. We show that this over-estimation did not affect B-CVTS performances compared to the situation of correct support upper-bound identification as presented in Section 5. 6. In particular, it even slightly improved its performance for α = 80%. This result is counter-intuitive, but it can be explained by the fact that the extra exploration induced by the larger upper bound may have sped up learning in this particular case, improving overall performances. On the other hand,

Non-Parametric Thompson Sampling for CVaR bandits

CVaR-UCB seems much more impacted by this over-estimation (regret is respectively increased by about 150%, 75% and 78% for α ∈ {5%, 20%, 80%}). Similarly U-UCB's performance is altered, despite its already unsatisfying results when fed the true upper bound. Conclusion of the experiments B-CVTS appeared to be a satisfying candidate for real-world alike problems, as shown with the planting date bandits. We empirically showed that the B-CVTS algorithm was best able to deal with a greater number of planting date arms than its UCB counterparts. We showed as well that B-CVTS remained the best performer despite considering a very unlikely support upper-bound estimation. We think that in many physical resourcebased problems, this should be reassuring for practitioners, in particular when compared with UCB algorithms' sensibility to the input upper bound.

Introduction

In this chapter we get back to the standard bandit problem, where the learner sequentially chooses an action (arm) and collect a reward, with the objective of maximizing the expected sum of rewards. We recall that this is equivalent to minimizing the regret, defined as

R T = E T t=1 µ ⋆ -µ At = K k=1 ∆ k E [N k (T )] , ( 6.1)
where N k (T ) = T t=1 1(A t = k) denotes the number of selections of arm k after T time steps, µ ⋆ = max j∈{1,...,K} µ j and ∆ k = µ ⋆ -µ k is called the gap between arm k and the largest mean.

We also recall that if the arms' distributions (ν 1 , . . . , ν K ) all belong to a family of distributions F, a uniformly efficient1 bandit algorithm on F must satisfy lim inf

T →∞ R T log(T ) ≥ k:∆ k >0 ∆ k K F inf (ν k , µ ⋆ ) , K F inf (ν k , µ ⋆ ) = inf G∈F {KL(ν k , G) : E G (X) > µ ⋆ } . (6.2)
A bandit algorithm is then called asymptotically optimal for a family of distributions F when its regret matches this lower bound. In Chapter 1 we showed that such algorithm exists for instance if F is a Single-Parameter Exponential Family (SPEF) [START_REF] Cappé | Kullback-leibler upper confidence bounds for optimal sequential allocation[END_REF][START_REF] Kaufmann | Thompson sampling: An asymptotically optimal finite-time analysis[END_REF], for which K F inf is simply the Kullback-Leibler divergence between the distribution of mean µ k and that of mean µ ⋆ in F; and for bounded distributions with a known upper bound [START_REF] Cappé | Kullback-leibler upper confidence bounds for optimal sequential allocation[END_REF][START_REF] Honda | Non-asymptotic analysis of a new bandit algorithm for semi-bounded rewards[END_REF][START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF]. In this chapter we will use the notation K B inf for this family for an upper bound B.

Motivations While many algorithms achieve optimal regret for bounded distributions with the sole knowledge of the upper bound, the algorithms tackling unbounded distributions (e.g SPEF, sub-Gaussian, sub-exponential) generally assume a known parametric model for the tails. While such assumption entails convenient properties on the theoretical side, the practitioner may have some difficulty to determine which setting/parameters correspond to her problem. Furthermore, this uncertainty raises the question of robustness with respect to these hypotheses. Several works have considered this question: [START_REF] Hadiji | Adaptation to the range in k-armed bandits[END_REF] show that adapting to an unknown bounded range requires a tradeoff between instance-dependent and worst-case regret, and recently (Agrawal et al., 2021b;[START_REF] Ashutosh | Bandit algorithms: Letting go of logarithmic regret for statistical robustness[END_REF] proved the impossibility of an instance-dependent logarithmic regret for light-tailed distributions without further assumptions on the tail parameters. The root cause for this is the lack of compactness of such families F, which allows mass to "leak" at infinity so that confusing distributions with mean µ * exist arbitrarily close to ν k , meaning In the first part of this thesis we proposed a family of algorithms based on sub-sampling, that can achieve strong theoretical guarantees under non-parametric assumptions on the arms (see Assumption 2.20). The kind of structure it requires on the arms, while being more flexible than the SPEF assumption, can still be somehow difficult to verify in some case-studies, e.g for the distributions we presented in Figure 3. Hence, in this chapter we propose a complementary approach, that is inspired by an optimal algorithm for bounded distributions, Non-Parametric Thompson Sampling (NPTS), that we already studied in Chapter 5 in the context of CVaR bandits. Building on NPTS, we propose in this chapter simple alternative setups allowing unspecified tail shapes but avoiding "mass leakage" to infinity, for instance with mild conditions linking the quantiles and the means of the distributions. As in the rest of this manuscript we consider the case of light-tailed distributions (that we define below). This problem is already non-trivial, so we let possible extensions for heavy-tail distributions for future work (e.g with tools like median-of-means, see [START_REF] Bubeck | Bandits with heavy tail[END_REF]). Definition 6.1 (Light-tailed distribution). We say that a distribution ν is light-tailed if there exists λ 0 > 0 such that ∀λ : |λ| ≤ λ 0 it holds that E X∼ν e λX < +∞ .

K F inf (ν k , µ * ) = 0.
Outline Following the central question that we consider in this thesis, we want to deign algorithms that require as little knowledge on the tails of distributions as possible. To this extent, NPTS [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF]) is a good candidate, considering its simple scheme that is sufficient to reach asymptotic optimality for bounded distributions with known bounds, and that in particular does not explicitly rely on the computation of the K B inf function contrarily to other asymptotically optimal algorithms. Furthermore, the flexibility of this algorithm has been demonstrated in Chapter 5 with its adaptation in a risk-aware setting. We provide an extension of the principle of NPTS that we call Dirichlet Sampling (DS): we combine the core elements of NPTS and the duel-based framework that we already used in the first part of this thesis for the SDA algorithms and that is inspired by (Chan, 2020). This framework allows to introduce data-dependent exploration bonuses using the history of two arms. We present the resulting algorithm and detail the technical motivations of this approach in Section 6. 2. We then introduce in Section 6.3 a first decomposition of the regret of DS algorithms under .3) where we slightly change the definition of N k (compared to the one in 6.1) to

Dirichlet Sampling Algorithms

quantity RT = E T r=1 K k=1 ∆ k 1(k ∈ A r ) = K k=1 ∆ k E[N k (T )] , ( 6
N k (T ) = T r=1 1(k ∈ A r ).
We recall that the T -round regret is an upper bound of the regret after T pulls. At the beginning of each round we define a reference arm (leader), and then organize pairwise comparisons called duels between this arm and the other arms (challengers). The leader is chosen as the arm with largest sample size, that is ℓ(r) ∈ argmax k∈{1,...,K} N k (r).We choose to break ties first in favor of the best empirical arm, then with a random choice. A major motivation for this choice is that the leader will have a sample size that is linear in the number of rounds, as at least one arm is chosen at each round. This ensures strong statistical properties that we will exploit to design the exploration bonus of DS strategies. For that reason, randomizing the index of the leader is also unnecessary: it competes against each challenger with its empirical mean. We suggest to do the same with all the arms k satisfying N k (r) = N ℓ (r).

These choices have a practical interest as they avoid the computation time of drawing the largest weight vectors. We believe this can be an alternative of independent interest to computationally intensive index policies. For instance, most of the computational cost of NPTS [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF] comes from drawing the random weights for the arm we define as the leader. Finally, we can remark that this use of the duel-based structure differs from SSMC (Chan, 2020) and SDA (see Part I of this thesis) since these algorithms use the empirical means of the challengers and sub-samples from the leader in the duels.

Challenger's index

We define an index that is not dependent on the round, but only on the history of the challenger and the leader available at this round, that we denote respectively by

X = (X 1 , . . . , X n ), Y = (Y 1 , . . . , Y N )
for simplicity of notation. We denote by Xn and ȲN their respective averages. We propose a duels in two steps, with a first comparison of the means and then a comparison using a Dirichlet re-sampled mean for the challenger. This is summarized in Algorithm 6.1 below.

1 Input: History X = (X 1 , . .4) This hypothesis is standard in the bandit literature, and is for instance satisfied by any lighttailed distributions (see Definition 6.1). We refer to [START_REF] Dembo | Large Deviations Techniques and Applications[END_REF] for techniques to derive the rate function of a distribution. We recall that we already made this assumption in the first part of this thesis (Assumption 2.4).

Y i ≤ y ≤ e -nI k (y) . ( 6
We now provide an upper bound on the round-regret presented in Section 6.2 for Algorithm 6.2. To simplify the notation we consider that there is only one optimal arm and, without loss of generality, that ∀k > 1, µ k < µ 1 . Furthermore, for simplicity we write the following theorem for an index µ(Y, µ), that depends on Y through the empirical mean of the history, denoted by µ. Theorem 6.4 (Generic regret decomposition of DS). Consider a bandit model ν = (ν 1 , . . . , ν K ), where all distributions in ν satisfy (C1). Then for any Dirichlet sampled mean depending only on the history of the leader through its empirical mean, the expected number of pulls of each arm k ∈ {2, . . . , K} under the round-based strategy of Algorithm 6.2

is upper bounded for each

ε ∈ [0, ∆ k ) by E [N k (T )] ≤ n k (T ) + B k T,ε + C ν,ε ,
where C ν,ε is independent on T and,

n k (T ) = E T -1 r=1 1(k ∈ A r+1 , ℓ(r) = 1) ,

Dirichlet Sampling Beyond Bounded Rewards

and denoting Y n the set of n first observations of arm 1,

B k T,ε = K k ′ =2 ⌈2 log(T )/I 1 (µ k +ε)⌉ n=1 sup µ∈[µ k ′ -ε,µ k ′ +ε] E 1 (µ(Y n ) ≤ µ) P(µ(Y n , µ) ≥ µ)
.

The proof follows the general outline of Chan (2020) that we already used in the analysis of LB-SDA in Chapter 3, and details all the components of C k ν,ε . This term is related to deviations of sample means for arm k and arm 1 and is typically bounded by a (problem-dependent) constant under light-tail concentration (Assumption 6.3) so it does not depend on µ but only on the rate functions and the means of each arm. The other two terms of the upper bound reflect the exploration strategy. n k (T ) is the expected number of pulls of arm k when the best arm is the leader; we interpret it as the sample size required to statistically separate both arms at horizon T . On the other hand, B k T,ε measures the capacity of the best arm to recover from a bad (small-sized) sample.

We now prove the theorem, but skip some details that were provided in Chapter 3 for the analysis of LB-SDA to avoid redundancy.

Proof. Thanks to the duel structure of DS, the fact that an arm is pulled or not depends of its status as a leader or a challenger. Furthermore, a challenger can be pulled only if it wins its duel against the leader. Consider an arm k ∈ {2, . . . , K}, we first write

E[N k (T )] ≤ E T -1 r=1 1(k ∈ A r+1 , ℓ(r) = 1) n k (T ) +1 + E T -1 r=1 1(ℓ(r) ̸ = 1) E T .
We already extracted the first term n k (T ) of Theorem 6.4, and further work on the term E T corresponding to upper bounding the expected number of rounds with a sub-optimal leader. As in Chapter 3 we consider two alternatives, defining the sequence a r = ⌈r/4⌉ for r ∈ N and the event

D r = {∃u ∈ [a r , r] : ℓ(u) = 1} .
We recall that for any round s ≥ a r the number of pulls of the leader is larger than b r := ⌈a r /K⌉, and consider the event ℓ(r) ∩ D r , that can only happen in case of a leadership takeover: a sub-optimal arm has the same sample size as arm 1 (larger than b r ) and a better empirical average. Summarizing this, starting the sum at any round r 0 we have

Regret Analysis and Technical Results

T

-1 r=1 P (ℓ(r) = 1, D r ) ≤ r 0 + T -1 r=1 K k ′ =2 r u=ar r n=br E 1 N k ′ (u) = N 1 (u) = n, Ȳk ′ ,n ≥ Ȳ1,n .
We classically use that for any

x k ′ ∈ R, { Ȳk ′ ,n ≥ Ȳ1,n } ⊂ { Ȳk ′ ,n ≥ x k ′ ∪ Ȳ1,n ≤ x k ′ }
, and thanks to Assumption 6.3 we finally obtain

T -1 r=1 P (ℓ(r) = 1, D r ) ≤ r 0 + K k ′ =2 T -1 r=1 r 2 e -brI 1 (x k ′ ) + e -brI k (x k ′ ) = O(1) .
This term is hence part of the constant C ν,ε . We now consider T -1 r=1 P(ℓ(r) = 1, Dr ). As in Chapter 3, we can upper bound these events by the number of total duels lost by arm 1 against a sub-optimal leader, using that

T -1 r=1 P(ℓ(r) = 1, Dr ) ≤ 9 × E T -1 r=1 K k ′ =2 1(ℓ(r) = k ′ , C k ′ r ) ,
where C k ′ r is the event corresponding to arm 1 losing a duel against the leader k ′ . We can now fix any sub-optimal leader k ′ and upper bound the term C k ′ r . We recall that arm 1 has two chances to win the duel: first with its empirical mean, and then with a Dirichlet sampled mean. Furthermore, the bad outcome can come from either bad estimation of arm k ′ or of arm 1. For any ε > 0, it holds that

C k ′ r ⊂ Ȳk ′ ,N k ′ (r) -µ k ′ ≥ ε, ℓ(r) = k ′ ∪ Ȳk ′ ,N k ′ (r) -µ k ′ ≤ ε, ℓ(r) = k ′ , Ȳ1,N 1 (r) ≤ µ k ′ + ε, µ Y N 1 (r) , Ȳk ′ ,N k ′ (r) ≤ Ȳk ′ ,N k ′ (r) .
Thanks to the concentration of the leader, the expected number of pulls caused by the first term can be upper bounded by

T -1 r=1 P Ȳk ′ ,N k ′ (r) -µ k ′ ≥ ε, ℓ(r) = k ′ = T -1 r=1 r n=⌈r/K⌉ P Ȳk ′ ,n -µ k ′ , ≥ ε = O(1).
For the simplicity of notation we keep the notation C k ′ r to define the remaining term. We then continue the analysis of C k ′ r by considering whether N 1 (r) ≥ n 1 (T ) or not, for some new function n 1 (T ). The idea is to choose n 1 (T ) such that C k ′ r is unlikely for n ≥ n 1 (T ) thanks to the

:= T -1 r=1 P C k ′ r , ℓ(r) = k ′ we obtain C k ′ ≤ T -1 r=1 P C k ′ r , N 1 (r) ≥ n 1 (T ), ℓ(r) = k ′ + T -1 r=1 P C k ′ r , N 1 (r) ≤ n 1 (T ), ℓ(r) = k ′ ≤ T -1 r=1 T -1 n=n 1 (T ) P( Ȳ1,n ≤ µ k ′ + ε) + T -1 r=1 P C k ′ r , N 1 (r) ≤ n 1 (T ), ℓ(r) = k ′ ≤ T -1 r=1 P C k ′ r , N 1 (r) ≤ n 1 (T ), ℓ(r) = k ′ + O(1) , if n 1 (T ) ≥ log(T ) I 1 (µ ′ k +ε) .
We then consider the remaining term, that we denote by

H r,n k ′ = N 1 (r) = n, Ȳk ′ ,N k ′ (r) -µ k ′ ≤ ε, Ȳ1,n ≤ Ȳk ′ ,N k ′ (r) , µ Y n , Ȳk ′ ,N k ′ (r) ≤ Ȳk ′ ,N k ′ (r) ,
and use it to write that

T -1 r=1 P C k ′ r , N 1 (r) ≤ n 1 (T ), ℓ(r) = k ′ ≤ T -1 r=1 n 1 (T ) n=1 P H r,n k ′ .
Following for instance [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF] we can further state that

T -1 r=1 1(H r,n k ′ ) = T -1 m=1 1   T -1 r=ar 0 1(H r,n k ′ ) ≥ m   ,
and define as τ n 1 , . . . , τ n m the m first rounds for which H r,n

k ′ hold. If 1 T -1 r=ar 0 1(H r,n k ′ ) ≥ m is true then H τ j ,n j
holds for any i ≤ m and all these τ i are finite, which provides

1   T -1 r=ar 0 1(H r,n k ′ ) ≥ m   ≤ m i=1 1 H τ n i ,n k ′ .
The remaining term can be upper bounded as

D k ′ T,ε := n 1 (T ) n=1 T -1 m=1 E m i=1 1 H τ n i ,n k ′ = n 1 (T ) n=1 T -1 m=1 E Yn m i=1 P µ Y n , Ȳk ′ ,N k ′ (τ n i ) ≤ Ȳk ′ ,N k ′ (τ n i ) Y n 1 H τ n i ,n k ′ .
We remove the dependency of the index in Y k N k ′ (τ n i ) , knowing that it only depends of its mean that is located in a small range around µ j . We finally obtain

Regret Analysis and Technical Results

D

k ′ T,ε ≤ n 1 (T ) n=1 T -1 m=1 sup µ∈[µ k ′ -ε,µ k ′ +ε] E Yn P (µ (Y 1,n , µ) ≤ µ) m 1 Ȳ1,n ≤ µ ≤ n 1 (T ) n=1 sup µ∈[µ k ′ -ε,µ k ′ +ε] E Yn P (µ (Y n , µ) ≤ µ) P (µ (Y n , µ) ≥ µ) 1 Ȳ1,n ≤ µ .
This concludes the proof if we define

B k T,ε = K k ′ =2 D k ′ T,ε in Theorem 6.4.
Theorem 6.4 is formulated to be as general as possible and can be regarded as a counterpart of existing results for the analysis of other randomized strategy, such as Theorem 1 of Kveton et al. (2019b) for General Randomized Exploration or Theorem 36.2 in (Lattimore and Szepesvári, 2020) for Thompson Sampling. Remark 6.5 (Further generalizations). Theorem 6.4 still holds under more general assumptions. First, we could replace the empirical averages by any robust estimator of the means in the first comparison step, and require Assumption 6.3 to hold for this estimator instead (e.g with medianof-means or truncated means for heavy-tailed bandits). Then, the exploration bonus could use any statistics on the leader's history that have concentration properties similar to Assumption 6.3 with slight adaptations of the proof. This can include quantiles (that we can concentrate e.g thanks to DKW inequality, see [START_REF] Massart | The tight constant in the dvoretzky-kiefer-wolfowitz inequality[END_REF])), or moments of higher order under some assumptions on the distributions.

Finally, Assumption 6.3 could actually be relaxed and require inequalities in o(n -3 ) (instead of exponential decay in n) to obtain the same final result (but different constant terms) with our proof scheme.

We will later analyze instances of Dirichlet Sampling where the first-order term of the regret is driven entirely by n k (T ). We therefore introduce the following assumptions to control the contribution of B k T,ε to the regret. Assumption 6.6 (Sufficient exploration of arm 1). For any ε > 0, and any n

1 (T ) = O(log T ) it holds that n 1 (T ) n=1 E Yn∼ν n 1 1( Ȳ1,n ≤ µ 1 -ε) P w∼D n+1 (µ(Y n , µ 1 -ε) ≥ µ 1 -ε)
= o(log T ) .

Regret Analysis and Technical Results

When all observations are distinct from each other this BCP has a closed formula, which has been derived for instance in [START_REF] Cho | The volume of simplices clipped by a half space[END_REF] as .5) This expression is obtained by computing the volume of the half-space of the simplex defined by the hyperplane n+1 i=1 w i Y i ≥ µ. Unfortunately, this formula is not very informative: for sorted data the terms are alternatively positive and negative, and can take large values (compensating each other). This makes the exact formula hardly tractable even for numerical simulations. We also add that the closed formula does not exist for a Dirichlet distribution with some parameters larger than 1.

P w∼D n+1 n+1 i=1 w i Y i ≥ µ = n+1 i=1 (Y i -µ) n + n+1 j=1,j̸ =i (Y i -Y j ) . ( 6
This quantity is of much interest as both the growth of n k (T ) and checking that Assumption 6.6 holds can be performed by respectively upper and lower bounding for the BCP. In Chapter 5 we provided such bounds for the CVaR case, that are still valid here because the standard setting is a special case of CVaR bandits with a risk level α = 1. These results resort on usual properties of the Dirichlet distribution that were given in Appendix 5.7. The lower bounds suggest non-trivial tuning of the bonus. We first exhibit a necessary condition when the bonus is not allowed to depend on the set of observations Y. Lemma 6.9 (Necessary condition with a data-independent bonus). Consider a fixed bonus B µ , and a distribution F (with CDF also denoted F ). Assumption 6.6 holds only if

B µ > µ + 1 1 -F (µ) E Y ∼F [(µ -Y ) + ] .
Proof. When all the observations are below the threshold Equation ( 6.5) provides

P w∼Dn n i=1 w i Y i + w n+1 B(µ) ≥ µ = n i=1 B -µ B -Y i ,
so plugging this term in Assumption 6.6 gives the expression

E n i=1 B -Y i B -µ 1(Y i ≤ µ) = E Y 1 ∼F B -Y 1 B -µ 1(Y 1 ≤ µ) n .
The assumption can then hold only if the expectation is smaller than 1, which is equivalent to

(B -µ)(1 -F (µ)) ≥ E [(µ -Y ) + ] ,
Dirichlet Sampling Beyond Bounded Rewards which gives the result.

This result is obtained using a "worst-case" scenario in which all the observations are below the threshold µ. Hence, it does not cover all possible trajectories, yet it suggests to investigate the properties of bonuses with a similar form. Since the right-hand side of the inequality requires a knowledge on the arms distributions that we would like to avoid, we use an empirical estimator for the expectation. This suggests to introduce some parameter ρ and data-dependent bonuses of the form Definition 6.10 (Canonical bonus). For a dataset Y = (Y 1 , . . . , Y n ), a parameter ρ and a threshold µ we define .6) We interpret ρ as the leverage of the empirical excess gap 1 n n i=1 (µ-Y i ) + w.r.t the threshold µ. We then tune ρ according to the hypothesis we make on the arm distributions, which is much less constraining than assuming knowledge of the shape of the entire tail. In all DS algorithms we proposed (see next section), we use Equation ( 6 .6) as the basis for defining the appropriate bonus. Finally, we provide in Lemma 6.11 a novel lower bound on the BCP that reveals that, in the general light-tailed unbounded case, without further processing of the data, DS cannot achieve a logarithmic regret when the empirical maximum of the observations tends to +∞ at some rate g(n). Y i ≥ g(n) ≥ µ for some function g. Denoting by

B(Y, µ, ρ) = µ + ρ × 1 n n i=1 (µ -Y i ) + . ( 6
∆+ n = 1 n n+1 i=1 (µ -Y i ) + the empirical excess gap, it holds that P w∼D n+1 n+1 i=1 w i Y i ≥ µ ≥ exp -n ∆+ n g(n) -µ .
Proof. We obtain this lower bound by truncating all the observations that are larger than the threshold except the maximum, allowing to use Equation 6.

-min(Y i , µ)) = exp - n i=1 log max j=1,...,n+1 Y j -min(Y i , µ) max j=1,...,n+1 Y j -µ = exp - n i=1 log 1 + µ -min(Y i , µ) max j=1,...,n+1 Y j -µ ≥ exp - n i=1 µ -min(Y i , µ) max j=1,...,n+1 Y j -µ = exp - n i=1 (µ -Y i ) + max j=1,...,n+1 Y j -µ ,
which yields the result.

In particular, we see in this expression that g(n) may hinder the exponential rate in n. In the next section we discuss three examples of DS algorithms and their theoretical guarantees. Before that, we state a result that will be useful in the analysis of DS algorithms using the canonical exploration bonus.

P w∼D n+1 n+1 i=1 w i Y i ≥ µ ≥ exp - n ρ .
The result is direct by replacing g(n) by the expression of the bonus in Lemma 6.11.

From optimality to robustness: three instances of DS

Building on the results from the previous section, we now instantiate the DS algorithms for three bandit problems. We first prove that optimal guarantees can be derived for DS with bounded distributions under a non-standard definition of the problem (i.e unknown upper bound but alternative assumptions), motivated by practical considerations. Then, we consider a natural extension to unbounded distributions using a simple truncation mechanism, ensuring logarithmic regret under assumptions on some quantile of the distributions. Finally we consider a simple DS algorithm, securing slightly larger-than-logarithmic regret for the entire family of light-tailed distributions. In the following we denote by B(X , µ, ρ) the bonus defined in Equation 6.6 that we call canonical bonus (Definition 6.10) for a set X , a mean µ and some Dirichlet Sampling Beyond Bounded Rewards parameter ρ. For simplicity we will keep a generic µ in our exposition, while its value is in practice the empirical mean of the leading arm. We will use this expression when detailing the value of the re-sampled means used in the algorithms that we propose.

A sketch of the proofs of the three theorems can be found in Section 6.5. They consist in deriving an expression for n k (T ) and showing that Assumption 6.6 (sufficient exploration) holds for the algorithms in the settings they tackle.

Bounded Dirichlet Sampling (BDS): optimality for an alternative family of bounded distributions

Let F [b,B] be the set of distributions supported in [b, B], and consider a bandit ν

= (ν 1 , . . . , ν K ) with ν k ∼ F [b k ,B k ] for some B k ∈ R.
If we assume that B k is known (case 1), then simply defining B k as the exploration bonus ensures an asymptotically optimal regret, with a direct adaptation of the proof of NPTS [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF]. However, the precise knowledge of the upper bound for each arm is sometimes inaccessible to the practitioner (e.g if the environment is new, or if no expert is available to provide a reasonable upper bound). We propose an alternative setting, with the family

F γ,p B = {∃B : ν ∈ F [b,B] , P ν ([B -γ, B]) ≥ p} ⊂ F [b,B] .
B k is unknown but we assume it is detectable in the sense that we will observe a sample from its neighborhood [B k -γ, B k ] with a reasonable probability of at least p, with known γ, p. In this case we propose the following bonus, allowing to obtain theoretical results in this setting for some values of ρ that we will precise later, B(Y, µ) := max{Y + +γ, B(Y, µ, ρ)} , where Y + = max{y : y ∈ Y}.

(6.7)

We summarize the re-sampled mean used by BDS in Algorithm 6. 3.

1 Input: Set Y = (Y 1 , . . . , Y n ), mean of the leader µ, parameters γ, ρ 2 Draw w = (w 1 , . . . , w n+1 ) ∼ D n+1 3 return n i=1 w i Y i + w n+1 max (max n i=1 Y i + γ, B(Y, µ, ρ)) Algorithm 6.3: Bounded Dirichlet Sampling re-sampled mean
We then provide in Theorem 6.13 the theoretical guarantees obtained for BDS under this alternative "bounded distributions" assumption. B , choosing the exploration bonus of Equation 6.7 with ρ ≥ -1/ log(1 -p) ensures that

E[N k (T )] ≤ log(T ) K Bρ,γ inf (ν k , µ 1 ) + O(1) , where B ρ,γ = max (B + γ, µ 1 + ρE X∼ν k [(µ 1 -X) + ])).
This setting is a first example of the interest of data-dependent bonuses. It makes sense in practice by avoiding for instance distributions with a small mass arbitrarily far from the rest of their support, which may not be likely in a real-world application. We now consider the unbounded case. Before that, we make a remark related to the B-CVTS algorithm of Chapter 5.

Remark 6.14. The changes of the proof allowed by the round-based structure allows to remove the assumption that the distributions are continuous that was needed in the theoretical guarantees of B-CVTS, as the term A 2 in the proof of Theorem 5.4 is no more considered here thanks to the first comparison of sample means. We could thus analyze a duel-based variant of B-CVTS for arbitrary bounded distributions.

Quantile Dirichlet Sampling (QDS): truncating the upper tail for logarithmic regret with unbounded distributions

Let us consider the family F [b,+∞] for some unknown b ∈ R. A natural way to extend algorithms designed for F [b,B] (where B < +∞) is to truncate the upper tail of the distributions. We propose a simple way to do this, by considering (as a parameter of the algorithm) a quantile α, denoted by q α (ν) for a distribution ν, and a truncation operator T α that (1) do not change a distribution below its α quantile, and ( 2) "summarizes" its upper tail by its expectation, known as Conditional Value at Risk (CVaR) 2 . Formally, we obtain

∀A ⊂ [b, q α (ν)] : T α (ν)(A) = ν(A) , and ∀x > q α (ν) : T α (ν) ({x}) = α1(x = C α (ν)) , with C α (ν) = E[X|X > q α (ν)].
We then propose Quantile Dirichlet Sampling (QDS), that computes the index of a challenger (say arm k, with observations Y k ) during a duel as follow: (1) apply T α to the empirical distribution, (2) compute the bonus B(Y k , µ, ρ), and (3) re-sample the truncated empirical distribution with weights drawn according to Dir(1, . . . , 1, n α ) where parameter n α is for the weight used with the empirical CVaR, and is the number of observations used to compute it (to avoid a bias in the re-sampled mean). This procedure is summarized Dirichlet Sampling Beyond Bounded Rewards in Algorithm 6.4. If presenting this algorithm and its guarantees is rather technical, its implementation is in fact quite simple. Furthermore, the computation time of these steps can be optimized in practice (keeping in memory the sorted data, quantile and CVaR).

1 Input: Sorted data Y = (Y 1 ≤ • • • ≤ Y n ), leader mean µ, quantile α, ρ 2 Set quantile index n α = ⌈nα⌉/n 3 Set C α = 1 n-nα+1 n i=nα Y i ;
▷ Compute the CVaR := average of largest data We can analyze this algorithm for the subset of distributions ) for some b > -∞ (lower-bounded support) and a known α > 0. Then, for any ε 0 > 0 small enough QDS with any parameters α ′ < α and ρ ≥

F α [b,+∞) = {ν ∈ F [b,+∞) : ∀µ > E ν (X), K F [b,+∞) inf (ν, µ) ≥ K M k inf (T α (ν), µ)} , where M k = max{q α (ν k ), µ 1 + ρE ν k [(µ 1 -X) + ]},
(1 + α ′ )/α ′2 satisfies E[N k (T )] ≤ log T K M C k inf (T α (ν k ), µ 1 ) -ε 0 + O(1) , with M C k = max{C α (ν k ), µ 1 + ρE ν [(µ 1 -X) + ],
and T α is the truncation operator we defined.

This result captures the continuum between bounded and light-tailed distributions. In our opinion, it sheds new light on the interpretation of infeasability results of e.g [START_REF] Ashutosh | Bandit algorithms: Letting go of logarithmic regret for statistical robustness[END_REF]: logarithmic regret can be achieved without specifying the tail with precise parameters, but some simple realistic assumptions (e.g avoiding very small mass at a very large value, for instance after some quantile) can be enough to avoid pathological distributions that make little sense in practice.

6.4

From optimality to robustness: three instances of DS Remark 6.16. The restriction to the semi-bounded case b > -∞ is due to our proof technique, based on a discretization of the support of the truncated distribution (see next section). Note that the actual value of b is not known by the algorithm. This is intuitive since K

F -∞,B inf = K F b,B
inf for all b, B ∈ R, as proved in Theorem 2 of [START_REF] Honda | Non-asymptotic analysis of a new bandit algorithm for semi-bounded rewards[END_REF]. Different theoretical tools could allow to prove a logarithmic regret for QDS in the doubly unbounded case, possibly with a symmetric treatment of the two tails. We leave this extension for future work.

One may wonder whether the couple quantile condition/truncation is necessary to achieve theoretical results as well as good practical performance. Our last algorithm tackle this question.

Robust Dirichlet Sampling (RDS) for light-tailed distributions

We call Robust Dirichlet Sampling (RDS) the algorithm with bonus B(Y, µ, ρ n ), where the leverage ρ n is a function of the sample size n = |Y|. We prove that while being very simple, RDS achieves a robust sub-linear regret bound when each arm comes from any unknown light-tailed distribution, that we define following 6.1 as the family

F ℓ = {ν ∈ F (-∞,+∞) : ∃λ ν > 0, ∀λ ∈ [-λ ν , λ ν ], E ν [exp(λX)] < +∞} .
RDS is the simplest instance of Dirichlet Sampling that we propose, as can be seen in Algorithm 6.5, and is also the one with the less restrictive family of distributions.

1 Input: Data Y = (Y 1 , . . . , Y n ), mean µ, (ρ n ) n∈N 2 Draw w = (w 1 , . . . , w n+1 ) ∼ D n+1 3 return n i=1 w i Y i + w n+1 B(Y, µ, ρ n ) Algorithm 6.5: Robust Dirichlet Sampling re-sampled mean
The regret bound of RDS only depends on the choice of an increasing sequence ρ n satisfying ρ n → +∞ and ρ n = o( √ n), and Theorem 6.17 shows that RDS attain slightly larger than logarithmic regret under very general assumptions. Theorem 6.17 (Robust regret bound for RDS). Let ν = (ν 1 , . . . , ν K ) a bandit model satisfying ν k ∈ F ℓ for all k. Consider any increasing sequence (ρ n ) n∈N with ρ n → +∞, ρ n = o(n). Then, for T large enough the expected number of pull of any sub-optimal arm k in RDS is upper bounded by

E[N k (T )] ≤ n η,ε 0 k (T ) + O(1) ,

Proof Sketch

for B-CVTS in Chapter 5: we want to find a large probability confidence ball for the empirical distribution of ν k in which the BCP can be conveniently upper bounded. Lemma 6.18

Y k n ⊂ B k,n ⇒ P µ(Y k n , µ) ≥ µ ≤ exp (-f k (n, B k,n , µ)) . 2. T -1 n=1 P Y k n / ∈ B k,n = O(1).
If these two conditions hold, then one can obtain

n k (T ) = m k (T ) + O(1)
,

where for any ε > 0, m k (T ) is the sequence satisfying f k (n k (T ), B k,n , µ 1 -ε) = log T .
Proof. As in several proofs presented in previous chapter our initial upper bound is

n k (T ) ≤ m k (T ) + E T -1 r=1 1(k ∈ A r+1 , ℓ(r) = 1, N k (r) ≥ m k (T )) ,
for any sequence m k (T ). We then consider the remaining term according to the "good" event

G r k = Y k N k (r) ∈ B k,N k (r) ∩ Ȳ1,N 1 (r) ≤ µ 1 -ε 1 ,
where ε 1 > 0. Without detailing the steps (that are similar to those for B-CVTS), we claim that under the first assumption of Lemma 6.18 this event contribute to the upper bound of n k (T ) with

C G = T exp -f k (m k (T ), B k,m k (T ) , µ 1 -ε 1 ) .
Then, if G is not true one of the two events it contains does not hold. The first additional term (empirical distribution of k not in the ball) is

C Ḡ,1 := T -1 r=1 T -1 n=m k (T ) E 1(k ∈ A r+1 , N k (r) = n, Y k N k (r) / ∈ B k,N k (r) )

Proof Sketch

from (Bhat and L.A., 2019b)) for two distributions ν and ν ′ it holds that

|C α (ν) -C α (ν ′ )| ≤ 1 1 -α W 1 (ν, ν ′ ) .
Then, Theorem 2 from [START_REF] Fournier | On the rate of convergence in Wasserstein distance of the empirical measure[END_REF] provides a concentration inequality for the Wasserstein distance. Combining these elements we obtain

+∞ n=1 P Yn (Y n / ∈ B k,n ) < +∞ .
We can now consider the BCP under Y n ∈ B k,n . The upper bound of lemma 5.8 still holds, even if it is used with the truncated distribution. Hence, the QDS index satisfies

P (µ(Y n , µ) ≥ µ) ≤ exp -(n + 1)K M Yn inf (T (ν Yn ), µ) . If Y n ∈ B k,n , then M Yn is upper bounded by M Yn ≤ max (C α (ν k ), B k,ρ,µ ) + max(ε 1 , ε 2 ) , We then define M C k = max (C α ′ (ν k ), B k,ρ,µ )
, that is independent of the run of the bandit algorithm. Finally, the definition of the Levy distance ensures that d (ν Yn , ν k ) ≤ ε ⇒ d (T (ν Yn ), T (ν k )) ≤ ε. Hence, we can use the continuity of K M k inf in all arguments (including M Yn , see e.g [START_REF] Honda | Non-asymptotic analysis of a new bandit algorithm for semi-bounded rewards[END_REF]) and obtain that for any ε 0 we can calibrate ε, ε 1 , ε 2 in order to obtain for any µ ≤ µ 1 -ε 1

P (µ(Y n , µ) ≥ µ) ≤ exp -(n + 1) K M C k inf (T (ν), µ 1 ) -ε 0 ,
which gives the first order term of the regret upper bound choosing m k

(T ) = log T K M C k inf (T (ν k ),µ 1 )-ε 0 in Lemma 6.18.
RDS In this setting, we only need to control the sample Y n through its mean, the "positive gap" used in the bonus, and a range on its maximum value. Hence, we fix some ε > 0 and consider

B k,n = Y ∈ R n : Ȳn ≤ µ k + ε, Zn ≤ ∆ + k + ε, σ(Y, µ) ≤ σ k,µ + ε, Y + ∈ [m n , M n ] ,
where Y + the maximum of the set Y, (m n ) n∈N , (M n ) n∈N are two fixed sequences, and Zn =

1 n n i=1 (µ -Y i ) + is the empirical excess gap.

Dirichlet Sampling Beyond Bounded Rewards

We start by claiming that the two conditions {µ(Y) ≤ µ k + ε} and {µ(Y + ) ≤ ∆ + k + ε} lead to a constant upper bound, just as in the previous paragraph. To obtain the same result with the event involving the standard deviation we consider the Wasserstein metric W p , that is defined between two distributions ν and ν ′ of real random variables as

L p (ν, ν ′ ) = inf R×R |x -y| p ξ(dx, dy) : ξ ∈ H(ν, ν ′ ) ,
where H(ν, ν ′ ) is the set of all probability measures on R × R with marginals ν and ν ′ . Then, the Wasserstein metric W p (ν, ν ′ ) is defined as W p (ν, ν ′ ) = L p (ν, ν ′ ) 1/p for p > 1. Two reasons motivate the use of this metric in our case: 1) concentration inequalities exist for L p for lighttailed distribution, and 2) the moments of order p are continuous with respect to the Wasserstein metric W p (see Theorem 6.9 in [START_REF] Villani | Optimal transport: old and new[END_REF]). These two properties make W p a good substitute for the Levy metric used for bounded distributions. Here we choose W 2 as we want to control moments of order 2, and obtain with the parameters of Theorem 2 of [START_REF] Fournier | On the rate of convergence in Wasserstein distance of the empirical measure[END_REF] a concentration inequality that is sufficient to conclude.

We now investigate possible values for the sequence m n and M n that would allow B k,n to happen with high probability. As we saw in Chapter 4, the maximum

Y + n of a set of n i.i.d random variables Y n = (Y 1 , . . . , Y n ) has an explicit distribution, which is (in terms of the cdf F k of ν k ) for any x ∈ R, P Yn∼ν n k (Y + n ≤ x) = F k (x) n .
First we calibrate the term M n , we calibrate it to ensure that P(Y 2 .

+ n ≤ M n ) ≥ 1 - 1 n log(n) 2 , so that M n = F -1 k 1 - 1 n(log n) 2 1 n ≤ F -1 k exp - 1 n 2 (log n)
This way, P(Y + n ≤ M n ) converges. Then we consider m n , and this time we want P(Y + n ≤ m n ) ≤ 1 n log(n) 2 to ensure the same convergence guarantees. We obtain

m n = F -1 k 1 n(log n) 2 1 n = F -1 k exp - log n + 2 log log n n .
Combining all these results, we obtain

T -1 n=1 P Yn∼ν n k (Y n / ∈ B k,n ) = O(1)
.

We now use Lemma 5.8 and the fact that for any η ∈ [0, 1)

and x ∈ (-∞, η], -log(1 -x) ≤ x + 1 1-η x 2 2 . Denoting M Yn = max Ȳn , B(Y n , ρ n , µ) , Y n+1 = B(Y n , ρ n , µ
) and using the representation of Dirichlet samples as normalized exponential variables, the Chernoff inequality 2 . We recall that we consider this upper bound under the event Y n ∈ B k,n , which ensures that 1) Ȳn ∈ [m n , M n ] with the sequences we defined, 2) ∆n ≥ µ -µ k + ε, 3) the bonus is upper bounded by µ + ρ n × (∆ + k + ε), and 4) the quadratic deviation satisfies σn (µ) ≤ σ k,µ + ε. For any ε 0 > 0, if we further assume that M n = o(m 2 n ), for any n large enough these results finally provide

w i Y i + w n+1 B(Y n , ρ n , µ) ≥ µ ≤ exp - n+1 i=1 log 1 -η Y i -µ M Yn -µ ≤ 1 1 -η exp - n i=1 log 1 -η Y i -µ M Yn -µ ≤ 1 1 -η exp n i=1 η Y i -µ M Yn -µ + η 2 2(1 -η) Y i -µ M Yn -µ 2 = 1 1 -η exp -nη ∆n M Yn -µ + n η 2 2(1 -η) σn (µ) 2 (M Yn -µ) 2 , where ∆n = 1 n n i=1 µ -Y i , σ2 n (µ) = 1 n n i=1 (Y i -µ)
P (µ(Y n , µ) ≥ µ) ≤ 1 1 -η exp -nη ∆ k -ε max(M n , B n ) -µ + n η 2 2(1 -η) (σ k,µ + ε) 2 (m n -µ) 2 ≤ 1 1 -η exp -nη ∆ -ε 0 max(M n , B n ) -µ ,
where

B n = µ + ρ n (E ν k [(µ -X) + ] + ε). The condition M n = o(m 2 n
) is satisfied for light-tailed distributions, as they generally have at most a poly-logarithmic growth of the maximum (e.g log(n) for exponential tails, √ log n for gaussian tails, . . . ) and so M n and m n are actually of the same order of magnitude. We then recover all the terms of Theorem 6.17 by matching the exponent of the upper bound withlog T .

Proof of sufficient exploration

In the following we denote by E n the term to upper bound to prove Assumption 6. 6. In the second term, the exploration bonus is larger than B, so we can use the proof technique of [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF], based on a discretization scheme and on Lemma 5. [START_REF] Cassel | [END_REF]. Hence, 1) and so Assumption 6.6 is satisfied by BDS if ρ satisfies the condition of Theorem 6.13.

BDS

T -1 n=1 E n = O(

QDS

We use the assumption that rewards are semi-bounded with a range [b, +∞] to apply again the proof based on discretization of [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF]. We first find a value y and a discretization step η (unknown to the algorithm) such that truncating the values Y i to min(Y i , y), and truncating each Y i < y to Y i = η Y i η preserves the ranking of the arms. We denote by S the number of items, and β ∈ N S the vector of counts for each item. We directly use Lemma 5.10 and for any β ∈ N S : ||β|| 1 = n we want to lower bound

K β = KL(β/n, ν 1 ) -K m β inf (β/n, µ k ) ,
where ν 1 denote the discretized/truncated version of ν 1 and m β denotes the maximum between the largest item with a non-zero coefficient in β and the exploration bonus.

We recall that QDS summarizes the information larger than the empirical (1 -α)-quantile by their mean (i.e the CVaR α of the empirical distribution). The truncation in y does not change that, and will simply makes this quantity smaller which will itself makes the BCP smaller. We use the result from Honda and Takemura (2010) (proof of Theorem 7) stating that for any β

Proof Sketch

since m β is at least larger than the exploration bonus. This means that for any ξ > 0 it holds that K B ≥ ξ on all the sub-space of empirical distributions satisfying KL(β/n, ν 1 ) ≥ (1 + ξ)/ρ. We now use Pinsker inequality to link the KL divergence with the total variation δ, in the sub-space where KL(β/n, ν 1 ) ≤ (1 + ξ)/ρ, δ(β/n, ν 1 ) ≤ 1 + ξ 2ρ .

If this quantity is small, we can control the probability of each measurable event. In particular, we want the quantile used by the algorithm to be strictly larger than the (1 -α)quantile of the assumption of Theorem 6.15. If the parameter of the condition of the theorem is α, and we run the algorithm with a parameter α ′ < α, we know that with proper tuning of ρ we will have F k,n (q 1-α (F k )) < 1 -α. This means that the true quantile q 1-α (ν k ) is present in the set Y n and is not truncated by the algorithm. In particular, if ρ ≥ 1+α ′ α ′2 this is satisfied, and finally

KL(β/n, ν 1 ) -K m β inf (β/n, µ k ) ≥ K F inf (β/n, µ 1 -η) -K q 1-α ′ inf (β/n, µ k ) ≥ K q 1-α inf (β/n, µ 1 -η) -K q 1-α ′ inf (β/n, µ k ) ≥ K q 1-α ′ inf (β/n, µ 1 -η) -K q 1-α ′ inf (β/n, µ k ) ≥ κ ,
for some κ > 0 and thanks to the definition of the family F α [b,+∞] . This result concludes the proof as it ensures that Assumption 6.6 is satisfied by the QDS algorithm on F α [b,+∞] .

RDS Corollary 6.12 gives a lower bound of the BCP in e -n ρn . We directly use this result and obtain E n ≤ e -n(I 1 (µ)-1/ρn) , and for n large enough ρ n > 2I 1 (µ), which is sufficient to obtain the convergence of +∞ n=1 E n . Hence, for RDS the increasing exploration bonus directly provides larger-than-logarithmic exploration.

Experiments: crop-farming and synthetic problems

we believe that the novel Dirichlet Sampling algorithms we introduce in Section 6.4 are a good alternative choice for this problem. In particular, the three algorithms we propose in this chapter are relevant in this setting: BDS keeps the bounded-support hypothesis but introduces the possible uncertainty on the bound, while the light-tailed hypothesis of RDS and the quantile condition of QDS look reasonable.

We compare the three DS algorithms with some of the algorithms introduced in Section 1.1 that assume bounded distributions. Under this assumption one can use classical algorithms such as UCB1 (Auer et al., 2002a) or Thompson Sampling with Beta prior using the binarization trick introduced in (Agrawal and Goyal, 2012a). These algorithms enjoy logarithmic regret without the optimal rate of [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF]. We also compare DS with two optimal algorithms: IMED [START_REF] Honda | Non-asymptotic analysis of a new bandit algorithm for semi-bounded rewards[END_REF] and NPTS [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF]. We recall that the former is based on the explicit calculation of the K inf function for bounded distributions. These algorithms require the explicit knowledge of an upper bound on the support of the arms distributions. To represent the fact that a tight bound is sometimes unknown to the practitioner (uncertain environment, possibility of yet unobserved black swan events...) we run two variants of the above algorithms, one with the exact maximum yield across all simulated data, which we believe is a strong prior information, and one with the same bound inflated by 50%, which we deem a conservative estimate. Finally, we include RB-SDA (presented in Chapter 2) that requires no parameter and is then agnostic of the choice of the upper bound. However, its theoretical guarantees are not clear here.

Tuning For BDS we choose the parameters ρ = 4, γ = 3500, corresponding to p ≈ 20% in the hypothesis of Theorem 6.13, which is conservative in our example. For QDS, we set ρ = 4 to be able to compare with BDS and a quantile 95%. Finally for RDS, we choose ρ n = log (1+n), which enters into the theoretical framework of Theorem 6.17.

Results

We report the performance of DS algorithms and their competitors on the bandit problem using the DSSAT simulator we introduced in the two settings we propose: the first with a tight upper bound, and the second with a "conservative" estimate (1.5 times larger).

We present the empirical regret of the algorithms for T = 10 4 and 5000 simulations in Figure 6.2. The first striking result is that the three DS algorithms perform similarly to the optimal algorithms using the exact upper bound (IMED and NPTS), and clearly outperform the non-optimal ones (UCB1 and binarized TS). The poor performance of non-optimal algorithms hints that this particular bandit instance is not easy and requires more sophisticated methods. Furthermore, Table 3 in (Baudry et al., 2021c) shows that if RB-SDA achieves good performance, it also exhibits larger dispersion than other methods (95% quantile is 0.99 × 10 6 , standard Dirichlet Sampling Beyond Bounded Rewards deviation is 0.26 × 10 6 ). It is not clear that RB-SDA operates in its theoretical scope in this setting. For these reason, we know focus on the comparison with IMED and NPTS only, plotting the regret of a selection of algorithms in Figure 6.3 for better visualization. Interestingly, RDS is the overall winner of the experiment considering the two settings (tight and conservative bounds). We see that if Dirichlet Sampling algorithms achieve similar regret to their competitors when the latter are allowed to use the "exact" upper bound, they compare favorably with the conservative estimate. Indeed, the performance of NPTS (and to a lesser extent IMED) is deteriorated by the conservative upper bound. Considering this, RDS seems to be the overall winner in both experiments. We think this demonstrates the merits of trading-off logarithmic regret (albeit only by a factor O (log log T )) for finite-time adaptation to the tail behavior via the leverage ρ n . As a side remark, note that our round-based implementation is more efficient than NPTS as it does not draw random weights for the leader, which is the most costly operation at each round.

Experiments on synthetic data

To further illustrate the properties of DS algorithms, we perform additional experiments on synthetic examples. First, we test the sensitivity of DS w.r.t its hyper-parameters, and check that their impact on the performance of the algorithms is moderate. Then, we show the merits of RDS in case of model mis-specification, inspired the robustness experiments of [START_REF] Ashutosh | Bandit algorithms: Letting go of logarithmic regret for statistical robustness[END_REF]. Finally, we consider the case of Gaussian mixtures, a common tool to model nonparametric distributions via kernel density estimation, and show that they fit the scope of DS but not that of usual bandit algorithms.

Sensitivity of BDS to its parameters

We study the sensitivity of BDS to its parameter ρ. Theorem 6.13 suggests to scale the exploration bonus B ρ,γ as ρ = -1/ log(1 -p), which is a proxy of an upper bound of 1/ (1 -F (µ 1 )) in Lemma 6.9. We believe this bonus to be rather conservative when p is small and the distributions considered exhibit little skewness; as an example, if a distribution is such that at most 25% of its mass is located to the right of the optimal mean reward µ * , ρ ≈ 4 should be a suitable tuning.

To investigate this, we consider a toy bandit instance with two arms following uniform distributions on [0, 1] and [0.2, 0.9] respectively (note that the upper bound is different for each arm yet the distribution of mass near their respective bounds is the same, thus fitting the setting of BDS). These distributions are shown in Figure 6.4, and in particular their means are 0.5 and 0.55 respectively. For γ = 0.1, we compute the expected regret of BDS obtained with the theoretical tuning ρ = -1/ log(1 -p) ≃ 9.5, and compare it with other choices of ρ. Figure 6.4 shows that only the most extreme tuning ρ = 50 exhibits significant, albeit still sublinear, regret. Small deviations from the theoretical tuning yields similar regret, the heuristic ρ = 4 discussed above being slightly better, which tends to confirm our belief that the analysis of Theorem 6.13 can be sharpened. Note that the exploration incentive given by ρ is necessary since smaller values (e.g ρ = 0.1) tends to accumulate more regret.

Experiments: crop-farming and synthetic problems

nor SPEF. Thanks to the different sets of assumptions in which they operate, both RDS and QDS are eligible algorithms to tackle the problem of sequential decision-making in a Gaussian mixture environment, at the cost of slightly larger-than-logarithmic regret and slightly lower K inf rate respectively.

We consider two arms distributed as a 50%-50% independent mixture of N (-0.3, 0.5 2 ) and N (1.3, 0.5 2 ) and a 10%-80%-10% independent mixture of N (-1.5, 0.5 2 ), N (0.6, 0.5 2 ) and N (2.5, 0.5 2 ). Note that both mixtures have total variance equal to 0.5 2 . Due to the lack of theoretically grounded benchmark, we run three SPEF algorithms (kl-UCB, IMED and Thompson Sampling) assuming the arms belong to the SPEF of Gaussian distributions with fixed variance 0.5 2 . This is an example of model misspecification.

We run RDS with ρ n = log(1+n), which matches the asymptotic growth rate of the maximum of i.i.d Gaussian samples, and QDS with α = 5%, ρ = 4. Note that the use of QDS in this context is technically out of scope of Theorem 6.15 since Gaussian mixtures are not lower bounded; we believe however that this is an artifact of our proof technique that could be avoided with a finer analysis.

Results are reported in Figure 6. 6. Both RDS and QDS outperform other existing methods; in particular, among the misspecified SPEF algorithms, only IMED exhibit comparable regret growth. This good performance of IMED is remarkable (across all our experiments), but we do not have the intuitions to explain its better consistency w.r.t for instance TS or kl-UCB. We finally note that as this bandit problem is complicated (small optimality gap, non-SPEF distributions), all algorithms have a relatively large variance. Chapter 7

Conclusion and Perspectives

In this thesis we proposed novel algorithms to tackle several variants of the Multi-Armed Bandit problem, motivated by a case-study in agriculture. This application of bandits made us consider specifically the regret minimization setting, where the performance of the algorithm is evaluated throughout the whole duration of the experiment. In the preamble of this manuscript we introduced some of the challenges raised by this real-world problem, and among them two questions were particularly central in our works. The first one is about deriving algorithms with strong theoretical guarantees and practical performance while using as little prior knowledge as possible on the arms' distributions. The second question that we examined with care is the evaluation of bandit algorithms, that led us to consider alternative performance metrics to the expected sum of reward. This point is central for real-world applications of machine learning algorithms, where the agents involved may be risk-averse. This led us to consider CVaR bandits, where the learner wants to pull as often as possible the arm with the largest Conditional Value at Risk for some risk level α, and its limit case when α → 0 that we related to the Extreme Bandit problem. Finally, to make algorithms more practical we also considered the performance of our approaches to incorporate non stationary rewards or batch feedback, that are commonly encountered in real problems. We developed two families of algorithms to tackle these questions, that we view as complementary approaches with their own merits and scopes.

Sub-Sampling Dueling Algorithms (SDA)

The family of algorithms considered in Part I of this thesis is based on the principle of pairwise comparisons between arms, that we call duels, where the arm that has been the most selected (leader) competes against each other arm (challengers) with sub-samples of its observations. The idea is that playing a variety of duels with "independent" sub-samples from the leader gives a fair chance for an under-sampled challenger to recover from potentially bad first rewards.

Dirichlet Sampling

using a limited memory, i.e that do not keep in memory all the rewards collected, avoiding the main drawback of approaches based on sub-sampling. We first showed that a variant of LB-SDA with a poly-logarithmic memory in terms of the time horizon has the same asymptotic performance as the vanilla LB-SDA, allowing for a significant reduction of the memory usage (initially linear in the time horizon). Secondly, we proposed a natural adaptation of LB-SDA for non-stationary environments, equipping it with a sliding window. We proved that the resulting SW-LB-SDA algorithm achieves similar theoretical guarantees as comparable benchmarks (such as Sliding Window UCB), but in potentially broader settings where the mean of the distributions would not be the only feature evolving with time. In our experiments, we showed the interest of this approach with a simple example with Gaussian arms with both evolving means and variance: while the competitors require either to know the variance during each phase (strong knowledge) or at least an upper bound of possible variances (weaker knowledge, but deteriorated performance when the variance is actually smaller), SW-LB-SDA adapts naturally to changes. Hence, we believe that this adaptation of LB-SDA shows the potential of fully non-parametric algorithms in bandits.

Finally, LB-SDA inspired us to develop novel algorithms for Extreme Bandits, that we introduced in Chapter 4. We replaced the comparison of empirical means in previous chapters by the comparison of a robust estimator of the "heaviness" of a tail, that we call Quantile of Maxima (QoMax). We first proposed an Explore-Then-Commit strategy using this estimator (QoMax-ETC), before proposing QoMax-SDA, adapting LB-SDA for QoMax comparisons. We proved strong theoretical guarantees for these two algorithms, under minimal assumptions on the tails of the distributions. For instance, some of our results are obtained only assuming that one tail "dominates" (i.e is heavier than) the others asymptotically (we refer to Definition 4.3). To the best of our knowledge, this is the least restrictive assumption considered in this literature, showing again the power of non-parametric approaches, and especially of algorithms based on sub-sampling. Furthermore, both QoMax-based algorithms perform very well in practice in the experiments that we implemented, and are more efficient than their competitors in terms of computation and memory cost.

Dirichlet Sampling

In Part II of this thesis we proposed extensions of the Non-Parametric Thompson Sampling algorithm of [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF]. This algorithm computes noisy evaluations of the empirical means of each arm by re-weighting their observations with weights drawn from a Dirichlet distribution, initializing their history with the known upper bound of the distributions' support. In Chapter 5 we analyzed an extension of this algorithm for CVaR bandits, that we call Bounded -CVaR Thompson Sampling (B-CVTS). We proved that this algorithm is asymptotically optimal when distributions are bounded with a known upper bound, in the sense that the

Conclusion and Perspectives

expected number of pulls of each sub-optimal arm matches the adaptation of the lower bound of [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF] for CVaR bandits. This result is interesting because B-CVTS is the first algorithm with such guarantees in CVaR bandits. Indeed, most existing competitors are adaptations of the optimism in face of uncertainty principle, that requires a careful design of confidence intervals on the empirical CVaR. Deriving such intervals is still an active research field, and so existing CVaR bandit algorithms may have sub-optimal performance because they are not optimally calibrated. For this reason, algorithms such as B-CVTS are particularly appealing in this setting, and the detailed experiments seem to prove the merits of this alternative approach compared to its competitors. In particular, we implemented an experiment using the DSSAT simulator that emulates the crop-management problem that we introduced in the foreword of this thesis, and for which B-CVTS clearly outperforms the other algorithms.

In addition, we proved some results that further advocate for the use of B-CVTS in practice: we first showed that rewards coming in batches do not change the theoretical guarantees of B-CVTS, and we then showed that using over-estimated upper bound of the support does not alter significantly the performance of the algorithm. In fact, we even showed that for α < 1 (i.e not in the "expectation" case) setting this upper bound to +∞ still allows to prove a logarithmic CVaR-regret.

In Chapter 6 we extended NPTS in a different direction, back in the standard (expectation) setting, and analyzed possible generalizations of NPTS for alternative assumptions on the arms that could include unbounded distributions. We carefully analyzed Boundary Crossing Probabilities for the Dirichlet distribution, in order to provide sound theoretical tuning of an exploration bonus that would replace the support's upper bound. As the empirical version of this bonus uses the history of two arms, we proposed to use the leader vs challenger framework introduced in (Chan, 2020) already used for Sub-Sampling Dueling Algorithms. This algorithmic structure has computational advantages (we can avoid sampling weights for the most pulled arm), and allowed us to provide theoretical guarantees for this strategy. We call any algorithm combining these factors an instance of Dirichlet Sampling (DS). Motivated by our theoretical results, we proposed three DS algorithms that progressively relax the initial assumption of NPTS on the distributions' support: Bounded Dirichlet Sampling (BDS) achieves asymptotic optimality when the upper bound is unknown but detectable, for Quantile Dirichlet Sampling (QDS) we obtained a logarithmic regret when the arms satisfy a mild quantile assumption, and Robust Dirichlet Sampling (RDS) ensures a slightly larger than logarithmic regret when assuming only that the distributions are light-tailed. Hence, these three algorithms exhibit a theoretical trade-off between the level of generality of the assumption on the arms and the performance that can be guaranteed. However, our experiments showed that the three algorithms actually perform very similarly in practice, advocating for the use of the most robust one.

Conclusions on our contributions

Conclusions on our contributions

We think that the works presented in the two parts of this thesis show the potential of nonparametric bandit algorithms, which we describe as algorithms that do not use a parametric model on the distributions for their implementation. For example, the Last Block Sub-sampling Dueling Algorithm (LB-SDA) and Robust Dirichlet Sampling (RDS) introduced respectively in Chapter 2 and 6 do not require any information on the arms for their implementation. As those two algorithms achieve strong theoretical and practical performance, the central message of this thesis is that using less information on the arms' distribution is not necessarily costly in terms of regret and/or practical performance. We even proved that in some cases the contrary can be true, as in some examples the algorithms are more robust to a potentially complicated or mis-specified model. We view the two families of algorithms that we studied as complementary: when the learner knows absolutely nothing on the arms' distribution except that they are light-tailed we recommend the use of RDS, if the distributions are bounded and can exhibit weird shapes (e.g as in Figure 3) NPTS or BDS are suitable, and when the learner can reasonably assume that Assumption 2.20 holds we recommend using either RB-SDA or LB-SDA. Indeed, we proved that these algorithms may have optimal theoretical guarantees even if the learner cannot precisely characterize the family of distributions.

We further showed that these two families of algorithms can be extended to broader settings, for instance considering alternative performance metrics. One of our main discoveries is that a simple adaptation of NPTS/DS can lead to optimal algorithms for CVaR bandits, and that similarly a variant of SDA with a robust estimator can achieve state-of-the-art performance for Extreme Bandits. Both settings could be of interest in practical use-cases such as our cropmanagement problem in agriculture. In the two cases the novel approaches that we introduced allowed to tackle the problems under a different perspective compared to existing literature, for instance by avoiding the conservative (and sometimes complicated) design of tight confidence intervals. These findings may open new doors for future improvements, and extensions to other settings that we did not consider during this thesis. Finally, we further demonstrated the flexibility of our algorithms by analyzing their natural extensions in usual variants of the bandit problems, respectively non-stationary bandits for LB-SDA and bandits with batch feedback for B-CVTS.

Perspectives

Our research was largely motivated by the crop-management problem introduced in the preamble in this thesis, and several challenges still remain open before implementing our algorithms in the real-world. For instance, we did not consider the use of contextual information in our algorithms. So far we would implement one model for each possible combination of

Conclusion and Perspectives

crop, soil, and climate types that we would encounter (or focus on one combination), while it may be possible to discover a general structure for these parameters that would allow to use a meta-model covering every combination. Some problems may also rise from the batch setting: spatial correlations between crop yields from the same years, or temporal correlations between crop yields from the same field across the seasons. Intuitively, we would like to reduce the weight of correlated data in the learning process. Furthermore, in such realistic application we may want to use all available information to improve our algorithms. For instance, the farmers may have access to weather predictions at the beginning of and during the season, or some ground measures may help adapt the strategies. So far, these considerations have been completely out of the scope of our works, that have been dedicated to solving some of the more fundamental problems we introduced. Finally, the non-stationarity (e.g due to climate change) may also be analyzed with more precise models: changes of distributions may have a structure (e.g following a climate model), and it may be possible to think about smarter ways to adapt to them if we knew something about this structure.

On the theoretical side, we did not consider some questions that could be of interest. For instance, we may want to analyze equivalents of the DS algorithms for CVaR bandits, and in particular what kind of performance could be expected from an algorithm like RDS. Also, we focused on the CVaR case as an example of alternative performance metric, but the NPTS/DS principle may work just as well for a broader variety of risk metrics.

Regarding the Sub-sampling Dueling Algorithms, several questions are also interesting for future research. For instance, we showed that Assumption 2.20 is sufficient to make SDA work, but we wonder if it is actually necessary. Furthermore, there may be a way to circumvent this assumption and to make SDA works in settings for which it currently fails. For instance, in [START_REF] Baransi | Sub-sampling for multi-armed bandits[END_REF] the authors suggest a larger amount of forced exploration to ensure with large probability that the best arm will be estimated "well enough" so that sub-sampling is sufficient to balance exploration and exploitation. This is a possibility that we could consider. We could also imagine algorithms working in two steps: in a first step the arms are sampled enough times to ensure that some condition is met (e.g: the support is "well-covered" for all arms), and in a second step we would use SDA. A last open question that we did not have time to consider during this thesis is about the generalization of SDA to structured setting. While the number of pulls accurately represent the "quantity of information" collected for an arm in the standard setting, this is no more true in the structured case (e.g in linear bandits). Finding alternative "information measures" and new ways to implement pairwise comparisons in structured bandits is an interesting and challenging open question. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 3.3 Evolution of the means: Left, Bernoulli arms (Fig. 3.4); Right, Gaussian arms (Figs. 3.5 and 3.6). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 3.4 Performance on the Bernoulli instance of Figure 3.3, on 2000 independent replications. . 98 3.5 Performance on a Gaussian instance with a constant standard deviation of σ = 0.5 averaged on 2000 independent runs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

List of Figures

Figure 1 -

 1 Figure 1 -Facing a complicated choice.

Figure 2 -

 2 Figure 2 -A very basic example of crop-management policy

1

  Input: Data Y = (y 1 , . . . , y n ), time t

1Algorithm 1 . 6 :

 16 Input: Data Y = (y 1 , . . . , y n ) upper bound B 2 Return: I NPTS = n i=1 w i y i + w n B, where w ∼ D n+1 Index of Non Parametric Thompson Sampling

  n ), arm 2 with Y = (y 1 , . . . , y m ), n ≥ m 2 Compute µ y = 1 m m i=1 y i 3 Draw (z 1 , . . . , z m ) without replacement in X , compute µ z = 1 m m i=1 z i 4 Return: argmax y,z {µ y , µ z } ; ▷ Return winning arm Algorithm 1.8: pairwise comparison in BESA

Figure 1 . 1 -Figure 1 . 3 -

 1113 Figure 1.1 -High risk aversion (α ≈ 20%) Figure 1.2 -Low risk aversion (α ≈ 80%)

Figure 1 . 4 -

 14 Figure 1.4 -Average maximum on 10 4 samples obtained from N (1, 1) and N (1, 1.7) for a number of observations ranging from 1 to 50.

Theorem 1 .

 1 18 (Theorem 1 in Besbes et al. (

Figure 2 . 1 -

 21 Figure 2.1 -Illustration of a duel step for a few sub-sampling algorithms. Each box represent an observation, and in each figure the framed box are the observations selected by the sampler. For each arm i ∈ {ℓ(r), k}, H i denotes the history available for i at round r

→

  The sampler fails to provide enough diversity by returning always the same subsample. Example 2 (distributions unadapted to sub-sampling): Consider an interval [a, b] ⊂ R, and a two-armed bandit where arm 1 is the best arm, P ν 1 ([a, b)) = p, and ν 2 is supported on (b, +∞). Then, the regret of SDA is at least in Ω(p f T T ): if the first f T observations from ν 1 are in [a, b) the arm is never pulled outside of forced exploration.

Lemma 2 .

 2 22 (Balance condition under Assumption 2.20). Under Assumption 2.20, the balance function of any arm k ≥ 2 can be upper bounded by

Figure 2 .

 2 2 further show that they are empirically matching the Lai and Robbins' lower bound

Figure 2 . 2 -

 22 Figure 2.2 -Regret as a function of time for xp 3B and xp 2G (Right), for T = 2×10 4 and 5000 simulations.The y axis is in logarithmic scale, and the x axis starts at T = 15×10 4 to illustrate the "asymptotic" regime of the algorithms (parallel straight lines correspond to a logarithmic regret with the same constant before the log).

1 :

 1 Duel step of LB-SDA

Theorem 3 . 1 (

 31 Asymptotic optimality of LB-SDA with Limited Memory). For any bandit model ν = (ν 1 , . . . , ν K ) ⊂ P K

t=1 4 t

 4 1(t ∈ [r, 4r]) ≤ 4 log(4r/(r -1)) ≤ 9 for r ≥ 2, we finally get:

Figure 3 . 1 -

 31 Figure3.1 -Illustration of a passive leadership takeover with a sliding window τ = 4 when the standard definition of leader is used. The bold rectangle correspond to the leader. A blue square is added when an arm has an observation for the corresponding round and the red square correspond to the information that will be lost at the end of the round due to the sliding window.

Theorem 3 . 4 (

 34 Upper bound on the dynamic regret of SW-LB-SDA). If the time horizon T and number of breakpoints Γ T are known, choosing τ = O( T log(T )/Γ T ) ensures that the dynamic regret of SW-LB-SDA satisfies

  .8) Proof. We start with the first claim. Under the considered event, (n ≥ f (τ ) and D τ k (r) = 0) arm k can be drawn only if it has won its duel against k * ϕ . The duel itself is a comparison between the mean of two blocks containing at least n = min(n, τ /(2K)) observations because of the definition of the leader in this part. For any constant ξ k , we have either µ τ k (r) ≥ ξ k or µ τ ℓ,k (r) ≤ ξ k , where µ τ k (r) and µ τ ℓ,k (r) denote respectively the mean used by arm k and arm ℓ in 3.4 LB-SDA in Non-Stationary Environments their duel at round r.

  Under these events k * ϕ competes with at most 2A ϕ,τ k * ϕ different sub-sample means in the entire window [r -τ, r -1]. This is due to the fact that the sub-sample changes only if k * ϕ is pulled (can happen at most A ϕ,τ k * ϕ times) or if k * ϕ loses one observation from the window [r -2τ, r -τ -1] due to the sliding window (which can also happen at most A ϕ

Figure 3 . 2 -

 32 Figure 3.2 -Cost of storage limitation on a Bernoulli instance. The reported regret are averaged over 2000 independent replications.

Figure 3 . 3 -

 33 Figure 3.3 -Evolution of the means: Left, Bernoulli arms (Fig.3.4); Right, Gaussian arms (Figs.3.5 and 3.6).

Figure 3 . 5 -

 35 Figure 3.5 -Performance on a Gaussian instance with a constant standard deviation of σ = 0.5 averaged on 2000 independent runs.

Figure 3 . 6 -

 36 Figure 3.6 -Performance on a Gaussian instance with time dependent standard deviations averaged on 2000 independent replications.

3. 5

 5 ExperimentsInput: K arms, horizon T , τ length of sliding windowInitialization: t = 1, r = 1, ∀k ∈ {1, ..., K} : N k = 0, N τ k = 0 while t < T do A = {}, ℓ = leader(N, Y, τ ) if r = 1 then A = {1, .. . , K} ; ▷ Draw each arm once end else for k ̸ = ℓ ∈ {1, ..., K} do 10 Compute D τ k (r) ; ▷ Compute the diversity flag 11 if N τ k ≤ log(τ ) or D τ k (r) = 1 then 12 A = A ∪ {k} ; ▷ k pulled because of diversity flag or forced exploration duel between ℓ and r with their history collected during the τ last round. ; ▷ k pulled by winning the duel Pull arm k, observe reward Y k,t

4. 2

 2 Comparing Tails of Distributions with Quantiles of Maxima enough. For such n we have

1 3 4 5

 345 Input: K arms, horizon T , quantile q, number of batches b T , number of samples per batch n T 2 for k = 1, . . . , K do Pull arm k, b T × n T times Allocate the data in b T batches of size n T Compute their QoMax, Ȳ q k,n T ,b T (Algorithm 4.1) 6 end 7 for t = K × n T × b T + 1, . . .

Proposition 4 . 10 (

 410 Regret of QoMax-ETC). Let π be an ETC policy sampling N T = n T × b T times each arm during the exploration phase. If T ≥ KN T ,

Assumption 4 .

 4 11. E Y + T = o(T ), and for any γ < 1 if

Theorem 4 .

 4 15 (Vanishing regret of QoMax-ETC). Consider a bandit ν = (ν 1 , . . . , ν K ) with ν 1 ≻ ν k for k ̸ = 1. Under Assumption4.11, for any quantile q ∈ (0, 1) and any sequence(b T , n T ) satisfying b T log(T )→ +∞ and n T → +∞ , the regret of QoMax-ETC with parameters (q, b T , n T ) is vanishing in the strong sense. Furthermore, for polynomial/exponential tails with positive tail gaps this result also holds for b T = Ω(log T ).

  and we conclude with Proposition 4.10 and Assumption 4.11. For polynomial or exponential tails, as the above inequality holds for any value of c k , b T = Ω(log T ) is sufficient to obtain P(I T ̸ = 1) = o(T -1 ).

  q, arm k, leader ℓ, current history, batch count and batch size:(Y m , b m , n m )

Algorithm 4 . 3 :

 43 4.1) ; ▷ QoMax used for the leader 5 Return: argmax m∈{k,ℓ} I m Duel (q-QoMax comparison)

12 4 4 :

 44 Input: queried arm k, history Y k of size n k × b k , target batch size S Add b k new data drawn from ν k in a new row of Y k ; ▷ Update existing batches 3 while b k < S do Add n k + 1 data drawn from ν k in a new column of Y k ; ▷ Create new batch 5 b k = b k + 1 ; CollectData procedure (without storage reduction trick)

Lemma 4 .

 4 18 (Upper bound on P(ξ T ) for QoMax-SDA). Consider a bandit ν

Figure 4 . 2 -

 42 Figure 4.2 -Experiment 1: Proxy Empirical Regret (left) and Number of pulls of the dominant arm (right), averaged over 10 4 independent trajectories for T ∈ {10 3 , 2.5 × 10 3 , 5 × 10 3 , 7.5 × 10 3 , 9 × 10, 10 4 , 1.5 × 10 4 , 2 × 10 4 , 3 × 10 4 , 5 × 10 4 }.

Figure 4 . 3 -

 43 Figure 4.3 -Statistics on the number of pulls of the best arm at T = 5 × 10 4 , Experiment 1.Algorithm Average (%) 1% 10% 25% 50% 75% 90% 99%

Figure 4 . 5 -

 45 Figure 4.5 -Experiment 3: Proxy Empirical Regret (left) and Number of pulls of the dominant arm (right), averaged over 10 4 independent trajectories for T ∈ {10 3 , 2.5 × 10 3 , 5 × 10 3 , 7.5 × 10 3 , 9 × 10, 10 4 , 1.5 × 10 4 , 2 × 10 4 , 3 × 10 4 , 5 × 10 4 }.

Figure 4 . 6 -

 46 Figure 4.6 -Statistics on the number of pulls of the best arm at T = 5 × 10 4 , Experiment 3.Algorithm Average (%) 1% 10% 25% 50% 75% 90% 99%

Figure 4 . 8 -

 48 Figure 4.8 -Experiment 6: Proxy Empirical Regret (left) and Number of pulls of the dominant arm (right), averaged over 10 4 independent trajectories for T ∈ {10 3 , 2.5 × 10 3 , 5 × 10 3 , 7.5 × 10 3 , 9 × 10, 10 4 , 1.5 × 10 4 , 2 × 10 4 , 3 × 10 4 , 5 × 10 4 }.

Figure 4 . 9 -

 49 Figure 4.9 -Statistics on the number of pulls of the best arm at T = 5 × 10 4 , Experiment 6.Algorithm Average (%) 1% 10% 25% 50% 75% 90% 99%

Figure 4 .

 4 Figure 4.11 -Experiment 7 (Log-normal arms): Number of pulls of the dominant arm, averaged over 10 4 independent trajectories for T ∈ {10 3 , 2.5 × 10 3 , 5 × 10 3 , 7.5 × 10 3 , 9 × 10, 10 4 , 1.5 × 10 4 , 2 × 10 4 , 3 × 10 4 , 5 × 10 4 }.

Algorithm 4 . 6 :

 46 Efficient Update of a list of maxima for QoMax-SDA Empirical evidences of the efficiency of the storage trick We propose simulations to verify that the solution we propose to store the data used by QoMAx-SDA is indeed efficient. We performed 1000 simulations for each sample size N ∈ [10 2 , 5 × 10 2 , 10 3 , 2 × 10 3 , 5 × 10 3 , 10 4 , 2 × 10 4 , 3 × 10 4 , 5 × 10 4 ]), and for 4 distributions: (1) a Pareto distribution with tail parameter 1.1, (2) a Pareto distribution with tail parameter 3, (3) an exponential distribution with parameter 1,

189

  Input: Level α, horizon T , K, supports X 1 , . . . , X K 2 Init.: t = 11, ∀k ∈ {1, ..., K}: β k = (1, . . . , 1) length M k 3 for t ∈ {2, . . . , T } do 4 for k ∈ {1, . . . , K} do 5 Draw w k ∼ Dir(β k ) ; ▷ Draw a probability from the posterior 6 Compute c k,t = C α (X k , w k ) ; ▷ Compute the corresponding CVaR 7 end Pull arm A t = argmax k∈{1,...,K} c k,t .Receive reward r t,At .10Update β At (j) = β At (j) + 1, for j as r t,At = x j k ; We further introduce the B-CVTS algorithm (Bounded-CVaR-Thompson-Sampling)

Proposition 5 . 5 (

 55 Decomposition in Post-CV and Pre-CV terms). Under B-CVTS, the number of pulls of each sub-optimal arms satisfy

  25.1 (14.1) Testing α = 1% We then check the robustness of B-CVTS to a smaller value of the parameter α by setting α = 1%. The bandit of Experiment 5 (Exp. 5) has six TGM arms with respective mean and variance parameters µ 135 = (0.3, 0.6), µ 246 = (0.25, 0.65), σ 12 = 0.05, σ 34 = 0.06, σ 56 = 0.07. This experiment allows to additionally check if adding different variances to the arms affects the performance of the algorithms. However, we keep the probability of each mode to 0.5. This problem provides the following CVaR values for each arm at level 1%: c 0.01 1:6 = [0

Figure 5 . 3 -

 53 Figure 5.3 -Regret comparison in DSSAT environment, averaged over 1040 experiment replications, α = 5% (Left) and α = 80%, along with 90% confidence intervals.

Figure 5 . 5 -

 55 Figure 5.5 -Regret comparison with the 7-armed DSSAT environment, averaged over 1040 experiment replications, α = 5% (Left) and α = 80%, along with 90% confidence intervals.

Figure 5 . 7 -

 57 Figure 5.7 -Regret comparison with the 4-armed DSSAT environment and an over-estimated upper bound, averaged over 1040 experiment replications, α = 5% (Left) and α = 80%, along with 90% confidence intervals.

1

 1 Ashutosh et al. (2021) also6.Introductionintroduce a robust variant of UCB, that trades off logarithmic regret for O (f (T ) log(T )), where f essentially tracks the possible mass leakage at infinity. These results puts into question the usual hypotheses under which bandit algorithms are designed: considering a parametric control of the tails is indeed sensitive to model mis-specification, but on the other hand the examples chosen to prove infeasability results seem a bit extreme for the practitioner.

Lemma 6 .

 6 11 (Lower bound for the BCP). Consider a set Y = (Y 1 , . . . , Y n+1 ) ∈ R n+1 and any threshold µ, and assume that max i∈{1,...,n+1}

Corollary 6 .

 6 12. Let Y = (Y 1 , . . . , Y n , Y n ) be a set and Y n+1 = B(Y, µ, ρ) for some parameters ρ, µ. Then, it holds that

6. 4

 4 From optimality to robustness: three instances of DS Theorem 6.13 (Theoretical guarantees of BDS). If ∀k ∈ {2, .. . , K} , ν k ∼ F γ,ρ

  and the second K inf is taken on the family F [b,M k ] (using previously introduced notations). Although technical, this condition essentially states that the bandit problem taken on the complete family F [b,+∞) is no harder than an alternative bandit problem considering the truncated distributions and a bounded family, with an upper bound depending on the 1 -α quantile and the leverage ρ of the exploration bonus. Theorem 6.15 (Logarithmic Regret of QDS). Consider a bandit model ν = (ν 1 , . . . , ν K ) satisfying ∀k, ν k ∈ F α [b,+∞

2 .

 2 We use the hypothesis P([B -γ, B]) ≥ p and the second component of the bonus, Ȳn + γ := max Y i + γ along with Corollary 6.12 to obtain Dirichlet Sampling Beyond Bounded RewardsE n ≤E Yn 1(µ(Y n ) ≤ µ)(1( Ȳn ≤ B -γ) + 1( Ȳn ≥ B -γ)) P ( µ(Y n , µ) ≥ µ) ≤ (1 -p) n e Y n ) ≤ µ)1( Ȳn + γ ≥ B) P ( µ(Y n , µ) ≥ µ) E n,The two terms correspond to the two possible expressions for the bonus. The term E n,1 gives the sufficient condition for the tuning of ρ in Theorem6.13 with ρ > -1 log(1 -p) ⇒ +∞ n=1 E n,1 = O(1) .

K

Figure 6 . 2 -

 62 Figure 6.2 -Average regret on 5000 simulations and horizon T = 10 4 . Dashed lines correspond to 5%-95% regret quantiles. UCB1, Binarized Thompson Sampling, Empirical IMED and NPTS are run with exact upper bounds around 1.5 × 10 4 kg/ha (left) and the conservative upper bound 1.5 × 10 4 kg/ha (right). BDS: ρ = 4. RDS: ρ n = log(1+n). QDS: ρ = 4, α = 5%.

Figure 6 . 3 -

 63 Figure 6.3 -Average regret on 5000 simulations and horizon T = 10 4 . Dashed lines correspond to 5%-95% regret quantiles. Empirical IMED and NPTS are run with exact upper bounds around 1.5 × 10 4 kg/ha (left) and the conservative upper bound 1.5 × 10 4 kg/ha (right).

Figure 6 . 4 -

 64 Figure 6.4 -Left: bandit with two uniform arms U(0, 1) and U(0.2, 0.9) (10 4 samples each). Right: average regret on 5000 simulations and horizon T = 10 4 of BDS for various values of ρ.

Figure 6 . 6 -

 66 Figure 6.6 -Left: Gaussian mixture arms (10 4 samples each). Right: average regret on 5000 simulations and horizon T = 10 4 . kl-UCB, IMED and Thompson Sampling are run assuming Gaussian arms with same variance as the mixtures. RDS: ρ n = log(1+n). QDS: ρ = 4, α = 5%.

1 1 . 1 1 . 2 1 . 4 2 . 1 3 . 1 3 . 2

 111214213132 Facing a complicated choice. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii 2 A very basic example of crop-management policy . . . . . . . . . . . . . . . . . . . . . . ix 3 Distribution of simulated dry grain yield (kg/ha) for seven different planting dates, all other parameters being equal. Reported on the x-axis are the distribution minimum, mean and maximum values. The optimal arm is the third one (mean 3630 kg/ha) if we want to maximize the expected yield. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi High risk aversion (α ≈ 20%) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Low risk aversion (α ≈ 80%) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.3 The Conditional Value-at-Risk (CVaR) of level α is the mean value of the blue area of the distribution, that stops at VaR α . The red line is the average µ of the distribution. . . . . . 15 Average maximum on 10 4 samples obtained from N (1, 1) and N (1, 1.7) for a number of observations ranging from 1 to 50. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Illustration of a duel step for a few sub-sampling algorithms. Each box represent an observation, and in each figure the framed box are the observations selected by the sampler. For each arm i ∈ {ℓ(r), k}, H i denotes the history available for i at round r . . 38 2.2 Regret as a function of time for xp 3B and xp 2G (Right), for T = 2 × 10 4 and 5000 simulations. The y axis is in logarithmic scale, and the x axis starts at T = 15 × 10 4 to illustrate the "asymptotic" regime of the algorithms (parallel straight lines correspond to a logarithmic regret with the same constant before the log). . . . . . . . . . . . . . . . . 58 Illustration of a passive leadership takeover with a sliding window τ = 4 when the standard definition of leader is used. The bold rectangle correspond to the leader. A blue square is added when an arm has an observation for the corresponding round and the red square correspond to the information that will be lost at the end of the round due to the sliding window. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 Cost of storage limitation on a Bernoulli instance. The reported regret are averaged over 2000 independent replications.

Table

  .2 -Overview of risk metrics

4 end 5 else 6 return Arm

  . . , X t ) and corresponding arms K = (k 1 , . . . , k t ), Bandit algorithm π chosen by π using (X t-τ +1 , . . . , X t ) and (k t-τ +1 , .. . , k t )

	7 end
	Algorithm 1.11: Generic sliding-window strategy

2 if t ≤ τ then 3 return Arm chosen by π using X and K 1 Input: Discount factor ρ, rewards Y = (y 1 , . . . , X t ) and corresponding arms K = (k 1 , . . . , k t ), Bandit algorithm π 2 return Arm chosen by π using Y with discounts ρ = (ρ t , ρ t-1 , . . . , ρ 2 , ρ, 1) and K Algorithm 1.12: Generic strategy with discounted rewards

• Sub-sampling for Efficient Non-Parametric Bandit Exploration, by Dorian Baudry, Emilie

	Introduction to some Bandit Problems
	• Efficient Algorithms for Extreme Bandits, by Dorian Baudry & Yoan Russac & Emilie Kauf-
	mann. Proceedings of the 25th International Conference on Artificial Intelligence and Statistics
	(AISTATS 2022). See Chapter 4. (Baudry et al., 2022).
	• Top-Two algorithms revisited, by Marc Jourdan & Remy Degenne & Dorian Baudry &
	Rianne de Heide & Emilie Kaufmann. Advances in Neural Information Processing Systems
	35 (NeurIPS 2022) (Jourdan et al., 2022). We do not detail this work in this thesis, as it
	tackles a quite different pure-exploration objective.
	Submitted works
	• Towards an efficient and risk aware strategy for guiding farmers in identifying best crop manage-
	ment, by Romain Gautron & Dorian Baudry & Myriam Adam & Gatien N. Falconnier &
	Marc Corbeels. Theorem 5.11 comes from this work.
	Kaufmann & Odalric-Ambrym Maillard. Advances in Neural Information Processing Systems
	34 (NeurIPS 2020). See Chapter 2. (Baudry et al., 2020).
	• Optimal Thompson Sampling strategies for support-aware CVaR bandits, by Dorian Baudry
	& Romain Gautron & Emilie Kaufmann & Odalric-Ambrym Maillard. Proceedings of the
	the Thirty-eighth International Conference on Machine Learning (ICML 2021). See Chapter 5.
	(Baudry et al., 2021a).
	• On Limited-Memory Subsampling Strategies for Bandits, by Dorian Baudry & Yoan Russac
	& Olivier Cappé. Proceedings of the the Thirty-eighth International Conference on Machine
	Learning (ICML 2021). See Chapter 2 and 3. (Baudry et al., 2021b).
	• From Optimality to Robustness: Adaptive Re-Sampling Strategies in Stochastic Bandits, by Do-
	rian Baudry & Patrick Saux & Odalric-Ambrym Maillard. Advances in Neural Information
	Processing Systems 35 (NeurIPS 2021). See Chapter 6. (Baudry et al., 2021c).

Part I Bandit Algorithms Based on Sub-Sampling Chapter 2

sample of ℓ used for the duel with k 7 if Ȳk,N k > Ȳℓ, Sr k then 8 A = A ∪ {k} ; ▷ k added to A if it wins the duel against ℓ 9 end 10 end 11 if |A| = 0 then 12

	1 Input: K arms, horizon of T rounds, Sampler SP, leader definition
	2 ∀k , N k = 1, H k = {Y k,1 } ;	▷ Each arm is drawn once
	3 for r ∈ {1, . . . , T } do
	4	A = {}, ℓ = leader((N k ) k∈{1,...,K} , (H k ) k∈{1,...,K} , ℓ) ; ▷ Initialize the round
	5	for k ̸ = ℓ ∈ 1, ..., K do
	6	Draw Sr k ∼ SP(N k , N ℓ , r) ; ▷ Sub-A = {ℓ} ; ▷ If no challenger in A then ℓ is pulled
	13	end
	14	for a ∈ A do
	15	Pull arm a, observe reward Y a,Na+1
	16	N a = N a + 1, H a = H a ∪ {Y a,Na } ; ▷ Update number of pulls and history

17 end 18 end Algorithm 2.1: Generic

SP-SDA

Table 2 .

 2 1 -Experiments with Bernoulli arms Name Number of arms Means {µ 1 , . . . , µ K

Table 2 .

 2 2 -Experiments with Gaussian arms with variance σ 2 = 1 Name Number of arms Means {µ 1 , . . . , µ K

Table 2 .

 2 3 -Regret and at T = 20000 for Bernoulli arms, with standard deviation

		Benchmark				SDA
	xp TS	PHE	BESA	SSMC RB	WR	LB	LDS
	1B 11.2	25.9	11.7	12.3	11.5	11.6	12.2	11.4
	(10.)	(87.9) (12.1)	(7.3)	(10.1) (10.2) (7.4)	(9.0)
	2B 22.9	24.0	22.1	24.3	22.0	21.5	24.0	21.8
	(29.2) (22.0) (25.2)	(38.2) (34.5) (17.3) (24.6) (24.5)
	3B 94.2	248.1 88.1	100.1	89.0	86.9	100.7 89.2
	(15.8) (25.5) (89.2)	(20.0) (19.8) (21.7) (21.3) (21.8)
	4B 108.1 216.5 147.5	119.9	105.1 106.9 119.6 106.8
	(45.1) (89.8) (209.8) (40.8) (41.1) (42.1) (42.7) (47.7)

Table 2 .

 2 4 -Regret and at T = 20000 for Gaussian arms, with standard deviation

				Benchmark					SDA
		xp TS	ReBoot BESA	SSMC RB	WR	LB		LDS
		1G 24.4	92.2	25.3		26.9	25.6	24.7	25.1		26.5
			(17.1)	(23.4)	(27.1)	(52.8) (62.8)	(20.6) (17.9) (140.2)
		2G 73.5	277.1	122.5	74.8	71.0	71.1	74.6		69.0
			(107.8) (41.3)	(585.5) (34.7) (152.2) (50.2) (35.1) (50.4)
		3G 49.7	190.9	72.1		51.3	50.4	50.0	51.2		48.6
			(26.9)	(29.6)	(410.3) (23.7) (156.5) (33.3) (22.4) (41.6)
	180							120
	160							110
	120 140			RB-SDA LB-SDA WR-SDA LDS-SDA	SSMC TS Lower Bound	90 100		RB-SDA LB-SDA WR-SDA LDS-SDA	SSMC TS Lower Bound
								80
	100
								70
	80	1.5 × 10 4	1.6 × 10 4	1.7 × 10 4	1.8 × 10 4	1.9 × 10 4		1.5 × 10 4	1.6 × 10 4	1.7 × 10 4	1.8 × 10 4	1.9 × 10 4

Table 2 .

 2 5 -Experiments with Truncated Gaussian arms Name Number of arms Means {µ 1 , . . . , µ K } and Std

Table 2 .

 2 6 -Regret at T = 20000 for Truncated Gaussian arms

				Benchmark				SDA
	xp	TS	NPTS PHE	BESA	SSMC RB	WR	LB	LDS
	1TG 21.9	4.2	22.3	1.4	1.5	1.4	1.4	1.5	1.4
		(20.4) (0.6)	(2.6)	(1.7)	(0.7)	(1.1)	(0.8)	(0.7)	(0.8)
	2TG 13.3	8	19.5	4.6	4.7	4.4	4.5	4.6	4.3
		(7)	(1.8)	(3.8)	(3.3)	(2.3)	(4.6)	(3.1)	(2.4)	(2.9)
	3TG 9.7	7.8	48.5	7.8	7.6	7.1	7.7	8.2	7.1
		(10.1) (4.5)	(217.8) (9.4)	(5)	(10)	(13.4) (27.5) (5.8)
	4TG 86.6	70	86	76.5	69.5	64.9	64.8	68.7	63.2
		(57.8) (39.4) (53.7)	(113.9) (40.9) (60.5) (43.9) (39.1) (51.1)

Table 2 .

 2

7 -Experiments with Exponential arms Name Number of arms Means {µ 1 , . . . , µ K } and Std

Table 2 .

 2 8 -Average Regret with Exponential Arms (with std) without forced exploration

	xp TS	IMED BESA	SSMC	RB	WR	LB	LDS
	1E 48.2	40.0	45.7	41.9	44.8	45.4	46.6	45.5
	(191.8)	(78.4)	(114.1)	(84.2)	(121.4)	(134.4)	(176.8)	(109.7)
	2E 3.8	3.4	4.2	3.6	4.1	3.9	3.9	5.4
	(9.9)	(3.6)	(25.1)	(41.9)	(14.3)	(13.4)	(8.7)	(49.5)
	3E 832.8	779.9	820.5	856.9	848.4	778.4	846.7	877.7
	(1065.1) (896.9) (1304.6) (1111.0) (1533.3) (1118.7) (1150.1) (1708.7)
	4E 258.3	234.6	525.4	251.3	272.6	262.1	263.8	258.4
	(519.6)	(126.6) (2115.1) (328.3)	(692.2)	(524.4)	(477.9)	(599.0)
	5E 25.6	24.0	55.7	25.6	25.5	25.0	26.5	24.7
	(51.2)	(33.6)	(219.9)	(23.6)	(46.7)	(24.0)	(36.8)	(37.6)
	6E 618.7	603.6	1184.2	627.9	595.7	616.0	652.6	605.9
	(672.3)	(576.8) (3096.4) (755.6)	(790.7)	(780.2)	(685.3)	(871.4)

Table 2 .

 2

	% of runs TS	IMED SSMC RB	WR	LB	LDS
	20%	319.8 336.0 335.0 261.0 290.0 326.0 261.8
	50%	626.0 650.0 661.0 532.0 568.5 642.0 536.0
	80%	1122.0 1080.0 1142.0 1006.0 1019.0 1143.2 1020.2
	95%	1924.1 1704.0 1846.0 2199.0 1817.2 1869.1 2134.1
	99%	4209.4 2632.9 3536.8 6813.1 4146.0 3762.3 7396.7

9 -Quantiles of the distribution of empirical regret at T = 10 4 for Experiment 3 with exponential arms, over 5000 runs.

Table 2 .

 2 10 -Average regret with exponential arms: SDA with forced exploration xp RB-SDA WR-SDA LB-SDA LDS-SDA

	1E 44.9	42.5	42.4	45.0
	(167.3)	(107.4)	(60.5)	(176.0)
	2E 3.6	3.4	4.0	3.6
	(9.2)	(2.2)	(27.9)	(11.2)
	3E 837.5	788.5	827.7	832.3
	(1466.1) (1222.1)	(1055.3) (1514.6)
	4E 244.8	238.9	251.7	246.0
	(403.3)	(250.8)	(248.5)	(323.4)
	5E 23.6	25.1	25.4	24.9
	(33.4)	(41.0)	(23.4)	(42.2)
	6E 578.9	595.1	631.2	577.8
	(651.9)	(561.3)	(446.4)	(652.7)

Table 2 .

 2

		11 -Average Regret on 10000 random
	experiments with Bernoulli Arms
	T	TS IMED PHE SSMC RB WR
	10 2 13.8 15.1	16.7 16.5	14.8 14.3
	10 3 27.8 31.9	39.5 34.2	31.8 30.9
	10 4 45.8 51.2	72.3 55.0	51.1 50.6
	2.10 4 52.2 57.6	85.6 61.9	57.7 57.3

Table 2 .

 2

		12 -Average Regret on 10000 random
	experiments with Gaussian Arms
	T	TS	IMED WR RB	SSMC
	10 2 41.2 45.1	38.3 38.1 40.6
	10 3 76.4 82.1	72.7 70.4 76.2
	10 4 118.5 124.0 115.8 111.8 120.1
	2.10 4 132.6 138.1 130.2 125.7 135.1

Table 3 .

 3 1 -Memory and computational costs at round T for existing subsampling algorithms.

	Algorithm	Memory	Computational cost
			Best-Worst case
	BESA Baransi et al. (2014)	O(T )	O((log T ) 2 )
	SSMC Chan (2020)	O(T )	O(1)-O(T )
	RB-SDA Baudry et al. (2020)	O(T )	O(log T )
	LB-SDA
	(this chapter)

  {}, ℓ = leader((n k ) k∈{1,...,K} , (Y k ) k∈{1,...,K} ) ; ▷ Define the leader

			4.4 QoMax-SDA
	1 Input: K arms, quantile level q
	2 exploration function f , batch function B
	3 Initialization: r = 1
	4 ∀k ∈ {1, ..., K}: n k = 1, Y k = {X k 1 } ;	▷ Draw each arm once
	5 for r ≥ 2 do
	7 end
	8 for k ̸ = ℓ ∈ {1, ..., K} do
	9	if n k < f (r) or Duel(k, ℓ) = k (Alg. 4.3) then
	10	A = A ∪ {k} ; ▷ k pulled if it is not enough explored or wins the duel
	11	end
	12 end
	13 if |A| = 0 then
	14	A = {l} ;	▷ Draw the leader if no winning challenger
	15 end
	16 for k ∈ A do
	17	CollectData(Y k , B(r)) (Alg. 6) ;	▷ Data collection

6 A = 18 n k = n k + 1 19 end Algorithm 4.5: QoMax-SDA (simplified data collection procedure)

  -Statistics on the distributions of maxima at T = 5 × 10 4 , Experiment 1. Results divided by 100 to improve readability.

										4.5 Practical performance
	Experiment 3
	-1.5						0.8
	-1.6						0.7
	-1.7						0.6
	-1.8		QoMax-SDA (q=1/2) QoMax-SDA (q=0.9)	ExtremeHunter ExtremeETC	0.5		QoMax-SDA (q=1/2) QoMax-SDA (q=0.9)	ExtremeHunter ExtremeETC
	-1.9		QoMax-ETC (q=1/2) QoMax-ETC (q=0.9)	MaxMedian ThresholdAscent	0.4		QoMax-ETC (q=1/2) QoMax-ETC (q=0.9)	MaxMedian ThresholdAscent
							0.3
	-2.0
							0.2
	-2.1
							0.1
	0	10000	20000	30000	40000	50000	0	10000	20000	30000	40000	50000
		Algorithm	Average (%) 1% 10% 25% 50% 75% 90% 99%
	QoMax-SDA (q = 1/2)		92	42	90	93	94	95	95
	QoMax-SDA (q = 0.9)		93	14	87	93	96	97	98
	QoMax-ETC (q = 1/2)		89	90	90	90	90	90	90
	QoMax-ETC (q = 0.9)		88	3	90	90	90	90	90
		ExtremeETC		71	3	3	90	90	90	90
		ExtremeHunter		79	3	5	89	90	90	90
		MaxMedian		72	0	0	0	100	100	100	100
		ThresholdAscent		53	46	50	52	53	55	56
	Figure 4.4 Algorithm		Average 1% 10% 25% 50% 75% 90%	99%
		QoMax-SDA (q = 1/2)	1852	41	81	130	245	547 1350
		QoMax-SDA (q = 0.9)	1042	39	78	128	239	529 1363
		QoMax-ETC (q = 1/2)	1058	40	79	126	232	530 1324
		QoMax-ETC (q = 0.9)	919	34	75	122	230	511 1301
		ExtremeETC		882	16	44	86	183	426 1089 9515
		ExtremeHunter		1092	21	61	104	208	477 1226 9799
		MaxMedian		785	3	37	83	180	436 1126 9240
		ThresholdAscent		748	27	51	82	156	351	853	7771

4

, Experiment 1.

  Statistics on the distributions of maxima at T = 5 × 10 4 , Experiment 6. Results divided by 100 to improve readability.

							4.5 Practical performance
	Experiment 7
	1.0
	0.8
	0.6
	0.4
			QoMax-SDA (q=1/2)	ExtremeHunter
			QoMax-SDA (q=0.9)	ExtremeETC
	0.2		QoMax-ETC (q=1/2)	MaxMedian
			QoMax-ETC (q=0.9)	ThresholdAscent
	0	10000	20000	30000	40000	50000
	Algorithm	Average (%) 1% 10% 25% 50% 75% 90% 99%
	QoMax-SDA (q = 1/2)	14	1	1	2	4		45	95
	QoMax-SDA (q = 0.9)	36	0	1	3	22		90	98
	QoMax-ETC (q = 1/2)	3	3	3	3	3	3	3	3
	QoMax-ETC (q = 0.9)	15	3	3	3	3	3	95	95
	ExtremeETC	79	3	3	95	95		95	95
	ExtremeHunter	87	3	85	95	95		95	95
	MaxMedian	1	0	0	0	0	0	0	5
	ThresholdAscent	43	27	34	38	43		53	60
	Figure 4.10 -Algorithm	Average 1% 10% 25% 50% 75% 90% 99%
	QoMax-SDA (q = 1/2)	60	5	9	12	21	41	91	635
	QoMax-SDA (q = 0.9)	120	6	10	15	28	64	155 1144
	QoMax-ETC (q = 1/2)	40	5	8	11	18	33	64	306
	QoMax-ETC (q = 0.9)	59	5	8	12	20	40	93	702
	ExtremeETC	267	6	14	24	47	108	266 2687
	ExtremeHunter	232	8	17	28	53	116	305 2620
	MaxMedian	35	0	7	10	17	30	60	306
	ThresholdAscent	136	7	12	18	33	70	170 1299

4

, Experiment 6.

Table 4 .

 4 2 -Statistics on the distributions of number of pulls of the best arm at T = 5 × 10 4 , Exp. 7

	Algorithm	Average 1% 10% 25% 50% 75% 90% 99%
	QoMax-SDA (q = 1/2)	94	85	94	95	95	95	95	95
	QoMax-SDA (q = 0.9)	97	89	96	97	98	98	98	98
	QoMax-ETC (q = 1/2)	90	90	90	90	90	90	90	90
	QoMax-ETC (q = 0.9)	90	90	90	90	90	90	90	90
	ExtremeETC	55	3	3	3	90	90	90	90
	ExtremeHunter	63	13	40	45	53	90	90	90
	MaxMedian	7	0	0	0	0	0	0	100
	ThresholdAscent	57	55	56	57	58	58	58	58

Table 4 .

 4 3 -Statistics on the distributions of maxima at T = 5 × 10 4 , Experiment 7. Results divided by 1000 to improve readability.

	Algorithm	Average 1% 10% 25% 50% 75% 90%	99%
	QoMax-SDA (q = 1/2)	1393	73	151	257	488 1090 2259 13764
	QoMax-SDA (q = 0.9)	1401	79	163	260	524 1171 2830 13839
	QoMax-ETC (q = 1/2)	1337	77	154	245	430 1007 2664 13651
	QoMax-ETC (q = 0.9)	1459	84	150	251	461	987 2419 12654
	ExtremeETC	957	6	12	30	214	581 1511 7422
	ExtremeHunter	867	32	85	156	297	666 1569 10855
	MaxMedian	76	0	0	0	0	0	15	1678
	ThresholdAscent	1043	43	94	160	311	667 1648 10715

Appendix A: proofs of section 4.4 (SDA) 4.6.2 Proof of Theorem 4.19

	Theorem 4.19 (Upper bound on the regret of QoMax-SDA). Under the assumptions of
	Lemma 4.18 it further holds that
	1. the regret of QoMax-SDA is vanishing in the strong sense for exponential tails
	2. the regret of QoMax-SDA is vanishing in the weak sense for polynomial tails.

.

  Remark 4.22. With this last result we can further say that the regret of QoMax-SDA is strongly vanishing if λ 1 is larger than 2+ √ 3, but we omitted this result in our statement since it is impossible to know in advance.

  and ε 1 , ε 2 are two small positive constants. We denoted by ( c k,t,f ) the noisy CVaRS computed by the algorithm for any arm k, season t and farmer f . Using the notation E

Table 5 .

 5 1 -CVaR regret (average and std) for Exp. 1 at T = 10000 for 5000 replications.

	α	U-UCB	CVaR-UCB B-CVTS
	10% 274.9 (1.8)	5.3 (1.5)	1.1 (0.5)
	50% 127.0 (19.3) 135.3 (41.1) 29.8 (17.2)
	90% 80.5 (10.4)	53.5 (6.7)	10.2 (17.9)

Table 5 .

 5 2 -CVaR regret (average and std) for Exp. 2 at T = 10000 for 5000 replications.

	α	U-UCB	CVaR-UCB B-CVTS
	10% 373.7 (4.1) 72.8 (9.6)	4.1 (2.3)
	50% 135.8 (8.9) 37.9 (7.5)	5.5 (2.7)
	90% 62.6 (7.1)	43.9 (5.1)	5.0 (1.8)

Table 5 .

 5 3 -CVaR regret (average and std) for Exp. 3 at T = 10000 for 5000 replications.

	α	U-UCB	CVaR-UCB B-CVTS
	10% 269.4 (1.8)	23.2 (4.8)	2.8 (1.5)
	50% 138.5 (12.4) 71.8 (19.0)	14.7 (8.3)
	90% 53.1 (6.6)	34.5 (6.6)	20.2 (22.4)

Table 5 .

 5 4 -CVaR regret (average and std) for Exp. 4 at T = 10000 for 5000 replications.

	α	U-UCB	CVaR-UCB B-CVTS
	10% 958.9 (4.8)	230.5 (25.3) 10.4 (3.2)
	50% 318.4 (12.2) 147.7 (17.9) 21.2 (6.4)
	90%

Table 5 .

 5 5 -CVaR regret (average and std) for Exp. 5 at α = 1%, for T ∈ {10 3 , 5 × 10 3 , 10 4 } for 5000 replications.

	T	U-UCB	CVaR-UCB B-CVTS
	1000	49.1 (0.3)	53.2 (5.6)	18 (37)
	5000	245 (1.1)	263.2 (24.7) 35.5 (51)
	10000 489.1 (2.2) 518.4 (45.0) 41 (66)

Table 5 .

 5 6 -CVaR regret (average and std) forExp. 6, α = 5%, averaged over 400 random instances for T ∈ {10 4 , 2 × 10 4 , 4 × 10 4 } for 5000 replications.

Table 5 .

 5 7 -Empirical yield distribution metrics in kg/ha estimated after 10 6 samples in DSSAT environment

			Empirical distributions estimated after #1e+06 samples
						day: 057
		3.0e-04				day: 072 day: 087 day: 102	day (action) .	CVaR α
		2.5e-04				yield upper bound CVaR @ 20%	5% 20% 80%	100%
	density	1.5e-04 2.0e-04					057 072	0 46	448 627	2238 2570	3016 3273
		1.0e-04					087	287 1059 3074	3629
		5.0e-05					102	538 1515 3120	3586
		0.0e+00
		0	2000	4000	6000	8000	10000
				dry grain yield (kg/ha)
	Figure 5.2 -Empirical simulated yields and
	respective CVaRs at 20% estimated after 10 6
	samples in DSSAT environment.

Table 5 .

 5 8 -Empirical yield regrets at horizon 10 4 in t/ha in DSSAT environment, for 1040 replications. Standard deviations in parenthesis.

Table 5 .

 5

			Empirical distributions estimated after #1e+06 samples
		4.0e-04				day: 057 day: 072
						day: 087
		3.5e-04				day: 102 day: 117		day (action)	CVaRα
		3.0e-04				day: 132 day: 147		5%	20%	80%	100%
		2.5e-04				yield upper bound CVaR @ 20%	057	0	448	2238	3016
	density	2.0e-04						072 087	46 287	627 1059 3074 2570	3273 3629
		1.5e-04						102	538	1515 3120	3586
		1.0e-04						117	808	1832 3299	3716
								132	929	1955 3464	3850
		5.0e-05						147	1122 2203 3745	4112
		0.0e+00
		0	2000	4000	6000	8000	10000
				dry grain yield (kg/ha)			9 -7-armed distributions CVaRs for
	Figure 5.4 -Monte-Carlo estimate of the dis-	different levels of α
	tributions using 10 6 samples from DSSAT; 7-
	armed problem (Left) and 4-armed problem
	with over-estimated upper bound.

Table 5 .

 5 10 -Results for DSSAT 7-armed experiment, empirical regret at T = 10000 in t/ha for 1040 replications. Standard deviations in parenthesis.

Table 5 .

 5 11 -Results for DSSAT Empirical regret at T = 10000 in t/ha for 1040 replications for the 4-armed experiment with over-estimated upper bound.

Regret Analysis and Technical Results thanks

  . . , X n ) of the challenger, history Y = (Y 1 , . . . , Y N ) of the leader Two chances to win: with empirical mean and DS mean to the duel-based structure and a standard assumption on the concentration of the arms' means. We then introduce several properties of Dirichlet sampling, that theoretically guide proper tuning of the exploration bonus used in DS. We finally instantiate the algorithm for three different problems and provide regret bounds in these settings. Starting with the regret decomposition, we exhibit general conditions to ensure guarantees that are independent of the index used. Allowing a different family of distribution F k for each arm k, the first one concerns the concentration of the mean of each distribution. (Concentration of means). For all ν k ∈ F k , there exists a good rate function I k satisfying I k (x) > 0 for x ̸ = µ k and for all x > µ k , y < µ k , and any i.i.d sequence Y 1 , . . . , Y n drawn from ν k

	6.3 Assumption 6.3 P n 1 i=1 n Y
	5 end
	6 Otherwise, leader wins.
	Algorithm 6.1: Generic Dirichlet Sampling duel step

2 Draw a Dirichlet Sampling (DS) mean µ(X , Y) for the challenger. 3 if max{ Xn , µ(X , Y)} ≥ ȲN then 4 Challenger wins ; ▷ i ≥ x ≤ e -nI k (x) , and P 1 n n i=1

Dirichlet Sampling Beyond Bounded Rewards first

  step of the duel with empirical means. Writing C k ′

4 From optimality to robustness: three instances of DS

	log(1 + x) ≤ x we obtain P w∼D n+1 n+1 w =	5. Combining this property with (max j=1,...,n+1 Y j -µ) n i=1 (max j=1,...,n+1 Y j n
	i=1

i Y i ≥ µ ≥ P w∼D n+1 n i=1 w i min(Y i , µ) + w n+1 max j=1,...,n+1 Y j ≥ µ 6.

4

  Draw w = (w 1 , . . . , w nα+1 ) ∼ Dir((1, . . . , 1, n α , 1)) ; ▷ Parameter 1 except for w nα 5 return nα-1 i=1 w i Y i + w nα C α + w nα+1 B(Y, µ, ρ) Algorithm 6.4: Quantile Adaptive Dirichlet Sampling re-sampled mean

  summarizes this principle. random variables drawn from ν k . For a given Dirichlet sampled mean µ, assume that for any n ∈ N there exists a subset B k,n ⊂ R n satisfying1. There exists a strictly increasing function f k such that for any threshold µ < µ 1 ,

	Lemma 6.18. Assume that ν k satisfies Assumption 6.3, and denote by Y k n = (Y k 1 , . . . , Y k n ) a set
	of n

i.e for two distributions F , G and λ ∈ [0, 1] it holds that U (λF + (1 -λ)G) ≤ max{U (F ), U (G)}.

as can be seen in the analysis of SSMCChan (2020), treating the general case only requires some additional notation.

samples in our case). However, we notice that even TS suffers from this problem(xp 2G)

The code for obtaining the different figures reported in the chapter is available at https://github.com/ YRussac/LB-SDA.

https://github.com/rgautron/DssatBanditEnv

That is, for each bandit on F, for each arm k with ∆ k > 0, then E[N k (T )] = o(T α ) for all α ∈ (0, 1].

Defined on the upper tail, contrarily to Chapter 5 where we consider the lower tail

Decision Support System for Agrotechnology Transfer, https:// dssat.net/

Acknowledgements

Decision Support System for Agrotechnology Transfer is an open-source project maintained by the DSSAT Foundation

Sub-Sampling Dueling Algorithms

Analysis of LB and RB samplers

In this section we further analyze the Last Block (LB) and Random Block (RB) samplers. Both are Block Samplers, as defined in 2.3, but the way they provide diverse sub-samples is very different: while RB explores a random part of the history at each step, LB simply returns the last observations. Our striking result is that both methods manage to ensure sufficient diversity. Lemma 2.13 (Random Block Sampler). The Random Block sub-sampling algorithm ensures sufficient diversity, as defined in 2. [START_REF] Cassel | [END_REF].

In the following we sketch the main results that allow to prove Lemma 2.13. We introduce the random variable X m,H,j that denotes the maximal number of pairwise non-overlapping subsets obtained in m i.i.d. samples from RB(j, H). We aim at upper bounding

for some values of m, H, j, and some specific value of x that will be specified later. Intuitively, this probability is decreasing in H and m, and increasing in j. Furthermore, if m is large compared to H we can reasonably expect to visit most of the history. Then, the proof of Lemma 2.13 mainly relies on the results we propose in Lemma 2.14, that are written with m = H. This is sufficient in our proof since during a run of RB-SDA we expect both the history size of the leader and the number of duels played to be linear in r, so we can set the two values equal to their minimum using that X a,b,j ≥ X a,a,j if a ≤ b.

Lemma 2.14 (Maximal number of non-overlapping subsets with RB-SDA). Consider a history size H, a number of draws that is also H, a sub-sample size j, and some α ∈ (0, 1). Then, it holds that

Furthermore, if α is small enough it holds that log (P (X H,H,1 ≤ αH)) = O(H) .

Proof. The first part of Lemma 2.14 consists in showing that it is sufficient to study X H,H,1 to get a result for any sub-sample size j. This result comes from block sampling: we can focus on the number of unique elements drawn by RB (starting observation for each block). If M such unique elements are available, then in the worst case these observations are successive in the Chapter 3

LB-SDA with Limited-Memory

In the previous chapter we introduced the family of Sub-Sampling Dueling Algorithms, based on the principle of pairwise comparisons between arms with sub-samples of the same size. We proposed various sub-sampling schemes and derived theoretical guarantees along with general intuitions on why this principle works in bandits. Strikingly, we showed that a very simple algorithm returning the Last Block of observations achieve strong theoretical guarantees.

In this chapter we propose to extend this Last Block Sub-Sampling Dueling Algorithm (LB-SDA), in two directions. First, we prove that its guarantees hold when limiting the algorithm memory to a polylogarithmic function of the time horizon. Then, we consider non-stationary scenarios in which the arm distributions evolve over time. We propose a natural variant of the algorithm in which only the most recent observations are used for sub-sampling, achieving optimal regret guarantees under the assumption of a known number of abrupt changes. Numerical simulations highlight the merits of this approach, particularly when the changes are not only affecting the means of the rewards. To allow for fair comparison, we use for SW-LB-SDA, the same value of τ = 2 T log(T )/Γ T that is recommended for SW-UCB [START_REF] Garivier | On upper-confidence bound policies for switching bandit problems[END_REF]. D-UCB uses the discount factor suggested by [START_REF] Garivier | On upper-confidence bound policies for switching bandit problems[END_REF], 1/(1 -γ) = 4 T /Γ T . The changepoint detection algorithms need extra information such has the minimal gap for a breakpoint and the minimum length of a stationary phase. For M-UCB, we set w = 800 and b = w/2 log(2KT 2 ) as recommended by [START_REF] Cao | Nearly optimal adaptive procedure with change detection for piecewise-stationary bandit[END_REF] but set the amount of exploration to γ = KΓ T log(T )/T following [START_REF] Besson | Efficient change-point detection for tackling piecewise-stationary bandits[END_REF]. In practice, using this value rather than the theoretical suggestion from [START_REF] Cao | Nearly optimal adaptive procedure with change detection for piecewise-stationary bandit[END_REF] improved significantly the empirical performance of M-UCB for the horizon considered here. For CUSUM, α and h are tuned using suggestions from [START_REF] Liu | A change-detection based framework for piecewise-stationary multi-armed bandit problem[END_REF], namely α = Γ T /T log(T /Γ T ) and h = log(T /Γ T ). On this specific instance, using ε = 0.05 (to satisfy Assumption 2 of [START_REF] Liu | A change-detection based framework for piecewise-stationary multi-armed bandit problem[END_REF]) and M = 50 gives good performance. For the EXP3S algorithm, following (Auer et al., 2002a) the parameters α and γ are tuned as follows: α = 1/T and γ = min(1, K(e + Γ T log(KT )/((e -1)T ).

This problem is challenging because a policy that focuses on arm 1 to minimize the regret in the first stationary phase also has to explore sufficiently to detect that the second arm is the best in the second phase. SW-LB-SDA has performance comparable to the forgetting TS algorithms and is the best performing algorithm in this scenario. Note that both TS algorithms use the assumption that the arms are Bernoulli whereas SW-LB-SDA does not. SW-klUCB performs better than D-klUCB and its regret closely matches the one from the changepoint detection algorithms. By observing the lower and the upper quartiles, one sees that the performance of CUSUM vary much more than the other algorithms depending on its ability to detect the breakpoints. Finally, EXP3S, which can adapt to more general adversarial settings, lags behind the other algorithms in this abruptly changing stochastic environment.

In the third experiment with Γ T = 3 breakpoints, the K = 3 arms comes from Gaussian distributions with a fixed standard deviation of σ = 0.5 but time dependent means. The Chapter 4

Sub-sampling for Extreme Bandits

In Chapters 2 and 3 we proved that an algorithm based on a simple sub-sampling scheme, LB-SDA, performs very well for some bandit problems. In this chapter we consider a variant of bandits where the learner seeks to collect the largest possible reward, known as Extreme Bandits. This problem is difficult because the algorithm has to compare the heaviness of the distributions' tails, with potentially little prior information. In that case the non-parametric nature of LB-SDA is appealing, but requires to carefully choose a criterion to compare two tails. To that extent, we introduce Quantile of Maxima (QoMax) after studying properties of the maximum of i.i.d random variables under mild assumptions. We show that QoMax allows to build a simple Explore-Then-Commit (ETC) strategy, QoMax-ETC, achieving strong asymptotic guarantees despite its simplicity. We then propose and analyze a more adaptive algorithm, QoMax-SDA, performing pairwise comparison of QoMax estimates inside LB-SDA. Strikingly, QoMax-ETC and QoMax-SDA are more efficient than existing approaches under several aspects: (1) their non-parametric nature allows to derive strong theoretical guarantees under very mild assumptions on the tails, (2) in the experiments that we performed they lead to overall significantly better empirical performance, and (3) they enjoy a significant reduction of the memory and time complexities. This chapter if adapted from [START_REF] Baudry | Efficient algorithms for extreme bandits[END_REF].

Comparing Tails of Distributions with Quantiles of Maxima

another, that are instrumental to design our algorithms. In Section 4.3, we first propose an Explore-Then-Commit algorithm using QoMax, for which we establish vanishing regret in the strong sense under the mild assumption that the bandit model has a dominant arm. Albeit simple, this approach requires some tuning which depends on the horizon T . To overcome this limitation, we propose in Section 4.4 the QoMax-SDA algorithm which combines QoMax with the LB-SDA strategy. We prove that it achieves vanishing regret for arms with exponential or polynomial tails and also provide some elements of analysis under the weaker dominant arm assumption. In Section 6.6, we highlight the efficiency of our algorithms which allow for a significant reduction of the storage and computational cost while outperforming existing approaches empirically.

Comparing Tails of Distributions with Quantiles of Maxima

In this section, we motivate our new Quantile of Maxima (QoMax) estimator used for comparing the tails of two distributions based on n i.i.d. samples of each. We first present the assumptions under which we are able to exhibit some properties of QoMax.

The first way to judge the heaviness of a distribution is to evaluate what probability is allocated into extreme values. To do that, we introduce the survival function of a distribution. This notion is central in the assumptions we will make on the arms' distributions. In the following we will consider two different assumptions: a first one where tails have a parametric asymptotic equivalent, and a mild non-parametric assumption.

Definition 4.2 (Exponential or polynomial tails). Let ν be a distribution of survival function

These semi-parametric assumptions (which say nothing about the lower part of the distribution) have been introduced by [START_REF] Bhatt | Extreme bandits using robust statistics[END_REF]. We remark that a polynomial tail is a weaker condition than the second-order Pareto assumption from [START_REF] Carpentier | Extreme bandits[END_REF] that

Sub-sampling for Extreme Bandits

Add a batch < l a t e x i t s h a 1 _ b a s e 6 4 = " S + D

o 7 g i z K 1 z 3 G f P e j J b u + k t s / F X q z S s 2 f N C m + P N 3 J I G X P s + z p + g u V 2 t E T 7 e q e w f F a M u Y x 0 b 2 K J 5 7 m I f h 6 i j Y W f + g E c 8 O U 3 n 2 r l 1 7 j 6 k T q n w r O H L c u 7 f A a s 3 l e k = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W X X P t 4 R 0 6 0 t y x m + Z N B 6 4 z

o 7 g i z K 1 z 3 G f P e j J b u + k t s / F X q z S s 2 f N C m + P N 3 J I G X P s + z p + g u V 2 t E T 7 e q e w f F a M u Y x 0 b 2 K J 5 7 m I f h 6 i j Y W f + g E c 8 O U 3 n 2 r l 1 7 j 6 k T q n w r O H L c u 7 f A a s 3 l e k = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W X X P t 4 R 0 6 0 t y x m + Z N B 6 4 z

o 7 g i z K 1 z 3 G f P e j J b u + k t s / F X q z S s 2 f N C m + P N 3 J I G X P s + z p + g u V 2 t E T 7 e q e w f F a M u Y x 0 b 2 K J 5 7 m I f h 6 i j Y W f + g E c 8 O U 3 n 2 r l 1 7 j 6 k T q n w r O H L c u 7 f A a s 3 l e k = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W X X P t 4 R 0 6 0 t y x m + Z N B 6 4 z

o 7 g i z K 1 z 3 G f P e j J b u + k t s / F X q z S s 2 f N C m + P N 3 J I G X P s + z p + g u V 2 t E T 7 e q e w f F a M u Y x 0 b 2 K J 5 7 m I f h 6 i j Y W f + g E c 8 O U 3 n 2 r l 1 7 j 6 k T q n w r O H L c u 7 f A a s 3 l e k = < / l a t e x i t > The second point is furthermore interesting for exploration as it allows to play more duels for a fixed number of draws. We provide some results on the memory saved thanks to those tricks in Appendix 4.7.

Finally, we can note that a forced exploration, through the function f (independent on T ), is necessary under general assumptions as in all existing algorithms.

Extreme Regret Analysis

We now provide an analysis of QoMax-SDA under the same assumption as before:

denote the number of pulls of arm k at time t. We start with a generic regret decomposition.

Sketch of proof.

For parametric tails, we can compute the growth rate of E[max t≤T X 1,t ] with respect to T . This permits to tune the values of M T and x T to properly balance the terms in the regret decomposition. The difference in the convergence for (1) and (2) comes from the fact that the exploration cost scales logarithmically with the time horizon when using exponential tails, whereas the dependency is polynomial with polynomial tails.

We note that there is no hope to upper bound the last term in the regret decomposition of Proposition 4.10 assuming only that arm 1 dominates the others, so we could not establish vanishing regret for QoMax-SDA under this assumption. However, at least weakly vanishing regret could be established using the conjecture we make in Remark 4.17. Even if we were not able to prove this, we note that QoMax-SDA achieves state-of-the-art performance for exponential and polynomial tails, and that Lemma 4.18 provides a strong indicator of the good performance of QoMax-SDA under more general assumptions, as it shows that the algorithm queries each sub-optimal arm O((log T ) 1 γ ) times. With this number of queries the total number of data collected from sub-optimal arms would be O((log T )

Knowing if it is possible to design an extreme bandit algorithm that would use only O(log(T )) data for sub-optimal arms as in the standard setting under mild assumptions on the tails is an interesting open question. We now turn our attention to the practical benefits of using QoMax-based algorithms.

Practical performance

In all of our experiments, we compare QoMax-SDA and QoMax-ETC with ThresholdAscent (Streeter and Smith, 2006b), ExtremeHunter [START_REF] Carpentier | Extreme bandits[END_REF], ExtremeETC [START_REF] Achab | Max k-armed bandit: On the extremehunter algorithm and beyond[END_REF] and MaxMedian [START_REF] Bhatt | Extreme bandits using robust statistics[END_REF]. We use the parameters suggested in the original papers (see details in [START_REF] Baudry | Efficient algorithms for extreme bandits[END_REF]). Namely, b = 1 for ExtremeHunter/ETC, s = 100, δ = 0.1 for ThresholdAscent, ε t = (t + 1) -1 for MaxMedian. For QoMax-ETC, we use b T = (log T ) 2 batches of n T = log T samples. This matches the size of the exploration phase of ExtremeETC and allows for a fair comparison. For QoMax-SDA, we choose γ = 2/3, which seems to work well across all examples. All the results presented in this section are obtained with these values.

Sub-sampling for Extreme Bandits

Time and Memory Complexity

We summarize in Table 4.1 the storage and computational time required by the different adaptive and ETC algorithms that we consider, with the aforementioned parameters. The smallest values in each category are colored in blue. We do not include ThresholdAscent in the table because the comparison is unfair, as it uses a fixed number of data but is not theoretically grounded. We refer the reader to [START_REF] Bhatt | Extreme bandits using robust statistics[END_REF] for the complexities of the baselines, and we refer to Appendix D.2 of [START_REF] Baudry | Efficient algorithms for extreme bandits[END_REF] for details of all computation, and the expression of the complexities according to the algorithms' parameters.

For QoMax-ETC, the memory needed is Kb T as we only store the current maximum of each batch during the exploration phase. The time complexity is O(max(Kb T n T , b T log b T )) by comparing the duration of the experiment and the time needed to compute the quantiles before exploiting. We just assumed that finding a quantile of a list of size n costs Ω(log(n)), plugging the values of b T and n T gives the result. The time complexity of QoMax-SDA is in O(KT log T ) as its main cost consists in sorting data online, just like MaxMedian. The storage of QoMax-SDA is obtained thanks to the two tricks: one allows to keep O(log T ) batches, the other O(log T ) samples per batch for the leader. On the contrary, the complexity for the challengers remains in O(log T log log T ), therefore the dependency in K only appears as a second order term.

Algorithm

Memory Time

Extreme Hunter

QoMax-SDA offers an exponential reduction of the storage cost compared to ExtremeHunter and MaxMedian, while being as computationally efficient as MaxMedian. On the other hand, choosing the same length for the exploration phase of the two ETC leads to a significantly smaller time complexity for QoMax-ETC. Hence, both QoMax-SDA and QoMax-ETC present a substantial improvement over their counterparts.

Practical performance

Empirical Performance

We compare the empirical performance of the QoMax algorithms with the different competitors on synthetic data. We reproduced 6 experiments from previous works: all experiments from [START_REF] Bhatt | Extreme bandits using robust statistics[END_REF] (Experiments 1-4 for us), and the experiments 1 and 2 from [START_REF] Carpentier | Extreme bandits[END_REF] (5-6 in this work). We also implement new experiments with other families of distributions to highlight the generality of our approach. We choose to present in this section our methodology for evaluating Extreme Bandits algorithms, and the results for a selection of experiments, that illustrate well our findings across all the settings we tested. The results for the other experiments can be found in [START_REF] Baudry | Efficient algorithms for extreme bandits[END_REF].

Empirical evaluation

We consider 4 performance criteria: I Empirical evaluation of the extreme regret, by averaging the maxima collected on each trajectory.

II Fraction of pulls of the optimal arm.

III Empirical distribution of the number of pulls of optimal arm, to observe potential failures for some trajectories or how greedy an algorithm can be.

IV Empirical distribution of the maximal reward and in particular some quantiles, that will be more robust that the empirical expected maximum.

In our experiments each criterion is estimated over N = 10 4 independent trajectories for different values of the horizon T . Most works report only criteria (I), and (II) was first proposed by [START_REF] Bhatt | Extreme bandits using robust statistics[END_REF]. Our analysis shows that the extreme regret of a strategy is closely related to its capacity to sample the optimal arm T -o(T ) times, so we think that (II) is indeed a good performance indicator. Criterion (III) gives a broader picture of what can happen in the experiments, and in our results we display the following quantiles of the empirical distribution of best arm pulls: [1%, 10%, 25%, 50%, 75%, 90%, 99%]. In particular, the lower quantiles are interesting to see how much an algorithm can under-sample the best arm.

Regarding (I), we note that estimating the expectation E[max t≤T X It,t ] featured in the extreme regret is very hard: we highlight the fact that this expectation is taken on the distribution of the maximum over T samples, which is heavy-tailed in many cases. For that reason, standard Monte-Carlo estimators will have a very large variance. Hence, we propose the following estimation strategy when a tight approximation of E[Y + 1,T ] is known. We first find the value q + T such that E[X + 1,T ] is equal to the quantile of order q + T of the distribution of Y + 1,T . We then compute the empirical quantile of order q + T of the set of maxima collected in each trajectory, denoted by We recall ξ T := {N 1 (T ) ≤ T -KM T }. First, using K k=1 N k (T ) = T , we remark that

We denote by r T the index of the round for which the number of observations equals or exceeds T . As at least one observation is collected at the end of the round it holds that r T ≤ T . Hence, we can obtain

Using b k (T ) = n k (T ) γ and Markov inequality gives

For all k ≥ 2, Lemma 4.21 shows that E[n k (T )] = O (log T )

with the tuning we choose for the algorithm. This is sufficient to conclude on the result. Lemma 4.21. Under assumption 4.11 and if ν 1 ≻ ν k for all k ∈ {1, . . . , K}, if we consider QoMax-SDA with parameters B(n) = n γ and f (r) = (log r) 1 γ , then for all k ≥ 2 there exists a constant C k such that the number of pulls of arm k at time T satisfies

We omit the proof of this result in this manuscript, as it is redundant with the proofs of Chapters 2 and 3. The interested reader can find the complete version in [START_REF] Baudry | Efficient algorithms for extreme bandits[END_REF]. The main ingredient is to replace the deviation inequalities of Assumption 2.4 by the ones we obtained for the QoMax estimates in Theorem 4.7. Furthermore, in this setting the forced exploration of f (r) = (log r) 1 γ queries (and hence log(r) batches due to the rule we set for the number of batches) ensures enough forced exploration so that we do not need to consider balance conditions, which provides the values we set for f (r) and b(r).

Sub-sampling for Extreme Bandits

data stored, and the maximum we observe on all 4 experiments is actually 23 which is very small compared to N = 5 × 10 4 .

Therefore, we conclude that the trick we introduced is indeed efficient and our experiments corroborate the intuition that it allows to store O(log N ) data out of N on average. We now prove it formally in Lemma 4.23 4.6. For any ν, it holds that

Proof. Denote the sorted random samples by

As the observations are i.i.d, all of them are equally likely to be in the last position. We consider I N the random variable denoting the index of the last observation, it holds that

Part II

Dirichlet Sampling Strategies for Bounded Rewards and Beyond

Chapter 5

Non-Parametric Thompson Sampling for CVaR bandits

In this chapter we consider the CVaR bandit problems introduced in Chapter 1 (Section 1.2). While existing works in this setting mainly focus on Upper Confidence Bound algorithms, we present a new Thompson Sampling approach for CVaR bandits on bounded rewards that is flexible enough to solve a variety of problems with finite rewards. Building on a recent work by [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF], we introduce B-CVTS for continuous bounded rewards and M-CVTS for multinomial distributions. On the theoretical side, we provide a non-trivial extension of their analysis that enables to upper bound their CVaR regret. Strikingly, our results show that these strategies are the first to provably achieve asymptotic optimality in CVaR bandits, matching the corresponding asymptotic lower bounds for this setting. We furthermore illustrate empirically the benefits of Thompson Sampling approaches, both in an experiment in agriculture using the DSSAT simulator and on various synthetic examples. The results presented in this Chapter were published in (Baudry et al., 2021a), and in Section 5.5 we present theoretical results that are not in the paper but are of interest for the practitioner. [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF] to the CVaR criterion. This result highlights the right complexity term that should appear when deriving regret upper bounds.

Non-Parametric Thompson Sampling for CVaR bandits

On the algorithms side, two powerful variants of Thompson Sampling were introduced recently by [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF] for the mean criterion, that enable to overcome the "parametric" limitation, in the sense that these approaches reach the minimal achievable regret given by the lower bound of [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF] respectively for discrete and bounded distributions: Multinomial Thompson Sampling (MTS) and Non- Parametric Thompson Sampling (NPTS). This timely contribution opens the room for a generalization to CVaR bandits, that we introduce in Section 5.2 with B-CVTS for bounded supports, and M-CVTS for multinomial arms. We then show that the two algorithms are asymptotically optimal (Theorem 5.4), and sketch the proof of the result for B-CVTS. Up to our knowledge, these are the first results showing asymptotic optimality of a CVaR regret minimization strategy. As expected, adapting the regret analysis from [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF]) is non-trivial; we highlight the main challenges of this adaption in Section 5. 4. For instance, one of the key challenge was to handle boundary crossing probability for the CVaR, and another difficulty comes in the analysis of the non-parametric B-CVTS due to regularity properties of the Kullback-Leibler projection. In Section 5.6, we provide empirical results on a simplified version of the case-study in agriculture introduced in the foreword of this thesis, using the well-established DSSAT crop-yield simulator [START_REF] Hoogenboom | The dssat crop modeling ecosystem[END_REF]. These simulations highlight the benefits of using strategies based on Thompson Sampling in this CVaR bandit setting against state-of-the-art baselines: We compare to U-UCB and CVaR-UCB as they showcase two fundamentally different approaches to build a UCB strategy for a non-linear utility function. The first one is closely related to UCB, the second one exploits properties of the underlying CDF, which may generalize to different risk metrics. As claimed in [START_REF] Tamkin | Distributionally-aware exploration for cvar bandits[END_REF], our experiments confirm that CVaR-UCB generally performs better than U-UCB. However, both TS strategies outperform UCB algorithms that tend to suffer from non-optimized confidence bounds. We complete this study with additional experiments on synthetic data that also confirm the benefits of TS.

Non-Parametric Thompson Sampling for CVaR Bandits

We present two novel algorithms based on Thompson Sampling and targeting the lower bound of Theorem 5.3 on the CVaR-regret, for any specified value of α ∈ (0, 1]. These algorithms are inspired by the first algorithms based on Thompson Sampling matching the Burnetas and Katehakis lower bound for bounded distributions in the expectation setting, recently proposed by [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF].

Non-Parametric Thompson Sampling for CVaR bandits

B-CVTS then selects the arm A t = argmax k c k,t . For α = 1, this algorithm coincides with the Non Parametric Thompson Sampling of [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF] (NPTS). NPTS can be seen as an algorithm that computes for each arm a re-weighted average of the past observations. Our extension to CVAR bandits required to interpret this operation as the computation of the expectation of a random perturbation of the empirical distribution, which can be replaced by the computation of the CVaR of this perturbed distribution. Note that this idea generalizes beyond using the CVaR, that can be replaced with any criterion. The resulting strategy differs from M-CVTS due to the initialization step using the knowledge of the support in M-CVTS. As we'll show, both actually have the same theoretical guarantees when distributions are multinomial.

Asymptotic optimality of the CVTS algorithms

In this section we prove, after defining this notion, that M-CVTS and B-CVTS are asymptotically optimal in terms of the CVaR regret defined in Equation ( 5.2) for the distributions they cover. [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF] first gave an asymptotic lower bound on the regret for parameteric distribution, that was later extended by [START_REF] Burnetas | Optimal adaptive policies for sequential allocation problems[END_REF] to more general classes of distributions. We present below an intuitive generalization of this result for CVaR bandits, first introducing the function that measures the complexity of a CVaR bandit problem.

Asymptotic Optimality in CVaR bandits

Non-Parametric Thompson Sampling for CVaR bandits

It follows from Pinsker's inequality that KL(ν

Therefore, in a bandit model in which all ν k are supported in [0, 1] (that is, all D k are equal to P([0, 1]), the set of probability measures on [0, 1]), it follows that

Combining this inequality together with the lower bound of Theorem 5.3, we obtain that the regret of an algorithm matching the lower bound is upper bounded by

, which is precisely the scaling of the CVaR regret bounds obtained for the U-UCB [START_REF] Cassel | A general approach to multi-armed bandits under risk criteria[END_REF] and CVaR-UCB [START_REF] Tamkin | Distributionally-aware exploration for cvar bandits[END_REF]. Assuming the above inequalities are tight for some distributions (which may not be the case), one may qualify these algorithms as "order-optimal", as their CVaR regret makes appear the good scaling in the gaps and in α, just like the UCB1 algorithm (Auer et al., 2002a) for α = 1. In this chapter we go beyond order-optimality, and we strive to design algorithms that are asymptotically optimal.

Regret Guarantees for M-CVTS and B-CVTS

In the next section we prove that both M-CVTS and B-CVTS match the lower bound of Theorem 5.3 for the families of distributions they respectively tackle, assuming that they are given the support of each distribution. Hence, under these hypotheses, the two algorithms are asymptotically optimal. Despite the recent development in CVaR bandits literature, to our knowledge no algorithm has been proved to match this lower bound yet. (I) Assume that ∀k, ν k is multinomial with known support

(II) Assume that ∀k ∈ {1, . . . , K}, ν k belongs the family

Non-Parametric Thompson Sampling for CVaR bandits

Now, we can first solve the problem inf q∈P M y,α,c KL(p, q) for a fixed value of y, satisfying y > c (else the feasible set is empty). We write the Lagrangian of this problem:

and want to solve max λ 1 >0,λ 2 >0 min q H(q, λ 1 , λ 2 ). To this end, we write

Setting the derivative to 0 yields

We can check that the inequality constraint is achieved. Moreover, exploiting the two constraints leads to λ 1 + λ 2 α(y -c) = 1. This finally gives

Note that this solution is only valid if λ 2 ≤ 1 α(y-c) . We have two possibilities: 1) the maximum is achieved in [0, 1 α(y-c) ), in this case we have

The other possibility is that the function is still increasing in λ 2 = 1 α(y-c) . For this case, we check the sign of

, that is the same as the one of (y-c)α 1 -E F (y-c)α (y-X) + . We see that the function can only be increasing if E F (y-c)α (y-X) + < 1, and the solution is then

(y-c)α . This concludes the proof.

This result matches the one of [START_REF] Honda | An Asymptotically Optimal Bandit Algorithm for Bounded Support Models[END_REF] for α = 1 and y = 1, and is similar to the one obtained by Agrawal et al. (2021b)[Theorem 6] for a more complex set of distributions (which is hence less explicit). Furthermore, Agrawal et al. (2021b)[ Lemma 4] prove the continuity of K α,Y inf under this condition, which is required in several part of our proofs.

Further upper bounds on Pre-CV and Post-CV

Upper Bound on the BCP Building on the dual form of the K α,D inf for discrete distributions we just introduced, we can derive an upper bound of the BCP using similar techniques as [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF]. We consider a known support Y = (y 1 , . . . , y n ) and the Dirichlet distribution D n defined in Section 5.2. We further denote by F Y the uniform distribution on Y (or empirical distribution of Y), and C α (Y) its CVaR.

where Ȳ = max{y 1 , . . . , y n }. Furthermore, the multiplicative factor n can be removed if α = 1.

Proof. We first use the formulation of the CVaR of Equation ( 5.3) for discrete distributions, and a union bound to obtain

We see that the multiplicative n comes from the union bound on the possible values of y for the quantile of order α. When α = 1 it is always equal to the maximum value in Y, hence the union bound is not necessary.

We then handle P α(y -c) -n i=1 w i (y -y i ) + ≥ 0 for a fixed value of y. We follow the path of [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF], using that a Dirichlet random variable w = (w 1 , . . . , w n ) can be written in terms of n independent random variables R 1 , . . . , R n following an exponential distribution, as

. Using this property and multiplying by n j=1 R j we obtain

where we used Markov's inequality for some t ∈ 0, 1 (y-c)α . We then conclude by deriving the MGF of the exponential variables,

We then put the sup on y inside the exponent, and recognize the dual form of K α,Y inf , which concludes the proof.

Lower Bounds on the BCP We now establish two lower bounds on the BCP. The first one concerns the case where we only want the CVaR of the noisy distribution to exceed the CVaR of the empirical distribution generated by the dataset Y. Lemma 5.9. Assume that Y = (y 1 , . . . , y n ) and y 1 < • • • < y n , then y ⌈nα⌉ is the empirical α quantile of the set and y 1 its minimum, and it holds that

Proof. We assume that Y is known and ordered, i.e y 1 ≤ y 2 ≤ • • • ≤ y n . We then write

Thanks to the definition of the CVaR provided by Equation ( 5.1) it holds that

First, if we know y 1 , . . . , y n then the second term is deterministic and the sup is actually achieved in y ⌈nα⌉ . Secondly, the inequality is true if at least one term in the left element satisfies it, so we can write

Further upper bounds on Pre-CV and Post-CV

As the variable

belongs to [0, 1] we can apply the lemma 17 of Riou & Honda and get

We conclude by omitting all the terms except (y ⌈nα⌉ -y 1 ) in the sum.

We finally study the case when the size of the support is |Y| = M , for some known M ∈ N and when the considered distributions are the frequencies of each observation in Y out of n ∈ N many observations, which we represent by the set Q M n = (β, p) ∈ N * n × P M : p = β n . Our last result is a lower bound on the BCP for distributions in Q M n .

Lemma 5.10 (Lower Bound on the BCP for multinomial distributions). For any (M, n) ∈ N 2 and (β, p) ∈ Q M n , let p ⋆ ∈ P M be any vector satisfying C α (X , p ⋆ ) ≥ c. It holds that

,

where

, and P Mult(n,q) (β) denotes the probability that a vector drawn from a multinomial distribution with n trials and probability q is equal to β.

Proof. We follow the sketch of the proof of Lemma 14 of Riou and Honda (2020) using Equation (5.3). We start by stating that there exists some p * such that K α,Y inf β n , c = KL β n , p * . The existence of p * is ensured by the fact that the function K α,Y inf is the solution of the minimization Non-Parametric Thompson Sampling for CVaR bandits

For that reason, for any ε, with well chosen (ε 1 , ε 2 ) we can write

Upper bounding (Pre-CV)

We recall that the (Pre-CV) term is expressed as

.

Inspired by [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF] we split this expectation into different regions depending of the value of the CVaR of the empirical distribution (including the term y 0 = B added at the beginning of the history of observations).

We split the upper bound on A into three terms

We now upper bound each of these three terms, for any value of ε 1 .

Upper bounding A 1 We do not detail this part because this region corresponds to the most favorable case, as the CVaR of arm 1 is reasonably estimated. Hence, the BCP admits an exponential upper bound that can be derived with similar techniques as for Lemma 5.8, and so

where Y is Y n but with each item only repeated once (the set built from Y n ). The compacity of the set of distributions supported in [0, B] ensures that δ > 0, and we finally obtain

Hence, A 3 is also upper bounded by a constant. This final result concludes the proof of optimality of B-CVTS for continuous bounded distribution, as we finally have

Additional theoretical results of practical interest

In this section we provide two results that are not presented in the initial paper on CVTS (Baudry et al., 2021a) but can be of interest for the practitioner. The first one is that the theoretical guarantees are preserved if CVTS runs in a batch setting where the number of participants in each batch is upper bounded by some constant, while the second shows that if the learner cannot provide an upper bound on the support setting B = +∞ still leads to logarithmic regret guarantees when α < 1.

Optimality in the batch setting

With the application of agriculture in mind, we consider running B-CVTS on T time steps that we call seasons and assume that at the end season t a number of arms n t ∈ [M ] will be drawn, where M ∈ N is an unknown upper bound on the possible number of draws for each season.

More precisely, for each season t ∈ [T ] and for each integer f ∈ [n t ] the B-CVTS algorithm chooses an action A t,f to play, where the choice of A t,f can depend only on the observations made up to (and including) season t -1. We call this algorithm Batched-B-CVTS.

With the following result we quantify the impact of the batched feedback on the CVaR regret of Batched-B-CVTS, and show that if the upper bound M is independent of the time horizon the algorithm remains asymptotically optimal, as in the purely sequential setting.

Additional theoretical results of practical interest

Proof. The analysis again relies on the upper and lower bounds we can obtain for the BCP if B = +∞. First, the pre-convergence term is trivially smaller than with a finite B, and does not need to be further examined here. We now have a look at the upper bound of the BCP, with a quite straightforward proof. Denoting by Y B the history augmented by B (Y ∞ for B = +∞) we first distinguish the case whether B is actually "used" in the noisy CVaR or not,

Now we can use that the marginal distribution of w n+1 is a beta distribution β (1, n), and that under the events considered in the second term the +∞ item is not considered in the computation of the CVaR, hence it could in fact be replaced by anything larger than the maximum value collected so far. For this reason, for bounded distributions we can simply write

where the second line comes from the fact that if w n+1 < 1 -α the last term is not used in the CVaR and could be anything larger than the other items, and the second term on the last line comes from the upper bound we provided in previous works. F B is the empirical distribution of the history augmented by B. Finishing the proofs requires only to use this

Experiments

First set of experiments We first consider experiments with two modes, each mode being equi-probable and having the same variance for simplicity (σ = 0.1) in all experiments. It is for example interesting to set up experiments with arms whose modes are relatively close, and other arms that have a large mass of probability close to the two support bounds (e.g one mode close to 1 and one close to 0).

We experiment 4 possible configurations of the modes, denoted by parameters (µ i ) i∈{1,...,4} : µ 1 = (0.2, 0.5) (arm 1), µ 2 = (0, 1) (arm 2), µ 3 = (0.3, 0.6) (arm 3), µ 4 = (0.1, 0.65) (arm 4).

Before detailing the experiments we can look at the CVaRs of these distributions for different values of parameter α. For example, arm 2 has a larger mean than the one with arm 1, but the 50% CVaR of 1 is larger. We represented the CVaR for each parameter for different values of α ∈ (0, 1] in Figure 5.1, with the thresholds α ∈ {0%, 10%, 90%} (used in our experiments) represented by the vertical lines. Interestingly, with these arms the most difficult problems are not necessarily those with smallest values of α. Indeed, for α = 80% it may be particularly difficult to choose between arm 2 and 3, or between arm 1 and 4, while arm 3 is the clear winner for α = 10% due to the distribution being very concentrated around 0.5. Furthermore, the arm 2 provides observations mostly around 0 and 1 but has a larger mean than the others, so it becomes the best arm for values of α that are close to 1. We run the algorithms for α = 10%, 50% and 90% on four bandit problems with the following characteristics : arm 1 and 2 (Exp. 1), arm 1 and 3 (Exp. 2), arm 1 and 4 (Exp. 3), and all arms from 1 to 4 (Exp. 4). In Tables 5.1, 5.2, 5.3 and 5.4 we report the results for the four considered problems (mean regret and standard deviation at T = 10000). On these examples B-CVTS significantly outperforms the two UCB algorithms for all levels of α.

Appendix A: Basic properties of the Dirichlet distribution

Appendix A: Basic properties of the Dirichlet distribution

We consider the Dirichlet distribution Dir(α) for some parameter α = (α 1 , . . . , α n ). Let w = (w 1 , . . . , w n ) be a random variable drawn from the distribution Dir(α). We first recall that w takes its values in the probability simplex P n = {p ∈ [0, 1] n : n i=1 p i = 1}. The distribution admits the following density,

where Γ denotes the Gamma function. In this manuscript we only consider integer values for the coefficient (α i ) i∈N , and for any m ∈ N Γ(m) = (m -1)!.

This distribution has convenient properties. First, using the notation n i=1 α i = N and interpreting α i /N as the frequency of an item in a set of observations drawn from a finite collection (empirical distribution), and w a random re-weighting of these observation providing a "noisy" empirical distribution, the Dirichlet distribution ensures that the new frequency of each item is unbiased with respect to the observed frequency, with a variance that is inversely proportional to the total number of items collected. For any i ∈ [1, n],

and the marginal density of each component of w is actually a distribution Beta(α i , N -α i ).

This explains the use of the Dirichlet distribution to generalize the Beta-Bernoulli Thompson Sampling.

In this manuscript we also use two main properties of the Dirichlet distribution, both using the relation between the Dirichlet distribution and the Exponential distribution. Let R 1 , . . . , R n be n i.i.d random variables drawn from exponential distributions with respective parameters α i , R i ∼ E(α i ). Then the vector w = (w 1 , . . . , w

The second property is a consequence of the first one, and is that the components of a random variable drawn from a Dirichlet distribution can be aggregated, providing another Dirichlet distribution: if w ∼ Dir(α), then w ′ = (w 1 , . . . , w i +w j , . . . , w n ) ∼ Dir((α 1 , . . . , α i +α j , . . . , α n )) (putting the sum in the i-th slot and removing the j-th slot without changing the other indices).

Chapter 6

Dirichlet Sampling Beyond Bounded Rewards

In Chapter 5 we showed that NPTS can be successfully of extended to CVaR bandits when distributions are bounded. However, the existence and knowledge of the upper bound may sometimes not be precisely accessible to the practitioner, raising the question of the robustness of bandit algorithms to model misspecification. In this chapter we extend this strategy for alternative assumptions on the distributions. We study a generic Dirichlet Sampling (DS) algorithm, based on pairwise comparisons of empirical indices computed with re-sampling of the arms' observations and a data-dependent exploration bonus. We propose variants of this strategy achieving respectively optimal regret when the distributions are bounded and logarithmic regret for semi-bounded distributions with a mild quantile condition. Furthermore, a simple tuning can lead to consistent guarantees inside a large class of unbounded distributions, at the cost of slightly larger than logarithmic regret. We finally provide numerical experiments further showing the merits of DS in the decision-making problem on synthetic agriculture data introduced in Chapter 5. The results we present were published in (Baudry et al., 2021c). Dirichlet Sampling Beyond Bounded Rewards general assumptions, and the technical results that allow to fine-tune the algorithm for different families (see Section 6.3.1). Then, we detail three instances of DS algorithms and their regret guarantees in Section 6.4: Bounded Dirichlet Sampling (BDS) tackles bounded distributions with possibly unknown upper bounds, Quantile Dirichlet Sampling proposes a first generalization to the unbounded case using truncated distributions. Last, Robust Dirichlet Sampling (RDS) has a slightly larger than logarithmic regret for any unspecified light-tailed unbounded distributions, making it a competitor to the Robust-UCB algorithm of [START_REF] Ashutosh | Bandit algorithms: Letting go of logarithmic regret for statistical robustness[END_REF]. Finally, we consider in Section 6.6 the use-case in agriculture introduced in Chapter 5 using the DSSAT simulator (see [START_REF] Hoogenboom | The dssat crop modeling ecosystem[END_REF]), which naturally faces all the questions (robustness, model specification) that motivate this work and shows the merit of DS over state-of-the-art methods for this problem.

Dirichlet Sampling Algorithms

In this section we introduce Dirichlet Sampling, a strategy extending the Non-Parametric Thompson Sampling algorithm of [START_REF] Riou | Bandit algorithms based on thompson sampling for bounded reward distributions[END_REF] outside the scope of bounded distributions with a known support upper bound. For this purpose, we build an adaptive strategy in a duel-based framework, already used in the sub-sampling based algorithms we introduced in Part I. We motivate this choice in the following.

Background

We introduced NPTS in Chapter 1 (see Algorithm 1.6), and recall that it is identical to the B-CVTS algorithm introduced in the previous chapter when the risk parameter considered is α = 1. The simplicity and strong theoretical guarantees of this algorithm are appealing for further generalization. As we fully depart from the Bayesian approach, considering alternative exploration bonuses, we derive a new family of algorithms under the name of Dirichlet Sampling. We keep the two principles of re-weighting the observations using a Dirichlet distribution and helping exploration by adding a bonus to the collected data, and explore how to apply them to more general (e.g unbounded) distributions. In particular, we allow in DS some pre-processing of the observations before re-weighting (see section 6.3.1 and 6.4) and motivate in Section 6.3.1 the use of a data-dependent bonus, that use information from several arms. The complexity introduced by such bonus in the analysis requires a change of algorithm structure, dropping the index policy for a leader vs challenger approach (Chan, 2020).

Round-based algorithm

We define a round as a step of the algorithm at the end of which a set of (possibly several) arms are selected to be pulled. Let A r ⊂ {1, . . . , K} be the subset of the arms pulled at the beginning of a round r, as in Chapters 2-3 we call T -round regret the

Dirichlet Sampling Beyond Bounded Rewards

In Dirichlet Sampling, these duels take place inside a round-based framework that we summarize in Algorithm 6.2. We write it for a generic "Dirichlet Sampled mean" µ that must be computed by a re-weighting of the observations augmented by an exploration bonus. As in NPTS, the weights are drawn with a Dirichlet distribution. For instance, we propose a standard way to define a Dirichlet Sampling mean with a data-dependent (instead of constant) bonus B(X , Y). Example 6.2. Consider a bonus B(X , Y) and weights (w 1 , . . . , w n+1 ) ∼ D n+1 , the following expression is a possible re-sampled mean with Dirichlet Sampling,

However, the algorithm structure in Algorithm 6.2 could be combined with any randomized index, which is of independent interest as we will see in Section 6. 3. In the next section we study the theoretical properties of Dirichlet Sampling, and discuss the choice of the index µ for different families of distributions.

Regret Analysis and Technical Results

In this section, we analyze the regret of DS algorithms. We first derive a general regret decomposition for the generic round-based strategy we described in Algorithm 6.2, that holds only

Dirichlet Sampling Beyond Bounded Rewards

The LHS represents the expected cost in terms of regret of underestimating the optimal arm; intuitively, it measures the expected number of lost duels before finally winning one when starting with low rewards. This is a classic decomposition in bandit analysis, and a counterpart of Assumption 6.6 holds for most index policies with provable regret guarantees, e.g Theorem 1 in Kveton et al. (2019b) (GIRO) or Lemma 4 in Agrawal and Goyal (2012a)) (Bernoulli Thompson Sampling). We find it noteworthy that this regret decomposition depends only on the distribution of the best arm and its randomized Dirichlet Sampling index when it is a challenger. Corollary 6.7 (Conditions for controlled regret). If Assumptions 6.3 and 6.6 hold for the DS index on the families of distribution (F k ) k∈{1,...,K} , the regret of the DS algorithm satisfies

Up to this point this result is quite abstract, but this standardized analysis allows us to instantiate the Dirichlet Sampling algorithm on different class of problems and calibrate it in order to ensure that Assumption 6.6 holds and to make n k (T ) explicit. In particular if n k (T ) = O(log T ), we recover the logarithmic regret. In the next section, we present technical results to justify calibrations of the DS index for several kind of families.

Technical tools: boundary crossing probability of a DS index

In this section, we highlight some key properties of a sum of random variables re-weighted by a Dirichlet weight vector that help us suggest a sound tuning of the bonus B(Y, Y) for different kind of families. We then detail such tuning. We first recall the definition of Boundary Crossing Probabilities (BCP), that we already used in Chapter 5.

Definition 6.8 (Boundary Crossing Probability (BCP)). Consider a set of n+1 observation points

Then, for any µ ∈ R, a "Boundary Crossing Probability" (BCP) conditionally on Y is defined as

where we recall that D n+1 is the Dirichlet distribution with parameter (1, . . . , 1) of size n+1, i.e the uniform distribution on the (n + 1)-simplex. We emphasize that here Y is considered fixed, and the only source of randomness comes from the weights w.

Dirichlet Sampling Beyond Bounded Rewards

where for any η ∈ (0, 1], ε 0 > 0, n η,ε 0 k (T ) is the sequence satisfying

In particular, if

The sequence M k,n is a high probability upper bound of the maximum of n observations from F k , that we discuss in next section. For light-tailed distributions, it holds that M k,n = O(log n) (using Jensen inequality as in the proof of Theorem 2.5 in [START_REF] Boucheron | Concentration inequalities : a non asymptotic theory of independence[END_REF]). Hence, choosing ρ n = O(log n) we can further obtain the simpler upper bound in O(log(T ) log log(T )). This slightly larger-than-logarithmic rate is a consequence of Lemma 6.11. In our opinion this is a small cost compared to the generality of the guarantees of RDS. We call the algorithm robust because these theoretical guarantees are obtained on the broad class of light-tailed distributions, without any additional assumption. We recommend the leverage function ρ n = O( log(1+n)), which corresponds to the growth rate of the maximum of sub-Gaussian samples and is empirically validated (see Section 6.6). We emphasize that RDS thus avoids all hyperparameter tuning, a desirable feature for the practitioner with little information on the problem at hand. Furthermore, in the next section we show that this algorithm performs very well in practice despite its non-logarithmic asymptotic guarantees.

Proof Sketch

In this part we provide a general overview of the proofs of Theorem 6.13, 6.15 and 6.17. The full proofs can be found in Appendix D of (Baudry et al., 2021c). Here, we try to highlight the key ingredients that lead to the results provided in the previous section.

We recall that the three family of distributions are light-tailed and hence satisfy assumption 6.3. So, starting from Corollary 6.7 our objective is to derive the dominant term n k (T ) and to prove that Assumption 6.6 is satisfied in the three examples we consider.

Deriving the dominant term: characterizing n k (T )

In this section we consider an arm k ∈ {2, . . . , K}, of distribution ν k . We first derive a general proof sketch that will be used for each algorithm. The idea is simple, and inspired by the proof

Dirichlet Sampling Beyond Bounded Rewards

and the second term (empirical mean of the leader over-estimated) can be upper bounded by C Ḡ,2 := T -1 r=1 re -⌈r/K⌉I 1 (µ 1 -ε 1 ) = O(1), using the concentration of the leader and a union bound on its sample size.

Combining these results, we obtain the following bound on n k (T ) for arm k as

If the assumptions of the lemma are satisfied and m k (T ) is chosen as suggested the result is proved.

The objective is then to find the proper confidence ball B k,n and associated rate function f k under the assumptions of the three theorems considered.

BDS

As the distributions are bounded B k,n can be chosen as a Levy ball, as in Chapter 5. Furthermore, Lemma 5.8 provides

where B BDS denotes the exploration bonus of BDS and F k,n is the empirical distribution associated to the data points and the bonus (counting as one observation). Setting m k (T ) as the dominant term of the theorem and using the continuity of the K inf for bounded distributions (in the upper bound too) provides the result.

QDS

We form B k,n with a Levy ball around the true distribution, that we augment with the concentration of the bonus and CVaR,

for some ε > 0, ε 1 , ε 2 > 0, denoting by ν Y the empirical distribution associated with a set Y,

If X ∼ ν k , (µ -X) + is also a light-tailed variable, so the corresponding term in the upper bound of n k (T ) is an additive constant. For the CVaR, we rely on the concentration of Wasserstein metrics for light-tailed distribution, using that (Lemma 2 Dirichlet Sampling Beyond Bounded Rewards 6.6 Experiments: crop-farming and synthetic problems 6.6.1 Application in agriculture: bandits with DSSAT We benchmark our algorithms using the DSSAT crop yield simulator 3 [START_REF] Hoogenboom | The dssat crop modeling ecosystem[END_REF], to emulate a simplified version of the decision problem in agriculture we introduced in the preamble of this manuscript. We consider the same use-case as in Section 5.6.2 of Chapter 5.

More specifically, we model the problem of selecting a planting date for maize grains among 7 possible options, everything else being equal, as a 7-armed bandit. We recall that the resulting distributions incorporate historical variability as well as exogenous randomness coming from a stochastic meteorologic model. We illustrate this in Figure 6.1 with their histogram computed on 10 6 samples. They are typically right-skewed, multimodal and exhibit a peak at zero corresponding to years of poor harvest, hence they hardly fit to a convenient parametric model (e.g SPEF/sub-Gaussian. . . ). This problem is the same as the one studied in the previous chapter, but this time we consider standard regret minimization instead of CVaR bandits. Benchmarks A natural choice for the learner would be to use algorithms adapted for bounded distributions with known support. Indeed, one could argue that crop yields are fundamentally bounded by a very large value, that can be provided with some expert knowledge. However this method may have limits when the upper bound cannot be estimated accurately (few data, new environment, . . . ), as a conservative bound can have a cost on the regret. For this reason,

Dirichlet Sampling Beyond Bounded Rewards

Robustness for light-tailed bandits: comparison with R-UCB-LT The study of statistically robust bandit algorithms is fairly recent, and as such is yet to have well-established benchmarks. [START_REF] Ashutosh | Bandit algorithms: Letting go of logarithmic regret for statistical robustness[END_REF] introduce R-UCB-LT, an adaptation of the standard sub-Gaussian UCB to enforce robustness w.r.t light-tailed distribution. We reproduce the setting of their experiment, namely two Gaussian arms N (1, 1) and N (2, 3), and compare several variants of both R-UCB-LT and RDS against a misspecified UCB1 (the misspecification takes the form of an overly optimistic 1-sub-Gaussian assumption, while the second arm is only √ 3-sub-Gaussian). Both R-UCB-LT and RDS rely on a slowly growing exploration bonus, denoted respectively by f and ρ; we run both algorithms with f and ρ equal to log 2 , log and √ log.

Results are reported in Figure 6. 5. As expected, the misspecified UCB1 exhibits much faster regret growth than the robust algorithms. However, RDS seems to outperform R-UCB-LT, the best average regret being achieved by RDS with ρ n = log(1+n) and ρ n = log(1 + n). Furthermore, the regret of RDS appears to be somewhat monotonic (slightly increasing) with respect to the hyperparameter ρ, and the best results are achieved by the one matching the asymptotic growth rate of the maximum of a i.i.d Gaussian samples, as recommended by Theorem 6.17. On the other hand, the best version of R-UCB-LT is obtained with f ≈ log (for which we do not find a theoretical intuition) and the performance gap is significant when other bonuses are considered. We also tested R-UCB-LT with powers of log log with similar results; we do not report these curves for the readability of the figures. In light of these results, RDS seems less sensitive to its parameter choice than R-UCB-LT. Gaussian Mixture Many real-world situations (loss profile of a portfolio of financial assets, crop yields, statistics of heterogeneous populations...) exhibit multimodal distributions. The Gaussian mixture model is perhaps the simplest example of such distributions and is ubiquitous in many areas of machine learning and engineering (speech recognition, clustering...), in particular as a nonparametric model for kernel density estimation. Still, to the best of our knowledge, it escapes the scope of current optimal bandit methods as it is neither bounded

Conclusion and Perspectives

The algorithms that we proposed are inspired by BESA [START_REF] Baransi | Sub-sampling for multi-armed bandits[END_REF] and SSMC (Chan, 2020), bridging the gap between the two approaches and avoiding some of their shortcomings.

In Chapter 2 we introduced in detail the SDA framework, and proposed several examples of sub-sampling algorithms, for instance Sampling Without Replacement (WR-SDA), Random Block Sampling (RB-SDA), or Last Block Sampling (LB-SDA). While WR-SDA matches BESA for two arms, RB-SDA and LB-SDA are closer in spirit to SSMC by using sequences of successively collected data, that we call blocks of observations. Hence, we call these sub-sampling algorithms Block Samplers. Our analysis can be divided in two parts. We obtained a first upper bound on the number of pulls of each sub-optimal arms for SDA using a Block Sampler, assuming only that the empirical means concentrate around the true means with an exponential rate. This property is quite general, as it is for instance satisfied by any light-tailed distribution. This upper bound showcases a term in O(log(T )) that would ideally be the dominant term of the upper bound, and the term T -1 r=1 P(N 1 (r) ≤ C log(r)) for a constant C, that sums the probability that the best arm (denoted by arm 1 for simplicity) is under-sampled. In a second part of the analysis, we showed that these probabilities are small if:

1. The sub-sampling algorithm allows the arms to play a sufficient diversity of duels 2. Playing many "diverse" duels is enough to avoid under-exploration of the best arm: it must be unlikely that that the best arm loses a very large number of duels against a sub-optimal arm, even if its first draws were unlucky.

We formalized these two intuitions, that allow to conveniently separate the properties that need to be satisfied by the sub-sampling algorithm (first statement) and by the arms' distributions (second statement). We then established that the Last Block and Random Block samplers are suitable sub-sampling algorithms, before exhibiting a sufficient condition (Assumption 2.20) on the arms' distributions to obtain logarithmic regret for SDA. Intuitively, this result states that sub-sampling works if obtaining "low" rewards is more likely for the sub-optimal arms than for the best arm. We further showed that this assumption holds for some widely used families of distributions, such as Single Parameter Exponential Families (SPEF) or some models where a reward is obtained by adding a random noise (whose distribution is common for all arms) to the mean. In the former case LB-SDA and RB-SDA are even asymptotically optimal, in the sense that their regret upper bound matches the lower bound of [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF]. Strikingly, these results hold without using any information on the arms' distributions for the implementation of the algorithms. Furthermore, their practical performance backs up the theory, and we showed that the instances of SDA introduced perform comparably to state-of-the-art bandit algorithms in many settings.

Considering these promising results, we decided to study possible extensions of these algorithms. In particular, in Chapters 3 and 4 we chose to work with LB-SDA due to its simplicity and low computational cost. In Chapter 3 we analyzed two variants of this algorithm

List of Figures

List of Algorithms

List of Algorithms Quantile Adaptive Dirichlet Sampling re-sampled mean . . . . . . . . . . . . . . . . . . . 6.5 Robust Dirichlet Sampling re-sampled mean . . . . . . . . . . . . . . . . . . . . . . . . . .

List of Tables

List of Tables

List of References

List of References

List of References

List of References