À mon père, Omar, et mon grand père, El Mekki, qui ont démontré par l'exemple, que le travail paie.

Other RL challenges RL is a very active field of research and has been applied to a different practical domains, e.g. robotics, advertisements, games, etc. Consequently, every potential application in this dynamic literature brings up new concerns and challenges specific to the particular setting's purpose. A common issue is the credit assignment, i.e. how to assign rewards to particular actions when a series of decisions has been made. Handling non-stationary environments is another issue that comes up often as the world is naturally non-stationary. Also, when used to solve real world problems, the issue of sample complexity arises, indeed RL algorithms may need a consequential amount of interactions before learning, and this can be very limiting in real life. Finally, an essential challenge in the deployment of RL algorithms is the safety and reliability of the agent. Indeed, a self-driving car cannot be trusted to drive in the real world if it's learning its policy from scratch and without prior knowledge.

Overall, the literature on RL is large and diverse, and as it continues to grow and evolve, researchers and practitioners raise new difficulties in conjunction with their respective applications. The latter is beneficial and tackling such challenges head on advances the state of the art and practicality of RL. The PhD student's example and many other applications of RL that ordinary mortals face consistently are a major personal incentives in this thesis.

Conclusion

It is particularly frustrating for our PhD student when they recognize that RL frameworks have the ability to address their conundrum and many other important challenges, xi yet the applicability of RL is hindered by practical limitations. It is also disheartening when certain RL problems are deemed resolved even though the theoretical framework is not wellaligned with reality.

Drawing inspiration from the previous examples and limitations, my particular interest in this thesis is to improve the practicality of RL frameworks and algorithms. In other words, I aimed to increase the applicability of RL algorithms by working towards reducing stringent assumptions, limiting the need for extensive prior knowledge, effectively utilizing available information, and ultimately replacing flawed structures or procedures with more realistic equivalents that better reflect the complexity of real-world scenarios. This way, my purpose is to make RL algorithms and theory more widely applicable to real-world problems.

C t

High probability confidence interval for the parameter at time t λ Regularization parameter

Reinforcement learning is a type of machine learning that involves trial and error interactions with an environment, i.e. learning from the consequences of actions. More precisely, the learner receives rewards or punishments for its actions, and it adjusts its behavior accordingly in order to maximize the reward.

In reinforcement learning, the learner tries to maximize its utility by taking decisions that lead to favorable outcomes. The agent learns to choose actions that will lead to the greatest reward over time, based on its past experiences and the feedback it receives from the environment. Reinforcement learning has been applied to a variety of applications, including natural language processing, robotics, and recommendation systems.

Motivation and theoretical formalism

the reduced amount of updates to the policy, which is usually correlated with an improved stability of the agent's performance.

In episodic RL, the performance of a policy π is measured by the total expected reward V π 1 starting from a state s ∈ S, the value function and the state-action value functions at step h ∈ [H] are defined as

Multi-armed bandits This is a special case of the MDP model where there is only one state, i.e. without a transition structure. In other terms, the agent chooses actions, observes the corresponding reward signals and repeats the process again, i.e. it stays in the same state.

The multi-armed bandits setting is classically motivated by slot machines, also known as one-armed bandits, of which the setting got its name. Given a fixed amount of coins T and a number K of slot machines (arms) with possibly different rewards, the agent allocates their wealth sequentially between the machines in order to gain as much money possible, under the assumption that each machine is characterized by a different payoff. Mathematically, arm k ∈ {1, . . . , K} is modeled by a distribution of probability D k , with mean µ k . At time t, the agent picks an arm j and receives reward r t sampled from the distribution D j . In an abstract sense, the agent seeks to identify the arm with the highest mean in the shortest amount of time.

Realistic reinforcement learning

Here we shed some light onto the subject of this thesis "towards realistic RL" and provide some high level insights to assure the reader of its relevance.

Reinforcement learning aims to solve a wide range of real-world problems, including natural language processing, robotics, and game playing. Therefore, it is only natural to expect these concrete problems to be plausible under the studied theoretical formalism and assumptions. For example, when training an autonomous car to navigate safely, we assume that the transition and reward models are well represented in an -efficiently-learnable model. In practice, applying reinforcement learning involves dealing with real-world challenges such as noisy or incomplete data, limited computational resources, and the need to balance exploration and exploitation without prior knowledge. As a result, the gap between reinforcement learning theory and practice can sometimes be significant, and bridging this gap requires a combination of theoretical understanding and practical experience.

For previous reasons and more, we make seeking realistic practices in theoretical reinforcement learning a driving goal for this thesis. This purpose is visible in our work with different

Outline and Contributions

The purpose of this section is to summarize the different contributions of this thesis in the fields of bandits, linear regression, and reinforcement learning.

Prior to outlining and elaborating our contributions, we provide an overview of our motivation at a high-level. In reality, our efforts and aspirations can be summarized in the question "how can we make the RL framework, algorithms, and assumptions more realistic?".

It is only natural that we begin this effort by turning to the simplest model for sequential decision making: Multi-armed bandits. We consider the finite number of arms case in Chapter 3 and study a pure exploration objective called Thresholding bandits. We propose a Frank-Wolfe based method, extend the setting to a wider class of losses, beat the state-of-the-art

List of

Résumé

Dans cette thèse de doctorat, nous considérons le défi de rendre l'apprentissage par renforcement (RL) plus adapté aux problèmes du monde réel sans perdre les garanties théoriques. Il s'agit d'un domaine de recherche très actif, car l'application au monde réel est l'objectif final de cette littérature ainsi que la motivation première des cadres spécifiques de l'apprentissage par renforcement. Les garanties théoriques sont, comme leur nom l'indique, l'assurance que la théorie peut fournir sur la performance et la fiabilité de nos stratégies. Le développement de ce domaine est crucial pour améliorer les algorithmes de RL interprétables. Notre travail est structuré autour de quatre contextes différents, nous commençons par une introduction au domaine et une revue générale de la littérature, y compris les bandits, les processus de Markov (MDP), certains objectifs d'apprentissage par renforcement, et quelques défis de RL réaliste.

La thèse se poursuit en spécifiant divers scénarios spécifiques ainsi que différentes approches pour relever quelques défis pertinents du RL. Nous nous attaquons d'abord à un scénario séquentiel d'identification de signe pour les bandits à bras multiples, où nous concevons une méthode générique pour définir des algorithmes, une nouvelle stratégie de preuve fournissant des limites d'erreur. Ensuite, nous présentons de nouvelles observations comparant les algorithmes adaptatifs aux oracles hors ligne. Notre deuxième contribution est une amélioration théorique de la régression linéaire séquentielle pour des limites de regret améliorées et une stabilité accrue, nous nous sommes inspirés de résultats bien établis et les avons adaptés au cadre stochastique, puis nous avons illustré les améliorations avec une application aux bandits linéaires. Une contribution significative de cette thèse est l'étude de la récente représentation de la famille exponentielle bilinéaire pour les MDPs à espaces continus. Nous avons pu faire des observations notables menant à des solutions explicites et à des garanties théoriques améliorées. Enfin, nous nous sommes attaqués au problème des gradients de politiques profondes où nous avons introduit une mesure d'erreur bien justifiée pour un apprentissage plus précis de la fonction de valeur. Le besoin de cette dernière amélioration a été fortement motivé par des travaux récents ainsi que par plusieurs expérimentations que nous avons fournies.

Les résultats de cette thèse démontrent un progrès dans la littérature RL, tant sur le plan pratique que théorique, offrant des perspectives et des solutions précieuses pour la communauté RL. Nous pensons que les méthodes proposées font partie des solutions pour combler le fossé entre la théorie du RL et ses applications, faisant de cette thèse une contribution significative au domaine.

Abstract

This thesis explores the challenge of making reinforcement learning (RL) more suitable to real-world problems without losing theoretical guarantees. This is an interesting active research area because real-world problems are the final goal and the first motivation for the different RL settings, and theoretical guarantees are like their name suggests, the assurances that the theory can provide about the performance and reliability of our strategies. Developing this field is crucial for improving interpretable RL algorithms. Our work is structured around four different RL settings, and begins with an introduction to the field and a general review of relevant literature, including bandits, Markov Decision Processes (MDPs), a number of reinforcement learning objectives, and relevant realistic RL challenges.

The thesis proceeds by specifying various specific scenarios as well as different approaches to address the relevant RL challenges. We first tackle an online sign identification setting for multi-armed bandits, where we investigate a generic method to design algorithms, a novel proof strategy providing SOTA error bounds, and we present unprecedented observations when comparing adaptive algorithms to offline oracles. Our second contribution is a theoretical improvement of sequential linear regression for improved regret bounds and increased stability, we took inspiration from well established results that we adapted to the stochastic setting, we illustrated the improvements with an application to linear bandits. Another significant contribution of this thesis is studying the recent bilinear exponential family representation for continuous MDPs, we were able to make notable observations leading to tractability and improved theoretical guarantees. Finally, we tackled the setting of deep policy gradients where we introduced a principled loss for a more accurate value function learning, the need for this improvement was strongly motivated by recent work as well a several experiments that we provided.

The results of this research demonstrate substantial progress in the RL literature both practically and theoretically, offering valuable insights and solutions for the RL community. We believe that the proposed methods show the potential to close the gap between purely theoretical RL and applications-motivated RL, making this thesis a significant contribution to the field.

To the layman reader

If you can't explain it to a six-year-old, you don't understand it yourself

Albert Einstein.

We are interested in a universal task, one that regularly faces every living entity, making decisions in sequential settings. Humans tackle such problems on a daily basis, and with varying importance or possible consequences. From choosing one's clothes every morning to deciding which career to pursue, and from impacting the personal comfort through the day to determining the future quality of life.

Naturally, the decisions must be made for a purpose, and can be part of a bigger plan. This so called purpose is a subjective concept that is almost never identical for different interacting entities, mathematically, it is commonly represented using a utility function that an entity seeks to maximize. This mapping is supposed to encode the entity's preferences and the impact of its choices. In this context, various problems with different complexities and specifities give rise to numerous possible formulations.

As part of our motivation for this manuscript, we shall consider a tangible example that speaks to everyone, concertizes our vision, and allows us deduce generic principles that will inspire the different problems we wish to tackle. For this purpose, I will examine the closest example to my personal life: the PhD student's journey.

Reinforcement learning Classical machine learning involves training a model to make predictions or decisions based on a fixed set of data. The goal is to build a model that is able to generalize to new, unseen data. In the latter, learning is based on the input data and the corresponding outputs that have been provided to it. Reinforcement learning, on the other hand, is interaction-based and the data is provided sequentially instead of a batch manner.

In simple terms, reinforcement learning is a way for a computer or robot to learn how to do something by trying different things and seeing which ones work the best. Imagine you're playing a game where you have to find the treasure. The computer or robot is like a little explorer trying to find the treasure, and it gets a reward every time it finds the treasure. The more treasure it finds, the more it will want to keep playing the game. The computer or robot will try different things to find the treasure, and if it does something that helps it find the treasure more often, it will keep doing that. If it does something that doesn't help it find the treasure, it will try something else. Through this process, the computer or robot will learn how to find the treasure more efficiently.

The case for RL instead of ML One of the key promises of RL is that it allows learners to improve through experience, without requiring explicit programming of rules or behaviors. This can be particularly useful in complex or dynamic environments, where it may be difficult to pre-define a fixed set of rules or behaviors. RL algorithms can adapt to changing environments and learn to make optimal decisions based on the feedback they receive.

Another promise of RL is that it has the potential to enable autonomous systems to learn and adapt to their surroundings in real-time. This can be beneficial in a variety of applications, such as self-driving cars, which need to be able to respond to changing traffic conditions and make decisions that maximize safety and efficiency.

Overall, the promise of RL is to enable agents to learn and adapt to their environments in order to optimize their performance and achieve their goals.

A primer on RL terminology

We define some necessary keywords of the RL jargon.

First, we define the environment, i.e. a context in which the learner operates and makes decisions. For example, the environment could be a physical space, such as an inverted pendulum that a person controls using their hands. Next, an action is a decision taken by the learner. For example, the action of a person to push the pendulum forward, backward, or to not push it. A state is the current situation or context in which the learner finds itself, like the angle or the abscissa of our pendulum. The state may include information about the environment, like the wind, the current velocity, or the person's past decisions. The learner is called an agent in RL, an entity that is learning and taking actions in an environment. Like the person that controls the inverted pendulum in our example.

Then, let's define some terms that encode the environment or agent's reactions and decisions. For instance, a policy is a strategy or set of rules that the agent uses in its interaction with the environment. The latter can be deterministic or randomized, e.g. an agent can play each possible action with a given probability. The reaction of the environment to an action in a given state is called a reward. A reward is the outcome or consequence of an action taken by the agent, it provides feedback to encourage or discourage the agent from taking specific actions. viii Finally, we define the most important quantity in reinforcement learning. The value function is a measure of the expected long-term reward or utility of a given state or action. The value function helps the agent decide which actions are most likely to lead to good long-term outcomes. The value function is defined in conjunction with a policy of the agent, and it helps the latter evaluate potential consequences of different actions and choose the ones that most likely lead to great rewards.

To summarize, we say that an agent interacts with an environment, decides which actions to play using their policy, receives a reward and moves to a new state. An agent represents the efficacy of a given policy by estimating its corresponding value function. Based on the value estimate, the agent updates their policy when they deem it appropriate and receives new rewards as feedback from the environment on how to improve their decision making process.

RL challenges RL is still an open field with many open questions, which is fortunate for us.

In the following, we enumerate and explain the main challenges facing an RL agent through a concrete example. Exploration-exploitation Consider an innocent undergraduate, embarking in the adventure that is a PhD, full of hope and happy to finally have the freedom of doing research in their favorite field. The latter is high spirited and doesn't know how to spend their energy, they are thus confronted with their first (of many) dilemma: probe the literature in depth or parse relevant papers and pursue a first result. While the former translates to spending months in exploring and searching for pertinent research, the latter can quickly lead to publishable content and to harnessing the student's strengths. This compromise is at the heart of the reinforcement learning challenge and is called the exploration-exploitation trade-off.

Definition. In reinforcement learning, the exploration-exploitation trade-off is the fundamental dilemma faced by a learner trying to learn about its environment and maximize its profit. On one hand, the learner needs to explore, or try out different actions and be in new states, in order to learn more about the environment and improve its decision-making. On the other hand, the agent also needs to exploit, or take advantage of what it has learned so far, in order to maximize its profit in the short term.

The exploration-exploitation trade-off arises because exploration can be risky and may not lead to an immediate payoff, while exploitation allows the agent to maximize its profit based on its current learner, but may not lead to long-term learning or optimization. As a result, the learner must find a balance between exploration and exploitation in order to learn effectively and maximize its reward over time.

Our PhD candidate faces the exploration-exploitation trade-off from the beginning of their studies, for instance, they must balance their desire to utilize known techniques to improve existing work whit their mission to learn key techniques from their area of interest. Therefore, this tension between exploration and exploitation manifests itself from the first months of their research.

Continuous spaces Let's say that our PhD student has now spent enough time to become well-informed about their chosen field. And let's assume for our case that their advisor has given them the freedom to choose their specific topic. The second challenge that confronts our candidate is selecting one specific subject. For instance, a scholar interested in RL has to choose from multi-armed bandits, structured bandits (linear, Lipschitz, generalized...), tabular MDPs, continuous MDPs, etc. And that for all of the latter, they can choose between stationary or non-stationary, theory or practice, regret minimization or pure exploration and much more. Agreed a layman reader wouldn't understand these technical terms, yet they would surely grasp the frustration that a junior candidate is faced with when they are spoiled for choice to this extent.

Definition. One major difficulty in sequential decision making is scaling the strategies to continuous spaces, i.e. environments with large or infinite number of states and actions. Indeed, it can be very challenging to handle the exploration-exploitation dilemma in continuous spaces. Indeed, balancing this trade-off may require the agent to explore a large (possibly infinite) number of states in order to learn about the environment effectively.

Function approximation Faced with the aforementioned challenge, it may be impractical for our PhD candidate to evaluate every possible course of action explicitly. In these cases, a popular solution is to resort to function approximation techniques. Our scholar can assume some kind of model for the rewards and or transitions, then they can optimize their policy by learning in a functional space. The latter can be easier and is especially interesting in the case of parametric spaces or functions with particular structures.

Definition. Function approximation in RL refers to the setting where some functional representation is assumed to approximate the value function or the policy. Major challenges x arise with the latter. For instance, these techniques can be computationally intensive if the class is overly intricate, do not necessarily provide accurate approximations if the class is overly simplistic, and may even be unrealistic.

Computational efficiency

Tractability is an important consideration in machine learning, indeed algorithms often involve complex computations. In RL, computational efficiency is even more regnant since algorithms almost always involve iterative processes, such as updating an estimator of the value function or re-optimizing the agent's policy after new observations. Definition. Computational efficiency is directly correlated to the performance and practicality of an RL algorithm in real-world scenarios. Unfortunately, and due to the nature of the setting, RL algorithms may need to update/optimize certain functions a number of times proportional to the amount of data. Therefore, RL methods can be computationally intensive and those with theoretical guarantees are often intractable. Accordingly, computational efficiency constitutes a key challenge in the development and application of RL algorithms, and can seriously affect the impact and popularity of this literature.

List of Symbols

Motivation and theoretical formalism

We are interested in sequential decision making, where a learner interacts with some environment and must take a sequence of decisions in order, such that the outcome of each decision may influence the options available for subsequent decisions. In this context, various specific problems with different complexities and diverse objectives give rise to numerous possible formulations. In particular, we are interested in the reinforcement learning framework, a sub-field of machine learning where data is not provided beforehand but is revealed sequentially as the learner interacts with the environment.

In this Chapter, we will clarify the main concepts of our chosen theoretical frameworks, then we will enumerate and provide high-level intuition for the different objectives studied in this manuscript. Finally, we will describe in further detail the tasks that are studied, and give a summary of our contributions.

Introduction

Markov decision process

The MDP framework is the most popular for reinforcement learning. In this structure, the learner that interacts with the world is called an agent, the agent's available decisions are named actions, and the utility that a decision achieves is called a reward. The literature studies reinforcement learning mathematically as a Markov decision processes (MDP) where the agent i) chooses actions based on past observations ii) observes an immediate reward sampled from an unknown function, and iii) moves to a new state following an unknown Markovian transition mapping. The latter implies that given the current state and action, the next state is independent of previous state-action pairs. In layman's terms, we say that a transition is Markovian if the future depends only on the present state and not on the past history.

Infinite-horizon RL A popular formulation of RL is the infinite-horizon Markov Decision

Problem (MDP) with -possibly-continuous states s ∈ S, actions a ∈ A, transition distribution s t+1 ∼ P(. | s t , a t ) and reward function r t ∼ r(s t , a t ). Let π(a|s) denote a stochastic policy, the agent repeatedly interacts with the environment by sampling action a t ∼ π(.|s t ), receives reward r t and moves to a state s t+1 .

In Infinite-horizon RL, the value of a policy in a given state is measured as:

V π (s) ≜ E τ ∼π ∞ t=0 γ t r t |s 0 = s ,
where γ ∈ [0, 1) is a discount factor accounting for the preference of present rewards over -possibly larger-future ones. This value function evaluates the performance of an agent and is often linked to the objective even though the latter can vary depending on the application.

Episodic RL Another prominent formulation for RL [START_REF] Osband | More) efficient reinforcement learning via posterior sampling[END_REF]; Azar, Osband, and Munos, 2017; Dann, Lattimore, and Brunskill, 2017), Episodic RL is a tuple P = ⟨S, A, P, r, K, H⟩, where the state (resp. action) space S (resp. A) might be continuous. In episodic RL, the agent interacts with the environment in K ∈ N episodes consisting of H ∈ N steps. Episode k starts by observing state s k 1 . Then, for t = 1, . . . H, the agent draws action a k t from a (possibly time-dependent) policy π t (s k t ), observes the reward r t ∼ r(s k t , a k t ) ∈ [0, 1], and moves to a new state s k t+1 ∼ P(. | s k t , a k t ) according to the transition function P. Roughly speaking, episodic RL was introduced as a way of breaking down the RL problem into smaller problems, where the agent learns from the rewards they receive at the end of each episode. This allows the agent to focus on the short-term decisions rather than the long-term ones and makes the problem more manageable. It is also more interesting in practice due to Introduction levels of wingspan, and we believe that the journey of realistic RL is still long and contains several -deceptively small-building blocks and contributions.

Reinforcement learning objectives

There are several different performance measures that can be used to evaluate the effectiveness of an agent in reinforcement learning. One common measure is the expected return, which is the sum of the discounted rewards that an agent receives over time. Another measure is the cumulative reward, which is the total reward received by an agent over a certain period of time. Other measures include the average reward per time step, which reflects the average reward received by an agent at each time step, and the average reward per episode, which reflects the average reward received by an agent over the course of an episode. These performance measures can be used to compare the performance of different agents or to assess the performance of a single agent over time.

Regret minimization

Regret minimization is a concept in reinforcement learning that refers to the idea of computing the policy leading to maximization of cumulative reward. The latter requires the agent to strategically control the actions in order to learn the transition and reward functions to a sufficient level of precision. This tension between learning the unknown environment and reward maximization is quantified as regret: the typical performance measure of an episodic RL algorithm. Regret is defined as the difference between the expected cumulative reward or value collected by the optimal agent that knows the environment and the expected cumulative reward or value obtained by an agent that has to learn about the unknown environment. Formally, the regret over K episodes is

R(K) ≜ K k=1 V π ⋆ 1 (s k 1 ) -V πt 1 (s k 1 )
.

where π t is the policy of the agent at episode t and π ⋆ is the optimal policy, i.e. for all possible initial states s ∈ S we have π ⋆ ∈ arg max π V π 1 (s) . We choose to defer the other definitions of regret to their respective chapters, e.g. those relating to Infinite-horizon RL or to Multi-armed bandits. In essence, the objective is similar: minimizing the regret incurred by an agent in every setting. The differences are manifested in the structures of interaction or in the different definitions of the value functions.

Outline and Contributions

Pure exploration

In reinforcement learning, pure exploration refers to situations where an agent's primary goal is to gather as much information about the world as possible, rather than maximizing the cumulative reward. In these settings, the environment may or may not reveal the rewards to the agent, and the goal is to learn as much as possible in order to return a policy that optimizes some criterion. The exploration and exploitation trade-off is also present in this setting, through the decision of whether to try a new action that may lead to a better reward or to continue using a known action to obtain a better estimate of its value and / or transition dynamics. Pure exploration settings can be useful for gathering data and understanding the structure of an environment.

To illustrate this objective, consider the Best-policy identification setting. In the latter, the agent observes both transitions and rewards to attempt returning an ε-optimal policy with large probability. In technical terms, if we denote V π the value function of the agent following policy π and V * the optimal value function then the objective is: given ε ∈ R + and δ ∈ (0, 1), interact with the environment and return as fast as possible a policy π such that:

P (∀s ∈ S, V π (s) ≥ V * (s) -ε) ≥ 1 -δ.
To conclude, regret minimization and pure exploration are two pertinent objectives that continue to attract the attention of researchers in the reinforcement learning field. While in pure exploration an agent's primary goal is to gather as much information about its environment as possible through exploration, regret minimization rather entails maximizing the cumulative reward which can be more restrictive for the agent since it cannot explore possibly fruitless actions. In both cases, the agent must balance the trade-off between exploration and exploitation.

Introduction

both theoretically and empirically, and apply our generic proof scheme to improve existing algorithms. The findings of this Chapter were published at the Neural Information Processing Systems conference in 2021 as a spotlight paper (top 3%).

In Chapter 4, we consider the popular linear regression problem. We uncover an old result of the adversarial setting and adapt it to the stochastic case, we show that this algorithm should be the default one instead of ridge regression. We apply this modification to linear bandits and show that it enables removing an omnipresent assumption. Our analysis is useful both theoretically and for the practitioner. The results of this Chapter were accepted for publication at the Neural Information Processing Systems conference in 2021.

In order to progress, we study continuous state action MDPs in Chapter 5. Bewildered by the popularity of the unrealistic linear MDP model, we consider a recently proposed MDP representation. We recall the expressive power of the latter and unveil a novel and crucial property about it that enables us to design an algorithm with tractable planning. We also show several results of independent interest, e.g. we show that we can forgo clipping value functions therefore removing a superfluous step of non-linearity. The contributions in this Chapter were published in the The AAAI Conference on Artificial Intelligence in 2023 as an oral presentation.

In Chapter 6, we study deep policy gradients, an infinite-horizon MDP setting where the value function and policy are modeled using deep neural networks and optimized using gradient descent. Driven by recently unveiled pitfalls of existing algorithms, we show that a simple modification in the value function loss can lead to significant improvement. We first motivate the latter and provide an intuitive proof of concept, then we demonstrate empirically that 1) our method improves over the standard training loss, and 2) our intuition is indeed accurate and our predicted are corroborated by the simulations. The contributions presented in this Chapter were accepted for publication at the International Conference on Learning Representations in 2021.

To summarize, our contributions are organized as depicted in Figure 1.1, between improvements of existing methods and contributions that introduce novel principles and algorithms. Our goal through all of them is unique: contribute to the RL research by improving algorithms that are both theoretically sound and user-friendly for practitioners, and to support the advancement towards a more realistic RL formalism. Remark 1.1. remarkbar The chapters in this thesis have been written to be standalone and can be read independently, catering to the diversity of settings studied. This allows for ease of readability and flexibility for the reader, who may only be interested in a specific topic without the need to go through the entire thesis.

Outline and Contributions

Frank-Wolfe for thresholding bandits

In Chapter 3, we focus on a pure exploration setting called thresholding bandit. In the latter, a learner interacts with a K-armed bandit, nature provides it with a threshold and after a known time horizon it asks the learner to output their answer for "which arms' means are larger than the threshold". An agent is evaluated by the total number of mistakes it makes, weighted or not, after ending the interaction. In this game, the learner's loss depends on the unknown bandit parameters. Therefore, at every time-step, the agent needs to act with two independent objectives in mind: improving the loss estimate, and sampling arms that can possibly yield the most significant loss reduction. The latter is a quintessential challenge facing an agent in this setting, and is a manifestation of the famous exploration-exploitation trade-off. This setting has been investigated in many recent papers and several algorithms were proposed with good empirical performances yet sub-optimal theoretical guarantees. We extend this bandit problem to more general loss functions, with possibly gap-dependent weights. Moreover, building on the popular Frank-Wolfe optimization approach, we are able to propose a generic method to design strategies for this setting. Our algorithms are (perhaps surprisingly) not based on typical RL concepts like optimism, explore-than-commit, nor successive elimination. Furthermore, we provide an simple and intuitive analysis that works for a wide class of index-based algorithms including all previously introduced ones for this setting. Our analysis not only yields to state-of-the-art theoretical guarantees for our algorithm but improves the bounds for existing algorithms as well. In addition, we evaluate our algorithm empirically and show how it outperforms existing techniques. Finally, we provide new non-trivial observations Introduction about this setting, e.g. how adaptive-methods can empirically outperform non-adaptive oracles by a significant margin.

The forward algorithm and application to linear bandits

In our quest of tackling increasingly complex sequential decision making settings, in Chapter 4 we are interested in studying bandit problems with a continuous action space. Several works in this direction make structural assumptions about the reward model, the most popular one being the linear rewards assumption. In the latter, it is also assumed that the reward function is bounded (usually in [0, 1]) and that the learner is aware of this. While considering bounded rewards is very reasonable, providing the bound to an agent before the interaction is restrictive in our opinion. We investigate methods of relaxing the latter by attacking the underlying linear regression challenge, specifically, we are interested in studying the stochastic linear regression setting without assuming a known bound on observations. We motivate this setting by highlighting the theoretical and empirical weaknesses of the standard ridge regression. This problem has been solved for adversarial setting in the seminal paper [START_REF] Vovk | On-line competitive linear regression[END_REF]) by adopting the forward algorithm, which ingeniously adds a regularization term depending on the future coordinate. Naturally, we investigated this method in the stochastic setting to determine whether it also alleviates the "bound knowledge" assumption in this case. We answer the latter positively, and we show the true bounds of standard regression when stripped from knowing the range of observations. Furthermore, we show that this algorithm lends itself very conveniently to the linear bandit setting. Finally, we show through numerical experiments the success due to this simple modification. The latter is indeed a small adjustment, yet it paves the way towards realistic scenarios in which the observations' range is not known a priori.

MDPs with Continuous state and actions spaces

We take the pursuit of realistic RL challenge to the next level by attempting to advance the work on continuous state-action MDPs. In Chapter 5, we consider the episodic RL setting in which an agent interacts with the environment in K ∈ N epochs of length H ∈ N. A learner in this framework must update its policy at the beginning of every epoch based on past information only. Currently, the main challenge in this setting is the representation choice. Indeed, while linear MDPs model has become ubiquitous in literature, it remains a purely abstract assumption that only helps the analysis without any concrete example or application.

In this perspective, we investigate a recently introduced representation based on the exponential family of distributions (Chowdhury, [START_REF] Sayak | Reinforcement Learning in Parametric MDPs with Exponential Families[END_REF]. Unlike the class of linear MDPs, the Bilinear exponential family is very expressive and includes real examples like Tabular, Factored MDPs, and Linear Quadratic Regulators. After a thorough motivation of this

Outline and Contributions

little-appreciated functional class, we unveil a novel observation assimilating the latter to an approximate linear representation. Consequently, we are able to provide a tractable algorithm for MDPs of this family in the episodic RL setting. We also provide a regret bound under mild regularity assumptions that exhibits an optimal dependence on H and K. Finally, the presented algorithm and analysis showcase certain improvements over the literature, e.g. clipping the value function is irrelevant.

Improved value estimator for deep policy gradients

In Chapter 6, probably our most practical work, we consider discounted infinite horizon MDPs with continuous state-action spaces. Our objective in this effort is to join the race for principled deep policy gradient algorithms with strong empirical performances. In fact, this research direction has been very prolific since the emergence of toy environments that mimic real life scenarios by encoding the laws of physics. Indeed, RL has been very successful in learning the dynamics of these tasks, ranging from playing various video games to controlling real robots.

We became interested in this problem after diverse observations by a handful of recent papers pinpointing significant discrepancies between the driving principles and insights that drive algorithms on and what is implemented in practice. For instance, the belief that neural network approximators allow for a smooth learning of value functions has been challenged showing that the deep policy framework fails to provide a decent fit. Motivated by these findings, we motivate then propose to adopt the residual error as a loss for critics in the actor-critic framework. We provide evidence that this methods fits the values better and demonstrate a consistent and logical performance boost on a variety of challenging tasks, including environments with sparse rewards signals. Finally, we provide empirical evidence that our method factually reduces the policy gradient variance, further demonstrating the soundness of our intuition, unlike previous methods that were shown to bias the estimators without variance reduction.

Introduction

This chapter provides an overview of several key concepts in Reinforcement Learning (RL), including multi-armed bandits, Markov Decision Processes (MDPs), and different RL objectives. We will also explore some of the open questions in realistic RL and the challenges that arise when dealing with large-scale and complex problems.

Multi-armed bandits are a simple and fundamental problem in RL that has been widely studied and has many practical applications. MDPs are a more general framework for RL problems and are widely used in many applications. We will discuss the key concepts of MDPs, such as states, actions, rewards, and the value function. We will also discuss different variants of the MDP framework with their possible applications.

Literature review

This chapter also examines different RL objectives, such as regret minimization and pure exploration, and various special manifestations of these objectives like best arm identification and thresholding bandits, which are important for many real-world applications. We also mention various concepts involved in solving this kind of RL problems, such as optimismunder-uncertainty, Thompson sampling and Track-and-stop Finally, we will review some of the open questions in realistic RL, such as the challenges of dealing with large-scale and complex problems. We will inspect the importance of representations and structures that can simplify the problem and make it more tractable. We will also highlight the importance of addressing these sample and computational complexity challenges in order to make RL more widely applicable to real-world problems.

Overall, the chapter provides a broad yet non-exhaustive introduction to the field of RL, with a focus on the key concepts, variants, and open questions that are relevant to the core of this thesis.

Remark 2.1. In this portion of the manuscript, we have made a conscious decision to be more verbose and steer away from discussing mathematical details. Our objective is not to provide a comprehensive, technical review of all relevant concepts, as there are already many comprehensive monographs available that serve that purpose. Instead, we aim to stimulate the reader's imagination and encourage discussion. By presenting the material in a more descriptive manner, we hope to generate a broader and more accessible understanding of the topic, and to encourage deeper exploration by the reader. Our focus is on creating a thought-provoking narrative that encourages the reader to engage with the material in a meaningful way.

Stochastic multi-armed bandits

A stochastic multi-armed bandit problem is a set of distributions indexed by the available actions. A decision maker sequentially samples from the different distributions for a number of consecutive rounds. This first trails of this setting date back to the fifties where the seminal works [START_REF] Robbins | Some aspects of the sequential design of experiments[END_REF][START_REF] Leung | Asymptotically efficient adaptive allocation rules[END_REF] introduced the stochastic multi-armed bandit problem. In a broader sense, these were among the pioneering works that paved the way for sequential statistics, i.e. where the sample size is not necessarily fixed or known in advance [START_REF] Harold | A method of sampling inspection[END_REF][START_REF] Wald | Sequential analysis[END_REF]. This novelty at the time was a significant step towards realistic and more efficient algorithms. Indeed, although the literature was far from being optimal, it still improved over known pre-defined time budgets. The latter was groundbreaking in the sense that it allowed statisticians a formalism where they could adjust their decisions continuously with the upstream of new information. This meant that the testing budgets were reduced and pointless samples were avoided.

Stochastic multi-armed bandits

In this modernization of research efforts on statistics, the departure was not from the fixed sample size exclusively, instead it also swept other outdated concepts from consideration, e.g. promoting multiple population statistics [START_REF] Isbell | On a problem of Robbins[END_REF][START_REF] Russell N Bradt | On sequential designs for maximizing the sum of n observations[END_REF][START_REF] Bellman | A problem in the sequential design of experiments[END_REF][START_REF] Vogel | An asymptotic minimax theorem for the two armed bandit problem[END_REF]. The latter was also motivated by the need for more applicability and better sample efficiency. Pragmatically, a statistician's goal is more often to compare multiple populations than just to confirm or infirm some property of a singular population. In the medicinal drug dosage problem for example, the doctor needs to determine the amount of medication to prescribe for their patient, and this can change over time with factors such as the resistance and reaction of the specific patient, the severity of their condition, the emergence of new drugs, and even to the comparison between several competing drugs with different properties. Overall, the fact that sequential decision making improves over classical statistics does not really require much motivation as it so clearly is an improvement in terms of applicability and prospects of gained efficiency.

Purpose and applications

Nowadays, multi-armed bandits serve a variety of real world objectives. Indeed, there exist multiple scenarios where one needs to make decisions in a sequential manner in order to maximize rewards. For instance, multi-armed bandit can be used to model situations in which an agent must allocate resources (Lattimore, Crammer, and Szepesvári, 2015; Verma et al., 2019; Fontaine, Mannor, and Perchet, 2020), such as advertising budget, to different options in order to maximize the return on investment. Another considerable field of application is clinical trials, indeed, bandits can be model the selection of treatments or therapies in clinical trials (Berry, 1978; Villar, Bowden, and Wason, 2015; Aziz, Kaufmann, and Riviere, 2021), allowing researchers to learn which treatments are most effective over time. Moreover, bandits are also of great interest in the field of network optimization, they help in the allocation of resources in communication networks [START_REF] Avner | Multi-user lax communications: a multi-armed bandit approach[END_REF][START_REF] Li | Almost optimal dynamically-ordered channel sensing and accessing for cognitive networks[END_REF][START_REF] Cai | An online learning approach to network application optimization with guarantee[END_REF][START_REF] Gai | Learning multiuser channel allocations in cognitive radio networks: A combinatorial multi-armed bandit formulation[END_REF][START_REF] Gai | Decentralized online learning algorithms for opportunistic spectrum access[END_REF], allowing network operators to learn which strategies are most effective at optimizing network performance.

There still exist a wide variety of problems that are not mentioned here, e.g. online advertising to model the selection of ads displayed to the users and personalization to optimize their engagement or satisfaction. Overall, multi-armed bandits are a useful tool for modeling and solving decision-making problems in which the agent must learn which actions most probably lead to good outcomes over time.

Variants A consequence of the abundances of potential applications including bandits is that they induce a myriad of variations over the original MAB problem. Here we provide a non-exhaustive number of the many influential settings based on MABs.

Literature review

Perhaps the most famous problem in this context is the contextual MAB. In contextual bandits [START_REF] Abe | Associative reinforcement learning using linear probabilistic concepts[END_REF][START_REF] Agarwal | Taming the monster: A fast and simple algorithm for contextual bandits[END_REF][START_REF] Beygelzimer | Contextual bandit algorithms with supervised learning guarantees[END_REF], the reward or utility of an action depends not only on the action itself, but also on the context in which the action is taken. For example, the utility of attending doctoral training may depend on the PhD student's prior education, interests, or current workload.

Combinatorial MABs is another compelling and popular setting. In combinatorial bandits (Anantharam, Varaiya, and Walrand, 1987; Caro and Gallien, 2007; Gai, Krishnamachari, and Jain, 2010; Chen, Wang, and Yuan, 2013), the agent must choose a combination of actions rather than a single action. This can be convenient for modeling scenarios where the learner must make multiple decisions or choices simultaneously, such as a PhD student allocating their time to multiple projects or papers.

Among the research directed at the previously mentioned continuous spaces challenge, Linear bandits were proposed as a structured variant of the (MAB) problem [START_REF] Abe | Associative reinforcement learning using linear probabilistic concepts[END_REF][START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF]. In linear bandits, the reward of an action is modeled as a linear function of a set of -known-features or characteristics associated with the action. Several research efforts introduced optimism based algorithms for this setting (Abbasi-Yadkori, Pál, and Szepesvári, 2011; Dani, Hayes, and [START_REF] Varsha | Stochastic linear optimization under bandit feedback[END_REF], along with near optimal regret bounds.

On the other hand, pure exploration objectives were also proposed with the multi-armed bandit setting. For instance, in thresholding bandits (Abernethy, Amin, and Zhu, 2016; [START_REF] Mukherjee | Thresholding bandits with augmented ucb[END_REF][START_REF] Tao | Thresholding bandit with optimal aggregate regret[END_REF], the final objective is to return arms whose reward exceeds a certain threshold value.

Markov Decision Processes

The Markov Decision Process (MDP) framework is a mathematical formalism for modeling problems of decision-making under uncertainty. There are many traces of this setting in the early fifties and sixties [START_REF] Massé | Les réserves et la régulation de l'avenir dans la vie économique[END_REF][START_REF] Wald | Sequential analysis[END_REF][START_REF] Kenneth J Arrow | Optimal inventory policy[END_REF][START_REF] Joseph Arrow | Studies in the mathematical theory of inventory and production[END_REF][START_REF] Dvoretzky | The inventory problem: I. Case of known distributions of demand[END_REF], this is not an exhaustive list as several objectives led to using similar frameworks at the time, e.g. inventory control, resource management, and sequential testing. Richard Bellman was the one who popularized MDPs in the early 1950s, indeed he introduced the framework rigorously and invented Dynamic programming in an effort to solve the optimization problems faced by engineers and economists [START_REF] Bellman | Dynamic programming[END_REF].

The motivation behind the MDP framework is to provide a flexible and generalizable tool for solving a wide range of optimization problems in which an agent must make decisions based on incomplete information about the environment. MDPs are particularly useful for

Markov Decision Processes

modeling problems in which the agent must take a sequence of actions over time, as they provide a way to represent the temporal structure of the problem.

The MDP framework has been widely applied in a variety of fields, including economics, operations research, engineering, and computer science. It has played a central role in the development of reinforcement learning (RL) [START_REF] Sutton | Introduction to reinforcement learning[END_REF]. MDPs have also been used to model a variety of real-world problems, such as inventory management, production planning, and robotic control [START_REF] Warren | Approximate Dynamic Programming: Solving the curses of dimensionality[END_REF].

Purpose

The purpose of adopting MDPs in RL is to provide a formal framework for defining and solving RL problems, that is more general and realistic than the multi-armed bandit. An MDP consists of a set of states, actions, and a transition function that defines the probabilistic transitions between states as a result of taking actions. It also includes a reward function that defines the rewards received by the agent for taking certain actions in certain states. The goal of the RL agent is to learn a policy, which is a function that specifies the action to take in each state, that maximizes the expected cumulative reward over time.

MDPs provide a way to represent the temporal structure of RL problems, as they allow the agent to take sequential actions in order to achieve a goal. They also provide a way to incorporate uncertainty into the decision-making process, as the transitions and rewards are typically stochastic and the agent must learn to make decisions based on incomplete information.

MDPs have been applied to a wide range of RL problems, including control problems such as robotic manipulation and autonomous driving, as well as problems in economics and finance, such as portfolio management and auction design. They have also been used to model a variety of real-world problems, such as inventory management, production planning, and network routing.

Variants There are many variants of MDPs that have been proposed in literature. The major defining factor of these diverse alternatives appears in the way the environment or decisionmaking problem is structured.

In Partially observable MDPs (POMDPs), the agent only has partial observability of its state in the environment. This is suitable when modeling situations in which the agent has limited or noisy information about the environment [START_REF] Thrun | Probabilistic robotics[END_REF], such as in problems involving sensor uncertainty or hidden state variables. A different variant is the Stochastic games where there are multiple agents that can take actions simultaneously and influence the state of the environment. This is suitable for modeling situations where the actions of multiple agents are co-dependent [START_REF] Kearns | Fast planning in stochastic games[END_REF], such as in multi-player games or social dilemmas. Another variant of MDPs is Decentralized MDPs, in which the agent must make decisions based on the actions of other agents, who may be pursuing their own goals or

Literature review

objectives. This fits scenarios where the agent must coordinate with other agents or respond to their actions [START_REF] Amato | Decentralized control of partially observable Markov decision processes[END_REF], such as in distributed control or multi-agent systems.

On the other hand, we find a different kind of MDP variants where the contrast is in terms of algorithmic choices or structural assumptions. In other terms, these are variants of MDPs that differ in the way agents make decisions or the way policies are chosen and updated.

Reinforcement learning, the most pertinent variant for our manuscript, is where the agent learns a policy or strategy by interacting with the environment and receiving feedback in the form of rewards. This is useful for settings where the agent must learn from experience [START_REF] Sutton | Introduction to reinforcement learning[END_REF], such as in problems involving exploration or uncertainty. Another algorithmic choice is Dynamic programming: in DP the agent uses a backward induction approach to solve the MDP by working from the final time step (horizon) backwards to the initial time step. Using the Bellman equations, the agent is able to compute the optimal value function and deduce the best policy to follow. This is suitable for interactions with a finite horizon or a known end time, and when the transition probabilities and rewards are known in advance. Please refer to the book [START_REF] Bertsekas | Dynamic Programming and Optimal Control[END_REF] for a complete overview. On a different notes, Monte Carlo methods were also proposed to solve MDPs. In MC algorithms, the agent estimate a policy's value function by sampling from the environment and averaging the rewards over multiple episodes. This is useful in problems with large state or action spaces, or when the transition probabilities or rewards are uncertain or hard to model.

Reinforcement learning objectives

The objectives in RL refer to the metrics used to evaluate the performance of an RL algorithm. These objectives typically reflect the ultimate goal of the agent, such as maximizing the cumulative reward, returning a policy within a certain distance of the optimal, or achieving a certain level of safety. Common performance objectives include the expected cumulative reward, the average reward per step, the value of the optimal policy, and the probability of reaching a goal state. Additionally, there are other objectives that are related to the stability and robustness of the algorithm, such as the variance of the achieved rewards and the convergence time. These performance objectives can be used to compare different RL algorithms and to guide the design and improvement of RL algorithms. In the real-world application of RL, it is crucial to identify the appropriate performance objectives and the according algorithm.

Regret minimization

Regret minimization is a key concept that is used in the field of reinforcement learning (RL) to evaluate the performance of RL algorithms. The idea is to compare the performance of a

Reinforcement learning objectives

strategy to that of an optimal decision-making algorithm, in terms of the cumulative rewards over time. Formally, regret is defined as the expected value of the difference between rewards of the optimal strategy and those obtained by the algorithm being evaluated.

There is a large body of literature on regret minimization in RL, with research focusing on various aspects of the problem such as designing algorithms with provably low regret, understanding the relationship between regret and other performance measures, and improving the sample complexity while keeping optimal bounds.

Most RL algorithms are based on the concept of optimism. Optimism in RL refers to the idea of taking actions that have a high potential for reward, i.e. the agent compares different actions optimistically by comparing their best possible outcomes [START_REF] Agrawal | Sample mean based index policies by o (log n) regret for the multiarmed bandit problem[END_REF]Kaelbling, 1993;[START_REF] Pack | Associative reinforcement learning: A generate and test algorithm[END_REF]. This can be accomplished by using optimistic initialization, which assigns a high initial value to the estimates of the value of the states or actions, or by using optimistic planning, which assumes that the unknown parts of the environment will be favorable. One example of an algorithm that uses optimism is UCRL (Upper Confidence RL), which is an optimistic algorithm that can be used for planning in uncertain environments.

Just as popular as the optimistic strategies is Thompson sampling. This is a strategy where the agent selects an action based on a probability weighting over the value of actions, this algorithm is older than Bandits or MDPs [START_REF] William R Thompson | On the likelihood that one unknown probability exceeds another in view of the evidence of two samples[END_REF]. The said distribution is updated based on the observed rewards, and the action is again selected by sampling from the current distribution. This strategy can be useful when the agent is uncertain about the true value of the actions and they have a reasonable prior distribution for the rewards. Thompson sampling is the most popular Bayesian Multi-armed Bandit algorithm [START_REF] Granmo | Solving two-armed Bernoulli bandit problems using a Bayesian learning automaton[END_REF] Another popular concept in designing RL algorithms is Explore-then-commit. This is a strategy in which an agent explores different options or actions at the beginning of the decisionmaking process and then commits to the action that has the highest estimated value and for the rest of the process (Gittins and [START_REF] John | A dynamic allocation index for the discounted multiarmed bandit problem[END_REF]. This strategy can be useful when the agent is uncertain about the true value of the actions or options but has a good idea of the relative value.

There exist other strategies in RL such as Elimination, in which an agent tries out different options or actions and eliminates the ones that have been found to be sub-optimal. This can be useful when the agent has a good idea of the relative value of the actions or options but is uncertain about their absolute value [START_REF] Even-Dar | Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems[END_REF]. Track-and-stop is another common strategy, it is used when a reasonable estimation of the optimal allocation is available, through e.g. a tractable lower bound, and the agent's policy is then trying to match this estimated Literature review allocation (Combes, Magureanu, and Proutiere, 2017; Garivier and Kaufmann, 2016; Degenne, Shao, and Koolen, 2020).

In addition to these, the research on regret minimization has also been extended to various specific scenarios such as adversarial setting and online learning with side observations. And it is worth noting that the mentioned strategies are not exhaustive nor mutually exclusive, indeed a number of recent efforts propose combinations of these algorithms.

Pure exploration

Pure exploration in RL refers to the problem of learning about the environment in order to identify the optimal policy or value function of the environment. In this problem, the agent may not know the reward function and it's objective is not to achieve a good reward but can rather be learning about the environment or returning a near optimal policy. Contrary to regret minimization, pure exploration encompasses a collection of disparate objectives, and with efforts focusing on various algorithmic. Also, in this setting and unlike the regret objective, the agent may be given a fixed confidence level instead of the time budget, in which case they need to decide when to stop the interaction and ensure an optimal response with the provided confidence level.

In the bandit literature, Best arm identification (BAI) is when the goal of the agent is to identify the arm with the highest expected reward in as few steps as possible. The best arm identification problem is a fundamental problem in the field of online learning and optimization. A number of algorithms have been proposed for BAI, some examples include UCB (Upper Confidence Bound) algorithm, Thompson Sampling and Bayesian optimization. Some key references for best arm identification problem are [START_REF] Domingo | Adaptive sampling methods for scaling up knowledge discovery algorithms[END_REF][START_REF] Bubeck | Pure exploration in multi-armed bandits problems[END_REF][START_REF] Audibert | Best arm identification in multi-armed bandits[END_REF]. The extension of the latter to MDPs is called Best policy identification (BPI). The BPI problem refers to the problem of learning the best policy in an environment with unknown dynamics and/or unknown reward functions. The objective is to identify, as fast as possible, the policy that maximizes the expected cumulative reward. This is generally more challenging than BAI problem and various algorithms have been proposed to solve it. A key reference for best policy identification problem is the book [START_REF] Fiechter | Efficient reinforcement learning[END_REF]; Zanette, Kochenderfer, and Brunskill, 2019; Al Marjani and Proutiere, 2021; Wagenmaker, Simchowitz, and Jamieson, 2022). A close variant of the latter is Reward free reinforcement learning. This is where there is no explicit feedback from the environment, in other words no rewards. RF methods rely on some kind of internal signal, such as the agent's curiosity or the uncertainty of its beliefs, to guide exploration. [START_REF] Ménard | Fast active learning for pure exploration in reinforcement learning[END_REF] provides methods for the two settings of BPI and Reward-free learning and discuss the juxtaposed relationship of these concepts. Note that different kinds of pure exploration objectives exist in literature. For instance, thresholding bandits is a variation of the multi-armed bandit problem

Realistic RL: Open questions and promising prospects

where the goal is to identify the relative positions of the arms' expected rewards from a certain threshold [START_REF] Steinwart | A Classification Framework for Anomaly Detection[END_REF]; Locatelli, Gutzeit, and Carpentier, 2016; Chen, Lin, and Zhou, 2015; [START_REF] Tao | Thresholding bandit with optimal aggregate regret[END_REF]. Thresholding bandits are relevant when the agent seeks to find all arms that have a pertinent level of reward instead of just finding the best one.

Regarding the prominent algorithms for this type of objectives, they are essentially similar to the popular strategies for regret minimization. For example, Optimism in the face of uncertainty is used for Best policy identification and reward free exploration [START_REF] Ménard | Fast active learning for pure exploration in reinforcement learning[END_REF], Thompson Sampling was also proposed for a variety of exploration objectives (Russo, [START_REF] Russo | A Tutorial on Thompson Sampling[END_REF], and so on and so forth of previously introduced algorithmic concepts. A special kind of concepts is of greater interest in this setting, namely the Information-theoretic measures. These are used to quantify the amount of information gained by the agent when it explores different states or actions. One of the most widely used measures is the mutual information, which quantifies the degree of dependence between the state of the environment and the actions taken by the agent. Another commonly used measure is the conditional entropy, which quantifies the uncertainty of the agent's belief about the environment given its observations. Several works are based on this concept (Russo and Van Roy, 2014; Wagenmaker and Jamieson, 2022).

Realistic RL: Open questions and promising prospects

Reinforcement Learning research is a large body of literature and one of the largest subfield of machine learning. Realistic RL refers to the part of literature and techniques that can be applied to complex and realistic environments, such as robotics, autonomous vehicles, or at least to large-scale simulations mimicking the world.

The field of reinforcement learning (RL) has seen significant progress in both theoretical and practical research. The former focuses on understanding the fundamental properties of RL algorithms and developing provable guarantees for their performance. In contrast, practical RL focuses on designing implementable RL methods that handle real-world problems and environments, this often involve the use of deep learning techniques. However, there is often a disjunction between the theoretical and practical RL communities, with the former focusing on idealized problems and assumptions, and the latter focusing on the empirical performance of methods in real-world settings. This can make it challenging to find common ground, for this reason, there is a growing interest in developing theoretical frameworks that better match the settings and assumptions of realistic RL problems.

Literature review

Representation

In reinforcement learning, the representation refers to how the agent's state, actions, and rewards are represented internally. Representation can be a defining factor of the problem's hardness and the subsequent algorithm complexity.

One of the most popular choices for representation in RL is the tabular one, where the agent's state is represented as a table with entries for each possible state, and the agent's action and value functions are also represented as tables. This method can be very sample efficient in small state spaces, but becomes infeasible in large or continuous state spaces, for several reasons such as the curse of dimensionality, the lack of differentiability, and the difficulty of exploring and learning in high-dimensional spaces. The curse of dimensionality refers to the exponential increase in the number of states as the dimensionality of the space increases. Another challenge is the lack of differentiability, which makes it difficult to use traditional optimization techniques such as gradient descent. Finally, exploring and learning in high-dimensional continuous spaces can be difficult, as the agent might need a very large number of interactions with the environment to learn a good policy.

For all previous reasons and more, continuous representations are indispensable in RL, and their choice is an important factor that can have a significant impact on the performance, sample efficiency, and tractability of the RL algorithm.

In this context, a popular direction in RL is the adoption of function approximation [START_REF] Richard S Sutton | Policy gradient methods for reinforcement learning with function approximation[END_REF][START_REF] Francisco S Melo | An analysis of reinforcement learning with function approximation[END_REF], where the agent's state, action, and value functions are represented by a parameterized function. This allows the agent to generalize from past experience to new states, and is supposed to be much more sample efficient in large or continuous state spaces. The choice of representation highly depends on the problem domain and the sensors available. Another important aspect to take in account is the generalization ability.

Structures in bandits

Structured multi-armed bandits deal with the problem of balancing exploration and exploitation in decision making when the action space is structured. Linear, Lipschitz, uni-modal and generalized linear are different possible assumptions that are commonly made about the underlying reward function in order to solve bandit problems efficiently.

The most popular bandits structure, and relevant to our manuscript in the linear reward assumption (Abbasi-Yadkori, Pál, and Szepesvári, 2011; Abeille and Lazaric, 2017). Indeed, linear bandits assume that the reward function is linear with respect to the parameters of the action. This assumption allows for the use of linear regression techniques (Vovk, 1997)

Realistic RL: Open questions and promising prospects

to estimate the parameters of the reward function and make more informed decisions. An extension of this structure is the Generalized linear multi-armed bandits [START_REF] Filippi | Parametric bandits: The generalized linear case[END_REF], where rewards are modeled by a Generalized Linear Model. A GLM is a flexible framework generalizes linear regression by assuming the response variable to be related to the explanatory variable through a link function on top of a linear model [START_REF] Ashworth | Generalized linear models[END_REF].

Note that there are many other relevant structures in literature, in we refer the interested reader to the following papers (Degenne, Shao, and Koolen, 2020; Combes, Magureanu, and Proutiere, 2017) for some examples of structures bandit instances as well as the state-of the art algorithms. Overall, it is true that structural assumptions in bandits have their flaws, but they are for most very realistic as they can model real-life scenarios, and they are useful in different settings because they entail more efficient solutions.

Structures in MDPs

Structured MDPs are an extension of the standard MDP framework where the state, action, or reward spaces have some additional structure. Examples of structured MDPs include Factored MDPs, which assume that the state space can be represented as a combination of smaller, independent components. This assumption allows for the use of techniques such as dynamic programming to solve the MDP more efficiently. MDPs with Graph feedback are another example, in which the state-action space is represented as a feedback graph, i.e. a directed graph where each node has an associated reward function, and each edge has an associated transition probability. [START_REF] Dann | Reinforcement learning with feedback graphs[END_REF] considers this setting and relates it to the bandits with side observations structure [START_REF] Mannor | From bandits to experts: On the value of side-observations[END_REF].

There exist certain techniques that allows researchers to handle generic structures, this often consists of tracking the optimal allocation given by lower bounds [START_REF] Ok | Exploration in structured reinforcement learning[END_REF]. However, the oracle allocation is often intractable and only currently available for infinite-horizon settings, i.e. obtaining tight -even intractable-lower bounds is still an active area of research in the finite horizon setting. Indeed, the stochasticity of transitions is a significant hurdle in this setting compared to bandits, especially when the state-action spaces are continuous. Currently, optimal finite time problem-dependent bounds for MDPs are still elusive, the current state of the art in this direction is the work (Tirinzoni, Al-Marjani, and Kaufmann, 2022) for the tabular episodic MDPs. This shows that the MDP literature has still got multiple open questions and the continuous state-action spaces area is still very challenging even to the purely theoretical researchers.

Linear MDPs is a recent class of representation where the dynamics and rewards are described by linear functions [START_REF] Jin | Provably efficient reinforcement learning with linear function approximation[END_REF]. This class of MDPs has gained a significant popularity because it entails a closed-form solutions using linear algebra, and they are computationally tractable and efficient. Also, using dynamic programming, the value can be represented Literature review as a unique function which makes linear MDPs ideal for studying the convergence properties of the RL algorithms and analyzing their performance. However, while it is true that the linear structure allows for efficient algorithms, the applicability of this class of MDPs is unsatisfactory. Indeed, the main limitation of linear MDPs is in assuming that the transition and value functions are linear. For instance, their isn't to this day a single example of finite dimensional linear transition function beyond tabular MDPs. This limited expressiveness doesn't seem to capture the complex relationships induced by stochastic transitions.

Overall, in MDPs the popular structures often suffer two possible limitations. The first consists of assumptions that are realistic and applicable, yet they entail intractable solutions and complicated algorithms that can't be implemented in practice. The second category includes structures for which very efficient solutions exist, unfortunately they are very unrealistic and therefore not suitable for modeling real world problems.

Compelling complexity measures

In this section, we will delve into the complexities of various classes of Markov Decision Processes (MDPs) and their significance in the field of Reinforcement Learning (RL). We will discuss three measures of complexity: Bellman rank, bilinear classes, and Eluder dimension, and how they are used to understand the properties of different classes of MDPs. Furthermore, we will also explore how these complexities are shaping the future of RL by providing insights into the fundamental limits of solving MDPs, studying the performance of different algorithms, and guiding the development of new methods and techniques. Understanding these complexities is crucial for making progress in the field of RL and for solving more challenging and realistic problems.

One way to measure the complexity of an MDP is by using the Eluder dimension [START_REF] Russo | Eluder dimension and the sample complexity of optimistic exploration[END_REF], which measures the efficiency of predicting the value of actions not taken based on observed sample data. Eluder dimension has been used to study the complexity of various classes of MDPs, such as -possibly infinite-linear mixtures [START_REF] Ayoub | Model-based reinforcement learning with value-targeted regression[END_REF], and to provide general analyses for MDPs with small Eluder dimension (Wang, Salakhutdinov, and Yang, 2020; [START_REF] Ishfaq | Randomized Exploration in Reinforcement Learning with General Value Function Approximation[END_REF]. It has been shown that the Eluder dimension provides optimal bounds for linear MDPs and generalized linear MDPs, it was also shown that this complexity is exactly equivalent to the information gain for reproducing kernel Hilbert spaces.

Another way to measure the complexity of an MDP is by using the Bellman rank [START_REF] Jiang | Contextual decision processes with low bellman rank are pac-learnable[END_REF]. Prior to defining the latter, let's define the average Bellman of a function f at step h when following policy π. This error is defined as the expected Bellman error of f at step h, when all previous actions were taken according to π. Now, the Bellman rank is basically the uniform 2.5 Realistic RL: Open questions and promising prospects upper bound on the rank of all matrices formed by the Bellman errors of all possible functions f when following a greedy policy π f ′ of any possible function f ′ . In the case of tabular MDPs, it is simply the rank of the transition matrix matrix that represents the optimal value function. Bellman rank has been used to study the complexity of various classes of MDPs [START_REF] Dong | Root-n-regret for learning in markov decision processes with function approximation and low bellman rank[END_REF].

A third structural family is the Bilinear class of MDPs [START_REF] Du | Bilinear classes: A structural framework for provable generalization in rl[END_REF], which connects the Bellman error, which measures the sub-optimality of a value function, to a sum of bilinear forms. Additionally, it allows for the use of data from a past value function to estimate a bilinear form for all value functions within the class. This essentially entails that data can be reused to evaluate multiple functions. The bilinear class subsumes block MDPs [START_REF] Du | Provably efficient RL with rich observations via latent state decoding[END_REF], linear MDPs [START_REF] Jin | Provably efficient reinforcement learning with linear function approximation[END_REF], linear quadratic regulators, factored MDPs (Kearns and Koller, 1999) and Bellman rank.

In conclusion, understanding the complexities of different classes of MDPs is crucial for making progress in the field of RL and for solving more challenging and realistic problems. The Bellman rank, Eluder dimension, and Bilinear classes are three ways to measure the complexity of an MDP and to study the properties of different classes of MDPs. These complexities are shaping the future of RL by providing insights into the fundamental limits of solving MDPs, studying the performance of different algorithms, and guiding the development of new methods and techniques.

Tractable RL

Reinforcement Learning is powerful for solving decision-making problems, and other than its sample complexity issues, it also comes with several computational challenges.

In RL, the policy and transition dynamics of the system should be modeled and the corresponding parameters must be estimated. In many problems, the transition dynamics are represented by a parametric probability density function, which can be complex and highdimensional, making it difficult to estimate the parameters. This issue can be addressed by using various methods such as Gaussian Processes [START_REF] Christopher | Gaussian processes for machine learning[END_REF], Maximum Likelihood Estimation [START_REF] Levina | Maximum likelihood estimation of intrinsic dimension[END_REF], and Variational Inference (Blei, Kucukelbir, and McAuliffe, 2017) to estimate the parameters of the transition dynamics.

Another manifestation of the computational efficiency hurdles is the optimization problem involving bilevel optimization problems where the outer problem is to optimize the policy, and the inner problem is to optimize the value function. In fact, a bilevel linear optimization problem is NP-hard in general [START_REF] Robert | The polynomial hierarchy and a simple model for competitive analysis[END_REF]. Even in practice, such problems are known to be computationally challenging and may require specialized optimization techniques such as

Literature review

gradient-based methods (Sinha, Khandait, and Mohanty, 2020) and meta-learning [START_REF] Hospedales | Meta-learning in neural networks: A survey[END_REF] to solve.

A third computational tractability hurdle is feature learning. For instance, the state space is often high-dimensional, making it difficult to learn the features of the state space that are relevant for the problem at hand. Practical approaches to deduce features for a task include using representation learning methods such as autoencoders [START_REF] Vincent | Extracting and composing robust features with denoising autoencoders[END_REF] and variational autoencoders (Kingma and Welling, 2013) that can be used to extract the relevant features from the high-dimensional state space. These methods learn a lower-dimensional representation of the state space that can be used to solve the RL task. On a more theoretical aspect, recent efforts in RL include causality structures (Bareinboim, Forney, and Pearl, 2015; Lu, Meisami, and Tewari, 2022; Lattimore, Crammer, and Szepesvári, 2015) that allow efficient and theoretically grounded methods for feature learning. Finally, we would like to mention research efforts for learning representations that are independent of RL, especially theoretically motivated ones like [START_REF] Duvenaud | Structure discovery in nonparametric regression through compositional kernel search[END_REF][START_REF] Malkomes | Bayesian optimization for automated model selection[END_REF].

Finally, RL also comes with several less critical challenges. For example, the problem's horizon is often unknown and this can make the practitioner's job more challenging, it can however be usually solved by the doubling trick [START_REF] Cesa-Bianchi | How to use expert advice[END_REF][START_REF] Cesa | Prediction, learning, and games[END_REF]) and we believe this should be more popular in literature. Also, the variance information even in the simple case of bandits is assumed to be known prior to learning, which is not realistic, and this can also be mitigated by specific adaptive regularization techniques [START_REF] Durand | Streaming kernel regression with provably adaptive mean, variance, and regularization[END_REF]. Moreover, the bounds on variables are also important, e.g. for deriving confidence intervals, and they are usually assumed to be known. Overall, these limitations can lead to infeasible or unstable decisions, and using available information to the fullest extent is crucial for the success of RL algorithms. This involves effectively utilizing the information obtained from the environment and making assumptions that are applicable to a wide range of real life settings.

In conclusion, RL is a powerful method for solving sequential problems, but it also comes with several computational challenges. The complexity of parameter estimation, optimization problems, and the tractability of learning features are some of the main issues that need to be addressed in order to make progress in the field of RL. There are various methods in literature to address these challenges and to improve the performance of RL algorithms. However, despite these advances, there are still open problems and challenges that need to be addressed, and further research is needed to improve the scalability and efficiency of RL algorithms.

Chapter 3

Online sign identification for multi-armed bandits

In the fixed budget thresholding bandit problem, an algorithm sequentially allocates a budgeted number of samples to different arms (distributions). It then predicts whether the mean of each arm is larger or lower than a given threshold. We introduce a large family of algorithms (containing most existing relevant ones), inspired by the Frank-Wolfe algorithm, and provide a thorough yet generic analysis of their performance. This allowed us to construct new explicit algorithms, for a broad class of problems, whose losses are within a small constant factor of the non-adaptive oracle ones. Quite interestingly, we observed that adaptive methods empirically greatly out-perform non-adaptive oracles, an uncommon behavior in standard online learning settings, such as regret minimization. We explain this surprising phenomenon on an insightful toy problem. Online sign identification for multi-armed bandits

Introduction

In a stochastic multi-armed bandit problem, a decision maker sequentially samples from different distributions in order to optimize a loss that depends on the unknown parameters of those distributions. As a consequence, a trade-off arises between gathering more samples from any possible distribution (to enhance the estimation of relevant parameters) and optimizing the allocation to minimize the final loss. We can distinguish two main categories of losses, focusing on "exploitation" vs "exploration". The former directly depends on the whole allocation of samples and the typical example is regret minimization (we refer to the recent monographs (Lattimore and Szepesvári, 2020; Bubeck, Cesa-Bianchi, et al., 2012;[START_REF] Slivkins | Introduction to Multi-Armed Bandits[END_REF] that cover this setting almost exhaustively). The later is a bit different; after the budget of samples is exhausted, the algorithms must answer one or several "questions" (on the different distribution) and its loss is related to the number of mistakes made; the typical application being best-arm identification and variants [START_REF] Audibert | Best arm identification in multi-armed bandits[END_REF][START_REF] Kaufmann | On the complexity of best-arm identification in multi-armed bandit models[END_REF].

We investigate a class of pure exploration problems, called "thresholding bandit" [START_REF] Locatelli | An optimal algorithm for the thresholding bandit problem[END_REF][START_REF] Tao | Thresholding bandit with optimal aggregate regret[END_REF]. The key property of this class is that a question is asked about each distribution, and the probability of making a mistake decreases with the total information gathered on that distribution solely. The typical question the algorithm must answer is "is the mean of the distribution above or below some threshold?" (say, 0, for simplicity); giving the wrong answer can either incur a unit cost -independently from the distribution -, or a data-dependent cost (say, the distance to the threshold that represents the "risk" of that distribution). A typical application of thresholding bandits is crowdsourcing [START_REF] Chen | Statistical decision making for optimal budget allocation in crowd labeling[END_REF] where the objective is to distinguish workers with positive (vs. negative) efficiency; another one is bandit binary classification [START_REF] Jain | A New Perspective on Pool-Based Active Classification and False-Discovery Control[END_REF]. Some care must be taken when designing a performance criterion for a thresholding bandit problem, since any non-stupid algorithm will eventually answer all questions correctly (hence have a 0 loss) if it has enough samples. Furthermore, if distributions are sub-Gaussian (a rather mild assumption that we are going to make), the probability of making a single mistake decreases exponentially fast with the number of samples. As a consequence, the focus must be on controlling the exponential decay constant. We illustrate that issue on the unit cost problem described as follows. There are K different σ-sub-Gaussian distributions; the mean of distribution k is denoted by µ k and the (variance-normalized) gap of distribution k to the threshold 0 is denoted by

∆ k := |µ k |/ √ 2σ 2 .
The algorithm has a budget of T samples to (sequentially) allocate to those distributions and, based on the N k,T samples of distribution k, it must decide the sign of µ k ; any mistake has a cost of one. We denote by E k ∈ {0, 1} an indicator of a wrong sign prediction of µ k after exhausting the budget of T samples. The loss is

Introduction

then L 1 T := k E k .
It is not difficult to see that the expected number of mistakes could be of order K k=1 exp(-N k,T ∆2 k ) . In particular, sampling evenly across distributions (i.e. choosing N k,T = T /K) gives an expected loss

E[L 1 T ] ≈ k exp(-T K ∆ 2 k
), which has an exponential decay in T . However, this uniform allocation is far from being optimal in term of the exponential decay constant. Computing an (approximate) optimal fixed allocation in hindsight is not difficult: just optimize the upper-bound of E[L 1 T ]. Since even the uniform allocation has a loss decaying exponentially, the performance of an algorithm should be measured not with respect to E[L 1 T ] (see (Kaufmann, Cappé, and Garivier, 2016)) but rather in terms oflog(E[L 1 T ])/T . The oracle that uses knowledge of the gaps ∆ k to optimize its fixed allocation verifies

lim sup T →∞ 1 T log(E[L 1 T ]) ≤ - 1 k 1/∆ 2 k .
This unit cost framework has been investigated recently [START_REF] Tao | Thresholding bandit with optimal aggregate regret[END_REF] with a simple yet effective algorithm called LSA (Logarithmic-Sample Algorithm) designed exclusively for this problem; it samples the distribution with the smallest current index defined as αN k,t ∆2 k,t + log N k,t , where ∆k,t is the empirical estimate of ∆ k and α is some parameter to be chosen. LSA is "optimal up to a constant", but the constant is unfortunately in the exponential decay, as it was proved that 2 lim sup

T →∞ 1 T log(E[L 1 T ]) ≤ - 1 16020 1 k 1/∆ 2 k for LSA.
As we shall see, thanks to our new refined and general proof methodology, we can improve this result drastically (that implies choosing a totally different parameter α = 1 instead of 1/10 as suggested originally) without modifying the algorithm.

Contributions

We investigate the thresholding bandit problem with a weighted number of errors loss. Our contributions are twofold: 1) a generic method to design algorithms, with a generic proof, showing good performance on the weighted number of errors loss. 2) new lower-bounds and counter-intuitive results for the unit cost problem.

A generic algorithm with performance guarantees

We propose a Frank-Wolfe inspired method to design bandit algorithms. We develop a proof technique to obtain loss bounds for the type of algorithms that our method produces, which we apply to the thresholding bandit Online sign identification for multi-armed bandits with losses

L T = K k=1 a k E k or L ∆ T = K k=1 ∆ k E k , ( 3.1)
where (a k ) k∈[K] are known costs. The class of algorithms we analyze includes both LSA and APT (Anytime Parameter-free Thresholding) [START_REF] Tao | Thresholding bandit with optimal aggregate regret[END_REF][START_REF] Locatelli | An optimal algorithm for the thresholding bandit problem[END_REF]. We obtain precise non-asymptotic loss bounds for E[L T ]; for instance, we improve the original bound of LSA by a factor 4005 (and APT by a factor 8). More importantly, we get a new algorithm whose expected error for the unit cost problem is within a factor 4 of the oracle. We emphasize again than those "constant" factors are in the exponential (and are not mere multiplicative constants).

Interestingly, this class of algorithms are not driven either by the "optimism under uncertainty" principle, a standard technique in multi-armed bandit [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF] nor "Explore-then-commit / Successive Elimination" [START_REF] Perchet | Batched bandit problems[END_REF][START_REF] Even-Dar | Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems[END_REF].

New insights on the thresholding bandit problem

First, the optimal allocation provided by the oracle of [START_REF] Tao | Thresholding bandit with optimal aggregate regret[END_REF] in the unit cost problem has a M-shape (see Figure 3.1) because of two concurrent phenomena. On the one hand, the arms close to the threshold should not be pulled too much because their sign is difficult (if not impossible) to identify and it is a waste of budget. On the other hand, the signs of the arms far from the threshold are quickly well estimated and therefore should not be chosen too often either. The middle arms are the ones that need to be pulled the most frequently. As T gets larger, more and more budget is allocated to difficult arms. In Section 3.2.3, we provide a lower-bound that shows that this M shape is actually impossible to achieve for a sequential algorithm. Typically, the hollow inside of the M shape corresponds to arms whose sign cannot be well-estimated. In particular, it is not possible to distinguish arms that are very close to the threshold from the arms that are at the top of the M and should be pulled the most frequently according to the oracle.

Our second insight is corroborated by numerical simulations in Section 3.7. We show empirically that our algorithms not only match but also surpass the optimal non-adaptive sampling of the oracle. We conjecture that our algorithms take advantage of the chance due to noise that can move its estimate of the arm away from the threshold. In particular, when all the gaps ∆ k are equal, the non-adaptive optimal allocation should be uniform, which is significantly outperformed by adaptive algorithms. This suggests that adaptivity is crucial for this problem and may inspire future research directions to the multi-armed bandit community in order to prove theoretical guarantees for such phenomena.

Introduction

Additional related work

Zero-one loss Most of the literature on thresholding bandits (Locatelli, Gutzeit, and Carpentier, 2016; Mukherjee et al., 2017; Cheshire, Menard, and Carpentier, 2020) aims at minimizing the probability of making any sign error, i.e., minimizing the loss

L * T = I{∃k ∈ [K], E k = 1} = max k E k . (3.2)
We already mentioned the algorithm APT [START_REF] Locatelli | An optimal algorithm for the thresholding bandit problem[END_REF], that gets an exponential decay of that loss (variants include variance estimation (Zhong, Huang, and Liu, 2017) and/or delayed feedbacks). Other algorithms exist, but based on the optimism principle (Katz-Samuels and Scott, 2018; Mukherjee et al., 2017). Unfortunately they suffer from a degraded exponential decay constant (by a factor bigger than 1000).

Another part of the literature focuses on the fixed confidence framework, where the objective is to answer some questions with some fixed probability of mistake (and obviously with a minimal sample budget). For instance, an objective could be to return any arm above some threshold as soon as possible [START_REF] Kano | Good arm identification via bandit feedback[END_REF]Degenne and Koolen, 2019), or the one closest to the threshold [START_REF] Garivier | Thresholding bandit for dose-ranging: The impact of monotonicity[END_REF], or just identifying that one arm is above that threshold [START_REF] Kaufmann | Sequential test for the lowest mean: From Thompson to Murphy sampling[END_REF], or even to control false discovery rates and variants (Jamieson and Jain, 2018; Jain and Jamieson, 2019).

Global loss, dynamic allocation and outliers detection

The loss considered in thresholding bandits can be seen as a variant of a "global loss" (i.e., essentially non-linear) that has been extensively studied in the bandit literature [START_REF] Agarwal | Stochastic convex optimization with bandit feedback[END_REF]; Agrawal and Devanur, 2014; Mannor, Perchet, and Stoltz, 2014). However, the major difference is, again, that the optimal allocation is time dependent and that the loss converges exponentially fast to zero (no matter the algorithm). Similarly, Frank Wolfe algorithms have been introduced in this setting [START_REF] Berthet | Fast Rates for Bandit Optimization with Upper-Confidence Frank-Wolfe[END_REF][START_REF] Fontaine | Regularized contextual bandits[END_REF]; even though our algorithms share some similarities, they are intrinsically different for the same reasons.

Similarly, the problem investigated could be seen as a special case of bandit resource allocations (Koopman, 1953; Chen, Lin, and Zhou, 2015; Salehi et al., 2016; Devanur et al., 2019; Fontaine, Mannor, and Perchet, 2020) but where the loss is always decreasing with respect to the budget allocated per resource (hence again leading to a zero loss exponentially fast).

Finally the global objective of thresholding bandits is to obtain a synthetic view of how the means of distributions are spread on the real line (which ones are above/below some threshold). In that aspect, this problem sheds some similarities with outlier detection in multiarmed bandits (Katariya, Tripathy, and Nowak, 2019; Zhuang, Wang, and Wang, 2017; Zhu, Katariya, and Nowak, 2020).

Online sign identification for multi-armed bandits

Preliminaries

We describe here the weighted number of errors setting, in which an error on arm k has a known cost a k > 0. The sum-of-gaps setting will be briefly investigated in Section 3.5.3. The environment is composed of K > 1 arms and an algorithm sequentially pulls them. After pulling arm k ∈ [K], it observes a sample from a distribution ν k with mean µ k , and that sample is independent of past observations. The distribution ν k is supposed σ-sub-Gaussian, that is

∀λ ∈ R : E X∼ν k [exp(λ(X -µ k ))] ≤ exp(σ 2 λ 2 /2) .
The total number of rounds (and samples) T is known in advance and called the horizon. After pulling T arms, the task of the algorithm is to classify the arms depending on whether µ k > θ or not, where θ is a known threshold that we conveniently set to 0 (although it could be any other value, even different from arm to arm, without significant change to the analysis). Let

s k ∈ {-1, 1} be the sign of µ k -θ, equal to 1 iff µ k -θ > 0.
The algorithm returns for all arms an estimated sign ŝk ∈ {-1, 1}. The objective is to minimize the expected weighted number of missclassified arms, where a mistake on arm k has a known cost a k > 0,

L T = K k=1 a k I{ŝ k ̸ = s k } = K k=1 a k E k . ( 3.3)
Note that the linear form of the loss is quite general: since E k ∈ {0, 1}, any separable loss

k f k (E k )
is the sum of a constant and k a k E k for some costs a k . We conclude this description of the problem with notations used in the design of algorithms. Let N k,t and μk,t = 1 N k,t t s=1 I{i t = k}X t be the number of times the learner has pulled arm k up to round t (included) and the subsequent empirical mean of arm k repectively. Define

further ∆k,t = |μ k,t -θ|/ √ 2σ 2 and ∆ k = |µ k -θ|/ √ 2σ 2
, respectively the empirical and the true (variance-normalized) gap of arm k to the threshold after t rounds.

Lower bound for the expected number of mistakes

Following the proof of [START_REF] Tao | Thresholding bandit with optimal aggregate regret[END_REF] in a slightly more generic fashion (using exponential families with one parameter instead of Bernoulli distribution), we obtain a lower bound on the performance of any algorithm from which we get Theorem 3.1. [START_REF] Tao | Thresholding bandit with optimal aggregate regret[END_REF]) Let (∆ 1 , . . . , ∆ K ) be a sequence of gaps. Then for any algorithm and time horizon T ≥ K, there exists an instance in which all arms

Theorem 3.1. (Similar to Theorem 20 in

Preliminaries k ∈ [K] have Gaussian distributions with variance σ 2 and mean in {∆

k , -∆ k } such that E[L T ] ≥ 1 4 min k N k =T K k=1 a k e -4N k ∆ 2 k .
Proof. First, denote the expected loss on a bandit problem µ:

E[L T (µ)] = K k=1 a k P µ {ŝ k ̸ = s k }. For each arm k ∈ [K], define two values µ k , μk ∈ R, with µ k < θ < μk . Let µ = (µ k ) k∈[K] .
For some fixed one-parameter exponential family, we denote by KL(a, b) the Kullback-Leibler divergence between distributions with mean a and b.

Given a vector λ ∈ R K with λ k ∈ {µ k , μk } for all k ∈ [K] and S ⊆ [K], let λ S be such that λ k,S ∈ {µ k , μk } and λ k,S ̸ = λ k for k ∈ S and λ j,S = µ j for j / ∈ S.
For S ∈ P([K]), let S ± i be equal to S ∪ {i} if i / ∈ S and to S \ {i} otherwise. Also, we denote by (s k (λ)) be the signs of (λ k ). Then the following holds

sup S∈P([K]) E[L T (µ S )] ≥ 1 2 K S∈P([K]) E[L T (µ S )] = 1 2 K S∈P([K]) K k=1 a k P µ S {ŝ k ̸ = s k (µ S )} = 1 2 K+1 S∈P([K]) K k=1 a k P µ S {ŝ k ̸ = s k (µ S )} + a k P µ S±k {ŝ k ̸ = s k (µ S±k )} = 1 2 K+1 S∈P([K]) K k=1 a k P µ S {ŝ k ̸ = s k (µ S )} + a k P µ S±k {ŝ k = s k (µ S )}.
For each arm k, we can bound the sum of the two probabilities from below. Let

E k,S = {ŝ k ̸ = s k (µ S )}. P µ S (E k,S ) + P µ S±k (E k,S ) ≥ 1 2 exp (-E µ S [N k,T ] KL(µ k,S , µ k,S±k )) ,
so that, when plugged back in the previous equation, we get

sup S∈P([K]) E[L T (µ S )] ≥ 1 2 K+1 S K k=1 1 2 a k exp (-E µ S [N k,T ] KL(µ k,S , µ k,S±k )) ≥ 1 4 1 2 K S min N : k N k =T K k=1 a k exp (-N k KL(µ k,S , µ k,S±k ))
Online sign identification for multi-armed bandits

≥ 1 4 min N : k N k =T K k=1 a k exp (-N k max{KL(µ k , μk ), KL(μ k , µ k )})
We finish the proof by recalling that for Gaussians with variance

σ 2 , KL(a, b) = (a-b) 2 2σ 2 .
As can be seen from the proof above, the result holds more generally for one-parameter exponential-families where the gaps on the r.h.s are replaced with the Kullback-Leibler divergence.

Non-adaptive oracle

We now derive an optimal but unrealistic oracle, which requires prior knowledge of the gaps as input. Consider the algorithm that pulls each arm N k,T times, a number fixed in advance, then returns the sign of the empirical mean μk,T . Using Hoeffding's inequality, the expected loss verifies:

E[L T ] = K k=1 a k P ((μ k,T -θ)(µ k -θ) < 0) ≤ K k=1 a k e -N k,T ∆ 2 k (3.4)
We define the non-adaptive oracle as the allocation N T which minimizes that upper bound. Its error probability has the same form as the lower bound of Theorem 3.1, but has a different constant in the exponential (1 instead of 4). We can solve that minimization problem and make the error bound more explicit.

Lemma 3.2. Suppose that the arms are ordered such that a

1 ∆ 2 1 ≤ . . . ≤ a K ∆ 2 K .
There is a set S = {k 0 , k 0 + 1, . . . , K} and a constant C S such that the oracle non-adaptive algorithm has

N k,T = 0 for k / ∈ S and N k,T = C S + log(a k ∆ 2 k ) /∆ 2 k for k ∈ S (

see the proof below for details). The expected loss of that non-adaptive oracle is

E[L T ] ≤ k / ∈S a k + k∈S a k exp     - T + j∈S 1 ∆ 2 j log a k ∆ 2 k a j ∆ 2 j j∈S 1 ∆ 2 j     . (3.5)
Proof. The objective of the non-adaptive oracle section is to find an explicit solution of

min k N k =T k a k e -N k ∆ 2 k .
Introducing the Lagrange multiplier γ ∈ R, it is straightforward that the solution is such that all N k which are nonzero verify ∂ ∂N k ( j a j e -N j ∆ 2 j ) = γ. Then there exists a set S and a

Preliminaries

constant γ S (T ) > 0 for which k / ∈ S =⇒ N k = 0 and for k ∈ S, N k ̸ = 0 and a k ∆ 2 k e -N k ∆ 2 k = γ S (T ) . That is, N k = 1 ∆ 2 k (γ S (T ) + log(a k ∆ 2 k )) . We remark that k ∈ S iff 1 ∆ 2 k (γ S (T ) + log(a k ∆ 2 k )) > 0, which then implies that if a 1 ∆ 2 1 ≤ . . . ≤ a K ∆ 2 K , then S = {k 0 , k 0 + 1, . . . , K} for some k 0 ∈ [K]. Using the condition k N k = T to determine γ S (T ), we get K k=k 0 1 ∆ 2 k (γ S (T ) + log(a k ∆ 2 k )) = T =⇒ γ S (T ) = T + K k=k 0 1 ∆ 2 k log 1 a k ∆ 2 k K k=k 0 1 ∆ 2 k . Finally, we can characterize k 0 . Notice that k ∈ S iff 1 ∆ 2 k (γ S (T ) + log(a k ∆ 2 k )) > 0, i.e. iff 1 ∆ 2 k   T + K j=k 0 1 ∆ 2 j log 1 a j ∆ 2 j K j=k 0 1 ∆ 2 j + log(a k ∆ 2 k )   > 0 ⇔ T > K j=k 0 1 ∆ 2 j log a j ∆ 2 j a k ∆ 2 k . Finally, let H k = K j=k+1 1 ∆ 2 j log a j ∆ 2 j a k ∆ 2 k , with H 0 = +∞ and H K = 0. Then k 0 is the unique element of [K] such that H k 0 < T ≤ H k 0 -1 .
Note that the oracle is not pulling arms 1, . . . , k 0 -1. These are the arms which are too close to the threshold (in a distance weighted by a k ) and thus too hard to classify to be worth trying. Giving up on those arms is not something that a non-oracle algorithm can do.

µ k = (-1) k k 2 K 2
for K = 50 arms. [START_REF] Chen | Combinatorial multi-armed bandit: General framework and applications[END_REF], and T = 500) the shape of the optimal allocation (arms near the threshold should not be drawn) as well as the empirical sampling distributions of several algorithms that pull all arms. Next, we illustrate how this optimal allocation evolves with the horizon T . Figures 3.2 and 3.3 represent the optimal non-adaptive sampling distribution if the means were known and the empirical sampling distribution of the algorithms for different numbers of iterations. As we can see, for the initial phase, the arms that are closest to the threshold should ideally not be drawn. Yet, as we will see in the next section, this is not possible for sequential algorithms. All arms must be sampled. We can see that this is indeed the case for all algorithms: the closer the arms are to the threshold, the more likely they are to be sampled.

k = (-1) k (k/K) 2 , k = 1, . . . ,

Preliminaries

A good algorithm must pull all arms

We provide a new lower bound for the thresholding bandit with unit-cost problem, to support the claim that it is not possible to avoid pulling the arms which are close to the threshold. Consider the following 4 Gaussian bandit models (with variances 1) with means

µ +ε = (ε, . . . , ε, µ K 0 +1 , . . . , µ K ) , µ ′ + = (µ K 0 +1 , . . . , µ K 0 +1 , µ K 0 +1 , . . . , µ K ) , µ -ε = (-ε, . . . , -ε, µ K 0 +1 , . . . , µ K ) , µ ′ -= (-µ K 0 +1 , . . . , -µ K 0 +1 , µ K 0 +1 , . . . , µ K ) .
where

0 < ε < µ K 0 +1 ≤ . . . ≤ µ K , the value µ K 0 +1
is large enough for the oracle to pull all arms on µ ′ + and ε ≤ log(2)/(2T ).

Lemma 3.3. If E μ[L T ] ≤ c 1 min k N k =T k e -c 0 N k ∆ 2 k for constants c 0 , c 1 on μ ∈ {µ ′ + , µ ′ -}, then max µ∈{µ +ε ,µ -ε } E µ   K 0 k=1 N k,T   ≥ 1 2(µ K 0 +1 -ε) 2 c 0 T + H log H + log K 0 32c 1 H .
where

H = K 0 ∆ 2 K 0 +1 + K k=K 0 +1 1 ∆ 2 k and H log = K 0 ∆ 2 K 0 +1 log 1 ∆ 2 K 0 +1 + K k=K 0 +1 1 ∆ 2 k log 1 ∆ 2 k .
The proof idea is that if an algorithm has an expected loss close to the loss of the non-adaptive oracle, then it must pull linearly the arms which are close to the threshold.

Proof. We will prove that the number of pulls of arms 1, . . . , K 0 cannot be too small. Formally, let n ε = K 0 k=1 E µ +ε N k,T be the expected number of pulls under µ +ε of the arms with mean ε. We aim at showing that that number cannot be zero. We first prove that

P µ +ε (L T > K 0 /2) ≥ 1 4 . ( 3.6)
This follows from the basic inequalities,

P µ +ε (L T (µ +ε ) > K 0 /2) + P µ -ε (L T (µ +ε ) ≤ K 0 /2) ≥ 1 2 e -nε KL(ε,-ε) ≥ 1 2 e -2ε 2 T
In particular, for ε

≤ log 2 2T , max{P µ +ε (L T (µ +ε ) > K 0 /2), P µ -ε (L T (µ -ε ) > K 0 /2)} ≥ 1 4
, and either Inequality 3.6 either holds for µ +ε , or we just need to switch the role of ε and -ε in this proof. Suppose now that Inequality 3.6 holds for µ +ε .

Online sign identification for multi-armed bandits

The Kullback-Leibler divergence between µ +ε and µ

′ + is k E µ [N k,t ] KL(µ k,+ε , µ ′ k,+ ) = K 0 k=1 E µ [N k,t ] KL(ε, µ k 0 +1 ) ≥ kl(P µ +ε (L T > K 0 /2), P µ ′ + (L T > K 0 /2)) ≥ kl( 1 4 , 2 K 0 E µ ′ + [L T ]) ≥ 1 4 log K 0 2E µ ′ + [L T ] -log 2 ,
We have proved that

n ε ≥ 1 KL(ε,µ K 0 +1 ) ( 1 4 log K 0 2E µ ′ + [L T ] -log 2)
and the final result is obtained by using the explicit form for the bound on

E µ ′ + [L T ].

An interesting class of algorithms

We introduce and analyse a new class of algorithms for the thresholding bandit problem that we call index-based algorithms. That class unifies several existing algorithms, including APT (Locatelli, Gutzeit, and Carpentier, 2016) and LSA [START_REF] Tao | Thresholding bandit with optimal aggregate regret[END_REF].

A generic algorithm

An index-based algorithm pulls the minimum of K quantities, one for each arm, that each depends only on the rewards and pull counts of the respective arm (it does not change when pulling other arms). In particular, we consider algorithms for which the sampled arm is

i t+1 ∈ arg min k∈[K] F N k,t , N k,t ∆2 k,t ; a k for a function F : N × R + × R * + →
R that depends on the pull counts, the information about the sign and the weight of the arm. Algorithm 3.1: Index-based algorithm for thresholding bandit

1 Inputs: an index function F : N × R + × R * + → R; a 1 , . . . , a K ∈ R * + ; σ > 0; and θ ∈ R 2 For t = 1, . . . , T do -for all k ∈ [K] define N k,t-1 = t-1 s=1 I{k = i s }, μk,t-1 = 1 N k,t-1 t-1 s=1 I{k = i s }X s , ∆2 k,t-1 = 1 2σ 2 μk,t-1 -θ 2 -pull i t ∈ arg min k∈[K] F N k,t-1 , N k,t-1 ∆2 k,t-1 ; a k . -observe X t ∼ ν it Define t max = max t∈[T ] min k∈[K] F N k,t , N k,t ∆2 k,t ; a k Return for each k ∈ [K] the sign ŝk = sign(μ k,tmax -θ)

An interesting class of algorithms

After T rounds, the algorithm recommends the sign of the arms at the round

t max ∈ [T ] at which min k∈[K] F N k,t , N k,t
∆2 k,t ; a k was maximal. This rule is used as opposed to returning the sign of all arms at time T to facilitate the analysis, which is based on the observation that there is a small probability of error when all arms have high index. The time t max should be close to T : in particular, only one arm is sampled (possibly several times) between t max and T (see Appendix A.1). In Section 3.4, we provide a generic analysis for index-based algorithms satisfying the assumption below.

Assumption 3.4. The index function

F (n, x; a) : N × R + × R * + → R is non-decreasing in n and x and lim n→+∞ F (n, ny; a) = +∞ for all y > 0, a > 0.
Intuitively, algorithms that verify Assumption 3.4 prefer pulling arms that were pulled the least (smallest n) and whose quantity of information about the sign (n ∆2 k,n ) is small. This class includes several algorithms from the thresholding bandits literature: APT (Locatelli, Gutzeit, and Carpentier, 2016) for F (n, x; a k ) = x and LSA [START_REF] Tao | Thresholding bandit with optimal aggregate regret[END_REF] for F (n, x; a k ) = x + log(n) (these algorithms are only defined for a k = 1). We now propose a generic method for designing an index-based algorithm.

Frank-Wolfe for Thresholding bandits

Our strategy to minimize the expected loss is inspired by the Frank-Wolfe algorithm [START_REF] Frank | An algorithm for quadratic programming[END_REF]) and aims at controlling an upper-bound on the loss, such as the right hand side of Inequality (3.4). Let's write that function as

B(N T ) = K k=1 a k e -N k,T ∆ 2 k
. The high-level idea is to sequentially estimate its gradient and move to the minimizer of its linear approximation. If the gaps were known, we could compute at time t + 1 the gradient of the bound with respect to N t ,

∇B(N t ) = (-a k ∆ 2 k e -N k,t ∆ 2 k
) k and use the Frank-Wolfe algorithm. The algorithm would pull i t+1 ∈ arg min u u ⊤ ∇B(N t ) for u in the simplex, which is simply arg min

k∈[K] (-a k ∆ 2 k e -N k,t ∆ 2 k
). The gaps are however unknown. We therefore compute an estimate of the gaps ∆k,t , with which we form the estimated gradient

∇B(N t ) k = -a k ∆2 k e -N k,t ∆2 k,t = -exp -N k,t ∆2 k,t -log(N k,t ∆2 k,t ) + log N k,t a k .
This gives a natural choice for the index function of our algorithm F (n, x; a k ) = x -log x + log(n/a k ). However, the latter is decreasing in x for x ∈ (0, 1), which in addition to violating Assumption 3.4, may lead to instability in the initial phase when the gaps ∆ k are poorly estimated by ∆k,n . We therefore propose a slight modification that preserves the asymptotic behavior of F and we call the resulting algorithm FWT (Frank-Wolfe for Thresholding bandits):

F (n, x; a k ) = max{x, 1} -log(max{x, 1}) + log(n/a k ) . (FWT)

Online sign identification for multi-armed bandits

Recovering APT Using different upper-bounds B on the expected loss may lead to different algorithms. In particular, we highlight a link between our Frank-Wolfe inspired method and the APT algorithm of (Locatelli, Gutzeit, and Carpentier, 2016), which was designed to minimize the loss

L T = K k=1 a k I{ŝ k ̸ = s k } = K k=1 a k E k .
Following our method with the choice B(N t ) = max k∈[K] e -N k,t ∆ 2 k results in exactly the same sampling rule as the one of the APT algorithm (the recommendation rule differs slightly since we recommend the sign at t max and not at T ). Indeed, the derivative of B with respect to N k,t is nonzero (and negative) if and only if N k,t ∆ 2 k = arg min j N j,t ∆ 2 j (ignoring the case in which there are several argmins, for which the tie breaking can be arbitrary). This leads to the choice

F (n, x; a k ) = x in Algorithm 3.1, which then pulls i t+1 = arg min k∈[K] N k,t
∆2 k . This is the sampling rule of APT.

Loss upper bound

We provide a loss upper bound that is valid for all index-based algorithms verifying Assumption 3.4. We then give a summary of the analysis outline and the resulting loss bounds.

Theorem 3.5. Let K ≥ 1, a 1 , . . . , a K > 0, T ≥ 1, and σ > 0. Let F : N × R × R * + → R that satisfies Assumption 3.4. Let C 1 , . . . , C K > max k F (0, 0; a k ). For all j, k ∈ [K], define • t j (C k ) a solution of the equation F (t, t∆ 2 j ; a j ) = C k , • S k ⊆ [K] and t j,0 (C k ) ∈ R + , a set and values such that for i / ∈ S k , P ∃n ≤ t i,0 (C k ), F (n, n ∆2 n,i ; a i ) ≥ C k = 1
. Then the expected loss of Algorithm 3.1 is upper-bounded as

E[L A T ] ≤ K k=1 a k   e • exp   - 1 2 T -j / ∈S k t j,0 (C k ) -j∈S k t j (C k ) j∈S k 1/∆ 2 j   + T • e -t k (C k )∆ 2 k   .
Before diving into the proof, we give a high-level intuition. The analysis is composed of two parts:

1. First we establish that for any arm j ∈ [K], with large probability, there is a time

τ j (C k ) such that F (τ j (C k ), τ j (C k ) ∆τ j (C k ),j ; a j ) ≥ C k . We prove that for all j, k ∈ [K], τ j (C k ) has

Loss upper bound

an exponential tail then use the fact that the algorithm pulls the minimal index to control the probability that the minimum never reaches C k .

2. We show that if an arm's index is large, then the probability of mistake on it is small.

Remark 3.6. The times t j (C k ) of Theorem 3.5 are the smallest numbers of samples such that

t j (C k ) ≥ τ j (C k )
with high enough probability. By determining those times, we derive explicit bounds for algorithms that verify Assumption 3.4.

We note that our analysis is valid for a more general class than the one studied in this paper. Indeed, 1) we analyze index functions slightly more general than F (n, x; a k ). Each arm has a potentially different index function F k (n, x), and 2) we consider algorithms which obey -a slightly more generic-Assumption 3.4, i.e. whose index can be written as

I k n = F k (n, n ∆2 n,k ),
where each F k is non-decreasing in both variables, and lim n→+∞ F k (n, ny) = +∞ for y > 0.

Proof roadmap: Set

(C k ) 1≤k≤K ∈ R + , so that it immediately follows E[L T ] = k a k P(ŝ k ̸ = s k ) ≤ K k=1 a k P(ŝ k ̸ = s k ∧ ∃T ′ ∈ [T ], I k N T ′ ≥ C k ) + a k P(∀T ′ ∈ [T ], I k N T ′ < C k ) ≤ K k=1 a k P(ŝ k ̸ = s k ∧ ∃T ′ ∈ [T ], I k N T ′ ≥ C k ) + a k P(F C k ) ,
where E stands for the complement of an event E.

The proof proceeds in two steps, which are proved in Section 3.4.1 and Section 3.4.2, in order to control both probabilities introduced above:

1. Lemma 3.8: with large probability, there is some time t for which the index I k N k,t is large for all k ∈ [K] (and all arms are well explored) 2. Theorem 3.10: if I k N k,t is large, then there is a small probability of mistake for arm k.

With large probability, all arms are well explored

We know from Lemma A.3 that any index based algorithm verifies

F C = { k τ k (C) ≤ T }.
Hence, to prove that F C happens with great probability, it suffices to show that k τ k (C) has an exponential tail.

We derive a bound on P( k τ k (C) > T ). To that end, we bound individually for each arm

P(τ k (C) ≥ t k + x)
for some t k to be defined and x ≥ 0, and conclude by Corollary A.7.

Online sign identification for multi-armed bandits Theorem 3.7. Under Assumption 3.4, the algorithm verifies, for all x ≥ 0,

P( τ k (C) > t k (C) + x) ≤ exp(-∆ 2 k x 2 ) , with t k (C) solution to F k (t, t∆ 2 k ) = C, if such a solution exists. Otherwise, if C < F k (0, 0) and no solution exists, τ k (C) = 0 with probability 1.
Proof. We first bound P(τ k (C) > t k + x), where t k is chosen later, and x ≥ 0.

P(τ k (C) > t k + x) = P(∀n ≤ t k + x, I k n < C) ≤ P(I k t k +x < C) = P(F k (t k + x, (t k + x) ∆2 k ) < C) First, by monotonicity of F k , this probability equals zero if C ≤ F k (t k + x, 0). If C > F k (t k + x, 0), we define δ t k +x,k,C such that ∆ δ t k +x,k,C t k +x,k = inf{∆ ≥ 0 | C < F k (t k + x, (t k + x)∆ 2 )}. In the following, we write F k (n, •) -1 (C) := inf{x | C ≤ F k (n, x)}. If F k (n, •) is increasing, this is its inverse, but we only suppose that F k (n, •) is non-decreasing. Note that x < F k (n, •) -1 (C) implies that F k (n, x) < C. With that definition, (t k + x)(∆ δ t k +x,k,C t k +x,k ) 2 = F k (t k + x, •) -1 (C).
As a consequence, we get

P(τ k (C) > t k + x) ≤ P(I k t k +x < C) = P(F k (t k + x, (t k + x) ∆2 t k +x,k ) < C) ≤ P((t k + x) ∆2 t k +x,k < F k (t k + x, •) -1 (C)) = P( ∆2 t k +x,k < (∆ δ t k +x,k,C t k +x,k ) 2 ) ≤ δ t k +x,k,C ,
where by definition,

δ t k +x,k,C = exp -(t k + x)(∆ k -∆ δ t k +x,k,C t k +x,k ) = exp - ∆ 2 k (t k + x) -F k (t k + x, •) -1 (C) 2 .
We intend to prove an exponential decrease with x. In order to have it, we will set t k such that the exponential is equal to 1 for x = 0, and then decreases as x grows.Let then t k be such that

C ≤ F k (t k , t k ∆ 2 k ).
It exists as soon as C ≥ F k (0, 0) (where the later is non-positive for specific algorithms we will consider). For all

t ≥ t k , F k (t, t∆ 2 k ) ≥ F k (t k , t k ∆ 2 k ) ≥ C, which leads to ∆ 2 k t -F k (t, •) -1 (C) ≥ 0. Note that since F k is non-decreasing in the first variable we have F k (t + x, •) -1 (C) ≤ F k (t, •) -1 (C)
for all t, x ≥ 0, and

δ t k +x,k,C = exp - ∆ 2 k (t k + x) -F k (t k + x, •) -1 (C) 2

Loss upper bound

≤ exp - ∆ 2 k (t k + x) -F k (t k , •) -1 (C) 2 ≤ exp -∆ 2 k √ t k + x - √ t k 2 .
Let Y k = max( τ k (C), √ t k ); we have proved that for all x ≥ 0,

P(Y k > √ t k + x) ≤ exp -∆ 2 k √ t k + x - √ t k 2 .
By setting

x = 2 √ λt k + λ for λ ≥ 0, we get P(Y k > √ t k + λ) ≤ exp -∆ 2 k λ 2 . Lemma 3.8. For all C > 0, t k such that F (t k , t k ∆ 2 k ) ≥ C. P( k τ k (C) ≥ T ) ≤ e × exp - T 2 -k t k k 1/∆ 2 k . Proof. Rewrite the event { k τ k (C) > T } using Y k = max( τ k (C), √ t k )
, so that:

P k τ k (C) > T = P k ((Y k - √ t k ) + √ t k ) 2 > T ≤ P 2 k (Y k - √ t k ) 2 + 2 k t k > T ≤ P k (Y k - √ t k ) 2 > T /2 - k t k . (3.7)
We now apply Lemma A.6 to

(Y k - √ t k ) 2 , which verifies P((Y k - √ t k ) 2 ≥ x) ≤ exp(-∆ 2 k x). From Equation (3.7), we obtain P( k τ k (C) > T ) ≤ P( k (Y k - √ t k ) 2 > T /2 - k t k ) ≤ e × exp - T /2 -k t k k 1/∆ 2 k .
Remark We can actually derive a tighter bound than (3.7)

P( k τ k (C) > T ) = P( k ((Y k - √ t k ) + √ t k ) 2 > T ) ≤ P      k (Y k - √ t k ) 2 + k t k   2 > T    = P   k (Y k - √ t k ) 2 > ( √ T - k t k ) 2   .

Online sign identification for multi-armed bandits

Roughly speaking, to get it, just write

∥(Y - √ t) + √ t∥ 2 ≤ (∥Y - √ t∥ + ∥ √ t∥) 2 . To get Equation (3.7), we further use (∥Y - √ t∥ + ∥ √ t∥) 2 ≤ 2∥Y - √ t∥ 2 + 2∥ √ t∥ 2 .
In the case of APT (at least) it leads to the same final bound on the algorithm because when we optimize further down, we set k t k = T /4 no matter which of these inequalities we use, value for which resulting exponents are equal. Corollary 3.9. Suppose now that there is a set S C such that for k / ∈ S C , P(τ k (C) > t k ) = 0. Then we can refine Lemma 3.8 to

P( k τ k (C) > T ) ≤ P( k∈S C τ k (C) > T - k / ∈S C t k ) ≤ e × exp - (T -k / ∈S C t k )/2 -k∈S C t k k∈S C 1/∆ 2 k .

When an arm index is large, the probability of mistake is small

The goal of this section is to bound

P(ŝ k ̸ = s k ∧ ∃T ′ ∈ [T ], I k N T ′ ≥ C). We define the random variable E k,t = I{(μ k,t -θ)(µ k -θ) < 0}; it is equal to 1 iff
there is an error on the sign at time t. The algorithm makes a mistake on arm k is E k,tmax = 1 since it returns the sign at that time.

Theorem 3.10. The algorithm using F k for its index definition verifies

P(ŝ k ̸ = s k ∧ ∃T ′ ∈ [T ], I k N T ′ ≥ C) ≤ P(∃T ′ ∈ [T ], E k,T ′ = 1 ∧ I k N T ′ ≥ C) ≤ T • inf{e -n k ∆ 2 k | F k (n k , n k ∆ 2 k ) < C} .
Proof. We use Lemma 3.13 (below): find n k as large as possible such that

F k (n k , n k ∆ 2 k ) < C.
Then, since the algorithm returns the sign of the arm at the time at which its index was maximal, we get

P(ŝ k ̸ = s k ∧ ∃T ′ ∈ [T ], I k N T ′ ≥ C) ≤ P(∃T ′ ∈ [T ], E k,T ′ = 1 ∧ I k N T ′ ≥ C) ≤ T • e -n k ∆ 2 k .
Lemma 3.11. For any δ k ∈ (0, 1), with probability at least 1 -δ k , and for all n ∈ [T ], it holds

N k,t (μ k t -µ k ) ≤ log T δ k .
Proof. This is a direct implication of Hoeffding's inequality with a union bound for timeuniformity.

Define

n k = 1 ∆ 2 k log T δ k .
Consider the following three facts (their definition will be useful for the following proofs):

1. If the concentration holds, then N k,t (μ k t -µ k ) ≤ √ n k ∆ k .
2. If there is a mistake at time t, then we have

3.5 Examples (a) μk t -µ k ≥ ∆ k . (b) (μ k t -µ k ) 2 ≥ ∆2 N k,t ,k + ∆ 2 k . 3. If I k N k,t > C then F k (N k,t , N k,t ∆2 t,k ) > C and N k,t ∆2 t,k ≥ F k (N k,t , •) -1 (C).
Lemma 3.12. If at time t, concentration holds and there is a mistake (1 and 2 are true), then

N k,t ≤ n k and F k (N k,t , (n k -N k,t )∆ 2 k ) ≥ I k N k,t .
Proof. First point: combine 1 and 2(a). Second point: use 1, then 2(b), then the definition of

I k N k,t : n k ∆ 2 k ≥ N k,t (μ k t -µ k ) 2 ≥ N k,t ∆2 t,k + N k,t ∆ 2 k ≥ F k (N k,t , •) -1 (I k N k,t ) + N k,t ∆ 2 k .
Lemma 3.13. If at time t, all three "if" are true, then

F k (n k , n k ∆ 2 k ) ≥ C.
Proof. Use the monotonicity of F k in the inequality of Lemma 3.12.We have

N k,t ≤ n k and n k -N k,t ≤ n k .

Examples

Here we explicit the earlier theorem for some previously existing algorithms in this setting as well as our newly proposed one. Then, we provide a numerical comparison of the different bounds so as to provide an intuition to the reader interesting in comparing them.

Explicit bounds for FWT, LSA and APT APT ((Locatelli, Gutzeit, and Carpentier, 2016))

We analyze the variant of this algorithm that returns the sign at the time t max when the minimal index was maximal.

Corollary 3.14. Suppose that for all

k ∈ [K], a k = 1. For all T ∈ N * , E[L APT T ] ≤ 2K √ e • T • exp - 1 4 T K j=1 1/∆ 2 j . Since max k E k ≤ k E k , the bound of Corollary 3.
14 is also a bound on the zero-one loss, which we can compare to the result of (Locatelli, Gutzeit, and Carpentier, 2016). Our result shows a 1/4 factor in the exponential instead of the worse 1/32 constant of the original paper.

Online sign identification for multi-armed bandits

Proof. This algorithm (in its variant that stops at t max ) corresponds to F (n, x) = x. To apply Theorem 3.5 we find that t

j (C k ) = C k ∆ 2 j is solution, then: E[L T ] ≤ K k=1 ea k exp - T /2 K j=1 1/∆ 2 j -C k + T • a k exp (-C k )
An optimal C k is such that e • exp -

T 2 K j=1 1/∆ 2 j + C k = T • exp(-C k ).
Then the bound becomes:

E[L T ] ≤ 2 √ e • T K k=1 a k exp -1 4 T K k=1 1/∆ 2 k .

LSA ((Tao et al., 2019))

This algorithm corresponds to F (n, x) = x + log n, we provide the following bound, which is significantly tighter than the one provided by its inventors.

Corollary 3.15. For all

T ∈ N * , E[L LSA T ] ≤ K k=1 e • a k exp   -   T 2 -K j=1 1 ∆ 2 j W (∆ 2 j exp(C k )) K j=1 1/∆ 2 j     + T a k exp -W (∆ 2 k exp(C k )) . Furthermore, if ∀k, j C k + log ∆ 2 j ≥ 1 we obtain: E[L LSA T ] ≤ K k=1 (1 + e) • a k exp   -   T 2 + K j=1 1 ∆ 2 j (log 1 ∆ 2 j + log(C k + log ∆ 2 j )) K j=1 1/∆ 2 j   + C k   + K k=1 T (C k + log ∆ 2 k ) ∆ 2 k a k exp (-C k ) . (3.8)
Proof. The stopping time t j (C k ) is solution to t∆ 2 j + log t = C k . This equation has a closed form solution:

t j (C k ) = 1 ∆ 2 j W (∆ 2 j exp(C k ))
, where W is the Lambert W function, and this entails the first bound.

Then, we can use an inequality on the Lambert function (cf Corollary 2.4 in [START_REF] Hoorfar | Inequalities on the Lambert W function and hyperpower function[END_REF])), For all x ≥ e we have

log x -log log x ≤ W (x) ≤ log x -log log x + log 1 + e -1 ,
this entails that if ∀k, j C k + log ∆ 2 j ≥ 1 we obtain the second more specific bound.

Examples

FWT (our algorithm) This algorithm corresponds to

F k (n, x) = max(1, x) -log max(1, x) + log n -log a k .
Using our generic analysis, we are able to prove the following result about FWT.

Corollary 3.16. Let S, S ′ be two sets with

S ′ ⊆ S ⊆ [K] and let C ∈ R be such that C ≥ 1 + max k∈S log 1 a k ∆ 2 k . Then, for all T ≥ 1 E[L FWT T ] ≤ k / ∈S ′ a k + e k∈S ′ a k exp   - 1 2 (T -j / ∈S a j e C-1 ) + j∈S 1 ∆ 2 j log 1 a j ∆ 2 j j∈S 1/∆ 2 j + C   + T k∈S ′ a k exp -C + log(1/(a k ∆ 2 k )) .
Moreover, if we have

T ≥ 2 K j=1 1 ∆ 2 j (2 + log a j ∆ 2 j max i a i ∆ 2 i (min k a k ∆ 2 k ) 2 -log T e 3 )
, we get the bound

E[L T ] ≤ 2 √ eT k a k exp     - 1 2 T /2 -j 1 ∆ 2 j log a j ∆ 2 j a k ∆ 2 k j 1/∆ 2 j     .
Up to a 1/4 factor, this is the exponent of the optimal non-adaptive oracle (cf Equation 3.4)

Proof. In order to find the times t j (C k ) of Theorem 3.5 we solve the equation:

max(1, n∆ 2 k ) -log max(1, n∆ 2 k ) + log(n∆ 2 k ) = C + log a k + log ∆ 2 k . Let D k = C + log a k + log ∆ 2 k .
We want a solution to max(1, x) -log max(1, x) + log x = D k . The function on the left, which we now denote by I, is increasing and bijective from R + to R.

• If D k ≥ 1, I(D k ) = D k . • If D k ≤ 1, I(e D k -1 ) = D k . • For all D k > 0, I(e D k -1 ) ≥ D k . Moreover, we have F k (a k e C-1 ) ≥ C, it comes P(τ k (C) > a k e C-1 ) = 0. Let C k > 0 and S k ⊆ {j ∈ [K] | C k + log a j ∆ 2 j ≥ 1}. Let S ′ be a set such that for all k ∈ S ′ , k ∈ S k . E[L T ] ≤ k / ∈S ′ a k + k∈S ′ a k P(F C k ) + k∈S ′ a k P(E k (T ) ∩ F C k ) ≤ k / ∈S ′ a k + e k∈S ′ a k exp   - 1 2 (T -j / ∈S k a j e C k -1 ) -j∈S k 1 ∆ 2 j log a j ∆ 2 j j∈S k 1/∆ 2 j + C k  
Online sign identification for multi-armed bandits

+ T k∈S ′ a k exp -C k -log a k ∆ 2 k .
Large T : The value of C k which equalizes the two terms indexed by k verifies

C k = 1 2 1 2 (T -j / ∈S k a j e C k -1 ) -j∈S k 1 ∆ 2 j log a j ∆ 2 j j∈S k 1/∆ 2 j - 1 2 log a k ∆ 2 k + 1 2 log T e .
The latter can be chosen if T is big enough such that S k = [K] for all arms, this is the case if

T ≥ 2 K j=1 1 ∆ 2 j (2 + log a j ∆ 2 j max i a i ∆ 2 i (min k a k ∆ 2 k ) 2 -log T e 3
), this entails the second bound of the corollary. General T : We choose a set S ⊆ [K] and set C k = C, a common value still to be determined, for all k ∈ S ′ . Then for all k ∈ S ′ , we set S k = S. We impose C ≥ 1 + max j∈S log 1

a j ∆ 2 j , such that the condition S k ⊆ {j ∈ [K] | C k + log a j ∆ 2 j ≥ 1} is verified.
This finished the proof for the first bound of the corollary. Remark 3.17. Note that the bounds for LSA and FWT are actually very similar, indeed Theorem 3.5 applies to LSA and FWT with the following times:

• LSA: t j (C k ) = W (e C k ∆ 2 j )/∆ 2 j and t j,0 (C k ) = e C k , • FWT: t j (C k ) = log(e C k a j ∆ 2 j )/∆ 2 j and t j,0 (C k ) = a j e C k -1
, Therefore, for the two algorithms, the times t j (C k ) are close (equal up to the log log terms in W ), thus their bounds are close as well. Note that LSA is only defined for a j = 1 for all j.

We highlight a notable property, in the regime where T is sufficiently large, FWT recovers the same exponent as in the non-adaptive oracle loss bound (3.5) (up to a factor 1/4). In the same regime of large T , the bound that we obtain for LSA is of the same order, but less explicit due to the function W . The latter is still impressive since the original theorem of [START_REF] Tao | Thresholding bandit with optimal aggregate regret[END_REF] for LSA exhibits an exponent significantly looser, of order exp -1 16020

T K j=1 1/∆ 2 j ,
i.e. 4005-times worse than our bound. We finally derive a bound for our newly introduced algorithm. )) (LSA), and Corollary 3.16 (FWT) for the particular case ∆ i = (i/K) 2 and a i = 1, for i = 1, . . . , K = 50. We can see that while the bounds of LSA, APT, and FWT are asymptotically similar, that of FWT starts to be significant for much smaller T . On the right, we can see the importance of the set S ′ in Corollary 3.16: the bounds first ignores all the arms, and suffers a

Numerical comparison

∆ i = (i/K) 2 . [right]
Evolution over time of the size of the optimal sets S (blue) and S ′ (orange) that minimize the bound of Corollary 3.16.

loss of 1 and then adds them one by one as soon as they can be classified. The bound derived in [START_REF] Tao | Thresholding bandit with optimal aggregate regret[END_REF] for LSA is not represented on the figures, since it is still bigger than K for the considered range of T . Online sign identification for multi-armed bandits

The sum-of-gaps objective

We show that our method applies for the sum-of-gaps objective K k=1 ∆ k E k . This is not a particular case of the setting discussed previously since a k was known to the algorithm, while ∆ k is unknown. It serves as a proof of concept for the extensibility of our method. The index given by FWT in this setting is F (n, x) = x ′ -3 2 log (x ′ ) + 3 2 log (n), where x ′ = max x, 3 2 . We can then bound the sum-of-gaps loss using our generic analysis by proceeding similarly to Theorem (3.5).

Corollary 3.18. (FWT for the sum-of-gaps objective) In the large horizon regime, specifically when

T ≥ 2 k j=1 1 ∆ 2 j 3 + 3 log ∆ j max i ∆ i (min i ∆ i ) 2 -log T
e , then we show that

E[ K k=1 ∆ k E k ] ≤ 2 √ eT k ∆ k exp   - 1 2 T 2 + j 3 2 1 ∆ 2 j log ∆ 2 k ∆ 2 j j 1/∆ 2 j    .
This can be useful for applications in which errors are more tolerated for arms that are close to the threshold.

Proof. The global loss we investigate in this section is

L T = k ∆ k E k .
First we write the Frank-Wolfe index:

arg min k N k,t ∆2 k,t -3 2 log N k,t ∆2 k,t + 3 2 log (N k,t
), then we slightly modify it to comply with Assumption 3.4 (see explanation above Equation FWT):

I k N k,t = max( 3 2 , N k,t ∆2 k,t ) - 3 2 log max( 3 2 , N k,t ∆2 k,t ) + 3 2 log N k,t .
This corresponds to the index

F (n, x) = 3 2 log n + max( 3 2 , x) -3 2 log max( 3 2 , x). Solving F (n, n∆ 2 k ) = C gives rise to two cases: • n = 1 ∆ 2 k (C + 3 2 log ∆ 2 k ) if C + 3 2 log ∆ 2 k ≥ 3 2 , • n = 3 2 exp( 2 3 C -1) otherwise. In that case, n ≤ 3 2 1 ∆ 2 k . Also, n = 3 2 exp( 2 3 C -1) is solution to F (n, 0) = C. Consider C k > 0, let S k = {j ∈ [K] | C k + 3 2 log ∆ 2 j ≥ 3 2 } and S ′ be a set such that for all k ∈ S ′ , k ∈ S k , then E[L T ] ≤ k / ∈S ′ ∆ k + k∈S ′ ∆ k P(F C k ) + k∈S ′ ∆ k P(E k (T ) ∩ F C k ) 3.6 Additional Experiments ≤ k / ∈S ′ ∆ k + e k∈S ′ ∆ k exp   - 1 2 (T -j / ∈S k 3 2 ∆ j e 2 3 C k -1 ) + j∈S k 3 2 1 ∆ 2 j log 1 ∆ 2 j j∈S k 1/∆ 2 j + C k    + k∈S ′ T ∆ k exp -C k - 3 2 log ∆ 2 k
Large T The values of C k that optimize the r.h.s of the previous inequality verify:

C k = 1 2 1 2 (T -j / ∈S k 3 2 ∆ j e 2 3 C k -1 ) + j∈S k 3 2 1 ∆ 2 j log 1 ∆ 2 j j∈S k 1/∆ 2 j - 1 2 log ∆ 3 k + 1 2 log T e
If T is big enough such that S k = [K] for all arms, which is true in the case considered in the corollary, then we prove the bound of the latter.

Additional Experiments

In this section, we illustrate on synthetic data the performance of APT, LSA, and FWT. The implemented algorithms respectively correspond to Algorithm 3.1 with the following choices:

F (n, x; 1) = x (APT) F (n, x; 1) = x + log(n) (LSA) F (n, x; 1) = (1 + √ x) 2 -log(1 + √ x) 2 + log(n) (FWT)
Note that we used a slighly different version for APT than the one proposed in the analysis Since most of the tested experiments obtained similar performance, we only provide the results of a few experiments. Although our theoretical upper bounds are slighly better for FWT, LSA and FWT generally have similar performance, while APT underperforms. This last point is not surprising since, although we provide in Corollary 3.14 an upper bound for APT that appears asymptotically similar to those of LSA and FWT, APT was not designed to minimize the sum of errors. APT was made to minimize the probability of making at least one error and thus focuses too much on arms with very small gaps that are very difficult to classify.

F (n, x; 1) = max{1, x} -log max{1, x} + log(n).
µ k = (-1) k k K [left] and µ k = (-1) k k 2 K 2
[right] for K = 50 arms.

Figure 3.6 shows the performance of the algorithms together with the non-adaptive oracle of Section 3.7. Figure 3.7 plots the ratio of error with respect to the non-adaptive oracle. Interestingly, in all of our experiments, APT and FWT perform better than it.

Beating the oracle? The benefits of adaptivity.

We argue that in some situations adaptive algorithms can greatly outperform the non-adaptive oracle of Section 3.2.2, i.e., the cost of non-adaptivity can be much higher than the cost of learning. The algorithms in the family we considered are all adaptive in the sense that they adapt their drawing strategy as more information is observed, at the cost of learning the parameter µ k . We illustrate the benefits of adaptivity in the following toy example.

Beating the oracle? The benefits of adaptivity.

The "optimal" non-adaptive algorithm may be worse than adaptive algorithms. Consider the following parametric problem. An arm distribution is parametrized by x ∈ R and is supported on {0, x}; a sample of that distribution is equal to 0 or x, each with probability 1/2. We assume that all arms have non-zero parameter and we will compute the optimal non-adaptive allocation.

We make the convention that if an algorithm sees only zeros for one arm, it returns any sign with probability 1/2. The error probability of a non-adaptive allocation N k T for arm k is half of the probability of seeing only zeros (since if anything else is observed, the arm can be classified with perfect accuracy). Hence the total error is

E L T = 1 2 K k=1 1 2 N k,T ≥ K 2 (T /K)+1 ,
which is minimized with the uniform allocation:

N k,T = T K for all k ∈ [K].
Consider now an adaptive procedure that sample each arm in turn, but stops sampling an arm as soon at it sees a non-zero value. We crudely prove an upper bound for its number of errors, by remarking that it is zero if the algorithm classifies all arms correctly and smaller than K otherwise. The number of samples required to perfectly classify an arm follows a geometric distribution with parameter 1/2. As a consequence, the number of required samples to classify all arms correctly follows a negative binomial NB(K, 1/2). Let Z be such a negative binomial random variable. The expected number of errors of the adaptive procedure is up to KP(Z > T ).

It then verifies

E L T ≤ KP(Z > T ) ≤ Ke -(log(2)/2)T Ee (log(2)/2)Z = K 2 T /2 1 + 1 √ 2 K ,
where the value log(2)/2 is chosen for simplicity (in [0, log 2)). In the regime where T is large, this is of order 1/2 T /2 , which for K > 2 is much smaller than 1/2 T /K for the uniform allocation.

This toy example differs drastically from more realistic situations, as one non-zero sample for an arm is sufficient to know the sign of the expectation perfectly. We therefore consider empirically more reasonable frameworks, closer to those analyzed in the paper: the distributions of K arms are either N (1, 1) or N (-1, 1). Since all gaps ∆ i are equal, the optimal non-adaptive oracle is also the uniform sampling. The results are illustrated on the left part of Figure 3.8 and highlight the fact that all the adaptive algorithms considered (APT, LSA or FWT) drastically outperform the oracle. The right part of the figure shows the same phenomenon on another example in which the gaps are not constant. In particular, we can see that FWT and LSA have similar performance while APT (not designed for this purpose) generally suffers from a larger error. This result was corroborated by most of our experiments. We refer to Appendix 3.6 for more details.

Discussion

An interesting research direction is to consider objective functions more general than (3.1). In particular, we believe that our approach (algorithm and analysis) can be generalized to loss functions that have the form

L T = K k=1 f (∆ k , E k )
under certain regularity assumptions on f . Moreover, we focused on separable losses (hence linear wlog) and the index based algorithms we analyze reflect that separability. An obvious and intriguing direction for further work is to replace that assumption. One might for example want to design an algorithm that minimizes the probability of making more than a given number of mistakes.

The fact that adaptive algorithms can beat non-adaptive oracles has already been observed empirically for fixed confidence identification (Simchowitz, Jamieson, and Recht, 2017; Degenne, Koolen, and Ménard, 2019), although only in cases where the non-adaptive oracle was worse only for small times and was still asymptotically optimal. The phenomenon we observe for fixed budget thresholding is much more significant and remains to be explained by theoretical arguments. Currently, the best theoretical bound for adaptive algorithms is still a factor 1 /4 away in the exponent from the non-adaptive oracle bound.

Chapter 4

Linear regression: an improved algorithm & application to linear bandits

We consider the problem of online linear regression in the stochastic setting. We derive high probability regret bounds for online ridge regression and the forward algorithm. This enables us to compare online regression algorithms more accurately and eliminate assumptions of bounded observations and predictions. Our study advocates for the use of the forward algorithm in lieu of ridge due to its enhanced bounds and robustness to the regularization parameter. Moreover, we explain how to integrate it in algorithms involving linear function approximation to remove a boundedness assumption without deteriorating theoretical bounds. We showcase this modification in linear bandit settings where it yields improved regret bounds. Last, we provide numerical experiments to illustrate our results and endorse our intuitions.1

Introduction and preliminaries

The forward regression algorithm, popularized in [START_REF] Vovk | Competitive on-line statistics[END_REF]; Azoury and Warmuth, 2001), shows competitive performance bounds in the challenging setup of online regression with adversarial bounded observations. We revisit the analysis of this strategy in the practically relevant alternative situation of stochastic linear regression with sub-Gaussian noise, hence possibly unbounded observations. When compared to the classical ridge regression strategy -its natural competitor -the existing analysis in the adversarial bounded case suggests the forward algorithm has higher performances. It is then natural to ask whether this conclusion holds for the stochastic setup. However, we show that in the stochastic setup, the existing adversarial analysis does not seem sufficient to draw conclusions, as it does not capture some important phenomena, such as the concentration of the parameter estimate around the regression parameter. It may further lead the practitioner to use an improper tuning of the regularization parameter. In order to overcome these issues, we revisit the analysis of the forward algorithm in the case of unbounded sub-Gaussian linear regression and provide a high probability regret bound on the performance of the forward and ridge regression strategies. Owing to this refined analysis, we show that the forward algorithm is superior in this scenario as well, but for different reasons than what is suggested by the adversarial analysis. We discuss the implications of this result in a practical application: stochastic linear bandits, both from theoretical and experimental perspectives.

Setup:

In the classical setting of online regression with the square loss, an environment initially chooses a sequence of feature vectors {x t } t ∈ R d together with corresponding observations {y t } t ∈ R. Then, at each decision step t, the learner receives feature vector x t and must output a prediction ŷt ∈ R. Afterwards, the environment reveals the true label y t and iteration t + 1 begins. In this article, we focus on the case when the data generating process is a stochastic linear model:

∃θ * ∈ R d such that ∀t ∈ N * : y t = x ⊤ t θ * + ε t ,
where {ε t } t is a noise sequence. At iteration t, strategy A computes a parameter θ A t-1 to predict

ŷA t = x ⊤ t θ A t-1 .
In the sequel, we omit the subscript A when the algorithm is clear from context. The learner's prediction incurs the loss:

ℓ A t = ∆ ℓ(x ⊤ t θ t-1 , y t ) = (ŷ t -y t ) 2
, the learner then updates its prediction θ t-1 to θ t and so on. The total cumulative loss at horizon T is denoted

L A T = T t=1 ℓ A t . We also let ℓ t (θ) = ℓ(x ⊤ t θ, y t ) (resp. L T (θ) = T t=1 ℓ t (θ)
) be the instantaneous (resp. cumulative) loss incurred by predicting θ at time t (resp. ∀t = 1, . . . , T ). Online regression algorithms are evaluated using different regret definitions, in the form of a relative cumulative loss to a batch loss; The quantity of interest in this paper is:

R A T = L A T -min θ L T (θ). (4.1)

Introduction and preliminaries

From the perspective of online learning theory, online regression algorithms are usually designed for an adversarial setting, assuming an arbitrary bounded response variable |y t | ≤ Y at each time step. While the mere existence of algorithms with tight guarantees in this general setting is remarkable, a practitioner may also consider alternative settings, in which analysis for the adversarial setup may be overly conservative. For illustration, we focus on the practical setting of bounded parameter ∥θ * ∥ 2 ≤ S and i.i.d zero-mean σ-sub-Gaussian noise sequences:

∀t ≥ 1, γ ∈ R : E [exp(γε)] ≤ exp(σ 2 γ 2 /2).
We emphasize that while previous results in literature are valid for the adversarial bounded setting, we will still shed new light on the performance of these strategies in a stochastic unbounded setup, which is neither more general nor more restrictive than the adversarial one, and discuss their implications for the practitioner. Let us recall the two popular online regression algorithms considered.

Online ridge regression [Algorithm 4.1]:

This folklore algorithm is defined in the online setting as the greedy version of batch ridge regression:

θ r t ∈ arg min θ L t (θ) + λ||θ|| 2 2 , ( 4.2)
where λ is a parameter and λ||θ|| 2 2 is a regularization used to penalize model complexity. Algorithm 4.1: Online ridge regression

1 Given θ 0 ∈ R d 2 for t = 1, . . . , T do 3 observe x t ∈ R d and predict ŷt = x ⊤ t θ r t-1 ∈
θ r t = G t (λ) -1 b t , where G t (λ) = λI + t q=1 x q x ⊤
q and b t = t q=1 x q y q . We may further denote G t instead of G t (λ) when λ is clear from context.

The forward algorithm [Algorithm 4.2]:

A subtle change to the ridge regression takes advantage of the next feature x t+1 to better adapt to the next loss:

θ f t ∈ arg min θ L t (θ) + (x ⊤ t+1 θ) 2 + λ||θ|| 2 2 . ( 4
θ f t-1 ∈ arg min θ L t-1 (θ) + (x ⊤ t θ) 2 + λ||θ|| 2 2 5 predict ŷt = x ⊤ t θ f t-1 ∈ R 6
observe y t and incur loss ℓ t ∈ R

end

Equivalently, the update step can be written:

θ f t = G -1 t+1 b t
, where G t is still defined same as before. Intuitively, the term (x ⊤ t+1 θ) 2 in Equation 4.3 is a "predictive loss", a penalty on the parameter θ in the direction of the new feature vector x t+1 . This approach can be linked to transductive methods for regression [START_REF] Cortes | On transductive regression[END_REF][START_REF] Tripuraneni | Single Point Transductive Prediction[END_REF]. [START_REF] Tripuraneni | Single Point Transductive Prediction[END_REF] describe two algorithms for linear prediction in supervised settings, and leverage the knowledge of the next test point to improve the prediction accuracy. However, these algorithms have significant computational complexities and are not adapted to online settings.

Related work

Linear regression is perhaps one of the most known algorithms in machine learning, due to is simplicity and explicit solution. In contrast with the batch setting (when all observations are provided), online linear regression started receiving interest relatively recently. The first theoretical analyses date back to [START_REF] Dean | Prediction in the worst case[END_REF]; Littlestone, Long, and Warmuth, 1991; Cesa-Bianchi, Long, and Warmuth, 1996; [START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF]. Under the assumption that the response variable is bounded |y t | ≤ Y , it has been shown that the forward algorithm [START_REF] Vovk | Competitive on-line statistics[END_REF]; Azoury and Warmuth, 2001) achieves a relative cumulative online error of dY 2 log(T ) compared to the best batch regression strategy. This bound holds uniformly over bounded response variables and competitor vectors, and is 4 times better than the corresponding bound derived for online ridge regression. [START_REF] Peter L Bartlett | Minimax fixed-design linear regression[END_REF] studies minimax regret bounds for online regression, and ingeniously removed a dependence on the scale of features in existing bounds by considering the beforehandknown features setting, where all feature points {x t } 1≤t≤T are known before the learning starts. Moreover, they derive a "backward algorithm" that is optimal under certain intricate assumptions on observations and features. Later on, [START_REF] Malek | Horizon-independent minimax linear regression[END_REF] proves that under new (tricky) assumptions on observed features and labels the backward algorithm is not only optimal but applicable in sequential settings as well. More recently, [START_REF] Gaillard | Uniform regret bounds over R d for the sequential linear regression problem with the square loss[END_REF] provides an optimal algorithm in the setting of beforehand known features without imposing stringent conditions as in (Bartlett et al., 2015; [START_REF] Malek | Horizon-independent minimax linear regression[END_REF]. The latter shows that the forward algorithm with λ = 0 yields a first-order optimal asymptotic regret bound

Adversarial bounds and limitations

uniform over bounded observations. However, due to the lack of regularization, their bound (cf Theorem 11 in [START_REF] Gaillard | Uniform regret bounds over R d for the sequential linear regression problem with the square loss[END_REF]) may blow up if the design matrix G t (0) is not full rank. It is hence not uniform over all bounded feature sequences {x t } t .

Paper outline and contributions:

In this paper, we continue the line of work initiated on the forward algorithm and advocate for its use in the stochastic setting with possibly unbounded response variables, in replacement for the ridge regression (whenever possible). To this end, we consider an online stochastic linear regression setup where the noise is assumed to be i.i.d σ-sub-Gaussian.

In Section 4.2 we recall the online performance bounds established for ridge regression and the forward algorithm in the adversarial case with bounded observations. Next, in subsection 4.2.3, we discuss some limitations of the adversarial results when comparing regression algorithms in the stochastic setting. For instance, these bounds compare the cumulative loss of a strategy to the value of the batch optimization problem, which may not be indicative of the real performance of the strategy (cf Corollary 4.5) and may encourage a sub-optimal tuning of the regularization parameter.

In Section 4.3, we study the performance of these algorithms using the cumulative regret with respect to the true parameter (cf Equation 4.6), which we believe is more practitionerfriendly than comparing to the batch optimization problem. We show in Theorem 4.6 how these two measures of performance are related. We provide in Theorems 4.7 and 4.9 a novel analysis of online regression algorithms without assuming bounded observations. This key result is made possible by considering high probability bounds instead of bounded individual sequences. We show that the regret upper-bound for ridge regression is inversely proportional to the regularization parameter. Consequently, we argue that following these results, forward regression should be used in lieu of ridge regression.

In Section 4.5, we revisit the linear bandit setup previously analyzed assuming bounded rewards: we relax this assumption and provide an -optimism in the face of uncertainty-style algorithm with the forward algorithm instead of ridge, which is especially well-suited for the bandit setup, and provide novel regret analysis in Theorem 4.15. We proceed similarly by revisiting a setup of non-stationary (abruptly changing) linear bandits.

Adversarial bounds and limitations

In this section, we recall existing results regarding the aforementioned ridge and forward algorithms. We then discuss their limits and benefits when considered from a stochastic perspective.

Linear regression: an improved algorithm & application to linear bandits

Adversarial regret bounds (known results)

One of the first theoretical analyses of online regression dates back to [START_REF] Vovk | Competitive on-line statistics[END_REF] and (Azoury and Warmuth, 2001), and is recalled in the theorem below. It is stated in the form of an "onlineto-offline conversion" performance bound.

L r T -min θ L T (θ) + λ∥θ∥ 2 2 ≤ 4(Y r ) 2 d log 1 + T X 2 λd ,
where X = max

1≤t≤T ∥x t ∥ 2 , and Y r = max 1≤t≤T |y t | , x ⊤ t θ t-1 .
The reader should note that this result compares the learner's online cumulative loss to the regularized batch ridge regression loss. As such, it is an online-to-offline conversion regret. This is different from the sequential regret that would compare to the minimum achievable loss. This theorem highlights a dependence on the range of predictions of the algorithm, as

Y r ≥ max 1≤t≤T x ⊤ t θ t-1 .

Remark 4.2. (Small losses)

The regret bound for ridge regression can be improved if the learner knows that the loss is small for the best expert, see Orabona, Cesa-Bianchi, and Gentile (2012). Note however that such techniques require prior knowledge of all the best expert loss L * T , their optimal bound is ∼ O( L * T log T ).

The forward algorithm has an enhanced performance in this setup according to this next result.

L f T -min θ L T (θ) + λ∥θ∥ 2 2 ≤ (Y f ) 2 d log 1 + T X 2 λd ,
where X = max Notice that in this result Y is different than in Theorem 4.1 and is independent from the algorithm's predictions. Moreover, Theorem 4.3 exhibits a bound that is at least 4 times better

Adversarial bounds and limitations

than Theorem 4.1. More precisely, Theorem 4.1 suggests that, in order to compare the two bounds, prior knowledge of Y f is required to further clip the predictions of online ridge regression in [-Y f , Y f ]; and that even with such knowledge the forward algorithm may be 4-times better than ridge regression. We believe that this unfortunately led researchers to turn away from analyzing more deeply what may happen.

Limitation in the adversarial setup: rigid regularization

To evaluate online regression strategies, a tight lower bound was derived in [START_REF] Gaillard | Uniform regret bounds over R d for the sequential linear regression problem with the square loss[END_REF]. The latter studied uniform minimax lower bounds in the setting of beforehand-known features (that is when (x t ) 1≤t≤T known in advance), which is very challenging for a lower bound. They show that, the minimax uniform regret bound is controlled as follows. 2019)) For all T ≥ 8, Y > 0 we have:

R ⋆ T,[-Y,Y ] ≥ dY 2 (log(T ) -(3 + log(d)) -log(log(d))) .
where

R ⋆ T,[-Y,Y ] def = inf A sup x 1,...,T ∈[0,1] d sup |yt|≤Y T t=1 (y t -ŷA t ) 2 -inf u∈R d T t=1 (y t -x ⊤ t u) 2
We will use this result to evaluate the optimality of ridge and forward regressions. First, we need to convert Theorems 4.1 and 4.3 to sequential regret bounds. Indeed, in their current form, they compare the cumulative loss of the learner to the value of a regularized batch optimization. This next result transforms them, and is a corollary of Theorems 11.7 and 11.8 of Cesa-Bianchi and Lugosi (2006).

≥ 1, (x t ) 1≤t≤T ∈ R d , (y t ) 1≤t≤T ∈ [-Y, Y ] such that ∥x t ∥ 2 ≤ X, for A ∈ {r, f} R A T ≤ c A (Y A ) 2 d log 1 + T X 2 λd + λ (Y A ) 2 T λ r T (G T (0)) ,
where r T = rank(G T (0)) and λ r T is its smallest positive eigenvalue, c r = 4 and c f = 1.

This bound suggests that to obtain a log(T ) bound, λ should not be chosen larger than about log(T )/T , due to the second term, this is the stringent regularization limitation.

Linear regression: an improved algorithm & application to linear bandits

Proof. Consider (w.l.o.g) ridge regression, denote X T the design matrix and y T the labels, then:

∥θ T ∥ 2 = G T (0) † b T 2 = y ⊤ T X ⊤ T G T (0) † G T (0) † X T y T ≤ y ⊤ T X ⊤ T G T (0) † X T y T λ r T (G T ) ≤ Y r T λ r T (G T )
,

where G T (0) † is the pseudo-inverse of G T (0), the last inequality is because X ⊤ T G T (0) † X T is an orthogonal projection on Im(X ⊤ ). Injecting in the previous theorem finishes the proof, these bounds hold for arbitrary bounded sequences. The proof for the forward algorithm proceeds in the same way by replacing G T by G T +1 and Y r by Y f . Choosing λ = 1/T yields a first order regret of 2dY 2 for the forward algorithm and 8dY 2 for ridge regression (with clipping and prior knowledge of Y ), which is at best twice the first order term from the lower bound. This suggests the presence of an optimality gap. Strikingly, [START_REF] Gaillard | Uniform regret bounds over R d for the sequential linear regression problem with the square loss[END_REF] shows that a non-regularized version of the forward algorithm achieves the optimal first order of dY 2 . However, it also suffers from an important weakness: Indeed, the (Y A ) 2 /λ r T (G T (0)) term in Corollary 4.5 is not uniformly bounded over feature sequences, but only on specific "well-behaved" features. In fact, double uniformity over features and observations is still an open question (see [START_REF] Gaillard | Uniform regret bounds over R d for the sequential linear regression problem with the square loss[END_REF]).

Limitations in the stochastic setting

Now that we have recalled the main properties of the forward and ridge algorithms in the adversarial setup, we advocate for the need of a complementary analysis of the previous algorithms in the stochastic unbounded setting by unveiling some key limitations.

Too unconstrained

The existing analysis being for a different setting, it naturally ignores crucial aspects of the stochastic setup. For instance, the quantity Y is uninformative and may be substantial. Let us look at how the term Y appears in the proofs of Azoury and Warmuth (2001). For ridge regression, the penultimate step to prove Theorem 4.1 writes:

L r T -min θ L T (θ) + λ∥θ∥ 2 2 ≤ T t=1 x ⊤ t θ t-1 -y t 2 x ⊤ t G -1 t x t first term ≤ 4(Y r ) 2 T t=1 x ⊤ t G -1 t x t . (4.4)
In an adversarial setting, the "first term" cannot be controlled without assuming bounded pre- Online regret terms Ridge1stTerm Forward1stTerm Forward2ndTerm For the forward algorithm, the final step in the proof of Theorem 4.3 writes:

dictions |x ⊤ t θ t-1 | ≤ Y r ,
L f T -min θ L T (θ) + λ∥θ∥ 2 2 ≤ T t=1 y 2 t x ⊤ t G -1 t x t first term - T -1 t=1 x ⊤ t+1 G -1 t x t+1 x ⊤ t+1 θ t 2 second term . ( 4.5)
Then, the analysis uses that |y t | ≤ Y f and disregards the negative contribution of the "second term". Illustrative example: Let us analyze these terms in an practice: consider d = 5, θ * ∈ R 5 , we sample 200 features uniformly in [0, 1] 5 and Gaussian noises (σ = 0.1). Figure 4.1 displays the instantaneous first regret term of both algorithms (with λ = 1) and the second regret term of the forward algorithm, averaged over 100 replicates. We remark that the first terms vanish quickly for ridge regression and are quite stable for the forward algorithm. On the other hand, they are essentially cancelled out by the second term. Overall, the two strategies perform on par on this example. This suggests that Theorems 4.1 and 4.3 can be misleading in this stochastic setup: for ridge regression they introduce a conservative 4(Y r ) 2 bound on (x ⊤ t (θ t-1 -θ * )) 2 , while in practice we observe that this term decreases rapidly to zero; for the forward algorithm, the bound ignores the effect of a negative term, which, as we see in Figure 4.1, is essential to explain why this algorithm may outperform ridge regression.

Time dependence:

In the stochastic case, it can be confusing to introduce Y A . Indeed, the latter hides a significant implicit time-dependence. For instance, for the forward algorithm

Y f = max 1≤t≤T |x ⊤ t θ * + ε t |.
Considering the tractable setting of Gaussian i.i.d noise with variance σ 2 and using the classical Sudakov minoration (Sudakov, 1969), we deduce that there Linear regression: an improved algorithm & application to linear bandits exists C > 0 such that:

∀T ≥ 1 : E[Y f ] ≥ E max 1≤t≤T ε t -X∥θ * ∥ 2 ≥ σC 2 log(T ) -X∥θ * ∥ 2 .
Since (Y f ) 2 appears in the previous performance bounds, this suggests that Y f actually increases the order of the regret bound to log(T ) 2 in this setting.

By focusing on the unbounded stochastic scenario, we hope in this paper to shed novel light on the practical performance of these strategies and better explain these phenomena.

High probability bounds

In this section, we analyze online ridge regression and the forward algorithm in the stochastic setting. We present our results in terms of the following intuitive regret definition:

RA T = L A T -L T (θ * ). (4.6)
This regret directly compares the cumulative loss of the learner to the cumulative loss of the oracle knowing the true parameter θ * . This contrasts with the online-to-batch conversion result that compares the loss of the learner to the value of a batch regularized optimization problem. Since we are in a stochastic setup, we further state results in high probability. More precisely, we state Theorems 4.6,4.7, 4.9 below holding with high probability uniformly over all T , and not simply for each T . As a first step, we prove that for T great enough, we can choose this definition instead of R T defined in Equation 4.1 without altering the bounds.

Theorem 4.6. (Regret equivalence) In the stochastic setting with sub-Gaussian noise, for all δ > 0 with probability at least 1 -δ, for all T > 0, (

x t ) 1≤t≤T ∈ R d such that ∥x t ∥ 2 ≤ X, |G T (0)| > 0 L T (θ * ) -min θ∈R d L T (θ) = o log(T ) 2 ,
in particular, it comes

R A T = RA T + o log(T ) 2
Theorem 4.6 justifies choosing RT to provide identical first order guarantees as R T . Indeed, in the following sections we prove high probability upper bounds of order O(log(T ) 2 ).

High probability bounds

Proof. Denote ∀T ≥ 1 : θ T = arg min θ∈R d L T (θ), then:

R T -RT = L T (θ * ) -L T (θ T ) = 2 T t=1 ε t (θ T -θ * ) ⊤ x t - T t=1 (θ T -θ * ) ⊤ x t 2 .
(4.7)

Denote S T = T t=1 ε t (θ T -θ * ) ⊤ x t and A T = T t=1 (θ T -θ * ) ⊤ x t 2
. Using Lemma B.1 we can prove that S T = o(A T ), this means that we can focus on bounding A T to obtain the desired result.

Using Lemma B.1 and Equation (4.7) we get that for all σ ′ , δ > 0 with probability at least

1 -δ: R T -RT ≤ 2(A T + 1/σ ′2 ) log σ ′2 A T + 1/δ -A T ≤ (A T + 1/σ ′2 ) (log(σ ′2 A T + 1) + 2 log(1/δ)) -A T ≤ A T + 1/σ ′2 log(σ ′2 A T + 1) + 2 log(1/δ) -A T ≤ 1 σ ′2 + 2(A T + 1/σ ′2 ) log(1/δ) (4.8)
The next step is to the use confidence intervals of Maillard (2016) (Theorem 3.3 therein) which hold once the design matrix is singular, see Theorem B.1 for its statement.

The latter entails, that for bounded features ∥x∥ ≤ X, we obtain λ max (G T (0)) ≤ T X 2 . Denote T 0 = inf t≥1 {|G t | > 0}, and for t ≥ T 0 :

β t = 2(1 + κ)(1 + α)σ 2 log κ d (e 2 λmax(G T )) δ
, then for all δ > 0 with probability at least 1 -δ:

A T = T t=1 (θ T -θ * ) ⊤ x t 2 ≤ A T 0 + T t=T 0 (θ T -θ * ) ⊤ x t 2 ≤ A T 0 + T t=T 0 β t ∥x t ∥ 2 Gt(0) -1 ≤ A T 0 + β T T t=T 0 ∥x t ∥ 2 Gt(0) -1 (4.9)
Then we bound the sum of features using Lemma B.2.

From equation (4.8), using

A T ≤ T t=1 ∥θ T -θ * ∥ 2 Gt ∥x t ∥ 2 G -1 t ≤ T t=1 ∥θ T -θ * ∥ 2 G T ∥x t ∥ 2 G -1 t
, then injecting Lemma B.2 with λ = 0, we find that for all δ > 0, with probability at least 1 -δ:

A T ≤ A T 0 + β T d log 1 + T X 2 /λ min (G T 0 (0))d

Linear regression: an improved algorithm & application to linear bandits

Then injecting this last inequality in equation (4.7) gives, for all δ, σ ′ > 0, with probability at least 1 -δ:

R T -RT ≤ 1 σ ′2 + σ ′ 2 log(1/δ) β T d log 1 + T X 2 /λ min (G T 0 )d + 1 .
Finally, we know -by definition-that R T ≥ RT , this concludes the proof for the equivalence of the two regret definitions.

Online ridge regression

We start our results by stating a new high probability regret bound for online ridge regression.

Theorem 4.7. In the stochastic setting with sub-Gaussian noise, for all δ > 0 with probability at least 1 -δ, for all T ≥ 0:

Rr T ≤ 2dσ 2 X 2 λ log(1 + X 2 /λ) log 1 + T X 2 /λd log (1 + T X 2 /λd) d/2 δ/2 + o(log(T ) 2 ),
where

X = max 1≤t≤T ∥x t ∥ 2 .
This result is interesting because the ranges of both predictions and observations do not appear, hence predictions clipping and/or a prior knowledge assumption on Y f are not required. On the other hand, a factor 1/λ appears in the worst case of a singular design matrix. This seems to be the price for no longer assuming bounded predictions. Another notable improvement is that this bound no longer involves λ∥θ∥ 2 2 terms. In particular, it is uniform over bounded sequences of observations. Proof. See Equation 4.12 for an explicit bound. In particular, the o(log(T ) 2 ) term is O(log(T ) 3/2 ).

Let's write the instantaneous regret:

rt = ℓ t (θ t-1 ) -ℓ t (θ * ) = θ ⊤ t-1 x t -θ ⊤ * x t ) 2 + 2ε t (θ ⊤ t-1 x t -θ ⊤ * x t ) (4.10)
The proof proceeds in three steps, that we detail hereafter and then we explain how to combine them for the final result.

First step: Confidence bound to control the concentration of θ t-1 around θ * . For this we use the confidence ellipsoid from Abbasi-Yadkori, Pál, and Szepesvári (2011), which states that for

High probability bounds

any δ > 0, with probability at least 1 -δ, for all t > 0:

∥θ r t -θ * ∥ Gt ≤ β t (δ) = σ d log 1 + tX 2 /λd δ + λ 1/2 S.
It comes, with probability at least 1 -δ:

(θ t-1 -θ * ) ⊤ x t ≤ ||x t || G -1 t-1 ||θ t-1 -θ * || G t-1 ≤ β t-1 (δ)||x t || G -1 t-1 .
Then, since β t is non-decreasing:

L t -L * t ≤ β T -1 T t=1 ∥x t ∥ 2 η t-1 + 2 T t=1 ε t (θ t-1 -θ * ) ⊤ x t . (4.11)
Second step: Next we bound the sum of feature norms. The main idea here is to use linear algebra techniques to obtain a telescopic sum. Lemma B.2 doesn't apply here because we have

||x t || G -1 t-1 instead of ||x t || G -1 t
. We derive a similar result for this sum of feature norms in Lemma B.3.

Third step:

To control the second term in the r.h.s of Equation 4.10, we use Martingale inequalities similar to the ones used for the confidence intervals to derive a uniform high probability bound. Indeed, Lemma B.4 proves that the second term in Equation 4.10 is a of order ∼ O(log(T ) log log T ). In fact, with high probability t s=1 (θ t-1 -θ * ) ⊤ x t

2

= O(log(T ) 2 ) therefore, with high probability S T is of order ∼ O(log(T ) log(log T )/δ). Consequently, with high probability, S T is second order.

Proof aggregation: By combining earlier results we find for any δ, σ ′ > 0, with probability at least 1 -δ, for all T ≥ 0:

Rr T ≤ σ d log 1 + T X 2 /λd δ/2 + λ 1/2 S 2 X 2 /λ log(1 + X 2 /λ) d log 1 + T X 2 /λd + σ 2 1/σ ′2 + t s=1 (θ s-1 -θ * ) ⊤ x s 2 log   2 1 + σ ′2 t s=1 (θ s-1 -θ * ) ⊤ x s ) 2 /δ   . ( 4

The forward algorithm

We analyze the forward algorithm and derive a high probability regret bound for it using similar techniques up to minor modifications.

Theorem 4.9. Assuming sub-Gaussian noise, with probability at least 1 -δ, for all T ≥ 0:

Rf T ≤ 2dσ 2 log 1 + T X 2 /λd log (1 + T X 2 /λd) d/2 δ/2 + o(log(T ) 2 ),
where X = max 1≤t≤T ∥x t ∥ 2 and the o(log(T ) 2 ) depends on λ (see Equation 4.13).

Proof. The proof proceeds similarly to that of Theorem 4.7. First, we need to bound the instantaneous regret.

rt = ℓ t (θ t-1 ) -ℓ t (θ * ) = θ ⊤ t-1 x t -θ ⊤ * x t ) 2 + 2ε t (θ ⊤ t-1 x t -θ ⊤ * x t )
We proceed in three steps like before.

First step:

We start by deriving a confidence ellipsoid for this new parameter estimate. This is the novel result of Theorem B.5 where we show that for any δ > 0, with probability at least 1 -δ, for allt > 0:

∥θ t -θ * ∥ Gt ≤ β t (δ) = σ d log 1 + tX 2 /λd δ + (λ 1/2 + X)S.
For the first term, with probability at least 1 -δ for all t ≥ 0:

(θ t-1 -θ * ) ⊤ x t ≤ ||x t || G -1 t ||θ t-1 -θ * || Gt ≤ β t-1 (δ)||x t || G -1 t ≤ β T -1 (δ)||x t || G -1 t .
Second step: We can use Lemma B.2 to bound the sum of feature norms. It comes

T t=1 (θ ⊤ t-1 x t -θ ⊤ * x t ) 2 ≤ β T (δ)d log 1 + T X 2 /λd
Third step: Again, we derive Lemma B.6, a high probability bound To control the second term in the r.h.s of (4.10).

The unregularized-forward algorithm

Proof aggregation: We combine previous results to finish the proof of the forward algorithm regret bound. For any δ, σ ′ > 0, with probability at least 1 -δ, for all T ≥ 0:

Rf T ≤ σ d log 1 + T X 2 /λd δ/2 + ( √ λ + X)S 2 X 2 /λ log(1 + X 2 /λ) d log 1 + T X 2 /λd + σ 2 1/σ ′2 + t s=1 (θ t-1 -θ * ) ⊤ x t 2 log   1 + σ ′2 t s=1 (θ t-1 -θ * ) ⊤ x t ) 2 /δ   . (4.13)
Theorem 4.9 exhibits a better bound than Theorem 4.7. In fact, the coefficient of the first order term for the forward algorithm only depends on the dimensionality and the noise variance, whilst for ridge regression, it also depends on the features' scale and on the regularization parameter λ. Remark 4.10. (Unrestrained regularization) Compared to existing results, this analysis lifts the "stringent regularization" that requires λ = 1/T or data-dependent regularization (cf [START_REF] Malek | Horizon-independent minimax linear regression[END_REF]) to obtain uniform bounds. Therefore, Theorems 4.7 and 4.9 are not a mere consequence of bounding Y 2 with high probability in previous deterministic theorems. For completeness, we also derive a high probability regret bound for a non-regularized version of the forward algorithm in Appendix 4.4; this algorithm was proven to be asymptotically first order minimax optimal in the adversarial bounded setting [START_REF] Gaillard | Uniform regret bounds over R d for the sequential linear regression problem with the square loss[END_REF].

The unregularized-forward algorithm

For the sake of completeness, we propose a high probability bound on the regret of a nonregularized forward algorithm -studied in the adversarial bounded case in Gaillard et al. (2019)which achieves the optimal asymptotic first order deterministic minimax bound of dY 2 log(T ). This algorithm is a simple yet elegant modification of forward regression, it avoids the exploding λ∥θ T ∥ 2 2 term by setting λ = 0. Consequently θ t = G † t+1 b t , where G † t is the pseudo-inverse of G t . Theorem 4.11. (Regret of the unregularized forward) The unregularized forward regression achieves, for any δ > 0, with probability at least 1 -δ for all T > 0:

Ru-f T ≤ 2(1 + κ)(1 + α)σ 2 log κ d (1 + T X 2 /γd) δ/4 log   |G † T | |G † T 1 |   + 2σ 2 log 4T 1 δ d + 1≤t≤T 1 ,t∈T log X 2 λ rt ( t s=1 x t x ⊤ t )
,

Linear regression: an improved algorithm & application to linear bandits

where κ, α ∈ R * + are peeling parameters (can be chosen), γ = min 1≤t≤T ∥x t ∥ 2 , and κ d (x) ∝ x d up to logarithmic factors and depends on κ and α (cf Theorem 5.4 in Maillard ( 2016)). T 1 = min {t ≥ 1, |G t | > 0} is, if it exists, the first time the design matrix is non-singular, otherwise T 1 = T , and T is the set of indices t such that rank(G t ) >rank(G t-1 ). The last term accounts for when the design matrix is singular, and is naturally unbounded (this was also the case in the adversarial case).

Asymptotically, with probability at least 1 -δ the first regret term is bounded as:

Ru-f T ≤ 2(1 + κ)(1 + α) log C(κ, α)(T X 2 /λd) d δ log (T -T 1 )X 2 /λd ,
where C(κ, α) is a function of the peeling parameters. We don't seek a more involved analysis to explicit this bound or improve on it, but we see that vaguely it leads to a bound similar to Theorems 4.7 and 4.9 provided that the term accounting for the singularity of the design matrix is controlled. The latter empowers the intuition that in the high probability analysis, the forward algorithm is first order minimax optimal even though concretely we cant be sure because we don't have access to uniform lower bounds.

Proof. The proof consists of two mains steps: the first is to use the following bound while the design matrix is singular:

R T (u) ≤ Y 2 T t=1 x T t η † t x t ⩽ dY 2 log T + dY 2 + Y 2 t∈[1,T ]∩T log X 2 λ rt ( t s=1 x s x ⊤ s )
where ∀M ∈ M d (R), λ 1 (M ) ≥ . . . ≥ λ d are M 's eigenvalues and r t = rank( t s=1 x s x ⊤ s )) and where the set T contains r T rounds, given by the smallest s ⩾ 1 such that x s is not mull, and all the s ⩾ 2 for which rank (G s-1 ) ̸ = rank (G s ).

The second step is a bound when the design matrix is invertible, using Theorem B.1. Denote

T 1 = inf t≥1 {|G t | > 0}, using Theorem 4.12: RT 1 ≤ Y 2   d log(T 1 ) + d + 1≤t≤T 1 ,t∈T log X 2 λ rt (G t )  
From standard results on sub-Gaussian noise, we also know that E[max 1≤t≤T ε t ] ≤ σ 2 log(T ) (see e.g. Kamath (2015)), then using the transformation of Laplace along with Markov's 4.4 The unregularized-forward algorithm inequality, ∀δ > 0 P ∀T ≥ 1, Y 2 ≤ 2σ 2 log(T /δ) ≥ 1 -δ, hence with probability at least 1 -δ:

RT 1 ≤ 2dσ 2 log T 1 δ log(T 1 ) + 2dσ 2 log T 1 δ + 2σ 2 log T 1 δ 1≤t≤T 1 ,t∈T log X 2 λ rt ( t s=1 x s x ⊤ s ) . (4.14)
And for T > T 1 , we bound R T -R T 1 using the same methodology as the proofs of Theorem 4.7 and Theorem 4.9. In fact, using the confidence bounds of Theorem B.1 we find, for all δ > 0, with probability at least 1 -δ:

∀t > T 1 : θ ⊤ t-1 x t -θ ⊤ * x t ) 2 ≤ β t-1 (δ)||x t || G † t .
We use the tail inequality of Lemma (B.4) to get, ∀δ > 0, with probability at least 1 -δ, ∀T > 0:

RT -RT 1 ≤ 2(1 + κ)(1 + α)σ 2 log κ d (1 + T X 2 /λd) δ/2 log   |G † T | |G † T 1 |   (4.15)
From (4.14) and (4.15) we obtain for all δ > 0, with probability at least 1 -δ:

RT ≲ 2(1 + κ)(1 + α)σ 2 log κ d (1 + T X 2 /λd) δ/4 log   |G † T | |G † T 1 |   + 2σ 2 log(T 1 ) δ/4 d + 1≤t≤T 1 ,t∈T log X 2 λ rt ( t s=1 x t x ⊤ t )
.

Tightness of the bounds

Here we clarify the impact of a tighter confidence width for regularized least squares that was proved concurrently with the writing of this paper. First we state the result then we discuss its implications.

Theorem 4.13. (Theorem 1 of Tirinzoni, Pirotta, et al. ( 2020)) Let δ ∈ (0, 1), n ≥ 3, and θ t be a regularized least-square estimator obtained using t ∈ [n] samples collected using an arbitrary bandit strategy π := {π t } t≥1 . Then,

P ∃t ∈ [n] : θ t -θ * Vt ≥ √ c n,δ ≤ δ

Linear regression: an improved algorithm & application to linear bandits

where c n,δ is of order O(log(1/δ) + d log log n).

This has important implications for Theorems 4.6, 4.7 and 4.9: in short it re-scales their regret upper-bounds from R T = O (dσ) 2 log(T ) 2 to R T = O dσ 2 log(T ) log(1/δ) . The first order (dσ) 2 log(T ) 2 in our results is the product of 1) d log T from the elliptical lemma, for bounding the sum of feature norms and 2) σ 2 log(T d /δ) the confidence ellipsoid width in the estimation of the regression parameter. It is the second term that is altered following the new result from Tirinzoni, Pirotta, et al. (2020). These tighter confidence intervals change the upper bounds to O(dσ 2 log(T ) log log(T )). The latter matches the popular lower bounds in excess risk literature (see e.g. Theorem 1 in Mourtada ( 2019)) up to sub-logarithmic terms suggesting the optimality of the forward algorithm in the stochastic setting.

Experiment

We provide experimental evidence supporting the fact that our novel high probability analysis better reflects the influence of regularization than results its adversarial counterpart. In Figures 4.2a and 4.2b we observe the effect of regularization on the performance of ridge regressions and on the Forward algorithm in a 5-dimensional regression setting. We vary λ ∈ {1/T, 1/ log(T ), 1, 10}, sample a zero mean Gaussian noise with σ = 0.1 and draw features uniformly from the unit ball. The results clearly highlight the robustness of the forward algorithm to λ, contrarily to ridge. In particular, for ridge regression, we observe the exact dependence on λ described by Theorem 4.7 in the first rounds of learning; as explained in Remark 4.8, once the collected features are enough for the design matrix G t (0) to become non-singular, the 1/λ virtually disappears from the first order regret bound and is replaced by the smallest eigenvalue of G t (0), making the regret significantly more stable.

Application: linear bandits

Application: linear bandits

The proposed analysis of forward regression in the stochastic setting suggests that using it could be useful for revisiting several popular setups that include linear function approximation. We apply this change for stochastic linear bandits hereafter and derive the novel regret bound obtained when using forward regression instead of the standard ridge regression.

Stationary bandits

Consider the setting of stochastic linear bandits, where at round t the reward of an action x t (from the action space X ⊂ R d ) is y t = ⟨x t , θ * ⟩ + ε t , where θ * ∈ R d is an unknown parameter and ε t is, conditionally on the past, a σ-sub-Gaussian noise. An upper bound S on the unknown parameter's norm is provided: ∥θ * ∥ 2 ≤ S. The (pseudo) regret in this setting is defined:

R T = T t=1 ⟨x * t , θ * ⟩ - T t=1 ⟨x t , θ * ⟩ = T t=1 ⟨x * t -x t , θ * ⟩, (4.16)
where x * t = arg max x∈X ⟨x, θ * ⟩. Traditionally, the following additional assumption is made.

Assumption 4.14. for all

x t ∈ X ⟨x t , θ * ⟩ ∈ [-1, 1] .
The "optimism in the face of uncertainty linear bandit" (OFUL) algorithm was introduced in (Abbasi-Yadkori, Pál, and Szepesvári, 2011). OFUL resorts to ridge regression, constructs a confidence ellipsoid for the parameter estimate, and chooses the action that maximizes the upperconfidence bound on the reward. Under Assumption 4.14, (Abbasi-Yadkori, Pál, and Szepesvári, 2011) we prove that the cumulative regret of OFUL satisfies, for δ > 0 with probability at least

1 -δ, ∀T > 0 R r T ≤ 4 T d log(λ + T X 2 /d) λ 1/2 S + σ 2 log(1/δ) + d log(1 + T X 2 /(λd)) ,
where X = max 1≤t≤T ∥x t ∥ 2 .

Forward variant [Algorithm 4.3]:

In a second phase, we propose the variant OFUL f in which we replace ridge regression by the forward algorithm. What this means is that the parameter estimate is a function of actions:

θ f t (x) = arg min θ∈R d t s=1 (y s -⟨x s , θ⟩) 2 + λ∥θ∥ 2 2 + ⟨x, θ⟩ 2 .

Linear regression: an improved algorithm & application to linear bandits

This fits perfectly since actions can be chosen. Implementation details are in Algorithm 4.3.

Algorithm 4.3: OFUL

f algorithm 1 Given λ, δ, S > 0 2 for t = 1, . . . , T do 3 x t = arg max x∈X ⟨x, θ f t (x)⟩ + ∥x∥ G -1 t-1,x ( √ λ + ∥x∥ 2 )S + σ 2 log (1+tX 2 t (x)/λd) d/2 δ , 4
where

X t (x) = max{∥x∥ 2 , max 1≤s≤t-1 ∥x s ∥ 2 }, G t-1,x = G t-1 + xx ⊤ and θ f t (x) = arg min θ∈R d t-1 s=1 (y s -⟨x s , θ⟩) 2 + λ∥θ∥ 2 2 + ⟨x, θ⟩ 2 5
play x t and observe y t .

end

Note that OFUL f only requires an upper bound S on ∥θ * ∥ 2 . We prove that OFUL f enjoys the same regret bound as OFUL and doesn't require Assumption 4.14. In stark contrast, we cannot show a similar bound for the standard OFUL without said assumption, it actually suffers a λ-dependent scaling factor in this case. Theorem 4.15. (Bandits with unbounded rewards) Without Assumption 4.14, for all δ > 0, OFUL r achieves with probability at least 1 -δ, for all T ≥ 1,

R r T ≤ 4 X X X 2 T d log(λ + T X 2 /d) λ λ λ log(1 + X X X 2 /λ λ λ) λ 1/2 S +σ 2 log(1/δ) + d log(1 + T X 2 /(λd)) ,
also, we show that for all δ > 0, OFUL f achieves with probability at least 1 -δ, for all T ≥ 1:

R f T ≤ 4 T d log(λ + T X 2 /d) (λ 1/2 + X)S + σ 2 log(1/δ) + d log(1 + T X 2 /(λd)) .
Proof. OFUL with forward regression: Lets decompose the instantaneous regret as follows:

r t = ⟨θ * , x * ⟩ -⟨θ * , x t ⟩ ≤ θt , x t -⟨θ * , x t ⟩ = θt -θ * , x t = θ t-1 -θ * , x t + θt -θ t-1 , x t = θ t-1 -θ * (G t-1 +xtx ⊤ t ) ∥X t ∥ (G t-1 +xtx ⊤ t ) -1 + θt -θ t-1 (G t-1 +xtx ⊤ t ) ∥x t ∥ (G t-1 +xtx ⊤ t ) -1 ≤ 2 β t-1 (x t , δ) ∥x t ∥ (G t-1 +xtx ⊤ t ) -1 , (4.17)
where θ t is the optimistic parameter estimate, i.e. the θ ∈ C t (x t ) that maximizes the upper confidence bound on the reward of action x t . The first inequality is since X t , θ t is optimistic, and the last step holds by Cauchy-Schwarz. Using Inequality (4.17) and the expression of the 4.5 Application: linear bandits confidence interval (C t (x)) for the forward algorithm at the action x:

θ ∈ R d : ||θ f t -θ|| Gt+xx ⊤ ≤ β t (x, δ) = ( √ λ + ||x|| 2 )S + σ 2 log (1 + tX 2 /λd) d/2 δ
we get that, with probability at least 1 -δ, for all n ≥ 0

R n ≤ n n t=1 r 2 t ≤ 8β n (δ)n n t=1 ∥x t ∥ (G t-1 +xtx ⊤ t ) -1 ≤ 4 nd log(λ + nL/d) (λ 1/2 + X)S + σ 2 log(1/δ) + d log(1 + nL/(λd))
where the last step follow from Lemma B.2.

OFUL with ridge regression: Now we derive a novel regret bound for online ridge regression, one that doesn't require the bounded rewards, i.e. Assumption 4.14. The proof follows exactly like the above one except the last step (control of the norm of actions) that now proceeds using Lemma B.3. The first step is to use the confidence ellipsoid for the ridge regression parameter (cf Theorem 2 of Abbasi-Yadkori, Pál, and Szepesvári (2011)). With probability at least 1 -δ, for all t ≥ 0, θ * lies in the set

C t =    θ ∈ R d : ∥θ r t -θ∥ Gt ≤ β t (δ) = σ d log 1 + tX 2 /λd δ + λ 1/2 S    .
Then

r t = ⟨θ * , x * ⟩ -⟨θ * , x t ⟩ ≤ θt , x t -⟨θ * , x t ⟩ = θt -θ * , x t = θ t-1 -θ * , x t + θt -θ t-1 , x t = θ t-1 -θ * G t-1 ∥X t ∥ G -1 t-1 + θt -θ t-1 G t-1 ∥x t ∥ G -1 t-1 ≤ 2 β t-1 (δ) ∥x t ∥ G -1 t-1 (4.18)
where θ t is the optimistic parameter estimate, i.e. the θ ∈ C t that maximizes the upper confidence bound on the reward of action x t . The first inequality is since X t , θ t is optimistic, and the last step holds by Cauchy-Schwarz. Using Inequality (4.18) we get that, with probability at least 1 -δ, for all n ≥ 0

R n ≤ n n t=1 r 2 t ≤ 8β n (δ)n n t=1 ∥x t ∥ G -1 t-1 ≤ 4 ndX 2 log(1 + nX 2 /λd) λ log(1 + X 2 /λ) λ 1/2 S + σ 2 log(1/δ) + d log(1 + nX 2 /(λd))

Linear regression: an improved algorithm & application to linear bandits

where the last step follow from Lemma B.3.

Remark 4.16. we can drop the dependence on X and S by bounding the second term in the index of OFUL and OFUL f (see line 285) by XS(1 + X/ √ λ) and then dropping this -constant-term at the expense of a looser index. Therefore, knowing the bounds x and S is not crucial. Furthermore, while we choose to adopt the pseudo-regret definition like in (Abbasi-Yadkori, Pál, and Szepesvári, 2011), we could also derive similar bounds for the regret involving rewards y t = ⟨x t , θ * ⟩ instead of their expected value, (y t ) t≥1 are unbounded.

Experiment We provide experimental evidence that the OFUL f variant improves OFUL for linear bandits; we find that it is generally as good as the standard OFUL r , and in some cases it can prove to be significantly more robust to aberrant regularization parameters. We consider a 100-dimensional linear bandit with 10 arms, the parameter vector is drawn from the unit ball, actions are such that ∥x t ∥ ≤ 200. Noise ε t L = N (0, 10 -1 ), λ = 10 -5 , δ = 10 In Figure 4.3, lines are average regret over 100 repetitions and shaded areas cover the region between dashed-lines that are the first and third quartiles. We observe that -as predicted by Theorem 4.15: OFUL f is particularly robust and choosing λ = 1/T incurs substantial regret for OFUL. Because of this phenomena, and for the same observations in the online stochastic regression setting, we advocate for the use of the forward algorithm instead of ridge regression whenever possible, to take advantage of its increased robustness to λ.

Application: linear bandits

Remark 4.17. Regarding the choice λ = 1/T : we use this specific regularization for two reasons: 1) to demonstrate the benefits of our stochastic analysis, since previous deterministic bounds suggest this λ is best, 2) to showcase the increased robustness of OFUL f compared to OFUL. In fact, more often than not, OFUL performs as good as OFUL, except when λ is small or X is large.

Non-stationary bandits

In this section, we study linear stochastic bandits in the non-stationary setting. We analyze then provide an experimental study of this setup. Consider non-stationary stochastic linear bandits, where the target parameter is varying with time:

θ * = θ * (t) ∈ R d , assuming that T -1 s=1 ∥θ * (s) -θ * (s + 1)∥ 2 ≤ B T .
One of the optimal algorithms in this setting is D-LinUCB of (Russac, Vernade, and Cappé, 2019), it defines θ t as

θ t = arg min θ∈R d t s=1 γ t-s (y s -⟨x s , θ⟩) 2 + λ/2∥θ∥ 2 2 .
D-LinUCB proceeds as follows:

Algorithm 4.4: D-LinUCB

1 Input: δ, σ, λ, X, S, γ > 0, dimension d ∈ N * . 2 Initialization: b = 0 R d , V = λI d , V = λI d , θ = 0 R d 3 for t ≥ 1 do 4 Receive X , compute β t-1 = √ λS + σ 2 log 1 δ + d log 1 + X 2 (1-γ 2(t-1) ) λd(1-γ 2 ) 5 for a ∈ X do 6 Compute UCB(a) = a ⊤ θ + β t-1 a ⊤ V -1 V V -1 a 7 A t = arg max a (UCB(a)) 8
Play action A t and receive reward X t 9

Updating phase:

V = γV + x t x ⊤ t + (1 -γ)λI d , V = γ 2 V + x t x ⊤ t + (1 -γ 2 )λI d 10 b = γb + Y t X t , θ = V -1 b
We recall the regret bound of standard D-LinUCB .

Linear regression: an improved algorithm & application to linear bandits

and ∀x ∈ X , t ≥ 1 : ⟨x, θ t ⟩ ≤ 1, the regret of the D-LinUCB algorithm is bounded for all γ, δ ∈ (0, 1) and integer D ≥ 1, with probability at least 1 -δ, by:

R r T ≤ 2XDB T + 4X 3 S λ γ D 1 -γ T + 2 √ 2β T √ dT × T log(1/γ) + log 1 + X 2 dλ(1 -γ) ,
where β T is the width of the confidence interval for θ * (T ).

Now we introduce D-LinUCB f , which uses the forward algorithm and defines an action dependent θ t as:

arg min θ∈R d t s=1 γ t-s (y s -⟨x s , θ⟩) 2 + λ/2∥θ∥
R f T ≤ 2XDB T + 4X 3 S λ γ D 1 -γ T + 2β T √ dT T log(1/γ) + log 1 + (2 -γ)X 2 dλ(1 -γ) .
Proof. This result is again a modification of the original proof consisting in bounding the sum of the actions' norms differently. Let us recall the notations

V t = t s=1 w s x s x ⊤ s + λ t I d + xx ⊤ and Ṽt = t s=1 w 2 s x s x ⊤ s + µ t I d + xx ⊤ .
To summarize the difference of this analysis -that no longer requires a bounded rewards assumption-at the step where we bound the sum of actions' norms, we replace Proposition 4 of Russac, Vernade, and Cappé (2019):

T t=1 min 1, ∥x t ∥ 2 V -1 t-1 Ṽt-1 V -1 t-1 ≤ 2 T t=1 log 1 + γ -t ∥x t ∥ 2 V -1 t-1 ≤ 2 log det (V T ) λ d ,
that requires the predictions to lie in the same range as the rewards with this inequality for

D-LinUCB f T t=1 ∥x t ∥ 2 V -1 t ṼtV -1 t ≤ T t=1 log 1 + γ -t ∥x t ∥ 2 V -1 t ≤ log det (V T ) λ d .
We don't provide the full proof of this result as it is cumbersome and not of special interest for our purposes since it is similar to the analysis for D-LinUCB except for the inequality above.

Remark 4.20. This result is fascinating as it first allows to remove an unnecessary assumption, and further yields a better bound than D-LinUCB r which suffers the factor X X X √ 2 λ λ λ log(1+X X X/λ λ λ) in its last regret term without Assumption 4.14.

Experiments for non-stationary linear bandits:

We now reproduce the experiments of (Russac, Vernade, and Cappé, 2019) for non-stationary linear bandits, and add D-LinUCB f to the pool of algorithms. We first simulate an abruptly changing environment of dimension 2 with 3 changes: for t < 10 3 : θ * = (1, 0); for 10 3 ≤ t ≤ 2.10 3 : θ * = (-1, 0); for 2.10 3 < t < 3.10 3 : θ * = (0, 1); for t > 3.10 3 : θ * = (0, -1). We observe in Figure 4.4a that both variants of D-LinUCB compare on par. Here LinUCB-OR denotes an oracle knowing the change points.

Second, we simulate a slowly changing environment where the parameter θ * starts at (1, 0) and moves counter-clockwise on the unit-circle up to the position (0, 1) in 3.10 3 steps then remains there, B T = 1.57. We see the results in Figure 4.4b, where we notice that in this setting as well, D-LinUCB f has very similar performance to standard D-LinUCB .

Remark 4.21.

In both experiments, we also reported the performances of SW-LinUCB, that is alternative version to D-LinUCB. SW-LinUCB is better suited for abrupt changes while D-LinUCB is better suited for slow changes.

Note that we added these final experiments to demonstrate the competitiveness of algorithms that use forward regression against their ridge counterparts in the same settings that were used by previous works. While we could have specified specific parameters to illustrate the robustness to regularization of algorithms that incorporate the forward algorithm; we estimate that the experiments presented in the main text already fulfilled this objective. Again, the purpose here is to show that using the forward algorithm improves the theoretical guarantees without deteriorating the performance.

Linear regression: an improved algorithm & application to linear bandits

Discussion

We revisited the analysis of online linear regression algorithms in the setup of stochastic, possibly unbounded observations. We proved high probability regret bounds for three popular online regression algorithms (cf Theorems 4.7, 4.9 and 4.11). These bounds provide novel understanding of online regression. In particular, Theorem 4.7 seems to be the first regret bound for ridge regression that does not require bounded predictions or prior knowledge of a bound on observations. Our novel bounds seem to correctly capture the nature of dependence with regularization, as indicated by Figure 4.2. Moreover, a new results from Tirinzoni, Pirotta, et al. ( 2020) can be incorporated in the proof mechanism to bring the high probability upper bounds to O(dσ 2 log(T ) log log(T )), which matches the optimal achievable bounds from the excess risk literature up to sub-logarithmic factors. Furthermore, we argue that replacing ridge regression by the forward algorithm whenever possible in algorithms that require linear approximations can be beneficial, we depict this in a case study involving linear bandits: First from a theoretical standpoint our results show that the OFUL f algorithm enjoys the classic first order regret bound while dropping Assumption 4.14; Second, we find that empirically, implementing OFUL with the forward algorithm makes the algorithm significantly more robust to extreme values of regularization, which is of practical interest. More broadly, we believe that the improvement resulting from replacing ridge regression with the forward algorithm could be extended to several other settings. For instance, Graph bandits are of interest as well: they consider linear function approximations using ridge regression, and make Assumption 4.14, see for example Theorem 1 of [START_REF] Valko | Spectral bandits for smooth graph functions[END_REF]; Meta-learning with linear bandits can also be enhanced using forward regression: see for example Lemma 1 and consequent results in [START_REF] Cella | Meta-learning with stochastic linear bandits[END_REF].

Chapter 5 Continuous MDPs: the Bilinear Exponential Family representation

We study the problem of episodic reinforcement learning in continuous state-action spaces with unknown rewards and transitions. Specifically, we consider the setting where the rewards and transitions are modeled using parametric bilinear exponential families. We propose an algorithm, BEF-RLSVI, that a) uses penalized maximum likelihood estimators to learn the unknown parameters, b) injects a calibrated Gaussian noise in the parameter of rewards to ensure exploration, and c) leverages linearity of the bilinear exponential family transitions with respect to an underlying RKHS to perform tractable planning. We further provide a frequentist regret analysis of BEF-RLSVI that yields an upper bound of Õ √ d 3 H 3 K , where d is the dimension of the parameters, H is the episode length, and K is the number of episodes. Our analysis improves the existing bounds for the bilinear exponential family of MDPs by √ H and removes the handcrafted clipping deployed in existing RLSVI-type algorithms. Our regret bound is order-optimal with respect to H and K.

Introduction

Reinforcement Learning (RL) is a well-studied and popular framework for sequential decision making, where an agent aims to compute a policy that allows her to maximize the accumulated reward over a horizon by interacting with an unknown environment [START_REF] Richard | Reinforcement learning: An introduction[END_REF].

Episodic
V π h (s) = ∆ E H t=h r(s t , a t ) | s h = s , Q π h (s, a) = ∆ E H t=h r(s t , a t ) | s h = s, a h = a .
Here, computing the policy leading to maximization of cumulative reward requires the agent to strategically control the actions in order to learn the transition functions and reward functions as precisely as required. This tension between learning the unknown environment and reward maximization is quantified as regret: the typical performance measure of an episodic RL algorithm. Regret is defined as the difference between the expected cumulative reward or value collected by the optimal agent that knows the environment and the expected cumulative reward or value obtained by an agent that has to learn about the unknown environment. Formally, the regret over K episodes is

R(K) ≜ K k=1 V π ⋆ 1 (s k 1 ) -V πt 1 (s k 1 )
.

Key Challenges. The first challenge in episodic RL is to tackle the exploration-exploitation trade-off.

This is traditionally addressed with the optimism principle that either carefully crafts optimistic upper bounds on the value functions (Azar, Osband, and Munos, 2017), or maintains a posterior on the parameters to perform posterior sampling [START_REF] Osband | More) efficient reinforcement learning via posterior sampling[END_REF], or perturbs the value function estimates with calibrated noise [START_REF] Osband | Generalization and exploration via randomized value functions[END_REF]. Though the first two approaches induce theoretically optimal exploration, they might not yield tractable algorithms for large/continuous state-action spaces as they either involve optimization in the optimistic set or maintaining a high-dimensional posterior. Thus, we focus on extending

Introduction

the third approach of Randomized Least-Square Value Iteration (RLSVI) framework, and inject noise only in rewards to perform tractable exploration.

The second challenge, which emerges for continuous state-action spaces, is to learn a parametric functional approximation of either the value function or the rewards and transitions in order to perform planning and exploration. Different functional representations (or models), such as linear [START_REF] Jin | Provably efficient reinforcement learning with linear function approximation[END_REF] [START_REF] Kearns | Efficient reinforcement learning in factored MDPs[END_REF], and linearly controlled dynamical systems (such as Linear Quadratic Regulators (Abbasi-Yadkori and Szepesvári, 2011)) as special cases (Chowdhury, [START_REF] Sayak | Reinforcement Learning in Parametric MDPs with Exponential Families[END_REF]. Thus, in this paper, we study the BEF of MDPs, i.e. the episodic RL setting where the rewards and transition functions can be modeled with bilinear exponential families.

The third challenge is to perform tractable planning 2 given the perturbation for exploration and the model class. Existing work [START_REF] Osband | Model-based reinforcement learning and the eluder dimension[END_REF]; Chowdhury, Gopalan, and Maillard, 2021) assumes an oracle to perform planning and yield policies that aren't explicit. The main difficulty in such planning approaches is calculating P(s ′ | s, a)V h (s) for all (s, a) pairs. This is not trivial unless the transition is assumed to be linear and decouples s ′ from (s, a), which is not known to hold except for tabular MDPs. This challenge received attention recently, e.g. [START_REF] Simon S Du | Is a good representation sufficient for sample efficient reinforcement learning?[END_REF] asks when misspecified linear representations are enough for a polynomial sample complexity in several settings. (Shariff and Szepesvári, 2020; Lattimore, Szepesvari, and Weisz, 2020; Van Roy and Dong, 2019) provide positive answers for certain linear settings. In this paper, we aim to design a tractable planner for the BEF representation.

In this paper, we aim to address the following question that encompasses the three challenges:

Can we design an algorithm with tractable exploration and planning for the bilinear exponential family of MDPs yielding a near-optimal frequentist regret bound?

Contributions. We address this question in three folds.

1. Formalism: We assume that neither rewards nor transitions are known, previous efforts on the bilinear exponential family of MDPs assumed knowledge of rewards. This makes the addressed problem harder, practical, and more general. We also observe that though the

Continuous MDPs: the Bilinear Exponential Family representation

√ d 4 H 5 K ✓ ✓ ✗ Linear V BEF-RLSVI (this work) √ d 3 H 3 K ✓ ✓ ✓
Bilinear Exp Family transition model can represent non-linear dynamics, it implies a linear behavior (see Section 5.2) in a Reproducible Kernel Hilbert Space (RKHS). This observation contributes to the tractability of planning.

Algorithm:

We propose an algorithm BEF-RLSVI that extends the RLSVI framework to bilinear exponential families (cf Section 5.3). BEF-RLSVI a) injects calibrated Gaussian noise in the rewards to perform exploration, b) leverages linearity of the transitions with respect to an underlying RKHS to perform tractable planning and c) uses penalized maximum likelihood to learn the parameters corresponding to rewards and transitions (cf Section 5.4). To the best of our knowledge, BEF-RLSVI is the first algorithm for the bilinear exponential family of MDPs with tractable exploration and planning under unknown rewards and transitions.

Analysis:

We carefully develop an analysis of BEF-RLSVI that yields Õ √ d 3 H 3 K regret which improves the existing regret bound for the BEF of MDPs with known rewards by a factor of √ H (Section 5.3.2). Our analysis builds on existing analyses of RLSVI-type algorithms (Osband, [START_REF] Osband | Generalization and exploration via randomized value functions[END_REF], but contrary to them, we remove the need to handcraft a clipping of the value functions [START_REF] Zanette | Frequentist regret bounds for randomized least-squares value iteration[END_REF]. We also do not need to assume anti-concentration bounds as we can explicitly control it by the injected noise. This was not done previously except for the linear MDPs. We illustrate this comparison in Table 5.1. We highlight three technical tools that we used to improve the previous analyses: 1) Using transportation inequalities instead of the simulation lemma reduces a √ H factor compared to [START_REF] Ren | A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning[END_REF], 2) Leveraging the observation that true value functions are bounded enables using an improved elliptical lemma (compared to (Chowdhury, Gopalan, and Maillard, 2021)), and 3) Noticing that the norm of features can only be large for a finite amount of time allows us to forgo clipping and reduce a √ d factor from the regret compared to (Zanette, Brandfonbrener, et al., 2020).

Bilinear exponential family of MDPs

Bilinear exponential family of MDPs

We introduce the BEF model (Chowdhury, Gopalan, and Maillard, 2021) and extend it to parametric rewards. Then, we make an important observation of linearity. Definition 5.1 (Bilinear exponential family model). We consider both transition and reward kernels to be unknown and modeled with bilinear exponential families. Specifically,

P (s | s, a) = exp ψ(s) ⊤ M θ p φ(s, a) -Z p s,a (θ p ) , ( 5.1)
P (r | s, a) = exp r B ⊤ M θ r φ(s, a) -Z r s,a (θ r ) , (5.2)
where φ ∈ (R q + ) S×A and ψ ∈ (R p + ) S are known feature functions, and B ∈ R p is a known scaling factor. The unknown reward and transition parameters are

θ p , θ r ∈ R d . M θ • = ∆ d i=1 θ • i A i
, where A i is a known p × q matrix for each i. Finally, Z denotes the log partition function:

Z p s,a (θ p ) = ∆ log S exp ψ(s) ⊤ M θ p φ(s, a) ds,
Z r is defined similarly.

A minor difference with the original BEF model and the one stated here is that, like (Li, Li, et al., 2021), we omit a base measure of the form h(s, s, a), all the BEF examples provided in (Chowdhury, Gopalan, and Maillard, 2021) still hold with this slight restriction. We denote V π θ p ,θ r ,h , (resp. Q π θ p ,θ r ,h ) the value (resp. state-action) value function for policy π in the MDP parameterized by (θ p , θ r ) at time h. A policy π ⋆ is optimal if for all s ∈ S, V π ⋆ θ,h (s) = max π∈Π V π θ,h (s). A learning algorithm minimizes the (pseudo) regret:

R(K) ≜ K k=1 V π ⋆ θ,1 (s k 1 ) -V π t θ,1 (s k 1 ) . (5.3)
Linearity of transitions. Now, we state an observation about the bilinear exponential family and discuss how it helps with the challenge of planning in episodic RL. Specifically, the popular assumption of linearity of the transition kernel is a direct consequence of our model. Indeed,

2ψ s ′ ⊤ M θ p φ(s, a) = -∥(ψ(s ′ ) -M θ p φ(s, a)∥ 2 + ∥ψ(s ′ )∥ 2 + ∥M θ p φ(s, a)∥ 2 .
Notice that the quadratic term is the Radial Basis Function (RBF) kernel. More precisely, for an RBF kernel with covariance Σ = I p and k(x, y) = ∆ exp -∥x -y∥ 2 /2 , we find

P s ′ | s, a = ⟨ϕ p (s, a), µ p (s ′ )⟩ H , (5.4)

Continuous MDPs: the Bilinear Exponential Family representation

where H is the RKHS associated with the k(., .), and

µ p (s ′ ) = (2π) -p/2 k ψ(s ′ ), . exp ∥ψ s ′ ∥ 2 /2 ϕ p (s, a) = k M ⊤ θ p φ(s, a), . exp ∥M ⊤ θ p φ(s, a)∥ 2 2 -Z s,a (θ p )
In Equation (5.4), s ′ is decoupled from (s, a), we see hereafter why this is crucial to reducing the complexity of planning.

Remark 5.2. Up to our knowledge, [START_REF] Ren | A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning[END_REF] is the only work providing an example of linear transitions for RL with continuous state-actions. They consider Gaussian transitions with an unknown mean (f ⋆ (s, a)) and known variance. It is a special case of the BEF model, where

ψ(s ′ ) = (s ′ , ∥s ′ ∥ 2 ) and M θ φ(s, a) = (f θ (s, a)/σ 2 , -1/σ 2 ).
Importance of linearity. To understand the planning challenge in RL, recall the Bellman equation: When ϕ p , µ p ∈ R τ , we can obtain Qh by computing τ integrals per timestep, reducing the stateaction space complexity to τ only. For our model, although ϕ p and µ p are infinite dimensional, we show in Section 5.4 ( § planning) that the planning complexity is still significantly reduced.

Q π h (s, a) = r(s, a) + s∈S P (s ′ | s, a)V π h+1 (s)

BEF-RLSVI: algorithm design and frequentist regret bound

We formally introduce and the Bilinear Exponential Family Randomized Least Squares Value Iteration algorithm (BEF-RLSVI) and provide a high probability regret bound.

BEF-RLSVI: algorithm design

BEF-RLSVI is based on RLSVI (Osband, Van Roy, and Wen, 2016) except it perturb the reward parameter only. The latter is reminiscent of Thompson Sampling, yet more explicit and with a better control of the optimism probability.

BEF-RLSVI: algorithm design and frequentist regret bound

Algorithm 5.1: BEF-RLSVI

Sample noise ξ k ∼ N 0, x k ( Ḡp k ) -1 such that Ḡp k = η α p A + k-1 τ =1 H h=1 (φ(s τ h , a τ h ) ⊤ A ⊤ i A j φ(s τ h , a τ h )) i,j∈[d]
5:

Perturb reward parameter: Update the penalized ML estimators θp (k), θr (k), see Equation (5.6) and Equation (5.7) 12: end for In line 4, BEF-RLSVI performs exploration by a Gaussian perturbation of the reward parameter. Contrary to optimistic approaches, this method is explicit and more efficient since it does not a involve high-dimensional optimization. Calculate Q θ,h (s, a) = E θr s,a [r] + ⟨ϕ p (s, a), V θ,h+1 (s ′ )µ p (s ′ )ds ′ ⟩ H 4: end for Line 3 can be approximated with O pH 3 K log(HK) complexity without damaging the sample complexity (cf § planning, Section 5.4). Therefore, planning is tractable.

θr (k) = θr (k) + ξ k 6: Compute (Q k θp ,
Remark 5.3. The observation of linearity (cf Equation (5.5) and Line 3) does not reduce BEF MDPs to linear MDPs because the former holds in an RKHS. Also, linearity is not in the representation parameter. Therefore, linear RL algorithms do not readily solve the BEF MDPs.

BEF-RLSVI: regret upper-bound

We state the standard smoothness assumptions on the model (Chowdhury, [START_REF] Sayak | Reinforcement Learning in Parametric MDPs with Exponential Families[END_REF][START_REF] Jun | Scalable generalized linear bandits: Online computation and hashing[END_REF][START_REF] Lu | Low-rank generalized linear bandit problems[END_REF].

Continuous MDPs: the Bilinear Exponential Family representation Assumption 5.4.

There exist constants α p , α r , β p , β r > 0, such that the representation model satisfies, for all (s, a) ∈ S × A , and for all θ, x ∈ R d

α p ≤ x ⊤ C θ s,a [ψ]x ≤ β p , α r ≤ Var θ s,a (r) x ⊤ B ⊤ Bx ≤ β r , where Var θ s,a (r) ≜ E θ s,a r 2 -E θ s,a [r] 2 , and C θ s,a [ψ (s ′ )] ≜ E P θ |s,a ∥ψ (s ′ ) ∥ 2 - E P θ |s,a ψ (s ′ ) 2
These inequalities imply a control over the eigenvalues of the Hessian matrices of the log-normalizers (cf Appendix C.2.3). We now state our main result.

Theorem 5.5. [Regret bound] Let

A ≜ ( ⊤ (A i A ⊤ j )) i,j∈[d] and G s,a ≜ (φ(s, a) ⊤ A ⊤ i A j φ(s, a)) i,j∈[d]
. Under Assumption 5.4 and further considering that

1. max{∥θ r ∥ A , ∥θ p ∥ A } ≤ B A , ∥A -1 G s,a ∥ ≤ B φ,A and E θ r [r(s, a)] ∈ [0, 1] for all (s, a). 2. noise ξ k ∼ N (0, x k ( Ḡp k ) -1 ) satisfies x k ≥ H β p β p (K,δ) α p α r + β r β r (K,δ) min{1, α p α r } 2α r 2 ∝ dH 2 ,
then for all δ ∈ (0, 1], with probability at least 1 -δ,

R(K) ≤ Hγ r K β r C d   β r (K, δ) 2α r +c x K d log(dK/δ)   Estimation error for no clipping ≈ dH + β r d √ x K Φ(-1) (1+ log(d/δ)) C d 1+ α r B φ,A H η Learning error for no clipping ≈ (dH) 3/2 + c β r x K dγ r K log(dK/δ) + β r x K dγ r K log(e/δ 2 ) Φ(-1) 1 + log(d/δ) Noise concentration ≈ d 3/2 H + β r β r (n, δ)γ r K 2α r Reward concentration ≈ d + 2H   2β p α p β p (K, δ)γ p K +(1+ γ r K ) log(1/δ 2 )   Transition concentration ≈ dH √ KH where for i ∈ [p, r], β i (K, δ) ≜ η 2 B 2 A + γ i K + log(1/δ), and
γ i K ≜ d log(1 + β i η B φ,A HK). Also, C d ≜ 3d log(2) log 1 + α r ∥A∥ 2 2 B 2 φ,A η log(2)
, Φ is the Gaussian CDF, and c is a universal constant.

Algorithm design

Theorem 5.5 entails a regret R(K) = O √ d 3 H 3 K for BEF-RLSVI, where d is the number of parameters of the bilinear exponential family model, K is the number of episodes, and H is the horizon of an episode. We now clarify how this contrasts with related literature.

Comparison with other bounds. The closest result is from (Chowdhury, Gopalan, and Maillard, 2021), it considers the same model for transitions but with known rewards. They propose a UCRL-type and PSRL-type algorithm, which achieve a O(

√ d 2 H 4 K) regret.
There are two notable algorithmic differences with BEF-RLSVI. First, they use intractable-optimistic upper bounds or high-dimensional posteriors, while we do explore with explicit perturbations. The second difference is in planning: while they assume access to a planning oracle, we do it explicitly with pseudo-polynomial complexity (Section 5. [START_REF] Sayak | Reinforcement Learning in Parametric MDPs with Exponential Families[END_REF]). In this case, our bound becomes R(K) = O( (S 2 A) 3 H 3 K), which is clearly not tight is S and A. This is understandable due to the relative generality of our setting. We are however positively surprised that our bound is tight in terms of its dependence on H and K.

Algorithm design

We discuss our choices for the design of BEF-RLSVI as well as the justification of tractability of some essential sub-procedures.

Continuous MDPs: the Bilinear Exponential Family representation

Building blocks of BEF-RLSVI

We present necessary details about BEF-RLSVI and discuss the key algorithm design techniques.

Estimation of parameters.

We estimate transitions and rewards from observations similar to EXP-UCRL (Chowdhury, Gopalan, and Maillard, 2021), i.e. by using a penalized maximum likelihood estimator θp (k) ∈ arg min

θ∈R d 1≤t≤k 1≤h≤H -log P θ s t h+1 | s t h , a t h + η pen(θ).
Here, pen(θ) is the trace-norm penalty: pen(θ) = 1 2 ∥θ∥ A where A = (tr(A i A ⊤ j )) i,j . By properties of the exponential family, the penalized ML estimator verifies, for i ≤ d:

1≤t≤k 1≤h≤H ψ s t h+1 -E θp k s t h ,a t h ψ s ′ ⊤ A i φ s t h , a t h = η∇ i pen θp k .
(5.6)

The above can be solved in closed form for simple distributions, like Gaussian, but it can be involved for other distribution (cf Appendix 5.4.2). For the reward, θ r is defined similarly:

θr (k) ∈ arg min θ∈R d 1≤t≤k 1≤h≤H -log P θ r t | s t h , a t h + η pen(θ),
Then, for all i ∈ [d]:

1≤t≤k 1≤h≤H r t -E θr k s t h ,a t h [r] B ⊤ A i φ s t h , a t h = η∇ i pen θr k (5.7)
Exploration. A significant challenge in RL is handling exploration in continuous spaces. The majority of the literature is split between intractable, upper confidence bound-style optimism or Thompson sampling algorithms with high-dimensional posterior and guarantees only in terms of Bayesian regret. In BEF-RLSVI, we adopt the approach of reward perturbation motivated by the RLSVI-framework (Zanette, Brandfonbrener, et al., 2020; Osband, Van Roy, and Wen, 2016). We show that perturbing the reward estimation can guarantee optimism with a constant probability, i.e. there exists ν ∈ (0, 1] such that for all k ∈ [K] and s k 1 ∈ S, [START_REF] Zanette | Frequentist regret bounds for randomized least-squares value iteration[END_REF] proves that this suffices to bound the learning error. However, their method clashes with not clipping the value function, as it modifies the probability of optimism. Thus, [START_REF] Zanette | Frequentist regret bounds for randomized least-squares value iteration[END_REF] proposes an involved clipping

P Ṽ1 (s k 1 ) -V ⋆ 1 (s k 1 ) ≥ 0 ≥ ν. ( Zanette

Algorithm design

procedure to handle the issue of unstable values. Instead, by careful geometric analysis (cf Lemma C.14), we bound the occurrences of the unstable values, and in turn, upper bound the regret without clipping. Note that unlike [START_REF] Ishfaq | Randomized Exploration in Reinforcement Learning with General Value Function Approximation[END_REF], BEF-RLSVI does not guarantee that the estimated value function is optimistic but still is able to control the learning error (cf Section 5.5).

Planning.

Recall that with our model assumptions, we can write the state-action value function linearly (Equation (5.5)). Using BEF-RLSVI, we have at step h:

Q π θp , θr ,h (s, a) = E θr [r(s, a)] + ϕ p (s, a), S µ p (s)V π θp , θr ,h+1 (s)ds .
Then, we select the best action greedily to compute Q h (s, a). Although ϕ p and ψ p are infinitedimensional, we show (next paragraph) that an approximation with dimensionality of order O pH 2 K log(HK) is possible, and that it doesn't increase the regret. Thus, the planning can be done in O pH 3 K log(HK)) , which is pseudo-polynomial in p, H and K, ergo tractable.

For details about the finite-dimensional approximation of our transition kernel, refer to Subsection 5.4.2. Now, we highlight the schematic of a finite-dimensional approximation of ϕ p and ψ p . We proceed in three steps. 1) We have with high probability S(V θp , θr ,h ) ≤ dH 3/2 (Section 5.5). 2) If we have a uniform ε-approximation of P θ p , we show that using it incurs at most an extra O εdH 5/2 K regret. 3) Finally, following [START_REF] Rahimi | Random features for large-scale kernel machines[END_REF], we approximate uniformly the shift invariant kernels, here the RBF in Equation (5.4), within ε error and with features of dimensions O pε -2 log 1 ε 2 ) , where p is dimension of ψ. Associating these three elements and choosing ε = 1/ (H 2 K), we establish our claim.

Tractable Planning and Maximum likelihood estimation

A Primer on random Fourier transforms. We start by defining the Random Fourier Transform and its most relevant property. Let us consider the transition model of Equation (5.1), we have andp(w, b) is a coupling among them.

P(s ′ | s, a, θ) = exp ψ(s ′ )M θ φ(s, a) -Z θ (s, a) = E p(w,b) f ψ(s ′ ), w, b f (M θ φ(s, a), w, b) , where f (x, w, b) = √ 2 cos(w ⊤ x + b) are the random Fourier bases. p(w, b) = N (0, σ -2 I) × U([0, 2π]), such that N is the Gaussian distribution, U is the Uniform distribution,
Notice that this provides an alternative approach to decompose the transition kernel and obtain linearity of the value function. Moreover, since ∀x,

w ∈ R d , b ∈ R, |f (x, w, b)| ≤ √ 2,
we can use Hoeffding's inequality to prove that a Monte-Carlo approximation of P(s ′ | s, a, θ) using N sample pairs of (w, b) guarantees an error smaller than ε with probability at least Continuous MDPs: the Bilinear Exponential Family representation 1 -2 exp(-N ε 2 /4). [START_REF] Rahimi | Random features for large-scale kernel machines[END_REF] proves a stronger result: it provides an algorithm approximating the Gaussian kernel for which the following uniform convergence bound holds. Lemma 5.6. Let M be a compact subset of R p with diameter diam(M). Then, using the explicit mapping z defined in Algorithm 1 in [START_REF] Rahimi | Random features for large-scale kernel machines[END_REF] with N samples, we have Pr sup

x,y∈M z(x) ′ z(y) -k(y, x) ≥ ε ≤ 2 8 σ p diam(M) ε 2 exp - N ε 2 4(p + 2)
where

σ 2 p ≡ E p [ω ′ ω]
is the second moment of the Fourier transform of k.

Further, it implies that if N = Ω p ε 2 log σp diam(M) ε , then sup x,y∈M |z(x) ′ z(y) -k(y, x)| ≤ ε with constant probability.
Application to planning in BEF-RLSVI. Since our regret analysis is done under the high probability event of bounded estimation parameters, we know that the spaces of ψ(s ′ ) and M θ φ(s, a) are bounded and the diameter depends on the dimensions. We abstain from explicating the exact diameter as it only influences the number of samples logarithmically. Using N ≈ p/ε 2 samples, we can construct a uniform ε-approximation of P(s ′ | s, a, θ).

Let's call Vh the estimated value function using Algorithm 3 with the above approximation of transition. Here, we elucidate the span of this estimation of value function. First we have:

V π H -V π H = s ′ ( P -P )(s ′ | s, a)r(s ′ , π(s ′ ))ds ′ ≤ εdH 3/2
Here, we use the facts that S V θ, θx ,h ≤ dH 3/2 (cf Section 5.5.2) and the error in approximating

P is bounded by ε, i.e. sup s ′ ,s,a |( P -P )(s ′ |s, a)| ≤ ε.
Assume that at step h + 1, we have V π h+1 -V π h+1 ≤ h+1 j=1 ε j α h+1,j . Then, we obtain

V π h -V π h ≤ s ′ ( P -P )(s ′ | s, a) V π h+1 (s ′ )ds ′ + s ′ P (s ′ | s, a)( V π h+1 -V π h+1 )(s ′ )ds ′ = s ′ ( P -P )(s ′ | s, a)(V π h+1 + V π h+1 -V π h+1 )ds ′ + s ′ P (s ′ | s, a)( V π h+1 -V π h+1 )(s ′ )ds ′ ≤ ε(dH 3/2 + h+1 j=1 ε j α h+1,j ) + h+1 j=1 ε j α h+1,j ≤ ε(dH 3/2 + α h+1,1 ) + h+1 j=2 ε j (α h+1,j-1 + α h+1,j ) + ε h+2 α h+1,h+1

Algorithm design

Using the fact that α 1,1 = dH 3/2 and with a proper induction, we find that:

V π 1 -V π 1 ≤ εdH 5/2 1 -ε H-h 1 -ε ≤ H→∞ εdH 5/2
This concludes the proof of the arguments provided in § Planning of Section 5.4.1. This means that the extra regret due to planning with the approximation by RFT features is of order O εdH 5/2 K . By choosing an ε of order 1/(H √ K), we deduce that approximating the probability kernel with O pH 2 K samples induces a tractable planning procedure without harming the regret.

Remark 5.7. The reader might be tempted to combine the finite approximation using RFT with algorithms from the linear reinforcement learning literature [START_REF] Jin | Provably efficient reinforcement learning with linear function approximation[END_REF]. However, note that the dimensionality of the linear space induced by RFT is polynomial in H and K. Consequently, applying algorithms designed with the assumption of linear value function would incur a linear regret.

Maximum likelihood estimation

The ML estimation is explicit for simple distributions like the Gaussian (Rogers and Young, 1977) and for Linearly controlled dynamical systems. But it requires integral approximations for generic transitions as mentioned in (Chowdhury, [START_REF] Sayak | Reinforcement Learning in Parametric MDPs with Exponential Families[END_REF]. However, we believe that this estimation problem is simpler than the planning problem since the latter traditionally involves approximating an integral for all pairs of (s ′ , a).

One of the popular solutions to this estimation are the Integral approximation techniques. [START_REF] Radford | Annealed importance sampling[END_REF] proposes to handle the MLe using simulated annealing, a method consisting in starting from a tractable distribution and updating it to resemble the distribution at hand. [START_REF] Vembu | Probabilistic structured predictors[END_REF] proposes MCMC techniques for approximating the partition function. (Carreira-Perpinan and Hinton, 2005) shows that optimizing a different objective, called the contrastive divergence leads to a good approximation of the ML.

Another popular solution to the ML estimation consists in Score matching. This is a technique that avoids approximating the partition function and is well studied in literature, see [START_REF] Jørgensen | Maximum likelihood estimation and large-sample inference for generalized linear and nonlinear regression models[END_REF]. More recently, [START_REF] Li | Exponential Family Model-Based Reinforcement Learning via Score Matching[END_REF] proposed an adaptation of this technique to the exact setting we consider. The latter shows that under certain conditions, that we are unable to verify, the estimation can be solved in O(d 3 ) time.

Other works (Shah, Shah, and Wornell, 2021) assume Bounded distribution support and natural parameter, and show tha, for a minimally represented k-parameter Exponential family, under boundedness of the support of the distribution and of the natural parameter,

Continuous MDPs: the Bilinear Exponential Family representation

an α-approximation of the MLE can be derived in O(poly(k/α)) time. The latter assumes a specific definition of compactness of the representation as well as knowledge of the support and shows how to re-parameterize the density to a specific class of exponential families that are easier to study.

Finally, [START_REF] Dai | Kernel exponential family estimation via doubly dual embedding[END_REF] studies exponential families such that the natural parameter belongs to some RKHS, moreover, it proposes a method that learns the Kernel parameters and improves over score matching in time and in memory complexity.

Regret analysis

We provide a high probability analysis of the regret of BEF-RLSVI under standard regularity assumptions of the representation. First we recall the regret definition then we separate the perturbation error from the statistical estimation:

R(K) = K k=1 (V ⋆ θ p ,θ r ,1 -V π k θ p ,θ r ,1 )(s k 1 ) = K k=1 V ⋆ θ p ,θ r ,1 -V π k θp , θr ,1 learning + V π k θp , θr ,1 -V π k θ p ,θ r ,1 Estimation (s k 1 )
For the estimation error, we use smoothness arguments with concentrations of parameters up to some novelties. Regarding the learning error, we show that the injected noise ensures a constant probability of anti-concentration. Applying Assumption 5.4 and Lemma C.12 leads to the upper-bound.

Estimation error

To show that the estimation error

K k=1 V θp , θr ,1 -V π k
θ p ,θ r ,1 can be controlled, we decompose it to an error that comes from the estimation of the transition parameter and one that comes from the estimation of the reward parameter:

V π θp , θr (s k 1 ) -V π θ p ,θ r (s k 1 ) = V π θp ,θ r (s k 1 ) -V π θ p ,θ r (s k 1 )
transition estimation

+ V π θp , θr (s k 1 ) -V π θp ,θ r (s k 1 ) reward estimation , ( 5.8)
we control each term separately in Section 5.5.1 and Section 5.5.1. Therefore, we obtain the following lemma controlling the estimation error.

Lemma 5.8. The estimation error satisfies, with probability at least 1 -5δ

K k=1 V θp , θr ,1 (s k 1 ) -V π θ p ,θ r ,1 (s k 1 ) ≤ 2H 2β p α p β p (N, δ)N γ p K + 2H 2N log(1/δ)

Regret analysis

+ KHd log (1 + α r η -1 B φ,A n) + C d Hd log(1 + αη -1 B φ,A H) × β r (n, δ) 2α r +c (max k x k )d log(dK/δ) β r + 2KHd log (1 + α r η -1 B φ,A n) log(1/δ)
where for i ∈ [p, r],

β i (K, δ) ≜ η 2 B 2 A + γ i K + log(1/δ), and
γ i K ≜ d log(1 + β i η B φ,A HK). Also, C d ≜ 3d log(2) log 1 + α r ∥A∥ 2 2 B 2 φ,A η log(2)
, and c is a universal constant.

Proof. It follows directly by combining Lemma 5.9 and Lemma 5.12 using a union bound.

Transition estimation

The goal of this section is to prove the following lemma which bounds the regret due to transition estimation.

Lemma 5.9. We have, with probability at least 1 -2δ

K k=1 V θp ,θ r (s k 1 ) -V π θ p ,θ r (s k 1 ) ≤ 2H 2β p α p β p (N, δ)N γ p K + 2H 2N log(1/δ)
where γ p K := d log 1 + β p η -1 B φ,A HK , and

β p (K, δ) ≜ η 2 B 2 A + γ p K + log(1/δ).
Note that in this error term, the reward is bounded since its parameter is exact, the value function is therefore smaller than H. Using the transportation of Lemma C.7 we obtain a bound in terms of kl divergences. We then notice that since the reward parameter is exact, the bound can be improved using Lemma C.12 under Assumption 5.4. We win a √ H factor compared to the analysis of (Chowdhury and Gopalan, 2019).

Proof. The proof proceeds in two parts. First, we will reveal a bound in terms of the induced local geometry, i.e. a bound in terms of KL-divergence. Second, we explicit the bound by transferring the induced local geometry to the euclidean one.

1)

Bound in terms of local geometry. We provide a bound on the estimation error of the transition in terms of KL divergences, for that end we show that the estimation error can be decomposed and well controlled. We start by writing the one-step decomposition:

Continuous MDPs: the Bilinear Exponential Family representation

V π θp ,θ r ,1 (s k 1 )-V π θ p ,θ r ,1 (s k 1 ) = E θp s k 1 ,a k 1 V π θp ,θ r ,2 -E θ p s k 1 ,a k 1 V π θp ,θ r ,2 + E θ p s k 1 ,a k 1 [V π θp ,θ r ,2 -V π θ p ,θ r ,2 ] = E θp s k 1 ,a k 1 V π θp ,θ r ,2 -E θ p s k 1 ,a k 1 V π θp ,θ r ,2 + V π θp ,θ r ,2 (s 2k ) -V π θ p ,θ r ,2 (s 2k ) + ζ k 1 = H h=1 E θp s hk ,a hk V π θp ,θ r ,h+1 -E θ p s hk ,a hk V π θp ,θ r ,h+1 + ζ hk where ζ hk = E θ p s hk ,a hk [V π θp ,θ r ,h+1 -V π θ p ,θ r ,h+1 ] -V π θp ,θ r ,h+1 (s h+1k ) -V π θ p ,θ r ,h+1 (s h+1k
) is a martingale sequence, and the last equality comes by induction. Here we consider the true reward parameter which verifies |E θ r [r(s, a)]| ≤ 1 by assumption, therefore |ζ hk | ≤ 2H. Using the Azuma-Hoeffding inequality [START_REF] Boucheron | Concentration inequalities: A nonasymptotic theory of independence[END_REF], with probability at least

1 -δ K k=1 H h=1 ζ hk ≤ 2H 2KH log(1/δ)
We finish bounding the first term using Lemma C.7, indeed

E θp s hk ,a hk V π θp ,θ r ,h+1 -E θ p s hk ,a hk V π θp ,θ r ,h+1 ≤ H 2 KL s hk ,a hk (θ p , θp ) ≤ H min 1, 2 KL s hk ,a hk (θ p , θp ) ,
the last inequality follows because ∀h, S(V θp ,θ r ,h+1 ) ≤ H.

Remark 5.10. Traditionally, the expected value difference bound follows from the simulation lemma [START_REF] Ren | A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning[END_REF]. The simulation lemma incurs an extra √ H factor compared to our bound.

We deduce that with probability at least 1 -δ:

K k=1 V θp ,θ r (s k 1 ) -V π θ p ,θ r (s k 1 ) ≤ H K k=1 min 1, H h=1 2 KL s hk ,a hk (θ p , θp ) + 2H 2KH log(1/δ) (5.9)
2) Bounding the sum of KL divergences. we explicit the bound of Inequality (5.9) using Assumption 5.4 along with properties of the exponential family (cf Section C.2.3). We have for all (s, a),

∀θ p , θ p′ , α p 2 θ p′ -θ p 2 Gs,a ≤ KL s,a θ p , θ p′ ≤ β p 2 θ p′ -θ p 2
Gs,a .

(5.10)

Regret analysis

This implies that

KL s,a θp (k), θ p ≤ β p 2 θ p -θp (k) 2 Gs,a ≤ β p ( Ḡp k ) -1/2 G s,a ( Ḡp k ) -1/2 1 2 θ p -θp (k) 2 Ḡp k , where Ḡp k ≡ Ḡp (k-1)H := G k + (α p ) -1 ηA and G k ≡ k-1 τ =1 H h=1 G s τ s ,a τ h . From Corollary C.2, with probability at least 1 -δ and for all k ∈ N θ p -θp (k) 2 Ḡp k ≤ 2β p (k, δ)/α p .
Also, using Lemma C.12, we have

T t=1 H h=1 min 1, ( Ḡp k ) -1/2 G s,a ( Ḡp k ) -1/2 ≤ 2d log 1 + α p η -1 B φ,A HK .
Combining these two results we obtain, with probability at least 1 -δ:

T t=1 H h=1 min 1, KL s t h ,a t h θp (k), θ p ≤ 2β p α p β p (K, δ)γ p K . ( 5.11)
Remark 5.11. Notice that the minimum with 1 is crucial, indeed, without it the bound deteriorates by a factor H as was the case in (Chowdhury, [START_REF] Sayak | Reinforcement Learning in Parametric MDPs with Exponential Families[END_REF].

3) Combining the bounds. By applying Cauchy-Schwarz in Inequality (5.9), we obtain, with probability at least 1 -δ, and for all

K ∈ N K k=1 V θp ,θ r (s k 1 ) -V π θ p ,θ r (s k 1 ) ≤ H 2 K k=1 H h=1 KL s hk ,a hk (θ p , θp ) + 2H 2KH log(1/δ).
Injecting Inequality (5.11) proves the desired result with probability at least 1 -2δ.

Reward estimation

Previous work uses clipping to help control this error, but in this case it can reduce the optimism probability. [START_REF] Zanette | Frequentist regret bounds for randomized least-squares value iteration[END_REF] proposes an involved clipping depending on the norms

∥(A i φ(s k h , a k h )) i∈[d] ∥ ( Ḡp k ) -1
, which is somewhat delicate to analyze and deploy. We remedy the situation acting solely in the proof. We provide the bound over the regret due to estimating the reward parameter in the following lemma.

Continuous MDPs: the Bilinear Exponential Family representation

Lemma 5.12. With probability at least 1 -3δ, the following result holds true.

K k=1 V π θp , θr ,1 (s k 1 )-V π θp ,θ r ,1 (s k 1 ) ≤   β r (K, δ) 2α r + c (max k≤K x k )d log(dK/δ)   β r × C d 1 + α r B φ,A H η + K log(e/δ 2 ) Hd log (1 + α r η -1 B φ,A HK),
where

β p (K, δ) ≜ η 2 B 2 A + γ p K + log(1/δ), and
γ p K ≜ d log(1 + β p η B φ,A HK). Also, c is a universal constant and C d ≜ 3d log(2) log 1 + α r ∥A∥ 2 2 B 2 φ,A η log(2)
.

Proof. The reward estimation error in Equation (5.8) can be written explicitly. Indeed, using Lemma C.11

V π θp , θr ,1 (s k 1 ) -V π θp ,θ r ,1 (s k 1 ) = E (s h ) 1≤h≤H ∼π| θp ,s k 1 H h=1 Var sh ,π(s h ) (r) 2 B ⊤ M θr -θ r φ(s h , π(s h )) ≤E H h=1 Var sh ,π(s h ) (r) 2 ∥ θr -θ r ∥ Ḡr k ∥(B ⊤ A i φ(s h , π(s h ))) 1≤i≤d ∥ ( Ḡr k ) -1 ≤∥ θr -θ r ∥ Ḡr k E H h=1 Var sh ,π(s h ) (r) 2 ∥(B ⊤ A i φ(s h , π(s h ))) 1≤i≤d ∥ ( Ḡr k ) -1 ≤∥ θr -θ r ∥ Ḡr k β r 2 E H h=1 ∥(A i φ(s h , π(s h ))) 1≤i≤d ∥ ( Ḡr k ) -1 = ∆ traj k , where traj k = ∆ H h=1 ∥(A i φ(s h , π(s h ))) 1≤i≤d ∥ (G r k ) -1
. Bad rounds. We separate the analysis of this estimation error into bad and good rounds.

Here we analyze the bad rounds, which are define by the following set:

T = {k ∈ N * , ∃h ∈ [H], ∥(A i φ(s h , π(s h ))) 1≤i≤d ∥ ( Ḡr k ) -1 ≥ 1}
These rounds are why clipping is necessary. Thanks to Lemma C.14, we know that the number of such rounds is at most O(d). Surprisingly, it depends neither on H nor on K.

We now show that the "bad rounds" incur at most O d 3/2 H 2 regret, independent of K. Thus, we can omit clipping for free.

Regret analysis

Remark 5.13. In non-episodic settings, the forward algorithm (Azoury and Warmuth, 2001) eliminates the span control issue. See [START_REF] Ouhamma | Stochastic Online Linear Regression: the Forward Algorithm to Replace Ridge[END_REF] for stochastic analysis and an application to linear bandits.

1) We know that ∥(A i φ(s h , π(s h ))) 1≤i≤d (A i φ(s h , π(s h ))) ⊤ 1≤i≤d ∥ 2 2 ≤ ∥A∥ 2 2 B 2 φ,A .
Then, according to Lemma C.14

|T | ≤ 3d log(2) log 1 + α∥A∥ 2 2 B 2 φ,A η log(2) .
2) Since G k is positive semi-definite, we have Ḡr k ⪰ (α r ) -1 ηA, and in turn, for all state-action couples (s, a), ( Ḡr

k ) -1 G s,a ≤ α r η A -1 G s,a ≤ α r B φ,A η .
This further yields

I + ( Ḡr k ) -1 H h=1 G s t h ,a t h ≤ 1 + H h=1 ( Ḡr k ) -1 G s t h ,a t h ≤ 1 + α r B φ,A H η .

Let us define Ḡr

k+H := Ḡr k + H h=1 G s k h ,a k h . Then, Ḡ-1 k+H G s,a = I + ( Ḡr k ) -1 H h=1 G s t h ,a t h -1 ( Ḡr k ) -1 G s,a .
Therefore, for all pairs (s, a),

∥(A i φ(s h , π(s h ))) 1≤i≤d ∥ ( Ḡr k ) -1 = ⊤ ((A i φ(s h , π(s h ))) ⊤ 1≤i≤d ( Ḡr k ) -1 (A i φ(s h , π(s h ))) 1≤i≤d ) = ⊤ ( 1 + α r B φ,A H η ( Ḡr k+H ) -1 G s,a ) ≤ 1 + α r B φ,A H η ∥(A i φ(s h , π(s h ))) 1≤i≤d ∥ ( Ḡr k+H ) -1 Since ∥(A i φ(s h , π(s h ))) 1≤i≤d ∥ ( Ḡr k+H ) -1 ≤ 1, we have ∥(A i φ(s h , π(s h ))) 1≤i≤d ∥ ( Ḡr k+H ) -1 ≤ min 1, ∥(A i φ(s h , π(s h ))) 1≤i≤d ∥ ( Ḡr k ) -1 . Consequently H h=1 ∥(A i φ(s h , π(s h ))) 1≤i≤d ∥ ( Ḡr k+H ) -1 ≤ Hd log(1 + α r η -1 B φ,A H).

Continuous MDPs: the Bilinear Exponential Family representation

3) From 1) and 2), we deduce that the total regret induced by rounds from T is bounded.

k∈T h∈[H] V π θp , θr ,1 (s k 1 ) -V π θp ,θ r ,1 (s k 1 ) ≤ ∥ θr -θ r ∥ Ḡr k β r 2 3d log(2) log 1 + α r ∥A∥ 2 2 B 2 φ,A η log(2) 1 + α r B φ,A H η Hd log(1 + α r η -1 B φ,A H) (5.12)
Remark 5.14. The bad rounds analysis is one of our most important contributions as it enables us to forgo clipping without consequences. Consequently, this is a novel method to control the reward estimation error that improves on existing work for whom clipping was essential.

Good rounds. Going forward we consider rounds from T . Let us define

ζ ′ k = ∆ traj k -E (s h ) 1≤h≤H ∼π| θp ,s k 1 traj k .
where traj k is the same quantity as traj but with a random realization of state transitions. Since all feature norms are smaller than one,

(ζ ′ k ) k is a martingale sequence with |ζ ′ k | ≤ Hd log (1 + α r η -1 B φ,A HK).
We deduce that with probability at least 1 -δ:

K k=1 ζ ′ k ≤ 2KHd log (1 + α r η -1 B φ,A HK) log(1/δ)
Therefore, we have with probability at least 1 -3δ:

k∈T c V π θp , θr ,1 (s k 1 ) -V π θp ,θ r ,1 (s k 1 ) ≤   β r (K, δ) 2α r + c (max k x k )d log(dK/δ)   × β r KHd log (1 + α r η -1 B φ,A KH) log(e/δ 2 ).
The last inequality follows from controlling the concentration of the reward parameter. First we observe that (Corollary C.4) with probability at least 1 -δ, uniformly over

k ∈ N, θ r -θr (k) 2 Ḡr k ≤ 2 α r β r (k, δ).
Second, we also have that for all k ≥ 1, with probability at least 1 -δ, ∥ξ k ∥ G r k ≤ c x k d log(d/δ), we then use a union bound. Combining with Equation (5.12) we find

K k=1 V π θp , θr ,1 (s k 1 ) -V π θp ,θ r ,1 (s k 1 ) ≤   β r (K, δ) 2α r + c (max k x k )d log(dK/δ)  

Regret analysis

× β r KHd log (1 + α r η -1 B φ,A HK) log(e/δ 2 ).
This concludes the proof of the reward estimation error.

Remark 5.15. If we use Lemma C.11 without the martingale difference sequence, it will lead to a linear regret. Indeed, the span of the sum of norms over an episode is of order √ H. Using the martingale technique instead allows us to retrieve a telescopic sum controlled using the elliptical lemma, this is essential to obtaining a sub-linear regret bound.

Learning error

We now start the control of an important regret term, due to the distance between the estimated value function and the optimal value function. Our main methodology here is to show that the estimated value is optimistic with a constant probability and that this suffices to control the error.

Lemma 5.16. If the variance parameter of the injected noise

(ξ k ) k satisfies x k ≥   H β p β p (k, δ) α p α r + β r β r (k, δ) min{1, α p α r } 2α r   ,
then the learning error is controlled with probability at least 1 -2δ as

K k=1 V ⋆ 1 (s k 1 ) -V π θp , θr + ξk ,1 (s k 1 ) ≤ dβ r √ x k 1 + log(d/δ) Φ(-1) H log (1 + α r η -1 B φ,A HK) × C d 1 + α r B φ,A H η + K log(e/δ 2 )
,

where for i ∈ [p, r], β i (K, δ) ≜ η 2 B 2 A + γ i K + log(1/δ), and
γ i K ≜ d log(1 + β i η B φ,A HK). Also C d = ∆ 3d log(2) log 1 + α r ∥A∥ 2 2 B 2 φ,A η log(2)
, and Φ is the normal CDF.

This result basically means that we are no longer obliged to follow optimistic value functions, the perturbed estimation is enough to have a tight bound on the learning error.

Continuous MDPs: the Bilinear Exponential Family representation

Stochastic optimism

The goal here is to show that by injecting our carefully designed noise in the rewards we can ensure optimism with a constant probability. Consider the optimal policy π ⋆ , we have:

(V θp , θr ,1 -V ⋆ θ p ,θ r ,1 )(s 1 ) ≥ (Q ⋆ θp , θr ,1 -Q ⋆ 1 )(s 1 , π ⋆ (s 1 )) ≥ V π ⋆ θp ,θ r (s 1 ) -V π ⋆ θ p ,θ r (s 1 ) first term + V π ⋆ θp , θr (s 1 ) -V π ⋆ θp ,θ r (s 1 ) second term + V π ⋆ θp , θr (s 1 ) -V π ⋆ θp , θr (s 1 )
third term

The first and second terms are perturbation free, we handle them like the estimation error, by concentration arguments for θp and θr . For the third term, using transportation of rewards (Lemma C.11) and anti-concentration of ξ k (Lemma C.6).

First term.

By assumption, the expected value of the reward following the true parameter satisfies

E θ r [r(s, a)] ∈ [0, 1], then S H t=1 E θ r [r(s t , π(s t ))] ≤ H.
Consequently, the first term can be controlled using Lemma C.7

V π ⋆ θ p ,θ r (s 1 ) -V π ⋆ θp ,θ r (s 1 ) ≤ H KL(P θp (s 2 , . . . , s H ), P θ p (s 2 , . . . , s H )) ≤ H E (st) t∈[H] ∼ θp |s k 1 H t=1 ψ(s t+1 ) ⊤ M θp -θ p φ(s t , π ⋆ (s t )) + Z p θ p (s t , π ⋆ (s t )) -Z p θp (s t , π ⋆ (s t ))
Using Taylor's expansion, for all h ∈ [H], ∃θ h ∈ [θ p , θp ] such that:

E (st) t∈[H] ∼ θp |s k 1 ψ(s t+1 ) ⊤ M θp -θ p φ(s t , π ⋆ (s t )) + Z p θ p (s t , π ⋆ (s t )) -Z p θp (s t , π ⋆ (s t )) = 1 2 ( θp -θ p ) ⊤ E (st) t∈[H] ∼ θp |s k 1 ∇ 2 s h ,π ⋆ (s h ) Z p (θ h ) ( θp -θ p ) ≤ β p 2 E (st) t∈[H] ∼ θp |s k 1 ∥ θp -θ p ∥ 2 G sh ,π ⋆ (s h ) . Define u k = ∆ H h=1 E (st) t∈[H] ∼ θp |s k 1 (A i φ(s h , π ⋆ (s h ))) i∈[d] , then V π ⋆ θ p ,θ r (s 1 ) -V π ⋆ θp ,θ r (s 1 ) ≤ H β p 2 H h=1 E (st) t∈[H] ∼ θp |s k 1 ∥ θp -θ p ∥ 2 G sh ,π ⋆ (s h ) ≤ H β p 2 θp -θ p H h=1 E (s t ) t∈[H] ∼ θp |s k 1 [G sh ,π ⋆ (s h ) ] ≤ H β p 2 θp -θ p u k u ⊤ k ≤ H β p 2 ( Ḡp k ) -1/2 u k u ⊤ k ( Ḡp k ) -1/2 ∥ θp -θ p ∥ Ḡp k 5.5 Regret analysis ≤ H β p 2 ∥u k ∥ ( Ḡp k ) -1 ∥ θp -θ p ∥ Ḡp k The third line follows because ∀x ∈ R d , ∥x∥ i=1 a i a ⊤ i ≤ ∥x∥ ( i=1 a i )( i=1 a i ) ⊤
, and the last one follows because ⊤ (AB) ≤ ⊤ (A) ⊤ (B) for any two real positive semi-definite matrices A and B. We deduce, with probability at least 1 -δ:

V π ⋆ θ p ,θ r (s 1 ) -V π ⋆ θp ,θ r (s 1 ) ≤ H β p β p (k, δ) α p H h=1 E (st) t∈[H] ∼ θp |s k 1 (A i φ(s h , π ⋆ (s h ))) i∈[d] ( Ḡp k ) -1
Second term. We have

V π ⋆ θp , θr (s 1 ) -V π ⋆ θp ,θ r (s 1 ) = E (st) t∈[H] ∼ θp |s k 1 H t=1 Var θ r t (r) 2 B ⊤ M θr -θ r φ(s t , π ⋆ (s t )) = ( θr -θ r ) ⊤ E (st) t∈[H] ∼ θp |s k 1 H t=1 Var θ r t (r) 2 (A i φ(s t , π ⋆ (s t ))) i∈[d] B ≤ √ β r 2 ∥ θr -θ r ∥ Ḡr k E (st) t∈[H] ∼ θp |s k 1 H t=1 (A i φ(s t , π ⋆ (s t ))) i∈[d] ( Ḡr k ) -1
The last inequality comes from Cauchy-Schwarz. Applying that the norm (sum) makes appear only symmetric matrices times the variances so that we can bound the latter by β r . We conclude that with probability at least 1 -δ,

V π ⋆ θp , θr (s 1 ) -V π ⋆ θp , θr (s 1 ) ≤ β r β r (k, δ) √ 2α r E (st) t∈[H] ∼ θp |s k 1 H t=1 (A i φ(s t , π ⋆ (s t ))) i∈[d] ( Ḡr k ) -1
We want to write all the norms in the same matrix. Therefore, with probability at least 1 -δ,

V π ⋆ θp , θr (s 1 ) -V π ⋆ θp , θr (s 1 ) ≤ β r β r (k, δ) min{1, α p α r } 2α r × E (st) t∈[H] ∼ θp |s k 1 H t=1 (A i φ(s t , π ⋆ (s t ))) i∈[d] ( Ḡp k ) -1
Third term. We have

V π ⋆ θp , θr ,1 (s 1 ) -V π ⋆ θp , θr ,1 (s 1 ) = E (st) t∈[H] ∼ θp |s k 1 H t=1 Var θ r j (r) 2 B ⊤ M θr -θr φ(s t , π ⋆ (s t )) = ξ ⊤ k E (st) t∈[H] ∼ θp |s k 1 H t=1 Var θ r j (r) 2 (A i φ(s t , π ⋆ (s t ))) i∈[d] B

Regret analysis

Combining the error bounds. Combining the upper bound of Equation (5.14) with the lower bound of Equation (5.16), we get, with probability at least 1 -δ:

V ⋆ 1 (s k 1 ) -V θp , θr + ξk ,1 (s k 1 ) ≤ E ξk | Ōk [V θp , θr + ξk ,1 (s k 1 ) -V 1 (s k 1 )]
Also, using the tower rule,

E ξk [V θp , θr + ξk ,1 (s k 1 ) -V 1 (s k 1 )] = E ξk | Ōk [V θp , θr + ξk ,1 (s k 1 ) -V 1 (s k 1 )]P( Ōk ) + E ξk | Ōc k [V θp , θr + ξk ,1 (s k 1 ) -V 1 (s k 1 )]P( Ōc k )
Therefore,

V ⋆ 1 (s k 1 )-V θp , θr + ξk ,1 (s k 1 ) ≤ E ξk [V θp , θr + ξk ,1 (s k 1 ) -V 1 (s k 1 )] -E ξk | Ōc k [V θp , θr + ξk ,1 (s k 1 ) -V 1 (s k 1 )]P( Ōc k ) /P( Ōk ) = E ξ k [V π θp , θr +ξ k ,1 (s k 1 ) -V π 1 (s k 1 )] -E ξ k | Ōc k [V θp , θr +ξ k ,1 (s k 1 ) -V 1 (s k 1 )]P( Ōc k ) /P( Ōk ).
The last line follows since ξ k and ξk are i.i.d.

The rest of the analysis proceeds similarly to the proof of the reward estimation.

Let us call the argument of the minimum in Equation (5.15) as ξ k . Using Lemma C.11, we find

V π θp , θr ,1 (s k 1 )-V π θp , θr +ξ k ,1 (s k 1 ) = E (s h ) 1≤h≤H ∼π| θp ,s k 1 H h=1 Var sh ,π(s h ) (r) 2 B ⊤ M θr -θr -ξ k φ(s h , π(s h )) ≤ E H h=1 Var sh ,π(s h ) (r) 2 ∥ θr -θr -ξ k ∥ Ḡp k ∥(B ⊤ A i φ(s h , π(s h ))) 1≤i≤d ∥ ( Ḡp k ) -1 ≤ ∥ θr -θr -ξ k ∥ Ḡp k E H h=1 Var sh ,π(s h ) (r) 2 ∥(B ⊤ A i φ(s h , π(s h ))) 1≤i≤d ∥ ( Ḡp k ) -1 ≤ ∥ ξk -ξ k ∥ Ḡp k β r 2 E H h=1 ∥(A i φ(s h , π(s h ))) 1≤i≤d ∥ ( Ḡp k ) -1
Then,

E ξk V π θp , θr ,1 (s k 1 )-V π θp , θr +ξ k ,1 (s k 1 ) ≤ β r 2 E ξk [∥ ξk -ξ k ∥ Ḡp k ]E (s h )∼π| θp H h=1 ∥(A i φ(s h , π(s h ))) 1≤i≤d ∥ ( Ḡp k ) -1 .

Continuous MDPs: the Bilinear Exponential Family representation

Also,

E ξ k | Ōc k [V θp , θr +ξ k ,1 (s k 1 )-V 1 (s k 1 )] ≤ β r 2 E ξk | Ōc k [∥ ξk -ξ k ∥ Ḡp k ]E (s h )∼π| θp H h=1 ∥(A i φ(s h , π(s h ))) 1≤i≤d ∥ ( Ḡp k ) -1 ≤ β r 2 E ξk [∥ ξk -ξ k ∥ Ḡp k ]E (s h )∼π| θp H h=1 ∥(A i φ(s h , π(s h ))) 1≤i≤d ∥ ( Ḡp k ) -1 .
We have a bound on the expected value of the sum of feature norms in the proof of Lemma 5.12. Also,

E ξk [∥ ξk -ξ k ∥ Ḡp k ] ≤ E ξk [∥ ξk ∥ Ḡp k ] + E ξk [∥ξ k ∥ Ḡp k ] ≤ E ξk [∥ ξk ∥ 2 Ḡp k ] + x k d log(d/δ) ≤ x k d + x k d log(d/δ)
The second line follows from Cauchy-Schwarz and by definition of ξ k . The last line is due to the fact that

x k ( Ḡp k ) -1 ∼ N (0, x k I d ), which implies ∥ ξk ∥ 2 Ḡp k ∼ N (0, dx k ).
We conclude the proof by taking the sum of feature norms from the proof of Lemma 5.12.

We conclude that with probability at least 1 -2δ:

K k=1 V ⋆ 1 (s k 1 ) -V θp , θr + ξk ,1 (s k 1 ) ≤ β r Φ(-1) ( x k d + x k d log(d/δ)) 3d log(2) log 1 + α r ∥A∥ 2 2 B 2 φ,A η log(2) 1 + α r B φ,A H η Hd log(1 + α r η -1 B φ,A H) + KHd log (1 + α r η -1 B φ,A HK) log(e/δ 2 ) .

Related works: functional representations of MDPs with regret and tractability

Our work extends the endeavor of using functional representations for regret minimization in continuous state-action MDPs. Now, we posit our contributions in existing literature.

Kernel value function representation. (pmlr-v119-ayoub20a) studies MDPs with a linear mixtures model then extends to an RKHS setting, this generalizes our work and that of (Yang and Wang, 2020). However, the paper proposes an Eluder-dimension analysis, for RKHS settings this leads to the result of (Yang and Wang, 2020), i.e. a regret H log(T ) d higher than

Related works: functional representations of MDPs with regret and tractability

for BEF-RLSVI.Recently, [START_REF] Huang | A short note on the relationship of information gain and eluder dimension[END_REF] shows that for RKHS, Eluder dimension and the information gain are strictly equivalent, which brings in the extra factor.

General functional representation. The Eluder dimension is a complexity measure often used to analyze RL with general function space, [START_REF] Huang | A short note on the relationship of information gain and eluder dimension[END_REF] asserts that "common examples of where it is known to be small are function spaces (vector spaces)". [START_REF] Dai | Sbeed: Convergent reinforcement learning with nonlinear function approximation[END_REF] provides the first convergence guarantee for general nonlinear function representations in the Maximum Entropy RL setting, where entropy of a policy is used as a regularizer to induce exploration. Thus, the analysis cannot address episodic RL, where we have to explicitly ensure exploration with optimism. In the episodic setting, (Wang, Salakhutdinov, and Yang, 2020) leverage the UCB approach for tabular MDPs and function spaces with bounded Eluder dimension, this strategy achieves a and achieve a Õ √ d 4 H 2 T regret for linear MDPs. [START_REF] Ishfaq | Randomized Exploration in Reinforcement Learning with General Value Function Approximation[END_REF] considers the same setting, proposes an RLSVI based algorithm, and achieves a Õ( √ d 3 H 4 K) for linear MDPs. However, the latter assumes an oracle perturbing the estimation to achieve anti-concentration while maintaining a bounded covering number, which is a counterintuitive mix of boundedness and anti-concentration. Indeed, [START_REF] Zanette | Frequentist regret bounds for randomized least-squares value iteration[END_REF] studied the linear MDP case, and while it managed to design an ingenious clipping verifying previous assumptions, the method is extremely intricate and the proof is involved and unlikely to extend for general value function spaces. To concertize our design, we focus on the general but explicit BEF of MDPs than any abstract representation. We also remove the requirement to clip with a novel analysis.

Bilinear exponential family of MDPs. Exponential families are studied widely in RL theory, from bandits to MDPs (Lu, Meisami, and Tewari, 2021; Korda, Kaufmann, and Munos, 2013; Filippi et al., 2010; Kveton and Hauskrecht, 2006), as an expressive parametric family to design theoretically-grounded model-based algorithms. (Chowdhury, [START_REF] Sayak | Reinforcement Learning in Parametric MDPs with Exponential Families[END_REF] first studies episodic RL with Bilinear Exponential Family (BEF) of transitions, which is linear in both state-action pairs and the next-state. It proposes a regularized log-likelihood method to estimate the model parameters, and two optimistic algorithms with upper confidence bounds and posterior sampling. Due to its generality to unifiedly model tabular MDPs, factored MDPs, and linearly controlled dynamical systems, the BEF-family of MDPs has received increasing attention [START_REF] Li | Exponential Family Model-Based Reinforcement Learning via Score Matching[END_REF]. [START_REF] Li | Exponential Family Model-Based Reinforcement Learning via Score Matching[END_REF] estimates the model parameters based on score matching that enables them to replace regularity assumption on the log-partition function with Fisher-information and assumption on the parameters. Both (Chowdhury, [START_REF] Sayak | Reinforcement Learning in Parametric MDPs with Exponential Families[END_REF][START_REF] Li | Exponential Family Model-Based Reinforcement Learning via Score Matching[END_REF]) achieve a worst-case regret of order Õ( √ d 2 H 4 K) for known reward. On a different note, [START_REF] Du | Bilinear classes: A structural framework for provable generalization in rl[END_REF][START_REF] Foster | The Statistical Complexity of Interactive Decision Making[END_REF]) also introduces a new structural framework for generalization in RL, called bilinear classes as it requires the Bellman error to be upper bounded by a bilinear form. Instead of using bilinear forms to capture non-linear structures, this class is not identical to BEF class of MDPs, and studying the connection is out of the scope of this paper. Specifically, we address the shortcomings of the Continuous MDPs: the Bilinear Exponential Family representation existing works on BEF-family of MDPs that assume known rewards, absence of RLSVI-type algorithms, and access to oracle planners.

Tractable planning and linearity. Planning is a major byproduct of the chosen functional representation. In general, planning can incur high computational complexity if done naïvely. Specially, [START_REF] Simon S Du | Is a good representation sufficient for sample efficient reinforcement learning?[END_REF] shows that for some settings, even with a linear ε-approximation of the Q-function, a planning procedure able to produce an ε-optimal policy has a complexity at least 2 H . Thus, different works (Shariff and Szepesvári, 2020; Lattimore, Szepesvari, and Weisz, 2020; [START_REF] Van | Comments on the du-kakade-wang-yang lower bounds[END_REF] propose to leverage different lowdimensional representations of value functions or transitions to perform efficient planning. Here, we take note from [START_REF] Ren | A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning[END_REF] that Gaussian transitions induce an explicit linear value function in an RKHS. And generalize this observation with the bilinear exponential. Moreover, using uniformly good features [START_REF] Rahimi | Random features for large-scale kernel machines[END_REF] to approximate transition dynamics from our model enables us to design a tractable planner. We provide a detailed discussion of this approximation in Section 5.4. More practically, [START_REF] Ren | A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning[END_REF][START_REF] Nachum | Provable representation learning for imitation with contrastive fourier features[END_REF] use representations given by random Fourier features [START_REF] Rahimi | Random features for large-scale kernel machines[END_REF] to approximate the transition dynamics and provide experiments validating the benefits of this approach for high-dimensional Atari-games. Thus, we propose the first algorithm with tractable planning for BEF-family.

Discussion

We propose the BEF-RLSVI algorithm for the bilinear exponential family of MDPs in the setting of episodic-RL. BEF-RLSVI explores using a Gaussian perturbation of rewards, and plans tractably (complexity of O pH 3 K log(HK) ) thanks to properties of the RBF kernel. Our proof shows that clipping can be forwent for similar RLSVI-type algorithms. Moreover, we prove a √ d 3 H 3 K frequentist regret bound, which improves over existing work, accommodates unknown rewards, and matches the lower bound in terms of H and K. Regarding future work, we believe that our proof approach can be extended to rewards with bounded variance. We also believe that the extra √ d in our bound is an artefact of the proof, and specifically, the anti-concentration. We will investigate it further. Finally, we plan to study the practical efficiency of BEF-RLSVI through experiments on tasks with continuous state-action spaces in an extended version of this work.

Introduction

Model-free deep reinforcement learning (RL) has been successfully used in a wide range of problem domains, ranging from teaching computers to control robots to playing sophisticated strategy games [START_REF] Silver | Deterministic policy gradient algorithms[END_REF][START_REF] Schulman | Highdimensional continuous control using generalized advantage estimation[END_REF]Lillicrap et al., 2016;[START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF]. State-of-the-art policy gradient algorithms currently combine ingenious learning schemes with neural networks as function approximators in the so-called actor-critic framework [START_REF] Richard S Sutton | Policy Gradient Methods for Reinforcement Learning with Function Approximation[END_REF][START_REF] Schulman | Proximal policy optimization algorithms[END_REF][START_REF] Haarnoja | Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF]. While such methods demonstrate great performance in continuous control tasks, several discrepancies persist between what motivates the conceptual framework of these algorithms and what is implemented in practice to obtain maximum gains.

For instance, research aimed at improving the learning of value functions often restricts the class of function approximators through different assumptions, then propose a critic formulation that allows for a more stable policy gradient. However, new studies (Tucker et These findings leave the reader skeptical about actor-critic algorithms, suggesting that recent research tends to improve performance by introducing a bias rather than stabilizing the learning. Consequently, attempting to find a better baseline is questionable, as critics would typically fail to fit it [START_REF] Ilyas | A Closer Look at Deep Policy Gradients[END_REF]. In [START_REF] Tucker | The Mirage of Action-Dependent Baselines in Reinforcement Learning[END_REF], the authors argue that "much larger gains could be achieved by instead improving the accuracy of the value function". Following this line of thought, we are interested in ways to better approximate the value function. One approach addressing this issue is to put more focus on relative state-action values, an idea introduced in the literature on advantage reinforcement learning (Harmon and Baird III, n.d.) followed by works on dueling [START_REF] Wang | Dueling network architectures for deep reinforcement learning[END_REF] neural networks. More recent work [START_REF] Lin | Ranking Policy Gradient[END_REF]) also suggests that considering the relative action values, or more precisely the ranking of actions in a state leads to better policies. The main argument behind this intuition is that it suffices to identify the optimal actions to solve a task. We extend this principle of relative action value with respect to the mean value to cover both state and state-action-value functions with a new objective for the critic: minimizing the variance of residual errors.

In essence, this modified loss function puts more focus on the values of states (resp. stateactions) relative to their mean value rather than their absolute values, with the intuition that solving a task corresponds to identifying the optimal action(s) rather than estimating the exact value of each state. In summary, this paper:

Related Work

• Introduces Actor with Variance Estimated Critic (AVEC), an actor-critic method providing a new training objective for the critic based on the residual variance.

• Provides evidence for the improvement of the value function approximation as well as theoretical consistency of the modified gradient estimator.

• Demonstrates experimentally that AVEC, when coupled with state-of-the-art policy gradient algorithms, yields a significant performance boost on a set of challenging tasks, including environments with sparse rewards.

• Provides empirical evidence supporting a better fit of the true value function and a substantial stabilization of the gradient.

Related Work

Our approach builds on three lines of research, of which we give a quick overview: policy gradient algorithms, regularization in policy gradient methods, and exploration in RL.

Policy gradient methods use stochastic gradient ascent to compute a policy gradient estimator. This was originally formulated as the REINFORCE algorithm [START_REF] Williams | Simple statistical gradient-following algorithms for connectionist reinforcement learning[END_REF]. [START_REF] Kakade | Approximately Optimal Approximate Reinforcement Learning[END_REF] later created conservative policy iteration and provided lower bounds for the minimum objective improvement. (Peters, Mulling, and Altun, 2010) replaces regularization by a trust region constraint to stabilize training. In addition, extensive research investigated methods to improve the stability of gradient updates, and although it is possible to obtain an unbiased estimate of the policy gradient from empirical trajectories, the corresponding variance can be extremely high. To improve stability, [START_REF] Weaver | The Optimal Reward Baseline for Gradient•Based Reinforcement Learning[END_REF] shows that subtracting a baseline [START_REF] Williams | Simple statistical gradient-following algorithms for connectionist reinforcement learning[END_REF] from the value function in the policy gradient can be very beneficial in reducing variance without damaging the bias. However, in practice, these modifications on the actor-critic framework usually result in improved performance without a significant variance reduction [START_REF] Tucker | The Mirage of Action-Dependent Baselines in Reinforcement Learning[END_REF][START_REF] Ilyas | A Closer Look at Deep Policy Gradients[END_REF]. Currently, one of the most dominant on-policy methods are proximal policy optimization (PPO) [START_REF] Schulman | Proximal policy optimization algorithms[END_REF] and trust region policy optimization (TRPO) [START_REF] Schulman | Trust Region Policy Optimization[END_REF], both of which require new samples to be collected for each gradient step. Another direction of research that overcomes this limitation is off-policy algorithms, which therefore benefit from all sample transitions; soft actor-critic (SAC) [START_REF] Haarnoja | Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF]) is one such approach achieving state-of-the-art performance.

Several works also investigate regularization effects on the policy gradient [START_REF] Jaderberg | Reinforcement learning with unsupervised auxiliary tasks[END_REF][START_REF] Namkoong | Variance-based Regularization with Convex Objectives[END_REF]; Kartal, Hernandez-Leal, and Taylor, 2019; Flet-Berliac and Preux, 2019; Flet-Berliac and Preux, 2020); it is often used to shift the bias-variance trade-off towards reducing the variance while introducing a small bias. In RL, regularization is often Deep policy gradient: improved learning of value functions used to encourage exploration and takes the form of an entropy term [START_REF] Ronald | Function optimization using connectionist reinforcement learning algorithms[END_REF][START_REF] Schulman | Proximal policy optimization algorithms[END_REF]. Moreover, while regularization in machine learning generally consists in smoothing over the observation space, in the RL setting, [START_REF] Thodoroff | Temporal Regularization for Markov Decision Process[END_REF] shows that it is possible to smooth over the temporal dimension as well. Furthermore, [START_REF] Zhao | Regularized policy gradients: direct variance reduction in policy gradient estimation[END_REF] analyzes the effects of a regularization using the variance of the policy gradient (the idea is reminiscent of SVRG descent [START_REF] Johnson | Accelerating stochastic gradient descent using predictive variance reduction[END_REF])) which proves to provide more consistent policy improvements at the expense of reduced performance. In contrast, as we will see later, AVEC does not change the policy network optimization procedure nor involves any additional computational cost.

Exploration has been studied under different angles in RL, one common strategy is ε-greedy, where the agent explores with probability ε by taking a random action. This method, just like entropy regularization, enforces uniform exploration and has achieved recent success in game playing environments [START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF][START_REF] Van Hasselt | Deep reinforcement learning with double q-learning[END_REF][START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF]. On the other hand, for most policy-based RL, exploration is a natural component of any algorithm following a stochastic policy, choosing sub-optimal actions with non-zero probability. Furthermore, policy gradient literature contains exploration methods based on uncertainty estimates of values (Kaelbling, 1993;[START_REF] Tokic | Adaptive ε-greedy exploration in reinforcement learning based on value differences[END_REF], and algorithms which provide intrinsic exploration or curiosity bonus to encourage exploration [START_REF] Schmidhuber | Developmental robotics, optimal artificial curiosity, creativity, music, and the fine arts[END_REF][START_REF] Bellemare | Unifying count-based exploration and intrinsic motivation[END_REF][START_REF] Flet-Berliac | Adversarially Guided Actor-Critic[END_REF].

While existing research may share some motivations with our method, no previous work in RL applies the variance of residual errors as an objective loss function. In the context of linear regression, [START_REF] Brown | On Small-Sample Estimation[END_REF]) considers a median-unbiased estimator minimizing the risk with respect to the absolute-deviation loss function [START_REF] Pham | The mean and median absolute deviations[END_REF]) (similar in spirit to the variance of residual errors), their motivation is nonetheless different to ours. Indeed, they seek to be robust to outliers whereas, when considering noiseless RL problems, one usually seeks to capture those (sometimes rare) signals corresponding to the rewards.

Preliminaries

Background and Notations

We consider an infinite-horizon Markov Decision Problem (MDP) with continuous states s ∈ S, continuous actions a ∈ A, transition distribution s t+1 ∼ P(s t , a t ) and reward function r t ∼ R(s t , a t ). Let π θ (a|s) denote a stochastic policy with parameter θ, we restrict policies to being Gaussian distributions. In the following, π and π θ denote the same object. The agent repeatedly interacts with the environment by sampling action a t ∼ π(.|s t ), receives reward r t and transitions to a new state s t+1 . The objective is to maximize the expected sum of discounted 6.4 Method: Actor with Variance Estimated Critic rewards:

J(π) ≜ E τ ∼π ∞ t=0 γ t r (s t , a t ) , ( 6.1)
where γ ∈ [0, 1) is a discount factor [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF], and τ = (s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , . . . ) is a trajectory sampled from the environment using policy π. We denote the value of a state s in the MDP framework while following a policy π by V π (s) ≜ E τ ∼π ∞ t=0 γ t r (s t , a t ) |s 0 = s and the value of a state-action pair of performing action a in state s and then following policy π by Q π (s, a) ≜ E τ ∼π ∞ t=0 γ t r (s t , a t ) |s 0 = s, a 0 = a . Finally, the advantage function which quantifies how an action a is better than the average action in state s is denoted A π (s, a) ≜ Q π (s, a) -V π (s).

Critics in Deep Policy Gradients

In this section, we consider the case where the value functions are learned using function estimators and then used in an approximation of the gradient. Without loss of generality, we consider the algorithms that approximate the state-value function V . The analysis holds for algorithms that approximate the state-action-value function Q. Let f ϕ : S → R be an estimator of V π with ϕ its parameter. f ϕ is traditionally learned through minimizing the mean squared error (MSE) against V π . At iteration k, the critic minimizes:

L AC = E s f ϕ (s) -V π θ k (s) 2 , ( 6.2)
where the states s are collected under policy π θ k , and V π θ k (s) is an empirical estimate of V (see Section 6.4.3 for details). Similarly, using f ϕ : S × A → R instead, one can fit an empirical target Qπ .

Method: Actor with Variance Estimated Critic

In this section, we introduce AVEC and discuss its correctness, motivations and implementation.

Defining an Alternative Critic

Recent work [START_REF] Ilyas | A Closer Look at Deep Policy Gradients[END_REF] empirically demonstrates that while the value network succeeds in the supervised learning task of fitting V π (resp. Qπ ), it does not fit V π (resp. Q π ). We address this deficiency in the estimation of the critic by introducing an alternative value network loss. Following empirical evidence indicating that the problem is the approximation error and not the estimator per se, AVEC adopts a loss that can provide a better approximation error, and Deep policy gradient: improved learning of value functions yields better estimators of the value function (as will be shown in Section 6.5.4). At update k:

L AVEC = E s (f ϕ (s) -V π θ k (s)) -E s f ϕ (s) -V π θ k (s) 2 , ( 6.3)
with states s collected using π θ k . Note that the gradient flows in f ϕ twice using Equation 6.3. Then, we define our bias-corrected estimator:

g ϕ : S → R such that g ϕ (s) = f ϕ (s)+E s [ V π θ k (s)- f ϕ (s)].
Analogously to Equation 6.3, we define an alternative critic for the estimation of Q π by replacing V π by Qπ and f ϕ (s) by f ϕ (s, a). Lemma 6.1 (AVEC Policy Gradient). If f ϕ : S × A → R satisfies the parameterization assumption [START_REF] Richard S Sutton | Policy Gradient Methods for Reinforcement Learning with Function Approximation[END_REF] then g ϕ provides an unbiased policy gradient:

∇ θ J (π θ ) = E (s,a)∼π θ [∇ θ log(π θ (s, a))g ϕ (s, a)] .
This result also holds for the estimation of V π θ with f ϕ : S → R.

Proof. we consider the case in which the state-action-value function of a policy π θ is approximated. We prove that given some assumptions on this estimator function, we can use it to yield a valid gradient direction, i.e. we are able to prove policy improvement when following this direction.

In this setting, the critic minimizes the loss of Equation 6.3 where the targeted function is Qπ θ (s, a). Therefore, when a local optimum is reached, the gradient of the latter expression is zero:

∇ ϕ L AVEC = E (s,a)∼π Qπ θ (s, a) -f ϕ (s, a) -E (s,a)∼π Qπ θ (s, a) -f ϕ (s, a) × ( ∂f ϕ (s, a) ∂ϕ -E (s,a)∼π ∂f ϕ (s, a) ∂ϕ = 0
In the expression above, the expected value of the partial derivative disappears because the term in the first bracket is centered:

E (s,a)∼π ( Qπ θ (s, a) -f ϕ (s, a) -E (s,a)∼π [ Qπ θ (s, a) -f ϕ (s, a)])E (s,a)∼π [ ∂f ϕ (s, a) ∂ϕ ] = E (s,a)∼π ∂f ϕ (s, a) ∂ϕ E (s,a)∼π [ Qπ θ (s, a) -f ϕ (s, a) -E (s,a)∼π [ Qπ θ -f ϕ ]] = 0.

Method: Actor with Variance Estimated Critic

Simplifying the gradient at the local optimum becomes:

E (s,
1 π θ (s, a) , ( 6.5)
We obtain:

∇ θ J = E (s,a)∼π θ [∇ θ log(π θ (s, a))g ϕ (s, a)] . (6.6)
The latter follows by combining the parameterization assumption in Equation 6.5 with Equation 6.4. Indeed this entails:

E (s,a)∼π θ ( Qπ θ (s, a) -g ϕ (s, a)) ∂π θ (s, a) ∂θ 1 π θ (s, a) = 0. (6.7)
Since the expression above is null, we get: Which finished the proof.

∇ θ J = E (s,a)∼π θ [∇ θ log(π θ (s, a
Remark 6.2. While the proof seems more or less generic, the assumption in Equation 6.5 is extremely constraining to the possible approximators. [START_REF] Richard S Sutton | Policy Gradient Methods for Reinforcement Learning with Function Approximation[END_REF] quotes J. Tsitsiklis who believes that a linear g ϕ in the features of the policy may be the only feasible solution for this condition.

Concretely, such an assumption cannot hold since neural networks are the standard approximators used in practice. Moreover, empirical analysis [START_REF] Ilyas | A Closer Look at Deep Policy Gradients[END_REF] indicates that commonly used algorithms fail to fit the true value function. However, this does not rule out the usefulness of the approach but rather begs for more questioning of the true effect of such biased baselines.

Building Motivation

Here, we present the intuition behind using AVEC for actor-critic algorithms. [START_REF] Tucker | The Mirage of Action-Dependent Baselines in Reinforcement Learning[END_REF] and [START_REF] Ilyas | A Closer Look at Deep Policy Gradients[END_REF] indicate that the approximation error ∥ V π -V π ∥ is problematic, suggesting that the variance of the empirical targets V π (s t ) is high. Using L AVEC , our approach reduces the variance term of the MSE (or distance to V π ) but mechanistically also increases the bias. Our intuition is that since the bias is already quite substantial [START_REF] Ilyas | A Closer Look at Deep Policy Gradients[END_REF], it may be possible to reduce the variance enough so that even though the bias increases, the total MSE reduces.

State-value function estimation.

In this case, optimizing the critic with L AVEC can be interpreted as fitting V ′π (s) = V π (s) -E s ′ [ V π (s ′ )] using the MSE. We show that the targets V ′π are better estimations of

V ′π (s) = V π (s) -E s ′ [V π (s ′ )] than V π are of V π .
To illustrate this, consider T independent random variables (X i ) i∈{1,...,T } . We denote

X ′ i = X i -1 T T j=1 X j and V(X) the variance of X. Then, V (X ′ i ) = V (X i ) -2 T V (X i ) + 1 T 2 T j=1 V (X j ) and V(X ′ i ) < V(X i ) as long as ∀i 1 T T j=1 V(X j ) < 2V(X i ),
or more generally when state-values are not strongly negatively correlated 2 and not very discordant. This entails that V ′π has a more compact span, and is consequently easier to fit. This analysis shows that the variance term of the MSE is reduced compared to traditional actor-critic algorithms, but does not guarantee it counterbalances the bias increase. Nevertheless, in practice, the bias is so high that the difference due to learning with AVEC is only marginal and the total MSE decreases. We empirically demonstrate this claim in Section 6.5.4.

State-action-value function estimation.

In this case, Equation 6.3 translates into replacing V π (s) by Qπ (s, a) and f ϕ (s) by f ϕ (s, a) and the rationale for optimizing the residual variance of the value function instead of the full MSE becomes more straightforward: the practical use of the Q-function is to disentangle the relative values of actions for each state [START_REF] Richard S Sutton | Policy Gradient Methods for Reinforcement Learning with Function Approximation[END_REF]. AVEC's effect on relative values is illustrated in a didactic regression with one variable example in Figure 6.1 where grey markers are observations and the blue line is our current estimation. Minimizing the MSE, the line is expected to move towards the orange one in order to reduce errors uniformly. Minimizing the residual variance, it is expected to move near the red one. In fact, L AVEC tends to further penalize observations that are far away from the mean, implying that AVEC allows a better recovery of the "shape" of the target near extrema. In particular, we see in the figure that the maximum and minimum observation values are quickly identified. Would the approximators be linear and the target state-values independent, the two losses become equivalent since ordinary least squares would provide minimum-variance mean-unbiased estimation. It should be noted that, as in all the works related to ours, we consider noiseless tasks, i.e. the transition matrix is deterministic. As such, there are no outliers and extreme state-action values correspond to learning signals. In this context, high estimation errors indicate where (in the state or action-state space) the training of the value function should be improved, as opposed to possible outliers.

Implementation

We apply this new formulation to three of the most dominant deep policy gradient methods to study whether it results in a better estimation of the value function. A better estimation of the value function implies better policy improvements. We now describe how AVEC incorporates its residual variance objective into the critics of PPO [START_REF] Schulman | Proximal policy optimization algorithms[END_REF], TRPO [START_REF] Schulman | Trust Region Policy Optimization[END_REF] and SAC [START_REF] Haarnoja | Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF]. Let B be a batch of transitions. In PPO and TRPO, AVEC modifies the learning of V ϕ (line 12 of Algorithm 6.1) using: for each gradient step do 11:

L 1 AVEC (ϕ) = E s∼B (f ϕ (s) -V π (s)) -E s∼B f ϕ (s) -V π (s) 2 , then V ϕ = f ϕ (s) + E s∼B [ V π (s) -f ϕ (s)], where V π (s t ) = f ϕ old (s t ) + A t such
θ ← θ -λ π ∇θ J ALGO (π θ ) 12: ϕ ← ϕ -λ V ∇ϕ L 1 AVEC (ϕ) 13:
end for 14: end for In SAC, AVEC modifies the objective function of (Q ϕ i ) i=1,2 (line 13 of Algorithm 6.2) using:

L 2 AVEC (ϕ i ) = E (s,a)∼B (f ϕ i (s, a) -Qπ (s, a)) -E (s,a)∼B f ϕ i (s, a) -Qπ (s, a) 2 , then Q ϕ i = f ϕ i (s, a) + E (s,a)∼B [ Qπ (s, a) -f ϕ i (s, a)],
: ψ ← ψ -λ V ∇ψ J V (ψ) and ϕ i ← ϕ i -λ Q ∇ϕ i L 2 AVEC (ϕ i ) for i ∈ {1, 2} 12: Update θ ← θ -λ π ∇θ J(π θ ); ψ ← βψ + (1 -β) ψ 13:
end for 14: end for Finally, notice that AVEC does not modify any other part of the considered algorithms whatsoever, which keeps their implementation and computational complexity unchanged.

Experimental Study

Remark 6.3. Theoretically, L AVEC is defined as the residual variance of the value function (cf Equation 6.3). However, state-values for a non-optimal policy are dependent and the variance is not tractable without access to the joint law of state-values. Consequently, to implement AVEC in practice we use the best-known proxy at hand, which is the empirical variance formula assuming independence:

L AVEC = 1 T -1 T t=1 f ϕ (s t ) -V π (s t ) - 1 T T t=1 f ϕ (s t ) -V π (s t ) 2 ,
where T is the size of the sampled trajectory.

(Greensmith, Bartlett, and Baxter, 2004) provides some support for this approximation by showing that weakly dependent variables tend to concentrate more than independent ones.

Experimental Study

In this section, we conduct experiments along four orthogonal directions. (a) We validate the superiority of AVEC compared to the traditional actor-critic training. (b) We evaluate AVEC in environments with sparse rewards. (c) We clarify the practical implications of using AVEC by examining the bias in both the empirical and true value function estimations as well as the variance in the empirical gradient. (d) We provide an ablation analysis and study the bias-variance trade-off in the critic by considering two continuous control tasks. We point out that a comparison to variance-reduction methods is not considered in this paper: [START_REF] Tucker | The Mirage of Action-Dependent Baselines in Reinforcement Learning[END_REF]) demonstrated that their implementations diverge from the unbiased methods presented in the respective papers and unveiled that not only do they fail to reduce the variance of the gradient, but that their unbiased versions do not improve performance either. Note that in all experiments we choose the hyperparameters providing the best performance for the considered methods which can only penalyze AVEC (cf Appendix D.1). In all the figures hereafter (except Figure 6.5c and 6.5d), lines are average performances and shaded areas represent one standard deviation.

Continuous Control

For ease of comparison with other methods, we evaluate AVEC on the MuJoCo (Todorov, Erez, and Tassa, 2012) and the PyBullet (Coumans and Bai, 2016) continuous control benchmarks (see Appendix D.2 for details) using OpenAI Gym (Brockman et al., 2016) Table 6.1 reports the results while Figure 6.2 and 6.3 show the total average return for SAC and PPO. TRPO results are provided in Section 6.5.2 for readability. We provide the list of hyperparameters and further implementation details in Appendix D.1.

When coupled with SAC and PPO, AVEC brings very significant improvement (on average +26% for SAC and +39% for PPO) in the performance of the policy gradient algorithms, improvement which is consistent across tasks. As for TRPO, while the improvement in performance is less striking, AVEC still manages to be more efficient in terms of sampling in all tasks.

Experimental Study

Overall, AVEC improves TRPO, PPO and SAC in terms of performance and efficiency. This does not imply that our method would also improve other policy gradient methods that use the traditional actor-critic framework, but since we evaluate our method coupled with three of the best performing on-and off-policy algorithms, we believe that these experiments are sufficient to prove the relevance of AVEC.

In addition, in Figure 6.3, we plot the total average return for AVEC coupled with SAC and PPO on the Walker2d task. Similar to considered other continuous control tasks from MuJoCo and PyBullet, AVEC brings a significant performance improvement (+26% for SAC and +33% for PPO), confirming the generality of our approach. Finally, in our experiments we do not seek the best hyperparameters for the AVEC variants, we simply adopt the parameters allowing us to optimally reproduce the baselines. Alternatively, if one seeks to evaluate AVEC independently of a considered baseline, further hyperparameter tuning should produce better results. Notice that since no additional calculations are needed in AVEC's implementation, computational complexity remains unchanged.

Comparison with TRPO

In order to evaluate the performance gains in using AVEC instead of the usual actor-critic framework, we produce some additional experiments with the TRPO (Schulman, Levine, et al., 2015) algorithm. Figure 6.4 shows the learning curves while Table 6.2 reports the results.

Sparse Reward Signals

Domains with sparse rewards are challenging to solve with uniform exploration as agents receive no feedback on their actions before starting to collect rewards. In such conditions AVEC performs better, suggesting that the shape of the value function is better approximated, encouraging exploration. The relative value estimate of an unseen state is more accurate: in Section 6.4.2, AVEC identifies extreme state-values (e.g., non-zero rewards in tasks with sparse rewards) faster. In Figures 6.5a and 6.5b, we report the performance of AVEC in the Acrobot and MountainCar environments: both have sparse rewards. AVEC enhances TRPO and PPO in both experiments. When PPO and AVEC-PPO both reach the best possible performance, AVEC-PPO exhibits better sample efficiency. Figures 6.5c and 6.5d illustrate how the agent improves its exploration strategy in MountainCar: while the PPO agent remains stuck at the bottom of the hill (red), the graph suggest that AVEC-PPO learns the difficult locomotion principles in the absence of rewards and visits a much larger part of the state space (green). This improved performance in sparse environments can be explained by the fact that AVEC is able to pick up on experienced positive reward more easily. Moreover, the reconstructed shape of the value function is more accurate around such rewarding states, which pushes the agent to explore further around experienced states with high values. Learning the Empirical Target. In Figure 6.6, we report the quality of fit (MSE) of the empirical target V π in the methods PPO and AVEC-PPO in the AntBullet and HalfCheetahBullet tasks. We observe that PPO better fits the empirical target than when equipped with AVEC, which is to be expected since vanilla PPO optimizes the MSE directly. This result put aside the remarkable improvement in the performance of AVEC-PPO (Figure 6.2) suggests that AVEC might be a better estimator of the true value function. We examine this claim below because if true, it would indicate that it is indeed possible to simultaneously improve the performance of the agents and the stability of the method.

Deep policy gradient: improved learning of value functions

Experimental Study

Deep policy gradient: improved learning of value functions

Learning the True Target. A fundamental premise of policy gradient methods is that optimizing the objective based on empirical return leads to a better policy. Which is why we investigate the quality of fit of the true target. To approximate the true value function, we fit the returns sampled from the current policy using a large number of transitions (3 • 10 5 ). Figure 6.7 shows that g ϕ is far closer to the true value function half of the time (horizon is 10 6 ) than the estimator obtained with MSE, then as close to it. Comparing Figure 6.7 with Figure 6.6, we see that the distance to the true target is close to the estimation error for AVEC-PPO, while for PPO, it is at least two orders of magnitude higher at all times. We further investigate these results in Figure6.8 where we study the variation of the squared bias and variance components of the MSE to the true target (MSE = Var + Bias 2 ).

Variation of the Bias and Variance terms: PPO. In Figure 6.8, we show the variation of the bias and variance terms in the MSE between the estimators (of AVEC-PPO and PPO) and the true target: . X-axis: we run PPO and AVEC-PPO and for every t ∈ {1, 2, 4, 6, 9} • 10 5 , we stop training, use the current policy to interact with the environment for 3 • 10 5 transitions, and use these transitions to estimate the true value function.

E[∥g ϕ -V π ∥ 2 2 ] = Bias(AVEC) 2 + Var(AVEC) and E[∥V ϕ (PPO) -V π ∥ 2 2 ] =
We observe that the variance reduction is more substantial than that of the bias. Using those results and Figure 6.7 showing that the distance of the estimator to V π is lower when using AVEC confirms that the variance reduction effect counterbalances the bias increase. Note that the % Variation of the Var term is always negative in our experiments, and that the shaded areas that suggest otherwise are merely due to a false assumption of symmetrical deviations, itself due to the assumption of Gaussianity needed to construct confidence intervals.

For completeness, we also analyze the distance to the true target for the Q-function estimator in SAC and AVEC-SAC in AntBullet and HalfCheetahBullet. Indeed, in Figure 6.9, we compare First, for SAC-like algorithms where the critic learns a state-action-value function, our results strongly suggest that state-actions with extreme values are identified more quickly. Second, for PPO-like methods where the critic learns the state-values, we show that the variance of the gradient is reduced and empirically demonstrate that this is due to a better approximation of the state-values. In sparse reward environments, the theoretical intuition behind a variance estimated critic is more explicit and is also supported by empirical evidence. In addition to corroborating the results in [START_REF] Ilyas | A Closer Look at Deep Policy Gradients[END_REF] proving that the value estimator fails to fit V π , we propose a method that succeeds in improving both the sample complexity and the stability of prominent actor-critic algorithms. Furthermore, AVEC benefits from its simplicity Chapter 7

General Conclusion and Perspectives

The past is gone, the future is unseen Rise and seize the opportunity in between Poet unknown 1In this thesis, we have made significant contributions to the fields of bandits, linear regression, and reinforcement learning. Our overarching goal was to make the RL framework, algorithms, and assumptions more realistic.

We began by studying the simplest model for sequential decision making, multi-armed bandits, in Chapter 3. We considered the finite number of arms case and studied a pure exploration objective called Thresholding bandits. We proposed a Frank-Wolfe based method of designing algorithms, which we made to extend and solve a wider class of losses. By doing so, we were able to improve the state-of-the-art both theoretically and empirically. Moreover, we applied our newly formulated, intuitive, and generic proof scheme to improve the bounds on the number of mistakes made by all previously existing algorithms in this setting. In terms of the limitations of our algorithm, it should be noted that our current state-of-the-art bound remains within a factor of 4 in the exponent from the ideal oracle. Additionally, during our research, we made an intriguing observation that adaptive algorithms can sometimes outperform nonadaptive (i.e. offline) oracles. A theoretical understanding of this phenomenon has yet to be developed. Lastly, we are also keen on extending this setting to large or continuous actions, as there are connections with other index-based algorithms such as IMED [START_REF] Honda | Non-asymptotic analysis of a new bandit algorithm for semi-bounded rewards[END_REF]. Further examination of these connections could lead to a deeper understanding of how to scale our generic algorithm.

General Conclusion and Perspectives

In Chapter 4, we delved into the linear regression problem and uncovered an old result from the adversarial setting, which we adapted for the stochastic case. Our findings revealed that this should be the default algorithm instead of the commonly used ridge regression. Additionally, we applied this modification to linear bandits and proved that it allows for the elimination of a pervasive assumption. Our analysis holds value not just in theory but also for practical use, as it presents a novel perspective on a frequently employed method and highlights novel insights about the effect of regularization. One of the main limitations of this method becomes evident in the context of MDPs, as it can be difficult to regularize using a random future explanatory variable. However, we were still able to eliminate the assumption of a known bound on observations in MDPs using other techniques in Chapter 5. Additionally, in Chapter 4, we discussed how the dependence on d in our bounds can be improved using Theorem 1 of [START_REF] Tirinzoni | An asymptotically optimal primal-dual incremental algorithm for contextual linear bandits[END_REF]. Possible avenues for future research building on this contribution include applying these techniques to more complex regression scenarios where the features are themselves dynamic, such as neural networks and regression trees.

To further our progress, we focused our interest on MDPs with continuous state action spaces in Chapter 5. At first we were intrigued by the ubiquity of the linear MDP model, this prevalence can be interpreted as a lack of optimism in finding efficient and effective representational assumptions for learning. Defying this attitude, we became interested in a recent MDP representation that has shown great promise and compelling expressive power, modeling real MDPs like Tabular, Factored, and Linear Quadratic Regulators. While investigating this model, we discovered a new and crucial property about it that allowed us to create an algorithm with tractable planning, which enjoys an optimal regret bound in its dependence on the number of episodes and their length. Furthermore, we presented several results that hold independent value, such as the ability to eliminate the need for clipping value functions, which removes an superfluous and omnipresent non-linearity. A major challenge we identified in our research is the dependence on d in our bound. From the available literature, it appears that this dependence is always suboptimal for tractable algorithms, and that a slight modification to UCB-style algorithms (which are intractable) can lead to the optimal dependence. We aim to investigate whether the increased dependence on dimensionality is a necessary trade-off for achieving tractability. Another area for future research is testing this representation in practice to evaluate its effectiveness. Additionally, a necessary direction for future work is learning features instead of providing them to the learner, which could be achieved through techniques such as causal discovery [START_REF] Zhang | Efficient reinforcement learning in block mdps: A model-free representation learning approach[END_REF][START_REF] Chalupka | Causal feature learning: an overview[END_REF] or automatic Bayesian search (Malkomes, Schaff, and Garnett, 2016).

In Chapter 6, the final contribution of this thesis, we examined a framework of decisionmaking known as deep policy gradients. This is an infinite-horizon setting that utilizes deep neural networks to model both the value function and the policy, and is optimized through the use of gradient descent. Inspired by some recently discovered -practical-limitations in existing algorithms, we proposed a minor modification to the value function loss which led to significant enhancements. Our proposed method was supported by both theoretical arguments and empirical evidence, showing improvements over traditional training loss and proving the accuracy of our insights. For instance, we validated independent findings, hypothesized possible implications for these shortcomings, and validated our intuition as well through practical experimentation on simulated environments. In terms of extending this work, one potential next step would be to find a balance between bias and variance in the mean square error loss, rather than completely eliminating bias as we have done. Additionally, some researchers propose that other parameters such as regularization or the weight of the stability penalty could also be optimized using gradient descent for automation [START_REF] Haarnoja | Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF]. Lastly, from a theoretical perspective, we are curious about the possibility of replacing the iterative optimization technique with theoretically sound algorithms for bi-level problems.

To summarize our contributions, they are organized between 1) improvements of existing methods, including reducing assumptions like the boundedness in linear regression Chapter 4 and simplifying algorithms like removing clipping Chapter 5 and slightly modifying the loss Chapter 6. And 2), notable novel contributions such as generic algorithm design schemes and general proof techniques Chapter 2, and discovering novel implications for useful structures Chapter 5 therefore leading to improved tractability and simpler algorithms. Our goal throughout this journey was unique: to contribute to the RL research by improving and proposing algorithms that are both theoretically sound and user-friendly for practitioners, and to support the advancement towards a more realistic RL formalism. We have made notable contributions to the research supporting this goal by proposing novel methods, adapting existing algorithms to new settings, and uncovering new properties of commonly used models. Overall, we believe that this thesis has made significant contributions to the field of RL and will have a lasting impact on the field.

Our directly relevant perspectives fall into two categories: 1) Designing tractable and optimal RL algorithms, for which we believe that the bilinear exponential family could be very perinent. Indeed it is an expressive representation enjoying many interesting properties that can be useful for other RL objectives. For instance, we intend to study this model for best policy identification as well as reward-free reinforcement learning. Furthermore, we wish to study the empirical performance of this representation, and possibly incorporate provable methods to automatically build the feature mapping (Malkomes, Schaff, and Garnett, 2016) instead of requiring it as prior knowledge. 2) Improving model learning in RL, we are interested in studying new RL settings and attempting to improve model estimation. This could take the form of theoretical improvements, e.g. in the mixture of experts setting, information about the level of confidence for each expert could be available and we don't know how to take advantage of it to make a final prediction. Improving model learning concerns empirical methods as well, e.g. in the deep learning literature, if validation data is available then the learned model could

General Conclusion and Perspectives

be calibrated in the hope of better adjusting to any eventual change of distribution. Since the coordinates of the test set are given, it should be studied whether incorporating this information could be beneficial, in the form of a regularization for example.

All in all, our goal in this thesis and for the future is to combine probability theory with machine learning and optimization techniques to tackle sequential decision making problems. Practically, we are interested in studying online learning and reinforcement learning problems and designing algorithms that can a) can be generic enough to model realistic scenarios or to constitute plausible approximations, and b) can be analyzed theoretically without stringent assumptions.

Complements on Chapter 4 B.1 Technical results

Lemma B.1. (Tail inequality) For all δ > 0, σ ′ > 0, with probability at least 1 -δ, for all T > 0:

|S T | ≤ 2(A T + 1/σ ′2 ) log √ σ ′2 A T + 1 δ
Proof. We use the method of mixtures, denote

M λ t = exp λε t (θ T -θ * ) ⊤ x t - λ 2 2 (θ T -θ * ) ⊤ x t 2 .
Without loss of generality, we can assume that (ε s ) s≥1 is 1-sub-Gaussian (this can be achieved by scaling features appropriately), then E[M λ t ] ≤ 1. Let Λ ∼ N(0, σ ′2 ) be a Gaussian random variable and define

M t = E[M Λ t |F ∞ ]. We have E[M t ] = E[E[M Λ t |Λ]] ≤ 1.
By making explicit M t and using Markov's inequality we get that for any stopping time τ , for all δ > 0, with probability at least 1 -δ:

|S τ | 2 1/σ ′2 + A τ ≤ 2σ 2 log √ 1 + σ ′2 A τ δ .
We conclude using the same stopping time construction in Proof B. Proof. Using the Weinstein-Aronszajn identity:

∥x t ∥ 2 G -1 t = 1 -|G t-1 |
|Gt| , and that z -1 ≥ log(z) leads to:

T t=T 0 ∥x t ∥ 2 G -1 t ≤ T t=1 -log |G t-1 | |G t | = log |G T | |G T 0 | .

B.1 Technical results

Since ∥x t ∥ 2 ≤ X, using the AM-GM inequality:

T t=T 0 log 1 + ∥x t ∥ 2 G -1 t-1 ≤ d log 1 + T X 2 /
T t=1 log 1 + ∥x t ∥ 2 G -1 t-1 = log G T G 0 .
Then since ∥x t ∥ 2 ≤ X and using the AM-GM inequality: ≤ λ max (G -1 t-1 )||x t || 2 2 ≤ X 2 /λ and the concavity of the function log we find:

T t=1 log 1 + ∥x t ∥ 2 G -1 t-1 ≤ d log 1 + T X 2 /λd .
T t=1 ∥x t ∥ 2 G -1 t-1 ≤ T t=1 X 2 /λ log(1 + X 2 /λ) log 1 + ∥x t ∥ 2 G -1 t-1
.

The last inequality can also be proved by noting that x → x/ log(1 + x) is non-decreasing which can be used to bound every feature norm.

Lemma B.4. (Tail inequality, see Corollary 8 of (Abbasi-Yadkori, Pal, and Szepesvari, 2012)) Define

S t = t s=1 ε s (θ s-1 -θ * ) ⊤ x s and let (F t ) t≥0 be a filtration such that x t is F t-1 measurable and ε t is F t measurable. Then S t is a martingale with respect to F t and for any δ > 0, σ ′ > 0, with probability at least 1 -δ, for all t ≥ 0:

|S t | ≤ σ 2 1/σ ′2 + t s=1 (θ t-1 -θ * ) ⊤ x t 2 log   1 + σ ′2 t s=1 (θ t-1 -θ * ) ⊤ x t ) 2 δ  

B.1 Technical results

Proof. Denote X t = (x ⊤ 1 , . . . , x ⊤ t ), ε t = (ε 1 , . . . , ε t ) ⊤ . Using

θ t = G -1 t+1 X ⊤ t (Xθ * + ε t ) = G -1 t+1 X ⊤ t ε t + G -1 t+1 (X ⊤ t X t + λI + x ⊤ t+1 x t+1 )θ * -G -1 t+1 (λI + x ⊤ t+1 x t+1 )θ * = G -1 t+1 X ⊤ t ε t + θ * -G -1 t+1 (λI + x ⊤ t+1 x t+1 )θ * ,
we get

|x ⊤ θ t -x ⊤ θ * | = |x ⊤ G -1 t+1 X t ε t -x ⊤ G -1 t+1 (λθ * + x t+1 x ⊤ t+1 θ * )| ≤ ∥x∥ G -1 t+1 ∥X ⊤ t ε t ∥ G -1 t+1 + √ λ + X ∥θ * ∥ 2 ,
where in the last inequality we used Cauchy-Schwartz inequality and that by the Sherman-

Morrison formula x ⊤ t+1 G -1 t+1 x t+1 = x ⊤ t+1 G -1 t x t+1 1+x ⊤ t+1 G -1 t x t+1 ≤ 1.
We know that:

||X ⊤ t ε t || G -1 t+1 ≤ ||X ⊤ t ε t || G -1 t
which allows us to use Theorem 1 from Abbasi-Yadkori, Pál, and Szepesvári (2011) that we recall directly after this proof. We conclude by plugging in x = G t+1 (θ t -θ * ).

Theorem. (Self-Normalized Bound for Vector-Valued Martingales). Let {F t } ∞ t=0 be a filtration. Let {η t } ∞ t=1 be a real-valued stochastic process such that η t is F t -measurable and η t is conditionally R -sub-Gaussian for some R ≥ 0 i.e.

∀λ ∈ R E e ληt | F t-1 ≤ exp λ 2 R 2 2
Let {X t } ∞ t=1 be an R d -valued stochastic process such that X t is F t-1 -measurable. Assume that V is a d × d positive definite matrix. For any t ≥ 0, define

Vt = V + t s=1 X s X ⊤ s S t = t s=1 η s X s .
Then, for any δ > 0, with probability at least 1 -δ, for all t ≥ 0,

∥S t ∥ 2 V -1 t ≤ 2R 2 log    det Vt 1/2 det(V ) -1/2 δ    .
Note that the deviation of the martingale ∥S t ∥ 2 V -1 t is measured by the norm weighted by the matrix V -1 t which is itself derived from the martingale, hence the name "self-normalized bound". Lemma B.6. (Tail inequality, Forward algorithm) Define S t = t s=1 ε s (θ s-1 -θ * ) ⊤ x s and let (F t ) t≥0 be a filtration such that x t is F t-1 measurable and ε t is F t measurable. Then S t is a martingale with Complements on Chapter 5

C.1 Concentrations

C.1.1 Concentration of the transition parameter

We recall the important concentration of the maximum likelihood estimator for general bilinear exponential families (cf Theorem 1 of (Chowdhury, Gopalan, and Maillard, 2021)).

Theorem C.1. Suppose {F t } ∞ t=0 is a filtration such that for each t, (i) s t+1 is F t -measurable, (ii) (s t , a t ) is F t-1 measurable, and (iii) given (s t , a t ) , s t+1 ∼ P p θ p (• | s t , a t ) according to the exponential family defined by Equation (5.1). Let θp (k) be the penalized MLE defined by Equation (5.6), and let Z p s,a (θ) be strictly convex in θ for all (s, a). Then, for any δ ∈ (0, 1], with probability at least 1 -δ, the following holds uniformly over all n ∈ N :

k t=1 KL st,at θp (k), θ p + η 2 θ p -θp (k) 2 A - η 2 ∥θ p ∥ 2 A ≤ log C p A,k δ ,
where

C p A,k = R d exp -η 2 ∥θ ′ ∥ 2 A dθ ′ / R d exp -k t=1 KL st,at (θ k , θ ′ )-η 2 ∥θ ′ -θ k ∥ 2 A dθ ′ . De- fine G s,a = ∆ φ(s, a) ⊤ A ⊤ i A j φ(s, a) i,j∈[d]
, we have

C p A,k ≤ det I + β p η -1 A -1 k t=1
G st,at , where β p = sup θ,s,a λ max C θ s,a [ψ (s ′ )] .

A proof of this result can be found in the work (Chowdhury, Gopalan, and Maillard, 2021). We provide an almost similar proof for the concentration of rewards in the next section. , we have

C A,k ≤ det I + β r η -1 A -1 k t=1
G st,at , where β r := ∥B∥ 2 2 sup θ,s,a Var θ s,a (r).

Proof. We proceed similar to the proof of Theorem 1 in (Chowdhury and Gopalan, 2019).

Step 1: Martingale construction. First, observe that by assuming strict convexity, the logpartition function Z r s,a becomes a Legendre function. Now for the conditional exponential family model, the KL divergence between P r θ r (• | s, a) and P r θ ′r (• | s, a) can be expressed as a Bregman divergence associated to Z r s,a with the parameters reversed, i.e. (θ r i + λ i ) B ⊤ A i φ(s, a) -Z r s,a (θ r ) dr = exp Z r s,a (θ r + λ) -Z r s,a (θ r ) -λ ⊤ ∇Z r s,a (θ r ) = exp B Z r s,a (θ r )

This implies E exp λ ⊤ S n | F n-1 = exp λ ⊤ S n-1 + B Z nn,an,θ r (λ) thus

E M λ n | F n-1 = M λ n-1 . Therefore M λ n ∞ n=0
is a non-negative martingale adapted to the filtration {F n } ∞ n=0 and actually satisfies E M λ n = 1. For any prior density q(θ) for θ, we now define a mixture of martingales

M n = R d M λ n q (θ r + λ) dλ (C.1)
Then {M n } ∞ n=0 is also a non-negative martingale adapted to {F n } ∞ n=0 and in fact, E [M n ] = 1.

Step 2: Method of mixtures. Considering the prior density N (0, (ηA) -1 ), we obtain from (C.1) that

M n = c 0 R d exp λ ⊤ S n - n t=1 B Z r
x t ,a t ,θ r (λ) -

C.1 Concentrations

Now, note that sup λ∈R d g k (λ) = η 2 θ rθr (k)

2 A
, where the supremum is attained at λ ⋆ = θr (k) -θ r . We then have g k (λ) = g n (λ) + sup Further, any optimal λ must satisfy ∇Z r n (θ r + λ) -∇Z r n (θ r ) = x n =⇒ ∇Z r n (θ r + λ) = ∇Z r n θr (n) .

One possible solution is λ = λ ⋆ . Now, since Z r n is strictly convex, the supremum is indeed attained at λ = λ ⋆ . We then have Since λ ⋆ = θr (n) -θ r , we have KL st,at θr (n), θ r + η 2 θ rθr (n)

c n = 1 R d exp -j∈{0,n} B Z r j (θ ′ , θ r + λ ⋆ ) dθ ′ = 1
2 A - η 2 ∥θ r ∥ 2 A ≥ log C A,n δ = P M n ≥ 1 δ ≤ δE [M n ] = δ
Step 3: A stopped martingale and its control. Let N be a stopping time with respect to the filtration {F n } ∞ n=0 . Now, by the martingale convergence theorem, M ∞ = lim n→∞ M n is almost surely well-defined, and thus M N is well-defined as well irrespective of whether N < ∞ or not. Let Q n = M min{N,n} be a stopped version of {M n } n . Then an application of Fatou's lemma θ ′ -θr (n) i φ (s t , a t ) ⊤ A ⊤ i A j φ (s t , a t ) θ ′ -θr (n)

j = β r 2 θ ′ -θr (n) 2 n t=1
G st,at , where β r := λ max BB ⊤ × sup θ,s,a Var θ s,a (r) and ∀i, j ≤ d, (G s,a ) i,j := φ(s, a) ⊤ A ⊤ i A j φ(s, a). Therefore, we obtain Proof. We prove the lemma as follows Proof. We prove these formulas by differentiating under the integral sign.

C A,n ≤ R d exp -η 2 ∥θ ′ ∥ 2 A dθ ′ R d exp -1 2 θ ′ -θr (n)
∇ i Z p s,
θ i -θ ′ i E θ s,a ψ s ′ ⊤ A i φ(s, a) -Z p s,a (θ) + Z p s,a θ ′ = 1 2 θ -θ ′ ⊤ ∇ 2 Z p s,a ( θ) θ -θ ′ ,
where in the last line, we used, by a Taylor expansion, that Z s,a (θ ′ ) = Z s,a (θ)+(∇Z s,a (θ)) ⊤ (θ ′ -θ)+

1 2 (θ-θ ′ ) ⊤ ∇ 2 Z s,a ( θ) (θ -θ ′ ) for some θ ∈ [θ, θ ′ ] ∞ .
The proof of the form of the KL divergence for the reward follows the same steps as above.

Complements on Chapter 5

A transportation lemma for rewards Lemma C.11. We provide a closed-form formula for the difference of expected rewards under two distinct parameters:

∃θ 3 ∈ [θ 1 , θ 2 ], E θ 1 s,a [r] = E θ 2 s,a [r] + Var θ 3 s,a (r) 2 B ⊤ M θ 1 -θ 2 φ(s, a)
Proof. Let's recall the gradient of the reward log partition function:

∇ i Z r s,a (θ r ) = E θ r s,a [r] B ⊤ A i φ(s, a)
then for all θ r′ we have:

E θ r s,a [r] = 1 B ⊤ M θ r′ φ(s, a) ∇ i Z r s,a (θ r ) ⊤ θ r′
Let θ 1 , θ 2 ∈ R d , using Taylor-Cauchy's formula there exists θ 3 ∈ [θ 1 , θ 2 ] such that:

E θ 1 s,a [r] = E θ 2 s,a [r] + 1 2B ⊤ M θ r′ φ(s, a) (θ 1 -θ 2 ) ⊤ ∇ 2 Z r s,a (θ 3 ) ⊤ θ r′
We know that ∇ 2 i,j Z r s,a (θ) = Var θ s,a (r) × φ(s, a) ⊤ A ⊤ i BB ⊤ A j φ(s, a), choosing θ r′ = θ 1 -θ 2 we find:

C.2.4 Elliptical potentials and elliptical lemma Elliptical lemma

Here we show a lemma that is popular for regret control in linear MDPs and linear Bandits. where B φ,A := sup s,a A -1 G s,a .

C.2 Technical results

Further, we have

C.2 Technical results

Proof. Let T be the set of rounds t when ∥a t ∥ V -1 t-1 ≥ 1 and G t = V 0 + t s=1 I T (s)a s a ⊤ s . Then

dλ + |T |L 2 d d ≥ trace (G n ) d d ≥ det (G n ) (Trace-determinant inequality) = det (V 0 ) t∈T 1 + ∥a t ∥ 2 G -1 t-1 ≥ det (V 0 ) t∈T 1 + ∥a t ∥ 2 V -1 t-1 ≥ λ d 2 |T |
where the third line follows from the matrix determinant lemma:

det Ḡn + a h a ⊤ h = det(I + a ⊤ h Ḡ-1 n a h ) det( Ḡn ).

Rearranging and taking the logarithm shows that .

Figure 1 -

 1 Figure 1 -Answer of DALL.E 2 for the prompt "PhD student in the wild, drawing style"

S set of states s ∈ S , 2 A 84 T

 284 range of integers {1, . . . , n} R + set of positive reals {τ ∈ R : τ ≥ 0} R set of real numbers N + set of positive integers N ∩ R + ∥x∥ Euclidean norm for a vector x ∈ R n |z| absolute value |z| = z + + z - z - negative part z -= z + -z z + positive part max(z, 0)e i normal basis vectors [0 . . . 0 1 . . . 0] ⊤ in R n for i ∈ [n], where 1 appears in the i th positionI n the identity matrix with dimension n × n M ⊤ transpose of a matrix M ∥A∥ 2 the induced L 2 matrix norm max i∈[n] λ i (A ⊤ A) P ≻ 0 a symmetric matrix P ∈ R n×n is positive definite o(•), O(•), Ω(•) Landau notations for positive functions: f (x) = o(g(x) means that g(x) ̸ = 0 and f (x)/g(x) → 0 for x → ∞, f (x) = O(g(x)) means that there exists x 0 , K > 0 such that f (x) ≤ Kg(x) from x ≥ x 0 , andf (x) = Ω(g(x)) means g(x) = O(f (x)) E expectationunder a probabilistic model P X set of probability measures on a measurable space X xix set of actions a ∈ A , 2 R(s, a) reward function R : s, a → R(s, a) ∈ [0, 1] P (s ′ | s, a) transition distribution s ′ ∼ P (s ′ | s, a) π policy π ⋆ optimal policy V state value function ( ⋆ for optimal value, π for policy value) , 5 Q state-action value function ( ⋆ for optimal value, π for policy value) , Bellman operator ( ⋆ for optimality, π for evaluation) D dataset Multi-armed bandits µ a reward vector (or function), µ a : A → R Linear regression x t Coordinate presented to the agent at time t y t Label of the t'th coordinate θ * ∈ R d , True parameter of the linear model θ t Estimation of the parameter at time t

Figure 1 . 1 -

 11 Figure 1.1 -Our contributions are divided by whether they improve existing algorithms and models or they propose entirely novel strategies. They are united in striving for reasonable structures.

1

 1

Contents 3 . 1 26 3. 2 30 3. 3 36 3. 4 38 3. 5 43 3. 6 49 3. 7 50 3. 8

 31262303364385436497508 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . An interesting class of algorithms . . . . . . . . . . . . . . . . . . . . . . . . . Loss upper bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Additional Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Beating the oracle? The benefits of adaptivity. . . . . . . . . . . . . . . . . . . Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Figure 3 . 1 -Figure 3 . 2 -

 3132 Figure 3.1 -Optimal and empirical sampling distributions with respect to µ.

Figure 3 . 3 -

 33 Figure 3.3 -Optimal sampling distribution and empirical sampling distribution with respect to µ when µ k = (-1) k k K for K = 50 arms.

Figure 3 .

 3 Figure3.1 illustrates on an example (µ k = (-1) k (k/K) 2 , k = 1, . . . , 50, and T = 500) the shape of the optimal allocation (arms near the threshold should not be drawn) as well as the empirical sampling distributions of several algorithms that pull all arms. Next, we illustrate how this optimal allocation evolves with the horizon T .

Figure 3 .

 3 Figure 3.4 compares the upper-bounds of Corollary 3.14 (APT), Theorem 3.5 (see also Equation (3.8)) (LSA), and Corollary 3.16 (FWT) for the particular case ∆ i = (i/K) 2 and a i = 1, for

Figure 3 . 4 -

 34 Figure 3.4 -[left] Comparison of the upper-bounds of Corollaries 3.14, 3.16 and that of the optimal non-adaptive oracle of Equation (3.5) (blue) when the gaps are of the form∆ i = (i/K) 2 . [right]Evolution over time of the size of the optimal sets S (blue) and S ′ (orange) that minimize the bound of Corollary 3.16.

Figure 3 .Figure 3 . 5 -

 335 Figure 3.5 compares the upper-bounds of Corollary 3.14 (APT), Equation (3.8) (LSA), and Corollary 3.16 (FWT) for the particular case ∆ i = i/K and a i = 1, for all i = 1, . . . , K and K = 50. We observe a behavior similar to that of Figure 3.4.

  The analysis and experiments work similarly for both versions. But the (1 + √ x) 2 version performs slightly better empirically while the max{1, x} version provides cleaner theoretical results. The experiments are averaged over 500 runs and consider arm distributions of the form ν k = N (µ k , 1). The gaps are thus ∆ k = |µ k |, for k = 1, . . . , K. The performance criterion is the sum of errors defined in Equation 3.4 with a k = 1. The experiments were run on a personal laptop with Intel Core i5, Dual Core, 3.1 GHz.

Figure 3 . 6 -Figure 3 . 7 -

 3637 Figure 3.6 -Sum of errors over time of the different algorithms when µk = (-1) k k K [left] and µ k = (-1) k k 2 K 2

Figure 3 . 8 -

 38 Figure 3.8 -[left] Median (and 0.25, 0.75 empirical quantiles obtained on 500 runs) of the ratio between the error suffered by each algorithm and that of the optimal non-adaptive oracle (µ k = (-1) k , k = 1, . . . , 100). [right] Ratio of the averaged errors (over 500 runs) of each algorithm with that of the oracle (µ k = (-1) k (k/K) 2 , k = 1, . . . , 50).

Contents 4 . 1 54 4. 2 57 4. 3 62 4. 4 67 4. 5 71 4. 6

 41542573624675716 Introduction and preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . Adversarial bounds and limitations . . . . . . . . . . . . . . . . . . . . . . . . High probability bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The unregularized-forward algorithm . . . . . . . . . . . . . . . . . . . . . . Application: linear bandits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 Linear regression: an improved algorithm & application to linear bandits

. 3 )Algorithm 4 . 2 : 3 observe x t ∈ R d 4 update parameter:

 34234 Linear regression: an improved algorithm & application to linear bandits The forward algorithm1 Given θ 0 ∈ R d 2 for t = 1, . . . , T do

Theorem 4 . 1 .

 41 (Theorem 4.6 2 of (Azoury and Warmuth,2001)) The online ridge regression algorithm satisfies:

Theorem 4 . 3 .

 43 (Theorem 5.6 of (Azoury and Warmuth, 2001)) The forward algorithm satisfies:

1≤t≤T∥xt∥2,

  and Y f = max 1≤t≤T |yt|.

Theorem 4 . 4 .

 44 (Gaillard et al. (

Corollary 4 . 5 .

 45 (Of Theorems 11.7 and 11.8 of[START_REF] Cesa | Prediction, learning, and games[END_REF]) For all T

Figure 4 . 1 -

 41 Figure 4.1 -Online regret. y-axis is logarithmic.

Theorem 4 .

 4 12. (Theorem 11 Gaillard et al. (2019)) For all T ⩾ 1, for all sequences x 1 , . . . , x T ∈ R d and all y 1 , . . . , y T ∈ [-Y, Y ], the unregularized forward algorithm achieves the regret bound

Figure 4 . 2 -

 42 Figure 4.2 -Online regret's (Instantaneous loss difference) dependence on λ. All axes are logarithmic. Lines are averages over 100 repetitions and shaded areas represent one standard deviation.

Figure 4 . 3 -

 43 Figure 4.3 -Cumulative regret. y-axis is logarithmic.

Theorem 4 . 18 .

 418 (Theorem 3 of Russac, Vernade, and Cappé (2019)) Assuming that T -1 s=1 ∥θ * (s) -θ * (s + 1)∥ 2 ≤ B T

Figure 4 . 4 -

 44 Figure 4.4 -Performance of several algorithms in an non-stationary environments, averaged over 100 runs, shaded areas represent one standard deviation.

1

 1

Contents 5 . 1 80 5. 2 83 5. 3 84 5. 4 87 5. 5 92 5. 6 104 5. 7

 518028338448759261047 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bilinear exponential family of MDPs . . . . . . . . . . . . . . . . . . . . . . . BEF-RLSVI: algorithm design and frequentist regret bound . . . . . . . . . . Algorithm design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Regret analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Related works: functional representations of MDPs with regret and tractability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 Continuous MDPs: the Bilinear Exponential Family representation

  Q πθ,h (s, a) = r(s, a) + ϕ p (s, a), S µ p (s)V π θ,h+1 (s)ds .(5.5)

Algorithm 5 . 2 :Backtracking 1 :

 521 Bellman Input Parameters θp , θr , initialize θ = ( θr , θp ) and for all s ∈ S, V H+1 (s) = 0 2: for steps h = H -1, H -2, • • • , 0 do 3:

  a)∼π ( Qπ θ (s, a) -f ϕ (s, a) -E (s,a)∼π [ Qπ θ (s, a) -f ϕ (s, a)])( ∂f ϕ (s, a) ∂ϕ ) = 0. (6.4) Then, if we denote g ϕ = f ϕ (s, a) + E (s,a)∼π [ Qπ (s, a) -f ϕ (s, a)], and use the policy parameterization assumption: ∂f ϕ (s, a) ∂ϕ = ∂π θ (s, a) ∂θ

  )) Qπ θ (s, a)] = E (s,a)∼π θ [∇ θ log(π θ (s, a)) Qπ θ (s, a)] -E (s,a)∼π θ [( Qπ θ (s, a) -g ϕ (s, a)) ∂π θ (s, a) ∂θ 1 π θ (s, a) ] = E (s,a)∼π θ [∇ θ log(π θ (s, a))g ϕ (s, a)].

Figure 6 . 1 -

 61 Figure 6.1 -Comparison of simple models derived when L AVEC is used instead of the MSE.

Figure 6 . 2 -

 62 Figure 6.2 -Comparative evaluation (6 seeds) of AVEC with SAC and PPO on PyBullet ("TaskBullet") and MuJoCo ("Task") tasks. X-axis: number of timesteps. Y-axis: average total reward.

Figure 6 . 3 -

 63 Figure 6.3 -Comparative evaluation (6 seeds) of AVEC with SAC (left) and PPO (right) on the Walker2d MuJoCo task. Lines are average performances and shaded areas represent one standard deviation.

Figure 6 . 4 -

 64 Figure 6.4 -Comparative evaluation of AVEC with TRPO. We run with 6 different seeds: lines are average performances and shaded areas represent one standard deviation.

Figure 6 . 5 - 6 . 5 . 4

 65654 Figure 6.5 -(a,b): Comparative evaluation (6 seeds) of AVEC in sparse reward tasks. X-axis: number of timesteps. Y-axis: average total reward. (c,d): Respectively state visitation frequency and phase portrait of visited states of AVEC-TRPO (green) and TRPO (red) in MountainCar.6.5.4 Analysis of the Variance Estimated Critic

Figure 6 . 6 -

 66 Figure 6.6 -L 2 distance to V π .

Figure 6 . 7 -

 67 Figure 6.7 -L 2 distance to V π . X-axis: we run PPO and AVEC-PPO and ∀t ∈ {1, 2, 4, 6, 9} • 10 5 we stop training, use the current policy to collect 3 • 10 5 transitions and estimate V π .

  Bias(PPO) 2 + Var(PPO) where V ϕ (PPO) is the value function estimator in PPO. Let us define what we study in exact terms: %Variation(Bias) = Bias 2 (AVEC-PPO)-Bias 2 (PPO) Bias 2 (PPO) and %Variation(Var) = Var(AVEC-PPO)-Var(PPO) Var(PPO)

1 . 2 ( 0 ∥x t ∥ 2 G - 1 t≤ d log 1 +

 120211 Theorem. (Theorem 3.3 of[START_REF] Maillard | Self-normalization techniques for streaming confident regression[END_REF])) (Ordinary Least-squares) Assume that N is a stopping time adapted to the filtration of the past. Then in the sub-Gaussian streaming regression model, for anyδ > 0, with probability at least 1 -δ, ∀T ≥ 1 if |G T (0)| > 0: ∥θ * -θ T ∥ 2 G T (0) ≤ 2(1 + κ)(1 + α)σ 2 log κ d (e 2 λ max (G T )) δ where κ d (x) is function of κ and α, κ d (x) = 2 3 π 2 log(x/e) 2 log(x) 12(d + 1) √ d) d x d + d for κ = α = 1.Lemma B.2. (Technical inequality) For all sequences {xt } t ∈ R d such that ∀t, ∥x t ∥ 2 ≤ X, for all λ ∈ R + , T 0 , T ∈ N T t=T T X 2 /λ min (G T 0 )dwhere G t = G t (λ).

Corollary C. 2 . 2 α

 22 The previous theorem implies a simple euclidean confidence region. Indeed, with probability at least1 -δ, for all k ∈ N θ pθp (k) p β p (k, δ), where β p (k, δ) = ∆ β p (k-1)H (δ) = 2 2 B 2 A + log 2C p A,k /δ .Proof. The result follows from the following simple calculations:

  KL s,a θ r , θ r′ := KL (Pθ r (• | s, a), P θ r′ (• | s, a)) = B Zs,a θ r′ , θ r . Now, for any λ ∈ R d , we introduce the function B Zn,α,θ r (λ) = B Zn,α (θ r + λ, λ) and define M λ n = exp λ ⊤ S n -n t=1

1 R d exp(-η 2 ∥θ ′ ∥ 2 Λ 2 ∥θ r ∥ 2 A

 1222 )dθ ′ . We now introduce the function Z r n (θ) = n t=1 Z r st,at (θ). Note that Z r n is a also Legendre function and its associated Bregman divergence satisfiesB Z r n θ ′ , θ = n t=1 Z r st,at θ ′ -Z r st,at (θ) -θ ′ -θ ⊤ ∇Z r St,at (θ) = n t=1 B Z r s t ,α t θ ′ , θFurthermore, we have n t=1 B Z r s t ,α t ,θ r (λ) = B Z r n ,θ r (λ). From the penalized likelihood formula (5.7), recall that∀i ≤ d, n t=1 ∇ i Z r st,at θr (k) + η 2 ∇ i ∥ θr (k)∥ 2 A = k t=1 r t B ⊤ A i φ (s t , a t ) . θr (k) -∇Z r st,at (θ r ) + ηA θr (k) = ∇Z r k θr (k) -∇Z r k (θ r ) + ηA θr (k) (C.3) We now obtain from (C.2) and (C.3) that M k = c 0 • expη R d exp λ ⊤ x k -B Z k ,θ * (λ) + g k (λ) dλ, (C.4)where we introduced g k (λ) = η 2 2λ ⊤ A θr (k) + ∥θ r ∥ 2 A -∥θ r + λ∥ 2 A and x k = ∇Z r k θr (k) -∇Z r k (θ r ).

-θ r 2 A- η 2 ∥θ r + λ∥ 2 A=

 22 λ∈R ⋆ g k (λ) -g k (λ ⋆ ) + η (λ -λ ⋆ ) ⊤ A (θ r + λ ⋆ ) + η 2 ∥θ r + λ ⋆ ∥ 2 A B Z r 0 θ r , θr (k) + (λ -λ ⋆ ) ⊤ ∇Z r 0 (θ r + λ ⋆ ) + Z r 0 (θ r + λ ⋆ ) -Z r 0 (θ r + λ) (C.5)where we have introduced the Legendre function Z r 0 (θ) = η 2 ∥θ∥ 2 A . We now have from (C.9) thatsup λ∈R d λ ⊤ x n -B Z r n ,θ r (λ) = B ⋆ Z r n ,θ r (x n ) = B ⋆ Z r n ,θ r ∇Z r n θr (n) -∇Z r n (θ r ) = B Z r n θ r , θr (n) .

λ

  ⊤ x n -B Z r n ,θ r (λ) = λ ⊤ x n -B Z r n ,θ r (λ) + B Z r n θ r , θr (n) -λ ⋆ x n -B Z r n ,θ r (λ ⋆ ) = B Z r n θ r , θr (n) + (λ -λ ⋆ ) ⊤ ∇Z r n (θ r + λ ⋆ ) + B Z r n ,θ * (λ ⋆ ) -B Z r n ,θ * (λ) -(λ -λ ⋆ ) ⊤ ∇Z r n (θ r ) = B Z r n θ r , θr (n) + (λ -λ ⋆ ) ⊤ ∇Z r n (θ r + λ ⋆ ) + Z r n (θ r + λ ⋆ ) -Z r n (θ r + λ) (C.6)Plugging Equation (C.5) and Equation (C.6) in Equation (C.4), we obtainM n = c 0 • exp   j∈{0,n} B Z r j (θ r , θ j )λ ⋆ ) ⊤ ∇Z r j (θ r + λ ⋆ ) + Z r j (θ r + λ ⋆ ) -Z r j (θ r + λ) (θ r + λ ⋆ ) ⊤ ∇Z r j (θ r + λ ⋆ ) -Z r j (θ r + λ ⋆ ) (θ r + λ) ⊤ ∇Z r j (θ r + λ ⋆ ) -Z r j (θ r + λ) j∈{0,n} (θ r + λ) ⊤ ∇Z r j (θ r + λ ⋆ ) -Z r j (θ r + λ) dλ R d exp j∈{0,n} (θ ′ ) ⊤ ∇Z r j (θ r + λ ⋆ ) -Z r j (θ ′ ) dθ ′ = c 0 c n • exp B Zn θ r , θr (n) + B Z 0 θ r , θr (n) -η 2 ∥θ r ∥ 2 A ,where we introduced c n = exp j∈{0,n} ((θ r +λ * ) ⊤ ∇Z r j (θ r +λ * )-Z r j (θ r +λ * ))R d exp j∈{0,n} ((θ ′ ) ⊤ ∇Z r j (θ r +λ * )-Z r j (θ ′ )) dθ ′ .

C. 1 2 A - η 2

 122 ConcentrationsyieldsE [M N ] = E lim inf n→∞ Q n ≤ lim inf n→∞ E [Q n ] = lim inf n→∞ E M min{N,n} ≤ 1, since the stopped martingale M min{N,n} n≥1is also a martingale. Therefore, by the properties of M n , (12) also holds for any random stopping time N < ∞. To complete the proof, we now employ a random stopping time construction as in Abbasi-Yadkori et al. (2011)We define a random stopping time N by N = min n ≥ 1 : min{∅} := ∞ by convention. We then haveP ∃n ≥ 1, n t=1 KL st,at θr (n), θ r + η 2 θ rθr (n) ∥θ r ∥ 2 A ≥ log C A,n δ = P[N < ∞] ≤ δ,which concludes the proof of the first part.Proof of second part: upper bound on C A,n . First, we have for some θ ∈ θr (n), (r) × φ(s, a) ⊤ A ⊤ i BB ⊤ A j φ(s, a) θ ′ -θr (n)

2 (β r nt=1Complements on Chapter 5 DerivativesLemma C. 8 .

 258 Gs t ,a t +ηA)dθ ′ = (2π) d/2 det(ηA) 1/2 × det (β r n t=1 G st,at + ηA) 1/2 (2π) d/2 = det I + β r η -1 A -1 n t=1 G st,at ,which completes the proof of the second part. (Gradients) We provide the derivatives of the log-partitions in closed form. As usual with exponential families, these are intimately linked to moments of the random variable. We have:∇ i Z p s,a (θ) = E θ s,a ψ s ′ ⊤ A i φ(s, a).And∇ i Z r s,a (θ) = E θ s,a [r] B ⊤ A i φ(s, a).

  a (θ) = S ψ s ′ ⊤ A i φ(s, a) exp d i=1 θ i ψ (s ′ ) ⊤ A i φ(s, a) S exp d i=1 θ t ψ (s ′ ) ⊤ A i φ(s, a) ds ′ ds ′ = E θ s,a ψ s ′ ⊤ A i φ(s, a) ∇ i Z r s,a (θ) = S rB ⊤ A i φ(s, a) exp r d i=1 θ i B ⊤ A i φ(s, a) S exp r d i=1 θ i B ⊤ A i φ(s, a) dr dr = E θ s,a [r] B ⊤ A i φ(s, a)Lemma C.9. (Hessians) The entries of the Hessians of the log partition functions are given by∇ 2 i,j Z p s,a (θ) = φ(s, a) ⊤ A ⊤ i C θ s,a ψ s ′ A j φ(s, a), where C θ s,a [ψ (s ′ )] = ∆ E θ s,a ψ (s ′ ) ψ (s ′ ) ⊤ -E θ s,a [ψ (s ′ )] E θ s,a ψ (s ′ ) ⊤ . Similarly, ∇ 2 i,j Z r s,a (θ) = Var θ s,a (r) × φ(s, a) ⊤ A ⊤ i BB ⊤ A j φ(s, a),where Var θ s,a (r) = ∆ E θ s,a r 2 -E θ s,a [r] 2 is the variance of the reward under θ.

∇ 2 iC. 2 2 θ

 222 ,j Z p s,a (θ) = S ψ s ′ ⊤ A i φ(s, a)ψ s ′ ⊤ A j φ(s, a)exp d i=1 θ i ψ (s ′ ) ⊤ A i φ(s, a) S exp d i=1 θ i ψ (s ′ ) ⊤ A i φ(s, a) ds ′ ds ′ -S ψ s ′ ⊤ A i φ(s, a) exp d i=1 θ i ψ (s ′ ) ⊤ A i φ(s, a) S exp d i=1 θ i ψ (s ′ ) ⊤ A i φ(s, a) ds ′ ds ′ (∇ j Z s,a ) (θ) = E θ s,a ψ s ′ ⊤ A i φ(s, a)ψ s ′ ⊤ A j φ(s, a) Technical results -E θ s,a ψ s ′ ⊤ A i φ(s, a) E θ s,a ψ s ′ ⊤ A j φ(s, a) =φ(s, a) ⊤ A ⊤ i E θ s,a ψ s ′ ψ s ′ ⊤ -E θ s,a ψ s ′ E θ s,a ψ s ′ ⊤ A j φ(s, a) =φ(s, a) ⊤ A ⊤ i C θ s,a ψ s ′ A j φ(s, a),where we introduce in the last line the p × p covariance matrix given byC θ s,a ψ s ′ = E θ s,a ψ s ′ ψ s ′ ⊤ -E θ s,a ψ s ′ E θ s,a ψ s ′ ⊤The proof of the form of the Hessian for the reward partition function follows the same steps as above.Lemma C.10. (KL Divergences) For any two θ, θ ′ and for some pair (s, a),∃ θ ∈ θ, θ ′ ∞ , KL P p θ (• | s, a), P p θ ′ (• | s, a) = 1 2 θ -θ ′ ⊤ ∇ 2 Z p s,a ( θ) θ -θ ′ , where [θ, θ ′ ] ∞ denotes the d-dimensional hypercube joining θ to θ ′ . Similarly ∃ θ ∈ θ, θ ′ ∞ , KL (P r θ (• | s, a), P r θ ′ (• | s, a)) = 1 -θ ′ ⊤ ∇ 2 Z r s,a ( θ) θ -θ ′ .Proof. We start by writing:log P p θ (s ′ | s, a) P p θ ′ (s ′ | s, a) = d i=1 θ i -θ ′ i ψ s ′ ⊤ A i φ(s, a) -Z p s,a (θ) + Z p s,a θ ′ , then KL P p θ (• | s, a), P p θ ′ (• | s, a) = d i=1

E θ 1

 1 s,a [r] = E θ 2 s,a [r] + Var θ 3 s,a (r) 2 B ⊤ M θ 1 -θ 2 φ(s, a).

First, consider the

  notations: G s,a := (φ(s, a) ⊤ A ⊤ i A j φ(s, a)) 1≤i,j≤d , Ḡe n ≡ Ḡe (k-1)H := G n + (α e ) -1 ηA , and G n ≡ G (k-1)H := k-1 τ =1 H h=1 G s τ s ,a τ h .Where e represents either r or p, we omit the superscript e w.l.o.g in the rest of this section.Lemma C.12. (Elliptical lemma and variant for bounded potentials) Let c ∈ R + , we can bound the sum of feature norms as follows log(1 + c) d log 1 + αη -1 B φ,A n .

,Complements on Chapter 5 ≤ d log 1 +

 51 + H h=1 G s h ,a h ) det( Ḡn )where the last line follows from the matrix determinant lemma:det Ḡn + a h a ⊤ h = det(I + a ⊤ h Ḡ-1 n a h ) det( Ḡn )We can now control the R.H.S. of the above equation, asα p ) -1 ηA) = log det I + αη -1 A -1 G N α p η -1 d tr A -1 G n (Trace-determinant (or AM-GM) inequality) ≤ d log 1 + α p η -1 B φ,A nThis concludes the proof of the first result.Second result. First, we have sup s,a ∥G s,a ∥ 2 ≤ ∥A∥ 2 B φ,A .Fix an episode k ∈ [K], n = (k -1)H, using Lemma C.

2 dλ

 2 Abbreviate x = d/ log(2) and y = L 2 /dλ, which are both positive. Then

  Introduction• Reda Ouhamma, Odalric-Ambrym Maillard, and Vianney Perchet. Stochastic Online Linear Regression: the Forward Algorithm to Replace Ridge.

	Advances in Neural Information
	Processing Systems 34 (2021), pp. 24430-24441 (used in Chapter 4, Chapter 5)
	• Reda Ouhamma, Debabrota Basu, and Odalric-Ambrym Maillard. Bilinear exponen-
	tial family of mdps: Frequentist regret bound with tractable exploration and planning.
	Association for the Advancement of Artificial Intelligence (2023) (used in Chapter 5)
	• Yannis Flet-Berliac, Reda Ouhamma, Odalric-Ambrym Maillard, and Philippe Preux.
	Learning value functions in deep policy gradients using residual variance. International
	Conference on Learning Representations (2021) (used in Chapter 6)

  RL. In this paper, we consider the episodic finite-horizon MDP formulation of RL, in short Episodic RL(Osband, Russo, and Van Roy, 2013; Azar, Osband, and Munos, 2017; Dann, Lattimore, and Brunskill, 2017). Episodic RL is a tuple P = ⟨S, A, P, r, K, H⟩, where the state (resp. action) space S (resp. A) might be continuous. In episodic RL, the agent interacts with the environment in episodes consisting of H steps. Episode k starts by observing state s k 1 . Then, for t = 1, . . . H, the agent draws action a k t from a (possibly time-dependent) policy π t (s k The performance of a policy π is measured by the total expected reward V π 1 starting from a state s ∈ S, the value function and the state-action value functions at step h ∈ [H] are defined as

	t ),
	observes the reward r(s k t , a k t ) ∈ [0, 1], and transits to a state s k t+1 ∼ P(. | s k t , a k t ) according to the
	transition function P.

  , bilinear (Du, Kakade, Lee, et al., 2021), and bilinear exponential families (Chowdhury, Gopalan, and Maillard, 2021), are studied in literature to develop optimal algorithms for episodic RL with continuous state-action spaces. Since the linear assumption is restrictive in reallife -where non-linear structures are abundant-, generalized representations have obtained more attention recently (Chowdhury, Gopalan, and Maillard, 2021; Li, Li, et al., 2021; Du, Kakade, Lee, et al., 2021; Foster et al., 2021). The BEF model is of special interest as it is expressive enough to represent tabular MDPs (discrete state-action), factored MDPs

Table 5 .

 5 1 -A comparison of RL Algorithms for MDPs with functional representations.

	Algorithm	Regret	Tractable	Tractable Free of	Model, assumptions
			exploration planning clipping
	Thompson sampling	√ d 2 H 3 K	✗	✓	N.A	Gaussian P
	(Ren et al., 2021)	(Bayesian)				Known rewards
	EXP-UCRL	√ d 2 H 4 K	✗	✗	N.A	Bilinear Exp Family (BEF)
	(Chowdhury, Gopalan, and Maillard, 2021)	(Frequentist)				known rewards
	SMRL (Li, Li, et al., 2021)	√ d 2 H 4 K	✗	✗	N.A	BEF, known rewards
	UCRL-VTR (pmlr-v119-ayoub20a)	√ d 2 H 4 K	✗	✗	N.A	Linear mixture model
	F-PHE-LSVI (Ishfaq et al., 2021) PHE-LSVI (linear-RL)	poly(dEH) √ d 3 H 4 K √ KH	✓	✗	✗	Eluder dimension, Tabular Anti-concentration
	UC-MatrixRL (Yang and Wang, 2020)	√ d 2 H 5 K	✗	✗	N.A	Linear factor MDP
	OPT-RLSVI (Zanette, Brandfonbrener, et al., 2020)

1 :

 1 Input: failure rate δ, constants α p , η and (x k ) k∈[K] ∈ R + 2: for episode k = 1, 2, . . . do

	3:	Observe initial state s k 1
	4:

  θr ,h ) h∈[H] via Bellman-backtracking, see Algorithm 5.2

	7:	for h = 1, . . . , H do
	8:	Pull action a k h = arg max a Q θp , θr ,h (s k h , a)
	9:	Observe reward r(s k h , a k h ) and state s k h+1 .
	10:	end for
	11:

  4.1). Moreover, we improve the regret bound by √ H thanks to an improved analysis, (cf Lemma C.12). But similar to all RLSVI-type algorithms, we pick up an extra √ d (cf (Abeille and Lazaric, 2017)). (Zanette, Brandfonbrener, et al., 2020) proposes a variant of RLSVI for continuous stateaction spaces, where there are low-rank models of transitions and rewards. They show a regret bound R(K) = O( √ d 4 H 5 K), which is larger than that of BEF-RLSVI by O( √ dH 2 ). In algorithm design, we improve on their work by removing the need to carefully clip the value function. Analytically, our model allows us to use transportation inequalities (cf Lemma C.7) instead of the simulation lemma, which saves us a √ H factor. This notion of regret is weaker than frequentist regret, hence this result is not directly comparable with Theorem 5.5. Tightness of regret bound. A lower bound for episodic RL with continuous state-action spaces is still missing. However, for tabular RL, (Domingues et al., 2021) proves a lower bound of order Ω( √ H 3 SAK). If we represent a tabular MDP in our model, we would need d = S 2 × A parameters (Section 4.3, (Chowdhury,

[START_REF] Ren | A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning[END_REF]

considers Gaussian transitions, i.e. s ′ = f * (s, a) + ε such that ε ∼ N 0, σ 2 . This is a particular case of our model. They propose to use Thompson Sampling, and have the merit of being the first to have observed linearity of the value function from this transition structure. But they do not connect it to the finite dimensional approximation of

[START_REF] Rahimi | Random features for large-scale kernel machines[END_REF]

unlike us (Section 5.4). Finally, they show a Bayesian regret bound of O( √ d 2 H 3 K).

  al., 2018; Ilyas et al., 2020) indicate that state-of-the-art policy gradient methods (Schulman, Levine, et al., 2015; Schulman, Wolski, et al., 2017) fail to fit the true value function and that recently proposed state-action-dependent baselines (Gu et al., 2016; Liu et al., 2018; Wu et al., 2018) do not reduce gradient variance more than state-dependent ones.

policy gradient: improved learning of value functions Algorithm 6.1:

  that f ϕ old (s t ) are the estimates given by the last value function and A t is the advantage of the policy, i.e. the returns minus the expected values (A t is often estimated using generalized advantage estimation[START_REF] Schulman | Highdimensional continuous control using generalized advantage estimation[END_REF]. AVEC for PPO or TRPO. J ALGO denotes the policy loss of either algorithm (described in[START_REF] Schulman | Proximal policy optimization algorithms[END_REF][START_REF] Schulman | Trust Region Policy Optimization[END_REF]). 1: Input parameters: λ π ≥ 0, λ V ≥ 0 2: Initialize policy parameter θ and value function parameter ϕ 3: for each update step do

	Deep 4:	batch B ← ∅
	5:	for each environment step do
	6:	a t ∼ π θ (s t )
	7:	s t+1 ∼ P (s t , a t )
	8:	B ← B ∪ {(s t , a t , r t , s t+1 )}
	9:	end for
	10:

for each iteration do 4: for each step do 5:

  a t ∼ π θ (a t |s t )

	6:	s t+1 ∼ P (. | s t , a t )
	7:	D ← D ∪ {(s t , a t , r t , s t+1 )}
	8:	end for
	9:	for each gradient step do
	10:	sample batch B from D
	11:	Update

where Qπ (s, a) is estimated using temporal difference (see

[START_REF] Haarnoja | Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF]

): Qπ (s t , a t ) = r(s t , a t ) + γE s t+1 ∼π [V ψ(s t+1 )] with ψ the value function parameter (see Algorithm 6.2). Algorithm 6.2: AVEC coupled with SAC.

1: Input parameters: β ∈ [0, 1], λ V ≥ 0, λ Q ≥ 0, λ π ≥ 0 2: Initialize policy parameter θ, value function parameter ψ and ψ and Q-functions parameters ϕ 1 and ϕ 2 and D ← ∅ 3:

Deep policy gradient: improved learning of value functions

  . Note that the

	Task	SAC	AVEC-SAC	PPO	AVEC-PPO
	Ant	3084	3650 ± 127 (+18%)	972 1202 ± 148 (+24%)
	AntBullet	1193	2252 ± 82 (+89%)	1174 2216 ± 99 (+89%)
	HalfCheetah	10028 11018 ± 102 (+10%) 1068 1403 ± 37 (+31%)
	HalfCheetahBullet 1255	1331 ± 184 (+6%)	1329 2223 ± 62 (+67%)
	Humanoid	4084	4472 ± 424 (+10%)	391	415 ± 4.6 (+6%)
	Reacher	-6.0	-5.0 ± 0.1 (+20%)	-7.4 -5.9 ± 0.3 (+25%)
	Walker2d	3452	4334 ± 128 (+26%) 2193 2923 ± 151 (+33%)

Table 6 .

 6 1 -Average total reward of the last 100 episodes over 6 runs of 10 6 timesteps. Comparative evaluation of AVEC with SAC and PPO. ± corresponds to a single standard deviation over trials and (.%) is the change in performance due to AVEC.PyBullet versions of the locomotion tasks are harder than the MuJoCo equivalents3 . We choose a representative set of tasks for the experimental evaluation; their action and observation space dimensions are reported in Appendix D.3. We assess the benefits of AVEC when coupled with the most prominent policy gradient algorithms, currently state-of-the-art methods: PPO[START_REF] Schulman | Proximal policy optimization algorithms[END_REF] and TRPO[START_REF] Schulman | Trust Region Policy Optimization[END_REF], both on-policy methods, and SAC[START_REF] Haarnoja | Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF], an off-policy maximum entropy deep RL algorithm.

Table 6 .

 6 2 -Average total reward of the last 100 episodes over 6 runs of 10 6 timesteps. Comparative evaluation of AVEC with TRPO. ± corresponds to a single standard deviation over trials and (.%) is the change in performance due to AVEC.

  Average cosine similarity between gradient measurements. AVEC empirically reduces the variance compared to PPO or PPO without a baseline (PPO-nobaseline). Trajectory size used in estimation of the gradient variance: 3000 (upper row), 6000 (middle row), 9000 (lower row). Lines are average performances and shaded areas represent one standard deviation.

											6.6 Discussion
				HopperBullet						Walker2DBullet
		0.14			AVEC-PPO PPO			0.22			AVEC-PPO PPO
	Average pairwise cos sim	0.10 0.11 0.12 0.13			PPO-nobaseline	Average pairwise cos sim	0.10 0.12 0.14 0.16 0.18 0.20			PPO-nobaseline
		0.09						0.08
		0.0	0.2	0.4 Timesteps 0.6	0.8	1.0 1e6		0.0	0.2	0.4 Timesteps 0.6	0.8	1.0 1e6
				HopperBullet						Walker2DBullet
		0.16			AVEC-PPO			0.20			AVEC-PPO
					PPO						PPO
	Average pairwise cos sim	0.08 0.10 0.12 0.14			PPO-nobaseline	Average pairwise cos sim	0.14 0.16 0.18			PPO-nobaseline
		0.06						0.12
		0.0	0.2	0.4 Timesteps 0.6	0.8	1.0 1e6		0.0	0.2	0.4 Timesteps 0.6	0.8	1.0 1e6
				HopperBullet						Walker2DBullet
		0.16			AVEC-PPO			0.16			AVEC-PPO
	Average pairwise cos sim	0.08 0.10 0.12 0.14			PPO PPO-nobaseline	Average pairwise cos sim	0.11 0.12 0.13 0.14 0.15			PPO PPO-nobaseline
		0.06						0.10
		0.0	0.2	0.4 Timesteps 0.6	0.8	1.0 1e6		0.0	0.2	0.4 Timesteps 0.6	0.8	1.0 1e6
	Figure 6.11 -

  λ min (G T 0 )d . Lemma B.3. (Technical inequality, Ridge regression) For all sequences {x t } t ∈ R d such that ∀t, ∥x t ∥ 2 ≤ X, for all λ ∈ R + , T ∈ N We use the Weinstein-Aronszajn identity: ∥x t ∥ 2

	T t=1	||x t || 2 G -1 t-1	≤	X 2 /λ log(1 + X 2 /λ)	d log 1 + T X 2 /λd
	Proof. G -1 t-1	= |Gt| |G t-1

|

-1, which leads to:

  This next part is what differs from Lemma B.2, using ||x t || 2

	G -1 t-1

  Suppose {F t } ∞ t=0 is a filtration such that for each t, (i) r(s t , a t ) is F t -measurable, (ii) (s t , a t ) is F t-1 measurable, and (iii) given (s t , a t ) , r(s t , a t ) ∼ P r θ r (• | s t , a t ) according to the exponential family defined by (5.2). Let θr (k) be the penalized MLE defined by Equation (5.7), and let Z r s,a (θ) be strictly convex in θ for all (s, a). Then, for any δ ∈ (0, 1], with probability at least 1 -δ, the following holds uniformly over all k ∈ N := R d expη 2 ∥θ ′ ∥ 2 A dθ ′ / R d expk t=1 KL st,at (θ k , θ ′ )-η 2 ∥θ ′ -θ k ∥ 2 A dθ ′ . Define G s,a = ∆ φ(s, a) ⊤ A ⊤ i A j φ(s, a)

												C.1 Concentrations
	C.1.2 Concentration of the reward parameter (contribution)
	Theorem C.3. k t=1	KL st,at	θr (k), θ r +	η 2	θ r -θr (k)	2 A	-	η 2	∥θ r ∥ 2 A ≤ log	C r A,k δ	,
	where C r A,k
												2
												G s τ h	,a τ h
							2 A	+	k-1 τ =1	H h=1	KL s τ h ,a τ

≤ (α p ) -1 η 2 θ pθp (k) h (θ k , θ) . i,j∈[d]

B

  Zn t ,a t ,θ r (λ)where ∀i ≤ d, we denote (S n ) i = n t=1 r (s t , a t ) -E θ r st,at [r] B ⊤ A i φ (s t , a t ). Note that M λ n > 0 and it is F n -measurable. Furthermore, we have for all (s, a), = exp -λ ⊤ ∇Z r s,a (θ r )

	E θ r s,a exp

d i=1 λ i r (s t , a t ) -E θ r st,at [r] B ⊤ A i φ (s t ,

a t ) Complements on Chapter 5 S exp d i=1

  R d expn t=1 KL st,at θr (n), θ ′ -η 2 θ ′ -θr (n)

					2 A ′ dθ ′
	Therefore, we have from (5) that
	C A,n :=	c n c 0	=	R d exp -η 2 ∥θ ′ ∥ 2 A dθ ′	2	dθ ′
					A
	An application of Markov's inequality now yields
	n
	P
	t=1

R d expn t=1 B Zs t ,a t θ ′ , θr (n) -η 2 θ ′ -θr (n)

  the last line is because G s,a = a h a ⊤ h , where a h = (A i φ(s h , a h )) i∈[d] . First result. Consider h ∈ [H], denote (λ h,i )i ∈ [d] the eigenvalues of a ⊤ Ḡn is positive definite hence λ h,i > 0, ∀h, i, then

	T t=1	H h=1	Ḡ-1/2 n	G s,a	Ḡ-1/2 n	≤ 2d log 1 + αη -1 B φ,A n +	3dH log(2)	log 1 +	α∥A∥ 2 2 B 2 φ,A η log(2)
	Proof. First we have
		∥ Ḡ-1/2 n	G s,a	Ḡ-1/2 n	∥ = ⊤ (	Ḡ-1/2 n	G s,a	Ḡ-1/2 n	Ḡ-1/2 n	G s,a	Ḡ-1/2 n	)
										≤ ⊤ ( Ḡ-1/2 n	G s,a	Ḡ-1/2 n	) = ⊤ ( Ḡ-1 n G s,a ) = ⊤ (a ⊤ h	Ḡ-1 n a h )
			H								H	d
		min{c,		Tr(a ⊤ h	Ḡ-1 n a h )} = min{c,	λ h,i }
			h=1							h=1	i=1
										≤	c log(1 + c)

h Ḡ-1 n a h .

Elliptical potentials: finite number of large feature norms (contribution) Lemma C.14

  [START_REF] Amato | Decentralized control of partially observable Markov decision processes[END_REF], we know that the number of timesh ∈ [h] such that Ḡ-1 n G s h ,a h ≥ 1 is smaller than 3d log(2) log 1 + α(∥A∥ 2 B φ,A ) 2 η log(2). Let us callT k := {h ∈ [H] Ḡ-1 (k-1)h G s h ,a h ≤ 1}, thenthe sum of the right hand side is similar to the first result. Although the sum is not contiguous, the previous bound holds since if h 1 < h 2 , det( Ḡn+h 1 ) ≤ det( Ḡn+h 2 ), this concludes the proof. We can also write from the lemma in terms of ∥(A i φ(s h , π(s h ))) 1≤i≤d ∥ ( Ḡr k ) -1 by skipping the norm upper bound at the beginning of the proof: . (Worst case elliptical potentials, adaptation of Exercise 19.3[START_REF] Lattimore | Bandit algorithms[END_REF] for matrices) Let V 0 = λI and a 1 , . . . , a n ∈ R d×p be a sequence of matrices with ∥a t ∥ 2 ≤ L for all t ∈ [n]. Let V t = V 0 + t s=1 a s a ⊤ s , then {t ∈ N * , ∥a t ∥ V -1

	T t=1	H h=1	Ḡ-1 n G s t h ,a t h	≤	3d log(2)	log 1 +	α∥A∥ 2 2 B 2 φ,A η log(2)	+	h∈T k	min{1, Ḡ-1 n G s t h ,a t h	}
	T t=1	min{c,	H h=1	∥(A i φ(s h , π(s h ))) 1≤i≤d ∥ ( Ḡr k ) -1 } ≤	c log(1 + c)	d log 1 + αη -1 B φ,A n .
	and
			T	H
			t=1	h=1	∥(A +	3dH log(2)	log 1 +	α∥A∥ 2 2 B 2 φ,A η log(2)
									t-1	≥ 1} ≤	3d log(2)	log 1 +	L 2 λ log(2)

Remark C.13. i φ(s h , π(s

h ))) 1≤i≤d ∥ ( Ḡr k ) -1 ≤2d log 1 + αη -1 B φ,A n

This chapter is based on a collaboration with Rémy Degenne, Pierre Gaillard, and Vianney Perchet(Ouhamma, Degenne, et al.,

2021) It was accepted for publication as a spotlight at the 32nd conference on advances in Neural Information Processing Systems (NeurIPS).

See Remark 1[START_REF] Tao | Thresholding bandit with optimal aggregate regret[END_REF]. This bound implies that LSA -with the specified choice of α = 0.1 needs 16000 times more samples than the oracle to achieve the same performances.

This chapter is based on a collaboration with Odalric Maillard and Vianney Perchet(Ouhamma, Maillard, and Perchet,

2021). It was accepted for publication at the 32nd conference on advances in Neural Information Processing Systems (NeurIPS).

This chapter is based on a collaboration with Debabrota Basu and Odalric Maillard(Ouhamma, Basu, and Maillard,

2023). It was accepted for publication as an oral at the 37th AAAI Conference on Artificial Intelligence.

By tractable planning, we mean having a planner with (pseudo-)polynomial complexity in the problem parameters, i.e. in the dimension of features, the horizon, and the number of episodes.

This chapter is based on a collaboration with Yannis Flet-Berliac, Odalric Maillard and Philippe Preux(Flet- Berliac, Ouhamma, et al.,

2021). It was accepted for publication at the The International Conference on Learning Representations (ICLR).

[START_REF] Greensmith | Variance reduction techniques for gradient estimates in reinforcement learning[END_REF] analyzes the dependent case: in general, weakly dependent variables tend to concentrate more than independent ones.

Bullet Physics SDK GitHub Issue.

Arabic proverbs collected and translated to english by @ArabicWords. The original wording of the verse is:

List of Acronyms

C CMDP

Constrained Markov Decision Process

Continuous MDPs: the Bilinear Exponential Family representation

Given the normal CDF Φ, we obtain that with probability at least Φ(-1)

Choosing x k ≥ H β p β p (k,δ) α p α r + β r β r (k,δ) min{1, α p α r } 2α r and using Lemma C. 6, we find that the perturbed value function is optimistic with probability at least Φ(-1).

Controlling the learning error

In this section we see the core difference with optimistic algorithms. On the one hand, optimistic approaches require the value function generating the agent's policy to be larger than the optimal one with large probability, and can therefore ensure that the learning error is negative. On the other hand, BEF-RLSVI only ensures that the value function is optimistic with a constant probability: intuitively when this event holds the learning happens, and if it does not then the policy is still close to a good one thanks to the decreasing estimation error.

Upper bound on V ⋆

1 . Let us draw ( ξk ) k∈ [K] i.i.d copies of (ξ k ) k∈ [K] . Define the optimism event at episode k: Ōk = {V θp , θr + ξk ,1 (s k 1 ) -V ⋆ 1 (s k 1 ) ≥ 0} (5. [START_REF] Al | Adaptive sampling for best policy identification in markov decision processes[END_REF] we know that P( Ōk ) ≥ Φ(-1). This event provides the upper bound:

Lower bound on V θp , θr . We define this bound with an optimization problem under concentration of the noise. Consider V 1 (s k 1 ) is the solution of

Under the concentration of our injected noise, we obtain V 1 (s k 1 ) ≤ V θp , θr (s k 1 ) (5.16)

Chapter 6

Deep policy gradient: improved learning of value functions

Policy gradient algorithms have proven to be successful in diverse decision making and control tasks. However, these methods suffer from high sample complexity and instability issues. In this paper, we address these challenges by providing a different approach for training the critic in the actor-critic framework. Our work builds on recent studies indicating that traditional actor-critic algorithms do not succeed in fitting the true value function, calling for the need to identify a better objective for the critic. In our method, the critic uses a new state-value (resp. state-action-value) function approximation that learns the value of the states (resp. state-action pairs) relative to their mean value rather than the absolute value as in conventional actor-critic. We prove the theoretical consistency of the new gradient estimator and observe dramatic empirical improvement across a variety of continuous control tasks and algorithms. Furthermore, we validate our method in tasks with sparse rewards, where we provide experimental evidence and theoretical insights. 1 the error between the Q-function estimator and the true Q-function for SAC and AVEC-SAC in AntBullet and HalfCheetahBullet. We note a modest but consistent reduction in this error when using AVEC coupled with SAC, echoing the significant performance gains in Figure 6.2. We conclude that AVEC improves the value function approximation and we expect that the gradient is more stable.

Empirical Variance Reduction.

We choose to study the gradient variance using the average pairwise cosine similarity metric as it allows a comparison with [START_REF] Ilyas | A Closer Look at Deep Policy Gradients[END_REF], with which we share the same experimental setup and scales. Figure 6.10 shows that AVEC yields a higher average pairwise cosine similarity, which means closer batch-estimates of the gradient and, in turn, indicates smaller gradient variance.

In Figure 6.11, we study the empirical variance of the gradient in measuring the average pairwise cosine similarity (10 gradient measurements) in two additional tasks: HopperBullet and Walker2DBullet. We also vary the trajectory size used in the estimation of the gradient. The variance reduction effect observed in several environments suggests that AVEC is the first method since the introduction of the value function baseline to further reduce the variance of the gradient and improve performance.

Ablation Study

In this section, we examine how changing the relative importance of the bias and the residual variance in the loss of the value network affects learning. For this study, we choose difficult tasks of PyBullet and use PPO because it is more efficient than TRPO and requires less computations than SAC. For an estimator ŷn of (y i ) i∈{1,...,n} , we write Bias = 2 . Consequently: MSE = Var + Bias 2 . We denote L α = Var + αBias 2 , with α ∈ R. In Figure 6.12, Bias-α means that we use L α and Var-α means that we use L 1 α . We observe that while no consistent order on the choices of α is identified, AVEC seems to outperform all other weightings. Note that, for readability purposes, the graphs have been split and the curves of AVEC-PPO and PPO are the same in Figures 6.12a and 6.12c, and in Figures 6.12b and 6.12d. A more extensive hyper-parameter study with more α values might provide even higher performances, nevertheless we believe that the stability of an algorithm is crucial for a reliable performance. As such, the tuning of hyperparameters to achieve good results should remain mild.

Discussion

In this work, we introduce a new training objective for the critic in actor-critic algorithms to better approximate the true value function. In addition to being well-motivated by recent studies on the behaviour of deep policy gradient algorithms, we demonstrate that this modification is both theoretically sound and intuitively supported by the need to improve the approximation error of the critic. The application of Actor with Variance Estimated Critic (AVEC) to state-ofthe-art policy gradient methods produces considerable gains in performance (on average +26% for SAC and +39% for PPO) over the standard actor-critic training, without any additional hyperparameter tuning. of implementation since no further assumptions are required (such as horizon awareness [START_REF] Tucker | The Mirage of Action-Dependent Baselines in Reinforcement Learning[END_REF] to remedy the deficiency of existing variance-reduction methods) and the modification of current algorithms represents only a few lines of code.

Deep policy gradient: improved learning of value functions

In this paper, we have demonstrated the benefits of a more thorough analysis of the critic objective in policy gradient methods. Despite our strongly favorable results, we do not claim that the residual variance is the optimal loss for the state-value or the state-action-value functions, and we note that the design of comparably superior estimators for critics in deep policy gradient methods merits further study. In future work, further analysis of the bias-variance trade-off and extension of the results to stochastic environments is anticipated; we consider the problem of noise separation in the latter, as this is the first obstacle to accessing the variance and distinguishing extreme values from outliers.

A.1 Properties of index-based algorithms

An algorithm is index-based if, at any round t, it pulls k t = arg min k I k N k,t-1 where the index I k N k,t depends only on the number of pulls and on rewards of arm k. That index does not change when other arms are pulled.

≥ C} be the event that at some time before T , all arm indices are above a value C. And let τ k (C) = min{n|I k n ≥ C} be the minimal number of pulls of arm k such that its index becomes greater than C.

We start with two immediate remarks about index-based algorithms.

Lemma A.1. If I j N j,t ≥ C, then at the next time t ′ when an index-based algorithm pulls arm j, it necessarily holds that

This next lemma explicits

Proof. We first prove the inclusion

Hence there is equality and we have

If there is some j such that N j,T > τ j (C), then after the time at which arm j was pulled τ j (C) times it verified I j N j,t ≥ C. Arm j is again pulled at least once at some time t ′ , and at that time we have by Lemma A.1 that for all k,

Then for all arms except at most one,

Proof. The algorithm switches arm only if the index of the pulled arm becomes strictly greater than the minimal index of the others. As a consequence, the value of the minimal index at times of arm changes is increasing. If two or more arms are pulled since t max , there is an arm change later than t max and the minimal index value at that time is higher than at t max . This is a contradiction.

A.2 Concentration lemmas

A.2 Concentration lemmas

First, we state a simple but useful lemma.

. For all δ ∈ (0, 1) such that ∆ δ n,k ≥ 0, Hoeffding's inequality implies that P( ∆n,k < ∆ δ n,k ) ≤ δ.

The next lemma will be used to bound the sum of exponentially tailed distributions.

Lemma A.6 ((Janson, 2018)). Let Z 1 , . . . , Z K be independent random variables and a 1 , . . . , a K ∈ R + be such that for all k ∈ [K] and x ∈ R + , P(Z k ≥ x) ≤ e -a k x . Then for all λ ≥ 0,

Corollary A.7. Let Y 1 , . . . , Y K be independent random variables and y 1 , . . . ,

The corollary is a direct application of Lemma A. 6 to

Appendix

Complements on Chapter 4

Proof. The proof of this result follows the same line in the proof of Theorem 1 of Abbasi-Yadkori, Pál, and Szepesvári (2011), first we define for λ ∈ R d , t > 0 :

Without loss of generality, we can assume that (ε s ) s≥1 is 1-sub-Gaussian (this can be achieved by scaling features). Let τ be a stopping time with respect to the filtration {F t } ∞ t=0 . Then M λ τ is well-defined almost surely and

Let Λ ∼ N(0, σ ′2 ) be a Gaussian random variable and define

By expliciting M t and using Markov's inequality we get that for δ > 0, with probability 1 -δ:

Next we use a stopping time construction from Freedman (1975): Define the bad event:

We are interested in bounding the probability that t>0 B t (δ) happens. Define τ (ω) = min{t ≥ 0 : ω ∈ B t (δ)} , with the convention that min ∅ = ∞. Then, τ is a stopping time. Further, t≥0 B t (δ) = {ω : τ (ω) < ∞} Thus, by Equation B.1:

Theorem B.5. (Confidence ellipsoid for the Forward algorithm) For any δ > 0, with probability at least 1 -δ, for allt > 0:

Complements on Chapter 4

respect to F t and for any δ > 0, σ ′ > 0, with probability at least 1 -δ, for all t ≥ 0:

Proof. The proof of this result proceeds in the exact same way as for Lemma B.4.

B.2 Experimental details and instructions:

The experiments were run on a personal laptop with Intel Core i7-8665U, CPU 1.90GHz × 8. Code for the experiments for online regression and linear bandits can be provided upon request to the authors. For the experiments of non-stationary linear bandits that we present next, we used an existing code from the Github page of Russac, Vernade, and Cappé (2019) and we added an implementation of D-LinUCB f to compare with previous algorithms.

Appendix

C.1.3 Gaussian concentration and anti-concentration

where Φ is the normal CDF.

Thanks to lower bounds on the error function, we have the following bound on the probability of anti-concentration Φ(-1) ≥ 1/(4 √ eπ).

C.2 Technical results

C.2.1 A transportation lemma

For any function f : X → R, we define its span as S(f

For a probability distribution P supported on the set X , let

2 denote the mean and variance of the random variable f (X), respectively. We now state the following transportation inequalities, which can be adapted from (Boucheron, Lugosi, and Massart, 2013) (Lemma 4.18).

Lemma C.7. (Transportation inequalities)

Assume f is such that S(f ) and V P [f ] are finite. Then it holds

C.2.2 Bregman divergence

For a Legendre function F : R d → R, the Bregman divergence between θ ′ , θ ∈ R d associated with F is defined as

. Now, for any fixed θ ∈ R d , we introduce the function

It then follows that B F,θ is a convex function, and we define its dual as

We have for any θ, θ ′ ∈ R d :

To see this, we observe that

Now an optimal λ must satisfy ∇F (θ) = ∇F (θ ′ + λ). One possible choice is λ = θ -θ ′ . Since, by definition, F is strictly convex, the supremum will indeed be attained at λ = θ -θ ′ . Plugin-in this value, we obtain

Note that (C.9) holds for any convex function F . Only difference is that, in this case, B F (•, •) will not correspond to the Bregman divergence.

C.2.3 Properties of the bilinear exponential family

In this section, we detail some useful results related to exponential families in our model.

Appendix

Complements on Chapter 6 D.1 Experiment Details

In all experiments we choose to use the same hyper-parameter values for all tasks as the bestperforming ones reported in the literature or in their respective open source implementation documentation. In Tables D.1, D.2 and D.3, we report the list of hyper-parameters for all continuous control tasks. Overall, we choose the hyper-parameters in a way to ensure the best performance for the conventional actor-critic framework. In other words, since we are interested in evaluating the impact of this new critic, everything else is kept as is. This experimental protocol may not benefit AVEC.

D.2 Details about the environments

Environment Description

Ant-v2 Make a four-legged creature walk forward as fast as possible.

AntBulletEnv-v0

Idem. Ant is heavier, encouraging it to typically have two or more legs on the ground (source: PyBullet Guide -url).

HalfCheetah-v2

Make a 2D cheetah robot run.

HalfCheetahBulletEnv-v0

Idem.

Humanoid-v2

Make a three-dimensional bipedal robot walk forward as fast as possible, without falling over.

Reacher-v2

Make a 2D robot reach to a randomly located target.

Walker2d-v2

Make a 2D robot walk forward as fast as possible.

Acrobot-v1

Swing the end of a two-joint acrobot up to a given height.

MountainCar-v0

Get an under powered car to the top of a hill.

Complements on Chapter 6 D.3 Dimensions of Studied Tasks

Table D.5 -Actions and observations dimensions.

List of Figures

[right] for K = 50 arms. . . . . . . . . . . .

List of Figures

[left]

Median (and 0.25, 0.75 empirical quantiles obtained on 500 runs) of the ratio between the error suffered by each algorithm and that of the optimal nonadaptive oracle (µ k = (-1) k , k = 1, . . . , 100).

[right] Ratio of the averaged errors (over 500 runs) of each algorithm with that of the oracle (µ k = (-1) k (k/K)

List of Algorithms

List of Tables

List of References

List of References

[12] Shipra Agrawal and Randy Jia. Optimistic posterior sampling for reinforcement learning: worst-case regret bounds. Advances in Neural Information Processing Systems 30 (2017).

List of References

List of References

List of References

List of References

List of References

List of References

Copyright 2019-2023, © Reda Ouhamma.