Shipra Agrawal

Csaba Szepesvári

Keywords: Thompson Sampling machine learning, decision-making, algorithm, thompson sampling, multi-armed bandit, linear bandit, linear quadratic control, reinforcement learning apprentissage automatique, processus décisionnel, algorithme, thompson sampling, bandit multi-bras, bandit linéaire, contrôle linéaire quadratique, apprentissage par renforcement

This dissertation is dedicated to the study of the Thompson Sampling (TS) algorithms designed to address the exploration-exploitation dilemma that is inherent in sequential decision-making under uncertainty. As opposed to algorithms based on the optimism-in-the-face-of-uncertainty (OFU) principle, where the exploration is performed by selecting the most favorable model within the set of plausible one, TS algorithms rely on randomization to enhance the exploration, and thus are much more computationally efficient. We focus on linearly parametrized problems that allow for continuous state-action spaces, namely the Linear Bandit (LB) problems and the Linear Quadratic (LQ) control problems. We derive two novel analyses for the regret of TS algorithms in those settings. While the obtained regret bound for LB is similar to previous results, the proof sheds new light on the functioning of TS, and allows us to extend the analysis to LQ problems. As a result, we prove the first regret bound for TS in LQ, and show that the frequentist regret is of order O( √ T ) which matches the existing guarantee for the regret of OFU algorithms in LQ. Finally, we propose an application of exploration-exploitation techniques to the practical problem of portfolio construction, and discuss the need for active exploration in this setting.

Introduction

This dissertation is dedicated to the study of Thompson Sampling algorithms, designed to address the exploration-exploitation dilemma that is inherent in sequential decisionmaking under uncertainty. Sequential decision-making is a complex and general process that consists in selecting a sequence of actions in order to achieve a task. Consider for instance the parking problem, where a driver (the agent) wants to park: each time the driver sees an available spot, it balances between parking immediately or waiting for another spot closer to its destination, running the risk of finding none and turning back. When the agent does not have the complete knowledge of its environment (e.g., the number of available parking spots), the agent has to deal with the uncertainty about the outcomes of its actions and adapts its decisions to its observations i.e., the empirical knowledge accumulated during the decision process. Therefore, the learning agent faces a dilemma between exploring (go further see whether there is any available spot) and exploiting (there was few available spots so far, it seems unlikely to find one further, park now). How to build intelligent agents that properly address this trade-off is one of the major goals of Artificial Intelligence, and is at the heart of Reinforcement Learning (RL). In RL, each time the agent selects an action, it receives a reward generated in an unknown fashion, and aims at cumulating as much reward as possible. While a large number of methods have been proposed so far to deal with the exploration-exploitation trade-off, two main principles emerged as the most effective and flexible solution to this problem: the optimism-in-the-face-uncertainty (OFU) and the Thompson sampling (TS). The former consists in selecting action that seems the more rewarding in the most favorable environment that is coherent with observations so far, while the latter introduces randomness in the way actions are chosen to enhance the exploration of the unknown environment with respect to the objective. The OFU principle has been intensively investigated in the last decades and led to the celebrated family of Upper Confidence Bound (UCB) algorithms, for which theoretical guarantees have been provided. TS is an old principle based upon Bayesian ideas, that has recently attracted lot of consideration because of its impressive empirical performances. However, theoretical guarantees for TS algorithms are still limited. Despite the success of RL in numerous domains, one of its main limitations is that the action and/or the state spaces are usually assumed to be finite (and of small cardinality) whereas real-world applications often deal with infinite (or large) state-action spaces. In order to overcome this limitation, a natural approach is to consider parametrized systems with continuous state-action spaces. However, this makes the control and the learning problems significantly harder. To this end, linearly parametrized systems are of major interest as they remain tractable but reflect the key difficulties in designing and studying exploration-exploitation algorithms for continuous state-action spaces settings. Additionally, they consist in robust, yet flexible models and thus are widely used in practice.

In this thesis, we analyse TS algorithms for the Linear Bandit (LB) problem and the Linear Quadratic (LQ) control problem. LB is a natural extension of the celebrated Multi-Armed Bandit (MAB) framework, where an agent has to sequentially select actions from an infinite set, at each round, and receives a reward that is randomly generated independently from the previous rounds, according to a linear model. Formally, the reward function is a noisy linear mapping between the action and an unknown parameter. One of the main limitations of the LB setting is that it cannot model problems where the environment is affected by the actions chosen by the agent over time. For such systems, one has to consider the environment's dynamics that makes both the learning and the control problems harder. LQ is a specific instance of Markov Decision Processes (MDP) with infinite state-action spaces, that assumes the dynamics to be linear in the state (which characterizes the system) and the action (chosen by the agent), and where the reward function is quadratic. LQ is a standard in control theory and offer the advantage of having explicit solutions for the optimal policy (i.e., the mapping from observations to actions) when the dynamics and the reward model are known.

Our approach

The original motivation of this thesis lies in addressing the dynamical portfolio allocation problem, where the dynamics of the market exhibit both return predictability i.e, the fact that traders have "views" on the future prices' movements, and price impact i.e., the fact that the trading orders affect the prices in an adverse way. When the volume of the transactions is large compared to the available liquidity, the execution of such orders drastically changes the supply and demand, and thus the prices. Moreover, empirical studies suggest that this effect is dynamical, i.e., that the market has memory of past trades and digest them slowly. While the objective is not to construct a complete and functional trading robot, we consider this use-case as a tool that helps us highlighting the practical issues that arise when constructing autonomous agent, and address them theoretically.

1.1. Our approach 3

In the concrete example of portfolio allocation, the challenges are numerous:

1. The dynamical nature of both the return predictability and the price impact effects requires one to use dynamic programming techniques in order to compute the optimal allocation. Moreover, the problem is intrinsically characterized by a large or a continuous state-action space: at short timescales (e.g., high-frequency trading), the prices and the traded volumes are discrete with fine-mesh that induces a very large state-action space; for longer horizons and larger volumes (e.g., asset management), the state-action space becomes so huge that both are usually modeled as continuous variables.

2. By nature, the dynamics of the financial market are unknown and have to be estimated. Moreover, to observe the price impact effects, the learner has to trade in a so-called bandit feedback scenario since it does not have access to the other participants' actions, and only observes the outcomes of its own decisions. This implies that one has to consider the online learning setting, where the task (i.e., the trading process) and the estimation are performed simultaneously.

3. Finally, for practical applications, it is of crucial importance that the models and the methods be robust (e.g., that two similar problems have similar solutions and performances) and tractable (i.e., computationally efficient).

We address the first point by considering parametrized systems, which can handle continuous state-action spaces. Nevertheless, this usually makes the control problem hard or impossible to solve. We overcome this issue showing that one can cast the portfolio optimization problem into a Linear Quadratic control problem for which closedform solutions have been derived when the parameters of the dynamics are known. This reformulation can be performed for any linear Markovian market model and thus encompasses numbers of econometric models used in the industry. Then, we consider online learning procedures for generic LQ systems that would allow us to perform the control and the estimation jointly. To tackle the exploration-exploitation trade-off, algorithms have been derived from the two popular principles that are optimism-in-theface-of-uncertainty and Thompson sampling. While the optimistic instance has been recently proved to suffer a low regret (i.e., have good performance), the computational cost of this strategy is prohibitive in this setting. Indeed, it requires at each policy re-evaluation to solve a non-convex, high-dimensional, optimization problem. On the other hand, TS stands as a good candidate to maintain tractability since it just requires a random sampling at each time step. Unfortunately, limited theoretical guarantee has been provided for this strategy which motivates our theoretical analysis for TS in LQ. However, the existing analysis for TS sampling in the simpler Linear Bandit setting (that does not take into account the dynamical effects) cannot be extended to LQ, mostly because of the structure of the proof. In particular, the analysis requires that the performance associated with any action under any parametrization concentrates as soon as the parameter concentrates around the true one. Unfortunately, while in LB the performance is well defined for every actions under any parametrization, it is no longer Chapter 1. Introduction the case in LQ where this performance can diverge to -∞ and thus the concentration property does not hold. To this end, we derive an alternative analysis for TS in LB that does not rely on this property and thus that can be extended to LQ. Moreover, it provides some intuitions about the functioning of the TS algorithm, that we leverage together with the existing proof for the optimistic algorithm in LQ to guarantee the first regret bound for TS in LQ.

Outline and contributions

In this section, we give an outline of the thesis and summarize the contributions.

Chapter 2: Exploration-Exploitation Dilemma in Sequential Decision-Making

The objective of this chapter is to introduce the concept of exploration-exploitation, present the state-of-the-art and the material that we need for the analyses of Ch. 3 and 4. We first introduce the exploration-exploitation dilemma in the well-known Multi-Armed Bandit setting, explaining the challenges that the learner faces and presenting the two most popular principles to address this issue, based respectively on optimism-in-faceof-uncertainty and Thompson sampling. Then, we present two extensions, namely the Linear Bandit problem and the Linear Quadratic control problem, that address the main limitations of MAB, i.e., the finite action space and the independency between rounds. We present algorithms based on the two principles in those frameworks and recall the available regret guarantees.

Chapter 3: Thompson Sampling in Linear Bandit

We derive an alternative proof for the regret of Thompson sampling (TS) in the stochastic linear bandit setting, where the reward is linear in the chosen arm (the selected action) according to an unknown parameter. While we obtain a regret bound of order O(d 3/2 √ T ) as in previous results, the proof sheds new light on the functioning of the TS. We leverage the structure of the problem to show how the regret is related to the sensitivity (i.e., the gradient) of the objective function and how selecting optimal arms associated to optimistic parameters does control it. Thus, we show that TS can be seen as a generic randomized algorithm where the sampling distribution is designed to have a fixed probability of being optimistic, at the cost of an additional √ d regret factor compared to a UCB-like approach. Furthermore, we show that our proof can be readily applied to regularized linear optimization and generalized linear model problems for which we prove the first O( √ T ) regret bound for the TS algorithm.

Chapter 4: Thompson Sampling in Linear Quadratic System

We consider the exploration-exploitation trade-off in Linear Quadratic control problems, where the state dynamics is linear and the cost function is quadratic in the state 1.2. Outline and contributions 5 and control. We analyze the regret of TS (a.k.a. posterior-sampling for reinforcement learning) in the frequentist setting, i.e., when the parameters characterizing the LQ dynamics are fixed. Despite the empirical and theoretical success in a wide range of problems from MAB to LB, extending those results to the LQ setting is highly challenging: 1) the standard line of proof that relies on classifying arms into saturated/unsaturated pool cannot be applied here as their associated optimal value could be infinite; 2) the TS functioning requires frequent policy updates, which is in contrast with the usual lazy update scheme used in most RL algorithm. As a consequence, it raises the issue of bounding the gap in the optimal value at the policy switches. We prove that TS achieves a O( √ T ) regret, thus matching the performance of the OFU approach and confirming the conjecture of [START_REF] Osband | Posterior sampling for reinforcement learning without episodes[END_REF]. We address the first point leveraging the ideas introduced in Ch. 3, stressing the link between the actual actions chosen by TS and the gradient of the optimal value function. We exhibit the need to trade-off the frequency of sampling optimistic parameters and the frequency of switches in the control policy, and show that lazy update schemes induce at best an overall regret of O(T 2/3 ). Finally, we derive a novel bound on the regret due to policy switches, thus allowing to update parameters and the policy at each step and overcome the limitations due to lazy updates.

Chapter 5: Application to Portfolio Construction

We propose an application of exploration-exploitation strategies for the concrete example of optimal portfolio allocation under price impact. We introduce a novel LQ formulation for the portfolio allocation problem, under the assumption of linear price dynamics, from which we obtain the optimal control and discuss the explorationexploitation trade-off arising from the presence of unknown parameters. We consider two problem instances with or without risk constraint and show that this affects the need for exploration. In the unconstrained case, a greedy strategy fails to achieve sub-linear regret, while Thompson Sampling or optimism-based algorithms effectively trade-off exploration and exploitation. On the other hand, the risk constraint modifies the structure of the policy, removing somehow the need for active exploration, and a greedy strategy is optimal. We discuss this counter-intuitive result and support it with numerical experiments.

Chapter 6: Summary and Future Work

We summarize the dissertation and discuss some directions for future research.

Chapter 2

Exploration-Exploitation Dilemma in Sequential Decision-Making

The objective of this chapter is to introduce the exploration-exploitation trade-off which is at the core of our work and is a standard problem in decision-making. We present the state-of-the-art methods and results and the material that we need for the analyses of Ch. 3 and 4. We first introduce the exploration-exploitation dilemma in the well-known Multi-Armed Bandit (MAB) setting, explaining the challenges that the learner faces and presenting the two most popular principles to address this issue, based respectively on optimism-in-face-of-uncertainty and randomness. Then, we present two extensions, namely the Linear Bandit (LB) problem and the Linear Quadratic (LQ) control problem, that address the main limitations of MAB, i.e., the finite action space and the independency between rounds. LB is a natural extension of MAB with continuous action space, where the reward depends linearly on an unknown parameter and the chosen arm. Despite the fact that the action set is allowed to change with time (in the contextual setting), it cannot be affected by the actions chosen by the learner.

On the other hand, Markov Decision Processes allow the system to be dynamically affected by the actions, and thus overcome this limitation. However, it usually assumes the state-action space to be finite. To this end, LQ stands as a powerful framework to extend the MDP model to continuous state-action spaces, while imposing the dynamic of the system to be linearly parametrized. We present algorithms based on the two principles in those frameworks and recall the available regret guarantees.

The multi-armed bandit problem

The Multi-Armed Bandit (MAB) problem is a sequential decision-making problem where an agent chooses at each time step an action to play and it receives a reward drawn from an unknown distribution. The aim of the agent is to maximize the cumulative Chapter 2. Exploration-Exploitation Dilemma in Sequential Decision-Making reward, i.e., the global payoff received from the chosen sequence of actions. The name Multi-Armed Bandit comes from the problem instance where a gambler (agent) faces multiple slot machines (one-armed bandits) with different unknown reward distributions and has to sequentially decide which machine (arm) to play. MAB problems have been originally introduced to study the problem of clinical trials (which treatment to use on patients suffering from the same disease) and is now used in a large number of applications (see e.g., [START_REF] Bubeck | Regret analysis of stochastic and nonstochastic multiarmed bandit problems[END_REF] such as allocation in finance, web-advertisement, routing etc... From a theoretical perspective, its popularity is due to the fact that it offers a simple framework to study the exploration-exploitation dilemma that is inherent of sequential decision making in unknown environment, where the agent balances between selecting highly rewarding actions based on the knowledge acquired so far, and playing poorly estimated actions (with potential low reward) to enhance his knowledge and select better actions in the future. This allocation problem had been extensively studied in statistics and became a standard in Reinforcement Learning (RL). In this section, we present the MAB setting and the challenges induced by the exploration-exploitation trade-off, the main principles used to tackle this issue and the algorithms that have been derived from those principles together with their theoretical guarantees.

Setting and challenges

We consider the stochastic MAB setting (see e.g., [START_REF] Bubeck | Regret analysis of stochastic and nonstochastic multiarmed bandit problems[END_REF][START_REF]Sample mean based index policies with o(log n) regret for the multi-armed bandit problem[END_REF], Auer et al. 2002a) where an agent selects at each time step t = 1, 2, . . . one arm (action) I t from a finite set of K arms i.e., I t ∈ {1, . . . , K}. Each arm i is associated with a distribution ν i of mean µ i so that when the agent plays arm I t , it receives a reward X It randomly generated (independently from the past) according to ν It .

Setting. The objective of the agent is to select a sequence of arms (I 1 , I 2 , . . . ) to maximize the associated cumulative reward. Denoting the optimal arm and optimal average reward as i ∈ argmax i=1,...,K µ i and µ = max i=1,...,K µ i , the equivalent objective is to minimize the regret of the strategy i.e., the difference between the optimal reward that would have been collected playing arm i at each time step and the reward actually collected. Formally, we consider the expected sudo-regret defined as

R n = nµ -E n t=1 µ It ,
where n is the total number of rounds. The expectation is taken w.r.t. any randomization in the choice of the sequence of arms. Finally, introducing T i (t) = t s=1 1{I s = i} the number of times the agent selected arm i up to time t, and ∆ i = µ -µ i the sub-optimality gap between arm i and i , it is possible to re-write the pseudo-regret as

R n = µ K i=1 E T i (n) -E K i=1 T i (n)µ i = K i=1 ∆ i E T i (n) .
(2.1)

The agent does not know about the reward distributions {ν i } i=1,...,K , and hence does not know about the average rewards {µ i } i=1,...,K , but collects knowledge about their value by observing the sequence of reward (X I 1 , X I 2 , . . . ) generated by its own sequence of actions (I 1 , I 2 , . . . ). As a result, the agent faces an explorationexploitation trade-off, where it balances between playing the most rewarding arm given its current knowledge to minimize its instantaneous regret, and playing badly estimated arms to collect information that would help it to improve its future performance.

Lower bound. Before introducing the main strategies used to tackle the explorationexploitation trade-off in the MAB problem, we highlight the inherent difficulty of this sequential decision making problem, by recalling the existing lower-bounds. In the parametric case where ν i = ν i (θ i ) are function of a unknown parameter θ = (θ 1 , . . . , θ K ) ∈ Θ, under mild assumptions on the set of parameter Θ, one has: Theorem 2.1.1 [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF]). For any adaptive strategy whose regret satisfies, for each θ ∈ Θ, the condition that as n → ∞, R n (θ) = o(n a ) for every a > 0,

(2.2) one has that for any θ ∈ Θ,

lim inf n→∞ R n (θ) log(n) ≥ i =i ∆ i KL(ν i ||ν i ) , (2.3)
where KL(ν i ||ν j ) = ∞ -∞ ν i (x) log ν i (x) ν j (x) dx is the Kullback-Leibler divergence.

Amongst the set of consistent strategies i.e., allocation rules satisfying Eq. 2.2, Thm. 2.1.1 guarantees that the regret scales at best as Ω(log n): from Eq. 2.1, it implies that an optimal strategy must select sub-optimal arms at most log n times. Additionally, Thm. 2.1.1 provides us with the optimal problem dependent constant that is function of the unknown distributions {ν i } i=1,...,K . To gain intuition about this result, consider the problem instance where rewards are generated according to Bernoulli distributions with parameters (µ 1 , . . . , µ K ) ∈ [0, 1] K pairwise disjoint (µ i = µ j for any i = j) and suppose that µ = 1/2 for sake of simplicity. Then, for all i = i , the Kullback-Leibler divergence can be lower and upper bounded as

2 ∆ i 2 ≤ KL(ν i ||ν i ) ≤ 4 ∆ i 2 ,
and thus the constant in Eq. 2.3 scales as i =i 1 ∆ i . Finally, a non-asymptotic minimax lower bound (see e.g., [START_REF] Cesa-Bianchi | Prediction, learning, and games[END_REF], i.e., a problem-independent regret bound is given by the following theorem: Theorem 2.1.2. Let sup ν denote the supremum over all distribution of rewards (i.e., over the MAB problem instances), and let inf algo denote the infimum over any adaptive strategy. Then, inf

algo sup ν R n = Ω √ nK .
(2.4)

Chapter 2. Exploration-Exploitation Dilemma in Sequential Decision-Making

Heuristics

In this subsection, we introduce and illustrate the two main principles that address the exploration-exploitation in MAB in particular and in RL in general, namely the optimism in face of uncertainty that leads to the celebrated Upper Confidence Bound (UCB) algorithms, and the Thompson Sampling that relies on randomization to enhance the exploration.

To highlight the difficulty of the problem, we first describe the behavior of a naive strategy that is known to suffer linear regret, where the choice of I t is made greedily given the current estimates of {ν i } i=1,...,K . First, notice that at time step t ≤ n, given the sequence of arms (I 1 , . . . , I t-1 ) selected so far, the learner observed the sequence of associated rewards (X I 1 ,1 , . . . , X I t-1 ,t-1 ) where X i,s is randomly drawn from ν i at time step s independently from the past. Thus, for each arm i, it can compute the empirical estimate µ t i of the mean µ i as

µ t i = 1 T i (t -1)
t-1 s=1 X i,s 1{i = I s }, for all i = 1, . . . , K.

(2.5)

The greedy strategy then consists in selecting the arm as

I t = argmax i=1,...,K µ t i .
As claimed above, this strategy is known to fail, as the algorithm can be stuck in a configuration where it keeps on selecting a sub-optimal arm. Consider for instance the 2-armed case where i = 1 and suppose that, at some time step t, due to the randomness of the rewards, µ 1,t < µ 2,t . Then, the algorithm will pick arm 2, improving the accuracy of µ 2,t but leaving µ 1,t unchanged. Thus, it may persistently pull arm 2, which unfortunately is the sub-optimal arm. We illustrate this configuration in Fig. 2.1. This sub-optimal behavior comes from the fact that the accuracy of the estimation is only improved over chosen arms, which speaks in favor of adjusting the score of arms w.r.t. the number of pull, or alternatively of adding some random perturbation to force the exploration over badly estimated arms. Leveraging this intuition, optimistic algorithms no longer select the arm with the maximum empirical average but the one with the maximum adjusted empirical average as I t = argmax i=1,...,K

µ t i + B i,t ,
where B i,t is the exploration bonus that quantifies how often arm i has been pulled up to time t. Since the agent still wants to take advantage of the accumulated knowledge, B i,t is designed to be an upper-bound for µ i i.e., such that µ i ∈ [ µ t i -B i,t , µ t i + B i,t ] with high probability, which explains the Upper Confidence Bound (UCB) name for this class of algorithms. As a result, such method overcomes the difficulty faced by the greedy strategy, as illustrated in Fig. 2.2, where we consider the same initial configuration as the one of Fig. 2.1. due to the lack of accuracy of the estimation (the black segments represent the h.p. confidence intervals). Here, µ t 2 ≥ µ t 1 so the greedy strategy selects the sub-optimal arm 2. Center: The agent observes a reward for arm 2 randomly generated according to ν 2 . Thus, µ 2 is re-evaluated and its accuracy improves while µ 1 are left unchanged. As a result, arm 2 is selected once again. Right: The greedy strategy keeps on selecting arm 2, improving the accuracy of its estimate, but is never able to discriminate that µ 1 ≥ µ 2 , and thus is persistently sub-optimal. (1,2) with a bonus B (1,2),t which is designed so that the score corresponds to the upper bound of the confidence interval (black segment). In this configuration, arm 2 is selected. Center: A reward is generated for arm 2, and its empirical mean and upper bound are re-evaluated while the one of arm 1 are left unchanged. As a result, the score of arm 1 is significantly boosted by the bonus B 1,t+1 compared to arm 2, and despite the fact that µ t+1 12 Chapter 2. Exploration-Exploitation Dilemma in Sequential Decision-Making Finally, another principle, named Thompson Sampling (TS), is built on Bayesian ideas where one assumes a prior over the average reward µ i 's and maintains this distribution as new data are collected. Then, a prediction µ i is sample for each arm i according to the posterior and the arm is selected as

I t = argmax i=1,...,K µ t i .
The original idea dates back from [START_REF] Thompson | On the likelihood that one unknown probability exceeds another in view of the evidence of two samples[END_REF] and has attracted considerable attention recently because of the impressive empirical performance of TS (see e.g., Chapelle and Li 2011). Furthermore, despite the Bayesian construction of this method, TS has also been shown to perform well in the frequentist setting, where parameters of the reward distribution of every arms are fixed, though unknown, implying that TS can be seen as a randomized algorithm and that the prior assumption only acts as a convenient tool to derive the sampling distribution. The underlying idea is that the posterior plays the role of the high-probability confidence interval of UCB i.e., w.h.p.

µ i ∈ [ µ t i -B i,t , µ t i + B i,t ]
, so TS can be seen as a randomized counterpart of UCB and hence avoids getting trapped in bad decisions. This point of view is somehow in contrast with the original Bayesian construction of TS and is at the core of our new analysis for TS in LB (see Ch. 3).

Algorithms and theoretical guarantees

We now present the main algorithms that had been derived from those heuristics and provide their associated theoretical guarantees. We first present the seminal UCB1 algorithm of Auer et al. (2002a) and its extensions UCB-V and KL-UCB introduced respectively by [START_REF] Audibert | Tuning bandit algorithms in stochastic environments[END_REF] and [START_REF] Cappé | Kullback-leibler upper confidence bounds for optimal sequential allocation[END_REF], [START_REF] Maillard | A finite-time analysis of multi-armed bandits problems with kullback-leibler divergences[END_REF]. Then, we present two instances of TS algorithms with Bernoulli and arbitrary reward distributions respectively. UCB algorithms. The UCB algorithm of Auer et al. (2002a) is based on the index

µ t i + 2 log t T i (t -1)
for each arm i = 1, . . . , K, where 2 log t T i (t-1) plays the role of the exploration bonus B i,t . This is motivated by the Chernoff-Hoeffding's inequality which ensures that

P µ i,t + 2 log t T i (t -1) ≤ µ i ≤ 1 t 4 , for all i = 1, . . . , K.
As a consequence, each index is an upper bound of the true means with high-probability. We report the formal algorithm in Fig. 2.3.

The multi-armed bandit problem 13

Initialization: Play each arm once, store the rewards in X = (X 1,1 , . . . , X K,1 ) and set T i = 1 for all i = 1, . . . , K 1: for t = {1, . . . , n} do 2:

Compute µ t i according to Eq. 2.5 for all i = 1, . . . , K 3:

Play arm I t = argmax i µ t i + 2 log t T i 4:
Receive reward X It,t

5:

Update T It = T It + 1 and X = (X, X It,t ) 6: end for Figure 2.3 -UCB1 algorithm for the MAB problem with K arms. Auer et al. (2002a) guarantee the following expected regret bound for the UCB1 algorithm:

Theorem 2.1.3. For all K > 1, if policy UCB1 is run on the MAB problem with K arms, with arbitrary reward distributions ν 1 , . . . , ν K with support in [0, 1], its expected regret over n steps is at most

R n ≤ 8 log(n) i =i 1 ∆ i + 1 + π 2 /3 i =i ∆ i .
The derivation of UCB1 relies on the Chernoff-Hoeffding's inequality which does not take into account the higher moments of the distributions but the empirical average only. To refine this result, [START_REF] Audibert | Tuning bandit algorithms in stochastic environments[END_REF] introduced a variant named UCB-V which takes into account the empirical variance and is based on a more advanced concentration inequality, namely an empirical version of the Bernstein's inequality instead of Hoeffding's. Formally, the arm selection is made as

I t = argmax i=1,...,K   µ t i + 2V t i log(t) T i (t -1) + 3 log(t) T i (t -1)   ,
where V t i is the empirical variance of arm i at time t. They show that the regret of UCB-V is bounded as

R n ≤ 10 log(n) i =i σ 2 i ∆ i + 2 ,
where σ 2 i is the variance of the distribution of arm i. This is a major improvement over Thm. 2.1.3 since it reflects the fact that an arm with low variance is easy to estimate and thus does not contribute much in the regret. Refining further this result, [START_REF] Cappé | Kullback-leibler upper confidence bounds for optimal sequential allocation[END_REF] and [START_REF] Maillard | A finite-time analysis of multi-armed bandits problems with kullback-leibler divergences[END_REF] introduced the KL-UCB algorithm using the Kullback-Leibler divergence kl(p, q) of two Bernoulli distributions with parameter p and q. Formally, one has:

kl(p, q) = p log p q + (1 -p) log 1 -p 1 -q , 14
Chapter 2. Exploration-Exploitation Dilemma in Sequential Decision-Making and the arms are selected according to the index

u i (t) = max q q ∈ µ t i , 1 : kl( µ t i , q) ≤ log t T i (t -1)
and I t = argmax i=1,...,K u i (t). They show that the regret suffered by KL-UCB is bounded as

R n ≤ log(n) i =i ∆ i kl(µ i , µ ) + O log n ,
which implies that KL-UCB is asymptotically optimal as it matches the lower bound in Thm. 2.1.1.

TS algorithms.

In order to provide intuition about the functioning of the Thompson Sampling algorithm, we first introduce it for the Bernoulli bandit problem of Chapelle and Li (2011) i.e., we assume that the reward distributions ν i 's are Bernoulli with unknown parameters µ i 's. In this setting, the beta distribution stands as a convenient tool, since this distribution is a conjugate prior w.r.t. the Bernoulli distribution. Formally, we denote as Beta(α, β) the beta distribution with parameters α > 0, β > 0, whose pdf is given by

f (x; α, β) = Γ(α + β) Γ(α)Γ(β) x α-1 (1 -x) β-1 .
When the prior over the reward distribution is Beta(α, β), after observing a Bernoulli trial, its posterior is simply Beta(α + 1, β) if the trial is a success, or Beta(α, β + 1) if the trial is a failure. Agrawal and Goyal (2012a) propose a specific instance of TS, where it is initially assumed that arm i has a prior Beta(1, 1) over µ i (which corresponds to the uniform distribution over [0, 1]), and maintain this distribution as Beta(S i (t) + 1, F i (t) + 1) where S i (t) and F i (t) are respectively the number of success (X i = 1) and failure (X i = 0) of arm i observed so far. Then, the algorithm randomly samples from these posteriors and plays the arm with the largest sample mean. The algorithm in summarized in Fig. 2.4.

Initialization:

For each arm i = 1, . . . , K, set S i = 0 and

F i = 0 1: for t = {1, . . . , n} do 2: For each arm i = 1, . . . , K, sample µ i (t) ∼ Beta(S i + 1, F i + 1) 3:
Play arm When the reward distributions are not Bernoulli, Agrawal and Goyal (2012a) propose an extension of this algorithm based on Bernoulli trials: after observing a reward X i,t ∈ [0, 1] that is generated from an arbitrary distribution, the algorithm performs a Bernoulli trial with probability of success X i,t , and updates the quantities S i (t) and Initialization: For each arm i = 1, . . . , K, set S i = 0 and F i = 0 1: for t = {1, . . . , n} do 2:

I t = argmax i µ i (t) 4: Receive reward X It,t 5: If X It,t = 1, update S It = S It + 1, else update F It = F It + 1 6: end for Figure 2.4 -
For each arm i = 1, . . . , K, sample µ i (t) from Beta(S i + 1, F i + 1) 3:

Play arm I t = argmax i µ i (t) and receive reward X It,t

4:

Perform a Bernoulli trial with success probability X It,t and observe output Xt F i (t) depending on whether this Bernoulli trial is a success or a failure. We summarize the algorithm in Fig. 2.5. Agrawal and Goyal (2012a) proved the following guarantee:

R n ≤ O   i =i 1 ∆ 2 i 2 log n   .
The bound in Thm. 2.1.4 is optimal w.r.t. the dependency on n, but sub-optimal w.r.t. the dependency on ∆ i in the constant. To overcome this, [START_REF] Kaufmann | Thompson sampling: An asymptotically optimal finite-time analysis[END_REF] derive a Bayes-UCB algorithm for Bernoulli rewards, which uses both the idea of UCB and TS, that achieves the lower bound of [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF]. Further, [START_REF] Korda | Thompson sampling for 1-dimensional exponential family bandits[END_REF] study an instance of TS wih Jeffreys prior, and show that the induced algorithm has a regret bounded by

O i =i ∆ i KL(ν i ||ν i )
log n , thus is asymptotically optimal, when the reward distributions ν i 's belong to a 1-dimensional canonical exponential family. Finally, [START_REF] Agrawal | Further optimal regret bounds for thompson sampling[END_REF] prove a O( nK log(n)) problem-independent regret bound for TS in MAB, that is near-optimal w.r.t. the problem-independent lower bound in Eq. 2.4, up to a factor log(n).

Extensions

The ability of the MAB problem to encode in a simple framework the hardness of the exploration-exploitation dilemma explains its popularity, and numerous variants and extensions have been studied over the last decade. For instance, while we focused in this section on the stochastic MAB problem, its adversarial counterpart have been studied by [START_REF] Bubeck | Regret analysis of stochastic and nonstochastic multiarmed bandit problems[END_REF] and [START_REF] Auer | The nonstochastic multiarmed bandit problem[END_REF], where the rewards are no longer assumed to be stochastic but chosen by an oblivious adversary. On the other hand, TS has also been studied in the Bayesian setting, where the parameters of the reward distributions are assumed to be generated from a true, yet unknown prior, and the regret is measured in average w.r.t. this prior [START_REF] Bubeck | Prior-free and prior-dependent regret bounds for thompson sampling[END_REF]. Further, enriching the structure of the rewards, [START_REF] May | Optimistic bayesian sampling in contextual-bandit problems[END_REF] considered the contextual bandit problem, where the reward is function of the action I t and a context x t . [START_REF] Russo | Time-sensitive bandit learning and satisficing thompson sampling[END_REF] address the discounted regret to take into account time preference when

Chapter 2. Exploration-Exploitation Dilemma in Sequential

Decision-Making the optimal action is costly to learn compared to near-optimal actions and propose a variation of the TS algorithm in this setting. Finally, despite existing results for the MAB problem with more arms than the possible number of rounds i.e., K n [START_REF] Wang | Algorithms for infinitely many-armed bandits[END_REF], the main limitation of this framework is that the number of arms is finite, thus the action space cannot be continuous. This motivates one of the major extension, the Linear Bandit (LB) problem, where the action set is embedded in R d and the reward is a noisy linear combination between the action and an unknown parameter. We present the LB problem in the next section.

The linear bandit problem

Despite the richness of the MAB framework and the flexibility of the model that allows very different reward distributions, one of its major limitation is its inability to model problems where the number of arms is infinite e.g., when the arm set is embedded in R d : without any further assumption on the problem, the lower bound of Eq. 2.3 clearly states that the regret would grow infinite (both because of infinite elements in the sum and because ∆ → 0). To address this issue, [START_REF] Dani | Stochastic linear optimization under bandit feedback[END_REF] formalized the Linear Bandit (LB) extension, that was introduced by Auer et al. (2002a), where the arms are vector of R d and the payoff is a noisy and unknown linear function of the arm. Given this additional structure, they derived an optimistic algorithm that relies on least square estimation but shares the same structure as UCB. Further, [START_REF] Li | A contextual-bandit approach to personalized news article recommendation[END_REF] propose the LinUCB algorithm that uses tighter confidence bounds and has been proved later by Abbasi-Yadkori et al. (2011b). To do so, they introduce a new concentration inequality for the least square estimates which allows them to improve the regret bound of [START_REF] Dani | Stochastic linear optimization under bandit feedback[END_REF] (see Abbasi-Yadkori et al. 2011a) since, as hinted in Sec. 2.1, the performance of optimistic algorithms are determined by the tightness of the confidence bounds. Similarly, a TS algorithm can be derived for the LB problem which has been shown to offer very good empirical performance. Agrawal and Goyal (2012b) provided the first regret analysis for the LB problem, and showed that up to a factor √ d, the TS algorithm offers the same performance as the UCB-like algorithm. We present here the LB setting and the least square concentration inequality, and provide the two algorithms together with their theoretical regret bounds.

Setting

We consider the stochastic linear bandit extension of [START_REF] Dani | Stochastic linear optimization under bandit feedback[END_REF]. Let X ⊂ R d be an arbitrary (finite or infinite) bounded set of arms. When an arm x ∈ X is pulled, a reward is generated as

r(x) = x T θ + ξ,
where θ ∈ R d is a fixed but unknown parameter and ξ is a zero-mean noise. An arm x ∈ X is evaluated according to its expected reward x T θ and for any θ ∈ R d we denote the optimal arm and its value by

x (θ) = arg max x∈X x T θ, J(θ) = sup x∈X x T θ.
Then x = x (θ ) is the optimal arm for θ and J(θ ) is its optimal value. At each step t, the learner selects an arm x t ∈ X based on the past observations (and possibly additional randomization), it observes the reward r t+1 = x T t θ + ξ t+1 , and it suffers a regret equal to the difference in expected reward between the optimal arm x and the arm x t . All the information observed up to time t is encoded in the filtration

F t = (F 1 , σ(x 1 , r 2 , . . . , r t , x t ))
, where F 1 contains any prior knowledge. The objective of the learner is to minimize the cumulative regret up to step T , i.e.,

R(T ) = T t=1 x ,T θ -x T t θ .

RLS estimation

The stochastic LB problem is characterized by bandit feedback, in the sense that the learner only observes the rewards without any additional information about the components of θ . However, an estimate θ t can be computed at each time step using the standard least square procedure. Formally, let (x 1 , . . . , x t ) ∈ X t be a sequence of arms chosen so far and (r 2 , . . . , r t+1 ) be the corresponding rewards, then θ can be estimated by regularized least-squares (RLS). For any regularization parameter λ ∈ R + , the design matrix and the RLS estimate are defined as

V t = λI + t-1 s=1 x s x T s , θ t = V -1 t t-1 s=1
x s r s+1 .

One of the many advantages of the RLS is that it offers strong theoretical guarantees.

β t (δ) = R 2 log (λ + tX 2 ) d/2 λ -d/2 δ + √ λS.
where X and S are constants such that x ≤ X for all x ∈ X and θ ≤ S.

Chapter 2. Exploration-Exploitation Dilemma in Sequential Decision-Making

Algorithms and theoretical guarantees

As for the MAB framework, two type of algorithms can be derived from the optimistic and random principles presented in Sec. 2.1. We first present the Optimism in Face of Uncertainty for Linear bandit (OFUL) algorithm of Abbasi-Yadkori et al. (2011a) which is a refined version of the ConfidenceBall algorithm of [START_REF] Dani | Stochastic linear optimization under bandit feedback[END_REF], and then present the TS algorithm for LB of Agrawal and Goyal (2012b) together with their respective theoretical guarantees.

OFUL algorithm.

According to the concentration inequality for RLS estimates, one can defined a confidence ellipsoid E RLS t at each time step as

E RLS t = θ ∈ R d | θ -θ t Vt ≤ β t (δ) ,
which is centered around θ t with orientation defined by V t and radius β t (δ). Thus, with high probability θ ∈ E RLS t so that E RLS t plays the role of the confidence bound of the UCB algorithm. Formally, the OFUL algorithm selects at each time step the optimistic

parameter θ t = argmax θ∈E RLS t J(θ),
and then chooses the optimal arm w.r.t. θ t as x t = x ( θ t ). We summarize the algorithm in Fig. 2.6.

Initialization: Set θ 1 = 0 and V 1 = λI 1: for t = {1, . . . , n} do 2:
Define the confidence ellipsoid

E RLS t = θ ∈ R d | θ -θ t Vt ≤ β t (δ) 3:
Select the optimal parameter θ t = argmax θ∈E RLS t

J(θ)

4:

Play the arm

x t = x ( θ t ) = argmax x∈X x T θ t 5:
Observe reward r t+1 = x T t θ + ξ t+1 6:

Update the RLS estimate θ t+1 and design matrix V t+1 7: end for Theorem 2.2.1. For any 0 < δ < 1, with probability at least 1 -δ, the regret of the OFUL algorithm in Fig. 2.6 is bounded as

R(T ) = O d log(T ) √ T + dT log(T /δ) .
Therefore, apart for logarithmic factors, the OFUL algorithm is optimal.

TS algorithm. To design the TS algorithm, Agrawal and Goyal (2012b) use a Gaussian prior for the unknown parameter θ . The motivation is that whenever the reward noise ξ t is conditionally Gaussian, the linear model ensures that the posterior is also Gaussian. Notice that none of those assumptions are required to be true but that they only provide a useful tool to obtain the sampling distribution. Formally, at each time step, a parameter θ t is randomly sampled according to N ( θ t , vV -1 t ), where v is a parameter, and the optimal arm w.r.t. θ t is chosen i.e., x t = x ( θ t ). The TS algorithm is summarized on Fig. 2.7.

Initialization: Set θ 1 = 0 and V 1 = λI 1: for t = {1, . . . , n} do 2:
Sample the parameter as θ t ∼ N ( θ t , vV -1 t )

3:

Play the arm

x t = x ( θ t ) = argmax x∈X x T θ t 4:
Observe reward r t+1 = x T t θ + ξ t+1 5:

Update the RLS estimate θ t+1 and design matrix V t+1 6: end for Notice that in line with MAB, the sampling of the TS algorithm for LB is made so that θ t spans the confidence ellipsoid E RLS t . The variance V -1 t is rescaled by the tuning parameter v, which is here to ensure that the sampling covers the whole ellipsoid with sufficient probability. In practice v = R 9d log(T /δ) if T is known, or replaced by v t = R 9d log(t/δ) at time t if the horizon T is unknown. Leveraging the proof structure of TS for MAB [START_REF] Agrawal | Further optimal regret bounds for thompson sampling[END_REF], Agrawal and Goyal (2012b) proved the following guarantee: Theorem 2.2.2. For any 0 < δ < 1, with probability at least 1 -δ, the regret of the TS algorithm in Fig 2 .7 is bounded as

R(T ) = O d 3/2 log(T ) √ T + d dT log(T /δ) .
The bound in Thm. 2.2.2 is √ d worst than the bound of OFUL. Despite the very good empirical performance of TS, whether this bound is tight or not is an open question. On the other hand, notice that the TS algorithm is computationally more efficient than OFUL, as the latter requires solving a bilinear optimization problem (i.e., arg max θ max x x T θ) whereas the computational complexity of the former is dominated by the sampling and the computation of the optimal action (i.e., arg max x x T θ).

Extensions

We presented in this section the LB problem with fixed arm set X . However, both Abbasi-Yadkori et al. (2011a) and Agrawal and Goyal (2012b) show that their results are still valid in the so-called contextual setting, where the arm set is allowed to vary with time,

Chapter 2. Exploration-Exploitation Dilemma in Sequential

Decision-Making i.e., replacing at each time step X by X t . While this setting is richer, it is limited to the case where the changes in the arm set are independent of the learner's actions and cannot handle the case where the learner's decisions affect the environment (and thus X t ). To do so, one has to consider a more complicated setting, such as Markov Decision Processes (MDP), that we present in the next section. On the other hand, efforts have been made to relax the linear assumption on the reward model which led to the Generalized Linear Model extension of LB that is widely used in practice (see [START_REF] Filippi | Parametric bandits: The generalized linear case[END_REF][START_REF] Li | Provable optimal algorithms for generalized linear contextual bandits[END_REF][START_REF] Jun | Scalable generalized linear bandits: Online computation and hashing[END_REF]. Finally, [START_REF] Lattimore | The end of optimism? an asymptotic analysis of finite-armed linear bandits[END_REF] recently analyzed the asymptotic problem-dependent regret in LB and showed that no algorithm based on TS or optimism can achieve the lower bound in the finite arm setting.

Markov decision processes and linear quadratic control

In this section, we consider the more challenging setting where the agent's actions influence the environment's dynamics, which permits to overcome one of the limitations of the contextual bandit framework. To leverage the existing result of dynamic control, we restrict to the case where the environment's dynamics are Markovian, i.e., we focus on Markov Decision Processes (MDP) [START_REF] Sutton | Introduction to Reinforcement Learning[END_REF]. This is motivated by the fact that it allows one to use Bellman operators to compute the optimal policy (see [START_REF] Bertsekas | Dynamic programming and optimal control[END_REF] for an introduction), i.e., the mapping from observations to actions which maximizes the total reward. We illustrate the agent/environment interactions in Fig. 2.8. At each time step t, the agent decides which action a t to take, based on his current knowledge, encoded in the state s t which characterizes the environment, according to a policy π. For each state-action pair, the environment returns a reward r t = R(s t , a t ) and the system evolves to the next state s t+1 according to a transition model P which is function of the current state and action (since we restrict to Markovian dynamics). Finally, the agent's objective is to find a policy π that maximizes the cumulative reward. When the transition and reward models are known, it is possible to compute the optimal policy using standard technics via the resolution of the well-known associated Bellman equation. For instance in the discrete case with finite state-action pairs, one can use Policy Iteration (PI) and Value Iteration (VI). On the other hand, when the transition and/or reward models are unknown, the agent faces an exploration-exploitation trade-off since it selects actions both to maximize the rewards and to get knowledge about the transition and/or reward model. In this section, we first provide a brief overview of the standard finite state and action space MDP's for which algorithms have been derived using the optimistic and random principles. However, as for MAB, the finite state-action space property is a major limitation, that can be overcome by looking at parametrized MDPs with continuous state-action space. To this end, we present the Linear Quadratic (LQ) control problem, where the transition model is parametrized according to a linear model, which stands as a standard in control theory. One of the main advantages of LQ is that the Bellman equation can be solved efficiently, and that the unknown dynamic can be estimated via least square. We will present the setting and the optimistic algorithm introduced by Abbasi-Yadkori and Szepesvári (2011).

Environment r t = R(s t , a t ) s t+1 ∼ P (•; s t , a t ) Agent a t = π(s t ) (r t , s t+1 ) a t

Markov decision process

Setting. A MDP is defined as a tuple M = (S, A, P, R) where S and A are respectively the state and action space that are assumed to be finite with cardinality S and A, P is the transition model that defines the underlying Markov chain modeling the environment dynamic and R is the reward function. Formally, R : S × A → R and P : S × S × A → S such that for any (s, s ) ∈ S 2 , for any a ∈ A, P (s , s, a) = P(s t+1 = s |s t = s, a t = a), is the probability of observing the next state s when action a is taken at state s and R(s, a) is the associated reward. The objective is to find a policy π : S → A mapping states to actions that maximizes the expected cumulative reward. Several instances of this problem have been studied depending on the horizon (e.g., finite horizon, infinite horizon with discount). Here, we focus on the infinite horizon with average reward, that consists in maximizing the expected average reward

ρ(M, π, s) = lim T →∞ 1 T E T -1 t=1 r(s t , a t ) s 0 = s, a t = π(x t ) ∀ t ≥ 1 .
The difficulty of learning the optimal policy in MDP depends on size of the state-action space S × A and on the transition model P . To measure this transition structure, [START_REF] Jaksch | Near-optimal regret bounds for reinforcement learning[END_REF] We further consider MDPs with finite diameter which are usually known as communicating. In this case, the optimal average reward ρ (M ) does not depend on the initial state (see e.g., Puterman 2014), i.e.,

ρ (M ) := ρ (M, s) = max π ρ(M, π, s).
Finally, we denote as π (M ) = arg max π ρ(M, π) the optimal policy w.r.t. M and define the regret of any adaptive strategies, i.e., a sequence of policy (π 1 , . . . , π T ) by:

R(T ) = T ρ (M ) - T t=1 r s t , π t (s t ) ,
for which [START_REF] Jaksch | Near-optimal regret bounds for reinforcement learning[END_REF] prove a lower bound of Ω √ DSAT .

UCRL2 algorithm. We now describe the UCRL2 algorithm, based on the optimistic principle, proposed by [START_REF] Jaksch | Near-optimal regret bounds for reinforcement learning[END_REF] (a similar algorithm named REGAL has also been introduced by [START_REF] Bartlett | REGAL: A regularization based algorithm for reinforcement learning in weakly communicating MDPs[END_REF]). In the general setting, the agent does not know about the transition model P and the reward function R of the true MDP M . However, it observes, at each time step t, the current state of the system s t and the associated reward r t = R(s t , a t ). As a result, it has access to empirical estimate P t (s , s, a) and R t (s, a) for any s , s, a ∈ S × S × A defined as

R t = t-1 τ =1 r t 1{s τ = s, a τ = a} t-1 τ =1 1{s τ = s, a τ = a} ; P t (s , s, a) = t-1 τ =1 1{s τ +1 = s , s τ = s, a τ = a} t-1 τ =1 1{s τ = s, a τ = a} . (2.6)
Using concentration inequalities, [START_REF] Jaksch | Near-optimal regret bounds for reinforcement learning[END_REF] introduce a confidence set C t for the MDP at time t, that contains the MDPs with transition model P and reward function R such that, for any state-action pair (s, a),

R(s, a) -R(s, a) ≤ 7 log(2SAt/δ) 2 t-1 τ =1 1{s τ = s, a τ = a} , P (•, s, a) -P (•, s, a) 1 ≤ 14S log(2At/δ) 2 t-1 τ =1 1{s τ = s, a τ = a} ,
where δ ∈ (0, 1), so that, with probability at least 1 -δ, M ∈ C t .

The structure of UCRL2 is very similar to UCB and OFUL, the only difference being that the chosen policy is kept constant for episodes instead of being re-evaluated at each time step: at the beginning of each episode, the agent selects the most optimistic MDP M within the confidence set, and compute the optimal policy π ( M ) associated with M ; then it follows this policy for a whole episode, it observes the states and rewards, and refines the confidence set given the new observations at the end of the episode. An episode ends as soon as the number of visit doubles for a state-action pair. This way of updating the policy is used both for theoretical purpose, and for reducing the computational complexity. We summarize the UCRL2 algorithm in Fig. 2.9.

Initialization: Set ν(s, a) = 0 and N (s, a) = 0 for all (s, a) ∈ S × A, s 1 , π 0 1: for t = {1, . . . , T } do Notice that as opposed to UCB or OFUL, finding the optimistic MDP M t and computing its optimal policy π t is a complicated task. To solve this issue, [START_REF] Jaksch | Near-optimal regret bounds for reinforcement learning[END_REF] introduced an extended value iteration procedure that allows them to compute directly the policy π t , and proved the following regret bound which is a significant improvement over the bound of the UCRL algorithm previously introduced by [START_REF] Auer | Logarithmic online regret bounds for undiscounted reinforcement learning[END_REF].

2: if ν(s t , π t-1 (s t )) > N (s t , π t-
Theorem 2.3.1. For any 0 < δ < 1, with probability at least 1 -δ, for any initial state s and any T > 1, the regret of the UCRL2 algorithm in Fig 2.9 is bounded as

R(T ) ≤ 34DS AT log T δ ,
PSRL algorithm. In line with MAB and LB, a Thompson Sampling algorithm for MDP, known as Posterior Sampling for Reinforcement Learning (PSRL) have been derived [START_REF] Strens | A bayesian framework for reinforcement learning[END_REF]. The idea is to assume a prior over the true MDP M i.e., over the transition model and reward function T and R and to maintain this distribution as new data are collected using prior-posterior update. Then, at each time step t, a MDP M t is sampled from the posterior, and the actions are chosen according to the policy π t that is optimal w.r.t. M t . From a practical perspective, PSRL is therefore much more efficient than UCRL2 since it does not require an extended value iteration procedure (to find the optimistic MDP) but a standard value iteration procedure to extract π t from Chapter 2. Exploration-Exploitation Dilemma in Sequential Decision-Making M t . Additionally, PSRL has been shown to offer very good empirical performances. However, the analysis of PSRL in the same frequentist setting as UCRL2 is significantly more difficult, and most theoretical guarantees are provided for the Bayesian regret, i.e., the expected regret w.r.t. to the prior over the true MDP, and are restricted to finite and episodic MDP. [START_REF] Osband | more) efficient reinforcement learning via posterior sampling[END_REF] proved the first regret bound for PSRL in finite MDPs of order O(S √ AT ). Osband and Roy (2014) studied finite factored MDPs (i.e., MDPs where dynamics and rewards are factored over the multidimensional representation of the state space), showing that their structure can be exploited to reduce the dependency on the number of states and actions in the final bound. The more general setting of learning in parameterized MDPs is studied in (Osband and Van Roy, 2014), where it is shown that the regret of PSRL depends on the dimensionality of the space of parameters rather its cardinality. Recently, [START_REF] Osband | On optimistic versus randomized exploration in reinforcement learning[END_REF] compared the behavior of randomized and optimism-based algorithm, showing that existing optimistic algorithms trade-off statistical efficiency with tractability while randomized approaches may enable simultaneous statistical and computational efficiency. Unfortunately, when moving from the episodic to infinite horizon setting the results of PSRL are very limited. While most of the results for UCRL hold in both cases, [START_REF] Osband | Posterior sampling for reinforcement learning without episodes[END_REF] reviewed in detail the challenges of extending episodic results to infinite horizon showing how previous attempts in proving regret for infinite horizon problem were possibly flawed [START_REF] Abbasi-Yadkori | Bayesian optimal control of smoothly parameterized systems[END_REF]. Notable exceptions are the work of [START_REF] Gopalan | Thompson sampling for learning parameterized markov decision processes[END_REF] who proved frequentist regret bounds in a slightly more general non-episodic setting under the assumption that the MDP is ergodic and that the initial state is positive recurrent under any policy, and the recent result of [START_REF] Agrawal | Posterior sampling for reinforcement learning: worst-case regret bounds[END_REF] who proved a O(D √ SAT ) high-probability frequentist regret bound for any communicating MDP in the infinite horizon with average reward setting.

Linear quadratic control

As for MAB, the main limitation of finite MDPs is their inability to model systems with continuous state-action space, or even large but finite state-action space, since as stressed by the lower bound of [START_REF] Jaksch | Near-optimal regret bounds for reinforcement learning[END_REF], the regret scales at best as √ SA w.r.t. the cardinality. To move from finite to continuous state-action space, the natural idea is to parametrize the MDP (see e.g., Abbasi-Yadkori and Szepesvári 2015), i.e., to impose a structure over the transition model and reward function so that one can learn the model without observing all state-action pairs. On the other hand, this makes the control problem, i.e., the resolution of the Bellman equation, much harder as Value Iteration (VI) and Policy Iteration (PI) cannot be applied. In order to overcome this issue, standard approaches consist in approximating the value function or discretizing the state-space to perform (VI) and (PI), thus narrowing down the interest of the continuous state-action space formulation. A notable exception is the Linear Quadratic (LQ) control problem which is a specific instance of parametrized MDP with continuous state-action space where the dynamics of the environment is linear in the state and control, and the cost function is quadratic (LQ is generally introduced as a cost minimization problem rather than a reward maximization problem). This setting stands as a standard in the control literature because the Bellman equation can be turned into a discrete Riccati equation, which can be solved efficiently. From a practical perspective, LQ problems have been intensively used and studied to address problems in many different fields such as robotics, economics, bioengineering, finance etc... We provide in this section an overview of the LQ theory. The interested reader may refer to [START_REF] Bertsekas | Dynamic programming and optimal control[END_REF] for a first introduction and to [START_REF] Lancaster | Algebraic riccati equations[END_REF] for a complete survey of the underlying Riccati equation. Finally, we present the OFU-LQ algorithm introduced by Abbasi-Yadkori and Szepesvári (2011) that uses the optimistic principle to address the exploration-exploitation trade-off in LQ system.

Setting. We adopt the standard notations of the LQ theory and consider the dynamic system of Fig. 2.8 where the environment is characterized by a state x t ∈ R n and control u t ∈ R d , and evolves according to the linear dynamic

x t+1 = Ax t + Bu t + x t+1 ,
where

A ∈ R n×n , B ∈ R n×d and { x t+1 } t≥0 is a martingale difference sequence w.r.t. the filtration F t = σ(x 0 , . . . , x t ) such that V( x t+1 |F t ) = Σ x .
The objective of the agent is to find a policy π : R n → R d mapping state to control that minimizes the average expected cost

J π = lim T →∞ 1 T E T -1 t=1 c(x t , u t ) x 0 = s, u t = π(x t ), ∀ t ≥ 1 .
The cost function is quadratic in the state and action as:

c(x, u) = x T Qx T + 2x T N u + u T Ru,
where Q, R, N are matrices of appropriate dimensions. For sake of convenience, we collect them into a single matrix

Q = Q N N T R
. Before stating the main result of the LQ theory, we introduce several properties characterizing the linear system.

Definition 2.3.2 (Ch.4 in [START_REF] Lancaster | Algebraic riccati equations[END_REF]). Let A ∈ R n×n and B ∈ R n×d be two real-valued matrices and let C = B, AB, . . . , A n-1 B be the associated controllability matrix. Then,

• The pair (A, B) is said to be controllable if and only if the controllability matrix C is of rank n (full row rank).

• The pair (A, B) is said to be stabilizable if and only if for any x ∈ Ker(C), Ax ≤ x .

The controllability of the pair (A, B) ensures that every state x is reachable from any state x in less than n time steps by using a suitable sequence of control. As a Decision-Making consequence, this property can be seen as the continuous counterpart of the finite diameter in MDP. Stabilizability is a weaker notion that ensures that the uncontrollable part of the linear system is stable. Formally, a matrix M is said to be stable if M 2 ≤ 1, where • 2 is the spectral norm. Symmetrically, we introduce the notion of observability and detectability that ensures that it is possible to retrieve the initial state of the system driven by x t+1 = Ax t + x t+1 from any sequence of n observations given by y t = Cx t .

Definition 2.3.3 (Ch.4 in [START_REF] Lancaster | Algebraic riccati equations[END_REF]). Let A ∈ R n×n and C ∈ R d×n be two real-valued matrices. Then,

• The pair (C, A) is said to be observable if and only if the pair (A T , C T ) is controllable.

• The pair (C, A) is said to be detectable if and only if the pair (A T , C T ) is stabilizable.

To ensure the existence and uniqueness of an optimal solution to the LQ problem, we consider the following assumption.

Assumption 2.3.1. The pair (A, B) is stabilizable and the matrix Q is symmetric positive definite.

In some case, the positive definiteness of Q is a too restrictive assumption, which can be relaxed by the weaker assumption: Assumption 2.3.2. The pair (A, B) is stabilizable, the matrix Q is symmetric non singular and the pair (Q, A) is observable.

Under those assumptions, it is well known that the LQ problem admits a unique optimal policy which is linear in the state.

Theorem 2.3.2 (Th.16.6.4 in [START_REF] Lancaster | Algebraic riccati equations[END_REF]). Under Asm. 2.3.1 or 2.3.2, the optimal solution of the LQ problem is unique and given by

π (x) = Kx, K = -(R + B T P B) -1 (B T P A + N T ), P = Q + A T P A -(A T P B + N )(R + B T P B) -1 (B T P A + N T ).
Further, the closed-loop matrix A + BK is asymptotically stable and the optimal cost is given by J π = Tr(P Σ x ).

Notice that the optimal control matrix K is function of a matrix P , that is the solution of a so-called discrete Riccati equation, which can be computed efficiently thus making the LQ solution tractable (see [START_REF] Laub | Invariant subspace methods for the numerical solution of riccati equations[END_REF][START_REF] Van Dooren | A generalized eigenvalue approach for solving riccati equations[END_REF][START_REF] Chun-Hua | Newtons method for discrete algebraic riccati equations when the closedloop matrix has eigenvalues on the unit circle[END_REF][START_REF] Laub | A schur method for solving algebraic riccati equations[END_REF].

RL in LQ systems. While Thm. 2.3.2 ensures that the optimal solution to the LQ problem is tractable for a given system with known matrices, it is often the case that the matrices are unknown and have to be estimated online, i.e., while controlling the system. To tackle this problem, Abbasi-Yadkori and Szepesvári (2011) introduced an optimistic algorithm called OFU-LQ, which addresses the induced exploration-exploitation tradeoff in LQ system. We recall here their setting, we present the OFU-LQ algorithm, and its theoretical guarantees. They consider the LQ system with cost function c(x, u) = x T Qx + u T Ru and assume that the cost matrices Q and R are known to the agent while the dynamic of the system is unknown. Additionally, they assume for sake of simplicity that Σ x = I. Formally, they denote as (A * , B * ) the unknown matrices of the true dynamics and collect them in a matrix θ T * = (A * , B * ). The objective of the learning strategy is to find a sequence of policy {π t } t that minimizes the regret w.r.t. the unknown optimal average cost J * = J π (θ * ) = TrP (θ * ) defined as:

R(T ) = T t=0 x T t Qx t + u T t Ru t -T J * ,
where u t = π t (x t ) for all t = 0, . . . , T .

Leveraging the linear structure of the state dynamic, at each time step, Abbasi-Yadkori and Szepesvári (2011) estimate θ * given the past control sequence (u 0 , . . . , u t-1 ) and associated states (x 0 , . . . , x t ) using RLS, for any regularization parameter λ > 0, as:

V t = λI + t-1 s=0 z s z T s ; θ t = V -1 t t-1 s=0 z s x T s+1 , (2.7)
where z t = (x t , u t ) T . A concentration inequality for this matrix RLS estimate can be derived in a straightforward manner from Prop. 2.2.1.

Proposition 2.3.1. For any δ ∈ (0, 1) and any F t -adapted sequence (z 0 , . . . , z t ), the RLS estimator θt is such that

Tr ( θt -θ * ) T V t ( θt -θ * ) ≤ β 2 t (δ); β t (δ) = n 2 log det(V t ) 1/2 det(λI) 1/2 δ + λ 1/2 S
:= {θ ∈ R n(n+d) s.t. Tr ( θt -θ) T V t ( θt -θ) ≤ β 2
t (δ)} be the confidence ellipsoid such that θ * ∈ E RLS t with probability 1 -δ. Abbasi-Yadkori and Szepesvári (2011) propose an algorithm that follows the same structure as UCRL2: at the beginning of each episode, an optimistic parameter θ t is chosen as

θ t = arg min θ∈Θ∩E RLS t TrP (θ),
where they add the constraint θ ∈ Θ (we refer to (Abbasi-Yadkori and Szepesvári, 2011) for the formal definition of Θ) to guarantee the controllability of θ Update Abbasi-Yadkori and Szepesvári (2011) prove the following regret bound for the OFU-LQ algorithm:

V t+1 = V t + x t x T
Theorem 2.3.3. For any 0 < δ < 1, for any time T , with probability at least 1 -δ, the regret of OFU-LQ algorithm is bounded as

R(T ) = O T log(1/δ)
where O hides logarithmic factors and problem dependent constant.

We conclude this section by noting that finding the optimistic parameter θ t is a computationally expensive task as θ :→ TrP (θ) is a non-convex function, while computing the optimal control K(θ) is cheap thanks to the Riccati equation solver. As a result, a TS algorithm would be much more efficient, as the optimistic step would be replaced by a sampling from the posterior. [START_REF] Abbasi-Yadkori | Bayesian optimal control of smoothly parameterized systems[END_REF] proposed a lazy PSRL algorithm for smoothly parametrized MDP, for which LQ is a specific instance, that they study in the Bayesian regret setting. Unfortunately, as hinted by [START_REF] Osband | Posterior sampling for reinforcement learning without episodes[END_REF], the proof suffers a flaw coming from the difficulty to move from episodic to non-episodic settings. Applying PSRL to LQ and providing regret bounds for this algorithm is one of main questions that motivates our work. We address it in Ch. 4.

Chapter 3

Thompson Sampling in Linear Bandit

In this chapter1 , we derive an alternative proof for the regret of Thompson sampling (TS) in the stochastic linear bandit setting. While we obtain a regret bound of order O(d 3/2 √ T ) as in previous results, the proof sheds new light on the functioning of the TS. We leverage the structure of the problem to show how the regret is related to the sensitivity (i.e., the gradient) of the objective function and how selecting optimal arms associated to optimistic parameters does control it. Thus, we show that TS can be seen as a generic randomized algorithm where the sampling distribution is designed to have a fixed probability of being optimistic, at the cost of an additional √ d regret factor compared to a UCB-like approach. Furthermore, we show that our proof can be readily applied to regularized linear optimization and generalized linear model problems.

Introduction

In this chapter, we focus on the Linear Bandit (LB) problem introduced in Sec. 2.2 and draw novel insights on the functioning of Thompson Sampling (TS) in this setting, where the value of an arm is obtained as the inner product between an arm feature vector x and an unknown global parameter θ . While TS has been originally introduced as a Bayesian heuristic [START_REF] Thompson | On the likelihood that one unknown probability exceeds another in view of the evidence of two samples[END_REF], it has been proved to offer good performance in the frequentist setting (Agrawal and Goyal, 2012b). We propose here an alternative frequentist analysis for the regret of TS in LB that stresses the randomized nature of the exploration performed by the algorithm. As opposed to the optimistic approaches, the main technical difficulty in analyzing TS lies in controlling the deviation in performance due to the randomness of the algorithm. Agrawal and Goyal (2012b) follows the MAB proof structure (as in (Agrawal and Goyal, 2012a)) classifying arms as saturated and unsaturated depending on whether their standard deviation is smaller or bigger than their gap to the optimal arm.2 While for unsaturated arms the regret is related to their standard deviation that decreases over time, they prove that TS has a small (but constant) probability to select saturated arms and it achieves a regret O d 3/2 √ T .

Contributions.

The major contributions of this paper are: 1) Following the intuition of Agrawal and Goyal (2012b), we show that the TS does not need to sample from an actual Bayesian posterior distribution and that any distribution satisfying suitable concentration and anti-concentration properties guarantees a small regret. In particular, we show that the distribution should over-sample w.r.t. the standard least-squares confidence ellipsoid by a factor √ d to guarantee a constant probability of being optimistic. 2) We provide an alternative proof of TS achieving the same result as Agrawal and Goyal (2012b). One of our major finding is that, leveraging the properties of support functions from convex geometry, we are able to prove that the regret is related to the gradient of the objective function, that is ultimately controlled by the norm of the optimal arms associated to any optimistic parameter θ. This provides a novel insight on the fact that whenever an optimistic parameter θ t is chosen, not only is its instantaneous regret small but the corresponding optimal arm x t = arg max x x T θ t represents a useful exploration step that improves the accuracy of the estimation of θ over dimensions which are relevant to reduce regret in any subsequent non-optimistic step. This approach allows us to avoid the introduction of saturated/unsaturated arms and it illustrates why any TS-like algorithm (not necessarily Bayesian) with a constant probability of being optimistic has a bounded regret. 3) Finally, we show how our proof can be easily adapted to regularized linear optimization (with arbitrary penalty) and to the generalized linear model, for which we derive the first frequentist regret bound for TS, which was first suggested by Agrawal and Goyal (2012b) as a venue to explore.

Preliminaries

The setting. We briefly recall the LB setting introduced in Sec. 2.2, and detail the assumptions that we impose on the problem structure as well as the additional material needed for our analysis. Let X ⊂ R d be an arbitrary (finite or infinite) set of arms. When an arm x ∈ X is pulled, a reward is generated as r(x) = x T θ + ξ, where θ ∈ R d is a fixed but unknown parameter and ξ is a zero-mean noise. An arm x ∈ X is evaluated according to its expected reward x T θ and, for any θ ∈ R d , we denote the optimal arm and its value by

x (θ) = arg max x∈X x T θ, J(θ) = sup x∈X x T θ. (3.1)
At each step t, the learner selects an arm x t ∈ X based on the past observations (and possibly additional randomization), it observes the reward r t+1 = x T t θ + ξ t+1 , and it suffers a regret equal to the difference in expected reward between the optimal arm x (θ ) and the arm x t . The objective of the learner is to minimize the cumulative regret up to step T , i.e.,

R(T ) = T t=1 x T θ -x T t θ .
Notation. We use • to denote the 2-norm and x T to denote the transpose of x ∈ R d . For a positive definite matrix M ∈ R d×d , we denote as • M the weighted 2-norm defined by x 2 M = x T M x for any x ∈ R d . We use λ min (M ) and λ max (M ) to denote the minimum and maximum eigenvalues of the positive semi-definite matrix M , respectively. We use 1{E} to denote the indicator function of the event E. Finally, we encode all the information observed up to time t in the filtration F x t = (F 1 , σ(x 1 , r 2 , . . . , r t , x t )), where F 1 contains any prior knowledge.

We impose the following assumptions on the problem structure and the noise ξ t+1 .

Assumption 3.2.1 (Arm set). The arm set X is a bounded closed (and hence compact) subset of R d such that x ≤ X for all x ∈ X . We also assume X = 1.

We focus here on the fixed arm set setting where X does not change with time while the original analysis of Agrawal and Goyal (2012b) has been derived for the contextual LB problem where the arm set X t can be chosen by an oblivious adversary at each time step t. However, our analysis still holds in this case, replacing X by X t and the optimal value function J by J t in every steps of the proof.

Assumption 3.2.2 (Bandit parameter).

There exists S ∈ R + such that θ ≤ S and S is known. Assumption 3.2.3 (Noise). The noise process {ξ t } t is a martingale difference sequence given F x t and it is conditionally R-subgaussian for some constant R ≥ 0,

∀t ≥ 1, E [ξ t+1 |F x t ] = 0, ∀α ∈ R, E e αξ t+1 | F x t ≤ exp α 2 R 2 /2 .
Technical tools. Let (x 1 , . . . , x t ) ∈ X t be a sequence of arms and (r 2 , . . . , r t+1 ) be the corresponding rewards, then θ can be estimated by regularized least-squares (RLS). For any regularization parameter λ ∈ R + , the design matrix and the RLS estimate are defined as

V t = λI + t-1 s=1 x s x T s , θ t = V -1 t t-1 s=1 x s r s+1 . (3.2)
For any 0 < δ < 1, we make use of Prop. 2.2.1 to define the high-probability ellipsoid E RLS t at each time step t as

E RLS t = θ ∈ R d θ-θ t Vt ≤ β t (δ ) , with β t (δ ) = R 2 log (1 + t/λ) d/2 δ + √ λS, (3.
x s 2 V -1 s ≤ 2 log det(V t+1 ) det(λI) ≤ 2d log 1 + t λ .
This result plays a central role in most of the proofs for linear bandit, since the regret is usually related to x s V -1 s and Prop. 3.2.1 is used to bound its cumulative sum. While Agrawal and Goyal (2012b) achieve this by dividing arms in saturated and unsaturated, we follow a different path that leverages the core features of the problem (structure of J(θ)) and of TS (probability of being optimistic).

Linear Thompson sampling

Agrawal and Goyal (2012b) define TS for linear bandit as a Bayesian algorithm where a Gaussian prior over θ is updated according to the observed rewards, a random sample is drawn from the posterior, and the corresponding optimal arm is selected at each step. As hinted by Agrawal and Goyal (2012b), we show that TS can be defined as a generic randomized algorithm constructed on the RLS-estimate rather than an algorithm sampling from a Bayesian posterior (see Fig. 3.1). At any step t, given RLS-estimate θ t and the design matrix V t , TS samples a perturbed parameter θ t as

θ t = θ t + β t (δ )V -1/2 t η t ,
where η t is a random sample drawn i.i.d. from a suitable multivariate distribution D TS , which does not need to be associated with an actual posterior over θ . Then the optimal arm x t = x ( θ t ) is chosen, a reward r t+1 is observed and V t and θ t are updated according to Eq. 3.2. Notice that the resulting distribution on θ t is obtained by rotation of η t according to the design matrix V t and by a rescaling β t (δ ). The computational complexity of TS is dominated by computation of x ( θ t ), which requires solving a linear optimization problem and by the sampling process from D TS . This is in contrast with OFUL (Abbasi-Yadkori et al., 2011a) presented in Fig. 2.6, which requires solving a bilinear optimization problem (i.e., arg max θ max x x T θ).

Input: θ1 , V 1 = λI, δ, T 1: Set δ = δ/(4T ) 2: for t = {1, . . . , T } do 3: Sample η t ∼ D TS and compute parameter θ t = θ t + β t (δ )V -1/2 t η t 4:
Compute optimal arm

x t = x ( θ t ) = arg max x∈X x T θ t 5:
Pull arm x t and observe reward r t+1 6:

Compute V t+1 and θ t+1 using Eq. 3.2 7: end for The key aspect to ensure small regret is that the perturbation η t is distributed so that TS explores enough but not too much. This translates into the following conditions on D TS . Definition 3.3.1. D TS is a multivariate distribution on R d absolutely continuous with respect to the Lebesgue measure which satisfies the following properties:

1. (anti-concentration) there exists a strictly positive probability p such that for any u ∈ R d with u = 1,

P η∼D TS u T η ≥ 1 ≥ p,
2. (concentration) there exists c, c positive constants such that ∀δ ∈ (0, 1)

P η∼D TS η ≤ cd log c d δ ≥ 1 -δ.
Once interpreted in the construction of θ t , the definition of D TS basically requires TS to explore far enough from θ t (anti-concentration) but not too much (concentration). This implies that TS performs "useful" exploration with enough frequency (notably it performs optimistic steps), but without selecting arms with too large regret. We introduce the high-probability ellipsoid E TS t as

E TS t = {θ ∈ R d | θ -θ t Vt ≤ γ t (δ )}, with γ t (δ ) = β t (δ ) cd log(c d/δ ).
The difference between E RLS t and E TS t lies in the additional factor √ d in the definition of γ t (δ ) and it is crucial for both concentration and anti-concentration to hold at the same time. In Sect. 3.5 we prove that any distribution satisfying the conditions in Def. 3.3.1 introduces the right amount of randomness to achieve the desired regret without actually satisfying any Bayesian assumption. Def. 3.3.1 includes the Gaussian prior used by Agrawal and Goyal (2012b), but also other types of distributions such as the uniform on the unit ball B d (0, √ d) or distributions concentrated on the boundary of E TS t (refer to App. 3.A for exact values of c, c , and p for uniform and Gaussian distributions).

Sketch of the proof

In this section we report a sketch of the proof providing a geometric intuition on the behavior of TS and how its actions (i.e., the sampled θ t and the corresponding x t ) influence the regret. For the sake of illustration, we consider the unit ball X = { x ≤ 1}, such that the optimal arm is just the projection of θ on the ball (x (θ) = θ/ θ ), and the optimal value is J(θ) = θ T θ/ θ = θ . We start by decomposing the regret using the definition of J(θ) as

R(T ) = T t=1 x T θ -x T t θ t + x T t θ t -x T t θ = T t=1 J(θ ) -J( θ t ) R TS (T ) + T t=1 x T t θ t -x T t θ R RLS (T )
,

where R TS depends on the randomization of TS and R RLS mostly depends on the properties of RLS.

Bounding R RLS (T ).

We first show that both RLS estimate θ t and TS parameter θ t should concentrate appropriately, by decomposing the regret R RLS (T ) as

R RLS (T ) = T t=1 x T t θ t -x T t θ + T t=1 x T t θ t -x T t θ t .
Since at each step t, θ t is sampled from D TS , the second term is kept under control by construction, while the first sum deals with the prediction error of RLS. As opposed to R TS , this error is not related to the exploration scheme and it is small for any sequence of arms. Intuitively, this is due to the fact that the RLS estimate is the minimizer of the regularized cumulative squared error θ T +1 = arg min θ T t=1 |r t+1 -x T t θ| 2 + λ θ 2 , so that x T t θ T +1 is an accurate prediction on the arms observed so far. The RLS minimizes the error in "hindsight" (i.e., after observing all rewards up to T ) and therefore it also

controls the online error |r t+1 -x T t θt+1 | 2 . By induction, T t=1 |r t+1 -x T t θ T +1 | 2 + λ θ T +1 2 = |r T +1 -x T T θ T +1 | 2 + T -1 t=1 |r t+1 -x T t θ T +1 | 2 + λ θ T +1 2 ≥ |r T +1 -x T T θ T +1 | 2 + min θ T -1 t=1 |r t+1 -x T t θ| 2 + λ θ 2 = |r T +1 -x T T θ T +1 | 2 + T -1 t=1 |r t+1 -x T t θ T | 2 + λ θ T 2 ≥ • • • ≥ T t=1 |r t+1 -x T t θ t+1 | 2 + λ θ 1 2 .
Having a small online error also implies a small prediction error |r t+1 -x T t θt | 2 . In fact, using a recursive version of Eq. 3.2, we have

θt+1 = θt + V -1 t x t (1 + x t 2 V -1 t ) -1 (r t+1 -x T t θt ), which, together with x t 2 V -1 t ≤ 1/λ, leads to |r t+1 -x T t θt+1 | ≥ λ 1 + λ |r t+1 -x T t θt |.
Since the cumulative prediction error is small, then the associated regret

T t=1 |x T t θ t -x T t θ
| is also small. This result can be seen as an intrinsic onpolicy error guarantee of RLS. Nonetheless, notice that while RLS minimizes the prediction error for any sequence of arms, this does not imply the consistency of the estimator. For instance, when the same arm x is repeatedly played, the unknown parameter θ is well-estimated in the direction of x (thus making R RLS (T ) small) but it is poorly estimated in any other directions. This shows the need for a careful exploration strategy to recover consistency and hence a sub-linear regret.

Bounding R TS (T ).

We denote by R TS t = J(θ ) -J( θ t ) each term in R TS (T ). For optimistic algorithms this term is bounded by 0 at any step since w.h.p. J( θ t ) ≥ J(θ * ) by construction. In the Bayesian regret analysis of TS, this term is equal to 0 by assumption that θ * is drawn from the same prior as θ t . On the other hand, in the frequentist analysis, we have to control the deviations caused by the random sampling of θ t . This is achieved by showing that the arms selected by TS provide "useful" information about θ and contribute to keep the regret small. We follow three steps: 1) we show that the regret is related to the sensitivity of J w.r.t. the errors in estimating θ and we bound the regret with the gradient of J(θ) at any optimistic θ; 2) we show how the gradient in a point θ is intrinsically related to its corresponding optimal arm x (θ); 3) since we prove that TS is frequently optimistic, then we can finally link x (θ) to x t = x ( θ t ) and Prop. 3.2.1 allows us to finally bound the overall regret.

Chapter 3. Thompson Sampling in Linear Bandit

true regret θ J(θ ) upper bound J(θ) J( θ τ ) ∇J( θ τ ) θ τ θ τ J( θ t ) θ t θ t E TS t E RLS t θ * p θ t θ t O X x t = x ( θ t ) Θ opt x Figure 3
.2 -Illustration of the steps 2) and 3) of the proof in R 1 and R 2 . Left: The regret at step t could be bounded by the gradient of the function J at a previous optimistic θ τ times the distance between θ τ and the current θ t . Notice that θ is always included in E RLS t (in dark gray) and thus θs sampled from E TS t (in light gray) are never too far. Right: TS has a constant probability of being optimistic thanks to the over-sampling of D TS .

Step 1 (regret and sensitivity of J). We first show why the exploration of TS should be well adapted to J(θ). Using the definition of J(θ) = θ we have

R TS t = J(θ ) -J( θ t ) = θ -θ t ≤ θ -θ t ≤ θ -θ t Vt λ min,t
, where λ min,t is the smallest eigenvalue of V t . This bound shows that it is sufficient to estimate θ accurately over all its components (i.e., λ min,t tends to infinity) to obtain a no-regret algorithm. Nonetheless, the desired regret bound of O( √ T ) is obtained only if λ min,t increases as O(t). While this could be achieved by a fully explorative algorithm (e.g., a round robin over the canonic vectors e i reduces the ellipsoid E TS t to a ball of radius λ min,t ), it would severely increase the second term of R RLS (T ) and cause an overall linear regret 3 . Fortunately, inspecting the definition of R TS t reveals that not all components of θ must be equally well estimated. In fact, we have w.h.p. that

R TS t ≤ sup θ∈E RLS t sup θ ∈E TS t J(θ) -J(θ ) .
This shows that R TS t is determined by the diameter of ellipsoid E TS t w.r.t. J, which suggests that the estimation of θ should be more accurate on the dimensions on which J is more sensitive. In the case of X unit ball, the most sensitive direction of J is θ / θ itself and Fig. 3.3 illustrates two opposite cases where the accuracy in the estimation of θ is the same (i.e., V t has the same eigenvalues) but the regret may be very different.

3 This happens because x t would be optimal w.r.t. a θ t , which is not in the ellipsoid The numerical experiment reported in Fig. 3.4 supports the fact that TS handles correctly the consistency of the estimates w.r.t. the sensitivity of J. In the 2-dimensional case, the eigenvalues λ max,t and λ min,t exhibit different divergence rate. While 1/λ max,t decreases as 1/t, 1/λ min,t decreases as 1/ √ t, which according to a direct consistency argument, is not enough to guarantee a √ T regret. However, the picture on the r.h.s shows that the diameter of the ellipsoid E TS t w.r.t J decreases as 1/ √ t. This is due to the fact that the sampling ellipsoid E TS t tends to align with the first configuration of Fig. 3.3. It implies that, on direction where J is very sensitive, the diameter of the ellipsoid shrinks appropriately (scaling with 1/λ max,t ≈ 1/t) whereas on direction where J is less sensitive, a slower decay (scaling with 1/λ min,t ≈ 1/ √ t) of the diameter of the ellipsoid still guarantees the deviation in J to be small. Therefore, the overall deviation in any direction decreases as 1/ √ t, inducing a √ T regret.

E RLS t . per-step regret E TS t,2 E TS t,1 θ O X Figure 3.3 -While E TS t,
10 2 10 3 10 4 10 -3 10 -2 10 -1 10 0 10 1 1/λ max, t 1/λ min, t 10 0 10 1 10 2 10 3 10 4 10 -2 10 -1 10 0 Let Θ opt = {θ : J(θ) ≥ J(θ )} be the set of optimistic parameters. In our example J(θ) = θ is convex thus we can make explicit the dependency of the regret on the sensitivity of J through its gradient evaluated at any θ ∈ Θ opt as (see Prop. 3.5.1 for the general case)

10 1 sup θ, θ ∈ E TS t J(θ) -J(θ ) 1/ t Figure 3.4 -
R TS t ≤ sup θ ∈E TS t J(θ) -J(θ ) ≤ sup θ ∈E TS t ∇J(θ) T (θ -θ ),
which shows that the regret of non-optimistic θ t is bounded by the gradient of J(θ) at any optimistic θ and its distance to any other point in the TS ellipsoid.

Step 2 (sensitivity of J and optimal arm). According to Prop. 2.2.1, the difference θ -θ in the previous inequality is well controlled whenever θ belongs to the ellipsoid, while the first term cannot be immediately controlled by the algorithm. Nonetheless, we notice that since J(θ) = θ , then ∇J(θ) = θ/ θ = x (θ) (see Lem. 3.5.2 for the general case). This shows how selecting the optimal arm associated to an optimistic θ is equivalent to controlling the gradient of J, which results in

R TS t ≤ sup θ ∈E TS t x (θ) T (θ -θ ).
From Prop. 3.2.1, we could conclude that the regret would be cumulatively small if x (θ) corresponded to the arms chosen by the TS (x t = x ( θ t )). As a result, we need a θ 1) that is optimistic (i.e., θ ∈ Θ opt ), 2) it belongs or is close to the ellipsoid E TS t and 3) it is used to select an arm x t . The first two requirements are at the core of the choice of the TS distribution in Def. 3.3.1 where the anti-concentration property guarantees enough probability to be optimistic, while the concentration property implies that θs are within a small ellipsoid. Let τ < t be any step when TS selects θ τ ∈ Θ opt with corresponding arm x τ = x ( θ τ ), then we have (see an illustration of this bound in Fig. 3.2 in the 1-d case)

R TS t ≤ sup θ ∈E TS t x T τ ( θ τ -θ ) ≤ x τ V -1 τ sup θ ∈E TS t θ τ -θ Vτ .
Introducing θ and using the fact that the design matrices forms a non-decreasing sequence (i.e., V τ V t ), we decompose

sup θ ∈E TS t θ τ -θ Vτ ≤ θ τ -θ Vτ + sup θ ∈E TS t θ -θ Vτ , ≤ θ τ -θ Vτ + sup θ ∈E TS t θ -θ Vt .
Since by Prop. 2.2.1, θ is contained in all ellipsoids E RLS t with high probability, then

R TS t ≤ β τ (δ ) + γ τ (δ ) + β t (δ ) + γ t (δ ) x τ V -1 τ , ≤ 2β T (δ ) + 2γ T (δ ) x τ V -1 τ .
Let K be the number of times θ t ∈ Θ opt , t k the corresponding steps, and ν k = t k -t k-1 , then the final regret can be written as

R TS (T ) ≤ 2 β T (δ ) + γ T (δ ) K k=1 ν k x t k V -1 t k
.

Step 3 (optimism). This bound shows the importance that TS is optimistic with high frequency. In fact, whenever θ t is in Θ opt , not only the corresponding instantaneous regret R TS t is upper-bounded by 0, but the exploration performed by playing arm x ( θ t ) has also a positive impact in controlling the regret for any subsequent non-optimistic step. Consider the extreme case when TS is never optimistic. Then, K = 1, ν 1 = T and R TS (T ) = O(T ). On the other hand, if TS is optimistic with a constant frequency, then we can easily show that R TS (T ) is bounded by O( √ T ). Consider the case where an optimistic θ is chosen with probability p. Since E[ν k ] = 1/p, we can prove that w.h.p.

R TS (T ) ≤ O(1/p √ T ) by Cauchy-Schwarz and Prop. 3.2.1 applied to K k=1 x t k 2 V -1 t k
, where K ≈ T . Unfortunately, sampling θ t from the RLS ellipsoid E RLS t may have a very small probability of being optimistic (see e.g., Fig. 3.2, where sampling uniformly in E RLS t has zero probability to return a θ t ∈ Θ opt ). For this reason, TS is required to draw θ t from a distribution over-sampling by a factor

√ d w.r.t. E RLS t
as in the definition of D TS . This guarantees a fixed probability p of being optimistic (see Lem. 3.5.3) and the final desired regret.

Formal proof

In this section we report the main steps of the regret analysis, while we postpone technical lemmas to the supplementary material. We prove the following result.

Theorem 3.5.1. Under assumptions 3.2.1,3.2.2,3.2.3, the regret of TS is bounded w.p.

1 -δ as R(T ) ≤ β T (δ ) + γ T (δ )(1 + 4/p) 2T d log 1 + T λ + 4γ T (δ ) p 8T λ log 4 δ ,
where δ = δ 4T . As anticipated in introduction, this bound is of order O(d 3/2 √ T ) and it entirely matches the result of Agrawal and Goyal (2012b). The analysis of the regret requires extra care in the definition of the filtrations. While in analyzing R RLS we consider all the knowledge up to step t (i.e., including the sampled parameter θ t ), in R TS we need to study the randomness of θ t conditional on all the information before sampling η t . We introduce an additional filtration besides F x t .

Definition 3.5.1. We define the filtration F t as the accumulated information up to time t before the sampling procedure, i.e., F t = (F 1 , σ(x 1 , r 2 , . . . , x t-1 , r t )).

Chapter 3. Thompson Sampling in Linear Bandit

Notice that θ t and V -1 t are both F t and F x t adapted, while θ t is a random variable w.r.t. F t and it is fixed when considering F x t . Hence we have

F 1 ⊂ F 2 ⊂ F x 2 ⊂ F 3 ⊂ F x
3 , . . . . We are now ready to introduce the high-probability events we use in the rest of the proof. Definition 3.5.2. Let δ ∈ (0, 1) andδ = δ/(4T ) andt ∈ [1, T ]. We define E t as the event where the RLS estimate concentrates around θ for all steps s ≤ t, i.e.,

E t = ∀s ≤ t, θ s -θ Vs ≤ β s (δ ) .
We also define E t as the event where the sampled parameter θ s concentrates around θ s for all steps s ≤ t, i.e.,

E t = ∀s ≤ t, θ s -θ s Vs ≤ γ s (δ ) .
Then we have that

E := E T ⊂ • • • ⊂ E 1 , E := E T ⊂ • • • ⊂ E 1 and we use E t = E t ∩ E t and E = E ∩ E. Lemma 3.5.1. [see proof in App. 3.D] Under Asm. 3.2.2, 3.2.3 we have P( E ∩ E) ≥ 1 - δ 2 .
Conditioned on F t and event E t , we have θ ∈ E RLS t , while on event E t we have θ t ∈ E TS t , then we directly bound the regret on event E as

R(T ) = R(T )1{E} = T t=1 J(θ ) -J( θ t ) 1{E} + T t=1 x T t θ t -x T t θ 1{E} ≤ T t=1 J(θ ) -J( θ t ) 1{E t } + T t=1 x T t θ t -x T t θ 1{E t } = T t=1 R TS t + T t=1 R RLS t , w.p. 1 -δ/2. Notice that the formal definitions of R TS t and R RLS t
involve the indicator of the high-probability events 1{E t }, which is the quantity of interest since we aim at bounding the regret on E. As discussed in Sec. 3.4, the main difficulty lies in bounding the regret term specific to TS. We first report the formal proof to bound R TS (T ), while the bound on R RLS (T ) and the overall regret is postponed to Sec. 3.5.4.

Similar to the sketch in Sect. 3.4, the proof follows three steps: 1) we use the convexity of J to upper-bound the regret by its expectation conditioned on being optimistic and to relate it to the gradient of J, 2) we relate the gradient of J to the arms chosen by TS over time, 3) we show that despite the randomization, TS has a constant probability of being optimistic.

Step 1 (regret and gradient of J(θ)).

On event E t , θ t belongs to E TS t and thus,

R TS t ≤ J(θ ) -inf θ∈E TS t J(θ) 1{ E t }.
Recalling that Θ opt is the set of all optimistic θs, we can bound the previous expression by the expectation over any random choice of θ in Θ opt t := Θ opt ∩ E TS t where we restrict the optimistic set to the high-probability sampling ellipsoid, that is

R TS t ≤ E J( θ) -inf θ∈E TS t J(θ) 1{ E t } F t , θ ∈ Θ opt t , where θ = θ t + β t (δ )V -1/2 t
η with η ∼ D TS is the TS sampling distribution. We now rely on the following characterization of J(θ) (see App. 3.C).

Proposition 3.5.1. For any set of arm X satisfying Asm. 3.2.1, J(θ) = sup x x T θ has the following properties: 1) J is real-valued as the supremum is attained in X , 2) J is convex on R d , 3) J is continuous with continuous first derivative except for a zero-measure set w.r.t. the Lebesgue's measure.

These properties follow from the fact that J is the support function of X and it shows that J is convex for any arm set X . As a result, we can directly relate R TS t to the gradient of J as

R TS t ≤ E sup θ∈E TS t ∇J( θ) T ( θ -θ)1{ E t } F t , θ ∈ Θ opt t ≤ E ∇J( θ) V -1 t sup θ∈E TS t θ-θ Vt 1{ E t } F t , θ ∈ Θ opt t ≤ 2γ t (δ )E ∇J( θ) V -1 t F t , θ ∈ Θ opt t 1{ E t }, (3.4)
where we used Cauchy-Schwarz from line 1 to line 2, the fact that θ ∈ E TS t and that E t is F t measurable from line 2 to line 3.

3.5.2

Step 2 (from gradient of J(θ) to optimal arm x (θ)).

In the sketch of the proof there was a direct relationship between ∇J(θ) and the optimal arm corresponding to θ by direct construction. In the next lemma, we show that this connection is true for any arm set X . Lemma 3.5.2. Under Asm. 3.2.1, for any θ ∈ R d , we have ∇J(θ) = x (θ) except for a zero-measure set w.r.t. the Lebesgue's measure.

Chapter 3. Thompson Sampling in Linear Bandit

Proof. The proof relies on the fact that, for any arm set X , J is the support function of X . Recalling Eq. 3.1,

x (θ) = arg max x∈X x T θ, J(θ) = sup x∈X x T θ.
By Asm. 3.2.1, x (θ) is well defined (since X is bounded) and x (θ) ∈ X (since X is closed). By Prop. 3.C.1, we know that J is twice differentiable almost everywhere. Thus, denoting as ∂J(θ) the sub-gradient of J in θ ∈ R d , it guarantees that it reduces to a singleton almost everywhere. This translate in ∂J(θ) = {∇J(θ)} except for a zero-measure set w.r.t. the Lebesgue's measure. Thus, one just has to ensure that x (θ) ∈ ∂J(θ) for all θ ∈ R d to conclude the proof. By definition, x (θ) T θ = J(θ) and for any θ ∈ R d , J( θ) ≥ x (θ) T θ. Therefore,

J( θ) -x (θ) T θ ≥ 0 := J(θ) -x (θ) T θ J( θ) ≥ J(θ) + x (θ) T θ -θ , ∀ θ ∈ R d
which is the definition of the sub-gradient.

Using Lem. 3.5.2 in Eq. 3.4, one obtains:

R TS t ≤ 2γ t (δ )E x ( θ) V -1 t F t , θ ∈ Θ opt t 1{ E t }. (3.5)
This property strongly connects the exploration of TS to the actual regret. In fact, together with Prop. 3.2.1, it implies that selecting the optimal arm associated with any optimistic θ is equivalent to reducing the weighted norm of the gradient of J and ultimately the regret R TS t . This motivates the next step where we show that since TS is often optimistic, then the arm x t = x ( θ t ) contributes to the reduction of the regret.

Step 3 (optimism).

The optimism of TS is a direct consequence of the convexity of J and the fact that the distribution of η is oversampling by a factor √ d w.r.t. the ellipsoid E RLS t . Since this is at the core of the TS sampling analysis, we detail the proof here but postpone convexity results in App. 3.B.

Lemma 3.5.3. Let Θ opt t := {θ ∈ R d |J(θ) ≥ J(θ )}∩E TS t be the set of optimistic parameters, θ = θ t + β t (δ )V -1/2 t η with η ∼ D TS , then, on E t , ∀t ≥ 1, P θ ∈ Θ opt t |F t ≥ p/2.
Proof. We need to study the probability that a θ drawn at time t from the TS sampling distribution is optimistic, i.e., J( θ) ≥ J(θ ), under event E t . More formally let

p t = P J( θ) ≥ J(θ )|F t .

Formal proof 43

Using the definition of E t we have that θ ∈ E RLS t (i.e., the true parameter vector belongs to the RLS ellipsoid) and then, under event E t , we can replace J(θ ) by the supremum over the ellipsoid as

p t ≥ P J( θ) ≥ sup θ∈E RLS t J(θ) F t .
By recalling the definition of the TS sampling process, we can write θ = θ t +β t (δ )V -1/2 t η, where η ∼ D TS and for notational convenience, we define the function

f t (η) = J( θ t + β t (δ )V -1/2 t η). Let θ t = arg max θ∈E RLS t J(θ) and η t be the corresponding η (i.e., η t is such that θ t = θ t + β t (δ )V -1/2 t η t ).
Since the supremum is taken within E RLS t , η t belongs to the unit ball (i.e., η t ∈ B d (0, 1)). As a result, we can rewrite the previous expression as

p t ≥ P f t (η) ≥ f t (η t ) F t .
Since the function f t inherits all the properties of J, notably its convexity in η, we know that the supremum on a convex closed set is reached at least at one point ηt and that it belongs to the boundary (see Prop. 3.B.1), which in our case corresponds to η t = 1. Moreover, let H t (η t ) be the hyperplane tangent to η t . H t (η t ) splits R d in two complementary subsets G t and G ⊥ t where G t does not contain the unit ball by convention. Formally, one has:

H t (η t ) := {η ∈ R d s.t. η T ηt = 1}, G t := {η ∈ R d s.t. η T ηt ≥ 1}.
Again, the convexity of f t ensures that f t (η) ≥ f t (η t ) for all η ∈ G t as proved in Prop. 3.B.2. As illustrated in Fig. 3.5 the probability of being optimistic is now reduced to the probability that η drawn from D TS falls into G t , which corresponds to

p t ≥ P η ∈ G t F t = P η T ηt ≥ 1 F t .
Notice that ηt is entirely defined by the filtration F t and the event E t and it is thus independent from η. As a result, we obtain from property 1 of Def. 3.3.1 of the TS sampling distribution, that

P η T ηt ≥ 1 F t ≥ p.
Finally, we show that this property is not affected, up to a second order term, by the high-probability concentration event. It relies on the fact that the chosen confidence level δ = δ/4T is small compared to the anti-concentration probability p of Def. 3.3.1. For sake of simplicity, we assume that T ≥ 1/2p which implies that δ ≤ p/2. For any events A and B, one has

P(A ∩ B) = 1 -P(A c ∪ B c ) ≥ P(A) -P(B c ) E TS t O B d (0, 1) η t H t G t G ⊥ t Figure 3.5 -Illustration of the probability of selecting an optimistic θ t . Since, by Def. 3.3.1, P( θ ∈ E TS t ) ≥ 1 -δ , applying the previous inequality to A := {J( θ) ≥ J(θ )} and B := { θ ∈ E TS t } leads to P( θ t ∈ Θ opt ∩ E TS t |F t ) ≥ p -δ ≥ p/2.
To control the regret, we now make use of the fact that the probability of being optimistic is constant under the event E t . Let g( θ) be an arbitrary non-negative function of θ, then we can write the full expectation as

E[g( θ)|F t ]1{ E t } ≥ E[g( θ t )1{ θ ∈ Θ opt t }|F t ]1{ E t } ≥ E g( θ) θ ∈ Θ opt t , F t P θ ∈ Θ opt t F t 1{ E t } ≥ p/2E g( θ) θ ∈ Θ opt t , F t 1{ E t }. Setting g( θ) = 2γ t (δ ) x ( θ) V -1 t
, we obtains an upper bound for Eq. 3.5 as

R TS t ≤ 4γ t (δ )/p E x ( θ) V -1 t |F t 1{ E t } ≤ 4γ t (δ )/p E x ( θ) V -1 t |F t ,
where 2/p can be interpreted as the expected time between any two optimistic samples. Finally, by construction

θ d = θ t |F t , thus x ( θ) d = x ( θ t )|F t . As a result, R TS t ≤ 4γ t (δ )/p E x ( θ t ) V -1 t |F t ,
and we can use Azuma's inequality to obtain the final bound with probability at least

1 -δ/2 R TS (T ) ≤ 4γ T (δ ) p T t=1 x t V -1 t + 8T λ log 4 δ , (3.6)
where x t is the optimal arm x ( θ t ) selected by TS. The proof is concluded using Cauchy-Schwarz and Prop. 3.2.1 to bound R TS (T ) and Prop. 2.2.1 to bound R RLS (T ).

Final bound

Bounding R RLS (T ). Similar to the sketch in Sect. 3.4, the proof relies on the fact that the RLS guarantees a small on-policy error. Let

R RLS (T ) = T t=1 R RLS t = T t=1 x T t θ t -x T t θ 1{E t },
the bound on R RLS is derived as in previous analysis (Abbasi-Yadkori et al., 2011b, Agrawal andGoyal, 2012b). We decompose the term in a sampling prediction error and a RLS prediction error as follow

R RLS (T ) ≤ T t=1 |x T t ( θ t -θ t )|1{E t } + T t=1 |x T t ( θ t -θ )|1{E t } By definition of the concentration event E t , |x T t ( θ t -θ t )|1{E t } ≤ x t V -1 t γ t (δ ), |x T t ( θ t -θ )|1{E t } ≤ x t V -1 t β t (δ ),
thus, one obtains:

T t=1 R RLS t ≤ γ T (δ ) + β T (δ ) T t=1 x t V -1 t (3.7)
Plugging everything together. Collecting bounds in Eq. 3.6 and Eq. 3.7, applying Cauchy-Schwarz and using Prop. 3.2.1 , one has, with probability at least

1 -δ/2, R RLS (T ) + R TS (T ) ≤ (4/p + 1)γ T (δ ) + β T (δ ) T t=1 x t V -1 t + 4γ T (δ ) p 8T λ log 4 δ ≤ (4/p + 1)γ T (δ ) + β T (δ ) √ T T t=1 x t 2 V -1 t 1/2 + 4γ T (δ ) p 8T λ log 4 δ ≤ (4/p + 1)γ T (δ ) + β T (δ ) 2d log(1 + T /λ) √ T + 4γ T (δ ) p 8T λ log 4 δ
Finally, by Lem. 3.5.1, the event E = t≤T E t holds with probability at least 1 -δ/2. Hence, a union bound argument ensures that with probability at least 1 -δ,

R(T ) ≤ R TS (T ) + R RLS (T ) ≤ β T (δ ) + γ T (δ )(1 + 4/p) 2T d log 1 + T λ + 4γ T (δ ) p 8T λ log 4 δ
where δ = δ 4T which proves Thm. 3.5.1.

Extensions

We provide an alternative proof for TS in LB, leveraging the properties of the optimal value function, the properties of the RLS estimate and the concentration/anticoncentration property of the sampling. As the functioning of the proof does not rely on the specific shape of the J function, we can readily apply it to similar, yet, more general linear problems. We present here two extensions: the regularized linear optimization problem and the generalized linear bandit.

Regularized linear optimization

Our proof holds for any arm set X and the corresponding constrained optimization problem max x∈X x T θ . Similarly, we can apply it to any regularized linear optimization problem max

x∈R d f µ,c (x; θ), with f µ,c (x; θ) = x T θ + µc(x)
, where µ is a constant and c(x) is an arbitrary penalty function of x (e.g., norm-regularization). While there always exists a set of constraints (corresponding to a set of arms X c,µ,θ ) such that the solution to the constrained and regularized problems coincides, such mapping is often unknown (e.g., c(x) = x 1 ) and thus TS cannot be run on X c,µ,θ but we need to directly deal with the regularized problem (i.e., sampling θ t and pulling arm

x t = arg max x f µ,c (x; θ t )).
In this case, it can be seen that the three main steps of our proof still hold. In fact 1) J(θ) is convex, 2) the gradient of J(θ) corresponds to the optimal arm x * (θ), 3) Lemma 3.5.3 holds unchanged since it relies on the convexity of J(θ) and the TS distribution D TS is the same. As a result, the regret bound follows.

Setting. We consider here the Regularized Linear Optimization (RLO) problem as an extension of the Linear Bandit problem. Given a set of arms X ⊂ R d and an unknown parameter θ ∈ R d , a learner aims at each time step t = 1, . . . , T to select action x t ∈ X which maximizes its associated reward x T t θ + µc(x t ) where µ is a known constant and c an arbitrary (yet known) real-valued function. Whenever arm x is pulled, the learner receives a noisy observation y = x T θ + ξ. As for LB, we introduce the function f (x; θ) = x T θ + µc(x), and denote as x (θ) = arg max x∈X f (x; θ) and J(θ) = max x∈X f (x; θ) the optimal action and optimal reward associated with θ. The regret is therefore defined as

R RLO (T ) = T t=1 f (x (θ ); θ ) -f (x t ; θ ).
Since this problem is just the regularized extension of the Linear Bandit, the TS algorithm is similar to Fig. 3.1 where r t is replaced y t and x t = arg max x∈X f (x, θ t ). Under the same assumptions, the regret shares the same bound and our line of proof holds.

Sketch of the proof. First, we decompose the regret

R RLO (T ) = T t=1 (f (x (θ ); θ ) -f (x t ; θ t )) + (f (x t ; θ t ) -f (x t ; θ )) = T t=1 J(θ ) -J( θ t ) =R TS (T ) + T t=1 x T t θ t -x T t θ =R RLS (T )
.

Since Prop. 2.2.1 holds thanks to the linear observations y t , R RLS (T ) is bounded as in the LB analysis. Finally, to bound R TS (T ), one just need to ensure that Prop. 3.5.1, Lem. 3.5.2 and Lem. 3.5.3 hold.

The convexity of the function f with respect to θ implies the convexity of J: ∀x ∈ X , ∀θ, θ ∈ R d , ∀α ∈ (0, 1),

J(αθ + (1 -α)θ ) = max x∈X f (x; αθ + (1 -α)θ ) ≤ max x∈X αf (x; θ) + (1 -α)f (x; θ ) ≤ αJ(θ) + (1 -α)J(θ ).
Then, J is real-valued and convex which implies its continuous differentiability thanks to Alexandrov's theorem. As a consequence, the first step of the proof holds. The equality between the gradient ∇J(θ) and the optimal arm x (θ) can be derived as in Lem. 3.5.2: for any θ, θ ∈ R d , by definition, J(θ) = f (x (θ); θ) and J( θ) ≥ f (x (θ); θ). Then,

J( θ) -f (x (θ), θ) ≥ 0 := J(θ) -f (x (θ), θ), J( θ) ≥ J(θ) + f (x (θ), θ) -f (x (θ), θ) = J(θ) + x (θ) T θ -θ , ∀ θ ∈ R d ,
which is the definition of the sub-gradient. Finally, the almost everywhere differentiability of J ensures the sub-gradient to be a singleton and hence equals the gradient. Therefore, Lem. 3.5.2 holds and so is step 2. Finally, since the optimism just relies on the convexity of J and on the over-sampling, it is satisfied in the RLO and step 3 holds. As a result, we obtain the same regret bound as in the LB.

On the other hand, the original proof by Agrawal and Goyal (2012b) could be less readily applied to this case. First notice that the mapping from µ and c(x) to the constrained set X c,µ,θ requires the unknown parameter θ . This means that if we pass from the regularized problem to the constrained problem at each time step t, we would be working on a set X c,µ, θt which keeps changing over time. While Agrawal and Goyal (2012b) study the contextual bandit problem where X t changes arbitrarily over time, in this case X t would change in response to θ t itself (i.e., it would not be available in advance) and the analysis would bound the per-step regret r t = max x∈X c,µ, θ t

x T θ -x T t θ, which does not correspond to the desired regret on f µ,c (the true optimal arm x (θ ) may not even be in X c,µ, θt ). Alternatively, we need to formulate a suitable definition of saturated and unsaturated arms for f µ,c (x; θ), which does not seem trivial and it may require developing a more ad-hoc analysis.

Generalized linear bandit

Another interesting extension is the generalized linear bandit (GLM) problem of [START_REF] Filippi | Parametric bandits: The generalized linear case[END_REF]. In this setting, the reward associated to arm x ∈ X is no longer drawn from the linear regression model but is generated as r(x) = µ(x T θ ) + ξ, where µ is the so-called link function, θ ∈ R d is a fixed but unknown parameter vector and ξ is a random zero-mean noise. One of the major advantage of this setting is that it encompasses logistic regression. It can model the case when the reward is in [0, 1] and thus became very popular in recommender system where the reward represents the probability of click. Similarly to the regularized optimization problem, a regret bound can be derived for the GLM problem using the same line of proof that we use for LB. It first relies on the fact that, under suitable assumptions about the link function µ, consistent estimates are available for θ together with high probability confidence ellipsoids. Then, we can show that the GLM optimal value function J GLM (θ) := sup x∈X µ(x T θ) is related to the LB optimal value function J(θ) as J GLM (θ) = µ(J(θ)) and thus, link the regret of GLM to the regret of LB. We present here how to apply our derivation to the generalized linear bandit (GLM) problem of [START_REF] Filippi | Parametric bandits: The generalized linear case[END_REF]. The regret bound is obtained by basically showing that the GLM problem can be reduced to studying the linear case.

The setting. Let X ⊂ R d be an arbitrary (finite or infinite) set of arms. Every time an arm x ∈ X is pulled, a reward is generated as r(x) = µ(x T θ ) + ξ, where µ is the so-called link function, θ ∈ R d is a fixed but unknown parameter vector and ξ is a random zero-mean noise. The value of an arm x ∈ X is evaluated according to its expected reward µ(x T θ ) and for any parameter θ ∈ R d we denote the optimal arm and its optimal value as

x ,GLM (θ) = arg max x∈X µ(x T θ), J GLM (θ) = sup x∈X µ(x T θ).
Then x = x ,GLM (θ ) is the optimal arm associated with the true parameter θ and J GLM (θ ) its optimal value. At each step t, a learner chooses an arm x t ∈ X using all the information observed so far (i.e., sequence of arms and rewards) but without knowing θ and x . At step t, the learner suffers an instantaneous regret corresponding to the difference between the expected rewards of the optimal arm x and the arm x t played at time t. The objective of the learner is to minimize the cumulative regret up to a finite step T ,

R GLM (T ) = T t=1 µ(x ,T θ ) -µ(x T t θ ) .
Assumptions. The assumptions associated with this more general problem are the same as in the linear bandit problem plus one regarding the link function. Formally, we require assumption 3.2.1, 3.2.2 and 3.2.3 and add: Assumption 3.6.1 (link function). The link function µ : R → R is continuously differentiable, Lipschitz with constant k µ and such that c µ = inf θ∈R d ,x∈X µ(x T θ) > 0.

Technical tools. Let (x 1 , . . . , x t ) ∈ X t be a sequence of arms and (r 2 , . . . , r t+1 ) be the corresponding observed (random) rewards, then the unknown parameter θ can be estimated by GLM estimator. Following [START_REF] Filippi | Parametric bandits: The generalized linear case[END_REF] one gets, for any regularization parameter λ ∈ R + ,

θ GLM t = arg min θ∈R d t-1 s=1 r s+1 -µ(x T s θ) x s 2 V -1 t , (3.8)
where V t is the same design matrix as in the linear case. Similarly to Prop. 2.2.1, we have a concentration inequality for the GLM estimate.

Proposition 3.6.1 (Prop. 1 in appendix.A in [START_REF] Filippi | Parametric bandits: The generalized linear case[END_REF]). For any δ ∈ (0, 1), under assumptions 3.2.1, 3.2.2, 3.2.3 and 3.6.1, for any F x t -adapted sequence (x 1 , . . . , x t , . . .), the prediction returned by the GLM estimator θ GLM t (Eq. 3.8) is such that for any fixed t ≥ 1,

θ GLM t -θ Vt ≤ β t (δ) c µ ,
and

∀x ∈ R d , µ(x T θ GLM t ) -µ(x T θ ) ≤ k µ β t (δ) c µ x V -1 t , x T θ GLM t -x T θ ≤ β t (δ) c µ x V -1 t ,
with probability 1 -δ (w.r.t. the noise sequence {ξ t } t and any other source of randomization in the definition of the sequence of arms), where β t (δ) is defined as in Eq. 3.3.

The Asm. 3.6.1 on the link function together with the properties of the GLM estimator implies the following: 1) since the first derivative is strictly positive, µ is strictly increasing and x ,GLM (θ) = arg max x∈X x T θ = x (θ) so we retrieve the optimal arm of the linear case. Similarly, J GLM (θ) = µ(J(θ)); 2) the concentration inequality of the GLM estimate involves the same ellipsoid as for the RLS (multiplied by a factor 1 cµ ). These two facts suggest to use then exactly the same TS algorithm as for the linear case (with a β t (δ ) multiplied by a factor 1 cµ , where δ = δ 4T ).

Sketch of the proof. From the previous comments, making use of the property of µ, one just need to reduce the GLM case to the standard linear case.

R GLM (T ) = T t=1 µ(x θ ) -µ(x T t θ ) , = T t=1 µ(x θ ) -µ(x T t θt ) + T t=1 µ(x T t θt ) -µ(x T t θ ) ≤ T t=1 µ(x θ ) -µ(x T t θt ) + T t=1 k µ x V -1 t θt -θ Vt .
The second term is bounded exactly as R RLS (T ). To bound the first one, we make use of the fact that

µ(x θ ) -µ(x T t θt ) ≤ k µ J(θ ) -J( θt ) , ifJ(θ ) -J( θt ) ≥ 0, µ(x θ ) -µ(x T t θt ) ≤ c µ J(θ ) -J( θt ) , otherwise.
Following the proof of the linear case, with high probability, for all t ≥ 1,

J(θ ) -J( θt ) ≤ 2γ t (δ ) c µ p E x t V -1 t |F t .
Since the r.h.s is strictly positive one can bound the first part of the regret, independently of the sign by,

T t=1 µ(x θ ) -µ(x T t θt ) ≤ 2k µ γ T (δ ) c µ p T t=1 E x t V -1 t |F t .
Finally, the same proof as in the linear case leads to the following bound for the Generalized Linear Bandit regret.

Lemma 3.6.1. Under assumptions 3.2.1,3.2.2,3.2.3 and 3.6.1, the cumulative regret of TS over T steps is bounded as

R GLM (T ) ≤ k µ c µ β T (δ ) + γ T (δ )(1 + 2/p) 2T d log 1 + T λ + 2k µ γ T (δ ) pc µ 8T λ log 4 δ
with probability 1 -δ where δ = δ 4T .

Other extensions

To go further, we can generalize our proof to the other convex optimization problems max x∈X f (x, θ), with linear observations (i.e., y = x T θ + ξ). If f (x, θ) is convex in θ, then J(θ) is convex as well, thus enabling the possibility to apply our line of proof. More precisely, the gradient of J to the arms played by TS should be related (step 2, Lem. 3.5.2) and the on-policy prediction error R RLS measured w.r.t. f should be bounded (Prop. 2.2.1). Whenever these properties are satisfied, the regret result follows.

Notice that while the original proof by Agrawal and Goyal (2012b) may be extended to cover some of these problems, its requirements are slightly stronger. In fact, the definition of saturated and unsaturated arms relies on the fact that f (x, θ n ) concentrates to f (x, θ) for any x, while in our case, we only need to bound R RLS , which corresponds to an on-policy error, where prediction errors are measured on the specific arms selected by the algorithm. While this advantage may appear abstract, let consider the reinforcement learning case, where f (x, θ) is the value function of a policy x in an environment θ.

In this case, f (x, θ ) may actually be unbounded for some x (i.e., the policy x does not control the system) and the definition of saturated/unsaturated arms could not be easily adjusted. This suggests that our proof could enable covering special RL cases as well. Finally, we remark that defining TS as a randomized algorithm and using convex geometry arguments in its analysis bears a strong resemblance with follow-the-perturbedleader algorithm and its regret analysis in adversarial linear bandit [START_REF] Abernethy | Fighting bandits with a new kind of smoothness[END_REF], suggesting that the two approaches may be strongly related.

Discussion

We developed an alternative proof for TS in LB with novel insights on the core elements of the algorithm (optimism) and the structure of the problem (support function J(θ)).

There are a number of possible applications of our results and future directions of investigation. The main open question is whether or not oversampling is needed to guarantee a √ T regret bound for TS. Since this worsens the bound by √ d, answering this question could improve the current frequentist bound from Õ(d 3/2 √ T ) to Õ(d √ T ), thus matching the bound achieved by OFUL. We first present numerical experiments that compare the Bayesian and frequentist versions of TS and exhibit the dependency of the constant of the frequentist regret w.r.t d. Then, we stress why oversampling is needed in the current analysis and discuss how to relax it.

Numerical experiments. To understand the impact of the oversampling, we compare two instances of the TS algorithm:

1) we denote as F reqT S the instance of the algorithm where, at each time step, the parameter is sampled as

θ t = θ t + β t V -1/2 t η with η ∼ N (0, I), for which we prove a Õ d 3/2 √ T regret bound,
2) we denote as BayesT S its Bayesian counterpart where, at each time step, the parameter is sampled as

θ t = θ t +V -1/2 t
η with η ∼ N (0, I), for which no frequentist regret guarantee exists.

We compute the regret over trajectories of length T = 200000 for values of d spanning [0, 30] and present the results in Fig. 3.6. The motivation for such long trajectories is that the regret curves exhibit slightly different regimes (w.r.t. t). Since we focus on the dependency on d, we discard this effect ensuring that each trajectory reaches the asymptotic regime. The parameter θ is fixed at the beginning of each trajectory as θ = (1, 0, . . . , 0). The reason for imposing θ = 1 is to remove the dependency on d in the norm of θ , which affects the regret through the constant S of Asm. 3.2.2. The RLS estimation is initialized as V 0 = λI with λ = 1 and θ 0 is randomly chosen on the unite sphere. Finally, the reward noise sequences {ξ t } t are generated i.i.d according to ξ t ∼ N (0, 1).

The intuition provided by this experiment is twofold: first, it stresses that the Õ d 3/2 √ T regret bound of the F reqT S algorithm is tight, so a factor √ d cannot be removed by a different analysis; secondly, it suggests that no oversampling is needed to guarantee a √ T regret and that a Õ d √ T regret bound could be derived for the BayesT S algorithm.

About optimism and oversampling. As illustrated in Sect. 3.4, in the current proof optimistic steps allows to bound the regret of non-optimistic steps. Nonetheless, it can be shown that some non-optimistic steps (even very pessimistic!) may indeed be as "informative" as optimistic steps and allow reducing the regret as well. a minor change in the line of proof, anticipating the use of the convexity of J, i.e.,

R TS t ≤ sup θ∈E TS t ∇J(θ ) T (θ -θ)1{E t } ≤ ∇J(θ ) V -1 t 2γ t (δ )1{E t }.
If we sample a θ such that the gradient at it ∇J( θ) (i.e., which coincides with the corresponding optimal action x ( θ)) has the same V -1 t -norm as ∇J(θ ), then we could apply the same reasoning as in the original sketch of the proof and bound the regret of any subsequent step. More formally, we can define the set Θ grad

t = {θ : ∇J(θ) V -1 t ≥ ∇J(θ ) V -1 t
} of parameters that have larger gradient than θ 's. Similar to Θ opt , if the probability of sampling θ in Θ grad t is lower-bounded by a constant p , then the proof can be reproduced with exactly the same arguments and result. Even further, we could relax the requirement and define Θ grad

t (α) = {θ : ∇J(θ) V -1 t ≥ α ∇J(θ ) V -1 t
}, with α < 1, which would allow even a bigger probability at the cost of an extra constant factor α in the final regret.

As illustrated in Fig. 3.7, in the case X = B d (0, 1), Θ grad t (α) corresponds to a cone whose overlap with E TS may actually be even larger than for Θ opt . This illustration shows that the set of useful explorative actions does not necessarily coincide with the set of optimistic parameters and that many more parameters in E TS may contribute to reduce the regret. This may explain the empirical success of TS and it may suggest that the oversampling by a factor √ d to ensure optimism may be a too strong requirement. Finally, we remark that a similar optimistic argument is employed by [START_REF] Agrawal | Further optimal regret bounds for thompson sampling[END_REF] in MAB. Nonetheless, in Lemma 2 they prove that the probability of being optimistic increases over time. This may suggest that E TS needs to be only a constant fraction bigger than E RLS , since the initial small probability of being optimistic would tend to a constant (or even to 1) later on during the learning process. Whether this argument holds and how to prove it remains an open question.

Discussion

53

E RLS t θ t p p θ t θ * E TS t O X x t = x ( θ t ) Θ opt Θ grad (α)
x

P(u T η ≥ 1) = 1 2 I 1-1 d d + 1 2 , 1 2
,
where I x (a, b) is the incomplete regularized beta function. In Prop. 3.E.2 we prove that

I 1-1 d d + 1 2 , 1 2 ≥ 1 8 √ 3π ,
and hence we obtain p = 1 16 √ 3π . Example 2: Gaussian case η ∼ N (0, I d ). The concentration property comes directly from the Chernoff bound for standard Gaussian random variable together with union bound argument. For any α > 0, we have

P( η ≤ α √ d) ≥ P(∀1 ≤ i ≤ d, |η i | ≤ α) ≥ 1 -dP(|η i | ≥ α).
Standard concentration inequality for Gaussian random variable gives, ∀α > 0,

P(|η i | ≥ α) ≤ 2e -α 2 /2 .
Plugging everything together with α = 2 log 2d δ gives the desired result with c = c = 2. Let η i be the i-th component of η for any 1 ≤ i ≤ d. Then η i ∼ N (0, 1). Since η is rotationally invariant, for any direction u of R d and an appropriate choice of basis, we have P(u T η ≥ 1) ≥ P(η 1 ≥ 1). From standard Gaussian properties (see Thm 2 of Chang et al. ( 2011)) we have

P(η 1 ≥ 1) = 1 2 erfc 1 √ 2 ≥ 1 4 √ eπ
which ensures the anti-concentration property with p = 1 4 √ eπ . Then define y = x + (x -x) for some x ∈ bound(C). By definition of the open set int(C), ∃ > 0 such that y ∈ int(C). Moreover, x ∈ [y, x] i.e.,

3.B Properties of convex function

x = (1 -t)x + ty, t = 1 1 + ∈]0, 1[
Using the convexity of f , one has

f (x ) ≤ (1 -t)f (x) + tf (y) < (1 -t)f (x ) + tf (y) ⇒ f (x ) < f (y)
which is impossible by assumption.

Proposition 3.B.2. Let f : R d → R be a convex function. Let B d (0, 1) be the unit d-dimensional ball and S d (0, 1) the associated unit sphere. Let x ∈ S d (0, 1) such that f (x ) ≥ f (x) for all x ∈ B d (0, 1), and let H(x ) be the hyperplan tangent to B d (0, 1) at the point x , which splits R d into two complementary subsets G(x ) and G ⊥ (x ) defined respectively by

H(x ) = {x ∈ R d s.t. x T x = 1}, G(x ) = {x ∈ R d s.t. x T x ≥ 1}, G(x ) ⊥ = {x ∈ R d s.t. x T x < 1}.
Then, ∀y ∈ G(x ), f (y) ≥ f (x ).

Proof. We first notice that from Proposition 3.B.1 x is well defined since the maximum is reached on the boundary. The associated subspace G(x ) is then

G(x ) := {y = x + u, u ∈ R d | u T x ≥ 0}.
We want to show that f (y) ≥ f (x ) for any y ∈ G(x ). We introduce the increasing sequence of subspace

G n = y = x + u, u ∈ R d | u T x ≥ u 2(n -1)
, n ≥ 2.

For any

y = x + u in G n , we associate x = x -1 2(n-1) u u
. By definition of y (and hence u), we have

x 2 = 1 + 1 2(n -1) 2 - 1 2(n -1) u u T x = 1 + 1 2(n -1) 1 2(n -1) - u T u x ≤ 1, which means that x ∈ B d (0, 1). Moreover let t = [2(n -1) u + 1] -1 , t ∈]0, 1[ one has x = (1 -t)x + ty. Since x ∈ B d (0, 1) then f (x ) ≤ (1 -t)f (x) + tf (y) ≤ (1 -t)f (x ) + tf (y) ⇒ f (x ) ≤ f (y).
Since the statement of the proposition holds for any G n , then we obtain the desired result for G(x ) by continuity of f . Let y ∈ G(x ), y = x + u. If u T x > 0, then ∃n ≥ 2 such that y ∈ G n and the proposition is satisfied. Otherwise, if u T x = 0, we introduce the sequences {u n } and {y n } defined as:

u n = u + u 1 -1 2(n-1) 2 x 2(n -1) = u + u n 2(n -1)
x ,

y n = x + u n .
By construction, y n ∈ G n and y n → y as n → ∞. Since the f (y n ) ≥ f (x ) for any n ≥ 2 we obtain the desired result taking the limit since f is continuous as a convex function on R d .

Theorem 3.B.1 (A.D. Alexandrov). Let f : R d → R be a convex function, then it is twice differentiable almost everywhere with respect to the Lebesgue's measure.

Proof. This result is an extension of the Rademacher's theorem for convex functions. A proof can be found in [START_REF] Niculescu | Convex functions and their applications: a contemporary approach[END_REF], Thm. 3.11.2.

3.C Properties of support function (proof of Proposition 3.5.1)

We study the support function of a set C, which is a function

f C : R d → R such that f C (θ) = sup x∈C x T θ
Those functions are at the core of convex geometry analysis.

Proposition 3.C.1. Let C ⊂ R d be a non-empty compact set and f C the associated support function. Then,

1. f C is real-valued and sup x∈C x T θ is attained in C, 2. f C is convex, 3. f C is continuous on R d
and twice differentiable almost everywhere with respect to the Lebesgue's measure.

Proof.

1. This comes directly from the compactness of C: since C is bounded, the support function is real-valued and since C is closed, the supremum is attained in C, 2. Let θ 1 , θ 2 two vectors of R d , and t ∈ (0, 1). By definition of the supremum, since f C is real-valued: We first bound the two events separately.

f C (tθ 1 + (1 -t)θ 2 ) = sup x∈C tx T θ 1 + (1 -t)x T θ 2 ≤ t sup x∈C x T θ 1 + (1 -t) sup x∈C x T θ 2 3.
Bounding E. This bound is a straightforward application of Proposition 2.2.1 together with a union bound argument. Let δ = δ/(4T ), then

∀1 ≤ t ≤ T, P θ t -θ Vt ≤ β t (δ ) ≥ 1 -δ from union bound, P T t=1 θ t -θ Vt ≤ β t (δ ) ≥ 1 - T t=1 P θ t -θ Vt ≥ β t (δ ) ⇒ P T t=1 θ t -θ Vt ≤ β t (δ ) ≥ 1 - T t=1 δ ⇒ P E ≥ 1 -T δ = 1 - δ 4 .
Bounding E. This bound comes directly from the concentration property of the TS sampling distribution. From the expression of θ t = θ t + β t (δ )V -1/2 t η t where η t is drawn i.i.d. from D TS , we have

∀1 ≤ t ≤ T, P   θ t -θ t Vt ≤ β t (δ ) cd log c d δ   = P   η t ≤ cd log c d δ   .
Then from Definition 3.3.1, we have

P   η t ≤ cd log c d δ   ≥ 1 -δ .
As before, a union bound over the two bounds ensures that P( E) ≥ 1 -T δ = 1 -δ 4 . Finally, a union bound argument between the two terms leads to P( E ∩ E) ≥ 1 -δ 2 .

3.E Hyperspherical cap and beta function

Proposition 3.E.1. Let V d (R) be the volume of the d-dimensional ball of radius R and let V cap d (h) the volume of the hyperspherical cap of height h = R -r > 0. Then, V cap d (h) = 1 2 V d (R)I 1-( r R ) 2 d + 1 2 , 1 2
where I x (a, b) is the incomplete regularized beta function.

Proof. The proof can be found in [START_REF] Li | Concise formulas for the area and volume of a hyperspherical cap[END_REF].

Proposition 3.E.2. Let I x (a, b) is the incomplete regularized beta function, ∀d ≥ 2, I 1-1 d d + 1 2 , 1 2 ≥ 1 8 √ 3π 58
B x (a, b) = x 0 t a-1 (1 -t) b-1 dt, B(a, b) = B 1 (a, b), I x (a, b) = B x (a, b) B(a, b)
Hence we seek for a lower bound on B 1-

1 d d+1
2 , 1 2 and an upper bound for B d+1 2 , 1 2 .

1. Let first find an lower bound for the incomplete beta function. Since t → t d-1

2 (1 -t) -1/2 is positive and increasing on [0, 1], for any d ≥ 2,

B 1-1 d d + 1 2 , 1 2 ≥ 1-d 2 1-3 2d t d-1 2 (1 -t) -1/2 dt ≥ 1 2d 3 2d -1/2 1 - 3 2d d-1 2 ≥ 1 √ 6d 1 - 3 2d d-1 2 ≥ 1 √ 6d 1 - 3 2d d 2
From the increasing property of x → (1 -α x ) x for any α < 1 the sequence 1

-3 2d d 2 d≥2
is increasing and

B 1-1 d d + 1 2 , 1 2 ≥ 1 √ 6d 1 - 3 2 × 2 2 2 = 1 4 √ 6d
2. Now we seek for an upper bound for

B d+1 2 , 1 2 . Since B(a, b) = Γ(a)Γ(b)
Γ(a+b) one has:

B d + 1 2 , 1 2 = Γ 1 2 Γ d+1 2 Γ d 2 + 1 = √ π Γ d+1 2 Γ d 2 + 1
From [START_REF] Chen | Completely monotonic function associated with the gamma functions and proof of wallis' inequality[END_REF], we have the following inequalities for the gamma function ∀n ≥ 1:

Γ(n + 1/2) Γ(n + 1) ≤ (n + 1/4) -1/2 Γ(n + 1/2) Γ(n + 1) ≥ (n + 4/π -1) -1/2
Together with Γ(x + 1) = xΓ(x) and treating separately cases where d is even or not, one gets ∀d ≥ 2 Γ d+1 2

Γ d 2 + 1 ≤ 2 d 3.
Using the obtained upper and lower bound we get:

I 1-1 d d + 1 2 , 1 2 ≥ √ d √ 2π × 4 √ 6d ≥ 1 8 √ 3π
Chapter 4

Thompson Sampling in Linear Quadratic System

We now consider the exploration-exploitation tradeoff in linear quadratic (LQ) control problems, where the state dynamics is linear and the cost function is quadratic in the state and control. We analyze the regret of Thompson sampling (TS) (a.k.a. posteriorsampling for reinforcement learning) in the frequentist setting, i.e., when the parameters characterizing the LQ dynamics are fixed. Despite the empirical and theoretical success in a wide range of problems from multi-armed bandit to linear bandit, extending those results to the LQ setting is highly challenging: 1) standard line of proof that relies on classifying arms into saturated/unsaturated pool cannot be applied here as their associated optimal value could be infinite; 2) the TS functioning requires frequent policy updates, which is in contrast with the usual lazy update scheme used in most RL algorithm. As a consequence, it raises the issue of bounding the gap in the optimal value at the policy switches.

In the chapter1 , we prove that indeed TS achieves a O( √ T ) regret in LQ problems, thus matching the performance of the OFU approach and confirming the conjecture of [START_REF] Osband | Posterior sampling for reinforcement learning without episodes[END_REF]. We address the first point leveraging the ideas introduced in Ch. 3, stressing the link between the actual actions chosen by TS and the gradient of the optimal value function. We exhibit the need to trade-off the frequency of sampling optimistic parameters and the frequency of switches in the control policy, and show that lazy update schemes induces at best an overall regret of Ω(T 2/3 ). Finally, we derive novel bound on the regret due to policy switches, thus allowing to update parameters and the policy at each step and overcome the limitations due to lazy updates.

Introduction

Designing algorithms to properly trade off exploration of an unknown environment and exploitation of the estimated optimal control policy is one of the most important challenges towards scaling reinforcement learning (RL) [START_REF] Sutton | Introduction to Reinforcement Learning[END_REF] to problems with large and/or continuous state and action spaces. To this end, we focus in this chapter, on a specific family of continuous state-action MDPs, the linear quadratic (LQ) control problems introduce in Sec. 2.3, where the state transition is linear and the cost function is quadratic in the state and the control. Despite their specific structure, LQ models are very flexible and widely used in practice (e.g., to track a reference trajectory). If the parameter θ defining dynamics and cost is known, the optimal control can be computed explicitly as a linear function of the state with an appropriate gain. On the other hand, when θ is unknown, an exploration-exploitation trade-off needs to be solved. [START_REF] Bittanti | Adaptive control of linear time invariant systems: the "bet on the best" principle[END_REF] and [START_REF] Campi | Adaptive linear quadratic gaussian control: the cost-biased approach revisited[END_REF], first proposed an optimistic approach to this problem, showing that the performance of an adaptive control strategy asymptotically converges to the optimal control. Building on this approach and the OFU principle, Abbasi-Yadkori and Szepesvári (2011) proposed a learning algorithm (OFU-LQ) with O( √ T ) cumulative regret. Abbasi-Yadkori and Szepesvári (2015) further studied how the TS strategy, could be adapted to work in the LQ control problem, but due to the difficulty to move from episodic to infinite horizon, no regret guarantee is available, either in a Bayesian or a frequentist sense.

Contributions.

In this chapter, we analyze the regret of TS in LQ problems in the frequentist case2 , where θ is a fixed parameter, with no prior assumption of its value, and prove a O( √ T ) regret bound for the 1-dimensional case (i.e., states are one dimensional). The analysis of OFU-LQ relies on three main ingredients: 1) optimistic parameters, 2) lazy updates (the control policy is updated only a logarithmic number of times) and 3) concentration inequalities for regularized least-squares used to estimate the unknown parameter θ. While we build on previous results for the least-squares estimates of the parameters, points 1) and 2) should be adapted for TS. Unfortunately, the existing frequentist regret analysis for TS in linear bandit due to Agrawal and Goyal (2012b) cannot be generalized to the LQ case. Leveraging the novel analysis of TS for LB presented in Ch. 3, we first prove that TS has a constant probability to sample an optimistic parameter (i.e., an LQ system whose optimal expected average cost is smaller than the true one) and then we exploit the LQ structure to show how being optimistic allows to directly link the regret to the controls operated by TS over time and eventually bound them. Nonetheless, this analysis reveals a critical trade-off between the frequency with which new parameters are sampled (and thus the chance of being optimistic) and the regret cumulated every time the control policy changes. In OFU-LQ this trade-off is easily solved by construction: the lazy update guarantees that the control policy changes very rarely and whenever a new policy is computed, it is guaranteed to be optimistic. On the other hand, TS relies on the random sampling process to obtain optimistic models and if this is not done frequently enough, the regret can grow unbounded which forces TS to favor short episodes. We first show that, sticking to lazy updates, the regret guarantee scales in O(T 2/3 ) at best, and then prove a O( √ T ) bounds in the frequentist regret when the policy updates are performed at each time step, thus confirming the conjecture in [START_REF] Osband | Posterior sampling for reinforcement learning without episodes[END_REF]. This result is enabled by a novel lemma that bounds the regret suffered at the switch between two episodes. We show that the regret incurred at policy switches is somehow related to the overall regret and that it can be bounded following similar steps. As a result, we are able to reduce the length of episodes even further (i.e., constant length or even one single step) at no additional cost and fully exploit the optimism of TS.

Preliminaries

We briefly recall the setting for the LQ problem introduced in Subsec. 2.3.2 and detail the assumptions that we impose on the problem structure as well as the additional material needed for our analysis. Most of the notations in this section are adapted from (Abbasi-Yadkori and Szepesvári, 2011).

The control problem. We consider the discrete-time infinite-horizon linear quadratic (LQ) control problem with state x ∈ R n and control u ∈ R d . Given state x t and control u t at time t, the next state and cost are computed as:

x t+1 = A * x t + B * u t + t+1 ; c(x t , u t ) = x T t Qx t + u T t Ru t , (4.1)
where A * , B * , Q, R are matrices of appropriate dimension and { t+1 } t is a zero-mean process. Following the setting of Abbasi-Yadkori and Szepesvári (2011), we assume Q and R are known, while the unknown parameters are summarized in θ T * = (A * , B * ). The objective of the learner is to find a stationary deterministic control policy π : R n → R d mapping states to controls that minimizes the asymptotic (i.e., infinite horizon) average expected cost

J π (θ * ) = lim sup T →∞ 1 T E T t=0 c(x t , u t ) , (4.2)
with x 0 = 0 and u t = π(x t ). We denote as π * (θ * ) the optimal policy of the LQ problem parametrized by θ * . We impose the following assumptions over the noise process and the linear system of Eq. 4.1.

K(θ * ) = -R + B T P (θ * )B -1 B T P (θ * )A, P (θ * ) = Q + A T P (θ * )A + A T P (θ * )BK(θ * ),
The optimal average cost is J * = J π * (θ * ) = Tr(P (θ * )). Finally, we also have that the closed-loop matrix A * + B * K(θ * ) is asymptotically stable. For sake of compactness, we introduce the matrix

H(θ * ) = I K(θ * ) T T
and rewrite the closed-loop matrix as

A * + B * K(θ * ) = θ T * H(θ * ).
We now construct a constraint set S such that for any θ ∈ S, there exists an optimal control K(θ). This is done by rejecting the pair (A, B) that are non-stabilizable, i.e., the one for which there exists no linear controller K such that the closed-loop matrix A+BK is asymptotically stable (i.e., has eigenvalues in the open unit disk). Fortunately, this set is of zero Lebesgue measure as provided by the following proposition.

Proposition 4.2.1 (Cor. 12 in [START_REF] Klamka | Controllability of dynamical systems[END_REF]). For given dimensions n and d, the set of dynamical systems which are controllable is open and dense in the space R n(n+d) of all dynamical system of the form (4.1).

Since controllability implies stabilizability (see Def. 2.3.2), the set of uncontrollable system is of zero Lebesgue measure and so is the set of unstabilizable system. Moreover, when the pair (A, B) is not stabilizable, there exists no control K such that A + BK is stable, thus, under any linear controller, the state process x t diverges exponentially. As a result, the associated "optimal" average cost is J(θ) = +∞. This property has two major implications: First, it is possible to define the optimal value function over the whole space R n(n+d) in a continuous manner by setting its value to +∞ wherever θ is a non-stabilizable pair; Second, for any sampling distribution absolutely continuous w.r.t the Lebesgue measure, the associated optimal value function is finite with probability one. However, it might sample parameters that are almost non-stabilizable (and hence of large optimal cost), which is still harmful from both theoretical and practical perspective. This motivates the introduction of the constraint set S defined as:

Definition 4.2.1. S = {θ ∈ R (n+d)×n s.t. J(θ) = Tr(P (θ)) ≤ D and Tr(θθ T ) ≤ S 2 }
This definition is implicit since it involves the optimal average cost but offers the advantages of unifying assumption A-2 and A-4 in (Abbasi-Yadkori and Szepesvári, 2011) in a tight way and implies the following guarantees. Proposition 4.2.2. S is a compact set. For any θ ∈ S, θ is a stabilizable pair (since J(θ) = +∞ otherwise) and there exist ρ < 1 and C < ∞ positive constants such that ρ = sup θ∈S A + BK(A, B) 2 and C = sup θ∈S K(θ) 2 .

Further, we assume that the true parameter belongs to the constraint set. Formally,

Preliminaries 63

Assumption 4.2.3. Let S be defined as in Def. 4.2.1, then θ * ∈ S.

Finally, we recall a result about the regularity of the Riccati solution.

Proposition 4.2.3 (proof in App. 4.A). Under Asm. 4.2.1 and for any LQ with parameters θ T = (A, B) and cost matrices Q and R satisfying Asm. 4.2.2, let J(θ) = Tr(P (θ)) be the optimal solution of Eq. 4.2. Then, the mapping θ ∈ S → Tr(P (θ)) is continuously differentiable. Furthermore, let A c (θ) = θ T H(θ) be the closed-loop matrix, then the directional derivative of P (θ) in a direction δθ, denoted as dP (θ)(δθ), where dP (θ)(δθ) ∈ R n×n , is the solution of the Lyapunov equation

dP (θ)(δθ) = A c (θ) T dP (θ)(δθ)A c (θ) + C(θ, δθ) + C(θ, δθ) T , where C(θ, δθ) = A c (θ) T P (θ)δθ T H(θ).
The learning problem. We consider the standard online learning setting where at each step t the learner receives the current state x t as input, it executes a control u t and it observes the associated cost c(x t , u t ); the system then transitions to the next state x t+1 according to Eq. 4.1. The learning performance is measured by the cumulative regret over T steps, where the costs cumulated over time are compared to the minimal cost obtained on average by the optimal policy. Formally we define

R T (θ * ) = T t=0 c t -J * (θ * ) .
Independently from the control problem, we need basic tools for the estimation of the parameter θ * . Let (u 0 , . . . , u t ) be a sequence of controls and (x 0 , x 1 , . . . , x t+1 ) be the corresponding states generated according to Eq. 4.1. For any regularization parameter λ ∈ R * + the regularized least-squares (RLS) and the associated design matrix are defined as

V t = λI + t-1 s=0 z s z T s ; θ t = V -1 t t-1 s=0 z s x T s+1 , ( 4.3)
where z t = (x t , u t ) T . While the RLS estimate of Eq. 4.3 slightly differs from the one of Ch. 3 since it is derived in matrix form, the concentration inequality still hold (see Prop. 2.3.1). Thus, for any 0 < δ < 1, we define the high-probability ellipsoid E RLS t at each time step t as

E RLS t = θ ∈ R n(n+d) θ-θ t Vt ≤ β t (δ ) , β t (δ ) = n 2 log det(V t ) 1/2 det(λI) 1/2 δ +λ 1/2 S, (4.4)
where δ = δ 8T and • M denote the weighted Frobenius norm associated with any positive definite matrix M , such that, for any θ ∈ R n(n+d) , θ 2 M = Tr θ T M θ . Under Asm. 4.2.1 and 4.2.3, Prop. 2.3.1 guarantees that θ ∈ E RLS t for all t ≤ T , with probability at least 1 -δ/8. Finally, we also have the standard result of RLS that, together with Prop. 2.3.1, shows that the prediction error on the points z t used to construct the estimator θ t is cumulatively small. Proposition 4.2.4 (Lem. 10 in (Abbasi-Yadkori and Szepesvári, 2011)). Let λ ≥ 1, for any arbitrary F t -adapted sequence (z 0 , z 1 , . . . , z t ), let V t+1 be the corresponding design matrix, then

t s=0 min z s 2 V -1 s , 1 ≤ 2 log det(V t+1 ) det(λI)
.

Moreover when z t ≤ Z for all t ≥ 0, then and

t s=0 z s 2 V -1 s ≤ 2 Z 2 λ (n + d) log 1 + (t + 1)Z 2 λ(n + d) β t ≤ nZ (n + d) λ log 1 + (t + 1)Z 2 λ(n + d) + 2 log(1/δ) 1/2 + λ 1/2 S.

Thompson sampling for LQ

We introduce a specific instance of TS for learning in LQ problems obtained as a modification of the algorithm proposed by Abbasi-Yadkori and Szepesvári ( 2015), where we replace the Bayesian structure and the Gaussian prior assumption with a randomized Gaussian process and we modify the update rule. The algorithm is summarized in Fig. 4.1. At any step t, given the RLS-estimate θ t and the design matrix V t , TS samples a perturbed parameter θ t . In order to ensure that the sampling parameter is indeed admissible, we re-sample it until a valid θ t ∈ S is obtained.We define θ t as

θ t = R S ( θ t + β t V -1/2 t η t ), (4.5)
where R S is the rejection sampling operator associated with the admissible set S, θ t is the RLS-estimate, V t is the design matrix and each entry of the perturbation matrix η t ∈ R (n+d)×n is a random sample drawn i.i.d. from N (0, 1). Then the control u t = K( θ t )x t is executed and the next state x t+1 and c t are observed. The new samples are then used to update θ t and V t .

Input: θ0 , V 0 = λI, δ, T , t 0 = 0 1: Set β t = β t (δ ) where δ = δ 8T according to Eq. 4.4 2: for t = {0, . . . , T } do

3: Sample θ t = R S ( θ t + β t V -1/2 t η t ) 4: Execute control u t = K( θ t )x t 5:
Observe state x t+1 and cost c t = c(x t , u t ) 6:

Compute V t+1 and θ t+1 using Eq. 4.3 7: end for The major difference of this instance of TS w.r.t. OFU-LQ and the algorithms of Abbasi-Yadkori and Szepesvári ( 2015) is that we are no longer using a lazy-update scheme and the policy is updated at each step. As shown in the Subsec. 4.5.4, this is possible because of the novel analysis of the regret suffered at each policy switch. The algorithm in Fig. 4.1 can be turned into a "proper" Bayesian algorithm if we follow the same approach as in (Abbasi-Yadkori and Szepesvári, 2015) and define a prior P 0 over θ * as a multivariate Gaussian conditioned on θ * ∈ S. More formally, let [θ * ] i be the i-th column of θ * then the density of the prior is proportional to n i=1 exp λ[θ * ] T i [θ * ] i 1{θ * ∈ S} where the indicator function guarantees that the resulting system is in the admissible set. As a result, Eq. 4.5 with β t = 1 is indeed the posterior over θ * at time t. On the other hand, TS can be seen as a randomized algorithm as suggested in Ch. 3. In this case, no prior is needed and the factor β t in Eq. 4.4 is allowed to change as in Eq. 2.8. We consider the latter instance here and use β t = β t (δ ) with δ = δ 8T .

Challenges and sketch of the proof

In this section, we highlights the challenges of proving a regret bound for TS in LQ problem, stressing the differences with the Bayes and OFU-LQ analysis. Then, we report a sketch of the proof, leveraging ideas presented in Ch. 3, to provide intuition on the behavior of TS. For the sake of illustration, we postpone the rigorous proof to Sec. 4.5. The following analysis only holds on some high probability events and we denote as , a numerical constant that varies from line to line. We start by decomposing the regret similarly to Abbasi-Yadkori and Szepesvári (2011, Sec. 4.2).

R(T ) ≤

T t=0 {J( θ t ) -J(θ * )} := R TS + T t=0 {z T t θ t P ( θ t ) θ T t z t -z T t θ * P ( θ t )θ T * z t } := R RLS + T t=0 {x T t P ( θ t )x t -E x T t+1 P ( θ t+1 )x t+1 |F t } := R mart + T t=0 E x T t+1 P ( θ t+1 ) -P ( θ t ) x t+1 |F t := R Gap
Before entering into the details of how to bound each of these components, we discuss what are the main challenges in bounding the regret.

Related work and challenges

Since the RLS estimator is the same in both TS and OFU, the regret terms R RLS and R mart can be bounded as in (Abbasi-Yadkori and Szepesvári, 2011). In fact, R mart is a martingale by construction and it can be bounded by Azuma's inequality. The term R RLS is related to the difference between the true next expected state θ T z t and the predicted next expected state θ T t z t . A direct application of RLS properties makes this difference small by construction, thus bounding R RLS . Finally, the R gap term is directly affected by the changes in model from any two time instants (i.e., θ t and θ t+1 ), while R TS measures the difference in optimal average expected cost between the true model θ * and the sampled model θ t . In the following, we denote by R gap t and R TS t the elements at time t of these two regret terms and we refer to them as consistency regret and optimality regret respectively. Optimistic approach. OFU-LQ explicitly bounds both regret terms directly by construction. In fact, the lazy update of the control policy allows to set to zero the consistency regret R gap t in all steps but when the policy changes between two episodes. Since in OFU-LQ an episode terminates only when the determinant of the design matrix is doubled, the number of episodes is bounded by O(log(T )), which bounds R gap as well (with a constant depending on the state bound X and other parameters specific of the LQ system).4 At the same time, at the beginning of each episode an optimistic parameter θ t is chosen, i.e., J( θ t ) ≤ J(θ * ), which directly ensures that R TS t is upper bounded by 0 at each time step.

Bayesian regret. The lazy PSRL algorithm of Abbasi-Yadkori and Szepesvári (2015) has the same lazy update as OFUL and thus R gap should be controlled the same way. Unfortunately, as hinted in [START_REF] Osband | Posterior sampling for reinforcement learning without episodes[END_REF], challenges of extending episodic TS results to infinite horizon introduced a flaw in this approach, so even in the Bayesian analysis, bounding R gap remains an open question. On the other hand, the random choice of θ t does not guarantee optimism at each step anymore. Nonetheless, the regret is analyzed in the Bayesian setting, where θ * is drawn from a known prior and the regret is evaluated in expectation w.r.t. the prior. Since θ t is drawn from a posterior constructed from the same prior as θ * , in expectation its associated J( θ t ) is the same as J(θ * ), thus ensuring that E[R TS t ] = 0. Frequentist regret. When moving from Bayesian to frequentist regret, this argument does not hold anymore and the (positive) deviations of J( θ t ) w.r.t. J(θ * ) has to be bounded in high probability. Abbasi-Yadkori and Szepesvári (2011) exploits the linear structure of LQ problems to reuse arguments originally developed in the linear bandit setting. Similarly, we could leverage the analysis of TS for linear bandit by Agrawal and Goyal (2012b) to derive a frequentist regret bound. Agrawal and Goyal (2012b) partition the (potentially infinite) arms into saturated and unsaturated arms depending on their estimated value and their associated uncertainty (i.e., an arm is saturated when the uncertainty of its estimate is smaller than its performance gap w.r.t. the optimal arm). In particular, the uncertainty is measured using confidence intervals derived from a concentration inequality similar to Prop. 2.3.1. This suggests to use a similar argument and classify policies as saturated and unsaturated depending on their value. Unfortunately, this proof direction cannot be applied in the case of LQR. In fact, in an LQ system θ the performance of a policy π is evaluated by the function J π (θ) and the policy uncertainty should be measured by a confidence interval constructed as |J π (θ * ) -J π ( θ t )|. Despite the concentration inequality in Prop. 2.3.1, we notice that neither J π (θ * ) nor J π ( θ t ) may be finite, since π may not stabilize the system θ * (or θ t ) and thus incur an infinite cost. As a result, it is not possible to introduce the notion of saturated and unsaturated policies in this setting and another line of proof is required. Another key element in the proof of Agrawal and Goyal (2012b) for TS in linear bandit is to show that TS has a constant probability p to select optimistic actions and that this contributes to reduce the regret of any non-optimistic step. In our case, this translates to requiring that TS selects a system θ t whose corresponding optimal policy is such that J( θ t ) ≤ J(θ * ). Lem. 4.5.5 shows that this happens with a constant probability p. Furthermore, we can show that optimistic steps reduce the regret of non-optimistic steps, thus effectively bounding the optimality regret R TS . Nonetheless, this is not compatible with lazy updates. In fact, while R TS is small when optimistic parameters θ t are sampled often enough, a crude bound of the consistency regret R gap requires to reduce the switches between policies as much as possible (i.e., number of episodes). If we keep the same number of episodes as with the lazy update of OFUL (i.e., about log(T ) episodes), then the number of sampled points is as small as T /(T -log(T )). While OFU-LQ guarantees that any policy update is optimistic by construction, with TS, only a fraction T /(p(T -log(T )) of steps would be optimistic on average. Unfortunately, such small number of optimistic steps is no longer enough to derive a bound on the optimality regret R TS . Summarizing, in order to derive a frequentist regret bound for TS in LQ systems, we need the following ingredients. 1) constant probability of optimism, 2) connection between optimism and R TS without using the saturated and unsaturated argument, 3) a novel approach to bound the deviation in the optimal value at the policy switch to guarantee small consistency regret.

Sketch of the proof

The outline of the proof is the following. We prove first the main result that states that TS keeps the average absolute deviation of the optimal value function small. Formally, let

∆ t = E |J( θ t ) -E J( θ t )|F x t , E t | F x t , E t ,
where E t is some high probability event, and F x t is a filtration encoding the knowledge up to time t (we postpone the formal definitions to Sec. 4.5), we show that

T t=0 ∆ t = T t=0 E |J( θ t ) -E J( θ t )|F x t , E t | F x t , E t = O √ T .
Then, we show how this implies bounds for the terms R gap and R TS . Thanks to the constant probability of being optimistic and the fact that sampling distributions does not change much between two subsequent steps, we relate R TS and R Gap to the average absolute deviation and thus bound the overall regret. Finally, we recall the bounds from (Abbasi-Yadkori and Szepesvári, 2011) derived for the OFUL strategy, but which still apply to R RLS and R mart . This is due to the fact that the TS strategy shares the principle of choosing a point within a high probability confidence ellipsoid (deterministically for OFUL, randomly for TS), and then of controlling the system with an optimal control w.r.t the chosen parameter. We sketch here the steps that are used to bound the regret terms specific to TS.

Average absolute deviation. The proof consists in the following chain of inequalities:

T t=0 E |J( θ t ) -E J( θ t )|F x t , E t | F x t , E t ≤ T t=0 E ∇J( θ t ) V -1 t |F x t , E t (1) ≤ T t=0 E H( θ t ) V -1 t |F x t , E t (2) ≤ T t=0 E z t V -1 t |F t-1 , E t-1 (3) ≤ T t=0 z t V -1 t + √ T (4) ≤ √ T .
The objective of the first inequality is to link the average absolute deviation of the optimal value function to its gradient, inspired by the discussion of Ch. 3 about the sensitivity of the performance w.r.t the randomness of the sampling. However, as opposed to LB, in the LQ setting no convexity argument can be used. We tackle this limitation introducing a modified Poincaré inequality (see Lem. 4.5.3). The Poincaré inequality is a major result in Sobolev spaces that allows to bound a function u by its derivative, as u L p ≤ C ∇u L p . We adapt it to our problem, by modifying the sharp L 1 -inequality derived by [START_REF] Acosta | An optimal poincaré inequality in l 1 for convex domains[END_REF] and apply it to the function

f t (η) = J( θ t + βV -1/2 t η) so that ∇ η f t = βV -1/2 t ∇ θ J.
The second inequality relates the gradient of the optimal value function to the control matrix. Thanks to Prop. 4.A.1, we show that for any θ ∈ S, for any p.s.d. matrix V , ∇J(θ) V ≤ H(θ) V . This stresses how the control chosen by TS over time is related to the sensitivity of the optimal value function. However, bounding the absolute deviation ∆ t by the control matrix chosen by TS is not enough to bound the cumulative sum, as Prop. 4.2.4 concerns the actual control. Formally, we would like to move from H( θ t ) to z t . Noticing that z t = H( θ t )x t , one can associate the control H( θ t ) to the chosen direction while x t plays the role of the amplitude. We rely on the weak dependence between those two quantities to show their equivalence. For sake of illustration, suppose for now that, conditionally to F t-1 , θ t and x t are independent. Making use of the fact that the state x t is excited by the noise t (see Eq. 4.1), one has

E(x t x T t |F t-1 , θ t ) = E(x t x T t |F t-1 ) I so H( θ t ) V -1 t ≤ H( θ t )E x t x T t |F t-1 , θ t V -1 t ≤ E H( θ t )x t x T t V -1 t |F t-1 , θ t ≤ XE z t V -1 t |F t-1 , θ t ,

Challenges and sketch of the proof 69

where X is an upper bound on x t . As a consequence, using Azuma's inequality and the law of expectation, one has

E H( θ t ) V -1 t |F x t E H( θ t ) V -1 t |F t-1 XE z t V -1 t |F t-1 .
In practice, θ t and x t are weakly linearly dependent, but this dependency scales in z t V -1 t . When both are distributed as Gaussian random variable (conditionally), it is possible to handle this weak dependence and derive a similar result. The last difficulty comes from the conditioning θ t ∈ S and x t ≤ X that breaks this Gaussian property. We overcome this issue by showing that this result still holds for truncated Gaussian random variable, given that the truncations are far enough from the means of the distributions.

Probability of being optimistic. The objective of this step is to relate J( θ t ) -J(θ * ) to ∆ t i.e., the point-wise deviation from θ * w.r.t. J to the average absolute deviation. Up to a martingale term, we equivalently aim to bound E(J( θ t )|F x t ) -J(θ * ). To this end, we rely on the probability of sampling optimistic parameters i.e., whose optimal average cost is smaller than the one of θ * . Formally, let

Θ opt = {θ ∈ R n(n+d) s.t. J(θ) ≤ J(θ * )},
for any θ ∈ Θ opt , one has:

E(J( θ t )|F x t ) -J(θ * ) ≤ E(J( θ t )|F x t ) -J(θ) ≤ |E(J( θ t )|F x t ) -J(θ)|.
This implies that

E(J( θ t )|F x t ) -J(θ * ) ≤ E |J( θ t ) -E(J( θ t )| F x t ) F x t , θ t ∈ Θ opt ≤ E |J( θ t ) -E(J( θ t )|F x t )| F x t P( θ t ∈ Θ opt |F x t ), = ∆ t P( θ t ∈ Θ opt |F x t ).
Therefore, R TS is bounded as soon as the probability of being optimistic is constant which is provided by Lem. 4.5.5. Intuitively, this means that optimistic samples induce controls that contribute to bound the deviation in the performance, and the constant probability of being optimistic ensures that we sample those parameters at a fixed frequency i.e., often enough.

Subsequent sampling distributions.

The objective of this step is to bound the regret due to the gap at the policy switch R gap . While standard approaches rely on lazy updates to control this term, this is in contrast with the functioning of TS that requires frequent updates in order to keep R TS small. This motivate our new line of proof, which relies on the fact that subsequent sampling distributions are close to each other. Unfortunately, due to the S constraint, this result is only provided for 1-dimensional LQ system (we discuss the general case in Sec. 4.6). The objective is again to show that E |J( θ t+1 ) -J( θ t )| F t is bounded by ∆ t . Supposed that θ t+1 and θ t have the same distribution, then, by definition, E |J( θ t+1 ) -J( θ t )| F t ≤ 2∆ t . The idea of the proof is to show that this still holds, provided that the distributions are close enough. For sake of illustration, we only sketch the proof when β t = β here. Let φ t (θ) be the pdf of the sampling step without the rejection sampling procedure (which can be done by appropriate extension of the J function), one has:

φ t (θ) = det(V t ) 1/2 β(2π) n(n+d)/2 exp - 1 2β 2 θ -θ t 2 Vt .
Noticing that V t+1 ≥ V t and that det(V t+1 ) ≤ det(V t ), one obtains

φ t+1 (θ) ≤ φ t (θ -θ t+1 + θ t ).
We use this inequality to re-write the expectation (together with some manipulation of the conditioning) and get

E |J( θ t+1 ) -J( θ t )| F t ≤ E |J( θ t -θ t+1 + θ t ) -E(J( θ t )|F x t )| F x t + ∆ t .
We conclude by using the Lipschitz property of J and Prop. 4.5.5, which guarantees that least square increments are cumulatively bounded.

Theoretical analysis

We prove the first frequentist √ T regret bound for TS in LQ systems of dimension n = 1 and arbitrary d. In order to isolate the steps which explicitly rely on this restriction, whenever possible we derive the proof in the general (n + d)-dimensional case. We discuss limitation and possible extension to the (n + d)-dimensional case in Sec. 4.6. Theorem 4.5.1. Consider the LQ system in Eq. 4.1 of dimension n = 1 and arbitrary d. Under Asm. 4.2.1 and 4.2.2 for any 0 < δ < 1, the cumulative regret of TS (Algorithm 4.1) over T steps is bounded w.p. at least 1 -δ as 5

R(T ) = O

log(1/δ)T .

Setting the stage

Filtration and sampling. To take into account the randomness of the sampling, we define the "extended" filtration F x t = (F t-1 , x t ). Note that both θ t and V t are F x t -measurable while θ t |F x t is a random variable (since it is not conditioned on F t ). Additionally, to handle the constraint S, we introduce θt ∼ N ( θ t , β 2 t V -1 t ) so that θ t |F x t d = R S ( θt ), where R S is a rejection sampling operator.

High-probability events. We introduce the following high probability events.

5 Further details can be recovered from the proof.

Theoretical analysis 71

Definition 4.5.1. Let δ ∈ (0, 1) and δ = δ/(8T ) and t ∈ [0, T ]. We define the confidence ellipsoids (RLS estimate concentration)

E RLS t := θ ∈ R d , θ -θ t Vt ≤ β t (δ ) ,
and (parameter θ t concentration around θ t )

E TS t := θ ∈ R d , θ -θ t Vt ≤ γ t (δ ) ,
where γ t (δ ) = β t n 2(n + d) log 2n(n + d)/δ . We also introduce their associated high probability events

E t = ∀s ≤ t, θ * ∈ E RLS s and E t = ∀s ≤ t, θ s ∈ E TS s respectively.
We also introduce a high probability event on which the states x t are bounded almost surely. Definition 4.5.2. Let δ ∈ (0, 1), X 1 , X 2 be two problem-dependent positive constants and t ∈ [0, T ] and let X = X 1 log X 2 δ . We define the event (bounded states) Ēt = ∀s ≤ t, x s ≤ X .

Then we have that

E := E T ⊂ • • • ⊂ E 1 , E := E T ⊂ • • • ⊂ E 1 and Ē := ĒT ⊂ • • • ⊂ Ē1 .
We show that these events do hold with high probability.

Lemma 4.5.1.

P( E ∩ E) ≥ 1 -δ/4. Corollary 4.5.1. On E ∩ E, P( Ē) ≥ 1 -δ/4. Thus, P( E ∩ E ∩ Ē) ≥ 1 -δ/2.
Lem. 4.5.1 leverages Prop. 2.3.1 and the sampling distribution to ensure that E ∩ E holds w.h.p. Furthermore, Corollary 4.5.1 ensures that the states remain bounded w.h.p. on the events E ∩ E. 6 As a result, the proof can be derived considering that both parameters concentrate and that states are bounded, which we summarize in the sequence of events E t = E t ∩ E t ∩ Ēt , which holds with probability at least 1-δ/2 for all t ∈ [0, T ].

Regret decomposition. Conditioned on the filtration F t and event E t , we have θ ∈ E RLS t , θ t ∈ E TS t and x t ≤ X. We decompose the regret and bound it on this event in line with (Abbasi-Yadkori and Szepesvári, 2011) (see details in App. 4.D).

6 This non-trivial result is directly collected from the bounding-the-state section in (Abbasi-Yadkori and Szepesvári, 2011). While our algorithm differs, it shares the same mechanism of picking a parameter θ t within a confidence ellipsoid w.h.p., which is the core idea of the proof. Finally, TS uses the ellipsoid E TS t instead of E RLS t which is harmless to the proof because the scaling remains the same in terms of x t 's.

Proposition 4.5.1. Let E t , E t and Ēt be the high probability events introduced in Def. 4.5.1-4.5.2, let E

t = E t ∩ E t ∩ Ēt , then R(T )1{E T } ≤ T t=0 J( θ t ) -J(θ * ) 1{E t } := R TS + T t=0 z T t θ t P ( θ t ) θ T t z t -z T t θ * P ( θ t )θ T * z t 1{E t } := R RLS + T t=0 x T t P ( θ t )x t 1{E t } -E x T t+1 P ( θ t+1 )x t+1 1{E t+1 }|F t := R mart + T t=0 E x T t+1 P ( θ t+1 ) -P ( θ t ) x t+1 1{E t+1 }|F t := R Gap (4.6)

Bounding the absolute deviation of the optimal value function

We prove in this subsection the main result of the proof that states that the conditional average absolute deviation of the performance J w.r.t. the TS distribution is cumulatively bounded. This is critical in the TS analysis: since the controls selected at each time step are based on random choices of θ t , one cannot expect to control the deviation in the performance everywhere but only in expectation. Those results holds for LQ systems of arbitrary dimension n and d.

Lemma 4.5.2. Consider the LQ system in Eq. 4.1. Under Asm. 4.2.1 and 4.2.2 for any 0 < δ < 1, the absolute deviation of the performance J w.r.t. the sampling of TS algorithm 4.1 is cumulatively bounded w.p. at least 1 -δ/12 as

T t=0 E |J( θ t ) -E J( θ t )|F x t , E t | F x t , E t ≤ γ abs √ T ,
where

γ abs = 16(1 + 1/β 2 0 )(1 + C)α 2 2(n + d)/λ log 1 + ((1 + C)α) 2 T /λ(n + d) + 2 log(24/δ) , γ = 4 n(n + d)β T Dnρ/(1 -ρ 2 ), α = (1 + 1/β 2 0 ) 2n log(3n) + (1 + C)X 2S + n(n + d)/β 0 .
The absolute deviation is taken w.r.t the actual sample parameter θ t i.e., the one with rejection sampling. However, in bounding the gap in the policy switch R Gap , we will use a slightly modified version of Lem. 4.5.2 for the parameter θt without rejection sampling. This is given by the following corollary: Corollary 4.5.2. Consider the LQ system in Eq. 4.1. Let J be the continuous extension of J over R n(n+d) defined by J(θ) = J(θ)1 S (θ) + D1 S c (θ). Let θt be a random variable defined by θt = θ t + β t V -1/2 t η t where each component of η t is conditionally distributed as N (0, 1). Then, for any 0 < δ < 1, the absolute deviation of the extended performance J of TS algorithm 4.1 w.r.t. to the random variable θt is cumulatively bounded w.p. at least 1 -δ/12 as

T t=0 E | J( θt ) -E J( θt )|F x t , θt ∈ E TS t , E t | F x t , θt ∈ E TS t , E t ≤ γ abs √ T ,
where

γ abs = 16(1 + 1/β 2 0 )(1 + C)α 2 2(n + d)/λ log 1 + ((1 + C)α) 2 T /λ(n + d) + 2 log(24/δ) , γ = 4 n(n + d)β T Dnρ/(1 -ρ 2 ), α = (1 + 1/β 2 0 ) 2n log(3n) + (1 + C)X 2S + n(n + d)/β 0 .
We first prove Lem. 4.5.2 and then show how it can be extended to Cor. 4.5.2 thanks to the extension of the performance function. Let

∆ t = E |J( θ t ) -E J( θ t )|F x t , E t | F x t , E t .
The proof follows four steps: 1) we introduce a modified Poincaré inequality to show that ∆ t is bounded at each time step by a quantity which depends on ∇J; 2) we link the gradient of the performance ∇J(θ) to the LQ control matrix H(θ); 3) we introduce the state x t , making use of the fact that H( θ t )x t = z t , to bound ∆ t by z t V -1 t

; 4) we conclude by using Prop. 4.2.4 to bound the cumulative sum.

Step 1) Absolute deviation and gradient. Let d = n(n + d), we introduce the mapping f t from the ball B(0, d ) to R + defined in Eq. 4.7,

f t (η) = J( θ t + β t V -1/2 t η) -E[J( θ t )|F x t , E t ], (4.7)
where the restriction on the ball is here to meet the E TS t confidence ellipsoid of the sampling. By definition of the sampling distribution, we can rewrite ∆ t as

∆ t = E ηt |f t (η t )| η t ∈ B(0, d ), θ t + β t V -1/2 t η t ∈ S 1{ Êt , Ēt }.
We now need to show that this formulation of the regret is related to the policy executed by TS. We prove the following result (proof in App. 4.C), that is a generic result of Sobolev's space. In particular, we use the notation of [START_REF] Acosta | An optimal poincaré inequality in l 1 for convex domains[END_REF] which are unrelated to the Chapter's notation.

Lemma 4.5.3. Let Ω ⊂ R d be a convex domain with finite diameter diam and denote as W 1,1 (Ω) the Sobolev space of order 1 in L 1 (Ω). Let p be a non-negative log-concave function on Ω with continuous derivative up to the second order. Then, for all u ∈ W 1,1 (Ω) such that Ω u(z)p(z)dz = 0 one has

Ω |u(z)|p(z)dz ≤ 2diam Ω ||∇u(z)||p(z)dz
Using Lem. 4.5.3 allows us to link ∆ t to ∇f t and thus to ∇J since, for any η and any

θ = θ t + β t V -1/2 t η, ∇f t (η) = β t V -1/2 t ∇J(θ).
Step 2) From gradient to control. To obtain a bound on the norm of ∇f t , we apply Prop. 4.A.1 (proof in App. 4.A) to get a bound on ∇J(θ) V -1 t for any θ ∈ S. First, notice that

∇J(θ) V -1 t ≤ sup δθ =1 Tr δθ T V -1/2 t ∇J(θ) = sup δθ =1 Tr dP (θ)(V -1/2 t δθ) ,
where the first inequality comes from the fact that the equality holds for δθ = V -1/2 t ∇J(θ)/ V -1/2 t ∇J(θ) and the second equality comes directly from the definition of the differential. Then, making use of Tr(A) ≤ n A 2 for any matrix A ∈ R n×n , we obtain:

∇J(θ) V -1 t ≤ n sup δθ =1 dP (θ)(V -1/2 t δθ) 2 .
We conclude using Prop. 4.A.1 which ensures that, for any θ ∈ S,

∀ δθ = 1, dP (θ)(V -1/2 t δθ) 2 ≤ 2Dρ/(1 -ρ 2 ) H(θ) V -1 t .
As a result, we have that θ ∈ S,

∇J(θ) V -1 t ≤ 2Dρ/(1 -ρ 2 ) H(θ) V -1 t . (4.8)
We are now ready to use the weighted Poincaré inequality of Lem. 4.5.3 to link the expectation of |f t | to the expectation of the norm of its gradient. From Lem. 4.2.3, we have f t ∈ W 1,1 (Ω), where Ω = B(0, d ) and its expectation is zero by construction. On the other hand, the rejection sampling introduces the conditioning θ t + β t V -1/2 t η ∈ S which is unfortunately not convex (S is not convex). However, we can still apply Lem. 4.5.3 considering the function ft (η) = f t (η)1( θ t + β t V -1/2 t η ∈ S) and diameter diam = d . As a result, we finally obtain,

∆ t ≤ γE H( θt ) V -1 t |F x t , θt ∈ S, θt ∈ E TS t , (4.9) where θt = θ t + β t V -1/2 t η t and γ = 4 n(n + d)β T Dnρ/(1 -ρ 2 ).
Step 3) From gradient to actions. Recalling the definition of H(θ) = I K(θ) T T we notice that the previous expression bound the regret ∆ t with a term involving the gain K(θ) of the optimal policy for the sampled parameter θ. This shows that the absolute deviation is directly related to the policies chosen by TS. To make such relationship more apparent, we now elaborate the previous expression to reveal the sequence of state-control pairs z t induced by the policy with gain K( θ t ). By noticing that z t = H( θ t )x t one just needs to include the state x t in Eq. 4.9. The intuition why this does not have a big impact on the bound is the following. The main contribution is coming from the direction H( θ t ) that intercepts the design matrix V -1 t .

On the other hand, the state x t can be seen as an amplitude i.e., once the direction is chosen, the exploration is made proportionally to the state. This is true thanks to the relative independence between x t and θ t . Finally, since x t is driven by the dynamic of Eq. 4.1, which is excited by t , its amplitude is lower bounded. Moreover, this property still holds when we constrain the state to be bounded, given that the bound is large enough. This is formalized in the following property on the conditional second order moment.

Proposition 4.5.2. Let θt ∼ N ( θ t , β 2 t V -1 t ), let x t be the state generated by any

F x t -measurable sequence of control {u t } t , let α t = 2n log(3n) + xt where xt = E(x t |F t-1 , θt , E t-1 ). Then, E x t x T t 1 { xt ≤αt} |F t-1 , θt , E t-1 1 8(1 + 1/β 2 t ) I.
We prove Prop. 4.5.2 in two steps. First, we deal with the conditioning and then with the boundedness of the state.

Proposition 4.5.3. Let θt ∼ N ( θ t , β 2 t V -1 t )
, let x t be the state generated by any

F x t - measurable sequence of control {u t } t , then, V x t |F t-1 , θt , E t-1 1 1 + 1/β 2 t I.
Proof. This proposition is based on the following property for Gaussian random variables: let X ∼ N (µ x , Σ x ), Y ∼ N (µ y , Σ y ) and Cov(X, Y ) = Σ xy , then,

V(X|Y ) = Σ x -Σ xy Σ -1 y Σ T xy and E(X|Y ) = µ x + Σ xy Σ -1 y (Y -µ y ).

This property still holds for matrix Gaussian distribution (by vectorization).

To exhibit the dependency, we write θt = a

t-1 + V -1 t z t-1 x T t + β t V -1/2 t
η t where a t-1 , z t-1 and V t are F t-1 -measurable quantities and η t |F t-1 ∼ N (0, I). Then, applying the Gaussian property to X = x t |F t-1 and Y = θt |F t-1 one obtains by vectorization, re-ordering and a little bit of algebra:

V x t |F t-1 , θt , E t-1 = 1 + 1 β 2 t z t-1 2 V -1 t -1 I, E x t |F t-1 , θt , E t-1 = E(x t |F t-1 , E t-1 ) + θt -E( θt |F t-1 , E t-1 ) T V -1 t z t-1 z T t-1 V -1 t + β 2 t V -1 t -1 V -1 t z t-1 . (4.10) Finally, since z t-1 2 Vt -1 = z T t-1 (V t-1 + z t-1 z T t-1 ) -1 z t-1 = z t-1 2 V -1 t-1 1+ z t-1 2 V -1 t-1 ≤ 1,
E x t x T t 1 { xt ≤αt} |F t-1 , θt , E t-1 1/8V(x t |F t-1 , θt , E t-1 ).
The proof of Prop. 4.5.4 relies on properties of truncated Gaussian random variable. The main ingredient is that, if the truncation takes place far enough from the mean, the Gaussian properties are preserved and the second order moment is greater than the variance. We postpone the proof to App. 4.D.

Thanks to Prop. 4.5.2, one has

H( θt ) V -1 t ≤ 8(1 + 1/β 2 t ) H( θt )E x t x T t 1 { xt ≤αt} |F t-1 , θt , E t-1 V -1 t ≤ 8(1 + 1/β 2 t )E H( θt )x t x T t V -1 t 1 { xt ≤αt} |F t-1 , θt , E t-1 ,
which, plugged in Eq. 4.9 leads to

∆ t ≤ γ8(1 + 1/β 2 t )E E H( θt )x t x T t V -1 t 1 { xt ≤αt} |F t-1 , θt , E t-1 F x t , θt ∈ S, θt ∈ E TS t ≤ γ8(1 + 1/β 2 0 )E E H( θt )x t x T t V -1 t 1 { xt ≤αt} |F t-1 , θt , E t-1 F x t , θt ∈ S, θt ∈ E TS t . Plugging θt = a t-1 + V -1 t z t-1 x T t + β t V -1/2 t
η t into Eq. 4.10, one can obtain the following bound for α t , conditioned on η t ∈ B(0, d ):

α t ≤ 2n log(3n) + E(x t |F t-1 , θt , E t-1 ) , ≤ 2n log(3n) + 2(1 + C)SX + 1/β t d (1 + C)X + (1 - 1 1 + z t-1 2 V -1 t /β 2 t ) x t .
Thus, making use of z t-1

2 V -1 t ≤ 1, one has x t ≤ α t =⇒ x t ≤ (1 + 1/β 2 t )( 2n log(3n) + (1 + C)X(2S + d /β t )) ≤ (1 + 1/β 2 0 )( 2n log(3n) + (1 + C)X(2S + d /β 0 )). Let α = (1 + 1/β 2 0 )( 2n log(3n) + (1 + C)X(2S + d /β 0 )), one obtains: ∆ t ≤ γ8(1 + 1/β 2 0 )E E H( θt )x t x T t V -1 t 1 { xt ≤αt} |F t-1 , θt , E t-1 F x t , θt ∈ S, θt ∈ E TS t , ≤ γ8(1 + 1/β 2 0 )αE E H( θt )x t V -1 t 1 { xt ≤α} |F t-1 , θt , E t-1 F x t , θt ∈ S, θt ∈ E TS t , ≤ γ8(1 + 1/β 2 0 )αE E H( θ t )x t V -1 t 1 { xt ≤α} |F t-1 , θ t , E t-1 F x t , θ t ∈ E TS t , ≤ γ8(1 + 1/β 2 0 )αE E z t V -1 t 1 { xt ≤α} |F t-1 , θ t , E t-1 F x t , θ t ∈ E TS t , ≤ γ16(1 + 1/β 2 0 )αE E z t V -1 t 1 { xt ≤α} |F t-1 , θ t , E t-1 F x t .
where we used that θ t d = θt |S from line 2 to line 3, that H( θ t )x t = z t from line 3 to line 4, and that P( θ t ∈ E TS t |F x t ) ≥ P( E t ) ≥ 1/2 to discard the conditioning from line 4 to line 5.

Step 4) Bounding the cumulative sum. Let

Y t = E E z t V -1 t 1 { xt ≤α} |F t-1 , θ t , E t-1 F x t ,
summing the previous bound over T leads to:

T t=0 ∆ t ≤ 16(1+1/β 2 0 )α T t=0 Y t , where α = (1+1/β 2 0 )( 2n log(3n)+(1+C)X(2S+d /β 0 )).
Noticing that

E(Y t |F t-1 ) = E( z t V -1 t 1 { xt ≤α} |F t-1 )
, by the law of iterated expectation,

Y t -z t V -1 t 1 { xt ≤α} t≥1 is a F t -
Y t ≤ T t=0 z t V -1 t 1 { xt ≤α} + (1 + C)α 2T log(24/δ), ≤ T t=0 z t V -1 t + (1 + C)α 2T log(24/δ), ≤ (1 + C)α 2(n + d)T /λ log 1 + ((1 + C)α) 2 T λ(n + d) + (1 + C)α 2T log(24/δ)
where we used Cauchy-Schwarz inequality and Prop. 4.2.4 in the last inequality. This ends the proof of Lem. 4.5.2.

Proof of Cor. 4.5.2. We conclude this section by proving Cor. 4.5.2, showing that

∆t = E | J( θt ) -E J( θt )|F x t , E t | F x
t , E t is upper bounded as ∆ t . The difference lies in the fact that no rejection sampling is considered here (i.e., θt versus θ t ) but the optimal value function J is extended to J in a constant fashion (i.e., J(θ) = J(θ)1 S (θ) + D1 S c (θ)). The proof is similar to the one of Lem. 4.5.2, the only difference lying in Step 1. Since J is a continuous extension of J, differentiable almost everywhere, we can apply the modified Poincaré inequality, which leads to

∆t ≤ 2d β t E ∇ J( θt ) V -1 t |F x t , θt ∈ E TS t .
However, by definition ∇ J(θ) = ∇J(θ)1 S (θ). Therefore,

∆t ≤ 2d β t E ∇J( θt ) V -1 t 1 S ( θt )|F x t , θt ∈ E TS t , ≤ 2d β t E ∇J( θt ) V -1 t |F x t , θt ∈ E TS t , θt ∈ S .
Finally, using Eq. 4.8, we obtain ∆t ≤ γE H( θt )

V -1 t |F x t , θt ∈ S, θt ∈ E TS t
, which is identical to Eq. 4.9 so the rest of the proof follows.

Bounding the optimality regret R TS

We prove in this section a √ T bound for the regret term R TS . The idea of the proof is to show that, thanks to the constant probability of being optimistic (see Lem. 4.5.5), one can link R TS to the average absolute deviation of the optimal value function and thus bound it with high probability according to Lem. 4.5.2. Unfortunately, since Lem. 4.5.5 only holds when n = 1, this restricts the result to the 1d case. However, we believe that this comes mainly from technical difficulties, and that both the ideas and the structure of the proof should hold in any dimension. We discuss its extension in Sec. 4.6. Lemma 4.5.4. Consider the LQ system in Eq. 4.1 of dimension n = 1 and arbitrary d. Under Asm. 4.2.1 and 4.2.2, for any 0 < δ < 1, the regret R TS incurred by running the TS algorithm 4.1 is cumulatively bounded w.p. at least 1 -δ/6 as

R TS = T t=0 {J( θ t ) -J(θ * )}1{E t } ≤ γ TS √ T
where γ TS = 2D 2 log(24/δ) + γ abs p . The structure of the proof is the following: 1) we first decompose the regret, introducing the average performance E J( θ t )|F x t , E t , and bound the martingale part of it, 2) we show that the probability of being optimistic is constant and we link the remaining term to the average absolute deviation of the performance function, 3) we conclude by using Lem. 4.5.2.

R TS decomposition. Let R TS t := {J( θ t ) -J(θ * )}1{E t }. Introducing E J( θ t )|F x t , E t , one can split R TS as R TS = T t=0 R TS,1 t + T t=0 R TS,2 t where R TS,1 t := J( θ t ) -E J( θ t )|F x t , E t 1{E t }, R TS,2 t := E J( θ t )|F x t , E t -J(θ * ) 1{E t }.
By definition, {R TS,1 t } t≥1 is a F x t -martingale difference sequence, bounded almost surely at each time step by 2D (thanks to Def. 4.2.1). Therefore, applying Azuma's inequality guarantees that, w.p. at least 1 -δ/12,

T t=0 R TS,1 t = T t=0 J( θ t ) -E J( θ t )|F x t , E t 1 Et ≤ 2D 2T log(24/δ).
Optimism and expectation. We now focus on the second term R TS,2 t . Let

Θ opt = {θ : J(θ) ≤ J(θ * )}
be the set of optimistic parameters (i.e., LQ systems whose optimal average expected cost is lower than the true one). Then, for any θ ∈ Θ opt , the per-step regret R TS,2 t is bounded by:

R TS,2 t ≤ E[J( θ t )|F x t , E t ] -J(θ) 1{E t } ≤ J(θ) -E[J( θ t )|F x t , E t ] 1{E t },
which implies that, for any random variable θ,

R TS,2 t ≤ E J( θ)-E[J( θ t )|F x t , E t ] F x t , E t , θ ∈ Θ opt ,
where we use first the definition of the optimistic parameter set and bound the resulting quantity by its absolute value. Since this inequality holds for any optimistic parameter, it still holds in expectation, conditioned on θ ∈ Θ opt . While the last inequality is true for any sampling distribution, it is convenient to select it equivalent to the sampling distribution of TS. Thus, we set

θ = R S ( θ t + β t V -1/2 t η) with η is component wise Gaussian N (0, 1) and obtain R TS,2 t ≤ E J( θ t ) -E[J( θ t )|F x t , E t ] F x t , E t , θ t ∈ Θ opt , = E J( θ t ) -E[J( θ t )|F x t , E t ]1{Θ opt } F x t , E t P θ t ∈ Θ opt | F x t , E t ≤ E J( θ t ) -E[J( θ t )|F x t , E t ] F x t , E t P θ t ∈ Θ opt | F x t , E t .
(4.11)

Probability of being optimistic. At this point we need to show that the probability of sampling an optimistic parameter θ t is constant at any step t. This is provided by the following lemma (proof in App. 4.B).

Lemma 4.5.5. Let Θ opt := {θ ∈ R d | J(θ) ≤ J(θ )} be the set of optimistic parameters and θ t = R S ( θ t + β t V -1/2 t η) with η be component-wise normal N (0, 1), then in the one-dimensional case (n = 1 and d = 1)

∀t ≥ 0, P θ t ∈ Θ opt | F x t , E t ≥ p,
where p is a strictly positive constant.

Summing up.

Integrating the result of Lem. 4.5.5 into Eq. 4.11 gives

R TS,2 t ≤ 1 p E J( θ t ) -E[J( θ t )|F x t , E t ] | F x t , E t . (4.12)
The most interesting aspect of this result is that the constant probability of being optimistic allows us to bound the worst-case non-stochastic quantity E[J( θ t )|F x t ] -J(θ * ) depending on J(θ * ) by an expectation E J( θ t )-E[J( θ t )|F x t ] | F x t up to a multiplicative constant (we drop the events E for notational convenience). The last term is the conditional average absolute deviation of the performance J w.r.t. the TS distribution. This connection provides a major insight about the functioning of TS, since it shows that TS does not need to have an accurate estimate of θ * but it should rather reduce the estimation errors of θ * only on the directions that may translate in larger errors in estimating the objective function J.

From Eq. 4.12, and applying Lem. 4.5.2, we obtain, by Azuma, that with probability at least 1 -δ/12,

T t=0 R TS,2 t ≤ 1/p T t=0 ∆ t ≤ γ abs p √ T .
Moreover, since with probability at least 1 -δ/12, one has T t=1 R TS,1 t ≤ 2D 2T log(24/δ), a union bound argument guarantees that, with probability 1 -δ/6, R TS ≤ 2D 2T log(24/δ) + γ abs p √ T .

Bounding the gap at policy switch R gap

We derive here a √ T bound for the regret term R gap which takes into account the deviation in the performance between two subsequent sampling P ( θ t+1 ) -P ( θ t ). The standard approach to control this term is to modify the algorithm by keeping the policy constant over episodes and performing policy update from time to time. Such approach offers two advantages: from a computational point of view, the fewest policy are updated, the cheapest (see Abbasi-Yadkori and Szepesvári 2011 for a lazy update scheme in the LQ setting for OFUL algorithm); from a technical point of view, it avoids quantifying and bounding the gap at policy switch. However, this technique does not suit well with the TS algorithm. We first show that TS has to solve a trade-off between frequently updating the policy to guarantee enough optimistic samples (and hence bound R TS ) and reducing the number of policy switches to limit the regret incurred at each change (and hence bound R gap ). This gives rise to a final bound of O T 2/3 . Then, we derive a new line proof that overcomes the trade-off of frequent versus lazy policy updates and thus leads to a √ T bound.

Trading-off frequent and lazy updates. We discuss in this paragraph the issue that arises when using lazy updates. Consider the modification of the TS algorithm 4.1 where the policy is kept constant over an episode. Denote as {t k } k=1,••• ,K the time steps at which the policy switch occurs, K the number of switches and τ k = t k+1 -t k + 1 the length of each episode. As a result, in line with (Abbasi-Yadkori and Szepesvári, 2011), the regret term R gap is directly bounded by 2DX 2 K. However, the bound of the optimality regret is modified accordingly: since the policy is kept constant over episode, R TS becomes

R TS = K k=1 τ k R TS t k = K k=1 τ k R TS,1 t k + K k=1 τ k R TS,2 t k , ≤ sup k τ k K k=1 R TS,1 t k + K k=1 R TS,2 t k .
Following the same proof as in Sec. 4.5.3, one obtains K k=1 R TS,1

t k + K k=1 R TS,2 t k ≤ √ K
, where is the appropriate numerical constant, and √ K comes directly from the use of Azuma's inequality.

Finally, consider for sake of simplicity the case where the length of episode is constant (e.g. τ k = τ = T /K for all k ≤ K), on obtains:

R TS + R gap ≤ τ √ K + K ≤ T / √ K + K ,
which is minimized by K = T 2/3 and leads to a O(T 2/3 ) overall regret. This synthesizes the trade-off faced by TS: by nature, thanks to randomization, the algorithm selects on average useful policies which contribute to control the regret. Thus the more it samples, the closer it gets to this average behavior. This speaks in favor of frequent updates. On the other hand, the available bound for R gap scales linearly with the number of policy updates, and hence is not compatible with frequent updates.To overcome this issue, we derive a new proof for R gap that allows us to change the policy at each time step without scaling linearly with the number of update. We prove the following result:

Lemma 4.5.6. Consider the LQ system in Eq. 4.1 of dimension n = 1 and arbitrary d. Under Asm. 4.2.1 and 4.2.2, for any 0 < δ < 1, the regret R gap incurred by running the TS algorithm 4.1 is cumulatively bounded w.p. at least 1 -δ/6 as

R gap = T t=0 E x T t+1 P ( θ t+1 ) -P ( θ t ) x t+1 1{E t+1 }|F t ≤ γ gap √ T + γ gap where                                        γ gap := 4D 2 log(24/δ) + X 2 (1 + 2γ gap,2 )γ abs /p + 2X 2 γ gap,1 γ lip γ gap,3 /p, γ gap := X 2 γ gap,2 Dδ/(2p) + n 2 pλ γ 2 gap,3 , γ lip := 2Dρ/(1 -ρ 2 )(1 + C), γ gap,1 := β T (1 + C)X + √ n, γ gap,2 := 1 + (1 + C)X λ 1/2 , γ gap,3 := (1 + C)X 2(n + d)/λ log 1 + ((1 + C)X) 2 T /λ(n + d) .
As for Lem. 4.5.2, this result only holds in the 1d case. This is due to the rejection sampling procedure that introduces boundary issue (w.r.t. the change in subsequent sampling distribution) and requires the use of the extended optimal value function J. However, we believe that this comes mainly from technical difficulties, and that both the ideas and the structure of the proof should hold in any dimension. We discuss its extension in Sec. 4.6.

The structure of the proof is the following: 1) we decompose the regret and show that the probability of sampling an admissible parameter (i.e., θ ∈ S) is constant. This allows us to replace the actual TS sampling distribution by the unconstrained one (i.e., we replace θ t by θt ). 2) We show that subsequent sampling distributions change slowly and make use of the Lipschitz property of J to link R gap to the average absolute deviation of the extended optimal value function. 3) We conclude by using Cor. 4.5.2.

Regret decomposition.

We consider here the 1-d case with n = 1. As a result, P (θ) = J(θ) and R gap can be re-written as

R gap = T t=0 E x T t+1 P ( θ t+1 ) -P ( θ t ) x t+1 1{E t+1 } F t , ≤ X 2 T t=0 E P ( θ t+1 ) -P ( θ t ) 2 1{E t+1 } F t , ≤ X 2 T t=0 E |J( θ t+1 ) -J( θ t )|1{E t+1 } F t .
Applying Azuma's inequality to the F t -martingale difference sequence

E |J( θ t+1 ) -J( θ t )|1{E t+1 } F x t -E |J( θ t+1 ) -J( θ t )|1{E t+1 } F t t≥1
implies that, w.p. at least 1 -δ/12,

R gap ≤ X 2 T t=0 E |J( θ t+1 ) -J( θ t )|1{E t+1 } F x t + 4D 2T log(24/δ).
Let J = J(θ)1 S (θ) + D1 S c (θ) be the continuous extension of the optimal value function.

Let Jt = E J( θt )|F x t , θt ∈ E TS t , E t where θt ∼ N ( θ t , β 2 t V -1 t ) then, w.p. at least 1-δ/12, R gap ≤ 4D 2T log(24/δ) + X 2 T t=0 R gap,1 t + X 2 T t=0 R gap,2 t , where R gap,1 t := E |J( θ t ) -Jt ||F x t , E t , R gap,2 t := E |J( θ t+1 ) -Jt ||F x t , E t+1 , (4.13)
where we used that 1{E t+1 } ≤ 1{E t }, and pushed the events in the conditioning to obtain the expressions of R gap,1 t and R gap,2 t .

Removing the S constraint.

Let θt ∼ N ( θ t , β 2 t V -1 t ). Given that on E t , θ t ∈ E TS t and that θ t d = θt |S for all t ≥ 1, the conditioning θ t ∈ E t implies that θt ∈ E TS t ∩ S. Since, on S, J(θ) = J(θ), one has R gap,1 t = E | J( θt ) -Jt | F x t , θt ∈ E TS t ∩ S, E t , R gap,2 t = E | J( θt+1 ) -Jt | F x t , θt+1 ∈ E TS t+1 ∩ S, E t+1 .
We now rely on the following corollary to handle the rejection sampling.

Corollary 4.5.3. Let θt ∼ N ( θ t , β 2 t V -1 t ), then P( θt ∈ S|F x t , θt ∈ E TS t , E t ) ≥ p where p is the same constant as in Lem. 4.5.5. Proof. The proof relies on Lem. 4.5.5, as in the worst-case configuration, the set of admissible parameters may coincide with the set of optimistic parameters (it can happen if D = J(θ * )). Formally, one just need to notice that, by definition of the admissible parameter set, ∀θ ∈ S, J(θ) ≤ D. Moreover θ * ∈ S. Therefore, ∀θ ∈ Θ opt , J(θ) ≤ J(θ * ) ≤ D and Θ opt ⊂ S. As a consequence,

P( θt ∈ S|F x t , θt ∈ E TS t , E t ) ≥ P( θt ∈ Θ opt |F x t , θt ∈ E TS t , E t ), ≥ P( θ t ∈ Θ opt |F x t , E t ) ≥ p.
Making use of Cor. 4.5.3, we get rid of the admissible constraint set

R gap,1 t ≤ E | J( θt ) -Jt |1{S} F x t , E TS t , E t P( θt ∈ S F x t , E TS t , E t ) ≤ 1/pE | J( θt ) -Jt |1{S} F x t , E TS t , E t R gap,2 t = E E | J( θt+1 ) -Jt | F x t+1 , E TS t+1 , E t+1 , S F x t , E TS t+1 , E t+1 ≤ E E | J( θt+1 ) -Jt |1{S} F x t+1 , E TS t+1 , E t+1 P( θt+1 ∈ S F x t+1 , E TS t+1 , E t+1 ) F x t , E TS t+1 , E t+1 ≤ 1/pE E | J( θt+1 ) -Jt |1{S} F x t+1 , E TS t+1 , E t+1 F x t , E TS t+1 , E t+1 =⇒      R gap,1 t ≤ 1/pE | J( θt ) -Jt | F x t , θt ∈ E TS t , E t R gap,2 t ≤ 2/pE | J( θt+1 ) -Jt | F x t , θt+1 ∈ E TS t+1 , E t (4.14)
where we replaced E t+1 by E t in the last expression, making use of the fact that

P(E t+1 ∩ E c t |F x t ) ≥ 1 -δ/8T ≥ 1/2.
E θ t+1 -θ t F x t , E t ≤ γ gap,1 z t V -1 t where γ gap,1 := (1 + C)X + √ n
Proof. Least Square updates can be written explicitly as

θ t+1 = θ t + V -1/2 t zt 1+ zt 2 V -1 t (x t+1 -θ T t z t ). Thus, θ t+1 -θ t ≤ z t V -1 t 1 + z t 2 V -1 t θ T * z t -θ T t z t + t+1 ≤ θ t+1 -θ t ≤ z t V -1 t 1 + z t 2 V -1 t (β t z t V -1 t + t+1 ) E θ t+1 -θ t F x t , E t ≤ z t V -1 t 1 + z t 2 V -1 t (β t z t V -1 t + √ n) ≤ z t V -1 t β t (1 + C)X + √ n 84 Chapter 4.
|β t+1 -β t | ≤ n z t 2 V -1 t 2
Proof. Using the expression of

β t = n 2 log |Vt| 1/2 |V 0 | 1/2 δ + √ λS and that |V t+1 | = |V t |(1 + z t 2 V -1 t
), one has

β t+1 ≤ β t + n log(1 + z t 2 V -1 t ) ≤ β t + n 1 + z t 2 V -1 t ≤ β t + n z t 2 V -1 t 2 .
First, we get rid of the conditioning in Eq. 4.14 which is a high probability event:

∀t ≥ 1, P( θt+1 ∈ E TS t+1 |F x t , E t ) ≥ 1 -δ/8T ≥ 1/2. Hence, R gap,2 t ≤ 2/pE | J( θt+1 ) -Jt | F x t , E t ≤ 2/pE E | J( θt+1 ) -Jt | F x t+1 , E t F x t , E t . Let θt d = θ t +β t V -1/2 t
η, where η ∼ N (0, I). We denote as φ t (θ) and φ(η) the gaussian pdf of θt |F x t and η respectively defined as

φ t+1 (θ) = det(V t+1 ) 1/2 β 2 (2π) (n+d)n/2 e -1/2β 2 θ-θ t+1 2 V t+1 φ(η)(η) = 1 1(2π) (n+d)n/2 e -1/2 η 2 .
Rewriting the expectation in the integral form, one has:

E | J( θt+1 )-Jt | F x t+1 , E t = R (n+d)n | J( θ)-Jt |φ t+1 ( θ)d θ = R (n+d)n | J( θ t+1 +β t+1 V -1/2 t+1 η)-Jt |φ(η)dη.
Using Prop. 4.2.2 and Prop. 4.2.3, it is clear that, for any θ ∈ S, ∇J(θ) ≤ 2Dρ/(1ρ 2 )(1 + C) := γ lip . Therefore, J is Lipschitz on S and by construction, so is J (with the same Lipschitz constant). Thanks to the Lipschitz property of J and using. Prop. 4.5.5 and 4.5.6, we have:

E | J( θt+1 ) -Jt | F x t+1 , E t = R (n+d)n | J( θ t+1 + β t+1 V -1/2 t+1 η) -Jt |φ(η)dη ≤| J( θ t+1 ) -J( θ t )| + R (n+d)n | J( θ t + β t+1 V -1/2 t+1 η) -J( θ t + β t V -1/2 t+1 η)|φ(η)dη + R (n+d)n | J( θ t + β t V -1/2 t+1 η) -Jt |φ(η)dη ≤γ lip θ t+1 -θ t + |β t+1 -β t | R (n+d)n V -1/2 t+1 η φ(η)dη + R (n+d)n | J( θ t + β t V -1/2 t+1 η) -Jt |φ(η)dη. Using that R (n+d)n V -1/2 t+1 η φ(η)dη ≤ n √ λ
for the second term and a change of variable for the third one, we get:

E | J( θt+1 ) -Jt | F x t+1 , E t ≤γ lip θ t+1 -θ t + |β t+1 -β t | n √ λ + |V t+1 | 1/2 |V t | 1/2 R (n+d)n | J( θ) -Jt |φ t ( θ)d θ Let γ gap,2 := 1 + (1+C)X λ 1/2
, using Prop. 4.5.5 and 4.5.6, we finally have that

E | J( θt+1 ) -Jt | F x t+1 , E t ≤γ lip θ t+1 -θ t + n 2 2λ z t V -1 t + γ gap,2 R (n+d)n | J( θ) -Jt |φ t ( θ)d θ = γ lip θ t+1 -θ t + n 2 2λ z t V -1 t + γ gap,2 E | J( θt ) -Jt | F x t .
Thus,

R gap,2 t ≤ 2/p γ lip E θ t+1 -θ t F x t , E t + n 2 2λ z t V -1 t + γ gap,2 E | J( θt ) -Jt | F x t , E t . Re-introducing the constraint θt ∈ E TS t gives E | J( θt ) -Jt | F x t , E t = E | J( θt ) -Jt |1{E TS t } F x t , E t + E | J( θt ) -Jt |1{E TS,c t } F x t , E t ≤ E | J( θt ) -Jt | F x t , E TS t , E t + 2D 1 -P( θt ∈ E TS t |F x t , E t ) ≤ E | J( θt ) -Jt | F x t , E TS t , E t + Dδ/(4T ),
and using Prop. 4.5.5, one has

E θ t+1 -θ t F x t , E t ≤ γ gap,1 z t V -1 t .
Finally, one obtains

R gap,2 t = 2 p γ gap,2 E | J( θt )-Jt | F x t , E TS t , E t +γ gap,1 γ lip z t V -1 t +γ gap,2 Dδ/(2pT )+ n 2 2λ z t V -1 t .
Summing up. Applying Cor. 4.5.2 provides us with a bound on the cumulative sum of

E | J( θt ) -Jt | F x t , E TS t ,

Final bound

Bounding R RLS and R mart The regret term R RLS is related to the prediction error of the least square. The following lemma provides an upper bound similar to the one derived in (Abbasi-Yadkori and Szepesvári, 2011) with a minor modification due to the different policy update rule between our TS sampling algorithms. We postpone the proofs to App. 4.D.

Chapter 4.

R RLS ≤ γ RLS √ T γ RLS := 2/ √ λ(1 + C 2 )X 2 SD(γ T (δ ) + β T (δ )) 2(n + d) log 1 + T (1 + C 2 )X 2 λ(n + d) .
On the other hand, as discussed in Sec. 4.4, the term R mart is, up to minor modification, a martingale sequence and thus is bounded w.h.p. Lemma 4.5.8. With probability at least 1 -δ/6, R mart ≤ γ mart √ T where γ mart := 2DX 2 2 log(12/δ).

Plugging everything together. We are now ready to bring all the regret terms together. Collecting all the results, one has

R RLS ≤ γ RLS √ T a.s. R mart ≤ γ mart √ T w.p. at least 1 -δ/6 R Gap ≤ γ gap √ T + γ gap w.p. at least 1 -δ/6 R TS ≤ γ TS √ T w.p. at least 1 -δ/6
Therefore, w.p. at least 1 -δ/2,

R RLS + R mart + R Gap + R TS ≤ (γ RLS + γ mart + γ gap + γ TS ) √ T + γ gap .
Finally, notice that the regret decomposition Eq. 4.6, is conditioned on the high probability sequence of events {E t } t≤T which holds w.p. at least 1 -δ/2 according to Cor. 4.5.1. Thus, a union bound argument ensures that, w.p. at least 1 -δ, R(T ) = O √ T where O hides some logarithmic factor and can be recovered from the proof.

Discussion

We derived the first O( √ T ) frequentist regret for TS in LQ control systems. Despite the existing results in LQ for optimistic approaches (OFU-LQ), the Bayesian analysis of TS in LQ, and its frequentist analysis in linear bandit, we showed that controlling the frequentist regret induced by the randomness of the sampling process in LQ systems is considerably more difficult and it requires developing a novel approach, inspired from Ch. 3 that directly relates the regret of TS and the controls executed over time. Furthermore, we stress the need for TS to sample new parameters (and hence choose new policies) at a high frequency, which is in contrast with the lazy update approach of OFU-LQ. The major implication is that the available bound for the gap at policy switch only guarantees a O(T 2/3 ) overall regret bound. To overcome this issue, we introduced a new line of proof that allows us to control the gap at policy switch, thus making possible to update parameters at each time step while ensuring a O( √ T ) overall regret. Despite the fact that most of the proof is derived in the general n dimensional case, two key steps are unfortunately restricted to the 1d case (i.e., when the state is 1 dimensional), which narrows down the final results. We believe that this comes mainly from technical difficulties and that it can be extended to n dimensions. We discuss here these limitations and how to relax it.

Probability of being optimistic

The first limitation comes from the need for optimism. The current proof of Lem. 4.5.5 uses the 1 dimension restriction to exhibit an optimistic set of parameters. Thanks to over-sampling, we show that the probability of sampling within this set is constant. While, inspired by the analysis of Ch. 3, we believe the way over-sampling is performed should guarantee the extension to n dimensions, the main limitation comes from the shape of the optimistic set. We highlight this issue and then try to tackle the problem at a higher level, removing the need for optimism.

Extension to n dimension. For sake of illustration, consider the n-dimensional problem which consists in n independent problems, of dimension 1. Formally, the structure of the problem is diagonal

A * =     a 1 * 0 . . . 0 a n *     , B * =     b 1 * 0 . . . 0 b n *     , Q =     q 1 0 . . . 0 q n     , R =     r 1 0 . . . 0 r n     .
and assume that the learner is aware of this structure. In this case, each system can be estimated and sampled independently, so one can write θ t = [ θ 1 t , . . . , θ n t ], and θ t = [ θ 1 t , . . . , θ n t ], with θ i t conditionally independent. Accordingly, each systems can be controlled independently, so Tr(P (

θ t )) = n i=1 J( θ i t ), Tr(P (θ * )) = n i=1 J(θ i * ). As a consequence, P Tr(P ( θ t )) ≤ Tr(P (θ * )) ≥ P J( θ i t ) ≤ J(θ i * ), ∀i ∈ [1, . . . , n] ≥ n i=1 P J( θ i t ) ≤ J(θ i * ) .
By independence, the probability of being optimistic can be narrowed down to the probability of being jointly optimistic in each direction. Finally, notice that Prop. 2.3.1 becomes Tr ( θi

t -θ i * ) T V i t ( θi t -θ i * ) ≤ β i t (δ) 2 , ∀i ∈ [1, . . . , n]
but that the sampling is made as

θ i t = R S θ i t + β t V i,-1/2 t η i t where β t ≥ √ nβ i t for all i ∈ [1, . . . , n].
In line with the analysis of Ch. 3, this over-sampling by a factor √ n ensures the joint probability to be constant. Despite the lack of generality due to the diagonal structure, this example stresses the intuition that over-sampling (coming from β t ) prevents a bad scaling of the probability of being optimistic with the dimension, and that the crucial difficulties lies in the characterization of the optimistic set Θ opt . While in the diagonal case, one can look at the joint probability of each system being optimistic, it is no longer possible in the general case, as optimizing the cost in a direction could incur a larger cost in another direction. In fact, the proof of Lem. 4.5.5 relies on the following steps. First, we exhibit a set Θ opt that contains optimistic parameters θ, leveraging the shape of the optimal average cost function J(θ) = Tr(P (θ)). Then, we show that the mass of this set, once transformed by the mapping

L : θ → 1 βt V 1/2 t (θ -θ t )
, is constant. This is motivated by the fact that the randomness of the sampling is due to η t , thus the probability of being optimistic is related to the volume of Θ opt under the parametrization corresponding to η t = L( θ t ).

In the 1-dimensional case, we are able to exhibit a set Θ opt which corresponds to the area intercepted by two parallel hyperplanes. This specific shape is invariant under the mapping L, thus the constant mass of Θ opt w.r.t. the Lebesgue measure guarantee a constant mass of L(Θ opt ) i.e., a fixed probability of being optimistic. On the other hand, in the n-dimensional case, we are only able to exhibit a set Θ opt that is a convex n-dimensional cone, of constant mass. Despite the fact that, once transformed by L, the obtained set is still a convex cone, its volume can shrink to zero (think of the case where one eigenvalue of V t diverge to ∞) and therefore does not guarantee a constant probability of being optimistic. Whether it is possible to exhibit a bigger set Θ opt of invariant mass w.r.t. the mapping L is, up to our knowledge, still an open question. On the other hand, we would like to point out the fact that this approach implicitly replaces the initial objective Tr(P (θ)) ≤ Tr(P (θ * )) by the more constrained version P (θ) P (θ * ), where is the inequality associated with p.s.d. matrices. It means that we require every eigenvalues to be smaller, or, equivalently to be more optimistic in every direction. This is, of course, more restrictive than comparing the trace where being very optimistic in a direction could compensate for a little pessimism in others. Comparing the eigenvalues rather than the trace is motivated by the fact that there exists no perturbation theory for trace of Riccati/Lyapunov solutions while some material is available in the matrix case.

Removing the need for optimism. While the above discussion is specific to LQ problem, as it focuses on the shape of the Riccati solution, we try here to tackle the issue at a higher level by discussing the need for optimism. As explained in Sec. 4.4, the core of the analysis is to show that, thanks to the Poincaré inequality and the relationship between the gradient of the optimal value function and the actual actions selected by TS, the average absolute deviation is cumulatively small. Formally, we show that

T t=0 ∆ t = T t=0 E |J( θ t ) -E(J( θ t )|F x t )| F x t = O( √ T ).
Notice that this bound holds for any sampling distribution, which stresses the intuition that it is a structural property of TS (as long as the gradient corresponds to actions).

On the other hand, the initial objective is to provide a bound on J( θ t ) -J(θ * ) or, up to a martingale term, on E(J( θ t )|F x t ) -J(θ * ). Since the only knowledge one has about the unknown parameter is that w.h.p. θ * ∈ E RLS t for all t ≤ T , one aims to bound sup

θ∈E RLS t E(J( θ t )|F x t ) -J(θ) . (4.15)
This motivates the structure of the sampling (see Eq. 4.5), which ensures that θ t spans E RLS t . Then the actual analysis makes use of the over-sampling (i.e., E RLS t ⊂ E TS t ) to guarantee a fixed probability of being optimistic, which provides a link between Eq. 4.15 and ∆ t as sup

θ∈E RLS t E(J( θ t )|F x t ) -J(θ) ≤ ∆ t /p.
However, this derivation is worst-case in the sense that it implicitly assumes that the function f : θ → E(J( θ t )|F x t ) -J(θ) is flat everywhere but over the optimistic set, where it is of high value, and in particular, completely discards almost optimistic points. Thanks to the regularity of the optimal value function J (and hence of f ), it seems that such inequality could be derived in a more global and tighter way. Formally, we would like to have access to an inequality of the form sup

θ∈E RLS t |f (θ)| ≤ c 0 E θ∼E TS t |f (θ)| (4.16)
Alternatively, we could also use an inequality of the form sup

θ∈E RLS t |f (θ)| ≤ c 1 E θ∼E TS t |f (θ)| + c 2 E θ∼E TS t ∇f (θ) . (4.17)
with c 0 , c 1 , c 2 constants that depend on the regularity of f , on the sampling distribution and on the dimension. Despite the fact that the converse of Eq. 4.16 holds with c 0 = 1, it is quite hard to construct counter-example for reasonably smooth function, since it requires f to be flat everywhere but on a small subset where it is very steep (and thus not that smooth). Furthermore, notice that Eq. 4.16 holds for linear functions while Eq. 4.17 holds in 1d (Taylor expansion). From a theoretical perspective, Eq. 4.16 seems related to Converse Holder Inequality, in the extreme case L ∞ versus L 1 , while Eq. 4.17 seems related to Sobolev inequalities. Proving those inequalities with small constant c 0 , c 1 , c 2 is a difficult and open question with implications way beyond the scope of this work. In particular, it will be a major breakthrough for TS analysis, stressing why its structural property (maintaining ∆ t small) ensures a small regret.

Bounding the gap at policy switch

The second limitation comes from the way we bound R gap , and is deeply related to the constraint S and the rejection sampling. The current proof of Lem. 4.5.6 relies on the fact that, when n = 1, P (θ) = J(θ) so that bounding P ( θ t+1 ) -P ( θ t ) 2 is equivalent to bounding |J( θ t+1 ) -J( θ t )|. This is motivated by the fact that J can be extended to J in a constant fashion, while P cannot and because the constant extension is necessary to get rid of the rejection sampling due to the S constraint. However, we believe that this is a technical detail of the proof and that the core idea, which consists in linking |J( θ t+1 ) -J( θ t )| to the average absolute deviation is still valid for P ( θ t+1 ) -P ( θ t ) 2 .

To highlight this intuition, we provide a sketch of the proof under the assumption that no rejection sampling is needed, and then discuss why this conjecture may hold.

The Gaussian sampling case. Assume for now, that the Riccati solution P (θ) is bounded everywhere (i.e., P (θ) 2 ≤ D, ∀θ ∈ R n(n+d) ). Therefore, Prop. 4.2.3 guarantee that P is Lipschitz everywhere with constant γ lip and no rejection sampling is needed, so θ t d = θt is actually sampled according to a Gaussian distribution. Thus, following the proof of Sec. 4.5.4, introducing Pt = E P ( θt )|F x t , E TS t , E t , one has:

R gap ≤ 4D 2T log(24/δ) + X 2 T t=0 R gap,1 t + X 2 T t=0 R gap,2 t where R gap,1 t := E P ( θt ) -Pt 2 |F x t , E TS t , E t R gap,2 t := E P ( θt+1 ) -Pt 2 |F x t , E TS t+1 , E t+1
As in Sec. 4.5.4, R gap,1 t is an average absolute deviation of the performance and can be bounded the same way through the use of a modified Poincaré inequality. Even though Lem. 4.5.3 is proved for real-valued function, at the cost of worsening a bit the constant, using algebraic manipulation and matrix norm equivalence, one can apply it component-wise and retrieve the same result as in Sec. 4.5.2, thus ensuring that w.h.p.

T t=1 R gap,1 t = O( √ T ). Therefore, what remains is to show that R gap,2 t = O R gap,1 t + √ T .
Using the same line of proof, one obtains:

R gap,2 t ≤ E P ( θt + θ t+1 -θ t ) -Pt 2 F x t , E t
where is the appropriate constant. Finally, the Lipschitz property of P together with θ t+1 -θ t cumulatively small (see Prop. 4.5.5) ensures that R gap,2 t is cumulatively 4.6. Discussion 91 closed to R gap,1 t and hence bounded.

Removing the constraint S. The above derivation requires P to be bounded everywhere and thus Lipschitz. Unfortunately, this property is not true, as stated in Prop. 4.2.1 Yet intuitively, we believe that the idea of the proof may still hold. A sufficient condition to deal with the boundedness of P would be to guarantee that (for t big enough) E TS t ⊂ S or that, w.h.p., θt ∈ S. This is tricky to verify theoretically, because it requires quantifying at which rate does E TS t shrink i.e., to look directly at the consistency of the estimates. On the other hand, it is likely to be true in practice since, the bigger the probability of sampling a parameter close to S c is, the faster E TS t shrinks. The reason is that parameters that are close to S c have high optimal value and thus induce very aggressive control: each time such a point is selected, a lot of knowledge is collected from the system which significantly reduces the uncertainty about the parameter estimates.

4.A.2 Proof of Prop. 4.2.3

Let θ T = (A, B) where A and B are matrices of size n × n and n × d respectively. Let R : R n+d,n × R n,n → R n,n be the Riccati operator defined by: R(θ, P ) :

= Q -P + A T P A -A T P B(R + B T P B) -1 B T P A, ( 4.18)
where Q, R are positive definite matrices. Then, the solution P (θ) of the Riccati equation is the solution of R(θ, P ) = 0. While Prop. 4.2.2 guarantees that there exists a unique admissible solution as soon as θ ∈ S, addressing the regularity of the function θ → P (θ) requires the use of the implicit function theorem.

Theorem 4.A.1 (Implicit function theorem [START_REF] Krantz | The implicit function theorem: history, theory, and applications[END_REF]. Let E and F be two Banach spaces, let Ω ⊂ E × F be an open subset. Let f : Ω → F be a C 1 -map and let (x 0 , y 0 ) be a point of Ω such that f (x 0 , y 0 ) = 0. We denote as d y f (x 0 , y 0 ) : F → F the differential of the function f with respect to the second argument at point (x 0 , y 0 ). Assume that this linear transformation is bounded and invertible. Then, there exists

1. two open subsets U and V such that

(x 0 , y 0 ) ∈ U × V ⊂ Ω, 2. a function g : U → V such that g(x) = y for all (x, y) ∈ U × V . Moreover, g is C 1 and dg(x) = -d y f (x, g(x)) -1 d x f (x, g(x)
) for all (x, y) ∈ U × V .

4.A. Control theory 93

Since R is positive definite, the Riccati operator is clearly a C 1 -map from S × S ++ n to S ++ n . Moreover, thanks to Prop. 4.2.2, to any θ ∈ S, there exists P ∈ S ++ n such that R(θ, P ) = 0. Thanks to Thm. 4.A.1, a sufficient condition for θ → P (θ) to be C 1 on S is that the linear map d P R(θ, P (θ)) : R n×n → R n×n is a bounded invertible transformation.

• Bounded. There exists M such that, for any P ∈ R n×n , d P R(θ, P (θ))(P ) ≤ M P .

• Invertible. There exists a bounded linear operator S : R n×n → R n×n such that SP = I n,n and P S = I n,n .

Property 4.A.1. Let θ T = (A, B) and R be the Riccati operator defined in equation (4.18). Then, the differential of R w.r.t P taken in (θ, P ) denoted as d P R(θ, P ) is given by:

d P R(θ, P )(δP ) := A T c δP A c -δP, for any δP ∈ R n×n , where A c = A -B(R + B T P B) -1 B T P A.
Proof. The proof is straightforward using the standard composition/multiplication/inverse operations for the differential operator together with an appropriate rearranging.

Clearly, d P R(θ, P ) is a bounded linear map. Moreover, thanks to the Lyapunov theory, for any stable matrix A c 2 < 1 and for any positive definite matrix Q, the Lyapunov equation A T c XA c -X = Q admits a unique solution. From Prop. 4.2.2, the optimal matrix P (θ) is such that the corresponding A c is stable. This implies that d P R(θ, P ) is an invertible operator, and θ → P (θ) is C 1 on S.

Therefore, the differential of θ → P (θ) can be deduced from the implicit function theorem. After tedious yet standard operations, one gets that for any θ ∈ S and direction δθ ∈ R (n+d)×n :

dJ(θ)(δθ) = Tr(dP (θ)(δθ)) = Tr(∇J(θ) T δθ),
where ∇J(θ) ∈ R (n+d)×n is the jacobian matrix of J in θ and, for any δθ ∈ R (n+d)×n , one has:

dP (θ)(δθ) = A c (θ) T dP (θ)(δθ)A c (θ) + C(θ, δθ) + C(θ, δθ) T , where C(θ, δθ) = A c (θ) T P (θ)δθ T H(θ). (4.19)
Proposition 4.A.1. For any θ ∈ S and any positive definite matrix V , one has the following inequality for the differential of P :

sup δθ =1 dP (θ)(V 1/2 δθ) 2 ≤ 2Dρ/(1 -ρ 2 ) H(θ) V .

Chapter 4. Thompson Sampling in Linear Quadratic System

Proof. From Eq. 4.19, we have, for any θ ∈ S, for any δθ F = 1,

dP (θ)(V -1/2 t δθ) 2 ≤ A c (θ) 2 2 dP (θ)(V -1/2 t δθ) 2 + 2 A c (θ) 2 P (θ) 2 δθ T V -1/2 t H(θ) 2 ≤ A c (θ) 2 2 dP (θ)(V -1/2 t δθ) 2 + 2 A c (θ) 2 P (θ) 2 δθ T V -1/2 t H(θ) ≤ A c (θ) 2 2 dP (θ)(V -1/2 t δθ) 2 + 2 A c (θ) 2 P (θ) 2 δθ H(θ) V -1 t ,
where we used the matrix norm equivalence from line 1 to line 2 and Cauchy-Schwartz from line 2 to line 3. Finally, on S, A c (θ) 2 ≤ ρ and

P (θ) 2 ≤ TrP (θ) ≤ D. Thus, dP (θ)(V -1/2 t δθ) 2 ≤ 2Dρ/(1 -ρ 2 ) H(θ) V -1
t which concludes the proof.

4.B Proof of Lem. 4.5.5

We prove here that, on E, the sampling θ ∼ R S ( θ t + β t V -1/2 t η t ) guarantees a fixed probability of sampling an optimistic parameter i.e., which belongs to Θ opt t := {θ ∈ R d | J(θ) ≤ J(θ )}. However, our result only holds for the 1-dimensional case as we deeply leverage the geometry of the problem. Figure 4.B.1 synthesizes the properties of the optimal value function and the geometry of the problem w.r.t the probability of being optimistic.

1) First, we introduce a simpler subset of optimistic parameters which involves hyperplanes rather than complicated J level sets. Without loss of generality we assume that

A * + B * K * = ρ * ≥ 0 and introduce H * = 1 K * ∈ R 1+d so that A * + B * K * = θ T H * . Let Θ lin,opt = {θ ∈ R d | |θ T H * | ≤ ρ * }.
Intuitively, Θ lin,opt consists in the set of systems θ which are more stable under control K * . The following proposition ensures those systems to be optimistic.

Proposition 4.B.1. Θ lin,opt ⊂ Θ opt t .

Proof. Leveraging the expression of J, one has when n = 1,

J(θ) = Tr(P (θ)) = P (θ) = lim T →∞ E T t=0 x 2 t (Q + K(θ) 2 R) = (Q + K(θ) 2 R)V(x t ), where V(x t ) = (1 -|θ T H(θ)| 2 ) -1
is the steady-state variance of the stationary first order autoregressive process x t+1 = θ T H(θ)x t + t+1 where t is zero mean noise of variance 1

and

H(θ) = 1 K(θ)
. Thus,

J(θ) = Q + K(θ) 2 R 1 -|θ T H(θ)| 2 -1 .
Hence, for any θ ∈ Θ lin,opt , (1

-|θ T H * | 2 ) -1 ≤ (1 -|θ T * H * | 2 ) -1 which implies that (Q + K 2 * R)(1 -|θ T H * | 2 ) -1 ≤ (Q + K 2 * R)(1 -|θ T * H * | 2 ) -1 = J(θ * ).
(A, B) ∈] -∞, -1] ∪ [1, ∞[×{0}}.
The rejection sampling procedure into S ensures P ( θ t ) to be well-defined. Moreover, S c does not overlap with Θ opt . 2) The introduction of the subset Θ lin,opt prevents using the actual -yet complicated -optimistic set Θ opt to lower bound the probability of being optimistic. 3) Even if the event E RLS holds, there exists an ellipsoid configuration which does not contain any optimistic point. This justifies the over-sampling to guarantee a fixed probability of being optimistic.

However, since K(θ) is the optimal control associated with θ,

J(θ) = (Q + K(θ) 2 R)(1 -|θ T H(θ)| 2 ) -1 = min K (Q + K 2 R)(1 -| 1 K θ| 2 ) -1 ≤ (Q + K 2 * R)(1 -|θ T H * | 2 ) -1 ≤ J(θ * ) As a result, P θ t ∈ Θ opt | F x t , E t ≥ P θ t ∈ Θ lin,opt | F x
t , E t and we can focus on Θ lin,opt .

2)

To ensure the sampling parameter to be admissible, we perform a rejection sampling until θ t ∈ S. Noticing that Θ lin,opt ⊂ Θ opt ⊂ S by construction, the rejection sampling 96 Chapter 4. Thompson Sampling in Linear Quadratic System is always favorable in terms of probability of being optimistic. Since we seek for a lower bound, we can get rid of it and consider

θ t = θ t +β t V -1/2 t η where η ∼ N (0, I 1+d ). 7 3) On E t , θ ∈ E RLS t
, where E RLS t is the confidence RLS ellipsoid centered in θ t . Since θ * is fixed (by definition), we lower bound the probability by considering the worst possible θ t such that E t holds. Intuitively, we consider the worst possible center for the RLS ellipsoid such that θ still belong in E RLS t and that the probability of being optimistic is minimal. Formally,

P θ t ∈ Θ lin,opt | F x t , E t = P θt∼N ( θt,β 2 t V -1 t ) θ t ∈ Θ lin,opt | F x t , E t ≥ min θ: θ-θ * V t ≤βt(δ ) P θt∼N ( θ,β 2 t V -1 t ) θ t ∈ Θ lin,opt | F x t )
Moreover, by Cauchy-Schwarz inequality, for any θ,

( θ -θ * ) T H * ≤ θ -θ * Vt H * V -1 t ≤ β t (δ ) H * V -1 t , thus, P θ t ∈ Θ lin,opt | F x t , E t ≥ min θ: θ-θ * V t ≤βt(δ ) P θt∼N ( θ,β 2 t V -1 t ) θ t ∈ Θ lin,opt | F x t ) ≥ min θ:|( θ-θ * ) T H * |≤βt(δ ) H * V -1 t P θt∼N ( θ,β 2 t V -1 t ) θ t ∈ Θ lin,opt | F x t ) = min θ:| θ T H * -ρ * |≤βt(δ ) H * V -1 t P θt∼N ( θ,β 2 t V -1 t ) | θ T t H * | ≤ ρ * | F x t )
Cor. 4.B.1 provides us with an explicit expression of the worst case ellipsoid. Introducing

x = θ T t H * , one has x ∼ N (x, σ 2 x ) with x = θH * and σ x = β t H * V -1 t . Applying Cor. 4.B.1 with α = ρ * , ρ = ρ * and β = β t (δ ) H * V -1 t , the last inequality becomes P θ t ∈ Θ lin,opt | F x t , E t ≥ min θ:| θ T H * -ρ * |≤βt(δ ) H * V -1 t P η∼N (0,I 1+d ) | θ T H * + β t η T V -1/2 t H * | ≤ ρ * | F x t ) ≥ P η∼N (0,I 1+d ) |ρ * + β t (δ ) H * V -1 t + β t η T V -1/2 t H * | ≤ ρ * | F x t )
Introducing the vector

u t = β t (δ )V -1/2 t H * , one can simplify |ρ * + β t (δ ) H * V -1 t + β t (δ )η T V -1/2 t H * | ≤ ρ * , ⇔ -ρ * ≤ ρ * + u t + η T u t β t β t (δ ) ≤ ρ * , ⇔ - ρ * u t -1 ≤ η T u t u t β t β t (δ ) ≤ -1.
Since η ∼ N (0, I 1+d ) is rotationally invariant and since βt βt(δ ) := 1, 7 In the 1-dimensional case, η is just a 1 + d standard Gaussian r.v. Finally, for all t ≤ T , u t is almost surely bounded: u t ≤ β T (δ ) (1 + C 2 )/λ. Therefore,

P θ t ∈ Θ lin,opt | F x t , E t ≥ P ∼N (0,1) ∈ 1, 1 + 2ρ * u t | F x t , E t .
P θ t ∈ Θ lin,opt | F x t , E t ≥ P ∼N (0,1) ∈ 1, 1 + 2ρ * /β T (δ ) (1 + C 2 )/λ := p Corollary 4.B.1. For any ρ, σ x > 0, for any α, β ≥ 0, arg min x:|x-α|≤β P x∼N (x,σ 2 x ) |x| ≤ ρ = α + β.
This corollary is a direct consequence of the properties of standard Gaussian r.v. Lemma 4.B.1. Let x be a real random variable. For any ρ, σ x > 0 Let f : R → [0, 1] be the continuous mapping defined by f (x) = P x∼N (x,σ 2

x ) |x| ≤ ρ . Then, f is increasing on R -and decreasing on R + .

Proof. Without loss of generality, one can assume that σ x = 1/ √ 2 (otherwise, modify ρ), and that x ≥ 0 (by symmetry). Denoting as Φ and erf the standard Gaussian cdf and the error function, one has:

f (x) = P x∼N (x,σ 2 x ) -ρ ≤ x ≤ ρ , = P x∼N (x,σ 2 x ) x ≤ ρ -P x∼N (x,σ 2 x ) x ≤ -ρ , = P x∼N (x,σ 2 x ) (x -x)/σ x ≤ (ρ -x)/σ x -P x∼N (x,σ 2 x ) (x -x)/σ x ≤ (-ρ -x)/σ x , = Φ((ρ -x)/σ x ) -Φ(-(ρ + x)/σ x ), = 1 2 + 1 2 erf((ρ -x)/ √ 2σ x ) - 1 2 - 1 2 erf(-(ρ + x)/ √ 2σ x ), = 1 2 erf(ρ -x) -erf(-(ρ + x)) .
Since erf is odd, one obtains f (x) = 1 2 erf(ρx) + erf(ρ + x) . The error function is differentiable with erf (z) = 2 π e -z 2 , thus

f (x) = 1 π exp -(ρ + x) 2 -exp -(ρ -x) 2 ă = - 2 π sinh (ρ -x) 2 ≤ 0
Hence, f is decreasing on R + and by symmetry, is increasing on R -.

4.C Weighted L1 Poincaré inequality (proof of Lem. 4.5.3)

This result is build upon the following theorem which links the function to its gradient in L 1 norm:

Theorem 4.C.1 (see [START_REF] Acosta | An optimal poincaré inequality in l 1 for convex domains[END_REF]. Let W 1,1 (Ω) be the Sobolev space on Ω ⊂ R d . Let Ω be a convex domain bounded with diameter D and f ∈ W 1,1 (Ω) of zero average on Ω then

Ω |f (x)|dx ≤ D 2 Ω ||∇f (x)||dx
Lem. 4.5.3 is an extension of Thm. 4.C.1. In practice, we show that their proof still holds for log-concave weight.

f ∈ W 1,1 (0, L) such that L 0 f (x)ρ(x)dx = 0 one has: L 0 |f (x)|ρ(x)dx ≤ 2L L 0 |f (x)|ρ(x)dx (4.20)
The proof is based on the following inequality for log-concave function.

Lemma 4.C.1. Let ρ be any non negative log-concave function on [0, 1] such that

1 0 ρ(x) = 1 then ∀x ∈ (0, 1), H(ρ, x) := 1 ρ(x) x 0 ρ(t)dt 1 x ρ(t)dt ≤ 1 (4.21)
Proof. Since any non-negative log-concave function on [0, 1] can be rewritten as ρ x) where ν is a concave function on [0, 1] and since x → e x is increasing, the monotonicity of ν is preserved and as for concave function, ρ can be either increasing, decreasing or increasing then decreasing on [0, 1]. Hence, ∀x ∈ (0, 1), either

(x) = e ν(
1. ρ(t) ≤ ρ(x) for all t ∈ [0, x], 2. ρ(t) ≤ ρ(x) for all t ∈ [x, 1].
Assume that ρ(t) ≤ ρ(x) for all t ∈ [0, x] without loss of generality. Then,

∀x ∈ (0, 1), H(ρ, x) := 1 ρ(x) x 0 ρ(t)dt 1 x ρ(t)dt = x 0 ρ(t) ρ(x) 1 x ρ(t)dt ≤ x 0 dt 1 x ρ(t)dt ≤ x 1 0 ρ(t)dt ≤ x ≤ 1
Proof of theorem 4.C.2. This proof is exactly the same as [START_REF] Acosta | An optimal poincaré inequality in l 1 for convex domains[END_REF] where we use lemma 4.C.1 instead of a concave inequality. We provide it for sake of completeness.

A scaling argument ensures that it is enough to prove it for L = 1. Moreover, dividing both side of (4.20) by 1 0 ρ(x)dx, we can assume without loss of generality that 1 0 ρ(x)dx = 1. Since 1 0 f (x)ρ(x)dx = 0 by integration part by part one has:

f (y) = y 0 f (x) x 0 ρ(t)dt - 1 y f (x) 1 x ρ(t)dt |f (y)| ≤ y 0 |f (x)| x 0 ρ(t)dt + 1 y |f (x)| 1 x ρ(t)dt
Multiplying by ρ(y), integrating on y and applying Fubini's theorem leads to

1 0 |f (y)|ρ(y)dy ≤ 2 1 0 |f (x)| x 0 ρ(t)dt 1 x ρ(t)dt
and applying (4.21) of lemma 4.C.1 ends the proof.

While theorem 4.C.2 provides a 1 dimensional weighted Poincaré inequality, we actually seek for one in R d . The idea of [START_REF] Acosta | An optimal poincaré inequality in l 1 for convex domains[END_REF] is to use arguments of [START_REF] Payne | An optimal poincaré inequality for convex domains[END_REF] to reduce the d-dimensional problem to a d -1 dimensional problem by splitting any convex set Ω into subspaces Ω i thin in all but one direction and such that an average property is preserved. We just provide their result.

Lemma 4.C.2. Let Ω ⊂ R d be a convex domain with finite diameter D and u ∈ L 1 (Ω) such that Ω u = 0. Then, for any δ > 0, there exists a decomposition of Ω into a finite number of convex domains Ω i satisfying

Ω i ∩ Ω j = ∅ for i = j, Ω = Ωi , Ω i u = 0
and each Ω i is thin in all but one direction i.e., in an appropriate rectangular coordinate system (x, y) = (x, y 1 , . . . , y d-1 ) the set Ω i is contained in

{(x, y) : 0 ≤ x ≤ D, 0 ≤ y i ≤ δ for i = 1, . . . , d -1}
This decomposition together with Treheorem 4.C.2 allow us to prove the d-dimensional weighted Poincaré inequality. Lem. 4.5.3. By density, we can assume that u ∈ C ∞ ( Ω). Hence, up ∈ C 2 ( Ω). Let M be a bound for up and all its derivative up to the second order. Given δ > 0 decompose the set Ω into Ω i as in lemma 4.C.2 and express z ∈ Ω i into the appropriate rectangular basis z = (x, y), where x ∈ [0, d i ], y ∈ [0, δ]. Define as ρ(x 0 ) the d -1 volume of the intersection between Ω i and the hyperplan {x = x 0 }. Since Ω i is convex, ρ is concave and from the smoothness of up one has: Those equation allows us to switch from d-dimensional integral to 1-dimensional integral for which we can apply theorem 4.C.2 at the condition that d i 0 u(x, 0)p(x, 0)ρ(x)dx = 0 (which is not satisfied here). On the other hand, we can apply theorem 4.C.2 to g(x) = u(x, 0) -

Proof of

d i 0 u(x, 0)p(x, 0)ρ(x)dx/ d i 0 p(x, 0)ρ(x)dx
with weighted function x → p(x, 0)ρ(x). Indeed, x → p(x, 0) is log-concave -as restriction along one direction of log-concave function, x → ρ(x) is log-concave -as a concave function, and so is x → p(x, 0)ρ(x) -as product of log-concave function. Moreover, g ∈ W 1,1 (0, d i ) and d i 0 g(x)p(x, 0)ρ(x)dx = 0 by construction. Therefore, applying theorem 4.C.2 one gets:

d i 0 |g(x)|p(x, 0)ρ(x)dx ≤ 2d i d i 0 |g (x)|p(x, 0)ρ(x)dx d i 0 |u(x, 0)|p(x, 0)ρ(x)dx ≤ 2d i d i 0 | ∂u ∂x (x, 0)|p(x, 0)ρ(x)dx - d i 0 u(x, 0)p(x, 0)ρ(x)dx d i 0 |u(x, 0)|p(x, 0)ρ(x)dx ≤ 2d i d i 0 | ∂u ∂x (x, 0)|p(x, 0)ρ(x)dx + (d -1)M |Ω i |δ (4.25)
where we use equation (4.24) together with Ω i u(z)p(z)dz = 0 to obtain the last inequality.

Finally, from (4.22)

Ω i |u(x,

4.D Regret proofs

We collect here the regret proofs that are directly inspired or collected from (Abbasi-Yadkori and Szepesvári, 2011). Since our framework slightly differs, minors differences coming from a different conditioning or the fact that we do not consider lazy updates, we provide it for the sake of completeness.

Proof of Lem. 4.5.1. Let δ = δ/8T . 1) From Prop. 2.3.1, P θ t -θ * Vt ≤ β t (δ ) ≥ 1 -δ . Hence,

P E = P T t=0 θ t -θ * Vt ≤ β t (δ ) = 1 -P T t=0 θ t -θ * Vt ≥ β t (δ ) ≥ 1 - T t=0 P θ t -θ * Vt ≥ β t (δ ) ≥ 1 -T δ ≥ 1 -δ/8
2) From Lem. A.2, let η be component-wise N (0, 1) then, for any > 0, making use of the fact that η ≤ n √ n + d max i≤n+d,j≤n |η i,j |,

P η ≤ ≥ P n √ n + d max i,j |η i,j | ≤ ≥ 1 - i,j P |η i,j | ≥ n √ n + d ≥ 1 -n(n + d)P X∼N (0,1) |X| ≥ n √ n + d .
Hence,

P E = P T t=0 θ t -θ t Vt ≤ γ(δ ) = 1 -P T t=0 θ t -θ t Vt ≥ γ(δ ) ≥ 1 - T t=0 P θ t -θ t Vt ≥ γ(δ ) ≥ 1 - T t=0 P η ≥ γ(δ )/β t (δ ) ≥ 1 - T t=0 P η ≥ n 2(n + d) log 2n(n + d)/δ ≥ 1 -T δ ≥ 1 -δ/8.
Finally, a union bound argument ensures that P(

E ∩ E) ≥ 1 -δ/4.
Proof of Cor. 4.5.1. This result comes directly from Sec. 4.1. and App. D of Abbasi-Yadkori and Szepesvári (2011). The proof relies on the fact that, on E, because θ t is chosen within the confidence ellipsoid E RLS t , the number of time steps the true closed loop matrix A * + B * K( θ t ) is unstable is small. Intuitively, the reason is that as soon as the true closed loop matrix is unstable, the state process explodes and the confidence ellipsoid is drastically changed. As the ellipsoid can only shrink over time, the state is well controlled expect for a small number of time steps. Since the only difference is that, on E ∩ E, θ t ∈ E TS t , the same argument applies and the same bound holds replacing β t with γ t . Therefore, there exists appropriate problem dependent constants X, X such that P( Ē| E ∩ E) ≥ 1 -δ/4. Finally, a union bound argument ensures that P

( E ∩ E ∩ Ē) ≥ 1 -δ/2. Regret decomposition. Let R(T ) = T t=0 x T t Qx t + u T t Ru t -J(θ * )
. Since {E t } t≤T is an decreasing sequence of events, one has

R(T )1{E T } ≤ T t=0 x T t Qx t + u T t Ru t -J(θ * ) 1{E t }.
From the Bellman optimality equations for LQ problems, we get that

J( θ t ) + x T t P ( θ t )x t = min u x T t Qx t + u T Ru + E xu,T t+1 P ( θ t )x u t+1 F t = x T t Qx t + u T t Ru t + E xut,T t+1 P ( θ t )x ut t+1 F t ,
where xu t+1 = θ T t z t + t+1 . Hence,

x T t Qx t + u T t Ru t -J(θ * ) 1{E t } = J( θ t ) -J(θ * ) 1{E t } + z T t θ t P ( θ t ) θ T t z t -z T t θ * P ( θ t )θ T * z t 1{E t } +x T t P ( θ t )x t 1{E t } -E x T t+1 P ( θ t )x t+1 1{E t } F t where we used in the last line that E t is F t -measurable. Noticing that E t+1 ⊂ E t , one has 1{E t+1 }1{E t } = 1{E t+1 } and since P ( θ t ) is positive definite, x T t+1 P ( θ t )x t+1 ≥ 0. Therefore, E x T t+1 P ( θ t )x t+1 1{E t } F t = E x T t+1 P ( θ t )x t+1 1{E t }(1{E t+1 } + 1{E c t+1 }) F t = E x T t+1 P ( θ t )x t+1 1{E +1 } F t + E x T t+1 P ( θ t )x t+1 1{E t }1{E c t+1 } F t ≥ E x T t+1 P ( θ t )x t+1 1{E +1 } F t = E x T t+1 P ( θ t ) -P ( θ t+1 ) x t+1 1{E +1 } F t + E x T t+1 P ( θ t+1 )x t+1 1{E +1 } F t
Bounding R mart . Re-ordering the term, using the fact that x 0 = 0 and that P ( θ t ) is definite positive by definition, one obtains:

R mart ≤ T t=1 x T t P ( θ t )x t 1{E t } -E x T t P ( θ t )x t 1{E t }|F t-1 + x T 0 P ( θ 0 )x 0 -E x T T +1 P ( θ T +1 )x T +1 1{E T +1 }|F T ≤ T t=1 x T t P ( θ t )x t 1{E t } -E x T t P ( θ t )x t 1{E t }|F t-1
which is turns to be a martingale. On E, x t ≤ X for all t ∈ [0, T ]. Moreover, since θ t ∈ S for all t ∈ [0, T ] due to the rejection sampling, Tr(P ( θ t )) ≤ D. From the definition of the matrix 2-norm, sup

x ≤X

x T P ( θ t )x ≤ X 2 P ( θ t ) 1/2 2 2 .

Matrix norm equivalence ensures that for any A ∈ R m,n , A 2 ≤ A . Therefore, P ( θ t ) 1/2 2 2 ≤ P ( θ t ) 1/2 2 = TrP ( θ t ) and, for any t ∈ [0, T ], sup x ≤X x T P ( θ t )x ≤ X 2 D so the martingale increments are bounded almost surely on E by 2DX 2 . Applying Azuma's inequality to R mart one obtains that, w.p. at least 1 -δ/6,

R RLS 1 = T t=0 x T t P ( θ t )x t 1{E t }-E x T t+1 P ( θ t+1 )x t+1 1{E t+1 }|F t ≤ 2DX 2 2T log(12/δ).
Bounding R RLS . The whole derivation is performed on the event E.

R RLS = T t=0 z T t θ t P ( θ t ) θ T t z t -z t θ * P ( θ t )θ T * z t = T t=0 θ T t z t 2 P ( θt) -θ T * z t 2 P ( θt) , = T t=0 θ T t z t P ( θt) -θ T * z t P ( θt) θ T t z t P ( θt) + θ T * z t P ( θt)
By the triangular inequality,

θ T t z t P ( θt) -θ T * z t P ( θt) ≤ P ( θ t ) 1/2 ( θ T t z t -θ T * z t ) ≤ P ( θ t ) ( θ T t -θ T * )z t .
Making use of the fact that θ t ∈ S by construction of the rejection sampling, θ ∈ S by Asm. 4.2.2 and that sup t∈[0,T ] z t ≤ (1 + C 2 )X 2 thanks to the conditioning on E and Prop. 4.2.2, one gets:

R RLS ≤ T t=0 √ D ( θ T t -θ T * )z t 2S √ D (1 + C 2 )X 2 ≤ 2SD (1 + C 2 )X 2 T t=0 ( θ T t -θ T * )z t
and one just has to bound T t=0 ( θ T t -θ T * )z t . Using Cauchy-Schwarz inequality, one has:

T t=0 ( θ T t -θ T * )z t = T t=0 (V 1/2 t )( θ t -θ * )) T V -1/2 t z t ≤ T t=0 θ t -θ * Vt z t V -1 t However, on E, θ t -θ * Vt ≤ θ t -θ t Vt + θ * -θ t Vt ≤ β t (δ ) + γ(δ ) ≤ β T (δ ) + γ(δ ). Therefore, R RLS ≤ 4SD (1 + C 2 )X 2 β T (δ ) + γ(δ ) T t=0 z t V -1 t .
The proof is conclude by using Cauchy-Schwarz inequality and Prop. 4.2.4. Proof. Explicit formula for the truncated normal distribution moment leads to

Proof of

V( | | | ≤ a) = 1 - 2a √ 2π e -a 2 /2 P(| | ≤ a) ≥ 1 - ae -a 2 /2 P(| | ≤ a)
.

Standard inequality for the Gaussian cdf guarantees that P(| | ≤ a) ≤ e -a 2 /2 . Hence, for all a ≥ 2 log(6),

V( | | | ≤ a) ≥ 1 - ae -a 2 /2 1 -e -a 2 /
(| | ≤ a) ≤ (1 -1/n) n ≥ 1/4
Proof of Prop. 4.5.4.

Denote as xt = E(x t | θt , F t-1 , E t-1 ) and Σ t = V(x t | θt , F t-1 , E t-1 ). By Prop. 4.5.3 Σ t 2 ≤ 1, hence, one has, E(x t x T t 1 { xt ≤αt} |F t-1 , θt , E t-1 ) ≥ E(x t x T t 1 { xt-xt ≤αt-ρ xt } |F t-1 , θt , E t-1 ), ≥ Σ 1/2 t E(y t y T t 1 { yt-ȳt ≤αt-ρ xt } |F t-1 , θt , E t-1 )Σ 1/2 t , ≥ Σ 1/2 t E(y t y T t 1 {|y i t -ȳ i t |≤ α t -ρ xt √ n ,∀i≤n} |F t-1 , θt , E t-1 )Σ 1/2 t ,
where y t = Σ -1/2 t

x t , ȳt = Σ -1/2 xt and y i denotes the i th coordinate of the n-dimensional vector y. By definition y t |θ t,Ft-1,Et-1 ∼ N (ȳ t , I) thus,

E(y t y T t 1 {|y i t -ȳ i t |≤ α t -ρ xt √ n ,∀i≤n} |F t-1 , θt , E t-1 ) = P(|y i t -ȳ i t |≤ α t -ρ xt √ n ,∀i≤n)E(y t y T t |F t-1 , θt , E t-1 , {|y i t -ȳ i t |≤ α t -ρ xt √ n ,∀i≤n}) ≥ P(|y i t -ȳ i t |≤ α t -ρ xt √ n ,∀i≤n)V(y t |F t-1 , θt , E t-1 , {|y i t -ȳ i t |≤ α t -ρ xt √ n ,∀i≤n}) = P(| | ≤ α t -ρ xt √ n ) n V( | | | ≤ α t -ρ xt √ n )I,
where ∼ N (0, 1). ,∀i≤n} |F t-1 , θt , E t-1 ) ≥ 1/8I. As a results,

E(x t x T t 1 { xt ≤αt} |F t-1 , θt , E t-1 ) ≥ 1/8Σ t = 1/8V(x t |F t-1 , θt , E t-1 )

Introduction

Modern finance theory is often thought to have started with the mean-variance approach of [START_REF] Markowitz | Portfolio selection*[END_REF]. This approach provides portfolio managers with a systematic treatment of the risk-return tradeoff by maximizing their own utility. This started intensive research to further develop the basic mean-variance theory. In particular, it raised questions about the relationship between risk and return, leading to the celebrated CAPM model [START_REF] Sharpe | Capital asset prices: A theory of market equilibrium under conditions of risk*[END_REF][START_REF] Jensen | The capital asset pricing model: Some empirical tests[END_REF] as well as finer modeling for the risk structure and the return predictability [START_REF] Fama | Common risk factors in the returns on stocks and bonds[END_REF]. However, one of the limitations of these approaches is their inability to take transaction costs into account: in a multi-step setting, performing such a strategy may be highly suboptimal as the rebalancing cost can be worse than the expected gain. This observation, of crucial importance for practitioners, led to dynamic allocation rules where portfolio managers anticipate this additional cost and track the Markowitz position by constraining the turnover (see e.g., [START_REF] Constantinides | Multiperiod consumption and investment behavior with convex transactions costs[END_REF][START_REF] Taksar | A diffusion model for optimal portfolio selection in the presence of brokerage fees[END_REF][START_REF] Morton | Optimal portfolio management with fixed transaction costs[END_REF][START_REF] Grinold | Signal weighting[END_REF].

When the volume of the transaction is large compared to the available liquidity, another effect known as price impact induces transaction costs: the execution of a large order drastically changes the supply and demand and thus affects the price in an adverse manner. The understanding of the market impact and the way to minimize it is an important topic for large investors and a large amount of literature addresses this question from different perspectives. Motivated by stylized facts and empirical studies which stress that markets digest very slowly modifications induced by large trade [START_REF] Bouchaud | How markets slowly digest changes in supply and demand[END_REF][START_REF] Brokmann | Slow decay of impact in equity markets[END_REF], [START_REF] Mastromatteo | Agent-based models for latent liquidity and concave price impact[END_REF], [START_REF] Donier | A fully consistent, minimal model for non-linear market impact[END_REF] derived a microstructure based model for the price impact from the dynamic of the latent order book. Following the work of [START_REF] Kyle | Continuous auctions and insider trading[END_REF], another stream of literature considers agent-based model to understand how information is incorporated into the prices and how it affects the liquidity. [START_REF] Huberman | Price manipulation and quasi-arbitrage[END_REF], [START_REF]No-dynamic-arbitrage and market impact[END_REF] study the effect of the price impact on the absence of price manipulation and derive various inequalities about the shape of the price impact function. On the other hand, a large part of the literature is dedicated to the minimization of the price impact. Two types of problem are usually considered: optimal execution (see e.g., [START_REF] Bertsimas | Optimal control of execution costs[END_REF][START_REF] Almgren | Optimal execution of portfolio transactions[END_REF][START_REF] Guéant | Optimal execution and block trade pricing: a general framework[END_REF][START_REF] Obizhaeva | Optimal trading strategy and supply/demand dynamics[END_REF] where investors seek to liquidate a given position within a certain period, and optimal allocation (see e.g., [START_REF] Gârleanu | Portfolio choice and pricing in illiquid markets[END_REF][START_REF] Lataillade | Optimal trading with linear costs[END_REF][START_REF] Gârleanu | Dynamic trading with predictable returns and transaction costs[END_REF][START_REF] Kallsen | The general structure of optimal investment and consumption with small transaction costs[END_REF][START_REF] Moreau | Trading with small price impact[END_REF] where investors try to dynamically control a portfolio to maximize their risk-profit utility under price impact. Finally, [START_REF] Park | Adaptive execution: Exploration and learning of price impact[END_REF] consider the optimal allocation problem of [START_REF] Gârleanu | Dynamic trading with predictable returns and transaction costs[END_REF] and introduce an adaptive algorithm, called CTRACE, that performs both the estimation and control when the impact model is unknown.

In this chapter, we consider the optimal portfolio allocation problem when the market exhibits dynamical return predictability and price impact. Our setting is close to the one of [START_REF] Park | Adaptive execution: Exploration and learning of price impact[END_REF], the main difference being that they assume the predictable part of the returns model to be known and focus only on the estimation of the impact, while we assume both to be unknown and estimate them jointly. We consider an investor who wants to dynamically allocate a portfolio of N assets in order to optimize a multi-horizon Markowitz cost function. Under the assumption that the market dynamics are linear, we propose a novel approach that allows us to cast this problem as a LQ problem, from which the optimal controller can be computed in an efficient way by solving the associated Riccati equation. Our method holds for any linear Markovian return dynamics [START_REF] Abeille | Lqg for portfolio optimization[END_REF] and thus can handle most of the standard financial model. In order to illustrate and discuss the exploration-exploitation issue in portfolio allocation, we focus here a synthetic example that exhibits the key features of the prices dynamics while remaining simple enough to provide intuition. We first describe the setting that we consider, how to encode the problem into a LQ framework and the obtained model, as well as its optimal solution. Additionally, we stress that the existence and uniqueness of the solution is directly related to the non-arbitrage property of the market model, highlighting the one-to-one relationship between the LQ theory and the financial intuition. Despite the generality and the flexibility of this LQ formulation, the main issue is that the parameters of the state-space are unknown and have to be estimated online i.e., while trading, since the impact effects are only generated by trades. Leveraging ideas presented in Ch. 4, we estimate the unknown parameters by Least Square (RLS) and investigate the exploration-exploitation trade-off in this setting. We compare three different algorithms: Certainty Equivalence (CE) which is the greedy strategy that consists in trading optimally given the current RLS estimate, Thompson Sampling (TS) where the exploration is based on randomization, and Optimism in Face of Uncertainty (OFU-LQ), where the exploration is based on optimism. We focus on two instances of the problem: at first, we consider the portfolio allocation problem without risk constraint, and show that the CE algorithm incurs a linear regret, while both TS and OFU-LQ incur a O( √ T ) regret. Then, we address the portfolio allocation problem with risk constraint and show that the regret of the greedy strategy is O(log T ), thus optimal, which means that no additional exploration is needed in this setting. Finally, we discuss why this surprising result is implied by the structure of the controller.

Setting the stage

The portfolio allocation control problem

Setting. We consider an investor whose objective is to dynamically construct a portfolio of N assets: at each time step, it can decide to rebalance his portfolio, using his current knowledge, in order to optimize his gain -encoded in the Profit and Loss (PnL) measure -with respect to a cost function that represents the risk-return trade-off. The seminal work of Markowitz suggests to balance between minimizing the PnL variance (i.e., the risk) and maximizing the PnL expectation (i.e., the return). Formally, one aims to where P nL t,t+1 represents the increment in the investor wealth, γ is a risk-tuning parameter and F t represents the information accumulated so far. To take into account the transaction costs, we explicitly do the distinction between the decision prices p t that are observed at every time step t and used to take the trading decision and the execution prices pt,t+1 , at which transactions occur, that are observed after the trade. Let Q t be the inventory position at time step t i.e., the number of shares of each asset hold at time t, and q t be the trade executed at time t one has:

P nL t,t+1 = Q T t+1 p t+1 -Q T t p t -q T t pt,t+1 , Q t+1 = Q t + q t .
(5.

2)

The introduction of the execution prices is of crucial importance when considering impact effects due to large trade: indeed, the decision prices correspond to the prices at which the supply and demand meet, which by definition is of zero liquidity. Whenever an investor seeks to buy (resp. sell) a large amount of shares, it has to offer a higher (resp. lower) price in order to find enough liquidity. A reasonable model is to assume that the execution prices are on average of the current and next decision prices as pt,t+1 = ηp t+1 + (1 -η)p t . Finally, for sake of simplicity we assume that η = 1 which corresponds to a setting where the investor executes at the next decision price. As a consequence, one can rewrite Eq. 5.2 as

P nL t,t+1 = Q T t r t+1 , r t+1 = p t+1 -p t , Q t+1 = Q t + q t .
As claimed in the introduction, we assume that returns r t+1 follow a linear model, and exhibit both impact effect and return predictability. We encode this into a state space of the form:

α t+1 = Φ α α t + α t+1 I t+1 = Φ I I t + q t r t+1 = β p α t + β I I t + β q q t + r t+1 (5.3)
where α t+1 and r t+1 and zero-mean noises, conditionally independent and of identity variance, for sake of simplicity. Φ α , Φ I , β p , β I and β q are matrices of parameters of appropriate dimensions. {α t } t≥0 is a stochastic process that represents the predictable part of the returns, while {I t } t≥0 is a deterministic process (w.r.t. trades) which takes into account the trades executed so far and quantifies the price impact effects. To visualize the dynamic of such systems, Impulse Response (IR) stands as a convenient tool: it consists in separately perturbing the inputs of the system (here α t+1 and q t ) and let the system evolves. We plot them in Fig. 5.1 for 1 dimensional system (r t+1 ∈ R), and describe the system dynamic. Similarly, to visualize the P nL dynamic, we plot in Fig. 5.2 the round trip response i.e., the P nL trajectory induced by a round-trip that consists in buying 1 share for the first 10 time steps, let the system evolve for the next 10 time steps, and selling back the 10 shares for the last 10 time steps to have a zero position at the end. The decision price is pushed up by the purchase and exhibits a concavity induced by the impact relaxation of past trades. The execution is made at the next decision price. Symmetrically, the decision price is pushed down when selling. On average over the trajectory, the execution price is high for buy transactions, low for sell transactions, which stresses the adverse effect of price impact. Right: PnL trajectory. Since the impact effects act in an adverse manner, the overall PnL is negative. If it increases at the beginning of the trajectory due to the purchase, it is purely artificial as the price mean-revert to zero when the trading stops, thus inducing a loss. This effect is accelerated when selling, because the price impact is symmetric.

110

Chapter 5. Application to Portfolio Construction LQ formulation. We introduce the state variable

x t =    Q t α t I t  
, and write Eq. 5.3 as x t+1 = A x t + B q t + x t+1 , r t+1 = 0 β α β I x t + β q q t + r t+1 ,

(5.4)

where

A :=    I 0 0 0 Φ α 0 0 0 Φ I    , B :=    I 0 I    , x t+1 =    0 α t+1 0    .
Further, we express the cost function c t = γV(P nL t,t+1 |F t ) -E(P nL t,t+1 |F t ) as a function of x t and q t :

γV(P nL t,t+1 |F t ) -E(P nL t,t+1 |F t ) = x T t Q x t + 2x T t N q t + q T t R q t ,
where

Q := 1 2    2γI -β p -β I -β ,T p 0 0 -β ,T I 0 0    , R := 0 , N := 1 2    -β q 0 0    , Q := Q N N ,T R .
Therefore, solving the dynamical Markowitz problem of Eq. 5.1 reduces to find the stationary deterministic control policy π mapping states to trades that minimizes the performance measured by the asymptotic (i.e., infinite horizon) average expected cost (5.5) with x 0 = 0 and q t = π(x t ). Since x t follows the linear dynamics of Eq. 5.4, we retrieve a similar LQ formulation to the one introduced in Subsec. 2.3.2.2

J π = lim T →∞ 1 T E T -1 t=0 x T t Q x t + 2x T t N q t + q T t R q t ,
Optimal control. As opposed to the LQ problem of Ch. 4, the cost matrices Q , R , N are no longer known but depend on the parameters of the returns model. However, they still can be estimated through a linear regression thanks to Eq. 5.3. Additionally, even if (A , B ) is a stabilizable pair, the mapping of the portfolio allocation problem into a LQ problem implies that the matrix Q is no longer positive definite which breaks the standard LQ assumptions (Asm. 2.3.1). Fortunately, one can still compute an optimal control under a more flexible criterion that we link to non-arbitrage. Assumption 5.2.1 (Noise). The noises { α t } t and { r t } t are F t -martingale difference sequences, conditionally independent, where F t is the filtration which represents the accumulated information up to time t. Furthermore, we assume that V( α t+1 |F t ) = I and V( r t+1 |F t ) = I for all t ≥ 0.

Assumption 5.2.2 (LQ). The matrices Φ α and Φ I are stable and the matrices A , B , Q , R , N are such that there exists a control matrix K which stabilizes the system (i.e., A + B K is stable) such that the hermitian matrix Ψ K (z) > 0 for all |z| = 1, z ∈ C, where

Ψ K (z) = Y T K (z -1 ) (Iz -1 -A ) -1 B I T H (Iz -A ) -1 B I Y K (z) Y K (z) = I + K(Iz -A -B K) -1 B .
Asm. 5.2.2 relies on a criterion over the Popov function Ψ K to guarantee the existence and uniqueness of an admissible controller (see [START_REF] Molinari | The stabilizing solution of the discrete algebraic riccati equation[END_REF]. While this stands as a technical tool for LQ in general, it is possible to link it to a financial argument in our specific portfolio allocation setting. In Lem. 5.2.1, we show that this is equivalent to the fact that the system allows no dynamical arbitrage i.e., that, in the absence of predictability, there exists no strategy which guarantees a positive profit. This "no free lunch" assumption is standard in finance, and justifies the validity of Asm. 5.2.2. We postpone the proof to App. 5.A.

Lemma 5.2.1. We denote as l p := {(x n ) n≥0 ∈ R N s.t. n≥0 |x n | p < ∞} Consider the deterministic system associated with Eq. 5.3 where the sequences of noise { α t } t and { r t } t are set to zero. Let RT be the set of admissible round-trip defined as RT = {q = (q 0 , q 1 , . . .

) ∈ l 1 ∩ l 2 }, if γ = 0, RT = {q = (q 0 , q 1 , . . . ) ∈ l 1 ∩ l 2 s.t. (Q 0 , Q 1 , . . . ) ∈ l 1 ∩ l 2 } if γ ∈ (0, ∞),
where

l 1 = {(q n ) n≥0 s.t. n≥0 |q n | < ∞} and l 2 = {(q n ) n≥0 s.t. n≥0 |q n | 2 < ∞}.
Let Ψ K (z) be the Popov function of Asm. 5.2.2. Then, Ψ K (z) > 0 if and only if, for any q ∈ RT , the PnL trajectory induced by q is such that ∞ t=0 P nL t,t+1 ≤ 0.

Under these assumptions, a slightly modified version of Thm. 2.3.2 (see Ch. 2, Subsec. 2.3.2) provides that the optimal policy is linear with the state, i.e., π * = Kx, where the optimal gain K is computed as

K = -R + B ,T P B -1 [B ,T P A + N ,T ], P = Q + A ,T P A + [A ,T P B + N ]K.
P is the unique solution to the Riccati equation associated with the control problem.

The optimal average cost is J

= J π * = Tr(P Σ x ) with Σ x = V( x t+1 |F t ) =    0 0 0 0 I 0 0 0 0   .
Finally, we also have the closed-loop matrix A + B K is asymptotically stable.

The learning problem

Structure of the parameters. As claimed in introduction, one of the major issue in this dynamic portfolio allocation problem is that the dynamic of the market is unknown by nature which translates in unknown parameters contained in Φ α , Φ I , β α , β I , β q . However, one can reasonably assume that part of those parameters are known to the controller, or at least can be estimated separately. We make the following assumptions and discuss their validity. Since the variable α t encodes the prediction made by the investor about the future prices move, it is reasonable to assume that it is observed at each time step. Furthermore, its dynamics is independent from the sequence of trades as it takes into account the uncontrolled prices move. As a result, Φ α can be estimated separately and its estimation converges almost surely to the true parameters (as provided by autoregressive process estimation) which implies that this does not participate to the exploration-exploitation dilemma. For sake of convenience, we assume that Φ α is known beforehand. Finally, the minimal representation assumption is here to ensure that the prediction model is not over-parametrized (all predictors are different) and significant (all predictors influence the returns dynamic). On the other hand, there is no reason for Φ I to be known a priori. Moreover, the impact variable I t has no physical meaning but takes into account the fact that the market digests slowly the modification induced by trades. Said differently, it is not a hidden variable but a latent variable that is here to encode the structure of the impact decay. However, one can overcome this difficulty using an over-parametrized state-space model of higher dimension for the learning than the true one. This idea is already used by [START_REF] Park | Adaptive execution: Exploration and learning of price impact[END_REF]: consider the 1-dimensional case where r t ∈ R and I t ∈ R; focusing on the impact effect only, the return modeling has a dynamic characterized by

I t+1 = Φ I I t + q t ; r t+1 = β I I t + β q q t .
Under Asm. 5.2.2, Φ I is stable which means that |Φ I | < 1 and by invariance, one can set Φ I ≥ 0. Hence, it is possible to approximate the above model using a vector Φ I = (Φ 0 I , . . . , Φ s I ) which spans [0, 1[ together with an augmented state Ĩt ∈ R s whose dynamic follows

Ĩt+1 =     Φ 0 I 0 . . . 0 Φ s I     Ĩt +     1 . . . 1     q t ; r t+1 = β T I Ĩt + β q q t .

Setting the stage 113

In this case, each component of Ĩt represents a different impact model whose effect on the return are weighted by the component of β I . As a result, the underlying difficulty in learning parameters Φ I , β I and β q can be summarized into the estimation of β I and β q as long as Φ I is sufficiently dense in [0, 1[. In particular if Φ I contains the true value Φ I , the augmented model includes the true one (where β I components are all zero but one equal to β I ) and the two estimation problems are equivalent. This stresses that Asm. 5.2.4 does not reduce the learning complexity of the problem, but acts as a choice of representation for the impact model. In particular, this does not influence the exploration-exploitation problem. Neglecting the approximation error, it is therefore possible to assume that Φ I is known by the controller and since, given Φ I , I t can be reconstructed from past trades, that the impact variable is observed at each time step. Finally, under Asm. 5.2.3 and 5.2.4, the matrices A , B of Eq. 5.4 are known while the unknown parameters are contained in β q , β α , β I . We collect them in θ ,T = [β q , β α , β I ].

Since θ is used to construct the costs matrices Q , R , N , it determines the LQ problem. Thus, for any θ satisfying Asm. 5.2.2, we denote as P (θ), K(θ) and J(θ) the optimal LQ quantities associated with this parametrization.

Regret definition. We consider the standard online learning setting where at each step t the learner receives the current state x t as input, the current return r t as observation, it executes a control q t and it suffers the associated cost

c t = c θ (x t , q t ) = x T t Q x t + 2x T t N q t + q T t R q t
The system then transitions to the next state x t+1 and the next observation r t+1 is generated according to Eq. 5.4. The learning performance is measured by the cumulative regret over T steps, where the costs cumulated over time are compared to the minimal cost obtained on average by the optimal policy. Formally we define

R T (θ ) = T t=0 c t -J(θ )
RLS estimates. From a learning perspective, this LQ problem slightly differs from the one of Ch. 4 although from a control point of view they share the same structure. In particular, the unknown parameter θ is used to defined the cost function rather than the state dynamics.3 On the other hand, one still needs an estimate about its value to compute the optimal control, thus requiring basic tools for the estimation of the parameter θ . Let (q 0 , . . . , q t ) be the sequence of trades, (α 0 , . . . , α t ) the sequence of predictors, (I 0 , . . . , I t ) the sequence of impact variables, and let (r 1 , . . . , r t+1 ) be the corresponding returns (i.e., observations) generated according to Eq. 5.4. For any regularization parameter λ ∈ R * + the regularized least-squares estimate (RLS) and the associated design matrix are defined as

V t = λI + t-1 s=0 z s z T s ; θ t = V -1 t t-1 s=0 z s r T s+1 , (5.6)
where z T t = (q T t , α T t , I T t ).

Algorithms

In Sec. 5.2, we showed how the portfolio allocation problem of Eq. 5.1 can be cast as an LQ problem, which allows us to compute the optimal control given the parameters of the price dynamics. Additionally, we discussed how, under reasonable assumptions about the parameters knowledge, one can use RLS to estimate the unknown component, thus providing us with all the material needed to perform adaptive strategies. We present in this section the algorithms CE, TS and the OFU-LQ (derived from the algorithm presented in Fig. 2.10, for the specific portfolio allocation problem instance) that we consider to address the exploration-exploitation trade-off in portfolio optimization. All of them are based on policy updates that consists in choosing a new parameter and following the policy which is optimal w.r.t this choice. To ensure that those parameters are coherent with Asm. 5.2.2 and 5.2.3, we constrain them to belong to an admissible set S that will be specified in the next section. We report in Fig. 5.1 the common structure of the three algorithms and then precise separately the sub-routine TRIGGER and SELECT that are specific to CE, TS and OFU-LQ and determine respectively the frequency of updates and the choice of the parameters.

Input: θ0 , V 0 = λI, δ, T , S, τ 1: Set δ = δ/(8T ) 2: for t = {0, . . . , T } do 3:

if TRIGGER(t, t 0 , τ, V 0 , V t ) = 1 then 4:

Select a new parameter θ t = SELECT( θ t , V t , δ , S) Execute control q t = K( θ t )x t 9:

Observe state x t+1 , observation r t+1 and suffer cost c t = c θ (x t , q t ) 10:

Compute V t+1 and θ t+1 using Eq. 5.6 11: end for Greedy strategy. First, we describe the method known as the Certainty Equivalence (CE) in the control literature [START_REF] Kumar | Stochastic systems: Estimation, identification, and adaptive control[END_REF], that is a greedy strategy where the optimal control is computed with respect to the current RLS estimate. While this algorithm is known to fail to achieve sub-linear regret in standard bandit problems, we use it as a base case to highlight the need or not for explicit exploration, and surprisingly notice that it offers very good performance as soon as we consider the risk constrained problem (see Subsec. 5.4.2). To reduce the computational complexity of the algorithm, we keep the same policy for 5.3. Algorithms 115 episode of constant length according to the TRIGGER sub-routine (see Fig. 5.2). Then, we select the greedy parameter θ t ∈ S by projecting θ t onto S. Randomized strategy. To overcome the potential failure of the CE algorithm, we consider the TS algorithm of Ch. 4. As discussed in the previous chapters, the idea is to sample new parameters θ t taking into account the knowledge acquire so far, around the current RLS estimate θ t but on the basis of the uncertainty i.e., with a variance that depends on the design matrix V t . Each time the policy is re-evaluted, a parameter θ t is sampled, and the optimal policy w.r.t to θ t is executed. Formally,

θ t = R S ( θ t + β t V -1/2 t η t )
, where R S is the rejection sampling operator associated with the admissible set S, θ t is the RLS-estimate, V t is the design matrix and each entry of the perturbation matrix η t ∈ R (n+d)×n is a random sample drawn i.i.d. from N (0, 1). The SELECT sub-routine in presented in Fig. 5.4, while the TRIGGER sub-routine is identical to the CE (Fig. 5.2). Optimistic strategy. Finally, we also compare the performance of CE and TS to the OFU-LQ algorithm introduced in (Abbasi-Yadkori and Szepesvári, 2011) presented in Fig. 2.10. We recall the TRIGGER and SELECT sub-routine in Fig. 5.5 and 5.6, based respectively on the doubling-schedule and on optimism.

Chapter 5. Application to Portfolio Construction

Input: V 0 , V t 1: if det(V t ) ≥ 2 det(V 0 ) then 2: TRIGGER(V 0 , V t ) = 1 3: V 0 = V t 4: else 5: TRIGGER(V 0 , V t ) = 0 6: end if Figure 5.5 -OFU-LQ TRIGGER sub-routine. Input: θ t , V t , δ ,S 1: Compute β t = β t (δ ) from Eq. 2.8 2: Define E RLS t = {θ s.t. Tr[( θt -θ) T V t ( θt -θ)] ≤ β 2 t } 3: Find θ t = argmin θ∈E RLS t ∩S J(θ) Figure 5.6 -OFU-LQ SELECT sub-routine.
From a theoretical perspective, applying the results of Ch. 4 and the one of Abbasi-Yadkori and Szepesvári (2011), one obtains the following guarantees for the regret of TS of OFU-LQ.

Corollary 5.3.1 (From Thm. 4.5.1). Consider the portfolio allocation problem in Eq. 5.1 of dimension n = 1. Under Asm. 5.2.1,5.2.2,5.2.3 and 5.2.4 for any 0 < δ < 1, the cumulative regret of TS Algorithm 5.1 over T steps is bounded w.p. at least 1 -δ as

R(T ) = O log(1/δ)T .
Corollary 5.3.2 (From (Abbasi-Yadkori and Szepesvári, 2011)). Consider the portfolio allocation problem in Eq. 5.1 of arbitrary dimension. Under Asm. 5.2.1,5.2.2,5.2.3 and 5.2.4 for any 0 < δ < 1, the cumulative regret of OFU-LQ Algorithm 5.1 over T steps is bounded w.p. at least 1 -δ as

R(T ) = O log(1/δ)T .
Those two results are stated as Corollary as they can be derived easily, following the same proof structure, from Ch. 4 and (Abbasi-Yadkori and Szepesvári, 2011). The minor differences come from the regret decomposition (the term R RLS no longer appears), the fact that x t is bounded w.h.p. since the matrices A + B K( θ t ) are stable by Asm. 5.2.2, and from a similar relationship between gradient and control (Prop. 4.A.1). Finally, despite the fact that the CE algorithm is widely used in practice (see e.g., Polderman

Experiments 117

1986), its performance relies on the consistency of the parameter estimation which may or may not occur. We exhibit this behavior through numerical experiments and discuss it in Sec. 5.5.

Experiments

In this section, we provide numerical experiments for the CE, TS and OFU-LQ algorithms. For sake of clarity, we consider the 1-dimensional case where r t ∈ R and use the following values for the true parameters of the system: Φ α = 0.9, Φ I = 0.7, β α = 0.1, β I = -1.5 and β q = 5. Notice that the impact parameters β I and β q are significantly larger than the prediction parameter β α which takes into account the fact that for large orders, the impact effects are huge compare to the predictions. Finally, the TRIGGER sub-routine of CE and TS is run with τ = 50. In the next subsections, we consider separately the case γ = 0 (no risk constraint) and γ = 0.1 (with risk constraint). We first detail the shape of the admissible set S, and then present the results of the strategies both in term of regret and in term of consistency of the estimation.

Optimal allocation without risk constraint

Whenever γ = 0, the optimization is made without taking into account the risk of the allocation strategy i.e., no control is made about the amplitude of the position Q t , and the objective is to maximize the profit only. This translates in the fact that the optimal policy only depends on the current prediction and impact variables α t and I t . Formally, the control matrix has the shape of K = 0 K α K I . However, the problem is still well defined due to impact effects: indeed, since trading is costly, it constrains the amplitude of the trades. The set of admissible parameters S which is the set of parameters such that there exists no dynamical arbitrage imposes the impact effect to be adversarial. Formally, one has

S := {θ = (β q , β α , β I ) s.t. β q > 0, β α = 0 and β q ≤ -(1 -Φ I )β I }
We first consider the CE algorithm and independently run multiple experiments. The estimate trajectories are plot in Fig. 5.1. It turns out that, on each trajectory, the sequence of estimates do converge, but that the convergence happens only in distribution. This is clear since running the same experiment, one obtains different limit points. Moreover, it is possible to characterize the support of this distribution, which depends on the parameter θ (see Subsec. 5.5.2). Finally, despite the fact that θ belongs to this set, the limit distribution does not concentrate around it, which means that no consistency is achieved and that the control can converge, with fixed probability, to a sub-optimal one. As a result, the regret of this strategy is linear (see Fig. 5.2). This is a well-known issue in RL and it motivates the use of more explorative algorithms, such as TS and OFU-LQ, that, thanks to randomness or optimism, are able to discriminate the true parameter and overcome the limitation of the CE. We plot in Fig. 5.2 the regret bounds for the 3 strategies, on average and in high probability. First, as expected from Fig. 5.1, on sees that the regret of the CE strategy is linear and that the high probability bound is significantly worse than the average bound. This is due to the fact that for lots of trajectories, the estimates do converge to a point close to θ . However, there is a small, yet fixed, probability that θ t converges to a value which is far from the true one, incurring point-wise a large linear regret. This is not the case of TS and OFU-LQ which, accordingly with Cor. 5.3.1 and 5.3.2, achieve a O( √ T ) regret, although, it may take some time to be smaller than the CE's because of the "expensive" exploration phase. Finally, we note that TS performs better than OFU-LQ (by a constant), which is often observed empirically although not provided by the theory. This may be due to the fact that optimism induces a more aggressive exploration than randomness, and that such aggressive behavior is not needed in practice and mostly comes from the looseness of the theoretical bounds used to derive the algorithms.

Optimal allocation with risk constraint

We now consider the case γ = 0.1, which corresponds to the setting where the cost function includes a risk term. The optimal policy becomes function of the current position Q t as well as the current prediction and impact variables α t and I t . Formally, the control matrix has the shape of K = K Q K α K I . The set of admissible parameters S simplifies into

S := {θ = (β q , β α , β I ) s.t. β q > 0, β α = 0 and β I ≤ 0}
This modification in the structure of the control K has a major consequence since it implies that the CE estimation is now consistent. To illustrate this effect, we plot in Fig. 5.3 the average trajectories of the estimates starting from different initial points. Since they all converge to θ , the convergence no longer occurs in distribution but almost surely while the rate of convergence is still 1/t. As a result, the regret of the CE is in log(T ) which is way better than the regret of TS and OFU-LQ. We show the average and high probability regret bounds of the CE algorithm in Fig. 5.4. While this situation is unusual in RL, where greedy strategies often fail to achieve a sub-linear regret, it provides support to the fact that this strategy had been widely used by the control community. In the next section, we discuss this result and show why adding a risk constraint to the portfolio allocation problem makes the greedy strategy consistent, and thus optimal.

From closed-loop consistency to consistency

In this section, we discuss why the RLS estimates of the CE strategy converge or not to the true parameters θ . We first show that the limit estimate of the CE belongs to a specific set, and then that it reduces to θ , thus ensuring consistency, when risk constraint is considered.

Closed-loop consistency

We introduce the set of self-coherent parameters which ensures the closed-loop consistency. The closed-loop consistency corresponds to the fact that the closed-loop dynamics of the system i.e., the dynamics of the system once controlled, is well estimated. The set of self-coherent parameters corresponds to the set of θ such that the predicted closed-loop dynamics is asymptotically equal to the observed one. We denote this set as SC which is formally defined as

SC := {θ ∈ S s.t (θ -θ) T z T 2 -→ T →∞
0, where q t = K(θ)x t for all t ≥ 1}, (5.7)

where the process {z t } t ≥ 0 is generated by following the policy q = K(θ)x. We first characterize the shape of SC and then show that the CE estimates converge to this set.

5.5. From closed-loop consistency to consistency 121 Lemma 5.5.1. Under Asm. 5.2.1,5.2.2,5.2.3 and 5.2.4, the set of self-coherent parameters satisfies

SC = θ ∈ S s.t θ T    K(θ) 0 I 0 0 0 I    = θ ,T    K(θ) 0 I 0 0 0 I    .
The result of Lem. 5.5.1 provides a characterization of the set SC which is independent of the covariate z t . We present the main ideas of the proof and postpone the formal derivation to App. 5.B. We use the structure of z t to introduce the control matrix K(θ) and the state process x t and show that, under the assumptions of Lem. 5.5.1, the state process admits a stationary distribution with positive definite variance. By construction, since

q t = K(θ)x t , x T t = (Q T t , α T t , I T t ), z T t = (q T t , α T t , I T t ), one has, for all t ≥ 0, z t =    0 0 0 0 I 0 0 0 I    x t +    I 0 0    K(θ)x t =    K(θ) 0 I 0 0 0 I    x t ,
where x t+1 = A +B K(θ) x t + x t+1 . Under Asm. 5.2.2, the matrix A c (θ) = A +B K(θ) is stable and so is the process {x t } t which admits a stationary distribution. Denoting as x ∞ the random variable following the stationary distribution, Asm. 5.2.1 guarantees that E(x ∞ ) = 0 and that V(x ∞ ) = Σ ∞ . Therefore, computing the expectatio of Eq. 5.7, one obtains that

SC ⊂ θ ∈ S s.t Tr (θ -θ) T    K(θ) 0 I 0 0 0 I    Σ ∞    0 0 K T (θ) I 0 0 I    (θ -θ) = 0 .
A sufficient condition to ensure Lem. 5.5.1 is thus that Σ ∞ is positive definite. Intuitively, it requires all components of x t to be persistently excited by x t+1 and that there exists no linear relationship between them. Notice that x t+1 has the specific structure x,T t+1 = (0, α,T t+1 , 0), which encodes the fact that the prediction α t are naturally excited by the random noise α t+1 , while the inventory position Q t and the impact variable I t are function of the prediction process {α t } t , and thus of { α t } t through the trading policy q = K(θ)x. As a result, all component of x t are persistently excited by { α t } t . Further, the structure of the dynamic and the control matrix K(θ) for any θ ∈ S impose that there exists no linear relationship between them, provided that Φ I = I i.e., provided that the dynamic of Q t and I t are disjoint. Since this is guaranteed by Asm. 5.2.2, one obtains the desired result.

We now show that the sequence of the CE estimates converge to SC. Lemma 5.5.2. Let θ t be the sequence of RLS estimates of the CE algorithm. Then, θ t converges in distribution to a random variable θ ∞ and θ ∞ ∈ SC a.s.

Applying the RLS properties of Prop. 4.2.4 to the portfolio estimation, one obtains: Corollary 5.5.1. Let λ ≥ 1, for any arbitrary F t -adapted sequence of control (q 0 , . . . , q t ), let θ t be the sequence of RLS estimates defined in Eq. 5.6, one has t s=0 (θ -θ t ) T z t = O √ T .

Cor. 5.5.1 guarantees that the on-policy error of the estimation is cumulatively bounded and that this is a structural property of the RLS, since it holds for any sequence of trades. As a consequence it implies that asymptotically, (θ -θ T ) T z T -→ T →∞ 0.

We now adopt an asymptotic reasoning. First, notice that without further assumption on the design matrix V t , no consistency can be guarantee for the RLS estimates, although the convergence still holds in distribution. We introduce the random variable θ ∞ which corresponds to the limit estimate, so that θ T d -→

T →∞ θ ∞ . Further, the regularity of the function K(θ) ensures that the control performed by the CE satisfy K( θ T )

d -→ T →∞ K( θ ∞ ).
Using the fact that the trades executed by the CE algorithm follows, neglecting the projection onto S, q t = K( θ t )x t , one obtains that θ ∞ satisfies Eq. 5.7 i.e., that θ ∞ ∈ SC.

Self-coherence set

Thanks to Lem. 5.5.2, we can exhibit, for any parametrization θ , a sufficient condition for the consistency of the CE estimates, or equivalently, a necessary condition for the failure of this greedy strategy. To support the experiments of Sec. 5.4, we illustrate both behavior depending on the presence of risk constraint.

Allocation without risk constraint. As discussed in Sec. 5.4.1, whenever γ = 0, the optimal policy does not depend on the current inventory position: for any θ ∈ S, K(θ) = 0 K α (θ) K I (θ) . As a result, the set of self-coherent parameters SC becomes: SC = θ = (β q , β α , β I ) ∈ S s.t.

β q K α (θ) + β α β q K I (θ) + β I = β q K α (θ) + β α β q K I (θ) + β I .

Since K(θ) is known, the set SC is characterized block-wise by two equations of three variables, and thus not reduced to a singleton. As a consequence, the CE asymptotic estimates can potentially be distributed onto this set, which stands as a necessary condition for the failure of the greedy strategy. This condition is common to RL problems that require additional exploration: to illustrate this, we plot in Fig. 5.1 the shape of such sets for the portfolio allocation problem (similar to the r.h.s of Fig. 5.1), the LB problem of Ch. 3 and the LQ problem of Ch. 4. Notice that if this provides us with a necessary condition, it is not a sufficient one. Despite the fact that the support of the distribution of the asymptotic estimates is not a singleton, it is does not imply that those points are attracting w.r.t the RLS estimation. Moreover, for specific problems, it is possible that this set coincides with the set of parameters that induces optimal control defined as K := {θ ∈ S s.t. K(θ) = K(θ )}. Therefore, this argument does not replace a regret analysis but provides some insight about the potential behavior of the estimation process. Finally, notice that the more we constrain the admissible parameter set S, the more we may reduce SC which is coherent with the intuition that as we inject knowledge about the parameters in the learning process (through the constraint), we make the exploration-exploitation trade-off easier to solve.

Allocation with risk constraint. When the optimal allocation is made under risk constraint (γ > 0), the optimal policy depends on the current inventory position thus, for any θ ∈ S, K(θ) = K Q (θ) K α (θ) K I (θ) with K Q (θ) = 0. Therefore, the set of self-coherent parameters SC becomes:

SC = θ = (β q , β α , β I ) ∈ S s.t.    β q K Q (θ) β q K α (θ) + β α β q K I (θ) + β I    =    β q K Q (θ) β q K α (θ) + β α β q K I (θ) + β I    ,
and is now characterized by three equations of three unknown variables. For sake of simplicity, consider the 1-dimensional case where θ ∈ R 3 . Since K(θ) is known to the controller, solving the three equations in SC leads to β q = β q , β α = β α and β I = β I . As a consequence, SC = {θ }, so Lem. 5.5.2 ensures that the CE estimates are consistent. Additionally, it allows us to characterize the convergence rate. Notice that by Least β q,t K Q ( θ t ) β q,t K α ( θ t ) + β α,t β q,t K I ( θ t ) + β I,t

    -     β q K Q ( θ t ) β q K α ( θ t ) + β α β q K I ( θ t ) + β I     ∼ t→∞ 1 t .
which implies that θ t -θ ∼ t→∞ 1 t . As a consequence, in the presence of risk constraint, the average regret of the CE strategy is bounded by O log(T ) .

Discussion. We end this section stressing how the difference in the structure of the controller, and hence the trades, modifies the behavior of the CE estimation. First, looking at Eq. 5.4, one observes that the inventory position Q t does not affect the open-loop i.e., uncontrolled dynamic of the returns, which only depends on α t , I t and q t . When the trading policy is of the form q t = K α α t + K I I t , which corresponds to the risk-free case, this creates an ambiguity in the prediction of the returns, that the CE strategy is unable to eliminate. A standard way to overcome this issue is to perturb or regularize the control to enhance the exploration, which is somehow what TS or OFU-LQ do. Following this intuition, one can understand trading policies of the form q t = (K α α t + K I I t ) + K Q Q t as a perturbed or regularized control. Moreover, in the presence of risk constraint, it coincides with the shape of the optimal control, which means that perturbing/regularizing is now optimal, thus the additional exploration is given for free (in term of regret). While this point of view is clear when the perturbation is exogenous i.e., independent of the state dynamic, it is less obvious here as Q t also depends on {α t } t and {I t } t . This is at the core of the proof of Lem. 5.5.1, which uses the fact that the dynamic of Q t is by construction different from the one of α t and I t , and thus, relatively independent. As a result, it still acts as an "exogenous" perturbation, maintaining the validity of the intuition.

Conclusion

In this Chapter, we present an application of adaptive strategies in LQ problems for portfolio allocation. We first highlight the interest of this framework, showing that under the assumption of linear Markovian prices dynamics, one can efficiently encode and solve the allocation problem with price impact and return predictability. In addition, we show that the LQ technical assumptions can be rephrased in term of non-arbitrage, thus allowing to use the richness of the Riccati theory while maintaining financial intuition. Further, we discuss the structure of the parameters and present an approximation procedure that enables us to use RLS to estimate the model. We recall and compare three strategies, both in terms of regret and consistency of the estimation, respectively based on greedy updates (CE), randomness (TS) and optimism (OFU-LQ). The obtained results differ significantly depending on the presence of risk constraint: in the risk-free setting, we observe that the greedy strategy fails to tackle the exploration-exploitation trade-off, while specifically designed algorithm such as TS

Conclusion 125

and OFU-LQ does, and we retrieve the linear and square-root regret bounds expected from the theory. On the other hand, under risk constraint, we show that the greedy strategy becomes consistent and exhibit a log(T ) average regret. We explain how this modification induces a different structure in the control, and why it guarantees the consistency, introducing the set of self-coherent parameters and comparing its shape in both cases. This has several implications: first, it stresses that the CE strategy may perform well in adaptive problems that exhibit a specific structure and explains its popularity in the control community. From a practical perspective, one shouldn't discard it a priori, even though it is usually expected to suffer a linear regret in standard RL problems. Finally, we believe that the study of the self-coherent parameters set provides a generic characterization of this issue in parametrized adaptive control problem, of interest beyond the scope of portfolio allocation.

We make use of Lemma 5.A.1 to rephrase (5.9) in terms of v sequence: for any q ∈ Q, let K be a stable control and v ∈ l 1 ∩ l 2 sequence such that q t = Kx t + v t . Denoting as v(z) the z-transform of v, the z-transform q(z) is:

q(z) = Y K (z)v(z).
Finally, equation (5.9) becomes: J(q) = |z|=1 q T (z -1 )Φ(z)q(z)dz = |z|=1 v T (z -1 )Φ K (z)v(z).

Therefore, rephrasing Thm. 5.A.1, there exists a unique symmetric stabilizing solution to the Riccati equation if and only if for some (and hence all) K such that A + BK is stable Φ K (z) > 0 for all |z| = 1 if and only if |z|=1 v T (z -1 )Φ K (z)v(z) > 0 for all v ∈ l 1 ∩ l 2 if and only if J(q) > 0 for any q ∈ Q.

5.A.4 From admissible trade sequence to round-trip

Finally, to prove Lem. 5.2.1, one just has to show that the set of admissible sequence coincides with the one of round-trip trajectories. Recalling the definition, one has RT = {q = (q 0 , q 1 , . . . ) ∈ l 1 ∩ l 2 s.t. Q = (Q 0 , Q 1 , . . . ) ∈ l 1 ∩ l 2 } if γ ∈ (0, ∞), RT = {q = (q 0 , q 1 , . . . ) ∈ l 1 ∩ l 2 }, if γ = 0.

We first deal with the generic case where γ ∈ (0, ∞) and then discuss the specific instance where γ = 0. By definition, Q ⊂ RT so we just need to prove that for any q ∈ RT , the associated state sequence is such that x ∈ l 1 ∩ l 2 . To do so, we denote as before q(z), Q(z) and x(z) the z-transform of q t , Q t and x t respectively. Then one has:

x(z) = (Iz -A) -1 Bq(z),

x(z) = (Iz -A) -1 B(z -1)Q(z).

Multiplying by (z -1) and taking the limit when z → 1 one has, since (z -1)(Iz -A) -1 converges to a constant matrix (finite) H:

(z -1)(Iz -A) -1 -→ z→1 H < ∞ =⇒ (z -1)x(z) ∼ z→1 H(z -1)Q(z).
Thanks to the final value theorem, one gets x t ∼ t→∞ Q t and since Q ∈ l 1 ∩ l 2 by definition of RT so does x. As a consequence, Q = RT . Finally, in the specific case where γ = 0, one does not need to stabilize the position Q t and thus does not need to feedback the current trade q t on the current position Q t . As a result, the same derivation can be applied to the sub-system with internal state x t = α T t I T t T which follows a stable dynamic by Asm. 5.2.2. Therefore, one directly obtains that q ∈ RT implies that x ∈ l 1 ∩ l 2 .

5.A.5 Plugging everything together

Thanks to the previous steps, we have that there exists a unique solution to the Riccati equation (since (A, B) is stabilizable) if and only if, for any q ∈ RT , J(q) > 0. Noticing that in the absence of noise, the cost function is equal, in term of portfolio allocation to J(q 0 , q 1 , . . . ) = ∞ t=0 -P nL t,t+1 proves Lem. 5.2.1. Because, in the absence of noise, there is no price predictability, this criterion states that every round-trip must be non-profitable so that impact effects are modeled to act in an adverse manner. Under such guarantee, a unique solution to the portfolio allocation exists.

5.B Proof of Lem. 5.5.1.

As discuss in Sec. 5.5.2, one just need to prove that, for any θ ∈ S, the state process generated by the trading policy q = K(θ)x admits a stationary distribution, with positive definite variance. Since A and B are known, by definition, for any θ ∈ S, A c (θ) = A +B K(θ) is stable. Therefore, x t admits a stationary distribution. Denoting as x ∞ the random variable which follows this stationary distribution, one has E(x ∞ ) = 0 and V(x ∞ ) = Σ ∞ where Σ ∞ = A c (θ)Σ ∞ A T c (θ) + Σ x , with Σ x = CC T and C T = 0 I 0 .

Further, we show the pair C T , A T c (θ) is observable.

Writing K(θ) = K Q (θ), K α (θ), K I (θ) and using Def. 5.B.1, this is provided if the matrix O T is of full row-rank. Formally, one has

O T =    0 K α (θ) (I + K Q (θ))K α (θ) + K α (θ)Φ α + K I (θ)K α (θ) I Φ α Φ 2, α 0 K α (θ) K Q (θ)K α (θ) + K α (θ)Φ α + (Φ I + K I (θ))K α (θ)    ,
which, is full-rank if and only if

K α (θ) (I + K Q (θ))K α (θ) + K α (θ)Φ α + K I (θ)K α (θ) K α (θ) K Q (θ)K α (θ) + K α (θ)Φ α (θ) + (Φ I + K I (θ))K α (θ)
is full-rank. Algebraic manipulations ensure that it is equivalent to consider the matrix Then, provided that the matrix A is stable, the solution is unique, symmetric and positive semi-definite. Further, if the pair (C, A) is observable, the solution is positive definite.

Chapter 6

Summary and Future Work

This chapter provides a summary of methods and analyses presented in this thesis, highlights some open questions that lay ground for future work and discusses the Linear Quadratic Gaussian (LQG) extension that consists in adding partial observability to the LQ problem of Ch. 4.

Summary

This thesis has been motivated by the study of Reinforcement Learning (RL) algorithms in linearly parametrized systems, that address the exploration-exploitation trade-off in sequential decision making. While linear models seem somehow limited to reflect the true dynamics of real systems, they remain accurate enough for lots of practical applications and offer many advantages that balance the induced approximation error. First, as stressed in the specific example on portfolio construction (see Ch. 5), they consist in robust, yet flexible models and thus can encode complicated features (e.g., dynamical predictability and impact effects) while maintaining tractability of both the solution of the control problem and of the estimation problem. Second, as a specific instance of parametrized system, they can handle problems with large and/or continuous state and action spaces, for which designing algorithms that properly trade-off exploration and exploitation stands as one of the main challenges in RL. Third, from a theoretical perspective, their relative simplicity allows one to focus on the underlying difficulty in the analysis of algorithms and to exhibit the key aspect of their functioning. Finally, they are the cornerstone of generic parametrized problems, and a good understanding of analyses in those settings is a prerequisite to any further developments. Two popular principles have been introduced to tackle the exploration-exploitation trade-off: one based on optimism and one based on Thompson sampling (TS). While the former have been intensively studied over the last decades and gave rise to many algorithms for which theoretical guarantees have been provided, the latter has recently generated significant interest due to the impressive empirical performance of the induced algorithms. We focused in this thesis on Thompson sampling-based algorithms and analyzed their performance in the Linear Bandit (LB) of Linear Quadratic (LQ) control problems. While TS is originally build on Bayesian ideas, we studied it in the frequentist setting and stressed the randomized nature of its functioning. In Ch. 3, we analyzed the regret of TS in the LB setting, and derived an alternative proof that sheds new light on the functioning of the algorithm. In particular, we leveraged the structure of the problem to show how the regret is related to the sensitivity of the objective function, and how the structure of the algorithm (i.e., selecting optimal arm w.r.t. a chosen parameter) takes this sensitivity into account. Then, we explain how the random nature of TS selects arms that control the sensitivity of the objective function, and hence the regret. Additionally, our analysis holds for any appropriate sampling distribution, which stresses that randomization is the key feature of TS while prior/posterior Bayesian update only stands as a convenient tool to obtain the sampling distribution. Further, our proof relies on the property of the objective function and can be readily applied to problems whose objective function shares the same structure (e.g., generalized linear model and regularized linear optimization).

In Ch. 4, we leveraged our novel analysis in LB and extended it to the LQ control problem. In this more complicated setting, we showed that the functioning of the algorithm is similar to the one of Ch. 3. We stressed the link between the actual actions chosen by TS and the Jacobian of the optimal value function, and showed how the randomization of TS induces actions that do control the sensitivity of the optimal value function. Further, we exhibited the need to trade-off the frequency of sampling parameters and the frequency of switches in the control policy, and showed that standard lazy update schemes induce at best an overall regret of O(T 2/3 ). We overcame this issue by deriving a novel bound on the regret due to policy switches, thus allowing to update parameters and the policy at each step and overcome the limitations due to lazy updates. As a result, we proved O( √ T ) regret bound for the regret of TS in LQ.

In Ch. 5, we presented an application example for portfolio allocation problems. We highlighted the interest of the LQ framework, by showing that it is possible to cast a complicated dynamical allocation problem, that takes into account the main features of the prices dynamics, into a LQ control problem. We investigated the explorationexploitation trade-off in this specific setting and compared the performance of TS with the optimistic-based algorithm for LQ (OFU-LQ) and with a naive greedy strategy. While TS and OFU-LQ achieve a O( √ T ) regret as expected from the theory, we showed that depending on whether or not risk constraint is considered, the greedy strategy suffers a O(T ) or a O(log T ) regret. We discussed this surprising result and showed that the greedy strategy may or may not be consistent. Further, we exhibited the support of the distribution of the limit estimates and explained the failure or the success of the greedy strategy depending on the shape of this set.

Future Work

The analyses provided in this thesis open a number of interesting questions and research directions. In particular, we discussed in Sec. 3.7 the need for optimism that is at the core of the TS analysis, and hinted that this is a sufficient condition rather than a necessary one. Investigating this aspect of TS is highly challenging because it requires to quantify how 'informative" actions induced by non-optimistic samples are, but may improve the regret bound by √ d and thus prove the empirical evidence that TS offers similar performance as OFUL. Additionally, it would allow to extend the results of Ch. 4 to the n-dimensional setting, since as discussed in Sec. 4.6, the need for optimism 2. Future Work 133 raises technical difficulties that would be overcome using a generic approach that relies on the shape of the objective function over the whole ellipsoid (and not only on the optimistic subset). Finally, we highlighted in Ch. 5 that a greedy strategy may guarantee consistent estimation, and thus achieve a O(log T ) regret, depending on the structure of the problem. A formal characterization of this matter would be of major interest, from both theoretical and practical perspective. In particular, we have collected preliminary theoretical and empirical evidence that the greedy strategy achieves a O(log T ) regret in the Linear Quadratic Gaussian (LQG) control problem. The LQG control problem is a natural extension of the LQ problem of Ch. 4, where the agent no longer observes the current state of the system x t , but a noisy linear transformation y t of it as y t = C * x t + y t , where { y t } t is a zero-mean noise process, and C * parametrizes the unknown observation model. As a result, it can be seen as a linearly parametrized Partially Observable Markov Decision Process (POMDP). From a practical perspective, it is very useful to model systems that do suffer from partial observability or that follow a more complicated linear dynamics than the one of LQ, e.g., autoregressive-moving-average with exogenous inputs (ARMAX) model versus autoregressive with exogenous inputs (ARX) model. From a theoretical perspective, the derivation of the optimal policy have been intensively studied [START_REF] Bertsekas | Dynamic programming and optimal control[END_REF] and relies on the separation principle that states that the optimal policy is linear in the Kalman filter estimate of the internal state, where the linear map is given by the LQ control matrix. One of the main challenge towards the analysis of explorationexploitation algorithms in LQG is the lack of theoretical guarantee for the estimation of the unknown parameters of the dynamics. Several methods have been proposed, such as the Subspace Method (SM) and the Prediction Error Method (PEM). The former is more popular in system identification, and gave rise to the celebrated N4SID algorithm while the latter is more popular in statistics, as it is based on Maximum Likelihood (ML). Despite this limitation, it is possible to sketch a regret analysis, assuming that confidence bounds do hold, in the same flavor as in Ch. 4. In particular, the regret decomposition is very similar to Eq. 4.6, and, up to technical difficulties, a regret analysis may be derived following the same proof structure. On the other hand, numerical experiments suggest that the greedy strategy, which consists in following the optimal policy w.r.t. the current estimate, achieves a O(log T ) regret, and thus, is optimal. Using ML to estimate the parameters in the 1-dimensional case (i.e., the state is 1-dimensional) and running the greedy strategy starting from different initial parameters, one retrieve similar results as in Fig. 5.3 which suggests that the greedy strategy induces consistent estimates, as in the portfolio example of Ch. 5. From a theoretical perspective, inspired from the discussions in Sec. 5.5, one should show that the self-coherent parameters set is reduced to a singleton, that is the true parameter. Since ML aims at minimizing the prediction error, the sequence of estimates are designed so that the predicted observation be equal to the true observation in average. However, this equality is more difficult to characterize compared to LQ as it involves the whole dynamics of the predictions, and not only the stationary distribution. While using the z-transform theory in the 1-dimensional case and equalizing the transfert functions

Figure 2 . 2 -

 22 Figure 2.2 -Illustration of the functioning of the optimistic strategy. Left: The agent associates an optimistic score to each arm by adjusting the empirical means µ t(1,2) with a bonus B (1,2),t which is designed so that the score corresponds to the upper bound of the confidence interval (black segment). In this configuration, arm 2 is selected. Center: A reward is generated for arm 2, and its empirical mean and upper bound are re-evaluated while the one of arm 1 are left unchanged. As a result, the score of arm 1 is significantly boosted by the bonus B 1,t+1 compared to arm 2, and despite the fact that µ t+1

  Figure 2.5 -Thompson Sampling for general MAB problem

Figure 2 . 6 -

 26 Figure 2.6 -OFUL algorithm.

Figure 2

 2 Figure 2.7 -TS algorithm.

Figure 2 . 8 -

 28 Figure 2.8 -Agent/environment interactions and dynamics in MDP.

Figure 2 . 9 -

 29 Figure 2.9 -UCRL2 algorithm.

Figure 2

 2 Figure 2.10 -OFU-LQ algorithm.

Figure 3 . 1 -

 31 Figure 3.1 -Thompson sampling algorithm for LB.

Figure 3 . 7 -

 37 Figure 3.7 -Illustration of the non-optimistic region that could contribute to reduce the regret.

  Proposition 3.B.1. Let f : R d → R be a convex function and C be a closed convex set of R d . Then, on C, f reaches its maximum on the boundary of C.3.B. Properties of convex function 55Proof. Let's denote as int(C) and bound(C) the interior and the boundary of the closed convex set C respectively. Assume that ∃x ∈ int(C) such that f (x ) > f (x) for any x ∈ bound(C) and f (x ) ≥ f (y) for any y ∈ int(C).

  The continuity is a consequence of the convexity of f C on the open convex set R d and the second order differentiability comes from Alexandrov's theorem 3.B.1.

Chapter 3 .

 3 Thompson Sampling in Linear Bandit Proof. The incomplete regularized beta function can be expressed in terms of the beta function B(a, b) and the incomplete beta function B x (a, b) where

Figure 4 . 1 -

 41 Figure 4.1 -Thompson sampling algorithm for LQ.

  Figure 4.B.1 -Optimism and worst case configuration. 1) In 1-D, the Riccati solution is well-defined expect for {(A, B) ∈] -∞, -1] ∪ [1, ∞[×{0}}.The rejection sampling procedure into S ensures P ( θ t ) to be well-defined. Moreover, S c does not overlap with Θ opt . 2) The introduction of the subset Θ lin,opt prevents using the actual -yet complicated -optimistic set Θ opt to lower bound the probability of being optimistic. 3) Even if the event E RLS holds, there exists an ellipsoid configuration which does not contain any optimistic point. This justifies the over-sampling to guarantee a fixed probability of being optimistic.

Theorem 4 .

 4 C.2. Let L > 0 and ρ any non negative and log-concave function on [0, L]. Then for any

  y)|p(x, y)dxdy -

  0)|p(x, 0)ρ(x)dx ≤ (d -1)M |Ω i |δ (4.22) Ω i | ∂u ∂x (x, y)|p(x, y)dxdy -0)|p(x, 0)ρ(x)dx ≤ (d -1)M |Ω i |δ (4.23) Ω i u(x, y)p(x, y)dxdy -d i 0 u(x, 0)p(x, 0)ρ(x)dx ≤ (d -1)M |Ω i |δ (4

  Prop. 4.5.4 We rely on the following properties of Gaussian/truncated Gaussian random variable. Property 4.D.1. Let ∼ N (0, 1), for any a ≥ 2 log(6), V( | | | ≤ a) ≥ 1/2.Chapter 4. Thompson Sampling in Linear Quadratic System

  nL t,t+1 | F t ) -E(P nL t,t+1 | F t )

Figure 5 Figure 5 . 2 -

 552 Figure5.1 -Impulse Response of the decision price. Left: An impulse of the predictor noise pushes the price up and the growth is exponential thanks to the auto-regressive dynamic of α t . Right: The impulse of the trade summarizes the impact model. The price is pushed up by the instantaneous impact γq t , then decreases exponentially, thank to the mean-reverting dynamic of I t .

Figure 5 . 1 -

 51 Figure 5.1 -Adaptive algorithm for Portfolio allocation.

Figure 5 . 4 -

 54 Figure 5.4 -TS SELECT sub-routine.

Figure 5 . 1 -

 51 Figure 5.1 -RLS estimates of the CE algorithm for 100 trajectories. Left: Empirical mean squared error of the RLS estimate θ t -θ ∞ 2 on average over the trajectories and rate of convergence in loglog plot. Right: Empirical distribution of estimates once the convergence is obtained. The line corresponds to the support of the distribution, which contains the true parameter θ .

Figure 5 . 2 -

 52 Figure 5.2 -Average and high probability regret bounds of the CE, TS and OFU-LQ algorithms.

Figure 5 . 4 -

 54 Figure 5.4 -Average and high probability bounds for the regret of the CE algorithm.

Figure 5 . 1 -

 51 Figure 5.1 -Shape of the set of self-coherent parameters SC, i.e., parameters for which the average predicted observation is equal to the observed one. Left: Portfolio allocation without risk constraint. Center: Linear Bandit problem with arm set X = {x ∈ R 2 s.t. x ≤ 1}. Right: LQ problem.

K

  α (θ) (I + K Q (θ))K α (θ) + K α (θ)Φ α + K I (θ)K α (θ) 0 (I -Φ I )K α (θ)By Asm. 5.2.2, Φ I is stable so I -Φ I is invertible and by Asm. 5.2.3, K α (θ) is full-rank which implies that O is full column-rank. Therefore, by Prop. 5.B.1, Σ ∞ is positive definite thus, SC is such that SC = θ s.t Tr (θ -θ) T Let A and C be two matrices of size n × n and d × n. The pair (C, A) is said to be observable if the observability matrix O = Th. 5.3.5 in (Lancaster and Rodman, 1995)). Let A and C be two matrices of size n × n and d × n. Let S be the solution of the Lyapunov equation S = A T SA + C T C.

  Illustration of the functioning of the greedy strategy. Left: at time step t, the empirical means µ t 1 and µ t 2 are computed and deviate from the true means µ 1 and µ 2

	1.0				1.0					1.0
		Play arm 2				Play arm 2					Play arm 2
	0.8				0.8					0.8
		µ 1				µ 1					µ 1
	0.4 0.6 Reward	μt 1		μt 2 µ 2	0.4 0.6	μt+1 1		μt+1 2	µ 2	0.4 0.6	μt+2 1		μt+2 2 µ 2
	0.2				0.2					0.2
	0.0	Arm 1	t	Arm 2	0.0	Arm 1	t + 1	Arm 2	0.0	Arm 1	t + 2	Arm 2
	Figure 2.1 -

  Thompson Sampling for Bernoulli Bandits.

Chapter 2. Exploration-Exploitation Dilemma in Sequential Decision-Making Definition 2.3.1. Let

  propose to consider the diameter D of the MDP which is defined as the time it takes to move from any state s to any other state s under an appropriate policy. T (s |M, π, s) be the random variable for the first time step in which s is reached from state s, for MDP M and stationary policy π.

			Then, the
	diameter of M is defined as:
	D(M ) := max s =s ∈S	min π	E T (s |M, π, s) .

Chapter 2. Exploration-Exploitation Dilemma in Sequential Decision-Making the

  system is optimally controlled w.r.t. θ t for the episode as u t = K( θ t )x t . Finally, they use a doubling schedule to determine the length of the episodes. In this specific LQ setting,Abbasi-Yadkori and Szepesvári (2011) propose to update the policy when the determinant of the design matrix det(V t ) doubles. We summarize the OFU-LQ algorithm in Fig.2.10.

	Initialization: Set θ 0 = O and V 0 = λI
	1: for t = {1, . . . , T } do
	2:	if det(V t ) > 2 det(V 0 ) then
	3:	Calculate θ t by Eq. 2.7
	4: 5:	Find θ t = arg min θ∈Θ∩E RLS t Let V 0 = V t	TrP (θ)
	6:	else
	7:	θ t = θ t-1
	8:	end if
	9:	Choose optimal control u t based on the current parameters, u t = K( θ t )x t
	10:	Execute control, observe next state x t+1
	11:	Save (x t+1 , z t ) into the dataset, where z T t = (x T t , u T t )
	12:

T t = ( A t , B t ). Then,

Let consider 52 Chapter 3. Thompson Sampling in Linear Bandit

	8000 10000 12000	Regret comparison w.r.t t av. Regret BayesTS h.p Regret BayesTS h.p Regret FreqTS av. Regret FreqTS	30000 40000 50000	Terminal av. Regret w.r.t d FreqTS BayesTS
	6000
	4000		20000
	2000		10000
	0 0	50000 100000 150000 200000 t	0 0	5	d 10 15 20 25 30
	Figure 3.6 -

Regret comparison between BayesT S and F reqT S algorithm. Left: empirical average and high probability regret w.r.t t for 100 trajectories with d = 10. Right: empirical average terminal regret R(T ) for T = 200000 w.r.t d for 100 trajectories.

Thompson Sampling in Linear Quadratic System Under

  The noise { t } t is a F t -martingale difference sequence, where F t is the filtration which represents the information knowledge up to time t. Furthermore, the noise is conditionallyGaussian, i.e., Asm. 4.2.1 and 4.2.2, Thm. 2.3.2 guarantees the existence and uniqueness of an optimal policy, such that π * (θ * ) = K(θ * )x, where,

	Assumption 4.2.1 (Noise).

t |F t ∼ N (0, I) for all t ≤ T . Assumption 4.2.2 (LQ). The cost matrices Q and R are symmetric p.d. and (A * , B * ) is stabilizable. 3 62 Chapter 4.

Proposition 4.5.5. Let

  { θ t } t≥1 be the sequence of least square estimate, ∀t ≥ 1,

	Thanks to Cor. 4.5.2, one directly obtains a bound for R gap,1 t	. The objective of next
	step is to show that R gap,2 t	is closed to R gap,1 t	because subsequent sampling distribution
	changes slowly.
	Bounding R gap,2 t	. This step relies on the following propositions.

Thompson Sampling in Linear Quadratic System Proposition 4.5.6.

  Let {β t } t≥1 = {β t (δ )} t≥1 be the sequence defined in Eq. 4.4, then, ∀t ≥ 1,

Thompson Sampling in Linear Quadratic System Lemma 4.5.7.

  Let If the TS is run over T time step according to algorithm 4.1, then, where

  Proof. Again, since P(| | ≤ a) ≤ e -a 2 /2 , one has, for any a ≥ 2 log(n), P

	2 ≥	2 log(6) 5	≥ 1/2
	Property 4.D.2. Let ∼ N (0, 1), for any n ≥ 2, for any a ≥ 2 log(n), P(| | ≤
	a) n ≥ 1/4.

  Noticing thatαt-ρ xt

		√	n	= 2 log(3n), Properties. 4.D.1-4.D.2 holds
	and E(y t y T t 1 {|y i t -ȳ i t |≤	α-ρ xt √ n

  The predictable part of the returns model is observed at each time step i.e., α t is F t -measurable and parameters contained in Φ α are known. Furthermore, we assume that this representation is minimal i.e., the eigenvalues of Φ α are pairwise disjoint and that β α is of full row-rank. The parameters contained in Φ I are known. As a consequence, the impact variable I t can be reconstructed from past trades (which are F t -measurable) and thus, I t is F t -measurable.

	Assumption 5.2.3. Assumption 5.2.4.

Résumé court en français : Cette thèse est dédiée à l'étude du Thompson Sampling (TS), une heuristique qui vise à surmonter le dilemme entre exploration et exploitation qui est inhérent à tout processus décisionnel face à l'incertain. Contrairement aux algorithmes issus de l'heuristique optimiste face à l'incertain (OFU), où l'exploration provient du choix du modèle le plus favorable possible au vu de la connaissance accumulée, les algorithmes TS introduisent de l'aléa dans le processus décisionnel en sélectionnant aléatoirement un modèle plausible, ce qui les rend bien moins coûteux numériquement. Cette étude se concentre sur les problèmes paramétriques linéaires, qui autorisent les espaces état-action continus (infinis), en particulier les problèmes de Bandits Linéaires (LB) et les problèmes de contrôle Linéaire et Quadratique (LQ). Nous proposons dans cette thèse de nouvelles analyses du regret des algorithmes TS pour chacun de ces deux problèmes. Bien que notre démonstration pour les LB garantisse une borne supérieure identique aux résultats préexistants, la structure de la preuve offre une nouvelle vision du fonctionnement de l'algorithme TS, et nous permet d'étendre cette analyse aux problèmes LQ. Nous démontrons la première borne supérieure pour le regret de l'algorithme TS dans les problèmes LQ, qui garantie dans le cadre fréquentiste un regret au plus d'ordre O( √ T ). Enfin, nous proposons une application des méthodes d'exploration-exploitation pour les problèmes d'optimisation de portefeuille, et discutons dans ce cadre le besoin ou non d'explorer activement.

This chapter is an extended version of our AI&Stats paper(Abeille and Lazaric,

2017a), which has been accepted for publication in the Electronic Journal of Statistics.

Here we refer to the definition introduced in the arXiv paper, which slightly differs from the original ICML paper.

This chapter is an extended version of our AI&Stats paper(Abeille and Lazaric,

2017b).

This setting is more challenging than its Bayesian counterpart in general as it encompasses it.

(A, B) is stabilizable if there exists a gain matrix K s.t. A + BK is stable, i.e., all eigenvalues are in (-1, 1). A formal characterization is given in Def. 2.3.2.

Notice that the consistency regret is not specific to LQ systems but it is common to all regret analyses in RL (see e.g., UCRL2[START_REF] Jaksch | Near-optimal regret bounds for reinforcement learning[END_REF]) except for episodic MDPs and it is always bounded by keeping under control the number of switches of the policy (i.e., number of episodes).

The mapping θ → Tr(P (θ)) is continuous (see Lem. 4.2.3). Thus, S is compact as the intersection between a closed and a compact set.

The continuity of the mapping θ → K(θ) together with the compactness of S justifies the finite positive constants ρ and C. Moreover, since every θ ∈ S are stabilizable pairs, ρ < 1.

This chapter is based on our paper(Abeille et al.,

2016) for generic portfolio construction using LQG systems that is submitted to International Journal of Theoretical and Applied Finance (under review).

This formulation slightly differs as it involves a cross-term x T t N q t , although it remains quadratic. Thanks to a change of variable, it is possible to retrieve the original formulation of Subsec. 2.3.2 (see[START_REF] Lancaster | Algebraic riccati equations[END_REF]).

Using state-space manipulations, it is possible to re-write the problem in the same form as in Ch.

However, this requires to augment the dimension of the system, thus for sake of clarity, we prefer to stick with this minimal model.

Proposition 4.2.2. S is a compact set. For any θ ∈ S, θ is a stabilizable pair (since J(θ) = +∞ otherwise) and there exist ρ < 1 and C < ∞ positive constants such that ρ = sup θ∈S A + BK(A, B) 2 and C = sup θ∈S K(θ) 2 .

1. When θ = (A, B) is not stabilizable, there exists no linear control K such that the controlled process x t+1 = Ax t + BKx t + t+1 is stationary. Thus, the positiveness of Q and R implies J(θ) = Tr(P (θ)) = +∞. As a consequence, θ / ∈ S.

Chapter 5

Application to Portfolio Construction

In this chapter 1 , we apply exploration-exploitation techniques to the problem of portfolio construction. While this is a standard in finance since the seminal work of Markowitz, several approaches have been recently proposed to integrate the price impact effects i.e., the fact that buying and selling shares modifies the supply and demand and hence the price at which transactions are made. This makes the control problem significantly more difficult since one has to anticipate the future costs implied by price impact. Moreover, by nature, the dynamics of financial markets are unknown and have to be estimated. However, in order to observe the price impact effects, asset managers have to trade directly on the market and this may induce the exploration-exploitation trade-off problem to balance between trading to make profit and trading to gain knowledge about the market dynamics. We introduce a novel LQ formulation for the portfolio allocation problem, under the assumption of linear price dynamics, from which we obtain the optimal control and discuss the exploration-exploitation trade-off arising from the presence of unknown parameters. We consider two problem instances with or without risk constraint and show that this affects the need for exploration. In the unconstrained case, a greedy strategy fails to achieve sub-linear regret, while Thompson Sampling or optimism-based algorithms effectively trade-off exploration and exploitation. On the other hand, the risk constraint modifies the structure of the policy, removing somehow the need for active exploration, and a greedy strategy is optimal. We discuss this counter-intuitive result and support it with numerical experiments.

Appendix 5.A Proof of Lem. 5.2.1.

We derive here the proof of theorem 5.2.1 which maps the existence and uniqueness guarantee of the LQR solution to a non-arbitrage criterion. The proof is structured as follow: first, we present the Popov criterion (see e.g., [START_REF] Molinari | The stabilizing solution of the discrete algebraic riccati equation[END_REF]) which guarantee the existence and uniqueness of a solution to the Riccati equation.

Secondly, we show that the deterministic and stochastic LQR share the same Riccati equation and hence, share the same existence and uniqueness condition. Then, we translate the Popov frequency domain criterion in terms of the cost function of the deterministic LQR and thus, in terms of non-arbitrage for admissible trade sequence. Finally, we show that the set of admissible trade sequence is the set of round-trip sequence.

5.A.1 The Popov criterion

Since the cost matrix Q N N T R associated with the LQR problem for portfolio construction presented in Section 5.2 is not positive definite by construction, the usual guarantee for the Riccati equation solution is violated. However, the existence of a unique admissible solution of 5.5 can still be provided using the Popov criterion.

Introducing the hermitian matrix:

from [START_REF] Molinari | The stabilizing solution of the discrete algebraic riccati equation[END_REF], we have the following theorem:

Theorem 5.A.1. Assume that the pair (A, B) is stabilizable then there exists a (necessarily) unique symmetric stabilizing solution P satisfying the associated Riccati equation if and only if for some (and hence all) K such that A + BK is asymptotically stable,

This frequency-domain criterion guarantees the global convexity of the problem based on a fairly complete existence theory (see e.g., [START_REF] Molinari | The stabilizing solution of the discrete algebraic riccati equation[END_REF][START_REF] Ionescu | General matrix pencil techniques for the solution of algebraic riccati equations: a unified approach[END_REF][START_REF] Van Dooren | A generalized eigenvalue approach for solving riccati equations[END_REF][START_REF] Wimmer | The algebraic riccati equation: conditions for the existence and uniqueness of solutions[END_REF]. The main drawback however is the use of the frequency-domain method involved. To get a better intuition about the existence of optimal solution, we link here the Popov criterion to a non-dynamical arbitrage criterion in line with [START_REF]No-dynamic-arbitrage and market impact[END_REF].

5.A.2 Deterministic LQR

First, let's notice that the LQR solution of the stochastic problem 5.5 involves the same Riccati equation (see e.g., Bertsekas 1995) -and hence shares the same conditions -as the deterministic LQR problem (5.8): minimize {qt} t=1,...,∞ ∈Q J(q 0 , q 1 , . . . (5.8) where Q := {q = (q 0 , q 1 , . . . ) ∈ l 1 ∩ l 2 such that x = (x 0 , x 1 , . . . ) ∈ l 1 ∩ l 2 } is the admissible control space which are the stabilizing sequences. Indeed, the stochastic LQR problem cost function is defined with an expectation regarding the noise process x t which is of zero conditional mean. Thanks to the linear structure of the dynamics, the noises vanish within the Bellman equation which is then the same as the one of the deterministic LQR. As a result, we can apply the Popov criterion on the deterministic problem to ensure the existence and uniqueness of a solution to the stochastic one.

5.A.3 From frequency to time domain

We now state the first corollary which derives directly from Thm. 5.A.1: Corollary 5.A.1. Assume that the pair (A, B) is stabilizable then there exists a (necessarily) unique symmetric stabilizing solution P if and only if J(q 0 , q 1 , . . . ) > 0 for any q ∈ Q.

The proof is straightforward using the z-transform theory. Let q ∈ Q be any admissible control sequence and denote as q(z) and x(z) the z-transform of the control sequence and associated state sequence respectively. Then, applying the z-transform theory to (5.8) and using Parceval's theorem leads to:

(5.9)

The following lemma provides another description for the admissible sequence q:

Lemma 5.A.1. q ∈ Q if and only if there exists a stable control K i.e., such that A + BK is stable, and a sequence v ∈ l 1 ∩ l 2 such that

Proof. Let K be a stable control and define v t = q t -Kx t for all t ≥ 1. By definition of Q, q and x belong to l 1 ∩ l 2 and so does v.

On the other hand, let v ∈ l 1 ∩ l 2 , K be a stable control and define q t = Kx t + v t for all t ≥ 1. Then, x t+1 = (A + BK)x t + v t and since A + BK is stable, v ∈ l 1 ∩ l 2 implies that x ∈ l 1 ∩ l 2 and so does q.

addresses this issue and proves that the self-coherent parameters set is reduced to a singleton, how to properly define and characterize this set in the general n-dimensional case, and proving that the N4SID or ML estimates of the greedy strategy converge to this set, are open questions that lay ground for future work and might provide good intuitions about exploration-exploitation issue in similar settings, such as POMDPs.

Appendix

A Concentration inequalities

Lemma A.1 (Hoeffding's inequality). Let X be a bounded r.v. in [a, b], of zero mean. Then, for any t ∈ R, E e tX ≤ e t 2 (b-a) 2 /8 . Lemma A.2 (Chernoff bound for Gaussian r.v.). Let X ∼ N (0, 1). For any t ≥ 0, then,

Then, for all t > 0 and all > 0,

B Convergence of random variables