Florian

Daniele Alexandre

Merwan, Pratik, Frederic, Tomas, Hadrien, Marta Marc

Lilian

Nicolas

Jean-Bastien Ronan

et Romain pour les tous les bons moments passés ensemble. Cette thèse m'a permis d'effectuer un stage à DeepMind où j'ai eu l'opportunité de travailler dans l'équipe de Thore Graepel que je tiens à remercier ainsi que les chercheurs et ingénieurs avec qui j'ai pu travailler Joel, Karl, Marc et Vinicius. Je terminerai en remerciant ma famille pour

General Notations

• the set ∆A is the set of the probability distributions on A,

• N the set of integer {0, 1, 2, . . .},

• R the set of real numbers,

• |S| the cardinal of set S,

• 1 s is the function defined on S, such that 1 s(s) = 1 if s = s and 0 otherwise,

• the L +∞ -norm is defined as f +∞ = sup x∈X f (x),
• I is the identity,

• the L p -norm of the function f (written

f p ) is f p p = x∈X f (x) p
• The L ρ,p of function f is f ρ,p is defined here as f p ρ,p = x∈X ρ(x)f (x) p

• a ∼ π means that a is a random variable drawn according to the law π

Statistic:

• Unbiased estimator: an estimator θn of the quantity θ is said to be unbiased if E θn -θ = 0.

• Consistent estimator: an estimator is said to be consistent if lim n→+∞ θn -θ in probability.

Reinforcement Learning and Multi-Agent Reinforcement

Learning Notations

v π = (I -γP π ) -1 r π v µ,ν = (I -γP µ,ν ) -1 r µ,ν v i π = (I -γP π ) -1 r i π Q-function Q π = (I -γP π ) -1 r Q µ,ν = (I -γP µ,ν ) -1 r Q i π = (I -γP π ) -1 r i

Introduction

In many areas of research and industry, interaction data are collected. For example, many databases record professional games of chess, checkers or go. In many areas of industry, records of interactions between humans or between human and machines are logged. These data demonstrate an empirical knowledge of the rules of an interaction that can't necessarily be explained. For example, it can be hard to program a machine to interact with a human. We intend to study whether or not it is possible to learn an interaction strategy from these data. We will take a machine learning perspective on this problem. We will design and evaluate algorithms to learn interaction strategies from data without prior knowledge. This document is devoted to the study of reinforcement learning methods in multi-agent environments. In reinforcement learning, machines are not explicitly programmed to perform a task but have to learn through a numerical reward to perform well. From a single agent environment to multi-agent environments, a large amount of problems can be studied and many agendas can be followed. The range of problems arising in multi-agent systems goes from the study of communication to the emergence of coordination and includes game theoretic equilibrium computation. There is a trove of scientifically relevant problems that are worth studying in multi-agent systems. First, we shall specify the agenda this work contributes to. In this thesis, we take a game theoretic approach to multi-agent learning. In classical game theory, each game player is supposed to know the rules of the game and is aware of the strategies each player can play and of their pay-off. This thesis shows that a machine can learn equilibria in games if the knowledge of the game is limited to samples.

In the single agent case, it is fairly straightforward to define what should be learnt. A single agent should learn how to behave in order to collect maximum rewards. In the multi-agent case, the optimal behaviour of a player must take into account the other players. If one agent changes its strategy, the others must adapt to this new strategic interaction. Co-adaptation is the reason why multi-agent systems are fundamentally different from single-agent systems. Despite the fact that players can adapt to changes of their opponents, one can still define stable strategic interaction called equilibrium in game theory. An equilibrium is a solution concept where players do not have an incentive to vary their current strategy even if they could. Equilibria have played a central role in the development of game theory and this thesis will study two forms of equilibria i.e., the minimax solution and the Nash equilibrium. In zero-sum games, the canonical solution concept is the minimax strategy which guarantees a minimal pay-off to a player whatever the opponent does. It also stands that if both players play their counterpart of the equilibrium, none of them will have an incentive to shift from Chapter 1. Introduction their current strategy. However, interactions are not necessarily purely adversarial in a complex multi-agent systems, players might pursue similar objectives in some situations and might have opposed ones in others. When no reward structure is assumed, the game is called a general-sum game as opposed to zero-sum two-player games. In general-sum games, when players choose their strategy secretly and independently the solution concept that is considered is the Nash equilibrium. Again, in a Nash equilibrium, no player has an incentive to switch from their current strategy but the goal is not to exploit the strategy of the others. Our aim is to find a strategy for each player such that none of them have an incentive to switch. However, in general sum games, playing a Nash equilibrium does not guarantee a minimal pay-off but it makes cooperation possible where the minimax solution explicitly prevents it by considering a worst case scenario.

Markov Games

Markov Decision Processes

Multistage game Normal form Games Figure 1.1 -Representation of the class of model we consider However, one major drawback of classical game theory is that one needs to have a perfect knowledge of the interaction between all players. In a general-sum game, all players must know their own reward and the reward of the opponents to find a Nash equilibrium. But, in many situations this knowledge is not available and can only be observed through past interactions or while playing the game. The first case is named the batch scenario and is the focus of the main part of this dissertation. The second case is the online scenario and is explored later. In the batch scenario, the only information on the game available to learn a strategy for all players are historical interaction data between players. For example, the usual way to teach chess is to first explain the rules of the game and then to give a few basic strategies. However, we won't teach the machine that way since this method is game specific and since it uses game specific knowledge. As a comparison, instead of giving the machine the rules of the game, we give it details of several games and their outcome. The class of algorithms studied here leverage this historical information to learn an optimal strategy without being told the rules of the game. The second case studied in this thesis is independent reinforcement learning. In this setting agents no longer have access to historical data and must learn their strategy online while interacting with other learning agents. Agents are blind to each other's actions and only receive their own reward signal.

In the end, since the knowledge of the system in which players interact is limited to 1. Structure of the dissertation 7 samples of interaction, finding an exact Nash equilibrium or an exact minimax solution of the game will not be possible. We will learn these equilibria from interaction samples. To understand to what extent this can be done, we will need to analyse the impact this approximate knowledge of the game has on the solution.

Structure of the dissertation

The next chapter (Chapter 2) introduces the necessary background to game theory, reinforcement learning, and machine learning used in the dissertation. We provide an historical overview of the existing Dynamic Programming (DP) techniques to solve Markov Decision Processes (MDPs) and zero-sum two-player MGs. We also exhibit links between Bellman residual minimization and DP. Finally, we provide an overview of independent RL in games. The rest of the document is organized in three parts and six chapters.

The second part (Part II) of this manuscript is dedicated to the study of Approximate Dynamic Programming (ADP) techniques to learn from batch data. In Chapter 3 we introduce a novel family of batch algorithms based on a generic ADP scheme Approximate Generalized Policy Iteration (AGPI) that includes Approximate Value Iteration (AVI) and Approximate Policy Iteration (API). These algorithms are approximate since our knowledge of the game is limited to samples. These errors accumulate from one iteration to an other and have an impact on the accuracy of the final solution of the algorithm. We provide a unified sensitivity analysis of these algorithms to errors and provide an empirical evaluation of the algorithm on the game of Alesia. Chapter 4 explores the use of non-stationary (cyclic) strategies to reduce the sensitivity of the error on the final solution. We show that there is an explicit trade-off between the size of the cycle and the quality of the final solution. Finally, in Chapter 5, we study to which extend using non stationary strategies can be used to solve general-sum MGs. In this case the size of the cyclic strategy is the only factor that improves the solution. Whilst in MDPs and in zero-sum two player MGs the number of iterations was improving the solution, this is not the case for general-sum MGs.

The third part of this dissertation (Part III) studies a Bellman residual approach to learn from batch data. In zero-sum two-player MGs (Chapter 6) we draw connections between an existing and unstable algorithm LSPI and Newton's method on the Bellman residual. We leverage this connection to improve these algorithms and provide two stable and efficient algorithms. Then, in Chapter 7, we propose a Bellman residual approach to learn stationary Nash equilibria from batch data in general-sum MGs (whilst Chapter 5 only provides a method to learn a non-stationary strategy).

Although this thesis mainly treat the batch scenario, we also are interested in how agents can reach an equilibrium while acting independently and online. Finding independent learning algorithms for cooperative and competitive scenarios is usually treated as a separate agenda in multi-agent RL and we propose in Part IV a unified learning rule that applies in both cooperative multi-stage games and in competitive multi-stage games.

Contributions

The mathematical model we consider to study multi-agent interaction is a Markov game (or stochastic game). It generalizes MDPs to multi-agent systems. A Markov Game (MG) is a temporally and spatially extended model of interaction between N players. It is composed of a state space S which contains all the states s in which agents can be. In every state s, players simultaneously take an action a i in a set A i (s). As a result of this joint action (a 1 , . . . , a N ), each player receives a reward r i (s, a 1 , . . . , a N ) and the system moves to the following state s with a probability defined by the transition kernel p(s |s, a 1 , . . . , a N ). Usually in these settings, each player's goal is to find a strategy π i (.|s) to maximize a long term objective. The long term objective studied in this thesis is the expected sum of γ discounted rewards (γ ∈ [0, 1[). Reinforcement learning is usually studied with a single agent using MDPs as a model while the canonical model used in standard game theory focuses on normal form games which is an MG with a single state. An-other common model studied in game theory are tree structured games named here multi-stage games. Multi-stage games are MGs with a probability transition function that forbids to visiting the same state twice. As an example, an MDP can model single agent problems such as Atari games, normal form games can model games such as rock-paper-scissors or the prisoner dilemma, and a multi-stage game can model the game of Alesia (also known as Oshi-zumo).

The rest of this section sums up the contributions of this thesis. Most of this work is published or under review.

Approximate Dynamic Programming for Zero-Sum Two-Player Markov Games: Chapter 3 provides an analysis of error propagation in Approximate Dynamic Programming applied to zero-sum two-player Stochastic Games. We provide a novel and unified error propagation analysis in L p -norm of three well-known algorithms adapted to Stochastic Games (namely Approximate Value Iteration, Approximate Policy Iteration and Approximate Generalized Policy Iteration). We analyse it's sensitivity to value function approximation error and greedy error on the final solution. In addition, we provide a practical algorithm (AGPI-Q) to solve infinite horizon γ-discounted two-player zero-sum Stochastic Games in a batch setting. This is an extension of the Fitted-Q algorithm (which solves Markov Decisions Processes from data) and can be non-parametric. Finally, we demonstrate experimentally the performance of AGPI-Q on a simultaneous two-player game, namely Alesia.

Background and Related Work

Markov Games (MGs) introduced by Shapley [START_REF] Shapley | Stochastic Games[END_REF] are a generalization of Markov Decision Processes (MDPs) introduced by Bellman [START_REF] Bellman | Dynamic Programming[END_REF]. As many of the algorithms employed to solve MGs are either a generalization of algorithms for MDPs or rely on algorithms for MDPs, the first section (Section 1) of this chapter will attempt to sum up existing work in the reinforcement learning community. This part will mostly focus on batch and on online learning algorithms. Then, in Section 2, normal-form games are described. This section introduces basic techniques to solve zero-sum two-player normal-form games which will be used as a subroutine in some algorithms for zero-sum two-player MGs. In Section 3, we introduce general-sum MGs. This section will emphasize the special case of zero-sum two-player MGs and introduce specific notation as it is the focus of Chapters 3, 4 and 6.

Markov Decision Processes

A Markov Decision Process (MDP) is the canonical model of sequential decision making under uncertainty as it is a temporally and spatially extended single-agent process. The situation is the following, an agent lies in a state space S (here we only consider discrete state spaces). At time t, the agent is in state s = s t and has to choose an action a = a t in the finite set A of possible actions in state s. As a result, the agent receives a reward r t = r(s t , a t ) and the state moves to state s t+1 with the probability p(.|s t , a t ). This model is a reductionist interaction model in multiple ways: first it assumes that the environment is stationary (i.e. the reward function r(s, a) and the transition kernel p (.|s, a) do not depend on the time t), second it assumes that no information is hidden from the agent (i.e. the agent completely knows the state of the environment). Finally, γ ∈ [0, 1[ is the discount factor of the MDP, it controls the horizon of the agent 1 1-γ (i.e. which is the horizon at which the agent will attempt to accumulate the maximum sum of rewards). In the end, an MDP M is fully represented by a tuple < S, A, r, p, γ >.

In such an environment, the goal of each agent is to find a policy. This is a decision rule an agent will apply in an MDP. This rule may depend on time, on the history of states, on the history of actions and on the history of rewards. It might be deterministic or stochastic. In an MDP, since the dynamics and the reward are Markovian (i.e. they only depend on the current state and action), the policy does not need to be history dependent [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF]. In technical terms, a policy {π t (.|s)} t∈N (here a nonstationary policy) is (for a fixed t) a mapping from a state to a distribution on the action space S → ∆A (where ∆A is the set of distributions over A and N is the set of positive integers {0, 1, 2, . . .}). The situation is the following: the process starts at state s 0 and the agent chooses an action a 0 with probability π 0 (.|s 0 ). The agent receives a reward r 0 = r(s 0 , a 0 ) and the environment moves to a next state s 1 . This process is repeated sequentially and produces a sequence of states s 0 , s 1 , s 2 , . . . , s t , . . . , a sequence of actions a 0 , a 1 , a 2 , . . . , a t , . . . and a sequence of rewards r 0 , r 1 , r 2 , . . . , r t , . . . . The goal of the agent is to cumulate a maximum amount of rewards over time. The criterion we will study in this dissertation is the expected γ-discounted sum of rewards written as v {πt} t∈N (s) where {π t } t∈N is the policy of the agent (in the following expression a ∼ π means that a is a random variable that behaves according to the law π). v {πt} t∈N (s) = E +∞ t=0 γ t r(s t , a t )|s 0 = s, a t ∼ π t (.|s t ), s t+1 ∼ p(.|s t , a t ) .

(2.1)

The goal of an agent is to find a policy {π t } t∈N that maximizes the value function v {πt} t∈N . A policy that satisfies that condition is said to be optimal. The set of optimal policies is not empty and contains at least one stationary policy (i.e., that does not depend on time) [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF]. Usually, the purpose of an agent is to find a stationary policy π since it is easier to store a time independent policy rather than a non-stationary one. However, it might be easier to find a good policy in the set of non-stationary policies rather than in the set of stationary ones [START_REF] Scherrer | On the Use of Non-Stationary Policies for Stationary Infinite-Horizon Markov Decision Processes[END_REF]. In the case of a stationary policy, the value function is defined as follows:

v π (s) = E +∞ t=0
γ t r(s t , a t )|s 0 = s, a t ∼ π(.|s t ), s t+1 ∼ p(.|s t , a t ) .

(2.2)

The value function achieved by an optimal policy is called the optimal value function and is defined as follows:

v * (s) = max {πt} t∈N v {πt} t∈N (s) = max π v π (s).
(2.3)

In addition to the value function, one usually defines the state-action value function or Q-function written Q π (s, a) for a stationary policy π. The Q-function Q π (s, a) corresponds to the expected γ-discounted return starting from state s if the agent takes a as the first action and then selects his next action according to the policy π. Q π (s, a) = E +∞ t=0 γ t r(s t , a t )|s 0 = s, a 0 = a, a t ∼ π(.|s t ), s t+1 ∼ p(.|s t , a t ) , (2.4) = r(s, a) + γ s ∈S p(s |s, a)v π (s ).

(2.5)

One can also define an optimal state-action value function as follow: (2.7)

Q * (s, a) = max π Q π (s,
Note that several policies might achieve such an optimal value function or Qfunction. We will often write an optimal policy π * .

Markov Decision Processes 13

Stochastic Matrices and Linear Algebra: To ease the notations and to simplify the proofs, we will see the value function as a vector of R |S| in the canonical basis {1 s } s∈S (R is the set of real numbers, |X| is the cardinal of the set X and 1 s is the function such that 1 s (s) = 1 and ∀s = s, 1 s (s) = 0). These notations will be introduced using a linear kernel for the sake of generality. Obviously, the vector v π = s∈S 1 s v π (s). In this vector space, the vector r π = s∈S 1 s E a∼π (.|s) [r(s, a)]. The probability transition kernel P π is the stochastic matrix that defines the following linear transformation:

P π v = s∈S 1 s s ∈S E a∼π(.|s) [p(s |s, a)] v(s ).
(2.8)

Furthermore, we will define r a = v π = (I -γP π ) -1 r π , where I is the identity application.

(2.9)

This property is a direct consequence of eq. 2.13 defined below.

Bellman Operator:

The key tools of dynamic programming are the so called Bellman operators. The first Bellman operator depends on a policy π and can be thought of as the solution to a one-step evaluation problem. Its application on the value v produces the value function [T π v] which is the expected return of the following game: the agent chooses an action a with probability π(.|s) and receives the random pay-off r(s, a) + γv(s ) (where s ∼ p (.|s, a)). It is a linear operator and is defined as follows:

[T π v] (s) = r π (s) + γ s ∈S p π (s |s)v(s ).

(2.10) Since this operator is linear in the value function, it can be written with the tools defined in the previous paragraph:

[T π v] = r π + γP π v.
(2.11)

The second Bellman operator is non-linear. Its value [T * v] for the value function v is the optimal return the agent can get in the game described above. It is often called the optimal Bellman operator and is defined as follows: (2.12)

[T * v] (s) = max
These two operators are γ-contractions with respect to the L +∞ -norm. Here the L +∞ -norm of function f : X → R is defined as f +∞ = sup x∈X f (x). This property of the optimal Bellman operator is leveraged in the value iteration algorithm (see Section 1.1.1). Thus, each of these two operators admits a unique fixed point. The Chapter 2. Background and Related Work fixed point of T π is v π and the fixed point of T * is v * . Meaning we have the two following identities:

T π v π = v π ,
(2.13)

T * v * = v * .
(2.14)

In addition to these two Bellman operators, we will define [T a v] (s) = r(s, a) + γ s∈S p(s |s, a)v(s ). From this definition, we can redefine the Q-function as Q π (s, a) = T a v π (s) and the optimal Q-function as Q * (s, a) = T a v * (s).

Respectively, one can define operators on Q-functions as follow: (2.16)

[B π Q] (
These operators are also γ-contractions and their respective fixed points are Q π (for operator 2.15) and Q * (for 2.16). The linear Bellman operator on a Q-function can be expressed using linear transformations. Let's write the state-action transition kernel: (2.17)

P π Q = s,a∈S×A
Then, the linear Bellman operator of 2.15 can be expressed as follow:

B π Q = r + γP π Q.
(2.18)

Greedy Policy: When given a value function v π , there is a simple way to retrieve a policy performing at least as well as π. The idea is to act greedily with respect to the value function v π . A policy π is said to be greedy with respect to a value function v if T π v = T * v. In this case, we say that π ∈ G(v). A policy π is said to be greedy with respect to a Q-function Q if for all s in S the relation holds E a∼π(.|s) [Q(s, a)] = max a Q(s, a). Again, we will say that π ∈ G(Q) if a policy π is greedy with respect to Q . One can prove the two following results: Theorem 2.1. A policy π is optimal if and only if it is greedy with respect to it's value v π .

v π = v * if and only if π ∈ G(v π ).
(2.19)

This first theorem proves that to find an optimal policy, it is sufficient to find an optimal value. This theorem will be extended for zero-sum two-player Markov-Games but has no extensions to general sum MGs.

Theorem 2.2. If π is greedy with respect to the value v π of policy π, then v π ≥ v π This second theorem proves that a simple way to improve a policy π is to play the greedy policy π with respect to the value v π . This theorem will be the basis of the Policy iteration algorithm (Section 1.1.2). These two theorems are standard in ADP and are proven in [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF].

Exact Algorithms

This section intends to give an overview of the common algorithms used to solve MDPs. The first family of algorithms are dynamic programming algorithms (Section 1.1.1,1.1.2,1.1.3), the second directly minimize the Bellman residual (sec 1.1.4).

Value Iteration

The Value Iteration (VI) algorithm [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF] is based on the γ-contraction property of the optimal Bellman operator. It repetitively applies the optimal Bellman Operator on a value function starting from an arbitrary value v 0 = 0.

Algorithm 1 Value Iteration

Input: An MDP M , a value v 0 = 0 and a maximum number of iterations K. for k=1,2,...,K-1 do for all

s compute v k (s) = max a r(s, a) + γ s ∈S p(s |s, a)v k-1 (s ) (or v k = T * v k-1 )
end for for all s, a compute

Q K (s) = r(s, a) + γ s ∈S p(s |s, a)v K-1 (s ) and find π K ∈ G(Q K ) (or find π K ∈ G(v K ) where v K = T * v K-1 ) Output: π K
The performance of policy π K can be expressed as a distance between v π and the value of an optimal strategy v * . The proof of the following result can be found in [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF]: (2.20) This algorithm performs iterations at a low computational cost but is slow to converge (in the number of iterations) compared to Policy Iteration.

v π K -v * +∞ ≤ 2γ K 1 -γ v * -v 0 +∞ .

Policy Iteration

The second iterative algorithm presented here is the Policy Iteration (PI) algorithm. It starts from an arbitrary value v 0 and proceeds in two steps. At iteration k it finds a greedy policy π k with respect to the value v k-1 and then computes the value of that policy. Finally, it returns π K after K iterations.

This algorithm produces an increasing sequence of values v k . This property is implied by theorem 2.2. Furthermore, this algorithm is guaranteed to converge in a finite number of iterations.

Again, one can prove the following result on the performance of policy π K :

v π K -v * +∞ ≤ 2γ K 1 -γ v * -v 0 +∞
∈ G(v k-1 ) compute v k = v π k = (I -γP π k ) -1 r π k end for Output: π K
The policy iteration is guaranteed to converge in a finite number of steps and its complexity is polynomial in the size of the state space and of the action space for a fixed γ (Hansen et al., 2013a[START_REF] Scherrer | Improved and generalized upper bounds on the complexity of policy iteration[END_REF]. In the end the Policy Iteration algorithm performs updates that are heavier VI's one. This is because the value update is required to solve a linear system. But PI converges faster in terms of the number of iterations compared to VI.

Modified Policy Iteration

Modified Policy Iteration (MPI) is an algorithm that bridges the gap between VI and PI by introducing a parameter m. The parameter m controls the cost of each iteration. This algorithm reduces to VI if m = 1 and to PI if m = +∞. Indeed, in Algorithm 3 if m = +∞, the value v k will be the fixed point of

T π k (i.e. the value v π k ). And if m = 1, since π k ∈ G(v k-1 ) we have T * v k-1 = T π k v k-1 and finally v k = T π k v k-1 = T * v k-1 .
Again, this algorithm enjoys the same guarantees of convergence as PI and VI and surprisingly the performance policy π k does not depends on m. (2.22) This algorithm also enjoys a lower bound of complexity which is detailed in [START_REF] Lesner | Non-stationary approximate modified policy iteration[END_REF] for a generalized algorithm.

v π K -v * +∞ ≤ 2γ K 1 -γ v * -v 0 +∞ .

Algorithm 3 Modified Policy Iteration

Input: An MDP M , a value v 0 = 0, a maximum number of iterations K and a parameter m. for k=1,2,...,K do

find π k ∈ G(v k-1 ) compute v k = (T π k ) m v k-1 end for Output: π K

Minimization of the Bellman Residual

The last method we will present here is the direct minimization of the Bellman residual. Indeed, solving an MDP means finding the fixed point of the operator T * . This means that we have to find a value v such that v = T * v. The idea is thus to minimize the norm of the Bellman residual v -T * v with some optimization method [START_REF] Baird | Residual Algorithms: Reinforcement Learning with Function Approximation[END_REF]. For example, the minimization of the L 2 -norm of the Bellman residual with Newton's method is in fact the policy iteration algorithm [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF]

(the L p -norm of the function f (written f p ) is f p p = x∈X f (x) p ).

Batch Algorithms

The previous section (Section 1.1) detailed several algorithms to exactly solve MDPs. These algorithms are based on the value function and proceed iteratively in two steps. The first step is a greedy step and requires knowing the model of the MDP (since for a given value v we need to find a policy π such that T * v = T π v). This section is devoted to batch algorithms and thus we will need to perform this step without knowledge of the model. That is why all the algorithms presented in this section perform iterations on the Q-function instead of the value function. The second step in these algorithms also requires the model and performs a linear transformation on the current value v. Again, since we don't have access to the model, we will replace that step with a supervised learning step. The underlying idea is that unbiased estimators of the Bellman operator can be built with samples. The algorithms introduced in Section 1.2.1 are an adaptation of the value iteration algorithm for the batch setting.

In Section 1.2.2 we will present an adaptation of policy iteration and in Section 1.2.3 we will show how errors are propagated through iterations of modified policy iteration. Finally, we will show how to adapt the Bellman residual minimization approach to the Batch setting.

Remark 2.1. An estimator θn of the quantity θ is said to be unbiased if E θn -θ = 0 and it is said to be consistent if lim n→+∞ θn -θ = 0 in probability.

Fitted-Q iteration and Neural Fitted-Q

These algorithms are value iteration like algorithms. Instead of performing an exact update

Q k = B * Q k-1 as the VI algorithm on Q-function would, it performs an ap- proximate update Q k B * Q k-1 .
Any batch algorithm requires as inputs a set of data D n = {(s j , a j , r j , s j )} j=1,...,n where for all j, the reward r j = r(s j , a j ) and where s j ∼ p(.|s j , a j ). At iteration k, the VI algorithm on Q-functions would perform the following step:

∀s, a ∈ S × A, Q k (s, a) = r(s, a) + γ s ∈S p(s |s, a) max b Q k-1 (s , b).
(2.23)

But since batch algorithms do not have access to the model, they will approximate that step with a regression. First, they will build a dataset {(x j , y j )} j=1,...,n where x j = (s j , a j ) and y j = r j + γ max a Q k-1 (s j , a) and then find a best fit in an hypothesis Chapter 2. Background and Related Work space F. This means finding Q k ∈ F such that:

Q k ∈ argmin f ∈F n j=1 l(f (x j ), y j ).
(2.24)

This approach was first studied by Bellman [START_REF] Bellman | Polynomial approximation-a new computational technique in dynamic programming: Allocation processes[END_REF] with polynomial approximations. Fitted-Q [START_REF] Ernst | Tree-Based Batch Mode Reinforcement Learning[END_REF] iteration considers extra-trees for the regression method while Neural fitted-Q [START_REF] Riedmiller | Neural Fitted Q Iteration-First Experiences with a Data Efficient Neural Reinforcement Learning Method[END_REF] considers neural networks. The sample efficiency of this method is analysed in (Antos et al., 2008a).

Algorithm 4 Fitted -Q & Neural fitted-Q Input: D n = {(s j , a j , r j , s j )} j=1,.
..,n some samples, q 0 = 0 a Q-function, F an hypothesis space and a number of iterations K. for k=1,2,...,K do for all j do q j = r(s j , a j ) + γ max a q k-1 (s j , a) end for q k = argmin q∈F n j=1 l(q(s j , a j ), q j ) where l is a loss function.

end for Output: q K

As a remark, Ernst [START_REF] Ernst | Tree-Based Batch Mode Reinforcement Learning[END_REF] also uses his algorithm for continuous action spaces. The only problem is that it is cumbersome to find the maximum of the Q-function over the actions. Thus, in their experiment they discretize the action space to approximate the maximum step. This example emphasizes the necessity of analysing approximations in the greedy step in approximate dynamic programming algorithms (Section 1.2.3).

Least Squares Policy Iteration and Bellman Residual Minimizing Policy Iteration

The Least Squares Policy Iteration (LSPI) algorithm [START_REF] Lagoudakis | Least-Squares Policy Iteration[END_REF] and the Bellman Residual Minimizing Policy Iteration (BRMPI) algorithms are approximate PI algorithms. They will perform two steps as PI, and both approximate the policy evaluation steps of PI. Again, as we work with batch data, LSPI and BRMPI will perform their iterations on the Q-function instead of the value function. At iteration k, the PI algorithm's evaluation step computes the state-action value function

Q k = Q π k .
In the case of LSPI and BRMPI, the Q-function are lying in a d dimensional linear function space . . . , φ d ] are linearly independent functions from S × A → R and can be thought of as vectors of size |S| × |A| in the vector space of canonical basis {1 s,a } s,a∈S×A . The feature Φ can be seen as a matrix of size (|S| × |A|) × d and ω a vector of size d. Furthermore, let ρ be a probability distribution over the state-action space S × A and we will write ∆ ρ the diagonal application that maps 1 s,a to ρ(s, a)1 s,a . Finally, the orthogonal projection onto F Φ with respect to the L ρ,2 -norm is the application Φ

F Φ = {Φω, ω ∈ R d }. The features Φ = [φ 1 ,
(Φ ∆ ρ Φ) -1 Φ ∆ ρ = Π ρ,Φ . The L ρ,p of function f is defined here as f p ρ,p = x∈X ρ(x)f (x) p .
Least Squares Policy Iteration: At each iterations for a policy π, the PI algorithm finds Q satisfying Q = T π Q. Instead, LSPI finds ω satisfying the projected fixed point equation:

Φω = Π ρ,Φ T π Φω.
(2.25)

The solution for ω of this equality is given by:

ω = A -1 ρ,Φ,π b ρ,Φ , (2.26)
where

A ρ,Φ,π = Φ ∆ ρ (Φ -γP π Φ), (2.27)
and b ρ,Φ = Φ∆ ρ r.

(2.28)

Given a batch of data D n = {(s j , a j , r j , s j )} j=1,...,n , the matrix A ρ,Φ,π and the vector b ρ,Φ can be estimated as follow:

ÂΦ,π,Dn = 1 n n j=1 Φ(s j , a j ) Φ(s j , a j ) -γ a∈A π(a|s j )Φ(s j , a) , (2.29) bΦ,Dn = 1 n n j=1
Φ(s j , a j )r(s j , a j ).

(2.30)

If the data in D n are independent and distributed according the distribution ρ, ÂΦ,π,Dn and bΦ,Dn are consistent estimators of A ρ,Φ,π and b ρ,Φ . Furthermore, Â-1 Φ,π,Dn can be be efficiently computed with the Scherman-Morison formula which provides an optimized version of the algorithm. The algorithm in its most simple version is presented in Algorithm 5.

Algorithm 5 Least Squares Policy Iteration (LSPI)

Input: A batch D n = {(s j , a j , r j , s j )} j=1,...,n , a feature space Φ, an initial parametrization ω 0 = 0 and a number of iterations K. for k=1,2,...,K do find π k ∈ G(Φω k-1 ) compute ÂΦ,π k ,Dn and bΦ,Dn as in formula 2.96 and 2.97. compute ω k = Â-1 [START_REF] Lagoudakis | Least-Squares Policy Iteration[END_REF]. We provide details here as it will be improved in Chapter 6 and since it illustrates bias issues in the Chapter 2. Background and Related Work estimation of the Bellman residual. These bias issues will be encountered in Chapter 6 and in Chapter 7.

Instead of finding the fixed point of the projected Bellman operator to approximate Q π , BRMPI finds ω that minimizes the Bellman residual Φω -T π Φω ρ,2 . The solution to this minimization problem is: (2.31) where

ω = C -1 ρ,Φ,π d ρ,Φ ,
C ρ,Φ,π = Φ (I -γP π ) ∆ ρ (I -γP π )Φ, (2.32) and b ρ,Φ,π = Φ (I -γP π ) ∆ ρ r. (2.33)
This approach can also be implemented with batch data but the resulting solution for this estimation proposed by Lagoudakis and Parr is a biased and inconsistent estimator of C ρ,Φ,π . The following estimators ĈΦ,π,Dn and bΦ,π,Dn of C ρ,Φ,π and b ρ,Φ,π are:

ĈΦ,π,Dn = 1 n n j=1 Φ(s j , a j ) -γ a∈A π(a|s j )Φ(s j , a) Φ(s j , a j ) -γ a∈A π(a|s j )Φ(s j , a) , (2.34) dΦ,π,Dn = 1 n n j=1 Φ(s j , a j ) -γ a∈A π(a|s j )Φ(s j , a) r(s j , a j ).
(2.35)

Algorithm 6 Bellman Residual Minimizing Policy Iteration (BRMPI)

Input: A batch D n = {(s j , a j , r j , s j )} j=1,...,n , a feature space Φ, an initial parametrization ω 0 = 0 and a number of iterations K. for k=1,2,...,K do find π k ∈ G(Φω k-1 ) compute ÂΦ,π k ,Dn and bΦ,Dn as in formula 2.34 and 2.35. compute ω k = Â-1 Φ,π k ,Dn bΦ,Dn end for Output: π K As described in [START_REF] Lagoudakis | Least-Squares Policy Iteration[END_REF] the BRMPI algorithm would be as in Algorithm 6. The bias of the estimator can be solved with a generative model of the MDP. If, for each sample of the bacth, the next state can be sampled two times independently (i.e. the batch would be D n = {(s j , a j , r j , s j , s j )} j=1,...,n where s j ∼ p(.|s j , a j )), one can get the following unbiased estimator from the batch:

ĈΦ,π,Dn = 1 n n j=1 Φ(s j , a j ) -γ a∈A π(a|s j )Φ(s j , a) Φ(s j , a j ) -γ a∈A π(a|s j )Φ(s j , a) .
(2.36)

This presentation of LSPI and BRMPI sums up useful background for this dissertation. These algorithms have been extensively studied in the literature. Although LSPI is a batch algorithm, it has been adapted to the online setting [START_REF] Buşoniu | Online least-squares policy iteration for reinforcement learning control[END_REF][START_REF] Li | Online exploration in least-squares policy iteration[END_REF]. The sample complexity of LSPI has been analysed by [START_REF] Lazaric | Finite-sample analysis of least-squares policy iteration[END_REF] and the one of BRMPI by Antos et al. (2008b). The sample-complexity of policy evaluation through Bellman residual minimization has been analysed in [START_REF] Maillard | Finite-Sample Analysis of Bellman Residual Minimization[END_REF].

Approximate Modified Policy Iteration

In the two previous sections (Section 1.2.1 and 1.2.2) we introduced approximations of VI and PI. These approximations are made in the evaluation step (i.e.

v k = T π k v k-1 for VI and v k = (T π k ) +∞ v k-1 for PI).
In this section, we present results that shed light on the robustness of these algorithms to errors. Imagine that instead of performing the iteration

v k = T π k v k-1 (where π k ∈ G(v k-1 )), the update of v k is made up to an error k such that v k = T π k v k-1 + k .
But approximations can also occur in the greedy step [START_REF] Gabillon | Classification-Based Policy Iteration with a Critic[END_REF], Scherrer et al., 2012) as discussed in Section 1.2.1 and, instead of taking a greedy action, some algorithms select suboptimal actions in some states. We shall write π ∈ Ĝ (v) if T * v ≤ T π v + . In this section we provide the state of the art sensitivity analysis of Approximate Modified Policy Iteration (AMPI) like algorithms. The generic scheme studied is described in Algorithm 7 and applies to all algorithms described in Section 1.2.1 and in Section 1.2.2.

Algorithm 7 Approximate Modified Policy Iteration

Input: An MDP M , a value v 0 = 0, a maximum number of iterations K and a parameter m. for k=1,2,...,K do find

π k ∈ Ĝ k (v k-1 ) compute v k = (T π k ) m v k-1 + k end for Output: π K
The following analysis is standard in approximate dynamic programming algorithms. As this kind of analysis will be done for several algorithms in Part II we detail here the case of AMPI of Scherrer et al. (2012). These bounds are decomposed in a sum of 3 terms: the value update error, the greedy error and a concentration term. Each of these terms is decomposed in a product of 3 quantities. All these terms can be controlled: the first one can be controlled by the accuracy of the value function update, the second one can be controlled making small approximation on the greedy step and the last one by the number of iterations of the algorithm. These analysis were performed in L ∞ -norm and then were performed in L p -norm [START_REF] Farahmand | Error Propagation for Approximate Policy and Value Iteration[END_REF], Antos et al., 2008a[START_REF] Munos | Performance Bounds in L p -norm for Approximate Value Iteration[END_REF].

The performance of AMPI after k-iterations measures in L ρ,p -norm the distance between v π k and v * . The measure ρ specifies the distribution over the state space on which the performance needs to be accurate. Thus, we will bound the norm v *v π k ρ,p . Since our method performs approximations at each iteration, our bound will depend on 1 , . . . , k-1 and 1 , . . . , k . These errors are supposed to be controlled in L σ,pq -norm where σ is the distribution on which the estimation is supposed to be accurate. For instance in Fitted-Q iteration, j , j ≤ k the error of the regression at iteration j. In that case, σ is the distribution of the data and d = 2 (since the L 2 loss is often used for trees). Theorem 2.3 shows the impact of the approximation on the final solution.

Theorem 2.3. Let ρ and σ be distributions over states. Let p, q and q' be such that 1 q + 1 q = 1. Then, after k iterations, we have:

v * -v π k ρ,p ≤ 2(γ -γ k )(C 1,k,0 q ) 1 p (1 -γ) 2 sup 1≤j≤k-1 j σ,pq value update error + (1 -γ k )(C 0,k,0 q ) 1 p (1 -γ) 2 sup 1≤j≤k j σ,pq greedy error , (2.37) + 2γ k 1 -γ (C k,k+1,0 q ) 1 p min ( v * -v 0 σ,pq , v 0 -T π 1 v 0 σ,pq ) contraction term . (2.38)
where

C l,k,d q = (1 -γ) 2 γ l -γ k k-1 i=l ∞ j=i γ j c q (j + d), (2.39)
with the following norm of a Radon-Nikodym derivative:

c q (j) = sup π 1 ,...,π j d(ρP π 1 ...P π j ) dσ q,σ
.

(2.40)

The contribution of the value update error can be divided in three parts:

2(γ -γ k )(C 1,k,0 q ) 1 p (1 -γ) 2 sup 1≤j≤k-1 j σ,pq value update error = 2(γ -γ k ) (1 -γ) 2 γ-sensitivity (C 1,k,0 q ) 1 p concentrability coeficient sup 1≤j≤k-1 j σ,pq -error .
(2.41)

The sensitivity to γ is of the order of the square of the horizon 1 1-γ . The sensitivity to the error depends on the maximum of the norm of the errors. The concentrability coefficient (C 1,k,0 q ) 1 p measures the impact of controlling the error with respect to measure σ and guaranteeing a performance on measure ρ. It is a discounted sum of the norm of the Radon-Nikodym derivative c q (j) (in discrete actions spaces, this derivative is the ratio of ρP π 1 ...P π j and σ). The larger j is the more c q (j) is discounted in the coefficient C l,k,d q . Indeed, if we only have access to samples to control the regression error of a part of the state-action space that is never visited from the part of the state-action space on which we want to give guarantees, then the concentrability coefficient will be infinite. As an example, the concentrability coefficient will always be finite if the measure σ is uniform. The reader can find in [START_REF] Scherrer | Approximate Policy Iteration Schemes: A Comparison[END_REF] a detailed and meticulous comparison of the different concentrability coefficients.

The greedy error term can be decomposed in the same way and is comparable to the value update error term:

(1 -γ k )(C 0,k,0 q ) 1 p (1 -γ) 2 sup 1≤j≤k j σ,pq greedy error = (1 -γ k ) (1 -γ) 2 γ-sensitivity (C 0,k,0 q ) 1 p concentrability coeficient sup 1≤j≤k j σ,pq -error . (2.42)
Finally, the last term of that sum is the contraction term and can be decomposed in three terms:

2γ k 1 -γ (C k,k+1,0 q ) 1 p min ( v * -v 0 σ,pq , v 0 -T π 1 v 0 σ,pq ) contraction term = (2.43) 2γ k 1 -γ γ-contraction (C k,k+1,0 q ) 1 p concentrability coeficient min ( v * -v 0 σ,pq , v 0 -T π 1 v 0 σ,pq )
error initial value .

(2.44)

This last term converges to zero at a geometrical rate (since the γ-contraction term is O(γ k )). It also depends on a concentrability coefficient (C k,k+1,0 q ) 1 p and on a term that reflects how close the initial solution v 0 is from the optimal solution v * .

Non-Stationary Policies in MDPs:

As described in this section, the sensitivity to γ is one of the strongest drawbacks of AMPI algorithms. One way to reduce this dependency is to use a non-stationary policy. In [START_REF] Scherrer | On the Use of Non-Stationary Policies for Stationary Infinite-Horizon Markov Decision Processes[END_REF], Scherrer shows that using cyclic policies reduces the dependency on γ. In this paper, they prove that playing the m last policies given by approximate value iterations (instead of playing π K at each time step, they play the cyclical policy π K , π K-1 , . . . , π K-m+1 , π K , π K-1 , . . . ). In the case of cyclical policies, the γ-sensitivity of approximate value iteration is

2(γ-γ k ) (1-γ)(1-γ m ) instead of 2(γ-γ k )
(1-γ) 2 for the value update error and

1-γ k (1-γ)(1-γ m ) instead of 1-γ k (1-γ) 2
for the greedy error term. The AMPI algorithm was adapted to use non-stationary strategies and analysed in [START_REF] Lesner | Non-stationary approximate modified policy iteration[END_REF].

Bellman Residual Minimization

This section on Bellman residual minimization introduces an approach existing in MDPs that we will study in MGs (part III).

Finding an optimal policy can be achieved in MDPs if one finds the optimal Qfunction Q * . This can be achieved by the minimization of the Optimal Bellman Residual (OBR) Q -B * Q ρ,2 . If π is greedy with respect to any Q the following bounds on Q * -Q π ρ,2 stands (Piot et al., 2014a)

Q * -Q π ρ,2 ≤ 2 1 -γ C 2 (ρ, π) + C 2 (ρ, π * ) 2 1 2 B * Q -Q ρ,2 .
(2.45)

Chapter 2. Background and Related Work

with C 2 (ρ, π) = || ∂ρ (1-γ)(I-γPπ) -1 ∂ρ || 2,ρ the Radon-Nikodym derivative of the Kernel (1 -γ)(I -γP π ) -1 .
The optimal Bellman residual minimization approach is to use a batch of data D n = {(s j , a j , r j , s j )} j=1,...,n to estimate the OBR. In (Piot et al., 2014a), the following estimator is proposed:

ĴOBR (Q) = n j=1 Q(s j , a j ) -r j -γ max a Q(s j , a) 2 .
(2.46)

The goal is now to minimize a loss where Q ∈ F with any relevant optimization technique:

Q = argmin Q∈F ĴOBR (Q).
(2.47)

This estimator is biased and not consistent (Piot et al., 2014b,a) in the case of a stochastic dynamic. Many techniques have been developed to improve the estimation of the OBR including [START_REF] Grunewalder | Modelling Transition Dynamics in MDPs With RKHS Embeddings[END_REF][START_REF] Taylor | Value Function Approximation in Noisy Environments Using Locally Smoothed Regularized Approximate Linear Programs[END_REF][START_REF] Maillard | Finite-Sample Analysis of Bellman Residual Minimization[END_REF].

Online Learning in MDPs

The last part of this dissertation will study independent learning in multi-stage games. As independent learning is a generalization to MGs of online learning in MDPs, we briefly introduce here two basic algorithms.

Whilst batch algorithms were designed to learn a near optimal behaviour without being able to access the environment, online algorithms were designed to learn a policy in interaction with the environment. At time t, the agent is in state s t and takes an action a t . With the reward r t collected, the agent will perform an update.

Q-learning

The Q-learning algorithm [START_REF] Watkins | Q-learning[END_REF] learns the optimal Q-function. This algorithm is an off-policy algorithm (there is a fixed policy used to behave which is different from the learnt policy). At each step the algorithm updates the Q-function in order to estimate the optimal Q-function Q * . For the on-policy version of the Qlearning algorithm, the policy to select the action a t at time t is usually an -greedy policy with respect to the current estimate of the Q-function.

Algorithm 8 Q-learning

Input: An MDP M , a Q-value Q 0 = 0, an initial state s 0 and a number of iterations T . for t=0,1,2,...,T-1 do take action a t according to an exploration policy (usually greedy policy), collect reward r t ∼ r(s t , a t ) and move to state

s t+1 ∼ p(.|s t , a t ), update Q t+1 : Q t+1 (s t , a t ) = Q t (s t , a t ) + α t r t + γ max a Q t (s t+1 , a) -Q t (s t , a t ) end for Output: Q T 1.3.2 SARSA
The SARSA algorithm will instead update the value according to the so called "Temporal difference" error. For a given behavioural policy π, the Q-learning algorithm will learn the optimal Q-function whilst SARSA will learn the Q-function associated with the behaved policy Q π .

Algorithm 9 SARSA

Input: An MDP M , a Q-value Q 0 = 0, an initial state s 0 and a number of iterations T . for t=1,2,...,T do take action a t according to an exploration policy (usually the greedy policy), collect reward r t ∼ r(s t , a t ) and move to state s t+1 ∼ p(.|s t , a t ),

Q t+1 (s t , a t ) = Q t (s t , a t ) + α t (r t + γQ t (s t+1 , a t+1 ) -Q t (s t , a t ))
end for Output: Q T

Normal-Form Games

Before introducing Markov Games (MGs), we will first shortly introduce Normal Form Games (NFGs) which is a stateless MG. In MDPs or in single agent problems, there is only a single way to define an optimal policy. In a multi-agent system, many solutions can be studied. In this section, we will introduce two of them: the minimax solution and the Nash equilibrium. These two notions will be generalized in the next section to Markov games.

In an N -player NFG, each player chooses actions a i in a finite set A i of available actions. Each player receives an individual reward r i (a 1 , . . . , a N ) ∈ R. For the sake of simplicity, we will write the joint action a = (a 1 , . . . , a N ) = (a i , a -i ) where a -i is the joint action of every player except player i. The goal of each player is to find a strategy π i ∈ ∆(A i ). If every player plays his strategy, player i receives the average reward E a∼π [r i (a)]. Again, the joint policy of all players is π and the joint policy of every player except player i is π -i . Finally, an NFG can be represented as a tuple < {A i } i∈{1,...,N } , {r i (a)} i∈{1,...,N } >.

In an NFG, the equivalent of a value function for player i would be r i π = E a∼π [r i (a)] and the equivalent of a Q-function would be r

i π -i (a i ) = E a -i ∼π -i r i (a i , a -i )

Nash equilibrium:

This thesis will focus on Nash equilibrium as a solution concept. In a Nash equilibrium, no player has an incentive to switch from his current strategy if other players stick to their own. The common way to define a Nash equilibrium is the following:

Definition 2.1. A strategy π is a Nash equilibrium if: ∀i ∈ {1, . . . , N }, r i π = max a i r i π -i (a i ). (2.48)
This definition can also be rewritten in reinforcement learning language as follows:

Definition 2.2. A strategy π is a Nash equilibrium if: ∀i ∈ {1, . . . , N }, π i ∈ G(r i π -i ), (2.49)
As in the previous section, G(r i

π -i ) = {π| E a i ∼π [r i π -i (a i )] = max a i r i π -i (a i )}
Finding a Nash equilibrium in Normal form games is considered as a hard algorithmic problem [START_REF] Daskalakis | The complexity of computing a Nash equilibrium[END_REF]. Several algorithms can be used to solve this problem such as the Lemke Howson algorithm or searching in the support space for the two-player case [START_REF] Shoham | Multiagent systems: Algorithmic, game-theoretic, and logical foundations[END_REF]. The problem is considered to be "hopelessly impractical to solve exactly" for more than two players [START_REF] Shoham | Multiagent systems: Algorithmic, game-theoretic, and logical foundations[END_REF]. These algorithms will not be presented in details here as these methods will not be used in this thesis(see (Nisan et al., 2007, Shoham and[START_REF] Shoham | Multiagent systems: Algorithmic, game-theoretic, and logical foundations[END_REF] for more involved explanations).

Zero-Sum Normal-Form Games

Zero-sum two-player NFG is one of the class of NFGs where a Nash equilibrium can be found in a reasonable time. A zero-sum two-player NFG is an NFG where N = 2 and where for all a 1 , a 2 ∈ A 1 × A 2 , r 1 (a 1 , a 2 ) = -r 2 (a 1 , a 2 ) = r(a 1 , a 2 ). In that case, player 1 will attempt to maximize his expected outcome while the second player's goal is to minimize it. In that case, the concept of a Nash equilibrium and the minimax solution are equivalent. In the specific case of zero-sum two-player games we will define µ the policy of player 1 and ν the policy of player 2. Definition 2.3. A pair of strategies (µ, ν) is a Nash equilibrium if:

µ ∈ argmax µ∈∆(A 1 ) min a 2 E a 1 ∼µ [r(a 1 , a 2 )],
(2.50)

ν ∈ argmin ν∈∆(A 2 ) max a 1 E a 2 ∼ν [r(a 1 , a 2 )].
(2.51)

General-Sum Markov Games 27

Furthermore, all Nash equilibria in zero-sum two-player NFGs achieve the same value:

v * = max µ∈∆(A 1 ) min a 2 E a 1 ∼µ [r(a 1 , a 2 )] = min ν∈∆(A 2 ) max a 1 E a 2 ∼ν [r(a 1 , a 2 )].
(2.52)

In a zero-sum two-player game, the minimax strategy can be found in the set of stochastic strategies. That's why in equation (2.50), the policy µ needs to be found in the set of distributions over A 1 . But a best response of player 2 against µ can always be found in the set of deterministic strategies. That's why in equation (2.50), a 2 is located in the set of deterministic strategies even if ν (defined in (2.51)) would be a valid best response.

This optimal value v * is the solution of two dual linear programs. The first linear program involves the strategy of player 1 (µ) and finds the value that maximizes over µ the minimum over a 2 of E a 1 ∼µ [r(a 1 , a 2 )]:

max v * Subject to a 1 ∈A 1 r(a 1 , a 2 )µ(a 1 ) ≥ v * , ∀a 2 ∈ A 2 a 1 ∈A 1 µ(a 1 ) = 1 µ(a 1 ) ≥ 0, a 1 ∈ A 1
It's dual involves the strategy of player 2 ν and finds the value that minimizes over ν the maximum over a 2 of E a 2 ∼ν [r(a 1 , a 2 )]:

min v * Subject to a 2 ∈A 2 r(a 1 , a 2 )ν(a 2 ) ≤ v * , ∀a 1 ∈ A 1 a 2 ∈A 2 ν(a 2 ) = 1 ν(a 2 ) ≥ 0, a 2 ∈ A 2
These two programs can be solved with any linear programming algorithms such as the simplex method or the interior point method. The complexity of a linear program depends on the number of constrains c = |A 1 | + |A 2 | + 1 or in the number of variables |A 1 | for the first linear program and |A 2 | the number of variables for the second one. For instance, the complexity of the simplex method is exponential in c in the worst case (for both linear programs) while the complexity of the interior point method is O(|A 1 | 3.5 ) for the first linear program and O(|A 2 | 3.5 ) for the second one [START_REF] Karmarkar | A New Polynomial-time Algorithm for Linear Programming[END_REF]. Linear programming was adapted to solve large turn taking partial information games [START_REF] Koller | Fast algorithms for finding randomized strategies in game trees[END_REF]. It was the state of the art technique to solve poker before Counter Factual Regret minimization (CFR) [START_REF] Zinkevich | Regret minimization in games with incomplete information[END_REF].

General-Sum Markov Games

General-sum Markov Games (MG) are a generalization of MDPs for the multi-player scenario and a generalization of NFGs to environments where the interactions also Chapter 2. Background and Related Work depend on state. In an MG, N players evolve in a discrete state space S. As in MDPs, the interaction is sequential and at time t, players belong to state s t . In that state, they all simultaneously choose an action a i t in the action space A i (a i t is the action of player i at time t) and receive a reward r i (s t , a 1 t , . . . , a N t ). Again, we will write a = (a 1 , . . . , a N ) = (a i , a -i ) where a -i is the joint action of every player except player i. Then, the state changes to state s t+1 ∼ p(.|s t , a 1 t , . . . , a N t ) where p(.|s, a 1 , . . . , a N ) is the transition kernel of the MG. The constant γ ∈ [0, 1] is the discount factor of the MG. Finally, an MG is a tuple < S, {A i } i∈{1,...,N } , {r i } i∈{1,...,N } , p, γ >.

In an MG, each player's goal is to find a strategy {π i t } t∈R . The strategy π i t at time t is a mapping from the state space to the a distribution over the action space (i.e. π i t ∈ ∆A S ). These strategies are independent and actions are supposed to be chosen simultaneously. Again, we will define {π t } t∈N as the joint strategy of all players and {π -i t } t∈N as the strategy of every player except i. If the strategy is stationary, it will be written π i , π and π -i . Furthermore, we will often use short notations such as π(b|s

) = π 1 (b 1 |s) × • • • × π N (b N |s) or π -i (b -i |s) = π 1 (b 1 |s) × • • • × π i-1 (b i-1 |s) × π i+1 (b i+1 |s) × • • • × π N (b N |s)
The value for player i in state s is his expected γ discounted sum of rewards starting from state s when all players play their part of the joint policy {π t } t∈N :

v i {πt} t∈N (s) = E +∞ t=0 γ t r i (s t , a t )|s 0 = s, a t ∼ π t (.|s t ), s t+1 ∼ p(.|s t , a t ) . (2.53)
When this strategy is stationary, the value function of player i is defined as: (2.54)

v i π (s) = E +∞ t=0 γ t r i (
In the case of stationary strategies, if every player except player i sticks to his strategy π -i the value the best response player i can achieve is: This value exists since when the strategy of the opponents π -i is fixed, the model reduces to an MDP for player i. This value of the best response is the optimal value of that MDP. The following Q-function can be defined:

v * i π -i (s) = max π i E +∞ t=0 γ t r i (
Q i π (s, a) = E +∞ t=0 γ t r i (s t , a t )|s 0 = s, a 0 = a, a t ∼ π(.|s t ), s t+1 ∼ p(.|s t , a t ) , (2.56) = r i (s, a) + γ s ∈S p(s |s, a)v i π (s ).
(2.57)

One can also define the Q-function for player i of the best response:

Q * i π -i (s, a) = max π i Q i π (s, a), (2.58) = r i (s, a) + γ s ∈S p(s |s, a)v i * π -i (s ).
(2.59)

These Q-functions are respectively the value for player i in state s if all players start from action a when they play the joint strategy π and when player i plays a best response to policy π -i .

Stochastic Matrices and Linear Algebra:

As for MDPs, we can define the linear operator as the transition kernel. The probability transition kernel P π is the stochastic matrix that defines the following linear transformation:

P π v = s∈S 1 s s ∈S E a∼π(.|s) [p(s |s, a)] v(s ).
(2.60) Furthermore, we will define

r i π = s∈S 1 s E a∼π [r i (
v i π = (I -γP π ) -1 r i π (2.61)
The linear Bellman operator on Q-function can be expressed with linear operators. Let's define the state-action transition kernel:

P π Q = s,a∈S×A 1 s,a s ,b∈S×A π(b|s )p(s |s, a)Q(s , b),
(2.62) and the reward:

r i = s,a∈S×A 1 s,a r i (s, a).
(2.63) with these definitions, we have:

Q i π = (I -γP π ) -1 r i .
(2.64)

Nash Equilibrium -Nash Equilibrium

A Nash equilibrium is a game theoretic solution concept. In a Nash equilibrium, no player can improve his own value by changing his strategy if the other players stick to their strategies [START_REF] Filar | Competitive Markov Decision Processes[END_REF]. In Markov games, this definition must stand in all states:

Definition 2.4. In an MG, a strategy π is a Nash equilibrium if

: ∀i ∈ {1, ..., N }, v i π = v * i π -i .
In an MDP, a Nash equilibrium is simply the optimal strategy. An -Nash equilibrium is a relaxed solution concept in game theory. When all players play an -Nash equilibrium the value they receive is at most sub-optimal compared to a best response. Formally [START_REF] Filar | Competitive Markov Decision Processes[END_REF]: Definition 2.5. In an MG, a strategy π is an -Nash equilibrium if: ∀i ∈ {1, ..., N },

v i π + ≥ v * i π -i 30
Chapter 2. Background and Related Work

Bellman Operator in General-Sum Games

As explained above, when the strategy of all opponents is fixed, the problem reduces to an MDP. Thus, two Bellman operator can be defined per player.

T i π v (s) = r i π (s) + γ s ∈S p π (s |s)v(s ) (2.65) T i π v = r i π + γP π v (2.66)
The Bellman operator of the best response of player i to strategy π -i :

T * i π -i v (s) = max π i   r i π i ,π -i + γ s ∈S p π i ,π -i (s |s)v(s )   (2.67)
Respectively, one can define operators on Q-functions as follow:

B i π Q (s, a) = r i (s, a) + γ s ∈S p(s |s, a)E b∼π(.|s ) [Q(s , b)] (2.68) = r + γP π Q (2.69) B * i π Q (s, a) = r i (s, a) + γ s ∈S p(s |s, a) max b i E b -i ∼π -i (.|s ) [Q(s , b)]
(2.70)

Again, the fixed point of

T i π is v i π , the fixed point of T * i π -i is v * i π -i , the fixed point of B i π is Q i π and the fixed point of B * i π is Q * i π .

Zero-Sum Two-Player Markov Games

As for NFGs we will introduce specific notations and highlight the properties specific to zero-sum two-player MGs. In a zero-sum two-player MG the reward of Player 1 is the loss of Player 2. Thus, for all s, a 1 , a 2 ∈ S × A 1 × A 2 we have r 1 (s, a 1 , a 2 ) = -r 2 (s, a 1 , a 2 ) = r(s, a 1 , a 2 ). In this setting, player 1 will attempt to maximize his rewards while player 2 will attempt to minimize his losses. In this specific setting, we will define the strategy of player 1 (µ) instead of π 1 and the strategy of player 2 (ν) instead of π 2 . The value of the joint policy (µ, ν) is:

v µ,ν = v 1 µ,ν = v 1 π 1 ,π 2 .
(2.71)

The value of the best response of player 2 to the strategy of player 1 µ is:

v µ = min ν v µ,ν = -v * 2 µ .
(2.72)

In an MG, there exists a unique value for all Nash equilibrium:

v * = min ν max µ v µ,ν = max µ min ν v µ,ν .
(2.73)

Bellman Operator: In zero-sum two-player MGs, five Bellman Operator can be described. Three of them correspond to the Bellman operators for general sum MGs (i.e. the Bellman operators introduced in eq. 2.74, eq. 2.75 and eq. 2.77).

T µ,ν v = r µ,ν + γP µ,ν v (2.74) T µ v = min ν T µ,ν v (2.75) T v = max µ T µ v (2.76) Tν v = max µ T µ,ν v (2.77) T v = min ν Tν v (2.78) The operator T µ,ν is either T 1 µ,ν or v → -T 2 µ,ν (-v). The operator T µ is v → -T * 2 µ (-v)
and Tν is T * 1 ν . However, operator T and T have no equivalent in general-sum games. These two operators are equivalent [START_REF] Patek | Stochastic Shortest Path Games: Theory and Algorithms[END_REF] and are γ-contractions. The fixed point of T and T is v * .

Q-Functions and Operators on the Q-Functions: These five operators have their counterparts for Q-functions. For the sake of completeness, we briefly define them in this paragraph. The linear operator B µ,ν is equal to B 1 µ,ν and equal to

Q → -B 2 µ,ν (-Q). The operator B µ is v → -B * 2
µ (-v) and the operator Bν is T * 1 ν . Finally, the equivalents of T and T are:

[BQ] (s, a 1 , a 2 ) = r(s, a 1 , a 2 ) + γ s ∈S p(s |s, a 1 , a 2 ) max µ∈∆A 1 min ν∈∆A 2 E b 1 ∼µ,b 2 ∼ν [Q(s , b 1 , b 2 )], (2.79) BQ (s, a 1 , a 2 ) = r(s, a 1 , a 2 ) + γ s ∈S p(s |s, a 1 , a 2 ) min ν∈∆A 2 max µ∈∆A 1 E b 1 ∼µ,b 2 ∼ν [Q(s , b 1 , b 2 )].
(2.80)

The fixed point of B µ,ν is the state-action value function of the joint strategy µ, ν and will be written

Q µ,ν = Q 1 µ,ν = -Q 2 µ,ν .
The fixed point of B µ is the Q-function of the best response of player 2 to the strategy µ and will be written

Q µ = -Q * 2
µ . The minimax state-action value function is Q * and is the fixed point of B or B.

Greedy Strategy in Zero-Sum Two-Player Markov Games:

The notion of greedy strategy in zero-sum two-player MGs is slightly different from the case of MDPs. In zero-sum two-player MGs we need to take the point of view of one player (player 1 or player 2). The canonical way to do it is to take the point of view of player 1 but the two point of view are dual. A strategy µ is said to be greedy with respect to a value function v if T µ v = T v and we will define G(v) = {µ| T µ v = T v}. A strategy µ is greedy with respect to a Q-function if for all s we have min

ν∈∆A 2 max µ∈∆A 1 E a 1 ∼µ,a 2 ∼ν [Q(s, a 1 , a 2 )] = min ν∈∆A 2 E a 1 ∼µ(.|s),a 2 ∼ν [Q(s, a 1 , a
2 )] and we will define it µ ∈ G(Q). We choose this definition of a greedy strategy with the intention to build policy iteration algorithms that converges to the minimax value v * but others operators could be considered if the problem was to find a best response to some strategy. Again, two results similar to the case of MDPs and can be found:

Chapter 2. Background and Related Work Theorem 2.4. A strategy µ is optimal if and only if it is greedy with respect to it's value v µ .

v µ = v * if and only if µ ∈ G(v µ ) (2.81)
This first theorem implies that the optimal policy can be retrieved from the optimal value. Theorem 2.5. If µ is greedy with respect to the value v µ of strategy µ, then v µ ≥ v µ This second theorem proves that a simple way to improve a strategy π is to play the greedy strategy π with respect to the value v π . This theorem will be the basis of the Policy Iteration algorithm for zero-sum two-player MGs (Section 3.4.2). These two theorems can be deduced from [START_REF] Patek | Stochastic Shortest Path Games: Theory and Algorithms[END_REF].

Exact Algorithms for Zero-Sum Two-Player Markov-Games

The Value Iteration (VI) algorithm for zero-sum two-player MGs [START_REF] Shapley | Stochastic Games[END_REF] will be detailed in Section 3.4.1 and is a generalization of the VI algorithm for MDPs (Section 1.1.1). As in MDPs, VI is relatively slow at the beginning even if iterations are computationally cheap. This motivates the use of algorithms that have more computationally intensive iterations such as the Policy Iteration algorithm. Howard's PI algorithm for MDPs (detailed in sec 1.1.2) has two extensions to zero-sum two-player MGs. The first one is Hoffman and Karp's Policy Iteration [START_REF] Hoffman | On nonterminating stochastic games[END_REF]. This algorithm is cumbersome since it requires solving an MDP as a subroutine (see Section 3.4.2). To address this issue, Pollatschek and Avi-Itzhak [START_REF] Pollatschek | Algorithms for Stochastic Games with Geometrical Interpretation[END_REF] proposed their own generalization of the Howard PI algorithm that only requires a joint strategy evaluation subroutine (see Section 3.4.3). Despite being efficient in practice, this algorithm was not proven to converge in all zero-sum two-player MGs. In fact, less than 10 years later Van Der Wal presented a simple counterexample [START_REF] Van Der Wal | Discounted Markov Games: Generalized Policy Iteration Method[END_REF] on which the Pollatschek and Avi-Itzhak's algorithm was oscillating. In the same paper, he presented his own improvement of the Hoffman and Karp's PI algorithm named the Generalized Policy Iteration (GPI) (detailed in Section 3.4.4). This algorithm is a generalization to zero-sum two-player MGs of MPI (Section 1.1.3). More than 10 year later, Filar and Tolwinski revisited the Pollatschek and Avi-Itzhak's algorithm. They proved that this algorithm was in fact a Newton's method and the L 2 -norm of the Optimal Bellman Residual (OBR). Furthermore, after showing that every local minimum of the OBR is in fact a global minimum, they suggest the use of a quasi-Newton method on the OBR instead of a Newton's method. This last method is detailed in 3.4.5.

Value Iteration

The value iteration algorithm (see Algorithm 10) was proposed by Shapley in the same paper introducing MGs [START_REF] Shapley | Stochastic Games[END_REF]. It iteratively applies the Bellman operator

T at each iteration. This algorithm produces a sequence of values v k and an implicit sequence of greedy strategies µ k with respect to v k . Each iteration's cost is |S| times the cost of solving a minimax problem. This cost thus depends on the choice of the algorithm for the minimax problem(see Section 2.2).

Algorithm 10 Value Iteration

Input: An zero-sum two-player MG M G, a value v 0 = 0 and a maximum number of iterations K. for k=1,2,...,K-1 do

v k = T v k-1 end for find µ K ∈ G(v K-1 ) Output: µ K
The performance of µ K is usually determined as the norm of the difference between v µ K and the minimax value v * . The prof of the following result can be found in [START_REF] Patek | Stochastic Shortest Path Games: Theory and Algorithms[END_REF]:

v µ K -v * +∞ ≤ 2γ K 1 -γ v * -v 0 +∞ . (2.82)
As for MDPs, the cost of each iteration is relatively low compared to other methods but the convergence of this method is slow in the number of iterations.

Policy Iteration by Hoffman and Karp

Hoffman and Karp's PI has two steps per iteration. Suppose we are at iteration k. The first step is to find a greedy strategy µ k with respect to the previous value v k-1 . This greedy step requires solving a minimax problem per state. The second step of this process is to find the value of a best response to µ k . This step requires building the MDP of the kernel E a 1 ∼µ(.|s) [p(.|s, a 1 , a 2 )] and of the reward E a 1 ∼µ(.|s) [r(s, a 1 , a 2 )] and to solve it by minimizing the sum of rewards instead of maximizing them. This second step computes v µ k . Any algorithm solving an MDP can be selected for this second step and the complexity of the overall method will depend on the algorithm selected. This algorithm reduces to policy iteration in the case the MG is an MDP (i.e. in the case

|A 2 | = 1).
This algorithm produces an increasing sequence of values v k . This property is implied by theorem 2.5.

Again, one can prove the following result on the performance of policy π K :

v µ K -v * +∞ ≤ 2γ K 1 -γ v * -v 0 +∞ . (2.83)
This bound shows the same asymptotic performance as the one of Value Iteration. But in practice, PI for zero-sum two-player MGs converges faster than VI (Van

Algorithm 11 Policy Iteration

Input: An zero-sum two-player MG M G, a value v 0 = 0 and a maximum number of iterations K. for k=1,2,...,K do , 1978). Further results can be proven on the number of iterations of PI for turn taking games (Hansen et al., 2013b). However, to the best of our knowledge, no better convergence guarantees exist in the case of simultaneous move games.

find µ k ∈ G(v k-1 ) (min max step) compute v k = v µ k (solving an MDP) end for Output: µ K Der Wal

The Algorithm of Pollatschek and Avi-Itzhak

As described in the previous section, Hoffman and Karp's PI algorithm is cumbersome since it requires solving an MDP at each iteration. Pollatschek and Avi-Itzhak's algorithm (Algorithm 12) is an attempt to address this issue by changing the second step of the algorithm. Instead of solving an MDP at the second step of the iteration, this algorithm evaluates the joint strategy µ k , ν k . Pollatschek and Avi-Itzhak's algorithm is said to be efficient in practice [START_REF] Van Der Wal | Discounted Markov Games: Generalized Policy Iteration Method[END_REF] but lacks theoretical guarantees of convergence to a minimax solution. In fact, Van Der Wal proved that it was oscillating in certain cases.

Algorithm 12 Algorithm of Pollatschek and Avi-Itzhak

Input: An zero-sum two-player MG M G, a value v 0 = 0 and a maximum number of iterations K. for k=1,2,...,K do

find µ k , ν k such that T µ k ,ν k v k-1 = T µ k v k-1 = Tν k v k-1 = T v k-1 compute v k = v µ k ,ν k = (I -γP µ k ,ν k ) -1 r µ k ,ν k end for Output: µ K

Generalized Policy Iteration

In the same paper proving the flaws of Pollatschek and Avi-Itzhak's algorithm (Van Der Wal, 1978), Van Der Wal gave his own solution to the Hoffman and Karp's PI problem. He introduces the Generalize Policy Iteration algorithm (GPI) that generalizes the MPI (Section 1.1.3) to MGs. GPI also introduces a parameter m that controls the cost of each iteration. This algorithm also reduces to VI if m = 1 and also reduces to Hoffman and Karp' PI if m = +∞. Indeed, in Algorithm 13 if m = +∞, the value v k is the value

v µ k . If m = 1, since µ k ∈ G(v k-1 ) we have T v k-1 = T µ k v k-1 and finally v k = T µ k v k-1 = T v k-1 .
Again, this algorithm enjoys the same guarantees of convergence as PI and VI and again the performance policy µ k does not depend on m.

v µ K -v * +∞ ≤ 2γ K 1 -γ v * -v 0 +∞ . (2.84)
But as shown by in [START_REF] Van Der Wal | Discounted Markov Games: Generalized Policy Iteration Method[END_REF], this algorithm converges faster than VI. And its cost per iteration is lower than the PI by Hoffman and Karp.

Algorithm 13 Generalized Policy Iteration

Input: An MDP M , a value v 0 = 0, a maximum number of iterations K and a parameter m. for k=1,2,...,K do

find µ k ∈ G(v k-1 ) compute v k = (T µ k ) m v k-1 end for Output: µ K

Minimizing the Bellman Residual: Filar and Tolwinski's Algorithm

We will take extra care to detail this algorithm as it inspired the work detailed in Chapter 6.

In an enlightening paper, Filar and Tolwinski prove that the Pollatschek and Avi-Itzhak algorithm is a Newton's method on the L 2 -norm of the Optimal Bellman Residual (OBR)

J OBR (v) = v -T v 2 = (v -T v) (v -T v). Indeed, the gra- dient of J OBR (v) is ∇J OBR (v) = (I -γP µ,ν ) (v -T v) and the Hessian matrix is HJ OBR (v) = (I -γP µ,ν ) (I -γP µ,ν ) where µ, ν are such that T µ,ν v = T µ v = Tν v = T v.
From value v k , the update v k+1 of the Newton's method is

v k+1 = v k -(HJ OBR (v k )) -1 ∇J OBR (v k ), (2.85) = v k -(I -γP µ k ,ν k ) (I -γP µ k ,ν k ) -1 (I -γP µ k ,ν k ) (v k -T v k ), (2.86) = v k -(I -γP µ k ,ν k ) -1 (v k -T µ k ,ν k v k ), (2.87) = v k -(I -γP µ k ,ν k ) -1 ((I -γP µ k ,ν k )v k -r µ k ,ν k ), (2.88) = v k -v k + (I -γP µ k ,ν k ) -1 r µ k ,ν k , (2.89) = (I -γP µ k ,ν k ) -1 r µ k ,ν k = v µ k ,ν k . (2.90)
Thus, the idea of Filar and Tolwinski is to make the use of a quasi-Newton method instead of Newton's method. At each iteration, they introduce a step size α k chosen by line search (Amijo's rule [START_REF] Filar | On the Algorithm of Pollatschek and Avi-ltzhak[END_REF]) instead of using a step size of 1. The update becomes:

v k+1 = (1 -α k )v k + α k v µ k ,ν k .
(2.91) Furthermore, they notice that the gradient of J OBR (v) is null if and only if v -T v = 0. Thus every local minimum of the OBR is a global minimum. Thus, they claim that their algorithm converges to a global minimum. The algorithm is described in Algorithm 14.

Algorithm 14 Filar and Tolwinski's Algorithm Input: An zero-sum two-player MG M G, a value v 0 = 0 and a maximum number of iterations K. for k=1,2,...,K do

find µ k , ν k such that T µ k ,ν k v k-1 = T µ k v k-1 = Tν k v k-1 = T v k-1 compute v µ k ,ν k = (I -γP µ k ,ν k ) -1 r µ k ,ν k find α k according to Amijo's rule for the function J OBR update v k+1 = (1 -α k )v k + α k v µ k ,ν k end for Output: µ K

Batch Algorithms for Zero-Sum Two-Player Markov-Games

Even if many batch algorithms were developed for MDPs, this topic remains superficially explored in Markov Games. The only algorithm handling batch data is LSPI for zero-sum two-player MGs [START_REF] Lagoudakis | Value Function Approximation in Zero-Sum Markov Games[END_REF] is a generalization of the LSPI algorithm from MDPs to zero-sum two-player MGs. Again, as any batch algorithm LSPI will perform its iterations on the Q-function and will take as input a batch of data D n = {(s j , a j , r j , s j )} j=1,...,n where for all j, the reward r j = r(s j , a j ) and where s j ∼ p(.|s j , a j )

Least-Squares Policy Iteration

The LSPI algorithm for MGs [START_REF] Lagoudakis | Value Function Approximation in Zero-Sum Markov Games[END_REF] is a batch version of the algorithm of Pollatschek and Avi-Itzhak algorithm. The first step of the iteration of the algorithm is the greedy part. This part is similar to the LSPI algorithm for MDPs but instead of finding an argmax, it will find a minimax solution. The second step of an iteration of LSPI finds an approximation of the Q-function of the joint policy found at the previous iteration. This approximation of the value function lies in a feature space

F Φ = {Φω, ω ∈ R d } where Φ = [φ 1 , . . . , φ d ] are linearly independent functions from S × A 1 × A 2 → R.
These features can be thought of as vectors of an |S| × |A 1 | × |A 2 |dimensional vector space. We will define ρ as a distribution of the state-action space and ∆ ρ the diagonal application that maps 1 s,a 1 ,a 2 to ρ(s, a 1 , a 2 )1 s,a 1 ,a 2 and finally, the projection on the feature space F Φ with respect to the L ρ,2 is Π ρ,Φ . Instead, LSPI finds ω satisfying the projected fixed point equation:

Φω = Π ρ,Φ B µ,ν Φω. (2.92)
The solution for ω of this equality is given by:

ω = A -1 ρ,Φ,µ,ν b ρ,Φ , (2.93) where A ρ,Φ,µ,ν = Φ ∆ ρ (Φ -γP µ,ν Φ), (2.94)
and b ρ,Φ = Φ∆ ρ r.

(2.95)

Given a batch of data D n = {(s j , a 1 j , a 2 j , r j , s j )} j=1,...,n , the matrix A ρ,Φ,µ,ν and the vector b ρ,Φ can be estimated as follow:

ÂΦ,µ,ν,Dn = 1 n n j=1 Φ(s j , a 1 j , a 2 j )   Φ(s j , a 1 j , a 2 j ) -γ a 1 ,a 2 ∈A 1 ×A 2 µ(a 1 |s j )ν(a 2 |s j )Φ(s j , a 1 , a 2 )   , (2.96) bΦ,Dn = 1 n n j=1 Φ(s j , a 1 j , a 2 j )r(s j , a 1 j , a 2 j ).
(2.97)

If the data in D n are independent and distributed according the distribution ρ, ÂΦ,µ,ν,Dn and bΦ,Dn are consistent estimators of A ρ,Φ,µ,ν and b ρ,Φ . Furthermore, Â-1 Φ,µ,ν,Dn can be be efficiently computed with the Scherman-Morison formula which provides an optimized version of the algorithm. The algorithm in it's most simple version is presented in Algorithm 15.

Algorithm 15 Least Squares Policy Iteration for MGs(LSPI)

Input: A batch D n = {(s j , a 1 j , a 2 j , r j , s j )} j=1,...,n , a feature space Φ, an initial parametrization ω 0 = 0 and a number of iterations K. for k=1,2,...,K do

find µ k , ν k such that µ k ∈ G(Φω k-1 ) and ν k is a policy such that E a 1 ∼µ(.|s), a 2 ∼ν(.|s) [Φω k-1 (a 1 , a 2 , s)] = min a 2 E a 1 ∼µ(.|s) [Φω k-1 (a 1 , a 2 , s)] compute ÂΦ,µ k ,ν k ,Dn
and bΦ,Dn as in formula 2.96 and 2.97. compute ω k = Â-1 Φ,µ k ,ν k ,Dn bΦ,Dn end for Output: µ K

Exact Algorithms for General-Sum Markov-Games

There is a trove of approaches to solve general sum MGs. This section will provide an overview of those methods with an emphasis on independent reinforcement learning methods.

Multi Objective Linear Programming and Homotopy: One way to solve MGs is to generalize the homotopy method. This method was studied widely in the two last decades and is model based [START_REF] Herings | Stationary equilibria in stochastic games: Structure, selection, and computation[END_REF], 2010[START_REF] Borkovsky | A user's guide to solving dynamic stochastic games using the homotopy method[END_REF]. Like many algorithms to solve MGs, it scales poorly with the number of players Chapter 2. Background and Related Work (the homotopy method scales exponentially). Multi Objective Linear Programming was studied in [START_REF] Dermed | Solving stochastic games[END_REF]. These two methods are only tractable for small size problems (as reported in [START_REF] Prasad | Two-Timescale Algorithms for Learning Nash Equilibria in General-Sum Stochastic Games[END_REF])

Rational Learning: Rational learning is guaranteed to converge to a Nash equilibrium [START_REF] Kalai | Rational learning leads to nash equilibrium[END_REF] but is not an independent learning procedure. In a Bayesian manner, players maintain a prior on the strategy of the opponents. This algorithm is only guaranteed to converge in repeated games.

Evolutionary Game Theory: In [START_REF] Akchurina | Multiagent Reinforcement Learning: Algorithm Converging to Nash Equilibrium in General-Sum Discounted Stochastic Games[END_REF], an evolutionary game theoretic approach is generalized to Markov games. This work uses numerical methods to find rest points of the replicator dynamic equation. This procedure is also model-based and hardly scale to large state spaces and large action spaces. Furthermore, no guarantee on the accuracy of the approximation is provided.

Two-timescale algorithms in MGs:

Two-timescale algorithms have been the subject of a wide literature in MDPs that starts from the seminal work of Borkar (1997a). These works mainly analyze the use of linear function approximations. However, when applied independently in a multiagent setting, those algorithms no longer have convergence guarantees (even without function approximation). In [START_REF] Prasad | Two-Timescale Algorithms for Learning Nash Equilibria in General-Sum Stochastic Games[END_REF], the authors attempt to provide an on-line model-free algorithm with guarantees. This approach requires each player to observe the rewards of the others. The authors claim that their algorithm is guaranteed to converge to a Nash equilibrium but the proof of this algorithm is broken (details in App. 14).

Sampling Based Algorithms in MSGs and Conterfactual Regret Minimization Algorithms (CFR):

A trove variety of algorithms exists to compute Nash equilibria in MSGs. A review of those algorithms can be found in [START_REF] Bošanský | Algorithms for computing strategies in two-player simultaneous move games[END_REF]. Those algorithms require that one can query the model at each stage of the game but they do not belong to the family of independent RL methods. Another family of algorithms related to our work are the CFR algorithms and their wide number of variations. Even if these algorithms have sample-based variations [START_REF] Lanctot | Monte carlo sampling for regret minimization in extensive games[END_REF], none of those variations are based on independent learning rules.

Independent Reinforcement Learning for Markov Games

Several major issues prevent direct use of standard RL algorithms with multi-agent systems. First, blindly applying single agent RL in a decentralized fashion implies that, from each agent's point of view, the other agents are part of the environment. Such an hypothesis breaks a crucial RL assumption that the environment is (at least almost) stationary [START_REF] Laurent | The world of Independent learners is not Markovian[END_REF]. Second, it introduces partial observability as each agent's knowledge is restricted to its own actions and rewards while its behavior should depend on others' strategies.

Independent reinforcement learning in games has been studied widely in the case of normal form games and include regret minimization approaches [START_REF] Bubeck | Regret analysis of stochastic and nonstochastic multiarmed bandit problems[END_REF][START_REF] Bubeck | Regret analysis of stochastic and nonstochastic multiarmed bandit problems[END_REF][START_REF] Cesa-Bianchi | Prediction, Learning, and Games[END_REF] or stochastic approximation algorithms [START_REF] Leslie | Generalised weakened fictitious play[END_REF]. However, to our knowledge, none of the previous methods have been extended to independent reinforcement learning in Markov Games or any intermediate models such as MSGs with guarantees of convergence both for cooperative and zero-sum cases. Addressing both cases are still treated as separate agendas since the seminal paper by [START_REF] Shoham | If multi-agent learning is the answer, what is the question[END_REF].

Q-Learning Like Algorithms

On-line algorithms like Q-learning [START_REF] Watkins | Q-learning[END_REF] are often used in cooperative multi-agent learning environments but fail to learn a stationary strategy in simultaneous zero-sum two-player games. They fail in this setting because, in simultaneous zero-sum two-player games, it is not sufficient to use a greedy strategy to learn a Nash equilibrium. In [START_REF] Littman | Markov Games as a Framework for Multi-Agent Reinforcement Learning[END_REF], the Q-learning method is adapted to guarantee convergence to zero-sum two-player MGs. Other adaptations to N -player games were developed [START_REF] Hu | Nash Q-Learning for General-Sum Stochastic Games[END_REF]Wellman, 2003, Greenwald et al., 2003). However, all these methods require the observation of the opponents' action and the two last ones are guaranteed to converge only under very conservative hypotheses. Moreover, these are not independent methods as each player needs to observe the rewards of the others.

Independent Policy Gradient

In MGs and in NFGs, previous work on decentralized learning includes the study of policy hill climbing [START_REF] Bowling | Rational and Convergent Learning in Stochastic Games[END_REF]. In this approach, each agent follows a gradient ascent on his expected outcome. This method enjoys limited guarantees and fails to converge in zero-sum games. The behavior of this algorithm can be improved using heuristics on the learning rate as reported in [START_REF] Bowling | Rational and Convergent Learning in Stochastic Games[END_REF]. It can also scale up using function approximation as in [START_REF] Banerjee | Adaptive policy gradient in multiagent learning[END_REF] which results in an actor-critic like method and fits in our setting.

Fictitious Play

Fictitious play is a model-based process that learns Nash equilibria in some subclass normal form games. It has been widely studied and required assumptions were weakened over time [START_REF] Leslie | Generalised weakened fictitious play[END_REF]Collins, 2006, Hofbauer andSandholm, 2002) since the original article by [START_REF] Robinson | An iterative method of solving a game[END_REF]. It has been extended to extensive form games (game trees) and, to a lesser extent, to function approximation [START_REF] Heinrich | Fictitious self-play in extensive-form games[END_REF]. However, theoretical work on this process is neither on-line nor decentralized except from [START_REF] Leslie | Generalised weakened fictitious play[END_REF] work which focus on NFGs. Fictitious play enjoys several convergence guarantees (Hofbauer and Sandholm, 2002) which makes it a good candidate for learning in simultaneous multi-stage games. The original fictitious play creates a sequence of policies π n . At each iteration n, players play their best response against the average policy played by the opponent until iteration n. This average policy per players evolve as follows:

π i n+1 = 1 - 1 n π i n+1 + 1 n B r i π -i n Where B r i π -i n
is a greedy policy of with respect to the reward r i

π -i n (formally we have B r i π -i n ∈ G(r i π -i n )).
This process is often studied as a continuous time process formalized as a differential inclusion. This continuous time process follows the following differential inclusion equation:

πi t ∈ G r i π -i t -π i t (2.98)
The process defined in Equation (2.99) has been weakened to allow the use of approximate best responses. In Hofbauer and Sandholm ( 2002), the approximate best response is defined in a generalized manner but can be thought as a softmax over the reward function. The differential inclusion defined above becomes a differential equation:

πi t ∈ B σ r i π -i t -π i t (2.99)
This process will be generalized to multi stage games in Chapter 8.

Part II

Approximate Dynamic Programming in Zero-Sum Two-Player Markov Games

Chapter 3

Approximate Dynamic Programming in Games : A Unified Analysis

This chapter details the contributions of [START_REF] Perolat | Approximate Dynamic Programming for Two-Player Zero-Sum Markov Games[END_REF]. The first part of this chapter reports a unified analysis of several approximate dynamic programming algorithms for MGs comparable to the one of section 1.2.3 on MDP. The second part of this chapter proposes a batch algorithm based on generalized policy iteration (Section 3.4.4). First we briefly recall the VI algorithm, the PI algorithm and the GPI algorithm. Then we present the scheme of approximation and analyse it. Finally, we propose our novel Batch algorithm for MGs and report its empirical evaluation.

Approximate Dynamic Programming: A Unified Scheme

The three algorithms we intend to analyse are VI, PI and GPI. Each iteration of these algorithms can be decomposed in two steps. The greedy step is common to all of them and the evaluation step implements a different strategy for each of them. Each iteration of the VI algorithm can be written as follows:

v k+1 = T v k .
This can equivalently be written in two steps as follows:

µ k+1 = G(v k ), v k+1 = T µ k+1 v k .
This two-step decomposition of these algorithms is straightforward for PI and GPI. Indeed, each iterations of PI can be written as:

µ k+1 = G(v k ), v k+1 = v µ k+1 = (T µ k+1 ) +∞ v k .
Finally, GPI bridges the gap between PI and VI:

µ k+1 = G(v k ), v k+1 = (T µ k+1 ) m v k .
It is clear that GPI generalizes VI and PI. In particular, an error propagation bound for VI (when m = 1) and PI (when m = ∞) will follow from the analysis we shall provide for GPI. Thus, only an approximate version of GPI will be analysed. This analysis will give insights on the VI and PI algorithms as they are limit cases.

Remark 3.1. In turn-based games, each state is controlled by a single player. In the zero-sum two-player MGs framework, they can be seen as a game where ∀s ∈ S, card(A 1 (s)) = 1 or card(A 2 (s)) = 1. In this special case, optimal strategies are deterministic (Hansen et al., 2013b). Furthermore, and as we already mentioned, the greedy step (see definition in Section 2) reduces to finding a maximum rather than a minimax equilibrium and is thus significantly simpler. Furthermore, one can see an MDP as a zero-sum two-player MG where one player has no influence on both the reward and the dynamics. Therefore, our analysis should be consistent with previous MDP analyses.

Approximate Generalized Policy Iteration

As its exact counterpart, Approximate Generalized Policy Iteration (AGPI) proceeds in two steps (Algorithm 16). Unlike GPI, each steps of AGPI accounts for errors in each step of the iteration. The greedy step can select a strategy that is not a minimax strategy. At iteration k, this strategy can be k suboptimal and we shall write

µ k ← Ĝ k (v k-1 ) for: T v k-1 ≤ T µ k v k-1 + k (3.1) or, ∀µ T µ v k-1 ≤ T µ k v k-1 + k . (3.2)
In other words, the strategy µ is not necessarily the best strategy, but it must be guaranteed to be away from the best strategy in the worst case. At iteration k the evaluation step may have an additive error k :

v k = (T µ k ) m v k-1 + k . (3.3)
In the evaluation step the strategy µ k is fixed and the operator

T µ k • = min ν T µ k ,ν • is applied m times.
From the point of view of the minimizer, this step finds an optimal value for a γ-discounted m-horizon problem.

Error Propagation

The goal of our analysis is to report the impact of previous errors { i } i∈{1,...,k-1} and { i } i∈{1,...,k} on the current strategy µ k . Thus, we are interested in bounding the difference

l k = v * -v µ k ≥ 0,

Algorithm 16 Approximate Generalized Policy Iteration

Input: An MDP M , a value v 0 = 0, a maximum number of iterations K and a parameter m. for k=1,2,...,K do find

µ k ∈ Ĝ k (v k-1 ) compute v k = (T µ k ) m v k-1 + k end for Output: µ K
where v * is the minimax value of the game (obtained when both players play the Nash equilibrium µ * and ν * ) and where v µ k is the value when the maximizer plays µ k and the minimizer plays the optimal counter-strategy against µ k . This is a natural measure of quality for the strategy µ k that would be output by the approximate algorithm.

The following elements will be key properties in our analysis. By definition, we have:

∀ν, ∀v, T µ v ≤ T µ,ν v.
(3.4)

In addition, we shall consider a few notations. The minimizer policies ν i k , νk , νk and ν k are policies that respectively satisfy:

(T µ k ) i+1 v k-1 = T µ k ,ν i k ...T µ k ,ν 1 k T µ k v k-1 . (3.5)
The strategy ν i k is an argmin strategy of Player 2 when Player 1 plays the strategy µ k with respect to the value T µ k ) i v k-1 . The strategy νk is is an argmin strategy for Player 2 when Player 1 chooses the strategy µ * with respect to the value v k :

T µ * v k = T µ * ,ν k v k .
(3.6)

The strategy νk (respectively ν k ) is also an argmin strategy for Player 2 when Player 1 chooses the strategy µ k with respect to the value v k (respectively v µ k ).

T µ k v k = T µ k ,ν k v k , (3.7) T µ k v µ k = T µ k ,ν k v µ k . (3.8)
In order to bound l k , we will study the following quantities similar to those introduced by Scherrer et al. ( 2012) (recall that T µ and T µ,ν are defined in Section 3.3 and the stochastic kernel is defined in Section 2.62):

d k = v * -(T µ k ) m v k-1 = v * -(v k -k ), (3.9) s k = (T µ k ) m v k-1 -v µ k = (v k -k ) -v µ k , (3.10) b k = v k -T µ k+1 v k , (3.11) x k = (I -γP µ k ,ν k ) k + k+1 , (3.12) y k = -γP µ * ,ν k k + k+1 . (3.13) Notice that l k = d k + s k .
We shall prove the following relations, similar to the one proved in Scherrer et al. (2012).

Lemma 3.1. The following linear relations hold:

b k ≤ γP µ k ,ν k γP µ k ,ν m-1 k ...γP µ k ,ν 1 k b k-1 + x k , (3.14) d k+1 ≤ γP µ * ,ν k d k + y k + m-1 j=1 γP µ k ,ν j k ...γP µ k ,ν 1 k b k , (3.15) s k ≤ (γP µ k ,ν k ) m ( ∞ i=1 γP µ k ,ν i k ...γP µ k ,ν 1 k b k-1 ). (3.16)
Contrary to the analysis of Scherrer et al. (2012) for MDPs in which the operator T µ is affine, it is in our case non-linear. The proof that we now develop is thus more technical.

Proof. Let us start with b k :

b k = v k -T µ k+1 v k , = v k -T µ k v k + T µ k v k -T µ k+1 v k . From equation (3.2) we have T µ k v k ≤ T µ k+1 v k + k+1 (since µ k+1 ∈ Ĝ k+1 (v k )) then: b k ≤ v k -T µ k v k + k+1 , = v k -k -T µ k v k =T µ k ,ν k v k +γP µ k ,ν k k + k -γP µ k ,ν k k + k+1 , = v k -k -T µ k ,ν k (v k -k ) T µ k ,ν k is affine +(I -γP µ k ,ν k ) k + k+1 . From Equation (3.3), we have v k -k = (T µ k ) m v k-1 . Thus, b k ≤ (T µ k ) m v k-1 -T µ k ,ν k (T µ k ) m v k-1 + x k (Eq. (3.5)) , = (T µ k ) m v k-1 -T µ k ,ν k T µ k ,ν m-1 k ...T µ k ,ν 1 k (T µ k v k-1 ) + x k (Eq. (3.5)), ≤ T µ k ,ν k T µ k ,ν m-1 k ...T µ k ,ν 1 k v k-1 -T µ k ,ν k T µ k ,ν m-1 k ...T µ k ,ν 1 k (T µ k v k-1 ) + x k (Eq. (3.4)), = γP µ k ,ν k γP µ k ,ν m-1 k ...γP µ k ,ν 1 k (v k-1 -T µ k v k-1 ) + x k , ≤ γP µ k ,ν k γP µ k ,ν m-1 k ...γP µ k ,ν 1 k b k-1 + x k .
To bound d k+1 , we decompose it in the three following terms:

d k+1 = v * -(T µ k+1 ) m v k , = T µ * v * -T µ * v k 1 + T µ * v k -T µ k+1 v k 2 + T µ k+1 v k -(T µ k+1 ) m v k
In this equation, term 1 can be upper-bounded as follows:

T µ * v * -T µ * v k = T µ * v * -T µ * ,ν k v k with νk defined in Eq. (3.6), ≤ T µ * ,ν k v * -T µ * ,ν k v k since ∀ν, T µ * . ≤ T µ * ,ν . = γP µ * ,ν k (v * -v k ).
By definition 2 is bounded by the greedy error:

T µ * v k -T µ k+1 v k ≤ k+1 (3.2) with µ ← µ * , k ← k + 1
Finally, bounding term 3 involves the b k quantity:

T µ k+1 v k -(T µ k+1 ) m v k , = m-1 j=1 (T µ k+1 ) j v k -(T µ k+1 ) j+1 v k , = m-1 j=1 (T µ k+1 ) j v k -T µ k+1 ,ν j k+1 ...T µ k+1 ,ν 1 k+1 T µ k +1 v k , ≤ m-1 j=1 [T µ k+1 ,ν j k+1 ...T µ k+1 ,ν 1 k+1 v k -T µ k+1 ,ν j k+1 ...T µ k+1 ,ν 1 k+1 T µ k +1 v k ] see (3.4), = m-1 j=1 γP µ k+1 ,ν j k+1 ...γP µ k+1 ,ν 1 k+1 (v k -T µ k +1 v k ), = m-1 j=1 γP µ k+1 ,ν j k+1 ...γP µ k+1 ,ν 1 k+1 b k .
Then d k+1 becomes:

d k+1 ≤ γP µ * ,ν k (v * -v k ) + k+1 + m-1 j=1 γP µ k+1 ,ν j k+1 ...γP µ k+1 ,ν 1 k+1 b k , ≤ γP µ * ,ν k (v * -v k ) + γP µ * ,ν k k -P µ * ,ν k k + k+1 + m-1 j=1 γP µ k+1 ,ν j k+1 ...γP µ k+1 ,ν 1 k+1 b k , ≤ γP µ * ,ν k (v * -( v k -k (Tµ k ) m v k-1 )) -P µ * ,ν k k + k+1 y k + m-1 j=1 γP µ k+1 ,ν j k+1 ...γP µ k+1 ,ν 1 k+1 b k , ≤ γP µ * ,ν k d k + y k + m-1 j=1 γP µ k+1 ,ν j k+1 ...γP µ k+1 ,ν 1 k+1 b k .

Chapter 3. Approximate Dynamic Programming in Games

Let us finally bound s k :

s k = (T µ k ) m v k-1 -v µ k , = (T µ k ) m v k-1 -(T µ k ) ∞ v k-1 , = (T µ k ) m v k-1 -(T µ k ) m (T µ k ) ∞ v k-1 , = (T µ k ) m v k-1 -(T µ k ,ν k ) m (T µ k ) ∞ v k-1 , with ν k defined in (3.8) ≤ (T µ k ,ν k ) m v k-1 -(T µ k ,ν k ) m (T µ k ) ∞ v k-1 since (3.4), = (γP µ k ,ν k ) m (v k-1 -(T µ k ) ∞ v k-1 ), = (γP µ k ,ν k ) m ( ∞ i=0 (T µ k ) i v k-1 -(T µ k ) i (T µ k v k-1 )), ≤ (γP µ k ,ν k ) m ∞ i=0 γP µ k ,ν i k ...γP µ k ,ν 1 k (v k-1 -T µ k v k-1 ), ≤ (γP µ k ,ν k ) m ∞ i=0 γP µ k ,ν i k ...γP µ k ,ν 1 k b k-1 .
From linear recursive relations of the kind of Lemma 3.1, Scherrer et al. (2012) show how to deduce a bound on the L p -norm of l k . This part of the proof being identical to that of Scherrer et al. (2012), we do not develop it here. For completeness however, we include it in Appendix 4.1 of this chapter.

Theorem 3.1. Let ρ and σ be distributions over states. Let p, q and q' be such that 1 q + 1 q = 1. Then, after k iterations, we have:

l k p,ρ ≤ 2(γ -γ k )(C 1,k,0 q ) 1 p (1 -γ) 2 sup 1≤j≤k-1 j pq ,σ value update error + (1 -γ k )(C 0,k,0 q ) 1 p (1 -γ) 2 sup 1≤j≤k j pq ,σ greedy error , + 2γ k 1 -γ (C k,k+1,0 q ) 1 p min( d 0 pq ,σ , b 0 pq ,σ ) contraction term .
where

C l,k,d q = (1 -γ) 2 γ l -γ k k-1 i=l ∞ j=i γ j c q (j + d), (3.17)
with the following norm of a Radon-Nikodym derivative:

c q (j) = sup µ 1 ,ν 1 ,...,µ j ,ν j d(ρP µ 1 ,ν 1 ...P µ j ,ν j ) dσ q,σ . (3.18)
This bound is again similar to the one of Section 1.2.3. It involves three terms similar to the one of theorem 2.3. The value update error, the greedy error and the concentration are comparable to the previous terms of theorem 2.3. The only major difference is the definition of the concentrability coeficient which generalizes the one for MDPs to zero-sum two-player MGs. The bound is consistent with previous analysis on MDPs (Section 1.2.3). Furthermore, if player 2 has no influence on the MG, the bound of theorem 3.1 is the one theorem 2.3. In addition, when p tends to infinity, the bound becomes:

lim inf k→+∞ l k ∞ ≤ 2γ (1 -γ) 2 + 1 (1 -γ) 2 ,
where and are respectively the sup of errors at the evaluation step and the sup of errors at the greedy step in ∞-norm. We thus recover the bounds computed by [START_REF] Patek | Stochastic Shortest Path Games: Theory and Algorithms[END_REF].

Remark 3.2. In the case of an MG with no discount factor but with an absorbing state the expression given in lemma 3.1 is still valid. Instead of having a γ-discounted transition kernel we would have a simple transition kernel. And if this transition kernel has the following property ∃l, sup

µ 0 ,ν 0 ,...,µ l ,ν l | l i=0 P µ i ,ν i ∞ ≤ γ < 1,
we could still have an upper bounds on the propagation of errors.

Empirical Evaluation

The analysis of error propagation presented in Section 1.2 is general enough to develop several implementations. From the moment one can control the error made at each iteration step, the bound presented in Theorem 3.1 applies.

Algorithm

In this section, we present the Approximate Generalized Policy Iteration-Q (AGPI-Q) algorithm which is an extension to MG of Fitted-Q. This algorithm is offline and uses the so-called state-action value function Q. The state-action value function extends the value function by adding two degrees of freedom for the first action of each player. More formally, the state-action value function

Q µ,ν (s, a, b) is defined as Q µ,ν (s, a 1 , a 2 ) = E[r(s, a 1 , a 2 )] + s ∈S p(s |s, a 1 , a 2 )v µ,ν (s ).
We assume we are given some samples ((s j , a 1 j , a 2 j ), r j , s j ) j=1,...,N and an initial Qfunction (here we chose the null function). As it is an instance of AGPI, each iteration of this algorithm is made of a greedy step and an estimation step. The algorithm is precisely described in Algorithm 17.

For the greedy step, the minimax policy for the maximizer on each matrix game defined by (q k (s j , a 1 , a 2 )) a 1 ,a 2 . In general, this step involves solving N linear programs; recall that in the case of a turn-based game this step reduces to finding a maximum.

Algorithm 17 AGPI -Q for Batch sample

Input: ((s j , a 1 j , a 2 j ), r j , x j ) j=1,...,N some samples, q 0 = 0 a Q-function, F an hypothesis space for k=1,2,...,K do Greedy step: for all j do ā1 j = argmax ā minb q k-1 (s j , ā, b) (solving a matrix game) end for Evaluation step: q k,0 = q k-1 for i=1,...,m do for all j do

q j = r(s j , a 1 j , a 2 j ) + γ min b q k,i-1 (s j , ā1 j , b) end for q k,i = argmin q∈F N j=1 l(q(s j , a 1 j , a 2 j ), q j ) Where l is a loss function. q k = q k,m end for end for
The evaluation step involves solving the MDP with an horizon m for the minimizer. This part is similar to fitted-Q iteration. At each step, we try to find the best fit over our hypothesis space for the next Q-function according to some loss function l(x, y) (often, l(x, y) = |x -y| 2 ).

Analysis

For this algorithm, we have k = 0 and k the error made on q k at each iteration. Let us note k,i the error for fitting the Q-function on the feature space.

We have

q k,i+1 = B µ k q k,i + i . Let us define ν k,i such as B m µ k q k,0 = B µ k ,ν k,m-1 ...B µ k ,ν k,0 q k,0 . Furthermore we have q k,i+1 ≤ B µ k ,ν k,i q k,i + k,i .
On the one hand, we have:

k = q k,m -B m µ k q k,0 , (3.19) ≤ B µ k ,ν k,m-1 ...B µ k ,ν k,0 q k,0 + m-1 i=0 P µ k ,ν k,m-1 ...P µ k ,ν k,i+1 k,i -B µ k ,ν k,m-1 ...B µ k ,ν k,0 q k,0 , (3.20) ≤ m-1 i=0 P µ k ,ν k,m-1 ...P µ k ,ν k,i+1 k,i . (3.21)
On the other hand (with νk,i as q k,i+1 = B µ k ,ν k,i q k,i + k,i ), we have:

k = q k,m -B m µ k q k,0 , (3.22) ≥ B µ k ,ν k,m-1 ...B µ k ,ν k,0 q k,0 + m-1 i=0 P µ k ,ν k,m-1 ...P µ k ,ν k,i+1 k,i -B µ k ,ν k,m-1 ...B µ k ,ν k,0 q k,0 , (3.23) ≥ m-1 i=0 P µ k ,ν k,m-1 ...P µ k ,ν k,i+1 k,i . (3.24)
From these inequalities, we can provide the following bound (the proof is given in Appendix 5):

l k p,ρ ≤ 2(γ -γ k )(1 -γ m ) (1 -γ) 3 (C 1,k,0,m,0 q ) 1 p sup i,l i,l pq ,σ (3.25) + 2γ k 1 -γ (C k,k+1,0 q ) 1 p min( d 0 pq ,σ , b 0 pq ,σ ), (3.26) with C l,k,l ,k ,d q = (3.27) (1 -γ) 3 (γ l -γ k )(γ l -γ k ) k-1 i=l k -1 i =l ∞ j=i+i γ j c q (j + d).
(3.28)

Complexity analysis

At the greedy step, the algorithm solves N minimax equilibria for a zero-sum matrix game. This is usually done by linear programming (Section 2.2). For state s, the complexity of such an operation is the complexity of solving a linear program with c s = 1 + card(A 1 (s)) + card(A 2 (s)) constrains and with card(A 1 (s)) variables. Let us note L(c s , card(A 1 (s))) this complexity. Then, the complexity of this step is bounded by N L(c, a) (with c = sup

s∈{x 1 ,...,x N } c s and a = sup s∈{x 1 ,...,x N } card(A 1 (s)))
. Using the simplex method, L(c, a) may grow exponentially with c while with the interior point method, L(c, a) is O(a 3.5 ) [START_REF] Karmarkar | A New Polynomial-time Algorithm for Linear Programming[END_REF]. The time to compute q j in the evaluation step depends on finding a maximum over A 2 (x j ). And the regression complexity to find q k,i depends on the regression technique. Let us note this complexity R(N ). Finally, the complexity of this step is mR(N ).

The overall complexity of an iteration is thus O(N L(c, a) + mR(N )); in general, the complexity of solving the linear program will be the limiting factor for games with a large amount of actions.

The Game of Alesia

Alesia [START_REF] Perolat | Approximate Dynamic Programming for Two-Player Zero-Sum Markov Games[END_REF][START_REF] Meyer | Learning Strategies in Games by Anticipation[END_REF], which resembles the Oshi-Zumo game in [START_REF] Buro | Solving the Oshi-Zumo Game[END_REF][START_REF] Bošanský | Algorithms for computing strategies in two-player simultaneous move games[END_REF] (meaning "the pushing sumo"), is a two-player board game where each player starts with N coins. They are positioned in the middle of a board with 2K + 1 different positions (see Figure 3.1). At each round, each player chooses secretly a certain amount of coins to bid (let's say (a 0 , a 1 )). If the player's budget is not null, he has to bet at least one coin. If player 1's bid is larger (respectively smaller) than the one of player 2, player 1 (respectively player 1) moves toward his side of the board. If the bids are equal, then players will stay on their current position. In all cases, all bets is discounted from the budget. This process continues until players have no coins left or until one player reaches one side of the board. The final position determines the winner. The game ends with a draw if no one succeeded to reach his side of the board. If player 2 (respectively player 1) reaches his side of the board, the reward is (-1, 1) (respectively (1, -1)).

5 4 3 2 1 5 4 3 2 1 (N ,N ) (N -a 0 , N -a 1 ) a 0 < a 1 Figure 3.1 -Alesia rules for K = 2.
We perform the evaluation of AGPI-Q on the Alesia game described above. We assume that both players start with a budget N = 20 et K = 2. As a baseline, we use the exact solution of the problem provided by VI. We have run the algorithm for k = 10 iterations and for m ∈ {1, 2, 3, 4, 5} evaluation steps. We have considered different sample set sizes, n = 2500, 5000, 10000. Each experiment is repeated 20 times. First, we generate n uniform samples (s j ) over the state space. Then, for each state, we draw uniformly the actions of each player in the set of their own action space in that state a 1 j , a 2 j , r j = r(s j , a 1 j , a 2 j ) and compute the next state s j . As hypothesis space, we use CART trees [START_REF] Breiman | Classification and Regression Trees[END_REF] which exemplifies the non-parametric property of the algorithm.

The performance of the algorithm is measured as the mean-squared error between the value function v k (s) = min b max a q k (s, a, b) where q k is the output of the algorithm AGPI-Q and the actual value function computed via VI. Figure 3.2 shows the evolution of performance along iterations for n = 10000 for the different values of the parameter m. Figure 3.3 shows the exact value function (Fig. 3.3(a)) and the approximated one v k (Fig. 3.3(b)). The complete list of experiments results can be found in the appendix, especially for different size of sample set n = 2500 and n = 5000.

For each size of sample set, the asymptotic convergence is better for small values of m. This conforms to Eq. (3.26), in which the term

2(γ-γ k )(1-γ m ) (1-γ) 3
increases with m. However, for small values of k, the mean-squared error is reducing when m is increasing. This is consistent with experimental results when using MPI for MDP: the bigger m, the higher the convergence rate. The price to pay for this acceleration of convergence towards the optimal value is an heavier evaluation step. This is similar to results in

Conclusion and Perspectives

53

the exact case [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF]. Overall, this suggests using large values of m at the beginning of the algorithm and reducing the values of m as k grows to get a smaller asymptotic error.

Conclusion and Perspectives

This work provides a novel and unified error propagation analysis in L p -norm of wellknown algorithms (API, AVI and AGPI) for zero-sum two-player MGs. It extends the error propagation analyses of Scherrer et al. (2012) for MDPs to zero-sum two-player MGs and of [START_REF] Patek | Stochastic Shortest Path Games: Theory and Algorithms[END_REF] which is an L ∞ -norm analysis for only API. In addition, we provide a practical algorithm (AGPI-Q) which learns a good approximation of the Nash Equilibrium from batch data provided in the form of transitions sampled from actual games (the dynamics is not known). This algorithm is an extension of Fitted-Q for zero-sum two-player MGs and can thus be non-parametric. No features need to be provided or hand-crafted for each different application which is a significant advantage. It appears that the provided bound is highly sensitive to γ (which is a common problem of ADP). This is critical and further work should concentrate on reducing the impact of γ in the final error bound. One solution explored in the next chapter is the use of non stationary strategies as they reduce the γ-sensitivity in the case of MDPs.

Appendix: Demonstration of Lemma 3.2

The following proof is similar to the one for MDP Scherrer et al. (2012) we write it for the sake of completeness. Furthermore, equation (3.34) is the starting point of the proof of (3.26).

We will use the same abusive but simplifying notation as in (Scherrer et al., 2012). We will note Γ n any product of n discounted transition Kernel. Then Γ n represents the set {γP µ 1 ,ν 1 , ... , γP µn,νn , with µ i , ν i random strategies}. Then Γ n is the represent a class of discounted stochastic matrix. One should read Γ n as there exists γP 1 , ..., γP n such as Γ n represents the product γP 1 ...γP n . For example we have the following property:

α 1 Γ i α 2 Γ j + α 3 Γ k = α 1 α 2 Γ i+j + α 3 Γ k .
We can rewrite Lemma 3.1 in this Simple way:

b k ≤ Γ m b k-1 + x k , (3.29) d k+1 ≤ Γd k + y k + m-1 j=1 Γ j b k , (3.30) s k ≤ Γ m ( ∞ i=1 Γ i b k-1 ). (3.31)
We prove these three inequalities:

b k ≤ k i=1 Γ m(k-i) x k + Γ mk b 0 d k ≤ k i=1 Γ i-1 y k-i + k-1 i=1 mi-1 j=i Γ j x k-i + mk-1 i=k Γ i b 0 + Γ k d 0 z k s k ≤ k-1 i=1 ∞ j=mi Γ j+m(k-i) x i + ( ∞ j=mk Γ j )b 0 z k
First inequality. After expanding (3.29) we get:

b k ≤ k i=1 Γ m(k-i) x k + Γ mk b 0 (3.32)

Chapter 3. Approximate Dynamic Programming in Games

Second inequality :

d k ≤ Γ k d 0 + k-1 j=0 Γ k-1-j (y j + ( m-1 l=1 Γ l )b j ) ≤ Γ k d 0 + k-1 j=0 Γ k-1-j [y j + ( m-1 l=1 Γ l )( j i=1 Γ m(j-i) x i + Γ mj b 0 )] = k i=1 Γ i-1 y k-i + k-1 j=0 m-1 l=1 j i=1 Γ k-1-j+l+m(j-i) x i + z k = k i=1 Γ i-1 y k-i + k-1 i=1 k-1 j=i m-1 l=1 Γ k-1+l+j(m-1)-mi x i + z k = k i=1 Γ i-1 y k-i + k-1 i=1 m(k-i)-1 j=k-i Γ j x i + z k = k i=1 Γ i-1 y k-i + k-1 i=1 mi-1 j=i Γ j x k-i + z k With z k : z k = k-1 j=0 m-1 l=1 Γ k-1-j+l+mj b 0 + Γ k d 0 = mk-1 i=k Γ i b 0 + Γ k d 0
last inequality (we replace b k-1 with it's expression (3.32)):

s k ≤ k-1 i=1 ∞ j=mi Γ j+m(k-i) x i + ( ∞ j=mk Γ j )b 0 z k And finally bounding l k l k = d k + s k (3.33) ≤ k i=1 Γ i-1 y k-i + k-1 i=1 ∞ j=i Γ j x k-i Depend on errors + z k + z k Depend on initial condition (3.34) η k = z k + z k = ∞ i=k Γ i b 0 + Γ k d 0
Then we prove the following lemma which is similar to that for MDP given by Scherrer et al. (2012):

4. Appendix: Demonstration of Lemma 3.2 57 Lemma 3.2. ∀k ≥ 1 |l k | ≤2 k-1 i=1 ∞ j=i Γ j | k-i | + k-1 i=0 ∞ j=i Γ j k-i + h(k), (3.35) with h(k) = 2 ∞ i=k Γ i |b 0 | or 2 ∞ i=k Γ i |d 0 | . (3.36)
Relation between b 0 and d 0 :

b 0 = v 0 -T µ 1 v 0 = v 0 -v * + T µ * v * -T µ * v 0 + T µ * v 0 -T µ 1 v 0 ≤ -d 0 + γP µ * ,ν 0 d 0 + 1 ≤ (I -γP µ * ,ν 0 )(-d 0 ) + 1 d 0 ≤ (I -γP µ * ,ν 0 ) -1 ( 1 -b 0 )
One should notice that in a discrete state space P µ,ν is a stochastic matrix. Then I -γP µ * ,ν 0 is invertible since γP µ * ,ν 0 has a spectral radius < 1.

then:

|η k | ≤ ∞ i=k Γ i ((I -γP µ * ,ν 0 ) |d 0 | + | 1 |) + Γ k |d 0 | ≤ ∞ i=k Γ i ((I + Γ) |d 0 | + | 1 |) + Γ k |d 0 | ≤ 2 ∞ i=k Γ i |d 0 | + ∞ i=k Γ i | 1 |
and also:

|η k | ≤ ∞ i=k Γ i |b 0 | + Γ k (I -γP µ * ,ν 0 ) -1 (| 1 | + |b 0 |) ≤ ∞ i=k Γ i |b 0 | + Γ k ∞ i=0 Γ(| 1 | + |b 0 |) ≤ 2 ∞ i=k Γ i |b 0 | + ∞ i=k Γ i | 1 |
because l k is positive we have then:

|l k | ≤ k i=1 Γ i-1 |y k-i | + k-1 i=1 ∞ j=i Γ j |x k-i | + |η k | ≤ k i=1 Γ i-1 (Γ | k-i | + k+1-i ) + k-1 i=1 ∞ j=i Γ j ((I + Γ) | k-i | + k+1-i ) + |η k | note that 0 = 0 ≤ k-1 i=1 (Γ i + ∞ j=i (Γ j + Γ j+1 )) | k-i | + k-1 i=1 (Γ i-1 + ∞ j=i Γ j ) k+1-i + Γ k-1 | 1 | + ∞ i=k Γ i | 1 | + h(k) ≤ 2 k-1 i=1 ∞ j=i Γ j | k-i | + k-1 i=0 ∞ j=i Γ j k-i + h(k) with h(k) = 2 ∞ i=k Γ i |b 0 | or 2 ∞ i=k Γ i |d 0 | (3.37)

Demonstration of Theorem 3.1

Let us recall Scherrer's lemma (demonstration can be found in Scherrer et al. (2012)).

Lemma 3.3. Let I and (J i ) i∈I be a sets of positive integers, {I 1 , ... , I n } a partition of I. Let f and (g i ) i∈I be function such as:

|f | ≤ i∈I j∈J i Γ j |g i | = n l=1 i∈I l j∈J i Γ j |g i | .
Then for all p, q and q such as 1 q + 1 q = 1 and for all distribution ρ and σ we have

f p,ρ ≤ n l=1 (C q (l)) 1 p sup i∈I l g i pq ,σ i∈I l j∈J i γ j .
with the concentrability coefficient written:

C q (l) = i∈I l j∈J i γ j c q (j) i∈I l j∈J i γ j .
Theorem 3.1 can be proven by applying lemma 4.2 with:

I = {1, ..., 2k} I = {I 1 , I 2 , I 3 }, I 1 = {1, 2, ..., k -1}, I 2 = {k, ..., 2k -1}, I 3 = {2k} 4. Appendix: Demonstration of Lemma 3.2 59 ∀i ∈ I 1 g i = 2 k-i J i = {i, i + 1, ...} ∀i ∈ I 2 g i = k-(i-k) J i = {i -k, i -k + 1, ...} ∀i ∈ I 3 g i = 2d 0 or 2b 0 J i = {k, k + 1, ...}
5 Appendix: Bound for AGPI-Q From (3.34) and with x k = (I -Γ) k and y k = -Γ k we can compute:

l k = d k + s k (3.38) ≤ k i=1 Γ i-1 Γ(-k-i ) + k-1 i=1 ∞ j=i Γ j (I -Γ) k-i + z k + z k (3.39) ≤ k i=1 Γ i-1 Γ m-1 l=0 Γ m-l-1 (-k-i,l ) + k-1 i=1 ∞ j=i Γ j (I -Γ) m-1 l=0 Γ m-l-1 k-i,l + z k + z k (3.40)
In (3.39) we use (3.24) to boundk-i and (3.21) to bound k-i Since l k ≥ 0:

|l k | ≤ k i=1 Γ i m-1 l=0 Γ m-l-1 | k-i,l | + k-1 i=1 ∞ j=i Γ j (I + Γ) m-1 l=0 Γ m-l-1 | k-i,l | + h(k) with 0,l = 0 (3.41) ≤ 2 k-1 i=1 ∞ j=i Γ j m-1 l=0 Γ l | k-i,m-l-1 | + h(k) (3.42) ≤ 2 k-1 i=1 m-1 l=0 ∞ j=i+l Γ j | k-i,m-l-1 | + h(k) (3.43)
with lemma 4.2 applied to:

I = {1, ..., (k -1)m + 1} I = {I 1 , I 2 }, I 1 = {1, 2, ..., (k -1)m}, I 2 = {(k -1)m + 1} ∀ξ ∈ I 1 , ξ = i + (k -1)l g ξ = 2 k-i,m-l-1 J ξ = {i + l, i + l + 1, ...} ∀i ∈ I 2 g i = 2d 0 or 2b 0 J i = {k, k + 1, ...}
The result is:

l k p,ρ ≤ 2(γ -γ k )(1 -γ m ) (1 -γ) 3 (C 1,k,0,m,0 q ) 1 p sup i,l i,l pq ,σ + 2γ k 1 -γ (C k,k+1,0 q ) 1 p min( d 0 pq ,σ , b 0 pq ,σ ).
With:

C l,k,l ,k ,d q = (1 -γ) 3 (γ l -γ k )(γ l -γ k ) k-1 i=l k -1 i =l ∞ j=i+i γ j c q (j + d).
(3.44)

Improved bounds using non-stationary Strategies

In this chapter, we explore a solution to decrease the γ-sensitivity of classical ADP algorithms [START_REF] Perolat | On the use of non-stationary strategies for solving two-player zero-sum markov games[END_REF]. Section 1.2.3 describes how the use of non-stationary policies can be leveraged in MDPs to reduce the γ-sensitivity from

2(γ-γ k ) (1-γ) 2 to 2(γ-γ k ) (1-γ)(1-γ m )
or to improve the concentrability coefficient. In that section, we explore how similar techniques can be used in zero-sum two-player MGs. From the point of view of player 1, the principle is the same. Instead of playing a stationary strategy µ K given by some algorithm, the player will play a cyclic strategy (µ

K , µ K-1 , . . . , µ K-m+1 , µ K , µ K-1 , . . . ).
To ease the notations, we will write such a strategy µ

K , µ K-1 , . . . , µ K-m+1 = µ k,m .
Using cyclic strategies in zero-sum two-player MGs provides a similar reduction in the γ-sensitivity of the algorithm as the one existing for MDPs. This chapter is organized in three parts. First, it provides definitions and properties required for this chapter. Then, we will present four ADP schemes using non-stationary strategies and provide their sensitivity analysis. Finally, an empirical evaluation of most algorithms is conducted in the last part of this chapter.

Non-Stationary Strategy in Zero-Sum Two-Player Markov Games

The value of a joint non-stationary strategy was defined in 2.53. If player 1 plays a nonstationary strategy of length M (a tuple (µ 0 , µ 1 , . . . , µ M -1 )), the value (v µ 0 ,µ 1 ,...,µ M -1 ) of this non-stationary strategy is the expected cumulative γ-discounted reward the player gets when his adversary plays the optimal counter strategy. Formally:

v µ 0 ,µ 1 ,...,µ M -1 (s) = min (νt) t∈N E +∞ t=0 γ t r µ i ,νt (s t )|s 0 = s, s t+1 ∼ P µ i ,νt (.|s t ), i = t mod(M ) . (4.1)
In other words, the strategy used in state s and at time t will depend on t. Instead of always following a single strategy the player will follow a cyclic strategy. At time t = 0 the player will play µ 0 , at time t = 1 he will play µ 1 and at time t = M j + i (∀i ∈ {0, . . . , M -1}, ∀j ∈ N ) he will play strategy µ i . This value is the fixed point of using non-stationary Strategies the operator T µ 0 . . . T µ M -1 . Thus, we have:

v µ 0 ,µ 1 ,...,µ M -1 = T µ 0 . . . T µ M -1 v µ 0 ,µ 1 ,...,µ M -1 . (4.2)
Proof. This property might appear intuitive but its proof remains technical. We want to show:

v µ 0 ,µ 1 ,...,µ M -1 (s) = min (νt) t∈N E +∞ t=0 γ t r µ i ,νt (s t )|s 0 = s, s t+1 ∼ P µ i ,νt (.|s t ), i = t [M ] .
Is the fixed point of operator T µ 0 . . .

T µ M -1 .
This property seems intuitive. However, it's demonstration is non standard. First we build an MDP with a state space of size M × |S|. Then we prove the value we are interested in is a sub-vector of the optimal value of the large MDP. Finally we prove the sub-vector is a fixed point of T µ 0 . . .

T µ M -1 .
Let us define the following MDP with the same γ:

S = S × {0, . . . , M -1}. For (s, i) ∈ S we have Ã((s, i)) = A(s) × {i}. p((s , i)|(s, j), (a 2 , j)) = δ {i=j+1[M ]} a 1 ∈A 1 (s) µ j (a 1 |s)p(s |s, a 1 , a 2 ) (4.3) r((s, j), (a 2 , j)) = a 1 ∈A 1 (s) µ j (a 1 |s)r(s, a 1 , a 2 ) (4.4)
The Kernel and reward are defined as follow:

Pν ((s , i)|(s, j)) = E a 2 ∼ν(.|(s,j)) [p((s , i)|(s, j), (a 2 , j))] One should notice that (s , i) ∼ Pν (.|(s, j)) then i = j + 1[M ] (obvious consequence of (4.3)) rν ((s, j)) = E a 2 ∼ν(.|(s,j)) [r((s, j), (a 2 , j))]
Instead of trying to maximize we will try to minimize the cumulated γ-discounted reward. Let ṽ * be the optimal value of that MDP: ṽ * (s) = min

(νt) t∈N E[ +∞ t=0 γ t rνt ((s t , i))|s 0 = (s, 0), st+1 ∼ Pνt (.|(s t , i)), i = t [M ]], (4.8) = min (νt) t∈N E[ +∞ t=0 γ t rνt ((s t , i))|s 0 = (s, 0), s t+1 ∼ P µ i ,νt (.|s t ), i = t [M ], ν t (.|s) = νt (.|(s, j))],
(4.9)

= min

(νt) t∈N E[ +∞ t=0 γ t r µ i ,νt (s t )|s 0 = (s, 0), s t+1 ∼ P µ i ,νt (.|s t ), i = t [M ], ν t (.|s) = νt (.|(s, j))], (4.10) = min (νt) t∈N E[ +∞ t=0 γ t r µ i ,νt (s t )|s 0 = s, s t+1 ∼ P µ i ,νt (.|s t ), i = t [M ]], (4.11) = v µ 0 ,µ 1 ,...,µ M -1 (s). (4.12)
Let ṽ be a value function and let ṽi , i ∈ {0, . . . , M -1} be the restriction to S × {i} of ṽ.

From (4.3) we also have:

B ν ṽ = rν + Pν ṽ =          T µ 0 ,ν 0 ṽ1 T µ 1 ,ν 1 ṽ2 . . . T µ M -2 ,ν M -2 ṽM-1 T µ M -1 ,ν M -1 ṽ0         
where ν j (.|s) ∼ ν(.|(s, j)) thus, we have:

Bṽ = min ν (r ν + Pν ṽ) =          T µ 0 ṽ1 T µ 1 ṽ2 . . . T µ M -2 ṽM-1 T µ M -1 ṽ0         
But from basic property of dynamic programming we have:

B M ṽ * = ṽ *
and finally, from the definition of B and from (4.12) we have:

(T µ 0 . . . T µ M -1 ṽ0 )((s, 0)) = ṽ0 ((s, 0)) = v µ 0 ,µ 1 ,...,µ M -1 (s)
This property is a key of our analysis. [START_REF] Scherrer | Approximate Policy Iteration Schemes: A Comparison[END_REF], while NSVI and NSPI have a reduced dependency over γ compared to their stationary counterparts. Here, in addition to defining the extensions of those algorithms, we also prove theoretical guarantees of performance.

Value Iteration and Non-Stationary Value Iteration

The structure of the NSVI algorithm and of the VI algorithm are very similar. As a reminder, the VI algorithm proceeds in two steps. The first step of each iterations is the greedy step and the second step is an evaluation step. We consider the case where, at iteration k, the greedy step picks an k -greedy strategy (i.e.

µ k ∈ G k (v k-1
)) and the evaluation step is made up to an evaluation error k . Iteration k can be formally described as follow:

T v k-1 ≤ T µ k v k-1 + k , (approximate greedy step) v k = T µ k v k-1 + k . (approximate evaluation step)
As in the previous chapter, we consider that those errors propagate from one iteration to the next, and we will analyse how far the final strategy may be from optimal. Again, to measure the performance of such an algorithm, we will bound (according to some norm) the distance between the optimal value and the value of the final strategy when the opponent is playing optimally.

The VI algorithm presented above produces a sequence of values v 0 , . . . , v k and, implicitly, strategies µ 0 , . . . , µ k . The non-stationary variation of VI for MGs, NSVI (Algorithm 18), simply consists in playing the m last strategies generated by VI for MGs. As described in the introduction, this strategy is written

µ k,m = µ k , µ k-1 , . . . , µ k-m+1 .
In the following, we provide a bound in L p -norm for NSVI in the framework of zero-sum two-player MGs. The goal is to bound the difference between the optimal value v * and the value v µ k,m = v µ k ,µ k-1 ,...,µ k-m+1 of the m last strategies generated by VI.

Algorithm 18 Non-Stationary Value Iteration

Input: An zero-sum two-player MG M G, a value v 0 = 0 and a maximum number of iterations K. for k=1,2,...,K do

find µ k ∈ G k (v k-1 ) v k = T µ k v k-1 + k end for Output: µ K,m = (µ K , µ K-1 , . . . , µ K-m+1 )
Usually, one is only able to control and according to some norm . pq ,σ and wants to control the difference of value functions according to some other norm . p,ρ (this is explained in 1.2.3 for MDPs and in the previous chapter for zero-sum two-player MGs). The following theorem provides a performance guarantee in L p -norm: Theorem 4.1. Let ρ and σ be distributions over states. Let p,q and q be positive reals such that 1 q + 1 q = 1, then for a non-stationary policy of size m and after k iterations we have:

v * -v µ k,m p,ρ ≤ 2(γ -γ k )(C 1,k,0,m q ) 1 p (1 -γ)(1 -γ m ) sup 1≤j≤k-1 j pq ,σ value update error + (1 -γ k )(C 0,k,0,m q ) 1 p (1 -γ)(1 -γ m ) sup 1≤j≤k j pq ,σ greedy error (4.13) + 2γ k 1 -γ m (C k,k+1,0,m q ) 1 p v * -v 0 pq ,σ contraction term , ( 4

.14)

With:

C l,k,d,m q = (1 -γ)(1 -γ m ) γ l -γ k k-1 i=l ∞ j=0 γ i+jm c q (i + jm + d)
and where:

c q (j) = sup µ 1 ,ν 1 ,...,µ j ,ν j d(ρP µ 1 ,ν 1 . . . P µ j ,ν j ) dσ q,σ
.

Proof. The full proof is left in appendix 6. Techniques used for the previous bound are similar to the ones used to analyse AGPI.

Each term of this bound is decomposed in three elements. The contraction term is essentially equivalent to the one of VI. The γ-contraction term 2γ k 1-γ m (instead of 2γ k 1-γ ) is slightly improved and converges to zero at a geometrical rate. The concentrability coefficient is improved in some sense [START_REF] Scherrer | Approximate Policy Iteration Schemes: A Comparison[END_REF]) (if it is infinite, it implies that the one of VI is infinite). Finally, the error on the initial value is slightly worse but we believe that extra care in the analysis would provide the same dependency on the initial value.

As for the error propagation bounds, the difference between VI and NSVI is way more significant (here we analyse both the value update error and the greedy error). Whilst the γ-sensitivity of VI is of the order of 1

(1-γ) 2 , the γ-sensitivity of VI is of the order of 1 (1-γ)(1-γ m ) . For instance, if γ = 0.99 (which is a typical choice in applications) we have 1

(1-γ) 2 = 10000 for VI and

1 (1-γ)(1-γ m )
if of the order of 5000 for m = 2, 2000 for m = 5 and 1000 for m = 10. Again, the concentrability coefficient is improved and the -error remains the same in both cases.

From an implementation point of view, this technique introduces an explicit tradeoff between memory and error. Indeed, m strategies have to be stored instead of 1 to decrease the value function approximation error from 2γ

(1-γ) 2 to 2γ

(1-γ)(1-γ m ) . Moreover, a benefit of the use of a non-stationary strategy in VI is that it comes from a known algorithm and thus needs very little implementation effort. using non-stationary Strategies Remark 4.1. We can notice the full bound matches the bound on stationary strategies in L p -norm of the previous chapter for the first and second terms. It also matches the one in L ∞ -norm for non-stationary policies of [START_REF] Scherrer | On the Use of Non-Stationary Policies for Stationary Infinite-Horizon Markov Decision Processes[END_REF] in the case of MDPs.

Policy Search by Dynamic Programming (PSDP)

PSDP was first introduced by [START_REF] Bagnell | Policy Search by Dynamic Programming[END_REF] for solving undiscounted MDPs and undiscounted Partially Observable MDPs (POMDPs), but a natural variant using nonstationary strategies can be used for the discounted case [START_REF] Scherrer | Approximate Policy Iteration Schemes: A Comparison[END_REF]. When applied to MDPs, this algorithm enjoys the best concentrability coefficient among several algorithms based on policy iteration, namely Non-stationary Policy Iteration (NSPI), Conservative Policy Iteration (CPI), Approximate Policy Iteration (API) and NSPI(m) (see [START_REF] Scherrer | Approximate Policy Iteration Schemes: A Comparison[END_REF] for more details). In this section, we describe two extensions of PSDP (PSDP1 and PSDP2) to two-player zero-sum MGs. Both algorithms reduce to PSDP in the case of MDPs but only one of them can be practically implemented in the case of a MG (PSDP2). Still, we present PSDP1 since its concentrability coefficient is close to the one given by Scherrer for PSDP in MDPs. The main difference between those two schemes is that PSDP1 involves the non-linear operator T µ whilst PSDP2 involves operator T µ,ν

Algorithm 19 PSDP1

Input: An zero-sum two-player MG M G, a value v 0 = 0 and a maximum number of iterations K. for k=1,2,...,K do

find µ k ∈ G(v σ k-1 ), v σ k-1 = T µ k-1 . . . T µ 0 0 + k-1 end for Output: µ K,K+1 = (µ K , µ K-1 , . . . , µ 0 )
PSDP1: A first natural extension of PSDP in the case of γ-discounted Markov games is the following. At each step the algorithm returns a strategy µ k for the maximizer, such that

T v σ k-1 = T µ k v σ k-1 where v σ k-1 = T µ k-1 . . . T µ 0 0 + k-1 .
Following any nonstationary strategy that uses σ k (= µ k , . . . , µ 0 ) for the k + 1 first steps, we have the following performance guarantee: Theorem 4.2. Let ρ and σ be distributions over states. Let p,q and q be positive reals such that 1 q + 1 q = 1, then we have:

v * -v µ k,k+1 p,ρ ≤ 2(C 1,k,0 µ * ,q ) 1 p 1 -γ sup 0≤j≤k j pq ,σ + o(γ k ), (4.15)
2. Algorithms 69 with:

C l,k,d µ * ,q = 1 -γ γ l -γ k k-1 i=l γ i c µ * ,q (i + d)
and where:

c µ * ,q (j) = sup ν 1 ,...,ν j ,µ j d(ρP µ * ,ν 1 . . . P µ * ,ν j-1 P µ j ,ν j ) dσ q,σ
.

Proof. The full proof is left in appendix 7.2.

The concentrability coefficient of this algorithm is similar to the one of its MDP counterpart: with respect to the first player, it has the advantage of depending mostly on the optimal strategy µ * . However, estimating at each iteration T µ k-1 . . . T µ 0 0 requires solving a control problem and thus might be computationally prohibitive. Therefore, we introduce a second version of PSDP for games, which does not require to solve a control problem at each iteration.

Algorithm 20 PSDP2

Input: An zero-sum two-player MG M G, a value v 0 = 0 and a maximum number of iterations K. for k=1,2,...,K do

find (µ k , ν k ) such that T v σ k-1 = T µ k v σ k-1 and T µ k v σ k-1 = T µ k ,ν k v σ k-1 , v σ k-1 = T µ k-1 ,ν k-1 . . . T µ 0 ,ν 0 0 + k end for Output: µ K,K+1 = (µ K , µ K-1 , . . . , µ 0 )
PSDP2: This algorithm creates a sequence of strategies ((µ k , ν k ),. . . ,(µ 0 , ν 0 )). At each step k the algorithm returns a pair of strategies

(µ k , ν k ) such that T v σ k-1 = T µ k v σ k-1 and T µ k v σ k-1 = T µ k ,ν k v σ k-1 (where v σ k-1 = T µ k-1 ,ν k-1 . . . T µ 0 ,ν 0 0+ k ).
To analyze how the error propagates through iterations, we will compare the value of the nonstationary strategy µ k,k+1 (=µ k ,. . . ,µ 0 ) against the value of best response to the optimal strategy. Theorem 4.3. Let ρ and σ be distributions over states. Let p,q and q be positive reals such that 1 q + 1 q = 1, then we have:

v * -v µ k,k+1 p,ρ ≤ 4(C 1,k,0 q ) 1 p 1 -γ sup 0≤j≤k j pq ,σ + o(γ k ), (4.16)
with:

C l,k,d q = 1 -γ γ l -γ k k-1 i=l γ i c q (i + d).
Proof. The full proof is left in appendix 7.1. using non-stationary Strategies PSDP1 and PSDP2 are different algorithms since one requires to solve a finite horizon γ-discounted MDP (PSDP1) as a subroutine whilst the second one (PSDP2) only requires to evaluate a joint strategy. Thus, the error term k in PSDP1 is an error due to solving a control problem, while the error term k in PSDP2 comes from a pure estimation error.

However, both algorithms share the same issue. Both algorithms require the storage of all strategies from the very first iteration. The algorithm PSDP1 needs to store k strategies at iteration k while PSDP2 needs 2k at the same stage. However PSDP2 alleviates a major constraint of PSDP1: it doesn't need an optimization subroutine at each iteration. The price to pay for that simplicity is to store 2k strategies and a worse concentrability coefficient.

Indeed, one can notice that ∀j ∈ N c µ * ,q (j) ≤ c q (j) and thus C l,k,d µ * ,q ≤ C l,k,d q . And thus, the concentrability coefficient of PSDP2 is worse than PSDP1's. Moreover, k,d,m q when m goes to infinity. This also means that if

C l,k,d,m q = (1 -γ m )C l,k,d q + γ m C l,k,d+m,m q meaning intuitively that C l,k,d q is C l,
C l,k,d q = ∞, then we have C l,k,d,m q = ∞.
In that sense, one can argue that PSDP2 offers a better concentrability coefficient than NSVI. Finally, PSDP1 and PSDP2 enjoy a better γ-sensitivity than NSVI for the value update error ( 41-γ instead of

2(γ-γ k ) (1-γ)(1-γ m ) )

Non Stationary Policy Iteration (NSPI(m))

The approximate version of PI has the same guarantees in terms of greedy error and approximation error as VI. Like VI, there exists a non-stationary version of policy iteration that was originally designed for MDPs by [START_REF] Scherrer | On the Use of Non-Stationary Policies for Stationary Infinite-Horizon Markov Decision Processes[END_REF]. Instead of estimating the value of the current policy at each iteration, it estimates the value of the non-stationary policy formed by the last m policies. Generalized to MGs, NSPI(m) estimates the value of the best response to the last m strategies.

Algorithm 21 NSPI Input: An zero-sum two-player MG M G, a value v 0 = 0 and a maximum number of iterations K. for k=1,2,...,K do

find µ k ∈ G(v k-1 ), v k = v µ k,m + k end for Output: µ K,m = (µ K , µ K-1 , . . . , µ K-m+1 )
Doing so, the algorithm NSPI(m) tackles the memory issue of PSDP. It allows controling the size of the stored non-stationary strategy. NSPI(m) proceeds in two steps. First, it computes an approximation

v k of v µ k,m (v k = v µ k,m + k ). Here, v µ k,m
is the value of the best response of the minimizer to strategy µ k,m = µ k , . . . , µ k-m+1 (defined in Section 1). Then it moves to a new strategy µ k+1 satisfying T v k = T µ k+1 v k .

Empirical Evaluation 71

Theorem 4.4. Let ρ and σ be distributions over states. Let p,q and q be positive reals such that 1 q + 1 q = 1, then for a non-stationary policy of size m and after k iterations we have:

v * -v µ k,m p,ρ ≤ 2γ(C 1,k-m+2,0,m q ) 1 p (1 -γ)(1 -γ m ) sup m≤j≤k-1 j pq ,σ + o(γ k ).
(4.17)

Proof. The full proof is left in appendix 8.

The NSPI dependency over γ and the concentrability coefficient involved in the NSPI bound are similar to those found for NSVI. However, in the MDP case the policy evaluation error is responsible for the error k and in the MG case the error comes from solving a full control problem.

Summary

To sum up, both NSVI and NSPI enjoy a reduced dependency over γ (i.e.

1 (1-γ)(1-γ m ) )
when considering non-stationary strategies. They exhibit similar concentrability coefficients but the origin of the error is different (a regression for NSVI and a policy evaluation or a control problem for NSPI). PSDP1 and PSDP2 enjoys an even better dependency on γ (i.e. 1 1-γ ). The concentrability coefficient of PSDP1 is better than that of PSDP2 which is better than those of NSVI and NSPI. However, the error involved in the analysis of PSDP1 is caused by solving a full control problem (which makes this algorithm impractical) while the error in PSDP2 comes from a simple regression.

Empirical Evaluation

The previous section provided, for each new algorithm, a performance bound that assumes a worst case error propagation. Examples suggesting the bound is tight were provided by [START_REF] Scherrer | On the Use of Non-Stationary Policies for Stationary Infinite-Horizon Markov Decision Processes[END_REF] for MDPs in the case of L ∞ analysis (p = ∞).

Since those examples apply on MDPs, they also apply to MGs. Those specific examples are pathological problems and, in practice, the bounds will generally be conservative. Furthermore, our analysis of the term k somehow hides the sources of errors that may vary a lot among the different algorithms. To ensure these techniques are relevant in practice and to go beyond the theoretical analysis, we tested them on synthetic MDPs and turn-based MGs, named Garnets [START_REF] Archibald | On the Generation of Markov Decision Processes[END_REF].

Garnets for MDPs:

A Garnet is originally an abstract class of MDPs. It is generated according to three parameters (N S ,N A ,N B ). Parameters N S and N A are respectively the number of states and the number of actions. Parameter N B is the branching factor defining the number of possible next states for any state-action pair. The procedure to generate the transition kernel p(s |s, a) is the following. First, one should using non-stationary Strategies draw a partition of [0, 1] by drawing N B -1 cutting points uniformly over [0, 1] noted (p i ) 1≤i≤N B -1 and sorted in increasing order (let us note p 0 = 0 and p N B = 1). Then, one draws a subset {s 1 , . . . , s N B } of size N B of the state space S. This can be done by drawing without replacement N B states from the state space S. Finally, one assigns p(s i |s, a) according to the following rule: ∀i ∈ {1, . . . , N B }, p(s i |s, a) = p i -p i-1 . The reward function r(s) depends on the experiment.

Garnet for two-player turn-based MGs

We are interested in a special kind of MGs, namely turn-based games. Here, turn-based games are two-player zero-sum MGs where, at each state, only one player has the control on the game. The generating process for this kind of Garnet is the same as the one for MDPs. Then we will independently decide for each state which player has the control over the state. The probability of state s to be controlled by player 1 is 1 2 .

Experiments

In the two categories of Garnets described previously we ran tests on Garnets of size N S = 100, with N A ∈ {2, 5, 8} and N B ∈ {1, 2, 5}. The experiment aims at analyzing the impact of the use of non-stationary strategies considering different amounts of samples at each iteration for each algorithm. Here, the reward for each state-action couple is null except for a given proportion (named the sparsity ∈ {0.05, 0.1, 0.5}) drawn according to a normal distribution of mean 0 and of variance 1. Algorithms are based on the state-actions value function (defined in 3.3):

Q µ,ν (s, a 1 , a 2 ) = r(s, a 1 , a 2 ) + s ∈S p(s |s, a 1 , a 2 )v µ,ν (s ).
The analysis of previous section still holds since one can consider an equivalent (but larger) MG whose state space is composed of state-action pairs. Moreover, each evaluation step consists in approximating the state-action(s) value function. We approximate the value function by minimizing a L 2 norm on a tabular basis with a regularization also in L 2 norm. All the following considers simultaneous MGs. To retrieve algorithms for MDPs, consider that the minimizer always has a single action. To retrieve algorithms for turn-based MGs, consider that at each state only a single player has more than one action.

Experiments are limited to finite states MGs. Moreover, Garnets have an erratic dynamic since next states are drawn without replacement within the set of states, thus the dynamic is not regular in any sens. Garnets are thus tough to optimize. Experiments on simultaneous games are not provided due to difficulties encountered to optimize such games. We believe Garnets are too hard to optimize when it comes to simultaneous games.

In all presented graphs, the performance of a strategy µ (which might be stationary or not) is measured as

v * -vµ u,2 v * u,2
where u is the uniform measure over the state-action(s) space. The value v µ is computed exactly with the policy iteration algorithm. In every curve, the confidence interval is proportional to the standard deviation. To compare algorithms on a fair basis, their implementation relies on a sample-based approximation involving an equivalent number of samples. In all tests, we could not notice a significant 3. Empirical Evaluation 73 influence of N A . Moreover the sparsity only influences the amount of samples needed to solve the MG.

NSVI

The NSVI algorithm starts with a null Q-functions Q 0 (s, a 1 , a 2 ). At each iteration we draw uniformly over the state-actions space (s i , a 1 i , a 2 i ) then we compute non-stationary strategies in VI is the reduction of the standard deviation of the value v µ k ,...,µ k-m+1 . Figure 4.1 shows the reduction of the variance when running a nonstationary strategy. Intuitively one can think of it as a way of averaging over the last m strategies the resulting value. Moreover, the larger m is the more the performance concentrates (the parameter m is varied in figure 4. 1, 4.4, 4.5, 4.6 and 4.7). A second using non-stationary Strategies advantage is the improvement of the average performance when N B (the branching factor of the problem) is low. One the negative side, since we are mixing last m strategies, the asymptotic performance is reached after more iterations (see Figure 4.1).

r i = r(s i , a 1 i , a 2 i ) and draw s i ∼ p(.|s i , a 1 i , a 2 i ) for i ∈ {1, . . . , N k }. Then we compute q i = r i + γ min a 2 E a 1 ∼µ k (.|s i ) [Q k (s i , a 1 , a 2 )]. The next state-actions value function Q k+1 is the best fit over the training dataset {(s i , a 1 i , a 2 i ), q i } i∈{1,...,N k } .

PSDP2

In practice PSDP2 builds N k rollout (indexed by j) {(s i,j , a 1 i,j , a 2 i,j , r i,j )} i∈{0,...,k+1} at iteration k. Where (s 0,j , a 1 0,j , a 2 0,j ) are drawn uniformly over the state-actions space and where s i+1,j ∼ p(.

|s i,j , a 1 i,j , a 2 i,j ), a 1 i+1,j ∼ µ k-i (.|s i+1,j ), a 2 i+1,j ∼ ν k-i (.|s i+1,j
) and r i+1,j = r(s i+1,j , a 1 i+1,j , a 2 i+1,j ). Then we build the dataset {(s 0,j , a 1 0,j , a 2 0,j ), From an implementation point of view, PSDP2 uses (k + 2) × N k samples at each iteration (parameter N k is varied in the figures 4.2 and 4.7). Furthermore, the algorithm uses rollouts of increasing size. As a side effect, the variance of k+1 i=0 γ i r i,j increases with iterations for non-deterministic MDPs and MGs. This makes the regression of Q k+1 less practical. To tackle this issue, we use a procedure, named resampling, that consists in averaging the cumulative γ-discounted reward k+1 i=0 γ i r i,j over different rollouts launched from the same state-actions triplet (s 0,j , a 1 0,j , a 2 0,j ). In figure 4.2 the two top curves display the performance of PSDP2 with (on the right) and without (on the left) resampling trajectories on deterministic MGs. The two figures on the bottom are however obtained on non-deterministic MGs. One can notice a significant improvement of the algorithm when using resampling on non-deterministic MGs illustrating the variance issue raised in the foregoing paragraph.

k+1 i=0 γ i r i,j } j∈{0,...,N k } .

PSDP1

We do not provide experiments within the PSDP1 scheme. Each iteration of PSDP1 consists in solving a finite horizon control problem (i.e. approximating v σ k = T µ k . . . T µ 0 0). The problem of estimating v σ k reduces to solving a finite horizon MDP with non stationary dynamics. To do so, one should either use Fitted-Q iterations or PSDP for MDPs. In the first case, one would not see the benefit of such a scheme compared to the use NSVI. Indeed, each iterations of PSDP1 would be as heavy in term of computation as NSVI. In the second case, one would not see the benefit compared PSDP2 since each iterations would be as heavy as PSDP2.

NSPI(m)

At iteration k, NSPI(m) approximates the best response of the nonstationary strategy µ k , . . . , µ k-m+1 . In the case of an MDP, this results in evaluating a policy. The evaluation of a stationary policy is done by an approximate iteration of T µ . This procedure can be done by a procedure close to Fitted-Q iteration in which the strategy µ is used at each iteration instead of the greedy policy. The evaluation of the non-stationary policy µ k , . . . , µ k-m+1 is done by approximately applying in a cycle T µ k , . . . , T µ k-m+1 . For MG, the subroutine will contain l × m iterations. At iteration All Garnets have a sparsity of 0.5 and γ = 0.9. Each step of the algorithm uses 2.25 × N A × N S rollouts. p ∈ {1, . . . , l × m} the subroutine computes one step of Fitted-Q iteration considering the maximizer's strategy is fixed and of value µ k-m+(p-1 mod(m))+1 and taking the greedy action for the minimizer. In this subroutine the dataset is fixed (it is only refreshed at each iteration of the overall NSPI(m) procedure). The parameter l is chosen large enough to achieve a given level of accuracy, that is having γ m×l below a given threshold. Note that for small values of k (i.e. k < m) this implementation of the algorithm finds an approximate best response of the non-stationary strategy µ k , . . . , µ 1 (of size k and not m). As for VI, the use of non-stationary strategies reduces the standard deviation of the performance as m grows.

A Comparison

From the theoretical analysis, one may conclude that PSDP1 is the best scheme to solve MGs. It's dependency over γ is the lowest among the analyzed algorithms and it exhibits the best concentrability coefficient. However, from the implementation point of view this scheme is a very cumbersome since it implies solving a finite horizon control problem of increasing size, meaning using an algorithm like Fitted-Q or PSDP as a subroutine at each iteration. PSDP2 tackles the main issue of PSDP1. This algorithm doesn't need to solve a control problem as a subroutine but a simple supervised learning step is enough. The price to pay is a worst bound and the storage of 2 × k instead of k strategies.

As in PSDP1, the NSPI algorithm needs to solve a control problem as a subroutine but it only considers a constant number of strategies. Thus NSPI solves the memory issue of PSDP1 and PSDP2. The γ-sensitivity of the error bound is reduced from roughly

1 (1-γ) 2 to 1 (1-γ)(1-γ m )
where m is the length of the non-stationary strategy considered. Nevertheless, the error term of PSDP2 derives from a supervised learning problem while it comes from a control problem in NSPI. Thus, controlling the error of PSDP2 might be easier than controlling the error of NSPI. The NSVI algorithm enjoys an error propagation bound similar to the NSPI one. However the error of NSVI derives from a simple supervised learning problem instead of a full control problem as for NSPI. Figure 4.4 compares NSPI and NSVI with the same number of samples at each iteration. It clearly shows NSVI performs better in average performance and regarding the standard deviation of the performance. For turn-based MGs the NSVI algorithm performs better than NSPI on Garnets. Furthermore one iteration of NSPI costs significantly more than an iteration of NSVI. Figure 4.4 also compares PSDP2 and NSVI. Even if PSDP2 uses k times more samples than NSVI at iteration k, it barely achieves the same performance as NSVI (in the case of a non-deterministic game this is not even the case, see Figure 4.6 in the appendix).

Chapter 4. Improved Approximate Dynamic Programming Algorithms using non-stationary Strategies

Conclusion

In this chapter, we explored the generalization of different techniques known for MDPs to zero-sum two-player MGs. These techniques make the use of non-stationary strategies to improve the error propagation bounds of several known dynamic programming algorithms. The theoretical analysis shows a reduced dependency over γ of nonstationary algorithms. For instance NSVI and NSPI have a dependency of

2γ (1-γ)(1-γ m ) instead of 2γ
(1-γ) 2 for the corresponding stationary algorithm. PSDP2 has a dependency of 2

(1-γ) over γ and it enjoys a better concentrability constant than NSPI and NSVI. The empirical study shows the dependency on the error is the main factor when comparing algorithms with the same budget of samples. The nature of the error seems to be crucial. NSVI outperforms NSPI since a simple regression produces less error than a policy evaluation or even a full control problem. NSVI outperforms PSDP2 since it is more thrifty in terms of samples per iteration.

One of the downside of this analysis is that it is limited to zero-sum two-player MGs whilst the use of cyclic strategies was suggested in general-sum games [START_REF] Zinkevich | Cyclic Equilibria in Markov Games[END_REF]. In the next chapter we will explore to which extent it is easier to find cyclic equilibria in general-sum MGs instead of a stationary one for the VI algorithm.

Appendix: NSVI

First, let us define a (somehow abusive) simplifying notation. Γ n will represents any products of n discounted transition kernels. Then, Γ n represents the class {γP µ 1 ,ν 1 . . . . .γP µn,νn , with µ i , ν i random strategies}. For example, the following property holds aΓ i bΓ j + cΓ k = abΓ i+j + cΓ k .

NSVI with a greedy and an evaluation error:

T v k-1 ≤ T µ k v k-1 + k , (approximate greedy step) v k = T µ k v k-1 + k . (approximate evaluation step)
The following lemma shows how errors propagate through iterations.

Lemma 4.1. ∀M < k:

v * -v µ k,M ≤ ∞ j=0 Γ M j 2Γ k |v * -v 0 | + 2 k-1 i=1 Γ i | k-i | + k-1 i=0 Γ i k-i . (4.18)
Proof. We will bound the error made while running the non-stationary strategy (µ k , . . . , µ k-M +1 ) rather than the optimal strategy. This means bounding the following positive quantity:

v * -v µ k,M
To do so let us first bound the following quantity:

T µ k v k-1 -v µ k,M (4.19) = T µ k v k-1 -T µ k . . . T µ k-M +1 v µ k,M , (with (4.2)) (4.20) = T µ k v k-1 -T µ k ,ν k . . . T µ k-M +1 ,ν k-M +1 v µ k,M , (4.21)
Where νk-i such as

T µ k-i ,ν k-i . . . T µ k-M +1 ,ν k-M +1 v µ k,M = T µ k-i . . . T µ k-M +1 v µ k,M ≤ T µ k ,ν k . . . T µ k-M +1 ,ν k-M +1 v k-M + i=M -1 i=1 γP µ k ,ν k . . . γP µ k-i+1 ,ν k-i+1 k-i -T µ k ,ν k . . . T µ k-M +1 ,ν k-M +1 v µ k,M , (4.22) since v i = T µ i v i-1 + i , ∀vT µ k v ≤ T µ k ,ν k v and since T µ k ,ν k is affine ≤ γP µ k ,ν k . . . γP µ k-M +1 ,ν k-M +1 (v k-M -v µ k,M ) + i=M -1 i=1 γP µ k ,ν k . . . γP µ k-i+1 ,ν k-i+1 k-i . (4.23)
We also have:

v * -v k = T v * -T v k-1 + T v k-1 -v k , (4.24) ≤ T µ * v * -T µ * v k-1 -k + k , (4.25) (since T µ * v k-1 ≤ T v k-1 and T µ * v * = T v * ) ≤ T µ * ,ν k-1 v * -T µ * ,ν k-1 v k-1 -k + k , (4.26) (with T µ * ,ν k-1 v k-1 = T µ * v k-1 and T µ * v * ≤ T µ * ,ν k-1 v * ) ≤ γP µ * ,ν k-1 (v * -v k-1 ) -k + k . ( 4

.27) using non-stationary Strategies

And we have:

v * -v k = T v * -T µ k v k-1 + T µ k v k-1 -v k , (4.28) ≥ T µ k v * -T µ k v k-1 -k , (4.29) (since T µ k v * ≤ T v * ) ≥ T µ k ,ν * k v * -T µ k ,ν * k v k-1 -k , (4.30) (where T µ k ,ν * k v * = T µ k v * and since T µ k v k-1 ≤ T µ k ,ν * k v k-1 ) ≥ γP µ k ,ν * k (v * -v k-1 ) -k . (4.31) (4.32)
which can also be written:

v k -v * ≤ γP µ k ,ν * k (v k-1 -v * ) + k . Using the Γ n notation . v * -v k ≤ Γ k (v * -v 0 ) - k-1 i=0 Γ i k-i + k-1 i=0 Γ i k-i , (4.33) v k -v * ≤ Γ k (v 0 -v * ) + k-1 i=0 Γ i k-i . (4.34) v * -v µ k,M = T v * -T v k-1 + T v k-1 -v µ k,M , (4.35) ≤ T µ * v * -T µ * v k-1 + T v k-1 -v µ k,M , (4.36) (since T µ * v k-1 ≤ T v k-1 ) ≤ T µ * ,ν k-1 v * -T µ * ,ν k-1 v k-1 + T v k-1 -v µ k,M , (4.37)
with νk-1 defined in (4.26)

≤ γP µ * ,ν k-1 (v * -v k-1 ) + T µ k v k-1 -v µ k,M + k , (4.38) ≤ Γ(v * -v k-1 ) + Γ M (v k-M -v µ k,M ) + M -1 i=1 Γ i k-i + k , With (4.23) (4.39) ≤ Γ(v * -v k-1 ) + Γ M (v k-M -v * ) + Γ M (v * -v µ k,M ) + M -1 i=1 Γ i k-i + k . (4.40)
Then combining (4.33), (4.34) and (4.40):

v * -v µ k,M ≤ Γ k (v * -v 0 ) - k-1 i=1 Γ i k-i + k-1 i=1 Γ i k-i + Γ k (v 0 -v * ) + Γ M k-M -1 i=0 Γ i k-M -i + M -1 i=1 Γ i k-i + k + Γ M (v * -v µ k,M ), (4.41) v * -v µ k,M ≤ 2Γ k |v * -v 0 | + 2 k-1 i=1 Γ i | k-i | + k-1 i=0 Γ i k-i + Γ M v * -v µ k,M . (4.42) (4.43)

Appendix: NSVI 81

And finally:

v * -v µ k,M ≤ ∞ j=0 Γ M j [2Γ k |v * -v 0 | + 2 k-1 i=1 Γ i | k-i | + k-1 i=0 Γ i k-i ]. (4.44)
Full analysis of NSVI Usually, one is only able to control the and according to some norm . q,µ and wants to control the difference of value according to some other norm . p,ρ where

f p,σ = s∈S |f (s)| p σ(s) 1 p
. Then the following theorem controls the convergence in L p -norm:

Theorem 4.5. Let ρ and σ be distributions over states. Let p,q and q be positive reals such that 1 q + 1 q = 1, then for a non-stationary policy of size M and after k iterations we have:

v * -v µ k,M p,ρ ≤ 2(γ -γ k )(C 1,k,0,M q ) 1 p (1 -γ)(1 -γ M ) sup 1≤j≤k-1 j pq ,σ + (1 -γ k )(C 0,k,0,M q ) 1 p (1 -γ)(1 -γ M ) sup 1≤j≤k j pq ,σ + 2γ k 1 -γ M (C k,k+1,0,M q ) 1 p v * -v 0 pq ,σ , (4.45)
With:

C l,k,d,M q = (1 -γ)(1 -γ M ) γ l -γ k k-1 i=l ∞ j=0 γ i+jM c q (i + jM + d)
and where:

c q (j) = sup µ 1 ,ν 1 ,...,
|f | ≤ i∈I j∈J i Γ j |g i | = n l=1 i∈I l j∈J i Γ j |g i | .
Then for all p, q and q such as 1 q + 1 q = 1 and for all distribution ρ and σ we have

f p,ρ ≤ n l=1 (C q (l)) 1 p sup i∈I l g i pq ,σ i∈I l j∈J i γ j .

82

Chapter 4. Improved Approximate Dynamic Programming Algorithms using non-stationary Strategies with the concentrability coefficient written:

C q (l) = i∈I l j∈J i γ j c q (j) i∈I l j∈J i γ j .
Theorem 4.5 can be proven by applying lemma 4.2 with:

I = {1, . . . , 2k} I = {I 1 , I 2 , I 3 }, I 1 = {1, 2, . . . , k -1}, I 2 = {k, . . . , 2k -1}, I 3 = {2k} ∀i ∈ I 1 g i = 2 k-i J i = {i, i + M, i + 2M, . . .} ∀i ∈ I 2 g i = k-(i-k) J i = {i -k, i -k + M, i -k + 2M, . . .} ∀i ∈ I 3 g i = |v * -v 0 | J i = {k, k + M, k + 2M, . . .}
With lemma 3 of Scherrer et al. (2012). we have:

v * -v µ k,M p,ρ ≤ 2(γ -γ k )(C 1,k,0,M q ) 1 p (1 -γ)(1 -γ M ) sup 1≤j≤k-1 j pq ,µ + (1 -γ k )(C 0,k,0,M q ) 1 p (1 -γ)(1 -γ M ) sup 1≤j≤k j pq ,µ + 2γ k 1 -γ M (C k,k+1,0,M q ) 1 p v * -v 0 pq ,µ .
(4.46)

Appendix: PSDP

In this section we prove the two theorems for PSDP schemes (theorem 4.2) and 4.3).

7. Appendix: PSDP 83

Appendix: PSDP2

First let us remind the PSDP1 algorithm.

v σ k = T µ k ,ν k . . . T µ 0 ,ν 0 0 + k (4.47) T v σ k = T µ k+1 v σ k and T µ k+1 v σ k = T µ k+1 ,ν k+1 v σ k (4.48) Let's note v µ k,k+1 = T µ k . . . T µ 0 v µ k,k+1 and only in this section v µ k,k-i = T µ k -i . . . T µ 0 v µ k,k+1 .
To prove theorem 4.2 we will first prove the following lemma:

Lemma 4.3. ∀k > 0: 0 ≤ v µ * -v µ k,k+1 ≤Γ k+1 µ * v * + Γ µ * Γ k v µ k,k+1 + k i=1 Γ i µ * k-i + k i=1 Γ µ * Γ i-1 k-i (4.49) With k = Γ k-1 -k Proof. v * -v µ k,k+1 = T µ * v * -T µ * v σ k-1 + T µ * v σ k-1 -T µ k v µ k,k-1 (4.50) ≤ γP µ * ,ν k (v * -v σ k-1 ) + T µ * v σ k-1 -T µ * v µ k,k-1 (4.51) ≤ γP µ * ,ν k (v * -v σ k-1 ) + γP µ * ,ν k (v σ k-1 -v µ k,k-1 ) (4.52) ≤ Γ µ * (v * -v σ k-1 ) (1) +Γ µ * (v σ k-1 -v µ k,k-1 ) (2)
(4.53)

To prove (1):

v * -v σ k = T µ * v * -T µ k ,ν k (v σ k-1 -k-1 ) -k , (4.54) = T µ * v * -T µ k ,ν k v σ k-1 + γP µ k ,ν k k-1 -k , (4.55) = T µ * v * -T µ k v σ k-1 + γP µ k ,ν k k-1 -k , (4.56) = T µ * v * -T µ * v σ k-1 + T µ * v σ k-1 -T µ k v σ k-1 + γP µ k ,ν k k-1 -k , (4.57) ≤ T µ * v * -T µ * ,ν k v σ k-1 + T v σ k-1 -T µ k v σ k-1 = 0 +γP µ k ,ν k k-1 -k , (4.58) ≤ T µ * ,ν k v * -T µ * ,ν k v σ k-1 + γP µ k ,ν k k-1 -k = k , (4.59) ≤ γP µ * ,ν k (v * -v σ k-1 ) + k , (4.60) ≤ γP µ * ,ν k . . . γP µ * ,ν 0 (v * ) + k-1 i=0 γP µ * ,ν k . . . γP µ * ,ν k-i+1 k-i , (4.61) ≤ Γ k+1 µ * (v µ * ) - k-1 i=0 Γ i µ * k-i + k-1 i=0 Γ i µ * Γ k-i-1 . (4.

62) using non-stationary Strategies

To prove (2):

v σ k-1 -v µ k,k-1 = v σ k-1 -T µ k-1 . . . T µ 0 v µ k,k+1 , (4.63) = T µ k-1 ,ν k-1 (v σ k-2 -k-2 ) + k-1 -T µ k-1 . . . T µ 0 v µ k,k+1 , (4.64) = T µ k-1 ,ν k-1 v σ k-2 -P µ k-1 ,ν k-1 k-2 + k-1 -T µ k-1 . . . T µ 0 v µ k,k+1 , (4.65) = T µ k-1 v σ k-2 -T µ k-1 . . . T µ 0 v µ k,k+1 -k-1 , (4.66) = T µ k-1 v σ k-2 -T µ k-1 ,ν k-1 . . . T µ 0 v µ k,k+1 -k-1 , (4.67) = T µ k-1 v σ k-2 -T µ k-1 ,ν k-1 . . . T µ 0 v µ k,k+1 -k-1 , (4.68) ≤ T µ k-1 ,ν k-1 v σ k-2 -T µ k-1 ,ν k-1 . . . T µ 0 v µ k,k+1 -k-1 , (4.69) ≤ γP µ k-1 ,ν k-1 (v σ k-2 -v µ k,k-2 ) -k-1 , (4.70) ≤ γP µ k-1 ,ν k-1 . . . γP µ 0 ,ν 0 (0 -v µ k,k+1 ) - k i=1 γP µ k-1 ,ν k-1 . . . γP µ k-i+1 ,ν k-i+1 k-i , (4.71) ≤ Γ k v µ k,k+1 - k i=1 Γ i-1 k-i . (4.72) Finally: v * -v µ k,k+1 ≤ Γ µ * (v * -v σ k-1 ) + Γ µ * (v σ k-1 -v µ k,k-1 ) (4.73) ≤ Γ k+1 µ * v * + k-1 i=0 Γ i+1 µ * k-1-i + Γ µ * Γ k v µ k,k+1 - k i=1 Γ µ * Γ i-1 k-i (4.74) ≤ Γ k+1 µ * v * + Γ µ * Γ k v µ k,k+1 + k i=1 Γ i µ * k-i - k i=1 Γ µ * Γ i-1 k-i (4.75)
Finally, noticing v * and v µ k,k+1 ≤ V max we have:

0 ≤ v µ * -v µ k,k+1 ≤ Γ k+1 µ * v * + Γ µ * Γ k v µ k,k+1 + k i=1 Γ i µ * k-i + k i=1 Γ µ * Γ i-1 k-i (4.76) ≤ 2γ k+1 V max + k i=1 Γ i µ * (Γ k-i-1 -k-i ) + k i=1 Γ µ * Γ i-1 (Γ k-i-1 -k-i ) (4.77) v µ * -v µ k,k+1 ≤ 2γ k+1 V max + 4 k i=0 Γ i | k-i | (4.78)
Lemma 4.2 concludes the proof of theorem 4.2.

Appendix: PSDP1

Below is reminded the scheme of PSDP1

v σ k = T µ k . . . T µ 0 0 + k (4.79) T v σ k = T µ k+1 v σ k and T µ k+1 v σ k = T µ k+1 ,ν k+1 v σ k (4.80)
First we will prove the following lemma:

Lemma 4.4. ∀k > 0:

v µ * -v σ k ≤Γ k+1 µ * v * + k i=1 Γ i µ * k-i (4.
= Γ k-1 -k
Proof. The proof comes from previous section. It is the bound of (1).

Noticing v µ k,k+1 ≥ v σ k -γ k+1 V max and v * ≤ V max we have: 0 ≤ v µ * -v µ k,k+1 ≤2γ k+1 V max + 2 k i=0 Γ i-1 µ * Γ k-i (4.82)
Lemma 4.2 concludes the proof of theorem 4.3. However one has to do the proof with c µ * ,q (j) instead of c q (j).

Appendix: NSPI

We remind the non-stationary strategy of length m µ k , . . . , µ k-m+1 is written µ k,m and in this section

µ k,m = µ k-m+1 , µ k , . . . , µ k-m+1 , µ k , . . . . Let also note T µ k,m = T µ k . . . T µ k-m+1 . Then we will have v µ k,m = T µ k,m v µ k,m and v µ k,m = T µ k-m+1 v µ k,m .
NSPI:

v k = v µ k,m + k (4.83) T v k = T µ k+1 v k (4.84)
First let's prove the following lemma.

Lemma 4.5. ∀k ≥ m:

0 ≤ v * -v µ k+1,m ≤Γ k-m+1 µ * (v * -v µm,m ) + 2 k-m j=0 Γ j µ * Γ( +∞ i=0 Γ im ) k-j (4.

85) using non-stationary Strategies

Proof. First we need an upper bound for:

v µ k,m -v µµ k+1,m = T µ k-m+1 v µ k,m -v µ k+1,m (4.86) ≤ T µ k-m+1 ,ν k-m+1 v µk,m -v µ k+1,m (4.87) with T µ k-m+1 ,ν k-m+1 v k = T µ k-m+1 v k ≤ T µ k-m+1 ,ν k-m+1 (v k -k ) -v µ k+1,m (4.88) ≤ T µ k-m+1 ,ν k-m+1 v k -γP µ k-m+1 ,ν k-m+1 k -v µ k+1,m (4.89) ≤ T µ k-m+1 v k -v µ k+1,m -γP µ k-m+1 ,ν k-m+1 k (4.90) ≤ T µ k+1 v k -v µ k+1,m -γP µ k-m+1 ,ν k-m+1 k (4.91) ≤ T µ k+1 (v µ k,m + k ) -v µ k+1,m -γP µ k-m+1 ,ν k-m+1 k (4.92) ≤ T µ k+1 ,ν k+1 v µ k,m -v µ k+1,m + γ(P µ k+1 ,ν k+1 -P µ k-m+1 ,ν k-m+1 ) k (4.93) with T µ k+1 ,ν k+1 v µ k,m = T µ k+1 v µ k,m ≤ T µ k+1 v µ k,m -v µ k+1,m + γ(P µ k+1 ,ν k+1 -P µ k-m+1 ,ν k-m+1 ) k (4.94) = T µ k+1 . . . T µ k-m+1 v µ k,m -T µ k+1 . . . T µ k-m+2 v µ k+1,m + γ(P µ k+1 ,ν k+1 -P µ k-m+1 ,ν k-m+1 ) k (4.95) = T µ k+1 . . . T µ k-m+1 v µ k,m -T µ k+1 ,ν k+1 . . . T µ k-m+2 ,ν k-m+2 v µ k+1,m + γ(P µ k+1 ,ν k+1 -P µ k-m+1 ,ν k-m+1 ) k (4.96) with T µ k+1 ,ν k+1 . . . T µ k-m+2 ,ν k-m+2 v µ k+1,m = T µ k+1 . . . T µ k-m+2 v µ k+1,m ≤ γP µ k+1 ,ν k+1 . . . γP µ k-m+2 ,ν k-m+2 Γk+1,m (T µ k-m+1 v µ k,m v µ k,m -v µ k+1,m ) + γ(P µ k+1 ,ν k+1 -P µ k-m+1 ,ν k-m+1 ) k (4.97)
Then: (4.98)

≤ (I -Γk+1,m ) -1 γ(P µ k+1 ,ν k+1 -P µ k-m+1 ,ν k-m+1 ) k (4.99)
Proof of the lemma:

v * -v µ k+1,m = T µ * v * -T µ k+1,m v µ k+1,m (4.100) = T µ * v * -T µ * v µ k,m + T µ * v µ k,m -T µ k+1,m+1 v µ k,m + T µ k+1,m+1 v µ k,m -T µ k+1,m v µ k+1,m (4.101) ≤ γP µ * ,ν * k,m (v * -v µ k,m ) + Γk+1,m (T µ k-m+1 v µ k,m -v µ k+1,m ) + T µ * v µ k,m -T µ k+1 v µ k,m (1) (4.102) Where T µ * v µ k,m = T µ * ,ν * k,m v µ k,m (4.103)
(1):

T µ * v µ k,m -T µ k+1 v µ k,m = T µ * v µ k,m -T µ * v k + T µ * v k -T µ k+1 v k ≤0 + T µ k+1 v k -T µ k+1 v µ k,m (4.104) ≤ γP µ * ,ν * k+1 (v µ k,m -v k ) + γP µ k+1 ,ν k+1 (v k -v µ k,m ) (4.105) with T µ * v k = T µ * ,ν * k+1 v k ≤ γ(P µ k+1 ,ν k+1 -P µ * ,ν * k+1 ) k (4.106)
And finally:

v * -v µ k+1,m ≤ Γ µ * (v * -v µ k,m ) + γ(P µ k+1 ,ν k+1 -P µ * ,ν * k+1 ) k (4.107) + Γk+1,m (I -Γk+1,m ) -1 γ(P µ k+1 ,ν k+1 -P µ k-m+1 ,ν k-m+1 ) k (4.108) ≤ Γ µ * (v * -v µ k,m ) + 2Γ( +∞ i=0 Γ im ) k (4.109) ≤ Γ k-m+1 µ * (v * -v µm,m ) + 2 k-m j=0 Γ j µ * Γ( +∞ i=0 Γ im ) k-j (4.110)
Theorem 4.6. Let ρ and σ be distributions over states. Let p,q and q be positive reals such that 1 q + 1 q = 1, then for a non-stationary policy of size M and after k iterations we have:

v * -v µ k,m p,ρ ≤ 2(γ -γ k-m+2 )(C 1,k-m+2,0,m q ) 1 p (1 -γ)(1 -γ m ) sup m≤j≤k-1 j pq ,σ + γ k-m (c q (k -m)) 1 p v * -v µm,m pq ,σ , (4.111)
Proof. The proof of the theorem 4.6 is done by applying lemma 4.2

Then theorem 4.4 falls using theorem 4.6. Chapter 5

Non-Stationary Startegies in General-Sum Markov Games

This chapter focuses on the use of non-stationary strategies in general-sum MGs. In general sum games, two key properties that exist in zero-sum two-player MGs are missing. Those two properties are: i) the optimal Bellman operator is a γ-contraction (which guarantees the convergence toward the optimal value) and ii) acting greedily with respect to the optimal value generates the optimal strategy. In general-sum MGs, the first property does not hold since there is no known Bellman operator converging to the value of a Nash equilibrium (there is no known equivalent of T * and T ). The existence of such an operator is unlikely since there is not a unique Nash equilibrium in general sum MGs. In addition, the existence of turn-based games with a unique stochastic Nash equilibrium contradicts the second property [START_REF] Zinkevich | Cyclic Equilibria in Markov Games[END_REF]. Indeed, acting greedily in that turn-based case would lead to a deterministic strategy which is not the unique stochastic Nash equilibrium we are looking for. Therefore and more generally, Zinkevich's example implies that no algorithm based on the value function would lead to a stationary Nash equilibrium in the case of general-sum MGs. This is unfortunate since value function based algorithms, such as VI, benefit from a wide literature from MDPs and zero-sum two-player MGs. As described in the two previous chapters, these algorithms can handle value functions approximation as well as errors in the greedy step. It also holds in a wide range of practical scenarios such as when the dynamic is known, when only a simulator is available or when only a batch of data is provided [START_REF] Perolat | Approximate Dynamic Programming for Two-Player Zero-Sum Markov Games[END_REF][START_REF] Ernst | Tree-Based Batch Mode Reinforcement Learning[END_REF][START_REF] Munos | Performance Bounds in L p -norm for Approximate Value Iteration[END_REF][START_REF] Bertsekas | Dynamic Programming and Optimal Control[END_REF]. But, even if VI does not converge, it seems, as mentioned by [START_REF] Zinkevich | Cyclic Equilibria in Markov Games[END_REF], that VI cycles after a certain number of iterations. Thus, as suggested in [START_REF] Zinkevich | Cyclic Equilibria in Markov Games[END_REF], it is worth studying to what extent the use of non-stationary (cyclic) strategies gives some guarantees of convergence to a Nash equilibrium. Loose guarantees of convergence under very strong hypotheses and under a constrained definition of an -correlated equilibrium are given in [START_REF] Zinkevich | Cyclic Equilibria in Markov Games[END_REF]. Even if their work focuses on correlated equilibrium (which is not the case here), two questions arise: i) how long would it take to converge to an -equilibrium (for a given ), ii) how long should be the cycle for a given ? In this chapter, we explore to which extent the techniques presented in the previous chapter can be used in general sum games. We first show that the non-stationary strategy found by the VI algorithm is an -Nash equilibrium where is only controlled by the length of the cycle and not by the number of iterations (which answers the two questions) contrary to MDPs. More precisely, we provide an upper bound of convergence O( γ m 1-γ ) where m is the length of the cycle whereas in

92

Chapter 5. On the use of non-stationary strategies

MDPs the upper bound is O(

γ k (1-γ)(1-γ m ) )
where k is the number of iterations. Second, we providing an example where the bound is tight up to a multiplicative constant. This example demonstrates that using value iteration with non-stationary strategies provides no better guarantees than approximating the infinite horizon γ-discounted problem with a finite horizon problem (of horizon m). In other words, playing the cyclic strategy (of VI) gives no better guarantee than playing first the m strategies of VI and then any arbitrary strategy.

Background on Cyclic Strategies and Nash Equilibrium

A cyclic strategy π cycle,M = (π 0 , . . . , π M -1 ) is a joint strategy where each player will play at time t the strategy π j where j ≡ t (mod M ). The value of such a strategy is the following:

v i π cycle,M (s) = E +∞ t=0 γ t r i π j (s t )|s 0 = s, s t+1 ∼ P π j (.|s t ), j ≡ t (mod M ) . (5.1)
This value is the fixed point of the operator T i π 0 . . . T i π M -1 . The best response of player i against the cyclic strategy of all other players π -i cycle,M is the following:

v * i π -i cycle,M (s) = max π i t ,t∈N E +∞ t=0 γ t r i π i t ,π -i j (s t )|s 0 = s, s t+1 ∼ P π i t ,π -i j (.|s t ), j ≡ t (mod M ) . (5.2)
This value is the fixed point of operator

T * i π -i 0 . . . T * i π -i M -1
. The demonstration of this property was done in the context of zero-sum games in the previous chapter.

Nash and -Nash Equilibrium in N -Player MGs:

A canonical definition of a Nash equilibrium for a cyclic strategy is the following: if everyone commits to play the cyclic strategy, no players has an incentive to deviate unilaterally from his strategy. And, in an -Nash equilibrium, no player has more than an incentive to deviate unilaterally from their own strategy. Formally: Definition 5.1. In a MG a cyclic strategy π cycle,M is a Nash equilibrium if ∀i:

v i π cycle,M = v * i π -i cycle,M
. Definition 5.2. In a MG, a cyclic strategy π cycle,M is an -Nash equilibrium if:

v i π cycle,M -v * i π -i cycle,M s,∞ i,∞ ≤ .
Remark 5.1. This definition of a cyclic Nash equilibrium is consistent with the one of [START_REF] Scherrer | On the Use of Non-Stationary Policies for Stationary Infinite-Horizon Markov Decision Processes[END_REF] when reduced to an MDP. However, the definition of [START_REF] Zinkevich | Cyclic Equilibria in Markov Games[END_REF] is stronger when reduced to Nash equilibrium. It requires the definition 5.2 to be valid for every cyclic permutation of the cyclic strategy.

The Value Iteration Algorithm for General-Sum MGs 93 2 The Value Iteration Algorithm for General-Sum MGs

In this section, we define the VI algorithm for general-sum MGs. Then, we prove a bound of performance of 4γ m 1-γ which is rather poor compared to the bound for MDPs and zero-sum two-player MGs ( 2γ k 1-γ m ). However, this bound is tight up to a multiplicative constant as shown in App. 3. Finally, we empirically illustrate the use of non-stationary strategies in general-sum games.

The VI algorithm is a popular method used to find optimal or near optimal strategies for MDPs [START_REF] Bertsekas | Dynamic Programming and Optimal Control[END_REF] or zero-sum two-player MGs [START_REF] Shapley | Stochastic Games[END_REF][START_REF] Perolat | Approximate Dynamic Programming for Two-Player Zero-Sum Markov Games[END_REF]. However, it is known not to converge toward a Nash equilibrium for general-sum MGs. [START_REF] Zinkevich | Cyclic Equilibria in Markov Games[END_REF] suggest to look for cyclic strategies instead of stationary strategies. Here, we present a non-stationary version of the VI algorithm for general-sum MGs (a stationary version is described in [START_REF] Kearns | Fast Planning in Stochastic Games[END_REF]). , π k-m+1 ) which is a 4γ m Rmax 1-γ -Nash equilibrium. Proof. The proof is left in Appendix 7

Algorithm 22 Non-Stationary Value Iteration

Input: constant m, k and (v 1 0 , . . . , v N 0 ) = (0, . . . , 0) for l = 1 to k do 1 Find π l such as ∀i ∈ {1, . . . , N },T i π l v i l-1 = T * i π -i l v i l-1 (Computing a Nash equilib- rium in each state). 2 ∀i ∈ {1, . . . , N }, compute v i l = T i π l v i l-1 (
The performance of the non-stationary VI algorithm for MDPs is 2Rmaxγ k

(1-γ)(1-γ m ) which is way better than the bound for general-sum MGs since the number of iterations k comes into account. In MDPs the number of iterations controls the quality of the strategy. In MGs, if we define the performance criterion as the norm of Def. 5.2, the corresponding bound is 4γ m Rmax 1-γ (cf. Th. 5.1). Thus, in the general-sum case, the quality of the joint non-stationary strategy only depends on the length of the cycle. As a comparison, the guarantee would be the same if one would play first the m strategies (π k , . . . , π k-m+1 ) and then an arbitrary strategy. This bound seems limited since it means that we can do no better than planning m steps forward with the VI algorithm. We show in Section 3 that for all k in N and for all m in {1, . . . , k} there exists an MG such that

v i π k,m -v * i π -i k,m s,∞ i,∞ ≥ γ m 1-γ (with R max = 1)
. This example shows that our bound is tight up to a multiplicative constant. Still, Theorem 5.1 shows that one can compute a cyclic Nash equilibrium from value functions which is not the case of a stationary Nash equilibrium [START_REF] Zinkevich | Cyclic Equilibria in Markov Games[END_REF].

94

Chapter 5. On the use of non-stationary strategies 3 Illustrating example of lower bound

-k • • • -2 -1 0 1 • • • k -1 b a 0 0 0 0 d 0 h 0 d 0 h 0 0 1 0 0
(a) Reward of Player 1.

-k In this section, we build an artificial example of a general-sum MG for which the performance criterion (cf Def. 5.2) matches, up to a multiplicative constant, the upper bound of Th. 5.1. The example is shown in Fig. 5.1. It is a turn-based two-player general-sum MG with state space S = {a, b, -k, -k + 1, . . . , k -2, k -1}. In every state only one action is available to players but two states {0, 1} where each player has two actions. The rewards depend on the state and on the action. In state 0, Player 1 (Fig. 5.1(a)) has two actions A 1 (0) = {h, d}. In state -1, Player 2 (Fig. 5.1(b)) has also two actions A 2 (-1) = {h, d}.

• • • -2 -1 0 1 • • • k -1 b a 0 0 0 0 d 0 h 1 d 0 h 1 1 1 0 0 (b) Reward of Player 2.
Since the game is turn-based, the greedy step which implies computing a Nash equilibrium in each state reduces to finding an argmax. When the argmax is not unique, we always choose action h over action d. As the greedy strategy is deterministic, instead of writing π = (π 2 (.| -1), π 1 (.|0)) for the sake of clarity we write π = (a 2 , a 1 ) with π 2 (a 2 | -1) = 1 and π 1 (a 1 |0) = 1. The value function will be written (v(a), v(b), v(-k), . . . , v(k -1)).

VI instantiation

The VI algorithm produces the following sequence of values and strategies: Obviously v 1 0 = v 2 0 = (0, . . . , 0) Then, for 0 < l < k we have:

Illustrating example of lower bound

95

π l = (h, h) (5.3) v 1 l = (0, 0, 0, . . . , v 1 l (k -l) = γ l-1 -γ l 1 -γ , . . . , γ 0 -γ l 1 -γ ),
(5.4)

v 2 l = (0, 0, 0, . . . , v 2 l (1) = γ 0 -γ l 1 -γ , . . . , γ 0 -γ l 1 -γ ).
(5.5)

And for l = k, Player 1 has an incentive to switch from action h to action d. Thus, we have:

π k = (h, d) (5.6) v 1 l = (0, 0, 0, . . . , v 1 l (0) = γ k-1 -γ k 1 -γ , . . . , γ 0 -γ k 1 -γ ), (5.7) v 2 l = (0, 0, 0, . . . , v 2 l (0) = γ 0 -γ k 1 -γ , . . . , γ 0 -γ k 1 -γ ).
(5.8)

Values, and Values of Bests Responses: Let's compute the performance criterion of Def. 5.2 for l in {1, . . . , k}.

If l < k we have:

v 1 π l,m = (0, 0, 0, . . . , 0, γ k-2 1 -γ , . . . , 1 1 -γ ),
(5.9)

v * 1 π -1 l,m = (0, 0, 0, . . . , 0, γ k-1 1 -γ , . . . , 1 1 -γ ),
(5.10)

v 2 π l,m = (0, 0, 0, . . . , 0, v 1 π l,m (1) = 1 1 -γ , . . . , 1 1 -γ ), (5.11) v * 2 π -2 l,m = (0, 0, 0, . . . , 0, v * 2 π -2 l,m (1) = 1 1 -γ , . . . , 1 1 -γ
).

(5.12)

When l < k, the cyclic joint strategy is in fact a stationary one (h, h). Then, the joint strategy when Player 1 plays his best response is (h, d) and is (h, h) when Player 2 plays his best response (since both actions are equivalent for him).

Therefore, the performance criterion is:

∀l < k, v i π l,m -v * i π -i l,m s,∞ i,∞ = γ k-1 1 -γ .
When l = k:

96

Chapter 5. On the use of non-stationary strategies v 1 π l,m = (0, 0, 0, . . . , 0,

γ k-1 1 -γ , . . . , 1 1 -γ ), (5.13) v * 1 π -1 l,m
= (0, 0, 0, . . . , 0,

γ k-1 1 -γ , . . . , 1 1 -γ ), (5.14) v 2 π l,m = (0, 0, 0, . . . , 0, v 1 π l,m (0) = 1 1 -γ , . . . , 1 1 -γ ), (5.15) v * 2 π -2 l,m = (0, 0, 0, . . . 0, v * 2 π -2 l,m (-m) = γ m 1 -γ , 0, . . . , . . . , 0, v * 2 π -2 l,m (0) = 1 1 -γ , . . . , 1 1 -γ ), (5.16) ∀i ∈ N, v * 2 π -2 l,m (-im) = γ im 1 -γ .
(5.17)

In this case, the cyclic joint strategy is π l,m = ((h, d), (h, h), . . . , (h, h)). The Player 1's best response is the joint strategy ((h, d), . . . , (h, d)) and the Player 2' best is the joint strategy ((d, d), (d, h), . . . , (d, h)).

When l = k the performance criterion is:

v i π l,m -v * i π -i l,m s,∞ i,∞ = γ m 1 -γ .
Then ∀k ∈ N, ∀m ∈ {1, . . . , k}, there exists an MG (our example for instance) such that:

v i π k,m -v * i π -i k,m s,∞ i,∞ ≥ γ m 1 -γ .
Remark 5.2. The reader should notice that the size of the example matters. Indeed, if one runs the VI algorithm for l > k, one would find a Nash equilibrium of the MG (i.e. (d, d)).

The reason why this example gives the proper lower bound is that Player 1's actions control Player 2's rewards (in state 0). Thus, one can design an MG where the VI algorithm takes an arbitrary long time until Player 1 changes his action (that is the role played by states 1, . . . , k -1). Until Player 1 switch from action h to action d in state 0 (as we said before this time can be arbitrarily long) Player 2 does not see the difference between action h and action d in state 1. That is why the number of iterations k does not come into account in the bound.

Approximate Value Iteration

In this section, we consider the same approach as in previous chapters to analyse the impact of such an approximation on the performance is to consider two sources of errors arising at each step of VI as illustrated in Algorithm 23.

Theorem 5.2. The approximate NSVI algorithm produces a strategy π k,m which is an -Nash equilibrium. Where:

= 4γ m R max 1 -γ + 1 -γ k 1 -γ sup l∈{0,...,k-1} i k-l i,∞ + 2 γ -γ k 1 -γ sup l∈{1,...,k-1} i k-l i,∞ .
(5.18)

Experiments 97 Algorithm 23 Non Stationary Approximate Value Iteration

Input: constant m, K and (v 1 0 , . . . , v N 0 ) = (0, . . . , 0)

for k = 1 to K do 1 Find π k such as ∀i ∈ {1, . . . , N },T i π k v i k-1 + i k ≥ T * i π -i k v i k-1 . 2 ∀i ∈ {1, . . . , N }, compute v i k = T i π k v i k-1 + i k end for Return: (π K , . . . , π K-m+1 )
Proof. The proof is left in Appendix 8.

Experiments

We empirically illustrate the previous theoretical results on Garnets (Chapter 4). They are randomly generated synthetic MGs with three parameters (N S , N A , N B ). The parameter N S is the number of states, N A is the number of actions and N B is a parameter controlling the branching factor of the MDP. The process to generate a Garnet is the same as the one for two player zero-sum MGs except for the reward. Here, reward function only depends on the state and is drawn according to a centered normal law of variance 1. Only a given ratio (the sparsity) of rewards are non-null.

π -i k,m -v i π k,m 2 \ v * i π -i k,m
2 according to k. It is a normalized way to quantify what a player could win by switching his strategy unilaterally. From now on, we will refer to this quantity as the performance of the algorithm even though the lower the better. The envelope in figures 5.2(a) and 5.2(b) are proportional to the standard deviation of the performance measure.

The bound of theorem 5.1 does guarantee to improve the performance as m grows. Figure 5.2(a) shows the performance of the stationary strategy π k of Players 1 and 2 according to the number of iterations. One can notice that, in the beginning, the performance seems to decrease exponentially toward a asymptotic regime. Then there is no more improvement. Figure 5.2(b) presents the performance for the cyclic strategy of length 10 (π k , . . . , π k-9 ) at iteration k. For the non-stationary strategy the performance still converges toward some stationary regime but the performance on that regime is improved.

Then we compare the performance with respect to m while varying γ. According to the bound of Theorem 5.1, v * i .9, 0.95, 0.99} for Player 1. In the worst case, the logarithm of the expected performance should decrease as m log(γ). This is confirmed even in the case of randomly generated MGs. We applied approximate VI on Garnets to illustrate the empirical validity of the bound of Theorem 5.2. To do so, in Algorithm 23, we consider step 1 as exact, but in step 2 we introduce an error i k generated according to a centered Gaussian distribution of standard deviation σ = 0.1. For small lengths of the non-stationary strategy, the performance is lower when γ is small. For small cyclic strategies, the term 4γ m Rmax 1-γ is dominant and it seems that the behavior of the algorithm is the same as the one on Fig. 5.3(a). Then, in Fig. 5.3(b) the performance seems to saturate for long cycles. Indeed, for large m the dominant term is

π -i k,m -v i π k,
2 γ-γ k 1-γ sup l∈{1,...,k-1} i k-l i,∞ + 1-γ k 1-γ sup l∈{0,...,k-1} i k-l i,∞ while the performance is normalized by the norm of v * i π -i k,m
(which is proportional to 1 1-γ ). Thus the sensitivity to γ is reduced for large m and all curves converge toward the same value.

Conclusion

The use of cyclic strategies has been the focus of recent works in approximate Dynamic Programming (DP) on MDPs [START_REF] Scherrer | On the Use of Non-Stationary Policies for Stationary Infinite-Horizon Markov Decision Processes[END_REF] and in zero-sum two-player MGs (see the previous chapter). However, these work concentrate on reducing the sensitivity to the error propagated through iterations. In this chapter, we highlight their use in exact DP for general-sum MGs.

Our main contribution is to show that Value Iteration (VI) can still be used to solve general-sum MGs when considering non-stationary strategies. Our approach relies on the fact that an infinite horizon γ-discounted MG can be well approximated by a finite horizon γ-discounted MG. We solve a m-finite horizon γ-discounted MG with somewhat similar techniques as in [START_REF] Kearns | Fast Planning in Stochastic Games[END_REF] (which gives us a non-stationary strategy). Then, we show that playing this strategy in a cycle gives a satisfactory non-stationary solution to the infinite horizon γ-discounted problem. Finally, we exhibit an example showing that this guarantee can only be improved by a multiplicative constant. This example emphasizes a strong limitation of the use of cyclic strategies in VI for generalsum games. Indeed, the lower bound demonstrates that playing a cyclic strategy does not grant better guarantees than playing first the m strategies of VI and then playing any random strategy. From definition 5.2, we know that we should bound the following L ∞ -norm

v i π k,m -v * i π -i k,m s,∞ i,∞
.

In other words, we want to compare ∀i ∈ {1, . . . , N } the value of π k,m (i.e.

v i π k,m = T i π k . . . T i π k-m+1 v i π k,m
) with the value of the best response of player i to others' joint strategy

π -i k,m = (π -i k , . . . , π -i k-m+1 ) (i.e. v * i π -i k,m = T * i π -i k . . . T * i π -i k-m+1 v * i π -i k,m
). Let's define πi l the strategy such as

T i πi k ,π -i k . . . T i πi k-m+1 ,π -i k-m+1 v * i π -i k,m = T * i π -i k . . . T * i π -i k-m+1 v * i π -i k,m
. Then ∀i ∈ {1, . . . , N } we have:

0 ≤v * i π -i k,m -v i π k,m (5.19) = T * i π -i k . . . T * i π -i k-m+1 v * i π -i k,m -T i π k . . . T i π k-m+1 v i π k,m , (5.20) = T * i π -i k . . . T * i π -i k-m+1 v * i π -i k,m -T * i π -i k . . . T * i π -i k-m+1 v i k-m + T i π k . . . T i π k-m+1 v i k-m -T i π k . . . T i π k-m+1 v i π k,m , (5.21) = γP π k . . . γP π k-m+1 (v i k-m -v i π k,m ) + T i πi k ,π -i k . . . T i πi k-m+1 ,π -i k-m+1 v * i π -i k,m -T * i π -i k . . . T * i π -i k-m+1 v i k-m , (5.22) ≤ γP π k . . . γP π k-m+1 (v i k-m -v i π k,m ) + T i πi k ,π -i k . . . T i πi k-m+1 ,π -i k-m+1 v * i π -i k,m -T i πi k ,π -i k . . . T i πi k-m+1 ,π -i k-m+1 v i k-m , (5.23) ≤ γP π k . . . γP π k-m+1 (v i k-m -v i π k,m ) + γP πi k ,π -i k . . . γP πi k-m+1 ,π -i k-m+1 (v * i π -i k,m -v i k-m ).
(5.24)

Finally, ∀i ∈ {1, . . . , N } (noticing that the three quantities v * i

π -i k,m , v i k-m and v i π k,m are smaller than Rmax 1-γ ): v * i π -i k,m -v i π k,m ≤ 4γ m R max 1 -γ .
(5.25)

8 Proof of Theorem 5.2

Here we write (π i k , . . . , πi 1 ) such as

T * i π -i k . . . T * i π -i 1 v i 0 = T i πi k ,π -i k . . . T i πi 1 ,π -i 1 v i 0 . 0 ≤ v * i π -i k,m -v i π k,m ≤ v * i π -i k,m -T i π k v i k-1 + T i π k v i k-1 -v i π k,m , (5.26) ≤ v * i π -i k,m -T * i π -i k v i k-1 + i k + T i π k v i k-1 -v i π k,m , (5.27) ≤ v * i π -i k,m -T i πi k ,π -i k v i k-1 + i k + T i π k v i k-1 -v i π k,m .
(5.28)

With v i k-1 = T i π k-1 v i k-2 + i k-1 , with T i π k v i k-1 ≥ T * i π -i k v i k-1 -i k and
noticing that for all v ≤ v and for all π we have T i π v ≤ T i π v , then:

8. Proof of Theorem 5.2 101 v * i π -i k,m -v i π k,m ≤ v * i π -i k,m -T i πi k ,π -i k T * i π -i k-1 v i k-2 + T i π k T i π k-1 v i k-2 -v i π k,m + i k + γP πi k ,π -i k i k-1 -γP πi k ,π -i k i k-1 + γP π k i k-1 , (5.29) ≤ v * i π -i k,m -T i πi k ,π -i k T i πi k-1 ,π -i k-1 v i k-2 + T i π k T i π k-1 v i k-2 -v i π k,m + i k + γP πi k ,π -i k i k-1 -γP πi k ,π -i k i k-1 + γP π k i k-1 , (5.30) ≤ v * i π -i k,m -T i πi k ,π -i k . . . T i πi 1 ,π -i 1 v i 0 + T i π k . . . T i π 1 v i 0 -v i π k,m
(5.31)

+ k-1 j=0 γP πi k ,π -i k . . . γP πi k-j+1 ,π -i k-j+1 i k-j
(5.32)

- k-1 j=1 γP πi k ,π -i k . . . γP πi k-j+1 ,π -i k-j+1 i k-j
(5.33)

+ k-1 j=1 γP π k . . . γP π k-j+1 i k-j . (5.34) With ṽi k-m = T i π k-m . . . T i π 1 v i 0 , with vi k-m = T i πi k-m ,π -i k-m . . . T i πi 1 ,π -i 1 v i 0 , noticing vi k-m and ṽi k-m ≤ 1
1-γ and from the proof of Th. 5.1 we have:

v * i π -i k,m -T i πi k ,π -i k . . . T i πi k-m+1 ,π -i k-m+1 . . . T i πi 1 ,π -i 1 v i 0 + T i π k . . . T i π k-m+1 . . . T i π 1 v i 0 -v i π k,m (5.35) = T * i π -i k . . . T * i π -i k-m+1 v * i π -i k,m -T * i π -i k . . . T * i π -i k-m+1 vi k-m + T i π k . . . T i π k-m+1 ṽi k-m -T i π k . . . T i π k-m+1 v i π k,m , (5.36) ≤ 4γ m R max 1 -γ .
(5.37)

And the three sums (5.32), (5.33) and (5.34) are bounded by: (5.38) which concludes the proof.

2 γ -γ k 1 -γ sup l∈{1,...,k-1} i k-l i,∞ + 1 -γ k 1 -γ sup l∈{0,...,k-1} i k-l i,∞ ,

Background

First let us recall quickly the PI algorithm for MDP and the (Pollatschek and Avi-Itzhak's algorithm. The two algorithms perform similar updates are detailed in the following table (table 6.1):

π k such as T π k v k-1 = T * v k-1 µ k , ν k such as T v k-1 = T µ k v k-1 = T µ k ,ν k v k-1 = Tν k v k-1 v k = v π k v k = v µ k ,ν k
The algorithm of Pollatschek and Avi-Itzhak does not compute the best response at each iteration as the PI algorithm of Hoffman and Karp. Instead, it computes the value of a joint strategy. The complexity of computing the fixed point of T µ k+1 ,ν k+1 is just inverting a matrix of size card(S) instead of solving an MDP (as in the algorithm of Hoffman and Karp). However, this scheme doesn't work in general [START_REF] Van Der Wal | Discounted Markov Games: Generalized Policy Iteration Method[END_REF].

Let us recall some part of the related work described in Section 3.4. [START_REF] Van Der Wal | Discounted Markov Games: Generalized Policy Iteration Method[END_REF] shows that Shapley's algorithm is slower in practice than the (Pollatschek and Avi-Itzhak's algorithm and the Hoffman and Karp's PI. It thus motivates the use of PI schemes. Between the two extensions of PI to games, the Pollatschek and Avi-Itzhak's algorithm is the less computationally intensive. In [START_REF] Filar | On the Algorithm of Pollatschek and Avi-ltzhak[END_REF], this algorithm is slightly modified to introduce a learning rate. The update is v k+1 = αv µ k+1 ,ν k+1 + (1 -α)v k where α is chosen according to Armijo's Rule. This comes from the fact that the Pollatschek and Avi-Itzhak's algorithm is a Newton's method on the L 2 -norm of the Bellman residual ||v -T * v|| 2 2 (proof in Section 3.4.5). The adaptation of the Pollatschek and Avi-Itzhak's algorithm introduced by [START_REF] Filar | On the Algorithm of Pollatschek and Avi-ltzhak[END_REF] uses a quasi-Newton method instead of a Newton's method on the objective J (v) = ||v -T * v|| 2 2 . To argue for the convergence of their algorithm towards the optimal value of the game, Filar and Tolwinski show first that a local minimum of the objective is also a global minimum and then they assure that their quasi-Newton method converges to a local minimum. However, they use a version of the Zoutendijk theorem [START_REF] Nocedal | Numerical Optimization[END_REF] to prove the convergence to a local minimum. This theorem requires the gradient of the objective function (L 2 -norm of the Bellman residual) to exists (in the Fréchet sense) and to be a Lipschitz function. This assumption does not hold in the case of MDPs or MGs because of the max and minmax operators respectively. The question whether there is convergence to a local minimum or not when the gradient is not Lipschitz is still an open problem in optimization. However, empirical evidence suggests quasi-Newton methods always converge to a Clarke stationary point even when the gradient is not Lipschitz Lewis andOverton (2009, 2013). This means that, despite Pollatschek and Avi-Itzhak's proof does not stand, there is good evidence that using quasi-Newton methods will empirically perform well. Again, there are no theoretical guarantees that quasi-Newton methods converge to a local optimum in the case of 2. Newton's Method on the OBR with Linear Function Approximation 107 non-convex and non-smooth functions [START_REF] Lewis | Nonsmooth Optimization via Quasi-Newton Methods[END_REF] (which is the case of the Bellman residual). But, as written by Lemaréchal it can be "good practice to use a quasi-Newton method in nonsmooth optimization" in his opinion it "is essentially due to the fact that inaccurate line-searches are made". He also indicates that there are "no theoretical possibility to prove convergence to the right point (in fact counterexamples exist)" (see [START_REF] Lewis | Nonsmooth Optimization via BFGS[END_REF]).

Newton's method and quasi-Newton methods: Newton's method is an optimization technique aiming at minimizing a function f : R n → R. Starting from x 0 ∈ R n , it computes the sequence (x n ) n∈N such that:

x n+1 = x n -[Hf (x n )] -1 ∇f (x n ), (6.1)
where Hf (x n ) and ∇f (x n ) are respectively the Hessian matrix and the gradient in x n . This method might be unstable [START_REF] Nocedal | Numerical Optimization[END_REF] and one way to soften it is to introduce a learning rate α k meeting the Wolf conditions and to compute the sequence:

x n+1 = x n -α k [Hf (x n )] -1 ∇f (x n ). (6.2)
This is a quasi-Newton method. When the function f is not differentiable (in the classical Fréchet sense), one can use more general definitions of differential (or gradient) such as the Clarke differential cla for instance. In Section 2, we discuss our choice of differential for our specific objectives which are not differentiable everywhere due to the use of the minmax operator.

Newton's Method on the OBR with Linear Function Approximation

As in Section 3.5.1 and in Section 1.2.2, in Chapter 2 Q-functions are represented as a linear combination of d linear independent features Φ = [φ 1 , φ 2 , ..., φ d ], where

φ i ∈ R S×A 2 . Thus, for each ω ∈ R d , we can define a Q-function Q ω = d i=1 ω i φ i = Φω.
We also define the linear span corresponding to Φ as Span(Φ) = {Q ω } ω∈R d . Moreover, it is usual to see Φ as a matrix where Φ [(s, a, b), i] = φ i (s, a, b). Thus, we have Q ω = Φω. In addition, we recall that for any element Q of R S×A 2 , its orthogonal projection Π ρ,Φ Q under a non-null measure ρ over S × A 2 is defined as:

Π ρ,Φ Q = argmin u∈Span(Φ) Q -u 2,ρ = Φ(Φ ∆ ρ Φ) -1 Φ ∆ ρ Q, (6.3)
Two canonical objectives to minimize can be defined in order to search for a Qfunction within Span(Φ): the OBR J OBR (ω) and the POBR J P OBR (ω): (6.4) Notice that explicit minimization of the OBR through gradient descent is not new (Piot et al., 2014a[START_REF] Baird | Residual Algorithms: Reinforcement Learning with Function Approximation[END_REF], as for the POBR with approximate stochastic gradient descent [START_REF] Maei | Toward Off-Policy Learning Control with Function Approximation[END_REF]. However, few works, at our knowledge, focus on the use of Newton's or quasi-Newton techniques to minimize those objectives. The Newton's method requires computing a gradient and a Hessian matrix. This means that one needs to compute the derivative of T * Φω with respect to ω.

J OBR (ω) = 1 2 ||Φω -T * Φω|| 2 2,ρ ,
J P OBR (ω) = 1 2 ||Φω -Π ρ,Φ T * Φω|| 2 2,ρ , (6.5) = 1 2 ||Φ (Φ ∆ ρ Φ) -1 Φ ∆ ρ (Φω -T * Φω)||
Computing the gradient of T * Φω with respect to ω As shown by [START_REF] Correa | Directional Derivative of a Minimax Function[END_REF], when for a fixed ω, there exists a unique pair of strategies (µ ω , ν ω ) such that T * Φω = T µω,νω Φω = T µω Φω = Tνω Φω, the gradient of T * Φ on ω, is simply the gradient of the linearized operator T µω,νω Φ = r + γP µω,νω Φ. From now on, we will note P µω,νω = P ω and T µω,νω = T ω . When there exist U ω and V ω such that, for all µ in U ω and for all ν in V ω , T * Φω = T µ,ν Φω = T µ Φω = Tν Φω, then the directional derivative of T * Φω (written [START_REF] Correa | Directional Derivative of a Minimax Function[END_REF]. In practice, when the minmax is not unique, a pair of strategies (µ ω , ν ω ) is chosen in U ω × V ω and T * Φ is linearized with T ω Φ to compute the gradient:

∂ d T * Φω) is min ν∈Vω max µ∈Uω ∂ d T µ,ν Φ
∇T * Φω = ∇T ω Φω = γP ω Φ.
For example, in the Pollatschek and Avi-Itzhak's algorithm (Φ = I), a couple (µ ω , ν ω ) is chosen such that T * Φω = T ω Φω = T µω Φω = Tνω Φω which is the pair of strategies for which the chosen gradient is γP ω Φ. In LSPI for games, the gradient is not the gradient of the linearized operator T ω . Rather, they choose T µ,ν such that T * Φω = T µ,ν Φω = T µ Φω and with ν a deterministic strategy. Thus, the LSPI algorithm as it has been described by [START_REF] Lagoudakis | Value Function Approximation in Zero-Sum Markov Games[END_REF] does not follow the gradient γP ω Φ. From now, we will call the Newton-LSPI algorithm the one with the Newton's gradient γP ω Φ. The reader should notice that this difference only appears when it comes to MGs and that Newton-LSPI is exactly LSPI when applied to MDPs since ν is not considered in MDPs. This also proves that LSPI for MGs is almost the (Pollatschek and Avi-Itzhak's algorithm when Φ = I as it almost perform the same greedy step. Interestingly, the example of Van Der Wal also applies and thus LSPI is not guaranteed to converge in MGs whether the function approximation is stable or not.

Newton's Method on the POBR

As mentioned in the previous section, the Newton's method on the POBR reduces to the LSPI algorithm but with a slightly different choice of the gradient. Here, we will consider the POBR with the linearized gradient (∇T * Φω = ∇T ω Φω = γP ω Φ) described previously. Let's write

A ω = Φ ∆ ρ (I -γP ω k )Φ and b = Φ ∆ ρ r
The POBR gradient is (for details see appendix 8):

∇J P OBR (ω) = A ω (Φ ∆ ρ Φ) -1 (A ω ω -b). (6.7)
The Hessian matrix is (for details see appendix 8):

HJ P OBR (ω) = A ω (Φ ∆ ρ Φ) -1 A ω . (6.8)

Newton's Method on the OBR with Linear Function Approximation 109

If the matrix A ω is invertible and if ρ is a non-null measure on S × A 2 , the Newton's method direction is:

[HJ P OBR (ω)] -1 ∇J P OBR (ω) = (A ω ) -1 (A ω ω -b).
(6.9)

The update of ω k of the Newton's method is thus:

ω k+1 = ω k -(A ω k ) -1 (A ω k ω k -b) = (A ω k ) -1 b,
which is the update of the Newton-LSPI algorithm [START_REF] Lagoudakis | Least-Squares Policy Iteration[END_REF]. The Newton-LSPI algorithm optimizes J P OBR (ω) as a cost function, which might never be zero (see Remark 7.1 below). Moreover, to our knowledge, controlling the POBR does not allow controlling the quality of the final policy. The main benefit of the Newton-LSPI algorithm is that the matrix A ω k is easy to estimate from batch data whether the state space is continuous or not. This makes Newton-LSPI a very practical algorithm [START_REF] Lagoudakis | Value Function Approximation in Zero-Sum Markov Games[END_REF].

It is well known that, for a fixed policy, the minimum of the projected Bellman

residual 1 2 ||Φ (Φ ∆ ρ Φ) -1 Φ ∆ ρ (Φω -T π Φω)|| 2
2,ρ is equal to 0 Koller and Parr ( 2000) for all but finitely many number of γ. Nonetheless, we can show that it is not the case for the POBR. In other words, the minimum of

1 2 ||Φ (Φ ∆ ρ Φ) -1 Φ ∆ ρ (Φω-T * Φω)|| 2 2,ρ
might be positive for a continuous interval of γ (see appendix 7.1).

Newton's Method on the OBR

Here we will note C ω = Φ (I -γP ω ) ∆ ρ (I -γP ω )Φ and e ω = Φ (I -γP ω ) ∆ ρ r.

Applying the Newton's method to the residual J OBR (ω) gives the following algorithm:

ω 0 = 0, (6.10
)

ω k+1 = ω k + (HJ OBR (ω k )) -1 ∇J OBR (ω k ). (6.11)
The gradient is (for details see appendix 8):

∇J OBR (ω) = C ω ω -e ω .
The Hessian matrix is (for details see appendix 8):

HJ OBR (ω) = C ω .
The update of ω k of the Newton's method is: [START_REF] Lagoudakis | Least-Squares Policy Iteration[END_REF] propose this algorithm as an alternative method to learn an optimal policy in MDPs and call it Bellman Residual Minimization Policy Iteration (BRMPI). Extension to games is easy as shown previously.

ω k+1 = ω k -(C ω k ) -1 (C ω ω k -e ω k ), = (C ω k ) -1 e ω k .

Comparison of BRMPI and Newton-LSPI

In MGs, the two algorithms previously described derive from the Pollatschek and Avi-Itzhak algorithm and are thus not guaranteed to converge [START_REF] Van Der Wal | Discounted Markov Games: Generalized Policy Iteration Method[END_REF]. They both follow the Newton's direction of either J OBR or J P OBR . On the one hand, the cost function J OBR (ω) controls the norm ||Q * -Q µ || 2 where µ is the maximizer's greedy strategy with respect to Φω and Q µ the fixed point of T µ (see Piot et al. (2014a) in the case of an MDP). In consequence, minimizing the OBR gives a nearly optimal strategy. However, there are no easy ways to estimate the matrix C ω and the vector e ω from data, especially when the state space is continuous. One possible way to obtain such estimates is to use embeddings in RKHS [START_REF] Grunewalder | Modelling Transition Dynamics in MDPs With RKHS Embeddings[END_REF]. Theorem 6.1. Let Q be a value function and let µ be a greedy strategy on Q (i.e. T * Q = T µ Q). Then we have:

||Q * -Q µ || 2,ρ ≤ 2 1 -γ C 2 (ρ, µ, ν) + C 2 (ρ, µ * , ν) 2 1 2 ||T * Q -Q|| 2,ρ
with µ, ν, µ * and ν strategies such that

T * Q * = T µ * Q * , T µ * ,ν Q = T µ * Q, T µ,ν Q µ = T µ Q µ and T µ,ν Q = T µ Q. And with C 2 (ρ, µ, ν) = || ∂ρ (1-γ)(I-γPµ,ν ) -1 ∂ρ || 2,ρ L ∈ -norm of the Radon-Nikodym derivative of the Kernel (1 -γ)(I -γP µ,ν ) -1 .
Proof. Proof in appendix 7.

On the other hand, applying the Newton's method to J P OBR (ω) is easy. Estimating A ω and b ω is computationally cheap. However, minimizing J P OBR (ω) does not guarantee to find a good strategy.

Concerning the stability, Newton-LSPI is clearly the less stable one. The Hessian matrix HJ P OBR (ω) might not even be invertible since A ω is not invertible for a finite number of values of γ [START_REF] Koller | Policy Iteration for Factored MDPs[END_REF]. For BRMPI however the Hessian HJ OBR (ω) is always invertible and the cosine between the Newton's update and the gradient is bounded away from zero. Theorem 6.2. For an MG with finite state space, there exists δ > 0 such that:

- J OBR (ω) (HJ OBR (ω)) -1 J OBR (ω) ||J OBR (ω)|| 2 ||(HJ OBR (ω)) -1 J OBR (ω)|| 2 ≤ -δ.

Proof. Proof in appendix 7

This means that, when on a point ω where the derivative is well defined, the Newton's update is guaranteed to decrease the OBR in a neighborhood of ω.

Batch Learning in Games

In the batch scenario, the dynamics and the reward function are known only through logs of interactions between the two players and the environment. These logs are a 4. Quasi-Newton's Method on the OBR and on the POBR 111 collection of tuple {(s i , a 1 i , a 2 i , r i , s i )} i∈{1,...,N } where (s i , a 1 i , a 2 i ) is drawn from a distribution ρ where r i = r(s i , a 1 i , a 2 i ) and where s i ∼ p(.|s i , a 1 i , a 2 i ). Thus, at each iteration of Newton-LSPI or of BRMPI, the objective will be to estimate one matrix and one vector.

Newton-LSPI with Batch Data

Newton-LSPI with batch data proceeds as follows. The goal is to estimate the matrix A ω and the vector b (the estimates will be noted Âω and b). The procedure is described in Algorithm 24 where µ ω and ν ω are the minmax strategies with respect to the approximate Q-function Φω.

The update of Âω is convenient since it allows computing ( Âω ) -1 very efficiently [START_REF] Lagoudakis | Least-Squares Policy Iteration[END_REF] with the Sherman-Morrison formula. Then doing Newton-LSPI with batch data corresponds simply in updating our parameter ω as follows:

ω k+1 = ( Âω k ) -1 b. (6.12)
Algorithm 24 LSPI-Matrix update

Input: D N = ((x j , a 1 j , a 2 j ), r j , x j ) j=1,.
..,N some samples, Φ a d dimensional feature space and ω.

for i ∈ {1, ..., N } do Âω + = φ(s i , a 1 i , a 2 i )[φ(s i , a 1 i , a 2 i )-γ b 1 ∈A 1 (s i ) b 2 ∈A 2 (s i ) µ ω (s i , b 1 )ν ω (s i , b 2 )φ(s i , b 1 , b 2 )] b + = φ(s i , a 1 i , a 2 i )r i end for output: Âω , b

BRMPI with Batch Data

A natural extension of BRMPI to batch data when the state space is finite is to find estimates of C ω and e ω . To do so we first estimate (I -γP ω ) (noted Bω ) and r (noted r). The procedure is described in Algorithm-25. Then, estimates of C ω and e ω would be respectively Ĉω = Φ B ω Bω Φ and êω = Φ B ω r. Those estimates are clearly biased estimates of C ω and e ω . Thus, the Newton's update is:

ω k+1 = ( Ĉω k ) -1 êω k . (6.13)
4 Quasi-Newton's Method on the OBR and on the POBR

In this section, a quasi-Newton's method is applied to the norm of the OBR and of the PORBR. This corresponds to the introduction of a learning rate in Newton-LSPI and

Algorithm 25 BRMPI-Matrix update

Input: D N = ((x j , a 1 j , a 2 j ), r j , x j ) j=1,...,N some samples, Φ a d dimensional feature space and ω.

for i ∈ {1, ..., N } do Bω (s i , a 1 i , a 2 i , s i , a 1 i , a 2 i ) + = 1 for b 1 ∈ A 1 (s i ) and b 2 ∈ A 2 (s i ) do Bω (s i , a 1 i , a 2 i , s i , b 1 , b 2 )+ = -γµ ω (s i , b 1 )ν ω (s i , b 2 ) end for r(s i , a 1 i , a 2 i ) ← r(s i , a 1 i , a 2 i ) + r i end for
BRMPI and their batch versions described in the herein-before section. In [START_REF] Filar | On the Algorithm of Pollatschek and Avi-ltzhak[END_REF] the learning rate is chosen to verify Amijo's rule for the exact case (Amijo's rule is satisfied when the first Wolf condition is satisfied eq. (6.14)). We choose to use a learning rate according to the Wolf conditions for the case with linear function approximation. There exist theorems to guarantee convergence to a local minimum when the gradient is a Lipschitz function which is not the case here. However, there are empirical evidences that using inexact line search improves convergence to a local minimum of the objective function [START_REF] Lewis | Nonsmooth Optimization via BFGS[END_REF].

In both Newton-LSPI and BRMPI, when the dynamics is known and in the batch scenario, the update takes the following shape:

ω k+1 = (Ξ ω k ) -1 ψ ω k
and the objective is to minimize a L 2 norm of the following shape (the corresponding notations is given in table 6.2):

f (ω) = Ψ ω ω -υ ρ,2 .
Let's note the Newton's update:

g ω k = (Ξ ω k ) -1 ψ ω k -ω k ,
The quasi-Newton method will instead perform the following update:

ω k+1 = (1 -α k )ω k + α k (Ξ ω k ) -1 ψ ω k ,
where the learning rate α is chosen to satisfy the Wolf conditions (0 < c 1 < c 2 < 1):

f (ω + αg ω ) ≤ f (ω) + c 1 αg ω .∇f (ω) (6.14) f (ω + αg ω ) ≥ f (ω) + c 2 αg ω .∇f (ω) (6.15)
Constants c 1 is typically chosen around 10 -4 and c 2 is usually chosen close to 0.9. Condition (6.14) ensures that the learning rate is not too large whereas condition (6.15) guarantees a large enough learning rate.

ω k A ω k C ω = Φ B ω B ω Φ Âω k Ĉω = Φ B ω Bω Φ ψ ω k b e ω = Φ B ω r b êω = Φ B ω r Ψ ω Φ(Φ ∆ ρ Φ) -1 A ω B ω Φ Âω Bω Φ υ Φ(Φ ∆ ρ Φ) -1 b r b r ρ ρ ρ uniform uniform

Experiments

To empirically illustrate the application of quasi-Newton methods to both MDPs and MGs, we ran experiments on Garnets (defined in Chapter 4). A Garnet is an abstract class of MDPs. It is described by a triplet (N S , N A , N B ). The parameter N B , the branching factor, denotes the number of reachable states from any state-action pair.

In one of our experiment, we enforced some regularity in the dynamics. Usually (as described in Chapter 4), for all (s, a) we generate the following transition distribution p(.|s, a) according to the following procedure: first one draws uniformly N B -1 points over [0, 1]. Let's note those numbers (p i ) 1≤i≤N B -1 (in increasing order) and p 0 = 0 and p N B = 1. Then we draw a subset of size N B of {1, ..., N S } written {s 1 , ..., s N B }. The second type of Garnet, {s 1 , ..., s N B } is drawn in a subset of {1, ..., N S } centered around s (meaning we will draw states s such that |s -

s | ≤ ζ in the experiment ζ = N B ).
Finally, we define the kernel as follows, ∀i ∈ {1, ..., N B }, p(s i |s, a) = p i -p i-1 . The reward function will depend on the experiment.

For simultaneous two-player zero-sum MGs the N A parameter will describe the number of actions both players will be allowed to play at each state in the MG. The kernel p(.|a 1 , a 2 , s) is generated according to the same procedure as for MDPs.

Results always show the performance of the strategy with respect to the iteration number. The performance of a strategy µ is quantified as the following ratio: ||Q * -Qµ|| 2 ||Q * || 2 . This is the normalized norm of the difference between the optimal Q-function of the MG and the Q-function of strategy µ considering the opponent plays his best response. Results are averaged over experiments and the envelope is proportional to the standard deviation.

The main purpose of the two following subsections is to emphasize the effects of quasi-Newton methods both on the stability and on the performance of the algorithms.

In those experiments, we can notice the importance of tuning constants c 1 and c 2 from equations (6.14) and (6.15). Here, c 1 = 10 -4 and c 2 = 0.9. We performed the line search as follows: if condition (6.14) was not checked we decreased the learning rate geometrically α k ← η × α k and, if condition (6.15) was not checked, we increased it geometrically α k ← 1 η ×α k (η = 0.9). This was done until both conditions were checked.

were non-zero (this ratio is called the sparsity). Experiments of figure 6.1 was done on type 2 Garnets (with a regular dynamic). The feature space Φ has d = 0.2 × N S × N A features. Those d features are vectors randomly generated according to a Gaussian law.

First, one can notice that quasi-Newton version of LSPI and respectively BRMPI did not under-perform LSPI and respectively BRMPI. Figure 6.1 illustrates that quasi-Newton methods do not under perform their respective counterparts. We noticed that the BRMPI algorithm often oscillates from one iteration to another. The use of quasi-Newton method usually eliminates that instability.

In Figure 6.1 the Garnet is drawn with N B = 10. Usually, MDPs with high branching factors are easier to optimize since a high branching factor has a tendency to smooth the optimal value function. Figure 6.1 shows the effect of the learning rate on the learning curve. Especially in the case of Newton-LSPI, introducing a learning rate improves both the stability and the performance of the algorithm. Also, we can notice that the BRMPI algorithm oscillates on MDPs.

Experiments on Markov Games

We ran experiments on MG Garnets to compare Newton's method and quasi-Newton method. In the hereafter experiment, the reward function only depends on the state. Only a ratio of rewards are non-zero and are drawn according to a normal law. Again, we compared Newton-LSPI and BRMPI with their quasi-Newton counterparts. The feature space is also randomly generated according to a Gaussian law. But, we needed more features compared to the MDP case to learn a strategy. In the following experiment, we used d = 0.8 × N A × N A × N S features.

Compared to experiments on MDPs, one can notice that the variance of the performance is higher even if we use more features, more samples on MGs with less actions. We do not fully understand why it is significantly harder to learn a good strategy in an MG compared to an MDP. The reason might be that simultaneous games are simply more complex to optimize or that comparing with the value of a best response is too conservative. Anyway, the use of quasi-Newton method appears to be useful even with this conservative performance criterion.

Here again, one can notice that quasi-Newton methods did not under-perform their counterparts. Although, we could not notice huge gaps in the performance (Figure 6.2 and 6.3) as in experiments on MDPs, quasi-Newton methods did reduce the unstable behavior of the strategy (Figure 6.2).

Conclusion

To sum up this chapter, we first pointed out the fact that the proof of convergence of Filar and Tolwinski's algorithm is based on assumptions that do not hold. Second, we found out that, as a consequence of the instability of the Pollatschek and Avi-Itzhak algorithm, LSPI for games does not converge whether the linear approximation is stable or not. Third, we showed that LSPI and BRMPI can be regarded as Newton's methods. Fourth, this naturally led to the use of quasi-Newton methods instead of Newton's method as they result in more stable solutions. And, finally, these slight modifications on algorithms for games dramatically improve the stability of one of the most popular model-free algorithms to solve MDPs on synthetic problems, namely LSPI. We make novel connections between approximate policy iteration schemes on both MDPs and MGs and optimization methods. It describes three algorithms as Newton's method on different Bellman residuals. This unified picture of three popular algorithms naturally led to the use of quasi-Newton methods which is believed, in the optimization community, to be steady and to perform better in practice [START_REF] Nocedal | Numerical Optimization[END_REF]).

Yet, this chapter rises three open questions: first, since LSPI and BRMPI with batch data are minimizing some empirical residual, a natural question is whether those estimators are good estimators of the residual or not. Few works exist on direct min-

× N A × N A × N S .
imization of the OBR [START_REF] Baird | Residual Algorithms: Reinforcement Learning with Function Approximation[END_REF], Piot et al., 2014a). Piot et al. proved that minimizing a specific estimate of the Optimal Bellman residual on MDPs is consistent in the Vapnik sense. A second question is whether it makes sense theoretically to minimize the POBR or not. Little is known on the quality of the solution given by the minimum of the POBR and research should be conducted in that sense. Finally, since the goal is to minimize some Bellman residual, a third question is whether there exist smarter methods than quasi-Newton ones to minimize the considered residual. There exists a trove of Newton-like methods that are known to perform well on non-smooth functions. One interesting perspective would be to study how the BFGS method [START_REF] Nocedal | Numerical Optimization[END_REF]Wright, 2006, Lewis and[START_REF] Lewis | Nonsmooth Optimization via Quasi-Newton Methods[END_REF] would perform, as it does not require to compute and invert the Hessian matrix at each iteration.

But in the end, minimizing the Bellman residual can only be used in zero-sum games as there exists no known Bellman operator for general sum MGs and as no value function based method can find a Nash equilibrium [START_REF] Zinkevich | Cyclic Equilibria in Markov Games[END_REF].

Appendix

Remark

The minimum of POBR J P OBR (ω) might not be 0. Indeed, let us consider the following simple example on an MDP with only two actions:

r = 0 1 , P a 1 = 1 2 1 2 0 1 , P a 2 = 1 2 1 2 1 0 , Φ = 1 2
One can notice that, if ω ≥ 0, then a 1 is greedy with respect to Φω. If ω ≤ 0, then a 2 is greedy with respect to Φω.

(Φ Φ) -1 Φ (T a 1 Φω -Φω) = 1 5 (2 + (5.5γ -5)ω) (Φ Φ) -1 Φ (T a 2 Φω -Φω) = 1 5 (2 + (3.5γ -5)ω)

Appendix 119

Then, for γ ∈ [ 10 11 , 1], the value of (Φ Φ) -1 Φ (T * Φω-Φω) is a piecewise linear function. It is decreasing when ω ≤ 0 and increasing when ω ≥ 0. For ω = 0 it is equal to 2 5 . Thus J P OBR (ω) is positive (J P OBR (ω) ≥ 4 25 ).

Proof of Theorem 6.1

Let us recall theorem 6.1 Let Q be a value function and let µ be a greedy strategy on v (meaning

T * v = T µ v). Then we have: ||Q * -Q µ || 2 ≤ C 1 -γ ||Q -T * Q|| 2 Proof. Q * -Q µ = T * Q * -T * Q + T µ Q -T µ Q µ (6.16) ≤ T µ * Q * -T µ * Q + T µ,ν Q -T µ,ν Q µ (6.17) ≤ T µ * ,ν Q * -T µ * ,ν Q + T µ,ν Q -T µ,ν Q µ (6.18) (6.19) with T * Q * = T µ * Q * , T µ * ,ν Q = T µ * Q and T µ,ν Q µ = T µ Q µ . We will note µ the strategy such that T µ,ν Q = T µ Q (I -γP µ * ,ν )(Q * -Q µ ) ≤ (γP µ * ,ν -γP µ,ν )(Q -Q µ ) (6.20) Now let's bound Q -Q µ Q -Q µ = Q -Q µ,ν = Q -(I -γP µ,ν ) -1 r (6.21) = (I -γP µ,ν ) -1 (Q -γP µ,ν Q -r) (6.22) = (I -γP µ,ν ) -1 (Q -T µ,ν Q) (6.23) ≤ (I -γP µ,ν ) -1 (Q -T µ Q) (6.24) ≤ (I -γP µ,ν ) -1 (Q -T * Q) (6.25)
and:

Q µ -Q ≤ Q µ,ν -Q ≤ (I -γP µ,ν ) -1 (T µ,ν Q -Q) (6.26) ≤ (I -γP µ,ν ) -1 (T * Q -Q) (6.27) Finally: Q * -Q µ ≤ [(I -γP µ,ν ) -1 + (I -γP µ * ,ν ) -1 ]|T * Q -Q| with C 2 (ρ, µ, ν) = || ∂ρ (1-γ)(I-γPµ,ν ) -1 ∂ρ || 2,ρ the Radon-Nikodym derivative of the Kernel (1 -γ)(I -γP µ,ν ) -1 .
Then we have:

||Q * -Q µ || 2,ρ ≤ 2 1-γ C 2 (ρ,µ,ν)+C 2 (ρ,µ * ,ν) 2 1 2 ||T * Q -Q|| 2,ρ 120
Chapter 6. Bellman Residual Minimization in Zero-Sum Games

Proof of Theorem 6.2

Let us recall theorem 6.2: For a MG with a finite state space, there exists δ > 0 such that:

- J OBR (ω) (HJ OBR (ω)) -1 J OBR (ω) ||J OBR (ω)|| 2 ||(HJ OBR (ω)) -1 J OBR (ω)|| 2 ≤ -δ Proof. If γ ∈ [0, 1[, then HJ (ω) is invertible and thus it is a definite positive symmet- ric matrix. Let's note L 1 2
ω the root square of (HJ OBR (ω)) -1 (then (HJ OBR (ω)

) -1 = L 1 2 ω L 1 2 ω = L ω ). Then: - J OBR (ω) (HJ OBR (ω)) -1 J OBR (ω) ||J OBR (ω)|| 2 ||(HJ OBR (ω)) -1 J OBR (ω)|| 2 = - ||L 1 2 ω J OBR (ω)|| 2 2 ||J OBR (ω)|| 2 ||L ω J OBR (ω)|| 2 (6.28)
We have:

||J OBR (ω)|| 2 = ||L -1 2 ω L 1 2 ω J OBR (ω)|| 2 ≤ ||L -1 2 ω || 2 ||L 1 2 ω J OBR (ω)|| 2 (6.29)
And:

||L ω J OBR (ω)|| 2 = ||L 1 2 ω L 1 2 ω J OBR (ω)|| 2 ≤ ||L 1 2 ω || 2 ||L 1 2 ω J OBR (ω)|| 2 (6.30) Then:
-||L

1 2 ω J OBR (ω)|| 2 2 ||J OBR (ω)|| 2 ||L ω J OBR (ω)|| 2 ≤ -1 ||L 1 2 ω || 2 ||L -1 2 ω || 2 Furthermore with ||.|| 2 ≤ c 1 ||.|| ∞ (here c 1 = 2 |S| × |A| 2 ) and with noticing 1 -γ ≤ ||(I -γP ω )|| ∞ ≤ 1 + γ: ||L -1 2 ω || 2 = ||∆ 1 2 ρ (I -γP ω )Φ|| 2 ≤ ||∆ 1 2 ρ || 2 ||(I -γP ω )|| 2 ||Φ|| 2 ≤ c 1 (1 + γ) 2 ρ max λ Φ max
Where λ Φ max is the largest eigenvalue of Φ Φ, and ρ max is the maximum of ρ WithFilar and Tolwinski (1991) we have that ||∆

||L 1 2 ω || 2 = sup ω ω (Φ (I -γP ω ) ∆ ρ (I -γP ω )Φ) -1 ω ω ω = 1 inf ω ω Φ (I-γPω) ∆ρ(I-γPω)Φω ω ω But: inf ω ω Φ (I -γP ω ) ∆ ρ (I -γP ω )Φω ω ω ≥ inf X X (I -γP ω ) ∆ ρ (I -γP ω )X X X inf ω ω Φ Φω ω ω ≥ 1 ||(∆ 1 2 ρ (I -γP ω )) -1 || 2 2 λ Φ min
-1 2 ρ (I -γP ω ) -1 || 2 ≤ c 1 (1-γ) 2 √ ρ min

Computation of the Gradient and of the Hessian 121

Then:

||L 1 2 ω || 2 ≤ c1 (1 -γ) 2 ρ min λ Φ min
Finally we have:

- J OBR (ω) (HJ OBR (ω)) -1 J OBR (ω) ||J OBR (ω)|| 2 ||(HJ OBR (ω)) -1 J OBR (ω)|| 2 ≤ - 1 -γ c 2 1 (1 + γ) 2 ρ min λ Φ min ρ max λ Φ max

Computation of the Gradient and of the Hessian

Let's remind the reader:

J OBR (ω) = 1 2 ||Φω -T * Φω|| 2 2,ρ , (6.31) = 1 2 (Φω -T * Φω) ∆ ρ (Φω -T * Φω), (6.32) J P OBR (ω) = 1 2 ||Φω -Π ρ,Φ T * Φω|| 2 2,ρ , (6.33) = 1 2 ||Φ(Φ ∆ ρ Φ) -1 Φ ∆ ρ (Φω -T * Φω)|| 2 2,ρ , (6.34) = 1 2 Φ(Φ ∆ ρ Φ) -1 ∆ ρ Φ ∆ ρ (Φω -T * Φω) ∆ ρ Φ(Φ ∆ ρ Φ) -1 Φ ∆ ρ (Φω -T * Φω) (6.35) ∇J P OBR (ω) = ∂ ∂ω Φ(Φ ∆ ρ Φ) -1 ∆ ρ Φ ∆ ρ (Φω -T * Φω) ∆ ρ Φ(Φ ∆ ρ Φ) -1 Φ ∆ ρ (Φω -T * Φω) (6.36) = Φ (I -γP ω ) ∆ ρ Φ(Φ ∆ ρ Φ) -1 Φ ∆ ρ Φ(Φ ∆ ρ Φ) -1 Φ ∆ ρ (Φω -T * Φω) (6.37) = Φ (I -γP ω ) ∆ ρ Φ(Φ ∆ ρ Φ) -1 Φ ∆ ρ (Φω -T * Φω) (6.38) = Φ (I -γP ω ) ∆ ρ Φ(Φ ∆ ρ Φ) -1 Φ ∆ ρ (Φω -T ω Φω) (6.39) = Φ (I -γP ω ) ∆ ρ Φ(Φ ∆ ρ Φ) -1 Φ ∆ ρ ((I -γP ω )Φω -r) (6.40)
Then obviously: Exact methods to solve general-sum MGs have been developed in the past two decades. These methods are generally hopelessly difficult when more than two players are interacting. Thus, extending exact algorithms to the batch scenario and to function approximation seems to be a risky path as all exact algorithms require too much computation. Our approach is to reduce the problem to the minimization of a loss function over the polices and over the state action value functions. This loss is a sum over players of two Bellman residuals. This loss is non convex and thus, the method is not guaranteed to converge when the model is known and when no function approximations are used. However, our method can be efficiently estimated with samples and support approximation both on the policy and on the state-actions value function.

HJ P OBR (ω) = ∂ ∂ω Φ (I -γP ω ) ∆ ρ Φ(Φ ∆ ρ Φ) -1 Φ ∆ ρ (Φω -T * Φω) (6.41) = Φ (I -γP ω ) ∆ ρ Φ(Φ ∆ ρ Φ) -1 Φ ∆ ρ (I -γP ω )Φ (6.
The first key ingredient of our method is to define as a norm a weaker notion of an -Nash equilibrium than the one in 3.1. Since this norm can not be directly optimized and estimated with batch data, we define a surrogate loss (a sum of Bellman residuals) that can be estimated with batch data and supports function approximation. As there are no theoretical guarantees of convergence of these methods, we evaluate its soundness on randomly generated MDPs and general-sum MGs.

Nash, -Nash and Weak -Nash Equilibrium

We have defined the Notion of Nash equilibrium in Section 3.1. It states that one player cannot improve his own value by switching his strategy if the other players do not vary their own one [START_REF] Filar | Competitive Markov Decision Processes[END_REF]. The goal here is to find one strategy for players which is as close as possible to a Nash equilibrium. The definition of Section 3.1 definition can be rewritten with Bellman operators: Definition 7.1. In an MG, a strategy π is a Nash equilibrium if ∃v such as ∀i ∈ {1, ..., N }, T i π v i = v i and T * i π -i v i = v i . Proof. The proof is left in appendix 7.

We will use the definition of -Nash equilibrium in to quantify the performance of a joint strategy in MGs. As in MDP the criterion to evaluate a policy is the distance of it's value v π to the optimal value v * , in MGs, we will use the maximum over player of the distance between a joint value v i π to the value of a best response v * i π -i . In the case of a single player MG (or MDP), a Nash equilibrium is simply the optimal strategy. Formally Filar and Vrieze (2012): Definition 7.2. In an MG, a strategy π is an -Nash equilibrium if: ∀i ∈ {1, ..., N },

v i π + ≥ v * i π -i or ∀i ∈ {1, ..., N }, v * i π -i -v i π ≤ , which is equivalent to: v * i π -i -v i π s,∞ i,∞ ≤ .
An -Nash equilibrium is a relaxed solution concept in game theory. When all players play an -Nash equilibrium the value they will receive is at most sub-optimal compared to a best response. Interestingly, when considering an MDP, the definition of an -Nash equilibrium is reduced to control the L +∞ -norm between the value of the players' strategy and the optimal value. However, it is known that approximate dynamic programming algorithms do not control a L +∞ -norm but rather an L p -norm [START_REF] Munos | Finite-Time Bounds for Fitted Value Iteration[END_REF] (we take the definition of the L p -norm of Piot et al. (2014a)). Using L p -norm is necessary for approximate dynamic programming algorithms to use complexity bounds from learning theory Piot et al. (2014a). The convergence of these algorithms was analyzed using supervised learning bounds in L pnorm and thus guaranties are given in L p -norm Scherrer et al. (2012). In addition, Bellman residual approaches on MDPs also give guaranties in L p -norm [START_REF] Maillard | Finite-Sample Analysis of Bellman Residual Minimization[END_REF], Piot et al. (2014a). Thus, we define a natural relaxation of the previous definition of the -Nash equilibrium in L p -norm which is consistent with the existing work on MDPs. Definition 7.3. In an MG, π is a weak -Nash equilibrium if:

v * i π -i -v i π µ(s),p ρ(i),p ≤ .
One should notice that an -Nash equilibrium is a weak -Nash equilibrium (as an L +∞ -norm is an upper bound of any L p -norm with respect to any probability measure). Conversely, a weak -Nash equilibrium is not always an -Nash equilibrium. Furthermore, both do not need to be equal. The notion of weak -Nash equilibrium defines a performance criterion to evaluate a strategy while seeking for a Nash equilibrium. Thus, this definition has the great advantage to provide a convincing way to evaluate the final strategy. In the case of an MDP, it states that a weak -Nash equilibrium only consists in controlling the difference in L p -norm between the optimal value and the value of the learned strategy. For an MG, this criterion is an L p -norm over players (i ∈ {1, . . . , N }) of an L p -norm over states (s ∈ S) of the difference between the value of the joint strategy π for player i in state s and of his best response against the joint strategy π -i . In the following, we consider that learning a Nash equilibrium should result in minimizing the loss v * i

π -i -v i π µ(s),p ρ(i),p
. For each player, we want to minimize the difference between the value of his strategy and a best response considering the strategy of the other players is fixed. However, a direct minimization of that norm is not possible in the batch setting even for MDPs. Indeed, v * i π -i cannot be directly observed and be used as a target. A common strategy to alleviate this problem is to minimize a surrogate loss. In MDPs, a possible surrogate loss is the optimal Bellman residual v -T * v µ,p (where µ is a distribution over states and T * is the optimal Bellman operator for MDPs) Piot et al. (2014a), [START_REF] Baird | Residual Algorithms: Reinforcement Learning with Function Approximation[END_REF]. The optimal policy is then extracted from the learnt optimal value (or Q-value in general). In the following section, we extend this Bellman residual approach to MGs.

Bellman Residual Minimization in MGs

Optimal Bellman residual minimization can't be generalized to general-sum MGs as it is because multi-player strategies cannot be directly extracted from value functions as shown in [START_REF] Zinkevich | Cyclic Equilibria in Markov Games[END_REF]). Yet, from Definition 7.1, we know that a joint strategy π is a Nash equilibrium if there exists v such that, for any player i, v i is the value of the joint strategy π for player i (i.e. T i π v i = v i ) and v i is the value of the best response player i can achieve regarding the opponent's strategy π -i (i.e. T * i π -i v i = v i ). We thus propose to build a second Bellman-like residual optimizing over the set of strategies so as to directly learn a weak -Nash equilibrium. The first (traditional) residual (i.e.

T * i π -i v i -v i ρ(i),p
) forces the value of each player to be close to their respective best response to every other player while the second residual (i.e. T i π v i -v i ρ(i),p ) will force every player to play the strategy corresponding to that value.

One can thus wonder how close from a Nash-Equilibrium π would be if there existed v such that

T * i π -i v i ≈ v i and T i π v i ≈ v i .
In this section, we prove that, if we are able to control over (v, π) a sum of the L p -norm of the associated Bellman residuals

( T * i π -i v i -v i µ,p and T i π v i -v i µ,p ), then we are able to control v * i π -i -v i π µ(s),p ρ(i),p .
Theorem 7.1. ∀p, p positive reals such that 1 p + 1 p = 1 and ∀(v 1 , . . . , v N ):

v * i π -i -v i π µ(s),p ρ(i),p ≤ 2 1 p C ∞ (µ, ν) 1 p 1 -γ (7.1) × N i=1 ρ(i) T * i π -i v i -v i p ν,p + T i π v i -v i p ν,p 1 p , ( 7
C ∞ (µ, ν, π i , π -i ) = ∂µ T (1-γ)(I-γP π i ,π -i ) -1 ∂ν T ν,∞ and C ∞ (µ, ν) = sup π C ∞ (µ, ν, π i , π -i ).
Proof. The proof is left in appendix 8.

This theorem shows that an -Nash equilibrium can be controlled by the sum over the players of the sum of the norm of two Bellman-like residuals: the Bellman Residual of the best response of each player and the Bellman residual of the joint strategy. If the residual of the best response of player i (

T * i π -i v i -v i p ν,p
) is small, then the value v i is close to the value of the best response v * i π -i and if the Bellman residual of the joint strategy T i π v i -v i p ν,p for player i is small, then v i is close to v i π . In the end, if all those residuals are small, the joint strategy is an -Nash equilibrium with small since v * i π -i v i v i π . Theorem 7.1 also emphasizes the necessity of a weakened notion of an -Nash equilibrium. It is much easier to control a L p -norm than a L ∞ -norm with samples. In the following, the weighted sum of the norms of the two Bellman residuals will be noted as:

f ν,ρ,p (π, v) = N i=1 ρ(i) T * i π -i v i -v i p ν,p + T i π v i -v i p ν,p .
Finding a Nash equilibrium is then reduced to a non-convex optimization problem. If we can find a (π, v) such that f ν,ρ,p (π, v) = 0, then the joint strategy π is a Nash equilibrium. This procedure relies on a search over the joint value function space and the joint strategy space. Besides, if the state space or the number of joint actions is too large, the search over values and strategies might be intractable. We addressed the issue by making use of approximate value functions and strategies. Actually, Theorem 7.1 can be extended with function approximation. A good joint strategy π θ within an approximate strategy space Π can still be found by computing π θ , v η ∈ argmin θ, η f ν,ρ,p (π θ , v η ) (where θ and η respectively parameterize π θ , v η ). Even with function approximation, the learned joint strategy π θ would be at least a weak -Nash equilibrium (with ≤ 2 1 p C∞(µ,ν)

1 p 1-γ f ν,ρ,p (π θ , v η ))
. This is, to our knowledge, the first approach to solve MGs within an approximate strategy space and an approximate value function space in the batch setting.

The Batch Scenario

In this section, we explain how to learn a weak -Nash equilibrium from Theorem 7.1 with approximate strategies and an approximate value functions. As said previously, we will focus on the batch setting where only a finite number of historical data sampled from an MG are available. As for previous batch algorithms, we will work on stateactions value functions (the definition of the Q-function can be found in Section 3 and the one of Bellman operators can be found in Section 3.2).

Thus, we will have to minimize the following function depending on strategies and Q-functions:

3. The Batch Scenario 127 f (Q, π) = N i=1 ρ(i) B * i π -i Q i -Q i p ν,p + B i π Q i -Q i p ν,p (7.
3)

The batch scenario consists in having a set of k samples (s j , (a 1 j , ..., a N j ), (r 1 j , ..., r N j ), s j ) j∈{1,...,k} where r i j = r i (s j , a 1 j , ..., a N j ) and where the next state is sampled according to p(.|s j , a 1 j , ..., a N j ). From Equation ( 7.3), we can minimize the empirical-norm by using the k samples to obtain the empirical estimator of the Bellman residual error.

fk (Q, π) = k j=1 N i=1 ρ(i) B * i π -i Q i (s j , a j ) -Q i (s j , a j ) p + B i π Q i (s j , a j ) -Q i (s j , a j ) p , (7.4)
For more details, an extensive analysis beyond the minimization of the Bellman residual in MDPs can be found in Piot et al. (2014a). In the following we discuss the estimation of the two Bellman residuals B * i

π -i Q i (s j , a j ) -Q i (s j , a j ) p and B i π Q i (s j , a j ) -Q i (s j , a j ) p in different cases.
Deterministic Dynamics: With deterministic dynamics, the estimation is straightforward. We estimate

B i π Q i (s j , a j ) with r i j + γE b∼π [Q i (s j , b)] and B * i π -i Q i (s j , a j ) with r i j + γ max b i E b -i ∼π -i [Q i (s j , b i , b -i )]
, where the expectation are:

E b∼π [Q i (s , b)] = b 1 ∈A 1 . . . b N ∈A N π 1 (b 1 |s ) . . . π N (b N |s )Q i (s , b 1 , . . . , b N )
and where

E b -i ∼π -i [Q i (s , b i , b -i )] = b 1 ∈A 1 . . . b i-1 ∈A i-1 b i+1 ∈A i+1 . . . b N ∈A N π 1 (b 1 |s ) . . . . . . π i-1 (b i-1 |s )π i+1 (b i+1 |s ) . . . π N (b N |s )Q i (s , b 1 , . . . , b N )
Note that this computation can be turned into tensor operations as described in Appendix-9 and thus the architecture can be implemented using relevant libraries.

Stochastic Dynamics:

In the case of stochastic dynamics, the previous estimator (7.3) is known to be biased [START_REF] Maillard | Finite-Sample Analysis of Bellman Residual Minimization[END_REF], Piot et al. (2014a). If the dynamic is known, one can use the following unbiased estimator:

B i π Q i (s j , a j ) is estimated with r i j + γ s ∈S p(s |s j , a j )E b∼π [Q i (s , b)] and B * i π -i Q i (s j , a j ) with r i j + γ s ∈S p(s |s j , a j ) max b i E b -i ∼π -i [Q i (s , b i , b -i )] .
If the dynamic of the game is not known (e.g. batch scenario), the unbiased estimator cannot be used since the kernel of the MG is required and other techniques must be applied. One idea would be to first learn an approximation of this kernel. For instance, one could extend the MDP techniques to MGs such as embedding a kernel in a Reproducing Kernel Hilbert Space (RKHS) [START_REF] Grunewalder | Modelling Transition Dynamics in MDPs With RKHS Embeddings[END_REF]), Piot et al. (2014b,a) or using kernel estimators [START_REF] Taylor | Value Function Approximation in Noisy Environments Using Locally Smoothed Regularized Approximate Linear Programs[END_REF], Piot et al. (2014a). Therefore, p(s |s j , a j ) would be replaced by an estimator of the dynamics p(s |s j , a j ) in the previous equation. If a generative model is available, the issue can be addressed with double sampling as discussed in [START_REF] Maillard | Finite-Sample Analysis of Bellman Residual Minimization[END_REF], Piot et al., 2014a).

Neural Network Architecture

Minimizing the sum of Bellman residuals is a challenging problem as the objective function is not convex. It is all the more difficult as both the Q-function and the strategy of every player must be learned independently. Nevertheless, neural networks have been able to provide good solutions to problems that require minimizing nonconvex objective such as image classification or speech recognition [START_REF] Lecun | Deep Learning[END_REF]. Furthermore, neural networks were successfully applied to reinforcement learning to approximate the Q-function [START_REF] Mnih | Human-Level Control Through Deep Reinforcement Learning[END_REF] in one agent MGs with eclectic state representation.

Here, we introduce a novel neural architecture1 that implements our Bellman residual approach. For every player, a two-fold network is defined: the Q-network that learns a Q-function and a π-network that learns the stochastic strategy of the players. The Q-network is a multilayer perceptron which takes the state representation as input. It outputs the predicted Q-values of the individual action such as the network used by [START_REF] Mnih | Human-Level Control Through Deep Reinforcement Learning[END_REF]. Identically, the π-network is also a multilayer perceptron which takes the state representation as input. It outputs a probability distribution over the action space by using a softmax. We then compute the two Bellman residuals for every player following Equation 7.5. Finally, we back-propagate the error by using classic gradient descent operations.

In our experiments, we focus on deterministic turn-based games. It entails a specific neural architecture that is fully described in Figure 7.6. During the training phase, all the Q-networks and the π-networks of the players are used to minimize the Bellman residual. Once the training is over, only the π-network is kept to retrieve the strategy of each player. Note that this architecture differs from classic actor-critic networks [START_REF] Lillicrap | Continuous Control with Deep Reinforcement Learning[END_REF] for several reasons. Although Q-network is an intermediate support to the computation of π-network, neither policy gradient nor advantage functions are used in our model. Besides, the Q-network is simply discarded at the end of the training. Our method directly searches in the strategy space by minimizing the Bellman residual.

Experiments

In this section, we report an empirical evaluation of our method on randomly generated MGs. This class of problems has been first described by [START_REF] Archibald | On the Generation of Markov Decision Processes[END_REF] for MDPs and has been studied for the minimization of the optimal Bellman residual Piot et al. (2014a) and in zero-sum two-player MGs [START_REF] Perolat | On the use of non-stationary strategies for solving two-player zero-sum markov games[END_REF]. First, we extend the class of randomly generated MDPs to general-sum MGs, then we describe the training setting, finally we analyze the results. Without loss of generality we focus on deterministic turn-based MGs for practical reasons. Indeed, in simultaneous games the complexity of the state actions space grows exponentially with the number of player whereas in turn-based MGs it only grows linearly. Besides, as in the case of simultaneous actions, a Nash equilibrium in a turn-based MG (even deterministic) might be a stochastic strategy [START_REF] Zinkevich | Cyclic Equilibria in Markov Games[END_REF]. In a turn-based MG, only one player can choose an action in each state. Finally, we run our experiments on deterministic MGs to avoid bias in the estimator as discussed in Section 3.

To sum up, we use turn-based games to avoid the exponential growth of the number of actions and deterministic games to avoid bias in the estimator. The techniques described in Section 3 could be implemented with a slight modification of the architecture described in Section 4.

Benchmark: Again we use artificially generated MGs to test our approach. The general procedure to generate a Garnet is described in Chapter 4. We chose this benchmark for two reasons. First, we chose them for the repeatability of our result on several instances of the same class of MGs. Second, the underlying model of the dynamics of the Garnet is fully known. Thus, it is possible to investigate whether an -Nash equilibrium have been reached during the training and to quantify the . It would have been impossible to do so with more complex games. We only investigate N B = 1 to avoid the bias issue and the dynamics is defined as follow. For each state and action (s, a), we randomly pick the next state s in the neighbourhood of indices of s where s follow a rounded normal distribution N (s, σs ). We then build the transition matrix such as p(s |s, a) = 1. Next, we enforce an arbitrary reward structure into the Garnet to enable the emergence of a Nash equilibrium. Each player i has a random critical state ŝi and the reward of this player linearly decreases by the distance (encoded by indices) to his critical state. Therefore, the goal of the player is to get as close as possible from his critical state. Some players may have close critical states and may act together while other players have to follow opposite directions. A Gaussian noise of standard deviation σnoise is added to the reward, then we sparsify it with a ratio m noise to harden the task. Finally, a player is selected uniformly over {1, . . . , N } for every state once for all. The resulting vector v c encodes which player plays at each state as it is a turn-based game. Concretely, a Garnet works as follows: Given a state s, a player is first selected according the vector v c (s). This player then chooses an action according its strategy π i . Once the action is picked, every player i receives its individual reward r i (s, a). Finally, the game moves to a new state according p(s |s, a) and a new state's player (v c (s )). Again, the goal of each player is to move as close as possible to its critical state. Thus, players benefit from choosing the action that maximizes its cumulative reward and leads to a state controlled by a non-adversarial player.

Finally, samples (s, (a 1 , . . . , a N ), (r 1 , . . . , r N ), s ) are generated by randomly select- ing a state and an action uniformly. The reward, the next state, and the next player are selected according to the model described above.

Evaluation: Our goal is to verify whether the-joint strategy of the players is an -Nash equilibrium. To do so, we first retrieve the strategy of every player by evaluating the π i -networks over the state space. Then, given π, we can exactly evaluate the value of the joint strategy v i π for each player i and the value of the best response to the strategy of the others v * i π -i . The value v i π is computed by inverting the linear system v i π = (I -γP π ) -1 r i π . The value v * i π -i is computed with the policy iteration algorithm. Finally, we compute the Error vs Best Response for every player defined as

v i π -v * i π -i 2 v * i π -i 2 .
If this metric is close to zero for all players, the players reach a weak -Nash equilibrium with close to zero. Actually, Error vs Best Response is a normalized quantification of how sub-optimal the player's strategy is compared to his best response. It indicates the sub-optimality margin of the policy of a player compared to his best response while other players keep playing the same strategies. If this metric is close to zero for all players, then they have little incentive to switch from their current strategy. It is an -Nash equilibrium. In addition, we keep track of the empirical norm of the Bellman residual on both the training dataset and the test dataset as it is our training objective.

Training parameters: We use N -player Garnet with 1, 2 or 5 players. The state space and the action space are respectively of size 100 and 5. The state is encoded by a binary vector. The transition kernel is built with a standard deviation σs of 1. The reward function is

r i (s) = 2 min(|s-ŝ i |,N S )-|s-ŝ i |) N S
(it is a circular reward function). The reward sparsity m noise is set to 0.5 and the reward white noise σnoise has a standard deviation of 0.05. The discount factor γ is set to 0.9. The Q-networks and π-networks have one hidden layers of size 80 with RELUs [START_REF] Goodfellow | Deep Learning. Book in preparation for[END_REF]. Q-networks have no output transfer function while π-networks have a softmax. The gradient descent is performed with AdamGrad [START_REF] Goodfellow | Deep Learning. Book in preparation for[END_REF] with an initial learning rate of 1e-3 for the Q-network and 5e-5 for the π-networks. We use a weight decay of 1e-6. The training set is composed of 5N S N A samples split into random minibatch of size 20 while the testing dataset contains N S N A samples. The neural network is implemented by using the python framework Tensorflow [START_REF] Abadi | TensorFlow: Large-scale machine learning on heterogeneous systems[END_REF]. The source code is available on Github (Hidden for blind review) to run the experiments.

Results:

Results for 1 player (MDP) and 5 players are reported in Figure 7.1. Additional settings are reported in the Appendix-9 such as the two-player case and the tabular cases. In those scenarios, the quality of the learned strategies converges in parallel with the empirical Bellman residual. Once the training is over, the players can increase their cumulative reward by no more than 8% on average over the state space. Therefore, neural networks succeed in learning a weak -Nash equilibrium. Note that it is impossible to reach a zero error as (i) we are in the batch setting, (ii) we use function approximation and (iii) we only control a weak -Nash equilibrium. Moreover, the quality of the strategies are well-balanced among the players as the standard deviation is below 5 points.

Our neural architecture has good scaling properties. First, scaling from 2 players to 5 results in the same strategy quality. Furthermore, it can be adapted to a wide variety of problems by only changing the bottom of each network to fit with the state representation of the problem.

Discussions:

The neural architecture faces some over-fitting issues. It requires a high number of samples to converge as described on Figure 7.2. [START_REF] Lillicrap | Continuous Control with Deep Reinforcement Learning[END_REF] introduces several tricks that may improve the training. Furthermore, we run additional experiments on non-deterministic Garnets but the result remains less conclusive. Indeed, the estimators of the Bellman residuals are biased for stochastic dynamics. As discussed, embedding the kernel or using kernel estimators may help to estimate prop- erly the cost function (Equation (7.4)). Finally, our algorithm only seeks for a single Nash-Equilibrium when several equilibria might exist. Finding a specific equilibrium among others is out of the scope of this thesis.

Conclusion

In this chapter, we present a novel approach to learn a Nash equilibrium in MGs.

The contributions are both theoretical and empirical. First, we define a new (weaker) concept of an -Nash equilibrium. We prove that minimizing the sum of different Bellman residuals is sufficient to learn a weak -Nash equilibrium. From this result, we provide empirical estimators of these Bellman residuals with batch data. Finally, we describe a novel neural network architecture to learn a Nash equilibrium from batch data. This architecture does not rely on a specific MG. It also scales to a high number of players for turn taking games. Thus it can be applied to a trove of applications. This chapter concludes this part on Bellman residual minimization. Whilst approximate dynamic programming was not a convincing approach to learn Nash equilibria, Bellman residual minimization seems to be a reasonable approach for general sum

Conclusion 133

MGs. This approach does not reduce to the existing ones in MDPs and in zero-sum two-player MGs. Indeed, general-sum MGs lack key properties of MDPs and zero-sum two-player MGs. In MDPs and in zero-sum two-player MGs only require finding the solution of a single fixed point equation whilst an N -player general sum MG requires solving 2N fixed point equations simultaneously.

Proof of the Equivalence of Definition 2.4 and 7.1

Proof of the equivalence between definition 7.1 and 2.4.

(7.1) ⇒ (2.4):

If ∃v such as ∀i ∈ {1, ..., N }, T i π v i = v i and T * i π -i v i = v i , then ∀i ∈ {1, ..., N }, v i = v i π i ,π -i and v i = max πi v i πi ,π -i . (2.4) ⇒ (7.1): if ∀i ∈ {1, ..., N }, v i π i ,π -i = max πi v i πi ,π -i ., then ∀i ∈ {1, ..., N }, the value v i = v i π i ,π -i is such as T i π v i = v i and T * i π -i v i = v i .
8 Proof of Theorem 7.1

First we will prove the following lemma. The proof is strongly inspired by previous work on the minimization of the Bellman residual for MDPs Piot et al. (2014a).

Lemma 7.1. let p and p be a real numbers such that 1 p + 1 p = 1, then ∀v, π and ∀i ∈ {1, ..., N }:

v i π i * ,π -i -v i π i ,π -i µ,p (7.5) ≤ 1 1 -γ C ∞ (µ, ν, π i * , π -i ) p p + C ∞ (µ, ν, π i , π -i ) p p 1 p T * i π -i v i -v i p µ,p + T i π v i -v i p µ,p 1 p , (7.6)
where π i * is the best response to π -i . Meaning v i π i * ,π -i is the fixed point of T * i π -i . And with the following concentrability coefficient

C ∞ (µ, ν, π i , π -i ) = ∂µ T (1-γ)(I-γP π i ,π -i ) -1 ∂ν T ν,∞ .
Proof. The proof uses similar techniques as in Piot et al. (2014a). First we have:

v i π i ,π -i -v i = (I -γP π i ,π -i ) -1 (r i π i ,π -i -(I -γP π i ,π -i )v i ), (7.7) = (I -γP π i ,π -i ) -1 (T i π i ,π -i v i -v i ). (7.8)
But we also have:

v i π i
Finally, using the same technique as the one in Piot et al. (2014a), we get:

v i π i * ,π -i -v i π i ,π -i µ,p (7.13) ≤ (I -γP π i * ,π -i ) -1 T * i π -i v i -v i µ,p + (I -γP π i ,π -i ) -1 T i π i ,π -i v i -v i µ,p , (7.14) ≤ 1 1 -γ C ∞ (µ, ν, π i * , π -i ) 1 p T * i π -i v i -v i ν,p + C ∞ (µ, ν, π i , π -i ) 1 p T * i π -i v i -v i ν,p , (7.15) ≤ 1 1 -γ C ∞ (µ, ν, π i * , π -i ) p p + C ∞ (µ, ν, π i , π -i ) p p 1 p T * i π -i v i -v i p ν,p + T i π v i -v i p ν,p 1 p .
(7.16) Theorem 7.1 falls in two steps:

max πi v πi ,π -i -v i π µ(s),p ρ(i),p ≤ 1 1 -γ max i∈{1,...,N } C ∞ (µ, ν, π i * , π -i ) p p + C ∞ (µ, ν, π i , π -i ) p p 1 p (7.17) × N i=1 ρ(i) T * i π -i v i -v i p ν,p + T i π v i -v i p ν,p 1 p , (7.18) ≤ 2 1 p C ∞ (µ, ν) 1 p 1 -γ N i=1 ρ(i) T * i π -i v i -v i p ν,p + T i π v i -v i p ν,p 1 p , (7.19) with C ∞ (µ, ν) = sup π i ,π -i C ∞ (µ, ν, π i , π -i )
The first inequality is proven using lemma 1 and Holder inequality. The second inequality falls noticing ∀π

i , π -i , C ∞ (µ, ν, π i , π -i ) ≤ sup π i ,π -i C ∞ (µ, ν, π i , π -i ).

Additional curves

This section provides additional curves regarding the training of the Network. One may notice that the tabular case works well for a 1 player game (MDP). Yet, the more players there are, the worth it performs. Experiments are run on 5 Garnets which is re-sampled 5 times to average the metrics.

(A.0) (B) (D) (C.1) (E) (H) (G) -γ × (F) reward (A.1) (C.0) (C.2)
Figure 7.6 -This scheme summarizes the neural architecture that learns a Nash equilibrium by minimizing the Bellman residual for turn-based games. Each player has two networks : a Q-network that learns a state-action value function and a πnetwork that learns the strategy of the player. The final loss is an empirical estimate of the sum of Bellman residuals given a batch of data of the following shape (s, (a 1 , . . . , a N ), (r 1 , . . . , r N ), s ). The equation (7.4) can be divided into key operations that are described below. For each player i : first of all, in (A.0) we compute Q i (s, a 1 , . . . , a N ) by computing the tensor Q i (s, a) and then by selecting the value of Q i (s, a) (A.1) given the action of the batch. Step (B) computes the tensor Q i (s , b) and step (C.0) computes the strategy π i (.|s ). In all Bellman residuals we need to average over the strategy of players Q i (s , b). Since we focus on turn-based MGs, we will only average over the strategy of the player controlling next state s (in the following, this player is called the next player). In (C.1) we select the strategy π(.|s ) of the next player given the batch. In (C.2) we duplicate the strategy of the next player for all other players. In (D) we compute the dot product between the Q i (s , b) and the strategy of the next player to obtain E b∼π [Q i (s , b)] and in (E) we pick the highest expected rewards and obtain max

b Q i (s , b). Step (F) aims at computing max b i E b -i ∼π -i [Q i (s , b i , b -i )]
and, since we deal with a turn-based MG, we need to select between the output of (D) or (E) according to the next player. For all i, we either select the one coming from (E) if the next player is i or the one from (D) otherwise. In (G) we compute the error between the reward r i to Q

i (s, a) -γ max b i E b -i ∼π -i [Q i (s , b i , b -i )]
and in (H) between r i and γE b∼π [Q i (s , b)] -Q i (s, a). The final loss is the sum of all the residuals.

Part IV Independent Learning in Games

Chapter 8

Actor-Critic Fictitious Play

This chapter contributes to filling a gap in the MARL literature by providing two online independent RL algorithms converging to a Nash equilibrium in multi-stage games both in the cooperative case and the zero-sum two-player case. Those two cases used to be treated as different agendas since the seminal paper of [START_REF] Shoham | If multi-agent learning is the answer, what is the question[END_REF] and we expect our work to serve as a milestone to reconcile them going further than normal form games [START_REF] Leslie | Generalised weakened fictitious play[END_REF]Collins, 2006, Hofbauer andSandholm, 2002).

We proposed two novel on-line and decentralized algorithms inspired by actor-critic architectures for Markov Games, each of them working on two time-scales. Those two algorithms perform the same actors' update but use different methods for the critic. The first one performs an off-policy control step whilst the second relies on a policy evaluation step. Although the actor-critic architecture is popular for its success in solving continuous action RL domains, we chose this architecture for a different reason. Our framework requires handling non-stationarity (because of adaptation of the other players) which is another nice property of actor-critic architectures. Our algorithms are stochastic approximations of two dynamical systems that generalize the work of [START_REF] Leslie | Generalised weakened fictitious play[END_REF] and Hofbauer and Sandholm (2002) on the fictitious play process from normal form games to multistage games [START_REF] Bošanský | Algorithms for computing strategies in two-player simultaneous move games[END_REF].

In the following, we first describe the necessary background in both game theory and RL (Section 1) to introduce our first contribution, the two-timescale algorithms (Section 2). These algorithms are stochastic approximations of two continuous-time processes defined in Section 3. Then, we study (in Section 3) the asymptotic behavior of these continuous-time processes and show, as a second contribution, that they converge in self-play in cooperative games and in zero-sum two-player games. In Section 4, our third contribution proves that the algorithms are stochastic approximations of the two continuous-time processes. Finally, we perform an empirical evaluation (in Section 5).

Specific Background

Multistage Game: We consider games that can be modeled as trees (see Fig. 8.1 in appendix). The state s is the root of the game tree and each state can be reached with some non-zero probability with at least a deterministic joint strategy. Furthermore, we enforce the fact that once a state is visited, it can never be visited again except state Ω, the end of the game. Formally, a multistage game is an MG with an initial state s ∈ S and an absorbing state Ω ∈ S. To enforce that a state can be only visited once, we define an ordering bijection φ : S → {1, . . . , |S|} where |S| is the cardinal of S, φ(s) = |S|, φ(Ω) = 1 and such that ∀s, s ∈ S × S\{(Ω, Ω)}, φ(s) ≤ φ(s ) ⇒ ∀π, P π (s |s) = 0. Moreover the reward in state Ω is null (meaning r(Ω, .) = 0) and no player has more than one action available in that absorbing state. Furthermore, for any state s ∈ S\{s, Ω} there exists a deterministic strategy π and a time t ≤ |S| such that P t π (s|s) > 0. This condition means that every states can be reached from state s with at least a deterministic strategy.

Specific Operators and Value Functions

We define here specific best responses operators and values. Instead of using argmax as a greedy strategy, we will use a smooth version of the argmax (i.e. a Logit choice function). In this subsection, we review notations introduced in Chapter 2 and define ones specific to this chapter.

Value Function and State-Action Value Function:

In a γ-discounted multistage game, (with γ ∈ (0, 1)), the value is defined as in MGs (see Section 3 of Chapter 2):

v i π (s) = E[ +∞ t=0 γ t r i π (s t )|s 0 = s, s t+1 ∼ P π (.|s t )], = I + γP π + • • • + γ |S| P |S| π r i π .

Specific Background 143

Note that for a multistage game, the value function is well defined even for γ = 1 since the process has an absorbing state Ω where the reward function is null. For t > |S|, we have that P t π r i π = 0 since after |S| steps (at most) the process ends up in the absorbing state Ω. We can also define a state-action value function per player, that is the value of performing action a i in state s and then following its strategy:

Q i π (s, a i ) = r i π -i (s, a i ) + γ s p π -i (s |s, a i )v i π (s ) (8.1).
Logit Choice Function: when given a choice between actions a ∈ A based on an expected value Q(a) (in the following Q(a) will be the state-action value function

Q i π (s, a i ) or Q * i σ,π -i (s, a i ) defined below)
, it is common not to take the action that maximizes Q(a) but to choose suboptimal actions so as to favor exploration. One common choice of suboptimal actions is to pick them according to the logit choice function

B η (Q)(a) = exp(η -1 Q(a)) ã exp(η -1 Q(ã)) . We will write C η (Q) = ã∈A Q(ã)B η (Q)(ã)
the expected outcome if actions are taken according to the logit choice function. This definition can be generalized to the notion of choice probability function (see remark 8.1) and will be written B σ (Q) and C σ (Q) where σ is a function in R ∆A . Here ∆A is the set of distributions over actions. In the case of the logit choice function, η is linked to σ through the following formula σ(π) = a ηπ(a) ln(π(a)).

Best Responses: If the strategy π -i of all opponents is fixed, the value of the best response is v * i σ,π -i and is recursively defined as follows (starting from Ω and going in increasing order with respect to φ):

v * i σ,π -i (s) = C σ (r i π -i (s, a i )+ s ∈S p π -i (s |s, a i )v * i σ,π -i (s )).
From the definition of that value function, we define the corresponding Q-function:

Q * i σ,π -i (s, a i ) = r i π -i (s, a i ) + γ s p π -i (s |s, a i )v * i σ,π -i (s ). (8.2)
Bellman Operators: We define the two following Bellman operators on the Q-function:

T i π Q i (s, a i ) = r i π -i (s, a i ) + γ s p π -i (s |s, a i )E b i ∼π i (.|s ) Q i (s , b i ) , and [T * i σ,π -i Q i ](s, a i ) = r i π -i (s, a i ) + γ s p π -i (s |s, a i )C σ Q i (s , .) . The value Q i π is the fixed point of the operator T i π and Q * i σ,π -i is the fixed point of the operator T * i σ,π -i . Further- more, a strategy πi is greedy with respect to a Q-function Q i if T * i σ,π -i Q i = T i πi ,π -i Q i .
Operators on the Value Function: we define the counterparts of those operators on value functions as follows: 8.3) From these definitions, we have the two following value functions:

T i π v i (s) = r i π (s) + γ s p π (s |s)v i (s ) , [T i π v i ] = r i π + γP π v i and [T * i σ,π -i v i ] = C σ (r i π -i (s, .) + γ s p π -i (s |s, .)v i (s )). (
v * i σ,π -i (s) = C σ (Q * i σ,π -i (s, .)) and v i π (s) = E b i ∼π i (.|s) [Q i π (s, b i )].
Smooth Nash Equilibrium: The goal in this setting is to find a strategy π i for each player that recursively (in increasing order with respect to function φ(.)) fulfills the following condition: ∀i,

E a i ∼π i (.|s) Q i π (s, .) = C σ (Q i σ,π (s, .)). As a consequence, we have that Q * i σ,π -i = Q i π and v i π = v * i σ,π -i .
Furthermore, in the case of a zero-sum two-player game, we have v i π = -v -i π (where v -i π is the value function of the opponent)

Proof. The full proof of this lemma is left in appendix 7. It shows that all bounded solutions of the time dependent ODE is a (T, δ)-perturbation of the other ODE. Now, we show that if player i follows the direction given by the local best response dynamics Eq. ( 8.4), it converges to a best response to π -i . Proposition 1. In any game, if the strategy π -i of the opponents is fixed, the process (8.4) converges to a best response.

Proof. The proof of that property is left in appendix 10 Now, let's consider the case where the strategy of other players π -i t converges to a stationary strategy π -i . In that case, one can show that, for a fixed i, the solutions of ODE (8.4) and (8.5) are the same as if π -i t was fixed.

Proposition 2. If π -i t → π -i , then the solutions of ODE (8.4) and (8.5) are the same as the one of:

πi t (.|s) = d π i t ,π -i (s)[B σ (Q i π i t ,π -i (s, .)) -π i t (.|s)] and πi t (.|s) = d π i t ,π -i (s)[B σ (Q * i σ,π -i (s, .)) -π i t (.|s)]
Proof. The proof of that property is left in appendix 11

Thus, if all other players converge to a stationary strategy, the strategy of player i converges to a best response to π -i .

Convergence in Multistage Games:

Concerning the convergence in self-play of those two fictitious play processes, one must first be aware that they will not converge in all multistage games. Even in normal form games, the fictitious play process is known to be unstable in some cases. For example, the fictitious play process is known to oscillate in some two-player games [START_REF] Shapley | Some topics in two-person games[END_REF]. More surprisingly, [START_REF] Hart | Uncoupled dynamics do not lead to nash equilibrium[END_REF] proved in the case of normal form games that there exists no uncoupled dynamics which guarantees Nash convergence. In their paper, they consider dynamics for each player which only depends on its own reward and possibly on the strategy of all players. They present an example in which this class of dynamical systems has no stable equilibrium. We were able to prove the convergence of our process in the case of zero-sum two-player and cooperative multistage games. Proposition 3. In a zero-sum two player multistage game, the process (8.4) and the process (8.5) converge to a smooth Nash equilibrium π.

Proof. The proof of that property is left in appendix 12 Remark 8.3. Proposition 3 can be adapted to study the cooperative case (i.e. when players receive the same reward ∀i, j r i (s, a) = r j (s, a)). The proof in the cooperative case also works by induction. The corresponding proposition and proof can be found in the appendix 9.

Stochastic Approximation with Two-Timescale

In this section, we provide a novel stochastic approximation theorem taking advantage of two independent techniques. First, we use two timescales because the process we are studying (defined on the strategy of each player called the actor) has a complex dependency on the strategy of all players through the Q-function (the critic). In an on-line setting, one can't have access to the Q-function of a given strategy (either Q i π or Q * i σ,π -i ) as it is an asymptotic solution of the critic process. Thus formally, we look for an asymptotically stable solution of ẏ(t) = g(µ(y(t)), y(t)) (actors' update) where µ(y) is the stable solution of another process ẋ(t) = f (x(t), y)(critics' update). In our case µ(y) is either Q i π or Q * i σ,π -i , y is the strategy and x the action-value function. Instead of waiting until the subroutine converges (the critic part) to iterate over the main routine (the actor part), Borkar (1997b) gives an elegant solution to that class of problems by using two timescales to update simultaneously x n and y n . The process x n will be updated according to a "fast" timescale α n . On that timescale, y n behaves as if it was stationary and thus, x n is an estimate of µ(y n ) (i.e. µ(y n ) -x n → 0). The process y n will move on a "slower" timescale β n . On that slower timescale, one can treat the process x n as µ(y n ) and thus, y n will converge to a stable solution of ẏ(t) = g(µ(y(t)), y(t)).

Second we use an averaging technique. Our process is on-line and follows a Markovian dynamics controlled by the policy π n . All policies and Q-functions are only updated on state s n . Thus, the two processes x n and y n also depend on a Markov process Z n controlled by y n . Again, [START_REF] Borkar | Stochastic approximation with 'controlled markov'noise[END_REF] shows that, in the case of a simple timescale, the process y n+1 = y n +α n f (y n , Z n ) tracks the solution of ẏ(t) = f (y(t), d y(t) ) where f (y(t), d y(t) ) is the average of f over the stationary distribution of Z n (the distribution d y ). Formally, f (y, d y ) = z∈Z f (y, z)d y (z). In our case, we need to study the following recursion: Those two equations admit as an attractor Q * i σ,π -i and Q i π . Then, the strategy recursion follows either ODE (8.4) (if the subroutine is defined by Eq. (8.23)) or ODE (8.5) (if the subroutine is defined by (8.22)).

x n+1 = x n + α n f (x n ,

Experiment on Alesia

The game of Alesia is described in Chapter 3. This game is challenging in many aspects. First, rewards are sparse and are received at the end of the game. Furthermore, strategies need to be stochastic [START_REF] Buro | Solving the Oshi-Zumo Game[END_REF], [START_REF] Bošanský | Algorithms for computing strategies in two-player simultaneous move games[END_REF]. The performance criterion used to compare algorithms is the difference of the value of the joint strategy v i π (s) and the value of the best response v i 0,π -i (s) without any perturbation σ (i.e. σ = 0) We ran experiments for a game with learning rates of the form α n = α 0 aα aα+n bα and β n = (β 0 , a β , b β ) = (0.01, 10 4 , 1.0) and (η 0 , a η ) = (0.1, 10 6 ). The results are displayed in figure 8.2 and shows that both algorithms learn a strategy that tend to be closer to its own best response over learning (meaning that both algorithms learn a Nash equilibrium). An empirical finding is that ACFP with a control critic converges faster than the one with a policy evaluation critic on that game.

Function approximation:

We also ran these experiments with function approximations both on the strategy and on the Q-function. The main issue in Alesia is that the number of actions available to each player depends on the remaining budget of the player. For each player, we aggregated states for which the budget of the player is fixed. The aggregated states are consecutive states with respect to the budget of the other players'. The results reported in Figure 8.3 show that our method is robust to this form of function approximation. We ran experiment with an initial budget of N = 15 on which the algorithm was able to learn a minimax strategy. The step-size parameters were (α 0 , a α , b α ) = (0.1, 10 5 , 0.9), (β 0 , a β , b β ) = (0.01, 10 5 , 1.0) and (η 0 , a η ) = (0.05, 10 8 ).

Conclusion

This chapter explores two independent RL algorithms that provably converge to Nash equilibria in zero-sum MSGs and in cooperative MSGs. This work studies extensions of the Fictitious play process from normal form games to MSGs. Compared to previous families of methods (Bowling andVeloso, 2001, Littman, 1994), these algorithms can be applied to a larger number of settings without heuristics and without knowing the opponents actions. The main contributions were algorithmic and theoretical but we also proposed an empirical evaluation that provides good evidence that our methods are applicable to real problems and could scale up thanks to function approximation. Several interesting research paths remain to be explored. The first is the study of the stability of the two dynamical processes in MGs. The proof given for MSGs propagates the convergence property from the end of the game to the initial state. Since MGs have a graph structure, this technique can't be applied. Thus, other stability arguments must be found to ensure convergence in MGs. The second research path could be to further study the use of function approximation. Whilst there is a wide literature dealing with the off-policy evaluation part or the off-policy control [START_REF] Watkins | Q-learning[END_REF], the update of the strategy using general function approximation remains challenging. The main issue is that the strategy space should support convex combination with a best response. The convex combination of the strategy and the best response could be fitted at each iteration using a classification neural network. Lemma 8.2: Let's study the two following ODE ẋ(t) = f (x(t)) and ẏ(t) = f t (y(t)) and let's assume that lim sup f (.) -f t (.) → 0 uniformly. Furthermore, let's assume that f is Lipschitz continuous. Then, any solution of ẏ(t) = f t (y(t)) converges to the set of attractors of ẋ(t) = f (x(t)).

Proof. let us fix t such as for all T > 0, f (.) -f t+T (.) ≤ . if x(t) = y(t), then:

x(t + T ) -y(t + T ) = This inequality means that, given T , for all δ we can choose t 0 large enough such that any trajectory of ẏ(t) = f t 0 +t (y(t)) is a (T, δ)-perturbation of ẋ(t) = f (x(t)). Then, lemma 8.1 concludes the proof.

8 Proof of Theorem 8.1 Proof. First, rewrite Eq. (8.8) as:

x n+1 = x n + α n f (x n , y n , Z n ), ( 8
y n+1 = y n + α n β n α n g(x n , y n , Z n ) .
Since the function g is Lipschitz in the two first arguments (A 1), since Z is discrete (A 2) and since x n and y n are bounded (A 4), then we have that βn αn g(x n , y n , Z n ) → 0 a.s.. Then from corollary 81 of chapter 6.3 of [START_REF] Borkar | Stochastic Approximation: A Dynamical Systems Viewpoint[END_REF] (first presented in [START_REF] Borkar | Stochastic approximation with 'controlled markov'noise[END_REF]) it follows that (x n , y n ) converges to an internally chain transitive invariant set of the ODE ẏ(t) = 0 and ẋ(t) = f (x(t), y(t), d y(t) ). In other words, x n -µ(y n ) → 0 a.s.. Second, let's write (8.8) as:

y n+1 = y n + α n g(µ(y n ), y n , Z n ) + (g(x n , y n , Z n ) -g(µ(y n ), y n , Z n )) . (8.19)
Since g(., ., .) is Lipschitz in the two first variables uniformly with respect to the third one, we have that g(x n , y n , Z n ) -g(µ(y n ), y n , Z n ) ≤ K x n -µ(y n ) (for some K). Then Eq. (8.8) can be rewritten as: (8.20) where n → 0 a.s.. Again, from corollary 8 of chapter 6.3 of [START_REF] Borkar | Stochastic Approximation: A Dynamical Systems Viewpoint[END_REF] In a cooperative game (as defined in [START_REF] Busoniu | A Comprehensive Survey of Multiagent Reinforcement Learning[END_REF]), the reward signal is as follows: ∀s, i, j r i (s, a) = r j (s, a)). Thus we have the property that ∀s, i, j v i π (s) = v j π (s)

y n+1 = y n + α n g(µ(y n ), y n , Z n ) + n ,
Proposition 4. In a cooperative two player multistage game, the process (8.4) and the process (8.5) converge to a smooth Nash equilibrium π.

Proof. The proof of this result works again by induction on the set S n = {s ∈ S|φ(s) ≤ n}. Let's suppose that for all states s in S n , the process converges to a smooth Nash equilibrium. This means that for all states in s ∈ S n and for all players i, the strategies π i t (.|s) converge to π i (.|s) such as v i π (s) = v * i σ,π -i (s). We also have that ∀i, j v i π (s) = v j π (s).

Let ŝ be the state such that φ(ŝ) = n + 1. Then, we define:

M i (ŝ, a i , a -i ) = r i (ŝ, a i , a -i ) + s ∈S p(s |ŝ, a i , a -i )v i π (s ) M i t (ŝ, a i , a -i ) = r i (ŝ, a i , a -i ) + s ∈S p(s |ŝ, a i , a -i )v i πt (s )
And:

M * i t (ŝ, a i , a -i ) = r i (ŝ, a i , a -i ) +

s ∈S p(s |ŝ, a i , a -i )v * i σ,π -i t (s )

Since the strategy π i t (.|s) converges to π i (.|s), we have v i πt (s) and v * i σ,π -i t (s) that converges to v i π (s) for all s ∈ S n and finally M i t (ŝ, a i , a -i ) and M * i t (ŝ, a i , a -i ) converges to M i (ŝ, a i , a -i ). Then, lemma 8.2 and results of convergence of stochastic fictitious play for N -player potential games from Hofbauer and Sandholm (2002) guarantees that the process in state ŝ will converge to a smooth Nash equilibrium of the normal form game defined by M i (ŝ, a i , a -i ). Finally, we have v i π (ŝ) = v * i σ,π -i (ŝ) and ∀i, j v i π (ŝ) = v j π (ŝ).

Proof of proposition 1

Proof. We prove the result by induction on the set S n = {s ∈ S|φ(s) ≤ n}. First, the property is true in state Ω since by definition, there is only one action available per player. Suppose that for all s ∈ S n , the process converges to a best response (i.e. v i π (s) = v * i σ,π -i ). Let ŝ be the state of order n + 1 (i.e. φ(ŝ) = n + 1). From the definition of the Q-function (Eq. (8.1) and (8.2)), we have that in ŝ the Q i π i t ,π -i (ŝ, .) converges to Q i π (ŝ, .) uniformly over the actions. Moreover, since for all s ∈ S n , v i π (s) = v * i σ,π -i we have that Q i π (ŝ, .) = Q * i σ,π -i (ŝ, .). From lemma 8.2, we get the convergence of π i (.|ŝ) to a best response if the distribution d π is non-null in all states (see remark 8.2). Proof. The proof comes from the fact that B σ Q i π i ,π -i t (s, .) converges to B σ Q i π i ,π -i (s, .) uniformly with respect to π i (since the Q-function is polynomial in π and B σ (.) is Lipschitz with respect to the Q-function. Moreover, since the state space is finite, simple convergence imply the uniform convergence) and the fact that B σ Q i π i ,π -i (s, .) is Lipschitz with respect to π i .

12 Proof of Proposition 3

Proof. The proof of this result works by induction on the set S n = {s ∈ S|φ(s) ≤ n}.

Again, the property is true in state Ω. Let's suppose that for all states s in S n , the process converges to a smooth Nash equilibrium. This means that for all states in s ∈ S n and for all players i, the strategies π i t (.|s) converge to π i (.|s) such as v i π (s) = v * i σ,π -i (s). We also have that v i π (s) = -v -i π (s). Let ŝ be the state such that φ(ŝ) = n + 1. Then, we define:

M i (ŝ, a i , a -i ) = r i (ŝ, a i , a -i ) + s ∈S p(s |ŝ, a i , a -i )v i π (s ) M i t (ŝ, a i , a -i ) = r i (ŝ, a i , a -i ) + s ∈S p(s |ŝ, a i , a -i )v i πt (s )
And:

M * i t (ŝ, a i , a -i ) = r i (ŝ, a i , a -i ) +

s ∈S p(s |ŝ, a i , a -i )v * i σ,π -i t (s )

Since the strategy π i t (.|s) converges to π i (.|s), we have v i πt (s) and v * i σ,π -i t (s) that converges to v i π (s) for all s ∈ S n and finally M i t (ŝ, a i , a -) and M * i t (ŝ, a i , a -) converges to M i (ŝ, a i , a -). Then, lemma 8.2 and results of convergence of stochastic fictitious play from Hofbauer and Sandholm (2002) guarantees that the process in state ŝ will converge to a smooth Nash equilibrium of the normal form game defined by M i (ŝ, a i , a -i ). Finally, we have v i π (ŝ) = v * i σ,π -i (ŝ) and v i π (ŝ) = -v -i π (ŝ).

Analysis of Actor-Critic Fictitious Play

The two on-line algorithms we present here (Algo 27) are stochastic approximations of the two dynamical systems (8.4) and (8.5). In an on-line setting, the interaction between players proceeds as follows. At a step n, players are in state s n and individually take an action a i n ∼ π i n (.|s n ). Each of them receives a reward r i (s n , a 1 n , . . . , a N n ) and the process moves from state s n to state s n+1 ∼ p(.|s n , a 1 n , . . . , a N n ). In addition, we consider the controlled Markov process Z n = (s n , a 1 n , . . . , a N n , s n+1 ) (controlled by the strategy π n ). Finally, if s n+1 = Ω, we restart from state s and Z n+1 = (s, . . . ). We define the following two-timescale processes (with α n and β n defined as in A 3):

π i n+1 = π i n + β n 1 sn 1 sn B σ (Q i n (s n , .)) -π i n (.|s n ) (8.21)
Lemma 11 shows that K 2 contains only unstable equilibrium and conclude that both processes converges to K 1 and thus to a Nash equilibrium since K 1 is not empty (this fact is proven early in the paper).

Unfortunately, that proof contains a mistake. The proof proceeds as follows: They show that if π * ∈ K 2 then there exists a i ,x and i such that g i x,a i (v i π , π -i ) > 0 and they conclude that consequently ∂f (vπ,π) ∂π i < 0. However, there is no direct link between the sine of g i x,a i (v i π , π -i ) and ∂f (vπ,π)

∂π i since f (v π , π) = N i=1 x∈S z∈A i (x)
π i (x, z)g i x,z (v i π , π -i ). This imply that both processes might converge in K 2 (i.e. not to a Nash equilibrium).

Part V

Conclusions and Future Work

Conclusion 161 1 Conclusion

This thesis is devoted to the study of learning in games from interaction data. We consider that the interaction can be modelled as a Markov Game (MG). This model is a generalization of the Markov Decision Process to the multi-agent setting and a generalization of normal form games to multi-state models. We studied two kinds of problems. The first one is the problem of learning from batch data, meaning that there is no way to collect new interaction data from the model. The second one is independent reinforcement learning where each agent plays independently one from an other without knowing the others behaviour.

In part II, we extend several ADP techniques from MDPs to zero-sum two-player MGs. In Chapter 3 we extend known ADP bounds for zero-sum two-player MGs from the L +∞ -norm to the L p -norm. Those ADP bounds are consistent with previous analysis of modified policy iteration in MDP. Based on these results, we propose a non-parametric batch algorithm to learn Nash equilibria in zero-sum two-player MGs. We tested this approach on the game of Alesia and Chapter 4 explores the use of non-stationary strategies to improve the guarantees of Chapter 3. To do so, we defined extensions of most ADP algorithms for zero-sum two-player MGs to the use of non-stationary strategies. These non-stationary algorithms improve both theoretically and empirically their stationary counterpart. The main improvement is to reduce the γ-sensitivity of the contribution of the error from O 1 (1-γ) 2

to O 1 (1-γ)(1-γ m ) . Furthermore, whilst the value iteration algorithm enjoys the worst guarantees, it is the one that performs the best in practice on randomly generated zero-sum two-player MGs. Chapter 5 investigates the use of non-stationary strategies in general-sum MGs as was suggested in previous work [START_REF] Zinkevich | Cyclic Equilibria in Markov Games[END_REF]. We prove that the γ-sensitivity of value iteration in general-sum MGs is improved by the use of non-stationary strategies but that it can't match the performance of value iteration in MDPs or in zero-sum two-player MGS. In the case of general-sum games, the only factor that counts is the length of the cycle and not the number of iterations.

Part III explores the Bellman residual approach to solve MDPs, zero-sum twoplayer MGs and general-sum MGs. In Chapter 6, we proved that a lot of existing batch algorithms to solve MDPs and zero-sum two player MGs could be seen as a Newton's method on some Bellman residuals. We leveraged this link and proposed the use of quasi-Newton methods to improve convergence at a small computation cost. Empirical evaluation on randomly generated MDPs and zero-sum two-player MGs show improvement both in stability and performance of our method compared to existing ones. However, this method can't be applied directly to general sum games. The next chapter (Chapter 7) generalizes this Bellman residual approach to general-sum MGs. This method relies on the minimization of two Bellman residuals per player and does not reduce to the one for MDPs and zero-sum two-player MGs. The minimization of the Bellman residual is not guaranteed. However, having a small Bellman residual guarantees having an -Nash equilibrium. This approach was empirically investigated on randomly generated MGs and shows promising results.

Part IV (Chapter 8) studies another scenario. Instead of being given a fixed amount of data we study an online learning problem. Here players must learn while they play in an independent manner. We proposed an independent RL algorithm that provably converges to zero-sum two-player Multi-Stage Games (MSGs) and cooperative MSGs and tested with function approximation.

Future Work

This thesis mainly contributes to the problem of learning from batch data and the algorithms we created and studied are based on Approximate Dynamic Programming and Bellman Residual Minimization. These approaches could be extended in other settings as in the MDP literature.

The first part of this dissertation studies batch learning using Approximate dynamic programming. But when a generative model of the game is provided, many other approaches can be used to perform each iteration of the inner loop of AGPI-Q. This inner loop is required to provide an approximate solution of a finite horizon MDP of length m. Using batch data restricts the number of possible methods (we only used approximate value iteration to solve that part). When a generative model of the game is provided, one can use a vast amount of methods. For instance, one could use Monte Carlo tree search to evaluate the best response of the computation.

We have seen in Chapter 3 that the use of value iteration gave the best asymptotic performance but that early convergence could be improved using large m. In that chapter, we suggest using large m at the beginning and reducing it after a while. Chapter 4 suggests using non-stationary strategies to improve the asymptotic performance of ADP methods. The value iteration algorithm using non-stationary strategies that had the best asymptotic performance but the early convergence is slowed down compared to stationary algorithms. Adapting AGPI-Q to the use of non-stationary strategies could improve early convergence while getting the same asymptotic performance.

In Chapter 6 we propose using a Bellman residual approach to learn from batch data. We proved that using quasi Newton methods on a Bellman residual is more stable and more accurate than using LSPI or BRMPI. These methods are hardly comparable to approximate dynamic programming methods in theory but an exhaustive empirical comparison can still be done and building a large benchmark of simultaneous move MGs where using stochastic strategies is compulsory, would definitely be a valuable contribution to that field of research.

We proved that LSPI could be improved using quasi Newton methods. And LSPI has been adapted to the online setting. In MDPs and in zero-sum two-player MGs, it could be interesting to investigate whether or not the use of second order methods could be used online to solve MGs. Many advances in stochastic second order methods have been made in recent years and it could be interesting to investigate its application to online learning in MDPs and in zero-sum two-player MGs.

In Chapter 7 we have generalized the Bellman residual approach to general-sum MGs but this method hardly scales when the actions are taken simultaneously. Future 2. Future Work 163 work in this direction should focus on whether or not it is possible to only get a dependency on the action of one player instead of the joint action. One idea could be to use importance sampling if the policy used to collect the data is known.

Finally, the last part of this dissertation focused on independent learning in Multi Stage Games (MSGs). We could provide an algorithm that is guaranteed to converge to a smooth Nash equilibrium in the case of a zero-sum two-player MSG and in the case of a cooperative MSG. In future work, we plan to investigate whether or not, this process converges in MGs (without assuming a tree structure). Another way to improve that work would be to propose different kinds of algorithms using more general function approximations than state aggregation.

As a long term perspective, many areas of MARL remain to explore. In this thesis we assumed that the reward function was a parameter of the problem but this assumption is not always realistic. In RL, this problem is addressed with different approaches such as inverse reinforcement learning , imitation learning or apprenticeship learning. These three approaches remain superficially explored in the MARL literature compared to the state of the art in RL. In this manuscript, we only studied the batch case and the online learning case. We could explore how to leverage batch data or demonstrations of an optimal policy to accelerate the learning process.

  s, a) + γ s ∈S p(s |s, a)v * (s ).

  |s, a)v(s ) and p π (s |s) = E a∼π(.|s) [p(s |s, a)]. With these notations we can define the value function as follows:

  s, a) = r(s, a) + γ s ∈S p(s |s, a)E a∼π(.|s ) [Q(s , a)] ,(2.15)[B * Q] (s, a) = r(s, a) + γ s ∈S p(s |s, a) max a [Q(s, a)] .

  )p(s |s, a)Q(s , b).

Figure 3

 3 Figure 3.2 -Mean-squared error (y-axis) between the estimated value function and the true value function at step k (x-axis). For N = 20 and n = 10000

  Figure 3.3 -Value functions at token position 3

Figure 3

 3 Figure 3.4 -Exact value for the game of ALESIA. From left to right the value of the game (for player's budget = 0...20) for token's position 5 to 1

Figure 4

 4 Figure 4.1 -Performance (y-axis) of the strategy at step k (x-axis) for NSVI for a strategy of length 10 (right) and length 1 (left). Results are averaged over 70 Garnets N S = 100 , N A = 5, N B = 1 (top) and N B = 2 (bottom). All Garnets have a sparsity of 0.5 and γ = 0.9. Each step of the algorithm uses 2.25 × N A × N S samples.

Figure 4

 4 Figure 4.2 -Performance (y-axis) of the strategy of length k + 1 at step k (x-axis) for PSDP2 without resampling (left) with a 10 times resampling (right). Results are averaged over 70 Garnets N S = 100 , N A = 5, N B = 1 (top) and N B = 2 (bottom).All Garnets have a sparsity of 0.5 and γ = 0.9. Each step of the algorithm uses 2.25 × N A × N S rollouts.

Figure 4

 4 Figure 4.3 -Performance (y-axis) of the strategy at step k (x-axis) for NSPI for a strategy of length 10 (left) and 30 (right). results are averaged over 70 Garnets N S = 100 , N A = 8, N B = 1 (top) and N B = 2 (bottom). All Garnets have a sparsity of 0.5 and γ = 0.9. Each step of the algorithm uses 2.25 × N A × N S samples.

Figure 4

 4 Figure 4.4 -Performance (y-axis) of the strategy at step k (x-axis) for NSVI, PSDP and NSPI. Results are averaged over 70 Garnets N S = 100 , N A = 8, N B = 1. All Garnets have a sparsity of 0.5 and γ = 0.9. Each step of NSPI and NSVI uses 2.25 × N A × N S samples. Each step of PSDP2 uses 2.25 × N A × N S rollouts.

Figure 4 Figure 4 Figure 4

 444 Figure 4.5 -Performance (y-axis) of the strategy at step k (x-axis) for NSVI for a strategy of length 10,5,2 and 1 from right to left. Those curves are averaged over 70 Garnet N S = 100 , N A = 5, N B = 1 ( for the two curves on the top) and N B = 2 (for the two curves on the bottom). All curves have a sparsity of 0.5. Each step of the algorithm uses 2.25 × N A × N S samples.

  Applying the Bellman operator) end for Return: (π k , . . . , π k-m+1 ) Theorem 5.1. The NSVI algorithm produces a strategy π k,m = (π k , . . .

Figure 5 . 1 -

 51 Figure 5.1 -Example of game matching the lower bound. Actions are in blue and Rewards in red

  Figure 5.2(a) and 5.2(b) show v * i

  m 2 should be lower than 4γ m Rmax a) reports an experiment where VI is ran with k = 200. It shows the logarithm of the expected performance of the cyclic strategy π k,m for m ∈ {5, 10, 15, . . . , 100} and γ ∈ {0

Figure 5

 5 Figure 5.2 -Performance (y-axis) as a function of k (x-axis) for VI with a stationary strategy 5.2(a) and a cyclic strategy of length 10 5.2(b). Results are averaged over 100 Garnets N S = 100 , N A = 10, N B = 2. All Garnets have a sparsity of 0.2 and γ = 0.99.

Figure 6

 6 Figure 6.1 -Performance (y-axis) of the strategy at step k (x-axis) for LSPI, BRMPI and the corresponding quasi-Newton method. Results are averaged over 50 Garnets N S = 100, N A = 8, N B = 10. All Garnets have a sparsity of 0.5 and γ = 0.99. The number of batch samples used is 0.5 × N A × N S .

Figure 6

 6 Figure 6.2 -Performance (y-axis) of the strategy at step k (x-axis) for Newton-LSPI, BRMPI and the corresponding quasi-Newton method. Results are averaged over 50 Garnets N S = 50, N A = 2, N B = 1. All Garnets have a sparsity of 0.3 and γ = 0.9. The number of batch samples used is 1.0 × N A × N A × N S .

Figure 6

 6 Figure 6.3 -Performance (y-axis) of the strategy at step k (x-axis) for Newton-LSPI, BRMPI and the corresponding quasi-Newton method. Results are averaged over 50 Garnets N S = 50 , N A = 2, N B = 4. All Garnets have a sparsity of 0.3 and γ = 0.9. The number of batch samples used is 2.0 × N A × N A × N S .

  42)And for J OBR∇J OBR (ω) = ∂ ∂ω [Φω -T * Φω] ∆ ρ (Φω -T * Φω) (6.43) = Φ (I -γP ω ) ∆ ρ (Φω -T * Φω) (6.44) = Φ (I -γP ω ) ∆ ρ ((I -γP ω )Φω -r) -γP ω ) ∆ ρ (Φω -T * Φω) (6.46) = Φ (I -γP ω ) ∆ ρ (I -γP ω )Φ (6.47)Chapter 7

  with the following concentrability coefficient (the norm of a Radon-Nikodym derivative).

Figure 7

 7 Figure 7.1 -(top) Evolution of the Error vs Best Response during the training for a given Garnet. In both cases, players manage to iteratively improve their strategies. A strong drop may occur when one of the player finds a important move. (middle) Empirical Bellman residual for the training dataset and the testing dataset. It highlights the link between minimizing the Bellman residual and the quality of the learned strategy. (bottom) Average Error vs Best Response averaged over every players, Garnets and every batch. It highlights the robustness of minimizing the Bellman residual over several games. Experiments are run on 5 Garnets which is re-sampled 5 times to average the metrics.

Figure 7

 7 Figure 7.2 -Impact of the number of samples on the quality of the learned strategy. The number of samples per batch is computed by N samples = αN A N S . Experiments are run on 3 different Garnets which are re-sampled 3 times to average the metrics.

Figure 7

 7 Figure 7.3 -(top) Evolution of the Error vs Best Response during the training for a given Garnet. (middle) Empirical Bellman residual for the training dataset and the testing dataset. (bottom) Average Error vs Best Response averaged over every players, Garnets and every batch. Experiments are run on 5 Garnets which is re-sampled 5 times to average the metrics.

  Figure 8.1 -Example of a two-player zero-sum multistage game with deterministic dynamics.

Figure 8 . 2 -

 82 Figure 8.2 -Performance of the strategy π i (y-axis) along iterations (x-axis).

βFigure 8

 8 Figure 8.3 -Performance of the strategy π i (y-axis) along iterations (x-axis).

  x(τ )) -f (y(τ ))] dτ + t+T t [f (y(τ )) -f τ (y(τ ))] dτ (8.12) (x -y)(t + T ) ≤ t+T t f (x(τ )) -f (y(τ )) dτ + t+T t f (y(τ )) -f τ (y(τ )) dτ (y)(τ ) dτ + T (8.14) Let's write g(T ) = (x -y)(t + T ) . We have g(T ) ≤ K T 0 g(τ )dτ + T Let's write h(T ) = T +K T 0 g(τ )dτ + K and h (T ) = +Kg(T ) ≤ +K(h(T )-K ) ≤ Kh(T )With the differential form of the Grönwall lemma:h(δ) ≤ h(0) exp(KT ) = K exp(KT )And we have (x -y)(t + T ) = g(T ) ≤ K (exp(KT ) -1)

  .15) y n+1 = y n + β n g(x n , y n , Z n )Functions f and g are jointly continuous in their arguments and Lipschitz in their two first arguments uniformly with respect to the third, A 2. The controlled Markov process Z n takes its value in a discrete space Z controlled by variable y n . The variable Z n+1 follows the transition kernel p(.|Z n , y n ) which is uniformly continuous in y n . Furthermore, let us suppose that if y n = y, the Markov chain Z n has a unique invariant distribution d y (.). From now on, we define f (x, y, d y ) = z∈Z f (x, y, z)d y (z) which is the function f averaged over the stationary distribution of the Markov chain defined in the previous assumption. Similarly, ḡ(x, y, d y ) = z∈Z g(x, y, z)d y (z). A 3. The sequences {α n } n≥0 and {β n } n≥0 are two positives decreasing step-size sequences satisfying: and β n = o(α n ) A 4. We need sup n x n < ∞ and sup n y n < ∞ A 5.

a t ∼ π(.|s t

  s t , a t )|s 0 = s,

), s t+1 ∼ p(.|s t , a t ) .

s t , a t )|s 0 = s, a t ∼ π(.|s t ), s t+1 ∼ p(.|s t , a t ) .

	(2.55)

Non-Stationary Strategy in Zero-Sum Two-Player Markov Games 65

	then:
	ṽ * ((s, 0))			(4.6)
			+∞
	= min (νt) t∈N	E[	t=0	γ t rνt (s t )|s 0 = (s, 0), st+1 ∼ Pνt (.|s t )],	(4.7)
	= min

(νt) t∈N E[ +∞ t=0 γ t rνt (s t )|s 0 = s, st+1 ∼ Pνt (.|s t )] (4.5)

1.

Improved Approximate Dynamic Programming Algorithms using non-stationary Strategies 2 Algorithms

  It becomes now possible to use similar proof techniques as the ones used in previous chapter.

	Chapter 4. This section presents extensions to two-player zero-sum MGs of three non-stationary
	algorithms, namely Policy Search by Dynamic Programming (PSDP), Non-Stationary
	Value Iteration (NSVI) and Non-Stationary Policy Iteration (NSPI), known for improv-
	ing the error bounds for MDPs. For MDPs, PSDP is known to have the best concen-
	trability coefficient

  The state-actions value function Q k+1 is the best fit over the training dataset. Strategies µ k+1 and ν k+1 are the exact minmax strategies with respect to Q k+1 .

  ,ν 1 . . . . .γP µ * ,νn , with µ i , ν i random strategies}. And with k

								81)
	With	Γ n µ *	representing	the	class	of	kernel	products
	{γP µ

*

Table 6

 6

	.1 -Policy Iteration and (Pollatschek and Avi-Itzhak's algorithm
	Policy Iteration	Pollatschek and Avi-Itzhak

Bellman Residual Minimization in Zero-Sum Games

	108	Chapter 6.
		2 2,ρ .	(6.6)

Table 6

 6

	.2 -Parameters for LSPI, BRMPI and their Batch version
	LSPI	BRMPI	Batch LSPI	Batch BRMPI
	Ξ

Bellman Residual Minimization in General-Sum Games

  None of the approaches described in the last four chapters are meant to find a stationary Nash equilibrium in general-sum MGs. The very reason why all the previous approaches can't find a stationary Nash equilibrium is that they are all value function based methods. Approximate dynamic programming algorithms iterate over a value function or a state-action value function and the Bellman residual approach developed in the previous chapter also estimates a value function. In all these methods, the policy is implicitly defined by the value function. In general-sum MGs, value function based methods can't find a stationary Nash equilibrium. Zinkevich et al. found an example of turn-taking general-sum MGs for which no value function based method can find a stationary Nash equilibrium.

138 Chapter 7. Bellman Residual Minimization in General-Sum Games

  Evolution of the Error vs Best Response during the training for a given Garnet. When the Garnet has a complex structure or the batch is badly distributed, one player sometimes fails to learn a good strategy. (right) Distribution of actions in the strategy among the players with the highest probability. This plots highlights that π-networks do modify the strategy during the training

	reward	+	-γ ×
			next player
								Dot product
		+						× +	Sum, product terms by terms
								Multiplexer
	action	Select	max	next player	Select	Y 𝑌 =	p 𝐴 𝑖 𝑖𝑓 𝑖 ≠ 𝑝 A B 𝐵 𝑖
								Matrices: 𝐽 x 1
	0 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Figure 7.5 -(left)Shared 250 500 750 1000 1250 1500 1750 epoch % Error vs Best Response Network policy Random policy weights state next state 𝑆 → 𝐴 𝑆 → 𝐴 𝑄 𝜃 𝑠, 𝑎 𝑄 𝜃 𝑠′, 𝑏	Distribution over action	0 0.10 0.15 0.20 0.25 0.30 0.35 0.40	250 next state 𝜋 𝜃 𝑠′ 𝑆 → 𝐴	500	750	1000 1250 1500 1750 epoch Action 0 Action 1 Action 2 Action 3 Action 4 Matrices: 𝐽 x 𝐴

  y n , Z n ), (8.7) y n+1 = y n + β n g(x n , y n , Z n ), Functions f and g are jointly continuous in their arguments and Lipschitz in their two first arguments uniformly with respect to the third. The controlled Markov process Z n takes its value in a discrete space Z controlled by variable y n . The variable Z n+1 follows the kernel p(.|Z n , y n ) which is uniformly continuous in y n . Furthermore, let us suppose that if y n = y, the Markov chain Z n has a unique invariant distribution d y (.).We note f (x, y, d y ) = z∈Z f (x, y, z)d y (z) the function f averaged over the stationary distribution of the Markov chain defined in the previous assumption. Similarly, ḡ(x, y, d y ) = z∈Z g(x, y, z)d y (z). Qi t (s, a i ) = d π (s)π i (.|s) T i π Q i t (s, a i ) -Q i t (s, a i )

	Where: Assumption 1. Assumption 2. And	(8.8)

  For any constant y, the ODE:

	dx(t) dt	= f (x(t), y, d y ),	(8.17)
	has a globally asymptotically stable equilibrium µ(y). That stable equilibrium µ(.) is a
	Lipschitz continuous function.
	A 6. The ODE:
	dy(t) dt	= ḡ µ (y(t)) , y(t), d y(t)	(8.18)
	has a globally asymptotically stable equilibrium y

* Theorem 8.1. (x n , y n ) → (µ(y * ), y * ) almost surely (a.s.).

Convergence in Cooperative Multistage Games 155 9 Convergence in Cooperative Multistage Games

, we have that y n → y * and x n → µ(y * ) 9.

https://github.com/fstrub95/nash_network

As written in chapter

[START_REF] Borkar | Stochastic Approximation: A Dynamical Systems Viewpoint[END_REF] in all stochastic approximation scheme studied in the book (except those in chapter 9) can be added a noise n that converges to 0 a.s.

Part III

Learning in Games : A Bellman Residual Approach

Chapter 6

Bellman Residual Minimization in Zero-Sum Games

The first part of this dissertation explored how ADP methods could be used to produce sample based algorithms for games. In MDPs, the Policy Iteration (PI) algorithm can also be seen as a Newton's method on the Optimal Bellman Residual (OBR) v-T * v 2 . However, in zero-sum two player MGs, the Newton's method on the OBR (Pollatschek and Avi-Itzhak's algorithm) does not reduce to the policy iteration for zero-sum twoplayer MGs (Hoffman and Karp's algorithm). In a nutshell, this chapter shows that two sample-based approximate PI algorithms for MDPs (Least-Squares Policy Iteration (LSPI) and Bellman Residual Minimizing Policy Iteration (BRMPI)) and one algorithm for two player MGs (LSPI for MGs) can be seen as Newton's methods on some form of the OBR.

This simple observation leads to several novel contributions. First it reveals that the LSPI algorithm for zero-sum two-player MGs (which is a sample-based algorithm using function approximation) is the Pollatschek and Avi-Itzhak's algorithm when the model is known and when no function approximation is used. Thus, the LSPI algorithm for MGs does not converge even when the model is known since the Pollatschek and Avi-Itzhak's algorithm can oscillate. All these algorithms (LSPI and BRMPI) suffers from oscillation issues. Our second set of contributions is to consider the use of the quasi-Newton methods instead of Newton's method to address this oscillation problem. This solution was introduced to solve the oscillation issue of the (Pollatschek and Avi-Itzhak's algorithm in [START_REF] Filar | On the Algorithm of Pollatschek and Avi-ltzhak[END_REF]. The modification of Filar and Tolwinski amounts to use a quasi-Newton method on the L 2 -norm of the OBR instead of a Newton's Method. Their proof of convergence is based on the assumption that the L 2 -norm of the OBR is smooth, which is untrue in general (as we show at the end of Section 1). Indeed, the derivative of the L 2 -norm of the OBR might be discontinuous and thus a quasi-Newton method is not anymore guaranteed to converge to a local minimum (counterexamples exist [START_REF] Lewis | Nonsmooth Optimization via Quasi-Newton Methods[END_REF]). But, in practice [START_REF] Lewis | Nonsmooth Optimization via Quasi-Newton Methods[END_REF], using quasi-Newton methods often leads to pretty good solutions even in the non-smooth case.

The resulting algorithms are more thrifty than other approximate PI like algorithms and do not suffer from the oscillation problem.

Algorithm 26 quasi-Newton LSPI

Input: D N = ((x j , a 1 j , a 2 j ), r j , x j ) j=1,...,N some samples, Φ a d dimensional feature space and ω 0 = 0. Constants c 1 (= 10 -4 ),c 2 (= 0.9), η(= 0.9), α 0 = 1, α min (= 10 -10 ) and α min (= 1.0). let's suppose there is an algorithm f such that Âω , b = f (D N , ω, Φ) (the procedure is described in Section 3.1) for k=1,2,...,K do

Experiments on Markov Decision Processes

The PI algorithm for MDPs is known to converge because it builds a sequence of increasing values while searching in an underlying finite policy space [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF]. This does not stand anymore when it comes to function approximation, since the argument of increasing values is not verified. As a result, the LSPI algorithm is not proven to converge in general for MDPs.

We ran experiments on Garnets to compare LSPI and BRMPI (Newton's methods) with their quasi-Newton counterparts (Softened LSPI described in Algorithm-26).

Here, the reward function depends on states and actions, and only a ratio of the rewards 144 Chapter 8. Actor-Critic Fictitious Play

Actor-Critic Fictitious Play

We present here our first contribution: the actor-critic fictitious play algorithm for MGs. This is an on-line and decentralized process. At each time-step n, all players are in state s n , players choose independently an action a i n according to their current strategy π i n (.|s n ) and observe independently one from another a reward signal r i n = r i (s n , a n ). The process is decentralized, meaning players do not observe others' actions nor their rewards. Then, the game moves to the following state s n+1 . If the process reaches the absorbing state Ω, we simply restart from the beginning (s).

Algorithm 27 On-line Actor Critic Fictitious Play

Input: An initial strategy π i 0 and an initial value v i 0 = 0. Two learning rates {α n } n≥0 , {β n } n≥0 satisfying assumption A 3 and an initial state s 0 = s. for n=1,2,... do Agent i draws action

critic step

Either an off-policy control step:

Or a policy evaluation step:

end if end for Return

The joint strategy π and values v i for all i.

The learning algorithm performs two updates. First, it updates the players' strategy (actors' update). The strategy π i n+1 is a mixture between the current strategy π i n and either a local best response

)). The actors' update is performed according to a slow timescale β n . Second, it performs the critics' update which evaluates the current strategy. It happens on a fast timescale α n on which we can consider that the strategy of every player is stationary. If at the actors' step we want to act according to a local best response dynamics, the critic step will perform a policy evaluation step. If we want to perform a global best response dynamics, the critic will perform an off-policy evaluation step. Thus, at the slow timescale, the

Therefore, we obtain canonically two algorithms.

In the next section, as a second contribution, we introduce the two dynamical processes corresponding to these algorithms and we prove that they possess desirable 3. Fictitious play in Markov Games 145 properties (i.e. rationality and convergence in self play for zero-sum two-player and cooperative games). The proofs rely non trivial techniques to propagate the Lyapunov stability property of the Fictitious play process of these ODE on the tree structure of the multistage games. Then in Section 4, as a third contribution, we show formally that these two algorithms (Algorithm 27) are stochastic approximations of those dynamical processes. This is again non-trivial as we need to prove the convergence of a two-time scale discrete scheme depending on a Markov chain.

Remark 8.1. In the previous section, we defined the logit choice function. One can generalize this notion with the concept of choice probability function (see Hofbauer and Sandholm (2002)). Imagine a player is given a choice of an action a ∈ A based on the outcome Q(a). Instead of choosing based on the outcome Q(a), he also obtains an additive random payoff a and chooses with respect to Q(a) + a . The vector ( a ) a∈A is a positive random variable taking it's value in R |A| and does not depend on Q(a). These choice probability functions allow us to get smooth Bellman operators and ease the analysis of our algorithms. A choice probability function is defined as follows:

This definition is equivalent to the following one where σ(.) is a deterministic perturbation:

This perturbation is said admissible Hofbauer and Sandholm ( 2002) if for all y, ∇ 2 σ(y) is positive definite on R 0 = {z ∈ R n : j z j = 0}, and if ∇σ(y) goes to infinity as y approaches the boundary of ∆A. For example,

if the σ(.) perturbation is defined as σ(π) = a ηπ(a) ln(π(a)). In that case, the noise ã follows a Gumbel distribution. From now on, we consider that σ(.) is fixed.

Fictitious play in Markov Games

In this section, we propose novel definitions for two perturbed best response dynamics in the case of MGs. These dynamics are defined as a set of Ordinary Differential Equations (ODE) that generalizes the continuous time Fictitious play process to MSGs Hofbauer and Sandholm (2002). Then, we prove the convergence of these processes in multistage games. To do so, we build on the work of Hofbauer and Sandholm (2002) on the stability of the Fictitious play process in normal form games. By induction on the tree structure of multistage games, we prove that our processes have stable attractors in zero-sum two-player and in cooperative multistage games. Later in Section 4, we will prove that our actor-critic algorithms track the solutions of that ODE and thus are guaranteed to converge in both settings. if we start the process with a strategy π 0 that gives a non-null probability for each action and each player, the distribution over states d π t (s) is never null since we consider that there is at least a deterministic strategy that reaches any state. Furthermore, since we consider smooth best response dynamics, the smooth best response will assign to each action some probability which is bounded away from zero (since the Q-function are bounded). Thus, we can always consider that we will visit each state with some minimal probability (i.e. d π t (s) ≥ δ > 0). Two properties are usually desirable for such a stochastic process which are rationality and convergence in self-play [START_REF] Bowling | Rational and Convergent Learning in Stochastic Games[END_REF]. Rationality implies that if other players converge to a stationary strategy, the learning algorithm will converge to a best response strategy. The convergence property received many definitions in the MARL literature and usually ensures convergence of the algorithm against a class of other algorithms or convergence in self-play. Here, we choose the later.

Rationality:

For the rest of this paragraph, let us study a fixed player i. First, if we consider the case where other players are stationary (i.e. π -i t = π -i ), the strategy of player i will converge to B σ (Q * i σ,π -i (s, .)) for the second process Eq. (8.5) (since

))does not depend on π i t the solution of the second process converges exponentially toward the best response strategy). For the first process, let us show that if the strategy π i follows the dynamics described by (8.4), it converges to a best response strategy. The proof of this property requires the following two technical lemmas.

(T, δ)-perturbation: Let us consider the following ODE where h(.) : R n → R n is a Lipschitz continuous function: ż(t) = h(z(t)) (8.6)

We consider the case where ODE (8.6) has an asymptotically stable attractor set J. A bounded measurable function y(.

The following technical lemma can be proved Borkar (1997b).

Lemma 8.1. Given , T > 0, there exists a δ > 0 such that for δ ∈ (0, δ), every (T, δ)-perturbation of Eq. (8.6) converges to J (def: the -neighborhood of J).

The next lemma is used all over this section. It proves that if, instead of following a given process ẋ(t) = f (x(t)), one follows a perturbed version of that process ẏ(t) = f t (y(t)) one converges to the set of stable equilibria of the unperturbed process if f t (.) converges toward f (.). Lemma 8.2. Let's study the two following ODE ẋ(t) = f (x(t)) and ẏ(t) = f t (y(t)) and let's assume that lim sup f (.) -f t (.) → 0 uniformly. Furthermore, let's assume that f is Lipschitz continuous. Then, any bounded solution of ẏ(t) = f t (y(t)) converges to the set of attractors of ẋ(t) = f (x(t)).

Stochastic Approximation with

Assumption 5. For any constant y, the ODE: ẋ(t) = f (x(t), y, d y ), has a globally asymptotically stable equilibrium µ(y). That stable equilibrium µ(.) is a Lipschitz continuous function.

Assumption 6. The ODE ẏ(t) = ḡ µ (y(t)) , y(t), d y(t) has a globally asymptotically stable equilibrium y * Theorem 8.1. (x n , y n ) → (µ(y * ), y * ) almost surely (a.s.).

Analysis of Actor-Critic Fictitious Play:

The analysis of our two algorithms is a direct application of the previous theorem and is left in appendix (App. 13). In a nutshell, the critic Q n is updated on a fast timescale (i.e. x n ) and we will prove that it's recursion satisfy assumption A 1, A 2, A 3, A 4, and A 5. The actor π n is updated on a slow timescale (i.e. y n ) and satisfy assumption A 1, A 2, A 3, A 4, and A 6. Thanks to the use of choice probability function, assumption A 1 is satisfied on the slow timescale since a choice probability function is lipschitz with respect to the Q-function. The two on-line algorithms we present here (Algo 27) are stochastic approximations of the two dynamical systems (8.4) and (8.5). In an on-line setting, the interaction between players proceeds as follows. At a step n, players are in state s n and individually take an action a i n ∼ π i n (.|s n ). Each of them receives a reward r i (s n , a 1 n , . . . , a N n ) and the process moves from state s n to state s n+1 ∼ p(.|s n , a 1 n , . . . , a N n ). In addition, we consider the controlled Markov process Z n = (s n , a 1 n , . . . , a N n , s n+1 ) (controlled by the strategy π n ). Finally, if s n+1 = Ω, we restart from state s and Z n+1 = (s, . . . ). We define the following two-timescale processes (with α n and β n defined as in A 3):

And:

Or:

Assumption A 1 and assumption A 2 are verified. Regarding assumption A 4, the strategy π i n is obviously bounded since it remains in the simplex. If Q i 0 = 0, the stateaction value function is bounded as follows Q i n (s, a) ≤ φ(s)R max (this can be shown by recursion). The two faster processes(Eq. (8.22) and (8.23)) track the two following ODE:

14. On the Guarantees of Convergence of OFF-SGSP and ON-SGSP 157

And:

Or:

Assumption A 1 and assumption A 2 are verified. Regarding assumption A 4, the strategy π i n is obviously bounded since it remains in the simplex. If Q i 0 = 0, the stateaction value function is bounded as follows Q i n (s, a) ≤ φ(s)R max (this can be shown by recursion). The two faster processes (Eq. (8.22) and (8.23)) track the two following ODE:

Those two equations admit as an attractor Q * i σ,π -i and Q i π . Then, the strategy recursion follows either ODE (8.4) (if the subroutine is defined by Eq. (8.23)) or ODE (8.5) (if the subroutine is defined by (8.22)).

On the Guarantees of Convergence of OFF-SGSP and ON-SGSP

The main contribution to the field of MARL related to our work is the paper of [START_REF] Prasad | Two-Timescale Algorithms for Learning Nash Equilibria in General-Sum Stochastic Games[END_REF]. These algorithms are not decentralized but one of them, ON-SGSP, is an on-line and model-free algorithm. This paper proposes two algorithms OFF-SGSP and ON-SGSP which are stochastic approximations of a dynamical system described in section 8. The authors claims that these algorithms converges to a Nash equilibrium of the game. The proof of the stability given in lemma 11 is wrong. In the following we point out the issue with this proof.

Using their notations:

• G is the set of Nash equilibrium (the feasible set of the optimization problem),

• K is the limit set of the dynamical system (and G ⊂ K)

• K 1 is the set of limits points of the dynamical system which are Nash equilibrium K ∩ G

• K 2 is the complementary of K 1 in K (i.e. K 2 = K\K 2 )