Jérôme Renault

Bruno Ziliotto

Hidden Stochastic Games and Limit Equilibrium Payoffs

Keywords: Stochastic games, Limit equilibrium payoffs, Hidden states

de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Most economic and social interactions have dynamic aspects, and equilibrium plays of dynamic games are typically not obtained by successions of myopic equilibria of the current one-shot interaction, but need to take into account the impact of actions on both current payoffs and future payoffs. In this paper we consider stochastic games with 2 players, where the actions taken by the players are perfectly observed at the end of every stage. We respectively denote by E δ and E δ , the sets of Nash equilibrium payoffs and sequential equilibrium payoffs of the δ-discounted game, and we write E ∞ for the set of uniform equilibrium payoffs of the dynamic game. We focus on studying the limit of E δ and E δ as players get extremely patient, i.e. as the discount factor goes to one.

Standard stochastic games were introduced by Shapley [START_REF] Shapley | Stochastic games[END_REF] and generalize repeated games: the payoff functions of the players evolve from stage to stage, and depend on a state variable observed by the players, whose evolution is affected by the players' actions. In the zero-sum case, E δ and E δ coincide with the discounted value, and Bewley and Kohlberg [START_REF] Bewley | The asymptotic theory of stochastic games[END_REF] proved its convergence as δ goes to one. In the general-sum case, Sorin [START_REF] Sorin | Asymptotic properties of a non-zero-sum stochastic game[END_REF] provided an example where lim δ→1 E δ and E ∞ are nonempty and disjoint. Vieille [START_REF] Vieille | Two-player stochastic games i: A reduction[END_REF][START_REF] Vieille | Two-player stochastic games ii: The case of recursive games[END_REF] proved that E ∞ is always nonempty, that is, there exists a uniform equilibrium payoff 1 .

Under the assumption that the dependence of the stochastic game on the initial state vanishes as δ goes to 1, several Folk theorems have been proven. More precisely, Dutta [START_REF] Dutta | A folk theorem for stochastic games[END_REF] assumes that the set of long-run feasible payoffs is independent of the initial state, has full dimension, and that minmax long-run payoffs do not depend on the initial state either. Fudenberg and Yamamoto2 [START_REF] Fudenberg | The folk theorem for irreducible stochastic games with imperfect public monitoring[END_REF] assume that the stochastic game is irreducible (all players but one can always drive the current state where they want, possibly in many stages, with positive probability). Hörner et al. [START_REF] Hörner | Recursive methods in discounted stochastic games: An algorithm for δ → 1 and a folk theorem[END_REF] generalize the recursive methods of [START_REF] Fudenberg | The folk theorem with imperfect public information[END_REF] to compute a limit equilibrium set in stochastic games with imperfect public monitoring, when this limit set does not depend on the initial state (this is the case when the Markov chain induced by any Markov strategy profile is irreducible).

It is important to note that this type of assumption excludes the existence of absorbing states3 with different equilibrium payoffs. Nonetheless, there are many situations in which the actions taken can have irreversible effects on future plays. This is the case in stopping games, in which players only act once and have to decide when to do so, or when the actions represent investment decisions, or extractions of exhaustible resources. In climatology, the notion of tipping point (a critical threshold upon which irreversibility may occur) plays an important role 4 .

Our contribution In this paper, we introduce the more general model of hidden stochastic games, and we refer to the above original model as standard stochastic games. In a hidden stochastic game, the players still perfectly observe past actions, but they no longer perfectly observe current states. Instead, they receive a public, possibly random, signal on the current state at the beginning of every stage. Accordingly, players have incomplete information on the sequence of states, but this incomplete information is common to both players. Hidden stochastic games are generalizations of hidden Markov decision processes (where there is a single agent), which explains their name. In addition, hidden stochastic games generalize repeated games with common incomplete information on the state. We believe this model to be meaningful in many interactions in which the fundamentals are not perfectly known by the players, such as competition games. Let us also mention that in climate change, the level of a tipping point is often unknown, and can be seen as a hidden state. It has been argued that collective action is more difficult to implement if uncertainty about the threshold is high: in this case, free riding makes it virtually inevitable that the tipping point will be crossed [START_REF] Barrett | Sensitivity of collective action to uncertainty about climate tipping points[END_REF].

Surprisingly enough, very few papers have already addressed stochastic games with imperfect observation of the state. During his PhD, Venel [START_REF] Venel | Commutative stochastic games[END_REF] studied zero-sum hidden stochastic games in which the players do not receive any signal on the state during the game. Under some commutativity assumption over transitions, he proved the existence of the limit value, as well as the stronger notion of uniform value (corresponding to uniform equilibrium payoffs). This model is also addressed in Gimbert et al. [START_REF] Gimbert | On values of repeated games with signals[END_REF], with different payoff functions. The second author [START_REF] Ziliotto | Zero-sum repeated games: counterexamples to the existence of the asymptotic value and the conjecture maxmin= lim v(n)[END_REF] showed that the commutativity assumption was necessary for Venel's result to hold, and provided an example of a zero-sum hidden stochastic game 5 with payoffs in [0, 1], in which the δ-discounted values have no limit when δ goes to one. Independently from the present work, Yamamoto [START_REF] Yamamoto | Stochastic games with hidden states[END_REF] also introduces stochastic games with observed actions and hidden states. With a rather different approach from ours, Yamamoto proves a Folk theorem under some irreducibility or connectedness assumptions.

For any given parameters ε in (0, 5/12) and r in (0, ε/5), we provide here an example of a 2player hidden stochastic game with all payoffs in [0, 1] and four actions for each player, with the following features:

1. The game is symmetric between the players: they have the same strategy sets, and in any state, switching the actions of Player 1 and 2 switches their payoff and keep the same transition.

2. The players have incomplete information on the current state, but the public signals received are informative enough for the players to know the current stage payoff functions at the beginning of every stage. As a consequence, the players know their current payoffs during the play.

3. There are 13 states, and for any initial state and discount factor, the set of sequential equilibrium payoffs contains a square with side 2r, thus it has full dimension.

4. For a specific initial state k 1 , there exist subsets ∆ 1 and ∆ 2 of discount factors, both containing 1 as a limit point, such that for all discount factors in ∆ 1 , the corresponding set of sequential equilibrium payoffs is exactly the square E 1 centered in (ε, ε) with side 2r, whereas for all discount factors in ∆ 2 , the set6 of sequential equilibrium payoffs is the square E 2 centered in (1 -ε, 1 -ε) with side 2r. In all cases, the associated square is also the set of Nash equilibrium payoffs, the set of (normal or extensive-form) correlated equilibrium payoffs, and the set of communication7 equilibrium payoffs of the discounted game. Given that these two squares are disjoint, there is no converging selection of equilibrium payoffs, and the game has no uniform equilibrium payoff.

5. Moreover, the example is robust to small perturbations of the payoffs: if one perturbs all payoffs of the game by at most 1 2 r(ε -5r), the set of discounted equilibrium payoffs of the perturbed game with initial state k 1 still does not converge, no converging selection of equilibrium payoffs exists and there is no uniform equilibrium payoff.

Our last example is thus robust in many aspects, and it seems impossible to affect a reasonable limit equilibrium payoff to this game. The hidden stochastic game model may be seen as a small departure from the standard model of stochastic game, but it seems very difficult for an expert to find any good answer to the informal question: "The game being played by extremely patient players, what are the possible outcomes?"

Comparison with the zero-sum example in [START_REF] Ziliotto | Zero-sum repeated games: counterexamples to the existence of the asymptotic value and the conjecture maxmin= lim v(n)[END_REF] The construction improves upon the zerosum example in [START_REF] Ziliotto | Zero-sum repeated games: counterexamples to the existence of the asymptotic value and the conjecture maxmin= lim v(n)[END_REF]. In particular, the present example has the following important additional properties:

-The oscillations of (E δ ) and (E δ ) are arbitrarily extreme: in both examples the payoffs lie in [0,1], but in [START_REF] Ziliotto | Zero-sum repeated games: counterexamples to the existence of the asymptotic value and the conjecture maxmin= lim v(n)[END_REF] the discounted value oscillates between 1/2 and 5/9, whereas in the present example, the set of discounted equilibrium payoffs oscillates between a square centered in (ε, ε) and a square centered in (1 -ε, 1 -ε), where ε and the square length can be arbitrarily small. We believe that this property is by itself a significant improvement of [START_REF] Ziliotto | Zero-sum repeated games: counterexamples to the existence of the asymptotic value and the conjecture maxmin= lim v(n)[END_REF]. Indeed, prior to our example, one may have hoped to prove a weak version of a Folk theorem in hidden stochastic games, such as "all the equilibrium payoffs accumulation points lie in some set".

Our example shows that this set may contain the extreme points of the payoff functions, and thus any general result is likely to fail.

-The equilibrium sets satisfy the interiority conditions that are standard in literature (see Properties 3 and 4 above). Indeed, in most papers on discounted Folk theorems, there are assumptions implying that the set of feasible and individually rational payoffs has non-empty interior, or at least that there exists a feasible and strictly individually rational payoff (see [START_REF] Fudenberg | The folk theorem in repeated games with discounting or with incomplete information[END_REF][START_REF] Fudenberg | The folk theorem for irreducible stochastic games with imperfect public monitoring[END_REF][START_REF] Hörner | Recursive methods in discounted stochastic games: An algorithm for δ → 1 and a folk theorem[END_REF][START_REF] Yamamoto | Stochastic games with hidden states[END_REF]. In our example, the fact that equilibrium sets have non-empty interiors implies these assumptions.

-The construction and the analysis are simpler.

-The game is symmetric.

Comparison with the examples in [START_REF] Renault | Limit equilibrium payoffs in stochastic games[END_REF] In a companion paper [START_REF] Renault | Limit equilibrium payoffs in stochastic games[END_REF], we provided examples of 2-player stochastic games with finitely many states and actions, in which neither E δ nor E δ converges: the limit of the equilibrium payoffs set may simply not exist in a stochastic game. However, compared to the example of this paper, these examples are limited in many aspects:

• These examples are not symmetric: Property 1 fails.

• The set of equilibrium payoffs has empty interior for any discount factor, thus Property 3 fails.

• They are not robust to the introduction of an extensive-form correlation device. Moreover, as in any stochastic game (see [START_REF] Renault | Limit equilibrium payoffs in stochastic games[END_REF]), the set of discounted stationary equilibrium payoffs converges, and thus the oscillations of (E δ ) and (E δ ) are not "extreme". Last, they have uniform equilibrium payoffs. Thus, Property 4 fails in every aspect.

• The limits of converging selections of E δ coincide with the uniform equilibrium payoffs. Thus, we believe that in these examples, the elements of the limit set of stationary equilibrium payoffs emerge as natural outcomes of the game played by extremely patient players.

Why did not we use the examples in [START_REF] Ziliotto | Zero-sum repeated games: counterexamples to the existence of the asymptotic value and the conjecture maxmin= lim v(n)[END_REF] and [START_REF] Renault | Limit equilibrium payoffs in stochastic games[END_REF] to prove our theorem? To prove our result, we first build a zero-sum hidden stochastic game with similar properties as the one of [START_REF] Ziliotto | Zero-sum repeated games: counterexamples to the existence of the asymptotic value and the conjecture maxmin= lim v(n)[END_REF].

Then, we twist it to obtain a symmetric game with equilibrium sets having full dimension. One may wonder why we build another zero-sum hidden stochastic game, instead of using directly the one of [START_REF] Ziliotto | Zero-sum repeated games: counterexamples to the existence of the asymptotic value and the conjecture maxmin= lim v(n)[END_REF]. There are two reasons for that. First, the oscillations in [START_REF] Ziliotto | Zero-sum repeated games: counterexamples to the existence of the asymptotic value and the conjecture maxmin= lim v(n)[END_REF] are between 1/2 and 5/9: therefore, to obtain "extreme" oscillations, we need to build a different example, even though it has the same flavor. Second, we believe that the presentation and analysis of this new zero-sum example are considerably simpler than in [START_REF] Ziliotto | Zero-sum repeated games: counterexamples to the existence of the asymptotic value and the conjecture maxmin= lim v(n)[END_REF].

As far as examples in [START_REF] Renault | Limit equilibrium payoffs in stochastic games[END_REF] are concerned, we could indeed twist them to obtain symmetric games. Nonetheless, the oscillations would not be extreme, since they are stochastic games where the state is observed; thus, a converging selection of (E δ ) always exist. Moreover, the full dimensionality assumption would not be satisfied for any state. Indeed, a typical feature of the examples in [START_REF] Renault | Limit equilibrium payoffs in stochastic games[END_REF] is that, for δ, δ large enough, any equilibrium payoff in E δ is close to a δ -equilibrium payoff of the δ -discounted game. Thus, if one modifies the game to obtain full dimensionality, players can use trigger strategies to ensure that any equilibrium payoff in E δ is close to an equilibrium payoff in E δ .

Organization of the paper Hidden stochastic games are introduced in section 2, and our example is presented in section 3. The presentation is done here in 5 progressive steps, starting with a Markov chain on [0, 1], then a Markov Decision Process, then a zero-sum stochastic game with infinite state space, a zero-sum hidden stochastic game and a final example. A few proofs are relegated to the Appendix.

Notations : N, N * , R and R + respectively denote the sets of nonnegative integers, positive integers, real numbers and nonnegative real numbers.

All limits of sets in the paper are taken with respect to the Hausdorff distance between nonempty compact subsets of an Euclidean space : d(A, B) = max{max a∈A d(a, B), max b∈B d(b, A)}. The notation d(A, B) ≤ ε means that: every point in A is at most distant of ε from a point in B, and conversely.

Hidden Stochastic Games

We enlarge the stochastic game model of Shapley [START_REF] Shapley | Stochastic games[END_REF] by assuming that players observe a public signal on the current state at the beginning of every period. Denote by K, I and J respectively the finite sets of states, actions for player 1 and actions for player 2, and let S be the finite set of public signals. Let u 1 and u 2 be the state-dependent utility functions from K × I × J to R, and q the transition function from K × I × J to ∆(K × S), the set of probabilities over K × S. Let π in ∆(K × S) be the initial distribution. The elements K, I, J, S, u 1 , u 2 , q and π are known by the players.

In the first period, a pair (k 1 , s 1 ) is selected according to π, and the players publicly observe s 1 , but not k 1 . The players simultaneously select actions i 1 ∈ I and j 1 ∈ J, then these actions are publicly observed, the stage payoffs are u 1 (k 1 , i 1 , j 1 ) for player 1 and u 2 (k 1 , i 1 , j 1 ) for player 2, and the play goes to period 2. In every period t ≥ 2, a pair (k t , s t ) is selected according to q(k t-1 , i t-1 , j t-1 ), k t is the state of period t but the players only observe the public signal s t . They then simultaneously select actions i t ∈ I and j t ∈ J. These actions are publicly observed, the stage payoffs are u 1 (k t , i t , j t ) for player 1 and u 2 (k t , i t , j t ) for player 2, and the play goes to period t + 1. Given a discount factor δ in [0, 1), the δ-discounted hidden stochastic game is the game with payoff functions (1 -δ) ∞ t=1 δ t-1 u 1 (k t , i t , j t ) and (1 -δ) ∞ t=1 δ t-1 u 2 (k t , i t , j t ). We respectively denote by E δ and E δ the sets of Nash equilibrium payoffs and sequential equilibrium payoffs of this game. This is a generalization of the standard stochastic game model, where one has S = K and s t = k t for all t, and part of the general model of repeated games ( [START_REF] Mertens | Repeated games[END_REF], [START_REF] Sorin | A first course on zero-sum repeated games[END_REF]) In the model of hidden stochastic game (HSG, for short), the players have incomplete information on the current state, but this information is common to the players, and can be represented by a belief p t on the state k t . Given the initial signal s 1 , the initial belief p 1 is the conditional probability induced by π on K given s 1 . The belief p t is a random variable which can be computed8 recursively from p t-1 by Bayes' rule after observing the public signal s t and the past actions i t-1 and j t-1 . We can thus associate an equivalent stochastic game to our HSG, in which the state variable p lies in ∆(K), and represents the common belief on the current state in the HSG, and in which actions and state variables are now publicly observed, in addition to the public9 signal s. A strategy in the HSG defines an equivalent strategy in the stochastic game, and vice-versa. In particular, the sets of equilibrium payoffs of the two games coincide. By definition, a stationary strategy in the associated stochastic game plays after every history a mixed action which only depends on the current state variable in ∆(K). We will say that a strategy σ in the HSG is stationary if the associated strategy in the stochastic game is stationary, that is if σ plays after every history a mixed action which only depends on the current belief in ∆(K).

Standard fixed-point arguments show that E δ and E δ are non-empty, and that there exists a stationary equilibrium in the δ-discounted associated stochastic game.

When there is a single player (for instance, when player 2 has a unique action), a hidden stochastic game is simply a partially observable Markov decision process (POMDP), and if in addition player 1 plays constantly the same mixed action, we obtain a hidden Markov model, which can be considered as the simplest model of dynamic Bayesian network. Hidden stochastic games generalize both standard stochastic games and POMDPs.

An interesting subclass of hidden stochastic games is the class of HSG with known payoffs, where at each stage the current payoff can be deduced from the last public signal and the current actions played. Accordingly, when players choose their actions, they know the current payoff function, as it is the case in a standard stochastic game. However, they may not exactly know the current state in K, and consequently, they are uncertain about the transition probabilities to the next state. HSG with known payoffs generalize standard stochastic games.

Example 2.1. Consider firms competing in a market for a natural exhaustible resource. At each period, firms decide how much resource to extract, and their revenue correspond to their period profit. The state variable is the remaining amount of natural resources (the stock). Firms have a common belief about the initial stock value, and are informed of the current stock value only when it goes below some threshold.

Example 2.2. Consider firms competing in a market for a single good. In each period, a firm chooses its selling price, as well as development and advertising budgets. The state variable represents the state of the market, and includes unobserved variables, such as the overall state of the economy.

Our main result is the following: Theorem 2.3. For each ε in (0, 5 12 ] and r in (0, ε/5), there exists a 2-player hidden stochastic game Γ having the following properties:

1. There are 13 states and public signals, four actions for each player, and all payoffs lie in [0, 1],

2. The game is symmetric between the players, and has known payoffs, 3. For all initial distributions and discount factors, the corresponding set of sequential equilibrium payoffs contains a square of side 2r, and, consequently, has full dimension.

4. There exist an initial state k 1 , perfectly known to the players, and two subsets ∆ 1 and ∆ 2 of [0, 1), both containing discount factors arbitrarily close to 1, such that:

-for all δ in ∆ 1 , the set of sequential equilibrium payoffs E δ is the square E 1 centered in (ε, ε) with side 2r, and for all δ in ∆ 2 , the set of sequential equilibrium payoffs E δ is the square

E 2 centered in (1 -ε, 1 -ε) with side 2r,
-for all δ in ∆ 1 ∪ ∆ 2 , the associated square is also the set of Nash equilibrium payoffs, the set of correlated equilibrium payoffs, and the set of communication equilibrium payoffs of the δdiscounted game, as well as the set of stationary equilibrium payoffs of the associated stochastic game in which the state variable corresponds to the belief on the current state in the original game,

-there is no converging selection of (E δ ) δ , and Γ has no uniform equilibrium payoff.

5.

The above conclusions are robust to perturbations of the payoffs, in the following sense. Consider, for η ∈ [0, r(ε-5r)

2

), a perturbed game Γ(η) obtained by perturbing each payoff of Γ by at most η. The initial state being k 1 , denote by E δ (η) (resp. E δ (η)) the corresponding sets of δ-discounted Nash (resp., sequential) equilibrium payoffs. We have:

∀δ ∈ ∆ 1 , E δ (η) ⊂ [ε -r -η, ε + r + η] 2 , and ∀δ ∈ ∆ 2 , E δ (η) ⊂ [1 -ε -r -η, 1 -ε + r + η] 2 .
There is no converging selection of (E δ (η)) δ , and Γ(η) has no uniform equilibrium payoff. Finally,

lim η→0 lim δ→1,δ∈∆ 1 E δ (η) = lim η→0 lim δ→1,δ∈∆ 1 E δ (η) = E 1 , lim η→0 lim δ→1,δ∈∆ 2 E δ (η) = lim η→0 lim δ→1,δ∈∆ 2 E δ (η) = E 2 , lim δ→1,δ∈∆ 1 lim sup η→0 d(E δ (η), E 1 ) = lim δ→1,δ∈∆ 1 lim sup η→0 d(E δ (η), E 1 ) = 0, lim δ→1,δ∈∆ 2 lim sup η→0 d(E δ (η), E 2 ) = lim δ→1,δ∈∆ 2 lim sup η→0 d(E δ (η), E 2 ) = 0.
the case ε = .3, r = .05 .65

P 1 P 2 1 E 1 E 2
The proof shows that property 4 can also be satisfied for r = 0. The rest of the paper is devoted to the construction of the example of Theorem 2.3, which elaborates and improves on the zero-sum construction of [START_REF] Ziliotto | Zero-sum repeated games: counterexamples to the existence of the asymptotic value and the conjecture maxmin= lim v(n)[END_REF]. We proceed in several steps, starting with a Markov chain on [0, 1], then a Markov Decision Process, then a zero-sum stochastic game with infinite state space, a zero-sum HSG and finally our non zero-sum example.

We believe that our step-by-step presentation of the example is simpler and more intuitive than in [START_REF] Ziliotto | Zero-sum repeated games: counterexamples to the existence of the asymptotic value and the conjecture maxmin= lim v(n)[END_REF], in which the example was directly presented in its full complexity. Another important difference is that though all payoffs lie in [0, 1], in [START_REF] Ziliotto | Zero-sum repeated games: counterexamples to the existence of the asymptotic value and the conjecture maxmin= lim v(n)[END_REF] the discounted value oscillates between 1/2 and 5/9, whereas in the present example the oscillations can be extreme. Along the construction (propositions 3.10 and 3.12), we obtain in corollary 3.13 a stronger result than in [START_REF] Ziliotto | Zero-sum repeated games: counterexamples to the existence of the asymptotic value and the conjecture maxmin= lim v(n)[END_REF] : Given ε > 0, there exists a zero-sum hidden stochastic game with payoffs in [0, 1] such that lim sup δ→1 v δ ≥ 1 -ε and lim inf δ→1 v δ ≤ ε. To achieve this goal, we need to generalize the zero-sum structure, and could not simply reuse the previous zero-sum example. Moreover, we tackle here properties which are of interest in the general-sum case : our game is symmetric, the equilibrium payoff sets all have non empty interior, and we also deal with correlated and communication equilibrium payoffs. We have tried to make the current counter-example as simple as possible while still having many striking properties, so that there is no point for future research in trying to prove existence of a limit equilibrium payoff in classes containing this example.

3 Proof of Theorem 2.3 Sections 3.1, 3.2, 3.3 and 3.4 are dedicated to the construction of a zero-sum hidden stochastic game with payoffs in [0, 1], such that the discounted value oscillates between ε and 1 -ε (ε > 0). This gives an example that satisfies the "extreme oscillations" property of Theorem 2.3. Since it is a zero-sum example, it is also robust to perturbations of the payoffs. Thus, the main work that remains to be done after that is to "thicken" its equilibrium payoff sets, and to make it symmetric. This is done in Section 3.5. The zero-sum example that we are going to construct in Sections 3.1-3.4 is largely inspired from [START_REF] Ziliotto | Zero-sum repeated games: counterexamples to the existence of the asymptotic value and the conjecture maxmin= lim v(n)[END_REF], which is a hidden stochastic game where (v δ ) oscillates between 1/2 and 5/9. For the reader that is not familiar with it, let us remind the motivation behind this kind of example. Recall that any hidden stochastic game is equivalent to a stochastic game where the state corresponds to the observed common belief of players. Since the belief space is infinite, the convergence result of Bewley and Kohlberg [START_REF] Bewley | The asymptotic theory of stochastic games[END_REF] does not apply, and prior to [START_REF] Ziliotto | Zero-sum repeated games: counterexamples to the existence of the asymptotic value and the conjecture maxmin= lim v(n)[END_REF], this was one of the simplest model where convergence of (v δ ) was still open. To make the presentation easier, we present first a stochastic game with infinite state space (Sections 3.1-3.2-3.3), that corresponds to the equivalent stochastic game of the final example. Then, in Section 3.4, we explain how this stochastic game can be encoded in a hidden stochastic game with finite state space.

A Markov chain on [0,1]

Given a parameter α ∈ (0, 1), we consider the following Markov chain with state variable q in [0, 1] and initial state q 1 = 1. Time is discrete, and if q t is the state of period t, then with probability α the next state q t+1 is α q t and with probability 1 -α the next state q t+1 is 1.

0 αq q 1 α 1 -α
Because of the transitions, the set of states that can be reached is the countable set {α a , a ∈ N}. This Markov chain can be viewed as follows: there is an infinite sequence X 1 , ..., X t , ... of independent Bernoulli random variables with success parameter α for t ≥ 2, and with X 1 = 0. At any period t the state of the Markov chain is α a if and only if the last a (but not a + 1) realizations of the Bernoulli variables have been successful, i.e. iff X t-a = 0 and X t = 1 for t -a + 1 ≤ t ≤ t.

In the next subsection, the variable q will be interpreted as a risk variable. Indeed, assume that a decision-maker observes the realizations of the Markov chain, and has to decide as a function of q when he will take a risky action, having probability of success 1 -q and probability of failure q. He would like q to be as small as possible, but time is costly and payoffs are δ-discounted, with δ ∈ [0, 1). For a in N, we denote by T a the stopping time of the first period for which the risk is α a , i.e.

T a = inf{t ≥ 1, q t ≤ α a }.

If a = 0, then T a = 1. If a ≥ 1, T a is a random variable which law can be computed by induction. Indeed, we have:

T a = T a-1 + 1 X 1+T a-1 =1 + 1 X 1+T a-1 =0 T a , = 1 + T a-1 + 1 X 1+T a-1 =0 (T a -1), (1)
where T a has the same law as T a and is independent from X 1+T a-1 . Consequently,

E(T a ) = 1 + E(T a-1 ) α .
The quantity E(T a ) grows exponentially with a, and this is an important feature of our counterexample: reaching the risk level α a+1 requires 1/α more time than reaching the risk level α a on average. The expectation of δ Ta will play an important role in the sequel. It can be easily computed by induction, since equation [START_REF] Barrett | Sensitivity of collective action to uncertainty about climate tipping points[END_REF] implies that E(δ Ta ) = E(δ T a-1 )((1 -α)E(δ Ta ) + αδ). We get 10 :

Lemma 3.1. ∀a ∈ IN, E(δ Ta ) = 1 -αδ 1 -α + (1 -δ)α -a δ -a-1 .

A Markov Decision Process on [0,1]

We introduce a player who observes the realizations of the above Markov chain and can choose as a function of the state q when he will take a risky action, having probability of success 1 -q and probability of failure q. In case of success, the payoff of the player will be R at all subsequent stages, where R is a fixed positive reward. Before he takes the risky action, the stage payoff is 0. Moreover, if the risky action is unsuccessful, the payoff is 0 at any subsequent stage. Overall payoffs are discounted with discount factor δ.

0 αq q 1 0 * R * α 1 -α q 1 -q 10
One can see also e.g. Lemma 2.2 and Proposition 2.6 in [START_REF] Philippou | A generalized geometric distribution and some of its properties[END_REF].

In this MDP with finite action set, there exists a pure stationary optimal strategy. Notice that a pure stationary strategy of the player can be represented by a non negative integer a, corresponding to the risk threshold α a . We define the a-strategy of the player as the strategy where he takes the risky action as soon as the state variable of the Markov chain does not exceed α a . The induced expected discounted payoff is E (1 -δ Ta )0 + δ Ta (α a 0 + (1 -α a )R) = R (1 -α a ) E(δ Ta ). Hence using Lemma 3.1, we obtain: Lemma 3.2. The payoff of the a-strategy in the MDP with parameter α and discount factor δ is:

(1 -α a )(1 -αδ)R 1 -α + (1 -δ)α -a δ -a-1 .
This payoff is proportional to R > 0, hence the optimal strategies do not depend on the value of R. Indeed, counting the reward in Dollars or Euros does not affect the strategic problem of the decision-maker. The problem is now to choose a non negative integer a maximizing the above payoff function.

Definition 3.3. Define, for all a in R + , s α,δ (a) = (1 -α a )E(δ Ta ) = (1 -α a )(1 -αδ) 1 -α + (1 -δ)α -a δ -a-1 ,
and let v α,δ = max a∈N s α,δ (a) denote the value of the δ-discounted MDP with parameter α and reward R = 1 .

The equality v α,δ = max a∈N s α,δ (a) is straightforward 11 because there exists a pure optimal stationary strategy in the δ-discounted MDP. The parameter α being fixed, we are now interested in maximizing s α,δ for δ close to 1. Differentiating the function (a → . Because δ ln(1-δ) converges to 1 as δ goes to 1, we obtain that lim δ→1 δ a * (α,δ) = 1.

α a * = 1 -δ 1 -α . Let ∆ 1 (α) = {1 -(1 -α)α 2a ,
Remark 3.5. The real α a * (α,δ) can be interpreted as the "optimal risk level". Because a * (α, δ) may not be an integer, the player will choose the closest integer smaller or larger than a * (α, δ). Hence, the case where a * (α, δ) is an integer, that is, when δ lies in ∆ 1 (α), is good for the player. On the contrary, the case where δ lies in ∆ 2 (α) is unfavorable to the player. Proposition 3.6.

1) v α,δ ---→ δ→1 1.

2) Fix α < 1/16. For δ ∈ ∆ 1 (α) high enough, the a * (α, δ)-strategy is optimal in the MDP and

lim δ→1,δ∈∆ 1 (α) 1 -v α,δ √ 1 -δ = 2 √ 1 -α .
3) For all α, lim inf δ→1,δ∈∆ 2 (α)

1 -v α,δ √ 1 -δ ≥ 1 α 1/2 (1 -α) .
The convergence property in 1) is very intuitive: when δ is high, the decision-maker can wait for the state variable to be very low, so that he takes the risky action with high probability of success. Points 2), when α < 1/16, and 3) give asymptotic expansions for the value v α,δ when δ goes to 1, respectively of the form

v α,δ = 1 -2 1-δ 1-α + √ 1 -δ ε α (δ) for δ ∈ ∆ 1 (α) and v α,δ ≤ 1 - 1-δ α 1/2 (1-α) + √ 1 -δ ε α (δ) for δ ∈ ∆ 2 (α), with lim δ→1 ε α (δ) = lim δ→1 ε α (δ) = 0.
The proof of Proposition 3.6 is based on simple computations that are presented in the Appendix.

A zero-sum stochastic game with perfect information

Fix two parameters α and β in (0, 1), and define a 2-player zero-sum stochastic game Γ α,β with infinite state space:

X = {(1, q), q ∈ [0, 1]} ∪ {(2, l), l ∈ [0, 1]} ∪ {0 * , 1 * } .
The initial state is (2, 1). The sum of the payoffs of the players is constant12 equal to 1. States 0 * and 1 * are absorbing states with, respectively, payoffs 0 and 1 to player 1. The payoffs only depend on the states, and the payoff of player 1 is 0 in a state of the form (1, q), and 1 in a state of the form (2, l). Each player has 2 actions: Wait or Jump. Transitions in a state (1, q) are controlled by player 1 only: if player 1 Waits in state (1, q), then the next state is (1, αq) with probability α and (1, 1) with probability 1 -α, as in the MDP of subsection 3.2. If player 1 Jumps in state (1, q), then the next state is 0 * with probability q and (2, 1) with probability 1 -q. Similarly, transitions in a state (2, l) are controlled by player 2 only: if player 2 Waits in state (2, l), then the next state is (2, βl) with probability β and (2, 1) with probability 1 -β, and if player 2 Jumps in state (2, l), then the next state is 1 * with probability l and (1, 1) with probability 1 -l. Payoffs are discounted with discount factor δ ∈ [0, 1), and the value of the stochastic game is denoted by v α,β,δ .

The strategic aspects of this game display strong similarities with those of the previous MDP. Indeed, Player 1's payoff is 0 in 0 * and in all states (1, q), q ≥ 0, and 1 in 1 * and in all states (2, l), l ≥ 0. Starting from state (1,1), the only possibility for Player 1 to obtain positive payoffs is to Jump at some period to try to reach the state (2, 1). If he waits for the state to be (1, q 0 ) with q 0 small, then the risk of reaching the state 0 * after jumping is low. Nonetheless, each period in a state (1, q), q ≥ 1 gives him a null payoff, thus he should not wait too long. The situation is symmetric for player 2. Player 1 Player 2

J 1 , 1 -q J 2 , 1 -l (1, 1)
(1, q)

(1, αq)

(1, 0) 0 * (2, 1)
(2, l)

(2, βl)

(2, 0)

1 * J 1 , q W 1 , α W 1 , 1 -α J 2 , l W 2 , β W 2 , 1 -β Figure 1: The stochastic game Γ α,β
As the game is discounted and states are controlled by a single player, it follows that there exists a pure stationary optimal strategy for each player 13 . Definition 3.7. For a in N, the a-strategy of Player 1 is the strategy where Player 1 Jumps in a state (1, q) if and only if q ≤ α a . Similarly, for b in N the b-strategy of Player 2 is the strategy where Player 2 Jumps in a state (2, l) if and only if l ≤ β b . Denote by g α,β,δ (a, b) the payoff of Player 1 in the stochastic game where Player 1 uses the a-strategy and Player 2 uses the b-strategy.

Assume that Player 2 uses a b-strategy. Then Player 1 faces a MDP with finite action sets, hence he has a pure stationary best reply, which is an a-strategy. Similarly, if Player 1 uses an a-strategy, Player 2 has a b-strategy best reply. We thus consider the game restricted to a-and b-strategies. Lemma 3.8. For a and b in N,

g α,β,δ (a, b) = 1 -s β,δ (b) 1 -s α,δ (a)s β,δ (b)
.

Proof: Recall that s α,δ (a) = (1 -α a )E α (δ Ta ), where T a is the random variable defined in subsection 3.1 and E α denotes the expectation for the Markov chain with parameter α. Similarly, one has

s β,δ (b) = (1 -β b )E β (δ T b ).
Starting from the initial state, with probability β b the first Jump of player 2 will end up in 1 * and the payoff for player 1 will be 1 in each period, and with probability 1 -β b the game will first stay T b stages in a state controlled by player 2 and then reach the state [START_REF] Barrett | Sensitivity of collective action to uncertainty about climate tipping points[END_REF][START_REF] Barrett | Sensitivity of collective action to uncertainty about climate tipping points[END_REF]. This gives:

g α,β,δ (a, b) = β b + (1 -β b )E β (1 -δ T b ) + δ T b g α,β,δ (a, b) ,
where g α,β,δ (a, b) denotes the payoff of the a-strategy against the b-strategy in the game with initial state (1, 1). Thus,

g α,β,δ (a, b) = 1 + s β,δ (b)(-1 + g α,β,δ (a, b)). Similarly, g ,α,β,δ (a, b) = α a 0 + (1 -α a )E α (δ Ta )g α,β,δ (a, b), hence g α,β,δ (a, b) = s α,δ (
a)g α,β,δ (a, b), and Lemma 3.8 is proved.

Assume that Player 2 plays a b-strategy, and denote by R the best payoff that Player 1 can obtain against this strategy from the state (1, 1) (if the play never reaches this state, then player 1 has nothing to do and gets a payoff of 1 in each period). We have seen that Player 1 has a best reply in the form of a a-strategy, and finding the best a is equivalent to finding a pure optimal strategy in the MDP of subsection 3.2 with reward R. As we have seen in subsection 3.2, this optimal value for a does not depend on R, and simply maximizes s α,δ (a). This implies that the best reply of player 1 does not depend on b, and the corresponding a-strategy is a dominant strategy of player 1 in the zero-sum stochastic game restricted to pure stationary strategies. The existence of dominant strategies in a zero-sum game is rather uncommon, and this is an important property of the present example. It can be verified analytically by looking at the function g α,β,δ : for all b, it is increasing in s α,δ (a), and for all a, it is decreasing in s β,δ (b). This proves 1) in the proposition below. Proposition 3.9. Let a # and b # be respectively maximizers of s α,δ (a) over a in N, and of s β,δ (b) over b in N.

1) The a # -strategy, resp. the b # -strategy, is a dominant strategy for player 1, resp. player 2, in the zero-sum stochastic game restricted to pure stationary strategies.

2) The a # -strategy, resp. the b # -strategy, is an optimal strategy for player 1, resp. player 2, in the zero-sum stochastic game Γ α,β .

3) The value of Γ α,β satisfies:

v α,β,δ = 1 -v β,δ 1 -v α,δ v β,δ
.

Proof: 2) The strategy profile induced by (a # , b # ) is a Nash equilibrium of the game Γ α,β restricted to pure stationary strategies. Against any pure stationary strategy of one player, the other one has a pure stationary best reply, thus this strategy profile is indeed a Nash equilibrium of the game Γ α,β . Hence the value of Γ α,β is the payoff induced by this strategy profile, and 3) follows.

Notice that v α,α,δ =

1 1+v α,δ ---→ δ→1 1
2 . We are interested in cases where α = β, and the next proposition is a building brick for our global construction. Proposition 3.10. For each ε > 0, there exists n 0 ∈ N * such that for all n ≥ n 0 , and α := 1/n and β := 1/(n + 1), we have:

lim sup δ→1 v α,β,δ ≥ 1 -ε, and lim inf δ→1 v α,β,δ ≤ ε.
Remark 3.11. The intuition for the result is the following. When the discount factor lies both in ∆ 1 (α) and ∆ 2 (β), Player 1 can take the "optimal level of risk", whereas Player 2 can only take the optimal level of risk time (n + 1) or divided by (n + 1). This gives a significant advantage to Player 1, and when n is large, the discounted value is close to 1. When the discount factor lies both in ∆ 2 (α) and ∆ 1 (β), Player 2 can take the "optimal level of risk", whereas Player 1 can only take the optimal level of risk time n or divided by n. This gives a significant advantage to Player 2, and when n is large, the discounted value is close to 0.

Proof: We proceed in 2 steps.

Step 1: Define ∆ 1 (α, β) := ∆ 1 (α) ∩ ∆ 2 (β), that is:

∆ 1 (α, β) = {δ ∈ [0, 1), ∃(a, b, η) ∈ N × N * × [-3/2, 3/2], δ = 1 -(1 -α)α 2a = 1 -(1 -β)β 2b+η }.
Discount factors in ∆ 1 (α, β) simultaneously favor player 1 and disfavor player 2 in their respective MDP: for δ ∈ ∆ 1 (α, β), we have by Proposition 3.

6 that v α,δ = 1 -2 1-δ 1-α + √ 1 -δ ε α (δ) and v β,δ ≤ 1 - 1-δ β 1/2 (1-β) + √ 1 -δ ε β (δ), with lim δ→1 ε α = lim δ→1 ε β = 0. Because v α,β,δ = 1-v β,δ 1-v α,δ v β,δ
is decreasing in v β,δ , we obtain:

v α,β,δ ≥ 1-δ β 1/2 (1-β) - √ 1 -δ ε β (δ) 1 -1 -2 1-δ 1-α + √ 1 -δ ε α (δ) 1 - 1-δ β 1/2 (1-β) + √ 1 -δ ε β (δ) , ≥ 1-δ β 1/2 (1-β) - √ 1 -δ ε β (δ) 1-δ β 1/2 (1-β) + 2 1-δ 1-α + √ 1 -δ ε (δ)
, where lim δ→1 ε = 0.

If ∆ 1 (α, β) contains discount factors arbitrarily close to 1, this implies that lim inf

δ→1,δ∈∆ 1 (α,β) v α,β,δ ≥ 1 1 + 2 β 1/2 (1-β) 1-α . ( 2
)
In the same vein, define ∆ 2 (α, β) := ∆ 2 (α) ∩ ∆ 1 (β). Discount factors in ∆ 2 (α, β) simultaneously disfavor player 1 and favor player 2 in their respective MDP, and similar computations as above show that if ∆ 2 (α, β) contains discount factors arbitrarily close to 1, lim sup

δ→1,δ∈∆ 2 (α,β) v α,β,δ ≤ 1 1 + 1 2 (1-β) α 1/2 (1-α)
.

(

) 3
Our goal, inspired by ( 2) and ( 3), is now to prove that there exist α and β arbitrarily small such that both ∆ 1 (α, β) and ∆ 2 (α, β) contain discount factors arbitrarily close to 1.

Step 2:

We want to prove that for n large enough, there exists an infinite number of (a, b, η)

∈ N * 2 × [-3/2, 3/2] satisfying 1 -(1 -α)α 2a = 1 -(1 -β)β 2b+η , that is, (ln β) -1 [ln((1 -α)/(1 -β)) + 2a ln(α)] = 2b + η. Let A(α, β) := (ln β) -1 ln((1 -α)/(1 -β))
and B(α, β) := (ln β) -1 ln(α) -1. The last equation can be written as

A(α, β) + 2B(α, β)a = 2(b -a) + η.
If B(α, β) < 1/4, then this equation has an infinite number of solutions (a, b, η)

∈ N * 2 × [-3/2, 3/2].
Set α n := 1/n and β n := 1/(n + 1). For n large enough, we have B(α n , β n ) < 1/4. This implies that ∆ 1 (α n , β n ) contains discount factors arbitrarily close to 1, and the proof is similar for ∆ 2 (α n , β n ). Let ε and n 0 ∈ N such that for all n ≥ n 0 , both ∆ 1 (α n , β n ) and ∆ 2 (α n , β n ) contain discount factors arbitrarily close to 1, and 1 + 2 2) and ( 3) yield lim sup δ→1 v α,β,δ ≥ 1 -ε, and lim inf δ→1 v α,β,δ ≤ ε, and the proof of proposition 3.10 is complete.

β 1/2 n (1-βn) 1-αn -1 ≥ 1 -ε, and 1 + 1 2 (1-βn) α 1/2 n (1-αn) -1 ≤ ε. For n ≥ n 0 , equations (

A zero-sum hidden stochastic game

The MDP and games considered so far have perfect information and infinite state space. We now mimic the previous construction with a hidden stochastic game with 6 states and 6 public signals.

Given α and β two parameters in (0, 1), the HSG Γ * (α, β) is defined as follows. The set of states is K = {(1, 1), (1, 0), (2, 1), (2, 0), 1 * , 0 * }, and the set of public signals is S = {s 1 , s 1 , s * 1 , s 2 , s 2 , s * 0 }. The players perfectly observe past actions and public signals, but not current states. As in the previous stochastic game, the sum of the payoffs of the players is always 1, and the states 0 * and 1 * are absorbing. The payoffs only depend on the states, player 1 has payoff 0 in states 0 * , (1, 0) and (1, 1), and payoff 1 in states 1 * , (2, 0) and [START_REF] Bewley | The asymptotic theory of stochastic games[END_REF][START_REF] Barrett | Sensitivity of collective action to uncertainty about climate tipping points[END_REF]. Each player has 2 actions corresponding to Wait and Jump, action sets are I = {W 1 , J 1 } and J = {W 2 , J 2 }. The initial probability π selects with probability 1 the state (2, 1) and the signal s 2 , thus players know that at period 1 the game is in state [START_REF] Bewley | The asymptotic theory of stochastic games[END_REF][START_REF] Barrett | Sensitivity of collective action to uncertainty about climate tipping points[END_REF]. Once in the absorbing state 0 * , resp. 1 * , the play stays there forever and the public signal is s * 0 , resp. s * 1 . Transitions from states (1, 0) and (1, 1) only depend on the action of player 1, whereas transitions from (2, 0) and (2, 1) only depend on the action of player 2. More precisely:

-If player 1 Jumps in state [START_REF] Barrett | Sensitivity of collective action to uncertainty about climate tipping points[END_REF][START_REF] Barrett | Sensitivity of collective action to uncertainty about climate tipping points[END_REF], the play goes to the absorbing state 0 * and the public signal is s * 0 , i.e. q((1, 1), J 1 ) selects (0 * , s * 0 ) a.s.

-If player 1 Jumps in state (1, 0), the play goes to state (2, 1) and the public signal is s 2 , i.e. q((1, 0), J 1 ) selects ((2, 1), s 2 ) a.s.

-If player 1 Waits in state (1, 1), the transition is as follows: q((1, 1), W 1 ) selects ((1, 1), s 1 ) with probability 1 -α, ((1, 1), s 1 ) with probability α 2 and ((1, 0), s 1 ) with probability α(1 -α).

-If player 1 Waits in state (1, 0), the transition is as follows: q((1, 0), W 1 ) selects ((1, 1), s 1 ) with probability 1 -α, and ((1, 0), s 1 ) with probability α.

Transitions from the states controlled by player 2 are defined symmetrically: q((2, 1), J 2 ) selects (1 * , s * 1 ) a.s., q((2, 0), J 2 ) selects ((1, 1), s 1 ) a.s., q((2, 1), W 2 ) selects ((2, 1), s 2 ) with probability 1 -β, ((2, 1), s 2 ) with probability β 2 and ((2, 0), s 2 ) with probability β(1 -β), and finally q((2, 0), W 2 ) selects ((2, 1), s 2 ) with probability 1 -β and ((2, 0), s 2 ) with probability β.

Payoffs are discounted with discount factor δ ∈ [0,

(2, 0)

1 * W 1 , 1 -α, s 1 W 1 , α(1 -α), s 1 W 1 , α 2 , s 1 W 1 , α, s 1 W 1 , 1 -α, s 1 J 1 , s 2 J 1 , s * 0 W 2 , 1 -β, s 2 W 2 , β(1 -β), s 2 W 2 , β 2 , s 2 W 2 , β, s 2 W 2 , 1 -β, s 2 J 2 , s 1 J 2 , s * 1 Figure 2: Transitions in Γ * (α, β)
Signals in states (1, 0) and (1, 1) are either s 1 or s 1 , and signals in states (2, 0) and (2, 1) are either s 2 or s 2 . Accordingly, the public signal always informs the players of the element of the partition {{(1, 0), (1, 1)}, {2, 0), (2, 1)}, {0 * }, {1 * }} that contains the current state, and the game has known payoffs.

In Γ * (α, β), player 1 would like to Jump in state (1, 0) and to Wait in state (1, 1), but the current state is not fully known to the players. Because of the previous partition, the belief of the players over the current state has at most two points in its support. Assume that this belief corresponds to the state being (1, 1) with probability q and (1, 0) with probability 1 -q. The current payoff is 0, and the transition only depends on player 1's action:

-If player 1 Jumps, the new state is 0 * with probability q and (2, 1) with probability 1 -q.

-If player 1 Waits, with probability 1 -α the public signal will be s 1 . By Bayes' rule the players can deduce that the new state is almost surely [START_REF] Barrett | Sensitivity of collective action to uncertainty about climate tipping points[END_REF][START_REF] Barrett | Sensitivity of collective action to uncertainty about climate tipping points[END_REF]. With probability α, the public signal is s 1 and the probability that the transition selects ((1, 1), s 1 ) is qα 2 . Consequently, by Bayes' rule the belief of the players over the new state is : (1, 1) with probability qα and (1, 0) with probability 1 -qα.

Consequently, the transitions and the payoffs here perfectly mimic those of the stochastic game of subsection 3.3. The equivalent stochastic game associated to the HSG Γ * (α, β) (see the beginning of section 2) corresponds to the game Γ(α, β), up to adding the observation of the public signal at the beginning of each period. This public signal plays no role on the payoffs and could only be used as a correlation device for the players, but in a zero-sum context this has no influence on the value. We obtain:

Proposition 3.12. The value of the δ-discounted hidden stochastic game Γ * (α, β) is the value v α,β,δ of the δ-discounted stochastic game Γ(α, β).

Using proposition 3.10, we obtain the following corollary, improving the result of [START_REF] Ziliotto | Zero-sum repeated games: counterexamples to the existence of the asymptotic value and the conjecture maxmin= lim v(n)[END_REF].

Corollary 3.13. For ε > 0, there exists a zero-sum hidden stochastic game with payoffs in [-1, 1] such that lim sup δ→1 v δ ≥ 1 -ε and lim inf δ→1 v δ ≤ -1 + ε.

The non-zero-sum game with oscillating equilibria

Fix ε ∈ (0, 5 12 ] and r in (0, ε/5). We finally construct a non zero-sum HSG Γ satisfying the conditions of Theorem 2.3. By Proposition 3.10, it is possible to fix α and β such that: lim inf δ→1 v α,β,δ < ε -5r and lim sup

δ→1 v α,β,δ > ε + 5r. Define ∆ 1 = {δ ∈ [1 -2r, 1), v α,β,δ < ε -5r} and ∆ 2 = {δ ∈ [ 1 1+2r , 1), v α,β,δ > ε + 5r}.
Because we want all payoffs of Γ to be in [0, 1], we first modify the zero-sum HSG Γ * (α, β) of subsection 3.4 by transforming all payoffs (1, 0) into (1 -r, r) and all payoffs (0, 1) into (r, 1 -r). That is, we apply the affine increasing transformation (x → r + (1 -2r)x) to the payoffs, and the game remains constant-sum. We obtain a new HSG Γ 1 with each payoff in [r, 1 -r], and the δ-discounted value of this new game is simply v δ = r + (1 -2r)v α,β,δ . We also define the HSG Γ 2 as the game Γ 1 where the identity of the players are exchanged: player 1 in Γ 2 plays the role of player 2 in Γ 1 , and vice-versa. Clearly, the value of Γ 2 is 1 -v δ . This game is crucial to enforce the symmetry property of the final example.

We now define our final HSG Γ. The states are the 6 states (1, 1), (1, 0), (2, 1), (2, 0), 1 * , 0 * of Γ 1 , 4 more states14 corresponding to the states (1, 1), (1, 0), (2, 1), (2, 0) of Γ 2 , and in addition 3 states k 1 , (ε, ε) * and (1 -ε, 1 -ε) * : k 1 is the initial state and is known to the players, and (ε, ε) * and (1 -ε, 1 -ε) * are absorbing states where the payoffs will partly depend on the actions played. Actions sets are I = {W 1 , J 1 } × {T, B} and J = {W 2 , J 2 } × {L, R}. The game Γ is defined as the "independent sum" of two different games played in parallel, the first game evolving according to the first coordinate of the actions, and the second game evolving according to the second coordinate of the actions. 1) In the first period, in state k 1 , the first coordinate of the players' actions 15 determines the following continuation game to be played:

W 2 J 2 W 1 (ε, ε) * Γ 2 J 1 Γ 1 (1 -ε, 1 -ε) * If (W 1 , W 2 ), resp. (J 1 , J 2
) is played in period 1, the game reaches the absorbing state (ε, ε) * , resp. (1 -ε, 1 -ε) * (considering absorbing payoffs simplify the analysis). If (W 1 , J 2 ), resp. (J 1 , W 2 ), is played in period 1, then from period 2 on the hidden stochastic game Γ 2 , resp. Γ 1 , is played. The payoffs of the first game in period 1 are respectively defined as (ε, ε), (1 -ε, 1 -ε), (0, 0) and (0, 0) if (W 1 , W 2 ), (J 1 , J 2 ), (W 1 , J 2 ) and (J 1 , W 2 ) is played.

2) In addition, in every period of Γ the players play, through the second coordinate of their actions, the following bimatrix game G, independently of everything else.

L R T r, r -r, r B r, -r -r, -r
The above game will ensure that the full dimensionality property of the example is satisfied. In each period, the payoffs in Γ are the sum of the payoffs of the two games. For instance, if the state is (ε, ε) * and the second components of the actions are (B, L), then the stage payoffs are ε + r for player 1 and ε -r for player 2. If (J 1 , W 2 ) is played in the first period, then at any subsequent stage the payoffs of the players are the payoffs in Γ 1 plus the payoffs in G. One can easily check that all payoffs lie in [0, 1].

We recall that past actions are perfectly observed. The public signals are those of Γ 1 or Γ 2 when these games are played, and we add one specific public signal for the initial state and each absorbing state (ε, ε) * and (1 -ε, 1 -ε) * , so that Γ has 13 public signals and is a hidden stochastic game with known payoffs. Moreover, the game is symmetric between the players.

Let us now analyze the game Γ. Note first that in G, each player chooses the payoff of the other player, hence any profile is a Nash equilibrium. We deduce that the equilibrium payoff set of G is the square of feasible payoffs [-r, r] 2 . For each initial probability and discount factor, the game described in 1) has a sequential equilibrium yielding some payoff (x, y). Combining independently such equilibrium with any sequential equilibrium of the repetition of G gives a sequential equilibrium of Γ. Thus, the square centered in (x, y) with side 2r is included in the set of sequential equilibrium of Γ for this initial probability and discount factor. This proves the third item of Theorem 2.3.

From now on, we consider the game Γ with initial state k 1 . The idea is quite simple: for δ in ∆ 1 , v δ is significantly smaller than ε and all equilibria of Γ will play (W 1 , W 2 ) in the first period; whereas for δ in ∆ 2 , v δ is significantly larger than ε and all equilibria of Γ play (J 1 , J 2 ) in period 1.

Proposition 3.14.

1) For δ in ∆ 1 , E δ = E δ is the square [ε-r, ε+r] 2 ,
and this is also the set of communication equilibria of the δ-discounted game, as well as the set of stationary equilibrium payoffs of the associated stochastic game.

2

) For δ in ∆ 2 , E δ = E δ is the square [1 -ε -r, 1 -ε + r] 2
, and this is also the set of communication equilibria of the δ-discounted game, as well as the set of stationary equilibrium payoffs of the associated stochastic game.

Proof: First consider, for any discount factor δ, the subgame induced by Γ after (J 1 , W 2 ) has been played in period 1, discounted from period 2 on. By playing optimally in the Γ 1 component, player 1 can secure a payoff of v δ -r, whereas player 2 can secure a payoff of 1 -v δ -r. Because the sum of the payoffs is not greater than 1 + 2r, all equilibrium payoffs of this subgame lie in the set [v δ -r, v δ +3r]×[1-v δ -r, 1-v δ +3r]. Symmetrically, equilibrium payoffs of the subgame induced by Γ after (J 1 , W 2 ) has been played in period 1, belong to the square

[1-v δ -r, 1-v δ +3r]×[v δ -r, v δ +3r].
1) Fix a discount factor δ in ∆ 1 . We have v δ = r + (1 -2r)v α,β,δ , so δv δ < ε -4r. Consider a Nash equilibrium (σ, τ ) of the δ-discounted game Γ, and denote by x, resp. y, the probability that σ plays W 1 , resp. τ plays W 2 at stage 1. We will show that x = y = 1. First, assume for the sake of contradiction that x < 1. By playing W 1 at period 1 and optimally in Γ 2 afterwards, player 1 can get a payoff higher than:

A := y(ε -r) + (1 -y)(δ(1 -v δ ) -r).
This should not exceed the payoff obtained against τ by playing J 1 at period 1 and following σ afterwards, and this payoff is not greater than

B := y(δ(v δ + 3r) + (1 -δ)r) + (1 -y)(1 -ε + r),
because if y > 0 the continuation strategies after (J 1 , W 2 ) should form a Nash equilibrium of the corresponding subgame. Because δv δ < ε -2r(1 + δ), we obtain that ε -r > δ(v δ + 3r) + (1 -δ)r. Because δv δ < ε -4r and δ ≥ 1 -2r, we have δv δ < ε -2r + δ -1, and this implies δ(1 -v δ ) -r > 1 -ε + r. Consequently, for all values of y in [0, 1] we have A > B, which is a contradiction. Hence we obtain x = 1, and by symmetry y = 1. All Nash equilibrium of Γ play W 1 and W 2 in period 1, and the set of Nash equilibrium payoffs E δ is included in the square [ε -r, ε + r] 2 . The players can combine (W 1 , W 2 ) in period 1 with the repetition of any given mixed Nash equilibrium of G, therefore any point in the square can be achieved at equilibrium, and E δ = [ε -r, ε + r] 2 . Considering sequential equilibria, or introducing a correlation device, even with communication, would not modify the above proof. This is the same with stationary equilibria of the associated stochastic game with state space ∆(K). This proves part 1) of the proposition.

2) We proceed similarly for δ in ∆ 2 . Showing δv δ > ε + 2r + δ(1 + 2r) -1 ≥ ε + 2r is enough to show that any Nash equilibrium of the δ-discounted game Γ plays (J 1 , J 2 ) with probability one at stage 1.

Because ε + r < 1 -ε -r, Proposition 3.14 clearly implies that no converging selection of (E δ ) δ exists. We now consider small perturbations of the payoffs. Let us first explain informally why they do not change significantly the equilibrium payoffs. Indeed, they do not change significantly the values of Γ 1 and Γ 2 . Thus, for δ in ∆ 1 , v δ is still significantly smaller than ε and all equilibria of Γ play (W 1 , W 2 ) in the first period; whereas for δ in ∆ 2 , v δ is much larger than ε and all equilibria of Γ play (J 1 , J 2 ) in period 1. Because the repeated bimatrix game G is also robust to small perturbations, the result follows. Let us now formalize these ideas. For η ∈ [0, r(ε-5r) Proposition 3.15.

1) For all

δ in ∆ 1 , E δ (η) ⊂ [ε -r -2η, ε + r + 2η] 2 . Moreover , lim η→0 lim δ→1,δ∈∆ 1 E δ (η) = lim η→0 lim δ→1,δ∈∆ 1 E δ (η) = E 1 , and lim δ→1,δ∈∆ 1 lim sup η→0 d(E δ (η), E 1 ) = lim δ→1,δ∈∆ 1 lim sup η→0 d(E δ (η), E 1 ) = 0. 2) For all δ in ∆ 2 , E δ (η) ⊂ [1 -ε -r -2η, 1 -ε + r + 2η] 2 . Moreover , lim η→0 lim δ→1,δ∈∆ 2 E δ (η) = lim η→0 lim δ→1,δ∈∆ 2 E δ (η) = E 2 , and lim δ→1,δ∈∆ 2 lim sup η→0 d(E δ (η), E 2 ) = lim δ→1,δ∈∆ 2 lim sup η→0 d(E δ (η), E 2 ) = 0.
3) There is no converging selection (x δ ) δ of (E δ (η)) δ .

4) The game Γ(η) has no uniform equilibrium payoff.

The proof is in the Appendix, and concludes the proof of Theorem 2.3. Lemma 4.1. For all α and δ in (0, 1),

1 -2 δ -a * -1 1 -δ 1 -α ≤ s α,δ (a * ) ≤ 1 -2 1 -δ 1 -α + 3 1 -δ 1 -α , (4)
And if a ≥ 0 is such that |a -a * | ≥ 1/4,

s α,δ (a) ≤ 1 - 1 √ α 1/2 1 -δ 1 -α + 1 -δ 1 -α (α + 1 α 1/2 ). (5)
This lemma implies that the maximum of s α,δ over R + is asymptotically 1 -

2 1-δ 1-α + o( √ 1 -δ),
whereas the maximum of s α,δ over N is asymptotically smaller than 1 -1

√ α 1/2
1-δ 1-α . Proof of lemma 4.1: For the LHS of (4), it is enough to notice that :

s α,δ (a * ) ≥ 1 -α a * 1 + 1-δ 1-α α -a * δ -a * -1 = 1 -1-δ 1-α 1 + 1-δ 1-α δ -a * -1 ≥ 1 -1-δ 1-α δ -a * -1 1 + 1-δ 1-α δ -a * -1 ≥ 1 -2 1 -δ 1 -α δ -a * -1 .
For inequality (5), we introduce l α,δ (a) = 4 and we write:

1-α a 1+ 1-δ 1-α α -a δ -a-1 . If a ≤ a * -1/4, we have α a ≥ α a * -1/4 = 1-δ α 1/2 (1-α) , and l α,δ (a) ≤ 1 -α a ≤ 1 - 1-δ α 1/2 (1-α) . If a ≥ a * + 1/4, we have α -a ≥ α -a * -1/
l α,δ (a) ≤ 1 1 + 1-δ 1-α α -a ≤ 1 1 + 1-δ α 1/2 (1-α) ≤ 1 - 1 -δ α 1/2 (1 -α) + 1 -δ α 1/2 (1 -α) .
And we get inequality (5) since :

s α,δ (a) = l α,δ (a) + (1 -δ) α 1-α l α,δ (a) ≤ l α,δ (a) + (1 -δ) α 1-α .
Finally notice that :

l α,δ (a * ) ≤ 1-α a * 1+ 1-δ 1-α α -a * = 1-1-δ 1-α 1+ 1-δ 1-α ≤ 1 -2 1-δ 1-α + 2 1-δ 1
-α , and use s α,δ (a * ) ≤ l α,δ (a * ) + 1-δ 1-α to obtain the RHS of (4), concluding the proof of lemma 4.1.

We now prove 2) of proposition 3.6. Fix α < 1/16, we have 1 α 1/4 > 2 so for δ close enough to 1, 2δ

-a * -1 + 1-δ 1-α (α + 1/ √ α) < 1 α 1/4 , which implies that: 1 -2 δ -a * -1 1-δ 1-α > 1 -1 √ α 1/2 1-δ 1-α + 1-δ 1-α (α + 1/ √ α). For δ ∈ ∆ 1 (α)
, the a * -strategy is available in the MDP, and this inequality shows that it is an optimal strategy. v α,δ = s α,δ (a * ), and (4) of lemma 4.1 implies lim δ→1,δ∈∆ 1 (α)

1-v α,δ 2
1-δ 1-α = 1.
We finally prove 3) of proposition 3.6, and consider δ ∈ ∆ 2 (α). The pure stationary strategies available in the MDP are a-strategies, with |a -a * | ≥ 1/4. Inequality (5) of lemma 4.1 then implies that:

v α,δ ≤ 1 -1 √ α 1/2 1-δ 1-α + 1-δ 1-α (α + 1/α 1/2
), hence the result.

Proof of Proposition 3.15

For any discount factor, the perturbed game issued from Γ 1 may no longer be zero-sum, but the quantity that player 1 can guarantee (whatever the strategy of the other player) in this game is close to v δ . More precisely, in the subgame induced by Γ(η) after (J 1 , W 2 ) has been played in period 1, player 1 can secure a payoff of v δ -r -η, whereas player 2 can secure a payoff of 1 -v δ -r -η.

Since the sum of the payoffs is now not greater than 1 + 2r + 2η, all equilibrium payoffs of this subgame lie in the set

[v δ -r -η, v δ + 3r + 3η] × [1 -v δ -r -η, 1 -v δ + 3r + 3η].
Symmetrically, all equilibrium payoffs of the subgame induced by Γ(η) after (W 1 , J 2 ) has been played in period 1, are in the set

[1 -v δ -r -η, 1 -v δ + 3r + 3η] × [v δ -r -η, v δ + 3r + 3η].
1) Fix δ in ∆ 1 , we have v δ < r + (1 -2r)(ε -5r) and δ ≥ 1 -2r. This implies:

v δ ≤ min{ε -4(r + η), ε -2(r + η) + δ -1}. (6)
Mimicking the proof of 1) of proposition 3.14, we obtain

A(η) = y(ε-r -η)+(1-y)(δ(1-v δ )-r -η) and B(η) = y(δ(v δ + 3r + 3η) + (1 -δ)(r + η)) + (1 -y)(1 -ε + r + η)
, so that A(η) and B(η) are obtained from the quantities A and B of that lemma by replacing the payoff r by the payoff r + η.

By inequality (6), we have A(η) > B(η). This implies that any δ-discounted Nash equilibrium of Γ(η) plays W 1 and W 2 at the first period, and E δ (η) ⊂ [ε -r -η, ε + r + η] 2 .

Fix now η in (0, r(ε-5r)

2

). Define Γ(η)(W 1 , W 2 ) as the subgame obtained from Γ(η) after (W 1 , W 2 ) has been played in period 1. Γ(η)(W 1 , W 2 ) is a repeated game, with stage payoffs η-close to the bimatrix:

(W 2 , L) (J 2 , L) (W 2 , R) (J 2 , R) (W 1 , T ) r + ε, r + ε r + ε, r + ε -r + ε, r + ε -r + ε, r + ε (J 1 , T )
r + ε, r + ε r + ε, r + ε -r + ε, r + ε -r + ε, r + ε (W 1 , B) r + ε, -r + ε r + ε, -r + ε -r + ε, -r + ε -r + ε, -r + ε (J 1 , B) r + ε, -r + ε r + ε, -r + ε -r + ε, -r + ε -r + ε, -r + ε By the Folk Theorem of Fudenberg and Maskin [START_REF] Fudenberg | The folk theorem in repeated games with discounting or with incomplete information[END_REF], the set E δ (η)(W 1 , W 2 ) of sequential equilibrium payoffs of Γ(η)(W 1 , W 2 ) converges, when δ goes to 1, to the set of feasible and individually rational payoffs of this game. So does E δ (η)(W 1 , W 2 ). And this set of feasible and IR payoffs converges, when η goes to 0, to the square E 1 = [-r + ε, r + ε] 2 . Since all Nash equilibria of Γ(η) play (W 1 , W 2 ) in period 1, we obtain lim η→0 lim δ→1,δ∈∆ 1 E δ (η) = lim η→0 lim δ→1,δ∈∆ 1 E δ (η) = E 1 .

Consider now the repetition of the bimatrix game G. Fix ε > 0, there exists δ such that for all δ ≥ δ and any payoff u in [-r, r] 2 , there exists a periodic sequence (i t , j t ) t of pure action profiles in {T, B} × {L, R} such that for all t 0 , playing the sequence (i t , j t ) t≥t 0 yields a δ-discounted payoff ε -close to u. Assume u = (u 1 , u 2 ) ∈ [-r + 2ε , r] 2 and η < min{ε , r(ε-5r) 2 }, we have u l -ε > -r + η for each player l = 1, 2. For δ ∈ ∆ 1 , δ ≥ δ , consider the strategy profile where: a) (W 1 , W 2 ) is played at stage 1, and for the second component of the actions, the above sequence of pure actions is played, with deviations after (W 1 , W 2 ) at stage 1 being punished by repeating (B, R) forever, and b) arbitrary fixed sequential equilibria are played after (W 1 , J 2 ), (J 1 , W 2 ) or (J 1 , J 2 ) at stage 1. This is a sequential equilibrium of the δ-discounted game Γ(η). Hence E δ (η) contains a point ε -close to u, and d(E δ (η), E 1 ) ≤ 2ε . So lim sup η→0 d(E δ (η), E 1 ) ≤ 2ε , and lim δ→1,δ∈∆ 1 lim sup η→0 d(E δ (η), E 1 ) = lim δ→1,δ∈∆ 1 lim sup η→0 d(E δ (η), E 1 ) = 0.

2) For δ in ∆ 2 , we have δv δ > δ(r + (1 -2r)(ε + 5r)). Because η < 1 2 (1 -ε -5r), we have r + (1 -2r)(ε + 5r) > ε + 4(r + η), and since ε -1 + 2(r + η) < 0, it implies r + (1 -2r)(ε + 5r) > 1 δ (ε -1 + 2(r + η)) + 1 + 2(r + η). So: δv δ > ε -1 + 2(r + η) + δ(1 + 2(r + η)).

Since δ ≥ 1 1+2r , the above also implies:

δv δ > ε + 2(r + η). (8)
We mimick the proof of 2) of proposition 3.14 and obtain quantities A (η) = y(ε + r + η) + (1y)((1-δ)(r+η)+δ(1-v δ +3r+3η)), and B (η) = y((1-δ)(-r-η)+δ(v δ -r-η))+(1-y)(1-ε-r-η). And the inequalities [START_REF] Fudenberg | The folk theorem for irreducible stochastic games with imperfect public monitoring[END_REF] and [START_REF] Gimbert | On values of repeated games with signals[END_REF] imply that B (η) > A (η), hence any δ-discounted Nash equilibrium of Γ(η) plays J 1 and J 2 at the first period. The rest of the proof of 2) is similar to the proof of 1).

3) We have ε + r + η < ε + r(1 + 1 2 ε -5r) < 1/2 since r < ε/5 and ε < 5/12. Hence there is no converging selection (x δ ) δ of (E δ (η)) δ .

4) It remains to prove that Γ(η) has no equilibrium payoff, i.e. that for ε small enough, there is no strategy profile which is an ε -equilibrium of all discounted games Γ(η) with high enough discount factors.

We proceed by contradiction, and assume that for each ε > 0, on can find a discount δ ε in (0,1), and a strategy profile (σ, τ ) = (σ ε , τ ε ) which is an ε -equilibrium of each game Γ(η) with discount δ > δ ε . Denote by x = x ε , resp. y = y ε , the probability that σ plays W 1 , resp. τ plays W 2 at stage 1. The δ-discounted payoff of player 1 induced by (σ ε , τ ε ) is by definition:

g δ 1 (σ, τ ) = E σ,τ (1 -δ) ∞ t=1
δ t-1 u 1 (k t , i t , j t ) .

We denote by g δ 1 (σ, τ |W 1 , W 2 ) the conditional payoff of player 1 given that (W 1 , W 2 ) is played at period 1, that is:

E σ,τ (1 -δ) ∞ t=1
δ t-1 u 1 (k t , i t , j t ) (i 1 = (W 1 , T ) or (W 1 , B)) and (j 1 = (W 2 , L) or (W 2 , R)) .

And we similarly define g δ 1 (σ, τ |W 1 , J 2 ), g δ 1 (σ, τ |J 1 , W 2 ), g δ 1 (σ, τ |J 1 , J 2 ) and similar quantities for player 2's payoff. We have:

g δ 1 (σ, τ ) = xyg δ 1 (σ, τ |W 1 , W 2 ) + x(1 -y)g δ 1 (σ, τ |W 1 , J 2 ) +(1 -x)yg δ 1 (σ, τ |J 1 , W 2 ) + (1 -x)(1 -y)g δ 1 (σ, τ |J 1 , J 2
). Because player 1 can secure the payoff v δ in the game Γ 1 , the fact that (σ, τ ) is an ε -equilibrium implies that:

g δ 1 (σ, τ |J 1 , W 2 ) ≥ δv δ -(r + η) -ε (1-x)y . Similarly, g δ 1 (σ, τ |W 1 , J 2 ) ≥ δ(1 -v δ ) -(r + η) -ε
x(1-y) , g δ 2 (σ, τ |W 1 , J 2 ) ≥ δv δ -(r + η) -ε x(1-y) , and g δ 2 (σ, τ |J 1 , W 2 ) ≥ δ(1 -v δ ) -(r + η) -ε (1-x)y . Since g δ 1 (σ, τ |W 1 , J 2 ) + g δ 2 (σ, τ |W 1 , J 2 ) ≤ 1 + 2r + 2η, we obtain:

g δ 1 (σ, τ |W 1 , J 2 ) ≤ 1 + 3(r + η) -δv δ + ε x(1 -y) (9)
g δ 1 (σ, τ |J 1 , W 2 ) ≤ 1 + 3(r + η) -δ(1 -v δ ) + ε y(1 -x) [START_REF] Lenton | Tipping elements in the earth's climate system[END_REF]

( 1 -

 1 α a ) 1-α+(1-δ)α -a ) and having δ go to 1 leads to the introduction of the following quantity. Definition 3.4. When δ ∈ [α, 1), we define a * = a * (α, δ) in R + such that:

  a ∈ N} be the set of discount factors δ such that a * (α, δ) is an integer, and let∆ 2 (α) = {1 -(1 -α)α 2a+η , a ∈ N * , η ∈ [-3/2, 3/2]} be the set of discount factors δ such that a * (α, δ) ∈ N + [1/4, 3/4].∆ 1 (α) and ∆ 2 (α) contain discount factors arbitrarily close to 1. The real a * (α, δ) can be expressed in closed form as a * (α, δ) = ln(1-δ)-ln(1-α) 2 ln α

The generalization of this result to games involving more than two players is a well-known open question in dynamic games.

Fudenberg and Yamamoto [7], as well as Hörner et al.[START_REF] Hörner | Recursive methods in discounted stochastic games: An algorithm for δ → 1 and a folk theorem[END_REF], address the more general case of imperfect public monitoring.

When an absorbing state is reached, the play will remain forever in that state, no matter the actions played.

for instance regarding the greenhouse gases concentration in the earth atmosphere, or a global temperature rise inducing loss of permafrost and potential Arctic methane release, see e.g.[START_REF] Lenton | Tipping elements in the earth's climate system[END_REF].

The example in[START_REF] Ziliotto | Zero-sum repeated games: counterexamples to the existence of the asymptotic value and the conjecture maxmin= lim v(n)[END_REF], constructed during the PhD of the second author, refutes two conjectures of Mertens[START_REF] Mertens | Repeated games[END_REF] for zero-sum dynamic games.

As an illustration, if ε = .3 and r = .05, for any discount factor in ∆1, the set of equilibrium payoffs is the square E1 = [.25, .35] 2 , and for any discount in ∆2 the set of equilibrium payoffs is the square E2 = [.65, .75] 2 .

Introduced in Myerson[START_REF] Myerson | Multistage games with communication[END_REF] and Forges[START_REF] Forges | An approach to communication equilibria[END_REF].

Notice that this belief does not depend on the strategy of the players, as in repeated games with incomplete information, but only on past actions played and public signals observed. Since actions are finite, there is a countable set of posteriors that can be reached during the game when the initial distribution π is given.

In the equivalent stochastic game, the public signal s provides no extra information on past actions or on the state variable. Its sole impact is that it may be used by the players as a correlation device. Notice that the equivalent stochastic game is not a standard stochastic game, due to the fact that its state space is infinite.

The maximum of s α,δ over N is achieved: indeed, 0 = s α,δ (0) = lim a→+∞ s α,δ (a).

Strictly speaking, the game is constant-sum and not zero-sum, but we make the usual language abuse.

Note that a and b-strategies are not fully defined in definition 3.7, because they do not specify the actions played in the absorbing states nor in the states controlled by the other player. Because these actions have no impact on the game, we will simply ignore them.

There is no need to duplicate states 0 * and 1 * .

At period 1, W1, W2, J1, J2 should not be interpreted as Wait or Jump.

), let Γ(η) be a HSG obtained from Γ by perturbing each payoff by at most η, and denote by E δ (η), resp. E δ (η), the corresponding set of δ-discounted Nash, resp. sequential equilibrium payoffs with initial state k 1 .

Both authors gratefully acknowledge the support of the Agence Nationale de la Recherche, under grant ANR JEUDY, ANR-10-BLAN 0112, and thank J. Bolte, T. Mariotti, T. Tomala and N. Treich for fruitful discussions. Support from the ANR-3IA Artificial and Natural Intelligence Toulouse Institute is gratefully acknowledged by the first author, and support by the PEPS "Jeunes Chercheurs" is gratefully acknowledged by the second author. This project benefited from the support of the FMJH Program PGMO RSG and from the support of EDF, Thales, Orange and Criteo.

Appendix

4.1 Proof of Proposition 3.6 1) Define â = â(α, δ) as the integer part of a * = a * (α, δ), we have v α,δ ≥ s α,δ (â(α, δ)). Since â > a * -1, we have

Consequently, lim δ→1 s α,δ (â(α, δ)) = 1, which implies that lim δ→1 v α,δ = 1.

We now turn to the proof of conditions 2) and 3) of proposition 3.6, and start with a lemma. a) By definition (σ, τ ) is an ε -equilibrium, so playing J 1 at period 1 then optimally afterwards against τ should not increase player1's payoff by more than ε , i.e. y sup σ g δ

). This implies:

We have

Together with inequality (9), it implies:

Rearranging terms, the above equation is equivalent to: 2ε + 2x(r + η)(2 -y) ≥ x(δv δ -ε).

x = x ε and y = y ε depend on ε . Consider δ in ∆ 2 , we have v δ > ε + 4(r + η). So there exists ε > 0, independent from ε , such that for all δ high enough in ∆ 2 :

Passing to the limit gives:

0. And by symmetry between the players, we also have

b) We finally write that playing W 1 at period 1 then optimally afterwards against τ should not increase player 1's payoff by more than ε , i.e.

This implies:

We have

Together with inequality (10), the above implies : 2ε + 2(r + η)

For δ ∈ ∆ 1 , we have v δ < ε -4(r + η) so for all δ high enough in ∆ 1 : ε -1 + δ(1 -v δ ) ≥ 4(r + η) and we obtain: ε r+η + (1 -x)(1 + y) ≥ 2(1 -x). We finally get a contradiction with lim ε →0 x = lim ε →0 y = 0.