Samy, Simon Antoine Anthony

Rémy Guillaume Grâce

Pierre C Patrick

Tout d'abord, je tiens à remercier les membres de mon jury, qui ont accepté d'assister à ma soutenance, et particulièrement Nathalie Bertrand et Antonín Kucera qui m'ont fourni de nombreux retours et suggestions de corrections, dans un délai très court. Nicolas et Patricia me connaissent depuis maintenant quatre ans, et je leur suis également profondément reconnaissant pour leur travail d'encadrement et leurs conseils, qui m'ont énormément appris sur la façon de présenter mes travaux, que ce soit à l'oral ou à l'écrit 1 . Plus encore, ils ont su, tout du long, m'apporter la motivation et surtout l'enthousiasme qui m'a permis d'apprécier la recherche, même lorsque je pensais, fréquemment et très souvent à tort, me trouver dans une impasse. J'ai bien sûr retrouvé cette dynamique avec d'autres chercheurs, tels qu'Arnaud Sangnier et Mickael Randour, avec qui j'ai apprécié les nombreuses discussions et découvertes enthousiastes devant un tableau tout griffonné. Une autre source d'inspiration non négligeable m'est venu de l'enseignement en tant que chargé de TD, que ce soit grâce à des encadrants dynamiques, autrefois mes professeurs, comme Serge Haddad, Paul Gastin, Stefan Schwoon, Sylvain Schmitz et Alain Finkel ainsi que des élèves, trop nombreux pour être cités ici mais toujours pleins de ressources.

Le LSV est un laboratoire très agréable à vivre où l'on s'intègre facilement 2 et les discussions et débats 3 se lancent facilement autour du café lors d'une pause. Je garde des souvenirs mémorables des thésards et stagiaires du fond du couloir , passés ou actuels, avec lesquels j'ai passés de très bons moments :

Chapter 1 Introduction

Stochastic behaviours are a particular form of uncertainty that also allow desynchronization and symmetry breaking mechanisms. Imagine for example two persons playing the wellknown rock-paper-scissors game as in Figure 1.1a, on a daily basis. The game consists in a series of rounds, where each of them consists in the two persons picking a symbol among , and . The game continues until a winner is determined, that is when a tie is not triggered. As these players often play this game against each other, they may start to learn from each other and infer over time how their opponent reasons. Each day that passes, the players can decide to change their mind according to what was played the day before, which will result in an endless oscillation of their played symbols, or keep playing the same way. This may require them to remember the beginning of the previous plays, or at least the beginning of the sequence of played symbols. One day, one of the players is bored of this game and instead of trying to "outsmart" their opponent as usual, decides at each round, to roll a dice and play if the value is 1 or 2, if the value is 3 or 4 and otherwise. As soon he decides to play according to this scheme, he secures his win with probability 1 2 . No matter what the other player does, there is no point in trying to guess an opponent behaviour nor remembering what he has played the day before, thus the game becomes easier to play for both of them. Playing at random may also be a time-saver, as every day, players are now guaranteed to spend only 3 2 rounds on average.

In this thesis, we study formal verification and synthesis of systems with stochastic behaviours. Randomization in the computer science field is indeed a key tool to desynchronise processes, to avoid deadlock situations or collisions in communication protocols. The global ; ; ; ; 2 ; 2 ; 2 (a) Rock-paper-scissors game. A tie resets the game.

;

(b) Simple schema of a wait/transmit ( / ) mechanism for two entities that share the same medium. Any collision resets the game.

. GAME THEORY AND COMPUTER SCIENCE

picture in this context is very close to our previous rock-paper-scissors game, as shown in Figure 1.1b. Many protocols rely on randomization, like CSMA/CD [START_REF]Part 3: Carrier sense multiple access with collision detect on (csma/cd) access method and physical layer specifications[END_REF], which requires its participants to randomize their sending in case of a previous collision. As a consequence, considering stochasticity in this protocol is of fundamental feature for its analysis [DKN + 12]. Another example of desynchronization through randomization occurs for the distributed assignment of link-local addresses as specified by the Zeroconf protocol [START_REF] Bohnenkamp | Cost-optimization of the IPv4 zeroconf protocol[END_REF][START_REF] Aboba | Dynamic configuration of IPv4 link-local addresses[END_REF].

With the rise of open-source softwares and the progresses of code disassembly, but also because communication protocols are publicly described, it is reasonable to assume that the source code of any program is in fact a public information. This has security implications since several algorithms and data structures have a reasonable complexity on average, but a huge complexity in the worst case, that can be triggered by an attacker. For example, with limited computational power and bandwidth, an attacker could trigger many collisions in the deterministic hash-table of a server-side program and bring down a large computational structure [START_REF] Crosby | Denial of service via algorithmic complexity attacks[END_REF]. In this context, randomization is not only a way to ensure fairness during a transmission, but also a natural counter-measure against an adversary.

Game theory and computer science

Game theory and strategic reasoning is a prolific domain, introduced by Von Neumann [START_REF] John Von Neumann | Zur theorie der gesellschaftsspiele[END_REF] with his famous minimax theorem, and later more complete formalisms [START_REF] Neumann | Theory of Games and Economic Behavior[END_REF]. This framework has been successfully applied to economics and sociology by Nash, Selten and others [START_REF] Nash | Equilibrium points in n-person games[END_REF][START_REF] Selten | Spieltheoretische Behandlung eines Oligopolmodells mit Nachfrageträgheit[END_REF], as it provides a relevant model for the behaviours and interactions of people and entities, that are seen as players or agents, each one of them making decisions that affect all players. Decisions or actions are chosen by each player, according to their strategy, which can possibly take into account some observation of the game and its environment. Each player is given a goal or objective, usually a utility function or more generally a preference relation. The utility is rewarded to each player according to the actions played by all the players.

These games are usually given as payoff matrices, that is to say the game is described as a punctual event in time. The previous example and related communication protocols include a sequential aspect, that is often represented in an automata theoretical framework in the computer science point of view. For example, the Church's problem asks whether a binary relation on infinite sequences of words, represented as a logical formula, can be implemented in terms of a circuit that takes the first infinite sequence as an input, and produces one of the corresponding possible infinite sequence as an output. The problem can be represented by a game played on a graph between two players, the first one proposing an output of the game by choosing an edge labelled with the corresponding letter of the output sequence, while the other tries to exhibit an input sequence, letter by letter that doesn't correspond to the output. Implementing the circuit corresponds to a winning strategy for the first player [START_REF] Escrig | Solving sequential conditions by finite-state strategies[END_REF], hence solving the game.

This methodology is typically well-suited for reactive systems, where we are interested in the synthesis of a controller that should satisfy a logical property against an unpredictable environment [START_REF] Abadi | Realizable and unrealizable specifications of reactive systems[END_REF]. Here, the first player is existential, and we shall name her "Eve", as we are interested in a single possible winning strategy, whereas the second player is universal, namely "Adam", and we consider all his possible behaviours. 1.2 Non-determinism, stochasticity and quantitative problems

When we say that a player is winning against all possible strategies, we adopt a worst-case approach since all possible actions are taken into account. One can legitimately argue that this form of uncertainty is very strong as we may consider irrelevant scenarios, that may not happen in practice. For example the simple transmission model depicted before, can be assumed to enjoy a fairness property, that is to say, the existence of infinitely many time slots available for emission. An intermediate notion of uncertainty is stochasticity, namely the environment, which is usually introduced as a particular player that takes no decision but has a stochastic behaviour. As opposed to a universal player, the environment has no objective and can help or work against the existential player. Introduction of probabilities into a model has two major consequences: first of all, some sets of possible executions from the worst-case approach are discarded. For example, a tie loop is improbable in the rock-paper-scissors game as soon as one player plays at random. This can be related to certain forms of fairness properties, that are otherwise harder to model. On the other hand, stochasticity brings new quantitative questions. For example, we can ask whether a player can win with probability larger than a given threshold.

Strategic optimization

Several concepts of optimality for strategies have been proposed, depending on the desired properties. An optimal strategy is the first natural solution concept where the strategy of a given player has to maximize the utility of that player, no matter the actions of the other players. This concept leads to the notion of value [START_REF] Shapley | A value for n-person games[END_REF]. In this context, games are assumed to be played by two players with antagonistic objectives, namely zero-sum. Although the resulting values for both players do not necessarily sum to zero, determinacy results state that this intuition is correct for a wide range of games when players play in turn without randomization but with regular objectives [START_REF] Martin | Borel determinacy[END_REF].

When allowing concurrent moves and randomization, optimal strategies may not exist even though the game is determined, as the value may be asymptotically reached [START_REF] Martin | The determinacy of Blackwell games[END_REF]. Consider for example the concurrent 2-player zero-sum game H (hide-or-run), depicted in Figure 1.2. Player 1 can either hide ( ) or run home (r) while player 2 tries to shoot him, with a snowball. If player 2 shoots while player 1 is hiding, she loses her snowball and loses the game. If player 2 shoots while the other player is running, she wins. This game has been shown by [START_REF] Kumar | Existence of value and randomized strategies in zero-sum discrete-time stochastic dynamic games[END_REF] to have no optimal strategy although player 1 can ensure to win with probability arbitrarily close to 1.

When the game involves more than two players, or when objectives are compatible from 1.4. OUTLINES one player to another, we may argue that the notions of optimal strategy and value are not suited anymore: players may not necessarily play against each other anymore. The equilibrium notion introduced by Nash [Nas50] plays a central role for the study of non-zero-sum games. In such an equilibrium, we are interested in particular strategies, one for each player, such that for each player, her strategy is optimal when played together with the particular strategies of the other players, that are already fixed. On a graph structure, an equilibrium forms a particular cooperation mechanism between the players, in order to visit profitable states. What happens when a player does not respect the agreement depends on the exact notion considered, as players in a Nash equilibrium prefer to retaliate to maintain their equilibrium, instead of continuing to optimize their own utility. This last observation justified the introduction of particular notions of Nash equilibria called subgame perfect equilibria [START_REF] Selten | Spieltheoretische Behandlung eines Oligopolmodells mit Nachfrageträgheit[END_REF], where equilibrium must be ensured from any state.

The assumed behaviour of the other players, that may want to optimize their own utility is called rationality. This hypothesis is particularly sound in the computer science area, as players are programs, autonomous systems, or individual devices, that interact together, through a protocol or some predefined patterns. Hence, a strategy for a player can be seen as a program or firmware, that we may, depending on the studied problem, try to synthesize or simply assume to exist [START_REF] Kupferman | Synthesis with Rational Environments[END_REF].

Outlines

We focus in this thesis on the power provided by stochasticity in games represented by finite control-flow graphs, and mainly consider reachability and safety objectives of a particular state. This choice can be considered as a first step to more general verification studies.

We are particularly interested in the study of the decidability and computability statuses of solution concepts from the perspective of the following parameters: (a) the number of agents involved, (b) the way they interact with each other, (c) the allowed memory in strategies, (d) the allowed stochasticity in strategies, (e) the unpredictability of environment (stochastic, and/or non-deterministic), (f) the relaxation of the solution concept.

We consider first a classical framework of concurrent games played on graphs by a fixed number of agents, with stochastic behaviours coming from the environment and the strategies. More precisely, we study how concurrent actions and stochastic strategies together harden the decidability problem of the existence of a Nash equilibrium. In this general situation, which we will see is undecidable, we investigate relaxed notions of equilibria to regain decidability.

Despite their very general specification, concurrent games have a rigid structure that we argue is not suited to the representation of concrete problems. One main challenge that we address is the study of a model that is flexible enough to capture the stochastic interactions between an arbitrary number of agents, while remaining simple enough to keep decidability and computation possible and tractable.

Therefore, we consider a model where the number of agents is a parameter, and consider behaviours for large values. Because of this parametrized setting, the model of interaction has to be discussed and made precise. We proceed to the study of interactions made through a shared variable with finite domain, that each agent can access asynchronously. The system is somehow turn-based, however, the order is chosen according to a (non-)deterministic or stochastic scheduler, seen as an environment. On the one hand, this model is more general than games as it is parametrized, that is to say it involves an arbitrary number of players.

On the other hand, it is more restricted since the communication primitives are limited to a particular case of shared-register access between the players. Again, we study which role a stochastic environment plays in this context. Then, we address the existence and even the synthesis of strategies for players by characterizing the memory and randomization that they may require. To simplify the study in this second part, our work focuses only on qualitative reachability properties, that is to say reachability with probability 1, or strictly less than 1.

Related work

The seminal result of Nash [START_REF] Nash | Equilibrium points in n-person games[END_REF], shows that any one-shot game has an equilibrium in mixed strategies. The crucial point of the proof is the introduction of strategy randomization to ensure continuity then existence of a fixed point of a well-chosen function, for example by applying Brouwer's fixed-point theorem. Such arguments can be adapted to some games played on graphs, for example in the finite horizon case, when the length of the play is bounded. Discounted utility functions is a special case of infinite horizon, where reward on each state exponentially decreases over time. Players are therefore encouraged to apply their best strategies in the beginning of the play, which ensures continuity then existence of a Nash equilibrium [START_REF] Fink | Equilibrium in a stochastic n-person game[END_REF]. The undiscounted infinite horizon case is usually harder, even when the game involves only one player interacting with her stochastic environment. Such a game can be seen as a Markov Decision Process, for which a family of asymptotically optimal strategies is known to exist [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF]. From a mathematical point of view, games with at least two players are also studied under the name Competitive Markov Decision Process with similar discounted and undiscounted objectives [START_REF] Filar | Competitive Markov Decision Processes[END_REF].

Here, our objectives consist in reachability and safety objectives on states, which can be seen as a particular notion of undiscounted objective. Existence is ensured for games with safety objectives for all players, that is to say staying inside a subset of safe states [START_REF] Secchi | Stay-in-a-set games[END_REF]. In the reachability case, the existence of a Nash equilibrium remains open, nonetheless, a relaxed notion of equilibria where each player can still deviate to win up to a fixed imprecision ε > 0 is known to exist for terminal-reachability objectives [START_REF] Chatterjee | On Nash equilibria in stochastic games[END_REF]. Extensions to ω-regular objectives have been considered in the two-player case [START_REF] Chatterjee | Two-player nonzero-sum ω-regular games[END_REF], while the general case remains open. A more complete survey of known results of Nash equilibria and stochastic games in general can be found in [START_REF] Chatterjee | A survey of stochastic ω-regular games[END_REF].

When restricting to deterministic games without randomization, a study of ω-regular objective is possible [START_REF] Brenguier | Nash Equilibria in Concurrent Games -Application to Timed Games[END_REF], by encoding any Nash equilibrium in a two-player game. Stochastic games have also been studied in the turn-based case by [START_REF] Ummels | Stochastic Multiplayer Games: Theory and Algorithms[END_REF], where the decision problem of the existence of a general Nash equilibrium is shown undecidable. More precisely, the author shows that it is undecidable to determine whether a game involving 11 players with terminal rewards and where a given player wins almost-surely has a Nash equilibrium. When restricting to memoryless strategies, we can decide in PSPACE whether there exists a Nash equilibrium whose average payoff profile lies between two given threshold profiles, which provides a method for approximating the resulting payoff. This has to be compared to the complexity of computing a Nash equilibrium in matrix games which is PPAD-complete, even with only two players [START_REF] Chen | Settling the complexity of computing two-player nash equilibria[END_REF]. Note that the two studies focus on decision and computation problems since the existence of a Nash equilibrium is not ensured in their restriction. Indeed, randomization and concurrent actions are the main ingredient of the existence proof.

Considering an arbitrary number of agents is a form of parametrized system, for which 1.6. STRUCTURE OF THE THESIS verification has been introduced by German and Sistla [START_REF] German | Reasoning about systems with many processes[END_REF]. In general, systems that interact together, concurrently, can be modeled as a vector-addition system, or equivalently a Petri net. Reaching a particular control state by any agent can be seen as a coverability property, which is known to be decidable and EXPSPACE-complete [START_REF] Rackoff | The covering and boundedness problems for vector addition systems[END_REF][START_REF] Lipton | The reachability problem requires exponential space[END_REF]. Therefore we realize that parametrized verification may be an already difficult problem and has mainly been studied in the non-stochastic case. However, some studies of population protocols, like [START_REF] Esparza | Verification of Population Protocols[END_REF], include fairness properties and ask for reachability of a strongly connected component, which relates to almost-sure reachability, that is to say a qualitative property. Petri nets and population protocols are nonetheless very powerful models, that we may want to restrict. Several restricted communication primitives are presented in [START_REF] Esparza | Keeping a crowd safe: On the complexity of parameterized verification (invited talk)[END_REF], and their impact on the complexity of verification problem is discussed. The non-atomic shared register protocols, chosen as the communication scheme of the second part of the thesis, have been for example studied in [START_REF] Esparza | Parameterized verification of asynchronous shared-memory systems[END_REF][START_REF] Durand-Gasselin | Model checking parameterized asynchronous shared-memory systems[END_REF]. The authors consider only the non-stochastic case, but do not restrict their analysis to finite control-flow graph and also consider agents described by pushdown machines. Similar models as reconfigurable broadcast networks [START_REF] Bertrand | Playing with probabilities in reconfigurable broadcast networks[END_REF], can also be studied with their stochastic behaviours, while local distributed strategies are introduced in [START_REF] Bertrand | Distributed Local Strategies in Broadcast Networks[END_REF].

Structure of the Thesis

After providing in Chapter 2 the basic concepts, tools and notations used along the thesis, the rest of the manuscript is divided into two independent parts:

• The first one is devoted to the study of concurrent stochastic games with terminalreachability objectives and a fixed number of agents. Chapter 3 defines the whole game framework, from stochastic arena to state-of-the-art results on Nash equilibria. Chapter 4 develops a proof of the undecidability of the existence of Nash equilibria and discusses several implications. Finally, Chapter 5 introduces a relaxed notion of equilibria that is proven to always exist and enjoy computability properties.

• In the second part, we shift to a more realistic model where the number of involved agents is not fixed a priori and information is only partial. Here, we mainly focus on qualitative reachability properties of our systems, assuming that the behaviour of agents is fixed. Chapter 6 introduces the parametric verification concepts and relates the studied model to the different communication classes already considered in the literature. Chapter 7 introduces formally the model and describes the considered semantics. Chapter 8 recalls some preliminary results on the deterministic case, or equivalently, on the stochastic case with positive probability. Chapter 9 studies the almost-sure reachability question. The techniques are then generalized to some extensions in Chapter 10. The last Chapter 11 tries to bridge the gap with the first part, by introducing a local strategy concept for agents, and presents decidability results in some restricted cases.

Scientific publications

Several results presented here complete previous conference publications that the author of this thesis has co-authored: Chapter 4 describes the undecidability result of [START_REF] Bouyer | Mixed Nash equilibria in concurrent games[END_REF], and exposes further discussions on the invisibility of played actions as well as several implications to the study of Nash equilibria in games with qualitative reachability and safety objectives. The existence and computation of equilibria under imprecise deviations developed in Chapter 5 have been published in [START_REF] Bouyer | Stochastic equilibria under imprecise deviations in terminal-reward concurrent games[END_REF]. Finally, the second part of the thesis extends [BMR + 16], whose main theorem is presented in Chapter 9, to the case of a distinguished agent, called a leader, and to more general reachability objectives. Extensions to multiple registers and strategy concepts are also studied as further improvements.

Chapter 2

Preliminaries

We briefly describe in this chapter some notations and concepts that will be intensively used in the sequel.

Usual notations

We respectively denote by N, Z, Q and R the sets of natural, relative, rational and real numbers, with the usual comparison relation ≤.

For a set A ⊆ R, we will write

A ≥0 = {x ∈ A | x ≥ 0}, A >0 = {x ∈ A | x > 0}.

Operations over a monoid

Let (R, •) a monoid with identity element 1, and A, B ⊆ R, two subsets. We extend notations on • to sets by A • B = {a • b | a ∈ A, b ∈ B}. For x ∈ R and n ∈ N,

x 0 = 1 A 0 = {1} and x n+1 = x n • x A n+1 = A n • A
We define the Kleene plus of A as the smallest subset of R containing A, and stable under • operation, and denoted A + = ∪ n>0 A n . Similarly, the Kleene star of A is written

A * = A ∪ {1} = ∪ n≥0 A n .
We give several examples of monoids that will be considered through this thesis.

Monoid over words

For a finite set Σ called an alphabet, we consider R the set of finite sequences of elements of Σ, namely words. In particular, ε is the empty sequence, or empty word. For a word w ∈ R, we denote by |w| ∈ N its length, and for any position i, for 1 ≤ i ≤ |w|, we write w[i] the i-th element of w. • is defined as the concatenation of two words: for any w 1 , w 2 , w 1 • w 2 is defined as

|w 1 • w 2 | = |w 1 | + |w 2 | ∀1 ≤ i ≤ |w 1 | (w 1 • w 2 )[i] = w 1 [i] ∀|w 1 | + 1 ≤ i ≤ |w 1 | + |w 2 | (w 1 • w 2 )[i] = w 2 [i -|w 1 |]

SETS AND MULTISETS

We can check that (R, •) is indeed a monoid with identity element ε, and that Σ * = R. For a word w ∈ Σ + , we write first(w) = w [START_REF]1 Simple games that require randomization[END_REF] and last(w) = w [|w|]. For w, w ∈ Σ * , we write w w whenever there exists w ∈ Σ * such that w = w • w .

With the exception of the last operator, we extend previous notions on words to ω-words, defined on set Σ ω .

Relations over a set

Let us now consider a set V and the class R of binary relations over V that is to say elements of the form E ⊆ V × V . For x, y ∈ V , we write xEy whenever (x, y) ∈ E. For E 1 and E 2 two binary relations, we denote by

E 1 • E 2 = {(x, y) ∈ V 2 | ∃z ∈ V xE 1 z ∧ zE 2 y}.
We can then define for any class C of binary relations C * and C + . By analogy with the previous notations, we write E -1 = {(y, x) | (x, y) ∈ E}. Notice that in general, 1 ⊂ E -1 • E but 1 = E -1 • E. E * and E + are called respectively the transitive and strict transitive closure of E.

Paths

Let E be a binary relation over V . We denote by paths (E) the set of paths on E, that is to say:

paths

(E) = π ∈ V + ∀i ∈ N 1 ≤ i < |π| ⇒ (π[i], π[i + 1]) ∈ E
For x, y ∈ V and π ∈ V , we write π : x E * y whenever π ∈ paths (E), π[1] = x and π[|π|] = y.

Partial functions

Partial functions from V to V are particular binary relations E over V , that are stable by • operations, hence that form a sub-monoid.

For a property ϕ, we will write 1 ϕ for the value 1 if ϕ holds and 0 otherwise. When X is a set, we also write 1 X for the function x → 1 x∈X . For any function f defined on a monoid R, and any x ∈ R, we write f (x • -) for the function: y ∈ R → f (x • y).

Sets and multisets

If A is an arbitrary set, [A] denotes its cardinal which is a natural number or ∞.

A multiset over A is a function µ : A → N. Its cardinality is denoted by |µ| = a∈A µ(a) ∈ N {∞} while its support is µ = {a ∈ A | µ(a) > 0}. The set of multiset of cardinality n is written N A n , while the whole set of multisets is N A = n N A n . ⊕ and respectively denote point-wise addition and subtraction of multisets.

Well-quasi-orders

Let A be a set equipped with a quasi-order , that is to say a reflexive and transitive binary relation over A. We say that (A, ) is a well-quasi-order (wqo for short), whenever any infinite sequence w ∈ A ω contains two elements w[i] and w[j] with 1 ≤ i < j such that w[i] w [j].

A subset U of A is upward closed whenever for all x y and x ∈ U , we also have y ∈ U . When (A, ) is a wqo, we see that any upward closed set U can be represented by a unique finite subset B ⊆ U , in the following way: U = {y | ∃x ∈ B.x y}. Remark that cardinality

[U ] is potentially infinite, although U can be finitely represented.

Probability theory

In order to give semantics to our different models, we will equip them with a probability distribution over their possible runs. A run will usually be represented as a maximal word, possibly of infinite length, whose all finite prefixes are paths in the considered model.

Theorem 2.1 (Carathéodory's criterion [START_REF] Halsey | Real Analysis[END_REF]). Let P a function of the form P : 2 Σ + → R ≥0 such that for all w ∈ Σ + , a∈Σ

P(w • a • (Σ * Σ ω )) ∈ {0, P(w • (Σ * Σ ω ))}
Then, P can be uniquely extended as a measure on the σ-algebra B(Σ) generated by the class of languages with a common prefix, namely {w • (Σ * Σ ω ) | w ∈ Σ * }. For any finite word w ∈ Σ + , we usually identify notations P(w) with P(w • (Σ * Σ ω ))

If w w and P(w ) > 0, then by induction, we show that P(w) > 0. Finite words w such that P(w) > 0 and ∀a P(w • a) = 0, are called terminal words. For any infinite sequence of words of the form (w i ) i>0 with ∀i w i w i+1 with ∀i P(w i ) > 0, the limit of such sequence is a finite or infinite word w ∈ Σ ω Σ + , which has measure lim i>0 P(w i ).

Our main goal is to study probabilistic properties. As the notation P suggests it, for a given prefix w 0 ∈ Σ + , if P(w 0 ) > 0, then ∀X ∈ B(Σ) → P(w 0 • X) is a probability measure. In the following, if A is an at most denumerable set, Dist(A) will denote the set of probability distributions over A. If µ is such a distribution, µ denotes the support of µ, that is the subset {a ∈ A | µ(a) > 0}. Pointwise addition for distributions will be written +, and multiplication by a scalar is written •, so that for any two distributions µ and µ on the same set A, and for any p ∈ [0, 1], p • µ + (1p) • µ is still a distribution on A.

Part I Mixed Strategies in Concurrent games

13 Chapter 3

Concurrent games

In this chapter, we introduce the game framework used in the first part. We describe how players can interact together, by the means of an arena, which roughly corresponds to a finite directed and labelled graph. Then, several classes of strategies are defined, that will later induce an outcome, seen as a probability measure over the possible runs in the arena. Game theory is focused on how players can enforce an outcome that is profitable for them, so we have to define how outcomes can be compared, that is to say to define payoffs for the players. Finally, we consider game theoretical concepts such as optimal strategies and Nash equilibria in our framework and discuss known results.

Definition 3.1. A concurrent arena A is a tuple A = (States, Agt, Act, Tab, (Allow i ) i∈Agt ) where

• States is a finite set of states;

• Agt is a finite set of players;

• Act is a finite set of actions;

• For all i ∈ Agt, Allow i : States -→ 2 Act is a function describing authorized actions in a given state for Player i;

• Tab : States × Act Agt → Dist(States) is the transition function.

A state s ∈ States is said terminal (or final ) if it allows no action for any player:

i∈Agt Allow i (s) = ∅. We write F A (or simply F when the underlying arena is clear from the context) for the set of terminal states of A.

We argue below that this model is very general, and captures several frameworks from the litterature.

We say that the arena is deterministic whenever the transition function is deterministic i.e., only makes use of Dirac distributions, stochastic otherwise. When refering to an arena with n = [Agt] players, we can refer to the probabilistic behaviour as a last player, usually called the environment, who resolves the probabilistic transitions. As a consequence, we will say that an arena is a n + 1/2-player arena if it is stochastic, or n-player otherwise.

We say that the arena is turn-based whenever for any s ∈ States, there is at most one player i ∈ Agt who has more than one allowed action: ∀j = i [Allow j (s)] ≤ 1.

STRATEGIES

q 0,0 q 0,1

q 1,0 bbb bab,bba baa p 1 -p q 1,1 †bb †ab, †ba †aa p 1 -p a * * Figure 3.
1 -Graphical representation of a 3+1/2-player arena. Player 1 can enforce transition from q 0,0 to q 0,1 by playing a. Player 1 has no move left from this state since Allow 1 (q 0,1 ) = { †}, on the other hand, players 2 and 3 can trigger a probabilistic transition if they both play action a.

Graphical representation. In the sequel, we usually identify each agent i ∈ Agt, with a positive integer from 1 to [Agt], that is to say Agt = 1, [Agt] .

Thus, an action profile of all players is a word A ∈ Act Agt , of length [Agt], and for any player i ∈ Agt, A[i] is the action played by player i.

An example of stochastic arena1 is represented in Figure 3.1. We provide several graphical conventions:

• Non-terminal states are drawn with non-empty circles filled with their state label.

• Rectangle states are terminal.

• If Tab(s 1 , A) is a distribution containing a unique state s 2 , we write an arrow from s 1 to s 2 labelled with the word A.

• If several action profiles link a state s 1 to another state s 2 , we can draw a single arrow labelled with the language describing the corresponding set of action profiles.

• The symbol * is a shortcut for the language of the whole alphabet Act.

• The action † is used whenever it is the only action allowed for a given player.

• If Tab(s 1 , A) is a distribution, we write a first A-labelled arrow to a stochastic node then several arrows to each state s 2 in the distribution support, labelled with the corresponding probability in the distribution.

Strategies

During a play, players in Agt choose their next moves in Act concurrently and independently of each other. They can even sample the played actions from a chosen distribution.

In order to make their choice, players must be aware of the current location of the game since the set of allowed actions may differ for each state. In the general setting, players are even given the complete sequence of visited states from the beginning of the play. Such sequence h ∈ States + will be called an history, and the current state will be denoted by last(h) = h [|h|]. The choices operated by each agent are represented by strategies, that we define below. Definition 3.2. A mixed strategy for player i ∈ Agt is a mapping σ i : States + → Dist(Act), with the requirement that for all h ∈ States + , σ i (h) ⊆ Allow i (last(h)). We denote by S A i the class of (general, mixed) strategies for player i in arena A. We will usually simply write S i when A is clear in the context.

We consider several subclasses of strategies, defined below:

Definition 3.3. Let σ i ∈ S i and k ∈ N, we define M(k) i , M i , F i as follows:
• σ i ∈ M(k) i if the strategy requires a memory of size k, that is to say, if there exists a deterministic finite-state automaton with state-space M of cardinality k + 1;

-transition function 2 δ : M × States → M, with initial state m 0 ∈ M; -for all hq, h q ∈ States + , if δ * (m 0 , h) = δ * (m 0 , h ), then σ i (hq) = σ i (h q).
Intuitively, such a strategy stores m = δ * (m 0 , h) the current state in the automaton, seen as a memory information, that can be updated iteratively. The decision of σ i from history h and memory m is expressed as a function of (last(h), m).

• σ i ∈ M i = M(0) i , if σ i is a stationary (or memoryless) strategy, that is to say ∀h ∈ States + σ i (h) = σ i (last(h)). • σ i ∈ F i if σ i requires finite memory, that is to say F i = k ≥0 M(k ) i .
• We also define the class of pure strategies S i in which all probability distributions are Dirac functions, that is, strategies are deterministic:

σ i ∈ S i ⇔ ∀h ∈ States + σ i (h) = 1
Each previously listed class can also be restricted to its pure counterpart:

M (k) i = M(k) i ∩ S i , M i = M i ∩ S i , F i = F i ∩ S i .
A strategy profile is a tuple σ = (σ i ) i∈Agt , in which σ i is a strategy for player i. Following the definitions introduced above, we consider the full class S A , or simply S, of mixed strategy profiles. More generally, given a strategy class C i for each player i ∈ Agt, we will write C the corresponding class of strategy profiles.

For a ∈ Act, i ∈ Agt and h ∈ States + , we write σ i (a | h) for the quantity σ i (h)(a). We also identify a with the Dirac distribution 1 {a} , allowing us to write σ i (h) = a whenever σ i (a | h) = 1.

VISIBILITY OF ACTIONS

A strategy profile σ naturally induces a distribution over Act Agt for any history h by

∀A = (a i ) i∈Agt ∈ Act Agt , σ(A | h) = i∈Agt σ i (a i | h)
For an action profile A ∈ Act Agt , σ(A | h) and σ(h) = A are defined similarly to the actions and strategies for a specific player. Remark 3.4. A strategy profile σ for a game G can be seen as a strategy σ C for a unique player C in a game G Agt where all agents are seen as only one agent C, a coalition, with set of allowed actions from state s, Allow C (s) = i∈Agt Allow i (s). Notice however the converse is not true: a strategy for player c may not be a strategy profile in the original game as the unique player may introduce dependencies between concurrently played actions. For example, from a state s, the coalition strategy σ C (s) = 1 2 1 {ab,ba} cannot be decomposed as a strategy profile for two players with two allowed actions a and b.

Visibility of actions

Note that strategies, as defined above, can only observe the sequence of visited states along the history, but they may not depend on the exact distributions chosen by the players along the history, nor on the actual sequence of actions played by the players. Usually, actions are made visible in the game models considered in the literature-see for instance [START_REF] Ummels | The complexity of Nash equilibria in infinite multiplayer games[END_REF] and [BBMU11, section 6] or [START_REF] Chatterjee | Partial-observation stochastic games: How to win when belief fails[END_REF] for discussions-and the results presented here may differ.

One motivation for action-visible models comes from the fact that invisibility of actions is a particular case of partial information, which is a concept usually inducing undecidable models [START_REF] Baier | On decision problems for probabilistic Büchi automata[END_REF]. However, we allow this little source of unobservability in our model, as it makes it more general, and recent results like [START_REF] Brenguier | Nash Equilibria in Concurrent Games -Application to Timed Games[END_REF] tend to prove that such a lack of observation is not crucial.

Moreover, we can still consider subclasses of arenas for which the observation of history determines which action has been played. Such arenas are defined below: Definition 3.5. An arena A is action-visible if, for every non-final states s 1 , s 2 ∈ States\F, there exists at most one action profile A ∈ Act Agt such that Tab(s 1 , A)(s 2 ) > 0.

Remark 3.6. We can see that any arena A can be transformed into A v which is actionvisible. Indeed, we can encode the played actions inside the current state, by setting

States v = States × Act Agt and Tab v ((s 1 , A 1 ), A)((s 2 , A 2 )) = Tab(s 1 , A)(s 2 ) • 1 A=A 2 . A strategy σ ∈ S A
can be seen as a strategy σ v ∈ S A v that ignores the second coordinate of each state that appears in any history.

When considering solution concepts like Nash Equilibria, or algorithms based on concurrent games, our framework is more general as it does not require action-visibility. Hence, existential and algorithmic results from Chapter 5 are stronger in this framework.

However, when considering hardness results, as the undecidability proof of Chapter 4, exploiting the lack of visibility can be seen as a weakness, as imperfect information in stochastic games usually leads to undecidability. Although similar undecidability results have been developed for example by [START_REF] Ummels | Decision problems for Nash equilibria in stochastic games[END_REF] without this hypothesis, we argue that this extra hypothesis allows us to narrow the gap between decidability and undecidability problems, since the undecidability result only leaves open the two-player case, which improves our understanding of concurrent games.

Semantics

For a given strategy profile σ in arena A, we can define a distribution over histories, that we will write P σ . This section is devoted to its definition and its related properties. Definition 3.7. Let σ ∈ S a strategy profile for A. We define P σ (h) by induction on h ∈ States + .

• For any s ∈ States, P σ (s) = 1

• For any h ∈ States + and s ∈ States,

P σ (h • s) = P σ (h) • A∈Act Agt σ(A | h) • Tab(last(h), A)(s) if last(h) / ∈ F, 0 otherwise. If last(h) / ∈ F, we check that s∈States P σ (h•s) = s P σ (h)• A∈Act Agt σ(A | h)•Tab(last(h), A))(s) = P σ (h)• A σ(A | h) = P σ (h)
Therefore, we can apply Carathéodory's criterion stated in Theorem 2.1 to extend P σ as a measure to any finite or infinite sequence on States. Notice that maximal sequences belong to the language (States\F) * • F States ω . Intuitively, for any history or infinite run h ∈ States + States ω , P σ (h) will denote the probability of generating h with strategy (profile) σ, when starting from state h [START_REF]1 Simple games that require randomization[END_REF]. Definition 3.8 (Conditional probability). Let h ∈ States + be a history, we define the conditional probability measure P σ (-| h) for any finite history or infinite run h ∈ States + States ω , by

• P σ (h | h) = P σ (h ) P σ (h) if P σ (h) > 0 and h h ; • P σ (h | h) = 0 otherwise.
We can check that such a conditional measure is indeed always a probability measure over (States\F) * • F States ω , as the choice of h fixes the initial state.

Remember that when considering a finite history h ∈ States + , we are in fact considering the probability of the whole cylinder h • ((States\F) * • F States ω ). This property also applies to conditional properties.

Another interpretation of P σ (h | h) is to say that arena and strategies are "initialized" with history h, and we are considering the probability that the resulting stochastic process generates a run that starts with h .

Outcome of a game

An arena as previously described, only gives information about how the agents can interact, to make the game progress. However, at this point, no information was given about the incentive of each player. Several choices can be made at this point to define such a concept:

• When the arena involves at most two players, we can define a set Ω of states that have to be reached. First player (if he exists) will try to enforce reaching such a state, whereas second player (if he exists) will try to avoid it.

• In the same setting, we can define a set W ⊆ States ω States * • F of winning runs for first player. In the previous setting, this approach boils down to

W = States * • Ω • (States ω States * • F).
• More generally, we can provide a total ordering relation over States ω States * •F, which will be called a preference relation for the first player. With the previous approach, this is equivalent to define ∀r, r r r ⇔ r ∈ W .

• When the game involves more than two players, the roles of the players is not necessarily antagonistic, so we can give to each player a set of target states, a set of runs, or a preference relation, to define their objectives.

In a deterministic setting, we have listed above any possible objective: when considering a deterministic arena A, a pure strategy profile σ and an initial state s 0 , we can define precisely the outcome of the game as the unique finite or infinite maximal path starting from s 0 , denoted Out s 0 ,A (σ). The last preference relation is in some sense the most general one, as it allows to compare any run, seen as an outcome, for each player. However, this approach is not suited anymore when introducing probabilities in our model, as the outcome cannot be defined as runs in the arena. More precisely, the outcome of a game starting from s 0 in arena A with arbitrary distributions consists in the resulting probability measure, that is to say Out s 0 ,A (σ) = P σ (-| s 0 ). As a consequence, comparing outcomes in a stochastic setting boils down to defining preference relations over probability measures of paths.

Instead of providing such a general and complex definition of objectives, we limit ourselves to the first and second settings where objectives are given as measurable functions from the set of possible runs to the real numbers. Definition 3.9. G = (A, s, (Φ i ) i∈Agt ) is given by an arena A, an initial state s, and for every player i ∈ Agt, a real-valued function Φ i : States * States ω → R, which is measurable over the set of runs of A.

As explained before, we will not define a general preference relation over outcomes, which are general distributions over runs, but give a more concrete characterization based on reward functions. For any history h, and any function measurable over the set of runs of A, we denote by E σ (φ | h) the expected value of the reward function φ induced by the probability measure

P σ (-| h).
As for strategy profiles, we write by extension E σ (Φ | h) for the tuple (E σ (Φ i | h)) i∈Agt when (Φ i ) i∈Agt is a family of measurable functions, one for each player.

Finally we say that an history h is activated, or enabled by a strategy profile whenever it is visited with positive probability under that profile.

Some measurable reward functions

In this thesis, we will be particularly interested in simple objective functions that consists in assigning a reward to a player only if she manages to reach a final state, or complementary, if she manages to visit as few different states as possible. Definition 3.10. An objective function φ is:

• a reachability objective if for any run r, φ(r) = max{φ(r[k]) | 1 ≤ k < |r| + 1}; • a safety objective if for any run r, φ(r) = min{φ(r[k]) | 1 ≤ k < |r| + 1};
• a terminal-reachability objective if for any run r,

φ(r) = φ(last(r)) if r ∈ States * • F, 0 otherwise.
We can easily check that terminal-reachability is a particular case of reachability objective where the payoff profile is determined by the final state reached in the game, and that each non-final state has reward 0.

The study of terminal reachability objectives is crucial for the understanding of general reachability objectives, so our study will mainly focus on this particular class of objectives. When considering games where all players have terminal-reachability objectives, the payoff functions are entirely characterized by the vectors Φ(f ) ∈ R Agt for each f ∈ F. Thus, a final state f will usually be directly denoted and graphically represented by its payoff profile Φ(f ). Definition 3.11. A reward objective φ is non-negative whenever each φ is a non-negative function. It is qualitative whenever each φ takes values in {0, 1}, in the opposite case, it is quantitative.

A reachability or safety objective φ, can be represented by a subset of states G ⊆ States representing the goal of the player, that is to say the states that she has to reach or, respectively, to not leave.

Qualitative reachability and safety objectives for Nash equilibria have been studied intensively in the general case of ω-regular objectives in [START_REF] Brenguier | Nash Equilibria in Concurrent Games -Application to Timed Games[END_REF]. In the quantitative case, terminal reachability objectives can be generalized to limit-average rewards, as in [START_REF] Ummels | The complexity of Nash equilibria in limit-average games[END_REF], where reward values are attached to each state and where the reward function consists in computing the upper or lower limit of the mean value of visited states. At the expense of a exponential blow-up3 , average-reward objectives can even encode general quantitative reachability and safety objectives.

Two-player zero-sum games

In this section, we develop known results in the 2 + 1/2-player case. More precisely, we revisit results of the zero-sum fragment. A 2-player stochastic game G with payoff function

Φ = (Φ 1 , Φ 2 ) is called zero-sum if Φ 1 + Φ 2 ≡ 0, that is to say ∀r ∈ States ω States + Φ 1 (r) + Φ 2 (r) = 0.
In such games, the roles of players 1 and 2 are totally antagonistic. When considering qualitative objectives, player 1 usually has a qualitative terminal reachability objective, while the second player goal has to avoid the same reachability set of states, that is to say ensure a safety condition.

Borel determinacy

In his seminal work, [START_REF] Martin | Borel determinacy[END_REF] studied Gale-Stewart games [START_REF] Gale | Infinite games with perfect information[END_REF] and established their determinacy in a very general setting. Such games are played by two players, who pick in turn one an element at a time from a possibly infinite set A, which generates an infinite word in A ω . Winning condition for player 1 is given as a subset Ω ⊆ A ω of infinite runs. Whenever Ω is a Borel set, the game is shown to be determined, that is to say there exists a deterministic winning strategy either for player 1, or for player 2.

This result can be rephrased in our framework to state that any deterministic, 2-player, qualitative zero-sum, turn-based game with qualitative measurable payoff function φ is determined, that is to say:

• either there exists a winning strategy for player 1, σ 1 ∈ S 1 , such that for all σ 2 ∈ S 2 ,

E (σ 1 ,σ 2 ) (φ | s 0 ) = 1,
• or there exists a winning strategy for player 1, σ 2 ∈ S 2 , such that for all σ 1 ∈ S 1 ,

E (σ 1 ,σ 2 ) (φ | s 0 ) = 0.
Notice this notion does not involve any probabilistic property, as the game and its strategies are assumed deterministic: all possible outcomes of such a game are therefore single runs. Notice however that a winning strategy is still winning against a randomized adversary, which allows us to generalize the result to randomized strategies.

However, the result is not valid anymore when considering games with stochastic transition functions, as both players may win with positive probability. A similar phenomenon occurs when allowing concurrent games. Such models form first examples of imperfect information as each player is not aware of the action played by the other player at a given step. Moreover, objectives are not necessary qualitative.

Nonetheless, some another notion of determinacy may be introduced, that captures the quantitative features and imperfect information of our model.

Value determinacy

In the quantitative setting, an interesting concept to introduce is the maximal expected value that a player can reach no matter what her opponent is doing. A dual approach consists in looking at the minimal expected payoff this opponent can achieve, when strategy of player 1 is not known in advance.

Note that "maximal" and "minimal" are here to be interpreted in an asymptotic manner, as we are not guaranteed that these optimal values can be reached. Definition 3.12. Let G be a stochastic two-player game, such that the reward function Φ 1 of player 1, is measurable. We define the upper value and the lower value from state s as the quantities:

ν 1 (s) = inf σ 2 ∈S 2 sup σ 1 ∈S 1 E (σ 1 ,σ 2 ) (Φ 1 | s) ν 1 (s) = sup σ 1 ∈S 1 inf σ 2 ∈S 2 E (σ 1 ,σ 2 ) (Φ 1 | s)
In general ν 1 (s) ≥ ν 1 (s). The converse inequality is the so-called value determinacy of a game. This unique value is then called value from state s, denoted by ν 1 (s).

Value determinacy is achieved for a large variety of zero-sum games including Blackwell games, that is to say concurrent games with a Borel measurable payoff function [START_REF] Martin | The determinacy of Blackwell games[END_REF].

In [START_REF] Martin | The determinacy of Blackwell games[END_REF], the author suggests that the corresponding strategies somehow require as much memory as needed to compute the payoff function: the more complex the reward, the more complex the strategy.

When the reward function is a terminal reachability objective, results from [START_REF] Secchi | Stationary strategies for recursive games[END_REF] and [START_REF] Kumar | Existence of value and randomized strategies in zero-sum discrete-time stochastic dynamic games[END_REF] allow to consider only memoryless strategies to asymptotically achieve the game value. This means there exist pairs of ε-optimal strategies that are memoryless, for both players and for any ε > 0: Theorem 3.13. Let φ be a terminal reachability payoff function of an action-visible zero-sum game G. Then for any state s, ν 1 (s) = ν 1 (s), denoted from now on ν 1 (s), and there exists a family of memoryless strategies (σ ε 1 ) ε>0 (resp. (σ ε 2 ) ε>0 ) such that for any state s:

∀ε > 0 inf σ 2 ∈S 2 E (σ ε 1 ,σ 2 ) (φ | s) ≥ ν 1 (s) -ε
and respectively, ∀ε > 0 sup

σ 1 ∈S 1 E (σ 1 ,σ ε 2 ) (φ | s) ≤ ν 1 (s) + ε

Action invisibility

The previous theorem is stated in the particular case of action-visible games. Although the proof can be adapted to the case of invisible actions, we give here a simpler argument in our setting, based on the fact that memoryless ε-optimal strategies are not aware of played actions.

Let us consider a stochastic game G and a terminal reachability payoff function φ. We define A the corresponding action-visible arena defined from A as in Remark 3.6. For a run r of A , we define φ (r) = φ(p(r [START_REF]1 Simple games that require randomization[END_REF]

) • p(r[2]) • • •)
where p is the projection on the first coordinate 4 .

From now on, we reason on a game G with arena A and objective φ , so S = S G denotes the class of strategies that see actions in the history. We define, the subclass of strategy profiles S u ⊆ S, that do not see actions, that is to say σ ∈ S u if, and only if,

∀h ∈ States + , ∀α, α ∈ (Act Agt ) |h| , σ((h[1], α[1]) • (h[2], α[2]) • • • (h[|h|], α[|h|])) = σ((h[1], α [1]) • (h[2], α [2]) • • • (h[|h|], α [|h|]))
We define the ν u 1 and ν u 1 by the following mapping over any s ∈ States and A ∈ Act Agt :

ν u 1 ((s, A)) = inf σ 2 ∈S u 2 sup σ 1 ∈S u 1 E (σ 1 ,σ 2 ) (φ | (s, A)) ν u 1 ((s, A)) = sup σ 1 ∈S u 1 inf σ 2 ∈S u 2 E (σ 1 ,σ 2 ) (φ | (s, A))
We can see that these quantities does not depend on A, and correspond in fact to the values in the original game G. Indeed, the reward function Φ and transition table do not observe any action. Also, we still have ν u ((s, A)) ≥ ν u ((s, A)). Moreover, S u ⊆ S so by introducing a pair of ε-optimal memoryless strategies (σ ε 1 , σ ε 2 ) for each ε > 0, from Theorem 3.13, we have:

∀ε > 0 inf σ 2 ∈S u 2 E (σ ε 1 ,σ 2 ) (φ | (s, A)) ≥ inf σ 2 ∈S 2 E (σ ε 1 ,σ 2 ) (φ | (s, A)) ≥ ν 1 ((s, A)) -ε
and respectively, ∀ε > 0 sup

σ 1 ∈S u 1 E (σ 1 ,σ ε 2 ) (φ | (s, A)) ≤ sup σ 1 ∈S 1 E (σ 1 ,σ ε 2 ) (φ | (s, A)) ≤ ν 1 ((s, A)) + ε
Moreover, these ε-optimal strategies are uniform, as they try to optimize the expected payoff from any state. Since this payoff does not depend on the played action, we argue that we can assume, without loss of generality, that σ ε 1 (s, A) and σ ε 2 (s, A) do not depend on A for any state s, that is to say

σ ε ∈ M u = S u ∩ M.
As a consequence, when ε goes to 0, we get the inequality

ν u 1 ((s, A)) ≥ ν 1 ((s, A)) ≥ ν u 1 ((s, A)) hence ν u 1 ((s, A)) = ν 1 ((s, A)) = ν u 1 ((s, A)
). We conclude that action-visibility hypothesis can be omitted from Theorem 3.13.

Values may not be achieved

We emphasize here on the fact that the value of a game may not be achieved exactly, and only ε-optimal values may exists. Consider the 2-player zero-sum game H (hide-or-run), depicted in Figure 1.2 on page 3 in the introduction.

First of all, we show that ν(s 0 ) = (1, -1), by considering a fixed strategy for player 1 who is hiding at each round with probability ε > 0. As a matter of fact, player 2 can only shoot, which generates average payoff (1 -2ε, 2ε -1), or wait, which yields payoff (1, -1) or iterate the game again. In both cases, the game almost-surely terminates, with a payoff for player 1 larger than 1 -2ε. Hence, ν 1 (s 0 ) = ν 1 (s 0 ) ≥ 1 -2ε for any ε > 0.

However, no optimal strategy σ 1 exists for player 1, as it would require to avoid state (-1, 1) almost-surely for any strategy of player 2. This would imply that player 1 plays deterministically in each history s n 0 . Moreover, such a strategy would require reaching (1, -1) almost-surely, that is to say, player 1 should not play hide action forever. However, when playing action run r, player 2 has positive chances to shoot her (for some strategy) so reaching (1, -1) almost-surely no matter the strategy of player 2 is impossible.

Nevertheless, we can identify some game classes that allow existence of optimal strategies for both players. For example, it is well-known that an optimal policy exists for Markov Decision Processes with reachability objectives [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF]. In our terminology, a policy is a strategy, and a Markov Decision Process is a 1 + 1/2-player game. In a multi-player setting, such results have been extended to the case of turn-based zero-sum games, with any limit average reward objective. For example, [START_REF] Liggett | Short notes: Stochastic games with perfect information and time average payoff[END_REF] shows that game values are achieved by pure memoryless strategies, for both players. Again, visibility of actions makes little difference to the result, and can be omitted.

As a consequence, deciding whether the value of a player in such a game is larger than a given threshold is in NP ∩ coNP (see [START_REF] Condon | The complexity of stochastic games[END_REF] for the qualitative reachability case, or [START_REF] Zwick | The complexity of mean payoff games on graphs[END_REF] for the study in the deterministic average payoff case), by guessing pure memoryless strategies achieving or refuting a value larger than the threshold. This has to be compared with parity games: where deciding which of the two players has a winning strategy is in NP ∩ coNP, as well as in UP ∩ coUP [START_REF] Jurdzinski | Deciding the winner in parity games is in UP ∩ co-UP[END_REF].

Notice also another consequence of the existence of pure memoryless strategies in the turn-based case: the values become rational, whenever the game involves rational numbers for discounted factors, probability transitions and rewards. On the contrary, concurrent games may have irrational values, even for terminal qualitative reachability objectives with rational probabilities [AM04, page 382].

Qualitative analysis

Let us consider qualitative terminal reachability objectives. Even when ν 1 (s 0 ) = 1, that is to say when player 1 can yield her maximal payoff, she can win in several manners:

• Limit-surely, if there exists a family of strategies ensuring payoff 1ε for any ε > 0,

• Almost-surely, if there exists a strategy ensuring an average payoff of 1,

• Surely, if furthermore this strategy ensures the game terminates in finite time.

The qualitative analysis focus on the existence of strategies for such winning modes, while the goal of the opposite player is only to spoil the winning condition. Notice that in the limit-sure case, a unique spoiling strategy should be resilient against a whole family of strategies that try to achieve a reachability objective with probability arbitrarily close to one, hence this context is slightly different from the existence of ε-optimal strategies for both players, as presented before.

In [START_REF] Luca De Alfaro | Concurrent reachability games[END_REF], the authors showed determinacy for each three winning conditions. When a winning strategy exists for player 1, no memory is required for winning (although randomization may be required). When a winning strategy for player 2 exists, that is to say a spoiling strategy, counting5 memory is required only for almost-sure reachability, whereas memoryless strategies suffice to spoil sure and limit-sure winning modes. This last result should be kept in mind as it relates to the value of a game, which in turn connects to Nash equilibrium concepts.

Nash Equilibria

We shift now to the study of games with more than two players. By analogy with the previous study, for any game with reachability and safety objectives, we denote by ν i (s) the value from state s for player i, in a new game played by i against the coalition composed of all agents except i. Definition 3.14. Let G be a stochastic n + 1/2-player game, with payoff function φ, s a state, and i ∈ Agt one of the players. Assume that Φ i is a terminal reachability or safety reward function, then we define ν i (s) by:

ν i (s) = inf σ∈S sup σ i ∈S i E σ[i/σ i ] (Φ i | s) = sup σ i ∈S i inf σ∈S E σ[i/σ i ] (Φ i | s)
Where σ[i/σ i ] denotes the strategy profile where strategy for player i has been changed to σ i . Furthermore, ν(s) can be seen as a vector of R Agt representing for each player, the supremum of her achievable payoffs, no matter what the other players do.

As opposed to the two-player case, it is not clear how actions from one player can be opposed to the actions of the other player, so the zero-sum condition is usually omitted, that is to say, there may exist two runs r 1 , r 2 such that i Φ i (r 1 ) = i Φ i (r 2 ).

Definition

To address this issue, John Nash introduced a new concept [START_REF] Nash | Equilibrium points in n-person games[END_REF], where emphasis is put on each individual objective, and strategy for each player should be stable with respect to the strategies of the others. Definition 3.15. Let G = (A, s, Φ) be a stochastic game. A strategy profile σ forms a Nash equilibrium for G, when no player has a profitable deviation; in other terms, for all i ∈ Agt and for all σ i ∈ S i , it holds

E σ[i/σ i ] (Φ i | s) ≤ E σ (Φ i | s)
Equivalently, the average payoff of an equilibrium σ, denoted by the vector E σ (Φ | s), satisfies, for all player i ∈ Agt:

E σ (Φ i | s) = sup σ i ∈S i E σ[i/σ i ] (Φ i | s)
This last form immediately implies that the average payoff of an equilibrium is component-wise larger than the value of the game from the same state.

Deterministic deviations

When defining the Nash equilibrium concept, we used the term deviation to denote a single strategy σ i ∈ S i for player i that may increase her payoff. Such deviation σ i is called deterministic, or pure, whenever σ i ∈ S i . An important remark when analyzing Nash equilibria is that randomization in our context is not required for deviations: if a profitable deviation has to be found, a deterministic one also exists. Randomizing its strategy is necessary only to ensure stability, that is to say prevent the other players from deviating. This observation is summarized in the following proposition: Proposition 3.16. Let G = (A, s, Φ) be a stochastic concurrent game with safety and reachability objectives and σ be a strategy profile. Then σ is a Nash equilibrium if, and only if, for all i ∈ Agt and all deterministic deviation

σ i ∈ S i , it holds E σ[i/σ i ] (Φ i | s) ≤ E σ (Φ i | s).
Proof. This result is similar to [UW11c, proposition 3.1] for turn-based games with qualitative Borel objectives (the payoff is 1 if the run belongs to the designed objective and 0 otherwise). Intuitively, for a given σ ∈ S, the composed game G σ -i where the actions of all players but i are fixed, is a one-player stochastic game (with possibly countably many states), that can be seen as a Markov Decision Process. Our safety and reachability payoffs can be interpreted as a total-reward payoff that can be taken only once (by encoding in each state the set of already visited states). Under our hypothesis, this Markov Decision Processes are known to have an ε-optimal pure memoryless6 strategy, for any ε > 0, see for example [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF]Theorems. 7.2.7 and 7.3.7]. As a consequence, if σ is not a Nash equilibrium, there exists a strategy improving the payoff of some player i by some positive quantity ε > 0, and a deterministic one that improves her payoff by ε/2 > 0.

Remark 3.17. It may not be possible, in general, to achieve the best profitable value in a deviation, and randomization or memory may be required. First remark that if the initial strategy profile σ has finite memory, the resulting Markov Decision Process has also a finite state space and an optimal deterministic and memoryless strategy exists in this case. However, the result does not hold in the case of infinite memory: consider for example the game composed of a unique non-terminal state s, with Tab(s, a * ) = s, Tab(s, ba) = (1, 0), Tab(s, bb) = (0, 0), and strategy profile ∀n ≥ 1,

σ 1 (a | s n ) = 1/2, σ 2 (b | s n ) = 1/2 n-1 .
If player 1 deviates by deciding to always play b with probability p > 0, then he yields payoff n≥0 p(1p) n • (1 -1 2 n ) = 1-p 1+p . Thus, the optimal value that player 1 can yield by deviating is 1, although this value cannot be achieved by any deviation. Moreover, if σ 1 ∈ M (n) 1 is a deterministic deviation using memory of size n, then he plays his first b action after at most n steps, which yields a payoff of at most 1 -1/2 n . In particular, there is no profitable memoryless deterministic deviation in this case.

Sub-game characterization

We explore some further characterization of a Nash equilibrium in terms of its sub-game. In order to do so, we extend the notion of Nash equilibrium to any history, in the following way: Definition 3.18. Let G = (A, s, Φ) be a stochastic game. A strategy profile σ forms a Nash equilibrium after a history h ∈ States + when the following conditions are met:

• h ∈ States + is enabled by σ from first(h);
• No player has a profitable deviation; in other terms, for all i ∈ Agt and for all σ i ∈ S i such that h is enabled from σ[i/σ i ], it holds

E σ[i/σ i ] (Φ i | h) ≤ E σ (Φ i | h)
We then write that (σ, h) is a Nash equilibrium for G.

This notion generalizes a classical Nash equilibrium σ in a game (A, s, Φ) as a Nash equilibrium (σ, s). Intuitively, we can assume that strategy σ i is not changed for any history h / ∈ hStates + , and that the game is initialized with an initial history h. In his seminal work, Nash concluded that Nash equilibria always exists for one-shot games, that is to say, games that terminates after one iteration. The result can be extended to games played on finite trees, as we can prove the result by induction. We have indeed the following results: Lemma 3.19. Let G be a stochastic concurrent game, and (σ, h) be a Nash equilibrium. If (σ, h) enables h , then, (σ, h ) is a Nash equilibrium.

The proof is immediate by expressing conditional probabilities from history h. A converse property is harder to define in the general case because we cannot assume that any history h that extends h is enabled by σ. More precisely, whenever an history h is not enabled by a strategy σ but can be enabled in a new strategy profile σ[i/σ i ] by the deviation of some player i, the other players but i have no incentive to improve their payoff, so (σ, h ) is not necessarily a Nash equilibrium.

When the game is action-visible, the other players have a perfect knowledge of the player that started a deviation and can retaliate to ensure as much stability of the strategy profile as possible. We conclude on the following property: Proposition 3.20. Let G be a stochastic action-visible concurrent game, σ be a strategy profile and h an history enabled by σ. We construct the following one-shot game G h = (A h , last(h), Φ h ) that starts from last(h), with same actions, and same transition functions but where any state s is supposed to be final, with payoff Φ h (hs) that equals:

• ν(s) whenever hs is not enabled by σ in the original game;

• E σ (Φ | hs) otherwise.

Assume moreover that

• for any disabled history hs, σ(hs-) is an optimal strategy that achieves value ν i (s) where i is the unique player who can perform transition from last(h) to s,

• for any enabled history hs, (σ, hs) is a Nash equilibrium,

• σ is a Nash equilibrium of G h .

Then, (σ, h) is a Nash equilibrium of G.

Again, the property is proven immediately by expressing conditional expectations from history h and possible deviations for any player i.

In the case of action invisibility, the situation is less clear, as players may not be immediately aware of the player actually deviating, hence we cannot directly consider the value function previously defined. This suggested some new constructions like the suspect game defined in [START_REF] Brenguier | Nash Equilibria in Concurrent Games -Application to Timed Games[END_REF], that encodes in a new 2-player game the set of possible deviating players. The first player wins if she manages to construct a Nash equilibrium while the other player tries to dismiss it by exhibiting a potential deviation, that can be done by one of the remaining players in set of suspects. However, this technique does not extend well when considering mixed strategies, as the resulting suspect game would contain a continuous set of allowed actions (distributions) from each state.

Subgame perfect equilibrium

An alternative and stronger notion consists in considering strategy profiles that yield a Nash equilibrium, when the game is initialized with any history, even if not enabled. With this approach, detecting deviations is pointless, as players always have to stick to the Nash equilibrium from any history. From a game theoretical point-of-view, players are more eager to pursue their own objective, instead of threatening each other to prevent deviations. Note however that such an equilibrium is less likely to occur, as it requires more stability than a Nash equilibrium. Definition 3.21. Let G = (A, s, Φ) be a stochastic game. A strategy profile σ forms a subgame perfect equilibrium for G, when for any history hs

∈ States + , (σ(h • -), s ) is a Nash equilibrium in G hs = (A, s , Φ(h • -)).
Equivalently, no player has a profitable deviation from hs : for all i ∈ Agt, h ∈ States + and σ i ∈ S i , it holds

E σ[i/σ i ](h•-) (Φ i (h • -) | s ) ≤ E σ(h•-) (Φ i (h • -) | s ) Where for any function f , f (h • -) is the function that maps each h to f (h • h ).
For prefix independent objectives, like terminal-reachability, and whenever the considered history hs is enabled by σ, (σ(h • -), s ) is a Nash equilibrium in G hs if, and only if, (σ, hs ) is a Nash equilibrium in G.

Example of equilibria

In the following section, we recall some known facts about the existence status of Nash equilibria, for games with terminal reachability objectives.

In general, several Nash equilibria may coexist (see e.g. later Figure 4.4a on page 39), in finite, countable, or uncountable number. They may require finite memory as depicted in a turn-based example of [START_REF] Ummels | Stochastic Multiplayer Games: Theory and Algorithms[END_REF].

Moreover, there exist games that admit no Nash equilibria.

Lemma 3.22. The game hide-or-run H of Figure 1.2 on page 3 has no Nash equilibria.

Proof. Since H is zero-sum, any Nash equilibrium σ is also an optimal strategy, which has been proven to not exist for this game.

Even if H is a terminal reachability game, it involves negative values as payoffs. A possible way to remove these negative values consists in shifting then rescaling the payoff profiles, which has no effect on the dynamics of the game, since expectation operator E is linear. Let us define φ = φ+1 2 , which assigns (1, 0) (resp. (0, 1)) to any history ending in the leftmost (resp. rightmost) final state, but also assigns [START_REF]1 Simple games that require randomization[END_REF][START_REF]1 Simple games that require randomization[END_REF] to the infinite run s ω 0 . As a consequence, the new game admits no Nash equilibrium, but is not terminal-reachability anymore.

In order to have only non-negative terminal-reachability payoffs, we could remove this last reward on loop s ω 0 , which corresponds to the game H depicted on Figure 3.2. But then one easily sees that the strategies σ 0 ( | s n 0 ) = 1 and σ 1 (s | s n 0 ) = 1 form a Nash equilibrium, contrary to a claim in [START_REF] Chatterjee | On Nash equilibria in stochastic games[END_REF]. In fact, it is not known whether there always exists a Nash equilibria in concurrent game with nonnegative terminal-reachability (Chapter 5 tries unsuccessfully to solve this issue).

However, the intuition behind this example remains valid as the authors of [CJM04] successfully establish the existence of a relaxed notion of Nash equilibrium, where players can deviate to improve their payoff by at most ε, for a fixed positive ε. The existence of such an ε-Nash equilibrium is proven with memoryless strategies, the take-away message being that as for the value of a zero-sum game, considering ε-optimality (here ε-Nash equilibrium) suffices to establish the existence of memoryless strategies.

Chapter 4

Decidability of Nash Equilibria

In this chapter, we study the existence problem of a Nash Equilibrium with arbitrary memory and precision in games with reachability and safety objectives. We will prove that the following general problem is undecidable: eNE INPUT:

A stochastic terminal reward game G with rational probability distributions and rewards. QUESTION: Whether there exists a Nash equilibrium for G.

Notice that from a complexity point of view, we need to ensure that G can be finitely represented to make the decision problem clear. This is why we require distributions and rewards to be rational, hence finitely representable.

We will focus on a more precise problem in this chapter, that we can describe as follows: cNE INPUT:

A deterministic terminal-reachability game G, with non-negative integral terminal rewards, and three players. QUESTION: Whether there exists a Nash equilibrium for G, which is 1-maximal, that is

E σ (Φ 1 | h) = max r Φ 1 (r).
One may notice our second problem cNE is way more restrictive, as we allow only three players, rewards ranging over N, and deterministic transition functions only. However, we require in this second problem a constraint, the 1-maximality, so cNE does not reduce immediately to eNE. This extra requirement can be justified by several consideration

• First of all, we are often not only interested in knowing whether a Nash equilibrium exists, but also in computing one. Given two payoff profiles x 1 , x 2 ∈ R Agt , we may be interested in checking whether there exists a Nash equilibrium, with payoff profile lying between x 1 and x 2 . In this particular setting, 1-maximality is a special case of constraints.

• When fixing to its maximal value the payoff of the first agent, we are interested in synthezing a strategy for this particular player, interacting with other components which are non-cooperative, but still behave rationally according to a Nash equilibrium. Thus, this problem is similar to the rational synthesis problem, studied in the non-deterministic setting by [START_REF] Kupferman | Synthesis with Rational Environments[END_REF] and [START_REF] Condurache | The complexity of rational synthesis[END_REF].

TOOLS

Undecidability of eNE will be derived from cNE by replacing 1-maximality with gadgets with possibly negative rewards. This chapter is devoted to the proof of undecidability of cNE, and its consequences. The core argument is a reduction of the non-halting problem of a Minsky machine [START_REF] Minsky | Computation: Finite and Infinite Machines[END_REF], namely a 2-counter machine. This proof is composed first of the study of some game constructions that help manipulate equilibrium payoffs and observation properties of strategies. Then, we will describe several modules, or gadgets, that will be able to carry on the encoding of a 2counter machine. By plugin together these modules, any control structure of such machine will be encoded and correctness of the reduction will be proven. Finally, we derive undecidability results for several classes of existence problems.

Similar proofs based on the non-halting problem of a Minsky machine appear in the litterature: first of all, [Umm08] proved a similar undecidability result for deterministic turnbased games, with 14 players and 1-maximality condition. This result was later improved by [DKM + 15] to 1-maximal equilibria with finite memory and pure strategies in turn-based games with at least 5 players. In the concurrent setting, [START_REF] Ummels | The complexity of Nash equilibria in limit-average games[END_REF] adapted his proof to the existence of a Nash equilibrium in 14-player concurrent deterministic games, without 1-maximality condition.

All these techniques rely on the encoding of any pair of counter values (c 1 , c 2 ) by an average payoff of 1 + 1 2 c 1 3 c 2 for some player, while another player plays an antagonistic role and gets an average payoff of 1 -1 2 c 1 3 c 2 . The reduction proceeds to construct several gadgets in order to test and update the counters and move to the next state of the counter machine. These constructions usually rely on the maximality condition, that is placed on a third player.

Keeping counter values as the simulated machine makes a transition to the next state usually requires to duplicate the two antagonistic players that encodes the counter. As a consequence, the resulting reductions cited before require at least 5 players. Here, the reduction exploits the concurrent action framework in order to avoid the duplication of the players and to keep reduction valid with 3 players. The proof is done at the expense of more complex gadgets, that we show to be reducible to single states each, thanks to the lack of action visibility.

Tools

One-shot games

In this section, we state useful properties of Nash equilibria in two-player two-action one-shot games (that is, in one step the game ends up in a terminal state).

Such games can be represented by a graph as shown in Figure 4.1a, where terminal states are labeled with their payoff functions φ for both players. Alternatively, these games, also known as one-shot games, can be represented as a payoff matrix as in Table 4.1b. Lemma 4.1. Consider the two-player two-action one-shot concurrent game G of Figure 4.1, and pick some strategy profile σ. If (σ, s 0 ) is a Nash equilibrium, then for every player i ∈ {1, 2}, it holds Proof. (σ, s 0 ) is a Nash equilibrium if, and only if, it is resilient to deterministic deviations (Proposition 3.16). Considering the deterministic deviation of player 1 returning move m, we get (omitting to mention s 0 in σ i (m) = σ i (m|s 0 )):

σ i (m | s 0 ) < 1 ⇒ [(d i -c i ) + (a i -b i )] • σ 3-i (m | s 0 ) ≤ d i -c i (4.1) σ i (m | s 0 ) > 0 ⇒ [(d i -c i ) + (a i -b i )] • σ 3-i (m | s 0 ) ≥ d i -c i (4.2) a 1 , a 2 b 1 , c 2 c 1 , b 2 d 1 , d 2
s 0 a 1 , a 2 b 1 , b 2 E k E k
a 1 σ 1 (m)σ 2 (m) + b 1 σ 1 (n)σ 2 (m) + c 1 σ 1 (m)σ 2 (n) + d 1 σ 1 (n)σ 2 (n) ≥ a 1 σ 2 (m) + c 1 σ 2 (n). As σ 1 (m) + σ 1 (n) = 1, we get b 1 σ 1 (n)σ 2 (m) + d 1 σ 1 (n)σ 2 (n) ≥ a 1 σ 1 (n)σ 2 (m) + c 1 σ 1 (n)σ 2 (n), which, assuming σ 1 (n) > 0 (or, equivalently, σ 1 (m) < 1), gives [(a 1 -c 1 ) -(b 1 -d 1 )] • σ 2 (m) ≤ d 1 -c 1 .
The other cases are similar.

k-action matching-pennies games

The classical matching-pennies games are a special case of the games of the previous section, where a i = d i and b i = c i : basically, there are two outcomes, depending on whether the players propose the same action or not. This game can be generalized to k ≥ 2 actions, as depicted on Figure 4.2. In this figure (and in the sequel), E k (resp. E k ) is a shorthand for all pairs of identical (resp. different) actions taken from a set of k actions Σ k = {c 1 , . . . , c k }. In other terms, E k represents the set

{c i c i | 1 ≤ i ≤ k}.
Lemma 4.2. In the k-action matching-pennies game, playing uniformly at random for the two players defines a Nash equilibrium. This is the unique Nash equilibrium of the game if, and only if, either (a 1 < b 1 and b 2 < a 2 ), or (a 1 > b 1 and b 2 > a 2 ). The payoff profile of this Nash equilibrium is

1 k • b 1 + 1 -1 k • a 1 , 1 k • b 1 + 1 -1 k • a 1 .
Proof. First of all, we check that when σ i (s 0 ) is uniform for both i = 1 and i = 2, σ is always a Nash Equilibrium. Indeed, for any i ∈ Agt and σ i a deviation,

P σ[i/σ i ] (s 0 • (b 1 , b 2 )) = P σ (s 0 • (b 1 , b 2 )) = 1 k .
Thus, the payoff profiles of σ and σ[i/σ i ] are equal to the announced result.

We prove now that this equilibrium is the only one if, and only if, (a 1 < b 1 and b 2 < a 2 ), or (a 1 > b 1 and b 2 > a 2 ).

TOOLS

(⇒) We prove the contraposition, by assuming the condition on payoffs is not satisfied, that is to say (a 1 ≥ b 1 or b 2 ≥ a 2 ), and (a 1 ≤ b 1 or b 2 ≤ a 2 ), and exhibit a pure Nash equilibrium in each case.

-If a 1 ≥ b 1 and a 1 ≤ b 1 , we let σ 1 (s 0 ) = c 1 and one of the best profitable actions for player 2 (c 1 if b 2 ≥ a 2 , c 2 otherwise). Player 1 has no incentive to deviate since she always wins a 1 , and player 2 already yields her maximal payoff.

-If a 1 ≥ b 1 and b 2 ≤ a 2 , then σ(s 0 ) = c 1 c 1 is a Nash equilibrium.

-The two last cases are symmetric.

(⇐) Assume a 1 < b 1 and b 2 < a 2 (the other case is symmetric) and σ is a Nash equilibrium. By shifting payoff of player 1 (resp. 2) by -a 1 (resp. -b 2 ) then by rescaling by b 1 -a 1 > 0 (resp. a 2b 2 > 0), we can assume without loss of generality that (a 1 , a 2 ) = (0, 1) and (b 1 , b 2 ) = (1, 0). The resulting game is of constant-sum and has value k-1 k , 1 k . Similarly, if player 1 does not play uniformly, some action c α occur with probability smaller than 1 k , then player 2 can ensure winning with probability bigger than 1 k by avoiding playing c α . If player 2 does not play uniformly, some action c α occur with probability smaller than 1 k , then player 1 can ensure winning with probability bigger than k-1 k by playing purely c α . Both cases are incompatible with the value payoff, thus are not Nash Equilibria.

(⇒) We prove the converse property, by noticing a Nash equilibrium that agrees on a final state exists whenever

a 1 = b 1 or a 2 = b 2 or a 1 ≤ b 1 and a 2 ≤ b 2 or a 1 ≥ b 1 and a 2 ≥ b 2 .

Embedded game

In this section, we present another technical construction that will be useful for our reduction: indeed, our reduction is modular, and consists in plugging several modules, sequentially, to encode the counter value updates. Some modules may require several branching transitions which are expected to lead to the same continuation of the simulation, from where a new Nash equilibrium encodes the updated values of the counters. However, since our strategies can observe the visited states, they can distinguish such branchings, that in turn can represent different Nash equilibria hence different counter values. We are willing to prevent this phenomenon to occur, to ensure a proper encoding of the simulation, by hiding the different histories that may occur inside a module.

The basic idea consists in transforming each module, seen as a terminal-reachability game G, into a one-shot game G , where all but one state are terminal, embedding all the histories and possible strategies in simple transitions. Intuitively, the states of G will be the maximal runs of G, which we will assume to be finitely many.

This method is presented in a refined manner, since our modules are in fact plugged together to form an entire simulation of a Turing machine. Some issues may arise when applying such a transformation:

• The whole game may have cycles, hence an infinite set of strategies and histories, that cannot be finitely described.

• Each agent now plays a whole strategy at a time. This means she can introduce a probabilistic correlation between her past sampled actions, and the new played actions, that were not possible otherwise. This may introduce new Nash equilibria that are irrelevant for the encoding.

In order to circumvent these problems, we will perform the embedding with respect to some subset of states, that we want to make undistinguishable, namely a sub-arena. We will see that Nash equilibria are indeed preserved when we additionally assume the sub-arena to be action-visible and without cycle, which is not true for the whole game.

Sub-arena

We proceed first tho define a sub-arena, which is intuitively just a subset of the states of the original arena, with the same structure, interpreted as a concurrent arena.

• Either Allow B (s) = Allow(s) and ∀A ∈ Allow(s), Tab(s, A) ⊆ States B , • Or Allow B (s) = ∅ (that is s ∈ F B ) and ∀A ∈ Allow(s), Tab(s, A) ⊆ States\States B .
Where Tab(s, A) denotes the support of distribution Tab(s, A). Any latter quantity or object relative to B will be denoted with an upper-script B.

By restricting the set of allowed actions, a previously internal state can become final in the sub-arena. When a state is not final, the sub-arena structure requires all its successors to be in the sub-arena.

In the sequel, the non-final nodes of a sub-arena B are marked on the figures with a different color, to differentiate them, as shown on the examples on Figure 4.3. For notational purpose, we write Int B = States B \F B the set of internal states of B.

Note that we do not require the sub-arena to be connected, and may have several disjoint connected components. This will appear to be useful as we want to mark all possibly problematic states in all the modules used by our reduction.

Embedded game

Our main goal, as stated before, is to hide from a strategic point of view, the events occurring in a sub-arena, and give information to the players, only about the entry point of the subarena, and the resulting exit point, which will be a final state of the sub-arena. Note that the normal game continues from those states. In the left game, players can enter the sub-arena from s 0 or s 1 . In the first case, agent 1 playing action b leads to state s 1 still in the sub-arena, then to states (x, y) or t 1 , which are both terminal in the sub-arena. In the right-hand side of the figure, state s 0 directly leads to (x, y) and t 1 without any intermediate state of the sub-arena. In order to do so, possible actions for player 1 have been replaced by a word consisting of both her actions from s 0 and s 1 , that is to say Allow 1 (s 0 ) = {ac 1 , ac 2 , bc 1 , bc 2 }. Similarly, both actions for player 2 are encoded simultaneously from s 0 : Allow 2 (s 0 ) = { †c 1 , †c 2 }. Whenever the played action of player 1 starts with letter a, the game continues to state t 0 . Otherwise, the second letter of her played action has to be compared to the second letter of agent 2, which determines whether state (x, y) or t 1 is reached. We conclude that both games behave similarly, in the sense that probabilities to reach one of the final states of the sub-arena t 0 , t 1 or (x, y) are the same. Moreover, there exists a single history connecting s 0 to any of them, once the transformation has been applied.

4.1. TOOLS s s 0 s 1 x, y t 0 t 1 E 2 E 2 a † b † E 2 E 2 . . . . . . s s 0 x, y s 1 t 0 t 1 E 2 E 2 (a * )( † * ) (bc 1 )( †c 2 ), ( bc
The construction is made precise in the following definition: Definition 4.5. Let G be a terminal-reachability game and B a sub-arena. We construct G = G/B the quotient game G by B, by extending the set of states to

• States = States {⊥} • If s ∈ States\Int B , Allow (s) = Allow(s) and Tab(s, -) = Tab (s, -). • If s ∈ Int B , ∀i Allow i (s) = M B i which is finite, and ∀σ ∈ M B ∀s ∈ F B Tab (s, σ)(s ) = P σ B ((Int B ) * • s | s)
is the probability to eventually reach final state s from s in arena B under strategy profile σ, considered here as an action profile in game G . We normalize by adding a transition to

⊥: ∀σ ∈ M B Tab (s, σ)(⊥) = P σ B ((Int B ) ω | s) the probability to never reach a final state.
More informally, each non-terminal state of the sub-arena B, is converted into a bigger state, where each allowed action corresponds to the whole strategy to a final state of the subarena. In the resulting game, what happens from the entrance to the exit of the sub-arena is encoded in one single state (for each entrance state). As a consequence, we can say that the strategy, when exiting the state, cannot distinguish between the possible plays inside the sub-arena. We formalize this notion below. Definition 4.6. We define, ε/B = ε and for any h

• s ∈ States + , (h • s)/B = h/B if h = ε and last(h) • s ∈ (Int A u ) 2 (h/B) • s otherwise
Intuitively, h/B only keeps from history h states that enter sub-arena B and terminal states that leave this sub-arena.

For example, back to Figure 4.3, we have for both games, s

• s 0 • s 1 • t 1 /B = s • s 0 • t 1 .
Definition 4.7. We say a strategy σ is blind to a sub-arena B, or B-blind for short, if for all

h, h 1 , h 2 ∈ States + , such that (h 1 /B) • h = (h 2 /B) • h, we have σ(h 1 • h) = σ(h 2 • h).
As we can expect, the previous notion allows us to consider Nash equilibria with blindness to some sub-arena, from the perspective of a standard Nash equilibrium, thanks to the following theorem: Theorem 4.8. Let G be a terminal-reachability game and B an action-visible sub-arena and payoff vector v ∈ R Agt . Assume that for two internal states s, s ∈ Int B , there exists at most one path (with actions) from s to s .

Then, there exists a Nash equilibrium with value v in G/B if, and only if, there exists a Nash equilibrium with value v in G which is blind to B. Remark 4.9. Notice the hypothesis on B is very strong, it implies first that the sub-arena is action-visible, but also that it has no cycle. In particular, this means that ⊥ state won't be reachable. If the sub-arena is only action-visible and cycle free, one can convert it to add finite memory inside the current state (as for the conversion from action-invisible to action-visible arena). This transformation is not developed further as Definition 4.5 would require encoding finite memory strategies as actions (instead of memoryless), and this more general result is not useful in our reduction.

Before proving the lemma directly, we introduce some notations and justify why the action-visible property is crucial:

Lemma 4.10. Let h be an history of B. Let Prev(h) = ({σ ∈ M B | P σ B (h) > 0}
) and assume B is action-visible, then for any i ∈ Agt, there exists a set Prev i (h) such that Prev(h) = i∈Agt Prev i (h).

Proof. We prove the result by induction on |h|.

• If |h| = 1, the result is immediate, as Prev(h) = M B and ∀i Prev i (h) = M B i .
• Let h • s be an history such that Prev i (h) are already proven to exist. Let σ ∈ M B , then

P σ (h • s) > 0 is equivalent to P σ (h) > 0 and P σ (last(h) • s) > 0. However, there exists at most one action profile A ∈ Act Agt such that Tab(last(h), A)(s) > 0. If A does not exist, we have Prev(hs) = ∅, otherwise, Prev(hs) = {σ ∈ Prev(h) | σ(last(h)) = A} = i∈Agt {σ i ∈ Prev i (h) | σ i (last(h)) = A i }, which concludes the induction step.
Thanks to this lemma, we can locally translate an embedded strategy profile into strategies for each player: Lemma 4.11. Let B an action-visible sub-arena, such that for any s 1 , s 2 ∈ Int B two internal states, there exists at most a path (with actions)from s 1 to s 2 . Let σ u ∈ S G/B , and σ ∈ S G such that for any h

∈ (Int B ) + ∀i ∈ Agt ∀a ∈ Act σ i (a | h) = σ u i (first(h)) σ i ∈ Prev i (h) σ i (h) = a 4.1. TOOLS
Then, for all s 1 ∈ States B , s f ∈ F B , we have

P σ u G/B (s 1 • s f ) = h∈(States B ) * P σ B (h • s f | s 1 )
Proof. If s 1 ∈ F B then both sides are equal to 0.

We assume now that s 1 ∈ Int B . For any s 2 ∈ Int B , we write w s 2 ∈ (Act Agt ) * the unique sequence of action profiles, if any, leading from s 1 to s 2 in B, denoted by history h s 2 .

Then, any h ∈ (States B ) * , such that

P σ B (h • s f | s 1 ) > 0 can be uniquely described as h = h s 2 for some s 2 ∈ Int B .
Previous sum can now be rewritten:

h∈(States B ) * P σ B (h • s f | s 1 ) = s 2 P σ B (h s 2 • s f ) = s 2 P σ B (h s 2 )P σ(h s 2 -) B (s 2 • s f ) = P σ u G/B (s 1 • s f )
We now turn to the proof of Theorem 4.8.

Proof. We describe first how to convert a strategy profile σ u for G/B into a strategy profile σ which is B-blind.

For any h ∈ States + , let h be its maximal suffix in Int B , and define σ i (a | h) by

σ i (a | h) = σ u i (a | h/B) if h = ε σ u i (h/B) ({σ i ∈ Prev i (h ) | σ i (h) = a}) otherwise
We can check that the corresponding strategy profile σ is indeed B-blind by applying the definition. Notice that the translation can be done the other way around, to obtain a strategy for G/B from a blind strategy profile σ.

For such pairs of strategy profiles, we show that they have the same pay-off. In order to do so, we introduce the measure µ(h) = h |h / ∈States * (Int B ) 2 ∧h /B=h P σ G (h ), which sums together all equivalent history probabilities in G for B-blindness. We prove that ∀h ∈ States + µ(h) = P σ u (h) by induction on |h|. For the inductive case, we have to distinguish the case where last(h) / ∈ Int B which is immediate, to the opposite case, where we apply Lemma 4.11. Since we are considering terminal-reachability games, and final states are preserved by embedding, we conclude that σ and σ u yield the same payoff vector v.

Finally, we have to show σ is a Nash equilibrium if, and only if, σ u is a Nash equilibrium. The reverse implication is immediate as we can extract from a deviation of σ u a deviation of σ which is still B-blind. The direct implication is concluded by noticing that a deviation can be assumed to be deterministic, hence, we can build a B-blind deviation, which can then be converted into a deviation in the embedded game.

On the action-visible hypothesis

We develop here why the restriction to action-visible games is crucial by considering the game structures depicted in Figure 4.4 on the next page.

Game C is obtained from game C by making actions from s 0 visible, as discussed in Remark 3.6. Game C will be shown to correspond both to games C/A and C /A . 1, 1

3, 0 0, 0 2, 2 s 1 s 0 ab, ba aa, bb E 2 E 2 (a) C, two equilibrium payoffs: 2, 1 2 , 5 2 , 1 3, 0 0, 0 2, 2 s a 1 s b 1 s 0 ab, ba aa bb E 2 E 2 E 2 E 2 (b) C , with an additional equi- librium payoff: 7 3 , 2 3 3, 0 0, 0 2, 2 s 0 a * b * , b * a * ac i ac j , bc i bc j , i = j ac i ac i , bc i bc i (c) C: Equivalent embedded game.
Payoff matrix from state s 0 , assuming the players agree on a deterministic action from state s b 1 but play uniformly at random from state s a 1 . There are only two possible equilibria in s 1 : either both players agree on playing the same action, yielding payoff (2, 2), or both players play uniformly yielding payoff profile (1, 1). From state s 0 , the second player has always an incentive to go to s 1 whereas the first player is always better off with profile (3, 0). We conclude that game C has two equilibrium :

               σ 1 (a | s 0 ) = 2 2 + 1 = 2 3 σ 2 (a | s 0 ) = -1 -1 -2 = 1 3 E σ (Φ i | s 0 ) = 7 3 , 2 3 (b
• σ i (s 0 ) uniform and σ 0 (s 1 ) = σ 1 (s 1 ), with payoff profile 5 2 , 1 ; • σ i (s 0 ) and σ i (s 1 ) uniform, with payoff profile 2, 1 2 . The previous equilibria can be achieved in the action-visible variant C , but a new equilibrium can occur: agents can play uniformly from s α 1 and agree on an action in the other state s β 1 (α = β). Let us consider the case where s a 1 is played uniformly, the other case being symmetric. From state s 0 , game can now be seen as a one-shot game with the following payoff table represented in Figure 4.5a. This game has a unique equilibrium that can be computed by Lemma 4.1 as seen in Figure 4.5b.

However, both embedded games C/A and C /A are somehow equivalent: let us denote the actions σ i ∈ M A from s 0 in C/A as a word w = σ i (s 0 )σ i (s 1 ) ∈ Act 2 . We represented in the same way actions in C /A as a word w = σ i (s 0 )σ i (s a 0 )σ i (s b 0 ). Even if C /A allows more actions, we argue that these extra actions are equivalent. Indeed, for any x, y ∈ Act, embedded action axy (resp bxy) in C /A is equivalent to embedded action ax (resp by) in C/A from the point of view of transition function. This equivalence relation is transposed to the expected payoffs of both players, and their possible deviations.

We conclude that C/A, C /A and C have the same set of Nash Equilibria. This implies that Theorem 4.8 cannot apply on C and sub-arena A which is not action-visible.

To summarize, we have in this section that several branchings in a module can be hidden from strategies thanks to an embedding of all possible strategies seen as simple actions. The reduction is shown to be sound whenever action are visible in the module and that there exists a unique path from every two states in the module. Notice we can replace the latter condition by a simple cycle-free hypothesis by using finite memory strategies as actions instead of simply memoryless strategies. However, the action-visible hypothesis is crucial for the correction of the proof. Notice also that paradoxically, the resulting embedded state has multiple actions (one for each strategy) pointing to the same final state in the sub-arena. This means that strategies should not see actions, which is the case in our model, in order to hide these branchings.

Modules

In this section, we present a game that will be a building block in our undecidability proof, which is based on an encoding of a two-counter machine.

Rescale game

We discuss here different ways of updating the values of a game when the strategy profile forms a 1-maximal Nash equilibrium. This will be useful later for updating counter values in the undecidability reduction. Assume the continuation of a game forms a Nash Equilibrium of payoff (1, 4+x, 4-x) for some x ≥ 0. We want to produce a new 1-maximal Nash Equilibrium with payoff (1, 4 + α • x, 4α • x) for some constant α. A game achieving such a property is described below.

Consider the 3-player game R k depicted on Figure 4.6: in this game, player 1 has two available actions a and b from r 0 , s k and s l , while the other two players can either continue (action c), or unilaterally decide to stop the game (action s) and go to a terminal state (where player 1 will have payoff 0). Notice that s k and s l have the same structure, but with different values for their immediate terminal successors. In node t k , only players 2 and 3 have a choice: they can either continue to final state n (when both of them play c), or decide to stop and possibly go to a k-action matching-pennies game. In Figure 4.6, we write S as a shorthand to represent any combination of moves of players 2 and 3 where at least one of them decides to stop (action s). Node n is a final node with some payoff (1, 4 + x, 4x) that will be later replaced by the continuation of the game, when considering the actual reduction. Notice that the sub-arena described by the set of states {r 0 , s k , s l , t k , (0, 4, 4), (0, 5, 3), (0, 5 + k, 3k), (1, 4 + k, 4k), (0, 5 + l, 3l), (1, 4 + l, 4l)} is action-visible. Notice that t k is a terminal node in this sub-arena, therefore not marked. Indeed, if t k was marked, there would have been two paths from r 0 to t k in the sub-arena, and Theorem 4.8.

We relate 1-maximal Nash equilibria from r 0 and those from n:

Proposition 4.12. Consider game R k from Figure 4.6, with some fixed parameter x ∈ R.

Then there exists a 1-maximal Nash equilibrium σ from r 0 if, and only if, Moreover, for any such Nash equilibrium, terminal state n is reached with positive probability and expected payoff profile equals

0 ≤ k • x ≤ 1.
(1, 4 + k • x, 4 -k • x).
We first give the proof of the necessary conditions:

Proof. Since the strategy profile has to be 1-maximal, any terminal state rewarding agent 1 with payoff 0 should be avoided. One of the two states s k or s l (or both) should be enabled by σ. However, action profile acc cannot be played with probability 1 by all three players, otherwise player 2 can deviate to win 5 + k (resp 5 + l) instead of 4 + k (resp 4 + l). This means that t k is also enabled by strategy profile σ. Players should then agree to go to final state n from t k in order to achieve 1-maximality.

Player 2 has no incentive to deviate from t k , so 4 + x ≥ 4, that is x ≥ 0. Value of state u k can be computed and equals v(u k ) = (0, 4 + 1/k, 4 -1/k). Since player 3 has no incentive to deviate from t k to enable state u k , we also have 4

-x ≥ 4 -1/k that is k • x ≤ 1.
The sufficient condition, and the exact computation value is carried on in the following lemma. We assume that 0 ≤ k•x ≤ 1, and start to build a strategy profile which is 1-maximal Nash equilibrium. We can already state that such profile must enable n with probability 1 from t k and play uniformly from state u k to avoid any deviation, from t k . Lemma 4.13. Assume that there is a Nash equilibrium from

s k •t k with payoff (1, 4+x, 4-x). If (σ, s k ) is a 1-maximal Nash equilibrium from s k , then the state t k is activated by σ, and the expected payoff of σ from s k is (1, 4 + (k + 1) • x x+1 , 4 -(k + 1) • x x+1
). Furthermore such a 1-maximal Nash equilibrium from s k exists and consists for player 1 in playing a with probability x/(x+1) and b with probability 1/(x+1), and for the other two players, in playing c almost-surely in s k , and then follow the given equilibrium from s k t k .

Proof. Let (σ, s k ) be a 1-maximal Nash equilibrium. Because the equilibrium is 1-maximal, players 2 and 3 do not play action s:

we have σ 2 (c | s k ) = σ 3 (c | s k ) = 1. Considering this 4.2. MODULES s c a b 1 2 0, 5 + k 0, 4 1, 4 + k 1, 4 + x s c a b 1 3 0, 3 -k 0, 4 1, 4 -k 1, 4 -x
Table 4.1 -Two-player projections of R k in s k assuming player 3 (left), resp. player 2 (right), plays c almost-surely fixed action for player 3, we can look at the 2-player game between players 1 and 2, which is represented in matrix form in Table 4.1 (left). Applying Lemma 4.1, using the fact that σ 2 (s

| s k ) < 1, we get that (x+1)•σ 1 (a | s k ) ≤ x.
The same argument applied to the projection to players 1 and 3 (see Table 4

.1 (right)) gives (x + 1) • σ 1 (a | s k ) ≥ x. Hence (x + 1) • σ 1 (a | s k ) = x.
This entails that x = -1 (actually, x will be forced to be nonnegative in the sequel), so that the action profile bcc has probability 1/(x + 1) in the Nash equilibrium (σ, s k ). We conclude that t k is reached with positive probability, and that

E σ (φ | s k ) = 1, (4 + k)x + (4 + x) x + 1 , (4 -k)x + (4 -x) x + 1 .
Conversely we check that the strategy profile from s k where player 1 plays a with probability x/(x + 1) and b with probability 1/(x + 1) in s k , and where the other two players play c almost-surely, is a 1-maximal Nash equilibrium.

Notice the same reasoning (and lemma) can be done from state s l = s k-1 , with the same conclusions.

We now consider the global game R k from its initial state r 0 , in order to conclude.

Lemma 4.14. Assume that 0 ≤ k • x ≤ 1. Then, any 1-maximal Nash equilibrium from r 0 has payoff (1, 4 + k • x, 4 -k • x)
. Moreover, such a strategy profile exists.

Proof. Since the equilibrium is 1-maximal, it holds

σ 2 (c | r 0 ) = σ 3 (c | r 0 ) = 1.
We first consider the cases when only one of the states s k and s l is enabled:

• if only s k is enabled, i.e. σ 1 (a | r 0 ) = 1
, then from the previous lemma, we have

E σ (Φ 2 ) = 4 + (k + 1) • x x+1
. This quantity should be greater than 5 (otherwise player 2 would better deviate), so that k • x ≥ 1, and using our hypothesis, k • x = 1. It follows that the payoff of σ from r 0 is (1, 5, 3) in this case.

• if only s l is enabled, i.e. σ 1 (a | r 0 ) = 1, the value for player 3 is 4 -(l + 1) • x
x+1 . This must be greater than or equal to 4 (otherwise player 2 has a profitable deviation). We get x = 0, and the expected payoff of σ is [START_REF]1 Simple games that require randomization[END_REF]4,4).

We now consider the case where both states s k and s l are enabled, i.e. 0 < σ 1 (a | r 0 ) < 1. We again separately consider the strategies of players 2 and 3, as shown in Table 4.2. Let us fix y i (for i ∈ {k, l}) and y such that

E σ (φ | r 0 s i ) = (1, 4 + y i , 4 -y i ), and E σ (φ | r 0 ) = s c a b 1 2 0, 5 0, 4 1, 4 + y k 1, 4 + y l s c a b 1 3 0, 3 0, 4 1, 4 -y k 1, 4 -y l
y l + 1 -y k ) • σ 0 (a | r 0 ) = y l .
Then the expected payoff for player 2 is

E σ (Φ 2 | r 0 ) = 4 + y = σ 1 (a | r 0 ) • (4 + y k ) + (1 -σ 1 (a | r 0 )) • (4 + y l ).
This simplifies as y = y l /(y ly k + 1). Replacing y l and y k with their values (Lemma 4.13), we end up with y = k • x. Now to reconstruct a 1-maximal Nash equilibrium from r 0 , it is sufficient for player 1 to play a at r 0 with probability k • x, and for the other two players, to play c almost-surely at r 0 , and then to follow the 1-maximal Nash equilibrium from s k and s l . This yields a 1-maximal Nash equilibrium from r 0 , with the announced expected payoff profile.

To summarize this section, we have built a module that takes a sub-game as an input (from state n), which is assumed to have an equilibrium of payoff (1, 4 + x, 4x), and returns a game which has an equilibrium of payoff (1, 4 + k • x, 4k • x) for a given integer k. Notice the reverse operation (dividing by k) is more natural, as it is sufficient to implement a kmatching-pennies state to implement it. This way of the rescaling game was harder, as it requires somehow to increase the gap of expected rewards between players 2 and 3. This has been achieved thanks to the extra 1-maximality requirement.

Testing game

We present in this section the construction of a game for comparing the expected payoffs in different nodes. This will be useful in our reduction to encode the zero-tests of our twocounter machine. Once again, this construction heavily relies on the 1-maximal character of our studied Nash Equilibria.

Consider the game T of Figure 4.7. Any non-terminal state is now marked, since the whole game is action-visible. Two terminal states, n 1 and n 2 , will be later replaced by continuation of another reduction, but are given here with 1-maximal Nash equilibria payoff profiles of the form (1, 4 + x, 4x) and (1, 4y, 4 + y) for some x, y ∈ R ≥0 . Proposition 4.15. Consider the game T of Figure 4.7, for some parameters x, y ≥ 0. There exists a 1-maximal Nash equilibrium from s 0 if, and only if, x = y.

Moreover, when this condition holds, the expected payoff its equals (1, 4 + x/2, 4x/2). Additionnaly, n 1 and n 2 are both enabled whenever x = y > 0.

Proof. If x = y, we build σ that plays uniformly from states s a 2 and s b 2 , yielding an equilibrium of value [START_REF]1 Simple games that require randomization[END_REF]4,4). Playing uniformly from s 0 then playing action profile †cc from s a 1 and s b Notice that state s 2 should be considered terminal, as it only carries a self-loop. We could replace it by a two-state loop. We could also see it as a terminal state with reward (0, 0, 0), but for technical reasons (in Section 4.3.2), we want the terminal rewards of players 1 and 2 to always sum to 8, which we could not achieve easily in this case.

1 43 4.2. MODULES s 0 s a 1 s b 1 0, 4, 4 s a 2 s b 2 n 2 : 1, 4 -y, 4 + y n 1 : 1, 4 + x, 4 -x †ab †ba †S †S †cc †cc †E 2 †E 2 †E 2 †E 2 †E 2 Figure 4.7 -Testing module T s 0 s 2 1, 3, 5 1, 4, 4
s 1 †aa †ba †ab †bb †E k †E k game C k : s 0 s 2 0, 3, 5 1, 3,
generates the announced payoff, and we can check that no deviation to (0, 4, 4) is profitable for any player.

Conversely, if σ is a 1-maximal Nash equilibrium from s 0 ,

• if n 1 (resp n 2 ) is enabled alone, then x = 0 (resp y = 0), otherwise player 3 (resp 2) could deviate to (0, 4, 4).

• Otherwise, both n 1 and n 2 are enabled, then at least one of the two states s a 1 or s b 1 is visited, which means by stability that x = y (otherwise one of the two players can deviate). Then, by stability, both players should play uniformly from s 0 .

Counting modules

Previous testing module allows us to test two sub-games which have similar (when reversing player 2 and 3) payoff profiles. Our reduction strategy will consist in simulating on the one hand a 2-counter machine, and on the other hand, check that the resulting encoding has a particular form. In order to check for this particular form, we introduce below games that can generate some specific families of Nash equilibria with a particular expected payoffs.

r k (n) = 1, 4 - 1 k n , 4 + 1 k n s(n) = 1, 4 - 1 n + 1 , 4 + 1 n + 1 Then the set of 1-maximal Nash equilibrium values is {s(n) | n ∈ N {∞}} in D, and {r k (n) | n ∈ N {∞}} in C k for all k ≥ 2.
Proof. Fix k ≥ 2. We begin with proving that these values are indeed the payoffs of Nash equilibria. For this, we define the witnessing strategy profiles. For all history h, we let

γ ∞ (hs 0 ) = abb γ ∞ (hs 1 ) = ac 1 c 2 δ ∞ (hs 0 ) = abb δ ∞ (hs 1 ) = bss
One easily observes that (γ ∞ , s 0 ) and (δ ∞ , s 0 ) are Nash equilibria with payoff [START_REF]1 Simple games that require randomization[END_REF]4,4) in C k and in D, respectively.

For n ∈ N, we define γ n and δ n inductively. First, we let γ 0 (s 0 ) = δ 0 (s 0 ) = aaa, which gives rise to Nash equilibria with payoff (1, 3, 5) from s 0 both in C k and in D.

Then, for n ∈ N, we define the strategies inductively on the length of the history. The base cases are γ n+1 (s 0 ) = δ n+1 (s 0 ) = abb and

γ n+1 1 (s 0 s 1 ) = a ∀1 ≤ j ≤ k. γ n+1 2 (c j | s 0 s 1 ) = γ n+1 3 (c j | s 0 s 1 ) = 1/k δ n+1 1 (a | s 0 s 1 ) = 1 n + 2 δ n+1 2 (s 0 s 1 ) = δ n+1 3 (s 0 s 1 ) = c.
The inductive case is

∀1 ≤ i ≤ 3, γ n+1 i (s 0 s 1 h) = γ n i (h) δ n+1 i (s 0 s 1 h) = δ n i (h)
We can indeed check that these strategy profiles form Nash equilibria with the expected payoffs.

Conversely, let us fix a 1-maximal Nash equilibrium (σ, s 0 ), and let us show that the expected payoff of that Nash equilibrium is one of the above values. By 1-maximality, players 2 and 3 have to play deterministically the same actions after all histories ending up in s 0 (state s 2 should not be enabled under (σ, s 0 )). We can reason on the number of histories enabled from s 0 . For that, we define N s 0 (σ) = sup{|h| | h enabled by σ and last(h) = s 0 } (this is somehow the maximal number of visits of s 0 enabled by σ).

• If N s 0 (σ) = +∞, then the transition †aa from s 0 is never taken (since it would then be played deterministically and it would then stop the game immediately). Since (σ, s 0 ) is 1-maximal, it then means that the terminal state (1, 4, 4) is reached almost-surely (only possibility for Player 1 to get payoff 1).

• Otherwise, N s 0 (σ) is finite and we reason by induction on this number:

first if N s 0 (σ) = 1, the game ends up immediately in [START_REF]1 Simple games that require randomization[END_REF]3,5).

if N s 0 (σ) > 1, then a transition to s 1 occurs with probability 1. If the path s 0 • s 1 • (1, 4, 4) has probability 1 under σ, then the results holds; otherwise s 0 s 1 s 0 is enabled from s 0 , and (σ , s 0 ), with σ : h → σ(s 0 s 1 h), is another 1-maximal Nash

MODULES

equilibrium such that N s 0 (σ ) < N s 0 (σ). By induction, it has an expected payoff of the form (1, 4x, 4 + x), with either x = 1 k n (for C k ) or x = 1 n+1 (for D) for some n ∈ N {∞}. If x = 0, the results holds immediately, as the expected payoff of (σ, s 0 ) is also [START_REF]1 Simple games that require randomization[END_REF]4,4). Now assume x > 0, and consider the game C k , and the distributions proposed by the strategies σ 2 and σ 3 after s 0 s 1 . For this to be a Nash equilibrium, both distributions must be uniform; this leads to payoff (1, 4 -x k , 4 + x k ), and proves the result. For D, if it were σ 1 (s 0 s 1 ) = b, then Player 2 would have a profitable deviation. Hence σ 1 (a | s 0 s 1 ) > 0, and the best response for players 2 and 3 is to play c. We can then analyze the projections on agents 1, 2 and 1, 3 (as done in Section 4.2.1) and apply Lemma 4.1, which yields

(x + 1)σ 1 (a | s 0 s 1 ) ≥ x and (x + 1)σ 1 (a | s 0 s 1 ) ≤ x; it follows σ 1 (a | s 0 s 1 ) = 1 n+2 and E σ (φ | s 0 s 1 ) = (1, 4 -1 n+2 , 4 + 1 n+2 ).

Description of the reduction

We now turn to the global undecidability proof of the constrained-existence problem in threeplayer games. The proof is a reduction from the recurring problem of a two-counter machine.

We first recall the definition of a two-counter machine as a tuple M = (Q, q 0 , ∆) where:

• Q is a finite set of states, • q 0 ∈ Q is an initial state, • ∆ ⊆ Q×Γ×Q is the transition table with Γ = {inc(j), dec(j), zero(j), !zero(j) | j ∈ {1, 2}}
is the set of operations on counters.

Without loss of generality, we assume that any considered machine M never decreases a counter with value 0. This can be syntactically enforced by placing a non-zero test before any decrement operation.

The semantics of M = Q, q 0 , ∆ is given as a transition system where configurations are tuples C = (q, c 1 , c 2 ) ∈ Q × N × N and for any two configurations C = (q, c 1 , c 2 ) and C = (q , c 1 , c 2 ), for every δ = (q, γ, q ) ∈ ∆, there is a transition C → δ C if, and only if:

• c k = c k + 1 and c 3-k = c 3-k , if γ = inc(k); • c k = c k -1 and c 3-k = c 3-k , if γ = dec(k); • c k = 0 and (c 1 , c 2 ) = (c 1 , c 2 ), if γ = zero(k); • c k > 0 and (c 1 , c 2 ) = (c 1 , c 2 ), if γ = !zero(k).
We fix for the rest of this section a two-counter machine M = (Q, q 0 , ∆), and we build a terminal-reachability game G M as depicted in Figure 4.9 on the facing page: for every machine state q ∈ Q (resp. every δ ∈ ∆) we have a corresponding state q (resp. state δ) in the game G M , as depicted on Figure 4.9b (resp. Figures 4.9c to 4.9f). These states are connected together thanks to the described translation which takes into account the transition table of M. We write φ for the terminal-reachability payoff function that is given by G M .

The states of the game G M are Q ∆ T , where T represents the set of states that were introduced by our reduction, as part of one of the different gadgets, with multiplicity. Moreover, we have seen that these modules may have some branching, that will be hidden by considering the embedded game construction with respect to B, which is the sub-arena composed of the marked states of the modules. For the sake of simplicity, we assume, from now on, that G M has already been quotiented by this sub-arena. In the sequel, any considered Nash equilibrium or strategy profile will therefore become implicitly a B-blind strategy profile in the original game. We will evaluate the existence of a 1-maximal Nash equilibrium from all the main nodes of the game. The relation between M and G M is made explicit thanks to the following predicate. Let s ∈ Q ∆, and c 1 , c 2 ∈ N. We denote by P (s, c 1 , c 2 ) the predicate:

in q0 0, 5, 3 * S * cc (a) Input gadget of G M q s †E n δ1 δ2 δ3 δ4 † δ 1 δ 1 † δ 2 δ 2 † δ 4 δ 4 † δ3 δ3 (b) Translation of q (where {δ 1 , ..., δ n } is the set ∆ q of transitions leaving q) δ q 1, 4, 4 †E k+1 †E k+1 (c) translation of δ = (q, dec(k), q ) δ R k+1 n = q r0 (d) Translation of δ = (q, inc(k), q ) δ R 2 r0 T s0 n 2 n 1 = q C 4-k (e) translation of δ = (q, zero(k), q ) δ R 2 r0 T s0 n 1 = q n 2 D 1, 4, 4 †E k+1 †E k+1 (f) translation of δ = (q, !zero(k), q )
∃σ ∈ S. σ is a Nash equilibrium from s and E σ (φ | s) = 1, 4 + 1 2 c 1 3 c 2 , 4 - 1 2 c 1 3 c 2 Lemma 4.17. Assume C = (q, c 1 , c 2 ) is a configuration of M such that P (q, c 1 , c 2 ) holds.
Then there are a transition δ and a configuration C = (q , c 1 , c 2 ) such that (i) C → δ C and (ii) P (δ, c 1 , c 2 ) and P (q , c 1 , c 2 ) hold.

Proof. Write σ for a 1-maximal Nash equilibrium witnessing the truth of predicate P (q, c 1 , c 2 ). In particular,

E σ (φ | q) = 1, 4 + 1 2 c 1 3 c 2 , 4 - 1 2 c 1 3 c 2

MODULES

In G M from q, only one state δ ∈ ∆ q is activated, otherwise σ would not be 1-maximal, as with positive probability the play would end up in state s of Figure 4.9b. Hence P (δ, c 1 , c 2 ) holds as well.

We write x = 1 2 c 1 3 c 2 , and we distinguish the different cases for δ. • First assume δ = (q, dec(k), q ). Since x > 0, the next state q has to be activated by σ, and (σ, qδ q ) needs to be a 1-maximal Nash equilibrium as well. Applying the analysis of k-action matching-pennies games of Section 4.1.2, it must be the case that the payoff of σ after qδ q is (1, 4 + y, 4y) with y = (k + 1)

• x. If k = 1, it is the case that y = 1 2 c 1 -1 3 c 2 , and if k = 2, it is the case that y = 1 2 c 1 3 c 2 -1 . Writing c k = c k -1 and c 3-k = c 3-k , we get that (q, c 1 , c 2 ) → δ (q , c 1 , c 2 ) and that P (q , c 1 , c 2 ).
• Then assume δ = (q, inc(k), q ). Applying Proposition 4.12, we get that q is activated by σ, and that there is a 1-maximal Nash equilibrium from q whose expected payoff is (1, 4 + y, 4y) with y = x/(k + 1). As in the previous case, writing c k = c k + 1 and c 3-k = c 3-k , we get that (q, c 1 , c 2 ) → δ (q , c 1 , c 2 ) and that P (q , c 1 , c 2 ).

• Assume δ = (q, zero(k), q ). Applying Proposition 4.12, we get that the first node s 0 of G t is activated by σ, and that there is a 1-maximal Nash equilibrium from that node whose payoff is (1, 4+x/2, 4-x/2). Then, applying Proposition 4.15, we get that q and the initial node of C k+1 are activated, and that σ after q and σ after entering C k+1 are 1-maximal Nash equilibrium. Furthermore, writing (1, 4 + z, 4z) and (1, 4y, 4 + y) for the payoffs of those equilibria respectively, we should have z = y and x/2 = z/2. In particular, P (q , c 1 , c 2 ) holds. Now thanks to Lemma 4.16, we know that there exists m such that y =

1 (4-k) m . This implies that c k = 0: (q, c 1 , c 2 ) → δ (q , c 1 , c 2 ).
• Finally, assume that δ = (q, !zero(k), q ). Again applying Proposition 4.12, we get that the first node of G t is activated by σ, and that there is a 1-maximal Nash equilibrium from that node whose payoff is (1, 4 + x/2, 4x/2). Then, applying Proposition 4.15, we get that q and node n 2 are activated, and that σ after q and σ after entering n 2 are 1-maximal Nash equilibria. Furthermore, writing (1, 4 + z, 4z) and (1, 4y, 4 + y) for the payoffs of those equilibria respectively, we should have z = y and x/2 = z/2. In particular, P (q , c 1 , c 2 ) holds.

Note that there is a 1-maximal Nash equilibrium from n 2 of payoff (1, 4y, 4 + y) with y > 0 if, and only if, there is a 1-maximal Nash equilibrium in D of payoff (1, 4-y , 4+y ) with y = (k + 1) • y. Now, thanks to Lemma 4.16, we know that there exists h such that

y = 1 h+1 , hence y = 1 (k+1)•(h+1) . This implies that c k > 0: (q, c 1 , c 2 ) → δ (q , c 1 , c 2 ).
We can now show the following correspondence between M and G M :

Proposition 4.18. The two-counter machine M has an infinite valid computation if, and only if, there is a 1-maximal Nash equilibrium from state in in game G M .

Proof. We use the reduction from the non-halting problem of a two-counter machine we have described. Given a two-counter machine M, we construct game G M as on Figure 4.9. For technical reasons, we syntactically require that each incrementation is followed by a non-zero test: since decrements are preceded with a zero-test, this enforces infinitely many visits to module G t in G M along any infinite run, so that infinite runs will have probability zero in strategy profiles we will build (we will see that later).

Assume a 1-maximal Nash equilibrium exists from the initial state in. Then it must be the case that its payoff profile is at least [START_REF]1 Simple games that require randomization[END_REF]5,3), hence equal to [START_REF]1 Simple games that require randomization[END_REF]5,3) as the payoffs sum of player 2 and 3 always sum to 8 and payoffs of player 1 are at most 1. As the predicate P (q 0 , 0, 0) is true, we can inductively apply Lemma 4.17 to build the corresponding valid infinite run of the counter machine.

Conversely assume that M has an infinite (valid) run

C 0 → δ 0 C 1 → δ 1 . . . with C i = (q i , c i 1 , c i 2 )
, and c 0 1 = c 0 2 = 0. We build a strategy profile σ inductively as follows: • in state in, players 2 and 3 should play c almost-surely;

• we assume we have built σ for the prefix C 0 → δ 0 C 1 → δ 1 . . . C i , and that the "main stream" of σ traverses successively the gadgets starting in in, q0 , δ0 , . . . , δi-1 and arrives in state qi , from which we now need to define the strategy profile σ. In state qi , the players should select transition δ i almost-surely, and then enter gadget state δi . We now distinguish the possible cases for δ i :

if δ i = (q i-1 , dec(k), q i ), then players 2 and 3 should play uniformly at random among the k + 1 actions;

-if δ i = (q i-1 , inc(k), q i ), then, in R k+1
, the players should follow the strategy described in Proposition 4.12 with x =

1 (k+1)•2 c i-1 1 •3 c i-1 2 ; -if δ i = (q i-1 , zero(k), q i ), then, in R 2
, the players should follow the strategy described in Proposition 4.12 with x = if δ i = (q i-1 , !zero(k), q i ), then we apply a strategy as described in the previous item (except that we replace the strategy in C 4-k by that in D).

1 2•2 c i-1 1 •3 c i-1 2 ; in G t ,
First, due to the hypothesis on (non-)zero-tests in every syntactic loop of the machine, under the above strategy profile, the game ends up almost-surely in a terminal state, where Player 1 has reward 1. This is because in any state δ where δ is a test-to-zero, or a test-to-nonzero, the game ends up in gadget C 4-k or D with probability 1/2, and the previously defined strategies from these gadgets ensure almost-sure termination. More formally, for every ε > 0, there is a length N ε that we can easily compute such that

P σ (reach terminal in no more than N ε steps | in) ≥ 1 -ε.
We write G M (N ε ) the game G M (N ε ) truncated after N ε computation steps of M, in which we replace any outgoing transition by a terminal node with reward (1, 0, 0). We note σ ε the truncated strategy profile. We have that

E σε (Φ j | in) ≤ E σ (Φ j | in) ≤ E σε (Φ j | in) + 8ε We can now show by induction on i ≤ N ε that E σε (φ | in) is (1, u ε 1 , u ε 2 ) with |5 -u ε 1 | ≤ 8ε and |3 -u ε 2 | ≤ 8ε, which entails that E σ (φ | in) = (1, 5, 3
). We can then show inductively that for every i,

E σ (φ | in • T * • q0 • T * • δ0 • T * • • • δi-1 • T * • qi ) = 1, 4 + 1 2 c i 1 3 c i 2 , 4 - 1 2 c i 1 3 c i 2
Assume that σ is not a Nash equilibrium, and pick σ j a deviation of Player j (with j ∈ {2, 3}) which improves her payoff. By Proposition 3.16 we can assume that σ j is deterministic. Under σ = σ[j/σ j ], we can first notice that the same gadgets are visited as under σ, since Player j cannot improve her payoff by switching the choice of the transitions (gadgets starting from q for any q ∈ Q). We also realize that in all states of the game, under σ, the choice of Player j is either deterministic (play c) or she plays matching-pennies games uniformly at random against Player 5j. Switching the choice of Player j in a matching-penny game does not change the probabilities of the two output-edges. So only a switch from action c to action s can possibly improve the payoff of Player j. This is not the case, since by construction, we have a local Nash equilibrium in every gadget. Hence, no deterministic deviation of Player j can improve her payoff.

Conclusions

We summarize our construction as follows: for any 2-counter machine M, we built a game G M which has the following properties:

• Deterministic with three players;

• Reward functions are terminal, with non-negative integers;

• Rewards of agents 2 and 3 always sum up to 8;

• Rewards of agent 1 are either 0 or 1, and agent 1 is always winning, its only purpose is to balance the game for the two other players;

• M has an infinite run if, and only if, G M has a Nash equilibrium which is 1-maximal.

The immediate consequence of our reduction is the following undecidability theorem:

Theorem 4.19. The constrained existence problem with non-negative rewards (cNE) and three players is undecidable.

We consider in the sequel several extensions of this result. We first state two straightforward corollaries in this section, and develop more involved extensions to terminal-reachability and safety games in the next sections.

First of all, we realize that in the reduction, there is a 1-maximal Nash equilibrium from in if, and only if, there is a Nash equilibrium with social welfare larger than or equal to 9, where the social welfare is defined as the sum of the expected payoffs of all players. As an immediate corollary, we get: Corollary 4.20. We cannot decide whether there exists a Nash equilibrium with some lower bound on the social welfare (or with optimal social welfare) in three-player terminal-reachability games with non-negative payoffs.

Unconstrained problem

We discuss in this section how to get rid of the 1-maximality condition. The main idea is similar to [START_REF] Brenguier | Nash Equilibria in Concurrent Games -Application to Timed Games[END_REF] where the initial state of the game is replaced by a branching to a submodule which is a game known to not have a Nash equilibrium in the considered model.

Here, the considered module is the game H of Section 3.6.4, which is known to not have a Nash equilibrium even in mixed strategies with arbitrary memory. Note however this game has negative terminal reward values, so the reduction is done at the expense of introducing negative terminal rewards. Lemma 4.21. Let G be a terminal-reachability game. We can build a terminal-reachability game G such that G has a 1-maximal Nash equilibrium if, and only if, G has a Nash equilibrium.

Proof. Without loss of generality, we can rescale payoffs in G and assume the maximal reward of player 1 is 1. The game G is depicted on Figure 4.10, where player 1 can decide in s 0 whether to go to H or to G. Assume there is a Nash equilibrium in G . Since H has no Nash equilibrium, in any Nash equilibrium of G from s 0 , player 1 will play action continue (with probability 1) in s 0 . This entails that G has a Nash equilibrium (since the payoffs are prefix-independent). Moreover, the payoff of player 1 in this Nash equilibrium must be 1, as otherwise player 1 could secure a better payoff by going to H (see proof of Lemma 3.22). Conversely, if there is a 1-maximal Nash equilibrium in G, then it gives rise to a Nash equilibrium in G by letting player 1 move to G in s 0 . This is easily seen to be a Nash equilibrium, in particular because deviating to H in s 0 cannot benefit to player 1. Corollary 4.22. The existence of a Nash equilibrium in a deterministic game with only three players and (relative) integral terminal rewards is undecidable. In particular eNE is undecidable.

s 0 G H continue s 0 stop
One drawback of the above proof is the use of the game H which has negative rewards so removing a constraint has been done at the expense of introducing negative rewards. In fact, game H can be replaced by any game with no Nash equilibria, such that player 1 can still secure a payoff 1ε for every ε > 0. For instance, one could use a game with limit-average payoff and nonnegative rewards only as shown in [START_REF] Ummels | The complexity of Nash equilibria in limit-average games[END_REF], hence giving undecidability results for the limit-average payoff case. However, it is not known if there exists a concurrent deterministic game with at most three players and non-negative rewards which has no Nash equilibrium. Existence of such game would imply undecidability of the existence of a Nash equilibrium with non-negative terminal rewards.

Qualitative objectives

In order to strengthen our result, we are interested in qualitative objectives, that is to say objective functions ranging over {0, 1}. More precisely, we study here how we can transform our previous reductions in order to use only qualitative terminal reachability and safety objectives. The table on the right gives the value of E y 8 for some values of y (notice that E y 8 ⊆ E y 8 when y ≤ y , so that for instance E 4 8 = 32).

Terminal-reachability games

We now explain how to extend our main theorem to games with terminal-reachability objectives (in other terms, with terminal payoffs in {0, 1}). The crucial point to achieve this is that in all our terminal states of G M , the sum of the rewards of players 2 and 3 is 8. Our construction amounts to replacing these terminal rewards with a simple module in which the payoffs of players 2 and 3 are (8, 0) and (0, 8).

Proposition 4.23. Let G = (A, s 0 , (Φ i ) i ) be a 3-player terminal-reachability game such that in any final state s, the terminal payoff Φ(s) = (x, y, z) satisfies the following conditions:

x ∈ {0, 1} y, z ∈ N y + z = 8
Then we can construct an arena A , and a qualitative terminal-reachability reward φ i for each player i, such that (σ, s 0 ) is a 1-maximal Nash equilibrium in G if, and only if, it is a 1-maximal Nash equilibrium in G = (A , s 0 , (φ i ) i ).

Proof. We replace every final node (x, y, 8y) with a constant-sum game as depicted in Figure 4.11. In this figure, for all i ∈ {1, 2}, the set Allow i (v x,y ) of allowed actions is the set {c i | i ∈ 0, 7 }, and for any k, y ∈ N, E y k = {c i • c j | ∃0 ≤ r < y. ij = r mod k}. Notice this definition generalizes the k-action construction of Section 4.1.2, where E k = E 0 k . By playing uniformly at random, player 2 can ensure winning (i.e. reaching state (x, 8, 0)) with probability y/8, whatever player 3 does. She then gets payoff y. Similarly, player 3 can ensure winning with probability 1y/8 by playing uniformly. We conclude that (x, y, 8y) is the only equilibrium payoff.

We built this way a new game G where all final payoffs are of the form (x, 8, 0) or (x, 0, 8). Every Nash equilibrium in G can be converted into a Nash equilibrium with the same payoff (by playing uniformly in every new node v x,y ) in G .

Conversely, if (σ, s 0 ) is a Nash equilibrium in G , then for every hv x,y ∈ States + ,

• If hv x,y is enabled by σ, we have E σ (φ | hv x,y ) = (x, y, 8y);

• Otherwise, hv x,y is not enabled and we can assume σ i (hv x,y ) is the uniform distribution for both i ∈ {1, 2}. This assumption does not change the final reward of the game (as hv x,y is not enabled) and preserves the equilibrium because a deviation of player 2 in this branch can already ensure at least payoff y (respectively at least 8-y for player 3).

Finally, every branch ending up in v x,y has payoff (x, y, 8-y) so (σ, s 0 ) is in fact an equilibrium in G with the same value.

To conclude, we can divide every terminal reward for players 2 and 3 by 8, so that every final state satisfies φ (s) ∈ {0, 1} Agt . By linearity, every 1-maximal Nash equilibrium in the original game is a Nash equilibrium in G with average payoffs for players 2 and 3 divided by 8.

We conclude with the following corollary: Corollary 4.24. The existence of a 1-maximal Nash equilibrium in a 3-player game with qualitative terminal rewards is undecidable.

Safety games

We explain now how the previous proof can also be transformed to prove the undecidability of the constrained existence of a Nash equilibrium in a safety game.

This result is interesting since [START_REF] Secchi | Stay-in-a-set games[END_REF] shows that there always exists a Nash equilibrium in safety games (called stay-in-a-set games), without constraints. We must remark that the existence result is established for action-visible games only. Nevertheless, the authors notice that the constructed strategies require relatively low memory, as players only have to remember the set of players that already lost their objective. This information does not depend on the played actions, and when encoded inside the current state, the authors remark that memoryless Nash equilibria are proven to exists. Therefore we conclude that the actionvisible hypothesis can in fact be omitted.

The following result comes in contrast:

Corollary 4.25. The existence of a Nash equilibrium in a 3-player game, with qualitative safety objectives, where player 1 loses almost surely, is undecidable.

By analogy, a strategy profile losing for player 1 will be called 1-minimal.

Proof. Proposition 4.23 along with the reduction presented in Proposition 4.18 allows us to compute for every two-counter machine M a concurrent qualitative terminal-reachability game G such that M does not halt if, and only if, G has a 1-maximal Nash equilibrium.

For each player i, we can write her payoff function as a function

φ r i = 1 States * •R i with R i ⊆ F.
We now define safety conditions for this arena by:

G 1 = States\R 1 ∀i ∈ {2, 3}. G i = States\F R i
We define reward function φ s i , for each player i, by

φ s i = 1 G ω i G * i .
First remark that we defined all internal states as winning for all players so an infinite run is a possible Nash equilibrium for the safety game. Let us now consider the constraint E σ (φ s 1 | s 0 ) = 0. In the following, we will say that a σ, s 0 is an 1-minimal Nash equilibrium if it is a Nash equilibrium of the safety game which satisfies the above constraint.

Let us notice that φ s 1 = 1-φ r 1 and for i ∈ {2, 3},

φ s i = 1 States * R i +1 (States\F) ω
The following analysis is mostly concerned with the term 1 (States\F) ω that is the difference between terminalreachability objectives and safety objectives. Based on the reduction of Proposition 4.18, we will show that this term can be neglected, in the original profile σ, but also when considering any deviation.

• Assume σ is a 1-minimal Nash equilibrium. We have E σ (φ r 1 | s 0 ) = 1 so σ is 1-maximal for the reachability objective.

Moreover, R 1 ⊆ F is reached with probability 1 so E σ (1 (States\F) ω | s 0 ) = 0. So ∀i ∈ {2, 3}, E σ (φ s i | s 0 ) = E σ (φ r i | s 0 ). For i and any deviation σ i ∈ S i , let σ = σ[i/σ i ], then E σ (φ s i | s 0 ) = E σ (1 States * R i | s 0 ) + E σ (1 (States\F) ω | s 0 ) ≥ E σ (φ r i | s 0 ). So E σ (φ r i | s 0 ) ≤ E σ (φ s i | s 0 ) ≤ E σ (φ s i | s 0 ) = E σ (φ r i | s 0 ).
We conclude that σ is a 1-maximal Nash equilibrium for the reachability objectives.

• Conversely, assume there exists a 1-maximal Nash equilibrium. Thanks to the reduction proof, we know this corresponds to an infinite run of M. Without loss of generality, we can assume such run has infinitely many counter tests, so that the underlying Nash equilibrium enables the testing module G t infinitely often. This strategy profile σ makes both players 2 and 3 play uniformly at random so even if one decides to deviate, there is still a fixed positive probability 1 4 to branch to submodules n 2 and eventually reach a final state. We conclude from this analysis, that for every deviation

σ i ∈ S i (i ∈ {2, 3}), E σ[i/σ i ] (1 (States\F) ω | s 0 ) = 0.
Hence σ is resilient to deviations of 2 and 3 for safe objectives.

It remains to check that player 1 has no incentive to deviate, so we have to carefully look at the states where several allowed actions are given to her. Such constructions happen only in modules R k and D, where player 1 can play actions a or b. However, since σ encodes a correct simulation of M, we know by construction that players 2 and 3 always play action c concurrently. Hence, a deviation for player 1 is not profitable, since it always yield the same payoff 1, for the reachability objective φ r 1 , or 0 for the safety objective φ s 1 . We conclude that the strategy profile σ is also a 1-minimal Nash equilibrium.

Unconstrained problem revisited

The previous game H used to get rid of 1-maximality condition in Corollary. 4.22 can be seen as a zero-sum 2-player game, where player 1 has a terminal reachability objective and second player has the complementary safety objective.

We introduce a family of decision problems, where we allow a given number of terminal reachability and several safety objectives: Definition 4.26. For w ∈ {R, S} * , we define the Nash equilibrium decision problem for w-qualitative objectives by: qNE(w) INPUT:

A |w|-player game G such that for all i ∈ [1, |w|],

• If w[i] = R, player i has a qualitative terminal reachability objective: there exists QUESTION: Whether there exists a Nash equilibrium for G.

G i ⊆ F such that Φ i (r) = 1 r∈States * G i . • If w[i] = S,
In the previous proof of Corollary. 4.25, we replaced reachability objectives of all players by safety objectives. We argue that this replacement can be done one by one for each player, with the same proof. We conclude that we can build games G M for any M, that have reachability objectives for player 1 (resp. 1 and 2) and safety objectives for players 2 and 3 (resp. 3), such that G M admits a 1-maximal Nash equilibrium if, and only if, M does not halt.

Mixing at least one reachability player with a safety player allows us to apply the same reduction as in Corollary. 4.22, to derive the following result: Corollary 4.27. Both problems qNE(RS 2 ) and qNE(R 2 S) are undecidable.

As a comparison, two particular classes of decision problems are trivial, since Nash equilibria always exist:

• Games with at most one reachability/safety player can be seen as Markov decision processes. Existence of an optimal policy is ensured, which corresponds to a Nash equilibrium.

• When only safety objectives are at stake, we can rephrase the main existence results of [START_REF] Secchi | Stay-in-a-set games[END_REF] on stay-in-a-set games.

Theorem 4.28. For any n ≤ 1, the decision problem qNE(R n ) is trivial (always true). For any n ∈ N, the decision problem qNE(S n ) is trivial (always true) hence decidable.

A summary of these qualitative results is depicted in Table 4.

Notice two cases remain open:

• qNE(RS) contains problems such as whether the value of a zero-sum reachability game has 0-optimal strategy. As H does not satisfy this property, we can already say that the decision class is not trivial. Moreover, the class contains instances of games that are not zero-sum.

• For n ≥ 2, qNE(R n ), no examples of instances (games) without Nash equilibria are known.

Summary

In this chapter we have shown the undecidability of the existence of a constrained Nash equilibrium in a three-player concurrent game with terminal-reachability or safety objectives.

Several similar results were shown in the literature, for example by [START_REF] Ummels | The complexity of Nash equilibria in limit-average games[END_REF], where the reduction is made for 14 players in a turn-based setting with arbitrary (relative integers) terminal rewards. Variants of the reduction are reducing the problem to the existence of a finite (arbitrary) memory Nash equilibrium. On the other side, our proof exploits intensively the concurrent aspects of the framework, as long as the action-invisibility property, to lower the undecidability bound to three players and qualitative objectives.

This lets open the two-player positive rewards case, where little is known. In fact, even the existence of Nash equilibria in such games is an open problem: it was believed until recently that there are two-player games with nonnegative terminal rewards having no Nash equilibrium [CJM04, UW11a], but the proposed example was actually wrong (as explained in Section 3.6.4). On the one hand, if one can find such a game with no Nash equilibrium, then Corollary. 4.22 extends to nonnegative terminal-reachability games, and possibly to qualitative terminal-reachability games.

On the other hand, the existence of exact Nash equilibria remains open, as classical proof techniques based on fixed point theorem fail to apply. As a matter of fact, terminal reachability cannot preserve continuity of payoff functions when cyclic behaviours occur. A classical example of such discontinuity is exhibited in game H, with strategy patterns consisting in waiting more and more in the internal state. Such phenomena are usually tackled by introducing relaxed equilibrium notions, as discussed in the next chapter.

Chapter 5

Games that almost-surely terminate

In the previous chapter, we have seen that deciding the existence of an arbitrary Nash equilibrium is undecidable in the general case. Moreover, there exist games without Nash equilibria, as soon as we consider three players and at least one reachability and one safety qualitative objective.

In the case of non-negative reachability objectives, the decidability status is still open, however undecidability can still be derived when looking at computability questions, since the constrained problem remains undecidable.

Among the causes of undecidability, we may invoke:

• The arbitrary precision for the strategies and payoffs.

• The arbitrary memory for the strategies.

In this chapter, we focus on a relaxed notion of Nash equilibrium, with limited precision. Such a relaxation ensures existence, even under memoryless strategies. As a consequence, we are able to develop efficient computation techniques, and bring back decidability for this model. We are mainly focusing on games with terminal-reachability objectives, with possibly negative rewards, although the presented results may be generalized to other objectives.

The chapter is divided into three sections: first of all we study a restriction of the strategy space to particular memoryless strategies, that ensures the existence of a "stable" point in the restricted space. Then, we characterize properties of such profiles, against arbitrary strategies, and interpret it as a particular notion of equilibrium. Finally, we explore computational questions on the newly introduced equilibrium notion.

Avoiding cycling behaviours

The classical approach for the proof of equilibrium profiles consists in defining a best-response function, that maps each strategy profile σ, to a set of memoryless strategy profiles where each player has an optimal value against the rest of the players in σ. As we can characterize the equilibrium profiles as fixed points of this best-response, it is sufficient to prove the existence of such fixed point, thanks to some regularity properties of the function. Usually, such regularity properties are derived from structural properties of the game, that ensures the run to eventually terminate. We can cite for example the following cases: Termination is ensured after one round. The existence of Nash equilibria in surely (within a fixed number of steps) eventually terminating games can be proven the same way, or by induction.

• For qualitative safety objectives, [START_REF] Secchi | Stay-in-a-set games[END_REF] showed that exact Nash equilibria exist, with linear memory. The key argument consists in bounding from below the probability for any player to lose, whenever players cannot agree on a run that stays safe for all of them.

• In [START_REF] Chatterjee | On Nash equilibria in stochastic games[END_REF], a discounted version of terminal reachability objectives is considered. The discounted factor λ can be interpreted as a probability to finish the game, with probability λ and reward 0, at each step. In this context, the authors showed the existence of a relaxed notion of Nash equilibrium in terminal-reachability games, namely ε-Nash equilibrium, where deviations may only improve the payoffs by at most ε > 0.

In general, the best-response function is not continuous when considering terminal reachability objectives (see later Remark 5.11 on page 63), as the probability to not terminate the game can be arbitrarily close to 0, as seen on the hide-or-run game of Figure 1.2. As opposed to [START_REF] Chatterjee | On Nash equilibria in stochastic games[END_REF], where termination is forced from the game structure, we explore here a dual approach, by considering a restriction on strategies, that enforces players to leave cycles of the game, hence terminating.

Example 5.1. Consider Figure 5.1 which displays an example of a turn-based game, where state i ∈ {1, 2} is controlled by player i. The number labelling each node corresponds to the player controlling the state. She can decide whether to stop (action s) or to continue (action c) playing the game. However, each player has an incentive to wait for the other player to terminate the game. Again, the payoff is (0, 0) if the play does not reach a terminal state. This game has pure Nash equilibria: for instance, the memoryless strategy profile where player 1 plays c and player 2 plays s is an equilibrium, with payoff (1, 1/3). Another solution concept would allow a trade-off between players who will commit a fixed probability each to exit the game (for example ε > 0). In general, such a trade-off is not a Nash equilibrium as the other player can change her mind (and continue to play c to yield payoff 1).

We fix for the rest of this section a stochastic concurrent game with terminal-reachability payoffs G = (A, s 0 , (Φ i ) iAgt ), with A = (States, Agt, Act, (Allow i ) i∈Agt , Tab)

Non-cycling games

Definition 5.2. A state s of A is said cycling if there exists a mixed strategy profile σ ∈ S such that no single player can enforce (by deviating) reaching a final state, that is:

∀i ∈ Agt ∀σ i ∈ S i , P σ[i/σ i ] (States * F | s) = 0.
The arena A (and by extension, the game G) is said cycle-free if it contains no cycling state.

Computing the set of cycling states can be done in polynomial time, for example by computing the set of states that are almost-surely winning for the first player, in the following turn-based zero-sum qualitative safety game where:

• Player 1 suggests an action profile σ(s) from any state s;

• Player 2 agrees or starts a deviation σ i (s) for an agent i that is now fixed;

• The game continues to the next state determined by both action propositions.

• Player 2 has a qualitative reachability objective equal to F.

Moreover, this game is determined by pure memoryless strategies, so we can assume in the previous definition that σ, σ ∈ M .

We further notice that from any cycling state, there is a Nash equilibrium with payoff zero for all the players (playing profile σ from the definition). They are therefore somehow pathological behaviours, that we may want to remove. This is formalized as follows:

Proposition 5.3. Assume G contains a final state with payoff profile 0 Agt . One can construct a new transition function Tab such that the resulting game G:

• has the same set of states, final states, rewards and allowed actions,

• is cycle-free,

• and has "fewer equilibria" than G, from any state s: for any σ ∈ S, there exists σ ∈ S such that:

E σ G (φ | s) = E σ G (φ | s) and ∀i ∈ Agt ∀σ i ∈ S i . E σ[i/σ i ] G (φ | s) = E σ[i/σ i ] G (φ | s)
Notice that since only the transition function is modified, strategies in G and G coincide, which allows us to consider σ[i/σ i ] as a strategy profile of G. Moreover, this proposition implies that any Nash equilibrium of G can be converted into an equilibrium of G with the same payoff. The theorem is stated in a more general context in the hope to be applied to later notions of equilibria. In particular, keeping the same strategy space allows us to keep more easily further relations between a strategy profile and its allowed deviations.

Proof. Let f ∈ F such that φ(f ) = 0 Agt and C = {s ∈ States | s cycling in A}.
We construct Tab from Tab by immediately ending the game to state f from any cycling state. More precisely, for every A ∈ Act Agt , and s / ∈ F: Tab(s, A) = f if s ∈ C; Tab(s, A) = Tab(s, A) otherwise We can easily see that the underlying defined arena has the same set of final states, and is cycle-free.

Let σ C ∈ M be a (partial) stationary strategy profile in G which allows to stay within C (and hence prohibits reaching a final state from every s ∈ C, even under single-player deviations). This is possible since C is the set of cycling states of A.

We prove that

∀σ ∈ S. E σ(φ | h) = E σ (φ | h)

whenever

• last(h) ∈ F, since we are considering the same reward function, or

• last(h) ∈ C, since both sides are equal to 0 Agt .

Let s ∈ States a fixed state. Moreover, we assume that s ∈ Int (otherwise the result is immediate). We decompose all maximal runs starting from s into the following disjoint union:

s • (Int ω Int * • F) = s • (Int\C) ω h∈s•(Int\C) * •F h h∈s•(Int\C) * •C h • (Int ω Int * • F)
Since Tab and Tab functions coincide on Int\C, we can show by induction that for any n ∈ N,

• If A n = s • (Int\C) n • (Int ω Int * • F ), then P σ G (A n ) = P σ G (A n ). (A n ) n is a decreasing sequence whose limit equals s • (Int\C) ω , hence P σ G (s • (Int\C) ω ) = P σ G (s • (Int\C) ω ); • For any h ∈ s • (Int\C) n • F, P σ G (h) = P σ G (h); • For any h ∈ s • (Int\C) n C, let A = h • (Int ω Int * • F ), then P σ G (A) = P σ G (A) (or P σ G (h) = P σ G (h) when using usual shorthand notations).
We conclude by applying the total probability formula:

E σ G (φ | s) = h∈s•(Int\C) * •F E σ G (φ | h) E σ G (φ | h) • P σ(h) P σ (h) + h∈s•(Int\C) * •C E σ G (φ | h) E σ G (φ | h) • P σ(h) P σ (h) = E σ G (φ | s)
Same reasoning applies to σ[i/σ i ] for any i ∈ Agt and any deviation σ i ∈ S i .

Thanks to this proposition, we restrict our analysis to the case of cycle-free games. In this context, we aim at developping an equilibrium concept that is ensured for cycle-free games, and that can be later extended to arbitrary games.

Strong components

The main argument of our existence theorem relies on the structure of the strategy profiles, which can be forced to terminate the game, even in the presence of deviations. We define in this section a set of constraints we impose on our strategies. These constraints should be tight enough for the game to terminate, thus implying the existence theorem of a stable profile, but should also be loose enough to allow "interesting" deviations. Notice that for defining the stabilization property, one could equivalently require the probability to be equal to 1, as C would be a recurring set under P σ . Definition 5.5. Let C ∈ SC be a strong component, and s ∈ C. An action a ∈ Act is an exiting action from C for a state s and player i if there exists σ ∈ S that stabilizes C such that:

P σ[i/(s →a)] (s • (States \ C)) > 0. We set Exit(C) = {(a, i, s) | a is an exiting action from C for a state s and player i}.

As an illustration, let us consider Figure 5.2, whose game has two strong components:

Exit({s 1 , s 2 }) = {(b, 1, s 1 ), (b, 2, s 2 )} Exit({t 1 , t 2 }) = {a, b} × {1, 2} × {t 1 , t 2 }

Fixed point analysis

We now restrict the set of strategy profiles in which we search for equilibria. Under this restriction, we will show that each play almost-surely reaches a final state, which will provide the main argument for the existence of a "stable strategy profile", that is yet to be defined.

For the rest of this section, we assume that A is cycle-free, hence for any C ∈ SC, Exit(C) = ∅. Definition 5.6. Let ε > 0. For every strong component C ∈ SC, we define the set of (ε, C)-exiting stationary strategy profiles as follows:

∆ ε (C) = {σ ∈ M | ∀(a, i, s) ∈ Exit(C) σ i (a | s) ≥ ε} We also let ∆ ε = C∈SC ∆ ε (C).
Note that, to be properly defined (non-empty), ∆ ε requires the assumption that the game arena is cycle-free: Lemma 5.7. For all ε ≤ 1

[Act] and A cycle-free, it holds ∆ ε = ∅. Proof. Consider the stationary strategy profile σ u which makes each player play uniformly at random over the set of allowed actions, at each state.

For any C ∈ SC, since Exit(C) is non-empty, this strategy profile is in ∆ ε (C). Hence

σ u ∈ ∆ ε .
Under a strategy profile of ∆ ε , almost-sure reachability of a final state is ensured, but even more precisely, the probability to reach a final state, after a finite number of steps, is bounded from below by a positive constant: Proposition 5.8. Assume ε > 0. Then there exist 0 < p < 1 and k ∈ N such that for every σ ∈ ∆ ε , for every s ∈ States, for every n ≥ 0,

P σ (States k•n • F ω | s) ≥ 1 -p n .
We remark that under different hypothesis, the conclusion of this proposition is similar to the one of [SS01, lemma 2.1, p483] for stay-in-a-set games. The proof is a direct consequence of the following lemma: Lemma 5.9. Let ε > 0 and X ⊆ States\F. For every s ∈ X, there exists p s X > 0 such that for every σ

∈ ∆ ε , P σ (X ≤ [X] • X | s) ≥ p s X .
Proof. Notice first that for any σ ∈ M, the absence of memory ensures that

P σ (X ≤ [X] • X | s) > 0 if, and only if, P σ (X * • X | s) > 0.
Assume that this were not the case: that is to say, for any k, there exists σ k ∈ ∆ ε such that

P σ k (X ≤ [X] • X | s) < 1 k .
One can assume by extraction that σ k converges to some σ ∈ ∆ ε (∆ ε is compact). By continuity, we get

P σ (X ≤ [X] • X | s) = 0 Hence, P σ (X * • X | s) = 0.
Let s 0 ∈ X be such that P σ ((X * s 0 ) ω | s) > 0, that is to say one of the states that can appear infinitely often with positive probability under σ from s (it is in X due to P σ (X * • X | s) = 0). We consider C ⊆ X minimal subset containing s 0 and such that if s 1 ∈ C and P σ (s 1 s 2 | s 1 ) > 0 then s 2 ∈ C. One can verify that such a set is a strong component stabilized by σ. However, σ ∈ ∆ ε hence a contradiction (by definition of ∆ ε ).

Our proof will rely on the following well-known fixed-point theorem, that we will apply to a well-adapted set of strategy profiles.

Theorem 5.10 ([Kak41]

). Let X be a non-empty, compact and convex subset of some Euclidean space. Let f : X → 2 X be a set-valued function on X with a closed graph and the property that f (x) is non-empty and convex for all x ∈ X. Then f has a fixed point.

A Nash equilibrium σ can be characterized as a strategy profile such that for each player i, σ i lies in the set of the best-response strategies against the other players, denoted by BR i (σ). By extension, the best-response function, which returns the profile of sets of best-responses for a given strategy profile, is written BR and its possible fixed points correspond to the Nash equilibria of the game ([Nas50]).

Remark 5.11. Nevertheless, for games over graphs, continuity of this best-response function is not ensured. More precisely, the graph of the function is not closed. Let us consider for example the game of Figure 5.1, and write any stationary strategy profile σ in this game as the tuple (σ 1 (s | 1), σ 2 (s | 2)). Then, if one player decides to stop the game with any positive probability, the other player has all incentive to deterministically continue the game, until reaching the terminal state (almost-surely), hence: BR((x, y)) = {(0, 0)} for every x, y > 0, where BR denotes the best-response function. In particular, let (x n ) n (y n ) n two sequences of positive probability numbers, both converging to 0, and let σ n ∈ BR((x n , y n )) for any n. Then σ n equals (and converges to) (0, 0). However, if one player decides to continue the game with probability 1, the only way to win some positive payoff 1/3 is to play the stopping action with positive probability, hence: BR((0, 0)) = {(x, y) | x, y > 0}. We conclude that the graph is not closed, so Theorem 5.10 cannot apply to the best-response function in this strategy profile space. This is not surprising as we know that Nash equilibria need not always exist (recall the example given in Figure 1.2).

In the following we will see that the (standard) best-response function will fit well in our setting, with strategy restrictions: Definition 5.12. We consider T ⊆ M a subset of stationary strategy profiles. Let BR T :

T → 2 T with BR T (σ) = σ ∈ T ∀i ∈ Agt. ∀s ∈ States. σ i ∈ argmax σ i s.t. σ [i/σ i ]∈T E σ[i/σ i ] (Φ i | s)
Note that BR M is the usual notion of best response function. The fundamental application of Theorem 5.10 combined with our study of almost-sure termination is given below: Lemma 5.13. Assume that A is cycle-free and 0 < ε ≤ 1

[Act] . Then BR ∆ε has a fixed point. Proof. We show that Theorem 5.10 applies:

• First notice that T = ∆ ε can be viewed as a non-empty compact convex subset of R N where N = Act × Agt × States. Moreover, T can be decomposed in a product of individual strategy sets for each player

T = T 1 × . . . T [Agt] where ∀i ∈ Agt T i = {σ i | ∀(a, s) (a, i, s) ∈ Exit(C) ⇒ σ i (a | s) ≥ ε}
Hence, for every (σ, σ ) ∈ T 2 , and i ∈ Agt, we still have σ[i/σ i ] ∈ T .

• Let k and p be the constants appearing in the statement of Proposition 5.8. For every n ≥ 0, we define g n for the function assigning to every pair of strategy profiles (σ, σ ) ∈ T 2 the following vector in R Agt×States :

  k•n j=0 f ∈F P σ[i/σ i ] (States \ F) j • f | s • ν i (f )   i∈Agt,s∈States
Then, we obviously see that for every (i, s)

∈ Agt × States, lim n→∞ g n (σ, σ ) i,s = E σ[i/σ i ] (Φ i | s).
Furthermore, as an application of Proposition 5.8, we get:

|E σ[i/σ i ] (Φ i | s) -g n (σ, σ ) i | ≤ K • p n
where K = max i∈Agt,f ∈F |ν i (f )|. This implies that the above convergence is indeed uniform, and that g ∞ : (σ, σ

) → E σ[i/σ i ] (Φ i | s) i,s
is therefore continuous on T 2 .

• Let us now show that the graph of BR T is closed. In order to do so, we consider a converging sequence of strategy profiles (σ k ) k>0 with limit σ ∞ and for each k > 0, σ k ∈ BR T (σ k ) converging to σ ∞ . We will prove that σ ∞ ∈ BR T (σ ∞ ). For a fixed σ , we have

E σ k [i/σ i ] (Φ i | s) ≤ E σ k [i/σ k i ] (Φ i | s), hence by continuity, E σ ∞ [i/σ i ] (Φ i | s) ≤ E σ ∞ [i/σ ∞ i ] (Φ i | s).
• It remains to show that BR T (σ) is convex. We fix i ∈ Agt and show that (BR T (σ)) i is convex hence the result. Let 0 < λ < 1 and σ , σ ∈ BR T (σ): this means that both vectors

(E σ[i/σ i ] (Φ i | s)) s and (E σ[i/σ i ] (Φ i | s))
s are maximal, and equal to some vector m i . Indeed, if two different maximal vectors exist, we take the combined strategy that uses best action in each state, this new strategy is still in T i .

By convexity of

T = ∆ ε , σ λ = σ[i/λ • σ i + (1 -λ) • σ i ] ∈ T , so ∀s, P σ λ (States * F | s) = 1.
This implies that the payoff vector (E σ λ (Φ i | s)) s is the unique solution of the equation

             ∀f ∈ F E σ λ (Φ i | f ) = ν i (f ) ∀s / ∈ F E σ λ (Φ i | s) = s Tab(s, σ λ (s))(s )E σ λ (Φ i | s ) = s λTab(s, σ[i/σ i ](s)) + (1 -λ)Tab(s, σ[i/σ i ](s)) (s ) • E σ λ (Φ i | s )
On the other hand, m i satisfies the following equation:

   ∀f ∈ F m i,f = ν i (f ) ∀s / ∈ F m i,s = s Tab(s, σ[i/σ i ](s))(s )m i,s = s Tab(s, σ[i/σ i ](s))(s )m i,s
We can check that (E σ λ (Φ i | s)) s∈States = m i is a valid solution, hence the actual value, so σ λ i ∈ BR T (σ) i .

Equilibria under imprecise deviations

We have proven that the best-response function admits a fixed point, when restricting the strategy profiles to subset ∆ ε . This set is somehow close to the whole class of memoryless strategies, as ε goes to 0: intuitively, the "volume" of strategy spaces M and ∪ ε>0 ∆ ε are equal. However, such restriction does not allow us to prove the existence of a Nash equilibrium: for a given fixed point σ ∈ ∆ ε of BR ε , and a deviation σ i ∈ M i of player i, it is not guaranteed that σ[i/σ i ] ∈ ∆ ε . Nonetheless, there exists another deviation σ i ∈ M i such that σ[i/σ i ] ∈ ∆ ε , which is close to σ i . More precisely, σ i and σ i can differ from each other by at most ε for each involved probability value.

Thanks to this observation, we introduce the following equilibrium concept, where deviations are required to have a certain form of robustness: Definition 5.14. An equilibrium under ε-imprecise deviations from state s 0 is a strategy profile σ ∈ S such that:

∀i ∈ Agt. ∀σ i ∈ S i . ∃σ i ∈ S i s.t. E σ[i/σ i ] (Φ i | s 0 ) ≤ E σ (Φ i | s 0 ) and d(σ i , σ i ) ≤ ε
where d(σ i , σ i ) is the supremum distance between the two distributions:

d(σ i , σ i ) = sup h∈States + ,a∈Act |σ(a | h) -σ (a | h)|
The intuition behind that definition is that, to have an incentive to deviate, a player should ensure to improve her payoff, even if her deviation is perturbed by ε (this corresponds to some noise the other players can add, or to a lack of precision in playing distributions). Said differently, a deviation is only considered profitable when all the surrounding (up to a distance of ε) strategies are also profitable.

Restricting to memoryless deviations

The notion of equilibria under imprecise deviations has been introduced in a very general setting with arbitrarily strategies and deviations. Lemma 5.13 provides the existence of a memoryless strategy profile, that is resilient to memoryless deviations, even when perturbed by a distance ε, in a memoryless manner. In other words, previous fixed-point lemma cannot directly conclude on the existence of an equilibrium under imprecise deviations.

Intuitively, we can even wonder if we can, as in the case of a memoryless Nash Equilibria, only consider pure memoryless deviations, which are in finite number. This subsection is devoted to this question, through the proof of the following key lemma: Lemma 5.15. Let s 0 be a state of a stochastic concurrent game G with terminal-reachability payoffs. For any stationary strategy profile σ ∈ M, it holds: σ is an equilibrium under εimprecise deviations from s 0 if, and only if,

∀i ∈ Agt. ∀σ i ∈ M i . ∃σ i ∈ M i . d(σ i , σ i ) ≤ ε ∧ E σ[i/σ i ] (Φ i | s 0 ) ≤ E σ (Φ i | s 0 )
In other terms, it is sufficient to consider memoryless deviations when checking if a stationary strategy profile is an equilibrium under imprecise deviations.

We prove this lemma by considering an intermediate 2 + 1/2-player game to represent deviations of player i and her counter-deviations at distance ε.

Let G a game, σ a stationary strategy profile and i ∈ Agt a player. We write G σ -i for the 1 + 1/2-player game obtained from G by assigning to all players, but player i, her strategy in σ. Note that for any σ i ∈ S i , we have E

σ[i/σ i ] G (Φ i | s) = E σ i G σ -i (Φ i | s)
In the following, we are mainly interested in the possible ε-imprecise deviations of player i alone in this new game.

In order to make the reduction clear, we consider in the following the particular case of games where each player is allowed at most two actions. When exactly two distinct actions are allowed, they will be noted a and b. The general case will be discussed in Remark 5.20.

For a stationary profile σ, we consider the 1 + 1/2-player game G σ -i as defined above (with player i alone, all other strategies being fixed) and construct a 2 + 1/2-player turnbased game with an additional antagonistic player î, whose role is to "change" the strategy of player i by a distance at most ε. Formally, for any state s where player i has two allowed actions a and b (resulting in distributions δ(s, a) and δ(s, b), resp.), we modify the game as follows:

• from s, player i is given the opportunity to move to one of the following four states:

(s, [0, ε]), (s, [0, 2ε]), (s, [1 -2ε, 1]) and (s, [1ε, 1]). For a 1 + 1/2-player game G for i, we denote by G ε the previous transformation. Our aim is to provide a correspondence between (stochastic) moves of player i from s in G, and her move from the corresponding state ŝ in G ε . Our notion of correspondence is defined as follows:

Definition 5.16. Let σ i , σ i ∈ S two strategies for the 1 + 1/2-player game G (played by i) such that d(σ i , σ i ) ≤ ε, and σ a strategy profile in G ε . We say that (σ i , σ i ) corresponds to σ if the following holds for any history ĥ ending in state s of G ε : Tab(s, σ i (π States ( ĥ))) = Tab(s, σ( ĥ))

where π States (h) is the projection on the letters corresponding to the original states States.

We now make explicit the purpose of the construction by establishing a correspondence between strategies in the original game and strategies in our 2 + 1/2-player version.

Lemma 5.17. For any σ i strategy of G, there exists a strategy σi in G ε for player i, such that, for any strategy σ i of G with d(σ i , σ i ) ≤ ε, there exists σî such that (σ i , σ i ) corresponds to σ.

Moreover, any pure memoryless strategy profile of G ε corresponds to some pair of strategies (σ i , σ i ) in G where σ i is pure memoryless and σ i is stationary.

The constructed game is a turn-based stochastic game with a quantitative terminal reachability objective, which can be interpreted as a special case of limit-average objective. Hence, thanks to a result of [START_REF] Liggett | Short notes: Stochastic games with perfect information and time average payoff[END_REF], such a game is determined with pure memoryless optimal strategies for both players.

As a consequence of this construction, we can infer two possible characterizations of imprecise deviations in stationary profiles:

Corollary 5.18. The value of G ε at state s can be expressed as the following quantity on game G:

sup σ∈M G inf σ ∈M G d(σ,σ )≤ε E σ (Φ i | s)
Corollary 5.19. A stationary strategy profile σ ∈ M G in G is an equilibrium under εimprecise deviations from state s 0 if, and only if,

∀i ∈ Agt. ∀σ i ∈ M G i . ∃σ i ∈ M G i s.t. E σ[i/σ i ] (Φ i | s 0 ) ≤ E σ (Φ i | s 0 ) and d(σ i , σ i ) ≤ ε
Remark 5.20. One can notice the construction of the deviation game and inferred results have been applied to states with only two allowed actions. In fact, the same reasoning can be generalized to an arbitrary number of allowed actions at the expense of an exponential blowup: player i has to announce simultaneously, for each allowed action a, if its probability in the expected distribution will be larger than ε and/or smaller than 1ε. Note however that for a given fixed bound on the number of actions, the size of G ε is still polynomial.

Existence theorem

We finally conclude with the following existence theorem of an ε-imprecise:

Theorem 5.21. Let G be a stochastic concurrent game with terminal-reachability payoffs, and let s 0 be a state of G. For every ε > 0, there always exists an equilibrium under ε-imprecise deviations from state s 0 .

The proof consists in compiling together the several previous results:

Proof. We first consider a cycle-free game G obtained from G through Proposition 5.3. Let ε = min(1/ [Act] , ε). Let us consider a fixed-point σ ∈ S G of BR ∆ ε , which exists thanks to Lemma 5.13. Such a profile σ is an equilibrium under ε -imprecise deviations from every state s, thanks to Corollary. 5.19. Since ε ≤ ε, this is also an equilibrium under ε-imprecise deviations.

Finally, we apply last implication of Proposition 5.3 to σ to obtain an equilibrium under ε-imprecise deviations of G.

0, 0 1, 0 s aa, ab, ba bb σ 1 (a | s) = 1 σ 2 (b | s) = ε Figure 5
.5 -A game with an ε-Nash equilibrium which is not an equilibrium under ε -imprecise deviation, for any ε > 0.

Discussions

We discuss the newly introduced notion of equilibrium under ε-deviation and compare it to ε-Nash equilibrium as introduced in [START_REF] Chatterjee | On Nash equilibria in stochastic games[END_REF].

Both notions generalizes Nash equilibria, that is to say any Nash equilibrium is both an ε-Nash equilibrium and an equilibrium under ε-deviation.

First of all, consider the game of Figure 5.5, and the strategy profile (σ 1 , σ 2 ): the payoff is then (0, 0), and player 1 can improve her payoff by ε by playing action b from s. So (σ 1 , σ 2 ) is an ε-Nash equilibrium but not an equilibrium under ε-imprecise deviations: any strategy at distance ε from σ 1 strictly improves the payoff of player 1.

In the previous game of Figure 5.1, the strategy profile where each player plays s with probability ε yields payoffs 1 -2/(6 -3ε) for player 1 and 1 -(2 -2ε)/(6 -3ε) for player 2 from the initial state. It is an equilibrium under ε-imprecise deviations. The only way to really improve the payoff for a player is to play with higher probability action c. But with the lack of precision, she might lose some payoff anyway. The payoff values get arbitrarily close to 2/3 as ε goes to 0. Such an equilibrium is neither a Nash equilibrium, neither a ε-Nash equilibrium, since the pure deviation c allows an improvement of almost 1/3. This concludes that the two concepts are incomparable.

Remark 5.22. For any ε > 0 small enough, there exists σ ε ∈ M fixed point of BR ∆ε . As ε goes to 0, one can extract a converging sequence of (σ ε ) ε , whose limit can be denoted by σ 0 .

When G is a one-shot game, one can prove that σ 0 is still a fixed point of BR ∆ 0 = BR M , that is to say σ 0 is a Nash equilibrium. Such particular Nash equilibrium is called tremblinghand equilibria (in [START_REF] Selten | A reexamination of the perfectness concept for equilibrium points in extensive games[END_REF]) because of its extra properties. However, this converging procedure does not work anymore in the sequential games framework, as depicted again on Figure 5.1 and strategy profile where each player exits with probability ε. At the limit, strategy profile σ 0 , which stays in the game almost-surely, is not a Nash equilibrium.

An interesting open question consists in restricting again the strategy space M in order to force convergence to a trembling-hand Nash equilibrium, if any. As a matter of fact, such proof may require additional assumptions on G, as some games require finite memory for a Nash equilibrium to exist (see [START_REF] Ummels | Stochastic Multiplayer Games: Theory and Algorithms[END_REF]), so looking at an equilibrium in M may fail otherwise.

Computing stationary equilibria under imprecise deviations

We describe a polynomial-space algorithm for computing stationary equilibria under imprecise deviations for non-negative terminal reward games. A similar proof for Nash equilibria in turn-based stochastic games is given in [START_REF] Ummels | The complexity of Nash equilibria in stochastic multiplayer games[END_REF]. We briefly describe the latter proof, which will help understand our current encoding. The algorithm proceeds by encoding a Nash Equilibrium as an existential first-order formula over the reals, whose satisfiability can be decided in PSPACE. The formula quantifies over all stationary strategy profiles and payoffs at each state, and expresses that:

1. the strategy profile σ under consideration is properly defined; 2. the payoff in each state corresponds to the real payoff of the strategy profile;

3. for any i, player i cannot benefit from deviating in G σ -i .

These properties cannot, in general, be expressed locally, but in the setting of [START_REF] Ummels | The complexity of Nash equilibria in stochastic multiplayer games[END_REF], one can first, non-deterministically, guess the support of the strategy. On the one hand, this allows us to compute (in linear time) the set of states from which F is never reached. Those states have payoff 0 for all agents, and the payoff in the other states (from which F is reachable with some positive probability) can be expressed as a combination of the payoff values of the successor states and the (local) strategy profile. On the other hand, we can also compute (still in linear time) the set of states that are reachable from s 0 . It is easy to see that player i has an incentive to deviate if, and only if, her payoff can be increased by deviating locally from such a reachable state. Hence we can express stability of the Nash Equilibrium as a (polynomial size) conjunction of inequalities.

Another way of expressing this stability property is by saying that for any player i, s 0 should yield a payoff in the equilibrium that is larger than the optimal value v i (s 0 ) in the Markov decision process representing the possible deviations of player i, namely G σ -i . Since the initial guess can be done in NPSPACE and the generated formula is of polynomial size, the whole algorithm runs in PSPACE.

In the case of equilibria under ε-imprecise deviations, we apply a similar technique but deviations are now to be considered as strategies for player i in G σ -i ε against the worst strategies of player i. In fact, we want to check that s 0 has a payoff (in the equilibrium) larger for player i than the maximal value she could get by imprecisely deviating. Thanks to Corollary. 5.18, this optimal value is the same as in G i = G σ -i ε , denoted by v ε,i (s). In order to compute these values for each game G i , we non-deterministically compute optimal strategies for players i and î. These strategies can be supposed to be pure memoryless. In order to do so, we first guess a strategy for player i in the game game G σ -i ε . Without knowing the exact probability values of this game (which depends on σ), we can still derive its structure since the support is known, thus we can compute the set of states for which player î can totally spoil i's payoff, that is, enforce a non-terminating run; such a run has payoff 0, which is optimal for player î. We later guess a pure memoryless strategy for player î keeping in mind that î has to play such a cycling strategy from any state where she is able to. From the other states, for which player i can still ensure positive probability to terminate, the value of the game can again be expressed locally as a combination of the guessed strategy profile and the values of the successor states. As for the previous algorithm for Nash Equilibrium in G, the optimality of both strategies can be expressed as stability by local deviations. Finally, stability by imprecise deviations in G consists in coding the fact that payoff in G for player i should be larger than the optimal value v ε,i (s 0 ).

We now make precise the result and the algorithm.

Theorem 5.23. Let k > 0. Let G = (A, Φ ν ) be a stochastic concurrent game with nonnegative terminal rewards with [Act] ≤ k. Let s 0 ∈ States and ε > 0. For every i ∈ Agt, we fix x i , y i ∈ R + two real numbers. We can decide in PSPACE whether there is a stationary equilibrium under ε-imprecise deviations σ from s 0 , such that for every i ∈ Agt,

x i ≤ E σ (Φ i | s 0 ) ≤ y i .
Remark 5.24. The previous theorem can be applied to compute some equilibria in the case of negative payoffs by considering the new payoff function ν = νmin ν ≥ 0. However,

φ = Φ ν -

Interaction models

In the first part, we studied systems with a fixed finite number of agents, that have in each round arbitrary interactions. Such agents may have different roles, objectives and strategies. Moreover, as noticed before, introducing partial information may lead to undecidability, therefore our previous model allows all agents to have a complete view (apart from action visibility) of the current global state. We may argue that more realistic models should include more structural properties, namely:

• All agents, except possibly one, for the environment, a server or some other leader mechanism, should have a similar structure, entailing some symmetry for the global system.

• Interaction and communication between agents should have some particular structure.

• Agents should be aware of their own current state only, and only be able to deduce others' statuses thanks to interaction and communication.

• Even the number of initial agents should a priori be unknown, potentially large.

In this part, we are mainly interested in verification problems on such systems, although we keep in mind strategy synthesis as a further goal. Since interaction between agents is encoded with a particular structure, we usually refer to such systems as protocols (instead of games) composed of processes (instead of agents).

Parametrized verification: a global picture. When dealing with several identical processes, the naive approach based on building, by composition, the concrete system for a fixed number of processes then applying classical analysis techniques can be replaced by more clever techniques based on symmetry reduction, as shown in [START_REF] Veith | Symmetry and model checking[END_REF], which focuses on verification of symmetric systems.

However, as pointed out by the fourth requirement, the number of processes is not fixed a priori, so these techniques only provide partial answers for our systems, where the number of processes can be a parameter. Therefore, we switch to another setting where we want to solve verification for parametrized systems. In such a parametrized setting, the natural question is to characterize the set of parameter values for which the system is correct.

Although the problem may seem to be harder in general, it allows us to consider large parameter values for which we may reach limit behaviours. When such a limit is reached, answering verification and synthesis questions becomes easy as it suffices to compute the answers for a finite number of parameter values up to some limit parameter.

Moreover, not only the latter approach is more general, but it might also turn out to be easier and more efficient, since large parameters may involve regularity properties in our model. For example, adding more and more agents in a system may result in a kind of saturation mechanism ensuring that each action can always be performed by at least one process. Such concepts are usually related to upward closure, as described in Section 2.4 of Chapter 2.

More formally, we are interested in determining the existence of a cut-off for a property ϕ, i.e., an integer N such that a system composed with more than N processes, ensures property ϕ.

Network of protocols. The presence of a leader often makes computation harder as it can ensure some additional synchronization mechanisms, so models that avoid its election may allow easier computations. Moreover, when the election mechanism can be generalized, we can imagine each agent to get its own identity, which usually leads to undecidable problems. For example, we can think of a system that allows each agent to encode one cell of the tape of a Turing machine, and be able to communicate (addressing) to its next and previous cell tape.

Processes may also be restricted to communicate within some communication graph. When topology does not correspond to a complete graph, broadcast communication with neighbours does not allow every process to receive a message. Therefore, some topologies may allow processes to earn identities, hence implying undecidability. However, we can usually recover decidability by allowing the communication topology to be unreliable, or to change over time as in [START_REF] Bertrand | Playing with probabilities in reconfigurable broadcast networks[END_REF]. In this context, allowing reconfiguration of a message-passing communication network will usually be equivalent to considering shared register with non-atomic operations. This dual view of our protocols explain why such systems will be later abusively named networks.

Different means of communication. Literature on parametrized verification introduces several models that usually vary by the choice of interaction scheme between processes. In his nice survey on parametrized models [START_REF] Esparza | Keeping a crowd safe: On the complexity of parameterized verification (invited talk)[END_REF], Esparza shows that minor changes in the setting, such as the presence of a controller in the system, might drastically change the complexity of the verification problems. Nonetheless, he summarizes the communication classes in the following four groups with decreasing expressivity:

• Broadcast communication;

• Shared register with atomic operations (locking);

• Rendez-vous communication;

• Shared register with (only) non-atomic operations.

Intuitively, two main models exist either based on message passing, or on shared variable accesses. Differences between these classes occur when looking at their ability to ensure that an information has been transmitted (last class cannot ensure such property) and whether the system can elect a leader (only two first classes can do so), which means some agent is eventually differentiated from the others. Further analysis of the relative expressiveness of some of those models can be found in [START_REF] Aminof | On the expressive power of communication primitives in parameterised systems[END_REF], yielding several reductions of the verification problems for some of those classes of models.

A seminal paper on parametrized verification of such distributed systems is the work of German and Sistla [START_REF] German | Reasoning about systems with many processes[END_REF]. In this work, the authors consider networks of processes all following the same finite-state automaton; the communication between processes is performed thanks to rendez-vous communication. Various related settings have been proposed and studied since then, which mainly differ by the way the processes communicate. Among those, let us mention broadcast communication [EFM99, DSZ10], token-passing [CTTV04, AJKR14], shared register with ring topologies [START_REF] Aiswarya | An automata-theoretic approach to the verification of distributed algorithms[END_REF], or shared memory [START_REF] Esparza | Parameterized verification of asynchronous shared-memory systems[END_REF][START_REF] Durand-Gasselin | Model checking parameterized asynchronous shared-memory systems[END_REF].

Asynchronous shared-memory systems. We consider a communication model where the processes asynchronously access a shared register, and where read and write operations on this register are performed non-atomically, i.e. one transition can only perform a read or write operation at a time. A similar model has been proposed by Hague in [START_REF] Hague | Parameterised pushdown systems with non-atomic writes[END_REF], where the behaviour of processes is defined by a pushdown automaton. The complexity of some reachability and liveness problems for shared-memory models have then been established in [START_REF] Esparza | Parameterized verification of asynchronous shared-memory systems[END_REF] and [START_REF] Durand-Gasselin | Model checking parameterized asynchronous shared-memory systems[END_REF], respectively. These works also consider networks with a specific distinguished process, called the leader, which runs a different program, from the beginning, and address the problem whether, for some number of processes, the leader can satisfy a given reachability or liveness property. In the case where there is no leader, and where processes are finite-state machines, the parametrized control-state reachability problem (asking whether one of the processes can reach a given control state) can be solved in polynomial time, by adapting the approach of [START_REF] Delzanno | On the complexity of parameterized reachability in reconfigurable broadcast networks[END_REF] for lossy broadcast protocols.

Example: network of sensors. As discussed before, such model is the weakest on the scale of communication mechanisms. For example, lack of atomicity does not allow our processes to implement mutual exclusion. However, we argue that this model still contains interesting questions.

As an illustration, let us consider a system composed of several small devices, for example sensors in a room, that report their data to a central receiver, through a wireless protocol. The wireless technology allows several communication channels (11 for example for 2.4Ghz WiFi). Because of short distance considerations, we can assume that radio resources are totally shared among sensors. The situation is depicted as a small C multi-threaded program, presented in Figure 6.1. When a sensor starts emitting on a channel i, it sends a particular message saying that channel i is in use, then the data, then another message saying that channel i is now free. A sensor usually tries to avoid sending data on a channel that is already in use, however, it has very low computation capacity and between the instant where it checks that a channel is not in use, and the instant where it effectively decides to emit, some other process may decide to transmit too, which results in data collision.

Sensors are not capable of avoiding such collisions, but can always try to change their channel in order to use all the radio resources. In this example, we may consider the following properties:

• Coherence: at any time, for all i ∈ [0, 10], if t[i] = 1, there exists at least one thread-/process transmitting on channel i.

// An a r r a y o f c u r r e n t l y used c h a n n e l s int t [ 1 1 ] = { 0 } ; void t h r e a d ( ) { // S e l e c t e d c h a n n e l f o r c u r r e n t t h r e a d int i = 0 ;

while ( 1 ) { i f ( t [ i ] == 0 ) { t [ i ] = 1 ;
// A c t u a l t r a n s m i s s i o n t [ i ] = 0 ; } e l s e i f ( rand ( ) < P) { // Branching w i t h p r o b a b i l i t y P i = ( i +1) % 1 1 ; } } } Figure 6.1 -Simple specification of wireless sensors (seen as threads) communicating through eleven radio channels. P is a constant between 0 and 1 describing the probability to switch to next channel if current one is already used.

• Optimality: all channels are eventually used.

• Repeated optimality: all channels are used infinitely often.

The resulting system is of course stochastic as there is some constant probability to switch to next channel if the current one is in use. Moreover, the composition of the multiple sensors is also non-deterministic, and may induce some stochastic behaviours. More formally, we consider the transcription of the program into a finite automaton, as depicted in Figure 6.2, with input alphabet Σ = {R i (d) , W i (d) | i ∈ [0, 10], d ∈ {0, 1}}. A non-deterministic semantics is usually given, as in [START_REF] Esparza | Parameterized verification of asynchronous shared-memory systems[END_REF], on a language theoretical approach, by considering the shuffling of an arbitrary number of words accepted by the process automaton, then intersected with the language of words over Σ * that are consistent with the shared register (for example,

W i (0) • R i (1) is not consistent).
We will adopt in the next chapter a more operational approach suited to the probabilistic setting.

Non-determinacy, fairness and qualitative stochasticity In this second part of the thesis, we further insert fairness assumptions in the model of parametrized networks with asynchronous shared memory, and consider again terminal reachability problems in this setting. There are different ways to include fairness in parametrized models: One approach is to enforce fairness expressed as a temporal-logic properties on the executions (e.g., any action that is available infinitely often must be performed infinitely often); this is the option chosen for parametrized networks with rendez-vous [START_REF] German | Reasoning about systems with many processes[END_REF] and for systems with disjunctive guards (where processes can query the states of other processes) in [START_REF] Außerlechner | Tight cutoffs for guarded protocols with fairness[END_REF]. Another equivalent 0, 9 0 11 i, 9

i, 10 i, 11

. . .

R i ( 0 ) R i (1) R i-1 (1) R i (1) W i (1) W i ( 0 ) R 0 ( 0 ) . . . R 0 (1) R 0 (1) . . . R 10 (1)
. . .

W 0 ( 0 )
. . . view consists in equipping our networks with a stochastic scheduler that, at each step of the execution, assigns the same probability to the available actions of all the processes. From a high-level perspective, both forms of fairness are similar. However, notice that expressing fairness via temporal logic has to be done with care, as some formulae may allow very regular patterns (e.g., round-robin execution of the processes), whereas the stochastic approach always implies considering all possible interleaved runs with probability 1.

77

Chapter 7

Parametrized register protocols

In this section, we define different categories of parametrized register protocols and provide an operational semantics based on a transition system. This first semantics will allow us to consider the reachability/safety problem, as studied in [START_REF] Esparza | Parameterized verification of asynchronous shared-memory systems[END_REF] or [START_REF] Durand-Gasselin | Model checking parameterized asynchronous shared-memory systems[END_REF], namely reachability under a non-deterministic scheduler. We will then introduce a stochastic scheduler by introducing probabilities on distribution and introduce stochastic properties to be studied.

First of all, we define the operations that can be operated on the shared register. Here W means write in the register a new value, no matter the current content of the register, while R means read the content of the shared register. This last operation can only be triggered if the correct value is already stored in the register.

Intuitively, we will define finite state automata over the alphabet Op (D). A protocol will be given as two automata, the leader and the contributor, the last one being possibly duplicated several times: Definition 7.2. A register protocol is given by P = (Q l , Q c , D, d 0 , q l , q c , T l , T c ), where • Q l is a finite set of control locations for the leader;

• Q c is a finite set of control locations for the contributors, disjoint from Q l ;

• D is a finite alphabet of data for the shared register, with initial data d 0 ;

• q l ∈ Q l (resp q c ∈ Q) is an initial state for the leader (resp contributors);

• T l ⊆ Q l × Op (D) × Q l is
q l q c q 1 q 2 q f 0 R(0) W(1) R(1) W(2) R(2) W(2)
Figure 7.1 -Example of a leaderless non-atomic register protocol with D = {0, 1, 2} and initial value 0. q l will usually be ommited for leaderless register protocols.

• T c ⊆ Q c × Op (D) × Q c is a finite set of transitions of the contributors.
Without further explicit precision, we will consider in the rest of the chapter a fixed register protocol P = (Q l , Q c , D, d 0 , q l , q c , T l , T c ). Definition 7.3. We say that P is

• atomic if T c ⊆ Q c × Op R,W (D) * × Q c and T l ⊆ Q l × Op R,W (D) * × Q l ; • non-atomic if moreover T c ⊆ Q c × Op R,W (D) × Q c ; • leaderless if T l = ∅.
Notice first that the leaderless concept is only relevant when dealing with non-atomic protocols, as atomicity allows the election of a leader (start with extra data value "not elected" then first contributor to play jumps to q l by writing the actual initial value). Then, a nonatomic register protocol is a particular case of atomic protocol, which is also a particular case of register protocol.

Because of the monotonous properties we will describe later, we will be mainly interested in non-atomic protocols. Hardness results will be developed in the leaderless non-atomic case.

The number of involved data values in any atomic register protocol is bounded by [T ]. This allows us to legitimately define the size of any atomic register protocol P by:

|P| = [Q c ] + [Q l ] + [T c ] + [T l ]
A graphical representation of a leaderless non-atomic protocol is depicted in Figure 7.1.

Remark 7.4. Notice we do not require our protocol to be complete. We say a protocol is complete if for each state and each register value, there exists an operation defined for this value, namely ∀q ∈ Q c (q,f,q )∈Tc dom (f) = D and ∀q ∈ Q l (q,f,q )∈T l dom (f) = D.

Non-deterministic transition system

We now present the non-deterministic semantics of distributed systems associated with our register protocols. We consider the asynchronous composition of the automaton of the leader, with several copies of the contributor automaton, the number of copies is not fixed a priori and can be seen as a parameter.

We immediately consider the following abstraction: for each contributor state q, we only keep track of the number of processes in this particular state q. Namely, a configuration will store a multiset of Q c , alongside the current state of the leader and the current memory data. This abstraction will reveal to be conceptually sound, as the copies will be indistinguishable, and will allow us to later consider easier comparison relations over configurations. Definition 7.5 (Transition system associated with P). We define the transition system (Γ, →) in the following way:

• Γ = Q l × N Qc × D is the set of configurations; • (q, µ, d) → (q , µ , d ) if there exists f ∈ Op (D) with d ∈ dom (f), d = f(d) and
either (q, f, q ) ∈ T l , µ = µ ; or q = q and there exist q 1 ∈ µ and q 2 ∈ µ such that (q 1 , f, q 2 ) ∈ T c and µ ⊕{q 1 } = µ ⊕ {q 2 }.

We write respectively (q, µ, d) (q,f,q )

-→ (q , µ , d ) and (q, µ, d)

(q 1 ,f,q 2 )
-→ (q , µ , d ) in the first and second cases. This allows us to characterize relation → as the set-union of all individual transition relations:

[→] = t∈Tc∪T l [ t →]
Notice that some configurations may not have any successor through transition relation → since completeness is not required (Remark 7.4).

Furthermore, we introduce the following notations.

Definition 7.6. The size of a configuration (q, µ, d) = γ ∈ Γ, or number of processes, is denoted |γ| = |µ| = q µ(q ). We write Γ n for the set of configurations of size n, which is finite. Data and multiset of configuration γ will be denoted respectively by ν (γ) = d and st(γ) = µ. For q ∈ Q c , we write γ(q ) as a shorthand for st(γ)(q ). If µ ∈ N Qc is another multiset, we will write γ ⊕ µ as a shorthand for (q, µ ⊕ µ , d). Moreover, we identify q and the multiset composed of one copy of q allowing us to write configuration γ ⊕ q = (q, µ ⊕ q , d) = (q, µ ⊕ 1 {q } , d).

Remark 7.7. One first important remark is that the transition relation → never introduces or removes processes: the number of processes along a path is fixed from the beginning. Although the transition system is infinite, the number of configurations of a given size is bounded.

Another natural abstraction of our system consists in losing track of the exact number of processes in each state, and only keep information of the set of available states. By analogy with distribution and multisets, this abstraction will be called the support of a configuration. Definition 7.8. Let γ = (q, µ, d) ∈ Γ. We define the support of γ by γ = (q, {q | µ(q ) > 0} , d). We write q ∈ γ when µ(q ) > 0.

One main question that arises from the transition system definition is the choice, at each step, of the transition to be taken. Remark first, that because of our multiset construction, our choice is already limited to picking an available state from the support instead of choosing exactly which process will be involved. Once this state is chosen, it remains to determine which transition involving this state has to be taken. We argue in this chapter that both these decisions can be taken by a single scheduler, that we will assume first to be nondeterministic, then stochastic. Intuitively, we are interested in the reachability or safety from a given configuration. In the first case, this means checking whether one can manage to reach a particular configuration when all processes cooperate together with full awareness of the 7.2. PARAMETRIZED REACHABILITY: A GLOBAL PICTURE state of the system, that is with a cooperative scheduler. In the safety case, we are interested in knowing whether the system is well designed to avoid a bad configuration, no matter the local specification of each process and the concrete interleaving, which are both chosen by a non-cooperative scheduler.

Parametrized reachability: a global picture

Let γ 0 , γ f ∈ Γ respectively be an initial and a target configuration. We are interested in checking whether γ 0 → * γ f holds. Note first that this implies |γ 0 | = |γ f | as → relation preserves configuration sizes. This problem has some drawbacks:

• It may be computationally hard. Intuitively, the problem can be encoded as a reachability problem in a |γ 0 |-safe Petri Net. Conversely, reachability in 1-safe Petri Net can be coded back to reachability problem in a particular network protocol with fixed origin and target. Due to the constant number of processes, we can enforce atomicity and presence of a leader can be enforced even in the non-atomic and leaderless case. However, [START_REF] Cheng | Complexity results for 1-safe nets[END_REF] showed that reachability in 1-safe Petri Net is PSPACE-complete.

• On the other hand, solving reachability question for a particular initial configuration does not solve reachability for other instances. Moreover, the number of processes can be seen as an undetermined variable, and we may be interested in reachability no matter the exact value of this parameter, except the possible assumption for this parameter to be large.

We consider now a more general reachability problem. As our number of processes is not fixed a priori, a reasonable goal is to analyze the behaviour of our system for an arbitrary number of processes. This number will be left as a parameter and we are interested in detecting changing behaviour of our system depending on this parameter.

First of all, we consider the whole set of initial configurations: Definition 7.9. Let U 0 = {(q l , n • q c , d 0 ) | n ≥ 1} be the set of initial configurations.

As previously mentioned, the size of a configuration does not change over time so we have to consider a set of target configurations U f , and the problem is now whether there exists

γ 0 ∈ U 0 , γ f ∈ U f such that γ 0 → * γ f .

Monotonicity

Upward closed reachability objectives

Elements of U 0 share the following property: they are all composed of the same leader state, register value, and set of contributor states (mainly {q c }). They can be expressed easily using the following ordering relation: Definition 7.10 (Ordering relation). We consider the ordering relation over Γ defined by γ γ if Given the previous definitions, the set of initial configurations is easily expressed as the following upward closure.

• γ = γ , • ∀q ∈ γ γ(q) ≤ γ (q).
U 0 = ↑{(q l , q c , d 0 }

In the rest of the chapter, we will assume that U f enjoy similar property, that is U f is an upward closed set, and we will now give several reasons to justify this decision.

Note first that this assumption allows various kinds of reachability objectives, such as the following:

• Coverability of a given contributor state q f by some process:

U f = {γ | γ(q f ) > 0}.
• Consensus to a given contributor state q f by all processes:

U f = {γ | γ(q f ) = |γ|}.
Another important feature of these objectives is their finite unique representation, as an upward closure of a finite set of incomparable elements. This will allow us to consider the computational complexity of a reachability problem from U 0 to U f , as we will be able to finitely describe any instance of our problem. This finite unique representation is due to results from Chapter 2 and the following result on well quasi-orders. Theorem 7.12 (Dickson's lemma). (Γ, ) is a well quasi-order (wqo).

A natural quantity for complexity study would be the size of a representation of an upward closed set U which will be likely a sum of the sizes of minimal elements γ∈min U |γ|. However, this quantity reveals to be large for some simple objectives, as coverability. Indeed, for coverability objective, min{γ | γ(q f ) > 0} = 2 [Qc]-1 , so the naive representation of such objectives is at least exponential. As this quantity is large, even for small minimal elements, we may lose some information about the difficulty of the reachability problem. We introduce below another notion of size, which will be more relevant for our later results, based on the maximal size of minimal elements appearing in an upward closed set. Definition 7.13. Let U ⊆ Γ an upward closed set. We define the size of U by

|U | = max γ∈min U |γ|
As announced before the definition, this new notion of size captures the simplicity of both previous objectives consensus and coverability. Indeed, we check that in both cases, we have

|U f | ≤ [Q c ] ≤ |P|.

Non-atomicity

We have seen that reachability is in general a computationally hard problem. A way to tackle this issue presented for example in [START_REF] Esparza | Parameterized verification of asynchronous shared-memory systems[END_REF] is to consider non-atomic register protocols, which enjoy monotonicity properties, as we expect the result of the parametrized reachability analysis to provide additional regularity properties. Definition 7.14 (Monotonous register protocol). We say that P is monotonous when for any γ 1 , γ 2 ∈ Γ with γ 1 → γ 2 , the two following properties hold:

MONOTONICITY

• For any γ 1 γ 1 , there exists γ 2 such that γ 1 → * γ 2 and γ 2 γ 2 .

• For any γ 2 γ 2 , there exists γ 1 such that γ 1 → * γ 2 and γ 1 γ 1 .

Our definition of monotonicity is very close to the concept of well-structured transition system (WSTS), defined in [START_REF] Finkel | Well-structured transition systems everywhere![END_REF]. We can indeed remark that a protocol is monotonous if, and only if, both (Γ, →, ) (first condition) and (Γ, → -1 , ) (second condition) are WSTS. Definition 7.15. We define, for any A ⊆ Γ:

Pre(A) = {γ ∈ Γ | ∃γ ∈ A γ → γ } Post(A) = {γ ∈ Γ | ∃γ ∈ A γ → γ }
WSTS are particularly well suited for backward reachability analysis, since predecessor operator maps upward closed sets to upward closed sets. Here, both the initial system and the reversed one are WSTS, so both predecessor and successor operators preserves upward closed sets. Lemma 7.16. Assume P is monotonous, then Pre, Post, Pre * , Post * preserve upward closed sets.

Our systems also enjoy an additional property: they preserve size of the configurations over time, which gives us an equivalent simpler definition based on incrementation of states: Lemma 7.17 (Monotonous register protocol). P is monotonous if, and only if, for any γ 1 , γ 2 ∈ Γ with γ 1 → γ 2 , the two following properties hold:

• For q 1 ∈ γ 1 , there exists q 2 ∈ γ 2 such that γ 1 ⊕ q 1 → * γ 2 ⊕ q 2 .
• For q 2 ∈ γ 2 , there exists

q 1 ∈ γ 1 such that γ 1 ⊕ q 1 → * γ 2 ⊕ q 2 .
Proof. We describe the proof of the first point, the second property is similar: Notice first that we can assume γ 1 → * γ 2 instead of γ 1 → γ 2 in the hypothesis as the result extends immediately to arbitrary paths by induction on their lengths.

⇒ Assume P is monotonous, and let q 1 ∈ γ 1 , then γ 1 γ 1 = γ 1 ⊕q 1 , so there exists γ 2 γ 2 such that

γ 1 → * γ 2 , hence |γ 2 | = |γ 1 | = |γ 1 | + 1 = |γ 2 | + 1 so there exists q 2 ∈ γ 2 = γ 2 such that γ 2 = γ 2 ⊕ q 2 .
⇐ Conversely, assume that γ 1 → γ 2 and consider γ 1 such that γ 1 γ 1 . We show by induction on |γ 1 | -|γ 1 | that there exists γ 2 such that γ 1 → * γ 2 and γ 2 γ 2 .

-

If |γ 1 | = |γ 1 | and γ 1 γ 1 , then γ 1 = γ 1 . -If |γ 1 | ≥ |γ 1 | + 1 and γ 1 γ 1 , then there exists q 1 ∈ Q c and γ 1 ∈ Γ, such that γ 1 =
γ 1 ⊕ q 1 and γ γ 1 . By induction hypothesis, there exists γ 2 such that γ 1 → * γ 2 and γ 2 γ 2 . By hypothesis, there also exists q 2 such that

γ 1 ⊕ q 1 → γ 2 ⊕ q 2 = γ 2 . Hence, γ 2 satisfies γ 1 → * γ 2 and γ 2 γ 2 .
In the rest of the chapter, we will be mainly interested in monotonous protocols. In particular, any non-atomic operations f can be repeated (stuttered) an arbitrary amount of time. This simple remark applies to any non-atomic protocol which therefore are monotonous. As depicted in Figure 7.2, each process can be mimicked by another one (the copycat) performing the same transitions immediately after each transition of the original process.

1 q 1 q 1 q 1 1 q 2 q 2 q 1 (q 1 , R (1) , q 2 ) 1 q 2 q 1 q 1 ( q 1 , R ( 1 )
, q

2

) ( q 1 , R ( 1 ) , q 2 ) 1 q 2 q 2 q 2 (q 1 , R (1) , q 2 ) 2 q 1 q 1 q 2 (q 2 , W (2) , q 1 ) 2 q 1 q 2 q 2 ( q 2 , W ( 2 ) , q 1 ) ( q 2 , W ( 2 ) , q 1 )
Figure 7.2 -Copycat lemma applied to path (1, 2 • q 1 ) → 3 (2, q 1 ⊕ q 2 ) of register protocol from Figure 7.1 by copying first process. Each of its transitions is copied (dashed) immediately after the original one, thanks to non-atomicity.

q 0 q 1 q l 0 R (1) R (0) • W (1) R (1)
Figure 7.3 -Example of a simple atomic protocol which is not monotonous. Indeed, (q l , q 0 , 0) → (q l , q 1 , 1) but for any n ≥ 2, we cannot have (q l , n • q 0 , 0) → * (q l , n • q 1 , 1). Lemma 7.18 (Copycat Lemma [START_REF] Durand-Gasselin | Model checking parameterized asynchronous shared-memory systems[END_REF]). Non-atomic register protocols are monotonous.

Proof. Let γ 1 , γ 2 ∈ Γ such that γ 1 → γ 2 and q 1 ∈ γ 1 . There exists (q, f, q ) ∈ T , such that γ 1 (q,f,q ) → γ 2

• If q = q 1 , then q 1 ∈ γ 2 so q 2 = q 1 satisfies γ 1 ⊕ q 1 → γ 2 ⊕ q 2 .

• If q = q 1 , then q 2 = q satisfies γ 1 ⊕ q 1 →→ γ 2 ⊕ q 2 . Non-atomicity is a crucial property for monotonicity, as illustrated in Figure 7.3 with a simple atomic protocol which is not monotonous.

Probabilistic transition system

As discussed earlier, we are interested in stochastic behaviours of our systems, which may lead to breaking symmetries. We will now introduce probabilities over our transition system and characterize the qualitative properties we will study. In order to do so, we will equip our transition system with probability over transitions between configurations. This implies replacing our non-deterministic scheduler by a stochastic scheduler. As opposed to reachability and safety properties, our new stochastic scheduler will neither play with nor against our protocols as transitions will be picked at random. Definition 7.19 (Probabilistic transition system). Let p : (γ, γ ) ∈ [→] → p(γ, γ ) ∈ R such that for all γ ∈ Γ, p(γ, -) is a probability distribution with support Post({γ}). We say that (Γ, p) is a probabilistic transition system of P.

⇐ Assume Post

* A (Γ n ∩ U 0 ) ⊆ Pre * (A), then (Post * A (Γ n ∩ U 0 ) ∪ Pre * (A), p

) forms a finite Markov chain with a set of transient states in Post

* A (Γ n ∩ U 0 ). ⇒ Let γ ∈ Post * (Γ n ∩ U 0 )
, which means there exists a path π : γ n 0 ∈ U 0 → * γ, with π ∈ (Γ\A) + . We initially have P n (π [START_REF]1 Simple games that require randomization[END_REF] • Γ * ∩ ♦A) = 1 and we show by induction that for any i,

P n ((π[i] • Γ * ) ∩ ♦A) = 1. Indeed, P n ((π[i] • Γ * ) ∩ ♦A) = γ P n (π[i] • γ ) • P n ((γ • Γ * ) ∩ ♦A) with γ P n (π[i] • γ ) = 1. In particular, for γ = π[i + 1], P n (π[i] • γ ) > 0 so P n ((π[i] • Γ * ) ∩ ♦A) = 1 implies P n ((π[i + 1] • Γ * ) ∩ ♦A) = 1. In particular, P n ((γ • Γ * ) ∩ ♦A) = 1, so γ ∈ Pre * (A).
As opposed to Post operator, Post A and even Post * A do not preserve upward closed sets, even when A is upward closed. For example, assume Post({(q, q 1 , d)}) = {(q, q 2 , d)} and We can check that simple reachability sets are generated by minimal elements of the form γ with ∀q γ(q) ≤ 1. In particular, both reachability and consensus objectives are simple reachability objectives.

A = ↑{(q, 2 • q 1 , d)} then Post A ( ↑{(q, q 1 , d)}) = Post({((q, q 1 , d)}) = {(q, q 2 , d)} and Post * A ( ↑ {(q, q 1 , d)}) = ↑{(q, q 1 , d)} ∪ Post 1 A ( ↑{(q, q 1 , d)}) ∪ Post 2 A ( ↑{(q, q 1 , d)}) ∪ . . . = ↑{(q, q 1 , d)} ∪ {(q, q 2 ,
When U f is a simple set reachability objective, Post U f and Post * U f preserve upward closed sets, thanks to the following lemma: Lemma 7.24. For any upward closed set U and a simple set

U f , U \U f is upward closed. Proof. Let γ γ with γ ∈ U \U f . Assume γ ∈ U f , then there exists γ m ∈ min U f ⊆ min Γ such that γ m
γ . This means that γ = γ = γ m and we have the following multiset inequalities: st(γ) ≤ st(γ ) and st(γ m ) ≤ st(γ ). This implies st(γ m ) ≤ st(γ) which is absurd since γ / ∈ U f . Hence, γ ∈ U \U f .

Cut-off property

We will study some limit behaviours of our systems according to the probabilistic reachability defined before. We introduce a notion of cut-off, similar to [START_REF] Außerlechner | Tight cutoffs for guarded protocols with fairness[END_REF].

Definition 7.25. Let I be a sub-interval of [0, 1] and A a set of configurations. We say that N ∈ N is a I-cut-off for reaching A, if either

• for all parameter n ≥ N , P n (♦A) ∈ I,

• for all parameter n ≥ N , P n (♦A) / ∈ I.

The cut-off is said positive in the first case, and negative otherwise.

A cut-off is a parameter value that permits restricting the reachability analysis to a finite number of parameters: beyond this parameter the answer to the reachability question has a fixed answer either positive or negative. Let us first remark that a cut-off needs not exist, however for a given interval I, existence of positive and negative cut-off are mutually exclusive properties.

When the target set A will be omitted, we will consider a generic upward closed set U f . As the previous Lemma 7.22 concerning qualitative probabilistic properties applies, we will distinguish the following particular cut-off properties:

• {0}-cut-off is a safety property;

• (0, 1]-cut-off is a reachability (with positive probability) property;

• {1}-cut-off is almost-sure reachability.

The following chapters will be mainly dedicated to the two following questions:

• Existence: does there exists a I-cut-off for reaching U f ?

• Decision problem: is this I-cut-off positive or negative ? Chapter 8

Probabilistic reachability and safety

Before diving into the almost-sure reachability problem, we will build some tool box from the analysis of non-deterministic schedulers, thanks to the study of (positive) reachability and safety cut-offs.

In this section, we revisit results from [START_REF] Esparza | Parameterized verification of asynchronous shared-memory systems[END_REF], where analysis of runs is based on two transition categories: useless write operations, that are immediately overridden, and first write operation for a given value d ∈ D. The safety problem from coverable objective boils down to the analysis for a small number, [D], of contributors, where each contributor is in charge of writing exactly one of the value d ∈ D for the first time. As a consequence, the safety (resp. reachability) cut-off problem is shown in co-NP (resp. NP), then proven to be co-NP-complete (resp. NP-complete) for non-atomic register protocols.

We adopt here a dual approach, where emphasis is put on the evolution of the set of contributor states in the support, that can therefore make the leader progress. The developed techniques, based on the symbolic graph, abstracting the system, will be refined later to analyze almost-sure reachability.

Existence

One first important remark consists in showing that safety and reachability are dual notions, which is not obvious from the definition, but is a direct consequence of the nature of our systems, that can ignore additional processes: let us denote N ∈ N {∞} as the minimal value such that P N (♦U f ) > 0 (with N = ∞ if it never holds). Then,

• If N = ∞, 0 is a negative cut-off for reachability, and a positive cut-off for safety.

• If N < ∞, by copycat lemma, N is a positive cut-off for reachability and a negative cut-off for safety, by minimality of N .

Symbolic graph

We introduce the main tool for the analysis of reachability under a non-deterministic scheduler, namely the abstraction of the number of states in a configuration.

Definition 8.1. The symbolic graph G P of P is a directed graph G P = (V, E) with

• V = {γ | γ ∈ Γ}. • E = (γ, γ ) γ, γ ∈ Γ : γ → γ .
Intuitively, the symbolic graph represents an abstraction through the support function, nodes are of the form s = γ with γ ∈ Γ and there is an edge from s to s if a transition is possible for some pair of concrete configurations, with respective support s and s . From the definition, we can obviously state than any path π ∈ paths (→) can be converted into a path in the symbolic graph by abstracting each configuration in the path : π ∈ paths (E).

An example of (partial) symbolic graph is depicted in Figure 8.1.

{q 0 }, 0 {q 1 }, 1 {q 1 }, 0 {q 1 }, 2 {q 2 }, 1 {q 0 , q 1 }, 0 {q 0 , q 1 }, 1 {q 0 , q 1 }, 2 {q 0 , q 2 }, 1 {q 0 , q 1 , q 2 }, 1 {q 0 , q 1 , q 2 }, 2 {q 1 , q 2 }, 1 {q 1 , q 2 }, 2
Nodes whose configurations cover q f Figure 8.1 -Solving coverability of q f for the protocol of Figure 7.1. Self-loops and target nodes are omitted.

The symbolic graph does not keep track of the number of contributor processes in each state: some process may have to be duplicated during transitions involving contributors. Hence, a natural notion to measure the length of a run is the number of transitions involving a contributor process. Definition 8.2. Let π ∈ paths (E) be a path. We denote by |π| l the leader length of π, defined as the quantity:

|π| l = i 1 ≤ i < |π| ∧ ∃γ, γ ∈ Γ ∃t ∈ T l γ t → γ ∧ γ = π[i] ∧ γ = π[i + 1]
Then, we define the contributor length of π by :

|π| c = |π| -|π| l
A first important remark is that we can build again a concrete run from a symbolic run, involving no more contributor processes than the contributor length of the path. Lemma 8.3. Assume P is monotonous. If there exists a path from s to s of contributor length n, then there exist a path from γ ∈ Γ to γ ∈ Γ with γ = s, γ = s and |γ| ≤ n

+ [Q c ].
Proof. We proceed by induction on the length of the path π ∈ paths (E) from s to s .

• If |π| = 1, π is restricted to the single state s = (q, X, d) so we define γ = γ = (q, q ∈X q , d).

• If sE * s Es , we apply induction hypothesis on sE * s with contributor length n, there exist γ, γ

∈ Γ with γ = s, γ = s , |γ| ≤ n + [Q c ] and γ → * γ .
If there exists t = (q, f, q ) ∈ T l such that (q, X, d) = s and (q , X, f(d)) = s then the contributor length is still n and we let γ = (q , st(γ ), Otherwise, there exists t = (q, f, q ) ∈ T c such that γ (q) > 0, (q, X, d) = s and (q, X , f(d)) = s with X ∪ {q } = X ∪ {q}. We can again rebuild γ by firing transition t from γ , except if we still have at the end q ∈ X and γ (q) = 1. In this last case, we apply Lemma 7.17 to path γ → * γ (monotonicity with state incrementation) to assume, without loss of generality, that γ (q) = 2 (hence γ (q) = 1) and

f(d)) hence γ t → γ . ( * , X 1 , * ) ( * , X 2 , * ) ( * , X 3 , * ) . . . ( * , X k , * ) [→] = k -1 ≤ 2 • [Qc] → ? • (→ + • →) * [→] ≤ [Q l ] hence [→] ≤ [Q l ] + 1
|γ| = |γ | ≤ n+ [Q c ] +1.
Note that a similar argument can be found in [START_REF] Durand-Gasselin | Model checking parameterized asynchronous shared-memory systems[END_REF]. The authors are considering coverability objectives. A path covering q f can be abstracted into a symbolic path, that can be compressed into another symbolic path, which is non-decreasing on the contributors set coordinate. By the previous lemma, this new symbolic path can still be realized, and still covers q f . This property ensures that the number of portions is bounded by [Q c ], hence a polynomial witness for coverability. Although this non-decreasing property is sufficient for the study of coverability objectives, it cannot be used anymore for our arbitrary upward closed objectives.

On the other hand, the particular structure of non-atomic protocols allows us to give another bound on the symbolic path, which is still polynomial in the size of the network; more formally, we are able to perform the following path compression: Lemma 8.4 (Diameter of the symbolic graph). Assume P is non-atomic (hence monotonous). If two nodes s and s are connected in G P , then there exists a path from s to s of contributor length smaller than

(2

• [Q c ] + 1) • ( [Q l ] + 1) + 2 • [Q c ]
An important remark on this bound is its independence with the number of involved data values: the quantity neither depends on [D] nor [T c ] nor [T l ]. Intuitively, density of the graph (Γ, →) has no influence on its diameter. We depict the two steps of the path reduction in Figure 8.2 and in the following proof:

Proof. Let π : s → * s . Thanks to the monotonicity property, we transform π into π satisfying the following properties:

1. first(π) = first(π ), last(π) = last(π ), 2. For each q ∈ Q c , there is at most one position i such that q / ∈ π [i] and q ∈ π [i + 1],

3. For each q ∈ Q c , there is at most one position i such that q ∈ π [i] and q / ∈ π

[i + 1],
With the previous requirements, it is easy to check that π is composed of at most 2 [Q c ] transitions of the contributors that changes the set of contributors. As a consequence, there are at most 2 [Q c ] + 1 contiguous portions of π where the set of contributors is fixed. For such a portion, we can assume, without loss of generality that the leader length is at most equal to [Q l ], otherwise, the leader process would have visited the same state twice. Since the set of possible operations performed by the contributor is fixed during a portion, there is at most one operation performed before and between each leader transition, hence at most [Q l ] + 1 transitions for the contributors.

We conclude this chapter with the following result:

Theorem 8.5. The two following problems are NP-complete:

ReachCoverCutOff INPUT:
A non-atomic register protocol P and a final state q f ∈ Q c . QUESTION: Whether there exists a cut-off for positive probability reachability of

U f = {γ | γ(q f ) > 0}.

ReachConsensusCutOff

INPUT:

A non-atomic register protocol P and a final state q f ∈ Q c . QUESTION: Whether there exists a cut-off for positive probability reachability of

U f = {γ | γ(q f ) = |γ|}.
Moreover, cut-off value for reachability of any upward closed target

U f is always polynomial in |P| + |U f |.
Proof. The proof structure is the same for any upward closed objective U f . If a (concrete) path manages to visit U f , we can compress this path thanks to lemmas 8.4 and 8.3 to involve only a polynomial number of processes. Moreover, thanks to monotonicity, this path is still valid for bigger parameters. This allows us to state that previous symbolic run provides a cut-off for reachability, and one can non-deterministically guess such a polynomial size run to decide whether U f is reached in the symbolic graph, which concludes membership of these problems to NP.

NP-hardness follows from reduction from 3SAT, as described in [START_REF] Esparza | Parameterized verification of asynchronous shared-memory systems[END_REF].

Chapter 9

Almost-sure reachability

We consider now the main problem introduced for probabilistic systems, namely the almostsure reachability cut-off properties. As we will see, this new property is not comparable with reachability and safety properties as scheduler is neither bad nor good in helping reaching a final configuration. This section will first give some intuitions about the difficulty of the almost-sure reachability problem and why the previous tools are not suited anymore. Then, we will prove that non-atomic protocols always have a cut-off for almost-sure reachability for any simple objective. Since the proof relies on well quasi-orders, the cut-off value may have an arbitrary size, so before diving into a decision procedure, we will study hardness results. Lower bounds for minimal (tight) cut-off values will be given, then the decision will be proven to be PSPACE-hard. Finally, we will provide a refinement of our symbolic graph that will allows us to develop a EXPSPACE decision procedure. Unless otherwise specified, we will mainly consider a non-atomic register protocol P with a simple reachability objective U f (see Definition 7.23). Our results will involve an arbitrary leader, but hardness results will be given in the stronger case of a leaderless protocol.

First examples 9.1.1 Atomicity prevents cut-off existence

Let us first consider the example depicted in Figure 9.1, with coverability of q f as simple target set U f . We argue that due to the atomic operations involved, the protocol has no cut-off. Intuitively, two processes from state q 0 can cooperate to cancel each other by going to sink state q 2 . If the number of initial processes is odd, there should always remain one process which always have the ability to trigger the transition to q f . In the long run, such event will happen with probability one. First of all, if n is even, we exhibit a run which sends all processes to state q 2 : (n

q 0 q 1 q 2 q f 0 R(0) W(1) R(1) W(0) R(1) W(2) R(2) W(0) R(0)
• q 0 , 0) → ((n -1) • q 0 ⊕ q 1 , 1) → ((n -2) • q 0 ⊕ q 1 ⊕ q 2 , 2) → ((n -2) • q 0 ⊕ 2 • q 2 , 0)→ 3 n-2
2 (n • q 2 , 0). In the other case, we can easily check that from any initial state in U 0 = ↑ {(q 0 , 0)}, the system can only reach configurations γ where γ(q 1 ) = 1 ν(γ) =0 (one q 1 when register differs from 0, none otherwise). We derive then:

• If ν (γ) = 2, then γ(q 2 ) is odd; • If ν (γ) = 2, then γ(q 2 ) is even; • If |γ| is odd, then for any successor γ , γ (q 2 ) < |γ |.
We conclude that for any odd parameter n,

P n (♦U f ) = 1.
This example gives us another reason to focus only on non-atomic registers protocols, for which such atomicity of processes cannot occur.

Symbolic graph is powerless

We have seen in the previous section that reachability in the symbolic graph is equivalent to reachability in (Γ, →).

A first intuition consists is checking almost-sure reachability in the symbolic graph to infer almost-sure reachability cut-off in the concrete system. Indeed, we can remark that the second property implies the existence of a positive cut-off. Theorem 9.1. Assume that P is monotonous and has a positive cut-off for almost-sure reachability of U f . Then, any reachable node s in G P from U 0 can reach U f . Proof. Assume P has a positive cut-off N . For any reachable node s in G P from U 0 , we can find by Lemma 8.3 γ 0 ∈ U 0 and γ such that γ 0 → * γ and γ = s. We apply monotonicity to assume that |γ| ≥ N (otherwise replace γ by γ ⊕ (N -|γ|) • q with same support, for some q ∈ γ). By qualitative almost-sure reachability, there exists

γ f ∈ U f such that γ → * γ f , thus s = γE * γ f ∈ U f .
However, we explain now that the converse property cannot be extended to the probabilistic transition system for almost-sure reachability. As a matter of fact, we consider the leaderless non-atomic protocol of Figure 7.1 and its symbolic graph depicted in Figure 8.1.

Nodes covering q f in the symbolic graph are reachable from any intermediate abstract configuration in the symbolic graph. This means that equipped with any relevant probability transitions, the symbolic graph satisfies almost-sure reachability for covering q f . However, some transitions require the presence of at least two processes in particular given states. This is in general not ensured by our abstraction. Consider for example, initial run with parameter n:

(n • q c , 0) → ((n -1) • q c ⊕ q 1 , 0) → ((n -1) • q c ⊕ q 1 , 1)
We can check that such configuration cannot reach U f = {γ | γ(q f ) > 0} anymore, as the only process in state q 1 has to read register value 2 from q 2 , which can only be written by itself, when going back to state q 1 . This transition exists in the symbolic graph, as ({q 2 }, 1) → ({q 1 , q 2 }, 2) though it is not concretely feasible.

q 2 q 1 γ η 1 η 2 (a) U ⊆ U therefore U U q 2 q 1 γ η 1 η 2 (b) U
U as γ can be arbitrarily increased on q 2 coordinate and stay outside U .

q 2 q 1 γ η 1 η 2
(c) U U as eventually γ will be replaced by configurations with more q 1 or q 2 states enforcing inclusion in U .

Figure 9.2 -Simple cases of ultimate inclusions analysis with

Q c = {q 1 , q 2 }, U = ↑ {γ} and U = ↑{η 1 , η 2 }.

Existence

Due to the monotonicity property and the wqo structure of the transition system, both Post * U f (U 0 ) and Pre * (U f ) are upward closed sets that are finitely generated. By analyzing further how the reachable states evolve with bigger parameters, we now prove that any monotonous register protocol has a cut-off for almost-sure reachability.

Thanks to Lemma 7.22, we know that for a given parameter n, P n (♦U f ) = 1 if, and only if,

Post * U f (Γ n ∩ U 0 ) ⊆ Pre * (U f )
, that is if, from any reachable configuration, there exists a finite suffix that can still reach U f . A positive cut-off for almost-sure reachability corresponds to a parameter N above which the preceding inclusion always holds. Moreover, since the number of state is preserved over time,

Post * U f (Γ n ∩ U 0 ) = Post * U f (U 0 ) ∩ Γ n
, which means we can start computing Post * U f (U 0 ) as an upward closed set, and later compute, for any n, the set PostU f * (Γ n ∩ U 0 ). Definition 9.2. We let U, U ⊆ Γ be two upward closed sets (for ). We say that U is ultimately included in U , noted U U , whenever there exists N ∈ N such that

∀k ≥ N. U ∩ Γ k ⊆ U From the previous remarks, Post * U f (U 0 ) Pre * (U f
) is equivalent to the existence of a positive almost-sure reachability cut-off for U f . However, it is not obvious from the given definition that ultimate inclusion is decidable. We give in the following lemma a more concrete characterization, based on the structure of upward closed sets. Indeed, in order to check inclusion of two upward closed sets U and U , we can reason on their minimal elements: U ⊆ U if, and only if, for all γ ∈ min U , there exists γ ∈ min U such that γ γ. Graphically, several cases of ultimate inclusion can occur, as depicted in Figure 9.2. For example, inclusion may not hold for small parameters, but can still hold after increasing the parameter value. Let us denote Γ ≥N = ∪ n≥N Γ n the upward closed set of configurations of size larger than N . Then, for any N , U ∩ Γ ≥N is upward closed and we can compute its minimal elements recursively, starting from U = U ∩ Γ ≥0 : for any γ ∈ min U ∩ Γ ≥N , γ has size at least N , and

9.2. EXISTENCE q 2 q 1 γ η 1 η 2 Γ k Γ k γ , with |γ | = η 1 (q 1 ) + η 2 (q 2 ) Figure 9.3 -Illustration of Lemma 9.3 on U
U in the case where U = ↑ {γ} with γ = {q 1 , q 2 } and γ has two uncomparable minimal elements in U , η 1 and η 2 . When k increases, the frontier of Γ ≥k is the diagonal Γ k . Eventually, this frontier will overpass γ (for example for k ). If ultimate inclusion occurs, it occurs at most at the point of coordinate (η 1 (q 1 ), η 2 (q 2 )) which corresponds to a configuration γ of size/diagonal

η 1 (q 1 ) + η 2 (q 2 ) ≤ [γ] • max(|η 1 |, |η 2 |) ≤ [Q c ] • |U | . • If |γ| > N , then γ is still a minimal element of U ∩ Γ ≥N +1 , • If |γ| = N , then γ cannot be a minimal element of U ∩ Γ ≥N +1 as γ / ∈ Γ ≥N +1
, however any configuration γ ⊕ q for any q ∈ γ is still part of U ∩ Γ ≥N +1 . If there exists another element γ in this set such that γ γ ⊕ q, then we should have

|γ | = |γ ⊕ q| = N + 1 hence γ = γ ⊕ q is minimal of size N + 1.
As the parameter N increases, it catches up the size of all minimal elements which eventually all fall down into the second category of minimal elements of size exactly N . These elements are updated by duplicating at each step any appearing state. The question remains then to determine whether these successive duplications of arbitrary appearing states can eventually lead to inclusion, or not. The next lemma states that one can consider one single state of the support to be duplicated. Moreover, we can bound the parameter to consider when looking at ultimate inclusion U U , by a polynomial in the size of U , as graphically shown in Figure 9.3. Lemma 9.3. U U if, and only if, for any γ ∈ min U and q ∈ γ, there exists k ∈ N such that γ ⊕ k • q ∈ U . Moreover, when this is the case, we have ∀k ≥

[Q c ] • |U | U ∩ Γ k ⊆ U Proof.
• (⇒) parameter k = N from the definition is sufficient.

• (⇐) Let γ ∈ min U , for each q ∈ γ, we define k(γ, q) ∈ N such that γ ⊕ k(γ, q) • q ∈ U . This means there exists η ∈ min U such that η γ⊕k(γ, q)•q. Without loss of generality we can assume k(γ, q) minimal, that is to say k(γ, q) = η(q)γ(q

) ≤ |U | -γ(q). Let us define k(γ) = q∈γ k(γ, q) ≤ [Q c ] • |U | -|γ|. Consider now γ ∈ U ∩ Γ k for k ≥ [Q c ] • |U |. There exists γ ∈ min U such that γ γ.
Hence,

q∈γ γ(q) -γ (q) = |γ| -|γ | ≥ [Q c ] • |U | -|γ | ≥ q∈γ k(γ , q)
so there necessarily exists q ∈ γ such that γ(q)γ (q) ≥ k(γ , q). It follows

γ γ ⊕ k(γ , q) • q ∈U γ. Corollary 9.4. If Post * U f (U 0 ) Pre * (U f ), then [Q c ] • |Pre * (U f )| is a positive cut-off for almost-sure reachability for U f . Otherwise, |Post * U f (U 0 )| is a negative cut-off for almost-sure reachability for U f .
Proof. Using previous Lemma 9.3, we derive the correct bound in the positive case.

Otherwise, there exists γ ∈ min Post * U f (U 0 ) and q ∈ γ such that for any

k ∈ N, γ ⊕ k • q / ∈ Pre * (U f ). Hence, |γ| ≤ |Post * U f (U 0 )| is a negative cut-off since for all k ≥ |γ|, γ k = γ ⊕ (k -|γ|) • q ∈ Post * U f (U 0 ) ∩ Γ k \Pre * (U f ).

Bound examples

The previous proof of almost-sure reachability cut-off existence is giving bounds on a possible cut-off, which are polynomial in |Post * U f (U 0 )| and |Pre * (U f )|. Such values are interesting to compute as they allow us to solve the decision problem: it is enough to check almost-sure reachability for a fixed parameter equal to the cut-off. However, the given bounds can be a priori large. For example, when U f is a coverable target, one can apply Petri Net bounds on coverability due to Rackoff [START_REF] Rackoff | The covering and boundedness problems for vector addition systems[END_REF] and expect |Pre * (U f )| to be bounded by a double exponential in |P|.

However, no general bound on an arbitrary upward closed set, especially for Post * U f (U 0 ), can be given directly.

Before diving into the decision problem, we can wonder what optimal bounds on the cutoff we can expect. In order to do so, we will consider some instances and compute the minimal parameter (lower bound) which is still a cut-off, namely a tight cut-off.

Linear cut-off

We start our study with a family of leaderless non-atomic register protocols (F n ) n>0 , depicted in Figure 9.4. For a fixed n, protocol F n has n + 1 states and n different data; intuitively, in order to move from s i to s i+1 , two processes are needed: one writes i in the register and goes back to s 0 , and the second process can proceed to s i+1 by reading i. Since backward transitions to s 0 are always possible and since states can always exit s 0 by writing a 0 and reading it afterwards, no deadlock can ever occur so the main question remains to determine if s n is reachable by one of the processes as we increase the number of initial processes. As shown in Lemma 9.5, the answer is positive: F n has a tight linear positive cut-off for coverability of s n ; it actually behaves like a "filter", that can test if at least n processes are running together. We exploit this property later in the next section. Lemma 9.5. Fix n ∈ N. The "filter" protocol F n , depicted in Figure 9.4, has a tight positive cut-off for almost-sure reachability of

U f = {γ | γ(s n ) > 0}, equal to n.
Proof. Let us consider the reachable configurations for m contributor processes in F n . We first prove that any reachable configuration γ ∈ Post * U f (U 0 ) ∩ Γ m satisfies:

∀j ≤ m. j k=0 γ(s k ) ≥ j + 1 ν(γ)=j+1
The proof is by induction: the invariant is satisfied by the initial configuration γ 0 = (s m 0 , 0). Let us now consider the run γ 0 → * γ → γ , in which γ satisfies the invariant, and with last transition (q, f, q ) ∈ T c .

• If f = R (0), then q = s 0 and q = s 1 . Along that transition, the right-hand-side term is unchanged; so is the left-hand-side term as soon as j > 0, so that the inequality is preserved for those cases. The case j = 0 is trivial.

• If f = R (i) with i > 0, then q = s i and q = s i+1 . We have st(γ ) = st(γ) s i ⊕ s i+1 and ν (γ ) = ν (γ) = i. Again, along this read-transition, the right-hand side term is unchanged, while the left-hand-side term is unchanged for all j = i.

It remains to prove the inequality for j = i. We apply the induction hypothesis in γ for j = i-1: since the transition (q, R (i) , q ) is available, it must hold that st(γ)(s i ) ≥ 1 and ν (γ

) = i = j + 1. Hence i-1 k=0 st(γ)(s k ) ≥ i-1+1 = i, and i k=0 st(γ)(s k ) ≥ i + 1. This implies i k=0 st(γ )(s k ) ≥ i.
• If f = W (i), then q = s i and q = s 0 . Thus st(γ ) = st(γ) s i ⊕ s 0 . For j = i -1, the left-hand-side term of the inequality is increased by 1, while the right-hand-side one is either unchanged or also increased by 1. The property is preserved in both cases.

For j = i -1, the left-hand-side term cannot decrease, while the right-hand-side term cannot increase. Hence the invariant is preserved.

As a consequence, if m < n, we have (for j = m) m k=0 st(γ)(s k ) = m for any reachable configuration γ, so that st(γ)(s n ) = 0.

Conversely, if m ≥ n, from any configuration γ and for any 0

≤ i < n, it is possible to reach γ i = (i • s 0 ⊕ (m -i) • s i+1 , i):
• for i = 0: all processes can go to s 0 , then write 0 in the register, and all move to s 1 :

γ → * (m • s 0 , d) W(0) ---→ (m • s 0 , 0) R(0) ---→ m (m • s 1 , 0); s 0 0 s 1 s 2 . . . s n-1 s n W(0) R(0) W(1) R(1) W(2) R(2) R(n-2) R(n-1) W(n-1)
Figure 9.4 -A "filter" protocol F n for n > 0.

s 0 0 s 1 s 2 . . . s n-1 s n q f W(0) R(0) R(0) W(1) R(1) W(2) R(2) R(n-2) R(n-1) W(n-1) W(i) i∈[0,n-1]
Figure 9.5 -The "reversed filter" protocol F n for n > 0, with negative cut-off for almost-sure reachability n.

• for 1 < i < n -1, assuming (i • s 0 ⊕ (m -i) • s i+1 , i
) can be reached, one of the processes in s i+1 can write i + 1 (going back to s 0 ), and the remaining mi -1 processes in s i+1 can go to s i+2 :

(i • s 0 ⊕ (m -i) • s i+1 , i) W(i+1) -----→ ((i + 1) • s 0 ⊕ (m -i -1) • s i+1 , i + 1) R(i+1) ----→ m-i-1 ((i + 1) • s 0 ⊕ (m -i -1) • s i+2 , i + 1) Thus from any γ ∈ Γ, configuration γ n-1 = ((n -1) • s 0 ⊕ (m -n + 1) • s n , n -1) is reachable. Furthermore, γ n-1 contains the final state s n since m ≥ n.
Hence, we deduce that there is a unique bottom strongly-connected component in Γ m , and that γ n-1 belongs to it: this configuration is reached with probability 1 from (m • s 0 , 0). It follows that P m (♦U f ) = 1.

Counter machine

We are now interested in finding non-trivial tight negative cut-offs. One way to build one such corresponding register protocol is to encode a mechanism able to count processes, and proceed to deadlock the system, avoiding reaching a target state q f , if enough processes are present. For example, the previous filter protocol can count in unary up to n with n + 1 states. We now modify the protocol as depicted in Figure 9.5: the previous target state s n for coverability is modified in order to allow the other processes to join in s n . The target state for coverability, denoted q f is only reachable from s 0 . Thus, as soon as one process has reached s n , there is positive probability that all processes join s n , and stay "trapped" in this location. This results in a linear negative cut-off (n + 2 states for cut-off value n).

A natural idea we develop below is to count in binary, namely encoding a binary counter by several processes, each of them (en)coding one bit of the current value. One main problem that arises with such encoding is the coherence of our counter, as we need to avoid several states to encode the same bit with different values at the same time. Intuitively, such property cannot be enforced a priori, as stated by the following lemma, related to the monotonicity property.

Lemma 9.6. Assume P is leaderless non-atomic. If π 1 : (q, µ 1 , d) → * (q, µ 1 , d 1 ) and π 2 :

(q, µ 2 , d) → * (q, µ 2 , d 2 ) then ∀d ∈ {d 1 , d 2 }, (q, µ 1 ⊕ µ 2 , d) → * (q, µ 1 ⊕ µ 2 , d ).
Proof. We prove the result by induction on |π 1 | + |π 2 |.

• If π 1 = ε, the result is immediate. From now on, we assume |π 1 | ≥ 1 and (by symmetry)

|π 2 | ≥ 1.
• If π 1 or π 2 starts with a read operation, we can apply this first transition and apply induction hypothesis on the resulting configuration directly (shared register value hasn't changed).

• If both π 1 and π 2 start with a write operation, then by monotonicity, (q,

µ 1 ⊕ µ 2 , d) → * (q, µ 1 ⊕ µ 2 , d 2 ) and (q, µ 1 ⊕ µ 2 , d) → * (q, µ 1 ⊕ 2 , d 1 )
. This last path starts with a write operation, so value register d can be replaced by d 2 hence the result.

One way to tackle this issue consists in detecting, after the binary count, that effectively no more that one process encoded each counter bit, thanks to our previous filter protocol for positive cut-off. A general pattern for encoding a counting mechanism can be described by three phases, also depicted in Figure 9.6.

• Initialization: the initial state allows processes to choose randomly which bit position they want to encode, or if they want to be part of the counted processes (tokens). After the first write, the initial register value is erased and transitions from this phase cannot occur anymore. Denote with k i (resp. k) the number of processes encoding the i-th bit (resp. the tokens).

• Simulation: Processes of the binary counter exchange the addition remainder thanks to the shared register. In order to enable the counting mechanism, one of the processes (tokens) to be counted perform a write transition, writing initial remainder 1, allowing the counter to progress. During the whole simulation phase, each state has positive probability to take an extra transition to the final target state.

• Check : Instead of writing a remainder for bit n + 1, the process encoding bit n writes a final value halt, meaning the counter managed to count, with possible errors, at least 2 n tokens, allowing counter processes to reach the filter module of parameter n + 1. This last filter has no more probabilistic transition to the target state so the only way to reach it is to play according to the filter gadget rules. When each bit of the counter is encoded by one single process (∀i k i = 1), the total amount of processes encoding bits is n so the final state cannot be reached and the counting is correct so k ≥ 2 n .

Notice that the whole pattern is meant to build a negative cut-off, as any possible error in the pattern will lead to reaching the target state: not enough encoding processes (∃i k i = 0), not enough tokens (k < 2 n ) or too many encoding processes ( i k i ≥ n + 1) ensure reaching q f almost-surely; the only run that avoids q f is the one that respects the three described phases.

Theorem 9.7. There exists a family of leaderless non-atomic register protocols, which admit negative tight cut-offs for almost-sure reachability with coverability objective whose value is exponential in the size of the protocol.

Proof. We first focus on the first part of the protocol of Figure 9.6, containing nodes named a i , b i , c i , d i and s i . This part can be divided into three phases: the initialization phase lasts as long as the register contains #; the counting phase starts when the register contains halt for the first time; the simulation phase is the intermediate phase.

in # tok sent sink W(1) R(halt) a 1 b 1 c 1 d 1 R (1)
W(0) R(1) W(2) a 2 b 2 c 2 d 2 R(2) W(0) R(2) W(3) a n b n c n d n R(n) W(0) R(n) R(#) s 0 s 1 s 2 s n q f W(f 0 ) R(f 0 ) W(f 1 ) R(f 1 ) W(f 2 ) R(f 2 ) R(f n-1 ) R(fn) R(m),m =halt R(i) i =1 R(#) R(i) i =2 R(i) i =n R(i) i =1 R(i) i =2 R(i) i =n R(halt) R(f i ),i∈[0,n] W(halt)
initialization simulation checking Figure 9.6 -Simulating an exponential counter: grey boxes contain the nodes used to encode the bits of the counter; yellow nodes at the bottom correspond to the filter module from Figure 9.4; purple nodes tok, sentand sinkcorrespond to the second part of the protocol, and are used to produce tokens. Missing read edges are assumed to be self-loops.

During the initialization phase, processes move to locations a i and tok, until some process in tokwrites 1 in the register (or until some process reaches q f , using a transition from a i to q f while reading #). Write γ 0 for the configuration reached when entering the simulation phase (i.e., when 1 is written in the register for the first time). We assume that st(γ 0 )(a i ) > 0 for some i, as otherwise all the processes are in tok, and they all will eventually reach q f . Now, we notice that if st(γ 0 )(a i ) = 0 for some i, then location d n cannot be reached, so that no process can reach the counting phase. In that case, some process (and actually all of them) will eventually reach q f . We now consider the case where st(γ 0 )(a i ) ≥ 1 for all i. One can prove (inductively) that d i is reachable when st(γ 0 )(tok) ≥ 2 i . Hence d n , and thus also s 0 , can be reached when st(γ 0 )(tok) ≥ 2 n . Assuming q f is not reached, the counting phase must never contain more than n processes, hence we actually have that st(γ 0 )(a i ) = 1. With this new condition, s 0 is reached if, and only if, st(γ 0 )(tok) ≥ 2 n . When the latter condition is not true, q f will be reached almost-surely, which proves the second part of our claim: the final location is reached almost-surely in systems with strictly less than n + 2 n copies of the protocol.

We now consider the case of systems with at least n + 2 n processes. We exhibit a finite execution of those systems from which no continuation can reach q f , thus proving that q f is reached with probability strictly less than 1 in those systems. The execution is as follows: during initialization, for each i, one process enters a i ; all other processes move to tok, and one of them write 1 in the register. The n processes in the simulation phase then simulate the consecutive incrementations of the counter, consuming one token at each step, until reaching d n . At that time, all the processes in tok move to sent, and the process in d n writes halt in the register and enters s 0 . The processes in the simulation phase can then enter s 0 , and those in sent can move to sink. We now have n processes in s 0 , and the other ones in sink. According to Lemma 9.5, location q f cannot be reached from this configuration, which concludes our proof.

Remark 9.8. The question whether there exists protocols with exponential positive cut-offs remains open. The family of filter protocols described previously is (only) an example of protocols with a linear tight positive cut-off.

PSPACE-hardness

The previous idea of checking afterwards whether a simulation was correct can also be exploited to encode a linear-bounded Turing machine [START_REF] Sipser | Introduction to the theory of computation[END_REF] instead of a simple binary counter. This technique is developed below to establish PSPACE-hardness for the cut-off problem.

We build a register protocol for which there is a negative cut-off for almost-sure coverability, if, and only if, the machine reaches its final state q halt with the tape head reading the last cell of the tape.

Write n for the size of the tape of the Turing machine. We assume (without loss of generality) that the machine is deterministic, and that it accepts only if it ends in its halting state q halt while reading the last cell of the tape. Our reduction works as follows (see Fig. 9.7): some processes of our network will first be assigned an index i in [1; n] indicating the cell of the tape they shall encode during the simulation. The other processes are stuck in the initial location, and will play no role. The state q and position j of the head of the Turing machine are stored in the register. During the simulation phase, when a process is scheduled to play, it checks in the register whether the tape head is on the cell it encodes, and in that case it performs the transition of the Turing machine. If the tape head is not on the cell it encodes, the process moves to the target location (which we consider as the target for the almost-sure reachability problem). Finally, upon seeing (q halt , n) in the register, all processes move to a (n + 1)-filter protocol F n+1 (similar to that of Fig. 9.4) whose last location s n+1 is the aforementioned target location.

If the Turing machine halts, then the corresponding run can be mimicked with exactly one process per cell, thus giving rise to a finite run of the distributed system where n processes end up in the (n + 1)-filter (and the other processes are stuck in the initial location); from there s n+1 cannot be reached. If the Turing machine does not halt, then assume that there is an infinite run of the distributed system never reaching the target location. This run cannot get stuck in the simulation phase forever, because it would end up in a strongly connected component from which the target location is reachable. Thus this run eventually reaches the (n + 1)-filter, which requires that at least n + 1 processes participate in the simulation (because with n processes it would simulate the exact run of the machine, and would not reach q halt , while with fewer processes the tape head could not go over cells that are not handled by a process). Thus at least n + 1 processes would end up in the (n + 1)-filter, and with probability 1 the target location should be reached.

We now formalize this construction, by describing the states and transitions of the protocol within these three phases. We fix a linear-bounded Turing machine M = (Q, q 0 , q halt , Σ, δ), where Q is the set of states, q 0 , q halt ∈ Q are the initial and halting states, Σ is the alphabet, and

δ ⊆ Q × Σ × Q × Σ × {-1, +1} is the set of transitions. We define the data alphabet D = {#} Q × Σ {f i | 0 ≤ i ≤ n}, and the set of locations P = {p init , p init , p sink } [1; n] × Σ × (Q ∪ {ε}) {s i | 0 ≤ i ≤ n + 1}.
The set of locations corresponds to three phases (see Fig. 9.7):

p init # p init 1, c 1 2, c 2 3, c 3 ... n, c n p sink R(#) R(D\{#}) R(#) R(#) R(#) W(q 0 ,1) 1, σ 1, σ, q 1, σ n, σ n, σ , q n, σ R(q,1) W(q ,2) R(q ,n) W(q,n-1) s 0 s 1 s 2 s n s n+1 W(f 0 ) R(f 0 ) W(f 1 ) R(f 1 ) W(f 2 ) R(f 2 ) R(f n-1 ) R(fn) R(q halt ,n) R(f i ),i∈[0;n] R(•,j),j =n R(#) (to sn+1) (to sn+1) (to s0) (to s0)
initialization simulation for transitions (q, σ) → (q , σ , +1) and (q , σ ) → (q, σ , -1) checking Figure 9.7 -Distributed protocol P M encoding the linear-bounded Turing machine M.

• The initialization phase contains p init , p init and p sink . From the initial state p init , upon reading # (the initial content of the register), the protocol has transitions to each state (i, σ i ) for all 2 ≤ i ≤ n, where σ i is the i-th letter of the initial content of the tape. If reading anything different from #, the protocol moves to the sink state p sink . Finally, there are transitions (p init , R (#) , p init ) and (p init , W (q 0 , 1) , (1, c 1 )), where q 0 is the initial state of the Turing machine: this pair of transitions is used to initialize the computation, by setting the content of the first cell and modifying the register, so that the initialization phase is over (there are no transitions writing # in the register).

• The second phase, called simulation phase, uses register alphabet Q × [1, n], in order to encode the state and position of the head of the Turing machine. The state space for the simulation phase is [1; n] × Σ × (Q ∪ {ε}): state (i, σ, ε) (written (i, σ) in the sequel) encodes the fact that the content of the i-th cell is σ; the states of the form (i, σ, q) are intermediary states used during the simulation of one transition: when in state (i, σ) and reading (q, i) in the register, the protocol moves to (i, σ, q), from which it moves to (i, σ ) and writes (q , j) in the register, provided that the machine has a transition (q, σ) → (q , σ , ji). If the active process does not encode the position that the tape head is reading (i.e., the process is in state (i, σ) and reads (q, j) with j = i) then it moves to the final state s n+1 of the third phase.

• The role of the counting phase is to count the number of processes participating in the simulation. When seeing the halting state in the register, each protocol moves to a module whose role is to check whether at least n + 1 protocols are still "running". This uses data {f i | 0 ≤ i ≤ n} and states {s i | i ∈ [0, n + 1]}, with transitions from any state of the simulation phase to s 0 if the register contains (q halt , n) or any of {f i | 0 ≤ i ≤ n}.

We now prove that our construction is correct:

Lemma 9.9. The register protocol P M , with coverability objective s n+1 , has a negative cut-off if, and only if, the Turing machine M reaches q halt in the last cell of the tape.

Proof. First assume that there is a negative cut-off: there exists N 0 such that for any N ≥ N 0 , starting from the initial configuration (p N init , #) of the system (Γ N , →) made of N copies of P M , the probability that at least one process reaches s n+1 is strictly less than 1. Since (Γ N , →, p) is a finite Markov chain, this implies that there is a cone of executions never visiting s n+1 , i.e., a finite execution ρ whose continuations never visit s n+1 . Since the register initially contains #, this finite execution (or a finite continuation of it) must contain at least one configuration where some process has entered the simulation part. Now, in the simulation phase, we notice that, right after taking a transition ((i, σ, q), W (q , i ± 1) , (i, σ )), the transition ((i, σ ), R (•, j) , s n+1 ) is always enabled. It follows that at the end of the finite run ρ, no simulation transition should be enabled; hence all processes that had entered the simulation part should have left it. Hence some process must have visited s 0 along ρ (because we assume that ρ does not involve s n+1 ). Moreover, by Lemma 9.5, for s n+1 not to be reachable along any continuation of ρ, no more than n processes must be able to reach s 0 along any continuation of ρ, hence at most n processes may have entered the simulation phase. On the other hand, for s 0 to be visited, some process has to first write (q halt , n) in the register; since the register initially contains (q 0 , 1), and no process can write (•, i + 1) without first reading (•, i), then for each i ∈ [1, n] there must be at least one process visiting some state (i, σ i ), for some σ i ; It follows that at least n processes must have entered the simulation phase.

In the end, along ρ, exactly one process visits (i, c i ), for each i ∈ [1, n], and encode the content of the i-th cell. As a consequence, along ρ, each cell of the tape of the Turing machine is encoded by exactly one process, and the execution mimics the exact computation of the Turing machine. Since the configuration (q halt , n) is eventually reached, the Turing machine halts with the tape head on the last cell of the tape.

Conversely, assume the Turing machine halts, and consider the execution of N ≥ n processes where exactly one process goes in each of the (i, c i ) and mimics the run of the Turing machine (the other processes going to p sink ). We get a finite execution ending up in a configuration where all processes are either in p init or in p sink , except for n processes that are in the counting phase. No continuation of this prefix ever reaches s n+1 , so that the probability that some process reaches s n+1 is strictly less than 1.

We conclude by stating the following complexity results: Theorem 9.10. The following problems are PSPACE-hard.

ASCoverCutOff INPUT:

A non-atomic register protocol P and a final state q f ∈ Q c . QUESTION: Whether there exists a cut-off for almost-sure reachability of

U f = {γ | γ(q f ) > 0}.

ASConsensusCutOff

INPUT:

A non-atomic register protocol P and a final state q f ∈ Q c . QUESTION: Whether there exists a cut-off for almost-sure reachability of

U f = {γ | γ(q f ) = |γ|}.

Decision procedure

As previous Section 9.2 proved that the existence of a cut-off is ensured, the question of the nature of cut-off (positive or negative) arises. Answering this question boils down to deciding the asymptotic behaviour of our system, when the parameter is large enough. One naive approach to this problem consists in computing a cut-off value as given by Corollary. 9.4, then simulating the system to decide whether almost-sure reachability holds for this particular parameter. However the corresponding value is polynomial in both the size of the Pre * (U f ) and Post * U f (U 0 ) minimal elements, for which we have no bound a priori. Notice however that the theory of well-quasi-orders ensures that these elements can be computed in finite time. For example, Post * U f (U 0 ) can be computed by saturation, by considering the sequence (X n ) n , with X 0 = U 0 and ∀n X n+1 = X n ∪Post U f (X n ). For each n, X n is an upward closed set, and its minimal elements can be computed iteratively. Moreover, this sequence is non-decreasing for inclusion, and is eventually stationary; otherwise, we could extract an infinite sequence of minimal elements which are incomparable, which would be contradictory with the wqo structure.

On the other hand, our previous analysis gives us some hints about the complexity of a decision procedure. First of all, Theorem 9.10 tells us the problem is PSPACE-hard. Then, our different constructions show examples of tight linear and exponential cut-off, so we can expect the minimal elements to be at least of the same size (up to a polynomial).

The following sections show how to deal with coverability objectives, which allow the usage of already known bounds; then we proceed to develop general techniques for any objectives. In particular, we will be able to derive upper bounds on tight cut-off values.

Refined symbolic graph

As the symbolic graph revealed to be insufficient to analyze almost-sure reachability, we consider now a refinement where it can keep track of up to a fixed portion of configurations.

In order to do so, we first introduce a new protocol that encodes part of contributors' configuration inside the leader structure. We will then derive results from the analysis of its resulting symbolic graph. Definition 9.11 (Fixed protocol). Let P = (Q l , Q c , D, d 0 , q l , q c , T l , T c ), a register protocol, and k ∈ N, we define P k by P k = (Q l , Q c , D, d 0 , q l , q c , T l , T c ), where

• Q l = Q l × N Qc k , • q l = (q l , q k c ) ∈ Q l • T l = ((q, µ), f, (q , µ)) | (q, f, q ) ∈ T l , µ ∈ N Qc k ∪ ((q, µ), f, (q, µ )) | ∃q t ∈ µ, q t ∈ µ (q t , f, q t ) ∈ T c ∧ µ ⊕ q t = µ ⊕ q t
Intuitively, fixed protocols of index k encodes an initial protocol with at least k processes, that are taken apart and considered as part of the leader process. This isolation technique will reveal to be convenient to develop decision procedures and bounding techniques that couldn't be achieved directly by the symbolic graph, which cannot keep track of any quantitative number of processes. As one may notice, non-atomic, monotonous, atomic properties on register protocols are preserved, but not the leaderless property. We can also easily check the following translation lemma:

Then, γ = (q, µ 1 ⊕ µ 2 ⊕ µ 3 , d) → * γ f ∈ U f , however, for each 1 ≤ i ≤ m, q i / ∈ µ 2 ⊕ µ 3 so γ (q i ) = µ 1 (q i ) = γ(q i ) < η i (q i ) hence η i γ which is contradictory to γ ∈ Pre * (U f ).
Hence, the hypothesis sE * U f K was flawed. This lemma gives a general approach to decide almost-sure reachability cut-off for any monotonous protocol, based on the exploration of a finite graph. The remaining challenge consists in analyzing the resulting complexity, which will depend on the exact system and objective considered.

Complexity bounds on covering

We consider first the complexity of the algorithm that can be deduced from Lemma 9.14, in the case of almost-sure reachability of a coverability target U f = {γ | γ(q f ) > 0}. Using results by Rackoff on the coverability problem in Vector Addition Systems [START_REF] Rackoff | The covering and boundedness problems for vector addition systems[END_REF], we can bound |Pre

* (U f )| (then K = [Q c ] • |Pre * (U f )|
) by a double-exponential in the size of the protocol.

Due to the symmetry abstraction that leads us to consider only multiset of states as configurations, the resulting symbolic graph of index K does not have another exponential blow up, as stated by Lemma 9.15. Therefore, it suffices to solve a reachability problem in NLOGSPACE [START_REF] Sipser | Introduction to the theory of computation[END_REF] on this doubly-exponential graph, that can be constructed on the fly: this boils down to NEXPSPACE complexity with regard to the protocol's size, hence EXPSPACE complexity, by Savitch's theorem [START_REF] Sipser | Introduction to the theory of computation[END_REF]. Lemma 9.15. Let Q be a finite set. Then

N Q n = |Q| + n -1 n ≤ (2n) |Q|-1
Proof. Note Q = {q 1 . . . q k }, then the following function is a bijection

           N Q k → (i 1 , . . . i k-1 ) ∈ N k-1 1 ≤ i 1 < i 2 < . . . < i k-1 ≤ n + k -1 γ →   j + j ≤j γ(q j )   1≤j<k Hence, |N Q n | = k+n-1 k-1 = 1 (k-1)! k-1 i=1 (n + i) = k-1 i=1 n+i i = k-1 i=1 1 + n i ≤ (2n) k-1
Theorem 9.16. The following problem is in EXPSPACE: ASCoverCutOff INPUT:

A non-atomic register protocol P and a final state q f ∈ Q c . QUESTION: Whether there exists a cut-off for almost-sure reachability of

U f = {γ | γ(q f ) > 0}.
Proof. Pre * (U f ) is exactly the set of configurations that can cover q f , i.e., configurations γ from which there exists a path γ → * γ with γ (q f ) > 0. Recall also that it can be written as an upward closure of minimal elements: Pre * (U f ) = ↑ {η 1 , . . . , η m }. Now, consider the value K in Lemma 9.15: it is defined as K

= [Q c ] • |Pre * (U f )|, with |Pre * (U f )| = max i |η i |
the maximal size of minimal configurations. The value of K can be bounded using classical results on the coverability problem in Vector Addition Systems (VAS) [START_REF] Rackoff | The covering and boundedness problems for vector addition systems[END_REF].

Intuitively, a b-dimensional VAS is a system composed of an initial b-dimensional vector v 0 of naturals (the axiom), and a finite set of b-dimensional integer vectors (the rules). An execution is built as follows: it starts from the axiom and, at each step, the next vector is derived from the current one by adding a rule, provided that this derivation is admissible, i.e., that the resulting vector only contains non-negative integers. An execution ends if no derivation is admissible. The coverability problem asks if a given target vector v = (v 1 , . . . , v b ) can be covered, i.e., does there exists a (possibly extendable) execution v 0 v 1 . . . v n = v such that, for all 1 ≤ i ≤ b, it holds that v i ≤ v i .

Our distributed system P can be seen as a [Q c ]-dimensional VAS where each transition of the contributor is modeled by a rule vector modifying the multiset of the current configuration. Formally, one has to take into account that available rules depend on the data stored in the shared register and that leader process can also perform transitions atomically. This can be achieved by either considering the expressively equivalent model of VAS with states (VASS, see e.g., [START_REF] Rosier | A multiparameter analysis of the boundedness problem for vector addition systems[END_REF]) or by adding O( [D]) dimensions to enforce this restriction. Over such a VAS(S), we are interested in the coverability of the vector corresponding to the multiset q f (i.e., containing only one copy of q f and no other state). In particular, we want to bound the size of vectors needed to cover q f , as it will lead to a bound on minimal elements η i of Pre * (U f ), hence a bound on the value K.

Results by Rackoff (hereby as reformulated by Demri et al. [START_REF] Demri | The covering and boundedness problems for branching vector addition systems[END_REF]lemma 3]) state that if a covering execution exists from an initial vector v 0 , then there is one whose length may be doubly-exponential in the size of the input: singly-exponential in the size of the rule set and the target vector, and doubly-exponential in the dimension of the VAS. Hence, for our distributed system P, seen as a VAS, this implies that if q f can be covered from a configuration γ, there is a covering execution whose length is bounded by some

L in 2 O( [Qc]• [D]) O( [Qc]+ [D])
. This bound on the length of the execution obviously also implies a bound on the number of processes actively involved in the execution (because at each transition, only one process is active). Hence, we can deduce that if a configuration γ = (q, µ, d) can cover q f (i.e., there exists a path γ → * γ with γ (q f ) > 0), then it is also the case of configuration γ = (q, µ , d), which we build as follows: ∀ q ∈ Q, µ (q ) = min{µ(q ), L}. That is, it also holds that there exists a path γ → * γ with γ (q f ) > 0.

By Lemma 9.14, for our algorithm to be correct, it suffices to consider the symbolic graph of fixed protocol of index [Q c ] • L and to solve sequentially two reachability problems over this graph. Let us study the size of this graph. Its state space is

V = Q c × N Qc L• [Qc] × 2 Qc × D. Hence, we have that: [V ] ≤ [Q c ]•(2 • L • [Q c ]) [Qc] •2 [Qc] • [D]
, which is doubly-exponential in both the state space of the protocol and the size of the data alphabet (because L is). Since reachability over directed graphs lies in NLOGSPACE [START_REF] Sipser | Introduction to the theory of computation[END_REF] with regard to the size of the graph, we obtain NEXPSPACE-membership for reachability queries with regard to the size of the protocol. Finally, by Savitch's theorem [START_REF] Sipser | Introduction to the theory of computation[END_REF], we know that NEXPSPACE = EXPSPACE, which allows us to define the following non-deterministic EXPSPACE algorithm:

1. For all nodes s 1 ∈ U 0 K and s 2 .

2. Guess a node s 3 ∈ U f K .

3. Check that s 2 is reachable from s 1 .

4. Check that s 3 is reachable from s 2 .

General bounding scheme

Our previous EXPSPACE algorithm solves the almost-sure reachability cut-off problem for a coverability objectives thanks to a result from Rackoff on coverability in vector addition systems. We will now develop a general bounding technique, adapted to our systems, to give general complexity for several classes of protocols and objectives.

The first natural idea to bound the size of Pre * (U f ) consists in considering the abstraction of any run γ → * U f in the symbolic graph, applying symbolic path reduction from Lemma 8.4, then constructing again a new path thanks to Lemma 8.3. We get this way a new run of the form γ → * U f with γ = γ and γ now of polynomial size. Assuming γ was minimal in Pre * (U f ) we can deduce some information on its size, since |γ | is polynomial. First of all, if st(γ) = 1, then necessarily γ γ and we conclude |γ| is also polynomial. Otherwise, we cannot conclude since st(γ) and st(γ ) may be component-wise incomparable. In this last case, we can at least derive that there exists q ∈ γ such that γ(q) < γ (q) which is polynomial.

The other components remains though unbounded but we can iterate this reasoning, by fixing already the size of the previously bounded component on q. We show in the next lemma that the fixed protocol defined in previous Section 9.4.1 is well suited for this purpose.

Lemma 9.17 (Bounding base). Assume P is non-atomic, U 0 and U f are simple sets. Let

γ ∈ min Post * U f (U 0 ) ∪ min Post * (U 0 ) ∪ min Pre * (U f ), then |γ| ≤ (1/2) • (18 • [Q c ] • [Q l ]) [Qc] [γ]
In particular,

|γ| ∈ O ( [Q l ]) O( [Qc]) O( [Qc]) ⊂ 2 O(|P|) O(|P|)
Before diving into the technical proof of this lemma, let us give some remarks about this result:

• For short, this lemma proves that not only the minimal predecessor configurations for a coverability objective (Rackoff bound), but any predecessor or successor minimal element, for any simple objective, have at most doubly exponential size.

• As for Lemma 8.4, the actual bound does not depend on the number of involved values in D, as opposed to the proof of previous Theorem 9.16 for the special case of coverability objectives.

• The contributors and the leader have different roles: for a fixed set of contributor control states, the bound is polynomial in the number of leader control states.

Proof. We first introduce an auxiliary function B, that will be useful to bound our configuration sizes. For all l ≥ 1, we define

B(0, l) = 0 ∀n ≥ 0 B(n + 1, l) = P (l) + B n, l(2P (l)) [Qc]-1 with P (l) = (2 • [Q c ] + 1)(l + 1) + 3 • [Q c ]
Notice that P (l) only depends on Q c and not on Q l . Moreover, it is monotone. B is also monotone:

• Monotone in l: we prove by induction on n ≥ 0, that for any l ≤ l , B(n, l) ≤ B(n, l ).

Result is true for n = 0, for n = n +1, we let k = l(2P (l)

) [Qc]-1 ≤ l (2P (l )) [Qc]-1 = k .
We apply induction hypothesis on n with k ≤ k to obtain B(n , k) ≤ B(n , k ) then B(n, l) ≤ B(n, l ) since we also have P (l) ≤ P (l ).

• Monotone in n: for any n ≥ 0, and any l ≥ 0,

B(n + 1, l) = P (l) ≥0 +B n, l(2P (l)) [Qc]-1 ≥l ≥ B(n, l)
We prove now a stronger result by induction on [γ],

|γ| ≤ B( [γ] , [Q l ])
• If [γ] = 0, we have |γ| = 0, hence the result for the base case.

• Let us now consider γ ∈ min Pre * (U f ) with [γ] ≥ 1 (reasoning on Post * (U 0 ) or Post * U f (U 0 ) is similar), which means there exists a path of the form γ → * γ f ∈ U f . We abstract this path into the symbolic graph, then reduce it (Lemma 8.4), and finally reconstruct it again (Lemma 8.3). Contributor length is at most equal to (2

• [Q c ] + 1)( [Q l ] + 1) + 2 [Q c ] then reconstruction can add up to [Q c ] processes.
The new resulting path is of the form γ → * γ f with |γ| ≤ (2

• [Q c ] + 1)( [Q l ] + 1) + 3 [Q c ] = P ( [Q l ]
), γ = γ and γ f = γ f . This last property implies that γ ∈ Pre * (U f ), since U f is a simple reachability objective.

There exists at least one q ∈ γ such that 0 < γ(q) ≤ P ( [Q l ]) (otherwise γ would have been smaller than γ which is minimal). Write k = γ(q), we have γ = (q l , µ ⊕ k • q, d) and γ q = ((q l , k • q), µ, d) as a configuration of P k . Thanks to Lemma 9.12, we notice that γ q ∈ min Pre * ( U f k ) in P k , and we can apply the induction hypothesis since [γ q ] = [γ] -1. Note that induction hypothesis is applied on another protocol, with the same control states for contributors, but with a refined set of control states for the leader, which is now

Q l = Q l × N Qc k . This set is of cardinality [Q l ] • N Qc k , hence |γ| = k + |γ q | ≤ k + B( [γ q ] , Q l ) = k + B( [γ] -1, [Q l ] • N Qc k ) ≤ B( [γ] , [Q l ]) since k ≤ P (l), N Qc k ≤ (2k) [Qc]
-1 and B is monotone.

We now proceed to bound B(n, l) in order to get the announced result. First, remark that for l ≥ 1, [Qc] . We show by induction on n that:

P (l) ≤ 9 • [Q c ] • l. Then l(2P (l)) [Qc]-1 ≤ (18 • [Q c ] • l)
∀n, l B(n, l) ≤ n-1 i=0 (1/2) i+2 ≤1/2 (18 • [Q c ] • l) [Qc] n
• For n = 0, this sums up to B(0, l) ≤ 0.

• For n ≥ 0,

B(n + 1, l) ≤ 9 • [Q c ] • l + B(n, l(18 • [Q c ] • l) [Qc]-1 ) ≤ 9 • [Q c ] • l + n-1 i=0 (1/2) i+2 18 • [Q c ] • l(18 • [Q c ] • l) [Qc]-1 [Qc] n ≤ 9 • [Q c ] • l + n-1 i=0 (1/2) i+2 (18 • [Q c ] • l) [Qc] n+1
We conclude the proof by noticing that 2

2 n+1 /2 n+2 ≥ 1 so for [Q c ] ≥ 2, 9 • [Q c ] • l ≤ (9 • [Q c ] • l) [Qc] n+1 ≤ (1/2) n+2 (18 • [Q c ] • l) [Qc] n+1 .
A first immediate consequence of this doubly exponential bounding, consists in giving an explicit bound on tight cut-off values, for any simple objective, thanks to Corollary. 9.4 on the cut-off existence.

Corollary 9.18. Assume P is a non-atomic protocol. Then P has a tight cut-off for almostsure reachability of any simple objective, of at most doubly exponential size.

Simulating the system for this particular value, or applying again the same technique as for coverability objective, lead us to a general EXPSPACE procedure: A non-atomic register protocol P and a final state q f ∈ Q c . QUESTION: Whether there exists a cut-off for almost-sure reachability of

U f = {γ | γ(q f ) = |γ|}.
Chapter 10

Extensions and discussions

As an illustration to the previous chapters, we describe several extensions of our framework for which previously developed techniques can be applied. We conclude our study in the last section, by a summary of complexity results, in the extended cases.

Model checking

In [START_REF] Durand-Gasselin | Model checking parameterized asynchronous shared-memory systems[END_REF], techniques based on the symbolic graph are developed to address the question of the liveness property in a non-atomic protocol under a non-deterministic scheduler, with coverability objective, and an extra Büchi automaton, encoding an LTL formula. This allows the authors to consider the model checking of network protocols, in various settings, where leader and contributor can be represented by finite state machines or pushdown systems. Intuitively, given an LTL formula ϕ and a network protocol P with transition system (Γ, →), we want to check whether there exists an infinite run π ∈ Γ ω satisfying ϕ. Existence of such a run is then proved to be equivalent to the existence of a particular lasso run π = π 1 • π ω 2 with π 1 and π 2 both being finite paths of P. Analyzing the existence of the symbolic graph in this context is sufficient, as we can consider enough processes in each state to keep the system unblocked, as the extra processes can be considered as "extra noise" and used over time to unlock the run periodically. Additionally, we can state that the existence of such an infinite run implies a cut-off property, as we can always add new processes, which will mimic (copycat) an existing process in the infinite run. However we may argue that such infinite run, that visits infinitely often some target state, does not provide a cut-off for positive probability. As a matter of fact, we check only the existence of one single infinite run. As opposed to a finite prefix that can witness a positive probability reachability, a single infinite run may have probability 0 to occur. From a nonprobabilistic perspective, processes and scheduler have to cooperate in the long run, which may seem unlikely to happen.

Therefore, it may be interesting to consider almost-sure repeated reachability and repeated reachability with positive probability. In the first case, the analysis is similar to the almostsure reachability:

• First of all, let us denote by ♦A the set of infinite paths that visit the set of configurations A infinitely often. Basically, ♦A represents the set of infinite runs that eventually reach A, ♦A = Γ * • A • (Γ * Γ ω ), and ♦A = (Γ * • A) ω .

• As in Lemma 7.22, we consider (Γ, p) a probabilistic transition system of P, and show that as for P n (♦A), qualitative values of P n ( ♦A) do not depend on p. Indeed, such property depends only on the support of p which is determined by the specification of P.

In particular, we can characterize almost-sure repeated reachability by P n ( ♦A) = 1 which is equivalent to the following property on Pre and Post:

Post * (U 0 ∩ Γ n ) ⊆ Pre * (A)
Intuitively, we replaced operator X → Post(X\A) by the more simple mapping Post(•), which means reachability should still hold after reaching A. Since Γ n is finite, there exists a positive probability to reach A again from any reachable configure γ. Indeed, this value is bounded from below by min{P({γ}Γ

* ∩ ♦A) >0 | γ ∈ Post * (A) [•]<∞ } > 0
• Let A = U f be a simple objective. Then, Pre * (U f ) is upward-closed. We adapt the proof of Corollary. 9.4 to show that either

[Q c ] • |Pre * (U f )| is a positive cut-off, or |Post * (U f )|
is a negative cut-off for almost-sure repeated reachability. Thanks to Lemma 9.17, we can also see that any tight cut-off is at most doubly exponential, and obtain an equivalent EXPSPACE decision procedure.

The second question, about the existence of a cut-off for positive probability of repeated reachability is surprisingly harder, and remains currently open: a positive probability indeed corresponds to finding a finite path from U 0 to some configuration γ from which repeated reachability holds almost-surely. If we denote by B the set of such configurations, we can remark that B is not upward closed: consider for example filter protocol F n example of Figure 9.4 with repeated reachability objective U f = {γ | γ(s n ) = 0}. Because of this lack of regularity, we cannot directly answer this last question, which could be possibly solved by additional work based on ideals, or through the use of the refined symbolic graph to keep some coordinates low.

r-register protocol

We have mainly developed tools for the analysis of non-atomic protocols, where processes communicate through a single shared variable. On the other side, the analysis of atomic protocols seems to be much harder, as one can expect that atomicity allows us to encode arbitrary Petri Nets. One may argue that there still remains a big gap between theoretical results on the cut-off existence, that are stated in the monotonous case, and algorithms and bounds that are only valid for non-atomic protocols. One may ask, if some monotonous systems, but not necessarily non-atomic still enjoy good combinatorial properties and if previous algorithms can be lifted to more complex frameworks. This section explores one possible way of improving our model by the use of several registers.

Tools enhancement

First of all, let us remark that our previous copycat Lemma 7.18 can be more generally stated this way:

Lemma 10.1 (Copycat lemma improved). Assume P involves only stuttering operations for contributor, namely, for all (q, f, q ) ∈ T c , (f • f) = f. Then P is monotonous.

The proof is the same as for the original lemma, as the copycat can mimic an already existing process thanks to this stuttering property.

On the other hand, bounds and algorithms were later given thanks to a path reduction in the symbolic graph, by Lemma 8.4. Recall the proof of this lemma, the crucial property is the ability for a fixed set of contributor states (in the support) to produce in one step the desired value on the shared register, in order to make the leader or the support progress. We relax this notion now, to allow a fixed number of steps.

Definition 10.2. Let k ∈ N >0 and X ⊆ Op (D). X is k-valued if for all d ∈ D, all Y ⊆ X and f ∈ Y + , there exists f ∈ ∪ i≤k Y i such that f(d) = f (d) P is k-valued if T c ⊆ Q c × X × Q c for some k-valued set of operations X.
Intuitively, for any current value d and sequence of operations applied on d, we can extract a (possibly re-ordered) sequence of operations of length at most k, that has the same effect on d. As expected the set of non-atomic operations Op R,W (D) is 1-valued: proof is done by picking the last write operation, if any, or any read operation otherwise.

Operations over r registers

Before explaining how previous results can be generalized to k-valued monotonous protocols, we give below an example of concrete k-valued set of operations, that is still stuttering. Definition 10.3 (Operations over a r registers). Let D a set and r ≥ 1. For i, j ∈ [1, r] and d ∈ D, we define the following operations over D r .

W i (d) : dom (W i (d)) = D r d → d [i/d] = d [1] • • • d [i -1] • d • d [i + 1] • • • d [r] R i (d) : dom (R i (d)) = D i-1 × {d} × D r-1-i d → d M i→j : dom (M i→j ) = D r d → d [j/d [i]] = d [1] • • • d [j -1] • d[i] • d [j + 1] • • • d [r]
We denote by Op R,W,M (D r ) the set of such register operations.

Our initial shared register is now split into r coordinates, each of them being called (abusively) a register. Non-atomic read and write operations are now done on one coordinate at a time, leaving the other registers unchanged.

As one can expect, when restricting to read and write operations, which are in some sense still non-atomic, we achieve r-valued monotonicity. This next result generalizes the case r = 1 studied in the previous chapters.

Lemma 10.4. For any r, and P such that

T c ⊆ Q c × Op R,W (D r ) × Q c , P is an r-valued monotonous protocol. Proof. First of all, P is monotonous, as Op R,W (D r ) is stuttering. Let us consider f ∈ Y * with Y ⊆ Op R,W (D r ) and d ∈ D r . For each i ∈ [1, r], if d[i] = (f(d))[i]) we define f i = Id. Otherwise, there exists f i = W i (d[i]) ∈ Y . Hence, f = f 1 • • • f r ∈ Y k with k = [{i | d[i] = (f(d))[i]}] ≤ r.
The previous proof handles the read and write case. However, the newly introduced move (or copy) operation can also be stuttered, so monotonicity is preserved. We prove in the next lemma that this new operation also keeps our protocol k-valued, where k is a polynomial in r.

Lemma 10.5. For any r, and P such that

T c ⊆ Q c × Op R,W,M (D r ) × Q c , P is an r 3 -valued monotonous protocol.
Proof. First of all, P is monotonous, as Op R,W,M (D r ) is stuttering. We

consider f = f 1 • • • f n with for all i, f i ∈ Y ⊆ Op R,W,M (D r ). We will construct another sequence of f i 1 , . . . f i k with k ≤ r 2 , such that f(d) = (f i 1 • • • f i k )(d).
Without loss of generality, we make the two following assumptions:

• There are no read operations in f, otherwise, we can remove them, since they have no side effect.

• The number l of write operations is smaller than the number r of registers, by pigeonhole principle (otherwise, some written value was useless). The remaining write operations can be seen as extra registers indexed from r + 1 to r + l with a unique move operation. For example, the i-th (1 ≤ i ≤ l) write operation W j i (d i ) is replaced by move operation M r+i→j i , assuming the initial value d satisfies d[r + i] = d i . We denote by r = r + l ≤ 2r the total amount of registers.

Thus, the rest of the analysis can be done with move operations only, whose sequence performs a mapping m from

each i ∈ [1, r], the final value f(d)[i] = d[m(i)
] is obtained by a finite path in Z. We can assume this path is cycle-free, and whenever, the path uses internal nodes of a strongly connected component C, it preserves all values except 1, which can be achieved with at most

[C] ( [C] -1) operations. With additional move operations between two components, we can assign register i its final value in at most r 2 , hence a total number of r 3 for the whole mapping m. 10.1, there exists a sequence of write and move operations, that cannot be expressed with less than a cubic number of operations. We conclude that up to a multiplicative constant, the r 3 bound given in the previous lemma is optimal.

As shown on examples of Figure

Discussion on the r-register extension

We revisit below previous results when considering k-valued monotonous protocols, instead of non-atomic ones.

Lemma 10.6 (Diameter of the symbolic graph). Assume P is k-valued monotonous. If two nodes s and s are connected in G P , then there exists a path from s to s of contributor length smaller than k

• (2 • [Q c ] + 1) • ( [T l ] + 1) + 2 • [Q c ]
The proof is similar to original Lemma 8.4 but between two transitions of the leader, we may need up to k contributor transitions, to achieve the next data value required by the leader. Since contributors may not be able to produce the same register values between two leader steps (even with the same support), we can now only bound the number of leader steps by [T l ] instead of simply [Q l ]. Note that in the leaderless case, this quantity remains small, otherwise, it can be of the same magnitude as [D] k . Lemma 10.7 (Bounding base). Assume P is k-valued and monotonous, U 0 and U f simple sets

. Let γ ∈ min Post * U f (U 0 ) ∪ min Post * (U 0 ) ∪ min Pre * (U f ), then |γ| ∈ 2 O(k•|P|) O(k•|P|)
In the leaderless case, since [T l ] = 0, we can even give a bound that is polynomial in k:

|γ| ≤ (1/2) • (6 • (2k + 1) • [Q c ] • [Q l ]) [Qc] [γ]
As a consequence, tight cut-off upper bounds and EXPSPACE procedure can be lifted to k-valued monotonous protocols, and even to protocols over r registers with read, write and move operations:

Corollary 10.8. The following problem is in EXPSPACE: ASCoverCutOff INPUT: An integer r ∈ N, a register protocol P with operation set Op R,W,M (D r ) and a final state q f ∈ Q c . QUESTION: Whether there exists a cut-off for almost-sure reachability of

U f = {γ | γ(q f ) > 0}.
Moreover, such a tight cut-off is doubly exponential.

Proof. Even if r is provided in binary form, one can first ensure that any of the r registers is used so that1 r ≤ 2 max( [T l ] , [T c ]) ≤ 2|P| (otherwise, we can rewrite P in polynomial time to make use of r < r registers). Moreover, P is r 3 -valued, so

max(|U 0 |, |U f |) ∈ 2 O(8•|P| 4 ) O(8•|P| 4 )
gives a doubly exponential bound for tight cut-off, hence an EXPSPACE decision procedure.

Comparison with non-atomic protocols

One may notice that multiple registers can be emulated through the use of the leader in the following way. We encode the current value d ∈ D r in the state of the leader, and allow him to write messages on the shared register, of the form "d

[i] equals v" for any i ∈ [1, r] and v = d[i].
When reading such message, the leader can also move to a state encoding the updated value.

From the point of view of contributors, any operation R i (d ) (resp W i (d )) can be converted into reading (resp writing) the message "d[i] equals d ". Any path in the r-register protocol can be converted in a path in the described 1-register protocol and conversely, therefore, positive probability and almost-sure reachability properties are preserved. However, this method has several drawbacks:

• First of all, this encoding can only deal with read and write operations.

• From a combinatorial point of view, the new state space of the leader Q l is now of cardinality [Q l ] × [D] r which can be huge. In the general case, the previous bound is also exponential in r, but leaderless case or other fragments may feature lower complexity.

• Last but not least, this encoding does not work for leaderless protocols, where we can extract a polynomial bound in r, and independent of [D].

In the leaderless case, such reduction seems impossible, and we conjecture that r-register protocols are indeed more expressive than non-atomic (or 1-register) protocols.

Process identifiers

As noticed before, non-atomic protocols allow us to bound the diameter of the symbolic graph by a quantity that only depends on the state space of the protocol, and not on the domain size [D], nor the number of transitions [T c ] + [T l ]. First of all, this means we can encode in only one transition a read operation that only checks that the current register value is different from a given value d, by adding several transitions (q 1 , R(d ), q 2 ) for any d ∈ D\{d}. This also means that D can virtually encode an infinite number of values, without any impact on the complexity results. We argue that our techniques can be adapted to encode processes with unique identifiers (pid ). We can indeed equip our protocols with a new write operation, writing the identifier of the current process to the shared register. If we add another operation checking that the process id corresponds to the one written in the shared register, monotonicity will be broken. However, we can still allow each process to execute an operation only if the shared register value differs from the process identifier.

Intuitively, such systems will allow the election of a leader process, but this single process will not be able to check that it is indeed the leader. We do not study further this model as it requires to redefine our whole transition system to incorporate unique process identifiers, then to redefine the well quasi order to compare configurations.

Conclusions

As a partial conclusion to the four last chapters, we present in Table 10.1 a summary of the bounds stated for positive reachability and almost-sure reachability of simple objectives in r-register protocols with read, write and move operations.

Chapter 11

Toward Strategy Synthesis

In this chapter, we try to address the local strategy synthesis problem. More precisely, we are interested in the (partial) determinization of under-specified protocols, in order to satisfy a given objective. Strategy synthesis is often considered from a global point of view, where power is given to the scheduler which can observe the whole system and choose the next transition accordingly.

Here, the scheduler is considered as a source of uncertainty, that cannot be controlled. We adopt a more distributed approach, where strategies are given to all processes (or agents) with their restricted observation power. Since processes are running the same automaton, it is reasonable to consider that the same local strategy is distributed among all agents. Once a strategy is defined over an under-specified network, we can consider the resulting system, which would be another protocol on which we can apply previously seen techniques.

We introduce concepts similar to the first part of the thesis, namely actions, then allowed actions, and finally strategy classes. Since we are looking at qualitative reachability questions, the exact probability distributions involved in a mixed strategy are not relevant, so our randomization only consists in picking a support of actions. In a non-stochastic context, this model would correspond to non-determinism, that is solved by the scheduler. Here, the scheduler has a stochastic behaviour, and will be pick an available transition at random. Several strategy classes will be considered which can require more or less memory and randomization.

The chapter is divided into three sections: we first introduce the local strategy semantics in the framework of non-atomic protocols, and justify why our analysis can be done without taking exact probability values into account. Then, we address the characterization of local strategies ensuring a positive cut-off, in the two simple cases of reachability and safety in a leaderless protocol with a coverability objective.

Definitions

Allowed actions and randomization

In the rest of the chapter, P = (Q l , Q c , D, d 0 , q l , q c , T l , T c ) is a given shared register protocol. We denote for simplicity

Q = Q l Q c and T = T l T c .
As in Definition 3.1, we define a set of allowed actions, and build strategies over these allowed actions. Here actions are subsets of transitions of P. When an agent chooses an action, the scheduler will be in charge of picking the exact transition to be played, non-deterministically or randomly. We additionally require that the subset of transitions are defined on the same domain as the initial transitions, thus no agent can block a run (to a finite prefix) that was initially live (infinite run). Definition 11.1 (Actions and allowed transitions). Let q ∈ Q be a state of P. A subset A ⊆ T is called an action from q if the two following conditions hold:

• All transitions start from q: A ⊆ {(q, f, q ) ∈ T | f ∈ Op (D) , q ∈ Q};

• Register operations of A are globally defined on the same domain:

(f, q )/ (q, f, q ) ∈ A dom (f) = (f, q )/ (q, f, q ) ∈ T dom (f)
We denote by Act(q) the set of actions from q.

A set of allowed actions for protocol P, from state q, is defined as

Allow(q) ∈ 2 Act(q) \{∅}
We have seen in the previous chapters that having more than one transition from a given state with the same register operation corresponds to some non-deterministic move that is resolved by the scheduler. In a probabilistic setting, this means that if action A contains two transitions that both accept the same register value d, the transition with value d will be randomized.

Graphically, when more than one action is allowed from a state, each action A is represented by an outgoing edge to an intermediary node, with several outgoing edges for each transition t ∈ A. This representation is similar to the stochastic nodes of our stochastic games, as the choice of the exact transition to be taken is left to the environment, or scheduler. For the sake of simplicity, when [Allow(q)] = 1, that is to say when only one action A is allowed, we omit the representation of action A and its intermediary node, and directly represent outgoing transitions from q. However, we have to keep in mind that the choice of the exact transition taken from A is left to the scheduler, which will choose at random.

For example, let us consider the simple protocol of Figure 11.1. When current register value is 0 and action A 1 is played, two transitions are available: (q 0 , R (0) , q 0 ) and (q 0 , R (0) , q 1 ). In this case, the scheduler will choose (at random) which transition effectively takes place.

Local strategies

We proceed now to define local strategies. Here, the word local means that strategies can only see a history for a given player, that is to say a word h ∈ Q + .

Intuitively, a strategy assigns to each history a decision, which is a distribution over the allowed actions. Since we are considering qualitative questions, from a state q only the support δ ⊆ Allow(q) ⊆ Act of the distribution is relevant. The exact transition to be taken is resolved in the following way:

• A process is chosen, which provides (thanks to its local strategy) a distribution support δ.

Reachability

We study first the local synthesis problem for positive reachability, that is to say, whether there exists a cut-off strategy σ for positive reachability. In this setting, the scheduler is cooperative and we are interested in the existence of a single path that agrees with σ.

Mixed strategies

As the local strategy will cooperate with the scheduler, a reasonable strategy is to play as many actions as possible, to enable as many paths as possible. This intuition is summarized below.

Lemma 11.11. Let σ, σ ∈ S two strategies such that

∀h ∈ Q + σ(h) ⊆ σ (h)
Then agree (σ) ⊆ agree (σ ).

Proof. Immediate: for any π ∈ agree (σ), we consider a corresponding sequence of multiset of histories ρ ∈ (N Q + ) |π| from Definition 11.6 and check that is valid for path π .

Let us denote by Reach ⊆ S the set of cut-off strategies for positive probability of covering q f . Then, the following theorem states that a memoryless randomized strategy suffices. Theorem 11.12. Let σ be the memoryless strategy playing all allowed actions at random, and assume that Reach = ∅. Then σ ∈ Reach. In particular, Reach ∩ M = ∅.

Proof. Let σ ∈ Reach, then there exist π ∈ paths (→) ∩ agree (σ ), γ 0 ∈ U 0 and γ f ∈ U f such that π : γ 0 → * γ f . For all h ∈ Q + , σ (h) ⊆ σ(h), so and π ∈ agree (σ) by Lemma 11.11, thus σ ∈ Reach.

The previous theorem basically states that the existence of a mixed cut-off strategy for positive probability can be reduced to the decision problem of a positive cut-off in the underlying protocol alone, which belongs to P for the particular case of non-atomic leaderless coverability.

Pure strategies

Another approach, studied in [START_REF] Bertrand | Distributed Local Strategies in Broadcast Networks[END_REF], considers pure strategies only. In their context, they show that polynomial memory suffices for a pure strategy to cover q f , if such strategy ever exists.

Theorem 11.13. Assume Reach ∩ S = ∅. Then Reach ∩ M (P (|P|, [Allow])) = ∅, where P is a polynomial.

The proof is very similar to the one of [BFS15, proposition 2], yet using another model, which only allows certain forms of actions2 and where players can see the local actions.

We argue that these differences are not relevant for the announced result, and give a sketch of the proof.

h 1 q 0 h 2 q 0 h 2 ∀h h 1 h ⇒ σ (h) = σ(h) σ (h 1 -) = σ(h 2 -)
Figure 11.2 -Illustration of the proof of Theorem 11.13, when h 1 h 2 , first(h 1 ) = first(h 2 ) and subtrees rooted in h 1 and h 2 contain the same set of important nodes (in red). Since, h 1 and h 2 have the same final state, we can replace strategy from subtree rooted in h 1 by the one rooted in h 2 (same allowed actions). This is possible since all the important nodes are in the subtree rooted in h 2 or outside the subtree rooted in h 1 , therefore all removed nodes and transitions can be generated from somewhere else.

Proof. Let σ ∈ Reach ∩ S. We consider π ∈ agree (σ) ∩ U 0 • Γ * U f and the sequence of multiset of histories ρ as given from Definition 11.6. At step i, we denote by ρ[i] ∈ Q + the history that is picked to evolve, that is the unique history (the same as in Definition 11.6) word h

= ρ[i] such that ρ[i + 1](h) = ρ[i](h) -1.
We define T ⊆ T as the set of transitions that can be seen:

T = 1≤i≤|ρ| σ(ρ[i])
In particular, there exists a transition t f = (q, f, q f ) ∈ T . We define the set H of seen histories as

H = {ρ[i] | 1 ≤ i ≤ |ρ|}
H is closed by prefix, and can be seen as a tree, with root q 0 (the initial state of P). For h ∈ H, the subtree rooted in h can be expressed as H ∩ (hQ * ). We define I ⊆ H, the set of important nodes, as the histories that trigger a transition for the first time in the run, namely:

I = {ρ[i] | 1 ≤ i ≤ |ρ| ∧ ∀j ∈ [1, i -1] σ(ρ[i]) ⊆ σ(ρ[j])}
We argue that we can convert σ, π and ρ simultaneously, such that T is kept unchanged, q 0 ∈ H, and additionally, for any h

1 , h 2 ∈ H, if last(h 1 ) = last(h 2 ) and I∩(h 1 Q * ) = I∩(h 2 Q * ), then h 1 = h 2 .
Intuitively, this last property states that there are no two subtrees H ∩ (h 1 Q * ) and H ∩ (h 2 Q * ), with roots ending in the same state, containing the same important nodes.

Assume, that the property is not satisfied for some h 1 and h 2 :

q 0 0 q 1 q f A 1 A 2 W (1) R (0) R (1)
Figure 11.4 -Example of a protocol with allowed actions requiring randomization for a local strategy to be a cut-off strategy for positive coverability of q f . In order to cover q f , a local strategy has to play A 1 for one of the process, in order to write value 1, then play A 2 for another process. However, each process only sees its own history, which is always q 0 . QUESTION: Whether there exists a cut-off strategy σ, which is pure, for positive probability reachability of

U f = {γ | γ(q f ) > 0}.
Proof.

• As stated by Theorem 11.12, we can check that U f is reached with positive probability in the original protocol P, which can be done in polynomial time.

• NP-completeness of pReachCoverStrat is similar to the proof of [BFS15, theorem 2].

Safety

We address now the existence problem of a cut-off strategy for safety from a coverability objective. The main contribution of the section is the proof that pure memoryless strategies suffice. Intuitively, both pure and memoryless classes are restrictions of the strategies, that can only reduce the set of accessible configurations.

We define by Safe ⊆ S the set of cut-off strategies for safety. While a cut-off strategy for reachability requires only a finite run to prove reachability, a cut-off strategy for safety has to avoid U f for any arbitrary run. Thus, under-approximation of the strategy by a finite tree is not a suitable technique anymore, as runs may explore arbitrary long histories. Our proof consists in transforming a cut-off strategy for safety σ into another strategy that is still safe, and memoryless "for small histories", then applying a limit/diagonal argument.

As for Lemma 9.6, we can sum two paths together in the leaderless case:

Lemma 11.15. Assume P is leaderless non-atomic. Let σ be a strategy, π 1 : (q, µ 1 , d) → * (q, µ 1 , d 1 ) and π 2 : (q, µ 2 , d) → * (q, µ 2 , d 2 ), that both agree with σ.

Then (q, µ 1 ⊕ µ 2 , d) → * (q, µ 1 ⊕ µ 2 , d ) with d = d 1 or d = d 2 ,
and there exists such a path that agrees with σ.

The proof is omitted, as it is similar to Lemma 9.6. In fact, this result can be derived from this previous lemma applied to the "meta protocol" P(σ) informally defined before.

For h ∈ Q + , we say that h is enabled by σ if there exists a run π ∈ agree (σ) ∩ U 0 • Γ * with associated histories ρ from Definition 11.6 such that h appears in ρ, that is to say ∃i h ∈ ρ[i], where ρ[i] corresponds to the support of multiset of histories ρ[i]. We denote by en(σ) the set of enabled histories in σ. Lemma 11.16

. Let σ ∈ S, h 1 , h 2 ∈ Q + such that 11.3. SAFETY • last(h 1 ) = last(h 2 ); • h 2 ∈ en(σ).
We define σ by σ :

h 1 • h ∈ h 1 • Q * → σ(h 2 • h) h / ∈ h 1 • Q * → σ(h)
Then, en(σ ) ⊆ en(σ).

Proof. Consider h f ∈ en(σ ), so there exists π ∈ agree (σ ) ∩ U 0 Γ * , with ρ the corresponding sequence of histories, such that h f ∈ last(ρ). We will convert ρ in order to agree with σ. h 2 is enabled by σ so there exists a run π ∈ agree (σ), with sequence of histories ρ such that h 2 ∈ last(ρ ). For any i such that ρ[i] = h 1 , we add a copy of π into π thanks to Lemma 11.15. Any further execution of process with history h 1 is replaced by one of the process with history h 2 added in π .

At the end of the transformation, ρ is a run that agrees with σ and contains history h f , hence h f ∈ en(σ).

Notice that en(σ) contains an history ending in U f if, and only if, σ / ∈ Safe. We conclude, that the property carried by the lemma is stronger than just keeping safety property. This extra property is relevant to converge when iteratively constructing a memoryless strategy.

The following theorem concludes that pure memoryless strategies suffice for safety cut-off.

Theorem 11.17. Assume Safe = ∅, then Safe ∩ M = ∅.

Proof. Let σ ∈ Safe and let be a total ordering over Q + compatible with the length function (∀h, h h h ⇒ |h| ≤ |h |). We define

P = {(h 1 , h 2 ) | h 1 , h 2 ∈ Q + , last(h 1 ) = last(h 2 ), h 2 h 1 , h 1 = h 2 }
and consider f : N → P that enumerates P in the lexicographic order: this is possible since is well-founded. For any n ∈ N, we construct σ n ∈ Safe by induction:

• σ 0 = σ
• σ n+1 is defined by applying Lemma 11.16 to σ n and the pair of histories (h 1 , h 2 ) = f (n) whenever h 2 ∈ en(σ n ), otherwise σ n+1 = σ n . In both cases, we have en(σ n+1 ) ⊆ en(σ n ).

Let k ∈ N, we define N (k) such that:

∀n ≥ N (k) f (n) = (h 1 , h 2 ) ⇒ |h 1 | > k
Indeed, there is a finite number of pair of words of length smaller than k.

For a fixed history h and n

≥ N (|h|), if f (n) = (h 1 , h 2 ), then |h 1 | > |h| and h / ∈ h 1 • Q * ,
so whether Lemma 11.16 was applied or not, we have σ n (h) = σ n+1 (h). We conclude that

∀h ∈ Q + ∀n ≥ N (|h|) σ n (h) = σ N (|h|) (h) q c 0 q v (p) W ((p, 0)) A p,0 W ((p, 1)) A p,1 R (0) C 1 C 2 q f R ( ( p 1 , 1 -i 1 ) ) R ((p 2 , 1 -i 2 )) R ( ( p 3 , 1 - i3 ) )
. . .

. . .

For each p ∈ AP For each clause C, containing literal p j if i j = 1, or ¬p j if i j = 0 (1 ≤ j ≤ 3)

Figure 11.5 -Reduction of 3SAT to the existence of a cut-off strategy for safety.

This allows us to define the limit strategy, σ ∞ on E ∞ = ∩ n≥0 en(σ n ):

σ ∞ : h ∈ E ∞ → σ N (|h|) (h)
We complete the definition of σ ∞ over Q + in the following way: for each q ∈ Q, let a q = σ(hq) for some fixed hq ∈ E ∞ (or any allowed action otherwise). Then for any h q / ∈ E ∞ , we let σ ∞ (h q) = a q .

We note that any history enabled by σ ∞ is in3 E ∞ : for any run that agrees with σ ∞ , consider its finite set of histories H. Then, for n = max h∈H N (|h|) we have ∀h ∈ H σ n (h) = σ ∞ (h) so the same run also agrees with σ n . This means that en(σ ∞ ) ⊆ en(σ n ) ⊆ E ∞ . In particular, σ ∞ ∈ Safe.

We also check that σ ∞ is in fact memoryless, by showing by induction on h 1 ∈ Q + (with respect to ), that

∀(h 1 , h 2 ) ∈ P ∩ (Q + × E ∞ ) ∀n ≥ N (|h 1 |) σ n (h 1 ) = σ n (h 2 )
If no such pair exists, the result is immediate. Otherwise, after N (|h 1 |), we have enumerated all pairs starting with h 1 as first coordinate. In particular, there exists some maximal h 1 for which Lemma 11.16 was applied for the last time, on (h 1 , h 1 ), hence, h 2 h 1 by maximality (remember h 2 is always enabled). We conclude by induction hypothesis applied on pair

(h 1 , h 2 ): for all n ≥ N (|h 1 |) ≥ N (|h 1 |), σ n (h 1 ) = σ n (h 1 ) = σ n (h 2 ).
Finally, we extract, for each state q, an action σ (q) ∈ Allow(q) ∩ 2 σ ∞ (q) in order to define σ ∈ M , with σ ⊆ σ ∞ . By Lemma 11.11, we conclude that σ ∈ Safe ∩ M .

We conclude our study of safety coverability by the following theorem:

Theorem 11.18. The two following problems are NP-complete: SafeCoverStrat INPUT:

A leaderless non-atomic protocol P, a set of allowed actions Allow and a final state q f ∈ Q. QUESTION: Whether there exists a cut-off strategy σ for safety from U f = {γ | γ(q f ) > 0}.

pSafeCoverStrat defined as SafeCoverStrat for pure strategies only. general simple objective. We conjecture that such generalizations will require memory, even in the safety case, but the existence of sufficient polynomial memory strategies is an open problem.

Finally, little is known about the existence of almost-sure reachability cut-off strategies. First of all, results from Chapter 9 states that the problem is at least PSPACE-hard, even without leader. Moreover, technical difficulties occur when defining a discrete characterization, in the flavour of lemmas 7.22 and 11.9. Indeed, a Pre/Post characterization would require carrying the intermediate memories of each process. Nonetheless, the low complexity classes of strategies for positive reachability and safety objectives raises hopes for the almost-sure reachability question, which somehow combines the previous properties.

Introduction

Les comportements stochastiques fournissent une modélisation d'évènements incertains qui permettent toutefois de fournir des mécanismes de désynchronisation voire pour sortir d'une situations symétriques. Imaginons par exemple que deux amis se retrouvent chaque jour pour jouer au jeu Pierre-Papier-Ciseaux . Les deux joueurs choisissent un symbole parmi , et , et la partie continue tant qu'un vainqueur n'est pas désigné. Comme ces deux amis jouent fréquemment l'un contre l'autre, il leur est possible de retenir la stratégie de leur adversaire. Chaque jour, chaque joueur peut décider de changer d'avis et jouer une autre séquence de symboles pour contrer les actions supposées de son adversaire, ce qui conduit à un changement quotidien de sa stratégie, ou continuer de jouer de la même façon. D'autre part, il leur est nécessaire de se rappeler des coups joués précédemment, en nombre potentiellement élevé. Un jour, un des joueurs décide d'arrêter de tenter de prévoir les coups de son adversaire et lui annonce qu'à chaque tour, il jouera au hasard en lançant un dé : si sa valeur est 1 ou 2, son coup sera , si le dé tombe sur 3 ou 4, et sinon. Conscient de cette stratégie, son adversaire n'a cependant aucune méthode pour s'assurer une victoire avec probabilité supérieure à 1 2 . Jouer au hasard contraint d'une part les deux joueurs à gagner de façon équiprobable, mais assure également que la partie s'achève rapidement en moyenne : on peut calculer un nombre moyen de tours avant victoire égal à 3 2 . Dans cette thèse, nous étudions la vérification formelle et la synthèse de systèmes aux comportements stochastiques. En effet, l'aléa en informatique est un élément clé dans la désynchronisation de processus, pour éviter les situations d'interblocage ou les collisions dans les protocoles de communications. Le concept fondamental dans ce domaine se rapproche ainsi de notre exemple de jeu Pierre-Papier-Ciseaux , présenté en figure 1.1b. De nombreux protocoles s'appuient ainsi sur la randomisation, c'est le cas notamment de CS-MA/CD [CSM00], pour lequel chaque participant choisit aléatoirement le délai avant une réémission pour cause de collision. Ainsi, l'aspect stochastique de ce dernier protocole est fondamental pour son étude [DKN + 12]. Une autre mise en application de la randomisation à des fins de désynchronisation apparaît par exemple dans le protocole distribué Zeroconf, permettant le choix d'adresses sur un lien local [START_REF] Bohnenkamp | Cost-optimization of the IPv4 zeroconf protocol[END_REF][START_REF] Aboba | Dynamic configuration of IPv4 link-local addresses[END_REF].

La randomisation permet également de se prémunir efficacement face à certaines formes d'attaques informatiques : du fait de spécifications ouvertes, de l'engouement pour l'open source, voire des progrès du désassemblage de programmes, il est plus que raisonnable de supposer que le fonctionnement interne de tout programme est publiquement connu, et que sa robustesse doit être assurée sous cette hypothèse. Cependant, certains algorithmes manipulant des structures de données ne possèdent une complexité raisonnable que lorsque les données en entrée suivent une distribution uniforme, et non dans le pire cas. Une absence d'aléa en entrée d'un système de traitement de données peut ainsi avoir des conséquences néfastes si un attaquant est en mesure de déclencher à dessein une exécution dans le pire cas. Pour illustrer ce propos, nous pouvons citer une attaque informatique consistant à exploiter le déterminisme des fonctions de hachage employées dans les langages de programmations modernes : avec peu de puissance de calcul et peu de bande passante, un attaquant a ainsi pu réaliser une attaque par déni de service de serveurs web, en provoquant de nombreuses collisions [START_REF] Crosby | Denial of service via algorithmic complexity attacks[END_REF].

Informatique et théorie des jeux

La théorie des jeux et le raisonnement stratégique constituent une discipline aux résultats prolifiques. Introduits par Von Neumann [START_REF] John Von Neumann | Zur theorie der gesellschaftsspiele[END_REF] grâce à son théorème du min-max, puis davantage formalisé [START_REF] Neumann | Theory of Games and Economic Behavior[END_REF], la modélisation systématique sous forme de jeux s'est révélée fructueuse dans de nombreux domaines, notamment l'économie grâce aux travaux de Nash ou Selten [START_REF] Nash | Equilibrium points in n-person games[END_REF][START_REF] Selten | Spieltheoretische Behandlung eines Oligopolmodells mit Nachfrageträgheit[END_REF], car elle assure une bonne représentation des interactions et comportements entre personnes ou entités, appelés des joueurs ou agents. Ceux-ci prennent des décisions, c'est-à-dire des actions que l'on modélise par des stratégies, prenant en compte leurs observations de l'état du jeu et de l'environnement. Chaque joueur possède un objectif, généralement une fonction d'utilité voire une relation de préférence. L'utilité ou récompense fournie à chaque joueur est déterminée en fonction de la partie effectivement jouée, c'est-à-dire de la façon dont les actions des différents joueurs ont été jouées.

Lorsque l'utilité est fournie en tant que nombre, les jeux sont représentés par des matrices, représentant pour chaque joueur le gain final après le déroulement d'une partie en un coup. Le jeu est donc supposé ponctuel dans le temps. En informatique et comme nous l'avons vu dans les exemples précédents, une composante séquentielle apparaît, que nous modélisons en général grâce aux formalismes issus de la théorie des automates. Par exemple, le problème de Church s'intéresse à la réalisation d'une relation binaire définie par une formule logique sur les séquences infinies de mots. Plus précisément, ce problème soulève la question de l'implémentation d'une telle formule, sous la forme d'un circuit, prenant en entrée un mot infini, et fournissant en sortie un second mot infini en relation avec le premier. La réponse à cette question peut être obtenue par l'analyse d'un jeu à deux joueurs joués sur un graphe, où le premier joueur propose une sortie, en choisissant certaines transitions, tandis que le premier joueur essaye de l'en empêcher, en choisissant des transitions pour le mot d'entrée, mettant en défaut la relation binaire. Implémenter un tel circuit revient dans ce contexte à exhiber une stratégie gagnante pour le premier joueur [START_REF] Escrig | Solving sequential conditions by finite-state strategies[END_REF], ce qui signifie habituellement résoudre le jeu.

Cette méthodologie est particulièrement adaptée à l'étude des systèmes réactifs, pour lesquels nous nous intéressons à la synthèse d'un contrôleur qui devra satisfaire des propriétés logiques tout en restant robuste face à un environnement, dont le comportement est imprévisible [START_REF] Abadi | Realizable and unrealizable specifications of reactive systems[END_REF].

Dans ces exemples, le premier joueur est dit existentiel, car nous nous intéressons à l'existence d'une seule stratégie gagnante, contre n'importe quelle stratégie du second joueur, que nous qualifions d'universel.

Non-déterminisme, aléa et problèmes quantitatifs

Lorsque l'on caractérise la victoire d'un joueur contre toutes les stratégies de son adversaire, nous adoptons une approche dans le pire cas puisque toutes les actions possibles sont considérées. Nous pouvons cependant remarquer que cette forme d'incertitude est très forte car elle considère des scenarios peu crédibles, qui ne se produisent pas forcément en pratique. À titre d'exemple, le modèle de transmission de données précédemment décrit peut vraisemblablement incorporer une composante d'équité entre les différents protagonistes, c'est-à-dire supposer qu'il existe forcément un instant lors duquel il est possible d'émettre sans interférence par aucune autre partie.

Une notion intermédiaire d'incertitude est ainsi obtenue en considérant les comportements stochastiques, que l'on peut représenter comme les actions d'un joueur tierce, l'environnement, qui ne prend pas de décision à proprement parler, mais reste source d'aléa. Contrairement à un joueur universel, l'environnement n'a pas d'objectif propre, et ses actions peuvent contribuer ou s'opposer aux objectifs du joueur existentiel dont nous cherchons une stratégie. Introduire des probabilités dans ce modèle a deux conséquences majeures : premièrement, certaines exécutions envisagées dans le pire cas sont mises de côté. Par exemple, la succession infinie de matchs nuls dans la boucle du jeu de Pierre-Feuille-Ciseaux est impossible, dès lors qu'un des joueurs joue aléatoirement. Cette propriété se rapproche en ce sens de la notion d'équité. D'autre part, les phénomènes stochastiques fournissent un nouvel aspect quantitatif aux problèmes traités. Il devient par exemple possible de se demander si un joueur peut gagner avec une valeur de probabilité supérieure à un certain seuil.

Optimisation de stratégies

Il existe plusieurs concepts d'optimalité pour une stratégie, qui dépendent des conditions de gain considérées. Premièrement, une stratégie optimale conceptualise une situation où un joueur donné doit maximiser son gain, contre toutes les actions possibles de son adversaire. La meilleure valeur atteignable, contre la pire riposte est alors appelée valeur du jeu [START_REF] Shapley | A value for n-person games[END_REF]. Dans ce contexte, le jeu est supposé joué par deux joueurs aux objectifs antagonistes, c'est-àdire à somme nulle. Bien que les valeurs du jeu pour le premier et deuxième joueurs ne soient pas nécessairement égales, de nombreux résultats de déterminisation établissent cette égalité pour de larges classes de jeux, notamment les jeux à tour déterministes, avec des objectifs réguliers [START_REF] Martin | Borel determinacy[END_REF].

Lorsque des actions concurrentes et randomisées sont autorisées, des stratégies optimales peuvent ne pas exister bien que le jeu puisse toujours être déterminisé, au sens où la valeur n'est atteignable qu'asymptotiquement [START_REF] Martin | The determinacy of Blackwell games[END_REF]. Nous pouvons illustrer ce phénomène grâce au jeu à deux joueurs à somme nulle H (hide-or-run courir ou fuir ) représenté par la figure 1.2. Le joueur 1 peut soit se cacher ( ) ou courir à l'abri (r) tandis que le second joueur peut choisir de jeter une boule de neige (s) ou attendre (w). Il a été démontré par [START_REF] Kumar | Existence of value and randomized strategies in zero-sum discrete-time stochastic dynamic games[END_REF] que ce jeu n'admet pas de stratégies optimales, bien que le premier joueur peut s'assurer une victoire avec probabilité arbitrairement proche de 1.

Lorsque le jeu implique plus de deux joueurs, ou qu'une partie des gains sont communs à plusieurs joueurs, nous remarquons que les notions de stratégies optimales et valeurs ne sont plus adaptées : les agents ne jouent plus nécessairement les uns contre les autres mais peuvent coopérer. Partant de cette observation, John Nash introduisit une nouvelle notion, dite d'équilibre, pour traiter les jeux à somme non nulle. Dans un tel équilibre, nous nous donnons une stratégie pour chaque joueur, que nous souhaitons optimale par rapport aux stratégies fixées des autres joueurs. Sur une structure de graphe, un équilibre décrit des coopérations entre les joueurs, pouvant s'accorder pour visiter certains états profitables. Lorsqu'un joueur ne respecte pas sa stratégie initialement prévue par l'équilibre, les autres joueurs peuvent réagir de plusieurs façon, en fonction de la notion d'équilibre choisie : ceux-ci peuvent riposter et abandonner leurs propres objectifs, ou décider de continuer à optimiser leurs propres objectifs malgré la déviation. Dans le premier cas, nous parlerons d'équilibres début de partie. Cela a également pour effet d'assurer des propriétés de régularité (continuité) des fonctions précédemment évoquées et encore une fois assure l'existence d'un équilibre de Nash [START_REF] Fink | Equilibrium in a stochastic n-person game[END_REF].

Le cas d'un horizon infini sans facteur d'atténuation est en général plus complexe à résoudre, même dans le cas d'un seul joueur interagissant avec son environnement au comportement stochastique. Dans ce dernier cas, le jeu peut être vu comme un processus de décision markovien, pour lequel une famille de strategies asymptotiquement optimale existe [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF]. Ce modèle est également étudié dans le cas général de plusieurs joueurs sous le nom de processus de décision markovien compétitifs [START_REF] Filar | Competitive Markov Decision Processes[END_REF].

Les fonctions d'utilité considérées dans cette thèse sont de simples objectifs d'accessibilité et de sureté de certains états, ce qui constitue un cas très particulier d'horizon infini. Ainsi, l'existence d'un équilibre est assurée pour les jeux munis d'objectifs de sûreté pour tous les joueurs, c'est-à-dire que chaque joueur a pour objectif de confiner le jeu à un certain sous-ensemble d'états marqués comme sûrs [START_REF] Secchi | Stay-in-a-set games[END_REF]. Dans le cas de l'accessibilité, l'existence d'un équilibre de Nash reste un problème ouvert, bien qu'une notion relâchée d'équilibre, où chaque joueur peut encore dévier en n'améliorant toutefois son gain que d'une petite valeur ε > 0, reste assurée [START_REF] Chatterjee | On Nash equilibria in stochastic games[END_REF]. Des extensions considérant des classes de gains définies par des langages ω-réguliers sont étudiées, notamment dans le cas de deux joueurs [START_REF] Chatterjee | Two-player nonzero-sum ω-regular games[END_REF], tandis que le cas général reste ouvert. Pour une étude plus précise des résultats connus sur les jeux stochastiques et particulièrement les équilibres de Nash, nous pourrons nous référer à [START_REF] Chatterjee | A survey of stochastic ω-regular games[END_REF].

Lorsque nous retirons la composant stochastique, les objectifs ω-réguliers peuvent être étudiés plus simplement, notamment par l'encodage d'un équilibre de Nash comme une stratégie gagnante dans un nouveau jeu à deux joueurs [START_REF] Brenguier | Nash Equilibria in Concurrent Games -Application to Timed Games[END_REF]. En ne retirant l'aspect concurrent, c'est-à-dire en considérant des jeux à tours, le problème de l'existence d'un équilibre dans le cas stochastique est déjà indécidable [START_REF] Ummels | Stochastic Multiplayer Games: Theory and Algorithms[END_REF]. Plus précisément, l'auteur montre qu'il est impossible de décider si un jeu impliquant 11 joueurs ou plus, avec des objectifs d'accessibilité terminaux, admet un équilibre où un joueur désigné gagne avec probabilité 1. Toutefois, si l'on se restreint à des stratégies sans mémoire, le problème peut être résolu en espace polynomial, même en imposant des conditions plus fortes sur les gains finaux des joueurs. Ceci fournit donc une méthode de calcul de la valeur moyenne d'un équilibre, pour chaque joueur. Nous pouvons aussi comparer ce résultat au problème du calcul d'un équilibre de Nash dans le cas d'un jeu à un tour, c'est-à-dire décrit par une matrice, qui est PPAD-complet même en présence de deux joueurs [START_REF] Chen | Settling the complexity of computing two-player nash equilibria[END_REF]. Notons cependant que ces études de restrictions sans stochasticité ou sans actions concurrentes se concentrent sur des problèmes de décisions et de calcul, car l'existence d'un équilibre n'est pas assurée. On rappelle en effet que ces deux ingrédients sont cruciaux pour la preuve d'existence.

Nous avons vu que la complexité de l'étude d'un système dépend du nombre considéré de joueurs, ou agents, que l'on peut voir comme un paramètre variable. La vérification de systèmes paramétrés constitue ainsi un autre axe d'étude, introduit notamment par German et Sistla [START_REF] German | Reasoning about systems with many processes[END_REF]. En général, les interactions entre agents peuvent être modélisées par des systèmes d'addition de vecteurs ou, de façon équivalente, par des réseaux de Petri, où chaque agent est représenté par un jeton. Accéder à un état de contrôle particulier par un agent signifie qu'un jeton a atteint une place particulière du réseau, ce qui correspond à une propriété de couverture. Le problème de décision correspondant à la couverture a été intensivement étudié par [START_REF] Rackoff | The covering and boundedness problems for vector addition systems[END_REF][START_REF] Lipton | The reachability problem requires exponential space[END_REF] qui l'établissent EXPSPACE-complet. Nous nous rendons ainsi compte que le problème de la vérification de systèmes paramétrés est un problème déjà difficile dans le cas non-stochastique. Cependant, des travaux sur des restrictions tels que les protocoles de population [START_REF] Esparza | Verification of Population Protocols[END_REF], incluant des notions d'équité, nous montrent que l'ajout d'aspects stochastiques peut simplifier certains problèmes de décision. D'autre part, les réseaux de Petri et les protocoles de population restent des modèles très généraux, que l'on peut restreindre, en fonction des primitives de communications que l'on s'autorise. Une discussion sur le choix de ces primitives est présentées par [START_REF] Esparza | Keeping a crowd safe: On the complexity of parameterized verification (invited talk)[END_REF], et nous prenons ici un des modèles les plus faibles, consistant en la communication au travers d'un registre partagé entre les agents, via des écritures non-atomiques. Ce modèle, précédemment étudiés par [START_REF] Esparza | Parameterized verification of asynchronous shared-memory systems[END_REF][START_REF] Durand-Gasselin | Model checking parameterized asynchronous shared-memory systems[END_REF] dans le cas non-stochastique, se rapproche fortement des réseaux de diffusion avec reconfiguration [START_REF] Bertrand | Playing with probabilities in reconfigurable broadcast networks[END_REF] pour lesquels des raisonnements stratégiques peuvent même être appliqués avec succès [START_REF] Bertrand | Distributed Local Strategies in Broadcast Networks[END_REF].

Publications scientifiques

Les résultats présentés dans ce manuscrit complètent des publications auxquelles l'auteur de cette thèse a pris part : Le chapitre 4 décrit le problème d'indécidabilité de l'existence d'un équilibre, publié dans [START_REF] Bouyer | Mixed Nash equilibria in concurrent games[END_REF], et discute le choix crucial d'un modèle de jeu où les joueurs n'observent pas les actions jouées. Le chapitre suivant 5 présente l'existence et le calcul d'une notion relâchée d'équilibre, tels que présentés dans [START_REF] Bouyer | Stochastic equilibria under imprecise deviations in terminal-reward concurrent games[END_REF]. Enfin, la seconde partie de la thèse étend [BMR + 16], dont le théorème principal est repris en chapitre 9. Les principales extensions concernent l'étude d'objectifs plus généraux, mais aussi la présence de plusieurs registres en chapitre 10.

Partie 1 : Jeux stochastiques

La première partie du manuscrit s'intéresse à l'étude des jeux stochastiques concurrents dans lesquels les joueurs sont en nombre fixé et possèdent des objectifs d'accessibilité terminale ou de sûreté.

Cadre théorique

Le chapitre 3 fournit une définition formelle des jeux considérés : un jeu concurrent G fait intervenir un ensemble fini Agt de joueurs, ou agents. L'ensemble d'états du jeu est noté States, est supposé fini, et possède un élément distingué s 0 à partir duquel toute partie est initiée. La partie continue lorsqu'à chaque tour, les joueurs choisissent chacun une action parmi un ensemble fini Act, ce qui induit une transition vers un nouvel état. Chaque joueur agit conformément à une stratégie, notée σ i pour le joueur i ∈ Agt, correspondant à une fonction associant à tout mot h ∈ States + une décision. Ainsi, h représente un historique, c'est-à-dire la séquence finie d'états visités depuis le début de la partie, en s 0 , jusqu'à l'état courant depuis lequel une décision est prise par chaque joueur. Ici, les décisions des joueurs sont des distributions de probabilité sur l'ensemble des actions possibles, ce qui signifie que le prochain état est choisi de manière probabiliste. Lorsque σ i est donné pour tout agent i ∈ Agt, on note σ le un profil de stratégies, qui nous permet alors de définir une loi de probabilité notée P σ sur l'ensemble des parties possibles. Le choix d'une stratégie σ i par un joueur i se fait dans l'optique de l'optimisation de sa fonction de gain, qui lui est propre. Dans cette thèse, nous considérons principalement des fonctions de gain, ou objectifs, dits d'accessibilité terminale, c'est-à-dire que le gain de chaque joueur est un nombre réel déterminé par l'état final dans lequel la partie s'est achevée. Lorsque la partie continue indéfiniment, le gain associé est toujours nul. De manière symétrique, on parle d'objectif de sûreté lorsque le joueur considéré a pour objectif de maintenir la partie dans un sous-ensemble d'état donné.

Formellement, une fonction de gain φ associe à tout mot fini ou infini dans States + States ω un nombre réel. Dans le cas des deux types de fonctions (accessibilité terminale et sûreté), on démontre aisément qu'une telle fonction est mesurable, ce qui permet de définir son espérance E σ (φ) pour toute loi de probabilité P σ où σ est une stratégie.

Un jeu possède un profil d'objectifs Φ, c'est-à-dire une fonction d'objectif Φ i pour chaque joueur i. Celui-ci cherche à maximiser cette fonction, c'est-à-dire la quantité E σ (Φ i ) malgré les interactions avec les autres joueurs.

Deux cas de figures se dégagent :

• Lorsque seuls deux joueurs interviennent et que la somme Φ 1 + Φ 2 = 0 est nulle, tout gain acquis par un joueur constitue une perte pour l'autre joueur. Sous des hypothèses satisfaites ici, on démontre (voir [START_REF] Kumar | Existence of value and randomized strategies in zero-sum discrete-time stochastic dynamic games[END_REF]) que le jeu est déterminé, c'est-à-dire :

ν 1 (s) = inf σ 2 ∈S 2 sup σ 1 ∈S 1 E (σ 1 ,σ 2 ) (Φ 1 | s) = ν 1 (s) = sup σ 1 ∈S 1 inf σ 2 ∈S 2 E (σ 1 ,σ 2 ) (Φ 1 | s)
De plus, ces valeurs sont atteintes asymptotiquement par des stratégies ne dépendant que de l'état courant, c'est-à-dire stationnaires. Elles ne sont pas nécessairement atteintes de manière exacte, comme le montre l'exemple de la figure 1.2.

• Lorsque plus de deux joueurs interviennent, ou que la somme de leur gain n'est plus constante, de nouveaux concepts d'optimalité ont été proposés notamment par Nash. Un profil de stratégie σ forme ainsi un équilibre de Nash (cf [START_REF] Nash | Equilibrium points in n-person games[END_REF]) lorsque :

∀i ∈ Agt ∀σ i E σ[i/σ i ] (Φ i | s) ≤ E σ (Φ i | s)
où σ[i/σ i ] désigne un nouveau profil de stratégie où le joueur i joue désormais la stratégie σ i .

Il est important de mentionner que la plupart des résultats établis dans la littérature considèrent des jeux où les actions sont observables, c'est-à-dire que la stratégie est une fonction définie sur des historiques de la forme (s 0 , A 0 , s 1 , A 1 , . . . s k ) où chaque A i correspond au profil d'actions choisi conjointement par les joueurs depuis l'état s i menant à l'état s i+1 . Dans cette thèse, les actions ne sont pas supposées observables (seule la séquence (s 0 , s 1 . . . s k ) ∈ States + est fournie à la stratégie), ce qui nous mène à montrer dans un premier temps que les théorèmes de déterminisation précédents peuvent être étendus à ce nouveau cadre.

Indécidabilité de l'existence d'un équilibre de Nash L'existence d'équilibre est assurée dans le cas de jeux se terminant en un tour, dès lors que l'on autorise des stratégies randomisées, grâce au théorème de Nash. Dans le cas de jeux sur des graphes, l'existence de stratégies optimales n'est pas assurée dans les jeux à somme nulle, ce qui implique a fortiori qu'un équilibre de Nash n'existe pas nécessairement.

Le problème de décision consiste ainsi à déterminer si, étant donné un jeu stochastique fourni en entrée, il existe une profil de stratégie formant un équilibre de Nash. Le chapitre 4 répond à cette question de manière négative : le problème est montré indécidable dès lors que • µ ∈ N Qc est un multiensemble des états des processus contributeurs, décrits par le second automate (q c , Q c , T c ).

À chaque instant, l'ordonnanceur choisit un processus parmi le leader et les différents contributeurs et réalise une transition, mettant à jour l'état q ou le multiensemble µ, résultant en une nouvelle configuration γ . On note généralement γ → γ lorsqu'une telle transition est possible de la configuration γ vers γ , ou de manière équivalente γ ∈ Post(γ) voire γ ∈ Pre(γ ).

Objectifs et régularité

On remarque aisément que l'ajout d'un ou plusieurs processus contributeurs n'altère pas pas la validité d'un chemin : si (q, µ, d) → * (q , µ , d ) et q a ∈ Q c est un état contributeur tel que µ(q a ) > 0, alors il existe q a ∈ Q c tel que µ (q a ) > 0 et (q, µ ⊕ q a , d) → * (q , µ ⊕ q a , d ). Ce fait est présenté dans le lemme de copie 7.18 et la figure 7.2 et justifie l'étude des opérations non atomiques. La principale question adressée dans cette seconde partie est l'accessibilité d'une configuration recouvrant un état fixé q f ∈ Q c , c'est-à-dire que l'on cherche à savoir si q f est atteint par n'importe lequel des processus contributeurs. L'ensemble de configurations est ainsi U f = {(q, µ, d) | µ(q f ) > 0}. On remarque que l'ajout de processus contributeurs supplémentaires ne nuit pas à l'accessibilité, puisqu'on peut choisir de conserver ces processus supplémentaires inactifs, et toujours atteindre U f . Un autre objectif traité dans cette thèse est le consensus, où l'on requiert désormais que tous les processus atteignent q f au même instant, c'est-à-dire U f = {(q, µ, d) | ∀q = q f µ(q ) = 0}.

On introduit la relation d'ordre définie pour tout couple de configuration (q, µ, d) et (q , µ , d ) par (q, µ, d) (q , µ , d ) si, et seulement si, q = q , d = d et pour tout q a , soit µ(q a ) = µ (q a ) = 0, soit 0 < µ(q a ) ≤ µ (q a ). Munis de cette relation d'ordre, les deux objectifs précédents sont exprimables aisément, car clos par le haut pour la relation , qui est un bel ordre. Cela signifie en particulier que les ensembles U f considérés sont engendrés par la clôture d'un nombre fini d'éléments minimaux. Dans les deux cas de la couverture et du consensus, ces éléments minimaux sont même de petite taille : si (q, µ, d) ∈ min U f , alors ∀q µ(q ) ∈ {0, 1}. Ces objectifs seront nommés dans la suite simple et la plupart des résultats s'appliqueront ainsi De la même manière, nous définissons l'ensemble des configurations initiales par U 0 = ↑ (q l , q c , d 0 ).

Ordonnanceur

Deux cas de figures d'ordonnanceurs sont considérés :

• Un ordonnanceur non déterministe : existe-t-il un chemin d'une configuration initiale γ 0 ∈ U 0 vers une configuration finale γ f ∈ U f .

• Un ordonnanceur probabiliste : étant donné une configuration initiale γ 0 ∈ U 0 et en considérant des transitions choisies de manière probabiliste, quelle est la probabilité d'atteindre U f .

Nous retenons ici la seconde approche, d'un point de vue qualitatif, en évaluant uniquement si la probabilité d'accessibilité est nulle, entre 0 et 1, ou égale à 1. En effet, dans ce cadre plus restreint, la question de l'accessibilité probabiliste ne dépend pas des valeurs de probabilités exactes choisies par l'ordonnanceur.

Analyse non déterministe

Le chapitre 8 s'intéresse à l'accessibilité avec probabilité strictement positive. On remarque premièrement qu'une telle propriété équivaut à l'existence d'un chemin avec ordonnanceur non déterministe. Nous revisitons ainsi des résultats présentés dans [START_REF] Esparza | Parameterized verification of asynchronous shared-memory systems[END_REF]. En particulier, nous montrons que tout chemin entre U 0 et U f peut être compressé en un nombre polynomial de transitions et impliquant donc un faible nombre de processus. Ceci est réalisé à l'aide d'un graphe symbolique, ne gardant pas en mémoire le nombre exact de processus contributeurs, mais uniquement l'ensemble des états activés par au moins un contributeur. Ce graphe a ainsi de bonnes propriétés de compatibilité avec la relation . D'autre part, nous rappelons une précédente preuve de réduction du problème 3SAT vers l'accessibilité dans un réseau, ce qui démontre que le problème d'accessibilité avec probabilité strictement positive est NP-complet.

Analyse probabiliste

Le problème de l'accessibilité presque sûre, c'est-à-dire avec probabilité 1, est traité dans le chapitre 9. Contrairement au cas précédent, l'ajout de nouveaux processus peut débloquer de nouveaux comportements qui a leur tour, pourraient empêcher l'accessibilité avec probabilité 1.

Valeur seuil

Cette nouvelle analyse repose donc sur un nouveau concept de valeur seuil, à partir de laquelle le paramètre fixant le nombre de contributeurs impose toujours une accessibilité presque sûre ou à l'inverse, impose toujours une accessibilité avec probabilité strictement inférieure à 1.

Formellement, en notant P n (♦U f ) la probabilité d'atteindre U f à partir d'une configuration initiale à n contributeurs, on montre que la suite P n (♦U f ) à valeur dans {0, 1} admet une limite, c'est-à-dire qu'il existe un entier N tel que :

• ou bien pour tout n ≥ N , P n (♦U f ) = 1 : N est alors une valeur seuil positive ;

• ou bien pour tout n ≥ N , P n (♦U f ) < 1 : N est alors une valeur seuil négative ; La preuve d'existence s'appuie sur l'analyse d'inclusion d'ensemble d'accessibilité et de coaccessibilité, qui est facilité par la structure de bel ordre de .

Bornes de complexité

Une fois l'existence d'une valeur seuil établie, il reste à déterminer s'il s'agit d'un seuil positif ou négatif, répondant ainsi à une analyse globale du problème. En effet, une valeur seuil fournit une indication sur le comportement limite d'un protocole. Afin de répondre à cette question, il suffit d'analyser notre système pour un paramètre étant une valeur seuil, que l'on veut donc la plus petite possible. Nous nous intéressons ainsi à des exemples de protocole dont la valeur seuil minimale est importante, ce qui fournit des bornes inférieures • Nous démontrons premièrement qu'il existe des protocoles dont la valeur seuil minimale est linéaire en la taille du protocole.

• À l'aide de ce premier résultat, il est possible de générer un nouveau protocole dont la valeur seuil est négative, et exponentielle en la taille du protocole.

• Cet exemple est ensuite généralisé, pour réaliser l'encodage d'une machine de Turing linéairement bornée. Un seuil négatif dans le protocole obtenu devient alors équivalent à l'arrêt de la machine de Turing. Le problème de décision d'un seuil négatif, puis a fortiori, d'un seuil positif, est donc PSPACE-difficile.

De manière orthogonale, nous développons une technique générale pour borner la taille d'une valeur seuil par une quantité doublement exponentielle en la taille du protocole. Ce résultat généralise ainsi les bornes doublement exponentielles de Rackoff [START_REF] Rackoff | The covering and boundedness problems for vector addition systems[END_REF] pour la couverture dans les réseaux de Petri. Ceci fournit d'autre part un algorithme de décision en espace (simplement) exponentiel, grâce au théorème de Savitch [START_REF] Sipser | Introduction to the theory of computation[END_REF].

Extensions

Des extensions sont présentées sont dans le chapitre 10 :

• Le model checking presque sûr d'un protocole, par rapport à une formule LTL ;

• La gestion d'un nombre fini de registres partagés, en introduisant des opérations de copie d'un registre à un autre ;

• La gestion d'identifiant de processus (pid), tant que la monotonicité est préservée.

Nous indiquons comment les techniques précédentes peuvent être adaptées, et dans les deux premiers cas, nous fournissons des bornes doublement exponentielles ainsi qu'un algorithme de décision en espace exponentiel.

Stratégies locales

Le dernier chapitre 11 introduit alors le concept de stratégies issues de la première partie de la thèse, et montre que certains problèmes de synthèse sont décidables, sous certaines hypothèses :

• Nous considérons uniquement l'accessibilité avec probabilité strictement positive, ou strictement inférieure à 1,

• pour des objectifs de couverture,

• avec des stratégies locales, c'est-à-dire n'ayant pour observation que la séquence d'état localement visités.

Dans les deux cas étudiés, accessibilité avec probabilité strictement supérieure à 0 ou strictement inférieure à 1, la mémoire requise par une stratégie locale peut être limitée à une quantité polynomiale, ce qui signifie encore une fois que les problèmes correspondant sont NP-complets.

Figure 1 . 1 - 1 1. 1

 1111 Figure 1.1 -Simple games that require randomization

Figure 1 . 2 -

 12 Figure 1.2 -H has no optimal strategy

Figure 3 .

 3 Figure 3.2 -A shifted version of game of Figure 1.2 on page 3.

Figure 4 . 1 -

 41 Figure 4.1 -Representations of a one-shot game

Figure 4 .

 4 Figure 4.2 -Matching-pennies game

Example 4 . 4 .

 44 As an intuition of the construction, let us consider the games of Figure 4.3.

Figure 4 . 4 -

 44 Figure 4.4 -C is not action-visible whereas C is and has another equilibrium. This second can be replaced by a single node which has the same Nash equilibria.

  ) Associated equilibrium

Figure 4 . 5 -

 45 Figure 4.5 -Computation of an additional equilibrium payoff in game C .

Figure 4 . 6 -

 46 Figure 4.6 -The rescale game R k for k ∈ {1, 2, 3} and l = k -1

Figure 4 . 8 -

 48 Figure 4.8 -The modules C k (for k ≥ 2) and D.Notice that state s 2 should be considered terminal, as it only carries a self-loop. We could replace it by a two-state loop. We could also see it as a terminal state with reward (0, 0, 0), but for technical reasons (in Section 4.3.2), we want the terminal rewards of players 1 and 2 to always sum to 8, which we could not achieve easily in this case.

Lemma 4 .

 4 16. Consider the games of Figure 4.8. For n ∈ N {+∞}, we define

Figure 4 . 9 -

 49 Figure 4.9 -Description of the encoding G M . Modules are R k and T are copied several times. n = q means the terminal state n of a module is replaced by further encoded state q.

  the players should follow the strategy described in Proposition 4.15, and in C 4-k , they should follow the strategies described in the proof of Lemma 4.16, for the correct values of the counters;

Figure 4 .

 4 Figure 4.10 -A game that has a Nash equilibrium if, and only if, G has a 1-maximal Nash equilibrium

Figure 4 . 11 -

 411 Figure 4.11 -Transformation of a terminal node (x, y, 8y) with an intermediate node v x,y .The table on the right gives the value of E y 8 for some values of y (notice that E y 8 ⊆ E y 8 when y ≤ y , so that for instance E 4 8 = 32).

Figure 5 . 1 -

 51 Figure 5.1 -The first player to quit the cycle loses

Figure 5 . 2 -

 52 Figure 5.2 -Example of an arena with some exiting actions

  Simple (ŝ, I) node with I interval of [0, 1] played by î, ensuring any distribution pTab(s, a) + (1p)Tab(s, b) for any p ∈ I, hence a deviation range I.s(ŝ, [0, ε]) (ŝ, [1ε, 1]) (ŝ, [1 -2ε, 1]) (ŝ, [0, 2ε])(b) Replaced node s, where i can choose between 4 ranges of probabilities to play action a.

Figure 5 . 3 -Figure 5 . 4 -

 5354 Figure 5.3 -Translation of a node s with initial allowed actions a and b.

Figure 6 .

 6 Figure 6.2 -Underlying (simplified) automaton for one thread of program of Figure 6.1. Each possible state of the thread is modeled by a node containing the current value of local variable i and current line number.

Definition 7 . 1 (

 71 Operations over a set). Let D be a set. We denote by Op (D) the set of register operations, which are partial functions from D to D.The exact definition domain of an operationf ∈ Op (D) is denoted by dom (f) ⊆ D. Given two operations f, f ∈ Op (D), the sequential composition f•f is defined on dom (f • f ) = dom (f) ∩ f -1 (dom (f )) by d → f (f(d)).For d ∈ D, W (d) and R (d) represent the following particular partial functions: W (d) : dom (W (d)) = D d → d R (d) : dom (R (d)) = {d} d → d We denote by Op R,W (D) the set of such register operations and Op R,W (D) * its closure by sequential composition.

  a finite set of transitions of the leader; 79 7.1. NON-DETERMINISTIC TRANSITION SYSTEM

Definition 7 . 11 .

 711 Let A ⊆ Γ, the upward closure ↑A is defined by ↑A = {γ | ∃γ ∈ A γ γ}.A is called upward closed when A = ↑A.

  d)} which are not upward closed sets. However, when minimal elements of an upward closed set A are minimal among elements of Γ, set subtraction by A preserves upward closed sets. Definition 7.23. A reachability set U f is called simple whenever it is upward closed and min U f ⊆ min Γ

Figure 7 . 4 -

 74 Figure 7.4 -Cut-off illustration for a property depending on parameter n: a cut-off value ensures that the property is always true or always false for bigger parameters.

Figure 8 . 2 -

 82 Figure 8.2 -Proof scheme of Lemma 8.4, red arrows → correspond to contributor transitions whereas blue arrows → correspond to leader transitions.

Figure 9 . 1 -

 91 Figure 9.1 -Example of a register protocol with atomic read/write operations.

Theorem 9 .

 9 19. The following problem is in EXPSPACE: ASConsensusCutOff INPUT:

  [1, r] to[1, r ]. Let us denote with Z ⊆ Op R,W,M D r the set of available move operations. Z can be seen as a graph, with node set [1, r ] and edges (i,j) ∈ [1, r ]×[1, r] for any M i→j ∈ Z.We consider the strongly connected components of Z. Inside such a strongly connected component C, whenever an internal edge is used, it is erasing an internal value. Moreover, [C] -1 values can be preserved inside C by moving them sequentially along a cycle of C. One sequential move (shift) of all values requires [C] -1 operations, hence [C] ( [C] -1) to perform a complete cycle of the values. For

  Definition 4.3. Let G = (A, s, φ) be a terminal-reachability game. A sub-arena B of arena A (and game G) denoted B ⊆ A (or B ⊆ G) is an arena B = (States B , Agt, Act, Tab, Allow B ), with a subset of states States B but the same set of agents, actions and transition function, such that for all s ∈ States B , we have

  Example of arenas, with differentiated states for a sub-arena composed of set of states {s i , t i | i ∈ {0, 1}}. Only s 0 and s 1 are marked, as internal states. Notice the only internal transition in the left arena is from s 0 to s 1 , which is action-visible. In the second arena, we exit immediately the arena when entering s 0 by hiding transitions that could have occur through s 1 .

	E 2
	2 )( †c 1 )	E 2
	(bc 1 )( †c 1 ), (bc 2 )( †c 2 )
	. . .	. . .
	Figure 4.3 -

Table 4 .

 4

2 -Two-player projections of R k in r 0 assuming player 3 (left), resp. player 2 (right), plays c almost-surely (1, 4 + y, 4y). Applying Lemma 4.1 twice, we get (

Table 4 .

 4 3 -Summary of the decidability status of exact Nash Equilibria in games with several agents with qualitative reachability or safety objectives.

	R	S	0	1	≥ 2
		0		Ensured (Stay-in-a-set games)
		1	Ensured (MDP)	??
		≥ 2	??	Undecidable

player i has a qualitative safety objective: there exists G i ⊆ States such that Φ i (r) = 1 r∈G ω i .

  min ν and Φ ν coincide only on runs that reach a final state since φ assigns positive valuemin ν to non-terminating runs. A possible work-around is to first compute the cyclefree arena A and exiting conditions ∆ ε , whose size is bounded by the number of pairs (a, i, s) ∈ Act × Agt × States. Then we can apply the previous theorem on game A, Φ ν with the extra formula σ ∈ ∆ ε . Thanks to this last constraint, we ensure that the run always terminates, thus the payoff functions coincide. Finally we conclude the computation by applying Proposition 5.3 to get back an equilibrium on G.

	Chapter 6	Part II
	Parametrized Stochastic Systems
		71

In fact, this arena can model a system of three agents that try to emit at the same time, inducing interferences, but where agent 1 has higher priority (or more transmission power). Probability of a collision when agents

and

are emitting simultaneously is denoted by p. The terminal states correspond to situations where at least player 2 or 3 have successfully transmitted their own message.

δ and m0 induce a general transition function δ * defined for any m ∈ M by δ * (ε, m) = m0 and for any hq ∈ States + , δ * (m, hq) = δ(δ * (m, h), q)

by encoding inside each state the set of already visited states

That is to say ∀s ∈ States, ∀A ∈ Act Agt , p((s, A)) = s.

A strategy requires counting memory whenever it depends on the current state and the length of the history, this is a particular case of infinite memory.

The strategy is memoryless in the composed game, hence, may require memory in the original game.

Remember that a move operation may involve two different registers, hence a factor

Remember that for a transition t = (q, f, q ) ∈ A to be valid, the current register value d must be in dom (f).

In the context of lossy broadcast messaging, each local strategy chooses which message to send, and which transition to take when receiving a message. This corresponds to actions of the form {(q, W (d) , q )} and {(q, R (d1) , q1) . . . (q, R (dn) , qn)} with di all different.

Note the inclusion may be strict, as we may have en(σ ∞ ) = E ∞ .

Remerciements

SCIENTIFIC PUBLICATIONS

Due to the lack of completeness (see Remark 7.4), there might exist configurations γ such that Post({γ}) = ∅, which is not a possible probability distribution support. We address this issue by defining p only over [→] seen a subset of Γ 2 . Definition 7.20 (Induced P by p). We consider a probabilistic transition system (Γ, p). For any sequence of configurations π ∈ Γ + , we define P(π) by induction on π:

• For any γ ∈ Γ, P(γ) = 1;

• For any π ∈ Γ * , γ, γ ∈ Γ, P(π

Intuitively, P(π) denotes the probability that any maximal run starting from π [START_REF]1 Simple games that require randomization[END_REF] has a finite prefix π. Notice that because of possible deadlocks, a configuration γ may not have any successor, which means Post({γ}) = ∅ and any run visiting γ is of finite length. Thanks to Carathéodory's criterion and in particular Theorem 2.1, we show that for any γ ∈ Γ, P(γ -) is a probability measure over maximal runs of (Γ, →).

Qualitative analysis

From the definition of p and P, we conclude that P(π) > 0 if, and only if, π ∈ paths (→), which means the exact probability values don't matter for positive probability property. We will continue the study of our network protocols by giving several properties that do not depend on the exact probability values, i.e. qualitative properties.

Remember first of all we are interested in reachability properties, which we define now in our stochastic context: Definition 7.21. Let n ∈ N and Π ⊆ paths (→) a measurable set of paths. We define P n (Π) = P(Π ∩ U 0 • Γ * n ) the probability of achieving a path from Π with n contributors. Let us consider A ⊆ Γ, we denote by ♦A = (Γ\A) * • A the set of finite paths eventually visiting A.

Notice that this definition is sound: ♦A = (Γ\A) * • A is always measurable, as a countable union of finite prefixes. For a given n ∈ N, the paths in U 0 • Γ * n are in fact of the form {γ n 0 } • Γ * n with γ n 0 = (q l , n • q c , d 0 ), which means P n actually defines a probability measure. We now remark that deciding eventual reachability with probability 0, positive, or 1 are all three qualitative properties that can be expressed without mentioning the exact probabilistic transition system. Lemma 7.22. Let A ⊆ Γ. Both properties P n (♦A) = 1 and P n (♦A) = 1 do not depend on the actual values of p. Moreover, we have the following characterizations.

Proof. We will describe here the proof for the second property P n (♦A) = 1, the other case is similar. First of all, notice that P n (♦A) = 1 ⇔ P n (♦A) = 1. We then prove the equivalence:

if, and only if, (q, µ 1 ⊕ µ 2 , d) → (q , µ 1 ⊕ µ 2 , d ) in P Definition 9.13 (Fixed subset). Let A ⊆ Γ, we define

As we can expect, for any upward closed set of configurations U of P, U k is still upward closed, although the number of minimal elements undergo an exponential blow up. Note however that minimal elements have similar sizes:

Thanks to Lemma 9.12, we also infer that Pre * (U )

Symbolic based algorithm

The main result of this section consists in showing that for a parameter large enough, the almost-sure reachability cut-off is preserved in the symbolic graph of the fixed protocol. More precisely, the parameter value

Notice that this quantity is polynomial in the size of the network and in the size of the predecessor set. Surprisingly, this quantity is independant of the sucessor set itself. Lemma 9.14. Assume P is monotonous. There is a negative cut-off for almost-sure reachability of U f in P, if, and only if, there is a node in the symbolic graph G P K that is reachable from ((q l , K • q c ), {q c }, d 0 ) but from which U f K is not reachable.

Proof. ⇐ We apply Theorem 9.1 to show that P K has no positive cut-off. Thanks to Corollary. 9.4, a negative cut-off should then exist for P K . We infer by translation Lemma 9.12 that P has also a negative cut-off.

⇒ Let N be a negative cut-off for almost-sure reachability of U f in P. We can assume without loss of generality that N ≥ K, and consider γ = (q, µ, d)

γ and because of support equality, there exists

and µ 3 contains the remaining N -K ≥ 1 states neither taken in µ 1 nor in µ 2 . Consider the configuration ((q, µ 1 ⊕ µ 2 ), µ 3 , d) in P K , which is reachable from U 0 K by translation Lemma 9.12, then its support s = ((q, µ 1 ⊕ µ 2 ), µ 3 , d) is reachable in G P K from ((q l , K • q c ), {q c }, d 0 ). Assume that sE * U f K in the symbolic graph to expose a contradiction. Then, there exists µ 3 such that

(a) r = n registers, Z = {(i, (i mod n) + 1)}, and desired mapping m : i → min(n, i + 1). Only value 1 can be erased. We have to shift all values to the right n -1 times. Each shift requires n -1 move operations, hence the whole mapping requires (n -1) 2 move operations.

(b) r = 2n and n additional write operations (pseudo registers 2n + 1 to 3n). Because of the middle component cycle, that has to keep n -2 of its initial values, each assignment i ∈ [n + 1, 2n] → i + n will require a first write inside the cycle, then n -1 shifts (with n -1 operation each), hence a global cost of at least n(n -1) 2 ∼ r 3 /8 ∈ Ω(r 3 ) operations.

we label the node with the desired value for the mapping by (m(i)). We omit the label when m(i) = i.

Positive probability

Almost-sure Characterization As discussed in Section 10.1, these results can easily be transposed to the almost-sure repeated reachability case in order to explore protocol model checking. As discussed in Section 10.3, similar results can be obtained when adding the ability to manipulate process identifiers inside registers, though the concrete study of this model is left as further work. We can also notice a complexity gap between our EXPSPACE procedure and the PSPACEhardness of the decision problem. In terms of tight cut-offs, this corresponds to a gap between the worst known case of an exponential (negative) cut-off and the general doubly exponential bounds on cut-off extracted from the general bounding scheme from Lemma 9.17.

As testified by our several constructions and experiments, we haven't managed to build a cut-off, or minimal elements (in the Pre * (U f )) bigger than a simple exponential. A reasonable conjecture of a simple exponential bound on |Pre * (U f )| would close this complexity gap. As a matter of fact, we saw in Theorem 9.16 that our procedure is based on the refined symbolic graph of index [Q c ] × |Pre * (U f )| whose exploration can be done in NLOGSPACE in its size. With such a conjecture, this graph would be of exponential size (instead of doubly exponential), thus giving a PSPACE procedure.

Another related further work is the study of protocols composed of pushdown systems, as considered in [START_REF] Esparza | Parameterized verification of asynchronous shared-memory systems[END_REF]. In their work, the authors show that in the non-deterministic scheduler setting (positive reachability probability), the reachability problem is NP-complete if either the leader or the contributor is a pushdown machine, and becomes PSPACE-complete if both leader and contributor are pushdown machine.

Figure 11.1 -Example of a leaderless non-atomic protocol, with two allowed actions from state q 0 : A 1 = {(q 0 , R (0) , q 0 ), (q 0 , R (0) , q 1 ), (q 0 , R (1) , q 0 )} and A 2 = {(q 0 , R (0) , q ⊥ ), (q 0 , R (1) , q f )}

Each of these three choices is made by the scheduler, at random. Again, we argue that exact probability values are irrelevant and that we can focus directly on the set of transitions {t ∈ A | A ∈ δ} a strategy can provide to the scheduler: Definition 11.2 (Local strategies for a protocol). A mapping σ : Q + → 2 T is a local strategy for P and Allow if for all h ∈ Q + , σ(h) = A∈δ A for some non-empty set δ ⊆ Allow(last(q)). We denote by S the set of local strategies for P and Allow, or strategies for short when P and Allow are clear from the context.

We can then define sub-classes of strategies as in Definition 3.3. Definition 11.3. σ is pure, when for all h ∈ Q + , σ(h) ∈ Allow(last(q)). We define the following classes:

• S ⊆ S the class of pure strategies;

• M(k) for any k ∈ N the class of strategies requiring a memory of size k;

• M = M(0) the class of memoryless strategies;

• We also define their pure counterpart:

Remark 11.4. For any q ∈ Q, Act(q) is stable by union, so Act(q) also contains all randomized actions. However, they may not be allowed, depending on the actual choice of Allow function and the strategy class we are considering. When for all state q ∈ Q, Allow(q) is stable by union, pure and randomized classes coincide. Indeed, in the qualitative context, a randomized strategy only picks the support of action distributions.

Remark 11.5. As for our stochastic games, agents do not see played actions inside the history, nonetheless the protocol P can be modified to encode this information inside the history. Moreover, agents do not see the register value before playing, as it would break monotonicity of the resulting protocol.

Agents neither see past register values, though this information can be encoded in the current state, as the past transitions. Indeed, we can transform each transition t = (q, R (d) , q ) (resp. t = (q, W (d) , q ) ) in several transitions t = ((q, d ), R (d) , (q , d)) (resp. t = ((q, d ), W (d) , (q , d))) for all possible d ∈ D.

Semantics

Once we are given a local strategy σ, we are able to build runs of our protocol that satisfy the local strategy. Notice that agents have partial observation, since they are given only the history about their own visited states along the run. A formal definition of such runs is developed below. Definition 11.6. Let σ a strategy, and π ∈ paths (→), we say that π agrees with σ if there exists ρ ∈ (N Q + ) |π| , a sequence of multisets of histories, such that

We will denote with agree (σ) the set of paths that agree with σ, and ρ will be called the corresponding sequence of π and σ.

Intuitively, we attach to the finite run π a sequence ρ of the same length that stores at any time, the growing histories for each process. Note that these histories have various lengths, depending on how much a single process has already moved. At a given time i ∈ [1, |π|], transition t = (q, f, q ) is taken if one of the processes in state q has an history h • q for which strategy σ allows t. If this condition holds, the transition is triggered, so ρ[i + 1] is defined from ρ[i] by replacing one occurrence of h • q by the new history h • q • q .

In particular, we can check that, at any time, when removing historical information from ρ, we get back the multi-set of states of processes:

Under the prefix relation , we can consider the set of maximal runs

As a denumerable intersection of measurable sets, max agree (σ) is proven to be measurable for P. This allows us to define a probability measure over this strategy: Definition 11.7. Let n ∈ N, Π ⊆ paths (→) a measurable set of paths, and σ a local strategy. If P n (max agree (σ)) > 0, we define P σ n (Π) = P n Π max agree (σ) = P n (Π ∩ max agree (σ)) P n (max agree (σ))

Otherwise, we let P σ n (Π) = 0.

Remark 11.8. Another possible way of defining P σ n would consist in defining a new protocol P σ from P, with a new state space Q c = Q + c , and

and a similar definition for Q l and T l . Intuitively, we would store, for each process its whole history h as the current state, then allow only transitions from σ(h). We can easily check that each valid path of this new protocol, can be projected by taking last state for each history in each configuration, to a path in P that agrees with σ. However, all our study is based on a formal model where protocols have a finite state space. In particular, technical difficulties may occur when trying to consider a well quasi order on configurations in infinite dimensions, or when trying to prove that qualitative properties do not depend on the exact transition probabilities (Lemma 7.22). In fact, these technical difficulties are avoided by directly considering the conditional probability. In particular, visiting the same state twice but with different memory states may infer different transition probabilities.

However, we keep in mind the intuition that for a finite memory local strategy σ, we could build a protocol P σ with a finite number of states, having the same qualitative properties as for P σ n (•).

The previous definition makes implicit the choice of a probabilistic transition system (Γ, p) for P in order to define P σ n . However, as in Lemma 7.22, we characterize qualitative properties without having to consider the exact probability values of p: Lemma 11.9. Let A ⊆ Γ and σ ∈ S. The property P σ n (♦A) = 1 does not depend on the actual values of p. Moreover, we have the following characterization:

Proof. The proof is immediate by applying the previous definition to state that P σ n (♦A) > 0 ⇔ P σ n (♦A ∩ max agree (σ)) > 0 ∧ P σ n (max agree (σ)) > 0

Cut-off property

Our main goal is the synthesis of strategies that will ensure a positive cut-off. This leads to the following definition.

Definition 11.10. Let σ be a local strategy, I a sub-interval of [0, 1] and A a set of configurations. We say that σ is a cut-off strategy for probability interval I and objective A, if there exists N ∈ N such that for all parameter n ≥ N , P σ n (♦A) ∈ I.

As before, we define three qualitative cut-off properties, namely {0}, (0, 1] and {1} cutoffs for a simple reachability objective U f , respectively for safety, positive reachability and almost-sure reachability.

The sections 11.2 and 11.3 respectively focus on the first two problems, in the restricted non-atomic, leaderless case, with coverability objective U f .

As opposed to the cut-off decision problem, these two problems are not dual in the local strategy synthesis framework.

Figure 11.3 -Example of a protocol with allowed actions requiring linear memory for a pure strategy to be a cut-off strategy for positive coverability of q n .

• If h 1 and h 2 are roots of disjoint subtrees, then

We can "remove" h 1 , h 2 and its successors from H.

• Otherwise, assume, without loss of generality, that h 1 is a parent of h 2 (h 1 h 2 ). Then, all important nodes lie in the subtree H∩(h 2 Q * ), that is to say H∩(h 1 Q * \h 2 Q * ) contains no important nodes. Thus, this last set can be "removed" and replaced by the subtree H ∩ (h 2 Q * ) now rooted from history h 1 . This operation is depicted on Figure 11.2.

Such procedure removes at each step at least one node, so we are guaranteed to terminate. Each node h that is "removed" may have enabled some transition t, however, since h / ∈ I, we ensure that some other node h ∈ I is produced and can enable the same transition t. Thanks to the copycat lemma, we can produce as many processes with history h as required, and before they are needed.

Once the announced property is achieved, we can bound the size [H] of the tree by [Q] • ( [Allow] + 1) which is a polynomial in |P| and [Allow]. In particular, the resulting strategy σ requires only polynomial memory.

As illustration of Theorem 11.13, we provide in Figure 11.3 an example of protocol and allowed actions, which require linear memory when considering pure actions only.

Summary

It is not always possible to "trade randomness for memory". Consider for example the protocol and allowed actions depicted in Figure 11.4: there exists a mixed cut-off strategy for positive coverability, however no pure strategy is a cut-off strategy.

We conclude our study of positive probability coverability by the following theorem:

Theorem 11.14. The following problem can be solved in polynomial time:

A leaderless non-atomic protocol P, a set of allowed actions Allow and a final state q f ∈ Q. QUESTION: Whether there exists a cut-off strategy σ for positive probability reachability of

The following problem is NP-complete: pReachCoverStrat INPUT:

A leaderless non-atomic protocol P, a set of allowed actions Allow and a final state q f ∈ Q.

Proof.

• Membership in NP is due to the existence of memoryless strategies stated in Theorem 11.17. Such a strategy σ is of linear size in the original protocol, and we can check that the resulting composed protocol P σ cannot reach q f . Because we are considering coverability objective of q f without leader, this check can be done in polynomial time (see [START_REF] Esparza | Parameterized verification of asynchronous shared-memory systems[END_REF] or consider an increasing support path in the symbolic graph).

• NP-hardness is proven by a reduction of 3SAT ( [START_REF] Cook | The complexity of theorem-proving procedures[END_REF]). Given an instance of this problem, with atomic proposition set AP and set of clauses X, we build the protocol depicted in Figure 11.5, where only non-trivial actions are allowed from states q v (p), for each p ∈ AP. Intuitively, the strategy chooses a valuation for p in q v (p), while the scheduler challenges the whole assignment by picking a clause C and checking that each literal of C has the wrong truth value, which leads to unsafe state q f . More formally, if there exists a cut-off strategy for safety from q f , we can assume by Theorem 11.17 that such strategy is memoryless and pure, and consider the corresponding valuation ν. Since, q f cannot be reached by the scheduler, we check that ν satisfies all the clauses in X. Conversely, if ν satisfies X, we consider the memoryless strategy that plays A p,ν(p) from q v (p) for each p ∈ AP and check that such strategy is safe.

Surprisingly, the safety problem in non-atomic protocols with a leader, but without strategies, is proven by [START_REF] Esparza | Parameterized verification of asynchronous shared-memory systems[END_REF] to be coNP-complete. Hardness is achieved thanks to the leader process which encodes the valuations of atomic propositions (in the dual reachability objective setting). This has to be compared with our proof of Theorem 11.18 which exploits strategies in order to encode valuations. On the other side, membership in NP is proved thanks to the lack of a leader (polynomial time check).

Conclusions

We have studied several techniques for the existence of cut-off strategies for positive reachability and safety objectives. Both problems are not dual anymore when shifting to synthesis of local strategies. On the one hand, allowing randomization or memory is crucial for positive reachability objectives:

• If we allow randomization, we have all incentive to consider the most randomized strategy.

• If we restrict to pure strategies, polynomial memory is required, and suffice.

We argue that randomizing all actions may be expensive in a practical setting, so even when randomization helps, we may be interested in synthesizing strategies that still record some information in order to minimize the amount of required entropy. Moreover, mixed strategies may drastically increase expected time to objective. Such quantitative questions were eluded on purpose and left as future work.

On the other hand, safety requires no memory. However, note that our techniques heavily rely on three assumptions: non-atomicity, lack of a leader and coverability objective. First property seems legitimate as the cut-off decision problem for atomic networks is already an open problem. However, Chapter 8 studied more general cases with a leader process and a

List of Tables

LIST OF TABLES

le jeu fait intervenir trois joueurs. La preuve s'appuie sur la réduction du problème de l'arrêt d'une machine à deux compteurs [START_REF] Minsky | Computation: Finite and Infinite Machines[END_REF], de façon similaire à une précédente preuve à 14 joueurs [START_REF] Ummels | The complexity of Nash equilibria in infinite multiplayer games[END_REF].

La preuve présentée ici s'appuie sur des caractéristiques supplémentaires, ce qui permet d'abaisser le nombre de joueurs nécessaires requis pour l'indécidabilité :

• Les jeux sont supposés concurrents, ce qui signifie que plusieurs joueurs choisissent simultanément la prochaine action, par opposition aux jeux à tour où au plus un joueur peut réaliser un choix non trivial depuis chaque état.

• Les actions sont inobservables, ce qui permet de réaliser des encodages plus complexes de la machine à deux compteurs.

De manière équivalente, la réduction nous permet de considérer le cas qualitatif, où les fonctions de gain ne prennent que des valeurs égales à 0 ou 1. Dans le cas de l'accessibilité terminale, cela revient à dire qu'un joueur gagne uniquement s'il arrive à atteindre un certain ensemble d'états finaux. Dans le cas d'un objectif de sûreté, le joueur ne gagne que s'il parvient à rester dans un ensemble d'états donné.

Nous établissons ainsi que l'existence d'un équilibre est indécidable dès lors que le jeu implique au moins trois joueurs aux objectifs qualitatifs, dont l'un au moins est un objectif d'accessibilité terminale et un autre un objectif de sûreté. Ceci est à mettre en regard d'autres configurations pour lesquelles l'existence d'un équilibre est assuré :

• Lorsqu'un seul joueur est présent, le jeu se résume à un processus de décision Markovien, pour lequel l'existence d'une stratégie optimale est connue.

• Lorsque tous les joueurs possèdent des objectifs de sûreté qualitatifs, [SS01] établit l'existence d'un équilibre de Nash.

Deux cas restent ainsi ouverts : l'existence d'un équilibre en présence de deux joueurs, ainsi que le cas où tous les objectifs considérés sont de sûreté.

Équilibres aux déviations imprécises

Le chapitre 5 conclut la première partie en introduisant une notion relâchée d'équilibres. Une première notion présentée par [START_REF] Chatterjee | On Nash equilibria in stochastic games[END_REF] établit l'existence d'ε-équilibres de Nash, pour tout ε > 0. Formellement, un tel équilibre est caractérisé par un profil σ tel que :

Nous établissons ici l'existence d'une notion duale, que nous montrons incomparable à la notion précédente : un profil σ est un équilibre sous déviation ε-imprécise lorsque :

où d(σ i , σ i ) désigne la distance entre les stratégies σ i et σ i vue comme le supremum des distances entre distributions depuis tous les historiques possibles.

Ainsi, lorsqu'une déviation σ i est proposée par le joueur i, toute déviation à distance inférieure à ε doit également améliorer le gain du joueur i afin de mettre en échec l'équilibre.

Nous montrons qu'un tel profil de stratégies σ existe nécessairement, en adaptant des techniques de Nash ainsi que [START_REF] Secchi | Stay-in-a-set games[END_REF]. De plus, σ peut être supposé stationnaire, ce qui ouvre la voie au développement d'un algorithme de calcul en espace polynomial.

Partie 2 : Réseaux paramétrés

Dans la seconde partie du manuscrit, nous nous intéressons à un modèle plus réaliste où le nombre d'agents n'est pas déterminé à l'avance, et l'observation de chaque joueur est imparfaite. Plus précisément, le nombre d'agents est un paramètre, et nous souhaitons répondre à des problèmes de vérification indépendamment du paramètre choisi.

Différentes méthodes d'interaction

Puisque le paramètre n'est pas fixé, il n'est plus possible de considérer le graphe d'état du système complet, et il est nécessaire de spécifier séparément le comportement de chaque agent, puis les modalités d'interaction entre agents. Dans le chapitre 6, nous discutons des différentes méthodes d'interactions envisagées dans l'étude des systèmes paramétrés. Un bilan présenté par Esparza [START_REF] Esparza | Keeping a crowd safe: On the complexity of parameterized verification (invited talk)[END_REF] propose ainsi une classification des moyens de communication, par expressivité décroissante :

• Les diffusions de messages globaux (broadcast) ;

• Les variables partagées avec verrou ;

• La communication par rendez-vous ;

• Les variables partagées avec opérations non atomiques (sans verrou).

C'est ce dernier mode de communication qui est étudié ici, du fait de ses propriétés de monotonicité. En effet, lorsque plusieurs agents ou processus communiquent entre eux, il est toujours possible de dupliquer n'importe quel processus en lui faisant exécuter les mêmes transitions au même instant, celles-ci étant soit des lectures, soit des écritures. L'hypothèse de non atomicité se révèle cruciale car elle interdit la lecture puis écriture simultanée du registre, opération qui ne pourrait pas être dupliquée.

Réseaux paramétrés

Nous définissons ainsi dans le chapitre 7 le modèle formel de réseaux paramétrés avec variable partagée. Celui-ci est composé d'un domaine de définition D de la variable partagée, sa valeur initiale d 0 ∈ D, ainsi que de deux automates finis. Chaque automate (q, Q, T ) décrit le comportement d'un processus, initialement dans l'état q ∈ Q, et dont les transitions, définies par l'ensemble T , sont étiquetées par des opérations de lecture R (d) ou d'écriture W (d), pour une certaine valeur de registre d ∈ D. Intuitivement, s'il existe une transition (q, R (d) , q ) ∈ T où d ∈ D est la valeur courante du registre partagé, alors tout processus dans l'état q peut effectuer une transition vers l'état q , sans altérer le registre. S'il existe une transition (q, W (d) , q ) ∈ T , alors tout processus dans l'état q peut effectuer une transition vers l'état q , en inscrivant la valeur d dans le registre partagé, et ceux peu importe la valeur courante du registre.

Nous appelons configuration d'un réseau paramétré P un triplet γ = (q, µ, d) où

• d est la valeur courante du registre partagé ;

• q est l'état courant du processus leader, décrit par le premier automate (q l , Q l , T l ) ; We study games played on graphs by an arbitrary number of players with nonzero sum objectives. The players represent agents (programs, processes or devices) that can interact to achieve their own objectives as much as possible. Solution concepts, as Nash Equilibrium, for such optimal plays, need not exist when restricting to pure deterministic strategies, even with simple reachability or safety objectives. As a matter of fact, symmetry is preserved when players behave deterministi-cally, which motivates the studies where either the players or the environment can use randomization. In the rst case, we show that classical concepts like Nash Equilibria, cannot be computed even with a xed number of agents and propose computable approximations. In the second case, we study systems composed of several copies of the same process, communiting through a shared register, and whose executions are determined by a stochastic scheduler.

Université Paris-Saclay

Espace Technologique / Immeuble Discovery Route de l'Orme aux Merisiers RD 128 / 91190 Saint-Aubin, France