Jean Yves Le Boudec

Yezekael Hayel

William H Sandholm

Konstantin Avrachenkov

Tamer Başar

Pierre Bern- Hard

Philippe Michelon

Nahum Shimkin

Fréderic Bonnans

Pierre Cardaliaguet

Patrick Combettes

Michael Eisermann

Stéphane Gaubert

Jean Charles Gilbert

Lucien Guillou

Barthel Gottfried

Anatoli Iouditski

Siegmund Kosarew

Jean Pierre Ponssard

Dinah Rosenberg

Eilon Solan

Sylvain Sorin

Tristan Tomala

Yannick Viossat

Patrick Witomski

Ephie Deriche

Jocelyne Gourret

Patricia Hjelt

Rebecca Lonberger

My sincere gratitude for all the delightful discussions. I wish to thank Tamer Başar

Chapter 1 Introduction

Recently, there has been a surge in research activities that employ game theory to model and analyze the performance of various networks, such as communication networks, computer networks, social networks, biologically inspired networks etc. There already exist several successful examples where game theory provides deeper understanding of complex network dynamics and leads to better design of efficient, scalable, and robust networks. Still, there remain many interesting open research problems yet to be identified and explored, and many issues to be addressed. Important analysis and applications have been done in the context of static games. Motivated by the dynamic behavior of most of the long-term systems and the understanding of prediction, learning and evolution, dynamic game theory has found several applications. Those include cooperative and non-cooperative models of repeated games, sequential games, stochastic games, differential games, evolutionary games etc. Evolutionary games in large population provides a simple framework for describing strategic interactions among large numbers of players. Traditionally, predictions of behavior in game theory are based on some solution concept, typically Cournot equilibrium [START_REF] Cournot | Recherches sur les principes mathématiques de la théorie des richesses[END_REF], Bertrand equilibrium [START_REF] Joseph | Théorie des Richesses: revue de Théories mathématiques de la richesse sociale par Léon Walras et Recherches sur les principes mathématiques de la théorie des richesses par Augustin Cournot[END_REF] or some extension/refinement thereof. These notions require the assumption of equilibrium knowledge, which posits that each user correctly anticipates how other players will act or react. The equilibrium knowledge assumption is too strong and is difficult to justify in particular in contexts with large number of players. As an alternative to the equilibrium approach, an explicitly dynamic updating choice is proposed, a model in which players myopically update their behavior in response to their current strategic environment. This dynamic procedure does not assume the automatic coordination of player's actions and beliefs, and it can derive many specifications of players' choice procedures. These procedures are specified formally by defining a revision of pure strategies called a revision protocol. A revision protocol takes current payoffs (also called fitness in behavioral ecology) and aggregate behavior as inputs; its outputs are conditional switch rates which describe how frequently players in some class playing strategy a who are considering switching strategies move to another strategy b, given that the current payoff vector and subpopulation state. Revision protocols are flexible enough to incorporate a wide variety of paradigms, including ones based on imitation, adaptation, learning, optimization, etc. The revision protocols describe the procedures players follow in adapting their behavior in the dynamic evolving environment such as evolving networks.

Birth of Evolutionary Game Theory in Engineering

The evolutionary games formalism is a central mathematical tool developed by biologists for predicting population dynamics in the context of interactions between populations. This formalism identifies and studies two concepts: the Evolutionary Stability, and the Evolutionary Game Dynamics. The unbeatable strategy has been defined by Hamilton (95;[START_REF] Hamilton | The genetical evolution of social behviour, I and II[END_REF] which is the analogous of strong equilibrium (resilient against multilateral deviations) in large systems. A weaker of notion of locally unbeatable strategy, the Evolutionarily Stable State or Strategy (ESS), has been defined by the biologists Maynard Smith & Price [START_REF] Smith | Game Theory and the Evolution of Fighting[END_REF]. The ESS is characterized by a property of robustness against invaders (mutations). More specifically, (i) if an ESS is reached, then the proportions of each population do not change in time. (ii) at ESS, the populations are immune from being invaded by other small populations. This notion is stronger than Nash equilibrium in which it is only requested that a single user would not benefit by a change (mutation) of its behavior. The ESS concept helps to understand mixed strategies in games with symmetric payoffs. A mixed strategy can be interpreted as a composition of the population. An ESS can be also interpreted as a Nash equilibrium of the one-shot game but a (symmetric) Nash equilibrium cannot be an ESS. As is shown in [START_REF] Van Damme | Stability and perfection of Nash equilibria[END_REF], ESS has strong refinement properties of equilibria such as proper equilibrium, perfect equilibrium etc. Before the ESS concept, Hamilton has introduced the concept of Unbeatable Strategy (95; 96), which is stronger than ESS. Although ESS and unbeatable strategy have been defined in the context of biological systems, it is highly relevant to engineering as well [START_REF] Vincent | Evolution and control system design[END_REF]. In the biological context, the replicator dynamics is a model for the change of the size of the population(s) as biologist observe, where as in engineering, we can go beyond characterizing and modeling existing evolution. The evolution of protocols can be engineered by providing guidelines or regulations for the way to upgrade existing ones and in determining parameters related to deployment of new protocols and services.

There have been a lot of work on non-cooperative modeling of power control using game theory (76; 166). There are two advantages in doing so within the framework of evolutionary games:

• it provides the stronger concept of equilibria, the ESS, which allows us to identify robustness against deviations of a fraction of players, and

• it allows us to apply the generic convergence theory of evolutionary game dynamics, and stability results that we introduce in next chapters.

Homogenous population

We consider the standard setting of evolutionary games : there is a large populations of players; each member of the population has the same pure action set A (a finite set).

Mixed strategies and population profile

Denote by ∆(A) the (|A| -1)-dimensional simplex of R |A| . Let x(t) be the |A|dimensional vector whose element x a (t) is the population share of strategy a at time t. Thus we, have ∑ a∈A x a (t) = 1 and x a (t) ≥ 0. We frequently use an equivalent interpretation where x(t) is a mixed strategy used by all players at time t; by a mixed strategy we mean that a player chooses at time t an action a with probability x a (t). With either interpretations, at each local interaction occurring at time t a given player can expect that the other player would play strategy a with probability x a (t). The vector x(t) will also be called the state of the population at time t. Define the expected payoff of a player with the action a when the population profile x(t) by f a (x(t)).

Equilibrium and Evolutionary Stability

The evolutionarily stable state or strategy (ESS) concept have the property that if it is reached, then the proportions of each population do not change in time. Suppose that, initially, the population state has been x ∈ ∆(A) during a long time. The average payoff in the population is ∑ a∈A x a f a (x). Now suppose that a small group of mutants enters this population playing according to a different profile mut (and persists using it during a time longer than the delays). If we call ǫ ∈ (0, 1) the size of the subpopulation of mutants after normalization, then the population profile after mutation will be ǫ mut + (1ǫ)x. After mutation, the average payoff of non-mutants who are randomly matched to mutants is given by ∑ a∈A x a f a (mut), and the average payoff of non-mutants will be given by ǫ ∑ a∈A x a f a (mut) + (1 -ǫ) ∑ a∈A x a f a (x).

Analogously, we can construct the average payoff of mutant ǫ ∑ a∈A mut a f a (mut) + (1 -ǫ) ∑ a∈A mut a f a (x).

A population x ∈ ∆(A) is an ESS if for all mut = x, there exists some ǫ mut ∈ (0, 1), which may depend on mut, such that for all ǫ ∈ (0, ǫ mut ) ǫ ∑ a∈A mut a f a (mut) + (1 -ǫ) ∑ a∈A mut a f a (x) (1.1)

< ǫ ∑ a∈A x a f a (mut) + (1 -ǫ) ∑ a∈A x a f a (x)
That is, x is ESS if, after mutation, non-mutants are more successful than mutants, in which case mutants cannot invade and will eventually get extinct. The number ǫ mut is called invasion barrier See [START_REF] Weibull | Evolutionary Game Theory[END_REF]. It is the maximum rate of mutants against which s is resistant. If

x is an ESS then x is a Nash equilibrium. Equivalently x is an ESS if and only if it meets best response conditions:

∑ a∈A mut a f a (x) ≤ ∑ a∈A x a f a (x), ∀ mut, ( 1.2)
∀ mut = x, ∑ a∈A mut a f a (x) = ∑ a∈A x a f a (x) ⇒ (1.3)
∑ a∈A mut a f a (mut) < ∑ a∈A x a f a (mut) (1.4) For population games with non-linear payoffs (i.e when x -→ f a (x) is non-linear function), define the solution and evolutionary stability concepts as follows: A population profile x is an equilibrium state if ∑ a∈A x a f a (x) ≥ ∑ a∈A mut a f a (x), ∀ mut This variational inequality is equivalent to :

( * ) ∀a ∈ A, x a > 0 =⇒ f a (x) = max b∈A f b (x)
This last condition is sometimes referred to "Wardrop first principle" [START_REF] Wardrop | Some theoretical aspects of road traffic research[END_REF] of optimality and can be easily obtained using the indifference condition at mixed equilibria. The condition ( * ) is sometimes written in the form: supp(x) ⊆ arg max a f a (x).

We say that a population profile x = (x a ) a∈A is evolutionarily stable if for each deviation strategy called "mutant strategy" mut = (mut a ) a∈A = x, there exists some ǫ mut > 0 such that ∀ǫ ∈ (0, ǫ mut ), ∑ a∈A (x amut a ) f a (ǫ mut + (1ǫ)x) > 0.

(1.5)

The strategy x is a neutrally evolutionary stable state (NESS) if the above inequality is nonstrict. If the inequality is non-strict and ǫ = 0 then the resulting inequality says that x is an equilibrium state. Thus, the ESS notion is stronger than Nash equilibrium.

A population profile x ∈ ∆(A) is an unbeatable state if for any mut = x,

∑ a∈A (x a -mut a ) f a (ǫmut + (1 -ǫ)x) > 0, ∀ǫ ∈ (0, 1) (1.6)
That is, x is an unbeatable state if, after mutation of any size (it can be all the population), nonmutants are more successful than mutants. In other words, any mutants of any size cannot invade the population. Note that an unbeatable state is an evolutionarily stable state, which implies neutrally stable state, which is an equilibrium of the one-shot game i.e if the population profile is at this state and no player has incentive to unilaterally change his/her action. The following inclusion holds:

∆ Unbeatable state ⊂ ∆ ESS ⊂ ∆ Neutally stable ⊂ ∆ Equilibrium state (1.7)
Note that evolutionarily stable state and unbeatable state may not exists (see for example the class of Rock-Paper-Scissor games (99; 90; 65; 234)). The following theorem gives another necessary and sufficient condition to be an ESS for evolutionary games with bilinear payoff functions. A proof can be found in [START_REF] Haigh | Game theory and evolution[END_REF]. Theorem 1.0.0.1. Let q be a symmetric (Nash) equilibrium for the matrix game with payoffs (F, F t ) where F t is the transposition matrix of F = (F(i, j) i,j ) and BRP(q) be the pure best response to q i.e

BRP(q) = j | ∑ k F(j, k)q k = max l ∑ k F(l, k)q k .
Define q as qj = q j if j ∈ BRP(q) 0 otherwise Let F the submatrix obtained from F by taking only the index i, j ∈ BRP(q). Then q is an evolutionarily stable strategy if and only if

∑ k∈BRP(q) (p k -qk ) ∑ j∈BRP(q)
Fk,j (p jqj ) < 0 for all p = q.

Note that the last condition is equivalent to ∀y ∈ Y, ∑ k,j∈BRP(q) y k y j Fk,j < 0 where

Y :=    z ∈ R |BRP(q)| \{0}, ∑ j∈BRP(q)
z j = 0, and qj = 0 =⇒ z j ≥ 0    and |BRP(q)| is the cardinal of the finite set BRP(q).

ESS is an important refinement of Nash equilibria as shown in the following example.

Example

Consider the following matrix game with two players. Each has one block of strategies 1 nd one pure strategy 2. The block contains at least two strategies. The corresponding payoffs are given in the tabular where the player I chooses a row r1 (the block ) or r2 and player II chooses a column c1 (block) or c2.

Player II c1 c2 r1 min(α, β), min(α, β) min(α, β), min(α, β)

Player I r2 min(α, β), min(α, β) max(α, β), max(α, β)

where α, β ∈ R. The game has an infinity of Nash equilibria: (i) one of the player chooses the first strategy (in the block) and the second an arbitrary strategy, (ii) Both players choose the second strategy. But the game has only one ESS (the second strategy). An allocation of payoffs is said Pareto optimal if the outcome cannot be improved upon without hurting at least one player. In this case the ESS is also Pareto optimal because the payoff obtained at ESS is the maximum payoff that each player can have.

When the block has only two pure actions, the game becomes the following: Each player has three strategies 1, 2 or 3. The corresponding payoffs are given in the tabular where the player I chooses a row r1, r2 or r3 and player II chooses a column c1, c2 or c3. c1 c2 c3 r1 min(α, β), min(α, β) min(α, β), min(α, β) min(α, β), min(α, β) r2 min(α, β), min(α, β) min(α, β), min(α, β) min(α, β), min(α, β) r3 min(α, β), min(α, β) min(α, β), min(α, β) max(α, β), max(α, β)

ESS is not unbeatable

We ask the following question: What happens when the size of mutants is greater than the invasion barrier of an ESS? To answer to question, we use the unbeatable strategy, a concept defined by [START_REF] Hamilton | Extraordinary sex ratios[END_REF], as a population profile which is resilient to any deviant strategy of any size. An unbeatable strategy is always an ESS, though an ESS is not necessarily unbeatable, as it may be beaten by large migrations into the population. The following example illustrates a situation where ESS is not unbeatable Chapter 1. Introduction c1 c2 c3 r1 max(α, β), max(α, β) max(α, β), max(α, β) min(α, β), min(α, β) r2 max(α, β), max(α, β) min(α, β), min(α, β) max(α, β), min(α, β) r3 min(α, β), min(α, β) min(α, β), max(α, β) min(α, β), min(α, β) Are strong Nash equilibria evolutionarily stable strategies?

Aumann was one of the first one to introduce a number of concepts in game theory. One of these concepts was a strong equilibrium [START_REF] Aumann | Acceptable Points in General Cooperative n-Person Games[END_REF], which is a pure Nash equilibrium, in which not only single players cannot benefit from changing their strategy (to a different pure strategy), but no non-empty subset of players can form a coalition, where a coalition (group) means that all of them can change their strategies together, and all gain from the change.

In a strong equilibrium (see also acceptable point ), no coalition (of any size) can deviate and improve the utility of every member of the coalition (while possibly lowering the utility of players outside the coalition). Clearly, every strong equilibrium is a Nash equilibrium, but the converse does not hold. In cases where a strong equilibrium exists, it seems to be a very robust notion. Considering strong equilibrium allows us to separate the effect of selfishness (which remains in strong equilibria) from that of lack of coordination (which disappears, since a strong equilibrium is resilient to deviations in coalitions).

In a n-player one-shot normal form game we say that a pure strategy profile is a k-strong equilibrium if it is robust by deviation of the group of size at most k(k ≤ n). A pure Nash equilibrium is resilient to pure deviations of groups of size k, if there is no group of size at most k, such that the strategy profile is not resilient to a pure deviation by the subset of the group. A k-strong equilibrium is a pure Nash equilibrium that is resilient to pure deviations of coalitions of size at most k. We will discuss how to define strong equilibrium in randomized strategies. In a Nash equilibrium no player can improve its own payoff by unilaterally changing its action. Thus, a Nash equilibrium corresponds to 1-strong equilibrium. It is easy to see that strong equilibrium is equivalent to a n-strong equilibrium.

We ask the following question: Are strong Nash equilibria evolutionarily stable strategies? We show that the answer is negative. To see this, consider an evolutionary game with constant payoff. It is clear that the game has many strong equilibria. None of theses equilibria are evolutionarily stable strategies.

Evolutionary game dynamics

As an alternative to the equilibrium approach, evolutionary game dynamics propose an explicitly dynamic updating choice, a model in which players myopically update their behavior in response to their current strategic environment. This dynamic procedure does not assume the automatic coordination of users' actions and beliefs, and it can derive many players' choice procedures. These procedures are specified formally by defining a revision of pure strategies. The revision of strategies describes the procedures users follow in adapting their behavior to in the dynamic evolving environment such as in evolving networks.

There is a large number of population dynamics have been used in the context of noncooperative games. Examples are the excess payoff dynamics, the fictitious play dynamics such as Brown-von Neumann-Nash dynamics [START_REF] Brown | Solutions of games by differential equations[END_REF], gradient-based game dynamics [START_REF] Rosen | Existence and Uniqueness of Equilibrium Points for Concave N-Person Games[END_REF], replicator dynamics [START_REF] Taylor | Evolutionary stable strategies and game dynamics[END_REF], pairwise comparison dynamics such as Smith dynamics [START_REF] Smith | The stability of a dynamic model of traffic assignment: an application of a method of Lyapunov[END_REF], fictitious play (Gilboa & Matsui, 1991), projection dynamics, best response dynamics (Fudenberg & Levine, 1991), Fudenberg & Tirole, 1991), Boltzman dynamics and logit dynamics [START_REF] Fudenberg | The Theory of Learning in Games Cambridge[END_REF], G-function based dynamics [START_REF] Vincent | Evolutionary Game Theory, Natural Selection, and Darwinian Dynamics[END_REF] etc. Much literature can be found is the extensive survey on evolutionary game dynamics in [START_REF] Hofbauer | Evolutionary game dynamics[END_REF] and in the book of Sandholm on Population Games and Evolutionary Dynamics [START_REF] Sandholm | Population Games and Evolutionary Dynamics[END_REF]. The replicator dynamics is one of the most studied evolutionary game dynamics. It has been used for describing the evolution of road traffic congestion in which the fitness is determined by the strategies chosen by all drivers [START_REF] Sandholm | Potential games with continuous player sets[END_REF]. It has also been studied in the context of the association problem in wireless networks in [START_REF] Shakkottai | The Case for Non-cooperative Multihoming of Users to Access Points in IEEE 802.11 WLANs[END_REF].

Most of applications using evolutionary game dynamics was inter-disciplinary between economics, biology and behavioral ecology. Recently, game theory evolving has found many applications in the field of wireless communications and networks. The standard formulations of evolutionary games consider only pairwise interactions such as random matching. However, interactions which are not pairwise frequently arise in communication networks, such as the cases of Code Division Multiple Access (CDMA) system [START_REF] Tembine | Evolutionary game dynamics with migration for hybrid power control in wireless communications[END_REF], Orthogonal Frequency-Division Multiple Access (OFDMA) in Worldwide Interoperability for Microwave Access (WiMAX) environment [START_REF] Altman | Evolutionary Power Control Games in Wireless Networks[END_REF] etc. It has been applied to problems such as congestion control [START_REF] Menasche | An evolutionary game-theoretic approach to congestion control[END_REF], distributed cooperative sensing over cognitive radio networks (232; 160), code division multiple access (CDMA), Orthogonal Frequency-Division Multiple Access (OFDMA) based Worldwide Interoperability for Microwave Access (WiMAX) environment [START_REF] Altman | Evolutionary Power Control Games in Wireless Networks[END_REF], reciprocal and non-reciprocal interference control, mobile medium access control and channel selection [START_REF] Tembine | Evolutionary games with random number of interacting players applied to access control[END_REF] and capacity region of Additive White Gaussian Noise (AWGN) (88; 242), Multihoming and association problems (184; 209). These examples will be discussed in chapters 3 and 4.

The purpose of this manuscript is threefold.

Part I: Delayed Evolutionary Game Dynamics with Migration

The first part of this manuscript contains evolutionary stability and dynamic foundation of evolutionary games. The chapter 2 presents evolutionary games with delayed payoffs. We introduce and study delayed evolutionary game dynamics in which each pure strategy has its own time delay. Delayed evolutionary game dynamics are in general a system of first order nonregular nonlinear differential equations or differential inclusions with time delays. We use the theory of delayed differential equations (DDE), which is a special class of functional differential equations to study bifurcation, periodicity, stability, convergence and chaocity of the system under time delays. Delays play an important role in many situations in networking such as in flow control and congestion problems. We examine the effects of time delays in various contexts : congestion control protocols, dissemination information, Hawk and Dove game and multiple access game with regret cost using various delayed evolutionary game dynamics such delayed replicator dynamics, delayed imitation by dissatisfaction dynamics, delayed best response dynamics, delayed Brown-von Neumann-Nash dynamics, delayed projection dynamics, delayed generalized Smith dynamics, delayed evolutionary game dynamics with migration etc.

The chapter 3 extends the basic pairwise interaction model of evolutionary games to cover more general interactions including non-reciprocal interactions, and random number of players. In order to make use of the wealth of tools and theory developed in biology literature, many works in the area of computer networks ignore cases where local interactions between populations involve more than two players in a local interaction. This restriction limits the modeling power of evolutionary games which are not useful in a network operating at heavy load, such as internet traffic and dense networks. This motivated us to consider more than two players interacting locally. Moreover, the interactions can be non-symmetric. The resulting framework leads to evolving games with non-linear expected payoffs (in term of population profile). Equi-librium analysis as well as dynamic behavior of networking games are presented. Conditions for existence and uniqueness of equilibrium are given in some class of games including submodular games, supermodular games, potential games, games with monotone payoffs etc. We also discuss about selection of equilibria using population dynamics.

We study the following applications:

• Power control in Wideband CDMA wireless networks: Wideband Code Division Multiple Access (W-CDMA) is wideband spread-spectrum channel access method that utilizes the direct-sequence spread spectrum method of asynchronous code division multiple access to achieve higher speeds and support more users compared to most time division multiple access (TDMA) [START_REF] Meshkati | Energy-Efficient Resource Allocation in Wireless Networks[END_REF]. Power control in wireless networks has become an important research area. Since the technology in the current state cannot provide batteries which have small weight and large energy capacity, the design of tools and algorithms for efficient power control is crucial. For a comprehensive survey of recent results on power control in wireless networks an interested reader can consult e.g., [START_REF] Meshkati | Energy-Efficient Resource Allocation in Wireless Networks[END_REF] and the reference therein. Power control protocols based on game theory have been designed for already ten years starting with the pioneering work (76; 115). Non-cooperative games provide a convenient framework for decentralization and distributed decision making in those applications, where as cooperative approaches of game theory have allowed to handle issues concerning fairness in power allocation. Most applications of game theory to power control consider mobile terminals as players of the same type and study strategic oneshot games with a fixed number of players. Here, we consider a population game with many simultaneous local interactions, where the game is played infinitely under some self-organizing process called "hybrid dynamic" and where each interaction concerns a random number of players (213; 11).

• OFDMA based IEEE 802. [START_REF] Altman | Constrained cost-coupled stochastic games with independent state processes[END_REF] Networks: OFDMA (Orthogonal Frequency Division Multiple Access) is recognized as one of the most promising multiple access technique in wireless communication system. This technique is used to improve spectral efficiency and becomes an attractive multiple access technique for 4 th generation mobile communication system as WiMAX.

In OFDMA systems, each user occupies a subset of subcarriers, and each carrier is assigned exclusively to only one user at any time. This technique has the advantage of eliminating intra-cell interference (interference between subcarriers is negligible). Hence the transmission is affected by intercell interference since users in adjacent sectors may have also been assigned to the same carrier. If those users in the adjacent sectors transmitted with high power the intercell interference may severely limit the SINR achieved by the user. Some form of coordination between the different cells occupying the spectral resource are studied in (134; 123). The optimal resource allocation requires complete information about the network in order to decide which users in which cells should transmit simultaneously with a given power. All of these results however, rely on some form of centralized control to obtain gains at various layers of the communication stack. In a realistic network as WiMAX, centralized multicell coordination is hard to realize in practice, especially in fast-fading environments. Decentralized schemes have proposed to the intercell interference in OFDMA-based IEEE 802. [START_REF] Altman | Constrained cost-coupled stochastic games with independent state processes[END_REF] Networks.

• Aloha-like protocols: Random Medium Access Control (MAC) algorithms have played an increasingly important role in the development of wired and wireless networks and the performance and stability of these algorithms, such as slotted-Aloha, Carrier Sense Multiple Access (CSMA) is still an open problem [START_REF] Bardenave | Performance of Random Medium Access Control An asymptotic approach[END_REF]. Distributed Medium Access Control, starting from the first version of Abramson's Aloha to the most recent algorithms used in IEEE802.11, have enabled a rapid growth of both wired and wireless networks. They aim at efficiently and fairly sharing a resource among users even though each user must decide independently (eventually after receiving some messages or listening) when and how to attempt to use the resource. MAC algorithms have generated a lot of research interest, especially recently in attempts to use multi-hop wireless networks to provide high-speed access to the internet with low-cost and low-energy consumption. we focus our attention to wireless networks, where the resources are receivers, base station or access points and where users interact because of interference, i.e., interfering users cannot transmit simultaneously. There is a collision if another user (mobile) transmits with a greater power level at the same range of the receiver.

We apply evolutionary game with random number of interacting players to slotted Aloha wireless networks, power control in Wideband CDMA networks and OFDMA-based IEEE 802. [START_REF] Altman | Constrained cost-coupled stochastic games with independent state processes[END_REF] networks. We then focus on evolutionary stability properties of correlated equilibrium in evolutionary games with arbitrary number of players in each local interaction. The performance and cost of random medium access control with power control is analyzed. This study is extended to the case of Signal-to-Interference-plus-Noise-Ratio (SINR) and Quality of Service (QoS) based Admission Control in which more than one user can have successful transmissions at the same time slot. A class of delayed evolutionary game dynamics for correlated equilibria is given based on assignment functions from the set of signals to set of actions. The chapter 4 presents a class of bio-inspired game dynamics with migration in a hybrid evolutionary game model in Code Division Multiple Access (CDMA) wireless data networks, in OFDMA-based WiMAX environment and in association problem between several types of technologies such as UMTS 3G, High Speed (Downlink/Uplink) Packet Access (HSDPA/HSUPA), IEEE 802.16 (WiMAX), IEEE 802.11 (WiFi), Long Term Evolution (LTE) etc. Considering both location of users (regions) and secondary actions association in each location, we transform the migration problem in term of revision of strategies. Each user has its own state (location, specific technology). An individual state has two components: the current location of the user and the current secondary action. In contrast to the standard formulations in which users can not change their class or type, we consider here that the class which corresponds to its location can be changed in time. This captures the mobility of users in the population. We show that the order of revision of strategies is important in the leading dynamics (speed of convergence, Nash stationarity, positive correlation etc) .

Part II: Stochastic Population Games

The second part of this manuscript focuses on stochastic population games of evolving games. The chapter 5 introduces stochastic population games. This class of games is described by : (i) classes of large population of players, (ii) each player from each class has its own state and a set of actions available in each state, (iii) each player interacts with other randomly selected players (in the same class or from other classes). The states and actions of each player in an interaction together determine the instantaneous payoff for all involved players. They also determine the transition probabilities to move to the next state. Each player wishes to maximize the expected payoff during its sojourn time in the system.

• Energy management in hybrid Aloha-like networks We study energy management in a distributed Aloha network with large number of mobile terminals. Each mobile can choose both the channels and powers (this is in contrast to standard Aloha model in which users are associated to the closest receivers). Each terminal is faced to a random number of interacting players which transmit at the channel. A terminal attempts transmissions during a finite horizon of times depending on the state of its battery energy. At each slot, each terminal have to take a decision on the transmission power based on the battery state. At each state of the battery, there are a finite power levels. At the lowest state of battery no power is available and the mobile have to replaced the battery by a new or to recharge its battery. A transmission is successful if no other user transmit during the slot or the mobile transmits with a power which is bigger than the power of all others transmitting mobiles at the same receiver. We model each battery state as a Markov decision process.

We analyze this problem stochastic population games with individual states.

• Access control in solar-powered broadband networks

Environmental energy is becoming a feasible alternative for many low-power systems, such as wireless sensor and mesh networks. However, this provides an unpredictable and limited amount of energy over time. The power storage elements, such as rechargeable batteries or super-capacitors, become very useful to increase the system lifetime and the system availability. In particular, solar power is made possible with the use of Photovoltaic cells. Comprised of several layers of material, these cells are able to produce electrical power from exposure to sunlight. Since in many geographic areas, nice weather is not guaranteed and is unpredictable, the nodes should be able to recover from blackout periods caused by the unavailability of energy.

We extend our study to the case where the mobiles use power storage element, such as rechargeable Solar-powered batteries, in order to have energy available for later use. By modeling the energy-level of Solar-powered batteries as a stochastic process, we study noncooperative interactions within large population of mobiles that interfere with each other through many local interactions. Each local interaction involves a random number of mobiles. The actions taken by a mobile determine not only the immediate payoff but also the state transition probabilities of its battery. We define and characterize the evolutionary stable strategies of the stochastic population game.

The stochastic population game with dynamic rechargeable battery depending on the weather (solar energy) is described as follows: There are many local interactions among individuals belonging to large populations of mobiles. The result of the interaction between mobiles depends on their current individual state. From time to time the individual state of a mobile varies. The action choice of mobiles involved in a local interactions as well as their individual states determine the not only the result of the interactions but also the transition probabilities to the other possible individual states. Each individual is thus faced with an Markov Decision Process (MDP) in which it maximizes the expected average cost criterion. Each individual knows only the state of its own MDP, and does not know the state of the other mobiles it interacts with.

The destination of some transmission occasionally may receive simultaneously a transmission from another terminal which results in a collision. It is assumed however that even when packets collide, one of the packets can be received correctly if transmitted at a higher power. As state of the MDP of a user we take its energy level. The immediate fitness (rewards) is the number of successful transmissions. By allowing the mobiles to be equipped with rechargeable solar powered batteries, the mobiles may have infinite life time and the criteria that is maximizing is the limit average Cesaro-type payoff.

Part III: Mean Field Limits

The third part of this manuscript focuses on a class of mean field games and asymptotic properties of stochastic games. An interesting and important problem in n-player games is that of determining what happens when the number of players increases. Mean field asymptotics have been proposed to analyze this problem and a connection between the equilibria of limiting game and finite game is obtained. In addition, the dynamics of the system evolution is characterized. Those dynamics extend the standard evolutionary game dynamics. The chapter 6 studies the transition between evolutionary games with finite number of players and evolutionary games with infinite number of players. When the size of the population goes to infinity, a new class of deterministic population dynamics is given and sufficiency conditions for convergence and evolutionary stability are derived. A fluid model of spatial random access games is presented. The chapter 7 provides a rigorous derivation of the asymptotic behavior of the stochastic evolving games as the size of the population grows to infinity. Under restricted class of strategies, the random process consisting of one specific player and the remaining population converges weakly to a jump process driven by the solution of a system of differential equations. The large population asymptotic of the microscopic model is equivalent to a macroscopic stochastic population game in which a local interaction is described by a single player against an evolving population profile. We illustrate our model to derive the equations for a dynamic evolutionary Hawk and Dove game with energy level. We derive a new class of dynamic games called differential population games. Different from standard differential game models, this class of games is described by large population of players in which each player from each class is facing a random vector that evolves according to the population dynamics (deterministic or stochastic differential equations/inclusions). We derive equilibrium and optimality solutions using Hamilton-Jacobi-Bellman-Issacs equations. This research is of importance for dealing with complexity in stochastic dynamic optimization of large-scale systems and has methodological implications in many complex systems arising in engineering and socio-economic areas, ecology and evolutionary biology.

Part I

Delayed Evolutionary Game Dynamics with Migration

Chapter 2

Delayed evolutionary game dynamics

In this chapter, we study some evolutionary games where competition between individuals from a large population occurs through many pairwise interactions between randomly selected individuals. There are two features that make evolutionary games attractive to networking and communication systems. Its evolutionarily stability concept is better adapted to large populations of players as it describes robustness against deviations of a whole fraction of the population. The second appealing element of evolutionary games, the evolutionary game dynamics, describes the evolution of strategies in time. We study in particular an evolutionary game describing competition between mobile terminals over the access to a channel and we revisit the well known Hawk and Dove game (or Chicken game). In addition to the study of properties of the ESS in these games, we study the effect of time delays on the convergence of various evolutionary dynamics to the ESS. Delays play an important role in many situations in networking such as in flow control and congestion problem. We assume that time delays are not necessarily symmetric. The case of a common time delay for all strategies can be found in Tao & Wang (1997). We provide instability conditions and illustrate the non-stable behavior with numerical examples.

Several previous papers have already studied evolutionary games with pairwise local interactions in the context of wireless networks. Evolutionary games have been studied in the context of unslotted Aloha in [START_REF] Bonneau | An Evolutionary Game Perspective to ALOHA with Power Control[END_REF]. They have identified conditions for the existence of non trivial ESS and have computed them explicitly. In [START_REF] Tembine | Delayed Evolutionary Game Dynamics applied to the Medium Access Control[END_REF], we have considered the multiple access game and studied delay effect under various models of evolutionary game dynamics with asymmetric delay based on the theoretic results on stability obtained in [START_REF] Tembine | Asymmetric Delay in Evolutionary Games[END_REF]. In [START_REF] Tembine | Multiple Access Game in Ad-hoc Network[END_REF], we have extended the model of ( 215) by including a regret cost, incurred when no user transmits, and studied the impact of that cost on the proportion of mobiles that transmit at equilibrium. In the last three papers, the delay is shown to have negative impact on the stability of the system. For other applications of evolutionary games concepts in networking, see (147; 240; 12; 211; 13) who study congestion control models. The time delay is shown to have negative impact on the stability of the system and "evolutionarily stable state" can be unstable.

We introduce below asymmetric time delays into classical models of populations dynamics. An action taken today will have its effect some time later. The delays can be symmetric or not. We then obtain delayed payoff (or payoff, fitness, cost) functions. The evolution of the system leads to delayed evolutionary game dynamics. Delayed evolutionary game dynamics are in general a system of first order non-regular nonlinear differential equations or differential inclusions with time delays. We use the theory of delayed differential equation (DDE) to study stability, convergence and non-convergence of the system under time delays. Evolutionary game dynamics with asymmetric time delays have been introduced in (214; 216; 215; 13). In the case of finite number classes where each class has a finite number of pure actions, a simple class of delayed evolutionary game dynamics is given by ẋe

a (t) = ∑ b∈A e x e b (t)β e,b a (x(t), {x(t -τ e b )} e,b ) -x e a (t) ∑ b∈A e x e b (t)β e,a b (x(t), {x(t -τ e b )} e,b ) (2.1)
where τ e a denotes the time delay associated to the pure strategy a of class e. See chapter 6 for a derivation of this expression from mean field interactions or from Kolmogorov forward equations.

We now consider simple pairwise interactions. For such games there is a direct relation to a two players matrix game representing the expected fitness obtained in an interaction between two individuals (selected at random); the expectation is with respect to the fraction of the population that uses each strategy. We introduce asymmetric time delays in evolutionary game dynamics and study asymptotic stability of evolutionary stable strategies under these dynamics. We give sufficiency conditions of stability of the ESS which can not invaded by small proportion of deviations. Using delay differential equations properties, we show that the interior ESS of generic two-players games is asymptotically stable for small delays (Theorems 2.1.5, 2.1.6) and apply this result in the Hawk and Dove Game. We show that for a small time delay of the hawk strategy the system is asymptotically stable independently the Dove's time delay. In the common access problem in wireless adhoc networks, we show that the game has a unique ESS in which nodes which transmit and nodes which does not transmit can co-exist. We show how the evolution and the ESS are influenced by the characteristics of the wireless channel and pricing:

• We give a necessary and sufficient stability condition (Propositions 2.2.2.1 and 4.3.3) which depend on the regret cost in the multiple access game,

• We show that ESS is stable for small transmission delay under BNN and replicator dynamics for the regret cost lower to unit. The ESS becomes unstable for large delays.

• We show that the fraction of transmitters of the population of mobiles oscillates around the ESS for all small regret cost under imitate the better dynamics (Proposition 2.2.3.1 and remark 2.2.3.2).

The rest of this chapter is organized as follows. In Section 5.4, we describe our model of population game with delayed payoffs and we give an example of channel access problem in which delays must be take in consideration the interactions and we analyze delays impact on stability of the ESS in many dynamics. Multiple access game is studied in Section 2.2. We give some numerical examples of oscillatory regime in the access game. Section 2.4 extends delayed evolutionary game dynamics to asymmetric population games which one can have simultaneously intra-population interactions and inter-population interactions.

Setting and notations

We consider the standard setting of evolutionary games : there is a large populations of players; each member of the population has the same pure strategy set A (a finite set). There are many pairwise interactions. Denote by ∆(A) the (|A| -1)-dimensional simplex of R |A| .

Delays and Fitnesses

The fitness for a player at a given time is determined by the action a taken by the player at that time, as well as by the actions of the population it interacts with, that was taken τ a units ago. More precisely, If player 1 chooses the strategy b and player 2 the strategy a then player 1 receives the payoff u(b, a) (called fitness) only τ b times later and player 2 receives the payoff u(a, b) after a delay of τ a .

Below we denote by f a (x(tτ a )) the fitness for an individual using the strategy a at time t when it encounters an individual that used strategy x(tτ a ) at time tτ a . We have f a (x(t -

τ a )) = ∑ b∈A u(a, b)x b (t -τ a ), and we set f (x(t)) = ∑ a∈A x a (t) f a (x(t -τ a )
) to be the average expected fitness of the whole population at time t.

The following example in the context of wireless communications illustrates our model.

Example

Consider a transmitter A and a very close receiver B. Transmitter A can decide whether to transmit with frequency f 1 or f 2 . If it transmits with frequency f i then its signal receives interference at receiver B from transmitters that are located at a distance of D i from the receiver, which translates to a time delay of τ i (where i = 1, 2). Thus indeed, the action of a player determines the delay it will take for actions of other players to be effective.

Delayed evolutionary game dynamics

Evolutionary game dynamics are models of strategy change commonly used in evolutionary game theory. A strategy which does better than the average or its opponent, increases in frequency at the expense of strategies that do worse than the average or the opposed action. Many evolutionary game dynamics models are used in the literature (see (168; 99) and the references therein) : replicator dynamics, best response dynamics, Brown-von Neumann-Nash dynamics, Smith dynamics, gradient dynamics, projection dynamics etc. We introduce below delays into classical models of populations dynamics. Let β b a conditional switch rate from the strategy b to the strategy a. The flow of the population is specified in terms of the functions β b a which determine the rates at which an player who is considering a change in strategies opts to switch to his various alternatives. The incoming flow of the action a is ∑ b∈A x b (t)β b a (x(t)), and the outgoing flow

is x a (t) ∑ b β a b (x(t))
where x a (t) represents the fraction of players of the population which use the action a at time t. Let

W a f (x(t), {x(t -τ k )} k ) = ∑ b∈A x b (t)β b a (x(t), {x(t -τ k )} k ) -x a (t) ∑ b∈A β a b (x(t), {x(t -τ k )} k ).
The evolutionary game dynamics (234; 168; 208) is given by

ẋa = W a f (x(t), {x(t -τ k )} k ), a ∈ A. (2.2)
For example the delayed replicator dynamics, delayed Brown-von Neumann-Nash dynamics and generalized projection dynamics are obtained respectively for

β b a (x(t), {x(t -τ k )} k ) = x a (t) max(0, f a (x(t -τ a )) -f b (x(t -τ b ))), β b a (x(t), {x(t -τ k )} k ) = max(0, f a (x(t -τ a )) -∑ b∈A x b (t) f b (x(t -τ b ))), β b a (x(t), {x(t -τ k )} k ) = h a (x(t), {x(t -τ k )} k ).

Delayed replicator dynamics

In the delayed replicator dynamics, the share of a strategy in the population grows at a rate equal to the difference between the delayed payoff of that strategy and the average delayed payoff of the population. The fitness acquired at time t will impact the rate of growth τ a time later. Thus the delayed replicator dynamics of x a (t) is given by

a ∈ A, ẋa (t) = x a (t) f a (x(t -τ a )) -∑ b∈A x b (t) f b (x(t -τ b )) .
The standard replicator dynamics is obtained when all the delays are zero.

Delayed BNN dynamics

Delays can be introduced in BNN dynamics, [START_REF] Brown | Solutions of games by differential equations[END_REF] as follows

a ∈ A, ẋa (t) = g a (x(t)) -x a (t) ∑ b∈A g b (x(t)) (2.3)
where

g a (x(t)) = max 0, f a (x(t -τ a )) -∑ b∈A x b (t) f b (x(t -τ b )) .
The function g a (x(t)) is the positive part of the excess payoff for strategy a when the population state is x(t) at time t.

There are a close relation between BNN dynamics with Nash's original proofs of his equilibrium theorem (see [START_REF] Hofbauer | Evolutionary game dynamics[END_REF]). BNN dynamics is interpreted in [START_REF] Hofbauer | Evolutionary game dynamics[END_REF] as follows: Suppose there are large users in the population in which there is steady influx and outflux. New users joining the system use only strategies that are better than average, and better strategies are more likely to be adopted. On the other hand, randomly chosen users leave the game. More precisely, the strategy x ∈ ∆(A) is adopted with probability proportional to the excess payoff g a (x).

Delayed imitate the better dynamics

We introduce delayed myopic imitation dynamics. We suppose that users in the population review their strategy and imitate the better's strategy on the time (imitation by dissatisfaction). In a symmetric two player game, the delayed dynamic is given by

b ∈ A, ẋb (t) = x b (t) ∑ a∈A x a (t) [ρ ab (x(t -τ a )) -ρ ba (x(t -τ b ))] , b ∈ A (2.4)
where

ρ ab (x) = g( f b (x(t -τ b )), f a (x(t -τ a ))), a, b ∈ A and g(a, b) = 0 if a < b and g(a, b) = 1 if a > b.

Delayed best response dynamics

A strategy Y ∈ ∆(A) is a best response to the trajectory x(t) at time t if ∑ a∈A Y a f a (x(t -τ a )) ≥ ∑ a∈A v a f a (x(t -τ a )), ∀v ∈ ∆(A).
We note by BR(x(t)) the best response strategies set to the strategy x(t) i.e

BR(x(t)) = Y ∈ ∆(A), ∑ a∈A Y a f a (x(t -τ a )) ≥ ∑ a∈A v a f a (x(t -τ a )), ∀v ∈ ∆(A)
A fraction of population revise their strategy and choose the best replies BR(x(t)) to the current population state x(t) at time t. The best response dynamics is given by ẋ ∈ BR(x)x.

Delayed logit dynamics

The delayed logit dynamics (called also delayed exponential weight dynamics or delayed Boltzman dynamics) is given by

a ∈ A, ẋa (t) = e 1 ǫ f a (x(t-τ a ))
∑ b∈A e

1 ǫ f b (x(t-τ b )) -x a (t) (2.5)
It can be shown that the delayed logit dynamics given by

a ∈ A, ẋa (t) = e f a (x(t-τ a ))T 0 ∑ b∈A e f b (x(t-τ b ))T 0 -x a (t) (2.6)
converge to the best response dynamics if T 0 → +∞. See [START_REF] Hofbauer | Evolutionary game dynamics[END_REF].

Delayed ray-projection dynamics

The Ray-projection dynamics with time delays is given by

a ∈ A, ẋa (t) = h a (x(t), {x(t -τ b )} b ) -x a (t) ∑ b∈A h b (x(t), {x(t -τ b )} b ) (2.7)
where g is a vectorial function of the the population profile and the vector of expected payoffs.

Delayed orthogonal projection dynamics

The orthogonal-projection dynamics with time delays is given by

a ∈ A, ẋa (t) = f a (x(t -τ a )) - 1 n ∑ b∈A f b (x(t -τ b )) (2.8)
where g is a vectorial function of the the population profile and the vector of expected payoffs.

Delayed target projection dynamics

ẋ(t) = proj TX ( f ({x(t -τ k } k )) -x(t) (2.9)

Definitions

Consider the following delay differential equations

dx a dt = ẋa (t) = Θ a (x a (t), x(t -τ 1 ), . . . , x(t -τ |A| )), a ∈ A, x(.) ∈ ∆(A) (2.10)
Denote by τ the maximum of τ a , a ∈ A

∀ t ≥ t 0 , |x(t) -x * | ≤ Le -ηt .
From these definitions, it follows that exponential stability implies asymptotic stability implies stability which implies stationary point of system.

Stability analysis of the delayed replicator dynamics

In this subsection, we introduce replicator dynamics for a population and |A| asymmetric delays with |A| ≥ 2.

ẋa (t) = x a (t) f a (x(t -τ a )) -∑ b∈A x b (t) f b (x(t -τ b )) , a ∈ A. ( 2
= x * a f a (z(t -τ b )) -∑ b∈A x * b f b (z(t -τ b )) , a ∈ A (2.12)
where z(t) = x(t)x * . This equation can be written as

ż(t) = K ∑ l∈A B l z(t -τ l )
where

z(t) = t (z 1 (t), . . . , z |A| (t)) and B l = b l ik ∈ R |A|×|A| , b l ik = x * i (1 -x * i )u ik if l = i -x * i x * l u ik if l = i. Characteristic equation det λI -K ∑ l∈A B l e -τ l λ = 0. (2.13)
A necessary and sufficient condition of stability of (2.12) is that all roots of the equation (3.7) have negative real parts.

Two actions

Suppose that each individual of the population only uses a pure strategy. Denote ξ(t) = x 1 (t) the proportion of individuals in the population using the first strategy at time t, then replicator dynamic of ξ(t) is given by

ξ(t) = -Kδξ(t)(1 -ξ(t)) × (u 12 -u 11 ) δ ξ(t -τ 1 ) + (u 21 -u 22 ) δ ξ(t -τ 2 ) -ξ * (2.
∀ mut = ξ * , (ξ * , 1 -ξ * )U mut 1 -mut -(mut, 1 -mut)U mut 1 -mut (2.16) = δ(ξ * -mut) 2 (2.17) Hence (ξ * , 1 -ξ * ) is an ESS.
The linearized version of (2.14) at the stationary point x * is given by

ż(t) = -γ [αz(t -τ 1 ) + βz(t -τ 2 )] (2.18) where γ = Kξ * (1 -ξ * ), α = u 12 -u 11 , β = u 21 -u 22 . Let ż(t) = -az(t -τ 1 ) -bz(t -τ 2 ) (2.19)
where a, b ∈ R and τ 1 , τ 2 are positive. The characteristic equation of (2.19) is given by

s = -ae -τ 1 s -be -τ 2 s (2.20)
Gopalsamy (1992) proved that when a, b, τ 1 , τ 2 are positive, a sufficient condition for all roots of (2.20) to have negative real parts is aτ 1 + bτ 2 < 1 and a necessary condition for the same is

aτ 1 + bτ 2 < π 2 .
The following theorem give sufficient conditions of stability of (2.19) at zero.

Theorem 2.1.5 [START_REF] Berezansky | On stability of some linear and nonlinear delay differential equations[END_REF], [START_REF] Li | Stability and bifurcation in delay-differential equations with two delays[END_REF]). Suppose at least one of the following conditions holds

1. a + b > 0, |a|τ 1 + |b|τ 2 < a + b |a| + |b| 2. a > 0, aτ 1 < a -|b| a + |b| 3. b > 0, bτ 2 < b -|a| |a| + b Then equation (2.
= -e -r 1 λ -Ae -r 2 λ (2.21)
where λ = s a , r 2 = aτ 2 . Since all roots of equation (2.21) have negative real parts when r 1 = 0, if the conclusion fails, then there must be exist some r 1 ∈ (0,

1 1 + A ] such that equation (2.21)
has purely imaginary roots ±iy, y > 0 satisfying cos(r 1 y) = -A cos(r 2 y) and ysin(r 1 y) = A sin(r 2 y)

(2.22) One has A 2 -1 = y 2 -2y sin(r 1 y) that is f (y) := y 2 + 1 -A 2 2y = sin(r 1 y) (2.23) Since -1 ≤ sin(r 1 y) ≤ 1 ⇒ y ∈ (1 -A, 1 + A). f (y) = y 2 1 + 1 -A 2 y 2 ≥ y 2 1 + 1 -A 2 (1 + A) 2 ≥ y 1 1 + A > r 1 y ≥ sin(r 1 y) (2.24)
a contradiction with (2.23).

Corollary 2.1.6.1. If one of the following conditions holds, then ξ * is stable

1. α + β = δ > 0, K (|α|τ 1 + |β|τ 2 ) < δ ξ * (1 -ξ * ) (|α| + |β|) 2. α > 0, Kτ 1 < α -|β| αξ * (1 -ξ * ) (α + |β|) . 3. β > 0, Kτ 2 < β -|α| βξ * (1 -ξ * ) (|α| + β) . 4. 0 < β < α, Kτ 1 < 1 δξ * (1 -ξ * )
The case of symmetric delay Consider the following linear delay differential equation

z(t) = -ωz(t -τ) (2.25)
with τ, ω > 0 and its characteristic equation

λ + ωe -λτ = 0 (2.26)
Necessary and sufficient condition of asymptotic stability as a function of the delay is given by the following lemma. A proof can be found in Gopalsamy (1992, Proposition

Multiple Access Game with Regret Cost

Multiple Access Game introduces the problem of medium access. We assume that mobiles are randomly placed over a plane. All mobiles use the same fixed transmission range of r. The channel is ideal for transmission and all errors are due to collision. A mobile decides to transmit a packet or not to transmit to a receiver when they are within transmission range of each other. Interference occurs as in the ALOHA protocol: if more than one neighbors of a receiver transmit a packet at the same time then there is a collision. The Multiple Access Game is a nonzero-sum game, the mobiles have to share a common resource, the wireless medium, instead of providing it. We suppose that a mobile has a receiver in its range with probability K. When a mobile m transmits to its receiver R(m), all mobiles within a circle of radius r centered at the receiver R(m) cause interference to the node m for its transmission to receiver R(m). This means that more than one transmission within a distance r of the receiver in the same slot result in a collision of all packets at the receiver. Each of the mobiles has two possible strategies: either to transmit (T) or to stay quiet (S). If mobile m transmits a packet, it incurs a transmission cost of δ ≥ 0. The packet transmission is successful if the other mobiles don't transmit (stays quiet) in that given time slot, otherwise there is a collision, and the cost (of collision risk) is ∆ ≥ 0. If there is no collision, mobile m gets a reward of V from the successful packet transmission after the τ T times later. We assume that the reward V is greater than the cost of transmission ∆. When all mobiles stay quiet, they have to pay a regret cost κ. The regret cost κ describes the behavior of mobiles when they are aware of the backoff delays. The interaction between two mobiles is represented in Figure 3.27. The value of successful transmission is normalized to unit.

T S T -∆K K(V -∆) S 0 -κK

Nash Equilibria, ESS and Pareto optimality

An allocation of payoffs is said Pareto-optimal if the outcome cannot be improved upon without hurting at least one user. This matrix game has two pure Nash equilibria (T, S) and (S, T) and a unique mixed Nash equilibrium given by (

1 -∆ + κ 1 + κ , ∆ 1 + κ
). Note that the pure strategies (T, S) and (S, T) are also optimal in the Pareto sense. The mixed equilibrium converges the pure strategy T when the regret cost is large.

Proposition 2.2.1.1. The mixed strategy (

1 -∆ + κ 1 + κ , ∆ 1 + κ
) is the unique ESS of the multiple access game.

Proof. This comes from the theorem 2.1.4. The strategy (

1 -∆ + κ 1 + κ , ∆ 1 + κ
) is the unique interior Nash equilibrium. It is the unique symmetric Nash equilibrium. Thus, it is the only candidate to be ESS because symmetric Nash equilibria set contains ESS set. At a mixed equilibrium, the strategies T and S must have the same fitness. We check the condition of ESS given in (1.3). For

all ξ = 1 -∆ + κ 1 + κ 1 -∆ + κ 1 + κ -ξ, ξ - (1 -∆ + κ) 1 + κ A ξ 1 -ξ = K 1 + κ ((1 + κ)ξ -1 + ∆ -κ) 2 > 0 We conclude that ( 1 -∆ + κ 1 + κ , ∆ 1 + κ ) an ESS.
Note that at the ESS, the two subpopulations obtain the same negative payoff equal to

-K ∆ 1 + κ .

Stability bound for ESS under replicator dynamics

If (ξ(t), 1ξ(t)) denotes de repartition of the population in the multiple access game, then the fitness of the subpopulation using the strategy T at time t is K(-ξ(tτ T ) + 1 -∆), and the fitness of the strategy S is -κK(1ξ(t)). The replicator dynamic equation becomes

ξ(t) = -Kξ(t)(1 -ξ(t)) [ξ(t -τ T ) + κξ(t) -1 + ∆ -κ] (2.28)
and its linearized version around the stationary point ξ * ,

ż(t) = - Kξ * (1 -ξ * ) 1 + κ [κz(t) + z(t -τ T )] , τ T > 0, ( 2.29)
is asymptotically stable if and only if all roots of the characteristic equation Proof.

λ + Kξ * (1 -ξ * ) 1 + κ κ + e -τ T λ = 0 (2.
• Suppose there exists a root λ of the characteristic equation (2.30) with ℜ(λ) > 0 and κ > 1 then

ℜ(λ) + Kξ * (1 -ξ * ) κ 1 + κ > Kξ * (1 -ξ * ) κ 1 + κ > Kξ * (1 -ξ * ) 1 + κ (2.31) > - Kξ * (1 -ξ * ) 1 + κ e -τℜ(λ) cos(τℑ(λ)) (2.32)
a contradiction with (2.30). Thus all roots of the characteristic equation (2.30) have real parts strictly negative.

• Suppose that κ = 1.

   ℜ(λ) + Kξ * (1 -ξ * ) κ 1 + κ = -Kξ * (1 -ξ * ) κ 1 + κ e -τℜ(λ) cos(τℑ(λ)) ℑ(λ) = Kξ * (1 -ξ * ) κ 1 + κ e -τℜ(λ) sin(τℑ(λ)) (2.33) If ℜ(λ) > 0 then ℜ(λ) + Kξ * (1 -ξ * ) κ 1 + κ = -Kξ * (1 -ξ * ) κ 1 + κ e -τℜ(λ) cos(τℑ(λ)) ≤ Kξ * (1 -ξ * ) κ 1 + κ i.e ℜ(λ) ≤ 0 : impossible. If ℜ(λ) = 0 then ℑ(λ) = 0
≤ (1 + κ) 2 K∆(1 -∆ + κ) 1 + κ 1 -κ × cos -1 (-κ) (2.34)
Note that when κ goes to 1, the bound

(1 + κ) 2 K∆(1 -∆ + κ) 1 + κ 1 -κ cos -1 (-κ) goes to infinity (sta- bility without conditions).
Proof.

• Suppose that τ < τ 0 :=

(1 + κ) 2 K∆(1 -∆ + κ) cos -1 (-κ) 1 + κ 1 -κ . When τ = 0 the system (2.30) becomes ℑ(λ) = 0 and ℜ(λ) = -K ∆(1 -∆ + κ) (1 + κ) 2 < 0. Let b = K ∆(1 -∆ + κ) (1 + κ) 2 , a = κb. If (2.30
) has a solution with real part strictly positive, then there must be some 0 < τ 1 < τ 0 such that (2.30) has purely imaginary roots ±ℑ(λ

) × i with ℑ(λ) > 0 satisfying a = -b cos(τ 1 y) y = b sin(τ 1 y) ⇔ a = -b cos(τ 1 y) y = b sin(τ 1 y), y > 0 ⇔ y = b 2 -a 2 τ 1 = τ 0
We proved that τ 1 = τ 0 which is a contradiction. Thus, if 0 < τ < τ 0 then all solutions of the characteristic equation (2.30) have negative real parts.

• Suppose that τ > τ 0 . Denote by s = λ/b, r = τb > 0, r 0 = τ 0 b. The equation (2.30) can be written as the zeros the function

φ(s, κ) := s + κ + e -rs
We want to prove that all zeros of φ(., κ) have negative real parts. The function φ is an analytic function in s and κ. Let

O := {s, ℜ(s) ≥ 0, |s| ≤ 1} One has r 0 ≥ π 2 . Indeed, arccos(-p) - π 2 1 -p 2 > 0, p ∈ (0, 1).
Thus when r > r 0 and r = 2kπ + π 2 , k ∈ N the function φ(s, 0) has no zeros in the boundary of O. Using Rouche's theorem φ(s, 0) and φ(s, κ) have same sum of the orders of zeros. When r > r 0 the zeros of φ(s, 0) is at least 2. Thus, φ(s, κ) has at least one root with positive real part.

Imitate the better dynamics

The imitate the better equation can be expressed as

ξ(t) = ξ(t)(1 -ξ(t))h(ξ(t -τ), ξ(t)) where h(ξ(t -τ), ξ(t)) =    1 if ξ(t -τ) + κξ(t) < 1 -∆ + κ 0 if ξ(t -τ) + κξ(t) = 1 -∆ + κ -1 if ξ(t -τ) + κξ(t) > 1 -∆ + κ Proposition 2.2.3.1. Suppose that κ < 1.
For all τ > 0, and for all initial conditions ∀ t ∈ (-τ, 0],

ξ(t) = ξ 0 ∈ (0, 1 -∆ + κ 1 + κ
) the solution of the imitation equation oscillates around the ESS.

Proof. Consider the following differential equations

       ξ(t) = ξ(t)(1 -ξ(t)) if ξ(t -τ) + κξ(t) < 1 -∆ + κ ξ(t) = 0 if ξ(t -τ) + κξ(t) = 1 -∆ + κ ξ(t) = -ξ(t)(1 -ξ(t)) if ξ(t -τ) + κξ(t) > 1 -∆ + κ ∀ t ∈ (-τ, 0], ξ(t) = ξ 0 ( * * ) ẇ(t) = w(t)(1 -w(t)) w(0) = ξ 0 , ξ 0 ∈ (0, 1 -∆ + κ 1 + κ ).
The solution of ( * * ) is given by w

(t) = 1 - 1 1 + χe t where χ = ξ 0 1 -ξ 0
. The function w goes to 1 when t goes to infinity. The functions ξ and w are equal in (0, t 1 ) where t 1 = t r + t * and t * the first time such that w(t) = ξ * (this time exists because w is a continuous function, starts at ξ 0 which is lower than ξ * and goes to 1),

t 1 = inf{t > 0, ξ(t -τ) + κξ(t) > 1 -∆ + κ}.
We can explicitly compute t * and t 1 ,

t * = log ξ * χ(1 -ξ * ) , t 1 = log ν + ν 2 + 4ξ * (1 -ξ * )e -τ 2χ(1 -ξ * )e -τ where ν = -(1 -ξ * )(1 + e -τ ) + 1 + κe -τ 1 + κ . One has t * + τ > t 1 > t * and ξ 1 = ξ(t 1 ) = w(t 1 ) > ξ * . Let consider the equation v(t) = -v(t)(1 -v(t)) v(t 1 ) = ξ 1 Then, ξ(t) = v(t) = 1 1 + e t χe 2t 1 on (t 1 , t ′ 1 )
where

t ′ 1 = log χe 2t 1 α + α 2 + 4(1 -ξ * )ξ * e -τ 2e -τ ξ * , α = -ξ * (1 + e -τ ) + 1 + κe -τ 1 + κ . ξ(t ′ 1 ) = v(t ′ 1
) is lower than ξ * . Remarks 2.2.3.2. Using the same arguments as in proposition 2.2.3.1, we extend the result to all initial conditions. Suppose that that κ < 1. Then, for all τ > 0, and for all initial conditions ξ(t) = ξ 0 ∈ (0, 1) on (-τ, 0), the solution of imitation equation oscillates around the ESS.

Numerical illustrations

Impact of regret cost

The ESS depends on the parameter κ. When κ is large, the ESS point is closed to (1, 0) : the pure strategy transmit. Our first numerical experiment studies the convergence of these dynamics for the case of the unit growth parameter K and a small delay τ = 0.02 as a function of the regret cost κ : we check the speed of convergence and the stability of the dynamics as a function of the cost κ. The state 2/3 is a stationary point for these parameters, for which 2/3 of the population choose to transmit. We took ∆ = 1/3. Figure 2.2 (resp. Figure 2.3) represents the trajectories of the population using the strategy T in the imitation dynamic (resp. replicator dynamic) when the initial condition is ξ(t) = 0.02, ∀t ∈ (-τ, 0).

Impact of delay of transmission with regret cost

The ESS does not depend on the delay but the delay has a big influence on the stability of the system. Our second numerical experiment studies the convergence of these dynamics for the case of the unit growth parameter K as a function of delay τ.We took ∆ = 1/3 and κ = 0.002. The resulting trajectories of the population using the strategy T is represented as a function of time. We evaluate the stability varying the delay τ between 0.02 and 15 time units in the replicator dynamic in Figure 2.4 and Figure 2.5. For τ = 0.02, we have stability but the convergence speed is slow. The other extreme is illustrated for τ = 15 which the trajectory oscillates rapidly and the amplitude is seen to be greater than 2/3. The system is unstable. Figure 2.6 represents the fraction of the population using the strategy T in the imitation dynamic in which the system is unstable for all τ T > 0. The amplitude of oscillation growth with the delay.

Impact of the delay of transmission without regret cost

We evaluate the stability varying the delay τ T between 0.02 and 10 time units in the replicator dynamic without regret cost in Fig. 2.7. For τ T = 0.02, we have stability but the convergence speed is slow. The other extreme is illustrated for τ T = 10 which the trajectory oscillates rapidly and the amplitude is seen to be greater than 2/3. The system is unstable. Figure 2.8 represents the trajectories of the population using the strategy T in the imitation dynamic in which the system is unstable for all τ T > 0. The amplitude of oscillation growth with the delay. The periodicity increases when the delay decreases. The best response at the time t when the population

ξ(t) is BR(ξ(t)) = I [0,1-∆] (ξ(t -τ T )) + [0, 1]I {1-∆} (ξ(t -τ T ))
where I A (.) is the indicator function of the set A. In Figure 2.9, the best response dynamics is represented for respectively τ T = 0.04, 1, 2. The mixed strategy (1/2, 1/2) is chosen for the best response to (1 -∆, ∆). We evaluate the stability varying the delay τ T between 0.04 and 8.4823 time units in the Brown-von Neumann-Nash in Fig. 2.10. The BNN dynamics becomes In these figures we observe cases of stable and of non-stable behavior. All turn out to confirm the stability conditions that we obtained above.

ξ(t) =    K(1 -ξ(t) 2 )(-ξ(t -τ T ) + 1 -∆) if ξ(t -τ T ) < 1 -∆ 0 if ξ(t -τ T ) = 1 -∆ Kξ(t) 2 (-ξ(t -τ T ) + 1 -∆) if ξ(t -τ T ) > 1 -∆ For τ T = 0.
ξ(t) τ T =0.04 τ T =1 τ T = 2

Notes

This chapter is based on our publications (13; 12; 215; 214). We considered evolutionary games with one population of users and studied delays impact on convergence to ESS for different types of evolutionary dynamics. Using some results from delay differential equation theory, we observed stability phenomena that are new with respect to non-delayed evolutionary game dynamics. In all the dynamics considered, delays were shown to have negative impact and ESS can be unstable when the delays are large. In the context of the access evolutionary game of mobile terminals, this suggest that updating MAC strategies in the terminals have to be done with care so as to avoid the oscillatory behavior that we observed in the non-stable regime.

Evolution of Transport Protocols in Wireless and Wired Networks

Todays Internet is well adapted to the evolution of protocols at various network layers. Much of the intelligence of congestion control is delegated to the end users and they have a large amount of freedom in the choice of the protocols they use. In the absence of a centralized policy for a global deployment of a unique protocol to perform a given task, the Internet experiences a competitive evolution between various variants of protocols. The evolution manifests itself through the upgrading of existing protocols, abandonment of some protocols and appearance of new ones. We highlight in this paper the modelling capabilities of the evolutionary game paradigm for explaining past evolution and predicting the future one. In particular, using this paradigm we derive conditions under which (i) a successful protocol would dominate and wipe away other protocols, or (ii) various competing protocols could coexist. In the latter case we also predict the share of users that would use each of the protocols. We further use evolutionary games to propose guidelines for upgrading protocols in order to achieve desirable stability behavior of the system.

When transferring data between nodes, flow control protocols are needed to regulate the transmission rates so as to adapt to the available resources. A connection that looses data units has to retransmit them later. In the absence of adaptation to the congestion, the on going transmissions along with the retransmissions can cause increased congestion in the network resulting in losses and further retransmissions by this and/or by other connections. This type of phenomenon, that leads to several 'congestion collapses' [START_REF] Van Jacobson | Congestion Avoidance and Contrnol[END_REF], motivated the evolution of the Internet transport protocol, TCP, to a protocol that reduces dramatically its throughput upon congestion detection.

The possibilities to deploy freely new versions of protocols on terminals connected to the Internet creates a competition environment between protocols. Much work has been devoted to analyze such competition and to predict its consequences. The two main approaches for predicting whether one version of a protocol would dominate another one are • Inter Population Competition (IRPC): One examines local interactions between connections of different types that interact with each other (by sharing some common bottleneck link). If a connection that corresponds to one version performs better in such an interaction, then the DC approach predicts that it would dominante and that the other version would vanish.

• Intra Population Competition (IAPC): In this approach one studies the performance of a version of a protocol assuming a world where all connections use that version. This is repeated with the other version. One then predicts that the version that gives a better world would dominate.

We address the dominance question with the evolutionary game paradigm and provide an alternative answer along with a more detailed analysis of this competition scenario. Our approach predicts whether one can expect one protocol to dominate the other or whether the two protocols can be expected to coexist. It provides with the tools for computing the share of the population that is expected to use each version in case the versions would coexist. Finally, it provides a description of the dynamics of the competition, which may result in a stable behavior that consists of a convergence to some equilibrium, or it may display instabilities and oscillations. By identifying the conditions for a stable behavior, one can provide guidelines for upgrading protocols so as to avoid undesirable oscillating behavior.

We provide a framework to describe and predict evolution of protocols in a context of competition between two types of behaviors: aggressive and peaceful. We compute the ESS for congestion protocols of different degree of aggressiveness. We identify cases in which at ESS, only one population prevails (ESS in pure strategies) and others, in which an equilibrium between several population types is obtained. To study this, we map the problems, whenever possible, into the Hawk and Dove Game. We then study the convergence of the replicator dynamics to it.

We develop a framework for controlling evolutionary dynamics (changing or upgrading protocols) through the choice of a gain parameter governing the replicator dynamics. We address the following two design issues concerning this choice:

(i) the tradeoff between fast convergence and stability. We identify a simple threshold on the gain parameter in the replicator dynamics such that the stability is only determined by whether we exceed or not the threshold.

(ii) the stability as a function of delays. We derive new stability conditions for the replicator dynamics in the Hawk and Dove game with non-symmetric delays and apply it to the evolution of the MAC and transport layer protocols.

We first provide in the next subsection the needed background on evolutionary games. We summarize work on competition between TCP versions in subsection 2.3.1. We then study the ESS for congestion control protocols (subsection 2.3.2). After that, we investigate the impact of the choice of some parameters in the replicator dynamics on the stability of the system in subsection 2.3.4. Finally we give some numerical investigations.

Competition Between Congestion Control Protocols

There are various versions of the TCP protocol among which the mostly used one is New-Reno. The degree of 'aggressiveness' varies from version to version. The behavior of New-Reno is approximately AIMD (Additive Increase Multiplicative Decrease): it adapts to the available capacity by increasing the window size in a linear way by α packets every round trip time and when it detects congestion it decreases the window size to β times its value. The constants α and β are 1 and 1/2, respectively, in New Reno.

In last years, more aggressive TCP versions have appeared, such as HSTCP (High Speed TCP) [START_REF] Souza | A High Speed TCP study: Characteristics and deployment issues[END_REF] and Scalable TCP (122; 72; 5). HSTCP can be modeled by an AIMD behavior where α and β are not constant anymore : α and β have minimum values of 1 and of 1/2, resp. and both increase in the window size. Scalable TCP is an MIMD (Multiplicative Increase Multiplicative Decrease) protocol, where the window size increases exponentially instead of linearly and is thus more aggressive. Versions of TCP which are less aggressive than the New-Reno also exist, such as Vegas [START_REF] Brakmo | Tcp vegas: New techniques for congestion detection and avoidance[END_REF].

Several researchers have analyzed the performance of networks in which various transport protocols coexist, see (200; 51; 1; 6; 135). In all these papers, the population size using each type of protocol is fixed. Some papers have already considered competition between aggressive and well behaved congestion control mechanisms within a game theoretic approach. Their conclusions in a wireline context was that if connections can choose selfishly between a well behaved cooperative behavior and an aggressive one then the Nash equilibrium is obtained by all users being aggressive and thus in a congestion collapse (89; 136).

We introduce in the next two sections two models of competition between TCP versions, both can be modeled within the framework of the Hawk and Dove game. This will allow us to predict whether a given version of TCP is expected to dominate others (ESS in pure strategies, which means that some versions of TCP would disappear) or whether several versions would co-exist. The first model is adapted to competition in wireless networks and the second to wireline networks.

Competition in Wireless Networks

During the last few years, many researchers have been studying TCP performances in terms of energy consumption and average goodput within wireless networks (180; 243). Via simulation, the authors show that the TCP New-Reno can be considered as well performing within wireless environment among all other TCP variants and allows for greater energy savings. Indeed, a less aggressive TCP, as TCP New-Reno, may generate lower packet loss than other aggressive TCP. Thus the advantage of an aggressive TCP in terms of throughput could be compensated with energy efficiency of a more gentle TCP version. (In Section 2.3.3 we shall illusutrate another consideration that affects the competition between TCP versions.) The goal of this section is to illustrate this point, as well as its possible impact on the evolution of the share of TCP versions, through a simple model of an aggressive TCP.

The model. We consider two populations of connections, all of which use AIMD TCP. A connection of population i is characterized with a linear increase rate α i and a multiplicative decrease factor β i . Let x i (t) be the transmission rate of connection i at time t. We consider the following simple model for competition.

• The RTT (round trip times) are the same for all connections.

• There is light traffic in the system in the sense that a connection either has all the resources its needs or it shares the resources with one other connection. (If files are large then this is a light regime in terms of number of connections but not in terms of workload).

• Losses occur whenever the sum of rates reaches the capacity C: x 1 (t) + x 2 (t) = C.

• Losses are synchronized: when the combined rates attain C, both connections suffer from a loss. This synchronization has been observed in simulations for connections with RTTs close to each other (2). The rate of connection i is reduced by the factor β i < 1.

• As long as there are no losses, the rate of connection i increases linearly by a factor α i .

We say that a TCP connection i is more aggressive than a connection j if α i ≥ α j and β i ≥ β j . Let β i := 1β i . Let x n and y n be the transmission rates of connection i and j, respectively, just before a loss occurs. We have x n + y n = C. Just after the loss, the rates are β 1 x n and β 2 y n . The time it takes to reach again C is

T n = C -β 1 x n -β 2 y n α 1 + α 2
which yields the difference equation:

x n+1 = β 1 x n + α 1 T n = qx n + α 1 Cβ 2 α 1 + α 2 where q = α 1 β 2 + α 2 β 1 α 1 + α 2 .
The solution is given by

x n = q n x 0 + α 1 Cβ 2 α 1 + α 2 1 -q n 1 -q .

HD game: throughput-loss tradeoff

In wireline, the utility related to file transfers is usually taken to be the throughput, or a function of the throughput (e.g. the delay). It does not explicitly depend on the loss rate. This is not the case in wireless context. Indeed, since TCP retransmits lost packets, losses present energy inefficiency. Since energy is a costly resource in wireless, the loss rate is included explicitly in the utility of a user through the term representing energy cost. We thus consider fitness of the form J i = Thp i -λR for connection i; it is the difference between the throughput Thp i and the loss rate R weighted by the so called tradeoff parameter, λ, that allows us to model the tradeoff between the valuation of losses and throughput in the fitness. We now proceed to show that our competition model between aggressive and non-aggressive TCP connections can be formulated as a HD game. We study how the fraction of aggressive TCP in the population at (the mixed) ESS depends on the tradeoff parameter λ.

Since |q| < 1, we get the following limit x of x n when n → ∞:

x = α 1 Cβ 2 α 1 + α 2 • 1 1 -q = α 1 β 2 C α 1 β 2 + α 2 β 1 .
It is easily seen that the share of the bandwidth (just before losses) of a user is increasing in its aggressiveness. Hence the average throughput of connection 1 is

Thp 1 = 1 + β 1 2 × α 1 β 2 α 1 β 2 + α 2 β 1 × C.
The average loss rate of connection 1 is the same as that of connection 2 and is given by

R = 1 T = α 1 β 1 + α 2 β 2 1 C where T = β 1 β 2 C α 1 β 2 + α 2 β 1
with T being the limit as n → ∞ of T n .

Let H corresponds to (α H , β H ) and D to (α D , β D ) such that α H ≥ α D and β H ≥ β D . Then, for i = 1, 2, Thp i (H, H) = Thp i (D, D). Since the loss rate for any user is increasing in α 1 , α 2 , β 1 , β 2 it then follows that J(H, H) < J(D, D), and J(D, H) < J(D, D). We conclude that the utility that describes a tradeoff between average throughput and the loss rate leads to the HD structure.

The mixed ESS is given by the following probability of using H:

x * (λ) = η 1 -η 2 λ η 3
where

η 1 = µ 1 + β 1 2 - 1 + β 2 4 C, η 2 = 1 C α 1 β 1 - α 2 β 2 , η 3 = C( 1 2 -µ) β 1 -β 2 2 , µ = α 2 (β 1 ) α 2 (β 1 ) + α 1 (β 2 )
.

where µ := 1µ. Note that η 2 and η 3 are positive. Hence, the equilibrium point x * decrease linearly on λ. We conclude that applications that are more sensitive to losses would be less aggressive at ESS (Braess type paradoxes do not occur here).

Competition in Wireline Networks

Tradeoff between throughput and fairness Consider a number of TCP connections that share a common bottleneck. We study the competition between the New-Reno version of TCP and the Scalable version of TCP that has been proposed [START_REF] Kelly | Scalable TCP: Improving Performance in Highspeed Wide Area Networks[END_REF] in the context wireline networks that are characterized by a very high speed and has long distance (and thus large delays). Scalable TCP is then much more aggressive than the New-Reno TCP as it is an MIMD (Multiplicative Increase Multiplicative Decrease) protocol, where the window size increases exponentially instead of linearly at the absence of losses. It is also more aggressive when losses are detected: it decreases its window size (and thus the transmission) rate to 0.875 times the one it had prior to the loss (instead of to halving it as is done in New-Reno). We begin by examining the Inter and Intra Population Competition.

Intra Population Competition

If the connections are all symmetric and use the same AIMD version, then we know that they share equally the bandwidth [START_REF] Altman | Parallel TCP Sockets: Simple Model, Throughput and Validation[END_REF]. Moreover if the number of connections or the buffer size is large then the connections use all the available bandwidth so that the throughput per connection is the capacity of the bottleneck link divided by the number of connections that share the bottleneck.

The situation is different when a bottleneck link is shared between symmetric MIMD connections. The connections suffer from a high level of unfairness under various scenarios. We briefly summarize some central findings in (6):

• The case of synchronized losses: if all connections suffer a loss at the same time then the bandwidth share of each connection remains constant in time. This is particularly harmful for new connections that may not be able to grab throughput.

• The case of asynchronous losses: in case that there are some asynchronous random losses (with or without synchronous losses that are due to congestion), the bandwidth is shared more fairly: the average throughput tend to be the same after a very long time. Remarks 2.3.3.1. The situation that we described corresponds to symmetric connections. In case the round trip times are not the same, the throughput share between the connections is inversely proportional to the RTT. In MIMD, in contrast, the connection with larger RTT gets almost no bandwidth.

As for the throughput, MIMD is able indeed to use all the available link capacity (provided that there are buffers; the buffer size needed for full utilization are much smaller than those needed in AIMD. Below we do not consider this aspect.)

Inter Population Competition

The way bandwidth is shared between TCP connections of different types (AIMD and MIMD) that share a common bottleneck is complex and depends on various factors. We shall restrict our discussion to links with high bandwidth and large buffering. In those conditions, an MIMD connection gets more throughput than AIMD does. As for fairness, the problems of no fairness (in the sense that the the initial share remains the same) or fairness on a very slow time scale, do not occur between two connections of different types (AIMD and MIMD).

The game formulation Consider a competition between a large population of users where a fraction of them use an AIMD fersion of TCP and another MIMD connection sharing a common bottleneck link We can now use the above description to define. the structure of the evolutionary game. There are many pairwise interactions described through the matrix game of the form given in Figure 2.11 where H (Hawk) stands for the MIMD TCP and D (Dove) stands for AIMD. Let the fitness of a player that uses an action i that interacts with a player that plays j be given as F(i, j) = θ(i, j)f (i, j) where θ(i, j) is the throughput part corresponding to the share of the bottleneck capacity that the player receives and f is a disutility for lack of fairness. f (i, j) is zero except for the case i = j = H; θ(i, j) = 0.5 when i = j, where as θ(H, D) = 1θ(D, H) = a where a is some positive constant smaller than 0.5. Denoting d = 0.5f (H, H) we obtain the fitnesses as given in Figure 2.11.

11: MIMD (H) versus AIMD (H)

Characterizing the ESS We note that the conditions defining a Generalized Hawk and Dove game are satisfied. We thus conclude the following:

• AIMD is never a dominant strategy (i.e. there is no ESS where all the population uses AIMD). To understand this, we note that if all used AIMD then a shift of a small fraction of the population to MIMD would give this fraction a larger fitness, since MIMD would get a larger throughput when interacting with AIMD but it would almost never suffer from the fairness problem since encounters between MIMD connections would be quite rare.

• MIMD may, on the other hand, be dominant. This occurs if d > a. In other words, if users did not care much about the fairness problems of MIMD, then its share in the population could be expected to grow to 1 and AIMD could disappear.

• if d < a then the unique ESS consists of a coexistence between AIMD and MIMD. At ESS, the fraction using MIMD is given by

p = 1 -2a 1 -2d

Notes

We have mentioned in the introduction two simplistic approaches for predicting which population would dominate. The Intra Population Competition (IAPC) compares a world with only one type of behavior and compares the corresponding utility; in our case it would amount to observing that F(D, D) > F(H, H) and thus would wrongly suggest that AIMD would dominate the MIMD and their share would be one, where as we know that this possibility never occurs

The Inter Population Competition (IRPC) compares only the interactions that involve H and D and it would predict that MIMD would dominate since it gets better fitness in these interactions. This prediction is again wrong; indeed, we have seen that both population may coexist insight of having F(H, D) > F(D, H) provided that d < a.

Delayed replicator dynamics

We introduce here the replicator dynamics which describes the evolution in the population of the various strategies. In the replicator dynamics, the share of a strategy in the population grows at a rate equal to the difference between the payoff of that strategy and the average payoff of the population. More precisely, consider N strategies. Let x be the N dimensional vector whose ith element x i is the population share of strategy i. Thus we have ∑ i x i = 1 and x i ≥ 0. Below we denote by J(i, k) the expected payoff (or the fitness) for a player using strategy i when it encounters a player with strategy k. With some abuse of notation we define J(i, x) = ∑ j J(i, j)x j .

Then the replicator dynamics is defined as

ẋi (t) = x i K J(i, x) -∑ j x j J(j, x) = x i K ∑ j x j J(i, j) -∑ j ∑ k x j J(j, k)x k
where K is a positive constant and ẋi (t) := dx i (t)/dt. Note that the right hand side vanishes when summing over i. This is compatible with the fact that we study here the share of each strategy rather than the size of the population that uses each one of the strategies.

In Equation (2.35), the fitness of strategy i at time t has an instantaneous impact on the rate of growth of the population size that uses it. An alternative more realistic model for replicator dynamic would have some delay: the fitness acquired at time t will impact the rate of growth τ time later. We then have

ẋi (t) = x i (t)K ∑ j x j (t -τ)J(i, j) -∑ j,k x j (t)J(j, k)x k (t -τ) (2.35)
where K is some positive constant. The delay τ represents a time scale much slower than the physical (propagation and queueing) delays, it is related to the time scale of (i) switching from the use of one protocol to another (ii) upgrading protocols.

Architecting evolution

We study the choice of two parameters in the replicator dynamics that impact the stability of the evolution process of protocols: the gain parameter K and the delay τ appearing in Equation (2.35). The standard replicator dynamics (2.35) appearing in the evolutionary game literature is defined with K = 1. K's other than one can be interpreted as if the utilities J are multiplied by a constant. Alternatively, it can be seen as scaling time. The parameter K can thus be used to accelerate the rate of convergence in (2.35).

The impact of K and τ on the stability

We consider below the case of two players and two actions. Define

δ 1 = J(B, A) -J(A, A), δ 2 = J(A, B) -J(B, B), δ = δ 1 + δ 2 , θ = δπ 2δ 1 δ 2
Guidelines for an evolution framework.

For K = 1, it has been shown in [START_REF] Tao | Effect of time delay and evolutionarily stable strategy[END_REF] that if the delay τ in (2.35) satisfies τ < θ then the mixed ESS is asymptotically stable, and if τ increases beyond θ then the ESS becomes unstable.

We make the observation that Equation (2.35) with K is equivalent to Equation (2.35) with all elements J(i, m) multiply by K. Thus we can use the result of [START_REF] Tao | Effect of time delay and evolutionarily stable strategy[END_REF] to conclude that the stability condition for general K is given simply by τK < θ. (2.36) This provides us with an important guideline for designing evolutionary protocols. In order for such a protocol to be scalable to any delay, the product of the adaptation speed parameter K and delay τ should be O [START_REF] Ait-Hellal | Analysis of TCP Vegas and TCP Reno[END_REF]. Thus the larger the delay is, the slower we should react to the fitness of a strategy being used.

We note that this type of scaling is quite familiar in other networking contexts: the internet transport protocol TCP has a throughput that scales according to 1/RTT (where RTT is the round trip delay). This scaling is obtained by a self clocking mechanism based on ACKs that trigger new transmissions.

Numerical illustrations

Impact of gain parameter

Our first numerical experiment on evolution of protocols studies the behavior of the replicator dynamics for the case of one delay unit as a function of K: we check the speed of convergence and the stability of the replicator dynamics as a function of the gain parameter K. We consider the following fixed parameters: we took τ = 1, δ = 2/3, and let K vary between 0.16 and 15. A unique mixed ESS exists for these parameters, for which the fraction of the population using H is 3/4. The resulting trajectories of the population ratio using the first strategy, H, as a function of time, is given in Fig. 2.14, 2.12, 2.13 . For K = 0.16, we have stability but the convergence speed is slow. The other extreme is illustrated for K = 15 which is seen to be unstable: it oscillates rapidly and the amplitude is seen to grow slowly.

Impact of delay

We now keep K constant and evaluate the stability varying the delay between 0.016 and 15 time units. When τ = 0.016 the system is stable but the rate of convergence to the interior equilibrium is not fast. For τ = 15 the system is unstable, the solution oscillates around the equilibrium x * = 3/4. In Figure 2.14 we display an oscillatory behavior of the population ratios as function of time for two different initial values of x(0) = 0.03 and x(0) = 0.97 with K = 15 and τ = 1. It corresponds to an unstable regime in which the ESS is not attained. The trajectories are seen to converge to periodic ones. The limit trajectories look the same and do not depend on the initial state except for a dependence through the phase. In this unstable regime, more than one protocol co-exist and the ratio of population sizes using the protocols has oscillations with large amplitude.

In both figures 2.14 and 2.13, we observe that we have stability when τK < 4π ≃ 12.56. Indeed, in figure refallfigtau, the parameter τ = 1, hence the condition of stability (2.36) becomes K < 12.56. This actually confirm that using K = 0.16, 1, 1.56, 12, the system is stable and using K = 15, the system is unstable. We observe the same behavior when keeping K constant and varying the delay τ.

Notes

In this section, we have studied evolutionary aspects of congestion control protocols in using the biological paradigm of evolutionary games. We have studied the questions of whether one could expect one type of protocol to wipe away another one or whether we may expect protocols to coexist. In the latter case we provided a quantitative characterization of the share that each protocol could be expected to have in the whole population at equilibrium. We then identified conditions under which there is a convergence to the equilibrium and obtained examined the oscillating behavior that occurs when there is no convergence. The conditions that guarantee convergence can be used as guidelines for deployment of new protocols so that the users upgrades would result in a stable system wide behavior.

Multi-class delayed evolutionary game dynamics

In this section we briefly describe delayed evolutionary game dynamics with multi-classes of players. Consider several classes of large population of players and finitely many pure strategies in each class. Let E = {1, 2, . . . , E} be the set of classes. the action set of class e is A e = {1, 2, . . . , n e }. The equilibria of the evolutionary game with delayed payoffs are exactly the rest points of the dynamic (2.42).

Delayed target projection dynamics ẋe (t) = proj TX e (F e ({x(t

-τ k } k )) -x e (t)
In figure 2.15 we illustrate the evolution of cooperation with three types in which one can move from one to another stable equilibrium because of time delays. We conclude that the delayed evolutionary game dynamics can have unpredictable trajectories even when the nondelayed dynamics leads to stable points.

Notes

In next chapter we pursue the study of delayed game dynamics and extend this analysis for evolutionary games with variable number of players in each local interaction. The chapter is based to our publications in (213; 11; 211; 218; 210).

Chapter 3

Evolutionary games with random number of interacting players

The classical evolutionary game formalism is a central mathematical tool developed by biologists for predicting population dynamics in the context of interaction between populations. In order to make use of the wealth of tools and theory developed in the biology literature, many works in the area of computer networks (52; 20; 214) ignore cases where local interactions between populations involve more than two individuals. This restriction limits the modeling power of evolutionary games which are not useful in a network operating at heavy load, such as ad-hoc networks with high density. This motivated us in this chapter to consider more than two users interacting locally and the interactions can be non-reciprocal.

Our main contributions in this chapter can be summarized in three points:

• The first objective of this work is to extend the evolutionary game framework to allow an arbitrary (possibly random) number of players that are involved in a local interaction (possibly non-reciprocal interaction);

• The second objective of this work is to apply the extended model to access games studied in [START_REF] Tembine | Multiple Access Game in Ad-hoc Network[END_REF] which we extend to more than two interacting nodes. In the context of MAC games, we study the impact of the node distribution in the game area on the equilibrium stable strategies of the evolutionary game. The interaction between more than two individuals in a population is a new concept in evolutionary game theory and has a lot of application in multiple access game in wireless networks. Considering this kind of games, we use the notion of expected utility as this game is not symmetric, indeed the number of players with which a given one interacts may vary from one to another; and also non-reciprocity property. We consider the following parameters in the access game: transmission cost, collision cost and regret cost. We analyze the impact of these parameters on the probability of successful transmission and give some optimization issues. The notion of correlated evolutionarily stable state (see Section 3.4) and coordination mechanism are used to improve the performance(reducing collisions).

• The third objective of this work is to apply evolutionary game models to study the interaction of numerous mobiles in competition in a wireless environment. The power control game in wireless networks is a typical non-cooperative game where each mobile decides about his transmit power in order to optimize its performance. This application of noncooperative game theory and tools is studied in several articles (145; 171; 146). The main difference in this article is the use of the evolutionary game theory which deals with population dynamics that is well adapted for studying the power control game in dense wireless networks. Specifically, we focus our first study in a power control game in a dense wireless network where each user transmits using orthogonal codes like in W-CDMA.

In the second scenario, we consider also uplink transmissions but inter-cell interferences like in WiMAX cells deployment. The utility function of each mobile is based on carrier (signal)-to-interference ratio and pricing scheme proportional to transmitted power. We provide and predict the evolution of population between two types of behaviors : aggressive (high power) and peaceful (low power). We identify cases in which at ESS, only one population prevails (ESS in pure strategies) and others, in which an equilibrium between several population types is obtained. We also provide the conditions of the uniqueness of ESS. Furthermore, we study different pricing for controlling the evolution of population. We introduced the benefit of correlation (BoC) and show that the random access game with two strategies has a BoC of 100%. In addition, we show that hierarchical solutions and canonical correlation devices are do not improve the performance the number of strategies is at least three.

The rest of this chapter is organized as follows. In Section 3.1 and 3.2, we illustrate some examples of non-reciprocal interactions and limitations of the pairwise interaction approach. In Section 3.3 we describe a general model of population games with random number of interacting players at each local interaction and study both evolutionary stability and evolutionary game dynamics. We then study in Section 3.4 a generalized multiple access game with a random number of players. In Section 3.5, we present the evolutionary game model to study the power allocation game in a dense wireless ad hoc network. In Section 3.6, we propose an evolutionary game analysis for the power allocation game in a context of inter-cell interferences in OFDMA-based WiMax cells. Numerical results of both wireless network architectures are proposed in each section.

Reciprocal and non-reciprocal interactions

Consider a population of users. Each individual needs occasionally to take some action. When doing so, it interacts with the actions of some M (possibly random number of) other individuals. All players have the same actions available, and same expected utility. We note however that the actual realizations need not be symmetric. In particular, (i) the number of players with which a given player interacts may vary from one player to another. (ii) We do not even need the reciprocity property: if player A interacts with player B, we do not require the converse to hold. We provide some examples of multiple access games to illustrate this non-reciprocity.

We now model interference control as local interactions between transmitters; for each transmitter there corresponds a receiver. We shall say that a transmitter A is subject to an interaction (interference) from transmitter B if the transmission from B overlaps that from A, and provided that the receiver of the transmission from A is within interference range of transmitter B.

Non-reciprocal pairwise interaction

Consider the example depicted at Figure 3.4. It contains 4 sources (circles) and 3 destinations (squares). A transmission of a source i within a distance r of the receiver R, causes interference to a transmission from a source j = i to receiver R. We see that Source A and Source C cause no interference to any other transmission but the transmission from A suffers from interference from source B, and the one from C suffers from the transmission of the top most source (called D). Source B and D interfere with each other at their common destination. Thus each of the four sources suffers interference from a single other source, but except for nodes B and D, the interference is not reciprocal.

A D C B Figure 3.1: Non-reciprocal pairwise interactions

It is easy to see that in this non-reciprocal pairwise interaction, the following configurations leads to equilibria:

• A and D transmit at the same time slot. There are 2 successful transmissions if the others stay quiet.

• A and C transmit at the same time slot. There are 2 successful transmissions if the others stay quiet.

• B and C transmit at the same time slot. There are 2 successful transmissions if the others stay quiet

• A, D and C transmit at the same time slot. There is 1 successful transmission (only from A) if B stay quiet.

• A, B and C transmit at the same time slot. There is 1 successful transmission (only from C) if D stay quiet.

• C, D transmit at the same time slot. There is 1 successful transmission (only from A) if B stay quiet.

• B, D transmit at the same time slot. There is no successful transmission in the system at that time.

Non-reciprocal interactions between groups three players

In Figure 3.2 there are four sources and only two destinations. Node A does not cause any interference to the other nodes but suffers interference from nodes B and D. Nodes B, C, D interfere with each other. This is a situation in which each mobile is involved in interference from two other mobiles but again the interference is not reciprocal.

The following configurations leads to equilibria:

• A and C transmit at the same time slot. There are 2 successful transmissions if the others stay quiet. This is a global optimum.

• Only B transmits. There is one successful transmissions if the others stay quiet.

A B C D Figure 3.2: Non-reciprocal interaction between groups three players

• Only D transmits. There is one successful transmissions if the others stay quiet.

• B and D transmit at the same time slot. There is no successful transmissions.

Interactions between random number of players

In this subsection the number of interfering nodes is not fixed. The mobile A suffers interference from 2 nodes, B and D suffer interference from a single other node and C does not suffer (and does not cause) interference.

A D C B

Figure 3.3: Interactions between a random number of players

The following configurations leads to equilibria:

• A and C transmit at the same time slot. There are 2 successful transmissions if the others stay quiet. This is a global optimum.

• B and C transmit at the same time slot. There are 2 successful transmissions if the mobiles A and D stay quiet. This is a global optimum.

• B and C transmit at the same time slot. There are 2 successful transmissions if the others stay quiet. This is a global optimum.

• D and C transmit at the same time slot. There are 2 successful transmissions if the others mobiles A and B stay quiet. This is a global optimum.

Note that if D or B moves from its position. The mobile A can play the role of C is this analysis and a new game between B or D and C will be played.

All examples exhibit asymmetric realizations and non-reciprocity. We next show how such a situation can still be considered as symmetric (due to the fact that we consider distributions of nodes rather than realizations). Assume that the location of the transmitters follow a Poisson distribution with parameter λ over the two dimensional plane. Consider an arbitrary user A. Let r be the interference range. Then the number K + 1 of transmitters within the interference range of the receiver of A has a Poisson distribution with parameter λπr 2 /2. Since this holds for any node, the game is considered to be symmetric. The reason that the distribution is taken into account rather than the realization is that we shall assume that the actions of players will be taken before knowing the realization.

Spatial non-reciprocal random access games

We assume that all sources (players) have the same actions set (but their decisions can be different). The number of transmitters with which a given transmitter interacts may vary from one transmitter to another. We do not even need the reciprocity property: if transmitter TA interacts with transmitter TB, we do not require the converse to hold. We provide some example of multiple access and interference control to illustrate this non-reciprocity.

We consider local interactions between transmitters; for each transmitter there corresponds a receiver. We shall say that a transmitter TA is subject to an interaction (interference) from transmitter TB if the transmission from TB overlaps that from TA, and provided that the receiver of the transmission from TA is within interference range of transmitter TB. Consider the example depicted in Figure 3.4. It contains 3 receivers or destinations represented by squares, with range N j , j = 1, 2, 3. A transmission of a source i within a distance r of the receiver Rj, causes interference to a transmission from a source j = i to receiver Rj but also i causes interference to any transmitter located in the an intersection range of coverage (the destinations of i and j can be different). We see that sources in area A1 := N 1 \(A12 ∪ A13 ∪ A123) cause no interference to any transmitter from A12 ∪ A13 ∪ A123 transmitting to R2 or R3. The transmitters located in area A1 cause no interference to any transmitter located in areas A2 or A3. but the transmission from sources in A1 suffers from interference from source located in A12, A13, A123. The sources in A1 interfere with each other at their common destination R1. Thus, the interference is not reciprocal. In term of games, a transmitter in areas A12 ∪ A13 ∪ A23 ∪ A123 play simultaneous several one-shot games with a unidimensional action (transmit or not) and receives a vector of payoff. Only the payoff obtained at the receiver in which he transmits can be non-zero.

Analysis of equilibria and optima

We consider arbitrary number of sources in each area. Let n ij be the cardinal of Aij and n 123 be the cardinal of A123. Denote by r 1 := n 1 + n 12 + n 13 + n 123 the number of transmitters located in N 1 = A12 ∪ A13 ∪ A1 ∪ A123 (the total number of transmitters covered by receiver R1. Similarly, r 2 := n 2 + n 12 + n 23 + n 123 , r 3 := n 3 + n 13 + n 23 + n 123 . A strategy profile is a (Nash) equilibrium if no source from any area can unilaterally deviate and improve its probability of success. -From each area A1, A2, A3, only one source transmits: only one source from A1, one from A2, one from A3 transmit and others stay quiet. These configurations are equilibria. There are three successful transmissions. Any permutation of the n 1 sources in A1, n 2 sources in A2 and n 3 sources in A3 when keeping the others sources in areas A12, A13, A23, A123 stay quiet lead a pure strategy equilibria that are also Pareto optima. They are global optima (social welfare) in the sense that the total sum of payoffs is maximized (one can have at most three successful transmissions at the slot).

Equilibria and Pareto optima

C2 Equilibria with two successful transmissions at the same slot:

-Only one source located in A12 transmits, one source from A3 transmits and the others stay quiet.

-Only one source located in A13 transmits, one source from A2 transmits and the others stay quiet. There are two successful transmissions.

-Only one source located in A23 transmits, one source from A1 transmits and the others stay quiet.

C1 Equilibria with one successful transmission at the same slot:

-Only one source located in A123 transmits and all the others sources stay quiet. There 3.2. Why random number of players?

is one successful transmission in all the three receivers. Since this source is located at the intersection between N 1 , N 2 and N 3 , there is interference to any transmission at any receiver. By permutation and selection of one source of the n 123 sources located A123, we obtain the others equilibria at this configuration. These equilibria are not global optima.

C0 At least two sources from each neighborhood N j , j = 1, 2, 3 transmit and the others play any mixed strategies. Hence, there are infinite number of equilibria of this type. These equilibria are not Pareto optima because each source gets zero which is Pareto dominated by the configurations of type C1 -C3.

Notice that if there is no pairwise intersections between the neighborhoods N j , j = 1, 2, 3, (the number n 12 , n 123 , n 23 , n 13 are null) then the interactions becomes reciprocal (if source i interferes with source j then source j interferes with source i) and the games in N 1 , N 2 , N 3 can be analyzed separately.

Notes

This model can be easily extended to finitely many intersecting neighborhoods. See Chapters 4 and 6.

Why random number of players?

The classical evolutionary game formalism is a central mathematical tool developed by biologists for predicting population dynamics in the context of interaction between populations. In order to make use of the wealth of tools and theory developed in the biology literature, many works in the area of computer networks (88; 20; 52; 214) ignore cases where local interactions between populations involve more than two individuals. This restriction limits the modeling power of evolutionary games which are not useful in a network operating at heavy load, such as ad-hoc networks with high density. This motivated us to consider a random number of users interacting locally.

We model and study interactions in wireless networks using the theory of evolutionary games which we extend to cover a random number of players. We study access games within a large population of mobiles that interfere with each other through many local interactions. Each local interaction involves a random number of mobiles. The games are not necessarily reciprocal as the set of mobiles causing interference to a given mobile may differ from the set of those suffering from its interference. We then apply evolutionary games to non-cooperative uplink power control in wireless networks. Specifically, we focus our study in a power control in W-CDMA and WiMAX wireless systems. We study competitive power control within a large population of mobiles that interfere with each other through many local interactions. We show how the evolution dynamics, the equilibrium and the evolutionary stability behavior are influenced by the characteristics of the wireless channel and pricing characteristics.

Model and notations

We describe in this part notations of our model.

• There is a large population of users.

• We assume that there are finite number of pure strategies in each population. Each member of the population chooses from the same set of strategies A = {1, 2, . . . , N}.

• Let Mix = ∆(A) := {(x a ) a∈A | x a ≥ 0, ∑ a∈A
x a = 1} the set of probability distributions over the N strategies. It is also interpreted as the set of distributions of strategies among the population, where x a represents of proportion of users choosing the strategy a. A distribution x is sometime called the "state" or "profile" of the population. It can also be interpreted as a strategy. M can be interpreted as the set of mixed strategies of a user.

• The number of opponents M of a user is a random variable in the finite set {0, 1, . . .}. In the bounded case, we will denote by k max the maximum number of opponents interacting simultaneously with a user. This value depends on the node density and the transmission range. When making a choice of a strategy, a player knows the distribution of M but not its realization.

• The payoff of all players functions (identical for each member of the population) of the player's own behavior and opponents' behavior. Each user's payoff depends on opponents' behavior through the distribution of opponents' choices and of their number. The expected payoff of a user playing strategy j when the state of the population is x, is given by

f a (x) = ∑ k≥0 P(K = k)u k (a, x, . . . , x), a ∈ A
where u k is the payoff function given that the number of opponents is k. Although the payoffs are symmetric, the actual interference or interactions between two players that use the same strategy need not be the same, allowing for non-reciprocal behavior. The reason is that the latter is a property of the random realization whereas the actual payoff already averages over the randomness related to the interactions, the number of interfering players, the topology etc.

• The game is played many times. Theorem 3.3.1 (Existence of equilibrium). For any distribution of random number K of players that interact locally such that the expected payoff is continuous in the simplex (or product of simplexes for multipopulation case), the evolutionary game with random number of interacting players has at least one equilibrium state.

Proof. We show that the game has a symmetric Nash equilibrium. We first remark that the generating function of K is continuous in (0, 1). Thus, F is lower semi-continuous in ∆(A) (which is a non-empty, convex and compact subset of Euclidean space R n+1 ). The existence of symmetric Nash equilibrium in mixed strategies follows from Kakutani fixed point theorem or from the existence of solutions of the following variational inequalities

find x ∈ ∆ n s.t ∑ b∈P (x b -mut b ) f b (x) ≥ 0, ∀ mut

Slotted Aloha

Multiple Access Game introduces the problem of medium access. We assume that mobiles are randomly placed over a plane. All mobiles use the same fixed transmission range of r. The channel is ideal for transmission and all errors are due to collision. A mobile decides to transmit a packet or not to transmit to a receiver when they are within transmission range of each other. Interference occurs as in the Aloha protocol: if more than one neighbors of a receiver transmit a packet at the same time then there is a collision. The Multiple Access Game is a nonzero-sum game, the mobiles have to share a common resource, the wireless medium. In this game, the parameter µ represents the probability that a mobile has its receiver R(i) within its range. When a mobile i transmits to R(i), all mobiles within a circle of radius r centered at R(i) cause interference to the node i for its transmission to R(i). This means that more than one transmission within a distance r of the receiver in the same slot cause a collision and the loss of mobile's i packet at R(i).

Each of the mobiles has two possible strategies: either to transmit (T) or to stay quiet (S). If mobile i transmits a packet, it incurs a transmission cost of δ ≥ 0. The packet transmission is successful if the other users don't transmit (stays quiet) in that given time slot, otherwise there is a collision and the corresponding cost is ∆ ≥ 0. If there is no collision, user i gets a reward of V from the successful packet transmission. We suppose that the reward V is greater than the cost of transmission δ. When all users stay quiet, they have to pay a regret cost κ. If κ = 0 the game is called degenerate multiple access game. Figure 3.27 represents an example of interaction of three nodes. The node 1 chooses a row, node 2 chooses a column and node 3 an array. The ESS corresponding to any number of nodes1 of this game is given in theorem 3.4.1. Let A := {T, S} be the set of pure strategies. An equivalent interpretation of strategies is obtained by assuming that individuals choose pure strategies and then the probability distribution represents the fraction of individuals in the population that choose each strategy. We denote by s (resp. 1s) the population share of strategy T (resp. S). The payoff obtained by a node with k opponents when it plays

T S T (-B θ , -B θ , -B θ ) (-B θ , 0, -B θ ) S (0, -B θ , -B θ ) (0, 0, V -δ) T S T (-B θ , -B θ , 0) (V -δ, 0, 0) S (0, V -δ, 0) (-κ, -κ, -κ)
T is u k (T, s) = (-∆ -δ) (1 -η k ) + (V -δ)η k where η k := (1 -s) k
, and the node-mutant receives u k (S, s) = -κ(1s) k when it stays quiet. The expected payoff of an anonymous transmitter node-mutant is given

f T (s) = µ ∑ k≥0 P(K = k)u k (T, s) = µ -(∆ + δ) + (V + ∆) ∑ k≥0 P(K = k) (1 -s) k = -µ(∆ + δ) + µ(V + ∆)G K (1 -s).
where G K is the generating function of K. Analogously, we have

f S (s) := µ ∑ k≥0 P(K = k)u k (S, s) = -µκ ∑ k≥0 (1 -s) k P(K = k).
The expected payoff of any individual in the population where s is the proportion of mobiles which transmit, is given by: s f T (s) + (1s) f S (s).

We next introduce two alternative information scenario that have an impact on the decision making. In the first case, a mobile does not know whether there are zero or more other mobiles in a given local interaction game about to be played. In the second case the mobile has this information, and consequently he transmits with probability one in case no other potential interferers are present. In addition to studying these two cases we shall also consider a third case called the "massively dense" ad-hoc network in which, whenever a mobile participates wishes to transmit, there is at least one other mobile that is involved in the local interaction game.

We denote

α := ∆ + δ V + ∆ + κ
, which represents the ratio between the collision cost -∆δ (cost when there is a collision during a transmission) and the difference between global cost perceived by a mobile -∆δκ (collision and regret) and the benefit Vδ (reward minus transmission cost). When the collision cost ∆ becomes high, the value α converges to one and when the reward or regret cost becomes high, the value α is close to zero.

Case 1 : Aloha without sensing

A transmitter does not know if there are other transmitters at the range of its receiver. Then, even when it is the only transmitter, it has to decide to transmit or not.

Theorem 3.4.1.

If P(K = 0) < ∆ + δ V + ∆ + κ =: α, then the game has a unique ESS s * 1 given by s * 1 = φ -1 ( ∆ + δ V + ∆ + κ ) where φ : s → ∑ k≥0 P(K = k) (1 -s) k . Proof. A mixed equilibrium s is characterized by f T (s) = f S (s) i.e φ(s) = ∆ + δ V + ∆ + κ
. The function φ is continuous and strictly decreasing monotone on (0, 1) with φ(1) = P(K = 0) and φ(0) = 1. Then the equation φ(s) = ∆ + δ V + ∆ + κ has a unique solution in the interval

(P(K = 0), 1). One has, f (s, y) -f (mut, y) = µ(V + ∆ + κ)(s -mut) (φ(y) -φ(s)) . Since s -ǫmut -(1 -ǫ)s = ǫ(s -mut), for y = ǫmut + (1 -ǫ)s one has ∑ j∈{T,S} (x j -mut j ) f j (y) > 0
(because φ is strictly decreasing continuous function) for all mut = s. This completes the proof.

When a mobile is never alone in his interference area, i.e. P(K = 0) = 0; the condition α > 0 is satisfied.

Case 2: Aloha with sensing

A mobile knows when it is the only transmitter at the range of its receiver, and when it is it will thus transmit with probability one. We can say then that the action set is T whenever a user has opponents in a local interaction. Theorem 3.

An anonymous user without opponents receives the fitness f

0 = V -δ. If P(K = 0) < ∆ + δ V + ∆ , then the game has a unique ESS s * 2 given by s * 2 = φ -1 ∆ + δ + κP(K = 0) V + ∆ + κ where φ : s → ∑ k≥0 P(K = k) (1 -s) k .
Proof. The proof is similar as in theorem 3.4.1.

Case 3: Massively Dense

In this case, we take into account only local interactions between users. Then, in this case, mobiles are never alone to transmit during a slot and we have ∑ k≥1

P(K = k) = 1.
Theorem 3.4.3. The game has always an unique ESS which is solution of the following equation

∑ k≥1 P(K = k)(1 -s) k = α.
The proof is similar as in theorem 3.4.1 using the monotocity of g. Proposition 3.4.3.1. The ESS given in theorems 3.4.1,3.4.2,3.4.3 is asymptotically stable in the replicator dynamics without delays for all non-trivial initial state (s 0 / ∈ {0, 1}).

Proof. The replicator dynamics is given by

d dt s(t) = (V + ∆ + κ)s(t)(1 -s(t))(φ(s(t)) -α).
The function φ is decreasing on (0, 1) implies that the derivative of the function s(1s)(φ(s)α) at the ESS is negative. These means that the dynamic is competitive. Hence, the ESS is asymptotically stable.

ESS and nodes distribution

In this subsection, we consider different nodes distributions. We study the existence and the uniqueness of ESS in the different nodes distributions. First one, we assume that all mobiles have the same number of neighbors n -1, i.e., P(K = j) = δ n-1 (j) and seconde one, we assume that nodes are randomly distributed on a plan following a Poisson point process with density λ.

Arbitrary number of opponent nodes

In this part, we suppose that the population of nodes is composed with many local interaction between n ≥ 2 nodes. Let A := {T, S} the set of strategies and assume that the strategy T has a delay τ T and the strategy S has the delay τ S . The payoff of a player using the action a i ∈ A against the other players when they use the multi-strategy a -i = (a 1 , . . . , a i-1 , a i+1 , . . . , a n ) is given by U i (a).

Each user plays the n-player following game

Γ n = (N , A, (U i ) i∈N ) where
• N is the set of users (nodes) and the cardinal of N is n,

• A the set of pure actions (the same for every user),

• for every user i in N , the payoff function

U i : A n → R is given by U i (a) =        V -δ if a i = T and a j = S, ∀ j = i 0 if a i = S and {j ∈ N | a j = T} ≥ 1 -∆ -δ if a i = T and {j ∈ N | a j = T} ≥ 2 -κ if a j = S ∀ j ∈ N
Let s be the proportion of nodes in the population using the strategy T. Then x = (s, 1s) is the state of the population. Let ∆(A)

:= {sT + (1 -s)S | 0 ≤ s ≤ 1} the set of mixed strategies. The average payoff is f (x, x) = µs (-∆ -δ) 1 -(1 -s) n-1 + (V -δ) (1 -s) n-1 -µκ(1 -s) n .
It is not difficult to see that the one-shot game Γ n has 2 n -1 Nash equilibrium, n of them are optimal in Pareto sense 2 :

• If only one node transmit and the others stay quiet, then the node which transmit gets the payoff Vδ and the others receive nothing and has no cost. This configuration is an equilibrium .

• There are exactly n pure equilibria and all these pure equilibria are Pareto optimal.

• k (1 ≤ k < n -1) of the n nodes choose to stay quiet and the nk others are active and play the optimal mixed strategy in the game

Γ n-k : 1 -α 1 n-k-1 , α 1 n-k-1
where α :=

∆ + δ V + ∆ + κ
. Thus, there are exactly

n-2 ∑ k=1 ( n k ) = 2 n -(n + 2) partially mixed Nash equilibria.
• The game has a unique strictly mixed Nash equilibrium given by 1α

1 n-1 , α 1 n-1
• the allocation of payoff obtained in these (partially or completely) mixed strategy are not Pareto optimal.

Note that the first interference scenario described in the previous section holds here because the number of interferes is fixed and is equal to n -1. Then from Theorem 3.4.1 with the function φ(s) = (1s) n-1 , the ESS exists and is uniquely defined by

s * = 1 -α 1 n-1 .
This result generalizes the ESS in the two-player case that we have shown in [START_REF] Tembine | Multiple Access Game in Ad-hoc Network[END_REF] that when there are pairing local interaction in a single population, the completely mixed (Nash) equilibrium of the two nodes stage game Γ 2 is an ESS.

Optimization of the total throughput

We assume here that all mobile has the same number of opponents, that is n -1. We look for the probability of success that can be achieved in a local interaction depending on the Dirac distribution and also cost parameters. At the equilibrium point, the probability of success of a node is given by s * (1s * ) n-1 and the total probability to have a successful transmission in a local interaction (total throughput) is given by

P succ (α, n) = nµs * (1 -s * ) n-1 = nµ(1 -α 1 n-1 )α, (3.1)
where µ is the probability that a mobile has a receiver in its range. The probability to have

a successful transmission P succ (α, n) = nµ(1 -α 1 n-1
)α goes to -µα log(α) when the number of nodes goes to infinity. Hence, when n is large, the maximum total throughput of a local interaction is obtained when α = 1/e and is closed to µ/e. The optimal total throughput is 2 An allocation of payoffs is Pareto optimal or Pareto efficiency if there is no other allocation that makes every node at least as well off and at least one node strictly better off. obtained when

α * = (1 - 1 n ) n-1 and the corresponding throughput is P succ (α * , n) = µ(1 - 1 n ) n-1 -→ µ e .
In Figure 3.10 we observe the total throughput depending on the number of interferes n. The parameters considered are µ = 0.8 and α = 1/3. We denote that the total throughput is increasing in that case with the number of interferes which it seems non intuitive. The reason is that the number of transmitted mobiles at the ESS, i.e. s * , is exponentially decreasing with n. Another important result is that it may have a finite number of interferers that maximize the total throughput like in figures 3.7(α = 0.2). When the ratio α is very small α = 0.05 the probability of success is decreasing in n as shown in Fig. 3.6. In Fig. 3.11 we represent the probability of success P succ (α, n) as a function α for several values of n. In figure 3.11, we observe the total throughput for different value of n and applying the result of the previous proposition, we obtain the optimal total throughput depending on α.

Impact of the time delays

Now, we study the effect of the time delays on the convergence of replicator dynamics to the evolutionarily stable states in which each pure strategy is associated with its own delay. Let τ T (resp. τ S ) be the time delay of the strategy (T) (resp. (S)). The delayed replicator equation becomes

ṡ(t) = µ s(t)(1 -s(t)) [ f T (s(t -τ T )) -f S (s(t -τ S ))] (3.2)
where f T (s(t)

) := -µ(∆ + δ) 1 -(1 -s(t)) n-1 + µ(V -δ) (1 -s(t)) n-1 and f S (s(t)) := -µκ (1 -s(t)) n-1 .
In order to study the asymptotically stability of the replicator dynamics (3.2) around the

unique ESS s

* 1 = 1 -( ∆ + δ V + ∆ + κ ) 1 n-1 , we linearize (3.2) at s * 1 .
We obtain the following linear delay differential equation

ż(t) = -µ(n -1)s(1 -s) n-1 ((V + ∆)z(t -τ T ) + κz(t -τ S )) (3.3)
where

z(t) = s(t) -s * 1 .
The following theorem gives a sufficient conditions of stability of (3.3) at zero. Theorem 3.4.5. Suppose at least one of the following conditions holds

• (V + ∆)τ T + κτ S < 1 (n -1)s(1 -s) n-1 µ • V + ∆ > κ and (V + ∆)τ T < V + ∆ -κ (n -1)s(1 -s) n-1 µ(V + ∆ + κ) • V + ∆ < κ and κτ S < -V -∆ + κ (n -1)s(1 -s) n-1 µ(V + ∆ + κ)

Then the ESS s is asymptotically stable.

A proof of the theorem 3.4.5 can be obtained using theorem 3 in (214) applying to equation (3.3).

A necessary and sufficient condition of stability of (3.3) at zero when delays are symmetric is given in theorem 3.4.6. Theorem 3.4.6 (symmetric delay). Suppose that τ T = τ S = τ, then, the ESS s * 1 is asymptotically stable if and only if

τ < π 2(n -1)µs * 1 (1 -s * 1 ) n-1 (V + ∆ + κ)
The proof uses the following well known lemma (see [START_REF] Tembine | Delayed Evolutionary Game Dynamics applied to the Medium Access Control[END_REF] and the references therein). Lemma 3.4.6.1. The trivial solution of the linear delay differential equation

ż(t) = -az(t -τ), τ, a > 0 is asymptotically stable if and only if 2aτ < π.

Numerical solutions

The numerical solutions given figure 3.12 are obtained when the ran-

dom variable K is a Dirac δ {n-1} . We took n = 4 = k max , ∆ = 1 4 = δ, V = 1.
The initial condition is 0.02 and the delays τ T and τ S between 0.02 and 7. For the small delays: τ T = 0.02, τ S = 0.02 and τ T = 3, τ S = 2 respectively, the system is stable. For the delay τ H = 7 and τ S = 5, the system is unstable and the proportion of transmitters in the cell oscillates around the ESS.

Poisson distribution

We consider that nodes are distributed over a plan following a Poisson distribution with density λ. The probability that a node has k neighbors is given by the following distribution.

Cases 1 and 2:

P(K = k) = (λπr 2 ) k k! e -λπr 2 , k ≥ 0. Case 3:P(K = k) = (λπr 2 ) k-1 (k -1)! e -λπr 2 , k ≥ 1.
Considering those node distributions and from previous theorems, the unique ESS s * for all cases, is solution of the following equation :

       e -λπr 2 s 1 = α for case 1 e -λπr 2 s 2 = α + κP(K = 0) V + ∆ + κ for case 2 (1 -s 3 )e -λπr 2 s 3 = α for case 3
Thus we obtain the following equilibria in the different scenario:

s * 1 = log α -1 λπr 2 , s * 2 = log (α + κP(K = 0) V + ∆ + κ ) -1 λπr 2 and s * 3 = 1 - LambertW(λπr 2 αe λπr 2 ) λπr 2 ,
where LambertW(s) is the LambertW function which is the inverse function of f (w) = we w .

Optimization of the total throughput in Poisson distribution

We look for the probability of success that can be achieved in a local interaction depending on distribution parameters and also cost parameters. We consider the Poisson distribution with parameters λ and r.

The probability to have a successful transmission in a local interaction (total throughput) is given by different equation depending on the scenario. In the case 1 we have:

P succ (α, λ) = µs * 1 ∑ k≥0 kP(K = k)(1 -s * 1 ) k = µs * 1 ∑ k≥0 k (λπr 2 ) k k! (1 -s * 1 ) k ≈ µs * 1 (1 -s * 1 )λπr 2 α,
In the case 2, we have:

P succ (ᾱ, λ) ≈ µs * 2 (1 -s *
2 )λπr 2 ᾱ. We derive immediately the following result: Proposition 3.4.6.2. The maximum total throughput under poisson distribution is attained when α = e h(λ,r) in the case 1 (resp. ᾱ = e h(λ,r) in the case 2) where h is one of the two functions defined by

(λ, r) ∈ R 2 + → -(1 + 2λπr 2 ) ± 1 + 4(λπr 2 ) 2 2 .
In the case 3, we have:

P succ (α, λ) = µs * 3 ∑ k≥1 kP(K = k)(1 -s * 3 ) k = µs * 3 ∑ k≥1 k (λπr 2 ) k-1 (k -1)! (1 - s * 3 ) k ≈ µαs * 3 (1 + λπr 2 (1 -s * 3
)). Proposition 3.4.6.3. There exists a unique α * 3 in which the total throughput is maximum when α = α * 3 . The α * 3 is given by α * 3 = (1s)e -λπr 2 s where s is the unique solution in [0, 1] of the following equation :

1 + γ -s(2 + 5γ + γ 2 ) + s 2 (4γ + 2γ 2 ) -γ 2 s 3 = 0
Proof. The derivative of the function H := ∂P succ ∂s s given by

H(s) = (1 + γ -s(2 + 5γ + γ 2 ) + s 2 (4γ + 2γ 2 ) -s 3 γ 2 )e -γs .
We prove that the above function is strictly decreasing on [0, 1]. For that, it is sufficient to study the following function

G(s) = 1 + γ -s(2 + 5γ + γ 2 ) + s 2 (4γ + 2γ 2 ) -s 3 γ 2 .
We have ∂G(s) ∂s is given by

∂G(s) ∂s = -(2 + 5γ + γ 2 ) + 2s(4γ + 2γ 2 ) -3s 2 γ 2 .
It is easy to show that the above function is always negative. Since H(0) = 1 + γ > 0 and

H(1) = -e -γ < 0 then the function H is positive for s ∈ [0, s) and is negative for s ∈ (s, 1]
where s is the solution of the equation G(s) = 0. Since s * is decreasing function on α, we conclude that function

P succ is positive if s ∈ [0, s) and is negative s ∈ (s, 1]. Since the optimal of function P succ is attained at α = (1 -s)e -λπr 2 s
The probability of success at the ESS in poisson distribution is represented in figures 3.13 and 3.14. We observe in particular case that when the number of interferes increases, i.e. the rate λ in the case of the Poisson distribution, the total throughput increases.

In the figures 3.15 and 3.16, we describe numerical application of our evolutionary game model with Poisson distribution of nodes under the replicator dynamics. We took

n = 4 = k max , ∆ = 1 4 = δ = κ, λ = 1 and V = 1.
The initial condition in all these figures is 0.02. In the figure 3.15 we compare the evolution of the fraction of transmitters varying the parameter of density λ between 0.1 and 5 for the case 1, 2 and 3 respectively. We observe that we have stability for all cases. In figure 3.16 represents the impact of the parameter µ on the velocity of the system. We took µ between 0.1 and 0.5 without delay. We observe that we have stability but the convergence speed becomes slow when µ decreases. Now, we study the effect of the time delays on the convergence of replicator dynamics to the evolutionarily stable states in which each pure strategy is associated with its own delay in Poisson distribution. The replicator dynamics becomes

ṡ(t) = µs(t)(1 -s(t)) [ f T (s(t -τ T )) -f S (s(t -τ S ))] (3.4)
where f T (s(t) t) and f S (s(t)) := -µκe -λπr 2 s(t) in the case 1.In order to study the asymptotically stability of the replicator dynamics (3.4) around the unique ESS, we linearize (3.4) at s * = s * 1 . We obtain the following linear delay differential equation where

) := µ -(∆ + δ) + (V + ∆)e -λπr 2 s(
ẏ(t) = -c 1 ((V + ∆)y(t -τ T ) + κy(t -τ S ))
c 1 := µs * (1 -s * )α 1 + s * (1 -s * )λπr 2
, and y(t) = s(t)s * . The following theorem give sufficient conditions of stability of (3.5) at zero. Theorem 3.4.7 (see [START_REF] Tembine | Asymmetric Delay in Evolutionary Games[END_REF]). Suppose at least one of the following conditions holds

(i) (V + ∆)τ T + κτ S < θ a , (ii) V + ∆ > κ and (V + ∆)τ T < (V + ∆ -κ)θ a V + ∆ + κ , (iii) V + ∆ < κ and κτ S < (-V -∆ + κ)θ a V + ∆ + κ where θ a := 1 s * (1 -s * )µα (1 + s * (1 -s * )λπr 2 )
Then the ESS s * is asymptotically stable.

A necessary and sufficient condition of stability of (3.5) at zero when delays are symmetric is given in theorem 3.4.8. Theorem 3.4.8 (symmetric delay). Suppose that τ T = τ S = τ. Then, the ESS s * is asymptotically stable if and only if

τ < π 2s * (1 -s * )µα (1 + s * (1 -s * )λπr 2 ) (V + ∆ + κ)
The fraction of transmitters in the population is represented in figure 3.18 for λ = 0.5 and r = 1. The delays τ T and τ S are between 0.02 and 7. The system is stable for τ T = τ S = 0.02 or τ T = 3, τ S = 2. For τ T = 7 and τ S = 5 the system is unstable. We display an oscillatory behavior of the population as function of time. The trajectory are seen to converge to periodic ones. All turn out to confirm the stability condition that we obtained in theorem 3.4.7. In the figure 3.17 we compare evolution of the fraction of transmitters varying the parameter of density λ between 0.1 and 5 for the case 1, 2 and 3 respectively. In this figure, the time delays are respectively 3 and 2. Note that in this figure the equilibrium point is decreasing function in the density parameter λ. Indeed, when the density of nodes increases, the number of mobiles share a receiver increases. To avoid collision, the nodes decrease the probability of transmission. We observe also that for λ = 5, we have stability but the convergence speed is slow than for λ = 0.1.

Coordination mechanisms to reduce collisions

In this subsection we focus on the concept of correlated evolutionarily stable states (CESS) in access games. A correlated equilibrium is introduced by Aumann (32) in 1974 and can be interpreted as a distribution of actions given to the mobiles by some referee(which can be the base station, the receiver) before to play each local game. For more details on correlated equilibrium, we refer the reader to (32; 33; 125). As in [START_REF] Koch | Evolution and Correlated Equilibrium[END_REF], we use the concept of correlated equilibrium from the perspective of bounded rationality.

We define a probability space Ω, 2 Ω , P n which generates signals on which the nodes can condition their strategic choices where Ω = A n . The set Ω is partitioned as follows I j (a) := {as 1 s 2 . . . s j-1 s j+1 . . . s n | s l ∈ A, l = j}, I j := {I j (a), a ∈ A}. then, I j has exactly |A| = 2 elements which are information sets of node j ∈ N . We define an assignment function(called also rule) profile α = (α 1 , . . . , α n ) as a mapping from the set of states Ω to mixed strategies set ∆(A). For all j, the assignment function of the node j, α j must satisfy : if for some w, α j (w) = s, then α j (w ′ ) = s, for all w ′ ∈ I j (s).

That is, for each element w ∈ I j , node j cannot distinguish states that are in the same information set. We denote the set of all pure assignment functions by AF : {g | g : A -→ A}. Thus, when a node chooses an assignment function α and when he receives the signal ω from the referee, he will choose the mixed action α(ω). We use α(s j |ω) to denote the probability assigned on s j under this mixed action α(ω).

Then α(w) = [α(s 1 |ω), . . . , α(s n |w)] ∈ ∆(S).
Given a referee (Ω, P n ), we define the identity assignment function as α id,n (s j |ω) = 1 if proj j (ω) = s j where proj j (ω) denotes the j-th element of the signal ω and α id,n (s j |ω) = 0 for all ω such that proj j (ω) = s j .

If each node use the identity assignment then, the resulting probability distribution of actions profile actually played will be P n , the same as the probability distribution of actions recommended by the referee. But when nodes use other assignments, a different distribution may result.

Given an assignment profile α and a probability distribution P n over A n , the expected payoff is given by

f n (α) = ∑ ω∈Ω P n (ω) ∑ a=(a 1 ,...,a n )∈A n u n (a) ∏ j∈N α(a j |w) = ∑ a=(a 1 ,...,a n )∈A n u n (a) ∑ ω∈Ω P n (ω) ∏ j∈N α(a j |w) = ∑ a=(a 1 ,...,a n )∈A n u n (a)Q α (a) where Q α (a) = ∑ ω∈Ω P n (ω) ∏ j∈N α(a j |w).
We say that two assignment functions profile α and β are equivalent if they induce the same value Q α = Q β Proposition 3.4.9.1. Suppose that P n (a) = P n (σ(a)) for all permutation σ (the distribution P n is said symmetric). If an assignment profile β is equivalent to α id,n then Q β (a) = P n (a), for all a ∈ A n .

Proof. This is because Q α id,n (a) = P n (a), ∀a ∈ A n . Now, suppose a small group of mutants appears. These mutants use a mutational assignment function α ′n . which is not equivalent to the identical assignment function α id,n ., but they cannot change the referee recommendation. Let ǫ be the portion of the population which are mutants (who use α ′ ,) and 1ǫ portion of the population are non-mutants who use α id,n . At each time, n nodes are randomly chosen to play the strategic game Γ n . In playing the game, the nodes have the same referee (Ω, P n ).

A probability distribution P n over A n is an CESS if non-mutants with identity assignment function perform better than mutants with assignment functions that are not equivalent to the identical assignment function. Definition 3.4.9.2. A CESS P n is a symmetric distribution probability over A n such that for every assignment function α ′n nonequivalent to the identical assignment function α id,n , there exists some

ǫ α ′n > 0 such that F n (α id,n , ǫα ′n + (1 -ǫ)α id,n ) > F n (α ′n , ǫα ′n + (1 -ǫ)α id,n ) (3.5) for all ǫ ∈ (0, ǫ α ′n ) where F n (β, α) = f n (β, α, . . . , α) Proposition 3.4.9.3. If s is an ESS then the product measure s ⋆(n) given by s ⋆(n) (a) = ∏ j∈N s j (a j ) is a CESS. Proof. It is easy to see that s ⋆(n) is a probability measure on A n . Since s is an ESS, (s, . . . , s) ∈ ∆(A) n is a Nash equilibrium of the n-player game i.e ∑ a∈A n u n (a) ∏ j∈N s j (a j ) ≥ ∑ a∈A n u n (b k , a -k ) ∏ j∈N \{k} s j (a j ) ≥ 0, ∀b k ∈ A.
This means that the product probability measure s ⋆(n) is an CESS.

Our principal motivation to study CESS is that, in general the expected payoff obtained in ESS can be improved by CESS distribution (see the example depicted in Fig. 3. [START_REF] Altman | Constrained Markov Decision Processes[END_REF]). Non-ESS can be CESS as shown in the following example. Examples 3.4.9.4. Consider the following matrix game , Ω = {S, T} 2

Γ 1 ((V = 4, ∆ = δ = κ = 1)) : S T S -1, -1 0, 3 T 3, 0 -2, -
I 1 = {{SS, ST}, {TS, TT}}, I 2 = {{SS, TS}, {ST, TT}}.
A rule of the node 1 (row) is to assigns the first element and the second element of I 1 respectively. A rule of the node 2 (column) is to assigns the first element and the second element of I 2 respectively.

For the node 1,

• SS means "always stay quiet: choose always S"(independently of the signal).

• TT means "always transmit: choose always T"(independently of the signal).

• ST means "obedient to recommendation: choose S if ω ∈ {SS, ST} and choose T if ω ∈ {TS, TT}".

• TS means "opposite of the recommendation: choose S if ω ∈ {TS, TT} and choose T if ω ∈ {SS, ST}".

For the node 2,

• SS means "always stay quiet: choose always S"(independently of the signal).

• TT means "always transmit: choose always T"(independently of the signal).

• ST means "obedient to recommendation: choose S if ω ∈ {SS, TS} and choose T if ω ∈ {ST, TT}".

P n (a) = 1 n if a ∈ Θ n 0 otherwise is a CESS with the payoff V -δ n > 0.
Moreover at each slot, we have a successful transmission with probability one (if each node have some packet to transmit at each slot) and the allocation of payoffs obtained at this CESS is Pareto optimal.

Recall the payoff obtained at ESS is not Pareto optimal.

Proof. Θ n has exactly n actions. Thus, P n is a probability measure. Let

B := {b ∈ A n-1 | numb T (b) = 1}. [u n (TS . . . S) -u n (SS . . . S)]P n (TS . . . S) = V -δ n > 0, and ∑ a -1 ∈A n-1 [u n (Sa -1 ) -u n (Ta -1 )]P n (Sa -1 ) = -∑ a -1 ∈A n-1 u n (Ta -1 )P n (Sa -1 ) = (n -1) (∆ + δ) n > 0.
Thus, P n is a CESS and the system has a successful transmission at each slot with probability one. The total payoff obtained at the CESS is exactly Vδ. Hence the allocation at this CESS is Pareto optimal.

The Figure 3.19 represents the payoff obtained at the CESS given in the proposition 3.7.4.6 and the payoff of the ESS varying the regret cost κ. We took ∆ = 2/10, V = 1, δ = 0.

W-CDMA Wireless Networks

In this section, the random number of opponents is induced by the geographical position of the mobiles compared to the base stations. We study in this section competitive decentralized power control in an wireless network where the mobiles uses, as uplink MAC protocol, the W-CDMA technique to transmit to a receiver. We assume that there is a large population of mobiles which are randomly placed over a plane following a Poisson process with density λ. We consider a random number of mobiles interacting locally. When a mobile i transmits to its receiver R(i), all mobiles within a circle of radius R centered at the receiver R(i) cause interference to the transmission from node i to receiver R(i) as illustrated in figure 3.20. We assume that a mobile is within a circle of a receiver with probability µ. We define a random variable R which will be used to represent the distance between a mobile and a receiver. Let ς(r) be the probability density function (pdf) for R. Then we have µ = R 0 ς(r)dr. Remarks 3.5.0.7. If we assume that the receivers or access points are randomly distributed following a poisson process with density ν, the probability density function is expressed by ς(r) = νe -νr .

For uplink transmissions, a mobile has to choose between High(H) power level and Low(L) power level. We denote by P H the high power and P L the low power levels. Let s be the population share strategy H. Hence, the signal P r received at the receiver is given by P r = gP i l(r), where g is the gain antenna and α > 2 is the path loss exponent. For the attenuation, the most common function is l(t) = 1 t α , with α ranging from 3 to 6. Note that such l(t) explodes at t = 0, and thus in particular is not correct for a small distance r and largen intensity λ. Then, it makes sense to assume attenuation to be a bounded function in the vicinity of the antenna. Hence the last function becomes l(t) = max(t, r 0 ) -α . First we note that the number of transmission within a circle of radius r 0 centered at the receiver is λπr 2 0 . Then the interference caused by all mobiles in that circle is

I 0 (s) = λπg(sP H + (1 -s)P L ) r α-2 0 .
Now we consider a thin ring A j with the inner radius r j = jdr and the outer radius r j = r 0 + jdr. The signal power received at the receiver from any node in A j is P r i = gP i r α i . Hence the interference caused by all mobiles in A j is given by

I j (s) =          2gλπr j dr( sP H + (1 -s)P L r α j ) if r j < R, 2µgλπr j dr( sP H + (1 -s)P L r α j ) if r j ≥ R.
Hence, the total interference contributed by all nodes at the receiver is

I(s) = I 0 (s) + 2gλπ(sP H + (1 -s)P L ) R r 0 1 r α-1 dr + µ ∞ R 1 r α-1 dr , = gλπ(sP H + (1 -s)P L )( α α -2 r -(α-2) 0 -2(1 -µ)R -(α-2)
).

Hence the signal to interference ratio SI NR i is given by

SI NR i (P i , s, r) =        gP i /r α 0 σ + βI(s) if r ≤ r 0 , gP i /r α σ + βI(s) if r ≥ r 0 ,
where σ is the power of the thermal background noise and β is the inverse of the processing gain of the system. This parameter weights the effect of interference, depending on the orthogonality between codes used during simultaneous transmissions. In the sequel, we compute the mobile's utility (fitness) depending on his decision but also on the decision of his interferers. We assume the user's utility (fitness) choosing power level P i is expressed by

f P i (s) = w R 0 log(1 + SI NR(P i , s, r))ς(r)dr -ηP i .
The pricing function P i define the instantaneous "price" a mobile pays for using a specific amount of power that causes interference in the system and η is a parameter. This price can be the power cost consumption for sending packets.

We are now looking at the existence and uniqueness of the ESS. For this, we need the following result. where I(s) = (s(P H -P L ) +

P L )c(r) and c(r) = λπg α α -2 r -(α-2) 0 -2(1 -µ)R -(α-2) if r ≥ r 0 and λπg r α-2 0 otherwise. Since W satisfies W(s) > 1 and W ′ (s) = -c(r)β(P H -P L ) g(P H -P L ) r α (σ + βI(s) + gP L r α ) 2 < 0. Hence, ∂ ∂s log 1 + SI NR(P H , s, r) 1 + SI NR(P L , s, r) = ∂ ∂s (log W(s)) = W ′ (s) W(s) < 0 i.e h ′ (s) < 0.
We conclude that h is strictly decreasing.

Using this lemma, we have the following proposition which gives pure strategies depending on the parameters. Proposition 3.5.0.9. For all density function ς, the pure strategy P H dominates From the last proposition, we can use the pricing η as a design tool for create an incentive for the user to adjust their power control. We observe that the ESS s * decreases when η increases. That means the mobiles become less aggressive when pricing function increases and the system can limit aggressive requests for SINR.

Numerical examples in W-CDMA Wireless Networks:

In the numerical examples below, we show how the pricing function can optimize the overall network throughput. we first investigate the impact of the different parameters and pricing on the ESS and the convergence of the replicator dynamic. We also discuss the impact of the pricing function on the system capacity.

We first show the impact of density of nodes and pricing on the ESS and the average rate. We assume that the receivers which are randomly placed over a plane following a Poisson process with density ν, i.e, ς(r) = νe -νr . We recall that the rate of a mobile using power level P i at the equilibrium is given by w R 0 log(1 + SI NR(P i , s, r))ς(r)dr. We took r 0 = 0.2, w = 20, σ = 0.2, α = 3, β = 0.2, R = 1. First, we show the impact of the density of nodes λ on the ESS and the average rate. In figures 3.21, we depict the average rate obtained at the equilibrium and the ESS, respectively , as a function of the density λ. We recall that the interference for a mobile increases when λ increases. We observe that the mobiles become less aggressive when the density increases. We observe that it is important to adapt the pricing as function of the density of nodes. Indeed, we observe that for low density of nodes, the lower pricing (η = 0.92) gives better results than higher pricing (η = 0.97). When the density of nodes increases, the better performance is obtained with higher pricing.

Our second experiment in W-CDMA studies convergence to the ESS of the W-CDMA system described above under replicator dynamics. Fig. 3.22 and 3.23 represent the fraction of population using the high power level for different initial states of the population: 0.99, 0.66, 0.25 and 0.03. We observe that the choice of receiver distributions change the ESS. In the next section we study another wireless architecture where interferences are between mobiles which are located in different cell. This typical interference problem occurs in WiMAX environment.

OFDMA-based IEEE802.16 Network

OFDMA (Orthogonal Frequency Division Multiple Access) is recognized as one of the most promising multiple access technique in wireless communication system. This technique is used to improve spectral efficiency and becomes an attractive multiple access technique for 4 th generation mobile communication system as WiMAX.

In OFDMA systems, each user occupies a subset of subcarriers, and each carrier is assigned exclusively to only one user at any time. This technique has the advantage of eliminating intracell interference (interference between subcarriers is negligible). Hence the transmission is affected by intercell interference since users in adjacent sectors may have also been assigned to the same carrier. If those users in the adjacent sectors transmitted with high power the intercell interference may severely limit the SINR achieved by the user. Some form of coordination between the different cells occupying the spectral resource are studied in (134; 123). The optimal resource allocation requires complete information about the network in order to decide which users in which cells should transmit simultaneously with a given power. All of these results however, rely on some form of centralized control to obtain gains at various layers of the communication stack. In a realistic network as WiMAX, centralized multicell coordination is hard to realize in practice, especially in fast-fading environments.

We consider an OFDMA system where radio resources are allocated to users on their channel measures and traffic requirements. Each carrier within a frame must be assigned to at most one user in the corresponding cell. In this way each carrier assignment can be made independently in each cell. Hence when a user is assigned to carrier, the mobile should determine the power transmission to the Base station. This power should take into account the interference experienced by the transmitted packet.

Consider the uplink of a multiple multicell system, employing the same spectral resource in each cell. Power control is used in an effort to preserve power and to limit interference and fading effects. For users located in a given cell, co-channel interference may therefore come from only few cells as illustrated in figure 3.24. Since the intra-cell interference is negligible, we focus on the users which use a specific carrier. Consider N cells, and a large number of population of mobiles randomly distributed over each channel and each cell. Since in OFDMA systems, each carrier is assigned exclusively to only one mobile at any time, we assume that the interactions between mobiles are manifested through many local interactions between N ′ mobiles where N ′ is the set of neighbors of a cell. We can ignore the interaction among more than N ′ mobiles that transmit simultaneously and that can cause interference to each other. Hence, in each slot, interaction occurs only between the mobiles which have been assigned to the same carrier.

Let g ij denote the average channel gain from user i to cell j. Hence, if a user in cell i transmits with power p i , the received signal strength at cell i is p i g ii , while the interference it occurs on cell j is p i g ij . Hence, the interference experienced by cell i is given by

SI NR i (p) = g ii p i σ i + ∑ j =i g ij p j
, where σ i is the power of the thermal noise experienced at cell i. The rate achieved by user i is given by r i (p) = log(1 + SI NR i (p)), where p denotes the power level vector of mobiles choice which are assigned to a specific carrier. We assume that the user's utility is given by u i (p) = r i (p)η p i . The above utility represents the weighted difference between the throughput that can be achieved as expressed by Shannon's capacity and the power consumption cost. We assume that all mobiles perform the on-off power allocation strategy. In this strategy, each mobile transmits with full power or remains silent. Let N i be the set of neighbors of a user in cell i. Hence the interference experienced by a user in the cell i is given by

SI NR i (p) = g ii p i σ i + ∑ j∈N i \{i} g ji p j
, where p j ∈ {0, P}, P is the power level of transmission.

Let s i the proportion of transmitters in the cell i. The couple (s i , 1s i ) with s i ∈ (0, 1), represents the state of the cell i. We denote by s -i the vector (s 1 , . . . , s i-1 , s i+1 , . . . , s N i ). The fitness of the cell i can be defined as follows:

f i (s i , s -i ) = s i f i P (s -i ) = s i ∑ a -i ∈{0,P} |N i |-1 u i (P, a -i )s -i (a -i ) where s -i (a -i ) =   ∏ j∈T(a -i ) s j     ∏ j∈N i \{T(a -i ) {i}} (1 -s j )   and
T(a -i ) = {k ∈ N i \{i}, p k = P},
is the set of neighbors transmitting. We have f i 0 (s -i ) = 0 for any multi-strategy s -i . A multi-strategy s = (s i ) i=1,...N is neutrally stable if for all s ′ = s there exists ǫ y ′ > 0 such that ∀ǫ ∈ (0, ǫ y ′ )

f i s i (ǫs ′ -i + (1 -ǫ)s -i ) ≥ f i s ′ i (ǫs ′ -i + (1 -ǫ)s -i ) (3.6)
for some i. The multi-strategy s is evolutionary stable if the inequality (3.6) is strict. Hence, an ESS is a neutrally stable strategy but the reciprocal is not true. See [START_REF] Samuelson | Evolutionary Games and Equilibrium Selection[END_REF] for more details on neutrally stable strategy. If s is neutrally stable strategy then s is Nash equilibrium. The best response (BR) of the user i to s -i is given by

BR i (s -i ) =    1 if f i P (s -i ) > 0, 0 if f i P (s -i ) < 0, [0, 1] if f i P (s -i ) = 0.
A strategy s is a Nash equilibrium is equivalent to s i ∈ BR i (s -i ), i = 1, . . . , |N|. In particular when η = 0, the strategy ON is a Nash equilibrium (ON is a dominant strategy).

The following proposition gives necessary conditions for pure strategies and strictly mixed strategies to be neutrally stable. Proposition 3.6.0.1. We decompose the different equilibrium strategy in the following items.

• If the strategy ON (P) is a neutrally stable equilibrium then η ≤ η min where η min = 1 P min i=1,...,N log 1 + g ii

σ i P + ∑ j∈N i , j =i g ji .
Moreover, ON becomes a strictly dominant strategy if η < η min .

• If the strategy OFF (0) is a neutrally stable equilibrium then

η ≥ 1 P max i log 1 + g ii P σ i =: η max .
For η > η max , OFF becomes a strictly dominant strategy (hence, an ESS).

• If ∀ j = i, g ij = g, g ii = ḡ, σ i = σ then the game becomes a symmetric game. Hence, it exists a symmetric equilibrium with the proportion s(= s i , ∀i) of transmitters, which must satisfy

Q(s) := n-1 ∑ k=0 a k s k (1 -s) n-k-1 ( n-1 k ) = ηP (3.7)
where a k = log(1 + ḡP σ + kgP ) > 0 represents the fitness obtained by user i when he transmits and k of the opponents of user i decided to transmit and the n -1k others stay quiet.

• Every strictly mixed equilibrium (symmetric or not) must satisfy f i s i (s -i ) = 0 for all i.

Proof. The two first assertions of the proposition 3.6.0.1 are obtained by Nash equilibrium conditions: the strategy ON (P) is a neutrally stable equilibrium implies that ∀i, 0 ≤ f i P (P, . . . , P) and the strategy OFF (0) is a neutrally stable equilibrium implies that ∀i, f i P (0, . . . , 0) ≤ 0. In the last assertion (symmetric case), the fitness of user i is s(Q(s) -ηP) where (s i , 1s i ) is state of the cell. Since Q(1) < 0 and Q(0) > 0 when η min < η < η max , then the equation (3.7) has a solution s in (0, 1) which implies the existence of an ESS.

Remark that the strategy ON is an ESS if η < η min . In order to prove the existence and uniqueness of the ESS in the symmetric case, we study roots of the polynomial Q in the following lemma. Lemma 3.6.0.2. Let 0 ≤ a n-1 < a n-2 < . . . < a 0 , η and P positive reals satisfying ηP ∈ (a n-1 , a 0 ). Then, the polynomial Q(s) -ηP has a unique root on (0, 1).

Proof. We show existence and uniqueness of the root of the polynomial Q on (0, 1).

Existence Q is a polynomial with real coefficients. Hence, Q is continuous in (0, 1). The image of the interval (0,

1) contains I = [a n-1 -ηP, a 0 -ηP]. The inequality a n-1 P := η min < η < a 0 P := η max , implies 0 ∈ I. Thus, there exists, a ∈ (0, 1) such that Q(a) = 0. Uniqueness We show that Q is strictly monotone on (0, 1). Let Q ′ be the derivative of Q. Q ′ is given by Q ′ (s) = ∑ k≥0 (k + 1)( n-1 k+1 )a k+1 s k (1 -s) n-2-k -∑ k≥0 (n -k -1)( n-1 k )a k s k (1 -s) n-2-k Since (k + 1)( n-1 k+1 ) = (n -k -1)( n-1 k
), one has

Q ′ (s) = - n-2 ∑ k=0 (n -1)! k!(n -2 -k)! [a k -a k+1 ]s k (1 -s) n-2-k
The inequalities a ka k+1 > 0, k = 0, . . . , n -2 and s ≥ 0, 1s ≥ 0 imply that Q ′ (s) < 0 on (0, 1). Hence, Q is strictly decreasing on (0, 1). We conclude that Q is bijective from (0, 1) to I and hence, Q has a unique root on (0, 1)

Given this result, we have the following proposition given the existence and uniqueness of the ESS in the symmetric game. The proof is immediate from the lemma 3.6.0.2 and from proposition 3.6.0.1. Proposition 3.6.0.3. The symmetric power control game has a unique strictly mixed equilibrium.

Numerical investigation in OFDMA-based IEEE802.16 networks

We consider below numerical examples of an OFDMA system. We shall obtain the ESS for several values of η. We assume that g ii = g and g ij = ḡ for all i = j and σ i = σ for all i. We consider below a numerical example for different values of N ′ (see figure 3.24 in which N ′ = 7). Let the noise σ = 0.1 and the power attenuation g = ḡ/4 = 0.9. In figure 3.25 (resp. 3.26), we plot the ESS versus η (resp. power level P). In both figures, the population ratio using the strategy ON is monotone decreasing in η. We observe that the parameter η which can be interpreted as the pricing per unit of power transmitted, can determine whether the ESS is aggressive (in pure strategies or in mixed strategies. It can determine in the latter case what fraction of the population will use high power at equilibrium. Pricing can thus be used to control interference.

Notes

In this chapter we have adapted the theory of evolutionary games with a random number of players in wireless networks. This adaptation is needed in order to apply this theory for the study of access game and particularly in wireless networks. (i) We have proposed different scenario based on the level of information for each player in the slotted Aloha model. In all cases, we have obtained the existence and uniqueness of the ESS, we have proposed optimization issues for the transmission probability of success and finally, we have studied the impact of delay in the convergence to the ESS of the replicator dynamics. (ii) We considered power control games with one or several populations of users and studied evolutionarily stable states in interaction of numerous mobiles in competition in a wireless environment. We have modeled power control for W-CDMA and OFDMA systems as a non-cooperative population game in which each user needs to decide how much power to transmit over each receiver and many local interactions between users at the same time has been considered. We have derived the conditions for existence and uniqueness of evolutionarily stable states and characterize the distribution of the users over each power level.

Correlated Evolutionarily Stable Strategies in Random Medium Access

In this section we study dynamic multiple access in distributed wireless networks with random number of users. We apply evolutionary game theoretic analysis to solve several problems: (a)

We address the stability of Aloha-like systems with power levels. Specifically, we consider an arbitrary number of receivers distributed in several locations. They receive packets from random number of users accessing the resource using Aloha-like algorithms. We provide an explicit expression of equilibria, correlated evolutionarily stable strategies, and prove some asymptotic stability results. (b) We apply correlation mechanism and evaluate the performance of random medium access when saturated users interact through interference. We introduce the benefit of correlation (BoC) to measure the gap between the probability of success at correlated evolutionarily stable states and the worst probability of success of evolutionarily stable states. We show that if only two power levels are available, the correlation mechanism reduces considerably the interference and the number of collisions. Moreover, the correlation mechanism is stable in long-term under several classes of bio-inspired evolutionary game dynamics. (c) Surprisingly, when the number of strategies is at least three, the game has no pure equilibrium, and the correlation mechanisms do not improve the probability of success and BoC=0.

Random Medium Access Control (MAC) algorithms have played an increasingly important role in the development of wired and wireless networks and the performance and stability of these algorithms, such as slotted-Aloha, Carrier Sense Multiple Access (CSMA) is still an open problem [START_REF] Bardenave | Performance of Random Medium Access Control An asymptotic approach[END_REF]. Distributed Medium Access Control, starting from the first version of Abramson's Aloha to the most recent algorithms used in IEEE802.11, have enabled a rapid growth of both wired and wireless networks. They aim at efficiently and fairly sharing a resource among users even though each user must decide independently (eventually after receiving some messages or listening) when and how to attempt to use the resource. MAC algorithms have generated a lot of research interest, especially recently in attempts to use multi-hop wireless networks to provide high-speed access to the internet with low-cost and low-energy consumption.

In this section we focus our attention to wireless networks, where the resources are receivers, base station or access points and where users interact because of interference, i.e., interfering users cannot transmit simultaneously. There is a collision if another user (mobile) transmits with a greater power level at the same range of the receiver. Motivated by the interest of evolving dense networks, evolutionary game theory was found to be an appropriate framework to apply in networks. It has been applied to problems such as congestion control [START_REF] Menasche | An evolutionary game-theoretic approach to congestion control[END_REF], distributed cooperative sensing over cognitive radio networks (232; 160), code division multiple access (CDMA), Orthogonal Frequency-Division Multiple Access (OFDMA) based Worldwide Interoperability for Microwave Access (WiMAX) environment [START_REF] Altman | Evolutionary Power Control Games in Wireless Networks[END_REF], reciprocal and non-reciprocal interference control, mobile medium access control and channel selection [START_REF] Tembine | Evolutionary games with random number of interacting players applied to access control[END_REF] and capacity region of Additive White Gaussian Noise (AWGN) [START_REF] Gajic | Game theoretic considerations for the Gaussian Multiple Access Channel[END_REF], Multihoming and association problems [START_REF] Shakkottai | Multihoming of Users to Access Points in WLANs: A Population Game Perspective[END_REF]. A one-shot random access game have been studied in [START_REF] Inaltekin | A one-shot random access game for wireless networks[END_REF]. In (232; 88; 11; 109; 213) the authors did not analyze the performance and the fairness properties of these approaches and the number of strategies limited is fixed to two.

We provide a general evolutionary game theoretic framework to analyze networks where interfering users share a resource using an Aloha-type access control. We consider access control with arbitrary number of strategies and variable number of mobiles around each receiver. We study a large population of communicating terminals using a aloha-like protocol with several levels of transmission power. We examine how to choose between these power levels in order maximize their probability of success minus the cost of energy consumption. We study several solution and stability concepts: the Nash equilibrium and the Evolutionarily Stable State (ESS) and the correlated evolutionarily stable strategy. The concept of ESS were introduced in mathematical biology by Maynard Smith and Price (187; 186) in the context of Evolutionary Games, which allow to describe and to predict properties of large populations whose evolution depends on many local interactions, each involving a finite number of individuals. Evolutionary game dynamics are models of strategy change commonly used in evolutionary game theory. A strategy which does better than the average or its opponent, increases in frequency at the expense of strategies that do worse than the average or the opposed action. Many evolutionary game dynamics models are used in the literature (229; 168; 99; 65; 234). We compare the performances of these notions with the global cooperative solution. The payoffs that we consider are functions of the probability to have a successful transmission and of the cost for the power levels. We study in particular the impact of the pricing for the use of the power levels on the system performance. We analyze various solutions concepts and refinement: Nash equilibria, Pareto optimality, strong equilibria, correlated evolutionarily stable state. In order to study the interactions between mobiles and their stationary strategy in the long run, we develop some evolutionary game dynamics (99; 168; 229; 90) in order to study the convergence and the stability of equilibria.

Related works

In [START_REF] Wang | Evolutionary Game Framework for Behavior Dynamics in Cooperative Spectrum Sensing[END_REF], an evolutionary game-theoretical framework with two strategies: Cooperate or Defect has been proposed for distributed cooperative sensing over cognitive radio networks. By employing the theory of replicator dynamics, the authors study the behavior dynamics of secondary users, and further propose a distributed learning algorithm that gradually converges to the Nash equilibrium. In (20; 212), an evolutionary access game with battery state-dependent energy management have been proposed for distributed Aloha networks with low density. The authors consider only two strategies: high or low power and interference between more than three mobiles is neglected. Our analysis extend (20; 213; 109) and covers general distribution (eventually with mobility) of mobiles and arbitrary number of strategies. We propose an evolutionary access control game with random number of interacting players and arbitrary number of actions. Our work extends previous evolutionary networking game theoretic works (232; 20; 11) where only two strategies is considered. We use the Price of Anarchy (PoA) concept (28; 29) in order to measure the gap between equilibrium and social optima in the population game. We show that the PoA of evolutionarily stable state is more than the PoA of Nash equilibria in any population game with arbitrary number of strategies. In the evolutionary access game with non-degenerate costs and more than three strategies, the PoA at ESS denoted by PoA ESS coincide with PoA NE . Moreover, PoS ESS = PoS NE = 0. In contrast, if we have only two strategies and m mobiles around the receiver, the PoS NE = 1 ≥ PoA NE = 0 = PoA ESS . We introduce and analyze correlated evolutionarily stable states (CESS) in access control. CESS is a refinement of correlated equilibrium. We show that the evolutionary access control game with arbitrary number of opponent around each receiver has a several CESS which are more robust in term of fairness, stability and efficiency than correlated equilibria (CE) [START_REF] Auman | Subjectivity and Correlation in Randomized Strategies[END_REF]. We define the benefit of correlation (BoC) that measures the improvement in term of probability of success by introducing correlation between mobiles in the population game. To the best to our knowledge, this work is one the first which study correlated evolutionarily stable states in networking context. For the two strategies case, we show that the BoC of the access game is 100% i.e PoS CESS = 1 and PoA NE = 0. As a consequence, for the model studied in (109) the global optimum is attained using CESS. We compute explicitly an equivalence class of CESS which is a social optimum and which is local asymptotically stable under the evolutionary dynamics. In particular, our contribution gives some situations where the three properties: equilibrium, stability, optimality (global cooperative solution) holds. For more than two strategies, we show that the game has no pure equilibrium (Theorem 3.7.3) and a unique mixed equilibrium which we compute explicitly (Theorem 3.7.6). We develop a general class of evolutionary game dynamics for CESS. Surprisingly, when the number of strategies is at least three, the correlation mechanism does not improve the probability of success.

In next subsection we formulate problem of access control with several actions and introduce the evolutionarily equilibrium concept, the efficiency metric "Price of Anarchy" and evolutionary game dynamics with time delays. In subsection 3.7.1 we analyze equilibria: Nash, ESS, CE, CESS and price of anarchy, price of stability and benefit of correlation in Dirac distribution. We compute explicitly the equilibria and the price of anarchy in general distribution in subsection 3.7.5. We extend our study to asymmetric random access games with several classes and random number of interacting mobiles and formulate the SINR-threshold based admission control and in subsection 5.5.

Access control with several power levels

We consider a wireless communication network with distributed receivers in which some mobiles contend for access on a common, wireless communication channel. We characterize this distributed multiple access problem in terms of many random access game at each time. Random multiple access games introduce the problem of medium access. We assume that mobiles are randomly placed over an area and a distributed receivers in the corresponding area . The channels are ideal for transmission and all errors are due to collision. A mobile decides to transmit a packet with some power level or not to transmit (null power level) to a receiver when they are within transmission range of each other. Interference occurs as in the Aloha protocol where the power control is introduced: if more than one neighbors of the receiver transmit a packet with a power level which is greater than the corresponding power of the mobile at the same time slot there is a collision. The evolutionary random multiple access game is a nonzero-sum dynamic game, at each time, the mobiles in the same range have to share a common resource, the wireless medium. We denote by µ the probability that a mobile has its receiver within its range. We assume that µ > 0 (if µ = 0 there are no receivers, and hence no successful transmission).

Strategies

We assume that for each packet, its source can choose the transmitted power among several power levels P = {p 0 , p 1 , p 2 , . . . , p n } with p 0 < p 1 < . . . < p n i.e a strategy for a mobile corresponds to the choice of a power level in P. p 0 = 0 means that the mobile does not transmit, p n is the maximum power level available to the mobiles.

Aloha-type payoff with pricing

If mobile j transmits a packet using a power p j , it incurs a transmission cost of c(p j ) ≥ 0. The packet transmission is successful if the other users in the range of its receiver use some power levels strictly lower than p j in that given time slot, otherwise there is a collision. If there is no collision, user i gets a reward of V from the successful packet transmission. If c(a) > V for some power a then a is dominated by 0 (not transmit). For the remainder, suppose that the reward V is greater than the cost of transmission max j c(p j ) < V. All packets of a lower power level involved in a collision are assumed to be lost and will have to be retransmitted later. In addition, if more than one packet of a higher power level is involved in a collision then all packets are lost. The power differentiation thus allows one packet of a higher power level to be successfully transmitted in collisions that do not involve other packets of the higher power level. Then, a transmission of mobile j is successful if its transmission power is strictly greater than the power levels used by the others mobiles at the same slot. When the number of mobiles which transmitting at the receiver is m + 1, the payoff is given by u

j m+1 : P m+1 -→ R u j m+1 (a j , a -j ) = -c(a j ) + V × 1 if a j > max k =j a k 0 otherwise
where a -j denotes (a 1 , . . . , a j-1 , a j+1 , . . . , a m+1 ), c : R + -→ R + is a pricing function. We assume that

c(p 0 ) = c(0) = 0 ≤ c(p 1 ) ≤ c(p 2 ) . . . ≤ c(p n )
Denote a population profile as the frequencies (fractions) of use of pure strategies in the population. A population profile can be represented by an element of the n-simplex ∆ n of the (n + 1)-dimensional Euclidean space R n+1 . The expected payoff of a mobile j with the power level p i when facing to m others mobiles is given by

f m,p i (x) = µu j m+1 (x, . . . , p i , x, . . . , x) (3.8) = µ ∑ a∈P m u 1 m+1 (p i , a 1 , . . . , a m ) m ∏ j=1 x a j (3.9)
where x p j is the fraction of mobiles with the power level p j ∈ P, and x = (x p 0 , x p 1 , . . . , x p n ) is the population profile. Lemma 3.7.1.1. f m,p 0 (x) = 0, and for i ≥ 1,

f m,p i (x) = µ -c(p i ) + V ∑ j 0 +j 1 +...+j i-1 =m x j 0 p 0 x j 1 p 1 . . . x j i-1 p i-1 f m,p i (x) = -µc(p i ) + Vµ(x p 0 + x p 1 + x p 2 + . . . + x p i-1 ) m
Denote by K the random variable representing the number of opponents of a anonymous mobile picked in a the population when the population profile is x and by G K (s) = E K (s K ) the generating function of M. The expected payoff of a mobile using the power level p i can be expressed as

F p i (x) = E K ( f K,p i (x)) = VµG K (x p 0 + x p 1 + x p 2 + . . . + x p i-1 ) -µc(p i )
for all i > 0 and F p 0 (x) = 0.

We observe that if x is stochastically dominated by y, i.e for all i < n,

G(v, x) |v=p j = F p j (x).
A population profile x is an evolutionarily stable state (ESS) if for all other population profile mut = x there exists a ǫ mut > 0 (which may depend on mut) such that,

∑ b∈P (x b -mut b )G(v, (1 -ǫ)x + ǫ mut) |v=b > 0, ∀ǫ ∈ (0, ǫ mut ) (3.10)
When the inequality (3.10) is non-strict, and ǫ = 0, we obtain that the probability distribution x is a symmetric Nash equilibrium (NE).

Bio-inspired and G-function game dynamics

We adopt a class of evolutionary game dynamics based on the generating fitness (payoff) function (G-function) and revision protocols developed respectively by Vincent [START_REF] Vincent | Evolutionary Game Theory, Natural Selection, and Darwinian Dynamics[END_REF] and Sandholm [START_REF] Sandholm | Population Games and Evolutionary Dynamics[END_REF] in which we introduce time delays. The time delays characterize the expected delay needed to know if a transmission is successful or not. An action p i taken at time t will have its effect at time t + τ p i . Delayed evolutionary game dynamics with asymmetric time delays have been introduced in (214). In the evolving MAC context, we construct evolutionary game dynamics from a model of individual decision making, we introduce revision of protocols, which describe how mobiles adjust their choices of strategies during the game. A revision protocol is a Lipschitz continuous map β :

R n+1 × (∆ n ) n+2 -→ R (n+1)×(n+1) +
that takes a G-function G vectors G = (G(b, .)) b∈P and population profile x as arguments, and returns nonnegative matrices with size (n + 1) × (n + 1) as outputs. The generating The scalar β p i p j (G; x 1 , . . . , x n+2 ) is called the conditional switch rate from strategy p i to strategy p j . If the agents receiving revision opportunities independently according to rate κ Poisson processes, then β p i p j (G, x 1 , . . . , x n+2 ) κ represents the probability that a mobile with the power level p i who receives a revision opportunity switches to transmit with the power level p j . The two maps β and G together define a delayed non-linear differential equation

d dt x p i (t) = ∑ b∈P x b (t)β bp i (G, x(t), x(t -τ p 0 ), . . . , x(t -τ p n ) (3.11) -x p i (t) ∑ b∈P β p i b (G, x(t), x(t -τ p 0 ), . . . , x(t -τ p n ))
with initial condition

x(t) = φ(t), ∀t ∈ (-max b∈P τ b , 0). (3.12)
Note that the delayed differential equation (3.11) combined the condition x(t 0 ) = x 0 can defined an infinite solutions (this class of differential equation is not covered by Cauchy-Lipschitz's theorem) . To guarantee uniqueness of solutions under locally Lipschitz property of β we need to impose a initial condition known in an interval with length at least max Then the delayed evolutionary game dynamics defined by (3.11) and (3.12) has a unique solution.

The dynamic is said positively correlated (PC) if

d dt x = 0 =⇒ ∑ b [ d dt x b ][G(v, x) |v=b ] > 0.
In [START_REF] Sandholm | Population Games and Evolutionary Dynamics[END_REF], Sandholm showed that in absence of time delays, replicator dynamics [START_REF] Taylor | Evolutionary stable strategies and game dynamics[END_REF]

β p i ,p j = x p j (t) max 0, G(w, x(t -τ w )) |w=p j -G(v, x(t -τ v )) |v=p i
we obtain the delayed replicator dynamics

d dt x p j (t) = x p j (t) G(w, x(t -τ w )) |w=p j - n ∑ k=0 x p k (t)G(w ′ , x(t -τ w ′ )) |w ′ =p k j = 0, . . . , n,
Analogously, the delayed Brown-von Neumann-Nash dynamics is obtained for β p i ,p j = max (0, gg) where

gg = G w (x(t -τ w )) |w=p j - n ∑ k=0 x p k (t)G(w ′ , x(t -τ w ′ )) |w ′ =p k
and the delayed θ-Smith dynamics for

β p i ,p j = max 0, G(w, x(t -τ w )) |w=p j -G(v, x(t -τ v )) |v=p i θ ,
with θ ≥ 1. Note that under delayed evolutionary game dynamics, the ESS can be unstable (see Fig.3.35). If the dynamics is regular, a sufficient condition of stability of an ESS x * is given by: (i) all roots of the Jacobian of the function H defined by

H p j (y) = y p j (t) G(w, y(t)) |w=p j - n ∑ k=0 y p k (t)G(w ′ , y(t)) |w ′ =p k
have negative real value the ESS x * and (ii)

J(x * ) ∞ max b∈P τ b < 1
where J(x * ) ∞ denotes the norm sup of Jacobian of H.

Dirac distribution

In this subsection, M is the Dirac distribution δ m-1 (the number of opponents in the same range is m -1). If there is no pricing for the energy consumption then, the high power level weakly dominates the others power levels. We show that the one-shot random access game between m mobiles and without cost c(.) ≡ 0 (degenerate case) has an infinite number of equilibria. Only (p n , . . . , p n ) is a symmetric equilibrium. Proposition 3.7.1.5. If c(.) ≡ 0, then the one-shot random access game between m mobiles has an infinite number of Nash equilibria and unique a symmetric Nash equilibrium (p n , . . . , p n ) which is an evolutionarily stable state. Moreover, the game has many Pareto optima 3 .

Proof. (i) Symmetric Equilibrium It is clear that for all j, u j (p n , x -j ) ≥ u j (y j , x -j ), ∀ x -j ∈ (∆ n ) m-1 i.e p n weakly dominates the others power levels in P \{p n }. Hence, the strategy p n is an equilibrium. Moreover the best reply to the population profile x is to play p n if x p n = 1, and to play any strategy z ∈ ∆ n if x p n = 1. Thus, (p n , p n , . . . , p n ) is the unique symmetric equilibrium. At the equilibrium (p n , p n , . . . , p n ), the payoff of each mobile is zero. Thus, (p n , p n , . . . , p n ) is not Pareto optimal because the allocation obtained at (p n , 0, . . . , 0) or (p n , p 1 , p 1 , . . . , p 1 ) Pareto dominates zero.

(ii) Pure Equilibria: Fix a mobile j which uses the action p n . Then any action profile of the others mobiles x -j ∈∈ P m-1 leads to a Nash equilibrium (no mobile can improve its probability of success by deviating unilaterally). In particular, if j 0 , j 2 , . . . , j n-1 such that j i of the m -1 mobiles p i and j 0 + . . . + j n-1 = m -1 then no mobile can improve its probability of success by deviating unilaterally and this kind of configuration where only mobile use p n is also Pareto optimal. Thus, the game has the binomial number ( m 1 ) times the cardinality of P m-1 which is mn m-1 pure Nash-Pareto equilibria (pure Nash equilibria which are also optimal in Pareto sense). The total number of pure Nash equilibria is given by the cardinality of the set

(j 0 , . . . , j n ) ∈ N n+1 | j n ≥ 1, n ∑ i=0 j i = m, j i ≥ 0 (iii)
Partially Mixed Equilibria: Any situation where at least one of the mobiles use the strategy p n , and other mobiles use an arbitrary mixed strategy, gives a mixed Nash equilibria. The allocation of payoff obtained in these partially mixed strategy need not to be Pareto optimal if at least one mobile chooses the strategy p n with strictly positive probability.

(iv) Evolutionarily Stable State: The unique symmetric equilibrium p n is evolutionarily stable.

A multi-strategy is a strong equilibria if is a configuration from which no coalition (of any size) can deviate and improve their payoff (probability minus cost) of every member of the coalition (group of the simultaneous moves), while possibly lowering the payoff of mobiles outside the coalition group. A strong equilibrium is in particular a strong equilibrium (by taking coalition of size) but also Pareto optimal (coalition of full size). The Theorem 3.7.2 below describes Nash equilibria, Pareto optimality and coalition proof of two strategies {p 0 , p 1 } access game between m mobiles. Theorem 3.7.2. Suppose that n = 1 and 0 = c(p 0 ) < c(p 1 ) < V. Then the one-shot random access game has (i) 2 m -1 number of Nash equilibria, (ii) m of them are Nash-Pareto equilibria, and strong equilibria, (iii) a unique fully mixed Nash equilibrium which is not Pareto optimal. (iv) a unique ESS given by 1 -(

c(p 1 ) V ) 1 m-1 , ( c(p 1 ) V ) 1 m-1
. (v) the price of anarchy is zero for both ESS and Nash equilibria.

Proof. (Proof the Theorem 3.7.2) We show the following results: (i) If only one node transmit with p 1 and the others stay quiet (p 0 ), then the mobile which transmit with p 1 gets the payoff Vc(p 1 ) and the others receive nothing and has no cost c(p 0 ) = 0. This configuration is an equilibrium point. (ii) there are exactly m pure equilibria and all these pure equilibria are Pareto optimal (because the maximum to total payoff is attained) and strong equilibria. (iii) k (1 ≤ k < m -1) of the m mobiles choose to stay quiet and the mk others are active and play the optimal mixed strategy in the game with mk mobiles denote by

Γ m-k : 1 -ν 1 m-k-1 , ν 1 m-k-1
where ν := c(p 1 ) V . Thus, there are exactly

m-2 ∑ k=1 ( m k ) = 2 m -(m + 2)
partially mixed Nash equilibria. (iv) The game has a unique strictly mixed Nash equilibrium

given by 1ν

1 m-1 , ν 1 m-1
. (v) the allocation of payoff obtained in these (partially or completely) mixed strategy are not Pareto optimal.

Pure equilibria Suppose that the node i transmits and the others N \{i} stay quiet. Then mobile i will obtain the maximum payoff Vc(p 1 ) > 0. Mobile i obtains 0 by deviating (stay quiet). Hence mobile i has not incentive to deviate unilaterally. A mobile j = i will receive 0 and have no cost in this configuration. Suppose that mobile j decide to deviate unilaterally then mobile j payoff will be -c(p 1 ) < 0. Hence, no mobile has incentive to deviate unilaterally from this configuration. We change the role of mobile i and another mobile j to obtain the others pure equilibria (permutation).

Completely mixed equilibrium

The payoff of node i is f m (s) = s i [(-c(p 1 )(1 -η) + (V -c(p 1 ))η]
where η = ∏ j =i (1s j ). The best response (BR i ) of the node i to the multi-strategy s -i of the others nodes is given by

BR i (s -i ) =            1 if η > c(p 1 ) V 0 if η < c(p 1 ) V [0, 1] if η = c(p 1 ) V
Note that at η will be independent of i. Thus, for i, k ∈ {1, . . . , m}, the quotient

∏ j =i (1 -s j ) ∏ j =k (1 -s m ) = 1 = 1 -s i 1 -s k . Hence, s 1 = . . . = s m . We conclude that s j = 1 -(m-1) √ ν, ∀j ∈ N is the unique strictly mixed Nash equilibrium.
Partially mixed equilibria Suppose that m > 2 and fix an integer k between 1 and m -2. Suppose k nodes stay quiet and mk transmit with positive probability.Without lost of generality, we can suppose that the k first nodes {1, 2, . . . , k} does not transmit and any node j with j > k participate the game Γ m-k . The optimal strategy for j (j > k) is s j = 1 -(m-k-1) √ ν. For j ≤ k, we show that mobile j have no incentive to deviate because payoff obtained by node j by deviating is

s i [(-c(p 1 ))(1 -ν) + (V)ν] -0 × (1 -s i )ν
which is lower than 0. For m = 2, the game becomes the two-player game

1\2 p 1 p 0 p 1 (-c(p 1 ), -c(p 1 )) (V -c(p 1 ), 0) p 0 (0, V -c(p 1 )) (0, 0)
It is easy to see that the Nash equilibria of the strategic game Γ 1 are (p 1 , p 0 ), (p 0 , p 1 ) and (1ν, ν).

Pareto optimality Note that the allocation of payoff obtained in these (partially or completely) mixed strategy are not Pareto optimal because the payoff of each mobile is zero at these configuration. This completes the proof.

The negative results (ii), (iii), (v) of Theorem 3.7.2 in term of performance can be improved by introducing the concept of correlated evolutionarily stable state (CESS). We will introduce CESS in next subsection and exhibit a class of CESS for which the system is stable and 100% efficient (the payoff at the CESS is a social welfare) and the system is stable in long-term. We first remark that the result of Theorem 3.7.2 (i),(ii),(iv) for two strategies ♯P = 2 does not holds for arbitrary number of strategies ♯P ≥ 3 as shown in the following examples (see Fig. 3.27) where pure equilibria may not exists.

The three strategies two-player game has no equilibrium in pure strategies (see Fig 3.28 and 3.27). The game has a unique completely mixed strategy

( V -c(p 2 ) V , c(p 2 ) -c(p 1 ) V , c(p 1 ) V ) ∈ ∆ 2
obtained by solving the indifference equations. For the remainder we assume that 0

p 2 p 1 p 0 p 2 (-c(p 2 ), -c(p 2 )) (V -c(p 2 ), -c(p 1 )) (V -c(p 2 ), 0) p 1 (-c(p 1 ), V -c(p 2 )) (-c(p 1 ), -c(p 1 )) (V -c(p 1 ), 0) p 0 (0, V -c(p 2 )) (0, V -c(p 1 )) (0, 0)
p 2 p 1 p 0 p 2 (-c(p 2 ), -c(p 2 ), -c(p 2 )) (-c(p 2 ), -c(p 1 ), -c(p 2 )) (-c(p 2 ), 0, -c(p 2 )) p 1 (-c(p 1 ), -c(p 2 ), -c(p 2 )) (-c(p 1 ), -c(p 1 ), V -c(p 2 )) (-c(p 1 ), 0, V -c(p 2 )) p 0 (0, -c(p 2 ), -c(p 2 )) (0, -c(p 1 ), V -c(p 2 )) (0, 0, V -c(p 2 )) A1 := p 2 p 1 p 0 p 2 (-c(p 2 ), -c(p 2 ), -c(p 1 )) (V -c(p 2 ), -c(p 1 ), -c(p 1 )) (V -c(p 2 ), 0, -c(p 1 )) p 1 (-c(p 1 ), V -c(p 2 ), -c(p 1 )) (-c(p 1 ), -c(p 1 ), V -c(p 1 )) (-c(p 1 ), 0, -c(p 1 )) p 0 (0, V -c(p 2 ), -c(p 1 )) (0, -c(p 1 ), -c(p 1 )) (0, 0, V -c(p 1 )) A2 := p 2 p 1 p 0 p 2 (-c(p 2 ), -c(p 2 ), 0) (V -c(p 2 ), -c(p 1 ), 0) (V -c(p 2 ), 0, 0) p 1 (-c(p 1 ), V -c(p 2 ), 0) (-c(p 1 ), -c(p 1 ), 0) (V -c(p 1 ), 0, 0) p 0 (0, V -c(p 2 ), 0) (0, -c(p 1 ), V -c(p 1 )) (0, 0, 0)
= c(p 0 ) < c(p 1 ) < c(p 2 ) < . . . < c(p n ) < V.
We have seen in Theorem 3.7.2 that with two strategies, the access game between m mobiles has several pure equilibria and Pareto optimal solutions. The following Theorem 3.7.3 shows that Theorem cannot be extended for three strategies and an equilibrium do not exist in pure strategies. Theorem 3.7.3. Let ♯P be the cardinality of P. The evolutionary random access game with more than two strategies ♯P ≥ 3, and m ≥ 2 mobiles has no pure equilibrium.

Proof. Fix a strategy profile (a 1 , . . . , a m ) ∈ P m with ♯P ≥ 3. Let T1 = arg max j a j . We distinguish two cases: ♯T1 = 1 or ♯T1 ≥ 2

• suppose that ♯T1 = 1 and let T1 = {j * }. One has in particular p j * > p 0 = 0. If a j * = p i < p n then any mobile k in P \T1 which deviates and uses a l > p j * > p 0 = 0 (the existence of a l is guaranteed because there are more than three strategies) will improve its payoff from -c(a k ) to Vc(a l ).

If a j * = p n then mobile j * can improve its payoff from Vc(p n ) to Vc(b j * ) by playing a second higher power level b j * ∈ T2 = arg max j / ∈T1 a j or one the others mobiles can save energy.

• suppose now that ♯T1 ≥ 2. Then if (a 1 , a 2 , . . . , a m ) = (p 0 , . . . , p 0 ) there is collision for all mobiles which transmit, otherwise (a 1 , a 2 , . . . , a m ) = (p 0 , . . . , p 0 ) and any mobile which deviates and uses a power greater than p 0 will have a successful transmission.

In both cases, there is at least one mobile can deviate and improve its payoff. Thus, (a 1 , . . . , a m ) is not an equilibrium. Since (a 1 , . . . , a m ) is an arbitrary pure strategy profile in P with ♯P ≥ 3.

We conclude that the game between m mobiles has no pure equilibrium if the number of pure strategies is at least three.

Theorem 3.7.4. Then the evolutionary access game with m mobiles and at least three strategies has a unique Nash equilibrium (in mixed strategies) given by

x p 0 = c(p 1 ) V 1 m-1 , 1 ≤ j < n, x p j = c(p j+1 ) V 1 m-1 - c(p j ) V 1 m-1 , x p n = 1 - n-1 ∑ j=1 x p j -x p 0 = 1 - c(p n ) V 1 m-1
Proof. The result follows by using the inverse of the function x m in (0, 1) and solving the system (Cramer invertible)

                 
x p 0 x p 1 x p 2 . . .

x p n-1 x p n          =                     c(p 1 ) V 1 m-1 c(p 2 ) V 1 m-1 c(p 3 ) V 1 m-1 . . . c(p n ) V 1 m-1 1                    
Proposition 3.7.4.1. The price of anarchy and the price of stability (PoS) of Nash equilibria in the random access game are both zero for ♯P ≥ 3.

Correlated evolutionarily stable state

In this subsection we focus on the concept of correlated evolutionarily stable state (CESS) in access games. A correlated equilibrium is introduced by Aumann (32) in 1974 and can be interpreted as a distribution of actions or messages given to the mobiles by some referee (which can be the base station or the receiver) before to play each local game. For more details on correlated equilibrium, we refer the reader to (32; 33; 125; 58). As in [START_REF] Koch | Evolution and Correlated Equilibrium[END_REF], we use the concept of correlated equilibrium from the perspective of bounded rationality. In term of medium access control, the correlated mechanism phase (the phase of messages reception from the receiver before transmission) can be seen as the analogue of the probing or listening phase in CSMA algorithm (users willing to use the channel must first probe -or listen to-the channel, and when it is not busy, they can decide to transmit).

Let N := {1, 2, . . . , m}. We define a probability space Ω, 2 Ω , P m which generates signals on which the mobiles can condition their strategic power choices where Ω = P m . The set Ω is partitioned as follows I j (b) := {ba 1 a 2 . . . a j-1 a j+1 . . . a m | a l ∈ P, l = j}, I j := {I j (b), b ∈ P }. then, I j has exactly |P | = n + 1 elements which are information sets of mobile j. We define an assignment function (called also rule) profile α = (α 1 , . . . , α m ) as a mapping from the set of states or signals Ω to mixed strategies set ∆(P ). For all j, the assignment function of the mobile j, α j must satisfy : if for some w, α j (w) = b, then α j (w ′ ) = b, for all w ′ ∈ I j (s).

That is, for each element w ∈ I j , mobile j cannot distinguish states that are in the same information set (same equivalence class). We denote the set of all pure assignment functions by AF : {g | g : P -→ P }. Thus, when a mobile chooses an assignment function α and when he receives the signal ω ∈ Ω from the referee, he will choose the mixed action α(ω). We use α(a j |ω) to denote the probability assigned on a j under this mixed action α(ω). Then α(w) = [α(p 0 |ω), . . . , α(p n |w)] ∈ ∆(P ). Given a referee (Ω, P), we define the identity assignment function as α id,m (a j |ω) = 1 if proj j (ω) = a j where proj j (ω) denotes the j-th element of the signal ω and α id,m (a j |ω) = 0 for all ω such that proj j (ω) = a j . If each mobile use the identity assignment then, the resulting probability distribution of actions profile actually played will be P, the same as the probability distribution of actions recommended by the referee. But when mobiles use other assignments, a different distribution may result. Given an assignment profile α and a probability distribution P over P m , the expected payoff is given by

f (α) = ∑ ω∈Ω P(ω) ∑ a∈P m u m (a) ∏ j∈N α j (a j |w) = ∑ a∈P m u m (a) ∑ ω∈Ω P(ω) ∏ j∈N α j (a j |w) = ∑ a∈P m u m (a)Q α (a)
where Q α (a) = ∑ ω∈Ω P(ω) ∏ j∈N α j (a j |w). We say that two assignment functions profile α and β are equivalent if they induce the same value Q α = Q β Proposition 3.7.4.2. Suppose that for all a ∈ P m P(a) = P(σ(a)), for all permutation σ (the distribution P is said symmetric). If an assignment profile β is equivalent to α id,m then Q β (a) = P(a), for all a ∈ P m .

Proof. One has, Q α id,m (a) = P(a), ∀a ∈ P m . The result follows immediately. Now, suppose a small group of mutants appears. These mutants use a mutational assignment function α ′m . which is not equivalent to the identical assignment function α id,m ., but they cannot change the referee recommendation. Let ǫ be the portion of the population which are mutants (who use α ′ ,) and 1ǫ portion of the population are non-mutants who use α id,n . At each time, m mobiles are randomly chosen to play the strategic access game. In playing the access game, the mobiles have the same referee (Ω, P)). A probability distribution P over P m is an CESS if non-mutants with identity assignment function perform better than mutants with assignment functions that are not equivalent to the identical assignment function. Definition 3.7.4.3. A CESS P is a symmetric distribution probability over P m such that for every assignment function α ′m nonequivalent to the identical assignment function α id,m , there exists some

ǫ α ′m > 0 such that F m (α id,m , ǫα ′m + (1 -ǫ)α id,m ) > F m (α ′m , ǫα ′m + (1 -ǫ)α id,m ) (3.13)
for all ǫ ∈ (0, ǫ α ′m ) where F m (β, α) = ∑ r∈AF β r f (r, α, . . . , α)

When the inequality (3.13) is non-strict and ǫ = 0, the probability distribution P is a correlated equilibrium (CE). Hence, a CESS is a CE. Proof. It is easy to see that x ⋆(m) is a probability measure on P m . Since x is an ESS, (x, . . . , x) ∈ ∆(P ) m is a Nash equilibrium of the m-player game without correlated device (Ω = ∅ or mobiles ignore the signals) i.e

∑ a∈P m u m (a) ∏ j∈N x(a j ) ≥ ∑ a∈P m u m (b k , a -k ) ∏ j∈N \{k} x(a j ) ≥ 0, ∀b k ∈ P.
This means that the product probability measure x ⋆(m) is a CE. Moreover,

u m (x ǫ , . . . , x ǫ , x, x ǫ , . . . , x ǫ ) > u m (x ǫ , . . . , x ǫ , y, x ǫ , . . . , x ǫ ), ∀y implies that ∑ ω∈Ω P(ω) ∑ a∈P m u m (a)x(a i |w) ∏ j∈N \{i} x ǫ (a j |w) > ∑ ω∈Ω P(ω) ∑ a∈P m u m (a)y(a i |w) ∏ j∈N \{i}
x ǫ (a j |w)

where P(a) = ∏ j∈N x(a j ), x ǫ := (1ǫ)x + ǫy.

More generally all convex combination of Nash equilibria (NE) are correlated equilibria and the set of CE is convex. This convexity properties of the CE's set is used in some algorithms based on linear or convex programming and generically, the complexity is less than in Nash equilibria in finite action games. We denote by ∆ CESS the set of CESS. Then, Lemma 3.7.4.5. The following inclusions holds:

∆ ESS ⊂ ∆ NE ⊂ ∆ CE , ∆ ESS ⊂ ∆ CESS ⊂ ∆ CE , (3.14)
One of the principal interest to study CESS is that, in general CESS remains stable for small delays in various game dynamics and the expected payoff obtained at ESS can be improved by CESS distribution. Non-ESS can be CESS as shown below.

We now construct of a class CESS for arbitrary number of users with two actions.

Number of mobiles which transmit with power p i

We define numb p i : P m -→ N as the number of transmitters of the power p i . For example, numb p n (p n , . . . , p n ) = m and numb p n (p n , a 2 , . . . ,

a m ) = 1 if a j = p n , ∀j ≥ 2. The set {b ∈ P m | numb p 1 (b) = 1, numb p 0 (b) = m -1} is
exactly the situations where only one of the mobiles transmits with the power p n and the others mobiles use the power level p 0 (do not transmit). numb T (b) = 1 if and only if there exists a unique j such that a j = p n i.e all permutations of the actions profile (p n , a 2 , . . . , a m ) and permutations satisfying p j < p n ∀j. There are mn m-1 possibilities.

Denote by Θ

m := {a ∈ P m | numb p 1 (a) = 1, numb p 0 (a) = m -1}.
Proposition 3.7.4.6. For two strategies, the probability distribution P m over P m defined as

P m (a) = 1 m if a ∈ Θ m 0 otherwise is a CESS with the payoff V -c(p 1 ) m > 0.
Moreover at each slot, we have a successful transmission with probability one (if each mobile have some packet to transmit at each slot) and hence, the CESS leads to a social optimum. The price of anarchy of the class of assignment functions with the distribution P m is one (i.e the proposed method is 100% efficient). This class of CESS is also energy efficient (the energy consumption is minimized).

Proof. Θ m has exactly m actions. Thus, P m is a probability measure. We show that every strategy α ∈ P used in P m , ∑ a -j ∈P m-1 P m (α, a -j ) > 0 and any alternative strategy b j ∈ P \{α}, it holds that ∑ a -j ∈P m-1 P m (α, a -j ) u(α, a -j )u(b j , a -j ) > 0 i.e Since n = 1, we have verify for the inequality for two strategies: for b j ∈ P \{p

1 } one has, [u m (p 1 , a -j ) -u m (b j , a -j )]P m (p 1 , a -j ) = V -c(p 1 ) + c(b j ) m > 0. By summing over a -j ∈ P m-1 , ∑ a -j ∈P m-1 [u m (p 1 , a -j ) -u m (b j , a -j )]P m (p 1 , a -j ) >
0 for all permutation on the position of j and b j = p 0 . Similarly, for b j ∈ P \{p 0 },

a -j ∈ Θ m-1 , one has, [u m (p 0 , a -j ) -u m (b j , a -j )]P m (p 1 , a -j ) = c(p 1 ) m > 0 Hence P m is a strict CE. Thus, P m
is a CESS and the system has a successful transmission at each slot with probability one. The total payoff obtained at the CESS is exactly Vc(p 1 ) which is strictly greater than the payoff obtained at the ESS (zero). The social optimum is Vc(p 1 ) which guarantees a successful transmission with the minimum power consumption. thus, the CESS is efficient in term of energy consumption (by considering par example the ratio between the probability of success and cost of energy consumption as the energy-efficient metric).

Benefit of Correlation

One of the advantages of a correlated evolutionary stable state is that it has a potential to reduce the distance between the optimal solution and the ESS solution obtained as an outcome of users which decide independently. Proposition 3.7.4.7. The following inequalities holds:

PoS CESS ≥ PoS ESS ≥ PoA ESS ≥ PoA NE ≥ PoA CE (3.15) PoS CE ≥ PoS NE ≥ PoS ESS ≥ PoA ESS ≥ PoA NE ≥ PoA CE (3.16)
Proof. Apply the Lemma 3.7.4.5.

We define the benefit of correlation (BoC) as PoS CESS -PoA NE . Corollary 3.7.4.8. The benefit of correlation in the random access game is 100% for Ω = Θ m .

BoC = 1

Probability of success: Comparison at NE, ESS and CESS

In the numerical examples below, we show how the pricing function can optimize the network throughput or probability of success. we first investigate the impact of the different parameters and pricing on the NE, ESS and CESS and the convergence of the replicator dynamic with time delays. We also discuss the impact of the pricing function on the system capacity. We took the parameters: three strategies P = {p 0 , p 1 , p 2 }, n = 2, V = 1, µ = 0.8 (80% of coverage). The expected probability of success of a mobile is given by n ∑ j=1 x p j G K (x p 0 + . . . + x p j-1 ). These numerical examples (Fig. 3.29,3.30, 3.31) confirm the optimality of CESS in random access game with two strategies: probability to have a successful transmission at the CESS is maximum and equal to the probability that a resource exists times the probability that a randomly selected mobile has a packet to transmit which we represent by µ = 0.8.

Limitation of CESS's approach in access game

We have seen that for ♯P ≤ 2 or for c(.) = 0, CESS can improve the performance of system and some class of CESS are social welfare. But for ♯P ≥ 3, the correlated equilibria is reduced the Nash equilibria. Hence, the correlation is not needed. If we reduce the set of signal (message)

= PoS ESS = PoA ESS = PoA NE . If ♯P ≥ 3 then PoS CESS = PoA CESS = PoS ESS = PoA ESS = PoS N E = PoA NE = 0.

Game dynamics for CESS

The class of game dynamics based on rule or assignment functions r ∈ AF can be extended to correlated strategies. These dynamics are systems of non-linear delay differential with ♯AF = m m equations. The revision protocol β gives a matrix with size m 2m . y be a distribution of probabilities on AF . Then y r is the fraction of the population of mobiles with the assignment function r. The evolutionary game dynamics is then given by where Ḡ is a G-function defined on a more general space that contains the set of assignment function satisfying Ḡv (y

) v=r = F r (y) = ∑ r-j ∈AF m-1 f m (r, r-j ) ∏ i =j y ri .

General distribution

Theorem 3.7.6. The evolutionary access game with random number of interacting mobiles around each receiver and at least three strategies has a unique fully mixed Nash equilibrium given by

x p 0 = G -1 K c(p 1 ) V , 1 ≤ j < n, x p j = G -1 K c(p j+1 ) V -G -1 K c(p j ) V , x p n = 1 - n-1 ∑ j=1 x p j -x p 0 = 1 -G -1 K c(p n ) V under the condition that P(K = 0) < c(p 1 ) V Proof. If P(K = 0) < c(p 1 ) V all the real numbers c(p 1 ) V , c(p 2 ) V , . . . , c(p n ) V are in set G K (I)
where I is the interval (0, 1) and G K is a bijection from I to (P(K = 0), 1). The result follows by using the inverse of the G-function of the random variable in (0, 1) and solving the system (Cramer invertible) In figures 3.33, 3.34 and 3.35, we describe numerical application of our evolutionary game model with Poisson distribution of mobiles under the delayed replicator dynamics in which each pure strategy is associated with its own delay (τ p 0 = 0, τ p 1 = τ). We took a poisson distribution with parameter λ = 5. in Poisson distribution. In the figure 3.35 we illustrate a situation where the ESS is not stable (for large delays) and in the figure 3.33 and 3.35 the dynamics converge to the ESS which is stable (for small delays). These figures show that the ESS can be unstable for large time delays. In the particular case of pairwise interactions K = 1 where the payoff functions F b are linear, there is a method to obtain the equilibrium from oscillating trajectories of the delayed replicator dynamics if its remains at the relative interior of the simplex starting from any interior function φ(t) ∈ int∆ n , t ∈ (-τ, 0), then the time average trajectories of the delayed replicator dynamics converge to the ESS i.e

         1 0 0 . . . 0 0 1 1 0 . . . 0 0 1 1 1 . . . 0 0 . . . . . . . . . . . . . . . 0 1 1 1 . . . 1 0 1 1 1 . . . 1 1                   x p 0 x p 1 x p 2 . . . x p n-1 x p n          =                  G -1 K c(p 1 ) V G -1 K c(p 2 ) V G -1 K c(p 3 ) V . . . G -1 K c(p n ) V 1                 
A T = 1 T -s T s x p j (t) dt -→ x * p j .
Since the access game has a unique interior equilibrium, the delayed replicator dynamics has a unique interior rest point. Since the set ∆ n is compact, there exists φ such that the subsequence A φ(T) converges (to some point x j ). Similarly, the vector A ξ(T) converges to x. We show that x is a rest point of the delayed replicator dynamics.

log(

x p j (ξ(T))

x p j (s) ) ξ(T) -s = 1 ξ(T) -s G w, ξ(T) s x(t -τ w ) dt w=p j (3.18) - 1 ξ(T) -s ∑ b∈P ξ(T) s x b (t)G(v, (x(t -τ b )) v=b dt
This implies that when T goes the infinity, one has F p j (x) = constant. By uniqueness of the interior equilibrium, we conclude that x = x * . Note that for the Brown-von Neumann-Nash dynamics the limit of the time average trajectories can be different than x * (see (99; 44) for details on the so-called Shapley triangle, Time Average of the Shapley Polygon TASP) and Birkoff center.

Extension

We describe two possible extensions of the access game studied in previous subsections. The first is the asymmetric case: multi-class of users and the cost function are not necessarily the same. We give a multi-class evolutionary game dynamics for this model. The second extension is a signal-to-noise-plus-interference ratio (SINR) based admission control in which a mobile has a successful transmission if its SINR of the mobile is greater than some fixed threshold β.

Dynamics for asymmetric random access games

We use the model of multi-class evolutionary game to study the evolution of population profile (frequencies) and the asymptotic stability of (strict) equilibria in long-term. For example, the delayed replicator dynamics becomes

SINR-threshold based admission control

We now focus in the case where the value V of successful transmission may depend on the power consumption such as in the effective throughput or capacity depending on the signal to noise plus interference ratio (SINR). The system can accept several mobiles depending on their SINR. The packets of mobiles with a very low SINR (less than β) are lost. In that case the appropriate game formulation is an evolving game with variable number of players and coupled constraints. When facing to m others mobiles, the mobile is subjected to the Quality of Service (QoS) constraints

SI NR j (a j , a -j ) = a j h j σ 2 + γ ∑ k =j a k h k ≥ β
where β is some threshold, σ 2 noise term, h is the gain, γ is the load. The channel decodes correctly if the SI NR j (a) ≥ β. This implies that, given the power level of the mobiles a -j , the mobile j has a successful transmission if its power level is in

P j (a -j ) := b ∈ P j | b ≥ β h j (σ 2 + γ ∑ k =j a k h k ) .
By doing this SINR-threshold admission control, more than one mobile can have successful transmission and the number of collisions is reduced compared the models described in previous Sections. The payoff is then

u j m+1 (a j , a -j ) = -c(a j ) + V 1 if a j ∈ P j (a -j ) 0 otherwise
Let k be sup{l,

p 1 h σ 2 + l p 1 γ
h ≥ β} where h = min h j , h = max h j . Then k mobiles can have successful transmission if they use the power level p 1 and the others mobiles use p 0 . The class of CESS constructed in section 3.7.1 can be formulated as follows:

Θm = {b ∈ P m | numb p 1 (b) = k, numb p 0 (b) = m -k}
Under the correlated strategy Pm given by the distribution Pm (a) = 1 ♯ Θm if b ∈ Θm and 0 otherwise, each user has Vc(p 1 ) ♯ Θm . The expected payoff is then k(Vc(p 1 )) and total energy consumption is kp 1 which is the minimum possible for k successful transmission.

Notes

Biological models and tools have inspired a growing number of studies and of designs of decentralized wireless networks. In various ways, this work falls in this category of biology-inspired algorithms and decision making theory. The evolutionary game theoretic framework proposed here incorporates several actions for each mobile for interference, and admission control. We have analyzed correlated evolutionary stable states which preserve robustness, stability and high performance compared to ESS and Nash equilibrium. An interesting extension of this work is the evolutionary communication equilibria (85) that extends the correlated evolutionary stable states by introducing more general signal (message) space.

Notes

In next chapter we develop a new class of evolutionary game dynamics which covers migration and mobility of players in a one-hop neighborhood. This location-dependent evolutionary game dynamics with multicomponent strategies is well adapted to hybrid systems. Example of applications to power control in heterogeneous networks is given. The chapter is based to our publications in (209; 210).

Chapter 4

Evolutionary game dynamics with migration

In this chapter, we propose a class of bio-inspired evolutionary game dynamics with migration for hybrid population games with many local interactions at the same time. Each local interaction concerns a random number of interacting players. The strategies of a player have two components. Specifically, each player chooses both (i) the region or subpopulation and (ii) a strategy among a finite set of secondary pure strategies in each region. We investigate what impact the restriction the actions, when updating a strategy, a player can change only the secondary strategies associate to the region at a time, has on the population dynamics. We apply this model to the integrated power control and base station assignment problem in a multi-cell in code division multiple access (CDMA) wireless data networks, in OFDMA-based WiMAX environment, and association between technologies in heterogeneous networks with large number of mobiles. We show that global neutrally evolutionary stable states are stationary points of hybrid mean dynamics called dynamics with multicomponent strategies under the positive correlation conditions. We give some convergence results of our hybrid model in stable population games and potential population games under some particular class of dynamics.

Introduction

Power control in wireless networks has become an important research area. Since the technology in the current state cannot provide batteries which have small weight and large energy capacity, the design of tools and algorithms for efficient power control is crucial. For a comprehensive survey of recent results on power control in wireless networks an interested reader can consult e.g., [START_REF] Meshkati | Energy-Efficient Resource Allocation in Wireless Networks[END_REF] and the reference therein. Power control protocols based on game theory have been designed for already ten years starting with the pioneering work (76; 115). Non-cooperative games provide a convenient framework for decentralization and distributed decision making in those applications, where as cooperative approaches of game theory have allowed to handle issues concerning fairness in power allocation [START_REF] Altman | Fair power and transmission rate control in wireless networks[END_REF].

Most applications of game theory to power control consider mobile terminals as players of the same type and study strategic one-shot games with a fixed number of players. Here, we consider a population game with many simultaneous local interactions, where the game is played infinitely under some self-organizing process called "hybrid dynamic" and where each inter-action concerns a random number of players [START_REF] Tembine | Evolutionary games with random number of interacting players applied to access control[END_REF]. We develop several class of bio-inspired game dynamics with migration (148) in a hybrid evolutionary game model in code division multiple access (CDMA) wireless data networks, in OFDMA-based WiMAX environment and in association problem between several types of technologies such as UMTS 3G, WiMAX, WiFi etc.

A hybrid non-cooperative game model for wireless communication has been studied in (4) as an extension of the classical non-cooperative power control game formulation in CDMA system in which mobiles were considered to be connected to the closest base station (BS).

We consider in this chapter an evolutionary game model and study a hybrid dynamic for updating actions along with its convergence to an equilibrium point. We apply this model to the integrated power control and base station assignment problem as in such that each mobile's action space consists of the choice of a power level and a base station in some region which corresponds to a cell. The first advantage to use evolutionary framework to model this problem is that, under suitable assumptions, evolutionary game dynamics give naturally an algorithm for converging to an equilibrium point. Evolutionary Game theory describes the evolution with some dynamic process and involves strategic interaction over time in large populations of users.

Contribution

In the first part of this chapter we develop several classes of bio-inspired game dynamics with migration for hybrid systems. We apply our hybrid evolutionary game model in code division multiple access (CDMA) wireless data networks, in OFDMA-based WiMAX environment.

The second part of the chapter focuses on multi-class Stackelberg approach:

• On one hand based on the leader-followers games with constraints in large population of players, general model of discrete and continuous time game dynamics are proposed.

• We apply evolutionary Stakelberg games1 in several contexts including CDMA cognitive radio networks, OFDMA-based WIMAX environment in wireless networks and association between several technologies in hybrid networks.

• we develop a coordination mechanisms between players. To be best to our knowledge, this work is one of the first to apply multilevel evolutionary games in networking context.

Structure

The chapter is structured as follows. We first provide in the next section the model with different second strategies set in each region and we develop the hybrid mean dynamics. After that we study the power control game in multi-cell CDMA and we give some numerical investigations. We give some convergence results in potential population games and stable population games [START_REF] Tembine | Stable Networking Games[END_REF] with constraints. We then describe a Stackelberg approach in population games with choice constraints.

The Hybrid Model

We consider a population game model with multicomponent strategies. The population game consists of

• A large number of players.

• Each member of the population has a multicomponent strategy: first strategy (region) and secondary strategy.

• The players with the same first strategy have to choose their secondary strategies in a finite set and compete in a local non-cooperative game with either a finite random number of interacting players or with all the players in the region. The set of strategies available to players in all the population is S and has typical elements (r, a), (r, b). Let A r = {p 1 r , . . . , p n r r } the pure secondary strategies set of the region r and, S r = {(r, a), a ∈ A r } the pure strategies of a r-player. Then S = ∪ r S r . Denote by l r the mass of the region r, l the total mass of the population,

X r = { x r ∈ R |A r | , x r a ≥ 0, ∑ a x r a = l r }
the set of mixed secondary strategies in the region r and,

X = { x ∈ R ∑ r |A r | , x r a ≥ 0, ∑ r ∑ a x r a = ∑ r l r = l}
the set of all mixed strategies. Without loss of generality, we normalize the total mass l of the population to one. We call state of the population an element of X and, state of the region r, an element of X r .

• Each player from each local interaction receives some payoff. By F r a (x) we denote the expected payoff (reward minus loss) of the second strategy a ∈ A r in the region r when its state is x r and the state of the other regions is x -r . If every user takes a part in some region in a non-cooperative game between K r + 1 users where K r is a random variable over {0, 1, 2, . . .}, then the payoff F r a (x) can be expressed as

F r a (x) = ∑ k≥0 P(K r = k)F r,k a (x)
where F r,k a (x) is the expected payoff of the strategy a obtained in the local interaction between k users in the region r when the state is x. We denote by F : X → R ∑ r |A r | , the payoff function of all the population.

In addition to the study of the equilibrium of the game we shall consider also some evolutionary dynamics and study its convergence properties. We shall assume that players revise their strategies and use the strategies with higher payoffs. The system evolves under some evolutionary game dynamic process that describes the change of strategies (incoming flow and outgoing flow) in the population.

Global Nash Equilibrium

We say that x = (x r a ) r,a is a Global Nash equilibrium (GNE) if for all deviation multi-strategy mut = (mut r a ) r,a ,

∑ r ∑ a (x r a -mut r a )F r a (x) ≥ 0. (4.1)
Wardrop equilibrium When the number of opponents is very large (possibly infinity), the corresponding Nash equilibrium is sometimes referred to Wardrop equilibrium. A state x is a Wardrop equilibrium if for any region r, all strategies being used by the members of the region r yield the same payoff, the payoff that would be obtained by members which chose the region r is lower for all strategies not used by players with the first action r.

∀r, ∀a,

x r a > 0 =⇒ F r a (x) = Fr = constant, x r a = 0 ⇒ F r a (x) ≤ F r ′ a ′ , ∀ a ′ , r ′ | x r ′ a ′ > 0
For the existence of such equilibria, we can transform to existence of solution the variational inequality (4.1) which is guaranteed if the function F r a (.) is continuous. The relation between the two definitions is given by the following equivalences (see also the interaction between several populations model in ( 168)):

x is GNE ⇐⇒ x ∈ arg max mut ∑ r ∑ a mut r a F r a (x) (4.2) ⇐⇒ [x r a > 0 =⇒ F r a (x) ≥ F r ′ a ′ (x), ∀r ′ , a ′ ] (4.3)

Global evolutionarily stable state (ESS)

We use here the notion of ESS defined by Taylor in [START_REF] Taylor | Evolutionary stable strategies and game dynamics[END_REF] in a multi-subpopulation game models but in our case the fitness of the region r can be independent of the state of the other regions, and users migrate to the regions with higher fitnesses. We say that x = (x r a ) r,a is a global evolutionarily stable state if for each deviation multi-strategy called "mutations" mut = (mut r a ) r,a = x, there exists some ǫ mut > 0 such that ∀ǫ ∈ (0, ǫ mut ),

Choice Constrained Equilibrium

A strategy x is a choice constrained equilibrium (CCE) if for all (r, a) such that x r a > 0 one has

F r a (x) = max r, b∈A r F r b (x). (4.5)
Note that a global ESS is a neutrally ESS which is a GNE. But a GNE can not be a global ESS. A CCE is a collection of constrained local Nash equilibria.

General Game Dynamics with Migration

We study a game-theoretic dynamics of a large population of users with migration. Users move to a region in which the average payoff of their power control strategy is higher. If user from region r chooses a base station from region r we will say that the user migrate to region r. Migration between regions could help the population to evolve towards a stationary point which is related to a Wardrop equilibrium (233; 179) or an evolutionary stable state.

Strategies with migration as replicators

We introduce here the replicator dynamics with migration which describes the evolution in the population of the various strategies. In the replicator dynamics, the share of a strategy in the population grows at a rate equal to the difference between the payoff of that strategy and the average payoff of the population under the migration constraints. More precisely, let (x t ) be the state of the population at time t. Thus we have ∑ r ∑ a x r a,t = 1 and x r a,t ≥ 0 where x r a,t represents the fraction of players playing a strategy (r, a) in period t.

We describe by approximating from stochastic influence [START_REF] Tanabe | The propagation of chaos for interacting individuals in a large population[END_REF] on the change in frequency of actions, the replication dynamics. We will describe more general class of evolutionary game dynamics in next subsection.

Suppose that in every period ∆t, each player learns with probability α∆t > 0 the expected payoff to the other opponent players and changes to the other's strategy if he perceives that the other's payoff is higher. However the information about the difference in the expected payoffs of the strategies is imperfect, so the larger the difference in the payoffs, the more likely the player is to perceive it, and change. Specially, we assume that the probability that a player using (r, a) will shift to (r, b) in some neighboring set N (r,a) is given by

x r a,t+∆t = µ r [F r a,t (x t ) -F r b,t (x t )] if F r a,t (x t ) > F r b,t (x t ) 0 if F r a,t (x t ) ≤ F r b,t (x t )
where µ is sufficiently small so that x r a,t ≤ 1 holds ∀ (r, b), (r, a). The expected fraction Ex r a,t+∆t of the population using (r, a) in period t + ∆t is given by

x r a,t -α∆tx r a,t ∑ (r,b)∈I x r b,t µ r [F r b,t (x t ) -F r a,t (x t )] + ∑ (r,b)∈N (r,a) \I α∆tx r a,t x r b,t µ r [F r a,t (x t ) -F r b,t (x t )] = x r a,t + αµ r ∆tx r a,t   F r a,t (x t ) - ∑ (r,b)∈N (r,a) x r b,t F r b,t (x t )   (4.6)
where I = (r, b) ∈ N (r,a) , F r b,t (x t ) > F r a,t (x t )

For large population, we can replace Ex r a,t+∆t by x r a,t+∆t . Taking the limit of x r a,t+∆tx r a,t ∆t when ∆t goes to zero, we then obtain Continuous time:

d dt x r a,t = αµ r G t (x t )
where

G t = x r a,t   F r a,t (x t ) - ∑ (r,b)∈N (r,a) x r b,t F r b,t (x t )   .
The constant µ r α changes the rate of adjustment to stationary point. In the classical replicator these parameters are fixed to one. The two parameters α and µ give us a framework for controlling game dynamics (changing or upgrading policy) through the choice of a gain parameter governing the replicator dynamics. The discrete time version of the dynamics is then given by Discrete time: x r a,t+∆t = x r a,t + αµ r G t (x t )∆t

Decomposable dynamics for hybrid power control games

We now describe more general decomposable dynamics with migration respectively to apply in hybrid power control games.

We assume that during any time-step ∆t, each individual among a fraction κ r ∆t of the region r takes part in some local interaction and receives payoffs. We allow a fraction of users to migrate to a region in which their strategies have higher payoffs. The flow in the region r is specified in terms of some functions ρ r ā,a which determines the rates at which a player who is considering a change in strategies opts to switch to his various alternatives in the region r, and some function η ρ r a, ā. We denote by M r a the function representing increase in the number of users inside of region r due to higher fitness. M r a is the difference between the intra-inflow and the intra-outflow.

M r a (x r ) := k r ∑ ā∈A r x r āρ r ā,a -x r a ∑ ā∈A r ρ r a, ā (4.7)
The inter-inflow of the region r is k r γ r ∑ where γ r is a parameter which represents the migration rate of region r, and k r is a growth parameter. By φ r a we denote a function which describes changes of the numbers of users playing the pure secondary strategy a in the region r due to migration from other region to the region r. By combining the two equations (4.7), (4.8) we obtain the continuous time mean dynamics with migration or mean dynamics with multicomponent strategies.

d dt x r a (t) := V r,a F (x) = M r a (x r (t)) + φ r a (x(t)) (4.9)
We denote by β the following revision protocol

β (r,b) (r,a) = k r ρ r ba if r = r k r γ r η (r,b) (r,a) if r = r

Some example of dynamics with multicomponent strategies

Here we give some class of dynamics with migration. We denote by [d] + the maximum between d and 0.

• Replicator dynamics

ρ r ā,a = x r a [F r a -F r ā ] + , η (r,b) (r,a) = x r a [F r a -F r b ] + • Smith dynamics ρ r ā,a = [F r a -F r ā ] + , η (r,b) (r,a) = [F r a -F r b ] + • BNN dynamics ρ r ā,a = [F r a -∑ ā F r ā x r ā] + , η (r,b) (r,a) = [F r a -∑ (r,b) F r b x r b ] +
• Combined Smith and BNN dynamics (in this order)

ρ r ā,a = [F r a -F r ā ] + , η (r,b) (r,a) = [F r a -∑ (r,b) F r b x r b ] + • Combined BNN and Smith dynamics ρ r ā,a = [F r a -∑ ā F r ā x r ā] + , η (r,b) (r,a) = [F r a -F r b ] +
Note that the order of combination of the revision protocols is non-commutative. We next define number of properties that classify evolutionary game dynamics with migration constraints.

Constrained Nash Stationarity (CNS) property

All rest points of the mean dynamic (4.9) are precisely the CCEs of the game being played i.e (M + φ)(x) = 0 if and only if x is CCE.

The replicator dynamics is one of the most studied dynamics in evolutionary game theory to describe evolution of the frequencies in the population but it is known that the replicator dynamics may not lead to equilibria (see (231; 168)). To guarantee constrained Nash stationary (CNS) properties, the mean dynamics must satisfy some conditions.

Positive Correlation (PC) (168)

∑ r,a F r a (x)(M r a (x r ) + φ r a (x r , x -r )) ≥ 0.

Positive correlation guarantees that every Nash equilibrium of the game is a stationary point of the dynamics (4.9) as shown in Result 4.3.3.

To see why this condition is so named, observe that by the condition ∑

r,a M r a (x r ) + φ r a (x r , x -r ) = 0, ∑ r,a F r a (x)(M r a (x r ) + φ r a (x r , x -r )) = ∑ r,a   F r a (x) - 1 l r ∑ (r,b) x r b F r b (x)   M r a (x r ) + φ r a (x r , x -r ) -0 = ∑ r l r Cov(M r (x r ) + φ r (x), F r (x))
where Cov denotes the covariance between strategy growth rates and payoffs in region r. Hence, condition (PC) holds if there is a positive correlation between growth rates and payoffs in each region.

Monotonicity condition: sign preserving (SP) Monotonicity condition on ρ and η defines the following class of dynamics. The couple of functions (ρ, η) preserves the sign if for all region r ρ r ā,a (x)

> 0 if ā, a ∈ A r and F r a (x) > F r ā (x) = 0 otherwise , 4.3. Equilibrium and rest point η (r,b) r,a (x) > 0 if a ∈ A r , b ∈ A r , F r a (x) > F r b (x) = 0 otherwise
The sign preserving property say that the inflow rate from the strategy a to ā inside the region r is positive if and only if the payoff to a exceeds the payoff to ā and the inflow rate from other regions to the region r is positive for a given strategy a if and only the payoff to r exceeds the payoff to r.

Equilibrium and rest point

In [START_REF] Sandholm | Population Games and Evolutionary Dynamics[END_REF]

ρ r ā,a [F r a -F r ā ] ≥ 0 and η (r,b) (r,a) [F r a -F r b ] ≥ 0. ∑ (r,a) ẋr a F r a = ∑ r,a, ā k r x r ā ρ r ā,a [F r a -F r ā ] ≥0 + ∑ (r,a),(r,b) k r γ r x r b η (r,b) (r,a) [F r a -F r b ] ≥0 (4.10)
Theorem 4.3.3. If V F is positively correlated then x is CCE implies that x is a stationary point of mean dynamics (4.9).

Proof. x is a CCE is equivalent to ∀z, such that ∑ (r,a)

z r a = 0, one has ∑ (r,a)

F r a (x)z r a ≤ 0. Now fix a region r, and define the vector z as follows

z r b = V r,b F (x) if r = r, b ∈ A r 0 if b / ∈ A r or r = r Then ∑ r,a z r a = ∑ (r,a) (r,b) x r b β (r,b) (r,a) -∑ (r,a) (r,b) x r a β (r,a) (r,b) = 0 (4.11) So, ∑ (r,a) V r,a F (x)F r a (x) = ∑ (r,a)
z r a F r a (x) ≤ 0 By (PC), this inequality implies that V r,a F (x) = 0.

Theorem 4.3.4 (Characterization of stationary points). Suppose that the functions ρ and η satisfy (SP). Let x be a stationary point of the mean dynamics (4.9). Then x is a CCE.

x r a > 0 ⇒ F r a (x) = max r, b∈A r F r b (x) (4.12)
Proof. We show that every stationary point of the mean dynamics is a CCE.

• x is a CCE if and only for all (r, a), x r a = 0 or there is no outflow from strategy (r, a) :

∑ (r,b) β (r,a) (r,b) = 0 • Suppose that V F (x) = 0. If (

Class of Games with Multicomponents Strategies

Stable Population Games

We say that F is a stable game if for all x, y ∈ X , Proof. Suppose that F satisfies

x, y ∈ X , ∑ r,a (x r a -y r a )(F r a (x) -F r a (y)) ≤ 0
and let x be a GNE. It is easy to see that every neutrally ESS is a GNE. Now we show that x is a neutrally ESS. So fix, an arbitrary vector y. Since F is stable and x is GNE, one has the system

     ∑ r,a (y r a -x r a )(F r a (y) -F r a (x)) ≤ 0 ∑ r,a (y r a -x r a )F r a (x) ≤ 0
Adding the two inequalities of the last system, we obtain that ∑ r,a (y r ax r a )F r a (y) ≤ 0.

Class of Games with Multicomponents Strategies

Taking y = (1ǫ)x + ǫ mut for arbitrary mut = x, we conclude that x is a neutrally ESS. To prove the convexity, we rewrite the GNE set as an intersection of convex sets {GNE} = y A y

where A y = {x ∈ X , ∑ r,a (y r ax r a )F r a (y) ≤ 0}.

Proposition 4.4.1.2. Suppose that the revision protocol has the form

β (r,b) (r,a) (x) = ξ r a (F r a (x) -F r b (x)) if b ∈ A r , a ∈ A r 0 otherwise
for some functions ξ r a : R → R + then the set of CCE is globally asymptotically stable in stable games.

Proof. Let the function B : X → R + be defined by

B(x) = ∑ (r,a), a∈A r (r,b), b∈A r x r a -F r a (x)+F r b (x) 0 ξ r b (θ) dθ.
The function B has the following properties: The set {x, B(x) = 0} is exactly the set of CCE.

(b) ∂ ∂x r ′ a ′ B = ∑ (r,b) -F r ′ a ′ +F r b 0 ξ r b (θ) dθ + ∑ (r,a) ẋr a ∂F r a ∂x r ′ a ′ (c) d dt B(x(t)) = ∑ (r ′ ,a ′ ) ∂B ∂x r ′ a ′ ẋr ′ a ′ = ẋ, DF(x) ẋ + ∑ (r ′ ,a ′ ) (r,a) β (r,a) (r ′ ,a ′ ) ∑ (r,b) -F r ′ a ′ +F r b 0 ξ r b (θ) dθ - -F r a +F r b 0 ξ r b (θ) dθ
Since F is a stable population game, one has ẋ, DF(x) ẋ ≤ 0. In other hand,

β (r,a) (r ′ ,a ′ ) > 0 ⇔ F r ′ a ′ > F r a .
Then, for any action (r, b), one has

F r b -F r ′ a ′ ≤ F r b -F r a .
Hence, the term (r ′ ,a ′ ) = 0, ∀(r, a), (r ′ , a ′ ) i.e V F (x) = 0. Thus, the function satisfies Lyapunov stability criterion for the set of stationary point of V F .

-F r ′ a ′ +F r b 0 ξ r b (θ) dθ - -F r a +F

Potential Population Games

We say that F is a full potential game if it exists a C 1 function f : X → R such that

∂ ∂x r a f = F r a (x).
Two strategies: if the game has only two strategies in each r, all strategy distributions lie on a line. Given any continuous payoff functions

F r 1 , F r 2 : X r = {(x r 1 , x r 2 ), x r i ≥ 0, x r 1 + x r 2 = l r } → R,
a potential function is given by

f (x) = N ∑ r=1 x r 1 0 (F r 1 (t, l r -t) -F r 2 (t, l r -t)) dt Proposition 4.4.2

.1. Global convergence holds in potential games under (PC).

Proof. Suppose that f is a potential function of the game and the dynamic ẋ = V F (x) is (PC).

Then f is a Lyapunov function of this dynamic.

d dt f (x) = ∑ r ∑ a∈A r V r,a F (x)F r a (x).
Moreover f satisfies d dt f (x) ≥ 0 with equally if and only if x is a CCE. It is known that if a dynamic admits a such Lyapunov function, all solution trajectories of the dynamic converge to an equilibrium. Combining with results 4.3.4 and (PC) we obtain the announced result.

Migration with constraints

In this subsection, we assume that when updating a strategy one can change only one component of the strategies at the same time. Players with the action (r, a) can use all actions in N (r,a) = ∪ r =r,( {(r, a), a ∈ A r } ∪ ∪ ā∈A r {(r, ā)}. Taking these considerations, a strategy x is a CCE if for all (r, a) such that x r a > 0 one has for all (r, a) such that y r a > 0 then y is also a CCE in the game without migration constraints. This kind of phenomena occurs in the well known Braess Paradox problem [START_REF] Braess | Uber ein Paradoxon aus der Verkehrsplanung[END_REF]

F r a (x) = max (r,b)∈N (r,a) F r b (x).

: increasing the number of regions (increasing the area of coverage, adding new resources such as deployment of new base stations) does not necessary lead to higher equilibria payoffs. If a CCE x of the unconstrained migration game satisfies max

(r,b)∈C (r,a) F r b (x) ≤ F r a (x)
then x is a Braess steady state. Given such a steady state x, if there are mutations, mutations cannot spread due to the migration constraints. Hence, the mutants' strategy can be invaded strictly at the steady state x.

Let x be a CCE of the game without migration constraints. If x satisfies F r a

(x) = max (r,b)∈N (r,a) F r b (x)
then x is also a CCE in the game with migration constraints.

Global Optimization

3. Let Opt 1 (resp. Opt 2 ) the set of CCEs with the maximum payoff in the unconstrained migration game (constrained migration game.) Then,

∀ x ∈ Opt 1 , y ∈ Opt 2 , x r a > 0 =⇒ F r a (x) ≥ F r a (y).
4. Suppose that the unconstrained migration game has a unique CCE. Let x (resp. y) be a CCE for the game without migration constraints (resp. the constrained migration game.) Then for every action (r, a) such that x r a > 0, one has F r a (x) ≥ F r a (y).

Inverse problem: reachable regions of a power level

In this subsection, we model the power levels as first strategies and regions as secondary strategies. Let P a finite set of power levels, A p = {r 1 p , . . . , r n p p } the pure secondary strategies set of the power level p ∈ P and, S p = {(r, a), r ∈ A p } the pure strategies of a p-player. Then S = ∪ p∈P S p . Given an energy (power level), the player will migrate from its location to a reachable region of this power level.

A strategy x is a CCE of this inverse problem if for all (r, a) such that x r a > 0 one has

F r a (x) = max b∈P, r∈A b F r b (x), (4.14)
and mean dynamic becomes

ẋr a = ∑ b∈P ∑ r∈A b x r b β (r,b) (r,a) -x r a ∑ b∈P ∑ r∈A b β (r,a) (r,b)

Global Optimization

In this section, we focus on global optimization of the payoff of all the population. The total payoff function is defined as

F total : X -→ R x → ∑ r ∑ a∈A r
x r a F r a (x r , x -r ).

The optimization problem is given by max

x ∑ r ∑ a∈A r x r a F r a (x) subject to ∀ r, ∑ r ∑ a∈A r x r a = l r ∀(r, a) ∈ S, x r a ≥ 0
The Lagrangian for this maximization problem is

L(x, λ, µ) = ∑ r,a x r a F r a (x) -∑ r λ r ∑ a x r a -l r + ∑ r,a
x r a µ r a Thus, the Karush-Kuhn-Tucker(KKT) first order necessary conditions for maximization are ∀r, ∀a ∈ A r , ∂ ∂x r a F total (x) = λ rµ r a , µ r a ≥ 0, µ r a x r a = 0 (4.15)

Hybrid power control in OFDMA-based IEEE802.16 network

OFDMA (Orthogonal Frequency Division Multiple Access) is recognized as one of the most promising multiple access technique in wireless communication system. This technique is used to improve spectral efficiency and becomes an attractive multiple access technique for 4 th generation mobile communication system as WiMAX.

In OFDMA systems, each user occupies a subset of subcarriers, and each carrier is assigned exclusively to only one user at any time. This technique has the advantage of eliminating intracell interference (interference between subcarriers is negligible). Hence the transmission is affected by intercell interference since users in adjacent sectors may have also been assigned to the same carrier. If those users in the adjacent sectors transmitted with high power the intercell interference may severely limit the SINR achieved by the user. Some form of coordination between the different cells occupying the spectral resource are studied in (134; 123). The optimal resource allocation requires complete information about the network in order to decide which users in which cells should transmit simultaneously with a given power. All of these results however, rely on some form of centralized control to obtain gains at various layers of the communication stack. In a realistic network as WiMAX, centralized multicell coordination is hard to realize in practice, especially in fast-fading environments.

We consider an OFDMA system where radio resources are allocated to users on their channel measures and traffic requirements. Each carrier within a frame must be assigned to at most one user in the corresponding cell. In this way each carrier assignment can be made independently in each cell. Hence when a user is assigned to carrier, the mobile should determine the power transmission to the Base station. This power should take into account the interference experienced by the transmitted packet.

Power control on the uplink Consider the uplink of a multiple multicell system, employing the same spectral resource in each cell. Power control is used in an effort to preserve power and to limit interference and fading effects. For users located in a given cell, co-channel interference may therefore come from only few cells (neighbors). Since the intra-cell interference is negligible, we focus on the users which use a specific carrier.

Consider N cells, and a large number of the population of mobiles randomly distributed over each channel and each cell. Since in OFDMA systems, each carrier is assigned exclusively to only one mobile at any time, we assume that the interactions between mobiles are manifested through many local interactions between K r + 1 mobiles where K r is a random variable which represents the number of opponent mobiles in the set of neighbors of a cell r. At each slot, interaction occurs only between the mobiles which have been assigned to the same carrier. We assume that in this model that users can choose many base stations available in his neighborhood and a finite power levels as a secondary pure strategies .

Let g r r denote the average channel gain from user in cell r to cell r. Hence, if a user in cell r transmits with power p r , the received signal strength at cell r is p r g rr , while the interference it occurs on cell r is p r g r r. Hence, the interference experienced by cell r is given by

SI NR r K r (p) = g rr p r σ 2 r + ∑ r∈Nr\{r} |Nr|=K r g r r p r ,
where σ 2 r is the power of the thermal noise experienced at cell r, N r be the set of neighbors of a user in cell r. The rate achieved by user in cell r is given by R r (p) = E K r log(1 + SI NR r K r (p)), where p denotes the power level vector of mobiles choice which are assigned to a specific carrier. We assume that the user's utility in cell r is given by u r (p) = R r (p)c r (p). The above utility 4.7. A hybrid evolutionary game in multicell CDMA system represents the weighted difference between the throughput that can be achieved as expressed by Shannon's capacity and the power consumption cost. We show existence and uniqueness of the equilibrium point under some conditions on the cost functions. The results are given in next section in hybrid model of multicell CDMA system but the technique used there are valid for the above game.

A hybrid evolutionary game in multicell CDMA system

We consider a large population of mobile terminals and many distributed base stations in a multicell CDMA wireless network model. The system consists of N cells which are called "regions". The number of users which transmit to a base station is a random variable. Each mobile connects to a base station which it chooses from of the set of base stations {1, 2, . . . , N} with an uplink power level from the set A. The action space of a mobile is given by {1, 2, . . . , N} × A. The level of services a mobile receives is described in terms of signal-to-interference ratio (SIR). The SIR obtained by mobile m at a base station located in region r is given by

SIR r,k m (a m , a -m ) = La m h r,m σ 2 r + ∑ 0≤l≤k,l =m a l h r,l
where a -m denote the vector (a 1 , . . . , a m-1 , a m+1 , . . . , a k ) the power levels of the others mobiles, σ 2 r is a constant which represents the variance of a noise power due to the factors other than the transmissions of other mobiles at the base station r. The term a m h r,m represents the power level received at base station from mobile using the power level a m and L = W/B > 1 is the spreading gain of the CDMA system, W is the chip rate and B is the data rate of the users. The payoff that mobile using the power level a m can send to the base station in region r at a given slot is given by

J r,k (a m , a -m ) = C r log 1 + SIR r,k m (a m , a -m ) -c r (a m )
where C r is the channel bandwidth of r and c r (.) is the cost function of region r. We assume that c r is an increasing cost function. The expression of J r,k + c r is known as Shannon capacity. The expected fitness of mobile m at the base station r when the state of the population at region r is x r is given by

F r a m (x r ) = ∑ k P(K r = k) ∑ a -m ∏ l =m
x r a l F r,k (a m , a -m ).

The total payoff of the population is

thp(x) := N ∑ r=1 ∑ a m ∈S
x r a m F r a m (x r ) Proposition 4.7.0.1 (two power levels).

Let A = {0, P} and h r,m = h r , r = 1, 2. Let W r j = C r log(1 + h r P σ 2 r + jh r P
), j = 0, . . . , k., and c r (P) positive reals satisfying c r (P)

∈ ( ∑ k p k W r k , a r 0 )
where

p k = P(K = k). Then, the polynomial Q r (ξ) := -c r (P) + ∑ k p k k ∑ j=0 W r j ( k j )ξ j (1 -ξ) k-j
has unique root x r * on the interval (0, 1). Proposition 4.7.0.2. The unique solution (x r * ) r=1,...,N given by the result 7.3.10 is a the unique interior Wardrop equilibrium.

Proof. The fitness of the strategy P in the cell r is given by F r P (x r P , 1x r P ) = Q r (x r P ), the fitness of the strategy 0 is zero. Then the expected fitness of the cell r is x r P Q r (x r P ). Hence, every interior Wardrop equilibrium (x r * ) r=1,...,N must satisfy Q r (x r P ) = 0, ∀r.

From result 7.3.10, these equations have a unique solution on (0, 1) under the conditions: Proof. For x = x * and every cell r, one has (x r Pmut r P )F r P (ǫ mut r + (1ǫ)x r ) > 0 for small ǫ. Hence, for all r,

∀r, Q r (0) = -c r (P) + W r 0 > 0, Q r (1) = -c r (P) + ∑ k p k W r k < 0
∑ a∈A x r a F r a (ǫ mut r + (1 -ǫ)x r ) > ∑ a∈A mut r a F r a (ǫ mut r + (1 -ǫ)x r ). We conclude that ∑ r ∑ a∈A x r a F r a (ǫ mut r + (1 -ǫ)x r ) > ∑ r ∑ a∈A mut r a F r a (ǫ mut r + (1 -ǫ)x r ).
This completes the proof.

Numerical investigation: Convergence to the equilibrium

Our numerical experiment studies the behavior of the replicator dynamics and smith dynamics with migration. We consider the following fixed parameters : we took N = 2 cells, K ∈ {0, 1, . . . , 500}, A = {0, P}, h r,m = h r , r = 1, 2, σ 2 r = r × 10 -3 , p k = 2 × 10 -3 , k = 1, . . . , 500, γ 1 = 0.5, γ 2 = 1, c r (P) = r, P = 1. An interior Wardrop equilibrium exists for these parameters, for which the fractions of the population using P is given by the result 4.7.0.2. The continuous replicator dynamics becomes

d dt x r P (t) = k r x r P (1 -x r P )Q r (x r P ) + γ r k r x r P [Q r (x r P ) -∑ r Q r(x r P )]
The continuous Smith dynamics becomes

d dt x r P (t) = k r ((1 -x r P )[Q r (x r P )] + -x r P [-Q r (x r P )] + ) + γ r k r ∑ r x r P [Q r (x r P ) -Q r (x r P )] + -x r P ∑ r [Q r (x r P ) -Q r (x r P )] +
The resulting trajectories of the population ratio each cell using the power level P, as a function of time, is given in Fig. 4.2,4.3 and 4.4. The throughput of transmitters, the total throughput and the total payoff of the whole population are represented respectively in Fig. 4.5, 4.6 and 4.7. In Fig. 4.5, we can see that the throughput of players which use the strategy P decreases when the number of transmitters increases. This because the SINR decreases when the number of transmitters increases. The throughput increases with the power level P. When the number of transmitters is high, a player will increase its SINR, but will decrease the SINRs of the others players, and the total cost will decrease. Thus, the total payoff becomes negative as shown in Fig. 4.7.

Notes

In the second part this manuscript we develop a new class of dynamic evolutionary games called stochastic population games which include Markov decision evolutionary games. This class is also related to the so-called anonymous sequential games. We examine both homogenous and heterogenous case and apply them to energy management and resource competition in wireless networks. The chapter is based to our publications in (217; 10; 212; 25).

Illustrating Examples

Battery state-deependent power management

Consider a large number of mobiles terminals controlling their transmission power and a distributed base stations. Each mobile has an amount of energy E when its battery is new (typically it is the case if the battery is new or if the battery is completely recharged). Each mobile implements a power control policy where the transmission power is allowed to depend on the energy level (state) of its battery. The available action (reachable base stations and powers) depends on the state of the battery. Given the remaining energy of its battery, the mobile have to choose the optimal power level. One of the important element for each mobile is its instantaneous throughput which can be characterized as a function of the signal to interference plus noise ratio (SINR) at the base station where he transmits. The battery is replaced only when it is completely empty. The cost of new battery cost is C. The new battery has the same energy of E. The mobile have to control both the power consumption as well as the time at which the batteries are changed. At each slot, each mobile is faced to a random number [START_REF] Tembine | Evolutionary games with random number of interacting players applied to access control[END_REF] of interacting players which transmit at the same base station. Each battery life-time game corresponds to a stochastic population game with finite horizon (absorbing state of battery when the energy is very small). The aim here is to find jointly the power levels and the base stations such that all users achieve as high payoff as possible, minimum guarantee (e.g. QoS requirement thresholds) but also to control the battery-state.

When batteries are recharged dynamically with different types of alternative energy such as renewable energies (solar, wind etc). The battery transition state becomes irreducible Markov decision process under each policy depending on an exogenous parameter which characterizes the good weather (good weather will correspond to the sun for the solar-power systems and to the wind for wind-powered systems). In this case, the interaction becomes a stochastic population game with infinite horizon and we shall consider time-average reward (discounted or not).

Energy management in hybrid Aloha-like networks

Consider a distributed Aloha network with large number of mobile terminals. Each mobile can choose both the channels and powers (this is in contrast to standard Aloha model in which users are associated to the closest receivers). Each terminal is faced to a random number of interacting players which transmit at the channel. A terminal attempts transmissions during a finite horizon of times depending on the state of its battery energy. At each slot, each terminal have to take a decision on the transmission power based on the battery state. At each state of the battery, there are a finite power levels. At the lowest state of battery no power is available and the mobile have to replaced the battery by a new or to recharge its battery. A transmission is successful if no other user transmit during the slot or the mobile transmits with a power which is bigger than the power of all others transmitting mobiles at the same receiver. The pairwise interactions case of this problem has been studied by Altman and Hayel in (20) as a stochastic evolutionary game. They have considered three states: Full, Almost Empty and Empty, and simultaneous interactions with more two users are neglected 1 . Their model can be extend to more than two opponent interactions and also to finitely many states as shown in next Section. We can also extend to the case where each terminal is faced to a random number (219; 213; 11) of interacting terminals which transmit at the same range and each terminal have to control an arbitrary transition state of its energy.

Markov decision process

A player arrives at some random time t 0 . It has a clock that dictates the times at which interactions with other players occur. It is involved in interactions that occur according to a Poisson process with rate λ. After a random number of time periods, the player leaves the system and is replaced by another one. This will be made precise below. During the player's life time, each time the timer clicks, the player interacts with another randomly selected player.

We associate with each player a Markov Decision Process (MDP) embedded at the instants of the clicks.

The parameters of the MDP are given by the tuple {S, A, Q} where • S is the set of possible individual states of the player • A is the set of available actions. For each state s, a subset A s of actions is available.

• Q is the set of transition probabilities; for each s, s ′ ∈ S and a ∈ A s , Q s ′ (s, a) is the probability to move from state s to state s ′ taking action a. ∑ s ′ ∈S Q s ′ (s, a) is allowed to be smaller than 1.

Define further

• The set of policies is U . A general policy u is a sequence u = (u 1 , u 2 , . . .) where u i is a distribution over action space A at time i. The dependence on time is a local one: it concerns only the individual's clock; a player is not assumed to use policies that make use of some global clocks. There can only be also one global clock for all individuals. A policy is an individual decision which defines the sequences of action which will be taken by the individual at each individual's clock.

• The subset of mixed (respectively pure or deterministic) policies is U M (respectively U D ). We define also the set of stationary policies U S where such policy does not depend on time.

Frequency state-action Often we encounter the notion of individual states in evolutionary games; but usually the population size at a particular state is fixed. In our case the choices of actions of an individual determine the fraction of time it would spend at each state. Hence the fraction of the whole population that will be at a given state may depend on the distribution of strategies in the population. In order to model this dependence we first need to describe the expected amount of time f η,u (s) that an individual spends at a given state s when it follows a strategy u and its initial state at time 1 is distributed according to a probability η over S. More generally, we define f η,u (s, a) the expected number of time units during which it is at state s and it chooses action a. f u := { f η,u (s, a)} is called the occupation measure corresponding to a policy u or frequency state-action.

More precisely, define p t (η, u; s, a) = P η,u (X t = s, A s = a) the probability for a user to be in state s, at time t, using action a under policy u when the initial state has a probability distribution η. Further define p t (η, u; s) = ∑ a p t (η, u; s, a).

Define f t η,u (s, a) = 1 t t ∑ r=1 p r (η, u; s, a). Denote f t η (u) := { f t η,u (s, a)}.
Define by Φ u η to be the set of all accumulation points of f t η (u) as t → ∞. Whenever Φ u η contains a single element, we shall denote it by f η (u).

Single population stochastic evolutionary games

Aim: To model and characterize evolutionary games where individuals have states that are described by controlled Markov chains. The action of an individual in a local interaction with another randomly selected individual determines not only the instantaneous fitness but also its probability to move to another state. The goal of a player is to maximize its time average fitness.

Mathematical methods:

The main mathematical tool is occupation measures (expected frequencies of states and actions). This tool is a central one in the theory of Markov Decision Processes. We make use of the geometric properties of the set of achievable occupation measures.

Key assumption: Under any pure stationary policy of an individual, its Markov chain has a single ergodic class of states.

We define and characterize a new concept of Evolutionarily Stable Strategies (OMESS), based on the concept of Occupation Measures. We relate this set to the concept of ESSet [START_REF] Thomas | On Evoloutionary Stable Sets[END_REF]. We present a way to transform the new type of evolutionary games into standard ones. We apply this novel framework to energy control in wireless networks.

A new rich class of evolutionary games is defined along with the corresponding new definitions of equilibrium. This combination of evolutionary games together with Markov Decision Processes introduces new dynamical features and modeling capabilities.

Keywords: evolutionary games, occupation measure evolutionarily stable strategy, Markov decision process, energy control in wireless networks.

Pairwise interaction we consider a large population of individuals. As in standard evolutionary games, there are many pairwise interactions between randomly selected pairs. Let r(s, a, s ′ , b) be the immediate reward that a player receives when it is at state s and it uses action a while interacting with a player who is in state s ′ that uses action b.

Denote by α(u) = {α(u; s, a)} the system state: α(u; s, a) is the fraction of the population at individual state s and that use action a when all the population uses strategy u. We shall add the index t to indicate a possible dependence on some time.

Consider an arbitrary tagged player and let S t and A t be its state and action at time t (as measured on its individual clock). Then his expected immediate reward at that time is given by

R t = ∑ s ′ ,a ′ α t (u; s ′ , a ′ )r(S t , A t , s ′ , a ′ ) := r(S t , A t , α t (u)).
Assume now that a player arrives at the system at time 1. The global expected fitness when using a policy v is then

F η (v, u) = lim inf t→∞ 1 t t ∑ m=1 E η,v [R m ].
When η is concentrated on state s we write with some abuse of notation F s (v, u) = F η (v, u). We shall often omit the index η (in case it is taken to be fixed).

Unless stated differently, we shall make throughout the following assumption.

Introduce the following assumptions.

A2(U):

When the whole population uses a policy u ∈ U, then at any time t which is either fixed or is an individual time of an arbitrary player, α t (u) is independent of t and is given by

α t (u; s, a) = f η,u (s, a) = π(s)u(a|s)
for all s, a where f η,u (s, a) is the single limit of f t η,u (s, a) as t → ∞ and π is the stationary distribution of the chain. A2: Assumption A2(U) holds for U = U s and for U = U M .

The validity of the Assumption depends on the way the infinite population model is obtained by scaling a large finite population model. This aspect is beyond the scope of this paper. Denote the set of all policies for which Φ u η is a singleton by U * . For u ∈ U * , the following holds:

F(v, u) = inf z∈Φ v η ∑ s,a z(s, a) ∑ s ′ ,a ′ f η,u (s ′ , a ′ )r(s, a, s ′ , a ′ ). (5.1)
The set of occupation measures will be shown to be a polytope whose extreme points correspond to strategies in U D . This will allow us to transform the stochastic population game or Markov decision evolutionary game to a standard evolutionary game.

Note that for any u ∈ U M , and for any strategies v and w,

Φ v η ⊂ Φ w η implies F(v, u) ≥ F(w, u). (5.2)
This, together with the fact that for any policy u and z ∈ Φ u η there exist a stationary policy v ∈ U * satisfying f v η = z, will motivate us to limit ourselves to policies in U * . When both u and v are in U * , the global expected fitness simplifies to

F(v, u) = ∞ ∑ t=1 E η,v R t = ∑ s,a f η,v (s, a) ∑ s ′ ,a ′ f η,u (s ′ , a ′ )r(s, a, s ′ , a ′ ).
(5.3) Assumption A1 would not hold if the policy of a player could depend on the absolute time or on the behavior (i.e. the actions) of other players. For example, in the standard replicator dynamics, the policy of a player adapts to the instantaneous fitness which depends also on the actions of the other players in the population. Thus A1 does not hold there. On the other hand, since players of a given class are undistinguishable, and since the lifetime distribution of a player depends only on his local time, we may expect Assumption A1 to hold. Checking A1 is beyond the scope of the paper. Definition 5.3.0.1. We shall say that two strategies u and u ′ are equivalent if the corresponding occupation measures are equal. We shall write u = e u ′ . The set of occupation measures equivalent to u is denoted by e(u) := {v|v = e u}.

Proposition 5.3.1.1. If u is an OMESS, then e(u) is an ESSet.

Proof. Let u be an OMESS. We take a measure v ∈ e(u). By definition of equivalent class we have one of the following condition holds:

• F(u, u) > F(v, u), • F(u, u) = F(v, u) and F(u, v) > F(v, v).
Thus each w ∈ e(u) is a Nash equilibrium. The second condition implies that for every w ∈ e(u) and v ∈ e(u) such that F(w, w) = F(v, w), we have F(w, v) > F (v, v). This implies by definition that e(u) is an ESSet.

In the following, we show that an ESSet is a weaker notion than OMESS: a problem with no OMESS may still have a non empty ESSet.

Consider a single state s and two actions h or l. Assume that the reward does not depend on the action. two pure stationary policies are u and v where u consists on playing always h and the policy v is to play always l. Then,

• the ESSet of the Markov game is all the feasible policies,

• u and v are not in the same equivalence class,

• F(u, w) = F(w, w), ∀w ∈ e(v) c = ∅. v is not an OMESS.
• the game has no OMESS

Computing the OMESS

Define the set of occupation measures achieved by all (individual) policies in some subset

U ′ ⊂ U as L η (U ′ ) = u∈U ′ f η,u (s, a).
It will turn out that the expected fitness of an individual (defined in next subsection) will depend on the strategy u of that individual only through f η,u . We are therefore interested in the following characteristic of L η (U) (see (119; 26; 178)): Lemma 5.3.1.2. L η (U) equals to the set Q η defined as the set of α = {α(s, a)} satisfying

∑ s ′ ∈S ∑ a∈A s ′ α(s ′ , a)[δ s ′ (s) -Q s ′ (s, a)] = η(s), ∀s, α(s, a) ≥ 0, ∀s, a. (5.6)
where

δ s ′ (s) is the Dirac distribution in state s ′ . (ii) We have: L η (U) = L η (U S ) = coL η (U D ) where coL η (U D ) is the convex hull of L η (U D ). (iii) For any α ∈ L η (U), define the individual stationary policy u ∈ U S by u s (a) =        α(s, a) ∑ a∈A s α(s, a) if ∑ a∈A s α(s, a) > 0 arbitrary number in [0, 1] if ∑ a∈A s α(s, a) = 0 Then f η,u = α.

Transforming stochastic population game the into a standard population game

Consider the following standard evolutionary game EG:

• the finite set of actions of a player is U D ,

• the fitness of a player that uses v ∈ U D when the other use a policy u ∈ U S is given by (5.8).

• Enumerate the strategies in U D such that U D = (u 1 , ..., u m ) where

m = ∏ s∈S |A s |,
• Define γ = (γ 1 , ..., γ m ) where γ i is the fraction of the population that uses u i . γ can be interpreted as a mixed strategy which we denote by γ. Remarks 5.3.1.3. Here the convex combination ǫ γ + (1ǫ) γ ′ of the two mixed strategies γ and γ ′ is simply the mixed strategy whose ith component is given by

ǫγ i + (1 -ǫ)γ ′ i , i = 1, ..., m. Combining Lemma 5.

. (i) γ is an equilibrium for the game EG if and only if it is an OMESS for the original stochastic population game. (ii) γ is an ESS for the game EG if and only if it is a OMESS for the original stochastic population game.

Proof. The statements hold if we allowed for only mixed policies; indeed, they follow from Lemma 5.3.1.2 and eq. (5.7). We have to check that if a mixed policy is an equilibrium or a OMESS when restricting to U M then it is also an equilibrium among all policies. This in turn follows from from Lemma 5.3.1.2 and eq. (5.8).

Non-pairwise interactions

We now assume the number of players in local interaction is more than two and this number is not known. Let r k (s, a, s 2 , a 2 , . . . , s k , a k ) be the immediate reward that a player receives when it is at state s and it uses action a while interacting with k -1 others players who is in state s j that uses action a j , j = 2, . . . k. Then the expected immediate reward at that time is given by

R t = ∑ k≥1 P(K = k) ∑ s 2 ,a 2 ,...s k ,a k k ∏ j=2 α t (u; s j , a j )r k (S t , A t , s 2 , a 2 , . . . s k , a k ) =: r(S t , A t , α t (u)).
Assume now that a player arrives at the system at time 1. The global expected fitness when using a policy v is then

F η (v, u) = lim inf t→∞ 1 t t ∑ m=1 E η,v [R m ].
When η is concentrated on state s we write with some abuse of notation F s (v, u) = F η (v, u). We shall often omit the index η (in case it is taken to be fixed).

For u ∈ U * , the following holds:

F(v, u) = inf z∈Φ v η ∑ s,a z(s, a) ∑ s 2 ,a 2 ,...s k ,a k r(s, a, s 2 , a 2 , . . . s k , a k ) k ∏ j=2 f η,u (s j , a j ).
(5.7)

When both u and v are in U * , the global expected fitness simplifies to

F(v, u) = ∞ ∑ t=1 E η,v R t = ∑ s,a f η,v (s, a) ∑ s 2 ,a 2 ,...s k ,a k r(s, a, s 2 , a 2 , . . . , s k , a k ) k ∏ j=2
f η,u (s j , a j )r(s, a, s 2 , a 2 , . . . s k , a k ).

(5.8)

Stochastic population games : multiclass case

Consider the following model of population game denoted by

Γ = (P, (Y p ) p∈P , (A p (y)) p∈P,y∈Y p, (Q p ) p∈P , (r p ) p∈P ),
where

•
The population is composed of several subpopulations. Each subpopulation contains a large number of players. P denotes the set of subpopulations (we assume that P is finite).

• Each player of each subpopulation p has its own state Y p (finite) and Markov transition structures Q p between the states.

• For every player i from the subpopulation p ∈ P and every state y ∈ Y p of i, A p (y) is the set of actions available. The action space of the subpopulation p is given by ∏ y∈Y p A p (y).

The set of all actions at all states is given by All p where All p = {(y, a), y ∈ Y p , a ∈ A p (y)}.

• We denote by ∆(Y p ) the (|Y p | -1)-dimensional simplex of R |Y p | and by q p : All p → ∆(Y p ) a transition rule between the states. The transition probability distribution between states is defined by Q p y,a,y ′ := q p (y ′ |y, a) = q p (y ′ |y 1 , a 1 , . . . , y t-1 , a t-1 , y, a)

for each y ′ , y ∈ Y p , a ∈ A p (y).

• For every subpopulation p ∈ P,

r p : ∏ p ′ ∏ y∈Y p ′ X p ′ (y) → R ∑ y∈Y p ′ |A p ′ (y)|
is the vector of all instantaneous payoff functions of a player from the class p ′ ,

X p ′ (y) =    (x p ′ (y, b)) b∈A p ′ (y) | x p ′ (y, b) ≥ 0, ∑ y∈Y p ′ ∑ b∈A p ′ (y) x p ′ (y, b) = m p ′   
where m p ′ is the mass associate to the subpopulation p ′ . Given a state y and strategy profile x p , x -p , the payoff obtained by playing the action a ∈ A p (y) is r p y,a (x) • The game is played many times.

Histories and Strategies

Histories A history h t at time t is a collection of states and actions (y 1 , a 1 , x 1 , . . . , y t-1 , a t-1 , x t-1 , y t ). We denote by H p t = (All p × X ) t-1 × Y p the set of histories of a member of the subpopulation p at time t.

At t = 1, H p 1 = Y p . Let H p ∞
be the set of all infinite histories of the subpopulation p endowed with the product σ-field and

H ∞ = ∏ p∈P H p ∞ .

Strategies

• Pure strategy A pure strategy of a player from subpopulation p at time t is a map σ p t : H p t -→ A p (y t ). The collection σ p = (σ p t ) t≥1 of pure strategy at each time constitutes a pure strategy of the subpopulation p. We denote by Σ p the set of all pure strategies of subpopulation p, by Σ = ∏ p Σ p the set of all pure strategy profiles. Note that the number of pure strategies is infinite.

• Stationary strategy: σ is stationary strategy if for each population p and every time t and histories,

h t = (y 1 , a 1 , x 1 , . . . , y t-1 , a t-1 , x t-1 , y t ), h ′ t = (y ′ 1 , a ′ 1 , x ′ 1 , . . . , y ′ t-1 , a ′ t-1 , x t-1 , y ′ t ) such that if y t = y ′ t one has σ t (h t ) = σ t (h ′ t )
i.e a stationary strategy is a history and time independent strategy which depends on the state only. • Behavioral strategy A behavioral strategy at time t is a function that assigns each finite history to a mixed action profile of the current state:

σ p t : H p t -→ ∏ p ∆(A p (y t )), p ∈ P.
• Mixed strategy A mixed strategy profile is a collection of probability distributions on Σ. Using Tychonoff's theorem, the set of all these notions of strategies is compact in the product set histories spaces in the sense of the weak-topology. A general mixed strategy is a probability distribution on the behaviorial strategies set.

For any strategy profile σ = (σ p ) p∈P and every initial state distribution profile µ = (µ p ) p∈P , a probability measure P σ,µ is induced by σ and µ. The stochastic process (y t , a t , x t ) t≥1 is defined on H ∞ in a canonical way, where the random variables y t , a t , x t describe the individual state, the action in this state and the population profile.

Cesaro-limit Fitness

We examine the limit average Cesaro-type payoff

F p µ (σ p , σ -p ) = E σ,µ lim inf T-→+∞ 1 T T ∑ t=1 r p y t ,a t (x t )
where E σ,µ denotes the expectation over the probability measure P σ,µ induced by σ, µ on the set of histories endowed with the product σ-algebra. Given a strategy σ and a initial state y, we define the expected time-average payoff. We denote by Π p the stationary limit average matrix:

Symbol element

Π p (σ p ) = lim t-→+∞ 1 t t ∑ j=1 (Q p ) j (σ p ).
The matrix Π p is well-defined, commutes with Q p and satisfies the projection equation:

Π p × Π p = Π p .
If F p is the vector (F p y (x)) y∈Y p , we have that F p (x) = Π p r p (x) for all stationary strategy profile x. Then, F p = Π p F p . Note that the function x -→ F p (x) is not necessarily continuous because the limit matrix Π p (x) can be discontinuous on x. Definition 5.4.2.1. A strategy σ is an ǫ-equilibrium if for all p,

F p (σ) + ǫ ≥ F p (σ ′p , σ p ), ∀ σ ′p ∈ ∆(Σ p ).
A 0-equilibrium is called equilibrium.

Notes

• When each member of each subpopulation has a single state, we obtain a population game model which each local interaction is repeated game.

• If there exists a subpopulation p * such that |Y p | = 1 for all p = p * . We obtain a stochastic population game with single class of controllers which is the subpopulation p.

Constrained stochastic population games

In addition to the model described in Section 5.4, we assume that players have (possibly coupled) average constraints on their actions in any state. The payoff of the subpopulation p is

E σ,µ lim inf t-→+∞ 1 t t ∑ k=1 r p y k ,a k (x k )
with σ p ∈ ∆(Σ p ) subject to 5.5. Constrained stochastic population games

• Orthogonal constraints:

p ∈ P, P σ p lim sup t-→+∞ 1 t t ∑ k=1 D p (y k , a k ) ≤ β p = 1,
where D p : All p -→ R is an individual cost function (independent of the strategies of the others players), β p ∈ R is a given cost threshold.

• Coupled constraints:

p ∈ P, P σ,µ lim sup t-→+∞ 1 t t ∑ k=1 C p y k ,a k (x k ) ≤ α p = 1
where C p : All p × X -→ R is a cost function which depends on the individual stateaction but also on the population profile i.e the strategies of the others players (in the same class or not).

A strategy σ is a constrained equilibrium if for all p,

F p (σ) ≥ F p (σ ′p , σ p ), ∀ σ ′p ∈ ∆(Σ p )
and σ p ∈ Λ(σ -p ) where Λ(σ -p ) is the set feasible strategies (that satisfy the orthogonal and coupled constraints) given the strategies of the others populations σ -p .

Constrained stochastic population games with unknown horizon

In general, the lifetime of an individual or of the system is not known. We shall integrate this in our interaction model. In this section we develop a general formulation of a local interaction with unknown stopping time. Players does not known the length of the local interaction but have a common probability structure on the stochastic local game. At time t, they assign some probability P(T = t) to the event {T = t} that the local interaction ends in time t.

t ≥ 1, P(T = t) ≥ 0, ∑ t≥1 P(T = t) = 1.
Fix an anonymous member of some subpopulation p, and a sequence of state-actions σ. A player from the class p will receive

F p µ (σ) = E σ,µ lim inf t→+∞ ∑ t k=1 P(T = k)(∑ k j=1 r p y j ,a j (x j )) ∑ t j=1 jP(T = j)
under the constraints: p ∈ P,

P σ lim sup t→+∞ ∑ t k=1 P(T = k)(∑ k j=1 C p y j ,a j (x j )) ∑ t j=1 jP(T = j) ≤ α p = 1, P σ p lim sup t→+∞ ∑ t k=1 P(T = k)(∑ k j=1 D p (y j , a j )) ∑ t j=1 jP(T = j) ≤ β p = 1, Theorem 5.5.1. F p µ (σ) = E σ,µ lim inf t→+∞ F p,t µ (σ) where F p,t µ (σ) := 1 ∑ t j=1 jP(T = j) t ∑ j=1 t ∑ k=j P(T = k) r p y j ,a j (x j )
Proof. We apply Fubini's theorem on finite summation to change the order between k and j in the expression of F p,t µ (σ) where

F p,t µ (σ) = ∑ t k=1 P(T = k)(∑ k j=1 r p y j ,a j (x j )) ∑ t j=1 jP(T = j)
.

Examples: This model generalizes the finite and infinite horizon payoff notions:

• If T is the Dirac measure concentrated on t * i.e P(T = j) = 0 if j = t * and P(T = t * ) = 1, we obtained the arithmetic average payoff

E σ,µ ∑ t * j=1 r p y j ,a j (x j ) t * • If T is the geometric distribution P(T = t) = (1 -δ)δ t-1
, then we obtain the average discounted payoff:

(1

-δ) E σ +∞ ∑ t=1 δ t-1 r p y t ,a t (x t )
• Note that when the expected horizon of local interaction is finite (for example when the lifetime of the system or of the user is finite -in expectation -but the end of the interaction is not known)2 , the average payoff can be rewritten as

F p µ (σ) = ∑ +∞ j=1 ∑ +∞ k=j P(T = k) E σ,µ r p y j ,a j (x j ) ∑ +∞ j=1 jP(T = j) = ∑ t≥1 P(T ≥ t)E σ,µ r p y t ,a t (x t ) E(T) (5.9)
The following theorem generalizes the Theorem 2.1 in [START_REF] Altman | Constrained Markov Games: Nash Equilibria[END_REF] for constrained games and also the Theorem 2.6 (ii) in ( 21) and the Theorem 1 in (83) for unconstrained product games. Theorem 5.5.2. Assume that each subpopulation has a single (aperiodic) ergodic class under each stationary strategy. Then the stochastic population game with individual independent states and unknown lifetime has an equilibrium in stationary strategies. Moreover, the constrained game has an equilibrium under Slater condition.

For the proof we need tightness properties of the measure generated by the frequencies state-actions under the distribution of the horizon. Proposition 5.5.2.1. Assume that for each subpopulation p and any stationary strategy σ p , the state process is an irreducible Markov chain with one ergodic class then, for any strategy σ the frequencies state-actions

( f p,t µ,σ (y, a)) p∈P,t≥1 where f p,t σ,µ (y, a) = ∑ t j=1 ∑ t k=j P(T = k) P σ (X j = y, a j = a|y 1 = µ) ∑ t j=1 jP(T = j) are tight.
The occupation measures in this extended model are characterized by the following convergence result: P σ,µ almost surely, the random variables that give the frequencies state-action

∑ t k=1 P(T = k) ∑ k j=2 δ y ′ (X p j ) -∑ y,a Q p yay ′ δ (y,a) (X p j-1 , a p j-1 ) ∑ t j=1 jP(T = j) (5.10)
goes to zero when t goes to infinity, for all y ′ ∈ Y p , p ∈ P. Hence, when

E(T) = ∑ k≥0 P(T > k) = ∑ k kP(T = k)
< +∞ then we obtain the equation:

∑ +∞ k=1 P(T = k) ∑ +∞ j=2 δ y ′ (X p j ) -∑ y,a Q p yay ′ δ (y,a) (X p j-1 , a p j-1 ) ∑ ∞ j=2 P(T ≥ j)
(5.11)

Energy Control in Wireless Networks

We next illustrate the stochastic population game setting with a problem that arises in dynamic power control in mobile networks.

Users participate in local competitions for the access to a shared medium in order to transmit their packets. An individual state of each mobile represents the energy level at the user's battery which, for simplicity, we assume to take finitely many values, denoted by S = {0, . . . , n}.

Each time the battery empties (which corresponds to reaching state 0), the mobile changes the battery to a new one (this corresponds to state n), and pay a cost C. We assume that each time a mobile reaches state zero, it remains there during a period whose expected duration is τ.

In each state s ∈ S \ {0}, each mobile has two available actions h and l which correspond respectively to high power p H and low power p L . We consider an Aloha-type game where a mobile transmits a packet with success during a slot if:

• with probability p, the mobile is the only one to transmit during this slot,

• the mobile transmits with high power and the other transmitting mobile uses low power or is in state 0.

Time average fitness criterion

The reward function r depends on a mobile's state as well as on the transmission powers, that is, the action of the mobile as well as that of the one it interacts with. Then we have for s = 0:

r(s, a, s ′ , a ′ ) = p + (1 -p)1 (s ′ =0) + (1 -p)1 {(a=h), (a ′ =l), (s ′ =0)} .
For s = 0 we take r(0, a, s ′ , a ′ ) = C/τ.

For each state s ∈ S \ {0}, the transition probability Q s ′ (s, a) may be non-zero (for both a ∈ {l, h}) only for s ′ ∈ {s, s -1}. Then, as the two possible transitions are to remain at the same energy level or move to the next lower one, we simplify the notation and use Q(s, a) to denote the probability of remaining at energy level s using action a.

To model the fact that the mobiles stays in the average τ units at state 0 and then moves to state n we set the transition probabilities from state 0 to any state other than n and 0 to be zero; the probability to move to n is 1/τ and that of remaining at 0 is 1 -1/τ.

The transition probabilities between energy levels which are motivated by the application of energy consumption satisfy:

• For all state s ∈ S \ {0}, we have Q(s, h) < Q(s, l) because using less power induces higher probability to remain in the same energy level.

• For all state s ∈ S \ {0} and for both actions a ∈ {l, h}, we have

Q(s, a) > Q(s -1, a)
because less battery energy the mobile has, less is the probability to remain at the same energy level.

We pursue the example described in Section 5.6 applying the latest proposition in order to obtain the OMESS for this stochastic population game. Indeed, we will find the OMESS for the related EG game which will be written as a matrix game with dimension 4. In order to find the equilibrium of this matrix game, we have to compute the fitness F(v, u) for all policies v and u. We use the renewal theorem to find the expected fitness par cycle of lifetime.

F(v, u) = p T η,v T η,v + τ T η,u T η,u + τ + (1 -p) T η,v (h) T η,v + τ T η,u (l) T η,u + τ + τ 1 T η,u + τ T η,v T η,v + τ -C 1 T η,v + τ
where T η,v (a) is the expected number of times the action a is used under the policy v starting from initial distribution η.

In a first step, we have to compute the occupation measure f u corresponding to each policy u ∈ {u 1 , u 2 , u 3 , u 4 }; for that we need the probability for a user to be in each state, at time t, using action a under policy u. At initial time t = 0, a mobile always starts with a battery full of energy, that is η = (0, 0, 1). We describe the matrix game with the four following matrices:

F 1 (u i , u j ) = T η,u i T η,u i + τ T η,u j T η,u j + τ , F 2 (u i , u j ) = T η,u i (h) T η,u i + τ T η,u j (l) T η,u j + τ , F 3 (u i , u j ) = 1 T η,u j + τ T η,u i T η,u i + τ ,
and

F 4 =                  1 X 1 + X 3 + τ 1 X 1 + X 3 + τ 1 X 1 + X 3 + τ 1 X 1 + X 3 + τ 1 X 1 + X 4 + τ 1 X 1 + X 4 + τ 1 X 1 + X 4 + τ 1 X 1 + X 4 + τ 1 X 2 + X 3 + τ 1 X 2 + X 3 + τ 1 X 2 + X 3 + τ 1 X 2 + X 3 + τ 1 X 2 + X 4 + τ 1 X 2 + X 4 + τ 1 X 2 + X 4 + τ 1 X 2 + X 4 + τ                  with X 1 = 1 1 -Q(1, l) , X 2 = 1 1 -Q(1, h) , X 3 = 1 1 -Q(2, l) , X 4 = 1 1 -Q(2, h) .
Then we obtain the following modified fitnesses depending on the policies in the following matrix:

F = p F 1 + (1 -p) F 2 + τ F 3 -C F 4 .
The OMESS of the stochastic population game which model energy control behaviors in wireless networks is obtained by finding the OMESS of the standard EG with the matrix of fitnesses given by F.

Energy control: absorbing state

We study in this Section an energy management non-cooperative game with an infinite number of players modeled as a stochastic population game. We consider pairwise interactions where each player has to choose between an aggressive or a non-aggressive action. The lifetime of each player depends on the level of aggressiveness of all his action during his life. The instantaneous reward of each player depends on the level of aggressiveness of his action and also of his opponent. We consider different restricted strategies and study the existence of evolutionary stable strategies.

We assume that each mobile terminal is an individual player of a global population and the interaction for transmission is interpreted as a fight for food (as in ( 139)). It is important to note that the Markov process of each player is controlled by himself, whereas in (139) the interaction determines the evolution of the individual process. Each player starts his life in the highest energy state or level, say state n. Starting from state n, each player will visit the different levels in the decreasing order until reaching the last state 0. The state 0 corresponds to the state Empty and the state n is the Full state of the battery. The other states 1, . . . , n -1 are intermediary states of the battery or energy. When the system is at the state 0 there is no energy and the player has to charge his battery or to buy a new battery. We will call terminal absorbing fitness the expected cost of charge or price of a new battery.

At each time slot there should be an interaction with another player and then an action should be taken by each player in the population. The actions taken by a player determine not only the immediate reward but also the transition probabilities to its next individual state (energy level). The instantaneous reward depends also on the level of aggressiveness of the others players.

Then, the evolution of the level of energy depending on the player's action can be modeled as a discrete homogeneous time Markov process.

Individual sequential decision

Each player has to choose between two possible actions in each state (different from the Empty state): high(h) and low(l). The strategy h is a aggressive strategy and the strategy l is a passive one. Since there are only two strategies h and l, the population aggressiveness can be described by a process (α t ) t≥1 where α t is the fraction of the population using the aggressive strategy, that is the action h, at time t. The stochastic process α t may depend on the history with length t. The main issue of this paper is to study how a player will manage its energy during his lifetime in order to optimize its total reward.

Binary Reward

We define a binary reward of a player in an interaction as follows: The player win the resource if (i) he has no opponents for the same resource during the time slot, or (ii) he uses the most aggressive action and its opponents are passive. Otherwise, he gets zero. We extend the reward function to mixed strategies by randomized the binary reward.

We denote by 1p the probability for any player of having an opponent at the same time slot. The probability of success (probability to win to the resource) of a player using the action h when its energy level is at the state s > 0 is given by

r(s, h, α) = p + (1 -p)[α × 0 + (1 -α) × 1] = p + (1 -p)(1 -α), (5.12)
where α is the fraction of the population who use the action h at any given time. Note that assuming that α is fixed in time does not mean that the actions of each player are fixed in time. It only reflects a situation in which the system attains a stationary regime due to the averaging over a very large population, and the fact that all players choose an action in a given individual state using the same probability low.

Similarly, when he use l the probability of success in state s = 0 is r(s, l, α) = p.

(5.13)

The expected reward of a player when its energy level is at the state s > 0 and he uses h with probability β s is then given by

r(s, β s , α) = β s r(s, h, α) + (1 -β s )r(s, l, α) = p + β s (1 -p)(1 -α).
(5.14)

Under this reward function, a player has more chance to win the resource with an aggressive strategy than a passive one, but a passive strategy save more energy for future. Remarks 5.7.2.1. Consider a pairwise one-shot interaction between a player i in state s = 0 and a player j in state s ′ = 0. Define the action set of each player as {h, l} and the payoff function as the binary reward defined above. From the probability to win, it is not difficult to see that the instantaneous reward r(s, β s , α) increases with β s . Then, the one-shot game is a degenerate game with infinite Nash equilibria.

We will see that the non-aggressive which is weakly dominated in the one-shot game is not necessarily dominated in the long-term dynamic game that we will describe later.

A pure strategy of a player at time t is a map α t : H t -→ {h, l}, where H t = All t-1 × S the set of histories at time t with All = {(s, a) | s = 1, . . . , n, a ∈ {h, l}}. The collection σ = (σ t ) t≥1 of pure strategy at each time constitutes a pure strategy of the population. We denote by Σ the set of all pure strategies.

Let σ be a strategy profile, and h t = (s 1 , a 1 , . . . , s t-1 , a t-1 , s t ) be a history of length t. The continuation strategy -→ σ h t from round t given the history h t can be defined as follows

-→ σ h t (s ′ 1 , a ′ 1 , . . . , a ′ t ′ -1 , s ′ t ′ ) = σ(s 1 , a 1 , . . . , s t-1 , a t-1 , s t , s ′ 1 , a ′ 1 , . . . , a ′ t ′ -1 , s ′ t ′
). The fitness obtained with the policy -→ σ , v = r( -→ σ h t ) is called continuation fitness after h t . If σ is a stationary policy then v depends only on the state s t .

For each state s = 0, the action space of each player becomes [0, 1]. We define as a player policy, the set of actions during his battery life, that is the sequence u = (u 1 , u 2 , . . .) where u i is the probability of choosing action h at time slot i. We consider the energy level of energy at time t as the random process (X t ) t≥1 and energy management is a Markov decision process.

For the remainder we shall consider only stationary policies where the probabilities u i is the same for all time i and we denote it by the vector β = (β 1 , β 2 , . . . , β n ) where β s is the probability of choosing action h in state s. We denote by h = (1, . . . , 1) and l = (0, . . . , 0) the two pure aggressive and non-aggressive policies. Given a stationary policy β and a strategy of the global population α = (α t ) t≥1 , the transitions in energy levels of the battery is described by the Markov process (X t ) with the transition law

∀s = 0, ∀a ∈ {h, l}, q(X t+1 = s ′ |X t = s, a) =    Q s (a) if s ′ = s 1 -Q s (a) if s ′ = s -1 0 otherwise
and q(0|0, a) = 1.

We assume two main characteristics of the transition probabilities of energy levels depending on the action taken by the player.

• First, ∀s = 0, Q s (h) < Q s (l) < 1, reflects that using action h, the probability to remain in any given energy level is lower than using action l,

• second, ∀a, the discrete function s -→ Q s (a) is non-decreasing. This assumption means that lower is the energy level, lower is the probability to remain in the same level for each action.

Note that each player controls the transition state of its own battery: Q is independent of the decision of the other players. Let T(β) the expected time to reach to the state 0 given the initial state Full n under the stationary policy β. Then the total reward of the player in all the lifetime of its battery, starting in state n, is given by

V β (n, α) = E β,α T(β) ∑ t=1 r(s t , β s t , α t ) = r(n, β n , α) + E β,α T(β) ∑ t=2 r(s t , β s t , α) |s t 1 = n .
(5.15)

Computing Fitness using Dynamic Programming

We define the dynamic programming operator Y(s, a, α, v) to be the total expected fitness of a player starting at stat s if

• It takes action a at time 1,

• at time 2 the total sum of expected fitness from time 2 onwards is v,

• At each time slot, the probability that another player uses action h, given that there is a interaction between individual players, is α.

We denote by v the continuation fitness function which depends on the state s and the global population strategy α. We have

Y(s, a, α, v) = r(s, a, α) + Q s (a)v(s, α) + (1 -Q s (a))v(s -1, α), ∀a ∈ {h, l}.
The function v is a fixed point of the operator φ defined by

v -→ (φv) := β s Y(s, h, α, v) + (1 -β s )Y(s, l, α, v).
Proposition 5.7.3.1. The expected fitness for a player during its lifetime is given by:

V β (n, α) = n ∑ i=1 r(i, β i , α) 1 -Q i (β i ) -C.
Proof. We compute the fitness by using the recursive formula defined by the operator φ. First, we have that v(0, α) = 0 and then

v(1, α) = φ(v) = β 1 [r(1, h, α) + Q 1 (h)v(s, α) + 0] + (1 -β 1 )[r(s, l, α) + Q s (l)v(1, α) + 0], = r(1, β 1 , α) + [β 1 Q 1 (h) + (1 -β 1 )Q 1 (l)]v(1, α).
The fixed point v(1, α) is given by

V β (1, α) = r(1, β 1 , α) 1 -[β 1 Q 1 (h) + (1 -β 1 )Q 1 (l)] = p + β 1 (1 -p)(1 -α) 1 -[β 1 Q 1 (h) + (1 -β 1 )Q 1 (l)]
. For all state s > 1 we have

v(s, α) = β s [r(s, h, α) + Q s (h)v(s, α) + (1 -Q s (h))v(s -1, α)], +(1 -β s )[r(s, l, α) + Q s (l)v(s, α) + (1 -Q s (l))v(s -1, α)] = r(s, β s , α) + [β s Q s (h) + (1 -β s )Q s (l)]v(s, α) + [β s (1 -Q s (h)) + (1 -β s )(1 -Q s (l))]v(s -1, α), = r(s, β s , α) + Q s (β s )v(s, α) + (1 -Q s (β s ))v(s -1, α), where Q s (β s ) := β s Q s (h) + (1 -β s )Q s (l). Since Q s (β s ) < 1 one has v(s, α) = r(s, β s , α) 1 -Q s (β s ) + v(s -1, α).
By rewriting

v(s, α) = s ∑ i=1 [v(i, α) -v(i -1, α)],
we obtain that

V β (s, α) = s ∑ i=1 r(i, β i , α) 1 -Q i (β i ) .
Hence, the expected fitness obtained by starting from the Full state n to the next new battery is then given by V β (n, α) -C.

Sojourn Times

We denote by T s (β s ) the sojourn time of a player at the state s with the policy β s . The sojourn time satisfies the following fixed point formula:

T s (β s ) = 1 + ∑ a∈{h,l} Q s (a)β s (a)T s (β s ). Hence, T s (β s ) = 1 1 -Q s (β s )
The fraction of times that the player chooses the aggressive h in state s is given by

αs (β) = β s T s (β s ) ∑ n i=1 T i (β i )
and the probability for any player to meet another player which is using action h is given by

α(β) := n ∑ s=1 αs (β) = ∑ s β s T s (β s ) ∑ n i=1 T i (β i ) ≤ 1. (5.16)
The function α gives a mapping between the strategy vector β of an anonymous player and the global population strategy α.

Reduced game

We compare the two type of attitudes: use always the high power ( h) or use always low power l. The payoffs corresponding to these strategies are described as follows:

h l h pT h T h l pT l pT l where T h = n ∑ s=1 1 1 -Q s (h) (resp. T l = n ∑ s=1 1 1 -Q s (l)
) is the expected lifetime of the battery under the policy h (resp. l). T h can be interpreted also as the hitting time to the state 0 starting from the state Full. This description gives us a two-player matrix game called reduced game. Generically, the reduced game has a unique ESS and global maximizer. Since the remaining energy of battery decreases with the power consumption: T h < T l one has:

• If p = 0. Then h is the unique ESS and 1 2 is the global maximizer.

• If 0 < p < T h T l then ν = 1 1 -p - p 1 -p T l T h
is the unique ESS and the global maximizer is ν 2 .

• If T h T l < p ≤ 1 then l is the unique ESS and global maximizer of the reduced game.

Pure stationary strategies Preliminaries

As already mentioned, our game is different and has a more complex structure than a standard evolutionary game. In particular, the fitness that is maximized is not the outcome of a single interaction but of the sum of fitnesses obtained during all the opportunities in the player's lifetime.

We obtain the following characterization of an Evolutionary Stable Strategy (ESS). Lemma 5.7.7.1. A necessary condition for β * to be an ESS is

for all β ′ = β * , V β * (n, α(β * )) ≥ V β ′ (n, α(β * )) (5.18)
A sufficiency condition for β * to be an ESS is that (5.18) holds with strict inequality for all β ′ = β * .

Using this corollary and this simple necessary condition to be an ESS, we are now able to determine the existence of an ESS. First we considering the restricted game to independent state actions, i.e. the probability of choosing the aggressive action h does not depend on the individual state. Second, we generalize to dependent state actions which is more complicated. However we need latter the following lemma in numerous proofs. Lemma 5.7.7.2. Given α(β ′ ), the function Z :

β s -→ p + (1 -p)(1 -α(β ′ ))β s 1 -Q s (β s ) is monotone. Proof. Since Z is a continuously differentiable function, Z is strictly decreasing if (1 -p)(1 -α(β ′ ))(1 -Q s (l)) < (Q s (l) -Q s (h))p, constant if (1 -p)(1 -α(β ′ ))(1 -Q s (l)) = (Q s (l) -Q s (h))p
and strictly increasing if

(1 -p)(1 -α(β ′ ))(1 -Q s (l)) > (Q s (l) -Q s (h))p.
Thus, given p, Q, the function Z is monotone.

State-independent strategies

We examine the case where the probability of choosing action h does not depend on the level of energy of the battery, that is ∀s β s = ρ. We first consider only h the fully aggressive strategy (the player uses always the high energy level available in the state) or l the fully passive strategy (the player uses always weak action level when it is possible) policies. Proposition 5.7.8.1. If p = 0, then the strategy h which consists to play always h (ρ = 1) cannot be an ESS. Otherwise, if p = 0 then h is an evolutionary stable strategy.

Proof. Suppose that p = 0. Then, for all stationary mixed policy β

′ V 0 (n, α(β ′ )) = p ∑ s 1 1 -Q s (l)
, and

V 1 (n, α(1)) = p ∑ s 1 1 -Q s (h) . Thus V 1 (n, α(1)) < V 0 (n, α(1)
). It is better to use the low power l if the opponent use high power h (the payoff is p and the lifetime is increased). Hence, the strategy h is not is a best reply to itself. This means that h is not a Nash equilibrium and then h is not an ESS. Suppose now that p = 0. Then the strategy l is weakly dominated, and

V h (n, α( h)) = 0 = V l (n, α(β ′ )), ∀β ′ .
Moreover, for all β ′ = h we have

V β ′ (n, α(1)) = 0 = V h (n, α( h))
, and

V h (n, α(β ′ )) = (1 -α(β ′ )) ∑ s 1 1 -Q s (h) > V β ′ (n, α(β ′ )) = (1 -α(β ′ )) ∑ s β ′ s 1 -Q s (β ′ s )
. This completes the proof.

This result is somehow logical as when, at each time slot, any player is in interaction with another player, the only strategy which permits to win the fight is to be aggressive. We have the same kind of result for the non-aggressive strategy. Proposition 5.7.8.2. The strategy l which consists to play always l (ρ = 0) is an ESS if and only if the subset of states

I 1 := {s, p < 1 -Q s (l) 1 -Q s (h) } is empty.
Proof. We have

V 0 (n, α(β ′ )) = p n ∑ s=1 1 1 -Q s (l)
, and

V β (n, α(0)) = n ∑ s=1 p + (1 -p)β s 1 -Q s (β) . Thus max β V β (n, α(0)) = ∑ s∈I 1 1 1 -Q s (h) + p ∑ s∈I 0 1 1 -Q s (l) + ∑ s∈I * 1 1 -Q s (h) < V 0 (n, α(0)) ( 5.19)
where

I 1 := {s, p < 1 -Q s (l) 1 -Q s (h) }, I 0 := {s, p > 1 -Q s (l) 1 -Q s (h) }, I * := {s, p = 1 -Q s (l) 1 -Q s (h) }.
The subsets I 0 , I 1 , I * constitute a partition of S. I * has at most one element. Using lemma 5.7.7.2 and the equation (5.19), l is a best response to itself if

∑ s∈I 1 1 1 -Q s (h) < p ∑ s∈I 1 1 1 -Q s (l) .
This inequality does not holds if I 1 is not empty.

We consider now stationary mixed strategies where each player determine the probability ρ of choosing the aggressive strategy at each energy level of his battery. Proposition 5.7.8.

If the same level of aggressiveness

β s = ρ ∈ (0, 1), ∀s is an ESS then it must satisfy p ∑ n s=1 (Q s (l)-Q s (h)) (1-Q s (ρ)) 2 (1 -p) ∑ n s=1 (1-Q s (l)) (1-Q s (ρ)) 2 = 1 -ρ. (5.20)
If p, Q and 0 ≤ ρ ≤ 1 satisfy (5.20), then ρ is an ESS.

Proof. The strategy ρ is a best reply to the strategy

β ′ if it satisfies V ρ (n, α(β ′ )) ≥ V µ (n, α(β ′ )), ∀µ ∈ [0, 1] n . Then, ρ is a best response to itself if V ρ (n, α(ρ)) ≥ V µ (n, α(ρ)), ∀µ ∈ [0, 1] n .
A necessary condition for an interior extremum of the function h :

x -→ n ∑ s=1 p + (1 -ρ)(1 -p)x 1 -Q s (x) is to satisfy h ′ (x) = 0 ⇐⇒ 1 -ρ = p ∑ n s=1 (Q s (l)-Q s (h)) (1-Q s (x)) 2 (1 -p) ∑ s (1-Q s (l)) (1-Q s (x)) 2 .
Note that α(ρ) = ρ.

We now show that for every p, Q such that

p ∑ n s=1 (Q s (l)-Q s (h)) (1-Q s (ρ)) 2 (1 -p) ∑ s (1-Q s (l)) (1-Q s (ρ)) 2 ≤ 1, every strategy (ρ, 1 -ρ) satisfying 1 -ρ = p ∑ n s=1 (Q s (l)-Q s (h)) (1-Q s (ρ)) 2 (1 -p) ∑ s (1-Q s (l)) (1-Q s (ρ)) 2 is an ESS. Let β ′ be a strategy such that V ρ (n, ρ) = V β ′ (n, ρ). Denote by C := V ρ (n, α(β ′ )) - V β ′ (n, α(β ′ ))
. We now prove that C is strictly positive for all β ′ = ρ.

The condition C > 0 is satisfied if

V ρ (n, α(β ′ )) > V β ′ (n, α(β ′ )), n ∑ s=1 p + (1 -p)(1 -α(β ′ ))ρ 1 -Q s (ρ) > n ∑ s=1 p + (1 -p)(1 -α(β ′ ))β ′ 1 -Q s (β ′ ) . Since V ρ (n, ρ) = V β ′ (n, ρ), one has [p + (1 -p)(1 -ρ)ρ] ∑ s 1 1 -Q s (ρ) = [p + (1 -p)(1 -ρ)β ′ ] ∑ s 1 1 -Q s (β ′ ) . This implies that ∑ s 1 1 -Q s (β ′ ) = p + (1 -p)(1 -ρ)ρ p + (1 -p)(1 -ρ)β ′ ∑ s 1 1 -Q s (ρ)
.

By restituting this value in C, we obtain

C > 0 if n ∑ s=1 p + (1 -p)(1 -α(β ′ ))ρ 1 -Q s (ρ) > [p + (1 -p)(1 -α(β ′ ))β ′ ] p + (1 -p)(1 -ρ)ρ p + (1 -p)(1 -ρ)β ′ ∑ s 1 1 -Q s (ρ) , (p + (1 -p)(1 -α(β ′ ))ρ) > [p + (1 -p)(1 -α(β ′ ))β ′ ] p + (1 -p)(1 -ρ)ρ p + (1 -p)(1 -ρ)β ′ , (p + (1 -p)(1 -α(β ′ ))ρ)(p + (1 -p)(1 -ρ)β ′ ) > [p + (1 -p)(1 -α(β ′ ))β ′ ](p + (1 -p)(1 -ρ)ρ).
Developing and simplifying the last expression, we obtain that

D > 0 is equivalent to [β ′ - ρ][α(β ′ ) -ρ] = [β ′ -ρ] 2 > 0.
This completes the proof.

We are now able to determine explicitly, if it exists, an ESS in stationary mixed strategies of our stochastic population game for energy management. Then, it is interesting to provide conditions of existence of such stationary mixed ESS. Proposition 5.7.8.4. A sufficiency condition of existence of an interior state-independent evolutionary stable strategy is given by p < p 0 :=

∑ n s=1 1 1-Q s (l) ∑ n s=1 (1-Q s (h)) (1-Q s (l)) 2 .
(5.21)

Proof. The function

ξ(p, ρ) = p n ∑ s=1 (Q s (l) -Q s (h)) (1 -Q s (ρ)) 2 -(1 -p)(1 -ρ) n ∑ s=1 (1 -Q s (l)) (1 -Q s (ρ)) 2 ,
is continuous on (0, 1) and ξ(p, 1) > 0, ∀p ∈ (0, 1). Thus, if (5.21) satisfied then ξ(p, 0) < 0 and 0 is in the image of (0, 1) by ξ(p, .). This implies that there exists ρ ∈ (0, 1) such that ξ(p, ρ) = 0.

> max s =0 1 -Q s (l) 1 -Q s (h)
and from Proposition 5.7.8.1, h is an ESS if and only p = 0. Since, max

s =0 1 -Q s (l) 1 -Q s (h)
> 0, we conclude that only one of two strategies can be an ESS.

(ii) Strictly mixed ESS (or interior ESS): A necessary condition to interior ESS is given by Equation (5.20). Let ρ and ρ ′ in (0, 1) be two solutions of (5.20) i.e ξ(p, ρ) = ξ(p, ρ ′ ) = 0. From Proposition 5.7.8.3, one has

V ρ ′ (n, α(ρ ′ )) ≥ V ρ (n, α(ρ ′ )) > V ρ ′ (n, α(ρ ′ ))
which is a contradiction. We conclude that if an ESS exists in state-independent strategies, it is unique.

Note that the inequality (5.21) is satisfied if

0 < p < min s =0 1 -Q s (l) 1 -Q s (h) . ( 5.22)
Moreover the following result holds: Lemma 5.7.8.6.

min s =0 1 -Q s (l) 1 -Q s (h) < p 0 < max s =0 1 -Q s (l) 1 -Q s (h) . (5.23) Proof. Let a = min s =0 1 -Q s (l) 1 -Q s (h) and b = max s =0 1 -Q s (l) 1 -Q s (h) . Then, ∀s, b > 1 -Q s (l) 1 -Q s (h) > a ⇐⇒ (5.24) ∀s, b(1 -Q s (h)) > 1 -Q s (l) > a(1 -Q s (h)) ⇐⇒ (5.25) ∀s, b(1 -Q s (h)) -(1 -Q s (l)) > 0 > a(1 -Q s (h)) -(1 -Q s (l)) ⇐⇒ (5.26) ∀s, b(1 -Q s (h)) -(1 -Q s (l)) (1 -Q s (l)) 2 > 0 > a(1 -Q s (h)) -(1 -Q s (l)) (1 -Q s (l)) 2 (5.27)
By taking the sum over s from one to n, one has,

b n ∑ s=1 1 -Q s (h) (1 -Q s (l)) 2 - n ∑ s=1 1 1 -Q s (l) > 0 > a n ∑ s=1 1 -Q s (h) (1 -Q s (l)) 2 - n ∑ s=1 1 1 -Q s (l) .
This means that ξ(b, 0) > ξ(p 0 , 0) > ξ(a, 0). Since, the function p -→ ξ(p, ρ) is strictly increasing, the last inequality implies that b > p 0 > a. This completes the proof.

The relation (5.22) gives another sufficient condition of existence of a stationary mixed ESS.

By proposition 5.7.8.2, there is a pure non-aggressive ESS (l) if and only if p > max s =0

(

1 -Q s (l) 1 -Q s (h) ).
Then in some particular cases of parameters p and transition probabilities, there is no pure independent stationary ESS nor mixed, when

p 0 < p < max s =0 ( 1 -Q s (l) 1 -Q s (h) ).
The cooperative optimal strategy for players is denoted by ρ * that is defined by:

ρ = arg max β V β (n, β).
This strategy gives the global optimum solution of the centralized system of our energy management model. We are interested in comparing the aggressiveness of the ESS to the global optimum solution. Proposition 5.7.8.7. Let ρ * be the ESS in stationary strategies of the stochastic population game and ρ the global optimum solution. Then we have

ρ ≤ min{ 1 2 , ρ * }.
Proof. The function

β -→ V β (n, β) = n ∑ s=1 p + (1 -p)(1 -β)β 1 -Q s (l) + β(Q s (l) -Q s (h))
is continuous and strictly decreasing on ( 1 2 , 1). Thus, the function has a global maximizer on [0, 1] and the global maximizer is lower than 1 2 .

Let ρ * be an ESS. Suppose that ρ > ρ * . Since ρ * is an ESS, ρ * satisfies

V ρ * (n, ρ * ) ≥ V ρ (n, ρ) and V ρ * (n, ρ) > V ρ (n, ρ) if V ρ * (n, ρ * ) = V ρ (n, ρ * ). Given a strategy β, the function α -→ V β (n, α) is strictly decreasing. Hence, V ρ * (n, ρ * ) > V ρ * (n, ρ) > V ρ (n, ρ).
The inequality

V ρ * (n, ρ * ) > V ρ (n, ρ)
is a contradiction with the definition of ρ. Hence, ρ ≤ ρ * . We conclude that a global maximizer is lower than 1 2 and ρ coincides with ρ * (it is the case if the ESS is a pure strategy) or ρ < ρ * .

The main important result here is that the ESS is more aggressive than the global optimum solution, i.e. ρ * ≥ ρ. This seems relatively intuitive because in a context of an evolutionary game, every player is somehow afraid to meet another player using an aggressive strategy, then the aggressive strategy is more used in the population.

State-dependent actions

We consider now that the action taken by each player depends on his energy level or state. We first compute explicitly the best response correspondence BR : S\{0} -→ 2 I where I is the compact set [0, 1]. Proposition 5.7.9.1. In stationary strategies, the best response to the strategy α(β ′ ) is determined by (β 1 , . . . , β n ) such that for all s = 1, . . . , n,

β s (α(β ′ )) =                1 if α(β ′ ) < 1 - p(Q s (l) -Q s (h)) (1 -p)(1 -Q s (l)) 0 if α(β ′ ) > 1 - p(Q s (l) -Q s (h)) (1 -p)(1 -Q s (l)) any strategy η s ∈ [0, 1] if α(β ′ ) = 1 - p(Q s (l) -Q s (h)) (1 -p)(1 -Q s (l))
Thus, the optimal stationary strategy β = (β 1 , . . . , β n ) is characterized by

β s =          1 if s ∈ I α := {j, Q j (l) -Q j (h) (1 -Q j (l)) < (1 -α)(-1 + 1 p )} 0 if s ∈ J α := {j, Q j (l) -Q j (h) (1 -Q j (l)) > (1 -α)(-1 + 1 p )}
Proof. The best reply β s to α(β ′ ) maximizes the function Z :

β s -→ p + (1 -p)(1 -α(β ′ ))β s 1 -Q s (β s ) defined in
(1 -p)(1 -α(β ′ ))(1 -Q s (l)) > (Q s (l) -Q s (h))p, zero if (1 -p)(1 -α(β ′ ))(1 -Q s (l)) < (Q s (l) -Q s (h))p,
and any strategy in [0, 1] is maximizer of Z if the equality

(1 -p)(1 -α(β ′ ))(1 -Q s (l)) = (Q s (l) -Q s (h))p
holds. We conclude that

β s (α(β ′ )) =                1 if α(β ′ ) < 1 - p(Q s (l) -Q s (h)) (1 -p)(1 -Q s (l)) 0 if α(β ′ ) > 1 - p(Q s (l) -Q s (h)) (1 -p)(1 -Q s (l)) any strategy η s ∈ [0, 1] if α(β ′ ) = 1 - p(Q s (l) -Q s (h)) (1 -p)(1 -Q s (l)) Let f (s) = T s (l) T s (h) = 1 -Q s (h) 1 -Q s (l) , (5.28)
be the ratio between the sojourn time with the low power l and the sojourn time with h in each state s. By taking assumption on the transition probabilities, we have that for all energy level s, f (s) > 1. This ratio can be interpreted also as the proportion of time obtained taken the non-aggressive action l in state s compared to the aggressive one h. Definition 5.7.9.2 (Threshold Policies). We define the two following pure threshold policies u 1 and u 2 :

• Risky Policy (RP) : there exists a state s such that ∀s ′ > s, u 1 (s ′ ) = L and ∀s ′′ < s, u 1 (s ′′ ) = H. This policy is called a control limit policy in [START_REF] Mcnamara | If animals know their own fighting ability, the evolutionary stable levels of fighting is reduced[END_REF].

• Carefully Policy (CP) : there exists a state s such that ∀s ′ > s, u 2 (s ′ ) = H and ∀s ′′ < s, u 2 (s ′′ ) = L.

That kind of threshold policies, keeping the same action until the level of energy is low, has been also obtained in [START_REF] Mcnamara | If animals know their own fighting ability, the evolutionary stable levels of fighting is reduced[END_REF] in a context of a dynamic Hawk and Dove game. The inequality

α(β ′ ) < 1 - p(Q s (l) -Q s (h)) (1 -p)(1 -Q s (l)) = 1 - p 1 -p ( f (s) -1) is equivalent to (1 -α(β ′ )) 1 -p p > f (s) -1, (5.29)
where f (s) -1 is exactly the gain in mean sojourn time in state s using action l instead of action h. Equation 5.29 can be rewritten:

p 1 -Q s (l) < 1 -(1 -p(1 -α(β ′ ))) 1 -Q s (h) ,
where the left side term is the average reward for a player during the time it is in state s and it uses action l, and the right term is the average reward during the time it is in state s and it uses action h. Then, the threshold policies are based on the comparison between the average instantaneous reward in each state depending on the action taken by each player which determines the instantaneous reward and also the remaining time in this state.

We have the following result showing that the best response strategies are u 1 or u 2 depending on the structure of the Markov Decision Process. Proposition 5.7.9.3. The best response is u 1 (resp. u 2 ) if f (s) is increasing (resp. decreasing).

Proof. First we assume that the function f is decreasing, meaning that more the level of energy is high, more is the gain of surviving time in each energy level. If

1 -Q n (h) 1 -Q n (l) ) < (1 -α)(-1 + 1 p ) then β s = plays h if s ≥ j(α, p) plays l if s ≤ j(α, p) -2 else if 1 -Q n (h) 1 -Q n (l) > (1 -α)(-1 + 1 p
) then β s consists to play always l where

j(α, p) := min{j, 1 -Q j (h) 1 -Q j (l) < (1 -α)(-1 + 1 p )}.
Note that one has at most one state s such that 1 -Q j (s)

1 -Q s (l) = (1 -α)(-1 + 1 p
). We have the inverse relations if f (s) is increasing.

Theorem 5.7.10 (Partially mixed equilibrium). If there exists a level s such that

β s = α * + (1 -Q s (l))(κ 2 α * -κ 1 ) 1 -(Q s (l) -Q s (h))(κ 2 α * -κ 1 ) ∈ (0, 1)
where

κ 1 = ∑ j∈I 1 1 -Q j (h) , κ 2 = κ 1 + ∑ j / ∈I, j =s 1 1 -Q j (l) , α * = 1 - p(Q s (l) -Q s (h)) (1 -p)(1 -Q s (l)) , I = {j = s, f (s) > f (j)}
and β j , j = s given by

β j =    plays h if (Q s (l) -Q s (h)) (1 -Q s (l)) > (Q j (l) -Q j (h)) (1 -Q j (l)) i.e f (s) > f (j) plays l otherwise    , then β = (β 1 , . . . , β n ) is an equilibrium.
Proof. By Proposition 5.7.9.1, the best reply has the form

BR(β s ) =                play action h if α(β s ) < 1 - p(Q s (l) -Q s (h)) (1 -p)(1 -Q s (l)) , play action l if α(β s ) > 1 - p(Q s (l) -Q s (h)) (1 -p)(1 -Q s (l)) , any strategy h, l or mixed if α(β s ) = 1 - p(Q s (l) -Q s (h)) (1 -p)(1 -Q s (l)) .
The fixed point equation gives β ∈ BR(β) with

α(β) = ∑ n s=1 β s 1-Q s (β s ) ∑ n s=1 1 1-Q s (β s ) .
Since f is monotone and Q s (l) < Q s (h), there exist at most one state s 0 such that

1 -Q s 0 (h) 1 -Q s 0 (l) = (1 -α(β))(-1 + 1 p ) (5.30)
and a mixed equilibrium is characterized by

β ′ s =    play action h if s ′ > s 0 , play action l if s ′ < s 0 , any strategy h, l or mixed if s ′ = s 0 .
where β s 0 satisfies equation 5.30. After some basic calculations, equation 5.30 has a unique solution given by

β s 0 = α * + (1 -Q s (l))(κ 2 α * -κ 1 ) 1 -(Q s (l) -Q s (h))(κ 2 α * -κ 1 ) ∈ (0, 1)
with

κ 1 = ∑ j∈I 1 1 -Q j (h) , κ 2 = κ 1 + ∑ j / ∈I, j =s 1 1 -Q j (l) , α * = 1 - p(Q s (l) -Q s (h)) (1 -p)(1 -Q s (l)) , I = {j = s, f (s) > f (j)}.
Note that a necessary condition of existence such s is that

p > (1 -c)(1 -Q s (l)) (1 -Q s (h) -c(1 -Q s (l))) with c = 1 + κ 1 (1 -Q s (h)) 1 + κ 2 (1 -Qs(h))
∈ (0, 1). We then obtain several equilibria by permutation on the state satisfying

β s = α * + (1 -Q s (l))(κ 2 α * -κ 1 ) 1 -(Q s (l) -Q s (h))(κ 2 α * -κ 1 )
∈ (0, 1).

Lemma 5.7.10.1. The partially mixed equilibrium is decreasing in p.

Proof. It suffices to proof in the unique state s in which β s ∈ (0, 1). The fraction of players with the aggressive action in state s is then given by

β s = α * + (1 -Q s (l))(κ 2 α * -κ 1 ) 1 -(Q s (l) -Q s (h))(κ 2 α * -κ 1 ) ∈ (0, 1)
where

κ 1 = ∑ j∈I 1 1 -Q j (h) , κ 2 = κ 1 + ∑ j / ∈I, j =s 1 1 -Q j (l) , α * = 1 - p(Q s (l) -Q s (h)) (1 -p)(1 -Q s (l)) , I = {j = s, f (s) > f (j)}. Since the function p -→ α * (p) = 1 - p(Q s (l) -Q s (h)) (1 -p)(1 -Q s (l))
is strictly decreasing, the denominator is non-decreasing in p and the numerator is strictly decreasing in p. Thus β s is decreasing in p. This completes the proof.

Dynamics

By proposition 5.7.9.3 we know that, when the function f is monotone, the best response strategies have a given structure defined as threshold policies RP or CP. In order to numerically observe the equilibrium policies in the population, we construct the replicator dynamics which have properties of convergence to equilibrium and ESS in evolutionary games. A replicator dynamic is a differential equation that describes the way strategies change in time as a function of the fitness. Roughly speaking they are based on the idea that the average growth rate per player that uses a given action is proportional to the excess of fitness of that action with respect to the average fitness.

When f is monotone increasing (resp. decreasing) the set of policies are u 1 1 , . . . , u n+1 1 (resp. u 1 2 , . . . , u n+1 2 ) where policy u i 1 (resp. u i 2 ) consists of taking action h (resp. l) in the i -1 first states. For all i = 1, . . . , n + 1 the proportion of the population playing u i 1 is denoted by σ i . The replicator dynamics describes the evolution of the different policies in the population in time and is given by the following differential equation:

σi (t) = σ i (t) n+1 ∑ j=1 σ j (t)G u i 1 ,u j 1 - n+1 ∑ k=1 n+1 ∑ j=1 σ k (t)σ j (t)G u k 1 ,u j 1 =: f i (σ(t)).
(5.31)

The replicator dynamics satisfies the so-called positive correlation condition:

f (σ) = 0 =⇒ ∑ j G u j 1 (σ) f j (σ) > 0.
It follows from (? ) that any equilibrium σ * (in the n-simplex of the (n + 1)-dimensional Euclidean space R n+1 ) of the evolutionary game is a rest point of the replicator dynamics. In order to describe the long-term behavior of the dynamics, we shall say that a stationary point (or rest point) σ * is stable under (5.31) if for every neighborhood N of σ * there exists a neighborhood N ′ N of σ * such that σ * ∈ N ′ =⇒ σ(t) ∈ N , ∀t ≥ 0. If σ * is a stable rest point of (5.31) then σ * is an equilibrium.

Numerical Illustrations

We consider an evolutionary game where each player has n = 5 levels of energy. Each player starts his lifetime full of energy (level 5) and has to decide between an aggressive or a nonaggressive action at each pairwise interaction with another player. We assume that all players are in the same population that means that all players have the same transition probabilities Q.

We consider the following transition probabilities

Q =      
0.1 0.05 0.2 0.1 0.5 0.4 0.7 0.6 0.8 0.7

      ,
where for i = 1, . . . , 5, Q i1 (resp. Q i2 ) is the probability to remain in energy level i using action l (resp. h).

State-Independent action

Each player chooses action h with a probability that does not depend on their level of energy. We denote this probability by ρ. First, using this transition probabilities, the threshold p 0 given by equation 5.21 induces the existence of an ESS is p 0 = 0.75 Then using result from proposition 5.7.8.4, if the probability p is less than 0.75, there exists an ESS which is the unique solution of the system defined in equation 5.20.

Second, we have

M = max s 1 -Q s (l) 1 -Q s (h)
= 0.9474, then using proposition 5.7.8.2 the subset of states I 1 is empty. Then, we have that if p > 0.9474 the unique ESS is to take always action l, that is ρ = 0.

We observe also on figure 5.1 that taking the pure aggressive action h is only an ESS when p = 0. It is somehow intuitive because if the system is such that every player meets always another player, the best strategy to survive is to be aggressive. Related to this comment, we observe also that the probability to choose an aggressive action is monotone decreasing with p.

Dependent state action

We consider that each player decide to take an aggressive action depending on their level of energy. Using the transition probabilities Q the function f defined by equation 5.28 is strictly increasing, that means that the best response strategy is a risky policy as u 1 . We describe the set of policies u 1 1 , u 2 1 , . . . , u 6 1 where the policy u i 1 has 5 components which are the action in each energy level from 1 to 5. For example, u 3 1 = (hhhll) means action h in states 1, 2 and 3; and action l in states 4 and 5. On figure 5.2, the replicator dynamics converge when p = 0.5 to the pure policy u 2 1 , i.e. each player takes action l only in the full energy level (state 5). On figure 5.3 the replicator dynamics converge to a mixed strategy between policies u 1 1 and u 2 1 where 87.41% of the population uses policy u 1 1 and 12.59% policy u 2 1 . When p = 0.9, we observe on figure 5.4 the replicator dynamics converge to the pure policy u 5 1 which consists to take action h only in state 1. Moreover, we observe that every rest points of the replicator dynamics in the three cases are stable and then there are equilibrium.

Finally, with those examples, we have observed that when the probability to meet another player decreases (means p increases), players become less aggressive and the equilibrium tends to non-aggressive policies.

Notes

We have studied an energy management non-cooperative population game using evolutionary Markov game framework. We have presented a problem considering the stochastic evolutionary games where each player can be in different state during his life, and has the possibility to take several actions in each individual state of each player. Those actions have an impact not only on the instantaneous fitness but also on the future individual's state of the player. Restricting the game to stationary mixed policies, we have determined explicitly the ESS of the stochastic population game. Considering more general dependent state policies, we have obtained

Access Control in Solar-Powered Broadband Networks

This Section studies both power control and multiple access control in solar-powered broadband wireless networks. We assume that the mobiles use power storage element, such as rechargeable Solar-powered batteries, in order to have energy available for later use. By modeling the energy-level of Solar-powered batteries as a stochastic process, we study noncooperative interactions within large population of mobiles that interfere with each other through many local interactions. Each local interaction involves a random number of mobiles. The actions taken by a mobile determine not only the immediate payoff but also the state transition probabilities of its battery. We define and characterize the evolutionary stable strategies (ESS) of the stochastic evolutionary game.

Environmental energy is becoming a feasible alternative for many low-power systems, such as wireless sensor and mesh networks. However, this provides an unpredictable and limited amount of energy over time. The power storage elements, such as rechargeable batteries or super-capacitors, become very useful to increase the system lifetime and the system availability. In particular, solar power is made possible with the use of Photovoltaic cells (see Fig. 5.5). Comprised of several layers of material, these cells are able to produce electrical power from exposure to sunlight. Since in many geographic areas, nice weather is not guaranteed and is unpredictable, the nodes should be able to recover from blackout periods caused by the unavailability of energy. In [START_REF] Niyato | Sleep and wakeup strategies in solar-powered wireless sensor/mesh networks: Performance analysis and optimization[END_REF], a stochastic model for a solar powered wireless sensor/mesh networks is used to analyze the following QoS measures for several stochastic policies pro- In this Section, we consider an evolutionary game approach with dynamic rechargeable battery depending on the weather (solar energy). We make use of stochastic evolutionary games in solar-powered broadband wireless networks. There are many local interactions among individuals belonging to large populations of mobiles. The result of the interaction between mobiles depends on their current individual state. From time to time the individual state of a mobile varies. The action choice of mobiles involved in a local interactions as well as their individual states determine the not only the result of the interactions but also the transition probabilities to the other possible individual states. Each individual is thus faced with an Markov Decision Process (MDP) in which it maximizes the expected average cost criterion. Each individual knows only the state of its own MDP, and does not know the state of the other mobiles it interacts with.

The destination of some transmission occasionally may receive simultaneously a transmission from another terminal which results in a collision. It is assumed however that even when packets collide, one of the packets can be received correctly if transmitted at a higher power. As state of the MDP of a user we take its energy level. The immediate fitness (rewards) is the number of successful transmissions. By allowing the mobiles to be equipped with rechargeable solar powered batteries, the mobiles may have infinite life time and the criteria that is maximizing is the limit average Cesaro-type payoff.

Consider the following setting of evolutionary games: there is a large populations of mobiles; each mobile has a finite number of transmission power level available. There are many local interactions at the same time. At each slot, some of the terminals have to take a decision on their transmission power based on their own battery state. At the lowest state of the battery no power is available and the mobile has to wait the time to have good weather for regain some energy. Each player has its individual states set S = {0, 1, 2, . . . , n}. Each mobile of the population has a finite action set in each state s : A(s). We assume that there are a random number of interacting mobiles in each local interaction. At time t, each mobile knows its own state s t and selects an action a t ∈ A(s t ). The mobile receives some payoff r(s t , a t , α t ) where α t is the composition of the population at time t (the j-th element of α s t t represents the fraction of mobiles choosing the action j) and the state of mobile goes to the state s t+1 with the probability q(s t+1 |s t , a t ). Each individual is thus faced with an MDP in which it maximizes the expected average fitness criterion. Each individual knows only the state of its own MDP, and does not know the state of the other players it interacts with. The transition probabilities of a player's MDP are only controlled by that player. Let ∆(S) be the (|S| -1)-dimensional simplex of the Euclidean space R |S| . The set of all action profiles at all states is given by All = {(s, a), s ∈ S, a ∈ A(s)}, then q : All × Θ → ∆(S) is a transition rule between the states where Θ = ∏ s∈S ∆(A(s)), ∆(A(s))

is the set of probability distribution on A(s). The vector [q(s t+1 = 0|s t , a t ), . . . , q(s t+1 = n|s t , a t )] satisfies ∀ j ∈ S, q(s t+1 = j|s t , a t ) ≥ 0, ∑ j∈S q(s t+1 = j|s t , a t ) = 1. A state s is absorbing state if q(s|s, a) = 1, ∀ a ∈ A(s), ∀α ∈ Θ. We examine the limit average Cesaro-type payoff given a population profile σ and an individual trajectory u,

F n (u, σ) = lim inf T-→+∞ E u,σ 1 T T ∑ t=1 r(s t , a t , α t )
where E u,σ,n denotes the expectation over the probability measure P u,σ,n induced by u, σ on the set of histories endowed with the product σ-algebra (initial state of the battery is n). Define further

• The subset U S of stationary policies; a stationary policy u is a policy in which the probability to choose an action a depends only on the current state s; it is denoted by u(a|s).

• The subset U D ⊂ U S of pure or deterministic stationary policies U D . A policy of this type can be viewed as a function from the states to the actions.

• The set U M of mixed strategies: A mixed strategy is identified with a probability γ over the set of pure stationary strategies. It can be considered as first choosing a pure stationary policy u with probability γ(u) and then keeping choosing forever the actions according to u. A general mixed strategy is a mixture of behaviorial strategy.

Occupation measure Often we encounter the notion of individual states in evolutionary games; but usually the population size at a particular state is fixed. In our case the choices of actions of an individual determine the fraction of time it would spend at each state. Hence the fraction of the whole population that will be at a given state may depend on the distribution of strategies in the population. In order to model this dependence we first need to introduce the expected average frequency f n,u (s) that an individual spends at a given state s when it follows a strategy u and its initial state at time 1 is n. Moreover, we define f n,u (s, a) the expected average frequency during which it is at state s and it chooses action a, f u,n = lim

T o ∞ 1 T T ∑ t=1 Q t (u) where Q(u) := [q(s ′ |s, a, u)] s ′ ,s,a .
We now define the equilibrium concept in the context of evolutionary games. A behaviorial stationary strategy u = (u 1 , . . . , u n ) with u i = (x i , y i ) is an evolutionary stable strategy (ESS) if for all strategy mut such that f mut = f u there exists ǫ mut > 0 such that ∑ s,a

( f u (s, a) -f mut (s, a))r(s, a, α ǫ ) > 0 where α ǫ = (1 -ǫ)α(u) + ǫα(mut).
Interpretation: Suppose that, initially, the population profile is α(u). Now suppose that a small group of mutants enters the population playing according to a different profile α(mut). If we call ǫ ∈ (0, 1) the size of the subpopulation of mutants after normalization, then the population profile after mutation will be ǫα(mut) + (1ǫ)α(u). After mutation, the average payoff of non-mutants will be given by ∑ s,a f u (s, a)r(s, a, α ǫ ). Analogously, the average payoff a mutant is ∑ s,a f mut (s, a)r(s, a, α ǫ ). That is, u is ESS if, after mutation, non-mutants are more successful than mutants. In other words, mutants cannot invade the population and will eventually get extinct. In case where the payoff function r is linear in last variable α (it is not the case in this model) then this definition is equivalent to α(u) is a strict symmetric Nash equilibrium or α(u) is a strictly better response than α(mut) given that the others mobiles mut. Note that if α is not linear, we can have α(ǫmut + (1ǫ)u) = ǫα(mut) + (1ǫ)α(u).

Stochastic modeling of the energy levels of Solar-powered battery

we assume that a battery has n + 1 energy states S = {0, 1, . . . , n}. The state 0 corresponds to the state Empty and the state n is the Full state of the battery. The other states 1, . . . , n -1 are intermediary states of the battery. We associate with each mobile a Markov Decision Process (MDP) which represents the transition probabilities between energy levels. Let X i t be the energy level of battery at time t. Given a stationary policy σ and a strategy of all the population α = (α t ) t≥1 , the transition probability of the energy level of battery is is described by the (first order, time-homogeneous) Markov process (X t ) where the transition probability law q which is given by

q sas ′ =        1 -R γ,s (a) -Q γ,s (a) if s ′ = s -1 R γ,s (a) if s ′ = s + 1 Q γ,s (a) if s ′ = s 0 otherwise , ∀1 ≤ s ≤ n -1, ∀ a s ∈ A(s), q nas ′ =    1 -Q γ,n (a) if s ′ = n -1 Q γ,n (a) if s ′ = n 0 otherwise , q 0as ′ =    γ if s ′ = 1 (1 -γ) if s ′ = 0 0 otherwise , where γ -→ R γ,s (a) ∈ [0, 1] is an increasing function ∀s, a with R 0,s (a) = 0, 0 ≤ R γ,s (a) + Q γ,s (a) ≤ 1.
The factor γ represents the probability to have a "good weather". If γ is zero, the state 0 is unique absorbing sate and expected lifetime of the battery is finite. For γ = 0, the Markov chain is irreducible i.e., there is only one class, that is, all states communicate with each other for stationary policy.

Battery-state dependent access control in solar-powered system

This subsection studies is a generalization of the random access game with unknown number of mobiles, finite state, and three strategies.The channel is ideal for transmission and all errors are due to collision. A mobile can transmit a packet using a power level among three available levels: transmit with high power P h , transmit with low power P l or does not transmit 0. We consider a general capture model where a packet transmitted by a mobile is received successfully when if and only if that mobile uses a transmission power which is greater than the power used by the others transmitters at that time slot. Given a population profile α = (x, y, 1xy) with 0 ≤ x, 0 ≤ y, x + y ≤ 1, the expected probability to have a successful transmission :

(i) When the mobile chooses P l , the reward is the probability that the others mobiles choose zero i.e no others mobiles transmit, i.e., r(s, P l , α) = ∑ k≥0

P(K = k) (1 -x -y) k = E((1 -x -y) K ) = G K (1 -x -y)
where G K is the generating function of K.

(ii) When the mobile chooses P h , the reward is the probability that no other mobiles transmit with the high power i.e.,

r(s, P

h , α) = E K K ∑ l=0 ( K l )(1 -x -y) l y K-l = E K (1 -x) K = G K (1 -x).
(5.32)

where ( K l ) is the binomial coefficient of K and l. (iii) When the terminal chooses 0, the reward is zero, i.e., r(s, 0, α) = 0 Since there are three strategies P h , P l and 0, the population aggressiveness can be described by a process (x t , y t ) t≥1 where x t (resp. y t ) is the fraction of the population using the high power level (low power) at time t. However, for each state s = 0, the action space becomes M := {(x, y), x ≥ 0, y ≥ 0, x + y ≤ 1}. A stationary policy of an user is a map β : S -→ M. The expected reward of a user when its battery is at the state s = 0 is then given by r(s, β, α) = x ′ s r(s, P l , α) + y ′ s r(s, P l , α), where β(s) = (x ′ s , y ′ s ) ∀s ∈ S.

Computing Equilibria and ESS Nash Equilibria and Pareto optimality

In this subsection, we study the existence and uniqueness of Nash equilibrium in different scenarios :

(iii)Mixed Equilibria: Any situation where at least one of the mobiles use the strategy P h , and other mobiles use an arbitrary mixed strategies, gives a mixed Nash equilibria. The allocation of payoff obtained in these mixed strategy are not Pareto optimal if at least one mobile chooses the strategy P h with positive probability.

In the figure 5.6, we plot the best response (power level) of a mobile for a given trajectory of a profile population, with P h = 10 and P l = 2. We observe that the best response is is to use a low power when the population is very aggressive (x = 1), and to use the high power in any state s = 0 when the population is less aggressive x < 1. In the figure 5.7, we plot the sojourn time of a mobile to stay in a state as function of its stationary strategy β = (x, y) with s = 0, Q s (0) = 0.95, Q s (P l ) = 0.65 and Q s (P h ) = 0.25. We observe that the sojourn time is decreasing function in aggressiveness of that mobile.

Power

Evolutionarily Stable Strategy

Under the condition γ > 0, we have the following result: Proof. Let mut such that α ′ (mut) = 1. For u = Ph the time average payoff is

F u (α ǫ ) = G K ǫ(1 -α ′ P h (mut)) n ∑ s=1 f Ph (P h ),
and

F mut (α ǫ ) = n ∑ s=1 f mut (s, mut s ) × mut P h G K (ǫ(1 -α ′ P h (mut))) + mut P l G K (ǫ(1 -α ′ P h (mut) -α ′ P l (mut))) .
Since the generating is increasing, G K (ǫ(1α ′ P h (mut)α ′ P l (mut))) ≤ G K (ǫ(1α ′ P h (mut)) with equality if and only α ′ P l (mut) = 0. We deduce that F mut (α ǫ ) is strictly greater F P h (α ǫ ) for all mut such that α ′ P h (mut) < 1.

The Figure 5.10 illustrates the expected long-term payoff versus the parameter γ. We took four states and R s,γ = γ, R s,γ = 0.3 * (1γ).

Power control in clouded weather

We assume now that the solar-powered system is in a clouded weather (γ = 0, "no sunlight") during a long period (clouded sky, raining time or due to the season). Power storage elements, such as supercapacitors (but finite in practice), in order to have energy available for later use has been proposed. Because of limited capacity (hence energy of the battery), the aggressive terminals (which use the high power) will be rapidly in the state 0 which becomes an absorbing state. The terminals with empty battery need an alternative solution such as external recharge or to buy a new battery. So there is an additional cost to survive in this situation. If the maximum lifetime of the battery (for example using the power "0") is finite then the power control in clouded weather can be modeled by the stochastic evolutionary game with total successful transmission before to reach to the absorbing state 0. Lemma 5.8.4.1. The total expected successful transmission during the lifetime of the battery under the strategy u is given by

1 1 -γ × 0 + n ∑ s=1 u s (P l )G K (1 -α P l -α P h ) + u s (P h )G K (1 -α P h ) 1 -Q 0,s (u s )
where Q 0,s (u s ) := u s (P h )Q 0,s (P h ) + u s (P l )Q 0,s (P l )

Proof. This says that the total reward total is the sum over the states of the expected successful transmission times the expected sojourn times spent in this state. The sojourn time in state s under the policy u s is 1 1 -Q 0,s (u s )

. This completes the proof.

The following Proposition holds in the stochastic evolutionary game with total reward. Proposition 5.8.4.2. (i) The strategy "stay quiet" (plays "0" in each state) cannot be an ESS. (ii) A necessary condition for the full aggressive strategy to be an ESS is G K (0

) = P(K = 0) = 0. Moreover if G K (1 -α P h ) n ∑ s=1 1 1 -Q 0,s (P h ) > max mut∈{P h ,P l } n n ∑ s=1 r(s, mut s , α) 1 -Q 0,s (mut s ) then Ph is an ESS. (iii) Similarly, if G K (1 -α P h ) n ∑ s=1 1 1 -Q 0,s (P l ) > max mut∈{P h ,P l } n n ∑ s=1 r(s, mut s , α) 1 -Q 0,s (mut s )
then Pl is an ESS.

Proof. The strategy "stay quiet" (play "0" in each state) cannot be an ESS because it is best reply to itself. Hence, the strategy not transmit can be invaded by mutations. If Ph is an ESS then

G K (0) n ∑ s=1 1 1 -Q 0,s (P h ) must be greater than G K (0) n ∑ s=1 1 1 -Q 0,s (P l )
. Since the power consumption is greater with P h than P l (P h > P l ), one has,

n ∑ s=1 1 1 -Q 0,s (P h ) < n ∑ s=1 1 1 -Q 0,s (P l ) .
This implies that G K (0) = P(K = 0) = 0. The other results are immediate by best response conditions.

Notes

Thanks to the renewable energy techniques, designing autonomous mobile terminal and consumer embedded electronics that exploit the energy coming from the environment is becoming a feasible option. However, the design of such devices requires the careful selection of the components, such as power consumption and the energy storage elements, according to the working environment and the features of the application. In this paper we have investigated power control interaction based on stochastic modeling of the remaining energy of the battery for each user in each local interaction. We have showed existence of equilibria and conditions for evolutionary stable strategies.

Wardrop equilibria in nonatomic stochastic power control games

Consider a large number of mobiles terminals controlling their transmission power in a base station. Each mobile has an amount of energy E when its battery is new (typically it is the case if the battery is new or if the battery is completely recharged). Each mobile implements a power control policy where the transmission power is allowed to depend on the energy level (state) of its battery. The available action depends on the state of the battery. Given the remaining energy of its battery, the mobile have to choose the optimal power level in terms of reward and cost. One of the important element for each mobile is its instantaneous throughput which can be characterized as a function of the signal to interference plus noise ratio (SINR) at the base station where he transmits. The battery is replaced or recharged only when it is completely empty. The cost of new battery or recharge cost is C. The new battery has the same energy of E. The mobile have to control both the power consumption as well as the time at which the batteries are changed or recharged. At each time slot, each mobile is faced to a large number of interacting users which transmit at the same base station. Each battery life-time game corresponds to a stochastic power control game with finite horizon in expectation (absorbing state of battery when the energy is very small). Our aim here is to find the power consumption policy at each state such that all users achieve as high throughput as possible, minimum guarantee (e.g. QoS requirement thresholds) but also to control the battery-state.

Potential Function and Optimization Problems

Lemma 5.9.0.1. The function V : R + -→ R defined by

V(x) = (C l -C h )x s + log 1 + ρ h x s + (1 -x s )ρ l σ 2 is a potential function. Proof. V(x) = -(C h -C l )x s + x s 0 ρ h -ρ l 1 σ 2 + ρ h t + (1 -t)ρ l dt where ρ h = P h 1 -Q s (h) , ρ l s = P l 1 -Q s (l)
.

Interpretation of the Potential Function

The above can be seen as the expression of capacity minus energy consumption. Lemma 5.9.0.2. The potential function is strictly concave.

Proof. The potential function is continuously twice differentiable, and the second derivative is given by

V ′′ (x) = - (ρ h -ρ l ) 2 (σ 2 + ρ h x + (1 -x)ρ l ) 2 < 0.
We conclude V is strictly concave (it can be seen also as a sum of strict concave function and the concave function). Proposition 5.9.0.3. The stochastic power control game has a unique Wardrop equilibrium.

Proof. The existence of Wardrop equilibrium is guaranteed by maximization of the potential function. Since V is a C 1 concave function, V has a maximum in compact convex set [0, 1] and maximizers are Wardrop equilibria. The uniqueness of the equilibrium is immediate from the strict concavity of the potential function given by Lemma 5.9.0.2.

Convergence to Wardrop equilibrium

Proposition 5.9.0.4. The unique equilibrium is asymptotically stable under the θ-Smith dynamics. Moreover, for any initial strategy distribution, the dynamics to converge to the Wardrop equilibrium.

Different types of renewable energy

Power control in wireless networks has became an important research area. Since the technology in the current state cannot provide batteries which have small weight and large energy capacity, the design of tools and algorithms for efficient power control is crucial.

Thanks to the renewable energy techniques, designing autonomous mobile terminal and consumer embedded electronics that exploit the energy coming from the environment is becoming a feasible option. However, the design of such devices requires the careful selection of the components, such as power consumption and the energy storage elements, according to the working environment and the features of the application.

Menache and Altman have studied in (142) a battery-energy dependent power control with finite number of mobiles as a dynamic non-cooperative game with power cost assumption. In this model we consider a stochastic population game approach with dynamic rechargeable battery based on renewable energy. Environmental energy is becoming a feasible alternative for many low-power systems, such as wireless sensor/mesh networks. Nevertheless, environmental energy is an exciting challenge. Because of the limited amount of energy over time, the power provided is unpredictable. Power storage elements, such as rechargeable batteries or supercapacitors, in order to have energy available for later use has been proposed. Alternative energy as solar, wind, or nuclear energy, that can replace or supplement traditional fossil-fuel sources, as oil, and natural gas is needed. We refer the reader to [START_REF] Niyato | Sleep and wakeup strategies in solar-powered wireless sensor/mesh networks: Performance analysis and optimization[END_REF] for advantageous to use renewable energy in broadband wireless networks such as Wi-Fi, WiMax or mesh networks.

We consider several class of large number of mobiles terminals controlling their transmission power and a distributed base stations. The mobiles with the same type of renewable energy (wind, solar, hydro) are in the same class or subpopulation. Each mobile of the subpopulation p has an amount of energy E p when its battery is at the full state. Each mobile implements a power control policy where the transmission power is allowed to depend on the energy level (state) of its battery. The available action (reachable base stations) depends on the state of the battery. Given the remaining energy of its battery, the mobile have to choose the optimal power level. One of the important element for each mobile is its instantaneous throughput which can be characterized as a function of the signal to interference plus noise ratio (SINR) at the base station where he/she transmits. The battery is recharged by different techniques of renewable energy (solar-power, wind-power etc). The mobile have to control both the power consumption as well as the level of its battery and its throughput. Each mobile is faced to a non-cooperative stochastic game with individual states with many others mobiles which transmit at the same base station or at the same range. The goal of a terminal is to find jointly the power levels and the base stations such that the terminal achieves as high payoff as possible, minimum guarantee (e.g. QoS requirement thresholds) but also to control the battery-state.

Battery-state transition

We consider the energy reserve of the battery type p, (X p t ) t≥1 and power level management as a Markov decision process. For each state y = 0, the action space is A p (y) with at least two elements, and A p (0) has at most one element (empty or singleton). Given a stationary policy σ and a strategy of all the populations the change in energy reserves of the battery type p is described by the (first order, time-homogeneous) Markov process (X p t ) with the transition law q p . ∀y = 0, n p , ∀a, the probability of transition q p (X

p t+1 = y ′ |X p t = y, a) is expressed as          1 -R p γ p ,y (a) -Q p γ p ,y (a) if y ′ = y -1 R p γ p ,y (a) if y ′ = y + 1 Q p γ p ,y (a) if y ′ = y 0 otherwise , q p (X p t+1 = y ′ |X p t = n p , a) =      Q p γ p ,n p (a) if y ′ = n p -1 1 -Q p γ p ,n p (a) if y ′ = n p 0 otherwise , and
q p (X p t+1 = y ′ |X p t = 0, 0) =    γ p if y ′ = 1 (1 -γ p ) if y ′ = 0 0 otherwise , where ∀ y, a, γ p -→ R p γ p ,y (a) ∈ [0, 1] is an increasing function with R p 0,y (a) = 0, 0 ≤ R p γ p ,y (a) + Q p γ p ,y (a) ≤ 1.
The factor γ p represents a function of the probability to have a "good weather"(for example, the sun for the solar-power battery, the wind for wind-power battery) and the probability for battery of type p to go from state 0 to state 1. If γ p is zero, the state 0 is absorbing. For γ p = 0 is the chain is communicating. Note that each user controls the transition state of its battery: q p is independent of the decision of the other mobiles.

Reward

We focus on utility function based on a simplified version of the signal to noise plus interference ratio (SINR). The battery-state have the property that more energy is available in high state.

Hence, that set of powers in y + 1 contains the set of power available in y. For example,

∅ ⊂ A p (0) = {pow p 0 } ⊂ A p (1) = {pow p 0 , pow p 1 } ⊂ A p (2) = {pow p 0 , pow p 1 , pow p 2 } ⊂ . . . A p (n p ) = {pow
= (λ 1 , λ 2 , λ 3 ) is SINR p y (a, x; λ, BS) = ag p p (ǫ 2 +(λ 1 -x 0 ) 2 +(λ 2 -y 0 ) 2 +(λ 3 -z 0 ) 2 ) α 2 N 0 + κI own (x p ) + κI other (x -p ) , a ∈ A p (y), p ∈ P, y ∈ Y p = {0, 1, 2, . . . , n p } where I own (x p ) = ∑ y,b bg pp x p (y, b)h pp I other (x -p ) = ∑ k =p ∑ y,b g kp bx k (y, b)h kp ,
where

h kp = λ∈D dµ k,BS (ǫ 2 + (λ 1 -x 0 ) 2 + (λ 2 -y 0 ) 2 + (λ 3 -z 0 ) 2 ) α 2
x j (y, a) is the fraction of the sub-population j in state y with the power level a, N 0 is the power of the thermal background noise, µ p,BS is the distribution of mobiles (in the 3-dimensional space) with the battery type p around the base station BS, D ⊆ R 3 is the domain (geographical placement of base stations and mobiles) and α is the path-loss and κ is the inverse of the processing gain of the system, it weights the effect of interferences, depending on the orthogonality between codes used during simultaneous transmissions. The coefficient κ is equal to 1 in a narrow band system, and is smaller than 1 in a broadband system that uses CDMA. The instantaneous expected reward r p y,a (x) of an user in state y is expressed as

λ∈D f SINR p y (a, x, λ, BS) dµ p (λ)
where f is a non-decreasing function with f (0) = 0. ǫ is a positive parameter (to eliminate of continuity problem at zero) and the g ij are positive gain parameters. The 3-dimensional vector (x 0 , y 0 , z 0 ) describes the position of the base station BS in R 3

Computing the interference term in presence of continuum of users

In order to compute explicitly the SINR term, we first need the following lemma: Lemma 5.10.0.5.

ν ≥ 1, b ν = +∞ 0 1 (1 + x 2 ) ν dx =      π 2 if ν = 1 √ π 2 Γ(ν -1 2 ) Γ(ν) if ν > 1.
where Γ is the Euler function Γ(x) = +∞ 0 e -t t x-1 dt.

Proof. For ν = n a positive integer, the polynomial (1 + z 2 ) n has two zeros z = ±i each zero with the order n.

Consider the circuit C R ′ = [-R ′ , R ′ ] ∪ {R ′ e iθ , 0 ≤ θ ≤ π}. Since the complex function z ∈ C -→ 1 (1 + z 2
) n has no zero on the circuit C R ′ , Using residue's theorem of complex analysis, we obtain the following result:

x≥0 1 (1 + x 2 ) n dx = πiRes(ξ(z), i).
The residue of ξ around z = i, a pole of order n, can be found by the formula:

Res(ξ, i) = 1 (n -1)! lim z-→i ( d dz ) n-1 [(z -i) n ξ(z)] Thus, b n = √ π 2 Γ(n -1 2 ) Γ(n)
. We then use the extension of the Euler function Γ on the positive real axis.

From the lemma 5.10.0.5, we derive immediately that, n ≥ 2,

+∞ 0 x 2 (1 + x 2 ) n dx = b n-1 -b n Proposition 5.10.0.6. h jp = 4π 2 ǫ 2α-3 (b α 2 -1 -b α 2 )
Proof. Using spherical coordinates from cartesian coordinates by the transformation

   λ 1 = r sin θ cos φ λ 2 = r sin θ sin φ λ 3 = r cos θ
and the volume element r 2 dr sin θdθdφ, one has,

h jp = 4π 2 +∞ 0 r 2 (ǫ 2 + r 2 ) α 2 dr = 4π 2 ǫ 2α-3 +∞ 0 r 2 (1 + r 2 ) α 2 dr i.e h jp = 4π 2 ǫ 2α-3 (b α 2 -1 -b α
2 ) Proposition 5.10.0.7. The highest payoff that a mobile with the battery type p can obtain against any strategies of others mobiles in the one-shot power control game is given by

vp y = D f     u p y g p p (ǫ 2 +(λ 1 -x 0 ) 2 +(λ 2 -y 0 ) 2 +(λ 3 -z 0 ) 2 ) α 2 N 0 + ∑ j ∑ y h jp u j y m j y g jp     dµ p
where u p y is the maximum power level available in the battery-type p in state y.

Proof. Since the payoff decreases when the others players increases their power levels (in average), the minmax point is obtained when they uses their high powers. The maximum payoff that a mobile with the battery type p can obtain against any strategies of others mobiles is then given by vp

y = max a∈A p (y) D f    ag p p (ǫ 2 +(λ 1 -x 0 ) 2 +(λ 2 -y 0 ) 2 +(λ 3 -z 0 ) 2 ) α 2 N 0 + ∑ j ∑ y h jp u j y m j y g jp    dµ p = D max a∈A p (y) f    ag p p (ǫ 2 +(λ 1 -x 0 ) 2 +(λ 2 -y 0 ) 2 +(λ 3 -z 0 ) 2 ) α 2 N 0 + ∑ j ∑ y h jp u j y m j y g jp    dµ p = D f    max a∈A p (y) ag p p (ǫ 2 +(λ 1 -x 0 ) 2 +(λ 2 -y 0 ) 2 +(λ 3 -z 0 ) 2 ) α 2 N 0 + ∑ j ∑ y h jp u j y m j y g jp    dµ p .
This completes the proof.

Notes

Our stochastic population game model with multi-class of of users can be extended to the local common resources states case. The model including the state of local resources allows us to take into account the local resources for which users are interacting cooperatively or noncooperatively.

In the third part of this manuscript we develop a new class of mean field games. Using stochastic processes results we will be able to drop the assumptions of stationary regime and derive weak convergence of the population profile process when the size of population goes to infinity. The two chapters show that in general, the validity of standard analysis such as fixed point analysis, stationarity and ergodicity conditions need to be justified. The chapters are based to our publications (210; 208; 218).

Part III

Mean Field Limits

Chapter 6

Mean field asymptotics of population games

We consider evolutionary games in finite population, in which each individual in a interacts with other randomly selected players. The types and actions of each player in an interaction together determine the instantaneous payoff for all involved players. They also determine the transition probabilities to use actions. We provide a rigorous derivation of the asymptotic behavior of this system as the size of the population grows. We show that the large population asymptotic of the microscopic model is equivalent to a macroscopic evolutionary game in which a local interaction is described by a single player against a population profile. We derive various classes of evolutionary game dynamics.

Introduction

We consider a population with finite number of players in which a small frequent interactions occurs among random number of players that are selected at random. Each player is thus involved in infinitely many interactions with other randomly selected players. Each interaction in which a player is involved can be described as one stage of an evolving game. The actions of the players at each stage determine an immediate payoff (also called fitness in behavioral ecology) for each player as well as the transition rates of a Markov chain associated with each player. The transition rate is determined of the change of actions and the system state. This model extends the basic pairwise interaction model in evolutionary games by introducing a random number of interacting player and the rate of transition for several players which has opportunities to changes its action. At each time slot, a one-shot game with unknown number of players replaces the matrix games, Instead of a choice of a (possibly mixed) action, a player is now faced with the choice of decision rules (called strategies) and revision of theses strategies that determine what actions should be chosen at a given interaction for given present and past observations. This model with a finite number of players, called mean field interaction [START_REF] Benaim | A Class Of Mean Field Interaction Models for Computer and Communication Systems Performance Evaluation[END_REF], become more complex to analyze if the number of players grows (because a huge action profile space is required to describe all the of players). Then taking the asymptotic as the number of players grows to infinity, the whole behavior of the population is replaced by a deterministic limit that represents the system's state or population profile, which is fraction of the population at each type that use a given action.

In this paper we study the asymptotic behavior of the system in which the population profile evolves in time. For large number of players, under mild assumptions (see Section 6.2), the mean field converges to a deterministic measure that satisfies a non-linear ordinary differential equation for a fixed stationary strategy. We show that the mean field interaction is asymptotically equivalent to an evolutionary game with random number of interacting players. At a given time, any given player sees an equivalent game against a collective of players whose state evolves according to an ordinary differential equation (ODE) which we explicitly compute. In addition to providing the exact limiting asymptotic, the ODE approach provides tight approximations for fixed large N of the random process. We derive several evolutionary game dynamics and learning-based dynamics such as Smith dynamics, replicator dynamics, logit dynamics, Brown-von Neumann-Nash dynamics etc. We give sufficient conditions for convergence to equilibria and asymptotic results for non-convergent dynamics. We then apply to Aloha-based protocols with power control, evolution of technologies and resource selection in heterogenous networks.

The setting

Consider the following interaction model consisting of :

• One population of players with size N ∈ N.

• Each player has its own state. A state of a player is has two components: the type and the action. The state of player j at time t is denoted by X N j (t) = (θ j , A N j (t)). Each player j has to make type-dependent sequential decisions in interaction where he will be involved. For each type θ ∈ Θ (finite), there is finite set of possible actions a ∈ A θ .

Revision of strategies

At every time slot, a random number of players B N (t) is picked at random; then, given that ♯B N (t) = k, an ordered sequence of k players (j 1 , j 2 , . . . , j k ) is picked at random uniformly among the N(N -1)(N -2) . . . (Nk + 1) possible ones. Then, the collection of k players does a transition independent of the past according to

L N (x 1 ,...,x k ;x ′ 1 ,...,x ′ k ) (m, k) := P X N j l (t + 1 N ) = x ′ l , l = 1, . . . , k |ζ N , ♯B N (t + 1 N ) = k ζ N := {X N j l (t) = x l , l = 1, . . . , k, M N (t) = m} J N k (m) := P(♯B N (t) = k|M N (t) = m) (6.4)

Instantaneous payoffs

The instantaneous payoff of a player is given by:

r N (x, m) = E B r N,B (x, M N |M N (t) = m = N ∑ k=1 r N,k (x, m)J N k (m) with r N,k (x, m) = ∑ x 2 ,...,x k g N,k (x, x 2 , . . . , x k ) Nm x 2 -1 x=x 2 N -1 × Nm x 3 -1 x 3 =x -1 x 3 =x 2 N -2 . . . Nm x k -∑ k-1 l=2 1 x k =x l -1 x k =x N -k
where g N,k (x, x 2 , . . . , x k ) is the payoff of the player state in state x and the k -1 other players respectively with type-action x 2 = (θ 2 , a 2 ), . . . , x k = (θ k , a k ).

Convergence to differential equation

The drift is defined as the expected change in M N in one time slot:

f N (m, t) = E M N (t + 1 N ) -M N (t)|M N (t) = m = ∑ k≥1 f N,k (m)J N k (m) 6.3. Convergence to differential equation with f N,k (m) = E M N (t + 1 N ) -M N (t)|M N (t) = m, ♯B N (t + 1 N ) = k = ∑ x ′ 1 ,...,x ′ k ∑ x 1 ,...,x k m x 1 Nm x 2 -1 x 1 =x 2 N -1 × Nm x 3 -1 x 3 =x 1 -1 x 3 =x 2 N -2 . . . × Nm x k -∑ k-1 l=1 1 x k =x l N -k × 1 N L N x;x ′ (m, k) k ∑ l=1 ( e x ′ l -e x l )
where x l = (θ l , a l ); x ′ l = (θ l , a ′ l ). We are interested in the case where the number N of players is large compared to the expected number of possible number of simultaneous interacting players.

The following diagram illustrates the problem.

X N j (t), M N [u, m 0 ](t) t -→ +∞ -X N j , ?̟ N [u, m 0 ] X j (t), ?m[u, m 0 ](t) N -→ +∞ ? t -→ +∞ -??
N -→ +∞ ? We study the mean field convergence i.e lim N-→∞ M N weakly or better almost surely where for any time t, the sequence of measures M N (t) ∈ ∆(S) satisfying some dynamical systems. Necessarily this requires the weak convergence of M N (0) to m(0). For B ⊆ S, define

M N (t)(B) := 1 N N ∑ j=1 δ {X N j (t)∈B}
We assume that the following limits exist as N goes to infinity. (a) L N

x;x ′ (m, k) → L x;x ′ (m, k), (b) r N (x, a, m) → r(x, a, m) and (c) J N k (m) → J k (m). Proposition 6.3.0.1. For every k, the random measure M N,k = 1 N(N -1)(N -2) . . . (N -k + 1) ∑ j 1 ,j 2 ,...,j k distinct δ (X N j 1 ,...,X N j k
) converge weakly to a deterministic measure on S k . In particular for k = 1 the population profile • the instantaneous payoff of a player with the type θ and action a is

M N,1 = M N = 1 N N ∑ j=1 δ X N
r θ,a (m(t)) := lim N-→∞ r N θ,a (m(t)|X N j = (θ, a), M N (t) = m(t))
The Proposition 6.4.0.3 says that for large N, all interactions can be replaced to any one player with an average or effective interaction and an appropriate payoff function. This reduces any large population problem into an effective one-player problem.

At the asymptotic regime we have the following: At each time t, a random number b(t) = k of players has opportunity to revise their strategies. The expected rate to move from x 1 , . . . ,

x k to x ′ 1 , . . . , x ′ k is ∑ k≥1 L x,x ′ (m(t), k)J k (m(t))

Equilibrium state analysis

We say that a population profile (the vector of frequencies) m * is a stationary equilibrium state if for all vector m of frequencies, the following variational inequality holds: Recall that the projection map on ∆ d which is convex and closed set is characterized by:

z ∈ R d , z ′ = Π ∆ d z ⇐⇒ z ′ -z, m -z ′ ≥ 0, ∀m ∈ ∆ d Thus, m * = Π ∆ d ( m * + ςr( m * ))
According to Schauder's fixed point theorem, given a map α : ∆ d -→ ∆ d , with α continuous, there is at least one m ∈ ∆ d , such that m = α( m). Observe that since the projection Π ∆ d and (I + ς r), are both continuous, Π ∆ d (I + ς r) is also continuous by composition.

It follows from convexity and compactness of ∆ d and the continuity of Π ∆ d (I + ς r) that the Brouwer-Schauder fixed point theorem can be applied to the map Π ∆ d (I + ς r). We conclude that at least one stationary equilibrium state exists.

Dynamics of evolving games

The general dynamics obtained from the mean field interaction is given

d dt m θ,a (t) = f θ,a (m(t))
where the drift limit can be expanded as

f y (m(t)) = ∑ k≥1 J k (m(t)) f k y (m(t)), f k y (m(t)) = ∑ x 1 ,...,x k k ∏ l=1 m x l (t) k ∑ j=1 η j x;y (m(t), k) -m y (t) k ∑ j=1 ∑ x -j ( k ∏ l=1,l =j m x l (t))η t y,x -j , where y = (θ, a), η j x;y (m(t), k) = ∑ x ′ -j L x;(y,x ′ -j ) (m(t), k), (y, x ′ -j ) = (x ′ 1 , x ′ 2 , . . . x ′ j-1 , y, x ′ j+1 , . . . , x ′ k ) and η t (y,x -j ) (m(t), k) = ∑ x ′ L (y,x -j );x ′ (m(t), k).

Single player selected per time slot

Suppose now that at each time slot, only one player between the N is randomly selected and has a chance to change its strategy, i.e. ♯B N = 1 w.p. 1.

Thus H2 and H3 are automatically satisfied. The resulting ODE becomes

d dt m x (t) = ∑ x ′ m x ′ L x,x ′ (m) -m w ∑ x ′ L x,x ′ (m) The term ∑ x ′ m x ′ L x ′ ,x (
m) is the incoming flow in to x ′ and the outgoing flow from x is m x ∑

x ′ L x,x ′ (m).
We then obtain a large class of standard evolutionary game dynamics.

γ-Smith dynamics γ-Smith dynamics are obtained for L θ,b;θ,a ( m) = λ max(0, r θ,a ( m) -r θ,b ( m)) γ d dt m θ,a = λ ∑ b m θ,b max(0, r θ,a (m) -r θ,b (m)) γ -λm s,a ∑ b max(0, r s,b (m) -r s,a (m)) γ
Replicator dynamics Replicator dynamics is obtained for

L θ,b;θ,a ( m) = λm θ,a max(0, r θ,a (m) - r θ,b (m)) d dt m θ,a = λ ∑ b m θ,b m s,a max(0, r θ,a (m) -r θ,b ( m)) -λm θ,a ∑ b m θ,b max(0, r θ,b (m) -r θ,a (m)) = λm θ,a r θ,a (m) -∑ b m θ,b r θ,b (m)
The following result gives possible rest points for linear payoffs (in the population profile) of the replicator dynamics even when the ODE does not necessarily converge.

Convergence of time averages under the replicator dynamics

Suppose that the replicator dynamics has a solution (m θ,a (t)) in the relative interior of the (d -1)-simplex i.e there is an ǫ > 0 such that ǫ ≤ m θ,a (t) for all t ≥ 0. Then

L θ,b;θ,a (m) = λ ∑ b m θ,b max(0, r θ,a (m) -∑ b m θ,b r θ,b (m)) d dt m θ,a = λ ∑ b m θ,b max(0, r θ,a (m) -∑ b m θ,b r θ,b (m)) -λm θ,a ∑ b max(0, r θ,b (m) -∑ b m s,b r θ,a (m))
Note that in general the trajectories of the mean field dynamics need not to converge.

Dynamics for pairwise local interaction

Suppose now that at each time slot, exactly two players among N are randomly selected i.e. ♯B = 2 w.p. 1. Here too H2 and H3 are automatically satisfied. Then the mean dynamics is given by

∑ x ′ 1 ,x ′ 2 ∑ x 1 ,x 2 m x l m x 2 L x 1 ,x 2 ;x ′ 1 ,x ′ 2 ( m) 2 ∑ l=1 ( e x ′ l -e x l ) (6.6)
where

L x 1 ,x 2 ;x ′ 1 ,x ′ 2 ( m)
is the probability that the first object in interaction movers to state x ′ 1 and the second one to state x ′ 2 given that the first was in state x 1 , the second in state x 2 , and the system state was m just before the interaction.

We can expand Equation (6.6) as

d dt m x = ∑ x 1 ,x 2 ,x ′ 2 m x 1 m x 2 L x 1 ,x 2 ;x,x ′ 2 ( m) + ∑ x 1 ,x 2 ,x ′ 1 m x 1 m x 2 L x 1 ,x 2 ;x ′ 1 ,x ( m) -m x ∑ x 2 ,x ′ 1 ,x ′ 2 m x 2 L x,x 2 ;x ′ 1 ,x ′ 2 ( m) -m x ∑ x 1 ,x ′ 1 ,x ′ 2 m x 1 L x 1 ,x;x ′ 1 ,x ′ 2 ( m) = ∑ x 1 ,x 2 m x 1 m x 2 η 1 x 1 ,x 2 ;x ( m) + ∑ x 1 ,x 2 m x 1 m x 2 η 2 x 1 ,x 2 ;x ( m) -m x ∑ x 2 m x 2 η t x,x 2 ( m) -m x ∑ x 1 m x 1 η t x 1 ,x ( m) with η 1 x 1 ,x 2 ;x = ∑ x ′ 2 L x 1 ,x 2 ;x,x ′ 2 , η 2 x 1 ,x 2 ;x = ∑ x ′ 1 L x 1 ,x 2 ;x ′ 1 ,x , η t x 1 ,x 2 = ∑ x ′ 1 ,x ′ 2 L x 1 ,x 2 ;x ′ 1 ,x ′ 2 (6.7)
The term η j x 1 ,x 2 ;x (resp. η t x 1 ,x , η t x,x 1 ) corresponds to the rate of transfer of a player with x 1 = (θ 1 , a 1 ) to switch into x = (θ, a) (resp. from x) after an encounter with another player with x 2 = (θ 2 , a 2 ).

Equilibrium and rest point

Proposition 6.4.5.1 (sufficient condition). Suppose that the drift limit f satisfies

f (m) = 0 =⇒ f (m), r(m) = ∑ s,a r s,a (m) f s,a (m) > 0 (6.8) where f y (m) = ∑ k≥1 J k (m) f k y (m), f k y (m) = ∑ x 1 ,...,x k k ∏ l=1 m x l k ∑ j=1 η j x;y (m, k) -m y k ∑ j=1 ∑ x -j ( k ∏ l=1,l =j m x l )η t y,x -j , where y = (s, a), η j x;y (m, k) = ∑ x ′ -j L x;(y,x ′ -j ) (m, k), (y, x ′ -j ) = (x ′ 1 , x ′ 2 , . . . x ′ j-1 , y, x ′ j+1 , . . . , x ′ k ) and η t (y,x -j ) (m, k) = ∑ x ′ L (y,x -j );x ′ (m, k).
Then any stationary equilibrium state is a rest point of ODE.

Proof. Suppose that the condition

f (m) = 0 =⇒ f (m), r(m) = ∑ y r y (m) f y (m) > 0
holds. Let m * be a stationary equilibrium state (it exists by Proposition 6.4.1.1). We want to prove that f (m * ) = 0. Since, m * is a stationary equilibrium state, it solves the variational inequality problem

∀h ∈ {z = (z s,a ) s,a , ∑ s,a z s,a = 0}, h, r(m * ) ≤ 0.
Define the vector x = (x s,a ) s,a such that x s,a = f s,a (m * ). By construction, the drift limit satisfies

∑ s,a f s,a (m * ) = 0. Thus, 0 ≥ x, r(m * ) = f (m * ), r(m * ) .
This condition is equivalent to f (m * ) = 0.

The condition given by Equation (6.8) is called positive correlation (168) condition. In the one object per time slot model with single internal state case, it has been shown that (see [START_REF] Sandholm | Population Games and Evolutionary Dynamics[END_REF]) that the replicator dynamics, the Brown-von Neumann-Nash dynamics, θ-Smith dynamics or general pairwise comparison dynamics are positively correlated. As a corollary, the set of rest points of all these dynamics contains the set of equilibrium states. Proposition 6.4.5.2. Suppose that the polymatrix of transition L satisfies

L s,a,x -j ;s,b (m) > 0 ⇐⇒ a, b ∈ A s , r s,a (m) < r s,b (m)
for each j, x -j and m. Then says that if a is not used if a is not a good strategy. We now proof the first part of the Proposition. Suppose that m such that f (m) = 0 with

f y (m) = ∑ k≥1 J k (m) f k y (m), f k y (m) = ∑ x 1 ,...,x k k ∏ l=1 m x l k ∑ j=1 η j x;y (m, k) -m y k ∑ j=1 ∑ x -j ( k ∏ l=1,l =j m x l )η t y,x -j , One has, ∑ y r y (m) f y (m) = ∑ y ∑ k≥1 J k (m) f k y (m)r y (m) = ∑ k≥1 J k (m) ∑ y f k y (m)r y (m) (6.9) We can rewrite ∑ y r y (m) f y (m) as ∑ k≥1 J k (m) ∑ y [A k y -B k y ] where A k y = k ∑ j=1 ∑ x 1 ,...,x k ∑ x ′ -j ( k ∏ l=1 m x l )L x j ,x -j ;y,x ′ -j (m, k)r y (m), B k y = m y   k ∑ j=1 ∑ x -j ( k ∏ l=1,l =j m x l )   ∑ (x ′ j ,x ′ -j ) L y,x -j ;x ′ j ,x ′ -j (m, k)r y (m)     (6.10) = k ∑ j=1 ∑ x -j ∑ (x ′ j ,x ′ -j ) (m y k ∏ l=1,l =j m x l )L y,x -j ;x ′ j ,x ′ -j (m, k)r y (m) (6.11)
By interchanging the roles of y and x ′ j we obtain that

B k y = k ∑ j=1 ∑ (x j ,x -j ) ∑ x ′ -j ( k ∏ l=1 m x l )L x j ,x -j ;y,x ′ -j (m, k)r x j (m) (6.12) Then, ∑ k≥1 J k (m) ∑ y f k y (m)r y (m) is exactly ∑ k≥1 k ∑ j=1 ∑ (x j ,x -j ) ∑ (y,x ′ -j ) ( k ∏ l=1 m x l )J k (m)L x j ,x -j ;y,x ′ -j (m, k) max[0, r y (m) -r x j (m)].
Since there exists at least one component y such f y (m) = 0, and

L x j ,x -j ;z,x ′ -j (m, k)[r z (m) -r x j (m)] = L x j ,x -j ;z,x ′ -j (m, k) max[0, r z (m) -r x j (m)] ≥ 0
for all z, we conclude there exists k ≥ 1 such that L x j ,x -j ;y,w ′

-j (m, k) max[0, r y (m) -r x j (m)] > 0.
This implies that ∑ y r y (m) f y (m) > 0.

Cooperative vs Competitive mean field limit

We say that the mean dynamics is cooperative

(resp. competitive) if all w = w ′ d dm w ′ f w (m) ≥ 0 (resp. d dm w ′ f w (m) ≤ 0).
In the bidimensional (three strategies) case it has been shown in [START_REF] Hofbauer | Evolutionary Games and Population Dynamics[END_REF] that the trajectories of cooperative game dynamics converge starting from almost any initial condition in the simplex. Example of cooperative mean dynamics includes dynamics of supermodular population games [START_REF] Sandholm | Population Games and Evolutionary Dynamics[END_REF]. Generic convergence of cooperative (resp. competitive) dynamics with strict inequality can be found in [START_REF] Smith | Monotone Dynamical Systems: An Introduction to the Theory of Competitive and Cooperative Systems[END_REF].

Dynamic equilibrium

• In many cases in evolving games, it is known that the trajectories of evolutionary game dynamics can be chaotic, may have cycle limits and may not converge. Under several evolutionary game dynamics, the expected payoff obtained in the cycle is not so far from candidate to be a "static" equilibrium payoff. This suggests a time dependent equilibrium approach in evolving games.

• a trajectory is a dynamic equilibrium if the time average payoff under this trajectory leads to an equilibrium payoff

Example: the trajectory of RSP leads to a "dynamic equilibrium" but it cycles around the Shapley triangle.

m : [t 0 , ∞[-→ ∆ n is a dynamic equilibrium if for any other trajectory, there exists t 1 sufficiently large such that lim inf

T 1 T -t 1 T t 1 m(t) -m ′ (t), r(m(t)) dt ≥ 0
Corollary: a "static" equilibrium state m * is a special case of dynamic equilibrium where the trajectory is t -→ m * (constant function).

A Spatial Non-Reciprocal Random Access Model

Random Access Control algorithms have played an increasingly important role in the development of wired and wireless networks and the performance and stability of these algorithms, such as slotted-Aloha, Carrier Sense Multiple Access (CSMA) is still an open problem. Distributed medium access control, starting from the first version of Abramson's Aloha to the most recent algorithms used in IEEE802.11, have enabled a rapid growth of wireless networks. They aim at efficiently and fairly sharing a resource among users even though each user must decide independently (eventually after receiving some messages or listening, sensing channels) when and how to attempt to use the resource. Random access algorithms have generated a lot of research interest, especially recently in attempts to use multi-hop wireless networks to provide high-speed access to the internet with low-cost and low-energy consumption.

In this section, we restrict our attention to wireless networks, where the resources are receivers, base station or access points and where users interact because of interference, i.e., interfering users cannot transmit simultaneously. There is a collision if another user (mobile) transmits with a greater power level at the same range of the receiver. Motivated by the interest of evolving dense networks, game theory evolving was found to be an appropriate framework to apply in networks. We provide asymptotic analysis of spatial random access based on mean field interaction where interfering users share resources placed on a undirected graph using an 6.5. A Spatial Non-Reciprocal Random Access Model Aloha-type access control. We consider several classes of users. Users are spatially distributed and the arrival processes around each vertex are assumed to be independent. The interactions between users are non-reciprocal in the sense that the set of users causing interference differ from the set of those suffering from these interferences. The scenario has been introduced and analyzed in [START_REF] Tembine | Evolutionary games with random number of interacting players applied to access control[END_REF] in the context of evolutionary games.

Model Description

We consider a spatial non-reciprocal random access model described as follows:

• Let define a system composed of an undirected graph G = (V, E) with finite vertices V = {1, 2, . . . , K}, and E is set of edges, and many classes of users distributed in 3D. For each vertex i ∈ V, the one-hop neighborhood of i is denoted by

N i = {i} ∪ {j ∈ V, (i, j) ∈ E}.
The set N i corresponds to the set of 1-adjacent nodes to i. .

B i t ∼ Bin   W i t , u i ∑ j∈N i W j t  
• There is a successful transmission at vertex i if only one of users in N i transmit at time t.

W i t+1 = W i t -1 {B i t =1} ∏ j∈N i \{i} 1 {B j t =0} + A i t
Given (W 1 t , . . . , W K t ), the random variable B 1 t , . . . , B K t satisfy the decoupling conditions. (W 1 t , . . . , W K t ) ∈ N K is an irreducible Markov chain.

Drift

Let us look at the expected change in W in one slot when all users use the strategy u 1 , . . . , u K . Define f (u 1 , . . . , u K , x) = E[W t+1 -W t |W t = x] (6.13)

The drift is given by

f i (u 1 , . . . , u K , x) =      α i - u i x i ∑ j∈N i x j (1 - u i ∑ j∈N i x j ) x i -1 ∏ j∈N i \{i} (1 - u j ∑ j∈N i x j ) x j if x i > 0 α i if x i = 0
For any sequence (x i t ) such that x i t -→ +∞, define the process

M x t k = W(⌊k ∑ i x i t ⌋) ∑ i x i t . M x k ∈ D([0, T], R K )
where D([0, T], R K ) is the space of cadlag or RCLL ( right continuous with left limits) functions from the compact interval [0, T] to R K endowed with the Skorokhod topology.

Let (x i t ) t be a sequence going to infinity. Then the collection {(M x k ) k , k ∈ N} has a compact close in the Skorokhod topology. Any limit m of subsequence of M x k is Lipschitz continuous.

If a mean field limit m satisfies ∀i, ∑ j∈N i m j (0) > 0 then m is solution of the ordinary differen-

tial equation i = 1, 2, . . . , K ṁi (t) = α i - u i m i (t) ∑ j∈N i m j (t) e -∑ k∈N i \{i} u k m k (t) ∑ j∈N k m j (t) e -u i m i (t)
∑ j∈N i m j (t)

The mean field limit dynamics is topologically equivalent to the Ray-projection evolutionary game dynamics with expected rate transition α iu i m i (t) ∑ j∈N i m j (t) e -∑ k∈N i u k m k (t) ∑ j∈N k m j (t)

Introduction

We consider a large population of players in which frequent interactions occur between small numbers of chosen individuals. Each interaction in which a player is involved can be described as one stage of a dynamic game. The state and actions of the players at each stage determine an immediate payoff (also called fitness in behavioral ecology) for each player as well as the transition probabilities of a controlled Markov chain associated with each player. Each player wishes to maximize its expected fitness averaged over time.

This model extends the basic evolutionary games by introducing a controlled state that characterizes each player. The stochastic dynamic games at each interaction replace the matrix games, and the objective of maximizing the expected long-term payoff over an infinite time horizon replaces the objective of maximizing the outcome of a matrix game. Instead of a choice of a (possibly mixed) action, a player is now faced with the choice of decision rules (called strategies) that determine what actions should be chosen at a given interaction for given present and past observations. This model with a finite number of players, called a mean field interaction model, is in general difficult to analyze because of the huge state space required to describe the sate of all players. Then taking the asymptotics as the number of players grows to infinity, the whole behavior of the population is replaced by a deterministic limit that represents the system's state, which is fraction of the population at each individual state that use a given action.

In this chapter we study the asymptotic dynamic behavior of the system in which the population profile evolves in time. For large N, under mild assumptions (see Section 7.3), the mean field converges to a deterministic measure that satisfies a non-linear ordinary differential equation for under any stationary strategy. We show that the mean field interaction is asymptotically equivalent to a Markov decision evolutionary game. When the rest of the population uses a fixed strategy u, any given player sees an equivalent game against a collective of players whose state evolves according to the ordinary differential equation (ODE) which we explicitly compute. In addition to providing the exact limiting asymptotic, the ODE approach provides tight approximations for fixed large N. The mean field asymptotic calculations for large N for given choices of strategies allows us to compute the equilibrium of the game in the asymptotic regime.

Related Work. Mean field interaction models have already been used in standard evolutionary games in a completely different context: that of evolutionary game dynamics (such as replicator dynamics) see e.g. [START_REF] Tanabe | The propagation of chaos for interacting individuals in a large population[END_REF] and references therein. The paradigm there has been to associate relative growth rate to actions according to the fitness they achieved, then study the asymptotic trajectories of the state of the system, i.e. the fraction of users that adopt the different actions. Non-atomic Markov Decision Evolutionary Games have been applied to firm idiosyncratic random shocks and to cellular communications. Nonatomic mean field games have been studied in (129; 48; 173; 114; 108). The authors in (129) develop a mean field approach for optimal control and differential games -continuous state and time-mean field games with continuum of players and their partial differential equation characterization.

Structure. The remainder of this chapter is organized as follows. In next section we present the model assumptions and notations. In Section 7.3 we present some convergence results of the ODE in the random number of interacting players. In Section 7.5 a resource competition between animals with two types of behaviors and several states is presented. All the sketch of proofs are given in Appendix.

Model description

Markov Decision Evolutionary Process With N Players

We consider the following model, which we call Markov Decision Evolutionary Game with N players.

• There are N ∈ N players.

• Each player has its own state. A state has two components: the type of the player and the internal state. The type is a constant during the game. The state of player j at time t is denoted by X N j (t) = (θ j , S N j (t)) where θ j is the type. The set of possible states X = {1, . . . , Θ} × S is finite.

• Time is discrete, taking values in N N := {0, 1 N , 2 N , . . .}.

• The global detailed description of the system at time t is X N (t) = (X N 1 (t), . . . , X N N (t)).

Define M N (t) to be the current population profile i.e M N x (t) = M N θ,s (t) be the size of subpopulation θ (independent of t by hypothesis). We do not make any specific hypothesis on the ratios MN θ N as N gets large (it may be constant or not, it may tend to 0 or not).

• Strategies and local interaction: At time slot t, an ordered list B N (t), of players in {1, 2, . . . , N}, without repetition, is selected randomly as follows. First we draw a random number of players K(t) such that

P(K(t) = k|M N (t) = m) = J N k ( m)
where the distribution J N k ( m) is given for any N, m ∈ {0, 1 N , 2 N , . . . , 1} ♯X . Second, we set B N to an ordered list of K(t) players drawn uniformly at random among the N(N -1)...(N -K(t) + 1) possible ones. By abuse of notation we write j ∈ B N (t) with the meaning that j appears in the list B N (t).

Each player such that j ∈ B N (t) takes part in a one-shot event at time t, as follows. First, the player chooses an action a in the finite set A with probability u θ (a|s) where (θ, s) is the current player state. The stochastic array u is the strategy profile of the population, and u θ is the strategy of subpopulation θ. A vector of probability distributions u which depend only on the type of the player and its internal state is called stationary strategy.

Second, say that B N (t) = (j 1 , . . . , j k ). Given the actions a j 1 , ..., a j k drawn by the k players, we draw a new set of internal states (s ′ j 1 , ..., s ′ j k ) with probability L N θ;s;a;s ′ (k, m),

where θ = (θ j 1 , ..., θ j k ), s = (s j 1 , ..., s j k ), a = (a j 1 , ..., a j k ), s ′ = (s ′ j 1 , ..., s ′ j k )

Then the collection of k players makes one synchronized transition, such that

S N j i (t + 1 N ) = s ′ j i i = 1, . . . , k
Note that S N j (t + 1 N ) = S N j (t) if j is not in B N (t).

It can easily be shown that this form of interaction has following properties: (1) X N is Markov and (2) players can be observed only through their state.

The model is entirely specified by the probability distributions J N , the Markov transition kernels L N and the strategy profile u. In this paper, we assume that J N and L N are fixed for all N, but u can be changed and does not depend on N (though it would be trivial to extend our results to strategies that depend on N, but this appears to be unnecessary complication). We are interested in large N.

It follows from our assumptions that 1. M N (t) is Markov.

2. for any fixed j ∈ {1, . . . , N}, (X N j (t), M N (t)) is Markov. This means that the evolution of one specific player X N j (t) depends on the other players only through the occupancy measure M N (t).

Payoffs

We consider two types of instantaneous payoff and one discounted payoff:

• Instant Gain: This is the random gain G N j (t) obtained by one player whenever it is involved in an event at time t. We assume that it depends on this player's state just before the event and just after the event, the chosen action, and on the states and actions of all players involved in this event. Formally, if player j ∈ B N (t) G N j (t) = g N (x j , a j , x ′ j , x B N (t)\j , a B N (t)\j , x ′ B N (t)\j )

where x j = X N j (t), a j is the action chosen by player j, x ′ j = X N j (t + ] of players other than j involved in the event, a B N (t)\j is the list of their actions and g() is some non random function defined on the set of appropriate lists. Whenever j is not in B N (t), G N j (t) = 0. We assume that G N j (t) is bounded, i.e. there is a non random number C 0 such that, with probability 1: for all j, t: |G N j (t)| ≤ C 0 • Expected Instant Payoff: It is defined as the expected instant gain of player j, given the state x of j and the population profile m. By our indistinguishability assumption, it does not depend on the identity of a player, so we can write it as r N (u, x, m) := E G N j (t) X N j (t) = x, M N (t) = m Note that this conditional expectation contains the case when j is not in B N (t), i.e. when G N j (t) = 0.

• Discounted Long-Term Payoff: It is defined as the expected discounted long term payoff of one player, given the initial state of this player and the population:

rN (u; x, m) := E( ∞ ∑ t=0 step 1/N e -βt G N j (t)|X j (0) = x, M N (0) = m)
where β is a positive parameter (existence follows from the boundedness of G N j ). The fact that it does not depend on the identity j of the player, but only on its initial state x and the initial population profile m, follows from the indistinguishability assumption.

We defined the Discounted Long-Term Payoff in terms of the instant gain, as this is the most natural definition. The following proposition shows that the alternative definition, by means of the expected instant payoff, is equivalent. Let τ N be the first time after t = 0 that X N j (t) hits in some given state. We show that rN = 1 N E τ N ∑ s=0 step 1/N e -βt r N X N j (s), M N (s) (7.1) Define for t ∈ N/N:

Z N t = t ∑ s=0 step 1/N e -βs G N (s) -r N X N j (s), M N (s)
we have, for 0 ≤ s ≤ t:

Q := E Z N t -Z N s |F N s = t ∑ u ′ =0 step 1/N e -βu ′ E G N (u ′ ) -r N X N j (u ′ ), M N (u ′ ) |F N s
which can be written as

t ∑ u ′ =0 step 1/N e -βu ′ E E G N (u ′ ) -r N X N j (u ′ ), M N (u ′ ) |F N u ′ |F N s = 0
thus Z N t is an F N tmartingale. Now τ N is a stopping time with respect to the filtration F N t thus, by Doob's stopping time theorem: EZ N t∧τ N = EZ N 0∧τ N = 0 Further, Z N t∧τ N ≤ K|τ N | for some constant K. Since τ N is almost surely finite and has a finite expectation, we can apply dominated convergence (with t → ∞) and obtain EZ N τ N = 0.

Focus on One Single Player

We are interested in the following special case (here we make the dependency on the strategy explicit). There are two types of players, i.e. Θ = 2. There is exactly one player (the player of interest) with type 1. All other players have type 2. In this case we use the notation R N (u 1 , u 2 ; s, m) for the discounted long-term payoff obtained by the player in type 0, when her strategy is u 1 and all other players's strategy is u 2 , given that this player's initial internal state is s and the initial type 2 subpopulation profile is m. Note that R N (u 1 , u 2 ; s, m) = rN (u 1 , u

Markov Decision Evolutionary Game

Player j may choose a strategy u j . We look for a (Nash) equilibrium u such that if all players use u then no player has an incentive to deviate from u. For any finite N one can map this into a standard Markov game. This is true for both the case where the number of players is known and in the case it is unknown when taking a decision. Therefore we know that a stationary equilibrium exists in the discounted case. A stationary equilibrium is solution of the fixed point equation: ∀j, u j,θ ∈ arg max v j,θ R N (v j,θ , u -j ; s, m)

By assuming symmetry per type we can show that a stationary equilibrium exists which is a solution of the fixed point equation ∀θ, u θ ∈ arg max

v θ
R N (v θ , u; s, m)

Markov Decision Evolutionary Team

We wish to find a stationary u that maximizes R N averaged over all players. u = (u 1 , . . . , u Θ ) ∈ arg max v R N (v; s, m)

Main Results

Scaling Assumptions

We are interested in the large N regime and obtain that, for any fixed j, (X N j , M N ) converges weakly to a simple process. This requires the weak convergence of M N (0) to some m 0 .

We assume that the parameters of the model and the payoff per time unit converge as N → ∞, i.e. Note that H1 excludes the case where the number of players involved in an event per time slot scales like N (i.e. synchronous transitions of all players at the same time). There may be large N asymptotic results for such cases (130) but the limit is not given by an ODE. In contrast, H1 is automatically true if the number of players involved in an event per time slot is upper bounded by a non random constant. We also need some technical assumptions, which are usually true and can be verified by inspection. H2 ∑ k J k ( m) > 0 for all m ∈ ∆ (∆ is the simplex { m : m θ,s ≥ 0, ∑ θ,s m θ,s = 1}). This ensures that the mean number of players involved in an event per time slot, ∑ k≥0 kJ k ( m) is non zero.

Define the drift of M N (t) as

f N (u, m) = E M N (t + 1 N ) -M N (t)|M N (t) = m is continuous.
Using Theorem 7.3.4 and Theorem 7.3.5 we obtain the following, which is the main result of this paper. It says that when N goes to infinity, the Markov Decision Evolutionary Game with N(t) of players becomes equivalent to the associated Macroscopic Markov decision evolutionary game. This reduces any multi-player problem into an effective one-player problem.

Sketch of Proof of Theorem 7.3.5

Since Skorohod's topology is induced by a metric, it is sufficient to show that whenever (X N j , m N ) → (x, m) in Skorohod's topology, we have: By ( 73), page 117, there is some sequence of increasing bijections λ n : [0, ∞) → [0, ∞) s.t. Third, let D be the set of discontinuity points of (x, m). Since (x, m) is cadlag, D is enumerable, thus it is negligible for the Lebesgue measure and Define the discounted stochastic evolutionary game with random number of interacting players in each local interaction in which each player in x with the mixed action u(.|x) receives r(u, x, m(t)) where m(t) is the population profile at t, which evolves under the dynamical system (7.4) and the between states follows the transition kernel L. Then, a strategy of a player is the same as in the microscopic case and the discounted payoffs R(u 1 , u 2 , s 0 , m 0 ) = ∞ 0 e -βt r(s(t), u 1 , m[u 2 ](t))dt is the limit of R N (u 1 , u 2 , s 0 , m 0 ) when N goes to infinity, where m[u 2 ] is the solution of the ODE ṁ = f (u 2 , m), m(0) = m 0 . It follows that the asymptotic regime of the microscopic game and the Markov decision evolutionary game (macroscopic game) are equivalent.

Case with Global Attractor

Assume that, for some strategy u, the ODE (7.4) has a global attractor m * (this may or may not hold, depending on the ODE). If in addition the model with N players is irreducible, with stationary probability distribution ̟ N for M N , then lim N-→∞ ̟ N = δ m * where δ m * is the Dirac mass at m * (follows from ( 43)). i.e. the large time distribution of M N (t) converges, as N → ∞, to the attractor m * . Also, (X N j (t), M N (t)) converges to a continuous time, homogeneous Markov jump process with time-independent transition matrix:

A(x 1 , x ′ 1 ; u) = ∑ k≥1 J k ( m)A k (s 1 , s ′ 1 ; m * , u)
Assume that the transition matrix A(x 1 , x ′ 1 ; u) is also irreducible and let π() be its unique stationary probability. Also let π N be the first marginal of the stationary probability of (X N 1 , M N ). It is natural in this case to replace the definition of the long term payoffs R N (u 1 , u 2 ; s, m) and R N (u 1 , u 2 ; s, m) by their stationary counterparts R N st (u 1 , u 2 ) := ∑ s π N (s)R N (u 1 , u 2 ; s, m * ) R st (u 1 , u 2 ) := ∑ s π(s)R(u 1 , u 2 ; s, m * )

Single player per type selected per time slot

Consider the special case where at each time slot, only one player per type between the N is randomly selected and has a chance to change its action, i.e. ♯B N = 1 w.p 1.

Thus H1 and H2 are automatically satisfied. The resulting ODE (see [START_REF] Benaim | Deterministic Approximation of Stochastic Evolution in Games[END_REF]) becomes

d dt m x (t) = ∑ x ′ m x ′ L x ′ ,x ( m, u, Θ) -m x ∑ x ′ L x,x ′ ( m, u, Θ)
The term ∑ x ′ m x ′ L x ′ ,x ( m, u, Θ) is the incoming flow in to x and the outgoing flow from x is

m x ∑ x ′ L x,x ′ ( m, u, Θ).
We then obtain a large class of state-dependent evolutionary game dynamics. Note that in general the trajectories of the mean dynamics need not to converge. In the case of single player Moreover, if the game has symmetric payoffs and strategies for each type, there is a symmetric per type stationary equilibrium. This completes the proof. Theorem 7.3.11 (Infinite N). Optimal strategies (resp. equilibrium strategies) exist in the limiting regime when N → ∞ under uniform convergence and continuity of R N → R. Moreover, if {U N } is a sequence of ǫ N -optimal strategies (resp. ǫ N -equilibrium strategies) in the finite regime with ǫ N -→ ǫ, then, any limit of subsequence U φ(N) -→ U is an ǫoptimal strategy (resp. ǫ-equilibrium) for game with infinite N.

Sketch of Proof of Theorem 7.3.11

Let (U N ) N be a sequence of solution of (FIX N ) i.e equilibrium in the system with N players. Choose a subsequence N k such that U N k converges to some point u when k goes to infinity. We can write R(U,U). Since R N (., .) is continuous and converges uniformly to R(., .), R N k converges uniformly to R, the second term R N k (U, U) -R(U, U) -→ 0 when N k -→ ∞ and the first term

R N k (U N k , U N k ) -R(U, U) = R N k (U N k , U N k ) -R N k (U, U) + R N k (U, U) -
R N k (U N k , U N k ) -R N k (U, U) can be rewritten as R N k (U N k , U N k ) -R N k (U, U) = R N k (U N k , U N k ) - R(U N k , U N k ) + R(U N k , U N k ) -R(U, U) + R(U, U) -R N k (U, U).
Each term goes to zero by continuity of R, convergence of U N k to U, and by uniform convergence of R N to R. Let U N be a ǫ N -equilibrium. Then, R N (U N , U N ) ≥ R N (v, U N )ǫ N , ∀v. Then any limit U of a subsequence of U N satisfies R(U, U) ≥ R(v, U)ǫ, ∀v. Similarly, if R N (U N , U N ) ≥ R N (v, v)ǫ N , ∀v then any omega-limit U of the sequence of U N satisfies R(U, U) ≥ R(v, v)ǫ, ∀v i.e U is an ǫ-optimal strategy. In particular if (U N ) N is a sequence of ǫ N -equilibria (resp. optimal strategies) with ǫ N -→ 0 when N goes to infinity then any accumulation point U of (U N ) N is a 0-equilibrium (resp. 0-optimal strategy).

Robust equilibrium

We now consider two types of players θ = 1, 2 and define the expected payoff of a player with type 2, the strategy v whenever the other players of type 1 use u when the proportion of the players of type 2 is α as RN (v, u, α, m 0 ) Definition 7.3.12.1. The strategy u is k-resilient to invasion if for all strategy v = u RN (v, u, α, m 0 ) ≤ RN (u, u, α, m 0 )

∀α ∈ {0, 1 N , . . . , k N }
Since a Nash equilibrium is resilient against unilateral deviation (1-resilient). A k-resilient strategy is in particular 1-resilient. Definition 7.3.12.2. The strategy u is resilient to invasion by small fraction of deviants (RID) if for all strategy v = u there exists an invasion barrier ᾱv = 0 such that R(v, u, α, m 0 ) < R(u, u, α, m 0 ) = R(u, u, 0, m 0 ) ∀α ∈ (0, ᾱv ).

The strategy which is resilient to invasion by small fraction of deviant is more robust than a Nash equilibrium. A Nash equilibrium u is not necessarily a RID.

∀N ≥ N 0 , R N -R ∞ ≤ γ N ≤ ǫ/4, R N (u, u, m 0 ) -R N (v, u, m 0 ) = R N (u, u, m 0 ) -R(u, u, m 0 ) + R(u, u, m 0 ) -R(v, u, m 0 ) +R(v, u, m 0 ) -R N (v, u, m 0 ) ≥ -ǫ/2 -2γ N ≥ -ǫ
Proposition 7.3.12.6. Let u be a strict equilibrium. Then u is a RID for the payoff function R defined as R = αR(v, u, m 0 ) + (1α)R(u, u, m 0 ).

Moreover u is non-invadable strategy (NIS) i.e the inequality for RID holds for any uniform threshold ǫ.

Proof. Since u is a strict equilibrium, one has: R(v, u, m 0 ) < R(u, u, m 0 ), ∀v = u, ∀v ∈ U b

This implies that R < R(u, u, m 0 ), ∀α ∈ (0, 1). This completes the proof for RID and for NIS. and the rate of transition of the process X 1 (t) from state x 1 = (θ 1 , s 1 ) to state x ′ 1 = (θ 1 , s ′ 1 ) is A(x 1 , x ′ 1 ; m(t), u). We derive a new class of dynamic games called differential population games. Different from standard differential game models, this class of games is described by large population of players in which each player from each class is facing a random vector that evolves according to the population dynamics and the individual state follows a jump and drift process. See [START_REF] Tembine | Dynamic Games and Networks Evolution[END_REF] for more details on differential population games.

Link with differential population

Illustrating example

We present in this section an example of a dynamic version of the Hawk and Dove problem where each individual has three energy levels. We derive the mean field limit for the case where all users follow a given policy and where possibly one player deviates. We then further simplify the model to only two energy states per player. In that case we are able to fully identify and compute the equilibrium in the limiting Markov decision evolutionary game. Interestingly, we show that the ODE converges to a fixed point which depends on the initial condition.

Consider an homogenous population of N animals. An animal plays the role of a player. Occasionally two animals find themselves in competition on the same piece of food. Each animal has three states x = 0, 1, 2 which represents its energy level. An animal can adopt an aggressive behavior (Hawk) or a peaceful one (Dove, passive attitude). At the state x = 0 there is no action. We describe the fitness of an animal (some arbitrary player) associated with the possible outcomes of the meeting as a function of the decisions taken by each one of the two animals. The fitnesses represent the following: • An encounter Hawk-Hawk results in a fight in which with p = 1/2 chances, one (resp. the other) animal obtains the food but also in which there is a positive probability for each one of the animals to be wounded 1/2 . Then the fitness of the animal 1 is 1 2

( v -c) + 1 2 (-c) = 1 2
vc, where the -c term represents the expected loss of fitness due to being injured. The vector of frequencies of states at time t is given by M N x (t) =

i\j (g N i , g N j ) X N i (t + 1 N ), X N j (t + 1 N ) D -D ( v 2 , v
1 N N ∑ j=1
1 {X N j (t)=x} for x = 0, 1, 2 and the action set is A x = {H, D} in each state x = 0, A 0 = {}.

The assumptions in Section 6.2 are satisfied (pairwise interaction, ♯B N (t) = 2) and the occupancy measure M N (t) converges to m(t).

ODE and Stationary strategies

Consider the following fixed parameters µ 1 = L 0,1 , µ 2 = L 0,2 . The population profile is denoted by m = (m 0 , m 1 , m 2 ) and the stationary strategy is described by the parameters v 1 , v 2 where v 1 := u(H|1), v 2 = u(H|2)

   ṁ2 = m 0 L 0,2 + m 1 L 1,2 (u, m) -m 2 L 2,1 (u, m) ṁ1 = m 0 L 0,1 + m 2 L 2,1 (u, m) -m 1 L 1,2 (u, m) -m 1 L 1,0 (u, m) ṁ0 = m 1 L 10 (u, m) -(µ 1 + µ 2 )m 0 where L 12 (u, m) =m 0 + v 1 v 1 m 1 2 + (1 -v 1 )m 1 + v 2 m 2 2 + (1 -v 2 )m 2 + (1 -v 1 ) (1 -v 1 )m 1 2 + (1 -v 2 )m 2 2 L 2,1 (u, m) = v 2 v 1 m 1 2 + v 2 m 2 2 + (1 -v 2 ) (1 -v 1 )m 1 2 + v 2 m 2 + (1 -v 2 )m 2 2 L 10 (u, m) := v 1 v 1 m 1 2 + v 2 m 2 2 + (1 -v 1 ) v 1 m 1 + (1 -v 1 )m 1 2 + v 2 m 2 + (1 -v 2 )m 2 2 ,
For B N = {j 1 , j 2 }, x ′ j , x i ∈ {0, This implies that 0 = E x 1 ∆ ∆ 0 e -βt r(u 1 , u 2 , x(t), m(t)) dt + e -β∆ -1 ∆ E x V(u 1 , u 2 , x(∆)) + E x V(u 1 , u 2 , x(∆)) -V(u 1 , u 2 , x) ∆ Using Ito's formula and Lebesgue integration properties, we obtain that: when ∆ goes to zero. Thus, we obtain βV(u 1 , u 2 , x) = r(u 1,x , u 2,x , x, m) + ∑

E x V(u 1 , u 2 , x(∆)) -V(u 1 , u 2 , x) ∆ goes to ∑ x ′ D m x ′ V(
x ′ (D m x ′ V(u 1 , u 2 , x ′ )) f x ′ (u 2 , m) (7.13) where u i,x = u i (H|x).

The global optimality is then given by the Hamilton-Jacobi-Bellman equation obtained by maximizing the right-hand side of the equation (7.13) over the action set. Theses equations are in general difficult to solve and the solutions are not necessarily regular (e.g. viscosity solutions). Numerical approaches based on multi-grid techniques of Hamilton-Jacobi-Bellman-Issacs equations can be found in [START_REF] Kushner | Numerical Methods for Stochastic Control Problems in Continuous Time[END_REF].

The case of two energy levels

In order to derive closed form expressions for solutions of our ODE, we consider two states, i.e., each animal has two states x = 1, 2 which represents its energy levels. Thus, the ODE can be expressed as follows: ṁ2 (t) = (1m 2 (t))L 1,2 (u, m)m 2 (t)L 2,1 (u, m) (7.14) which can be rewritten as ṁ2 (t) = a 1 + a 2 m 2 (t) + a 3 (m 2 (t)) 2 (7.15)

with

a 1 = 1, a 2 = u 2 2 -2 < 0, a 3 = 1 -u 2 2 > 0.
Let m[u, m 0 ](t) be the solution of the ODE given u and a initial distribution m(0) = m 0 . We distinguish two cases: Case 1: u 2 = 1 (fully aggressive when it is possible): the ODE becomes ṁ2 (t) = 1 - m 2 [u, m 0 ](t) = γ -(u) + γ + (u)γ -(u) 1c 2 e (γ + (u)-γ -(u))a 2 t , (7.17)

c 2 = 1 + γ + (u) -γ -(u) m 2 (0) -γ -(u) , γ -(u) = 2 -u 2 /2 -(2 + u 2 2 /4) 1 2 1 -u 2 < 1, γ + (u) = 2 -u 2 /2 + (2 + u 2 2 /4) 1 2 1 -u 2 > 1
Note that in both cases there is a unique strategy-dependent global attractor. lim

t-→∞ m 2 [u, m 0 ](t) = γ -(u) if u 2 = 1 2/3 if u 2 = 1
The expected instant payoff of a player using the stationary strategy v when the population profile is m[u, m 0 ](t), is given by

r(v, u, 2, m[u, m 0 ](t)) = v[ v -cm 2 u 2 ] + (1 -v)r(v, u, 1, m[u, m 0 ](t)) r(v, u, 1, m[u, m 0 ](t)) = 1 2 (1 -m 2 [u, m 0 ](t)u 2 ) v
where m 2 [u, m 0 ](t) is given by (7.16) (resp. (7.17)) for u 2 = 1 (resp. u 2 = 1). Now, we can compute explicitly the best response against u for a given initial m 0 . Let

β 2 (u,

Figure 2 . 1 :

 21 Figure 2.1: Access game between two mobiles.

3 Figure 2 . 2 :

 322 Figure 2.2: Effect of κ on stability of delayed imitation dynamics, τ = 1.

Figure 2 . 3 :

 23 Figure 2.3: Effect of κ on stability of replicator dynamics, τ = 8.

15 Figure 2 . 4 :

 1524 Figure 2.4: Effect of τ T in delayed replicator dynamic for κ = 0.002

Figure 2 . 5 :Figure 2 . 6 :

 2526 Figure 2.5: Effect of τ T in delayed replicator dynamic for κ = 2.

Figure 2 . 8 : 2 Figure 2 . 9 :

 28229 Figure 2.8: Effect of τ T on velocity and stability of imitation dynamics.

Figure 2 . 10 :

 210 Figure 2.10: Effect of τ T on velocity and stability in BNN dynamics.

Figure 2 .

 2 Figure 2.11: MIMD (H) versus AIMD (H)

Figure 2 . 12 :

 212 Figure 2.12: Effect of K on stability

Figure 2 . 13 :

 213 Figure 2.13: Effect of τ on stability

Figure 2 . 14 :

 214 Figure 2.14: Effect of initial state

  02,0.02),K=1 τ= (2,2),K=1 τ=(4,4),K=1 τ=(4,4.4),K=1

Figure 2 . 15 :

 215 Figure 2.15: Moving from equilibrium to another.

Figure 3 . 4 :

 34 Figure 3.4: Non-reciprocal interference control

Figure 3 . 5 :

 35 Figure 3.5: Multiple access game with three nodes. B θ = ∆ + δ

Figure 3 . 6 :

 36 Figure 3.6: Impact of n in the probability of success in Dirac distribution for α = 0.05

Figure 3 . 7 :

 37 Figure 3.7: Impact of n in the probability of success in Dirac distribution for α = 0.2.

Figure 3 . 8 :Figure 3 . 9 :

 3839 Figure 3.8: Impact of n in the probability of success in Dirac distribution.

Figure 3 . 10 :

 310 Figure 3.10: Impact of n in the probability of success in Dirac distribution for α = 1/3.

Figure 3 . 11 :

 311 Figure 3.11: Probability of success in Dirac distribution.

Figure 3 . 12 :

 312 Figure 3.12: Delay effect in Dirac Distribution.

Figure 3 . 13 :Figure 3 . 14 :

 313314 Figure 3.13: Probability of success in Poisson distribution (cases 1,2).

Figure 3 . 15 :

 315 Figure 3.15: Evolution of the fraction of transmitters varying the density parameter λ.

5 Figure 3 . 16 :Figure 3 . 17 :

 5316317 Figure 3.16: Impact of µ on the velocity of the replicator dynamics without delay

Figure 3 . 18 :

 318 Figure 3.18: Impact of the time delay on replicator dynamics (case 1).

  Number of nodes which transmit: active nodesWe define numb T : A n -→ N as the number of transmitters of an action. Par example, numb T (T, . . . , T) = n and numb T (T, S, . . . , S) = 1. The set {b ∈ A | numb T (b) = 1} is exactly the situations where only one of the nodes transmit (and the others stay quiet). numb T (b) = 1 if and only if there exists a unique j such that b j = T i.e all permutations of the actions profile T S . . . S n-1 times . Denote by Θ n := {b ∈ A n | numb T (b) = 1}. Proposition 3.4.9.6. The probability distribution P n over A n defined as

Figure 3 . 19 :

 319 Figure 3.19: Payoffs obtained at a CESS and the ESS.

R

Figure 3 . 20 :

 320 Figure 3.20: Interferences at the receiver in uplink CDMA transmissions.

Figure 3 . 21 :

 321 Figure 3.21:The average rate at equilibrium versus λ for η = 0.92, 0.97.

Figure 3 . 22 :

 322 Figure 3.22: Convergence to the ESS in W-CDMA system : uniform distribution.

Figure 3 . 23 :

 323 Figure 3.23: Convergence to the ESS in W-CDMA system : quadratic distribution.

Figure 3 . 24 :

 324 Figure 3.24: Hexagonal cell configuration

Figure 3 . 25 :

 325 Figure 3.25: The population ratio of ON at equilibrium versus η.

Figure 3 . 26 :

 326 Figure 3.26: The population ratio of strategy ON versus P.

  F b (x) ≥ F b (y), ∀b ∈ P. The function F is extended to a more general generating function[START_REF] Vincent | Evolutionary Game Theory, Natural Selection, and Darwinian Dynamics[END_REF] G defined in a linear space that contains ∆ n × ∆ n . The G-function satisfies

b∈P τ b =: τ. Lemma 3 . 7 . 1 . 2 .

 3712 If β is locally Lipschitz, φ is continuous on (max b∈P τ b , 0) and F is generated by regular G-function G : R n+1 × R n+1 -→ R satisfying G(v, x) |v=b = F b (x).

3 . 7 . 1 . 3 .

 3713 (or general imitation dynamics), Smith dynamics (189) (pairwise comparison dynamics), projection dynamics (168), Brown-von Neumann-Nash dynamics (55) (excess payoff dynamics) satisfy the positive correlation (PC) property. Moreover Brown-von Neumann-Nash dynamics, projection dynamics and Smith dynamics satisfy the property that every rest point of the dynamics is an equilibrium of the game. It is easy to see that the parameter µ > 0 and the times does not change the set of equilibria. Hence, the following holds: Corollary Delayed imitation dynamics, delayed pairwise comparison dynamics, delayed projection dynamics, excess payoff dynamics and projection dynamics satisfy the positive correlation (PC) property. Moreover Brown-von Neumann-Nash dynamics, projection dynamics and Smith dynamics satisfy the property that every rest point of the dynamics is an equilibrium of the game. Examples 3.7.1.4. If

Figure 3 . 27 :

 327 Figure 3.27: No pure equilibria. Three strategies

Figure 3 . 28 :

 328 Figure 3.28: No pure equilibrium under pricing.

Proposition 3 . 7 . 4 . 4 .

 3744 If x is an ESS then the product measure x ⋆(m) given by x ⋆(m) (a) = ∏ j∈N x j (a j ) is a CESS.

  We consider three examples of distribution: geometric distribution Geo(p = 0.3) and Dirac distribution δ m , m = 30.

Figure 3 . 29 :

 329 Figure 3.29: Probability of success -Dirac distribution m = 30

Figure 3 . 30 :

 330 Figure 3.30: Geometric probability of success with parameter p = 0.3

Figure 3 . 31 :

 331 Figure 3.31: Probability of success at CESS

  t) = ∑ r∈AF y r(t)β rr ( Ḡ, y)y r ∑ r∈AF β r r( Ḡ, y) (3.17)

Proposition 3 . 7 . 6 . 1 .

 3761 For any distribution of the random variable K, and ♯P ≥ 3, the following results holds :PoS NE = PoS ESS = 0 = PoA ESS = PoA NENon-convergence and instability to the ESSIn figure3.32, we plot the trajectories of replicator dynamics without time delays and three strategies. We observe the convergence to the interior rest point starting from any in the relative interior of the simplex. Moreover, the interior rest point is asymptotically stable. The parameters are V = 1, c(p 2 ) = 1/2, c(p 1 ) = 1/4, c(p 0 ) = 0.

Figure 3 . 32 :

 332 Figure 3.32: Three powers: replicator dynamics

3 Figure 3 . 33 : 10 Figure 3 . 34 :

 333310334 Figure 3.33: Convergence and stability of ESS for small delays

20 Figure 3 . 35 :

 20335 Figure 3.35: Instability of ESS: non-convergence.

  amut r a )F r a (ǫ mut + (1ǫ)x) > 0. (4.4) If the inequality (4.4) is non-strict the corresponding equilibrium is called Neutrally Stable Strategy.

  b) which determine the rates at which a player who is considering a change in strategies opts to switch to his various alternatives from other regions into region r. The functions ρ and η are called revision protocols. The two revision protocols depend on the state of the population and the payoff functions. The inflow inside the region r is k r ∑ outflow inside the region r, k r x r a ∑ ā∈A r

  a) and inter-outflow of the region r is γ r k r x

  φ r a (x r , x -r ) = k r γ r

  r, b) is an optimal strategy then sign preserving assumption implies that there is no outflow from (r, b) i.e x r b) = 0. This condition is exactly∀(r, a), x r a = 0 or β (r,a) (r,b) = 0, ∀(r, b), b ∈ A r .This last alternative says that F r a (x) ≥ F r b (x), ∀(r, b), b ∈ A r. This completes the proof.

Proposition 4 . 4 . 1 . 1 .

 4411 In stable population games, the set of GNE is convex and coincides with the set of neutrally ESS.

Remarks 4 . 4 . 3 . 1 .

 4431 We denote by C (r,a) = S\N (r,a) = ∪ r {r} × A r \N (r,a) 1. Let y ∈ X be a CCE of the game with migration constraints. If y satisfies max (r,b)∈N (r,a) F r b (y) ≥ max (r,b)∈C (r,a) F r b (y)

Figure 4 . 1 :

 41 Figure 4.1: The hybrid model

Proposition 4 . 7 .0. 3 .

 473 The Wardrop equilibrium (x r * ) r=1,...,N is also a global ESS.

Figure 4 . 2 :

 42 Figure 4.2: RD: convergence to Wardrop equilibrium

Figure 4 . 3 :

 43 Figure 4.3: RD: fraction of mobiles using the power levels P in cell 1 and cell 2

Figure 4 . 4 :

 44 Figure 4.4: Smith dynamics

Figure 4 . 5 :

 45 Figure 4.5: The expected throughput of the transmitters.

Figure 4 . 6 :

 46 Figure 4.6: The expected throughput of the population vs x P .

Figure 4 . 7 :

 47 Figure 4.7: The total payoff of all the population vs x P .

  3.1.2 with eq. (5.7) we obtain: Proposition 5.3.1.4

Lemma 5 . 4 . 1 . 1 .

 5411 The number of pure stationary strategies is ∏ p∈P ∏ y∈Y p |A p (y)|.

Proposition 5 . 7 . 8 . 5 .

 5785 The game has at most one ESS in state independent strategies. Proof. (i) Uniqueness of pure ESS: We have to examine the two strategies: h and l. From Proposition 5.7.8.2 l is an ESS if and only p

Figure 5 . 1 :

 51 Figure 5.1: ESS ρ with state-independent actionsFinally, we also verify the result of the proposition 5.7.8.7 comparing the ESS with the global optimum solution.

Figure 5 . 2 :

 52 Figure 5.2: Replicator dynamic with p = 0.5

Figure 5 . 3 :

 53 Figure 5.3: Replicator dynamic with p = 0.1

Figure 5 . 4 :

 54 Figure 5.4: Replicator dynamic with p = 0.9

Figure 5 . 5 :

 55 Figure 5.5: Solar-power mechanism

Figure 5 . 6 :Figure 5 . 7 :

 5657 Figure 5.6: best reply (line) during several slots.

Figure 5 . 8 :Figure 5 . 9 :Figure 5 . 10 :

 5859510 Figure 5.8: Payoff -uniform distribution

Proposition 5 . 8 . 3 . 2 .

 5832 The payoff function has a representation in term of occupation measure, i.e., F n,u (σ) = ∑ s,a f u (s, a)r(s, a, α) where α = (x, y) is the profile of population under stationary strategy σ. Proposition 5.8.3.3. The fully aggressive strategy Ph which consists to transmit with the maximum power (P h ) in each state s = 0 is an Evolutionarily Stable Strategy.

Figure 5 . 11 :

 511 Figure 5.11: Generic battery state transition rule.

  The signal to noise plus interference ratio of a mobile with the battery type p in state y at the position λ

Theorem 5 . 10 . 1 .

 5101 Each mobile with the battery type p can guarantee the payoff ∑ y) is the probability to be in state y under the maximum power strategy. Proof. Π p y is the frequency of visit of the battery state in y. From Proposition 5.10.0.7, each mobile with the battery type p can obtain at least vp y against any strategies of others mobiles. Each mobile of subpopulation can then obtain at least ∑ y Π p y vp y which is an equilibrium payoff. This completes the proof.

Figure 6 . 1 :

 61 Figure 6.1: Non-Commutative Diagram.

m

  *m, r( m * ) ≥ 0, where r( m) = (r x (m)) x∈S , and ., . is the inner product in the d dimensional Euclidean space R d . We recall that given a closed convex and non-empty set ∆ d and a continuous function r, solving the variational inequality problem defined by ∆ d and r means finding a vector m * such that m *m, r( m * ) ≥ 0 Geometrically, the task of variational inequality problem is to find a vector m * such that the image of m * under the reward function r will form an angle more than or equal to 90ř with any vector with tail m * and head m ∈ ∆ d . Proposition 6.4.1.1 (Existence). For any distribution B and any continuous function r on the nonempty, convex and compact subset ∆ d of the Euclidean space R d , the evolving game has a least one stationary equilibrium state. Sketch of the Proof. We first show that m * ∈ ∆ d is a solution of the variational inequality problem if and only if for any ς > 0, m * is a fixed point of the map Π ∆ d (I + ς r) from ∆ d to ∆ d . Multiplying the inequality m *m, r( m * ) ≥ 0 by ς > 0, and adding m * , mm * to both sides of the resulting inequality, one obtains mm * , m * -[ m * + ςr( m * )] ≥ 0.

lim

  a (t) = 0 where RP is the set of rest point of replicator dynamics andd RP (m) = min m * ∈RP mm * .This says that the time average value of the population profile over the time interval [0, T] converges to the set of interior rest point of the replicator dynamics as T goes to infinity.Logit dynamics Logit dynamics (called also regularized version of best response dynamics orBoltzman dynamics) when ǫ > 0, L θ,b;θ,a (m) = λ e r θ,a (m) ǫ ∑ b e r θ,b (m) ǫ d dt m θ,a = λ e r θ,a (m) ǫ ∑ b e r s,b (m) ǫ λm θ,a BNN dynamics Brown-von Neumann-Nash dynamics for

1 .

 1 The mean dynamics satisfies the condition of Proposition 6.4.5.1, Equation (6.8).2. m * is an equilibrium state if and only if for all x j = (θ, a), one of the following conditions is satisfied:(a) ∑ b L s,a,x -j ;s,b (m) = 0 (b) m * s,a = 0.3. Any rest point of the ODE is a stationary equilibrium state.Proof. The point (3) follows from the points 2 -(a) and 2 -(b). The point The point 2 -(a) of Proposition is the optimality of action a used in state s and the point 2 -(b)

Proposition 7 . 2 . 2 . 1 .

 7221 For all player state x and population profile mrN (u; x, m) = E( ∞ ∑ t=0 step 1/N e -βt r N (u, X N j (t), M N (t))|X j (0) = x, M N (0) = m)Sketch of proof of Proposition 7.2.2.1

  ( m) → J k ( m) L N θ;s;a;s ′ (k, m) → L θ;s;a;s ′ (k, m) r N (u, x, m) → r(u, x, m) (7.2)Our main scaling assumption isH1 ∑ k k 2 J k ( m) < ∞ for all m ∈ ∆.This ensures that the second moment of the number of players involved in an event per time slot is bounded.

e 0 e

 0 -βt r N (v, X N j (t), m N (t))dt = ∞ -βt r(v, x(t), m(t))dt (7.5)

T 0 e 0 e 7 . 3 . 6 (

 00736 -βt d (x(λ N (t)), a, m(λ N (t))), (x(t), a, m(t) dt= T -βt d (x(λ N (t)), m(λ N (t))), (x(t), m(t)1 t / ∈D dt Now lim N-→∞ λ N (t) = t and thus for t / ∈ D lim N-→∞ d (x(λ N (t)), m(λ N (t))), (x(t), m(t) = 0 and thus by dominated convergence lim N-→∞ T 0 e -βt d (x(λ N (t)), m(λ N (t))), (x(t), m(t)) dt = 0 (7.12)and for N large enough the second term in the right-hand side of Equation (7.11) can be made smaller than ǫ/3. Finally, for N large enough, h N ≤ ǫ. This completes the proof. Theorem Asymptotically equivalent game). When N goes to infinity we have (a) the discrete time process X N 1 converges in distribution to the continuous time process X 1 (b) rN (u; x, m) → r(u; x, m) and (c) R N (u 1 , u 2 ; s, m) → R(u 1 , u 2 ; s, m) Sketch of Proof of Theorem 7.3.6

7. 4 . 7 . 3 . 12 . 3 .Proposition 7 . 3 . 12 . 5 .

 47312373125 Link with differential populationProposition If u is a RID then u is an equilibrium. Proof. ∀v, R(v, u, m 0 ) = R(v, u, 0, m 0 ) ≤ R(u, u, 0, m 0 ) = R(u, u, m 0 ) Proposition 7.3.12.4 (RID for Infinite N). If {U N } is a sequence of k N -resilient against deviantsstrategies in the finite regime with k N N -→ ǫ > 0, then, any limit of subsequence U φ(N) -→ U is a neutrally RID1 for game with infinite N. If u is an ǫ 2 -equilibrium of the Markov decision evolutionary game with infinite players then there exists N 0 such that for all N ≥ N 0 , u is an ǫ-equilibrium of the mean field interaction with N players.Proof. R(u, u, m 0 ) ≥ R(v, u, m 0 )ǫ/2, ∀v ∈ U bSince R N goes to R uniformly, choose N 0 satisfying

0 e

 0 Each individual maximizes its long-term payoffr(u; x, m) = E ∞ -βt r(u, X 1 (t), m(t))|X 1 (0) = x, m(0) = m 0subject to the population profile evolution given by d dt m(t) = f (u, m(t))

•

  An encounter Hawk-Dove or Dove-Hawk results in zero fitness to the Dove and in v of value for the Hawk that gets all the food without fight. The state of the Hawk (the winner) is incremented a = 1 {x ′ H =min(x H +1,2)} and the state of the Dove is b = 1 {x ′ D =max(x D -1,0)} . • An encounter Dove-Dove results in a peaceful, equal-sharing of the food which translates to a fitness of v 2 to each animal and the state of each animal change with the sum of the

  βΨ(x) = max u 1,x ,u 2,x {r(u 1,x , u 2,x , x, m) + ∑ x ′ (D m x ′ Ψ(u 1 , u 2 , x ′ )) f x ′ (u 2 , m)}and optimality conditions of the best response to u 2 is given byβΦ(u 2 , x) = max a∈{H,D} {r(a, u 2,x , x, m) + ∑ x ′ (D m x ′ Φ(u 2 , x ′ )) f x ′ (u 2 , m)}

3 2 m 2 3 [ 1 -

 231 (t) and the solution has the formm 2 [1, m 0 ](t) = 2 c 1 e -3 2 t ] (7.16) with c 1 = 1 -3 2 m 0 and m 1 [u, m 0 ](t) = 1m 2 [u, m 0 ](t)Case 2: u 2 = 1, (less aggressive in state 2)

  = u 21u 11 , δ 2 = u 12u 22 and δ = α + β = δ 1 + δ 2 .

	Kτ <	δπ 2δ 1 δ 2	(2.27)
	and unstable if Kτ > δπ δπ 2δ 1 δ 2 where δ 1 The case Kτ = 2δ 1 δ 2

1.2.8) Lemma 2.1.6.2. A necessary and sufficient condition for all roots of (2.26) to have negative real parts is 2ωτ < π Suppose that τ 1 = τ 2 = τ, then the trivial solution of (2.18) is asymptotically stable if is called bifurcation point. By using lemma 2.1.6.2 with ω = γδ, we conclude that the dynamic (2.14) is asymptotically stable at the stationary point ξ * = δ 2 δ if K and τ satisfy (2.27).

  04, we have stability but the convergence speed is slow. The system is unstable when τ T becomes greater than 8.4823. Effect of τ T on velocity and stability of replicator dynamics.

		1
		0.9
	ξ(t)	0.8
	Proportion of transmitters :	0.3 0.4 0.5 0.6 0.7
		0.2										τ=0.02, K=1
		0.1										τ=3, K=1
												τ=10, K=1
		0	0	10	20	30	40	50	60	70	80	90	100
						Replicator dynamics	time t
	0.2 0.3 0.8 0.9 1 Figure 2.7: 0 0.4 0.5 0.6 0.7 Proportion of transmitters:	2	4	6	8	10	12	14	16	18	20
							Imitation dynamics	time t

Delayed Replicator Dynamics with multi-class A

  fraction of member of a subpopulation of class e grows when its payoff is greater than the expected average payoff of all the class e in the population.

	Denote x(t) = (x e (t)) e∈E the population profile at time t. The composi-tion of class e is then describe by x e (t) = [x e 1 (t), . . . , x e n e (t)] where x e a (t) is fraction of players at time t with the strategy a in class e. Denote by F e a (x(t)) the expected payoff of a user with a in class e when the population profile is x(t) at time t. Denote τ e a the time associate to the strategy
	a of class e. The delayed evolutionary game dynamics is given by
		ẋe a (t) = ∑ b∈A e	x e b (t)β e,b a (x(t), {x(t -τ e b )} e,b )	(2.37)
		-x e a (t) ∑ b∈A e	x e b (t)β e,a b (x(t), {x(t -τ e b )} e,b )	(2.38)
	d dt	x e a (t) = µ e x e a (t) F e a (x(t -τ e a )) -∑ b∈A e	x e b (t)F e b (x(t -τ e b ))	(2.39)
	Every equilibrium of evolutionary game with delayed payoffs is a rest point of the dynamic
	(2.39).

Delayed Brown-von Neuman-Nash Dynamics Denote

	Imitation by dissatisfaction Users in the population review their strategy and imitate the
	better strategy at that time. g(F e a (x(t -τ e a )), F e b (x(t -τ e b )), g(a, b) = 1 if a > b, otherwise g(a, b) = 0. d dt x e a (t) = µ e x e a (t) ∑ x e b (t)(ρ e ba (x(t)) -ρ e ab (x(t))) where ρ e ba (x(t)) = b∈A e
	Delayed orthogonal projection dynamics The orthogonal projection dynamic is a myopic
	adaptive dynamic in which a subpopulation grows when its expected payoff is greater than the
	arithmetic average payoff of all the population.	d dt	x e a (t) = µ e F e a (x(t -τ e a )) -	1 n e ∑ b∈A e	F e b (x(t -τ e b ))
	Delayed ray-projection dynamics		d dt	x e a (t) = µ e F e a (x(t -τ e a )) -x e a (t) ∑ b∈A e	F e b (x(t -τ e a ))
	Delayed θ-Smith dynamics
	d dt	x e a (t) = µ e ∑ b∈A e	x e b (t) max(0, F e a (x(t -τ e a )) -F e b (x(t -τ e b ))) θ	(2.42)
		-µ e x e a (t) ∑ b∈A e	max(0, F e b (x(t -τ e b )) -F e a (x(t -τ e a ))) θ
									g e a by the positive part of the excess
	payoff of subpopulation a of class e In continuous time, Brown-von Neuman-Nash Dynamics is
	given by
		d dt	x e a (t) = µ e g e a (x(t)) -x e a (t) ∑ b∈A e	g e b (x(t))	(2.40)
	where g e a (x(t)) = max 0, F e a (x(t -τ e a )) -∑ b∈A e	x e b (t)F e b (x(t -τ e a )) . The equilibria of evolution-
	ary game with delayed payoffs are exactly the rest points of the dynamic (2.40).
	Delayed Best Response Dynamics Members of each subpopulation revise their strategy
	and choose the best replies BR(x(t)) at the current population state x(t).
				d dt	x e (t) ∈ µ e (BR e (x(t)) -x e (t))	(2.41)
	where BR e (x -e (t)) = arg max y∈∆(A e )	∑ b∈A e	y b F e b (x(t -τ e b )) ,
	The equilibria of evolutionary game with delayed payoffs are exactly the rest points of the
	dynamic (2.41).
	Delayed Logit dynamics	d dt	x e a (t) = µ e	  	e ∑ b∈A e e F e a (x(t-τ e a )) η e F e b (x(t-τ e b η e	))	-x e a (t)	  

  2

		S	T
	, P 1 :	S p 11 p 12
		T p 21 p 22

  • TS means "opposite of the recommendation: choose S if ω ∈ {ST, TT} and choose T if ω ∈ {SS, TS}". ST -p 11p 12 + 3p 22 + 3p 21 -p 11 + 3p 21 -2p 22 -p 12 + 3p 22 -2p 21 -2p 21 -2p 22 TS -p 21p 22 + 3p 11 + 3p 12 -p 21 + 3p 11 -2p 21 -p 22 + 3p 12 -2p 11 -2p 12 -2p 11 TT 3 3p 11 + 3p 21 -2p 12 -2p 22 3p 12 + 3p 22 -2p 11 -2p 21 -2 The mixed strategy s = (1/3, 2/3) is the unique ESS of the game with payoff matrix Γ 1 . The expected payoff obtained at the ESS is -1/3. In this example, every probability distribution P 1 satisfying p 11 < p 12 2 , p 22 < 2p 21 , p 12 = p 21 is a CESS. In particular,

	Thus, AF = {SS, ST, TS, TT} and the extended game becomes
		SS		ST		TS	TT
	SS	-1	-p 11 -p 21		-p 12 -p 22	0
	Remarks 3.4.9.5. P 1 :	S	S 0	T 1/2
				T 1/2	0

is a CESS (but the mixed strategy (1/2, 1/2) is not an ESS) and the assignment functions profile (ST, ST) and (TS, TS) give the payoff 3/2 which is greater than -1/3.

Lemma 3.5.0.8.

  For all density function ς defined on [0, R], the function h : [0, 1] → R defined as The function h is continuous. Using derivative properties of integral with parameter, we can see that the derivative function of h is the function h ′ : [0, 1] → R defined as

	s -→	0	R	log	1 + SI NR(P H , s, r) 1 + SI NR(P L , s, r)	ς(r) dr is continuous and strictly monotone.
	Proof. The function s -→ log the interval [0, R]. s -→	0	1 + SI NR(P H , s, r) 1 + SI NR(P L , s, r) R ∂ ∂s log 1 + SI NR(P H , s, r) ς(r) is continuous and integrable in r on 1 + SI NR(P L , s, r) ς(r).
	We show that the term	∂ ∂s	log	1 + SI NR(P H , s, r) 1 + SI NR(P L , s, r)	is negative. Let W(s) :=	1 + SI NR(P H , s, r) 1 + SI NR(P L , s, r)	.
								gP L r α

The function W can be rewritten as W(s) = 1 + g(P H -P L ) r α σ + βI(s) +

  P L is preferred to P H : Analogously, the lower power dominates the higher power if and only if f P L (P H ) > f P H (P H ) and f P L (P L ) > f P H (P L ) i.e Suppose that the parameters w, η, P H and P L satisfy the following inequality h(1) < η w (P H -P L ) < h(0). Then the game has no dominant strategy. A mixed equilibrium is characterized by f P H (s) = f P L (s). It is easy to see that this last equation is equivalent to h(s) = η w (P H -P L ). From the lemma 3.5.0.8, we have that the equation h(s) = η w (P H -P L ) has an unique solution given by s * = h -1 η w (P H -P L ) . We now prove that this mixed equilibrium is an ESS. To prove this result, we compare s * f P H (mut) + (1s * ) f P L (mut) and mut f P H (mut) + (1mut) f P L (mut) for all mut = s * . The difference between two values is exactly w(s *mut)(h(mut)h(s * )). According to lemma 3.5.0.8, h is decreasing function. Hence, (s *mut)(h(mut)h(s * )) is strictly positive for all strategy mut different from s

					the strategy P L if and
	only if	η w	(P η w	(P H -P L ) >	0	R	log	1 + SI NR(P H , P L , r) 1 + SI NR(P L , P L , r)	ς(r) dr.

H -P L ) < R 0 log 1 + SI NR(P H , P H , r) 1 + SI NR(P L , P H , r) ς(r) dr = h(1). For all density function ς, the pure strategy P L dominates the strategy P H if and only if η w (P H -P L ) > R 0 log 1 + SI NR(P H , P L , r) 1 + SI NR(P L , P L , r) ς(r) dr = h(0). Proof. We decompose the existence of the ESS in several cases. 1. P H is preferred to P L : The higher power level dominates the lower if and only if f P H (P H ) > f P L (P H ) and f P H (P L ) > f P L (P L ). These two inequalities implies that η w (P H -P L ) < R 0 log 1 + SI NR(P H , P H , r) 1 + SI NR(P L , P H , r) ς(r) dr.

2.

The following result gives a sufficient condition for the existence of the ESS. Proposition 3.5.0.10. For all density function ς, if h(1) < η w (P H -P L ) < h(0), then there exists an unique ESS s * which is given by s * = h -1 η w (P H -P L ) . Proof. * . We conclude that the mixed equilibrium (s * , 1s * ) is an ESS.

  where E denotes a index set for class or subpopulation, x(t) = (x 1 (t), . . . , x ♯E ), x e (t) = (x e p k ) k=0,...,n e the state of class e with n e + 1 actions. The number τ e p k denotes time delays of the action p k in class e and G(v, x(tτ e v )) v=p k = F e p k (x(tτ e p k )) the associated expected payoff received at time t.

	d dt	x e p j (t) = x e p j (t) G e (w, x(t -τ e w )) |w=p j -	n ∑ k=0	x p k (t)G e (w ′ , x(t -τ e w ′ )) |w ′ =p k	(3.19)
				j = 0, . . . , n e , e ∈ E

  , Sandholm showed that in absence of migration (par example if γ r = 0, ∀ r), replicator dynamics (or general imitation dynamics), Smith dynamics (pairwise comparison dynamics), projection dynamics, Brown-von Neumann-Nash dynamics (excess payoff dynamics) satisfy the positive correlation property. Moreover Brown-von Neumann-Nash dynamics and Smith dynamics satisfy (CNS) property. Here, we extend these results to evolutionary game dynamics with migrations.

Theorem 4.3.1. Suppose that ρ and η generate one of the following dynamics: replicator, Smith, BNN dynamics. Then, the resulting dynamics with multicomponent (ρ, η) in (4.9) is (PC). Theorem 4.3.2. Suppose that the functions ρ and η satisfy (SP). Then, The multicomponent dynamics (4.9) satisfies (PC). In particular, Smith dynamics is (PC).

Proof. Sign preserving property implies that

Table 5 .1: Notations

 5

  Lemma 5.7.7.2. From Lemma 5.7.7.2, Z is monotone. The maximizer is one if

  • Time is slotted. During each time slot t a random number A t of users arrive in the system.Denote by A i t the number of users arriving in vertex i at time t. We assume that arrival process {A i t } t are i.i.d in time with expectation E(A i t ) = α i . Denote by W i t the random variable representing the number of users at vertex i at time t. A user at vertex i require a service with some probability. The user transmits with some probability

	u i ∑ j∈N i W t j	.
	If B i t is the total number of users requiring a service at time t. Then, the random variable B i t follows a binomial law with parameters W i u i t and j ∑ j∈N i W t

  1} ♯X . and M N θ,s (t) is the fraction of players who belong to population of type θ (also called subpopulation θ) and have internal state s. Also let

	1 N	N ∑ j=1	1 {X N j (t)=x} . At each
	time t, M N (t) is in the finite set {0, , . . . , MN 1 N , 2 N θ = N

N

∑ s∈S

  N ), x B N (t)\j [resp. x ′ B N (t)\j ]is the list of states at time t [resp. at time t + 1 N

	1

  2 ; (1, s), m ′ ) ′ = m 2,s ′ for all s ′ ∈ S.

	with m ′ 1,s ′ =	1 N	1 s=s ′ and m ′ 2,s

  ′ , m ′ )) = mm ′ +1 x =x ′ (7.7) Let K ′ = sup x∈S,v,m∈∆ d |r(x, v, m)r(x ′ , v, m ′ )| mm ′ < ∞by hypothesis. It is easy to see that for all x, x ′ ∈ S and m, m ′ ∈ ∆ d :r(x, v, , m)r(x ′ , v, m ′ ) ≤ K ′ d((x, m), (x ′ , m ′ ))By (73), page 117, there is some sequence of increasing bijections λN : [0, ∞) → [0, ∞) s.t. λ N (t)λ N (s) ts → 1uniformly in t and s and d (x N (t), m N (t), (x N (λ N (t)), m N (λ N (t))) → 0 uniformly in t over compact subsets of [0, ∞). Thus there is some N 0 ∈ N such that for N ≥ N 0 and t ∈ [0, T]: d (x N (t), m N (t), (x N (λ N (t)), m N (λ N (t))) ≤ -βt d (x N (t)m N (t)), (x(λ N (t)), m(λ N (t)) dt

	Thus, by Equation (7.6): Thus, by the triangular inequality for d: h N ≤ ≤ ǫ 3 + K ′ T 0 e +K ′ h ǫβe βT 3K ′ ≤ 2ǫ 3 + K ′	(7.8)	(7.10)

λ n (t)λ n (s) ts → 1 uniformly in t and s and y n (t)y(λ n (t)) → 0 uniformly in t over compact subsets of [0, ∞). Fix ǫ > 0, arbitrary and consider

h N := | ∞ 0 e -βt r N (X N (t), v, m N (t))dt -∞ 0 e -βt r(x(t), v, m(t))dt| ≤ ∞ 0 e -βt |r N (x N (t), v, m N (t))r(x(t), v, m(t))|dt First let K = sup x∈S,v,m∈∆ |r(x, v, m)| < ∞

by hypothesis, and pick some time T large enough such that e -βT K/β ≤ ǫ/3. Thus h N ≤ ǫ/3 + T 0 e -βt |r(x N (t), v, m N (t))r(x(t), v, m(t))|dt (7.6) Second, we use the distance on E defined by d((x, m), (x N ≤ ǫ/3 + K ′ T 0 e -βt d (x N (t), m N (t)), (x(t), m(t)) dt (7.9) T 0 e -βt d (x(λ N (t)), m(λ N (t))), (x(t), m(t) dt T 0 e -βt d (x(λ N (t)), m(λ N (t))), (x(t), m(t)) dt (7.11)

  -βt r(u 1 , u 2 , x(t), m(t)) dt s.t. ṁ(t) = f (u 2 , m(t)), m(0) = m 0 , x(0) = x. V(u 1 , u 2 , x) = E x -βt r(u 1 , u 2 , x(t), m(t)) dt + E x e -β∆ V(u 1 , u 2 , x(∆))

						1, 2},
	d dt	m x =	∑ x 1 ,x 2 ,x ′	1 x 1 ,x 2 ,x ′	m x 1 m x 2 L x 1 ,x 2 ;x ′ 1 ,x (u, m)
			-m x ∑ x 2 ,x ′ 1 ,x ′ 2	m x 2 L x,x 2 ;x ′ 1 ,x ′
				0	∆	e -βt r(u 1 , u 2 , x(t), m(t)) dt + E x	∞ ∆	e -βt r(u 1 , u 2 , x(t), m(t)) dt
					∆
		= E x	0		e

2 m x 1 m x 2 L x 1 ,x 2 ;x,x ′ 2 (u, m) + ∑ 2 (u, m)m x ∑ x 1 ,x ′ 1 ,x ′ 2 m x 1 L x 1 ,x;x ′ 1 ,x ′ 2 (u, m)

7.5.2 Computation of R(u 1 , u 2 ; s, m).

We want to compute the value

V(u 1 , u 2 , x) := E x ∞ 0 e

  u 1 , u 2 , x ′ ) d dt m x ′ , where D m x ′ V is the derivative of V in a weak sense, e -β∆ -1 -βt r(u 1 , u 2 , x(t), m(t)) dt -→ r(u 1 , u 2 , x, m 0 )

					∆	-→
	-β, and the term
	E x	1 ∆	0	∆	e

  play Hawk if β 2 (u, x, m 0 , t) > 0 play Dove if β 2 (u, x, m 0 , t) < 0 = max(2/3, m 0 ). Since the solution of the ODE is strictly monotone in time for each stationary strategy, there is at most one time for which β 2 is zero. It is easy to see that if v 2c > 2 3 then the strategy which to play Hawk in state 2 and Dove in state 1 is an equilibrium.

	Index
	Access control, 148		OFDMA, 107
	AIMD, 32
	Aloha, 14		Pricing, 77
	Aloha protocols, 165 Asymmetric time delays, 14		Random number of players, 45 Robust equilibrium, 191
	Battery lifetime, 128 Benefit of Correlation, 87		SINR, 157 Skorokhod topology, 177
	CDMA, 108		Solar-powered systems, 148
	CESS, 74		Stochastic population game, 123
	Choice constrained equilibrium, 98	TCP, 30
	Delayed best response dynamics, 40	Threshold policies, 142
	Delayed BNN dynamics, 40 Delayed Boltzman dynamics, 40	WiMAX, 107
	Delayed imitation dynamics, 41
	Delayed logit dynamics, 40
	Delayed projection dynamics, 41
	Delayed replicator dynamics, 40
	Delayed Smith dynamics, 41
	Drift, 176
	This implies that it is better to play Hawk for Dynamic equilibrium, 175 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Proportion of Hawks ergodic chain, 118 Evolutionarily stable state, 15 Evolutionary game dynamics with migration, 98 Fairness, 35 Frequency state-action, 117 Hybrid power control, 100 Indistinguishability, 165 MAC protocols, 49 MDP, 117 where γ 0 v 2c > γ 1 + γ 1 2 3 4 5 6 7 8 9 0 Mean field dynamics, 172	10
	Mean field interaction, 164 MIMD, 32	time Figure 7.1: Global attractor for u 2 = 1
	Multicomponent dynamics, 97
	Non-reciprocal interaction, 175

2, m 0 , t) = r(H, u, 2, m[u, m 0 ](t))r(D, u, 2, m[u, m 0 ](t)).

The best response, BR(x, u, m[u, m 0 ](t)), against u at t is BR(x, u, m[u, m 0 ](t)) =

The one-shot game with n nodes has

n -1 Nash equilibria and a unique ESS.

An allocation of payoffs is Pareto optimal or Pareto efficient if there is no other feasible allocation that makes every user at least as well off and at least one user strictly better off

The classical Stackelberg games model rational players with asymmetric roles. In this paper, we don't necessary consider rational players. Players update their strategies by trial and error-detection and revising their protocols of strategies process (learning).

Note that this assumption does not holds in dense networks.

Note that the expected horizon can be finite and P(T = t) > 0. It is the case forP(T = t) = δ t-1 (1δ), δ ∈ (0, 1).E(T) = 1 1δ

An allocation of payoffs is Pareto optimal or Pareto efficient if there is no other allocation that makes every node at least as well off and at least one node strictly better off.

A neutrally RID is a weaker notion of RID with non-strict inequality

Acknowledgments

Project BIONETS, Project WiNEM, Project POPEYE for having provided the financial support which made this work possible. Last but the least, gratitude to all, who helped directly or indirectly.

Chapter 5

Stochastic Population Games

This chapter studies non-cooperative population games with several individual states. The population consists of local resources distributed in several areas and several classes of players. Each local resource has a finite number of states. Each member of each class of the population has (i) its own state that he/she controls (ii) a finite set of actions in each pair individual stateresource state, (iii) its instantaneous reward depends on its state, the resource state and the population's profile at the given time, (iv) time-average (coupled) constraints. We apply this model to power management in ALOHA-like wireless networks, solar-powered broadband wireless networks, and battery state-dependent power control in wireless networks with several types of renewable energies.

Introduction

Randomness is implicitly hinted in the requirement of robustness against mutations, that we may view as random deviations. But the assumption of a very large population tends to hide this source of randomness since that randomness tends to average out. The evolutionary games without state may provide an interpretation in which the deterministic game is a limits of games with finitely many players who may take random actions. Such an interpretation can be found in [START_REF] Corradi | Continuous Approximations of Stochastic Evolutionary Game Dynamics[END_REF]. Yet, other sources of randomness have been introduced into evolutionary games. Some authors have added small noise to the replicator dynamics in order to avoid the problem of having the dynamics stuck in some local minimum, see (42; 86; 112) and references therein. The ESS can then be replaced by other related notions such as the Stochastic Stable Equilibrium (SSE) [START_REF] Foster | Stochastic Evolutionary game dynamics[END_REF].

In this paper we introduce a class of stochastic population games with individual states for both players and resources. There are many local interactions among individuals belonging to large populations of players. Each individual stays permanently in the system; from time to time it moves among different individual states, and interacts with other player. The actions of the player along with those with which it interacts determine not only the immediate fitness of the player but also the transition probabilities to the next state it will have. Each individual is thus faced with an MDP in which it maximizes the expected average cost criterion. Each individual knows only the state of its own MDP, and does not know the state of the other players it interacts with. The transition probabilities of a player's MDP are only controlled by that player. The local interactions between players can be viewed as a stochastic game with Note that if u and u ′ are equivalent policies for a given player then for any v used by the rest of the population, the fitness under u and under u ′ are the same.

We now define the notion of evolutionary stable set in standard population games. An ESSet is a set of Nash equilibria which have the following special properties (see [START_REF] Cressman | Evolutionary Dynamics and Extensive Form Games[END_REF]. Definition 5.3.0.2. A set E of symmetric Nash equilibrium is an evolutionarily stable set (ESSet) if, for all q ∈ E, we have J(q, p) > J(p, p) for all p ∈ E and such that J(p, q) = J(q, q). Note that for all strategies p and p ′ in an ESSet E, we have J(p ′ , p) = J(p, p). The concept of ESSet is stronger than Nash equilibrium and there are some simple matrix games in which such an equilibrium set does not exist (see Weibull, 1995, page 48 example 2.7).

In [START_REF] Thomas | On Evoloutionary Stable Sets[END_REF], the author defines Evolutionary Stable Sets and presents an example of ESSet containing a continuum of (Nash) equilibrium strategies, none of which can be an Evolutionarily Stable Strategy (ESS). Another example is of an ESS, a special case of an ESSet restricted to one point.

The ESSet is robust against perturbation by a strategy which is outside the ESSet, but any strategy in the set need not to be robust against perturbation. For instance, every ESSet is asymptotically stable for the replicator dynamic [START_REF] Cressman | Evolutionary Dynamics and Extensive Form Games[END_REF]. Every ESSet is a disjoint union of equilibria.

The stronger notion of equilibrium from evolutionary game theory is the Evolutionarily Stable Strategy (ESS). The concept of ESSets generalize the standard concept of ESS as it is a one-element ESSet.

Defining the Occupation Measure ESS

With the expression (5.8) for the fitness, we observe that we are again in the framework of evolutionary games and can use definition of Theorem 5.3.1 for the Occupation Measure ESS (OMESS) in the stochastic population game: Definition 5.3.0.3. (i) A strategy u ∈ U * is an equilibrium for the stochastic population game if and only if it satisfies F(u, u) ≥ F(v, u).

(5.4)

ESS (OMESS) for the stochastic population game if and only if

• it is an equilibrium, and

We could use the following as an equivalent Definition of OMESS for stochastic population game. Theorem 5.3.1. A strategy u is said to be OMESS if for every v = e u there exists some ǫ v > 0 such that the following holds for all ǫ ∈ (0, ǫ v ):

In equation (5.5) we use a convex combination of two policies. We delay the definition of this to the next section (see Remark 5.3.1.3).

The following result links between the OMESS and the ESSets.

Deterministic strategies

The action taken in each state s is determined with a given distribution the first time the player is in this state. We are able to compute the ESS of this game by transforming our stochastic population game into a standard Evolutionary Game depicted as a matrix game. Lemma 5.7.6.1. The number of pure stationary strategies is 2 n . Proof. A pure strategy is an application from to set of states S\{0} to the actions {h, l}. Thus, the number of the pure strategies is the cardinal of the set {h, l} S\{0} which is exactly 2 n .

Denote by U = {β : y ∈ S\{0} -→ {h, l}} = {h, l} S\{0} the set of pure stationary strategies. The cardinal of U is 2 |S|-1 . Define the following finite population game Γ d = (U, G) : S\{0} is the set of actions. ∆(U) the set of probability measure on U (this corresponds to be mixed strategies obtained by randomizing the pure stationary strategies).

• the payoff matrix is given by

If I denotes the set of states in which the high power is used then

where s ≥ 1, u s ∈ {0, 1} and

.

The size of the matrix is 2 n × 2 n .

The strategy w ∈ ∆(U) is an equilibrium of Γ d if for all u ∈ U,

where

From Nash-Glicksberg's fixed point theorem, and the symmetry of the game, the following result holds: Lemma 5.7.6.2. The game Γ d has at least one symmetric equilibrium (in ∆(U)).

Any symmetric equilibrium of the matrix game Γ d induced an equilibrium of the population game in stationary strategies (this is because the set of symmetric mixed strategies in the matrix game contains the set of stationary strategies). Any strategy u ∈ U satisfying

is an ESS of the evolutionary game. Note that the non-strict inequalities G uu ≥ G u ′ u , ∀ u ′ ∈ U, are necessary condition for u to be an equilibrium in the matrix game Γ d .

Two mobiles K = δ 2 The interaction in each slot and each non-empty individual state is described in the following tabular. Mobile 1 chooses a row, mobile 2 chooses a column. The payoff of mobile 1 (resp. mobile 2) is the first (resp. the second) component of the vector payoff.

1\2

"0" P l P h "0" (0, 0) (0, 1) ⋄ (0, 1) ⋄⋆ P l

(1, 0) ⋄ (0, 0) (0, 1) ⋄⋆ P h (1, 0) ⋄⋆ (1, 0) ⋄⋆ (0, 0) ⋆ The Nash equilibria are represented by ⋆ and Pareto optimal allocation 3 are represented by ⋄.

Three mobiles K = δ 3 The instantaneous reward is presented in the matrix game. Mobile 1 chooses a row, mobile 2 chooses a column and mobile 3 chooses an array M i .

(0, 0, 1) ⋄⋆ (0, 0, 1) ⋄⋆ (0, 0, 0) ⋆ (0, 0, 1) ⋄⋆ (0, 0, 1) ⋄⋆ (0, 0, 0) ⋆ (0, 0, 0) ⋆ (0, 0, 0) ⋆ (0, 0, 0) ⋆ Proposition 5.8.3.1. In any state s = 0, the one-shot local interaction between p ≥ 2 mobiles has an infinite (Nash) equilibria, ( p 1 )2 p-1 = p2 p-1 of them are Pareto optimal, and a unique symmetric (Nash) which is the strategy h (independently of the state).

Note that this one-shot game has a unique evolutionary stable state (see [START_REF] Altman | Evolutionary Power Control Games in Wireless Networks[END_REF]).

Proof. The Nash equilibria and Pareto optimality of the local interaction in state s = 0 can be described as follows:

(i) Symmetric Equilibrium It is clear that r(s, P h , α) ≥ r(s, P l , α) ≥ r(s, 0, α), ∀ α i.e P h weakly dominates P l which weakly dominates 0. Hence, the strategy P h is an equilibrium. Moreover the best reply to the population profile α = (x, y, 1xy) is to play P h if x = 1, and to play any strategy z ∈ [0, 1] if x = 1. Thus, (P h , P h , . . . , P h ) is the unique symmetric equilibrium. At the equilibrium (P h , P h , . . . , P h ), the reward of each mobile is zero. Thus, (P h , P h , . . . , P h ) is not Pareto optimal because the allocation obtained at (P h , P l , . . . , P l ) or (P h , 0, 0, . . . , 0) Pareto dominates zero.

(ii)Pure Equilibria: Fix a mobile m which uses the action P h . Then any action profile of the others mobiles b -m ∈∈ {0, P l , P h } p-1 leads to a Nash equilibrium (no mobile can improve its probability of success by deviating unilaterally). In particular, if k (0 ≤ k ≤ p -1) of the p -1 mobiles choose P l and the pk -1 others use 0 then no mobile can improve its probability of success by deviating unilaterally. The mobile m has exactly

equilibria in which he/she has successful transmission. By changing the role of m, we get ( p 1 )2 p-1 = p2 p-1 pure equilibria with successful transmission. All these pure equilibria are Pareto optimal: if only one terminal uses the high power P h and the others mobiles use 0 or P l ,, then the mobile with the high power P h gets the payoff 1 and the others gets the payoff 0.

Let d = ♯S be the total number of states.

• Time is discrete and takes its value in the set

The global detailed description of the system at time t is

• (Indistinguishability per type) We assume that all players with the same type are indistinguishable i.e play the same role when facing an interaction. This implies that the transition kernel of X N (t) is invariant with respect to changes in the labeling of players. In other words, for any permutation σ of the index set {1, 2, ..., N}:

Thanks to indistinguishability per type assumption we will be able to reduce the study of

to a lower dimension one. This will be used to describe the dynamics of the system.

• Define M N (t) to be the current population profile or occupancy measure i.e

where N θ is the number of players with the type θ. For simplicity we denote by

. . , 1} ♯S . and M N θ,a (t) is fraction of population of type θ (also called subpopulation θ) using action a at time t.

• Local interaction: At time t, a random set of players B N (t) ⊂ 2 {1,2,...,N} are randomly selected from N players for an interaction. Each player j in the set B N (t) takes part in a one-shot game at time t. Each player selected has a revision opportunity of its strategies.

The state of player j can change only if j is selected for an interaction.

The instantaneous payoff of a player j when he/she moves from the state

after an interaction is given by

We assume that for any permutation σ of the index set {1, 2, ..., N}:

Note that the game is not necessarily symmetric since the set of strategies and the payoff function can be different between the types.

The instantaneous expected payoff of a player j in state x at time t is given by

where

We assume that the payoff :

is continuous and bounded. This payoff will represent in averaging over the random number b(t) = ♯B N (t) of interacting players. We will explicit the dependency in next section.

It follows from the indistinguishability per type assumptions that

is Markov. This means that the evolution of one specific player X N j (t) depends on the other players only through the occupancy measure M N (t).

Chapter 7

Stochastic mean field games

We introduce a particular class of stochastic mean field games also called Markov Decision Evolutionary Games with N players, in which each individual in a large population interacts with other randomly selected players. The states and actions of each player in an interaction together determine the instantaneous payoff for all involved players. They also determine the transition probabilities to move to the next state. Each individual wishes to maximize the total expected discounted payoff over an infinite horizon. We provide a rigorous derivation of the asymptotic behavior of this system as the size of the population grows to infinity. We show that under any Markov strategy, the random process consisting of one specific player and the remaining population converges weakly to a jump process driven by the solution of a system of differential equations. We characterize the solutions to the team and to the game problems at the limit of infinite population and use these to construct almost optimal strategies for the case of a finite, but large, number of players. We show that the large population asymptotic of the microscopic model is equivalent to a (macroscopic) Markov decision evolutionary game in which a local interaction is described by a single player against a population profile. We illustrate our model to derive the equations for a dynamic evolutionary Hawk and Dove game with energy level.

Note that we make explicit the dependency on the strategy u but not on J and L, assumed to be fixed.

It follows from our hypotheses that lim

exists.

H3

We assume that the convergence in Equation (7.3) is uniform in m and the limit is Lipschitzcontinuous in m. This is in particular true if one can write, for every strategy

, m), with φ u defined on [0, ǫ] × ∆ where ǫ > 0 and Φ u is continuously differentiable.

converges uniformly in m, m ′ and the limit is Lipschitz-continuous in m, m ′ . This is in particular true if one can write, for every strategy u, as ξ u,x;y (1/N, m, m ′ ). with ξ defined on [0, 1] × ∆ × ∆ and ξ u,x;y is continuously differentiable.

Our model satisfies the assumptions in ( 43), therefore we have the following result: Theorem 7.3.2 (( 43)). Assume that lim N-→∞ M N (0) = m 0 in probability. For any stationary strategy u, and any time t, the random process M N

with initial condition m 0 .

Convergence results

We focus on one player, without loss of generality we can call her player 1, and consider the process (X N 1 , M N ). For any finite N, X N 1 and M N are not independent, however in the limit we have the following: Theorem 7.3.4. Assume that lim

The discrete time process (X

where θ = (θ 2 , ..., θ k ), s = (s 2 , ..., s k ) a = (a 1 , ..., a k ), s ′ = (s ′ 2 , ..., s ′ k )

Note that, contrary to results based on propagation of chaos, we do not assume that the distribution of player states at time 0 is exchangeable. In contrast, we will use Theorem 7.3.4 precisely in the case where player 1 is different from other players. Theorem 7.3.4 motivates the following definition.

Sketch of Proof of Theorem 7.3.4

To prove the weak convergence of Z N , we check the following steps: Without loss of generality, we took the set of states as S = {0, 1, 2, . . . , ♯S} X N j has a jump r with probability

and M N is the continuous process with drift f N .

• We introduce of XN j by scaling with step size

in some sense by a discrete time process ZN = ( XN , mN ) where mN (k) = m(⌊Nt⌋) m solution of the ODE with XN j is the discrete time jump process with transition matrix

We show that d(X N j , XN j ) -→ 0 for any compact of time intervals.

We derive the weak convergence of Z N to (X, m) where m is deterministic and X is random.

Approximation by a discrete time process

The following lemma follows from the lemma 1 and 3 in Benaim and [START_REF] Benaim | Deterministic Approximation of Stochastic Evolution in Games[END_REF]Weibull ( ,2008)), in which we incorporate behaviorial strategies. Lemma 7.3.4.1. For every t > 0 there exists a constant c such that for every ǫ > 0 and N large enough one has

for all m 0 ∈ ∆ d , all every stationary strategy u.

Since C is independent of N, and (e -ǫ 2 C ) N is summable, we can use the dominated convergence theorem: for all ǫ > 0,

By Borel-Cantelli's lemma, for every fixed t < ∞, the random variable ν N,t := sup 0≤τ≤t M N (τ)m(τ) ∞ converges almost completely towards 0. This ν N,t implies that converges almost surely to 0.

We introduce of XN j by scaling with step size 1 N . Then, Z N = (X N , M N ) is approximate in some sense by a discrete time process ZN = ( XN , mN ) where mN (k) = m(⌊Nt⌋) m solution of the ODE where XN j is the discrete time jump process with transition matrix q N i,i+r ( mN (k)) =

Hence, we can write

and this holds for any ǫ arbitrary small. We define the metric

) -→ 0 when N goes to infinity.

Convergence of the discrete time process

To prove the weak convergence of ( XN j , MN ), we check the following steps:

• the discrete time empirical measures MN are tight (follows from Sznitman for finite states) and converges to a martingale problem. The limit m is deterministic measure and is solution of ODE which has the unique solution m (given m 0 , u). Thus, m = m.

• Conditionally to MN , XN j converges to a martingale problem. The jump and drift process X with time dependent transition is given by the limit of the marginal of A N (.| MN , m 0 , x 0 , u). We derive the weak convergence of ( XN j , MN ) to ( X, m) where m is deterministic and X is random. For this we use the Theorem 17.25 and its discrete time approximation in Theorem 17.28 pages 344-347 in Kallenberg. Definition 7.3.4.2. To a game as defined in Section (7.2.1) we associate a "Macroscopic Markov Decision Evolutionary Game", defined as follows. There is one player, (player 1), with state X 1 (t) and a population profile m(t). The initial condition of the game is X 1 (0) = x, m(0) = m 0 . The population profile is solution to the ODE (7.4) and X 1 (t) evolves as a jump process as in Theorem 7.3.4. Further, let r(u; x, m) be the discounted long-term payoff of player 1 in this game, given that X 1 (0) = x and m(0) = m 0 , i.e.

We also consider, as in Section (7.2.3), the case with Θ = 2 types and define by analogy R(u 1 , u 2 ; s, m) as the discounted long-term payoff when player 1 starts in state s and the population profile starts in state m, with player 1 using strategy u 1 and other players strategy u 2 .

In order to exploit the convergence in distribution of the process focused on one player, we need that the payoff be continuous in the topology of this convergence. This is stated in the next theorem. Theorem 7.3.5. Let E = S × ∆ and D E [0, ∞) the set of cadlag functions from [0, ∞) to R, equipped with Skorokhod's topology. The mapping

(u, s(t), m(t)) dt

selected in each time slot of 1/N and linear transition in m, the time averages under the replicator dynamics converge its interior rest points or the boundaries of the simplex.

Equilibrium and optimality

Let U s be the set of strategies. Consider the optimal control problems

The strategy u is an ǫ-optimal strategy for the N-optimal control problem if

Also consider the fixed-point problems

Note that the definition of equilibrium and optimal strategy may depend on the initial conditions. If, for any u ∈ U s , the hypotheses in Section (7.3.7) hold, then we may relax this depen- dency. Theorem 7.3.10 (Finite N). For every discount factor β > 0 the optimal control problem (OPT N ) (resp. the fixed-point problem (FIX N )) has at least one 0-optimal strategy (resp. 0-equilibrium). In particular, there a ǫ N -optimal strategy (resp. ǫ N -equilibrium) with ǫ N -→ 0.

Sketch of Proof of Theorem 7.3.10

We show that for every discount factor β > 0 the optimal control problem (OPT N ) (resp. the fixed-point problem (FIX N )) has at least one 0-optimal strategy. It follows from the existence of equilibria in stationary strategies for finite stochastic games with discounted payoff: The set of pure strategies is a compact space in the product topology (Tykhonov theorem). Thus, the set of behavioral strategies Σ j is a compact space and also convex as the set of probabilities on the pure strategies. For every player j and every strategy profile σ the marginal of the payoffs and constraints functions are continuous for any β > 0 : α j -→ R N j (α j , σ -j , s, m 0 ). Moreover, the stationary strategies is convex, compact and upper and lower hemi-continuous (as a correspondence). Define γ j (s, m 0 , σ) = arg max

Then, γ j (m 0 , σ) ⊆ Σ j is a non-empty, convex and compact set and the product correspondence

is upper hemi-continuous (its graph is closed). We now use the Glicksberg generalization of Kakutani fixed point theorem, and there is a stationary strategy profile σ * such that σ * ∈ γ(s, m 0 , σ * ).

Notes

The goal of this chapter has been to develop mean field asymptotic of interactions with large number of players using stochastic games. Due to the curse of the size of the population, the applicability of atomic stochastic games has been severely limited. As an alternative, we proposed a method for Markov decision evolutionary games where players make decisions only based on their own state and the global system state. We have showed under mild assumptions convergence results, where asymptotics were taken in the number of players. The population state profile satisfies a system of non-linear ordinary differential equations. We have considered very simple class of strategies that are functions only of player's own state and the population profile. We applied to Hawk-Dove interaction with several energy level and formulated the ODEs. We show that the best response depends on the initial conditions.

List of Figures

List of publications

Journal papers

Index

Short abstract

This manuscript presents dynamic foundations of population games with variable number of players and their solutions and stability concepts. We first introduce delayed evolutionary game dynamics and study their stability. Applications to both wired and wireless networks are presented. We then introduce mobility and spatial aspects of players distribution into the network dynamics. This leads to a new class of game dynamics with multicomponent strategies and migration constraints called evolutionary game dynamics with migration. We derived such dynamics for hybrid systems such as power control in heterogenous networks, switching between technologies and migration between different classes of users. After that we focus on stochastic population games with multiple classes of players in which each player has its own state and facing to an evolving vector which represents the population profile. We use this model to analyze resource and energy constrained interactions in wireless networks. Finally, we present a class of mean field games. When taking the asymptotics of finite systems, we derive a new class of game dynamics called mean field game dynamics. This class contains the standard evolutionary game dynamics based on revision of pure actions. We apply this model to analyze spatial random access game and dynamic resource competition game with individual states. We establish a link between mean field games and differential population games in which each player optimizes its long-term objective during its sojourn time in the system subject to the constraint that the population profile evolves according to some mean field game dynamics.