S M Perlaza

S Lasaulce
email: lasaulce@lss.supelec.fr

H Tembine

M Debbah

Learning to Use the Spectrum in Self-Configuring Heterogenous Networks: A Logit Equilibrium Approach

In this paper, we study the particular scenario where several transmitter-receiver pairs communicate subject to mutual interference due to the usage of the same frequency bands.

In particular, we focus on the case of heterogeneous networks, where radio devices have different interests (utility functions), transmit configurations (sets of actions), as well as different signal processing and calculation capabilities. The underlying assumptions of this work are the followings: (i) the network is described by a set of states, for instance, the channel realization vector; (ii) radio devices are interested in their long-term average performance rather than instantaneous performance; (iii) each radio device is able to obtain a measure of its achieved performance at least once after updating its transmission configuration. Considering these conditions, we model the heterogenous network by a stochastic game. Our main contribution consists of a family of behavioral rules that allow radio devices to achieve an epsilon-Nash equilibrium of the corresponding stochastic game, namely a logit equilibrium. A thorough analysis of the convergence properties of these behavioral rules is presented. Finally, our approach is used in the context of a classical parallel interference channel in order to compare with existing results.

INTRODUCTION

Self-configuring heterogeneous wireless networks is the term generally used to identify distributed networks where radio devices from different classes exploit common resources (e.g., frequency bands typically). Often, the difference from one device to another relies mainly of the physical layer technology, e.g., Wi-Fi, Bluetooth, Zigbee, etc, and also on the type of applications the devices are designed for. Within this framework, coordination between radio devices is lim-ited due to the unfeasibility of any kind of message exchange, and thus, each radio device must autonomously determine its own optimal transmit configuration. This is one of the reasons why game theory and the concept of Nash equilibrium (NE) [START_REF] Nash | Equilibrium points in n-person games[END_REF] has been widely accepted in the analysis of this kind of networks. The relevance of the NE stems from the fact that once it is achieved, each radio device's transmission configuration is optimal with respect to the transmit configuration of all the other devices. Thus, the NE is clearly a desired solution from each radio device's standpoint and appears to be relevant in wireless networks where neither coordination at all nor cooperation is possible. Nonetheless, achieving NE is not an easy task. As the main constraint, we highlight the fact that radio devices are not able to observe neither the transmit configuration (e.g., the transmit power level or the channel selected) for the other devices nor the instantaneous global state of the network, i.e., channel realizations, energy constrains and quality of service requirements of all the active radio devices. Thus, the lack of information naturally constraints each radio device to determine its optimal transmit configuration at a given instant. From this perspective, an increasing interest has been observed in the design of behavioral rules to allow radio devices to achieve an NE configuration as a result of a short interaction with its counterparts, similar to a learning process [START_REF] Lasaulce | Game Theory and Learning in Wireless Networks: Fundamentals and Applications[END_REF]. In this direction, the best response dynamics (BRD) [START_REF] Fudenberg | Game theory[END_REF] and fictitious play (FP) [START_REF] Brown | Iterative solution of games by fictitious play[END_REF] has been largely used in wireless communications [START_REF] Scutari | Optimal linear precoding strategies for wideband non-cooperative systems based on game theory -part II: Algorithms[END_REF][START_REF] Perlaza | On the base station selection and base station sharing in self-configuring networks[END_REF][START_REF] Scutari | Potential games: A framework for vector power control problems with coupled constraints[END_REF][START_REF] Scutari | The MIMO iterative waterfilling algorithm[END_REF][START_REF] Pang | Distributed power allocation with rate constraints in Gaussian parallel interference channels[END_REF] and have been shown to converge to NE in certain network topologies. The main constraint in BRD, FP and its variants is the fact that each radio device must observe the transmit configurations of all the other devices, the actual game state and possess a closed form expression of the utility function, which is clearly a very demanding condition in practical scenarios. In some network topologies and depending on the performance metric, this condition can be weakened and a simple broadcast message from each receiver might be enough to implement either BRD or FP [START_REF] Scutari | Optimal linear precoding strategies for wideband non-cooperative systems based on game theory -part II: Algorithms[END_REF]. However, the amount of signaling might be too high depending on the number of dimensions of the scenario, e.g., the number of frequency bands or transmit antennas. More elaborated behavioral rules for achieving equilibria are based on reinforcement learning (RL) [START_REF] Bush | Stochastic models of learning[END_REF][START_REF] Sastry | Decentralized learning of Nash equilibria in multi-person stochastic games with incomplete information[END_REF][START_REF] Young | Strategic learning and its limits (arne ryde memorial lectures sereis)[END_REF]. In RL, the information required by each radio device is simply an observation of its own achieved performance at least every time it changes its transmit configuration. The principle of RL is as follows. After observing the current value of its utility, each radio device updates a probability distribution over all its feasible transmit configurations (or actions). At each update, the probability of the played action increases or decreases depending on the observed utility. In the wireless communication domain, this idea has been used and has been proved to converge to NE in some particular radio resource allocation scenarios [START_REF] Xing | Stochastic learning solution for distributed discrete power control game in wireless data networks[END_REF][START_REF] Zhong | Game theoretic multimode precoding strategy selection for mimo multiple access channels[END_REF]. The main advantages of RL with respect to BRD and FP are numerous (provided it converges to NE). For instance, RL is less demanding in terms of information: only a numerical observation for the achieved utility at each game stage is sufficient to implement the RL rule. However, aside from all the attractive advantages of RL, it has a critical drawback: each observation of the utility is used to directly update the probability distributions without maintaining an estimate of the performance achieved with each transmit configuration. This fact might lead the network to converge to a stationary state which is not an NE. We say stationary, in the sense that none of the radio devices changes its transmit configuration since it is unable to identify that other transmit configurations might bring a higher performance. Consider for instance, the simple power allocation game described in [START_REF] Perlaza | How can ignorant but patient cognitive terminals learn their strategy and utility[END_REF] and general examples in [START_REF] Leslie | Convergent multiple-timescales reinforcement learning algorithms in normal form games[END_REF]. Motivated by this observation, in this paper, we introduce a kind of behavioral rules which are known in the domain of Markov decision processes as actor-critic algorithms [START_REF] Konda | Actor-critic-type learning algorithms for markov decision processes[END_REF][START_REF] Sutton | Policy gradient methods for reinforcement learning with function approximation[END_REF][START_REF] Konda | Actor-critic algorithms[END_REF]. Here, each radio device simultaneously learns both the time-average performance achieved with each of its transmit configurations and the equilibrium probability distribution. This estimation helps to solve the problem encountered in behavioral rules based on reinforcement learning, where convergence is observed but the final network configuration does not correspond to an NE. In particular, contrary to the RL algorithms described above, whenever these behavioral rules lead to a stationary network configuration, it corresponds to a logit equilibrium (LE), which is indeed, an epsilon-close Nash equilibrium concept.

The paper is organized as follows. In Sec. 2 the problem of spectrum sharing is formulated. In Sec. 3, such a problem is modeled by a stochastic game and the concept of -equilibrium and state-independent behavioral strategies is introduced. In Sec. 4 a particular -equilibrium known as logit equilibrium is introduced. In Sec. 5 and Sec. 6 a family of behavioral rules which allow radio devices to achieve a logit equilibrium are presented and a thorough analysis of its convergence in our particular system model is presented. In Sec. 7, we provide some numerical examples in order to evaluate the performance of the proposed behavioral rule. The paper is concluded by Sec. 8

SYSTEM MODEL

In the following, we describe a decentralized self-configuring network where transmitters aim to optimize their individual spectral efficiency by setting up their spectrum access scheme, i.e., number of channels to access, allocated power per channel. Nonetheless, as we shall see in Sec. 3, the contributions of this paper can be applied to scenarios where radio devices have different performance metrics and different sets of transmit parameters to set up. Consider a set K = {1, . . . , K} of transmitters and a set J = {1, . . . , J} receivers. Each ransmitter sends private information to its respective receiver trough out a set S = {1, . . . , S} of orthogonal channels. Here, the orthogonality is assumed in the frequency domain. All transmitters simultaneously use the same set S of channels and thus, communications are subject to mutual interference. Let h (s) j,k (n) represent the channel realization between transmitter k and receiver j over channel s at time n. In our analysis, flat fading channels are assumed during the frame period, i.e., the channel realization is assumed time-invariant during the transmission of one frame, however, the channel might vary from frame to frame period. Denote by h(n) = h (s) j,k (n) ∈ C J•K•S the vector of channel realizations at interval n and let H be the finite set of all possible channel realization vectors (in practice, relevant quantities like channel quality indicators in 3G cellular systems are quantized). Let h (i) be the i-th element of the set H, with i ∈ {1, . . . , |H|}. For each channel use, the vector h(n) is drawn from the set H following a probability distribution ρ = (ρ h (1) , . . . , ρ h (|H|) ) ∈ (H). That is,

ρ h (i) = Pr h(n) = h (i)
, with n ∈ N. The vector of transmitted symbols x k (n), ∀k ∈ K, is an S-dimensional random variable with zero mean and covariance matrix

P k (n) = E x k (n)x H k (n) = diag (p k,1 (n), . . . , p k,S (n))
. For all (k, s) ∈ K × S, p k,s (n) represents the transmit power allocated by transmitter k over channel s. A power allocation (PA) vector for transmitter k ∈ K is any vector

p k (n) = (p k,1 (n), . . . , p k,S (n)) ∈ A k ,
where, ∀k ∈ K, S s=1 p k (n) = p k,max and p k,max is the maximum transmit power of transmitter k. Following this notation, the power allocation vector p (s) k represents the s-th element of the set A k . We denote by N k = |A k | the cardinality of the set A k . We respectively denote the noise spectral density and the bandwidth of channel s ∈ S by N0 and Bs. The total bandwidth is denoted by B = S s=1 Bs, independently of the receiver. We denote the individual spectral efficiency of transmitter k ∈ K as follows,

u k (h(n), p k (n), p -k (n)) = s∈S Bs B log 2 (1 + γ k,s (n)) [bps/Hz],
(1) where γ k,s (n) is the signal-to-interference plus noise ratio (SINR) seen by player k over its channel s at time n, i.e.,

γ k,s (n) = p k,s (n)g (s) j k ,k (n) N0Bs + i∈K\{k} p (s) i (n)g (s) j k ,i (n) , (2)
where, j k ∈ J is the index of the receiver with which transmitter k is associated. Moreover, for all (j, k) ∈ J × K and

n ∈ N, g (s) j,k (n) h (s) j,k (n) 2 .
In the following of this paper, our interest focuses on designing behavioral rules to determine how radio devices choose their power allocation vectors at each interval n in order to to maximize the time-average of their individual spectral efficiency (1).

GAME MODEL

Game Formulation

Consider the stochastic game G described by the 5-tuple

G = (K, {A k } k∈K , {ū k } k∈K , H, {ρ h } h∈H ) . (3)
The sets K = {1, . . . , K}, H = h (1) , . . . , h (H) and

A k = {A (1) k , . . . , A (N k ) k
}, ∀k ∈ K, represent the set of players, the set of network states and the set of actions of player k, respectively. In this analysis, such sets are assumed finite, nonempty, and time-invariant sets. In the game G, each player represents an active transmitter of the network. The set of actions of a given transmitter corresponds to the set of all its feasible transmission configurations, for instance, the power allocation vectors, etc. The set H is the set of all possible channel realizations. The vector ρ = (ρ h (1) , . . . , ρ h (|H|) ) ∈ (H) is the probability distribution described in Sec. 2. The function ūk is the long-term performance metric of player k, for all k ∈ K, and will be described later. The game G is played stage by stage ad infinitum. At each stage n, every player k adopts an action e.g., a power allocation vector p k (n) ∈ A k . At the end of the stage, player k observes a numerical value ũk (n) of its achieved performance e.g., the individual spectral efficiency ũk

(n) = u k h(n), p k (n), p -k (n) .
Note that these observations might be noisy [START_REF] Perlaza | Game Theoretic Approaches to Spectrum Sharing in Decentralized Self-Configuring Networks[END_REF]. However, this case is not the focus of the present work. Note that all the information gathered by player k at stage

n is the 2-tuple (a k (n), ũk (n)) ∈ A k × R. We denote by θ k (n) the available information gathered by player k up to interval n, i.e., θ k (n) = {(a k (0), ũk (0)), . . . , (a k (n -1), ũk (n -1))}. (4)
We refer to θ k (n) as the private history of player k at time n. The set of all possible private histories of player k at time n is denoted by Θ k (n), and,

Θ k (n) = (A k × R) n . (5)
At each game stage, transmitters select their respective PA vector following a probability distribution

π k (n) = π k,A (1)
k (n), . . . , π k,A (N k ) k (n) ∈ (A k )
which is built based on its private history θ k (n). Here,

∀n k ∈ {1, . . . , N k }, π k,p (n k ) k
(n) represents the probability that player k plays action p

(n k ) k ∈ A k at time n, i.e., π k,A (n k ) k (n) = Pr p k (n) = p (n k ) k . ( 6
)
Such probability is built as follows. There exists a function for each time interval n, which we denote by σ k,n , and is defined as follows,

σ k,n : Θ k (n) → (A k ) . (7)
Thus, at each time n player k determines its probability distribution π k (n) based on its private history θ k (n). Each player k possesses an infinite sequence of functions,

σ k = {σ k,n } n>0 , (8)
such that at each game stage n, it is able to generate the corresponding distribution π(n). We refer to the sequence of functions σ k as the behavioral strategy (BS) of player k.

In the following, we denote by Σ k , the set of all possible BS of player k and let Σ = Σ1 × . . . × ΣK be the set of all BS profiles. Now note that, given any behavioral strategy σ, the initial channel selection profile p(0) ∈ A induce a set of sequences of probability distributions {π k (n)} n>0 , for all k ∈ K. Hence the set of sequences {π k (n)} n>0 induced by p(0) ∈ A together with the initial probability distributions π(0) = (π1(0), . . . , πK (0)) induce a probability distribution over all the possible sequences of action profiles {p(0), p(1), . . . }.

We denote the expectation with respect to such probability distribution by E (π(0),σ) . Then, the long-term expected performance of player k can be measured by the function, ūk : Σ1 × . . . × ΣK → R+, where,

ūk (σ k , σ -k |π(0) ) = lim n→∞ 1 n E (π(0),σ) n-1 i=0 ũk (i) . ( 9
)
The function [START_REF] Leslie | Convergent multiple-timescales reinforcement learning algorithms in normal form games[END_REF] captures the situation in which the interaction between all transmitters in the network lasts many time intervals and the instantaneous performance is insignificant compared with the performance in all the other time intervals.

In the following, the game G is analyzed assuming that the aim of each player k is to choose a BS σ k ∈ Σ k such that it maximizes its performance metric [START_REF] Leslie | Convergent multiple-timescales reinforcement learning algorithms in normal form games[END_REF] given the BS σ -k ∈ Σ -k adopted by all the other players and regardless of the initial action profile a(0) ∈ A. In particular, we look for a the BS profile σ * = (σ * 1 , . . . , σ * K ) ∈ Σ1 × . . . ΣK such that none of the players can obtain a performance improvement by unilaterally using other BS. We provide a more precise concept of this expected solution of the stochastic game G in the following subsection.

Nash Equilibrium and -Equilibrium

In the following, we describe the concept of -equilibrium and Nash equilibrium in the context of the stochastic game G. First, let us define the -equilibrium as follows, Definition 1 ( -equilibrium in the game G). Let > 0. In the game G, a strategy profile σ * ∈ Σ1 × . . . × ΣK is an -equilibrium if it satisfies, for all k ∈ K and for all

σ k ∈ Σ k , ūk (σ * k , σ * -k |π(0) ) ūk (σ k , σ * -k |π(0) ) -, (10)
independently of the initial probability distributions π k (0) ∈ (A k ), ∀k ∈ K.

An -equilibrium can be interpreted as a BS profile such that, none of the players can obtain an improvement higher than by unilaterally changing its own BS. Note also that, by letting = 0 in Def. 1, the classical definition of Nash equilibrium is obtained.

In the following section, we discuss the feasibility of achieving these equilibrium concepts in the game G.

Stationary State Independent Behavioral Strategies

Stationary state independent behavioral strategies (SSI-BS) are considered the simplest class of BS in stochastic games [START_REF] Solan | Stochastic games. Encyclopedia of Database Systems[END_REF][START_REF] Neyman | Stochastic Games and Applications[END_REF]. Let the set of stationary state independent behavioral strategy (SSI-BS) profiles be denoted by Σ and let a given SSI-BS profile be defined as follows, Definition 2 (Stationary State Independent BS). Consider the game G and let σ ∈ Σ be a behavioral strategy. Then, σ is said to be stationary state-independent (SSI) if for all k ∈ K and any two private histories

θ k (n) ∈ Θ k (n) and θ k (m) ∈ Θ k (m), with n = m, it follows that σ k,n (θ k (n)) = σ k,m (θ k (m)) , (11)
independently of the states h(n) and h(m).

From Def. 2, it can be implied that for a player k, an SSI-BS does not depend on any of the previous channel selections p k (0), . . . , p k (n -1) and neither on the previous nor current channel realizations h(0), . . . , h(n). Thus, a SSI-BS σ k ∈ Σ can be identified by a vector π = (π1, . . . , πK ) ∈ (A1) × . . . × (AK ), such that, ∀k ∈ K and

∀ (θ k (n), h(n)) ∈ Θ k (n)×H, it holds that σ k,n (θ k (n)) = π k .
In the sequel, we indifferently use the infinite set of sequences σ k = {π k }n>0 or the vectors π k , with k ∈ K, to refer to the SSI-BS σ. Moreover, with a slight abuse of notation, we indifferently write either ūk (σ k , σ -k ) or ūk (π k , π -k ) to denote the achieved performance of player k given the SSI-BS σ. In the following, we restrict the analysis of the game G to the set of SSI-BS. The justifications for this choice are manifold. First, note that none of the players is able to identify the current network state at each stage of the game. Moreover, given the information gathered by player k up to stage n -1, i.e., θ k (n -1), with k ∈ K and n > 1, it is not possible to infer any information about the network state h(n). This implies that, player k is unable to calculate an optimal probability distribution π k (n) at each stage n, since the ignorance of the network state at each stage implies the ignorance of a closed form expression of the instantaneous performance metric u k and the long-term performance metric ūk . Thus, regardless of the stage n and all the information gathered up to such stage n, all players face the same scenario.

LOGIT EQUILIBRIUM

In this section, we introduce the concept of an -equilibrium known as logit equilibrium. We define the logit equilibrium in the context of the game G in SSI-BS and we conclude this section claiming its existence and providing some insight on its uniqueness in the game G.

Logit Equilibrium in SSI-BS

Before we provide a formal definition of the logit equilibrium, we introduce the idea of logit best response.

Definition 3 (Logit Best Response).

Consider the game G and let the vector π -k ∈ (A1) × . . . × (A k-1 ) × (A k+1 ) × . . . × (AK ) represent a given SSI-BS profile, with k ∈ K. Then, the logit best response of player k, with parameter γ k > 0, is the probability distribution β

(γ k ) k (ū k (•, π -k )) ∈ (A k ) such that, β (γ k ) k : R N k → (A k ) is the logit function, β (γ k ) k ( ūk (•,π -k )) =   β (γ k ) k,A (1) k
(ūk(•,π-k)),...,β (γ k ) k,A (N k ) k (ūk(•,π-k))   
and ∀n k ∈ {1, . . . , N k },

β (γ k ) k,A (n k ) k (ū k (•, π -k )) = exp γ k ūk (e (n k ) k , π -k ) N k m=1 exp γ k ūk (e (m) k , π -k ) . ( 12
)
From Def. 3, it can be implied that at each stage of the game, every power allocation vector of a given transmitter has a non-zero probability of being played, i.e., ∀k ∈ K and ∀n k ∈ {1, . . . , N k } and ∀γ k ∈ R+, it holds that, β

(γ k ) k,A (n k ) k (ū k (•, π -k )) > 0.
More generally, it can be stated that the logit best response in SSI-BS is represented by a probability distribution that assigns high probabilities to the power allocation vectors associated with a high average individual spectral efficiencies and low probability to power allocations associated with low average individual average performance. Finally, note also that conversely to the case of the best response in the case of Nash equilibrium [START_REF] Nash | Equilibrium points in n-person games[END_REF], the logit best response of player k is unique for all the SSI-BS profiles the other players might adopt. Using Def. 3, we define the logit equilibrium as follows, Definition 4 (Logit Equilibrium in SSI-BS). Consider the game G and let the vector π * = (π * 1 , . . . , π * K ) ∈ (A1)×. . .× (AK ) represent a stationary state-independent behavioral strategy (SSI-BS). Then, π * is a logit equilibrium SSI-BS profile with parameter γ = (γ1, . . . , γK ) if for all k ∈ K, it holds that,

π * k = β (γ k ) k ūk e (N k ) 1 , π * -k , . . . , ūk e (N k ) N k , π * -k . ( 13
)
At the logit equilibrium, since all actions are played with non-zero probability, at some given game stages the actions taken by player k do not maximize the instantaneous performance u k , which negatively impacts the long-term performance ūk . In [START_REF] Perlaza | Game Theoretic Approaches to Spectrum Sharing in Decentralized Self-Configuring Networks[END_REF], it has been shown that the maximum loss of performance player k might experience is not higher than 1 γ k ln(N k ), which confirms that the logit equilibrium falls in the class of -equilibrium described in Def. 1.

Existence and Uniqueness of the LE

The main result regarding the existence of an LE in the game G is the following.

Theorem 5 (Existence of the LE). The stochastic game G has at least one logit equilibrium in the set of stationary state-independent behavioral strategies.

The proof of Theorem 5 is presented in [START_REF] Perlaza | Game Theoretic Approaches to Spectrum Sharing in Decentralized Self-Configuring Networks[END_REF] The uniqueness of the LE in the game G in SSI-BS is strongly related to the parameters γ k , with k ∈ K. For instance, when ∀k ∈ K, γ k → 0, there exits a unique LE in SSI-BS and corresponds to the vectors π k = 1 N k (1, . . . , 1) ∈ (A k ). This LE (uniform probability distributions over the sets A1, . . . , AK ) is unique, independently of the number of NE the game G might possess. Conversely, when ∀k ∈ K, γ k → ∞, the set of LE becomes identical to the set of NE and thus, the game G exhibits as many LE as NE might exist in G in SSI-BS. The number of NE (and thus, the number of LE) of the game G is often not unique. In fact, it has been already shown that even the simplest cases exhibit several NE. For instance, when K = {1, 2} and H and J are unitary, the number of NE can be 1 or 2 in pure strategies [START_REF] Perlaza | On the base station selection and base station sharing in self-configuring networks[END_REF]. When, J = K = {1, 2} and H is unitary, the number of NE can be 1, 2 or 3 in pure strategies [START_REF] Rose | On the Nash equilibria in decentralized parallel interference channels[END_REF].

LEARNING LOGIT EQUILIBRIA

In this section, we design behavioral strategy profiles σ = (σ1, . . . , σK ) ∈ Σ such that given the information gathered by player k at each stage n, i.e., given the sets {θ k (n)}n>0 for all k ∈ K, it is able to generate infinite sequences {π k (n)}n>0, such that, limn→∞ ||π k (n) -π * k || = 0, where π * = (π * 1 , . . . , π * K ) ∈ (A1) × . . . × (AK ) is a logit equilibrium in SSI-BS of the game G (Def. 4). An important remark is that achieving a logit equilibrium implies that at a given game stage, radio devices are able to build logit best responses in order to face the other players behavioral strategies. Nonetheless, building such a probability requires each player to know the vector ūk (•, π -k (n)).

Thus, radio devices must estimate their corresponding vector ūk (•, π -k (n)) at each game stage n based on their current history θ k (n) in order to generate their logit best response. Let the N k -dimensional vector

ûk (n) = ûk,A (1) k (n), . . . , ûk,A (N k ) k (n) (14)
be the estimation, at the game stage n, that player k possesses of the vector ūk (•, π -k (n)). In the following, we present a result initially introduced in [START_REF] Perlaza | How can ignorant but patient cognitive terminals learn their strategy and utility[END_REF][START_REF] Perlaza | Game Theoretic Approaches to Spectrum Sharing in Decentralized Self-Configuring Networks[END_REF], which allows radio devices to simultaneously estimate the vector ūk (•, π -k (n)) and determine the probability distribution π k (n), with which it chooses the power allocation vector p k (n).

Theorem 6 (LE Behavioral Strategy). Consider the game G and assume that for all k ∈ K and for all n k ∈ {1, . . . , N k }, it holds that for all n ∈ N,

                   ûk,p (n k ) k (n) = ûk,p (n k ) k (n-1)+ α k (n) 1 p k (n-1)=p (n k ) k π k,p (n k ) k (n)   ũk (n-1)-û k,p (n k ) k (n-1)   , π k,p (n k ) k (n) = π k,p (n k ) k (n-1)+ λ k (n)   β (γ k ) k,p (n k ) k ( ûk (n))-π k,p (n k ) k (n-1)   , (15) where
, p k (0) ∈ A k , ûk (0) ∈ R N k and π k (0) ∈ (A k )
are arbitrary initializations. Consider also the following assumptions and for all (j, k) ∈ K 2 , the learning rates α k and λj satisfy that (B0) lim

T →∞ T n=0 α k (n) = +∞ and lim T →∞ T n=0 α k (n) 2 < +∞, (B1) lim T →∞ T n=0 λ k (n) = +∞ and lim T →∞ T n=0 λ k (n) 2 < +∞,
and,

(B2) lim n→∞ λj(n) α k (n) = 0.
Then, if the set of coupled stochastic approximation algorithms (15) converge, it holds that,

lim n→∞ π k (n) = π * k , (16)
lim n→∞ ûk,p (n k ) k (n) = ūk (e (N k ) n k , π * -k ), (17)
where

π * k ∈ (A k ) satisfies that, π * k =β (γ k ) k ūk e (N k ) 1 ,π * -k ,...,ū k e (N k ) N k ,π * -k . ( 18
)
The proof of Theorem 6 is presented in the most general case of the game G in [START_REF] Perlaza | Game Theoretic Approaches to Spectrum Sharing in Decentralized Self-Configuring Networks[END_REF], based on previous results on stochastic approximations [START_REF] Leslie | Convergent multiple-timescales reinforcement learning algorithms in normal form games[END_REF][START_REF] Borkar | Stochastic approximation with two timescales[END_REF]. The behavioral rule in Theorem 6 has been proved to convergence in several classes of games, e.g., potential games [START_REF] Monderer | Potential games[END_REF]. However, in general, the game G does not fall in any of those classes and thus, the convergence must be proved.

CONVERGENCE ANALYSIS

In the following, we consider some particular cases of the scenario described above and we present some results on the convergence of the proposed algorithms.

6.1 Parallel Multiple Access Channels (J = 1) Note that when J = 1, the scenario described before reduces to a parallel multiple access channel [START_REF] Cover | Elements of information theory[END_REF], and thus, the corresponding game has a potential function φ, defined as follows,

φ(π) = h∈H s∈S log 2 σ 2 s + K i=1 pi,s h (s) 1,i (n) 2 ρ h K j=1 πj,p j . (19)
Then, from Theorem 4.6.1 in [START_REF] Perlaza | Game Theoretic Approaches to Spectrum Sharing in Decentralized Self-Configuring Networks[END_REF], the following holds, Proposition 7. When J = 1, the algorithm in [START_REF] Perlaza | Game Theoretic Approaches to Spectrum Sharing in Decentralized Self-Configuring Networks[END_REF], with λ1 = . . . = λK , converges to a logit equilibrium of the resulting game G.

In particular, the BRD has been used to achieve equilibria in this context [START_REF] Scutari | Optimal linear precoding strategies for wideband non-cooperative systems based on game theory -part II: Algorithms[END_REF] and proved to converge to NE. Other algorithms such as FP and its variants are also shown to converge [START_REF] Perlaza | On the fictitious play and channel selection games[END_REF]. Nonetheless, the advantages of the algorithm in [START_REF] Perlaza | Game Theoretic Approaches to Spectrum Sharing in Decentralized Self-Configuring Networks[END_REF] over the BRD (and FP) stem from the fact that there is no synchronization required for the radio devices to coordinate in sequential or simultaneous updates of their configuration [START_REF] Rose | Learning equilibria with partial information in wireless networks[END_REF]. Here, the only requirement for each device is to observe a numerical value of its SINR each time it updates its configuration, independently of all the other devices' strategy update timing. Moreover, contrary to the case of simultaneous BRD, the convergence is always ensured [START_REF] Perlaza | On the fictitious play and channel selection games[END_REF].

Parallel Interference Channels (J = 2 )

When J = 2 receivers, the resulting network topology describes in the simplest case, i.e., when ∀j ∈ J , |Kj| = 1, the parallel interference channel [START_REF] Cover | Elements of information theory[END_REF]. In the most general case, i.e., when ∀j ∈ J , |Kj| > 1, it describes a 2-cell multicarrier cellular channel (in uplink), e.g., an OFDM cellular network system. In both cases, the resulting game is no longer a potential game and thus, convergence of the BRD (and FP) are not always observed [START_REF] Scutari | The MIMO iterative waterfilling algorithm[END_REF][START_REF] Scutari | Simultaneous iterative water-filling for Gaussian frequency-selective interference channels[END_REF]. Nonetheless, using our algorithm, convergence is always achieved, as we show in the following proposition, Proposition 8. When J = 2, λi = λ1, ∀i ∈ K1, λj = λ2, ∀j ∈ K2 and

lim n→∞ λ1(n) λ2(n) = 0 ( 20
)
the algorithm in (15) converges to a logit equilibrium of the resulting game G.

The proof of proposition 8 follows from Theorem 4.6.2 in [START_REF] Perlaza | Game Theoretic Approaches to Spectrum Sharing in Decentralized Self-Configuring Networks[END_REF].

NUMERICAL ANALYSIS

In this section, we provide a numerical analysis of the performance achieved by radio devices following the behavioral rule proposed in this article. First, we focus on the achieved performance in limited time. Here, our interest focuses in determining the sum spectral efficiency when the learning period is limited. Note that the theoretical analysis requires infinite time for convergence, which is not practically appealing. Second, we focus on the impact of the number of choices each radio devices might possess at a given time. Here, we verify the counter intuitive result which states that increasing the set of choices each radio device possesses during the whole game realization might lead to worse global performance.

Convergence in Finite Time

In the following, we say that a learning algorithm converges to either a LE, if there exists a number n ∈ N1, given a sufficiently small number > 0, such that,

∀k ∈ K, ||π k (n) -π * k || < , (21)
where π * = (π * 1 , . . . , π * K ) is a LE strategy profile, respectively. We measure the convergence time, as the number of iterations required to observe convergence in the sense of [START_REF] Sastry | Decentralized learning of Nash equilibria in multi-person stochastic games with incomplete information[END_REF], assuming that all convergence updates occur periodically at a constant frequency. We refer as achieved performance to the time-average utility observed by the radio devices from the first strategy update up to convergence. As another note, we point out the fact that there exist other ways to measure convergence other than Euclidian distance as in [START_REF] Sastry | Decentralized learning of Nash equilibria in multi-person stochastic games with incomplete information[END_REF]. For instance, using the Kullback-Leibler divergence as in [START_REF] Mertikopoulos | Distributed learning policies for power allocation in multiple access channels[END_REF]. In the following, we evaluate numerically the convergence time and the achieved performance by radio devices in the particular cases described above. Consider the decentralized wireless network described in Sec. 2, when there exist only two transmitters, i.e., K = {1, 2}, two receivers, i.e., J = {1, 2} and two channels S = {1, 2}. We limit the set of power allocation vectors, to the channel selection case, i.e.,

A k = p (s)
k = p k,max es : ∀s ∈ S, es = (es,1, . . . , es,S) , ∀r ∈ S \ s, es,r = 0, and es,s = 1} .

In order to facilitate a fair comparison of the behavioral rule in Theorem 6 with existing results [START_REF] Rose | Learning equilibria with partial information in wireless networks[END_REF], we consider the set H is unitary. This implies that the stochastic game reduces to play the same game repeatedly ad infinitum. At each stage, the corresponding one-shot game G (h) might have either one NE in pure strategies or two NE in pure strategies plus one NE in mixed strategies [START_REF] Rose | On the Nash equilibria in decentralized parallel interference channels[END_REF]. Here, we generate 10000 channel realizations. For each channel realization, we calculate the sum of individual spectral efficiencies at the NE. We treat separately the case of unique and multiple equilibria. Similarly, for each channel realization, we determine the sum of individual spectral efficiencies achieved by both transmitters when their learning time is limited to a fixed number of time intervals (game repetitions). In Fig. 1, we plot the average sum of individual spectral efficiencies as a function of the number of times the game is let to be repeated. On the left, we consider the case of a unique equilibrium and on the right, the case of multiple equilibria is considered. In Fig. 1, it is clearly seen that the longer the radio devices are left to learn, the better their achieved performance. Interestingly, after certain number of iterations, radio devices are able to achieve an average sum spectral efficiency which is better than the worst NE, i.e., the NE with the lowest sum spectral efficiency. Another important remark, is the impact of the parameter γ k , for all k ∈ K. In Fig. 1, it is also shown that the smaller γ k the more interest player k has in playing uniformly all its own actions. Conversely, when the parameter γ k is large, the corresponding radio devices is tempted to use more often the best configuration and thus, it achieves a better performance.

Impact of the Number of Choices

In this subsection, we increase the number of available channels and we let each transmitter to use either a unique channel or any subset of adjacent channels. Thus, if we consider S channels, the cardinality of the sets A k is S 2 (1 + S), for all k ∈ K. Here, we generate 10000 channel realizations

) , α2(n

) = 1 n ( 2 3
) and λ1(n) = 1 n . Moreover, SN R =

) , α2(n) = 1

n ( 2 3
) and λ1(n) = 1 n . Moreover, SN R = p k,max σ 2 = 10 dBs.

and we let radio devices to learn through 1000 game repetitions. In Fig. 2, it is shown that increasing the number of available channels leads to a loss of spectral efficiency. This observation is due to the fact that letting each radio device to use an additional channel implies increasing the number of possible power allocation vectors. This implies that the radio device has to try more power allocation vectors in order to estimate the individual spectral efficiency it obtains with each of them. However, such power allocation vectors might not be those bringing the highest individual spectral efficiency, and thus, using them reduces the average individual spectral efficiency.

CONCLUSIONS

This paper presents a framework to learn distributed strategies in communication networks where the transmitters may not know much about the structure of the game, have different action sets, have different utility functions, and only know the realizations of their utility and nothing else (in particular the utility function is not assumed to be known). The interesting case where transmitters learn at different speeds is addressed, which is also a possible feature in heterogeneous networks. In this context, we have shown that the proposed (behavioral) strategies are independent of the closed-form expression of the performance metric and the set of transmit configurations of each radio device. Here, the required conditions for our results to be valid are: Each device must possess a finite set of feasible actions and it must be able to observe its achieved instantaneous performance at least once after each update of its transmit configuration. When the proposed behavioral rule is used and convergence is observed, then the system is said to achieve a Logit equilibrium. A convergence analysis was carried out for a particular scenario considering as performance metric the individual spectral efficiency.

Figure 1 :

 1 Figure 1: Achieved sum spectral efficiency in the two-transmitter two-receiver two-channel game, when transmitters are limited to channel selection. Here α1(n) = α2(n) = 1 n ( 3 4

p k,max σ 2 =

 2 10 dBs. (left) Case of unique NE. (right) Case of multiple NE.

  Number of Channels (S)Sum Spectral Efficiency in bps/Hz. Logit Equilibrium, ∀k ∈ K, 1 γk = 0.2 Logit Equilibrium, ∀k ∈ K, 1 γk = 0.4 Logit Equilibrium, ∀k ∈ K, 1 γk = 1

Figure 2 :

 2 Figure 2: Achieved sum spectral efficiency in the two-transmitter two-receiver N-channel game, when transmitters are limited to use only one channel or any combination of adjacent channels. Here α1(n) = α2(n) = 1 n ( 3 4

The long-term behavior of the decentralized network described in the previous section can be modeled by a stochastic game. In the following, we formulate such stochastic game and the concept of epsilon-equilibrium. Finally, we describe a particular class of behavioral strategies, to which we restrict the analysis of our game.


S M Perlaza

M Debbah

Modeling Noisy Feedback in Decentralized Self-Configuring Networks

This paper introduces a generalization of the notion of Nash equilibrium (NE), namely quantal response equilibrium (QRE). In the QRE, radio devices choose their transmit/receive configuration taking into account that the estimation of their own performance contains a noise component. Here, it is shown that the notion of QRE neatly models decentralized self-configuring networks (DCSN) where feedback messages are impaired by quantization noise or decoding errors. The main contribution of the paper is twofold. First, we show that under the presence of noise in the estimation expected utility, the notion of NE no longer holds, as players cannot be considered rational. Second, we introduce a learning technique that converges to a QRE in a fully decentralized fashion. We present numerical results in the context of a channel selection problem in a parallel multiple access channel in order to illustrate our theoretical results.

I. INTRODUCTION

Game theory is by now accepted as one of most suited theories for the study of decentralized self-configuring networks (DSCN) [START_REF] Lasaulce | Game Theory and Learning in Wireless Networks: Fundamentals and Applications[END_REF]. In particular, the concept of Nash equilibrium (NE) [START_REF] Nash | Equilibrium points in n-person games[END_REF] has played a key role in the performance analysis of DSCN. The interest on NE relies mainly on two reasons. First, it is a network state at which the transmit configuration of each radio device is optimal with respect to the configurations adopted by all the other devices. Second, the NE can be the result of the interaction of several radio devices following particular behavioral rules, for instance, best response dynamics, fictitious play, etc, [START_REF] Rose | Learning equilibria with partial information in wireless networks[END_REF]. From this point of view, radio device or network designers can develop algorithms that allow DCSN to achieve NE in a fully decentralized way [START_REF] Lasaulce | Game Theory and Learning in Wireless Networks: Fundamentals and Applications[END_REF]. Nonetheless, the application of the notion of NE in DSCN is largely constrained by several real-implementation constraints. As a first constraint, we identify the fact that radio devices are assumed rational. That is, radio devices must be capable of reliably determining the performance they achieve by playing each of their actions (transmit configurations). This assumption clearly does not always hold, since often this procedure involves feedback messages that are subject to quantization and decoding errors. Thus, the assumption of rational radio devices simply fails. Some references, where rationality is the main assumption, are for instance [START_REF] Larsson | Competition Versus Cooperation on the MISO Interference Channel[END_REF], [START_REF] Berry | Shannon meets nash on the interference channel[END_REF] in the analysis of interference channels and [START_REF] Meshkati | A game-theoretic approach to energy-efficient power control in multicarrier cdma systems[END_REF], [START_REF] Belmega | Power allocation games for mimo multiple access channels with coordination[END_REF] in the analysis of the multiple access channels.

In this article, we introduce the idea of quantal response equilibrium (QRE) [START_REF] Mckelvey | Quantal response equilibria for extensive form games[END_REF], [START_REF] Mckelvey | Quantal response equilibria for normal form games[END_REF], which can be considered as a generalization of the notion of NE. At the QRE, players choose their actions taking into account that the observed utility contains an unknown noise component. This implies that players are not always able to choose the action that maximizes their individual utility. Indeed, the noise component might lead them to choose any of their feasible actions. As a result, the assumption of rationality is no loger needed, which makes this notion of equilibrium better suited for DCSN. The notion of QRE can be specialized for certain particular types of noise. For instance, when the noise component added to the observed average utility follows a Gumble distribution, the QRE is reduced to the notion of logit equilibrium (LE) [START_REF] Anderson | The logit equilibrium: A perspective on intuitive behavioral anomalies[END_REF] already used in [START_REF] Perlaza | Learning to use the spectrum in self-configuring heterogeneous networks[END_REF] in the context of spectrum sharing. Another special case, the probit equilibrium, appears when the noise is Gaussian distributed [START_REF] Anderson | The logit equilibrium: A perspective on intuitive behavioral anomalies[END_REF].

The main contribution of this article is a game theoretical framework based on the notion of QRE to study DSCN when radio devices are not fully rational due to the imperfect estimation of their individual performance. This comprehends mainly the case of noisy feedback. We also show that the QRE, as the NE, can be learnt by radio devices in a fully decentralized manner. We provide some behavioral rules than convergence to QRE.

The sequel of this paper unfolds as follows. In Sec. II, we present the general system model which is later described by a one-shot game. In Sec. III, we introduce the notion of QRE. In Sec. IV, we present a behavioral rule that allows players to achieve QRE in the general context of DCSN. In Sec. V, we present some numerical examples to illustrate our results in the context of a parallel multiple access channel. Finally, in Sec. VI, we state our conclusions.

II. NOTATION AND GAME THEORETIC MODEL

A. Notation

In the sequel of this paper, vectors and scalars are respectively denoted by boldface lower case symbols and lower case symbols. The expectation operator with respect to a random variable x is denoted by E

x [•].
The indicator function is denoted by 1 {condition} and it equals 1 (resp. 0) when condition is true (resp. false). The set of non-negative and strictly positive integers is denoted by N = {0, 1, 2, . . .} and N 1 = N \ {0}, respectively. The space of real vectors of dimension n ∈ N is denoted by R n . Let X be a finite set. We denote by (X ) the unit simplex over the elements of X , that is, the set of all probability distributions over the elements of the set X . The cardinality of X is denoted by |X | ∈ N and the set of all subsets of X including the set X itself is denoted by 2 X . We denote by {e n,n = 1. Given a vector a = (a 1 , . . . , a N ) in a given space of dimension N ∈ N, we denoted by a -n , with n ∈ {1, . . . , N }, the vector a -n = (a 1 , . . . , a n-1 , a n+1 , . . . , a N ) in the corresponding space of dimension N -1. With a slight abuse of notation, we often write a = (a n , a -n ) to highlight the n-th component of the vector a.

B. Game Theoretical Model

Assume that the game in normal form G = K, {A k } k∈K , {u k } k∈K models a DSCN. The set K = {1, . . . , K} contains the indexes of all transmit-receiver pairs. A transmitter-receiver pair k, with k ∈ K, chooses its transmit/receive configuration out of a set of N k possible choices. We denote the n-th choice by A (n) k and the set of all possible choices by

A k = {A (1) k , . . . , A (Nk) k }.
Here, a transmit/receive configuration describes, for instance, a particular power allocation policy, modulationcoding scheme, decoding order, scheduling policy or any combination of these parameters. In the following, we refer to the transmit-receiver pairs and their corresponding transmit/receive configurations as players and actions, respectively. An action profile of the game G is a vector a = (a 1 , . . . , a K ) ∈ A, where A = A 1 × . . . × A K . The utility function u k : A → R, for all k ∈ K, measures the benefit (in the sense of Morgestein -von Neumann [START_REF] Neumann | Theory of games and economic behavior[END_REF]) player k obtains when it plays a specific action given the actions adopted by all the other players. Typical utility functions in DSCN are transmission rates, bit error rates, spectral efficiency, etc. See [START_REF] Lasaulce | Methodologies for analyzing equilibria in wireless games[END_REF] and references therein. The game G is played as follows. Player k chooses its action a k following a probability distribution

π k = π k,A (1) k , . . . , π k,A (N k ) k ∈ (A k ).
Here, ∀k ∈ K and

∀n k ∈ {1, . . . , N k }, π k,A (n) k represents the probability that player k plays action A (nk) k ∈ A k , i.e., π k,A (n k ) k = Pr a k = A (nk) k . (1)
A vector π k is often called mixed strategy of player k.

The expected utility ūk :

(A 1 ) × . . . × (A K ) → R of player k is defined as follows, ūk (π k , π -k ) = a∈A Ñ K j=1 π j,aj é u k (a k , a -k ). (2)
The information available for player k is a vector,

ūk (•, π -k ) = ūk e (Nk) 1 , π -k , . . . , ūk e (Nk) Nk , π -k , (3)
where, ūk e (Nk) m , π -k is the utility that player k hope to achieve by playing the action A (m) k when all the other players play the mixed strategies π -k . However, the actual utility achieved by player k by playing action

A (m) k is a random variable ũk (e k,m , π -k ) obtained as follows, ũk (e k,m , π -k ) = ūk (e k,m , π -k ) + ε k,A (m) k , (4)
where, ∀k ∈ K and ∀m ∈ {1, . . . , N k }, ε k,A (m) k is the realization of a random variable which represents an additive noise on the expected utility ūk . Sources of such noise are the quantification of the control variables such as transmit power, or decoding errors on the feedback messages used to build the vector ūk (•, π -k ). In this work, it is assumed that the additive error is independently and identically distributed (i.i.d) for all the players. The behavioral assumption is that, player k must choose the action

A (nk) k , where n k satisfies n k ∈ arg max m∈{1,...,Nk} ũk (e k,m , π -k ), (5)
given that all the other players play the complementary mixed strategy profile π -k . It is very important to note that such a maximization problem must be carried out without knowing the exact realization of the noise component. Note also that in the absence of the noise component, this behavioral rule corresponds to a fictitious play [START_REF] Brown | Iterative solution of games by fictitious play[END_REF]. Under these assumptions, it becomes clear that players are unable to choose the action that maximizes the truth utility ūk . Indeed, the noisy component can lead players to take any of their actions. In the following section, we introduce the game theoretical tools tackling such scenario.

III. EQUILIBRIUM NOTIONS In this section, we introduce the notion of QRE and a special case known as logit equilibrium.

A. Quantal Response Equilibrum

For doing so, we first focus on the determination of the mixed strategy π k that player k must use to select its action a k in order to maximize its actual utility function ũk given a vector ūk = (ū k (e k,1 , π -k ), . . . , ūk (e k,Nk , π -k ))). Thus, consider the vector

ξ k = ξ k,A (1) k , . . . , ξ k,A (N k ) k ∈ R Nk and let ξ k,A (n k ) k
, with n k ∈ {1, . . . , N k }, be distributed according to a joint distribution with density function f k , where the marginal of

f k exists for each ξ k,A (n k ) k , with n k ∈ {1, . . . , N k } and E ξ k,A (n) k = 0.
When f k satisfies these conditions, f k is said to be admissible [START_REF] Mckelvey | Quantal response equilibria for normal form games[END_REF]. Now, assuming that the truly achieved utility follows the form of ( 4), the best response can be obtained as follows. , for all k ∈ K and for all n k ∈ {1, . . . , N k }. Hence, given a vector

Let E k,A (n k ) k : R Nk → R Nk
ūk (•, π -k ) = (ū k (e k,1 , π -k ) , . . . , ūk (e k,Nk , π -k )), it yields E k,A (n k ) k (ū k (•, π -k )) = ¶ ζ k ∈ R Nk : ∀m ∈ {1, . . . N k }, ūk (e k,nk , π -k ) + ζ k,A (n k ) k ūk (e k,m , π -k ) + ζ k,A (m) k .
Thus, the probability distribution over the set of actions A k which maximizes the chances that player k chooses the action which brings the highest (expected) utility based on the vector ūk (•, π -k ), can be determined by using the function

β k : (A 1 ) × . . . × (A k-1 ) × (A k+1 ) × . . . × (A K ) → (A k )
, where, given a explicit form for the expected utility ūk , it follows that

β k (π -k ) = β k,A (1) k (π -k ) , . . . , β k,A (N k ) k (π -k ) and ∀n ∈ {1, . . . , N k }, β k,A (n) k (π -k ) = ζ∈E k,A (n k ) k ( ūk(•,π-k)) f k (ζ) dζ. (6)
Such probability distribution β k (π -k ) is known as the statistical reaction function (SRF) or quantal response function (QRF) [START_REF] Mckelvey | Quantal response equilibria for normal form games[END_REF]. Finally, we define the QRE as follows.

Definition 1 (Quantal Response Equilibrium): The strategy profile π

* = (π * 1 , . . . , π * K ) ∈ (A 1 ) × . . . × (A K ) is a quantal response equilibrium of the game G = K, {A k } k∈K , {u k } k∈K , if for all k ∈ K, π * k = β k π * -k . (7)
It is important to remark that Def. 1 implies a fixed point equation. For instance, let ζ :

(A 1 ) ×, . . . × (A K ) → (A 1 ) × . . . × (A K ) be defined as follows, ζ(π) = (β 1 (π -1 ) , . . . , β K (π -K )) . (8)
Then, if π * is a logit equilibrium it holds that π * = ζ (π * ). This observation leads to the following result. Theorem 2 (Existence of the LE [START_REF] Mckelvey | Quantal response equilibria for normal form games[END_REF]): The game G has at least one QRE. The proof of Theorem 2 (in [START_REF] Mckelvey | Quantal response equilibria for normal form games[END_REF]) relies on the fact that the mapping ζ is a continuos mapping from a compact convex set into itself. Thus, by Brouwer's fixed point theorem, there exists at least one π * such that π * = ζ (π * ). In the following, we deal with a particular case of QRE, known as logit equilibrium [START_REF] Anderson | The logit equilibrium: A perspective on intuitive behavioral anomalies[END_REF].

B. Logit Equilibrium

Assume now that the noise components ξ k,A (1) k , . . . , ξ k,A (N k ) k follow a Gumbel distribution (GD) with location and shape parameters µ ∈ R and

1 γk ∈ R + , respectively. That is, Pr ξ k,A (k) k < z = exp (-exp (γ k (µ -z))). (9)
The relevance of the GD is that it models the maximum value of a set of independent and identically distributed random variables. From ( 6), it follows that the SRF can be written as follows,

β (γk) k,A (n k ) k (π -k ) = exp γ k ūk (e (nk) k , π -k ) Nk m=1 exp γ k ūk (e (m) k , π -k ) , (10)
where the index (γ k ) is used to differentiate the particular case from the general case in [START_REF] Meshkati | A game-theoretic approach to energy-efficient power control in multicarrier cdma systems[END_REF]. In the next section, we provide some behavioral rules that allow achieving QRE in a fully decentralized fashion.

IV. LEARNING QRE In the following, we basically focus on the design of the behavioral rules that allow radio devices to achieve a logit equilibrium. Note that this assumes that the noise component in (4) follows a Gumble distribution. Nonetheless, the same analysis hold for other assumption on the noise distribution as long as the density f k aforementioned is admissible. Before we start, let the logit function β : R N +1 → (A k ), with parameter x ∈ R + , be defined as follows,

β (x, y) = Ä β1 (x, y) , . . . , βN (x, y) ä , (11)
with βs (x, y) = exp (x y s ) NK m=1 exp (x y m )

, y = (y 1 , . . . , y N ), s ∈ {1, . . . , N } and N ∈ N. We relate the smoothed best response in [START_REF] Anderson | The logit equilibrium: A perspective on intuitive behavioral anomalies[END_REF] with [START_REF] Perlaza | Learning to use the spectrum in self-configuring heterogeneous networks[END_REF], as follows,

β (γk) k (π -k ) = β (γ k , ūk (•, π -k )) . (12)
Using these elements, our main result in this section is summarized in the following theorem. Theorem 3: Let the following algorithm be defined, for all k ∈ K and for all n k ∈ {1, . . . , N k } and n ∈ N 1 , as follows,

ūk,A (n k ) k (n) = ūk,A (n k ) k (n -1) + α k (n)1 ak(n)=A (n k ) k (ũ k (n) -ū k,A (n k ) k (n -1) , π k,A (n k ) k (n) = π k,A (n k ) k (n -1) + λ k (n) Ä βnk (γ k (n), ūk (n)) -π k,A (n k ) k (n -1) , γ k (n) = γ k (n -1) + ν k (n)Ψ k (n),
where,

(γ k (0), ūk (0), π k (0)) ∈ R + × R Nk + × (A k ) is an arbitrary initialization of player k, Ψ k : R → R is a positive continuos non-decreasing function and for all (j, k) ∈ K 2 , (i) lim T →∞ T t=1 α k (t) = +∞, lim T →∞ T t=1 α k (t) 2 < +∞ (ii) lim T →∞ T t=1 λ k (t) = +∞, lim T →∞ T t=1 λ k (t) 2 < +∞ (iii) lim T →∞ T t=1 ν k (t) = +∞, lim T →∞ T t=1 ν k (t) 2 < +∞ (iv) lim n→∞ λ j (n) α k (n) = 0, lim n→∞ ν j (n) λ k (n) = 0, (13) and either,
(a) ∀k ∈ K, λ k = λ or (b) ∀k ∈ K \ {K}, lim n→∞ λ k (n) λ k+1 (n) = 0. (14)
Then, if the algorithm (13) converges, it holds that,

lim n→∞ π k (n) = π * k , (15)
lim n→∞ ūk,A (n k ) k (n) = ūk (e k,A (n k ) k , π * -k ), (16)
where

π * = (π * 1 , . . . , π * N ) ∈ (A 1 ) × . . . × (A K ) is a (unique) NE.
The reasoning behind the proof of Theorem 3 uses elements of stochastic approximations [START_REF] Borkar | Sthocastic approximation: A dynamical system viewpoint[END_REF] and in particular, the notion of multiple time scales as introduced in [START_REF]Stochastic approximation with two timescales[END_REF] and [START_REF] Leslie | Convergent multiple-timescales reinforcement learning algorithms in normal form games[END_REF]. A formal proof of Theorem 3 will be provided in a future publication. Here, we limit our analysis to some numerical examples to justify the validity of our results due to the space constraints.

Note that Theorem 3 states that the learning process converges, if it does, to a NE instead of a QRE. This effect is due to the fact that the parameter γ k depends on time and it is forced to grow to infinity, which implies that the SRF asymptotically approaches a uniform distribution over the actions which maximize ūk given π -k . Thus, one can state that if the noise component can be faded out in time, a NE selection process arises naturally since the NE to which the system converges is determined by the initial state of the network. This implies using additive noise to solve the problem of equilibrium selection. If on the contrary, the noise component can not be altered, as it models external variables, then it implies that Ψ k (n) = 0, for all n ∈ N 1 . In this case, the system converges to a logit equilibrium with parameters γ 1 , . . . , γ K , which is, in deed the result stated in [START_REF] Leslie | Convergent multiple-timescales reinforcement learning algorithms in normal form games[END_REF]. In the following, we provide some numerical results to illustrate our theoretical results.

V. NUMERICAL RESULTS

Consider a parallel multiple access channel with a set K = {1, . . . , K} of transmitters. Each transmitter sends private information to the receiver by using a set S = {1, . . . , S} of orthogonal channels. Here, the orthogonality is assumed in the frequency domain. All channels are available to all the transmitters and thus, communications are subject to mutual interference. The receiver implements single user decoding. The aim of each radio device is to select a single channel and a power level such that its individual (Shannon) transmission rate is maximized. Note that under this formulation transmitting at the maximum power is a dominant action, and thus, at the (Nash) equilibrium those actions should not be used. An action or power allocation (PA) vector for transmitter k ∈ K is any vector

p k (n) = (p k,1 (n), . . . , p k,S (n)) ∈ A k , where, ∀k ∈ K, A k = p (s)
k = p k,max e s : ∀s ∈ S, e s = (e s,1 , . . . , e s,S ) ∀r ∈ S \ s, e s,r = 0, and e s,s ∈ [0, 1]} .

Thus, the utility of radio device k can be written as follows,

u k (p k (n), p -k (n)) = s∈S log 2 (1 + γ k,s (n)) [bps/Hz],
where the bandwidth per channel is normalized to 1Hz. and γ k,s (n) is the signal-to-interference plus noise ratio (SINR) seen by player k over its channel s at time n.

In Fig. 1, we show the average sum Shannon rates (network spectral efficiency) as a function of the number of players K, with S ∈ {2, 4, 8}. Therein, we assume that players are able to know the exact realization of the noise component and thus, they can play the action which maximizes the truth utility ũk . Note how the existence of the noise component makes the players to take different actions and thus, achieve a different performance in terms of ūk .

In Fig. 2, we show the convergence of the strategy of player 1. Note that since γ k is time dependent, and thus, the noise component can be faded out in time, the dynamics converges to a NE after a short transient. Indeed, the transmitters end up transmitting at the maximum power using different channels, which correspond to a NE.

VI. CONCLUSIONS

In this paper, we have presented a mathematical framework to study the effect of bounded rationality in wireless communications. The notion of quantal response equilibrium has been introduced in order to model the fact that players are not able to properly estimate their own utility and thus, they are subject to choosing other actions than the optimal one. Here, we have also presented a behavioral rule which allows the networks to achieve LE or NE in a fully distributed manner, relaying only in the observation of the utility function.

N

  } the set of vectors of the canonical base spanning the space of the N -dimensional real vectors. Here, ∀n ∈ {1, . . . , N }, e (N ) n = e (N ) n,1 , . . . , e (N ) n,N , and ∀s ∈ {1, . . . , N } \ {n}, e (N ) n,s = 0 and e (N )

  specify the set of errors that lead player k to choose action A (nk) k

Fig. 1 . 2 k= 10 Fig. 2 .

 12102 Fig. 1. Average sum Shannon rates (network spectral efficiency) as function of the number of players K for different number of channels S. Average SNR p k,max σ 2 k = 10 dBs, K = S = 2, with Q = 4 power levels per channel.


B Baratti

M R Laterza

E Balestra

Macrì S Mingrone

F Virzì

P S Puccio

S Ravenda

Kusamura

MD. Marcello Deraco
email: marcello.deraco@istitutotumori.mi.it

MD D Baratti

MD B Laterza

MD M R Balestra

MD E Mingrone

MD S Virzì

MD F Puccio

MD P S Ravenda

MD S Kusamura

Deraco M Md

Baratti D Md

Laterza B Md

MD Balestra Mr

Virzì S Md

MD Ravenda Ps

Advanced Cytoreduction as Surgical Standard of Care and Hyperthermic Intraperitoneal Chemotherapy as Promising Treatment in Epithelial Ovarian Cancer: Review of Literature Data

Keywords: Cytoreductive Surgery, Peritonectomy, Hyperthermic Intraperitoneal Chemotherapy, Epithelial Ovarian Cancer, Peritoneal Carcinomatosis

  L'archive ouverte pluridisciplinaire

Introduction

Epithelial ovarian cancer (EOC) is one of the most common gynecologic malignancies and the fifth most frequent cause of cancer death in women. [ 1 ] Most patients have widespread disease at presentation. [ 2 ] The conventional clinical approach for advanced EOC is based on CRS followed by adjuvant chemotherapy. Clinical studies have shown that cisplatin/taxol-based first-line chemotherapy achieves the highest response rates, with a high proportion of complete responses.[ 3 , 4 ] However, about 2 thirds of the patients relapse and ultimately die of chemoresitant disease.

The role of Cytoreductive Surgery and Residual Disease (RD)

A prominent meta-analysis supporting the importance of CRS in the treatment of EOC was published by Bristow et al. [ 5 ] They evaluated retrospectively the relative effect of percent maximal CRS and other prognostic variables on survival among 81 cohorts of patients including 6,885 with stage III/IV EOC, treated with cisplatin based chemotherapy.

Maximal cytoreduction was concluded to be one of the most powerful determinants of survival.

Several factors concur for the accomplishment of an optimal cytoreduction: accurate preoperative identification of resectable cases, experience of the surgical team, favorable clinical condition of the patient, biological aggressiveness of the disease, the surgeon and patient's willingness to challenge the inherent increased morbidity of an extended operation. However, the essence of the problem is not operational. Although universally accepted as standard of care, the term CRS has been subject of a myriad of interpretations resulting in different concepts of optimal RD and advisable limits of radicality.

A recent retrospective review of 360 patients stage IV EOC who underwent primary surgery followed by 6 cycles of IV platinum/paclitaxel has provided important element to clear the issue. Median OS for microscopic, 0.1 to 5.0 cm, and >5.0 cm RD was 64, 30, and 19 months, respectively. The authors concluded that ultraradical surgeries might be targeted for selected patients in whom microscopic RD is achievable. [ 6 ] A survey of the Society of Gynecologic Oncologists (SGO) revealed 12.0% to think of removing all visible disease as "optimal", 13.7% use a 0.5 cm threshold, 60.8% a 1 cm threshold, 3.6% a 1.5 cm threshold, 8.7% a 2.0 cm threshold and 1.3% use other criteria such as the total estimated weight and/or volume of RD. [ 7 ] The large spectrum encompassed by the different definitions of optimal cytoreduction could be resumed in 2 school of thoughts. The first one takes advantage of the high initial chemosensibility of the disease and tends to differ radical operations with multivisceral resections whenever it could be necessary to attain minimal RD in order to avoid presumed morbidity. Supporters of such a police state that evidence of survival benefit coming from ultraradical procedure is not sufficient and attribute a major prognostic significance to chemotherapy.

The second one -which is based on the assumption that the RD is the most important prognostic factor -is surgically oriented and tend to push the limits the radicality as far as technically it is possible to be done. Extensive procedures including multi-organ resections, in order to achieve highest rate of complete cytoreduction, is systematically pursued, despite potential increase in side effect rates. Representatives of the second school of thought are the minority in the EOC management scenario. However, interesting data have been published thus far that substantiate a surgically aggressive police.

Eisenkop et al. reported on 213 patients with Stage IIIC EOC undergoing complete

cytoreduction before initiation of systemic chemotherapy. The median and estimated 5year survival were 75.8 months and 54%, respectively. Survival was influenced by the requirement of diaphragm stripping and the extent of mesenteric and serosal implants that required removal. [ 8 ] Investigators from Memorial Sloan Kettering Cancer Center have introduced a more radical surgical approach including advanced procedures -such as diaphragm stripping and/or full thickness resection, splenectomy, hepatectomy, porta hepatis dissection, bowel resections -to approach peritoneal carcinomatosis (PC), especially in the upper abdomen.

A recent report on 398 patients assigned to 2 non-randomized groups, but well balanced with respect the main covariates, allowed the conclusion that such aggressive surgical policy resulted in increased optimal cytoreduction rates and significantly improved oncological outcome. [ 9 ] However, supporters of the first school of thought argue that complete cytoreduction in most cases is attained thanks to relatively small tumour burdens and presumed diminished biological aggressiveness of neoplasms rather than to maximal surgical effort. [ 10 ] Eisenkop et al. observed that the need to remove a large number of peritoneal implants correlates with biological aggressiveness and diminished survival, but not significantly enough to preclude long-term survival or justify abbreviation of the operative effort. [START_REF] Eisenkop | What are the current surgical objectives, strategies, and technical capabilities of gynecologic oncologists treating advanced epithelial EOC?[END_REF] On the other hand, an exploratory analysis in the context of a large randomized trial (SCOTROC) has shown a significant benefit in PFS correlated with optimal surgery in stage IIIC to IV disease, that is limited to patients with less advanced disease. [ 11 ] The question of relative influence of tumor biology and actual impact of complete cytoreduction on outcome has been further elucidated by a provocative study evaluating tumor-infiltrating lymphocytes and/or tumor mitotic activity on 134 patients with stage III/IV serous or poorly differentiated ovarian adenocarcinoma undergoing primary CRS. [ 12 ] Patients whose tumors exhibited low Ki67 expression and high intraepithelial CD8(+) frequency had a better outcome with a 5-year survival rate of 73.3%. The finding worthy of note was that patients with aggressive biology -low frequency of intraepithelial CD8(+) T cells or high Ki67 expression -were more likely to benefit from aggressive CRS. On the other hand, survival was similar for patients with brisk CD8(+) T cells or low Ki67 who had optimal or suboptimal cytoreduction. The study represents the first biological evidence that optimal cytoreduction is not a direct consequence of a favorable biological behavior, and that aggressive surgical approach is justified in poor prognostic subsets.

As for other Peritoneal Surface Malignancies (PSM), the advent of Peritonectomy

Procedures is one of the most important contribution for the standardization of CRS. [ 13 ] The adaptation of the original technique to the different diseases continues to be a challenge for the surgeons treating patients with PSM. The original systematic codification and further modifications,[ 14 ] allow the best intraoperative surgical strategy by the following peritonectomy and multivisceral resections procedures:

•

Hyperthermic Intraperitoneal Chemotherapy

The evolution of management of advanced EOC in the last decade has been characterized by the validation of intraperitoneal chemotherapy (IP). The GOG-172 phase III trial favored the intravenous (IV) plus IP chemotherapy over IV chemotherapy in primary stage III EOC. Eleven out of 19 studies employed the CRS+HIPEC as primary treatment for a proportion of their patients and 11/19 studies employed it as salvage therapy. The overall rate of severe perioperative morbidity ranged from 0 to 40% and mortality rate from 0 to 10%. The median OS following treatment ranged from 22 to 64 months with a median DFS ranging from 10 to 57 months. In patients with optimal cytoreduction, a 5-year OS ranged from 12% to 66%.

It is hard to ascertain in which extent the apparent survival advantage reported by these uncontrolled clinical studies resulted from selection bias. Moreover, the investigations involved heterogeneous patient population, at different time points of disease evolution, usually heavily pre-treated, and therefore, with different tumor chemoresistances.

Controversial Points

The growing body of literature observed in recent years testing the CRS+HIPEC in EOC -a direct consequence of its increasing popularity in the scientific community -has been responsible for the emergence of various controversies.

Timing of CRS and HIPEC

A major controversial issue is the time point of the natural history of EOC for the performance CRS+HIPEC. The recent data recommending maximal surgical effort [START_REF] Bristow | Survival effect of maximal cytoreductive surgery for advanced ovarian carcinoma during the platinum era: a meta-analysis[END_REF] and IP [START_REF] Jaaback | Intraperitoneal chemotherapy for the initial management of primary epithelial EOC[END_REF] chemotherapy in the primary setting represents a indirect evidence that CRS+HIPEC could be tested as upfront treatment in the context of a phase III trial.

The CRS+HIPEC could also be tested after neoadjuvant chemotherapy (NACT) in an attempt to increase the likelihood of optimal cytoreduction. A recent large randomized study has compared the primary cytoreduction with NACT. [ 22 ] The investigators reported similar results in terms of PFS and OS between the study arms and lower morbidity associated with NACT. These results have been corroborated with a recent meta analysis. [ 23 ] No studies have been conducted thus far testing the effects of CRS+HIPEC following NACT.

In order to further minimize the morbidity, the CRS+HIPEC has been recently tested as consolidation treatment by some investigators.[ 24 , 25 ] The consolidation setting represents an attracting option as the tumor burden after completion of adjuvant systemic chemotherapy is expected to be low and as well as the inherent the possible complication(s) related to cytoreduction. Kim et al. reported on 18 stage Ic-IIIc EOC patients with a negative second-look who were submitted to HIPEC using paclitaxel. The 8 year OS rate resulted to be of 84%. [START_REF] Kim | Consolidation hyperthermic intraperitoneal chemotherapy using paclitaxel in patients with epithelial ovarian cancer[END_REF] In contrast, Pomel et al. have not found the same favorable results. They employed oxaliplatin for HIPEC and verified the emergence of unacceptable rate of post operative bleeding that constrained the premature closure of the trial. [START_REF] Pomel | Hyperthermic intra-peritoneal chemotherapy using oxaliplatin as consolidation therapy for advanced epithelial ovarian carcinoma. Results of a phase II prospective multicentre trial. CHIPOVAC study[END_REF] The last time point in which CRS+HIPEC has been tested is as second-line treatment. The influence of secondary CRS (without HIPEC) on survival outcomes has been addressed in a substantial number of publications and has been recently systematically reviewed.[ 26 ] However these were non-controlled studies of secondary CRS investigating the impact on survival. A consistent survival data comparing secondary CRS with chemotherapy is expected to be provided by the ongoing randomized trial AGO-OVAR OP4. [ 27 ] Indirect evidence could also be extracted from the recently published exploratory analysis of the DESKTOP I trial that shed new insights not only on the value of a aggressive surgical approach in relapsing EOC, but also on the prognostic implications of PC. [ 28 ] Two hundred fifty patients with recurrent EOC (50% with PC) undergoing secondary cytoreduction were included in the analysis. The authors concluded that although the PC represented a negative predictor of complete resection it had no effect on prognosis if complete resection could be achieved.

Two subsets of patients must be clearly distinguished in the circumstance of a second-line treatment: platinum resistant and platinum sensitive ones. Experimental data sustains that a major synergism between cytotoxicity and the heat is achievable in cisplatin resistant cell lines with respect to platin sensitive ones.

Time To Chemotherapy

Clinicians often question whether a long postoperative time after patients undergo CRS+HIPEC could affect the Time To Chemotherapy (TTC) with a delayed start of adjuvant chemotherapy and consequently a reduced benefit of the extensive procedure.

Aletti evaluated retrospectively data from 218 patients with stage IIIC/IV EOC. Mean TTC interval was 26 days (range, 7-79 days). TTC was not a predictor of OS allowing the conclusions that TTC interval should not be used to justify using a surgically more conservative approach for advanced EOC. [ 33 ] Staging System Another controversy regards the need for a new staging system (SS). It is expected that the surgical paradigm in the treatment of EOC is going to change to a more aggressive and radical operative approach in the near future. This means that the actual

Drugs

Various drug combinations for EOC have been tested by experimental and phase I/II studies on HIPEC. Several parameters should guide the choice of the drug: pharmacokinetic profile, tumour chemosensibility and toxicity. Ideally the drug must be water-soluble and of high molecular weight in order to guarantee a low peritoneal clearance. This, combined with a high systemic clearance, results in pharmacological advantage expressed by the ratio AUCperitoneal/AUCplasma. The penetration ability of drug in the tumour is a function of passive diffusion (related to the AUCpe/AUCpl ratio), removal from by the capillary blood flow, temperature modulation. Finally, the influence of temperature in the cytotoxicity should also be of concern. The optimal choice of the drugs for HIPEC represents an another unclear topic in EOC treatment and is still impossible to be done on the basis of the literature data. Agents that have already been employed are Cisplatin, [START_REF] Bartlett | A phase I trial of continuous hyperthermic peritoneal perfusion with tumor necrosis factor and cisplatin in the treatment of peritoneal carcinomatosis[END_REF] Doxorubicin, [START_REF] Rossi | Hyperthermic intraoperative intraperitoneal chemotherapy with cisplatin and doxorubicin in patients who undergo cytoreductive surgery for peritoneal carcinomatosis and sarcomatosis: phase I study[END_REF] Caelyx, [START_REF] Harrison | Phase I trial of pegylated liposomal doxorubicin with hyperthermic intraperitoneal chemotherapy in patients undergoing cytoreduction for advanced intra-abdominal malignancy[END_REF] Mitomycin C, [START_REF] Markman | Phase II trial of intraperitoneal Paclitaxel in carcinoma of the ovary, tube and peritoneum: a Gynecologic Oncology Group Study[END_REF] Mitoxantrone, [START_REF] Nicoletto | Pharmacokinetics of intraperitoneal hyperthermic perfusion with mitoxantrone in EOC[END_REF] Carboplatin, [START_REF] Lentz | Intraperitoneal hyperthermic chemotherapy using carboplatin: a phase I analysis in ovarian carcinoma[END_REF] Oxaliplatin, [START_REF] Fagotti | Secondary cytoreduction plus oxaliplatin-based HIPEC in platinum-sensitive recurrent ovarian cancer patients: a pilot study[END_REF] Gemcitabine [START_REF] Morgan Rj | Phase I trial of intraperitoneal gemcitabine in the treatment of advanced malignancies primarily confined to the peritoneal cavity[END_REF] and Paclitaxel. [START_REF] De Bree | Cytoreductive surgery and intraoperative hyperthermic intraperitoneal chemotherapy with paclitaxel: a clinical and pharmacokinetic study[END_REF]

Conclusion

CRS represents one of the most powerful determinants of outcome and the strongest clinician-driven predictor of survival in patients with stage IIIC/IV EOC according to literature evidence. The resection of all visible implants should be considered the standard of the surgical approach. Peritonectomy Procedures, as a result of a change in surgical paradigm, could dramatically improve outcomes and should be advisably considered in EOC treatment.

The current tendency of recognizing the importance of maximal surgical effort in the management of EOC confirms the need to revise the actual staging criteria (FIGO). A new staging system should be able to guide the surgeon not only to identify the subsets characterized by different prognosis, but also provide information about resectability and potential post operative morbidity.

The appropriate management of advanced EOC patients requires the presence of a multidisciplinary team including surgeons specialized in CRS. A well structured and long lasting training program is necessary to allow gynaecologists and surgeons to overcome the steep learning curve of CRS using peritonectomy procedures. The performance of at least 130 procedures is necessary to consider the physician an expert in cytoreduction using the Sugarbaker technique. [ 43 ] At this point national committees are advisable to be constituted to carefully evaluate if centres proposing to set up a peritoneal surface malignancy program meet all the requirements in terms of technical and human resources to perform the combined procedure safely and with optimal expertise. Considering the potential advantages of HIPEC associated to CRS (Table 1), trials testing its efficacy should be encouraged in all time points of natural history of the disease, especially in up-front and platinum-sensitive recurrent disease. In this sense 3 randomized controlled trials are ongoing. The first proposes to test the efficacy of secondary cytoreduction with or without HIPEC and has been conducted by the Netherlands Cancer Institute. [ 44 ] The second trial proposes to test the procedure in primary and secondary setting and has been conducted by St George Hospital, in Sydney. [ 45 ] The third is a multicentric randomized trial proposed by a French group (CHIPOR) and aims to test the HIPEC for relapsing disease.

Conflict of interests

The authors do not have any conflict of interest to disclose.

  Upper Right Peritonectomy: right diaphragmatic peritonectomy with Glisson's capsule dissection; lesser omentectomy, stripping of the omental bursa ± cholecystectomy ± gastric antrectomy or total gastrectomy • Upper Left Peritonectomy: left diaphragmatic and parietal peritonectomy with splenectomy and greater omentectomy; • Pelvic Peritonectomy: pelvic parietal peritonectomy ± sigmoidectomy ± hysterectomy and salpingo-oophorectomy; • Right Parietal Peritonectomy ± right/total colectomy; • Mesenteric Peritonectomy implants on visceral surfaces could be alternatively removed by electrosurgical local dissection.

  [ 15 ] Successively, a Cochrane meta-analysis of all randomized IP versus IV trials showed a hazard ratio of 0.79 for disease-free survival (DFS) and 0.79 for OS, favoring the IP arms.[16 ,16 17 ] Other prominent advance in the EOC management has been the advent of hyperthermic intraperioneal chemotherapy (HIPEC). It has been employment in the treatment of PSM following CRS and positive outcome results have been reported for PC from appendiceal cancer,[ 18 ] colorectal cancer,[ 19 ] and peritoneal mesothelioma.[ 20 ] The theoretical rationale for HIPEC for the treatment of advanced EOC is to combine the demonstrated pharmacological activity of IP chemotherapy in this disease with the advantage of intraoperative hyperthermia, that exerts an enhancement of cytotoxicity, enabling a microscopic cytoreduction, following a macroscopic cytoreduction obtained with CRS. Recently Chua TC. et al. have published a systematic review of 19 observational studies that reported the efficacy of CRS+HIPEC for advanced EOC.[ 21 ] The HIPEC protocol varied in each institution. The most commonly used chemotherapy was Cisplatin.

  [ 29 ] Based on this favorable in vitro evidence investigators from National Cancer Institute of Milan launched a phase III trial testing the CRS+HIPEC in platinum resistant EOC. The few patients that were recruited unfortunately presented high rate of morbidity and the trial was prematurely closed due to poor patient accrual.[Dr. Raspagliesi, NCI Milan, 30 ] Platin sensitive relapse has been deemed as the best clinical circumstance to test the combined treatment.[ 31 ] However one could argue raising the hypothesis that the favorable oncological results presented by such studies would be achievable in any case irrespective of treatment (secondary cytoreduction +/-HIPEC vs systemic platin based chemotherapy) due to the favorable biological profile of the tumor. According to the consensus statement (Milan 2006), the procedure could be employed in all time points (primary treatment, interval debulking, secondary cytoreduction, salvage treatment) with the exception of platinum resistant relapsing disease.[ 32 ] The studies published since then do not allow further conclusion other than that established by the 2006 consensus statement.

FIGO

  FIGO staging would become insufficient to help defining therapeutic strategies and new prognostic subsets. Recently Chéreau et al. has compared the International Federation Gynecology and Obstetrics (FIGO), Peritoneal Cancer Index (PCI), Eisenkop, Aletti, Fagotti, and Fagotti-modified scores regarding the accuracy to predict resectable disease, postoperative complications and oncological outcomes. In predicting resectability, Fagottimodified and PCI outperformed other scores. There was a strong association between the occurrence of postoperative complications and Aletti, PCI, and Eisenkop scores. A significant differences among FIGO, PCI, Eisenkop, Fagotti-modified, and Aletti scores were observed concerning their correlation with PFS (P < .05).[ 34 ]

  [ 46 ]

Table 1 :

 1 Literature results of patients with peritoneal carcinomatosis from EOC treated with CRS+HIPEC : most relevant experiences

	Pavlov et al.	56	26	38	NR	NR	NR	48
	(2009)
	Fagotti et al.	25	10	NR	NR	NR	NR	31
	(2009)
	Guardiola et al.	47	14	NR	NR		NR	49
	(2009)
	Di Giorgio et al.	47	20	24	26	NR	17	50
	(2008)
	Bae et al.	67	NR	NR	NR	NR	54	51
	(2007)
	Cotte et al.	81	19	28	55	NR	66	52
	(2007)
	Rufian et al.	33	NR	48	66	46	NR	53
	(2006)
	Raspagliesi et	40	11	32	NR	NR	37	54
	al. (2006)
	Ryu et al.	57	26	NR	41	NR	15
	(2004)

OS: Overall Survival; DFS: Disease Free Survival; Med: Median; mos: Months;