Csilla Filo
email: fcsilla@human.pte.hu

REGIONAL INTEGRATED DEVELOPMENT OF VOCATIONAL TRAINING IN BARANYA COUNTY

Keywords: vocational training, region development, education, indicators, labour market

In my study, I present the new training co-ordination of Baranya County. The aim of development was the training institution network modernization and effective work to ensure labour market needs and coordinate training, theoretical training and community service needs is community spaces, well-equipped classrooms and workshops to develop LLL-create the conditions

Regional integrated development of vocational training in baranya county

I. BACKGROUND

In 90's years the quality and quantity of vocational training decreased in Hungary. The actors in the economy have experienced that have not received the expected amount and quality workforce, are not given adequate support to improve competitiveness.

The training content, technical quality is not followed the technological development, the number of vocational school students, and the actual distribution of significant tension between the demand for labour generated, the applicability of school leavers, and often do not meet performance expectations in the labour market. The change, modernization, competitiveness and productivity is vital to determining the business sector and training institutions also engaged, in addition to social consequences is significant. The solution should be to expect decisive action to direct the training institutions of the system, but the maintenance managing organizational change is inevitable, but also the future as well. The availability of skilled labour, vocational training and employability of those entering the system is an important professional requirement. The harmonious functioning of labour markets and the fundamental task of improving the competitiveness of firms managed to increase the level of employment, training system development, the demand and supply side, more flexible fit. These challenges are very complex response. The local planning to the next level, the region should reach the level of requirements. This study wants to present a potential development, which created an innovative advanced training .

The geographic characteristics, area and settlement structure of the region

The South Transdanubia region encompasses the land between the Danube, and the Dráva rivers (which at the same time serves as a natural border with Croatia), as well as between Lake Balaton, in the south-western part of Hungary. With an area of 14 169 km2, and a population of 977,000 people at the beginning of 2005, its population density of 69 people/km2 makes it the most sparsely inhabited region in the country. Pécs -a large town with a population of 160,000 people -is the centre of the region, and Kaposvár and Szekszárd are its affiliate centres. These cities are the centres of the region's economic and cultural life. The surface of South Transdanubia is diverse, and the quality of its environment is generally favourable. Mining (black coal and uranium) that once dominated the region has diminished over the past decade, causing a significant economic downturn in the area of Pécs, Komló, Szászvár, and Nagymányok. The limestone that is the bulk of elevated areas is suitable for cement production. The role of andesite mining at Komló is going to increase in connection with the motorway construction activity planned in the region. Clay can be found in large quantities in the hills, together with gravel and sand along the Dráva river.

1.2. Level of development in the region, income situation

There is a clear correlation between differences in development level within the region, and the region's settlement structure. Deprived micro regions lined up in a block along the Dráva River, and situated in a mosaic-like way along the internal borderlines of the counties are a stark contrast to the development seen in the surroundings of major towns, and the micro regions alongside Lake Balaton. According to the regulation accepted in 2004 (Government Decree 64/2004) 18 of the region's 24 micro regions are disadvantaged, and of these, 9 belong to the group of those in the most disadvantageous situation. This means that almost half (47.5%) of inhabitants in the region live in disadvantaged areas, and within that, close to 20% of the population reside in the most disadvantaged areas. The latter proportion is only higher in two regions -North Great Plain (40%), and North Hungary (37.6%).

Demographics

The demographic conditions in the South Transdanubia region are developing less favourable than the national average. The region's population decreased by 39,986 people between 1990 and 2009. The decrease of population exceeds the national average as a result of the drop in live births in the region, and the migration of its population.

The extent of migration is increasing year by-year.

Looking at the domestic migration difference indicators of the Central Statistical Office (KSH) it can be concluded that while an average of 1225 people migrated from the region between 2000 and 2003, the loss due to migration was 1743 people in year 2009. The insufficient number (especially jobs demanding higher qualifications) of workplaces (braindrain), moreover the lack of services -mainly in micro-village, deprived, disadvantaged areasare the major reasons for migration. The development of population trends are also negatively influenced by unhealthy lifestyle, and bad living circumstances. The fast rate of population decrease deteriorates the chances for improving employment indicators, because it is typically the population of working age and with a willingness to work that migrates from the region, and in the long run this is going have effect to the competitiveness of the region.

Employment and unemployment

Comparing the rate of employment to the European Union, as well as nationwide data it can be concluded that employment rate in the South Transdanubia region is low.

Unemployment rate in Hungary in 2009 (level NUTS 4)

source:KSH 2010.

Looking at the rate of unemployment, the region's situation is less favourable than the national average, but does not reach the European Union's rate. Significant differences can be experienced within the region from the perspective of employment: while in centre areas with major towns unemployment is below the national and EU average, it is more then triple than the regional centre average in the Sellye (26.2%), Szigetvár (25.4%), as well as the Csurgó (21.9%) micro regions (2009 data from the Employment Centres).

The strong seasonality of jobs is an unfavourable phenomenon, especially in the area of the Preferential Holiday Resort District of Lake Balaton (due to seasonality in tourism), but the same can be observed in the processing industry and agriculture during summer months. The number of graduates and highly trained unemployed people is also large, their retraining and reintegration, however, is easier than that of people permanently unemployed.

Unemployment rate in

II. DEVELOPMENT OF VOCATIONAL TRAINING

The period of secondary education starts in year 9 and continues till year 10 or 12, depending on the type of school. In year 10 (school leaving age) the end of education is marked by a general knowledge examination, whereas in year 12 by a secondary school-leaving examination. Accordingly, at vocational schools an ISCED level 3.3, and at secondary vocational schools an ISCED level 5 qualifications can be obtained, respectively. The general knowledge examination can be taken at three kinds of school (grammar, secondary vocational and vocational schools), according to requirements set by different syllabuses. In this respect the Hungarian school system is similar to the German, where the equivalent of grammar school is the Gymnasium, that of secondary vocational school is the Realschule+ Fachoberschule, and that of vocational school is the Berufsschule. The institutional system in Hungary is rather complex, because the relatively homogeneous structure, which better reflected the school system, has been changed since the political transformation. Recently, this system can not adapt to the 21st century challenges. The main problems of vocational training in the South-Transdanubian region.

  Hungary in 2009 (level NUTS 5) source:KSH, 2010.The continuous trend of increasing unemployment causes difficulties in achieving the Lisbon objectives, since it makes improving employment rate and competitiveness indicators difficult. The region's labour market is not flexible, since the rate of full-time employment, and occupation in an employment relationship dominates the labour market. The number of people employed on fixedterm labour contracts is low in the region, which well represents the lack of flexible employment forms on the labour market.It is a positive development that number of active population is increasing after the decline over the decade following the fall of communism. Despite of that, the South Transdanubia region -lagging significantly behind Central Transdanubia -is ranked fourth with an activity rate of 51.6%, which is 2.9% less than the national average. (Central Statistical Office (KSH), 2009) The population of working age population and the number of registered unemployed in the South -Transdanubia source: KSH, 2010 ed:FILÓ Average rates of employment differ sharply between men and women. The figures show the different between men and women . This is less distinct from the EU 25 average rates of employment, which were 71.8% for men and 55.2% for women. The number of men and women of registered unemployed surce: KSH, 2010 ed: FILÓ 1.5 Vocational training Even in the years before the political transformation (1989), there was a lot of criticism concerning the structure of vocational training adjusted to the socialist economy. It became evident that the structure of vocational training based on large-scale and planned economy, and adjusted to a deformed economic structure could not meet the manpower demands of market economy. Presumably the major fault of the system of vocational training lay in the fact that 70-80% of the trades taught were connected with industry (heavy industry, machine industry, light industry, agricultural large-scale industry, and industrial services) and only 20-30% was connected with services, trade, office management, tourism, etc. According to the 1990 national census data, nearly 50% of manual workers, comprising three quarters of the active population, had only primary education, that is, they were unqualified. With the transformation of employment, in the era of mass unemployment, unqualified workers comprise some 60% of the unemployed, and within that, 80% of the permanently unemployed are unqualified. The composition of the unemployed according to qualifications, that there is correlation between qualifications and the chances of becoming unemployed. The significant discrepancy between the demands of a changed labour market and the professional composition of the labour force, as well as the rigidity of the system of vocational training and the immobility of manpower, accounts for the great number of qualified unemployment. Taking into consideration the still high number of unqualified as well as qualified manpower made redundant due to the structural transformation of the economy, the continuous re-training and inservice-training of manpower is an important task (labour market vocational training, vocational training of adults). To achieve this, regional manpower-development and -training centres came into being in 1994. The main task of vocational training is the training of experts to meet the requirements of the national economy in numbers demanded by the labour market. Thus vocational training is becoming the pulling power of the economy because it has an influence on the competitiveness of the region economy.

•

  the vocational training does not provide sufficient competencies • training can not follow the economic needs • decreases the attractiveness of vocational training • expensive and inefficient the vocational training We can see on the next figure, that steadily increasing the number of unemployed and unskilled as well in parallel. source: KSH, 2010 ed: FILÓ Number of unemployment and unqualifiedThe current training system is not able to perform tasks because increasing the number of those leaving school system without acquiring qualifications; the labor market, the students performed, not by the economy required professional knowledge, competencies, not a practical training, inadequate qualifications increase the number of long-term unemployed, can not be completed as required in the EU over 70% of employment. The operation of the system due to the teaching profession today do not meet the needs of the labor market. The figure below shows that the marketable professions (red diagram) are present in small proportion of the training spectrum. The blue diagram shows the over-education professions of training offer. ed:FILÓ Skills shortages and over-education It is followed the inadequate labor market indicators. In respect of the macro, micro and employment objectives of the Integrated Guidelines summarizing the Lisbon Strategy's objectives, the following will receive direct support: growth, as well as three of the ten micro-economic guidelines: business environment, small and medium size enterprises, and infrastructure. With respect to the employment objectives, the operational program covers flexibility, human capital investment, and education.2.1 Development directionIn 2008 it started a new approach to vocational training system. The first direction was to create a consortium for development where more institutions founded an integrated vocation development association. In this program they made the base of Institution Management System. The aim was the training institution network modernization and effective work to ensure labour market needs and coordinate training, theoretical training and community service needs is community spaces, well-equipped classrooms and workshops to develop LLL-create the conditions for learning. The program is a priority goal of socioeconomically backward region to increase the competitiveness of the vocational level, improving the efficiency, by providing training in the area labour market for highly qualified, modern knowledge, skilled in the use of high technology, changing economic conditions and technological flexibility to adapt workforce to meet the demand for targeted, an integrated institutional set up and run by the area of labour market flexibility to adapt to the needs of vocational training offerings, and the rational use of the principle of government expenditures in coordination with the training in order to effectively and efficiently. The priorities: • Institutional Support for Cooperative Systems • Professional collaboration (agreed with the matching of training) • Establish a central training locations • Joint utilization of infrastructure capacity (their practice of shared use of existing, labor-market assistance) • Common human resource planning • Joint employment of teachers • Start a joint training improvements (the existing capabilities in order) • Companies into the organization of vocational training

The target groups • The leaders of institutions, teachers, trainers • The institutions involved in vocational and adult education student • The disadvantaged students and their families Thus, the school training workshops designed to control the labor market requirements. The system also performs community functions, which provide primarily to provide comprehensive services to their students' career choices and career guidance, career counseling field, and disadvantaged youth and their families also offer additional services.


M Bennis
email: bennis@ee.oulu.

S M Perlaza

M Debbah

Learning Coarse Correlated Equilibria in Two-Tier Wireless Networks

AbstractIn this paper, we study the strategic coexistence between macro and femto cell tiers from a game theoretic learning perspective. A novel regret-based learning algorithm is proposed whereby cognitive femtocells mitigate their interference toward the macrocell tier, on the downlink. The proposed algorithm is fully decentralized relying only on the signal-to-interferenceplus-noise ratio (SINR) feedback to the corresponding femtocell base stations. Based on these local observations, femto base stations learn the probability distribution of their transmission strategies (power levels and frequency band) by minimizing their regrets for using certain strategies, while adhering to the cross-tier interference constraint. The decentralized regret based learning algorithm is shown to converge to an ǫ-coarse correlated equilibrium (ǫ-CCE) which is a generalization of the classical Nash Equilibrium (NE). Finally, numerical results are shown to corroborate our ndings where, quite remarkably, our learning algorithm achieves the same performance as the classical regret matching, but with substantially much less overhead.

I. INTRODUCTION

Wireless data traf c has been increasing exponentially in recent years in which the emergence of novel wireless services such as social networking and gaming has introduced stringent quality-of-service (QoS) and data rate constraints on nextgeneration wireless cellular networks. This increase led mobile operators to explore new ways to achieve network coverage improvements, higher spectral ef ciencies, and OPEX/CAPEX reductions. In view of this, femtocell technology (and small cells in general) represents a novel networking paradigm based on the idea of deploying low-power, low-cost base stations underlaying the legacy macrocell network [1].

The deployment of future heterogeneous cellular networks supporting macro, pico, and femtocells coexisting on the same spectrum and in the same geographical area entails new technical challenges for mobile operators. These challenges encompass co-tier and cross-tier interference, coverage holes due to large transmit power disparity across small cells, handover optimization, and heterogenous backhaul design. Furthermore, because femtocells are user-deployed, selforganizing network (SON) capabilities requiring innovative interference and mobility management are of vital importance. For this reason, both academia and standardization bodies are

The authors would like to thank the Finnish funding agency for technology and innovation, Elektrobit, Nokia and Nokia Siemens Networks for supporting this work. This work has been performed in the framework of the ICT project ICT-4-248523 BeFEMTO, which is partly funded by the EU. currently looking at SON aspects in which self-con guration and self-optimization are deemed instrumental to make the deployment of femtocell networks feasible.

Interference management for femtocell networks can be found in a number of works such as [START_REF] Lopez-Perez | Interference avoidance and dynamic frequency planning for WiMAX femtocells network[END_REF], [START_REF] Choi | Dealing with loud neighbors: the bene ts and tradeoffs of adaptive femtocell access[END_REF], [START_REF] Bennis | Niyato Distributed learning strategies for interference mitigation in femtocell networks[END_REF] among others where a variety of techniques including dynamic frequency planning, dynamic spectrum occupation, power control, closed/open/hybrid access group [START_REF] Choi | Dealing with loud neighbors: the bene ts and tradeoffs of adaptive femtocell access[END_REF], context awareness capabilities were studied. Interference management solutions based on reinforcement learning (RL) techniques such as Qlearning, cooperative Q-learning, replication dynamics and their variants were studied in [START_REF] Bennis | Niyato Distributed learning strategies for interference mitigation in femtocell networks[END_REF], [START_REF] Bennis | Decentralized Cross-Tier Interference Mitigation in Cognitive Femtocell Networks[END_REF]. In this paper, we propose a fully decentralized algorithm for interference mitigation from the femto-to-macrocell tier, on the downlink, as shown in Fig. 1. The basic idea is that each femtocell base station (FBS) learns the regret of its actions taken at every time instant, aiming at minimizing its average regret over time. At the same time and owing to the twotier hierarchy, femtocells need to mitigate their interference brought onto the macrocell. A players regret is de ned as the difference between its average utility when playing the same action in all previous stages of the game, and its average utility obtained by constantly changing its actions. The underlying assumption in our work is that feedback messages from macrocell users (MUEs) to their serving macrocell base station (MBS) containing their instantaneous signal to interference plus noise ratio (SINR) can be decoded by all FBSs. The repetitive observation of the SINR is used by the FBS to dynamically con gure how often different frequency bands are used such that, a minimum time-average SINR level can be guaranteed at the macrocell tier. Finally, our proposed algorithm allows us to capture fundamental issues, namely, (1) joint estimation of players own utility function due to perturbation and regret minimization, and (2) alleviate the setback of the classical regret matching procedure in terms of information requirement.

This paper is organized as follows. In Section II, the system and game model are presented. Section III describes the regret-based learning procedure carried out by femtocells to learn their optimal transmission strategies and mitigate their interference towards the macrocell tier. The distributed regret based learning algorithm is described in Section IV. Finally, numerical results are presented in Section V, and Section VI concludes this paper.

II. MODELS

A. Notations

Boldface upper and lower case symbols represent vectors and scalars. Given a random variable z, the expectation with respect to the probability distribution of z is denoted by E z [.]. The indicator function is denoted by 1 {condition} and it equals 1 (resp. 0) when condition is true (resp. false). Given a nite set A, △(A) represents the set of all probability distributions over the elements of the nite set A. Let the vector e

B. System Model

Let us assume M = 1 macrocell base station operating over a set S = {1, . . . , S} of S frequency bands. Let Γ 0 = (Γ (1) 0 , . . . , Γ (S) 0 ) denote the minimum average SINR offered by the MBS to its macrocell user equipment over its corresponding spectrum band. Consider now a set K = {1, . . . , K} of K femtocells underlaying the macrocell. Each femtocell can use any of the available frequency bands to serve its corresponding femto end-users (FUE) as long as it does not induce a lower average SINR than the minimum required by the MUE. At each time interval each FBS serves one FUE over one of the available channels following a time division multiple access (TDMA) policy. Designate the MBSs transmit power on a given sub-carrier to be p be the variance of the additive white Gaussian noise at receiver k, which is assumed to be constant over all sub-carriers for simplicity. Let p k,max with k ∈ K be the maximum transmit power of FBS k. For all k ∈ K, let the S-dimensional vector

p k (t) = p (1) k (t), ..., p (S)
k (t) denote the power allocation (PA) vector of FBS k ∈ K at time t. Here p (s) k (t) is the transmit power of femtocell k over frequency band s at time t. All FBSs are assumed to transmit over one frequency band only at each time t at a given power level not exceeding p k,max . Let L k ∈ N be the number of discrete power levels of FBS k and denote by q (ℓ,s) k its ℓ-th transmit power level when used over channel s, with (ℓ, s) ∈ L k × S, with L k = {1, . . . , L K }. Denote also by q (0,0) k , with k ∈ K, the S-dimensional null vector, i.e., q (0,0) = (0, . . . , 0) ∈ R S . Thus, FBS k has N k = L k • S + 1 possible PA vectors and for all t ∈ N, p k (t) ∈ A k , where

A k = q (0,0) ∪ q (ℓ,s) k : (ℓ, s) ∈ L k × S .
(1)

The SINR of MUE on carrier s is:

γ (s) 0 = |h (s) 0,0 | 2 p (s) 0 σ (s) 0 2 + k∈K |h (s) 0,k | 2 p (s) k femtocells , (2)
The SINR for FBS k ∈ K serving its femto-user FUE is given as follows:

γ (s) k = |h (s) k,k | 2 p (s) k σ (s) k 2 + |h (s) k,0 | 2 p (s) 0 macrocell + j∈K\{k} |h (s) k,j | 2 p (s) j femtocells . (3)
Finally, all FBSs are interested in optimizing a given interference mitigation metric denoted by φ k : R S•K → R + , which determines at each instant t the impact of the interference on the macro system based on the observation of all the SINR levels γ

(s) k and γ (s) 0 , with (k, s) ∈ K × S.
The interference mitigation metric considered in this work is:

φ k p k , p -k = S s=1 log 2 1 + γ (s) k .1 γ (s) 0 >Γ (s) 0 . (4)
This metric at a given instant t is different from zero only if the macrocell satis es at time t the minimum SINR level at least over one of the available channels. Hence, as long as the macrocell tier sees its QoS requirement satis ed, femtocells obtain a positive reward/payoff.

C. Game Theoretic Model

The cross-tier interference mitigation problem described in the previous section can be modeled by a normal-form game G = K, {A k } k∈K , {φ k } k∈K . Here, K represents the set of FBSs in the network and for all k ∈ K, the set of actions of FBS k is the set of power allocation vectors A k described in (1). We denote by A = A 1 × ... × A K the space of actions, and φ k : A k → R + is the payoff function of femtocell k.

At each time t and for all k ∈ K, FBS k chooses its action from the nite set A k following a probability distribution

π k (t) = π k,q (0,0) k (t), π k,q (1,1) k (t), ..., π k,q (L k ,S k ) k (t) where π k,q (l k ,s k ) k is the probability that femtocell k plays action q (l k ,s k ) k at time t, i.e., π k,q (l k ,s k ) k = Pr p k (t) = q (l k ,s k ) k . ( 5
)
where

(l k , s k ) ∈ {1, ..., L K } × S ∪ {(0, 0)}.
In what follows, we give a de nition of the equilibrium concept to be used in the sequel of this paper.

D. ǫ-Coarse Correlated Equilibrium (ǫ-CCE)

Since the action set A is discrete and nite, the game G admits at least one equilibrium in mixed strategies. In what follows, we formally de ne the ǫ-coarse correlated equilibrium (ǫ-CCE):

De nition 1 (ǫ-Coarse Correlated Equilibria):

The proba- bility distribution π ∈ ∆(A) is a ǫ-coarse correlated equilib- rium if ∀k ∈ K and ∀a ′ k ∈ A k , a -k ∈A -k φ k (a ′ k ,a -k )π -k,a -k - a∈A φ k (a k ,a -k ) πa ǫ, (6)
where

π -k, a -k = a k ∈A k π(a k , a -k ) is the marginal probabil- ity distribution w.r.t. a k .
Note that, by letting ǫ = 0, the classical de nition of coarse correlated equilibrium is obtained. An CCE is a probability distribution over the set of action pro les of the game from which no player has incentives to deviate. When a player observes the values of other players actions, the classical regret matching procedure exhibits convergence which is described in the following theorem.

Theorem 1 (Hart and Mas-Colell [START_REF] Hart | A simple adaptive procedure leading to correlated equilibrium[END_REF]): If every player plays according to the adaptive procedure of regret matching, then the empirical distribution of play converges almost surely as time goes in nity to the set of correlated equilibrium distributions of the game G.

Remark 1: It is worth noting that correlated equilibria which are a generalization of Nash equilibria are more relevant within the context of decentralized and dense networks (such as femtocells), as it induces players to coordinate their actions, and hence reach better overall performance than the Nash approach (with no coordination). The following section describes how femtocells learn these equilibria in a totally decentralized manner.

III. REGRET-BASED LEARNING PROCEDURE FOR CROSS-TIER INTERFERENCE MITIGATION

Let us assume that a given FBS k ∈ K compares the time-average of its utility observations φk (n) obtained by constantly changing its actions following a particular strategy π k , with the case where it would have played the same action in all previous stages of the game, while other players use their current strategies π -k . Our behavioral assumption is that all femtocells are interested in choosing the probability distribution π * ∈ △ (A) that minimizes the regret, where the regret of player k for not having played action q (ℓ k ,s k ) k from n = 1 up to time t is calculated as follows:

r k,q (ℓ,s) k (t) = 1 t t n=1 φ q (ℓ,s) k , p -k (n) -φk (n), (7)
Note that if r k,q (l k ,s k ) k (t) > 0, player k ∈ K would have obtained a higher average utility by playing action q

(ℓ k ,s k ) k
during all the previous stages. Thus, player k regrets for not having done it. On the contrary, if r k,q (l k ,s k ) k (t) 0, player k does not regret at all. Hence, given a vector of regrets up to time t, where r k (t) = r k,q (0,0) k (t), . . . , r k,q (L k ,S k ) k (t) , player k would be inclined towards taking actions with the highest regret, that is:

π k,q (l k ,s k ) k (t) = max 0, r k,q (l k ,s k ) k (t) (l,s)∈A k max 0, r k,q (l,s) k (t) . ( 8
)
Remark 2: At each time t > 0, FBS k calculates its regret vector r k (t) = r k,q (0,0)

k (t), . . . , r k,q (L k ,S k ) k (t)
. This hinges on the fact that: (1) each FBS knows the explicit expression of its own utility function φ k (., .), and (2) each FBS observes the actions of all the other players at each time t, p -k (t). Clearly, these assumptions are unrealistic in practice due to the distributed nature of femtocell networks, and thus techniques for relaxing the information conditions are paramount. The following section shows how to deal with such a case. Surprisingly as will be shown, one can design variants of the regret matching procedure which requires no knowledge whatsoever of other players actions, and yet yields similar performance.

IV. LEARNING THE ǫ-COARSE CORRELATED EQUILIBRIUM (ǫ-CCE)

As previously noted, the classical regret matching learning approach is unsuitable for solving the cross-tier interference mitigation problem due to the amount of required information at every FBS k. Here, we describe our novel distributed learning algorithm which signi cantly relaxes this assumption. To do that, femtocells face a trade-off between minimizing their regret and estimating their achieved time-average utility by playing a particular action at each time t. Hence, a suitable behavioral rule for each femtocell would be choosing the actions which yield high regrets more likely than those yielding lower regrets, but in any case always letting a non-zero probability of playing any of the actions. Formally speaking, the behavioral rule described above can be modeled by the probability distribution β k (r + k (t)) satisfying:

β k (r + k (t)) ∈ arg max π k ∈△(A k ) p k ∈A k π k,p k r k,p k (t) + κ k H(π k ) , (9)
where, we denote by r + k (t) the vector of positive regrets:

r + k (t) = max (0, r k (t)).
The temperature parameter κ k > 0 represents the interest of FBS k to choose other actions rather than those maximizing the regret in order to improve the estimations of the vectors of regrets [START_REF] Nash | Equilibrium points in n-person games[END_REF]. The unique solution to the right hand side of the continuous and strictly concave optimization problem in (9) is written as:

β k (r + k (t)) = β k,q (0,0) k (r + k (t)), β k,q (1,1) k (r + k (t)), ..., β k,q (L k ,A k ) k (r + k (t)) (10)
where for all k ∈ K and for all (l k , s k ) ∈ L k × S:

β k,q (l k ,s k ) k (r + k (t)) = exp κ k r + k,q (l k ,s k ) k (t) p k ∈A k exp κ k r + k,p k (t) , (11)
where

β k,q (l k ,s k ) k (r + k (t)
) > 0 holds with strict inequality regardless of the regret vector r k (t). In what follows, the distributed regret based learning algorithm for cross-tier interference mitigation between macro and femtocell tier is formally described.

A. Distributed No-Regret Learning Algorithm

The distributed learning procedure carried out independently by every FBS k ∈ K is described below where ∀k ∈ K and

∀(ℓ k , s k ) ∈ {1, . . . , L k } × S ∪ {(0, 0)}:                                φk,q (ℓ k ,s k ) k (t) = φk,q (ℓ k ,s k ) k (t-1)+ ν k (t)1 p k (t)=q (ℓ k ,s k ) k φk (t)- φk,q (ℓ k ,s k ) k (t-1) , r k,q (l k ,s k ) k (t) = r k,q (l k ,s k ) k (t-1)+λ k (t) φk,q (ℓ k ,s k ) k (t-1)- r k,q (l k ,s k ) k (t-1)-φk (t) , π k,q (l k ,s k ) k (t) = π k,q (l k ,s k ) k (t-1)+ α k (t) β k,q (l k ,s k ) k (r k (t))-π k,q (l k ,s k ) k (t-1) κ k (t) = κ k (t-1)+ǫ k (t).Ψ(t) (12)
where π k (0) and r k (0) are arbitrary initial actions and regrets, Ψ is a non-decreasing function, ν k , α k , λ k and ǫ k are learning rates 1 chosen such that:

1 In this work, femtocells are assumed to have similar learning rates, i.e., ∀k ∈ K, λ k = λ, α k = α. Different learning rates across players is left for future work. lim

T →∞ T t=0 α(t) + λ(t) = +∞ (13) lim T →∞ T t=0 α(t) 2 + λ(t) 2 < +∞, (14)
lim t→∞ λ(t) ν(t) = 0, (15)
lim t→∞ α(t) λ(t) = 0 (16) lim t→∞ ǫ(t) α(t) = 0 (17)
Remark 3: We would like to stress that in contrast to the classical regret matching approach where each player k perfectly knows its own utility function φ(q

(l,s) k , p -k (t))
as well as other players chosen strategies, our proposed algorithm is totally decentralized and is composed of three phases. First, using its own instantaneous observed utility, each player is able to estimate its expected utility with each of its actions. Second, the estimated utility function allows players to compute their regrets of playing a given action. Third, players update the probability distribution of their transmission strategies. Moreover, conditions (13)-( 17) are necessary to guarantee that the regret processes r k (t) always see the process φk (t) as fast processes always calibrated to the current value of the regret, and likewise, for the strategy distribution processes π k (t) which sees the regret processes as fast processes always calibrated to the current values, as per (15). Finally, it is important to note that the temperature parameter κ k is time-dependent in order to account for the fact in the beginning of the learning process femtocells may decide to explore all their actions (small κ) to favorite the estimation of the expected utility and corresponding regrets. Then as κ k increases, the behavioral rule approaches the classical regret learning. That is, the ǫ (of the ǫ-CCE) is made smaller as time increases which ensures the asymptotic convergence to CCE. A formal proof of the convergence of this algorithm relies on stochastic approximation theory using the notions of multiple time scale and is not included here for space limitation.

V. SIMULATION RESULTS

Let us consider one macrocell with radius R m = 500 underlaid with K femtocells each of radius R f = 20 m, transmitting over an arbitrary number of carriers S, with L transmit power levels. The minimum SINR of the macrocell UEs is given by Γ

0 = Γ (1) 0 , . . . , Γ (S) 0 where Γ (1) 0 = • • • = Γ (N ) 0
= 3 dB is assumed. The transmission power of the macro BS is set to 43 dBm, whereas the FBS adjusts its power through the various learning schemes to a value of maximum 10 dBm. The channel is represented as a combination of path-loss fading and lognormal shadowing with standard deviation of 8 and 4 dBm for outdoor and indoor communications, as per [START_REF]3G Home NodeB Study Item Technical Report[END_REF].

In Figure 2, we plot the average femtocell sum-rate for K = 2 FBSs underlaying one macrocell over s = 2 sub- carriers, highlighting the convergence behavior of our proposed algorithm as well as the benchmark case (classical regret matching). We observe that surprisingly the proposed learning approach yields the same performance as the RM in the longterm. This is quite signi cant given the fact that unlike the RM approach, our algorithm is totally decentralized and does not require any information about other players strategies.

Figures 34plot the convergence of the probability distribution over the set of actions taken by both FBSs. As time goes by FBS k = 1 increases the probability to transmit with the maximum power level on frequency band s = 1, while the probability of transmitting on the other bands decreases. On the other hand, the probability that FBS k = 2 transmits with maximum transmit power level on carrier s = 1 decreases with time, whereas the probabilities of transmitting over carrier s = 2 with maximum transmit level increases. It can also be seen that although femtocells do not communicate with each other, they implicitly coordinate their transmissions by using different frequency bands with very high probability.

VI. CONCLUSIONS

In this paper, the problem of cross-tier interference mitigation was studied from a game theoretic learning perspective. Owing to their implicit coordination, a regret-based learning algorithm was proposed where femtocells jointly estimate their own utility function and learn their transmission strategies in a purely decentralized manner, relying only on local information. The considered behavioral rule was shown to converge to an ǫ-CCE where femtocells optimize their exploration and exploitation tradeoff. Remarkably, it was shown that the proposed algorithm achieves the same performance of the classical regret matching approach with much less overhead.

Fig. 1 .

 1 Fig. 1. Two-tier network topology including a mix of high-power (Macro-BS) and low-power base stations (Pico-BS and Femto-BS). MUE/FUE/PUE stand for macro/femto/pico user equipments.

  ∈ R S denote the s-th vector of the canonical base spanning the space of real vectors of dimension S. Here, ∀n ∈ {1, ..., S} \ {s}, e

  | 2 denote the channel gain between the MBS and its associated MUE in sub-carrier s ∈ S. Likewise, |h (s) i,j | 2 denotes the channel gain between transmitter j and receiver i on sub-carrier s, and let σ(s) k 2

Fig. 2 .

 2 Fig. 2. Convergence of the proposed learning algorithm as well as the classical regret matching in terms of Ergodic transmission rate, for K = 2 femtocells, S = 2 channels, and L = 2 power levels.

Fig. 3 .

 3 Fig. 3. Probability distribution over the set of actions of femtocell 1.

Fig. 4 .

 4 Fig. 4. Probability distribution over the set of actions of femtocell 2.


Romain Couillet

Samir Medina Perlaza

Hamidou Tembine

Mérouane Debbah

A mean field game analysis of electric vehicles in the smart grid

In this article, we develop a mean field game model for the economical analysis of the integration of purely electrical vehicles (EV) or electrical hybrid oil-electricity vehicles (PHEV) in the smart grid energy market. The framework we develop allows for a consistent analysis of the evolution of the price of electricity, of the timely demand, and possibly of the energy reserves in the grid, when EV or PHEV owners buy and sell electricity from their cars, selfishly but rationally, based on collective price incentives.

I. INTRODUCTION

It is widely recognized [START_REF] Simpson | Cost-benefit analysis of plug-in hybrid electric vehicle technology[END_REF], [START_REF] Bradley | Design, demonstrations and sustainability impact assessments for plug-in hybrid electric vehicles[END_REF], [START_REF] Kintner-Meyer | Impacts assessment of plug-in hybrid vehicles on electric utilities and regional us power grids. part i: Technical analysis[END_REF] that the future intense penetration of electrical vehicles (EV) and plug-in hybrid electrical vehicles (PHEV) in the energy market will generate an important additional energy demand, and therefore a new strain to the electricity suppliers. At the same time, EV and PHEV owners are not only energy consumers as EVs and PHEVs can charge as well as discharge their battery contents from or to the electrical grid. From this perspective, these vehicles can be seen as an additional large dimensional energy buffer to flatten the peaky daily and seasonal energy demands. That is, EVs or PHEVs can connect to the smart grid and sell their energy surpluses, when needed. It is therefore an important economical and social challenge to enforce charge and discharge policies to EVs and PHEVs in an optimal manner. Here, optimality must be interpreted in the sense of maximum individual revenue obtained by the EV and PHEV owners when participating in the energy trades, as well as maximum performance of the electricity grid, e.g. low probability for blackouts, large energy reserves, etc. The price at which the energy is sold or bought depends obviously on the total demand and offer of all electrical vehicles present in the network (and also on the existing energy consumption by ancillary services in the grid). The electricity price is set by the energy market, and we therefore assume the existence of communications between EVs and the grid. Relevant important aspects of EV-to-grid communication protocols can be found in e.g. [START_REF] Tara | Battery storage sizing in a retrofitted plug-in hybrid electric vehicle[END_REF], [START_REF] Gonder | Using global positioning system travel data to assess real-world energy use of plugin hybrid electric vehicles[END_REF], [START_REF] Pan | Locating PHEV exchange stations in V2g[END_REF]. Since each EV action impacts the satisfaction of all other EVs, it impact the next actions of the latter in return, all of them being therefore in competition.

This competitive interaction in which each vehicle owner decides the amount of energy to be sold or bought at any time given a global price can be analyzed using tools from dynamic game theory [START_REF] Bas | Dynamic Noncooperative Game Theory[END_REF]. For instance, in [START_REF] Saad | A noncooperative game for double auction-based energy trading between PHEVs and distribution grids[END_REF], the coexistence of a number of PHEV groups aiming to sell part of their stored energy to the smart grid are studied using non-cooperative game theory. The authors propose in particular an algorithm based on best response dynamics to allow PHEV groups to reach a Nash equilibrium point [START_REF] Nash | Equilibrium points in n-person games[END_REF]. Nonetheless, in practical scenarios, the number of vehicles is extremely large and, thus, elements from classical game theory are difficult to handle and in general do not bring enough insight about the global behavior of the market.

To overcome this problem, in this paper, we study the energy trade when the number of vehicles tends to infinity and all vehicles can be considered identical. Within this framework, we model the game as a mean field game [START_REF] Lasry | Jeux à champ moyen. i -le cas stationnaire[END_REF], [START_REF]Jeux à champ moyen. ii -horizon fini et contrôle optimal[END_REF]. As opposed to N -player games where the objective is to follow the evolution of the state X (i) t of every individual player i ∈ {1, . . . , N } at every time instant t of the game, the object of mean field games is rather to obtain the optimal distribution m (t, X) of the players being in state X at time t (all players being in the same state at the same time behave similarly). As a main consequence, it is possible in these games to follow the state trajectory of all players at once and to capture the behavior of the players depending only on (i) their initial state X 0 and (ii) the joint distribution m (0, •), at time t = 0. The notion of Nash equilibrium in the context of mean field games is extended to the notion of mean field equilibrium [START_REF] Lasry | Jeux à champ moyen. i -le cas stationnaire[END_REF], [START_REF]Jeux à champ moyen. ii -horizon fini et contrôle optimal[END_REF]. A mean field equilibrium is now a fixed point solution to a coupled system of stochastic partial differential equations which includes a (backward) Hamilton-Jacobi-Bellman (HJB) equation and a (forward) Fokker-Planck-Kolmogorov (FPK) equation. The HJB equation determines the optimal controls α t = α t (X t ), or actions, of the players (the EV owners) in state X t at time t, given any density measure m, while the FPK equation determines the density m for a given control function α t .

In [START_REF] Guéant | Mean field games and oil production[END_REF], a mean field game approach to the study of oil production in unlimited time is developed where the selfish players are oil producers and the mean field variable is the oil selling price. In this article, we develop a similar framework to [START_REF] Guéant | Mean field games and oil production[END_REF] but on a finite time horizon, applied to both EVs and PHEVs, with vehicle owners as the selfish players and electricity price as the mean field variable of interest. The mean-field property relates presently to the fact that every player's action is driven not by the individual actions of each other player but by the collective (or mean) behavior of all players. Indeed, we will assume that the price for electricity is driven both by the smart grid which may enforce policies at specific time instants but more importantly by the total amount of energy bought at time t by the players. The state evolution under study will be here the evolution of the quantity X t of electricity in the battery of the EV owners or the joint quantity Z t = [X t Y t ] T of electricity and oil in the battery and tank of PHEV owners. This state is controlled by the quantity α t (X t ) of electricity, or (α X t (Z t ), α Y t (Z t )) of electricity and oil, bought or sold by players in state X t , or Z t , at time t.

The reminder of this article unfolds as follows. In Section II, we introduce the mean field game model for the electrical vehicle problem under consideration. In Section III, we develop the fundamental equations describing a mean field equilibrium of the systems under study. In Section IV, we run numerical simulations and derive conclusions in both study scenarios. Finally, in Section V, we conclude this work.

Notations:

The notations ∂ x F (X, Y ), ∂ x F (X, Y ), ∂ 2
xx F (X, Y ) and ∂ 2 yy F (X, Y ) stand for the first and second partial derivatives of the function F along the variables X and Y . The notation dX for X a random process is the differential notation following Itô's definition. The notation [x] + stands for max(x, 0). The function δ A is the indicator function on the set A.

II. MODEL

In this section, we develop the two models of EVs and PHEVs under study.

A. Electrical vehicles

Consider the game to be played by N electrical vehicle owners who decide at all time on the optimal battery level given a utility function encompassing the economical cost or gain of filling or emptying the battery, the (psychological) cost of operating on the battery at certain time instants of the day, and the disutility for the user to have a nearly empty battery.

For player i ∈ {1, . . . , N }, we denote X (i)

t ∈ [0, 1] its state variable, corresponding to its battery level at time t ∈ [0, T ]. We assume that player i modifies X (i) t by a quantity α (i) t dt, at time t. We also assume that player i consumes a quantity g (i) t dt of energy at time t. We therefore have the state evolution for each player given by the equation d dt X

(i) t = α (i) t -g (i) t
where we implicitly assumed the differentiability of the state variable

X (i)
t . This supposes in particular that electricity will be bought or sold constantly, instead of by bursts. This is a rather unrealistic assumption which will be made more realistic when we consider the mean field limit.

The objective for player i is to minimize the cost for an initial condition {X (j) 0 } j for all players over the time window [0, T ], given by

v (i) 0, {X (j) 0 } j ; {α (i) t } t , {α (-i) t } t = T 0 α (i) t p t ({α (j) t } j ) + h i (t, α (i) t ) + f i (t, X (i) t ) dt + k i (X (i) T )
where α

(-i) t = [α (1) t , . . . , α (i-1) t , α (i+1) t , . . . , α (N ) t
] T are the controls of all players but i at time t, the term p t ({α

(j) t } j )
is the price of electricity (identical for all players) given the instantaneous energy demand j α (j) t

(we assume that both buying and selling prices are equal), h i (t, α) is the psychological cost for player i to perform a change αdt in the battery level at time t (some time periods are more appropriate than others to purchase or sell energy depending on the player's routine), f i (t, X) is the cost for player i to be in state X at time t (typically, one does not wish to have a low battery level in certain periods of time), and k i (X) is the (terminal) cost of being at state X at time T (this avoids the undesirable effect that all energy is sold in the last instants of [0, T ]). Details on specific choices for these functions are provided later.

In game theory, this formulation is known as an N -player continuous-time differential game of pre-specified fixed duration T [START_REF] Bas | Dynamic Noncooperative Game Theory[END_REF]. The notion of equilibrium in this game corresponds to the state of simultaneous satisfaction of all players in the sense of Nash [START_REF] Nash | Equilibrium points in n-person games[END_REF], [START_REF] Aubin | Mathematical methods of game and economic theory[END_REF]. That is, the family of controls {α

(i) t } t,i engender a Nash equilibrium if v (i) 0, {X (j) 0 } j ; {α (i) t } t , {α (-i) t } t ≤v (i) 0, {X (j) 0 } j ; {α (i) t } t , {α (-i) t } t for each i, with {α (i) t } t any admissible family of controls of player i over t ∈ [0, T ].
The interest of Nash equilibria in the EV interaction lies in the fact that, at such state, all the EVs are using a control policy which is optimal with respect to the control policy of all the other EVs. Otherwise stated, if the system is at equilibrium, the unilateral deviation of any player would lead it to a higher cost and, thus, none of the players has a particular interest on unilaterally changing its control functions. However, analyzing the Nash equilibrium of a game where N is a large number is a very complex problem. In fact, even if a Nash equilibrium exits, it would lead to solutions that are inherently difficult to exploit. In particular, it is clear that, under this formulation, any change in the battery level of a given player impacts all other players which must react as a consequence. Instead, we will consider a simplification of this framework by assuming a fluid limit as N grows large. The foundations of this approach were established in [START_REF] Aumann | Markets with a continuum of trackers[END_REF]; see also [START_REF] Carmona | Nash equilibria of games with a continuum of players[END_REF] for recent results. Within this framework, instead of considering the vector (X

(1) t , . . . , X (N ) t
) of state spaces at time t, we consider the variable X t with density measure m(t, X t ) at time t, defined as

m(t, X) = lim N →∞ 1 N N i=1 δ {X (i) t ≤X} (X)
assuming the existence of such a fluid limit. The control variable is now a variable α t with density. Since it would be inappropriate to assume that all individual players consume exactly the same density of electricity per time instant, we consider instead that the energy consumption is a random variable with mean g t dt and with variance σ 2 t at time t; this is modelled through the variable g t [dt + σ t dW t ], where W t is a Brownian motion and the differential notation dW t must be interpreted in the sense of Itô [START_REF] Itô | Multiple Wiener integral[END_REF]. All other cost functions are supposed identical for all players, so we substitute in particular the functions h, k and f to h i , k i and f i , respectively, and denote p t (m(t, •)) the price of electricity given the density m(t, •) at time t.

With these updated notations, the objective for any given user is to solve the following stochastic control problem

v(0, X 0 ) = inf (αt) t∈[0,T ] E T 0 C(α t , X t , m, t)dt + k(X T ) dX t = α t dt -g t [dt + σ t dW t ] + dN t (1)
where dN t is a reflective variable to ensure that X t remains in [0, 1].

Regarding the cost function, C is given by

C(α t , X t , m, t) = α t p t (m(t, •)) + h(t, α t ) + f (t, X t ).
It would stand then to reason to consider that p t (m) writes as a function of the total instantaneous demand α t (X)m(t, X)dX. However, for computational ease, we will instead consider that prices are fixed not by the total consumption α t (X)m(t, X)dX but by the total anticipated consumption g t + d dt Xm(t, X)dX , where both quantities only differ by an additional Brownian motion term when σ t > 0. That is, we assume that the energy providers do not have the information on the instantaneous demand at time t but are able to track the density m(t, •) at all time. This is a reasonable assumption as this only requires for the market to obtain the status information feedback from a not-necessarily large sample of electrical vehicles at different (sampled) time instants.

We therefore define p t as

p t (m) = D(t, •) -1 g t + d dt Xm(t, X)dX
where D(t, p) is the total energy demand function at time t for a given price p. This function is central to create incentive behavior among players. For instance, at peak demand periods for electricity services other than EVs, p may be increased for negative EV electricity demand and decreased for positive EV electricity demand so to enforce car owners to sell rather than buy electricity. This fluid limit formulation now defines a mean field game, as introduced in [START_REF] Lasry | Jeux à champ moyen. i -le cas stationnaire[END_REF], [START_REF]Jeux à champ moyen. ii -horizon fini et contrôle optimal[END_REF]. Our specific problem setting is as a matter of fact very close to the work [START_REF] Guéant | Mean field games and oil production[END_REF]. In this framework, the solutions to the control problem (1) can be seen as generalizations of Nash equilibria [START_REF] Nash | Equilibrium points in n-person games[END_REF] in the equivalent N -person game. It is in general difficult to prove that the mean field solutions are well-defined limits of the N -person game, see e.g. the discussions in [START_REF] Guéant | Mean field games and applications[END_REF]. Similarly, it is in general difficult to prove the existence and uniqueness of solutions to (1) and, if so, to derive a numerical method that is provably converging to the solution as the discretization step tends to zero, see e.g. [START_REF] Achdou | Mean field games: Numerical methods[END_REF]. We will not try to prove any of these aspects here, our main target being rather to informally explore the potentials of the mean field game setting to the economical problem of optimal policies for EV and PHEV penetrations in the smart grid. Similar to [START_REF] Guéant | Mean field games and oil production[END_REF], only the numerical results will convey a justification of the correct behavior of our method.

In the next section, we improve the EV framework by turning the purely electrical vehicles into PHEVs, introducing therefore the possibility for players to select between two alternative sources of energy.

B. Hybrid vehicles

We now consider that players choose to buy, sell and consume either electricity with the elastic price defined as in Section II-A or a fix-priced alternative energy source, oil. We consider immediately the mean-field game limit. In this scenario, the state function becomes a two-dimensional vector

Z t = [X t Y t ] T , where Y t ∈ [0, 1]
is the content of the oil tank. We now replace the control α t by a vector γ t = [α X t α Y t ] T , where α X t ∈ R is the content of electricity bought or sold by the vehicle owner at time t and α Y t ≥ 0 is the oil content bought by the owner at time t (oil cannot be sold once bought). In addition, we denote β(t, Z t ) ∈ [0, 1] the relative quantity of electricity being consumed by the PHEVs at time t. That is, we assume that all vehicles have a common policy of energy consumption in the considered time window. Typically, taking β(t, Z t ) = X t /(X t + Y t ) translates a policy where energy is consumed indistinctly of the energy source. Note that, depending on the typical distances covered by PHEV owners at time t (e.g. weekdays against weekends), β(t, Z t ) may explicitly depend on t. Alternatively, we may have considered β(t, Z t ) an additional control variable which can be set optimally by the car owner depending on the status of the energy market. For simplicity of analysis and because this involves a somewhat unrealistic constant calculus from the PHEV owners, we do not consider this scenario here. We now call m the density for the variable Z t .

The price for electricity is given by the function p X t (m) updated as follows:

p X t (m) = D(t, •) -1 g t Z β(t, Z)m(t, Z)dZ + d dt X Xm(t, Z)dX .
The price for oil is given by p Y = p Y t , supposed constant throughout [0, T ], which is a satisfying assumption on a daily or even weekly basis. The optimal control problem now formulates as:

v(0, Z 0 ) = inf (γt) E T 0 C(γ t , Z t , m, t)dt + k(Z T ) dZ t = α X t α Y t dt - β(t, Z t ) 1 -β(t, Z t ) g t [dt + σ t dW t ] + dN t
where v(0, Z 0 ) = v(0, Z 0 ; {γ t } t ) assumes the dependence on {γ t } t and where

C(γ t , Z t , m, t) = α X t p X t (m) + α Y t p Y + h(t, γ t ) + f (t, Z t )
with similar notations as previously. Now that both mean field game formulations have been laid out, we concentrate on solving these optimal control problems in the next section.

III. MEAN FIELD EQUILIBRIUM

In this section, we analyze the mean field game described in the previous section. In this context, the notion of Nash equilibrium introduced previously is extended to the notion of mean field equilibrium [START_REF] Lasry | Jeux à champ moyen. i -le cas stationnaire[END_REF], [START_REF]Jeux à champ moyen. ii -horizon fini et contrôle optimal[END_REF].

As already mentioned, the mean field equilibrium is obtained as a solution to a coupled system of stochastic partial differential equations constituted by a backward HJB equation and a forward FPK equation. In this section, we detail these fundamental equations in the EV and PHEV frameworks of Section II-A and Section II-B, respectively.

A. Electrical vehicles

Consider the optimization problem of Section II-A, and define the following value function,

v(u, X u ) = inf (αt) t∈[u,T ] E T u C(α t , X t , m, t)dt + k(X T ) .
Then, the measure m (t, X t ) and the control functions α t at the mean field equilibrium satisfy the following system of coupled differential equations given the initial distribution m(0, •). First, an HJB equation determines the optimal trajectory v (t, X t ) in the control problem [START_REF] Simpson | Cost-benefit analysis of plug-in hybrid electric vehicle technology[END_REF]. That is,

0 = ∂ t v(t, X) + inf α {α∂ x v(t, X) + C(α, X, m (t, X), t)} -g t ∂ x v(t, X) + 1 2 g 2 t σ 2 t ∂ 2 xx v(t, X). (2)
Then, an FPK equation determines the measure m (t, X t ) of the game state. Precisely, this is

∂ t m(t, X) = -∂ x [(α t -g t )m(t, X)] + 1 2 g 2 t σ 2 t ∂ 2 xx m(t, X) (3
) where α t is the argument of the infimum in [START_REF] Bradley | Design, demonstrations and sustainability impact assessments for plug-in hybrid electric vehicles[END_REF]. Details of how these equations are obtained can be found in e.g. [START_REF]Stochastic differential equations: an introduction with applications[END_REF].

We assume here that the cost h(t, α) for control is quadratic and reads

h(t, α) = 1 2 h t α 2
with h t > 0 representing the unwillingness of the car owner to buy or sell energy at time t. This choice is seemingly non-natural as it implies that users are more willing to buy or sell small quantities rather than large quantities of energy. Nonetheless, under the mean field game formulation, this has to be understood as the fact that, on average, only a limited population of users at time t is willing (or able) to buy energy. For instance, during nighttime, many EVs are plugged to an electrical source so that h t is smaller at night (more users can ask for energy). As such, intuitively, making the (psychological) cost of buying or selling energy larger for larger amounts of energy forces only part of the population to buy or sell. As for the particular choice of a quadratic cost rather than any other cost function, it is convenient for calculus mostly and could be easily replaced by more appropriate functions.

Under these conditions, from Pontryagin's minimum principle [START_REF]Stochastic differential equations: an introduction with applications[END_REF], we immediately have an expression of the optimal control α t as the minimizer of inf α {α∂ x v(t, X t ) + C(α, X, m , t)} .

Precisely, we have

α t = - 1 h t [∂ x v(t, X t ) + p t (m )]
possibly submitted to some boundary conditions to ensure that X t ∈ [0, 1] at all times. In the remainder of the article, we will assume this condition always met, so that at no time we will consider EV owners with completely full or completely empty batteries.

The HJB equation now becomes

0 = ∂ t v(t, X) - 1 h t [∂ x v(t, X) + p t (m )] + g t ∂ x v(t, X) - p t (m ) h t [∂ x v(t, X) + p t (m )] + f (t, X) + 1 2h t [∂ x v(t, X) + p t (m )] 2 + 1 2 σ 2 t g 2 t ∂ 2 xx v(t, X)
which can be simplified as

∂ t v(t, X) = 1 2h t (∂ x v(t, X) + p t (m )) 2 + g t ∂ x v(t, X) -f (t, X) - 1 2 σ 2 t g 2 t ∂ 2 xx v(t, X) and the FPK equation is ∂ t m(t, X) = 1 h t [∂ x v (t, X) + p t (m(t, X))] + g t ∂ x m(t, X) + 1 h t ∂ 2 xx v (t, X)m(t, X) + 1 2 g 2 t σ 2 t ∂ 2 xx m(t, X).
This defines the two fundamental differential equations to be solved for the optimal EV policy. We now turn to the resolution of the PHEV problem.

B. Hybrid electric-oil vehicles

Under the notations of Section II-B, and similar to the previous section, we consider the value function

v(u, Z u ) = inf (γt) t∈[u,T ] E T u C(γ t , Z t , m, t)dt + k(Z T )
where we remind that Z t = [X t Y t ] T ] is a two-dimensional vector, with a given initial distribution m(0, •).

As in the EV formulation, we consider the cost function h as quadratic, as follows

h(t, γ t ) = 1 2 h X t (α X t ) 2 + 1 2 h Y t (α Y t ) 2 .
The HJB equation is here given by

-∂ t v = inf (γ) p X t α X + p Y α Y + h(t, γ) + (α X -g t β)∂ x v +(α Y + g t (β -1))∂ y v + f (t, Z t ) + 1 2 σ 2 t g 2 t β 2 ∂ 2 xx v + 2β(1 -β)∂ 2 x,y v + (1 -β) 2 ∂ 2 yy v with notation γ = [α X α Y ] and β = β(t, Z t ).
Due to the parameter decoupling, the optimal (α X t ) and (α Y t ) are easily found as

(α X t ) = - 1 h X t (p X t (m ) + ∂ x v) (α Y t ) = - 1 h Y t (p Y + ∂ y v)
with possibly additional constraints to ensure (α Y t ) ≥ 0 and Z t ∈ [0, 1] 2 . We will assume all these conditions met for simplicity.

The FPK equation is in turn given by

∂ t m = -∂ x (α X t -βg t )m -∂ y (α Y t + (β -1)g t )m + 1 2 g 2 t σ 2 t β 2 ∂ 2 xx m + (1 -β) 2 ∂ 2 yy m + 2β(1 -β)∂ 2 xy m
where we remind that β = β(t, Z t ) depends explicitly on Z t . Assuming σ t = 0 to obtain compact forms, we then have, after substitution, the HJB equation

∂ t v = 1 2h X t ∂ x v + p X t (m ) 2 + 1 2h Y t ∂ y v + p Y t 2 + g t β∂ x v + g t (1 -β)∂ y v -f
where m is solution to the FPK equation

∂ t m = 1 h X t (∂ 2 xx v ) + 1 h Y t (∂ 2 yy v ) + g t [∂ x β -∂ y β] m + 1 h X t (p X t (m) + ∂ x v ) + βg t ∂ x m + 1 h Y t (p Y + ∂ y v ) + (1 -β)g t ∂ y m.
with v the solution to the HJB equation. For β(t, Z t ) = Xt Xt+Yt , we have in particular

g t [∂ x β -∂ y β] m = 1 X t + Y t g t m.
In all generality, we cannot go further with the resolution of these equations which need to be solved numerically. This is the target of the next section in which simulation results are provided.

IV. SIMULATIONS

In this section, we provide simulation results for the electrical vehicle schemes developed in Section II-A and Section II-B.

A. EV analysis

We first consider the scenario of Section II-A. We assume a realistic 24-hour scenario where players have an average consumption rate and a possibility to buy or sell that depend on specific periods of the day (t = 0 stands for midnight and t = 1 for midnight a day after). However, for the sake of simplified interpretation, we consider only a rough approximation of the system parameters. Note in particular that all parameters chosen below are designed in the interest of readability of the results and do not rely on explicit models taken from the literature.

Specifically, the demand function D(t, p) is an affine function of the price. We take here p = (D(t, p)) + + 0.5, which ensures a positive selling price. The major trend is satisfying as it assumes that the more electricity is drained from the smart grid, the more expensive additional withdraws of energy. The threshold p t ≥ 0.5 provides a price for the electricity sold. Also note that D does not depend explicitly on t and, therefore, we assume here that the reserves in the electricity grid do not fluctuate sufficiently during the day to impact the price of the electricity dedicated to electrical vehicles. We may typically have chosen to change the threshold 0.5 so to encourage selling electricity at peak hours. We take the mean consumption rate g t in such a way that g t > 0 at any time and with two consumption peaks in the morning and in the evening (to model e.g. EV owners leaving to work and returning). This is depicted precisely in Figure 1. The variance σ 2 t on the consumption is taken equal to 0.01 at all time, ensuring a standard deviation of the order of 10%. We then consider h t to be maximal at night (since most EVs are then plugged) and minimal in the middle of the day (assuming that most EVs cannot be plugged at workplaces). This is given precisely in Figure 2. The cost f is given by f (t, X) = 10(1 -X) 2 in order to constrain the players to fill their batteries. Finally, the boundary and initial constraints for the resolution of the system of HJB and FPK equations enforce the following policies:

• m(0, •) is a triangle distribution centered at 0.5 and with support [0.3, 0.7], • v(T, X) = 2(1 -X) 2 is a cost on being far from X = 1, • ∂ x m(0, •) = ∂ x m(1, •) = ∂ x v(0, •) = ∂ x v(1, •) = 0 in
order to force the energy content to lie in [0, 1].

To solve the system of equations in (m, v), we proceed by solving sequentially the HJB and FPK equations using a simple fixed-point algorithm until convergence. We do not ensure here that this algorithm does converge, neither do we ensure that the solution obtained is the solution sought for. Using a finite difference method on a sampling of 192 points on both time and battery level axes, the above scheme leads to the density evolution m depicted in Figure 3. A few observations can be already made from this figure. From t = 0 to t = 0.1, the tendency is for the EVs to charge large amounts of electricity, which is due to the fact that players are more willing to charge in nighttime than in daytime. Then, due to both the increase in consumption and the decreased willingness to buy electricity (or the impossibility to do so for some players) around t = 0.3, the average quantity diminishes. The low consumption at t = 0.5, in spite of the low willingness to buy electricity, then generates a slight increase in the mean. At t = 0.7, the sharp electricity consumption then generates a large loss. After t = 0.8, electricity is bought again so to satisfy the final conditions. It is interesting to note that, due to the small variance σ 2 t that was chosen, the overall tendency is for m to concentrate into a single mass. This is a usual phenomenon which determines the steady state if time were to continue with constant values for all time-dependent system parameters. From the expression of m , v , and the equations derived in Section III, it is now possible to obtain much information about the system. In particular, it is interesting to follow the electricity demand D(t, p t (m )) and the price p t (m ) defined here as (we remind that both quantities only differ by an additional term 0.5 if both are positive)

p t (m ) = 0.5 + g t + d dt Xm (t, X)dX + .
This is depicted in Figure 4, where we see that the overall tendency is to buy larger amounts of electricity early in time.

The price is therefore decreasing with time but for a slight increase around the peak time t = 0.7. The price curve has very sharp edges that can be interpreted as before as a consequence of the evolution of g t and h t mainly. With this elementary linear price policy, it is therefore possible to reduce the expected refills at peak hours, e.g. around t = 0.7.

Obviously, this interpretation has to be taken with extreme care. In particular, note that at t = 1 the demand is very low, while it should coincide with the demand at time t = 0 if we had been analyzing a weekly or monthly behavior instead of a single day behavior. Such a property should appear if the terminal cost constraint were reinforced.

B. PHEV analysis

In this second section, we wish to analyze the behavior of hybrid vehicles as described in Section II-B. Since solving three-dimensional differential equations is time-consuming, we only provide results for the time scale discretized in 12 samples and for the "spatial" scales discretized both in 16 samples. For each differential equation, the resolution is performed by iterating the resolution of the two-dimensional differential equations along time and electricity scales for each fixed oil tank level, and time and oil scales for each fixed battery level. Then the system of HJB and FPK differential equations is solved by further iterating a fixed point algorithm as in the previous section. For simplicity of interpretation, we consider here a time-independent scenario where both g t = 0.2 and h t = (125, 125) T are constant with time. 1We take the electricity price policy to be, as above, p X = D(t, p X ) + + 0.5. The oil price is set to p Y = 0.7. This is a natural choice as it is expected that an approximate quantity g t = 0.2 will be asked for at any time to cover for the energy consumed, hence a price for electricity p X t 0.7. We impose a constraint f (t, Z) = 20(2-X -Y ) 2 , where Z = [X Y ] T . The relative consumption β of oil and electricity is proportional to the total quantity of energy, that is β(t, Z) = X/(X + Y ) and therefore 1 -β(t, Z) = Y /(X + Y ). We take σ t = 0 for simplicity. The boundary constraints are identical to those in the previous section. As for the terminal constraint on v, it imposes that v(T, Z) = 10(2 -(X + Y )) 2 .

We consider the scenario where m(0, •) is a (properly truncated and scaled) Gaussian distribution with mean (0.4, 0.6) T and covariance 0.02I 2 , with I 2 the 2 × 2 identity matrix. That is, we assume that, initially, most vehicles have more oil than electricity. This is depicted in Figure 5. We then let the system evolve freely under the above set of constraints. It is natural to guess that the overall behavior is a decrease of either or both quantities of oil and electricity to zero if the prices are too high, or an increase of either or both quantities to one, if the prices are more reasonable. What is interesting to observe is the trajectory jointly followed by the players. The resulting distribution m (1, •) is depicted in Figure 6. What we observe in the aforementioned conditions is that the initial distribution has shifted towards an increase of both electricity and oil levels, with a bigger increase of the mean battery level. Another observation is that the distribution tends to stretch along the X = Y diagonal in the figure, translating the fact that oil and electricity are seen almost as equivalent goods due to the loosely constraining energy cost policy.

Among the different further analyses, in Figure 7, we consider a section of the distribution of the optimal transaction policy (α X t ) and (α Y t ) at time t = 0 + , for Y t = 0.5 and Y t = 0.9 (we remind that both (α X t ) and (α Y t ) are functions of t, X t and Y t ). That is, we observe the initial behavior of players with half-filled oil tanks and almost completely filled oil tanks. It is observed that, for users with a very low level of electricity, buying electricity is an appealing choice. This can be interpreted by the fact that, as few players are in strong need for energy, it is possible to acquire a large quantity of electricity at a reasonable price. Those players with low reserves of electricity are the main beneficiaries. For users with already a reasonable level of electricity though, electricity and oil are seen as equivalent goods. As a matter of fact, our results also show that, at time t = 0 + , the price of electricity equals p X t = 0.706 p Y t . That is, the players with low electricity levels draw as much electricity as is needed to reach an equilibrium price with oil. Now, it is also observed that, for users with large quantities of oil, electricity becomes a compelling purchase in order to further increase the total quantity of energy (since f imposes X t + Y t to be close to 2), hence a larger incentive for buying electricity when the battery level is not large. When both battery and tank levels are alike, we see that the quantity of electricity purchased is the same as the quantity of oil purchased. Obviously, from the very generic setting of both EV and PHEV problems, many more scenarios can be carried out so to evaluate the actual impact of the EVs and PHEVs on realistic smart grid scenarios. The simulations above and their interpretations only provide a framework of understanding of fully rational vehicle owner's behavior.

V. CONCLUSION

In this article, we proposed a game theoretical framework to model the behavior of electrical vehicle and hybrid vehicle owners aiming at selfishly maximizing their satisfaction, under electricity pricing policy constraints. As the number of selfish players is large, players are alike, and the pricing policy depends on the action of all participants, we then turned the problem into a mean field game, for which we obtain the fundamental differential equations. We solved these equations numerically, from which conclusions were drawn which give new insights on the way to optimize the electrical vehicle penetration in the smart grid.

Fig. 1 .

 1 Fig. 1. Mean energy consumption gt of EVs as a function of time.

Fig. 3 .Fig. 4 .

 34 Fig.3. Density solution m (t, X) as a function of the time t and the battery level X.

Fig. 5 .Fig. 6 .

 56 Fig. 5. Initial distribution m(0, •) at time t = 0, as a function of both levels of battery and oil tank.

9 Fig. 7 .

 97 Fig.7. Optimal transactions at time t = 0 + for players with different oil and battery levels.

Such a large value for the entries of ht is motivated by faster algorithm convergence reasons, although it inhibits as a counterpart fast variations of m along time.