Benjamin Camus
email: benjamin.camus@irisa.fr

Fanny Dufossé
email: fanny.dufosse@inria.fr

Anne Blavette
email: anne.blavette@ens-rennes.fr

Martin Quinson
email: martin.quinson@irisa.fr

Anne-Cécile Orgerie
email: anne-cecile.orgerie@irisa.fr

Network-aware energy-efficient virtual machine management in distributed Cloud infrastructures with on-site photovoltaic production

Keywords: Distributed cloud computing, renewable energy, energy-efficient scheduling, on-site production, network-aware migration, consolidation

come

Network-aware energy-efficient virtual machine management in distributed Cloud infrastructures with on-site photovoltaic production

I. INTRODUCTION

The growing appetite of Internet services for Cloud resources leads to a consequent increase in data center (DC) facilities worldwide. This increase directly impacts the electricity bill of Cloud providers. Indeed, electricity is currently the largest part of the operation cost of a DC [START_REF] Bertoncini | Next Generation Data Centers Business Models Enabling Multi-Resource Integration for Smart City Optimized Energy Efficiency[END_REF]. Resource over-provisioning, energy non-proportional behavior of today's servers, and inefficient cooling systems have been identified as major contributors to the high energy consumption in DCs [START_REF] Goudarzi | Hierarchical SLA-Driven Resource Management for Peak Power-Aware and Energy-Efficient Operation of a Cloud Datacenter[END_REF].

On-site renewable energy production and geographical energy-aware load balancing of virtual machines (VM) allocation can be associated to lower the brown (i.e. not renewable) energy consumption of DCs. Yet, combining these two approaches remains challenging in current distributed Clouds. Indeed, the variable and/or intermittent behavior of most renewable sources -like solar power for instance -is not correlated with the Cloud energy consumption, that depends on physical infrastructure characteristics and fluctuating unpredictable workloads. Moreover, while literature tackles this issue, existing work often does not consider heterogeneous VMs running over several time slots, network constraints (bandwidth, latency and topology) of distributed Clouds, the energy cost of turning on/off servers and of migrating VMs, and the stochastic behavior of renewable supplies.

The main contribution of this paper is NEMESIS: a Network-aware Energy-efficient Management framework for distributEd cloudS Infrastructures with on-Site photovoltaic production. The originality of NEMESIS lies in its combination of a greedy VM allocation algorithm, a networkaware live-migration algorithm, a dichotomous consolidation algorithm and a stochastic model of the renewable energy supply in order to optimize both green and brown energy consumption of a distributed cloud infrastructure with onsite renewable production. Our solution employs a centralized resource manager to schedule VM migrations in a networkaware and energy-efficient way, and consolidation techniques distributed in each data center to optimize the Cloud's overall energy consumption.

NEMESIS has been implemented on the SimGrid toolkit [3] using real Cloud traces. This simulator embeds accurate energy consumption models and fine-grain VM and livemigration abstractions. The simulation-based evaluation shows that NEMESIS outperforms classical greedy allocations and algorithms from the literature with a realistic workload that comprises heterogeneous VMs and does not exhibit day/night patterns.

The rest of the paper is organized as follows. Section II presents related work. Section III details the employed models and Section IV describes our NEMESIS approach. Simulation results are presented in Section V. Section VI concludes this work and introduces future work.

II. RELATED WORK

As their increasing electricity bill raises environmental issues, Cloud providers resort more and more to renewable energy sources [START_REF]Clicking Green: who is winning the race to build a green Internet[END_REF]. On one hand, the renewable production can be adjusted through the use of energy storage devices. Yet, this solution is costly and far from ideal as these devices present charge and discharge maximal rates, depth of discharge lower bounds,as well as strong aging effects and non-negligible energy losses ratios. Furthermore, green energy availability highly depends on the data centers' location that is fixed upon construction.

On the other hand, Cloud providers can try to adjust the workload to the energy production. For instance, opportunistic scheduling aims at postponing Cloud's workload during lowproduction periods to wait for renewable energy availability [START_REF] Chen | Green-aware workload scheduling in geographically distributed data centers[END_REF]. Distributed Cloud systems can perform geographic load balancing and follow-the-sun resource management in order to increase the green energy use [START_REF] Rahman | A Survey on Geographic Load Balancing Based Data Center Power Management in the Smart Grid Environment[END_REF]. It consists in scheduling the virtual machine allocations to data centers where renewable energy is available. Such techniques can also be combined with consolidation algorithms that optimize the number of used resources by migrating virtual machines (VM) and switch off unused resources [START_REF] Dabbagh | Toward energyefficient cloud computing: Prediction, consolidation, and overcommitment[END_REF]. Yet, Cloud data centers are required to provide high availability to their customers, and consequently, parts of the workload cannot be reshaped.

While follow-the-sun and consolidation techniques can save energy in distributed Cloud infrastructures, existing frameworks often do not consider network constraints [START_REF] Rahman | A Survey on Geographic Load Balancing Based Data Center Power Management in the Smart Grid Environment[END_REF]. Indeed, as Cloud traffic demands diversify, network resources, inside and in-between the data centers, are often stretched to their limits and, in many cases, become a performance bottleneck [START_REF] Lin | EAFR: An Energy-Efficient Adaptive File Replication System in Data-Intensive Clusters[END_REF]. If not carefully taken into account, network can be a major issue for energy-efficient resource management techniques and it can make them unfeasible in practice.

III. MODEL

In this section, we describe the considered Cloud model and its underlying assumptions. We consider a distributed Cloud infrastructure comprising several data centers (DC) geographically distributed and connected through an IP telecommunication network. The DCs host a given amount of servers which are homogeneous in terms of memory, number of cores and energy consumption. On the contrary to other works in the literature, we take into account the complete IP network topology, i.e. the different links (with their latency and bandwidth constraints) and their topology.

Each DC produces its own green energy thanks to photovoltaic panels (PV). The energy production of the PV is not known in advance as it strongly depends on the meteorological context of each DC. When the local green energy production of a DC is not sufficient, the traditional, regular electrical grid is in charge of powering the Cloud. Following the worst case scenario, all the supply coming from the regular grid is considered as brown energy.

The user management of the Cloud is assumed to be centralized. At each time slot of 5min, heterogeneous VMs (in terms of memory, CPU and execution time requirements) are submitted to the Cloud. They will run for a given period of time a priori known. The Cloud manager is free to locate a VM on any server with enough resources to run the VM (i.e. no over-commitment). The VM location can be changed at runtime by using live migrations.

We consider four states for a server: POWERING ON, POWERING OFF, ON and OFF. Before being ON (resp. OFF), a server remains in the POWERING ON (resp. POWERING OFF) state for a given fixed duration. When the servers are ON, we assume their power consumption to change linearly with CPU usage [START_REF] Li | Opportunistic Scheduling in Clouds Partially Powered by Green Energy[END_REF]. Each of the three other states is associated with a static power consumption.

The power consumption of a server is the sum of a fixed part P idle and a second part proportional to the number n c of running CPUs n c × P c . It consumes electric power P ON when turning on, and P OF F when turning off. The number of servers on, at time slot t, on data center DC i , is denoted U i (t). Thus the static power of DC i cost is U i (t)×P idle . The migration of a VM during t will consume an average power of P c × t M 5min during the full duration of t (i.e. 5min). Note that we assume here that VM migrations are shorter than the duration of a timeslot (the algorithms we propose in the following section guarantee this point). Finally, we count the switch ON/OFF of servers:

U + i (t) × P ON + U - i (t) × P OF F , where U + i (t) = max(0, U i (t) -U i (t -1)) and U - i (t) = max(0, U i (t -1) - U i (t)).
In order to determine the migration time of a VM, we consider the live migration model of [START_REF] Velho | On the Validity of Flow-level TCP Network Models for Grid and Cloud Simulations[END_REF]. The live migration of a VM from a server A to a server B comprises several steps: reservation, iterative pre-copy, stop-and-copy, commitment and activation. In our migration model, we consider the execution time of steps reservation and activation to be negligible. For sake of simplicity, we also average the duration of dirty-pages iterative transfers as this highly depends on the application running on the VM. Consequently, the migration of a VM consists in (1) transferring all the pages, (2) sending a message to notify the end of the stop-and-copy step, and (3) sending the commitment message. Hence, following the TCP/IP model of [START_REF] Velho | On the Validity of Flow-level TCP Network Models for Grid and Cloud Simulations[END_REF], the time T m to migrate a VM is equal to: T m = 3L m + Sm /ρm with S m the memory size of the VM, L m the latency of the route used for the transfer (equals to 13.01 times the sum of the physical latency of the links [START_REF] Velho | On the Validity of Flow-level TCP Network Models for Grid and Cloud Simulations[END_REF]), and ρ m the throughput of the transfer. The transfer throughput can be limited by the physical bandwidth or the latency of the route. Therefore, it is equal to: ρ m = min (0.92B m , W /2Lm) with B m the physical bandwidth and W = 4194304B, the TCP maximum window size. Please, note that [START_REF] Velho | On the Validity of Flow-level TCP Network Models for Grid and Cloud Simulations[END_REF] determined the values of the different constants of the model with rigorous experiments made on real TCP network. We consider that the migration requires a full core on the destination server. Thus during the migration, as the source server still runs the VM, both the source and the destination servers consume energy.

IV. OUR APPROACH

Our approach consists in a centralized Algorithm 1, NEME-SIS, that operates four main steps. In the first step, further detailed in Section IV-B, Algorithm 2 allocates waiting VMs to servers on DCs according to the availability of servers and to the on-site green electricity production. This algorithm mainly focuses on allocating the VMs on the least possible number of servers. Then, Algorithm 3 revisits decisions of Algorithm 2. (Algorithm 3) Migrate running VMs;

(Algorithm 5)

for 1 ≤ i ≤ M do Consolidate DC i (Algorithm 6)
Then, as detailed in Section IV-C, Algorithm 5 determines migration of running VMs between data centers. Finally in Section IV-D, on each DC, Algorithm 6 determines if some servers can be turned off after intra DC migration.

A. Expected brown consumption computation

We first explain how to compute and forecast the brown consumption of the Cloud. Using our server model, we can compute the power consumption P of a DC. This electricity is provided by both local PVs, and by brown electricity. Similarly to [START_REF] Camus | A stochastic approach for optimizing green energy consumption in distributed clouds[END_REF], the PV production is modeled by a normal law N (Eg i (t), p i (t)) truncated in 0. This model is based on the common assumption that the forecast error of PV production follows a normal distribution [START_REF] Lorenz | Irradiance forecasting for the power prediction of grid-connected photovoltaic systems[END_REF]. Then, for any time slot, the expected brown consumption is:

Ec i (P, t) = (P -Eg i (t)) Φ(P ) -Φ(0) 1 -Φ(0) -p i (t) 2 φ(0) -φ(P ) 1 -Φ(0) , with φ(x) = 1 pi(t) √ 2π e -1 2 x-Eg i (t) p i (t) 2
, and

Φ(x) = 1 2 1 + erf x-Egi(t) pi(t) √ 2
. Details of computations are given in [START_REF] Camus | A stochastic approach for optimizing green energy consumption in distributed clouds[END_REF]. Concerning the expected remaining green energy, we obtain the following result. The expected green power used is:

EgU i (t) = P -Ec i (P, t) = P -(P -Eg i (t)) Φ(P )-Φ(0) 1-Φ(0) + p i (t) 2 φ(0)-φ(P ) 1-Φ(0)
and the expected green power production is

EgP i (t) = Eg i (t) + p i (t) 2 φ(0) /1-Φ(0)
Then, the expected remaining green power is:

EgR i (t) = EgP i (t) -EgU i (t) = Eg i (t) -P + (P -Eg i (t)) Φ(P )-Φ(0) 1-Φ(0) + p i (t) 2 φ(0) 1-Φ(0) -p i (t) 2 φ(0)-φ(P ) 1-Φ(0) = (Eg i (t) -P ) 1-Φ(P ) 1-Φ(0) + p i (t) 2 φ(P ) 1-Φ(0)
These formulas permit to compute the expected brown energy consumption and the remaining green power for the next time slot only. However, when allocating VMs, the brown energy consumption needs to be evaluated over the whole execution time. Nevertheless, evaluation on all time slots could be too expensive to compute for very long VMs. We thus define a parameter n eval . This parameter corresponds to the number of time slots where our algorithm evaluates the expected power consumption for a given computation. For example for a VM v of execution time t e , we will evaluate the brown power consumed at current time slot t and then at n eval -1 other time slots to evaluate the global brown extra cost related to this VM, for a given DC. We limit this evaluation to the current day light. This evaluation does not make sense during night as PV are not producing energy, and our forecast model does not predict the next day weather. We define t max = min(t sunset , t + t e ) as the last time slot for our evaluation, and t k = t + k × tmax-t n eval , for k between 0 and n eval -1. Evaluations over less than n eval time slots are done over all time slots, for example if t e ≤ n eval .

Thus, if an evaluation over many time slots is requested at time slot t, then the algorithm will compute the brown power consumption for many time slots. The evaluation for the next time slot has been detailed above. For further time slots, the power consumption of the DC will depend on tasks allocation and migrations done in between. However, the computation of the requested power is done based only on currently allocated (or pre-allocated) VMs, as if no more VMs would be submitted any more. Considering VMs finishing at previous time slot t k -1 and still running at time slot t k , the power consumption is computed, and the expected power consumption is determined.

B. Allocation

In the following, we say that a server is incomplete if it is not yet fully loaded (without considering overcommit). Algorithm 2 pre-allocates all the incoming new VMs at current time slot. These pre-allocations are then revisited by Algorithm 3.

min 1≤i≤M Ec i (P i + P idle + c L[0] * P c + E ON ) -EP i allocate l on s
Each VM is characterized by its execution time, its CPU requirement and its memory usage. A given server can run a VM only if CPU and memory requirements of the VM are satisfied by this server. We define the volume of a VM as the product of these two values.

Algorithm 2 executes a Best-Fit algorithm based on the volume. More precisely, it orders VMs by decreasing volume and servers of all DCs by increasing free volume, and allocates VMs one by one on the server of minimum free volume that can execute it. If many servers with same free volume can execute the VM, the server with minimum expected brown consumption cost is selected. As this expected cost is the same for two servers of the same DC (homogeneous servers), this implies to select the best location among DCs with servers of minimum free volume. This is especially the case when no incomplete server can run the VM and a new server needs to be turned on.

The complexity of this greedy algorithm is in worst case O(n w × n S + n w log(n w ) + n S log(n S )) with n w the number of waiting VMs and n S the number of servers over all DCs. It corresponds to a worst case where, for all waiting VMs and all server on, the free volume of the server is higher than the volume of the task but it cannot run it.

The second allocation step is executed by Algorithm 3. This second pass permit to lowered the possible negative impact of the Best-Fit algorithm that mostly focused on volume issues. . More precisely, it evaluates the expected green energy not used before the night. This step permits to determine the DCs that will send VMs in this process and those that will receive new ones. VMs pre-allocated on DCs with lower ERGE will be moved to DCs with higher ERGE. The evaluation of ERGE for each DC is done at maximum over n eval time slots, as detailed in Subsection IV-A. In increasing order of ERGE, DCs consider the possibility to move their pre-allocated VMs to greener DCs by decreasing order, and execute a Best-Fit algorithm. Algorithm 3 focuses on VMs allocated on incomplete servers, that is, servers that could be profitably turned off in the consolidation process. Each VM is moved if it reduces the Cloud's overall brown energy consumption, based on our evaluation.

Algorithm 3 has a complexity in O( N pa i log(N pa i )) with N pa i the size of Lt i . Its complexity is thus of lower order than the complexity of Algorithm 2.

C. VM migration

The following step of NEMESIS determines VMs migrations inter DCs. This step consists in moving running VMs to allocate them on other DCs with the objective to reduce the brown energy consumption.

Due to bandwidth constraints, a DC can only migrate VMs one by one. Thus, the amount of VMs a DC can send in a single time slot is bounded by the sum of migration times. In addition, both DCs have to be synchronized during the migration time. Thus, if many DCs send VMs to the same DC, they can not communicate during the same time.

Algorithm 5 first lists the VMs to migrate for each DC, denoted List to migrate i . For this step, we first order VMs running on DC i by decreasing remaining execution time. The first VMs are added to the list until the sum of migration times reaches the duration of the time slot. As migration are done one by one, there is no need to consider more VMs for migration. As in Algorithm 3, the DCs with lower ERGE will send VMs to DCs with higher ERGE. For given receiving and sending DCs, VMs are ordered by decreasing volume and allocated one by one if some servers can host them, if the expected brown power consumption is reduced and if the migration time constraints are fulfilled as illustrated in the example of Figure 1. Then, if some VMs remain to migrate, DC 1 sends VMs to DC M -1 . In the following, if last migration from DC i is done to DC j , then VMs from DC i+1 will be migrated first to DC j , and then to DC j-1 . Algorithm 4: Compute t limit and t send (for Algorithm 5)

t limit = t send ; f orward = 1 -f orward; if f orward = 1 then t send = 0; else t send = ∆t;
For given receiving and sending DCs, the time interval is determined by variables t limit and f orward. If f orward equals 1, tasks migrations are scheduled between time 0 and t limit , and between t limit and ∆t in the other case. Then, t send corresponds to the bound for the next task migration. If f orward equals 1, first task migration occurs at time 0 and t send is the time of the end of the last scheduled migration.

If f orward equals 0, we start at the end of the time slot. The first allocation of a migration is scheduled to finish at time ∆t and t send corresponds to the beginning of the last scheduled migration.

Algorithm 5: Migration algorithm if n receive = 2 then 20: DC receiving --;n receive = 0; Finally, Algorithm 5 is designed as follows. It first orders DCs by increasing ERGE, and creates a list of VMs to migrate. Then, it considers DCs one by one in increasing order of ERGE (from the minimum green energy available to the maximum). For each DC, it considers a first DC where to send VMs. Then, in decreasing order of VM volume, it determines if the migration is possible (if a server can run it and if migration time is available) and preferable (in terms of brown power consumption). After all VMs of the list are examined, the same is done with a second receiving DC. This is done until all DCs have been considered for either sending or receiving. Algorithm 5 has a complexity in O( N r i log(N r i )) with N r i the number VMs running on DC i .

D. Consolidation

In the consolidation algorithm, Algorithm 6, the objective is to turn off the maximum possible servers through migrations intra DC. This algorithm executes independently on each DC. It first orders servers by decreasing empty volume. It then tests if the first servers can be turned off. In order to determine the number of servers to turn on, it executes a dichotomy on the number of servers. Then for any number of servers, it selects all VMs on these first servers, and tries to allocate it by a Best-Fit algorithm on the remaining servers. For |L| incomplete servers, it first tries to turn off k = |L|/2 servers. Then, k is increased if a solution exists and decreased in the other case. The algorithm continues until the optimal decision is decided.

s i ) + log(N s i )(N r i log(N r i ) + N r i N s i )) with N s i the number of incomplete servers of DC i . Finally, NEMESIS has a simplified coarse-grain complexity in O(n S × n V M log(n w × n V M ) with n V M
the number of running and waiting VMs at current time slot, and n S the number of servers on. As shown in this section, this value is a large overestimate of the real complexity.

V. VALIDATION

A. Experimental setup

We evaluate the performance of NEMESIS through a modeling an simulation process. We use the SimGrid simulation framework [START_REF] Casanova | Versatile, scalable, and accurate simulation of distributed applications and platforms[END_REF] to perform our experiments with real-world IaaS Cloud and PV traces. SimGrid is a versatile platform (i.e. it can be used to study cluster, grid, peer-to-peer, Cloud, wide/local-area networks, . . . ) dedicated to the scalable simulations of distributed applications and platforms. It is grounded on sound simulation models (of CPUs, TCP/IP networks, VMs, and energy consumption) which are theoretically and experimentally assessed by numerous scientific works. Using these models, SimGrid accurately simulates the resources usage (i.e. CPU and bandwidth sharing), the execution time and the energy consumption of a distributed application code -NEMESIS in our case. In our experiments, we study a Cloud infrastructure with characteristics equivalent to the French experimental testbed Grid'5000 [START_REF]Grid'5000[END_REF]. Thus, our Cloud comprises a total of 1,035 servers spread across 9 geographically distributed DCs. The DCs are linked together thanks to 10 Gigabit IP links. Inside each DC, the servers use 1 Gigabit Ethernet links.

The characteristics of each server are based on the Taurus servers of Grid'5000. These servers are equipped with 2 Intel Xeon E5-2630 CPU with 6 cores each, 32GB memory, 598GB storage. To determine the power consumption of each node, we implement the power model of [START_REF] Li | Opportunistic Scheduling in Clouds Partially Powered by Green Energy[END_REF] that is based on real measurements made on Taurus nodes. These measurements notably state that a Taurus server consumes 8W when powered OFF, 97W when idle, and 220W at 100% CPU load. Moreover, with this model a server consumes 200W during 10 seconds when powering off, and 127W during 150 seconds when powering on.

We use the Eucalyptus IaaS Cloud traces of [START_REF] Wolski | Using parametric models to represent private cloud workloads[END_REF] that combines the traces of six different real production systems. It consists in the list of VM arrival times, required numbers of cores and execution lengths. We assume all the VMs to require 2Go of RAM per required core (following the typical characteristics of T2 VM instances of Amazon EC2 [START_REF]Amazon ec2 instances[END_REF]). This workload trace spans over one week. We scale the workload to 80% of the total computing capacity of the Cloud. For the VMs to be always deployable in our Cloud, we limit the maximum number of cores a VM can requires to 6. The simulated workload is not a favorable scenario for on-site PV production as it does not follow a day/night pattern with reduced workload during nights when green production is null. Such an unfavorable workload is typical of IaaS Clouds [START_REF] Wolski | Using parametric models to represent private cloud workloads[END_REF] and thus, represents a realistic scenario. As these Cloud traces do not contain the CPU utilization over time of each VM, we assume that during its entire duration, each VM uses its allocated CPU resources at maximum. This constitutes the least favorable case from an energy point-of-view as it is the most consuming scenario.

We use real recordings of green power production collected by the Photovolta project [START_REF]Photovolta project[END_REF] of the University of Nantes. These data correspond to the power produced by a single Sanyo HIP-240-HDE4 PV panel (with a total power of 5,52 Wc and a surface of 5.52m 2 ) updated every five minutes over one week. In order to have heterogeneous trajectories between DCs (and thus to represent solar irradiance differences between sites spread across a country), we select recordings starting at different dates. For each DC, the number of PV panels is dimensioned to have one PV for 3 servers and PV signals are scaled accordingly.

The controller implementing NEMESIS runs at each time slot of five minutes. It records all the amounts received from the green power sources during the current day. The controller computes at each time slot the standard deviations p i (t) using this history. Following our approach in [START_REF] Camus | A stochastic approach for optimizing green energy consumption in distributed clouds[END_REF], our implementation of NEMESIS computes each expected green power production Eg i (t) by averaging a reference green power production trajectory (the Photovolta project recording of the 20th August 2013 in our case which is the day with the best yield) scaled according to the last green power production received from i.

We compare NEMESIS performance against four approaches of the literature:

• Round-Robin distributes the VMs fairly among the DCs regardless of their green power production;

• First-Fit deploys each VM on the first (according to an arbitrary predefined order) DC which can host it. • Modified Best Fit Decreasing (MBFD) [START_REF] Beloglazov | Energy-aware resource allocation heuristics for efficient management of data centers for Cloud computing[END_REF] is a reference approach for reducing power consumption in Clouds. It relies on a decreasing best-fit algorithm to allocate incoming VMs and to perform consolidation of running VMs. The consolidation consists in performing live migrations of VMs that run on underused servers (i.e. servers that have less than 50% of CPU used) and shutting down these servers to save energy. On the contrary to NEMESIS, MBFD does not take into account the network constraints and the local green power productions. The remaining execution time of the VMs and the energy cost of the live migration are also not considered when migrating a VM. • OOD-MARE [START_REF] Khosravi | Online virtual machine migration for renewable energy usage maximization in geographically distributed cloud data centers[END_REF] is another approach from literature consisting in allocating the incoming and running VMs according to the current local green power productions. With this approach, a Most Available Renewable Energy (MARE) algorithm deploys the VMs on the DC that has the highest amount of available green energy. According to an Optimal Online Deterministic (OOD) policy, the running VMs that start consuming brown energy are sequentially re-allocated using live migrations. On the contrary to NEMESIS, OOD-MARE does not rely on green production forecasts and does not perform intra-DC consolidation. It also does not consider the VM remaining execution time when performing live migration. Finally, the energy cost of these live migrations is neglected. Like NEMESIS, these algorithms firstly employ servers already ON, and switch OFF those who are idle. As the performance of First-Fit and MBFD strongly depends on the order of the DCs, we test two opposite configurations corresponding to the best and the worst possible contexts. To define these contexts, we sort DCs according to the total amount of on-site green energy they provide. The best (resp. worst) context corresponds to the case where the PV traces are sorted in a decreasing (resp. increasing) order.

B. Results analysis

We simulate the Cloud behavior over one week. Table I shows the total cumulative energy consumption. We can see that NEMESIS consumes significantly less brown energy than the other approaches: at least 3.18%, and 13.26% maximum. It also slightly reduces the overall (i.e. brown and green) energy consumption.

As they do not take account of the local energy productions, First-Fit and MBFD have very variable performance depending on the scenarios considered : between the worst and the best scenarios the brown energy consumption differs of around 1 MWh. It is worth noting that MBFD has the highest overall energy consumption despite being designed to save energy. This is caused by its consolidation strategy that performs too many VM live migrations and neglects their energy cost. Thus, the energy consumed by the live migrations is more important than the energy saved by powering OFF the servers. Moreover, as MBFD does not take into account network constraints, it performs too many simultaneous migrations. These migrations compete for the network bandwidth, and take then significantly longer to terminate. As a consequence, the energy costs due to migrations is increased. Figure 2 shows the simulated power consumption over time of each DC in the Cloud using NEMESIS. As expected, NEMESIS uses in priority the DC that has the highest power production (e.g. Grenoble and Rennes DCs). We can also observe that the consolidations significantly lower the power consumption of the concerned DC. For instance in the DC of Rennes, a consolidation of 22 VMs occurs at time 344,100 sec and lower the consumption of about 500 W.

Figure 3 shows the contributions of the three main algorithms constituting NEMESIS: Algorithms 3 (migration of preallocated VMs), 5 (inter-DC migration of running VMs) and 6 (consolidation). We measure the improvement given by an algorithm by disabling it and checking the resulting increase in the total cumulative brown energy consumption. We can see that the energy consumption (both brown and global) is mainly reduced by live migrations of VMs to other DCs (i.e. Algorithm 5). The two other algorithms have only a limited positive impact on the energy consumption. This simulationbased validation shows promising results considering it employs a non-favorable workload for solar energy production (i.e. not exhibiting day/night patterns).

C. Comparison with a theoretical lower bound

Considering that our scheduling problem is NP-hard, an optimal off-line solution cannot be computed in reasonable time. However, to provide indirect guarantees, we compute a theoretical (i.e. unreachable) lower bound for the total and brown energy consumption. To compute this lower bound, we consider a simpler problem where all the servers and energy productions of the Cloud are grouped in a single DC. We also neglect the delays and energy cost of VM migrations. Each time a VM starts or stops, we reallocate all the running VM to the servers using a best-fit decreasing algorithm. We employ in priority the servers that are powered ON, and we switch off all the idle servers. We found a theoretical lower bound of 10.62 MWh (resp. 17.2 MWh) for the brown (resp. total) energy consumption. The brown power consumption of NEMESIS is then 9.56% above this unreachable lower bound. This experiment demonstrates that the brown power consumption of NEMESIS is less than 9.56% above the optimal solution.

VI. CONCLUSION AND FUTURE WORK

In this paper, we propose NEMESIS, an energy-efficient IaaS resource management framework to cope with distributed Cloud infrastructures with on-site PV production. As major Cloud providers resort more and more to solar energy, practical and realistic solutions are needed to optimize both the green and the brown consumption of DCs.

Our approach combines greedy resource allocation, inter DC network-aware live-migration, intra DC VM consolidation and PV production prediction algorithms. To the best of our knowledge regarding the state of the art, our work differs from the literature on the following points: 1) we deal with network constraints when performing VM migrations, 2) we rely on an original stochastic model to predict brown energy production, 3) we take into account on/off switching costs and delays.

In the simulation-based evaluation, we consider a realistic context as we use (1) real workload trace, (2) existing cloud infrastructure, (3) real photovoltaic power productions traces, and (4) an energy consumption model that is based on real wattmeter measurements. Moreover, our simulations are grounded on the SimGrid models for CPU usage, TCP/IP networks and VM live migrations that have been validated by numerous scientific works. That is why, despite the inherent limits of simulation, we can put a high confidence on our simulation results.

Despite a very unfavorable context with a workload that does not follow a day-night pattern contrary to photovoltaic power productions, the results show that significant amounts of energy can be saved with NEMESIS compared to classical allocation and consolidation algorithms from literature. The gains become more important when comparing our solution with the algorithms currently deployed on real production systems --i.e. round-robin consumes 6.9% more brown energy than our solution whereas first-fit consumes between 3.2% (in an ideal configuration) and 13.3% more brown energy. Other algorithms from the literature also perform well (i.e. OOD-MARE), but NEMESIS is able to save a bit more energy (the 1.6% and 3.2%). We also examine the contribution of each NEMESIS component to the overall energy gain.

Our future work includes dealing with heterogeneous servers and studying the scaling properties of NEMESIS by increasing the number of DCs. But, it will require to find real Cloud workload traces in order to further validate our framework and convince Cloud providers to implement it. We also considered only independent VMs in this work. We plan to extend our proposal to take account of multi-VM applications and their QoS constraints that may limit the VM deployment and migration in the Cloud.

Algorithm 1 :

 1 General algorithm Allocate new VMs;(Algorithm 2) Migrate pre-allocated VMs;

Algorithm 2 :

 2 Centralized VM allocation algorithm L = sorted list of new VMs (arriving at time t) by decreasing volume S = sorted list of incomplete servers in all DCs by increasing size of available number of vCPU for all l ∈ L do find list S opt of servers with minimum volume v find server s ∈ S opt with minimum cost:

Fig. 1 :

 1 Fig. 1: Example of VM migrations with 5 DCs To simplify the synchronizations, we limit each DC to send VMs to only two DCs during a time slot, or to receive VMs from only two DCs. For given receiving and sending DCs, we determine the VMs to migrate and the migration date. Migrations are done consecutively in the same time interval. The pair of sending and receiving DCs are determined as follow. First, worst DC DC 1 sends VMs to the best DC DC M .Then, if some VMs remain to migrate, DC 1 sends VMs to DC M -1 . In the following, if last migration from DC i is done to DC j , then VMs from DC i+1 will be migrated first to DC j , and then to DC j-1 .

Algorithm 6 :

 6 Consolidation on each DC Order list L of incomplete servers by decreasing empty volume k = |L|/2 ; i = |L|; sol = true while i >= 2 do i = i/2 if sol then k+ = i else k-= i if VMs on the k first servers of L can be allocated on the |L| -k last ones then sol = true else sol = f alse if sol=false then k --Allocate VMs on the k first servers of L on the |L| -k last ones Turn off the k first servers Algorithm 6 has a complexity in O(N s i log(N

Fig. 2 :Fig. 3 :

 23 Fig. 2: Power consumption per DC with NEMESIS.

Algorithm 3 :

 3 Allocation Revision. Order DC by increasing ERGE Let Lt i the list of pre-allocated VMs on incomplete servers of DC i for all DC sending in {1..M -1} do DC receiving = M Order Lt DC sending by decreasing volume while DC sending < DC receiving AND Lt DC sending not empty do for all VM t ∈ Lt DC sending do if a server of DC receiving can run t AND Exp Cost with M igration < EP DC sending + EP DCreceiving then Pre-allocate t on DC receiving Remove t from Lt DC sending DC

receiving --Algorithm 3 first orders DCs by expected remaining green energy (ERGE)

1 :

 1 Order DC by increasing ERGE 2: Create lists List to migrate 3: DC receiving = M ; DC sending = 1;n receive = 0 4: t send = 0; t limit = ∆t; f orward = 1; 5: while DC sending < DC receiving do

	6:	Order List to migrate DC sending by decreasing
		volume;
	7:	Compute t limit and t send ;
	8:	Determine migration decisions;
	9:	if migration decision then
	10:	n receive + +;
	11:	if List to migrate DC sending is not empty then
	12:	DC receiving --;
	13:	n receive = 0;
	14:	Compute t limit and t send ;
	15:	Determine migration decisions;
	16:	if migration decision then
	17:	n receive + +;
	18:	DC sending + +;
	19:

TABLE I :

 I Total cumulative energy consumption in the best/worst contexts (if different). The differences with NEME-SIS are in parenthesis.

	Approaches	Overall consumption	Brown Consumption
	NEMESIS	17.6 MWh	11.6 MWh
	OOD-MARE	17.9 MWh (1.6%)	12 MWh (3.2%)
	MBFD	best worst	18 MWh (1.9%)	12.2 MWh (4.47%) 13.3 MWh (13.9%)
	First-Fit	best worst	17.7 MWh (0.7%)	12 MWh (3.4%) 13.2 MWh (13.3%)
	Round-Robin	17.8 MWh (1%)	12.4 MWh (6.9%)

ACKNOWLEDGMENTS

This work has been supported by the Inria exploratory research project COSMIC (Coordinated Optimization of SMart grIds and Clouds).


Benjamin Camus
email: benjamin.camus@irisa.fr

Anne Blavette
email: anne.blavette@ens-rennes.fr

Fanny Dufossé
email: fanny.dufosse@inria.fr

Anne-Cécile Orgerie
email: anne-cecile.orgerie@irisa.fr

Harnessing the geographical flexibility of distributed computing clouds for cooperative self-consumption

Keywords: cooperative self-consumption, spatial flexibility, computing clouds

Smart grids allow to efficiently perform demandside management in electrical grids in order to increase the integration of fluctuating and/or intermittent renewable energy sources in the energy mix. In this paper, we consider a distributed computing cloud partially powered by photovoltaic panels as a self-consumer that can also benefit from geographical flexibility: the computing load can be moved from one data center to another one benefiting from better solar irradiance conditions. The various data centers composing the cloud can then cooperate to better synchronise their consumption with their photovoltaic production. The simulation-based results show that our approach can significantly increase the self-consumption of renewable energy and improve the state-of-the-art management techniques for energy-efficient distributed clouds.

I. INTRODUCTION

Demand-side management is key for increasing the level of renewables in the energy mix. This concept implies modifying the consumer's energy consumption by means of different incentives (e.g. economic) with respect to the fluctuations of the renewable electricity production in order to maintain the safe and stable operation of the power system. So far, a number of flexible load types has been extensively studied for demand-side management applications, such as controllable water heaters, air-conditioning equipment, refrigerators, etc. as well as electric vehicles. Traditionally, flexible loads are characterized by a certain degree of temporal flexibility. This means that their energy consumption can be reduced partially or entirely (i.e. interruptible loads) during a certain amount of time, usually with a rebound effect, or that their activation can be postponed (i.e. deferrable loads).

However, it may also be interesting to consider spatial flexibility. Spatial flexibility can be defined as the ability of a load to migrate physically from one node in the electrical network to another in a sufficiently short amount of time to be relevant for demand-side management. Spatial flexibility may be used for addressing local network issues, such as line congestion or voltage control. It is important to note that one approach to solve the mentioned local network issues consists in curtailing the renewable electricity generation, thus losing energy. Hence, spatial flexibility can represent an alternative to curtailment. Another advantage of spatial flexibility consists in accompanying the spatial fluctuations inherent to variable renewable electricity generation from wind or photovoltaic sources. In other words, such loads could be expected to migrate from one region to another of the power system if the electricity generation from renewables becomes more important in the latter than in the former. This represents a relevant manner to further harness renewables, in complementarity with temporal flexibility.

Spatial flexibility can be provided by loads such as distributed computing clouds. Clouds include a large number of computing servers (gathered in data centers) which can be virtualized, thus offering on-demand virtual machines (VMs) to end-users that are adapted to their needs in terms of computing power. These VMs can be deployed in any data centers, and can be displaced from one to another, which is usually referred to as "VM migration". In particular, these VMs can migrate from a data center where renewable electricity generation is becoming scarce to another with more favorable conditions.

Hence, spatial flexibility can represent an interesting approach for cloud managers, as they are becoming more and more aware of the increasing financial and environmental impact of their data centers ever growing energy consumption [START_REF] Shehabi | United States Data Center Energy Usage Report[END_REF]. Hence, more and more data centers are being equipped with photovoltaic (PV) panels in the perspective of self-consumption. VM migration can thus allow to increase the cloud self-consumption, while not impacting the quality of service provided to the cloud customers.

Although a large number of studies is available on VM migration for better harnessing renewables [START_REF] Khosravi | Online virtual machine migration for renewable energy usage maximization in geographically distributed cloud data centers[END_REF], [START_REF] Dandres | Consideration of marginal electricity in real-time minimization of distributed data centre emissions[END_REF], none has compared this approach with the alternative which consists in exchanging photovoltaic energy between the data centers through the electrical network.

Considering that data centers are supposed to be relatively distant from one another, in order to harness renewables spatial diversity, the energy exchange between them is virtual rather than physical. In other words, it is considered that the data centers can exchange energy as part of a collective selfconsumption scheme.

Collective self-consumption policies and decrees have been recently adopted or are under development in particular in Europe [START_REF]Promotion of the use of energy from renewable source[END_REF], and especially in countries such as France [START_REF] Légifrance | French Republic Official Journal[END_REF] and Germany [START_REF]Mieterstrom: Energiewende im eigenen Haus[END_REF]. This concept consists in allowing multiple consumers and renewable energy producers, all located in a small geographical area, to establish power purchase agreements (or similar schemes) between themselves. Legislation differs between countries, in particular regarding the definition of the geographical area which can span from a single site (e.g. a building equipped with PV panels on its roof) to a wider, but still local, geographical area where consumers and producers are connected to the same low-voltage feeder. The collective self-consumption scheme proposed by the authors in this paper is inspired from existing or under development policies, but it represents an extended, multi-site version. This version is deemed to require the establishment of only relatively simple measures, facing no deep technical challenges, and able to unlock the potential of spatial flexibility inherent to distributed computing clouds.

However, it is important to take into account that absorbing energy from, or injecting energy into, the grid leads to Joule losses, transmission and distribution equipment aging, etc. as well as grid operator management costs. This impact can be taken into account as a cost based on the network use tariff. This tariff is usually calculated based on the amount of energy and power transmitted by, or to, a grid-connected installation through the grid. Hence, self-consuming the PV electricity generated locally, i.e. not absorbed from, or injected into, the grid, reduces this cost. On the contrary, energy exchange between data centers, or between data centers and other customers or power plants increases this cost. On the other hand, migrating a VM requires an additional amount of energy (compared to exchanging energy through the electrical network), which also has a cost, as it will be detailed in Section II. Hence, this paper will focus on the comparison in terms of costs between the VM migration and the energy exchange approaches, and a combination of these.

Section II provides some background on Cloud computing. The hypotheses, models and the self-consumption optimization algorithm are detailed in Section III. Finally, the results will be described in Section IV, and future work is presented in Section V.

II. BACKGROUND ON CLOUD COMPUTING

Distributed cloud infrastructures consist of several data centers linked through dedicated telecommunication networks. Each data center comprises servers hosting virtual machines that run end-user applications, such as online services (e.g. emailing, file sharing). Upon creation, a virtual machine (VM) is allocated with virtual resources: memory (RAM), computing power (CPU) and disks. These allocated virtual resources are mapped to the physical resources of the server hosting the VM. Depending on its physical characteristics, a given server can host several VMs at the same time. Using its computing resources increases the power consumption of the server. Yet, servers also consume power while being idle but power on. This idle power consumption can reach 50% of the maximal power consumption of recent servers, thus justifying energy saving policies based on server shutdown techniques [START_REF] Heinrich | Predicting the Energy-Consumption of MPI Applications at Scale Using Only a Single Node[END_REF]. These policies consist in consolidating VMs, i.e. in gathering them on a fewer number of servers, so that a reduced number of the latter remains on, while the unused servers are switched off. Displacing virtually one VM from one hosting server to another is usually referred to as "migration", as mentioned earlier [START_REF] Li | Energy-efficient migration and consolidation algorithm of virtual machines in data centers for cloud computing[END_REF]. However, these operations -switching on/off servers and migrating VMs -consume time and energy. Cloud managers should thus carefully optimize their use of these operations as well as the initial allocation of VM onto servers. As the electricity bill of data centers keeps increasing with the success of Internet-based services, Cloud providers resort more and more to on-site renewable energy sources [START_REF]Clicking Green: who is winning the race to build a green Internet[END_REF], [START_REF] Chen | Emaas: Cloud-based energy management service for distributed renewable energy integration[END_REF].

III. HYPOTHESES, MODELS AND METHODOLOGY

In this section, we introduce the considered hypotheses and models for the flexible consumer (distributed cloud) and the smart grid. Then, we describe our methodology to optimize self-consumption in a cooperative manner between the cloud data centers.

A. Consumer model: distributed cloud 1) Cloud workload: Clouds are supplying computing power by the means of VMs allocated upon request to end-users at any time. When receiving a user request, the cloud manager decides to which data center (DC) and to which server the VM will be allocated.

We use the Eucalyptus Cloud traces of [START_REF] Wolski | Using parametric models to represent private cloud workloads[END_REF]. This workload combines the traces of six different real cloud systems. It includes the list of VM arrival times, required numbers of cores (i.e. virtual CPU allocated to VMs) and execution lengths. This workload is not a favorable scenario with respect to photovoltaic self-consumption, as it does not follow a day/night pattern, that is with a reduced workload during night time when photovoltaic electricity generation is null. Such an unfavorable workload is typical of Infrastructure-as-a-Service Clouds [START_REF] Wolski | Using parametric models to represent private cloud workloads[END_REF], thus representing a realistic scenario.

2) Telecommunication network: Once a VM is allocated to a server, it can be re-allocated to another server (in the same DC or not) during its lifetime without service interruption by using VM live-migration mechanisms [START_REF] Li | Energy-efficient migration and consolidation algorithm of virtual machines in data centers for cloud computing[END_REF]. Such a migration requires using the telecommunication network linking the two concerned servers (host and destination). The bandwidth capacities of telecommunication links limit the number of VM migrations that can be executed simultaneously, thus limiting the spatial flexibility of the cloud load.

In order to use a realistic distributed cloud infrastructure, in our experiments, we use the characteristics of the French experimental testbed Grid'5000 [START_REF]Grid'5000[END_REF]. Thus, the studied cloud comprises a total of 1,035 servers spread across 9 DCs in Luxembourg and 8 different cities in France (Grenoble, Lille, Lyon, Nancy, Nantes, Rennes, Sophia and Toulouse). The DCs are linked together thanks to 10 GBps Ethernet links. Inside each DC, the servers use 1GBps Ethernet links.

3) Server power model: The characteristics of each server are based on the Taurus servers of Grid'5000. Each server is equipped with 2 Intel Xeon E5-2630 CPU with 12 cores in total, 32GB memory and 598GB storage. In order to determine the power consumption of each node, we implement the power model of [START_REF] Li | Opportunistic Scheduling in Clouds Partially Powered by Green Energy[END_REF] that is based on real measurements made on Taurus nodes. These measurements notably state that a Taurus server consumes 8W when powered off, 97W when idle, and 220W at 100% CPU load. Moreover, this model provides also the energy consumption and duration corresponding to powering off and powering on a server (average values over 10 real measurements). These values are used in our simulations as well as the relation between CPU load and power consumption for non-idle servers. Similarly, we use the migration model provided in [START_REF] Velho | On the Validity of Flow-level TCP Network Models for Grid and Cloud Simulations[END_REF] to determine for each VM how much time it takes to be migrated, and consequently the energy consumption on both servers (the one initially hosting the VM and the one receiving the VM after migration).

4) On-site electricity generation: Each DC produces its own renewable electricity thanks to photovoltaic (PV) panels. It is important to note that the PV electricity generation is considered as stochastic, as it can only be predicted with a certain margin of unknown error. Also, the PV electricity generation is highly dependent on the meteorological conditions at each DC. The PV production forecast is modeled as a normal law truncated in 0 in order to compute the expected production and to optimize VMs allocation. When the local renewable electricity generation is insufficient with respect to the DC consumption, the additional energy provider is in charge of powering the DC.

We use real recordings of PV panels power profiles collected as part of the Photovolta project [START_REF]Photovolta project[END_REF] carried out at the University of Nantes, France. The data considered here corresponds to the power produced by 4 Sanyo HIP-240-HDE4 PV panels at a time resolution of five minutes over one week. In order to have heterogeneous trajectories between DCs, we select recordings starting at different dates. Thus, using these real traces we take account of the temporal and spatial variations of the solar irradiance, and therefore of the intermittent nature of PV panels electrical production. For each DC, the number of PV panels is dimensioned to have one PV panel for 3 servers and PV signals are scaled accordingly.

5) Cloud manager: The centralized cloud manager deals with incoming VM requests by end-users and controls the servers in each DC. It is responsible for VM allocation and migration decisions, and for switching on/off servers. While VMs can be submitted by users at any time, for optimization purpose, our allocation algorithm (detailed in Section III-C) is executed periodically. We choose a time step of 5 minutes as photovoltaic traces are provided with this granularity. It means that every 5 minutes, the cloud manager launches the VM allocation process and deploys the new VMs requested during the previous 5 minutes.

B. Smart grid model

The cloud acts as an aggregator of self-consuming data centers which are supplied either by themselves or by another energy supplier, if necessary, as allowed by collective selfconsumption rules.

A data center is considered to consume its local PV electricity in priority. Then, it injects the excess of PV electricity into the electrical network, if any. Otherwise, it consumes electricity from the electrical network. This electricity will be considered as exchanged from another data center if one or more data centers inject PV electricity during the same time slot. Otherwise, it will be considered as supplied by the additional energy supplier. The sum of the PV electricity injections into the grid and exchanged with other data centers is considered to constitute a "virtual energy pool" into which data centers in deficit of such electricity can tap. In the absence of energy storage equipment, the remaining amount of PV electricity, if any, is deemed to be sold and consumed by other electricity customers during this time interval. Time slots ∆t g of 10 minutes will be considered. This duration corresponds to the smallest temporal resolution of load curves which is provided to customers connected at the distribution level by smart meters in France [START_REF] Boillot | Advanced Smartgrids for Distribution System Operators[END_REF]. The amount of energy exchanged between data centers through the virtual pool is thus defined as follows:

E pool (t g ) = min i (max(0, E P Vi (t g ) -E DCi (t g ))) i (max(0, E DCi (t g ) -E P Vi (t g )))
where E P Vi (t g ) is the PV generation on DC i at time slot t g , and E DCi (t g ) the local energy consumption. The local self-consumption ratio R L SC (t g ) over each time slot ∆t g is calculated as the ratio of the PV energy consumed locally by DCs by the total amount of energy consumed by the cloud, such as:

R L SC = tg i min(E P Vi (t g ), E DCi (t g )) tg i (E DCi (t g ))
Similarly, the cooperative self-consumption ratio is the ratio of PV energy consumed by the cloud, including the virtual pool, by the total amount of consumed energy:

R C SC =
tg (E pool (t g ) + i min(E P Vi (t g ), E DCi (t g )))

tg i (E DCi (t g )) 1) Energy cost: The energy cost C E is composed of a fixed and of a variable share. The fixed share C E,f consists in standing charges which have been estimated to 45.79e/month, based on data available in the case of a small industrial customer powered by French electricity supplier EDF [START_REF] Edf | Official website[END_REF]. The variable share C E,v of the energy cost (in e/kWh) was estimated based on available data on the kWh unit price paid by residential customers in France, in the absence of data specific to data centers. A fixed electricity price has been considered for the electricity purchased from the additional energy supplier, as proposed by many energy suppliers [START_REF]Fixed price electricity contracts[END_REF], [START_REF] Ovo Energy | Official website[END_REF], contrary to a variable price indexed on the energy market prices. The kWh unit price considered in this paper is equal to 0.15e/kWh.

However, a data center may also present an excess of PV electricity which it injects into the grid. This energy can either be exchanged with other data centers through the virtual pool or not. In the first case, the energy exchange is considered to be realised at no cost for the cloud. In the latter case, the PV electricity excess is considered to be sold at a tariff of 0.06 e/kWh, as described in French decree of May, 9 than 100 kWp [START_REF] Légifrance | French Republic Official Journal[END_REF], as it is the case here. These rules can be expressed as:

C E = C E,f + C E,v
where the variable share C E,v , corresponding to the amount of energy ∆E, can be expressed as:

C E,v =    -0.06∆E if ∆E is injected into the grid 0.15∆E if ∆E is purchased 0 if ∆E is exchanged between DCs
The cost C E is calculated over each time slot of 10 minutes.

2) Network use tariff: As mentioned earlier, power transmission in the electrical network leads to Joule losses, equipment aging, etc. as well as grid operator management costs. This impact can be taken into account as a cost represented by the network use tariff. This cost is paid to grid operators (TSOs and DSOs) by energy suppliers. However, it is eventually paid by energy customers as their bills include a contribution to this charge. Hence, in this paper, the network use tariff was considered to be included in the energy cost when this energy is not exchanged between data centers. However, it is necessary to define this network use cost when it is the case.

This cost was estimated based on the French network use tariff called TURPE (standing for "Public Electrical Network Use Tariff") [START_REF]Turpe 5 HBA/BT -Tarifs d'utilisation des réseaux publics de distribution d'électricité[END_REF]. This tariff follows four main rules: it is independent of the location of the grid-connected installations and of the distance between the electricity injection and absorption points. However, it depends on the contract power and on the amount of energy consumed. It is important to re-emphasize that the amount of energy on which the network use cost is calculated is restricted to the energy exchanged between data centers through the virtual pool, as the energy cost includes implicitly the network use cost in other cases. Finally, the network use tariff depends on the period j considered, which is usually defined based on the season, the day in the week, and the hour in the day.

The energy E j that is consumed by the cloud (and originating from the virtual pool) over period j can be expressed as:

E j = tg∈j E pool (t g )
The network use tariff is composed of a fixed share and of a variable share, in additional to penalties if the contract power is exceeded. The fixed share corresponds to costs related the network management and was estimated at 397.92e/year. Considering that the actual maximal power absorbed from, or injected into, the grid never exceeds the contract power level, no penalties are applied in this case. As regards the variable share C Nv , it depends on both the contract power for all periods and on the amount of energy consumed, and it can be expressed as: C Nv = i bS i + 4 j c j E j where S i is the contract power of data center i, b is the power coefficient, and c j is the energy coefficient over period j. Four periods j are considered: peak hours and season (period 1, as indicated in Table I), off-peak hours and season peak (period 2), peak hours and season valley (period 3), off-peak hours and season valley (period 4). In addition, two tariff options are proposed: "long use time" and "short use time". As data centers operate 24/7, the "long use time" option was selected. The corresponding numerical values chosen for coefficient c j can be seen in Table I. Coefficient b was calculated as equal to 18.34e/kVA/year.

C. Optimization method

Our approach is named SCORPIUS: Self-Consumption Optimization of Renewable energy Production In distribUted cloudS. It relies on the previously described virtual pool and costs, and is intended to increase cooperative self-consumption between the data centers. It also employs a probabilistic model for the photovoltaic electricity generation, as mentioned in Section III-A4, that allows to approximate the expected PV production on each DC at any time slot. SCORPIUS can be divided in four main steps.

In the first step, it allocates incoming requested VMs to servers on DCs according mostly to the availability of servers. This algorithm mainly focuses on allocating the VMs on the least possible number of servers. It relies on a Best-Fit heuristic which is a well known algorithm in the computing cloud community [START_REF] Johnson | Worst-case performance bounds for simple one-dimensional packing algorithms[END_REF]. Second, SCORPIUS revisits VM allocation decisions by considering possible reallocations to different DCs based on the predicted PV electricity resources at each data center. More precisely, the VMs pre-allocations are modified by using a Best-Fit heuristic based the expected PV production at each DCs.

Third, SCORPIUS compares the cost of migrating VMs from DCs with insufficient photovoltaic electricity generation to other DCs with more favorable solar conditions, with the costs corresponding to exchanging the considered amount of energy between data centers through the virtual pool, or to purchasing it from the additional energy supplier. It is important to note that migrating VMs requires an additional amount of energy compared to the case when energy is transmitted through the electrical network. This is explained by the fact that, during the migration phase of duration T m , a given VM runs on two DCs (the hosting and the receiving DCs). Hence, electrical power is consumed by both DCs during this period.

Due to bandwidth constraints, a DC can only migrate VMs one by one. Thus, the amount of VMs a DC can send in a single time slot is bounded by the sum of migration times. In addition, both DCs have to be synchronized during the migration time. Hence, if many DCs send VMs to the same DC, they can not communicate simultaneously.

Figure 1 illustrates this communication issue and shows how our algorithm works with 3 DCs (DC1, DC2 and DC3) sending 4 VMs to 2 other DCs (DC4 and DC5) during the same time slot. First, all DCs are sorted by their expected remaining photovoltaic energy for the rest of the day. The idea here consists in sending VMs from the DCs with the lowest expected remaining photovoltaic energy to those with the highest one. In Figure 1, DC1 starts sending one VM to DC5. When the migration is completed, DC1 has no longer any VM running on it, so it can be switched off. In parallel to this, a VM of DC2 is migrated to DC4. When the migration from DC2 to DC4 is completed, the migration of the VM from DC3 to DC5 can start. In parallel to this migration, when the migration from DC1 is terminated, the second VM from DC2 can be migrated to DC4. This migration plan, resulting from our algorithm, is presented in Figure 1.

Finally, SCORPIUS executes individually on each DC a consolidation algorithm using intra-DC migrations and based on a dichotomy on the number of servers in order to turn off as many servers as possible.

IV. RESULTS

A. Test process

Our approach is tested based on numerical simulations using the SimGrid framework which is dedicated to the simulation of distributed IT systems [START_REF] Casanova | Versatile, scalable, and accurate simulation of distributed applications and platforms[END_REF]. SimGrid embeds sound simulation models of CPUs, TCP/IP networks, VMs, and servers energy consumption, which have been assessed both theoretically and experimentally. Thanks to this platform, we accurately simulate the resources usage (i.e. CPU and bandwidth sharing), the execution time and the cloud energy consumption.

We compare SCORPIUS performance against the roundrobin algorithm that is traditionally implemented in cloud schedulers. This latter algorithm distributes the workload uniformly among the DCs by allocating incoming VMs to the DCs following a circular order. However, contrary to our approach, round-robin does not take into account the switching off unused servers, nor VMs migration.

B. Results analysis 1) Weekly energy consumption and variables costs:

We simulate a week of execution of the cloud and determine the corresponding costs. These results only include the variable share of these costs upon which our optimization approach has an influence. This variable share depends on (1) the cost of energy purchased from the additional energy supplier, (2) the PV electricity excess injected into the grid and consumed by other customers, and (3) the network use cost corresponding to the energy exchange between data centers through the virtual pool. The results are shown in Figure 2 and Table II. It must be noted that all these results show the performance on the entire cloud, i.e. including all the data centers.

It can be observed that, when the cloud is managed using SCORPIUS, it consumes significantly less energy than when round-robin is used. More precisely, this difference is equal to 7.6 MWh over the period of one week considered here. This represents approximately 30% of the cloud weekly energy consumption (relative to the case where round-robin is used). This is the combined effects of switching off unused servers and consolidation through VM migration. Consequently, compared to the case where the round-robin is used, energy purchases from the additional energy supplier are reduced by 5.2 MWh, and energy injections into the grid are increased by 2.3 MWh, leading to a total gain of 959.03 e for one week.

2) Total annual cost: The total annual cost include both variable and fixed costs. The variable costs are calculated over the week considered in this study, and they are then multiplied by the number of weeks during a year (52). Monthly fixed costs are multiplied by the number of months in a year [START_REF]Grid'5000[END_REF] to obtain the annual equivalent cost. The total annual costs obtained with the two algorithms considered here are shown in Table II. It must be noted that using SCORPIUS leads to a difference of approximately 50 ke. This amount is extremely significant as it represents around 37% of to the total annual cost with round-robin. Moreover, as SCORPIUS does not require a specific infrastructure, its deployment does not imply any extra capital cost compared to round-robin.

3) Cloud self-consumption ratios: The self-consumption ratios are calculated over each time slot of 10 minutes. The profile of R L SC is shown on the second row of Figure 2. If both day and night times are taken into account, an average local (resp. cooperative) self-consumption ratio of around 34% (resp. 38%) is obtained when SCORPIUS is used, and of around 30% (resp. 36%) when round-robin is used. However, this self-consumption ratio is equal to zero in both cases during night time when there is no PV electricity generation. Hence, it would be more relevant to consider the performances of both algorithms during day time only. Under these conditions, using SCORPIUS leads to an average local self-consumption ratio of around 54%, while this ratio is only equal to around 46% in the case where round-robin is used. The cooperative self-consumption ratio R C SC is then equal to around 61% with SCORPIUS and 56% with round-robin.

The local self-consumption ratio increase, in the case when SCORPIUS is used, leads to a reduced need for energy exchange between the data centers. Under the conditions considered here, the amount of energy which is exchanged between the data centers through the virtual pool (and equal to 0.76 MWh) is almost reduced by half with respect to the round-robin case.

V. CONCLUSION AND FUTURE WORK

This paper focuses on the self-consumption optimization of a computing cloud partially powered by PV panels located at each data centers. The study includes economical costs corresponding not only to the cost of energy, but also to the network use cost. This latter cost is based on the network use tariff. An approach for cloud management called SCOR-PIUS is proposed and compared to the traditionally used and well-known round-robin algorithm. The results show that the increase in terms of self-consumption and total consumption is very significant when the SCORPIUS approach is used. Future work will focus on evaluating our approach with economical models from other countries.

Fig. 1 .

 1 Fig. 1. Example of migration plan with 5 DCs.

Fig. 2 .

 2 Fig. 2. Cloud performance each time slot over one week.

TABLE I NUMERICAL

 I VALUES OF THE ENERGY COEFFICIENT c j .

		Period 1	Period 2	Period 3	Period 4
	c j (ce/kWh)	4.18	2.81	1.89	1.74

ACKNOWLEDGMENTS

This work has been supported by the Inria exploratory research project COSMIC.