Ismael Samaye

Paul Leloup

Gilles Sassatelli

Abdoulaye Gamatié

Energy-efficient Hardware Reuse for Sustainable Data Centres (Extended Abstract)

Keywords: Computer systems organization → Architectures, Other architectures, Heterogeneous (hybrid) systems Sustainability, data centre, system reuse, renewable energy

As technologies progress in data centres, older-generation servers are often replaced by more energy-efficient ones. As a result of this trend, the problem of electronic waste is exacerbated. This extended abstract discusses a few recent insights about sustainable data centres. Then, it presents an on-going approach to extend the lifetime of old-generation servers in data centres and reducing the amount of brown energy and water needed for cooling.

INTRODUCTION

Recent advancements have been made in the field of information technology, particularly in the areas of cloud services that make life more convenient (e.g. the Internet of Things, streaming) and data centres that provide these services. However, this progress is not without consequences, especially for the environment, including an increase in 𝐶𝑂 2 emissions as a result of the production of electricity to fill the demand for data centres, which constitute approximately 1 percent of total global energy consumption, as indicated in [START_REF] Jones | How to stop data centres from gobbling up the world's electricity[END_REF]. In order to meet this energy demand, data centres commonly utilize non-green energy sources that contribute to significant carbon dioxide emissions [START_REF] Masanet | Recalibrating global data center energy-use estimates[END_REF]. The emissions of carbon dioxide from data centres account for 0.3% of the total emissions of 𝐶𝑂 2 on the planet.

Additionally, in [START_REF] Gupta | Chasing carbon: the elusive environmental footprint of computing[END_REF], authors point out that data centre hardware and infrastructure also contribute significantly to the amount of 𝐶𝑂 2 released. Besides the emission of 𝐶𝑂 2 , data centres are also concerned about water stress problems in some regions where they are located [START_REF] Abu | The environmental footprint of data centers in the united states[END_REF]. Electronic waste is another issue that contributes to pollution. As a matter of fact, toxic elements can be found in the metals and electronic components extracted from data centre equipment and deposited in the environment. There are two types of equipment included in this category: the data centre infrastructure, which consists of mechanical and electrical components (e.g. transformers, generators, air conditioners, racks, cables), and computing components (e.g. servers). The equipment is periodically replaced to maintain and improve its performance. Generally, infrastructure components are replaced every 10 or 15 years. In contrast, computing components are more frequently changed. A white paper from the SuperMicro Company [START_REF]Data Centers & The Environment[END_REF] examines the server renewal cycle in data centres. Various approaches to the management of electronic waste are also discussed. The paper argues that green IT is driven by the replacement of older systems with newer ones that can handle more intensive workloads. In 2020, only 68% of surveyed respondents renewed their servers annually or every 2 to 3 years compared to only 37% in the 2019 survey. For those who keep their servers for six years or more, only 8% reported doing so in 2020, compared to 23% in the 2019 survey. For organizations that refresh very frequently, about every year, 7% did so in the 2019 survey, but this number increased significantly to 26% in the 2020 survey. This enthusiasm for new, more energy-and workload-efficient systems is a trap. As people will want to use such a system, this will lead to an increase in the frequency of replacement, which means an increase in the amount of e-waste and also in the pollution caused by mineral extraction and manufacturing of servers.

Regarding the e-waste issue, various approaches are surveyed in [START_REF]Data Centers & The Environment[END_REF], as shown in Figure 1. One can observe an increase in the efforts to mitigate the e-waste issue in 2020 compared to 2019. To build green data centres, authors in [START_REF] Wang | Sustainability-aware resource provisioning in data centers[END_REF] suggest the reduction of 𝐶𝑂 2 emission during the data centre construction and operational phases while taking into account server lifetime. In a recent study [START_REF] Bashroush | Optimizing server refresh cycles: the case for circular economy with an aging moore's law[END_REF], the authors addressed the issue of electronic waste issue from a different perspective. Instead of replacing the whole servers for upgrading the compute infrastructure, they advocate renewing only some components within servers, thus reducing the electronic waste. Then, they also propose an approach to renewing servers in data centres by upgrading servers that have been in use for five to six years, thereby extending the life of the servers and reducing the amount of electronic waste dumped in nature. Despite the fact that this approach reduces the amount of electronic waste, some electronic waste is still emitted due to the replacement of certain components. We also have to take into account the possibility of errors in the calculations due to the fact that some of the server components are old.

Similarly to [START_REF] Bashroush | Optimizing server refresh cycles: the case for circular economy with an aging moore's law[END_REF], we aim at reducing electronic waste due to data centres. In our approach, it is strongly advised that old-generation servers be moved to companion data centre infrastructures using mostly renewable energy, thereby mitigating the energy efficiency penalty caused by their age.

OUR PROPOSAL FOR HARDWARE REUSE

Different strategies can be used to increase energy efficiency in data centres: one is to increase the computing performance of the data centre while maintaining (or, even better, decreasing) its power consumption; another is to maintain the data centre's computing performance while reducing its energy consumption.

Here, to achieve energy efficiency, we advocate a novel computing concept, referred to as Genesis, recently implemented in a demonstrator [START_REF] De Melo Da Silva | Optimization of data and energy migrations in mini data centers for carbon-neutral computing[END_REF] [4] [START_REF] Gamatié | Modeling and analysis for energy-driven computing using statistical model-checking[END_REF]. As illustrated in Figure 2, Genesis is a companion mini data centre typically deployed on the roof of the building that hosts servers and energy harvesting and storage facilities. Genesis is composed of modules or nodes that are interconnected. The nodes can exchange energy and computational data (e.g. virtual machines). A node is comprised of a solar panel, a battery, a server, and a local control system. In the event that a node lacks enough energy in its battery to execute a workload, it can request energy from another node. Or, it can send the workload to a node with sufficient energy. Local controllers are responsible for this energy exchange. The essence of our solution is to migrate non-urgent or noncritical workloads (e.g. workloads with low quality of service requirements) from indoor data centres to an outdoor Genesis data centre. The migrated workloads are therefore executed on servers that have been repurposed by Genesis and powered by free solar energy.

By extending the use of old-generation servers, the approach directly reduces e-waste. Moreover, the energy efficiency issue related to old-generation servers is mitigated by only leveraging clean renewable energy. In particular, it eliminates the amount of energy required for cooling indoor servers since Genesis is deployed outdoors to benefit from ambient cooling. As a result, this reduces the carbon footprint and water stress on the environment associated with conventional data centres.

We evaluated the potential of our proposed approach by integrating two typical old-generation servers into the Genesis prototype: a server fitted with an Intel Xeon processor E5-1620 v2 with 10M cache and operating at 3.70 GHz, and a less powerful Dell Pow-erEdge T20 server. The former server was used six years ago in the world-class supercomputer hosted by CINES in Montpellier, France (https://www.cines.fr/en). The latter belongs to an old generation of Dell servers. The benchmarked configuration of Genesis comprises a 2 KWh battery and 300 W solar panel per node.

Given the above configurations, a node integrating the most powerful server can run for 10h and 6.5h at 50% and 100% CPU stress levels respectively, by only using its associated battery budget. When considering the less powerful Dell server option, a node can run for 27 h and 19 h at 50% and 100% stress levels respectively. Note that under favorable solar irradiation conditions, an empty node battery can be completely filled with energy within about 18 h, using a single solar panel.

Based on the above compute capacity, Genesis should be able to perform some workloads in conventional data centres while providing better sustainability.

Figure 1 :

 1 Figure 1: E-waste plans 2019 and 2020 surveys [2]

Figure 2 :

 2 Figure 2: Integration of Genesis with conventional data centre for sustainable compute infrastructure (illustration partly based on macrovector / Freepik -https://fr.freepik.com/vec teurs-libre/composition-interieure-isometrique-du-cloudcomputing-server-datacenter_4027518.htm)


Thierry Legou
email: thierry.legou@lpl-aix.fr

Alain Marchal

Yohann Meynadier

Carine André

3D Palatography

Static palatography has been widely used to investigate consonant articulation. The method is based on the observation of the tongue print (a black paste is spread on the tongue before the production) either directly onto the palate (direct palatography) or onto a pseudo palate (indirect palatography).

This study has two main objectives. The first one is to provide an automatic analysis of the tongue print characteristic (size, shape, position onto the palate,…) in order to allow an objective description of the linguopalatal contact.

The second one concerns the shape of the palate. Indeed, a simple picture (2D) analysis does not take into account the shape (3D) of the palate vault. To alleviate this 2D limitation, we propose a method based on the use of structured light which enables the computation of the palate shape.

Introduction

Static palatography is an old method which has been used over years to study consonants articulations during speech production [START_REF] Marchal | La palatographie[END_REF]. The method consists of analyzing the contact area where the tongue touches the palate during the articulation. In order to visualize this contact area, before the consonant production, a paste made half of charcoal and half of olive oil is painted over the tongue. After the consonant production, the articuloratory contact leaves a dark print onto the palate which is analyzed to characterize the articulation. Palatography is said direct if the tongue touches the real palate. In this case, the tongue print is observed using a mirror (see figure 1). Palatography is said to be indirect, if an artificial palate is used. In this case, once the palate is placed in the mouth, the consonant is produced, and the artificial palate is then removed for its analysis. Static Palatography (direct and indirect) had been used in many studies for more than a century, but in the sixties Electropalatography (EPG) came in [START_REF] Kuzmin | Mobile palatography as a toll for acoustic study of speech sound[END_REF][START_REF] Kydd | Continuous Palatography[END_REF], and permitted the study of dynamic contacts tongue/plate. Electropalatography permits contact analysis over time, but it also shows limitations. Indeed, only points of the palate where electrodes are located are considered, the artificial palate can disturb the speaker. Lastly, an EPG plate is expansive and can make difficult studies which would require a large number of speakers [START_REF] Witting | New techniques of palatography[END_REF].On the other hand, static palatography takes into account the whole tongue print onto the palate. Moreover, the fact that associated costs to such studies is quite low, it permits to consider a large number of speakers. The ensuing study has two main objectives. The first one is to propose an automatic analysis of the tongue print with the extraction of its characteristics to make comparison inter and intra speakers. The second one aims at knowing the shape of the palate without making an impression of the speaker's palate. Information about the palate shape is quite important to understand specificities of certain articulators behavior as previous studies pointed it out [START_REF] Brunner | The influence of the palate shape on articulatory token-totoken variability[END_REF]. Knowing the palate shape also permits to compensate an intrinsic limitation due to the picture (2D) analysis whereas the palate is not flat. In order to permit palate classification we propose extraction of the 3D shape to make palate comparison possible.

Method

Shooting system

The shooting system used is inspired from the one described by Anthony [START_REF] Anthony | New method for investigating tongue positions of consonants S[END_REF]. We have slightly modified it (see figure1), keeping only the main mirror, and we have added a system to project the structured light in order to determine the palate's shape.

Figure1: Static direct Palatography, shooting system (inspired from the one described by Ladefoged, 1957) In order to be able to extract real dimensions from the picture, a calibration target is present on each picture taken. This permits a conversion pixel to mm, even if camera adjustments are not recorded. In case of indirect static palatography the shooting system is much simpler, as the artificial palate is removed from the mouth and place on a table to take the picture.

Picture analysis

Before any information extraction from the picture, the operator has to set three markers, thus in order to define automatically sixteen areas to perform tongue print shape analysis (see Figure 2). The first marker is set between the two incisors and the two other markers, on the rear side of last molars. In both case of direct or indirect palatography, the procedure to set markers and the picture analysis are identical. As analysis of the picture is essentially based on pixel counting, this is why the required format for pictures is BMP (BitMaP). Pictures processed must be 256 color BMP pictures. This palette size is large enough for color analysis in case of palatography study. If one wants to process pictures with a larger palette, then a palette conversion must be first performed. Various characteristics of the tongue print can be provided like its location onto the palate (referred to markers) and its "shape". To extract information about the print shape, for each area (16 in total, see figure 2) are computed: the center of gravity, the percentage of the considered area touched. Combining information for each of the sixteen areas, it is possible to determine symmetry parameters for the whole print (antero/posterior, left/right). An average pattern can be computed over several pictures. The proposed division of the palate (16 areas) can be overlaid on the one described by Lenz [7] which distinguished 6 different areas: alaveolar, palatal (pre, medio, post) and two velar areas (pre and post).

Palate shape calculation

Knowing the palate shape is quite important, and Ladefoged in 1957, already emphasized that the study of the plate's impression was mandatory to get a better estimate of the print than the one we can extract from a picture (2D). Moreover, several studies have previously pointed out that the palate shape impact articulator's behavior. So, once being able to classify palates' shape, then it is possible to run studies with atypical palates (wide, narrow, arched, flat, cleft) including the palate shape as a factor.

Principle

The principle consists of a projection of a structured light (a grid) onto the palate.

As the palate is not flat, columns and rows of the grid are distorted. Taking into account these distortions enables computation of the palate shape which caused the observed distortions. The shape we want to measure is placed at a distance D from the light source. Each beam of the light source is separated by an angle θ. To do so, we have tested a laser source which by construction projects a matrix 7*7, and also tested the use of a strong white LED behind a screen drilled. From figure 3 it appears that the information we want to get is the height of the shape (A) at a certain abscise X1'. If the shape was flat and thin then the spot would have hit it at abscise X1. To calculate the height (A) the principle is to measure the gap between X1 and X1', respectively the expected abscise in case of the shape which would extremely fine, and the observed abscise where the spot hits the real shape.

The following equation permits to calculate A, from controlled parameters (θ, D) and from the observed gap spotted B on figure 3.

A=B.(tanθ θ θ θ) -1 Equation 1

We can see that for a given horizontal axis (X1), a height of +A or -A leads to the same gap B. To deals with this ambiguity, we introduce a constraint that concavity is the expected global shape instead of convexity.

Figure 4: View of an impression of a palate with a structured light grid projected on it. Lines show shifts between expected spots (if the palate was flat) and observed spots on the shape to reconstruct.

To generalize Equation 1, to all spots of the grid:

Xs.DEFx=Xo DEFx: distortions matrix along X Ys.DEFy=Yo DEFy: distortions matrix along Y With: Xs,Ys,Xo,Yo : matrix of coordinates respectively of the structured light grid without distortions (Xs and Ys) and of the observed grid with distortions (Xo and Yo).

Measuring distortions of the grid (gaps between the original grid and the observed one) enables the computation of the shape. To validate this method we used it successfully to characterize known shapes like a ball. Once tested, this method had been used on a real impression of a palate. This configuration corresponds to the case of indirect static palatography when the palate is removed from the mouth and set in front of the picture shooting system (see figure below). This tests shows that the method permits using structured light to obtain "Z information" from a picture (2D).

Use of the 3D shape

Many studies consider the shape of the palate as valuable information for the optimization of the analysis of palatogrammes. In order to describe palates or to classify them according to the shape of their vault we propose parameters (possibly previously described).

Conclusion and perspectives

The automatic analysis of palatograms may revive interest in static palatography, as it provides an accurate description of the tongue print and then permits inter and intra speakers comparisons. The knowledge of the palate shape can be a key point to understand certain consonant productions. Several tests performed on known shapes and on impression of palate have validated the method. It can be used for static direct and indirect palatography.

For Electroplatography studies, the description of the palate can also help. A set of virtual electrodes laid out on a palatogram provides an interesting overlap with usual EPG studies. When EPG gives a digital pattern of contacts (tongue/electrodes), static palatography takes over, going beyond the electrodes lay out and consider the whole print.

Figure 2 :

 2 Figure 2: Palatogram with the 3 yellow markers and 16 analysis areas (3 areas are coloured to highlight them)

Figure 3 :

 3 Figure 3: Use of a structured light projected on the shape. Angle θ and the distance D are controlled parameters.

Figure 5 :

 5 Figure 5: View of a palate impression lighted by the structure grid. For lines drawn on the picture, the profile of the vault was rebuilt.This tests shows that the method permits using structured light to obtain "Z information" from a picture (2D).

Figure 6 :

 6 Figure 6: Palate description parameters.


Nathan C Éli
email: nathan.celie@insa-lyon.fr

Bernard Lyon

Margot Gaetani-Liseo
email: margot.gaetani-liseo@univ-lyon1.fr

Amine Lahyani
email: amine.lahyani@enit.utm.tn

Ali Sari
email: ali.sari@univ-lyon1.fr

Optimal multi-criteria management of energy storage systems in a micro-grid

Keywords: Charging Systems and Infrastructures, Microgrids, Energy management system, Vehicule-To-Grid, Battery degradation, Reinforcement learning

With the aim of relocating consumption and production, energy management at the scale of a micro-grid seems to be a lever to improve system efficiency, lifespan and profitability. The purpose of our work is to implement an energy management policy using reinforcement learning techniques in order to maximize the profits and considering different time-varying phenomena: variable power production from a PV system, variable power consumption from a DC load and variable energy price. This work will also take into account the battery degradation as a cost to be minimized. In the near future, this application would enable a better integration of electric vehicles in the grid.

I. INTRODUCTION

T he integration of Renewable Energies Sources (RES) in the power grid and electromobility seems to be a solution to reduce GreenHouse Gases emissions (GHG) and contribute to tackle climate changes and global warming. These considerations have led to major developments of renewablebased technologies such as PhotoVoltaic (PV) and wind power generation. However, these technologies are intermittent and do not have inertia, which leads to power grid instability and management issues. As a solution, Energy Storage Systems (ESS), such as batteries, are added in Micro-Grids (MG) connected to the grid, in order to improve flexibility and allow Energy Management System (EMS) with complex strategies making the all system more reliable and resilient [START_REF] Choudhury | Review of energy storage system technologies integration to microgrid: Types, control strategies, issues, and future prospects[END_REF]. Moreover, the environmental footprint of buildings is estimated at 30% of GHG emission and energy consumption [2]. More than being a solution for the building themselves, optimal EMS would enable a improve the ESS integration such as stationary batteries, and/or second life batteries and/or Electric Vehicles (EV) in the MG.

Our work aims at finding an optimal energy management policy on a given MG in order to improve its sustainability and its profitability. The MG considered in this paper had been scaled in articles [START_REF] Gaetani-Liseo | Identification of ESS Degradations Related to their Uses in Micro-Grids: application to a building lighting network with VRLA batteries[END_REF] and [START_REF] Gaetani-Liseo | Consideration of energy storage systems and their degradation in microgrids sizing and energy management : influence of model accuracy (Prise en compte des systèmes de stockage de l'énergie et de leurs dégradations dans la gestion et le dimensionnement des micro-réseaux : influence de la précision des modèles, fr)[END_REF].

Markov Decision Processes (MDP) and Agent-based algorithms such as Q-learning have shown a good efficiency in high dimension sequential optimisation problems and will be used in our study. This techniques are already used for solving such problems and they are mainly focused in the cost of sold and purchased energy. The purpose of our work is to add the degradation of the ESS both in the model and as a decision variable.

Section II presents a brief literature review on what have been done so far in terms of Reinforcement Learning (RL) techniques applied to EMS in MG. Section III aims at giving a model to the system considered in this study. Section IV gives the formulation of the problem that has to be solved by the RL algorithms. Section V presents some results, discussions and perspectives of this work. Section VI gives the conclusions.

II. STATE OF THE ART

Most of the subjects which deal with optimal EMS in MG are mainly focused on the amount of purchases and sells of energy. For example, the work presented in [START_REF] Abedi | Battery energy storage control using a reinforcement learning approach with cyclic time-dependent Markov process[END_REF] contributed to show the interest of Q-learning methods in energy management problems with economic criteria, compared to deterministic methods and the relevance of a good discrete size. The work done in [START_REF] Muriithi | Optimal Energy Management of a Grid-Tied Solar PV-Battery Microgrid: A Reinforcement Learning Approach[END_REF] on the time basis of a day consisted in a comparison between a trading and a non-trading case, and proved the relevance of the latter. Moreover, in this type of optimisation problem the aging aspect is barely considered and could represent a non-negligible potential blow on a long term basis.

Other non-economic criteria are often taken into account without being considered as part of the objective function. For instance, the study carried out for the stoRES project in 2019 was aiming at maximizing the self-consumption ratio on a bunch of sites in Europe considering the evolution of batteries efficiency [START_REF]LIVING LAB developed within the StoRES project Computation details and hypothesis[END_REF].

Hence, the purpose of our work is to take into account the ESS degradation both as a part of the cost and as a component of a state.

III. SYSTEM MODELING

The MG considered in this paper consists of a PV production system (i.e. solar panels), an ESS (i.e. a battery pack), an electric DC load (i.e. a lighting network with a set of LEDs) and a connection to a large scale network which enables imports and exports of electrical energy. In Fig. 1, we show the different potential power flows in the MG. The possibility of a Vehicle-To-Grid (V2G) configuration could also be considered with a direct flow between the power grid and the ESS, in both directions. In a first approach this possibility is not implemented.

In our case, the algorithms and the model are implemented in Python language.

A. Micro-grid specifications

The PV and ESS specifications have already been set in [START_REF] Gaetani-Liseo | Identification of ESS Degradations Related to their Uses in Micro-Grids: application to a building lighting network with VRLA batteries[END_REF]. This work suggests a new optimal sizing algorithm using a power/energy consideration. An optimal compromise between PV and ESS sizes is proposed with 0.65kW p 1 of PV panels and an ESS with a nominal energy E B 0 , of 1.87kW h. The ESS energy efficiency, η B , is 0.83, considered as constant. The DC-DC converter connected between the ESS and the DC bus has an efficiency η DC-DC of 0.96 as measured in [START_REF] Gaetani-Liseo | Impact on Energy Saving of Active Phase Count Control to a DC/DC Converter in a DC Micro Grid[END_REF].

The amounts of production and consumption power taken from data have already been affected by the efficiency of the converters.

In the configuration considered in this work, the PV installation had been sized for the PV installation enable full self-consumption [START_REF] Gaetani-Liseo | Identification of ESS Degradations Related to their Uses in Micro-Grids: application to a building lighting network with VRLA batteries[END_REF].

B. Input data

The input data of PV power production come from the database of the building integrated photovoltaic (BiPV) described in [START_REF] Gaetani-Liseo | Identification of ESS Degradations Related to their Uses in Micro-Grids: application to a building lighting network with VRLA batteries[END_REF], as well as the power consumption data available in [START_REF] Gaetani-Liseo | Consideration of energy storage systems and their degradation in microgrids sizing and energy management : influence of model accuracy (Prise en compte des systèmes de stockage de l'énergie et de leurs dégradations dans la gestion et le dimensionnement des micro-réseaux : influence de la précision des modèles, fr)[END_REF] and [START_REF] Gaetani-Liseo | Identification of ESS Degradations Related to their Uses in Micro-Grids: application to a building lighting network with VRLA batteries[END_REF]. The power load profile is the consumption of 1 Standard Test Conditions, i.e. 1000W/m 2 , AM 1.5, 25°C

the DC lighting network [START_REF] Gaetani-Liseo | Identification of ESS Degradations Related to their Uses in Micro-Grids: application to a building lighting network with VRLA batteries[END_REF]. The database consists of two years (730 days) of readings, collected at a 1 min time-step, between 2016-01-01 and 2017-12-31. PV power production and DC load power consumption are respectively denoted p P V t and p load t . These powers are both given in kiloWatts [kW]. A database description is detailed in table I. Data related to PV production have been verified thanks to another database extracted from the software PVGIS [START_REF]JRC Photovoltaic Geographical Information System (PVGIS) -European Commission[END_REF] with the appropriate location and installation characteristics.

In order to find some patterns, each day has been divided into 48 time intervals where each time interval has a duration ∆t taken at 30 min and the recordings of p P V t and p load t in the input data have been averaged for every time-step interval.

The power profile p P V t , p load t , as well as all time-dependant variables are discrete-time variables with a step time ∆t. Hence the notation t corresponds to one of the 48 time intervals of a given day as described in equation [START_REF] Choudhury | Review of energy storage system technologies integration to microgrid: Types, control strategies, issues, and future prospects[END_REF].

∀t, t = n∆t | n ∈ N (1)

C. Power flows along the DC bus

The electrical architecture of the study MG is represented in Fig. 2. The power produced by the PV system p P V t can only be positive or null. The power received by the DC load p load t can only be positive or null. The ESS can both receive or deliver power and we set that p B t is positive when battery is discharging and negative when it is charging. The power that is purchased or sold from and to the power grid, called p g t , is taken positive when energy is bought and negative when energy is sold. These conventions are illustrated in the schematic diagram of Fig. 2. A balance power, p bal t , is derived from Fig. 2 as a difference between PV power and load demand, and defined with equation (2).

p bal t = p P V t -p load t (2)
If we sum up the powers along the DC bus we get (3).

p g t + p B t + p P V t -p load t = 0 (3)
And then equation ( 3) is simplified with equation (2) to get (4).

p g t + p B t + p bal t = 0 (4)
In our case, only full power flows are considered which means that the power p B t going to or from the ESS cannot be split or lowered. The same rule is applied to the powered exchanged with the power grid p g t . Hence p B t and p g t cannot be non-zero on a same time interval.

D. Battery model

In our work, the ESS charge level is given in terms of energy as a State of Energy (SoE) which is given by equation 5 during charge and by equation 6 during discharge operating modes.

SoE t+∆t = SoE t -η DC-DC × η B × P B t × ∆t E B 0 ( 5
)
SoE t+∆t = SoE t - 1 η DC-DC × P B t × ∆t E B 0 (6)
Where E B 0 denotes the initial amount of energy in the ESS.

The ESS State of Health (SoH), with energy consideration is given by equation 7, so called SoH E t .

SoH E t = E B 0 -E B,loss t E B 0 (7)

IV. PROBLEM FORMULATION

The EMS aims at finding an optimal policy for battery charge and discharge in order to ensure system profitability taking as inputs the production and consumption powers as well as ESS ageing. The goal of this optimization can be expressed in the paradigm of the Q-learning as a Reward to be maximized.

A global Reward over the whole time of the experience, noted T , has to be maximized. It is defined as r total in equation [START_REF] Gaetani-Liseo | Impact on Energy Saving of Active Phase Count Control to a DC/DC Converter in a DC Micro Grid[END_REF].

r total = T t=0 r t (8)
Where each Reward, r t , is the opposite of a cost, noted c t , as defined in equation ( 9). The cost c t is split into three components, namely the cost of the power sold to the grid, c S t , the cost of the power purchased from the grid, c P t , and the cost of the battery degradation c deg t , as described in equation [START_REF]Tarif Rachat Photovoltaïque au 1er Trimestre 2023[END_REF]. Those costs have not been normalized in order to show their actual impact on the policy.

r t = -c t ( 9
)
c t = -c S t + c P t + c deg t ( 10
)
The ESS degradation cost will be explained in subsection IV-A. The costs c S t and c P t are both defined as follows :

c S t = p g,S t
× rate S t × ∆t (11a)

c P t = p g,P t × rate P t × ∆t (11b)
Where the power exchanged with the power grid p g t is split into two variables p g,S t and p g,P t for sale and purchase modes, respectively.

Rates of sold and purchased energy, respectively rate S t and rate P t are based on the french market which uses an on-peak / off-peak system for purchased energy and a constant rate for sold energy.

The energy sales rate is considered constant during the period of the study at rate S t = 0.133 C/kWh, which is the rate for the first semester of 2023 in France, for a PV installation lower than 3kW p, according to [START_REF]Tarif Rachat Photovoltaïque au 1er Trimestre 2023[END_REF].

In order to highlight the influence of time periods in energy purchase from the main grid, a fluctuation of energy cost during the day has to be considered. The french system imposes a format of alternating between on-peak and off-peak hours. In order for these rates fluctuations to be relevant to the algorithms, they must occur during system's operating periods, i.e. during times when load consumption and PV panels production are not zero (approximately between 7 a.m. and 6 p.m. each day). The chosen format for on and off -peak hours is given in TABLE II. Considering an installation of an apparent power lower than 6kV A and for the energy rates in France for 2022-2023, we get the hourly rates from [START_REF]History of regulated electricity sales rates for residential consumers[END_REF].

A. Cost of battery degradation

The novelty of this article is to consider the battery degradation cost c deg t in the reward function, using the energy loss, ∆E B,loss t , in the amount of "storable" energy. The integration of the ESS degradation is based on the degradation model of [START_REF] Sun | Accelerated aging and degradation mechanism of LiFePO4/graphite batteries cycled at high discharge rates[END_REF]. Hence the loss of energy can be interpreted as an aging cost that is updated at each time-step according to equation [START_REF] Sun | Accelerated aging and degradation mechanism of LiFePO4/graphite batteries cycled at high discharge rates[END_REF] and inspired by the work done in [START_REF] Dragicevic | Capacity Optimization of Renewable Energy Sources and Battery Storage in an Autonomous Telecommunication Facility[END_REF].

c deg total = T 0 c B × ∆E B,loss t (1 -SoH E lim ) × E B 0 = T 0 c deg t (12)
Thus, the degradation cost at each time interval is given by [START_REF] Dragicevic | Capacity Optimization of Renewable Energy Sources and Battery Storage in an Autonomous Telecommunication Facility[END_REF].

c deg t = c B × ∆E B,loss t (1 -SoH E lim ) × E B 0 (13)
In equations ( 12) and ( 13), c B stands for the investment cost of a new ESS (a set of batteries pack) and the factor (1 -SoH E lim ) represents the admissible relative loss in term of energy. This way, if the E B,loss t energy losses are added up after each time interval, when we reach a total loss of (1 -SoH E lim ) compared to the nominal energy available E B 0 (i.e. it remains SoH E lim of E B 0 ), then the ESS has to be changed at the price of the initial cost c B . The parameter SoH E lim is set to 0.7 in this study.

The determination of ∆E B,loss t

[%] is based on results given in [START_REF] Sun | Accelerated aging and degradation mechanism of LiFePO4/graphite batteries cycled at high discharge rates[END_REF] which shows the evolution of this loss as a function of full equivalent cycles (FEC) in a Lithium Iron Phosphate (LFP) storage system. Only a characteristic at low current solicitation is considered. The original FEC data has been translated in terms of exchanged energy, considering the initial energy of our storage system, E B 0 . This conversion enables to give a ∆E B,loss t(carac) at each time interval, as shown on Fig. 3 with the black curve. Our aging model takes as inputs the power exchanged by the ESS and its SoE. Only the cycling aging is taken into account in our study. At each times step, the loss ∆E B,loss t is calculated and a cost of degradation is computed according to 13. This loss depends on the amount of energy that has been exchanged since the beginning of the simulation E B t , the amount of energy that is going to be exchanged between t and t + ∆t. The determination of ∆E B,loss t(carac) is represented by the blue lines on Fig3 which is performed at each time interval. The storable amount of energy in the ESS is decreasing along the simulation.

The characteristic above gives a percentage of loss. Equation 14converts it in terms of actual amount of energy.

∆E B,loss t = ∆E B,loss t(carac) × E B 0 ( 14
)
Then the loss calculated in 14 enables to update SoH E t at each time step as already described in 7 and 13.

B. Specifications of the Q-learning algorithms

The formalism of RL algorithms relies on definition of the different parameters as close as possible to the physics of the MG.

In order for the future policy not to end up in forbidden states, some global boundaries (i.e. external of the algorithms) have to be defined, such that the ESS remains between its energy limits E B min and E B max . The flowchart of fig 4 gives the method of determination of the allowed action in each situation.

After the execution of the algorithm described above, the returned actions are given as inputs to the RL algorithm.

The maximum amounts of power that can be admitted and delivered by the ESS in a time-step ∆t, namely at P P V max and P load max are also set, according to the manufacturer specifications. Each state s t is defined by a timing component, some noncontrollable exogenous components and controllable components [START_REF] Mbuwir | Battery Energy Management in a Microgrid Using Batch Reinforcement Learning[END_REF] and is given according to equation [START_REF] Lennartson | Introduction to Discrete Event Systems -Letcure notes[END_REF].

s t ∈ S | s = t, d j , SoE t , p P V t , p load t , rate P t , SoH E t ( 15
)
Where t is the t th time-step of a day (out of 48 for a 30minute time-interval), d j is the j th day of a year, SoE t is the level of energy of the ESS at this time-step, p P V t is the level of power production at this time-step, p load t is the level of power demand during this time-step and SoH E t is the actual battery state of health in term of energy.

The latter component SoH E t has not been used so far and might have its importance in the chosen policy.

Each component has an a priori importance in the actions chosen on a specific state and we want to know the influence of each one in the global optimal policy.

In order to keep a reasonable number of states, the levels SoE t , p P V t and p load t only take a finite number of values. The databases have been simplified in order to keep a reasonable number of states. Thus, the powers have been rounded to the closest tens of watts and the whole have been shifted to start from zero. Since rate S t is considered as constant, it will not be regarded as a part of a state.

The formulation described in [START_REF] Lennartson | Introduction to Discrete Event Systems -Letcure notes[END_REF] will conduct to a large amount of states but not every single state will eventually be explored by the agent, as long as the trade-off between exploration and exploitation [START_REF] Lennartson | Introduction to Discrete Event Systems -Letcure notes[END_REF] is appropriated to the problem. The ϵ parameter has been set to decrease exponentially so that more and more importance is given to exploitation.

Three actions depending on the operating state of the battery are considered: a ∈ A = [-1; 0; 1], defined as follows:

• action = -1 "Discharge"

• action = 0 "Idle"

• action = 1 "Charge"

V. RESULTS, DISCUSSIONS AND PERSPECTIVES

The first operation consists in the establishment of the optimal policy, for a specific day. This day have been chosen for its representativeness of the whole period. The algorithm performs 5000 episodes over this day and a Return is computed for each of them. The evolution of the Return after each episode is given in Fig. 5. Each thin blue line represents the occurrence of a maximum return and shows the infinite occurrence of the best policy after some episodes. In order to avoid the algorithm to choose a non-optimal policy, the return is checked backward from the last iteration such that the last occurrence of the maximum value sets the optimal policy.

Starting from this initial optimal policy, each following day is executed as a single episode, so that the policy can adapt to the whole time of the experiment. In order to compare the Q-Learning results in the same conditions, three system configurations with different EMS strategies have been implemented :

• A passive installation, without any PV platform nor ESS is studied for energy cost basic analysis. • A second topology integrating PV production is considered and enables purchases and sells of energy surplus or deficits.

• A third configuration with the full MG described in Fig. 1, with basic rules EMS (called in literature naive, rulesbased or decision tree strategies). It prioritizes ESS charge and discharge at each time step, based on the SoE t value limits and on the p bal t sign according to equation (4). If the SoE t limits are reach it uses the main power grid. These configurations led to first results concerning the amounts of sold and purchased electrical energy during the experiment time period. The basic EMS strategies can be compared with the strategy derived from the Q-learning algorithms but we have to consider in this last configuration the impact of the exchanges of energy on its aging in order to achieve the actual comparison.

However, these first results showed the interest of a storage system in minimizing the amount of purchased energy as shown in the second and third raws of the table III. However, this distribution highly depends on the a priori balance between the different costs. Since non-normalized data have been used, some bias can influence the optimal policy, such as the initial price of the ESS or the typical cost of energy. The choice of a specific country, with its rules and rates of electrical energy would change the optimal policy as well as the quantitative results.

Furthermore, a specific day is chosen for the establishment of the initial optimal policy. The choice of this day might influence the establishment of the policy. These different conditions and can have major impacts on the quantitative results and it would be worth doing an impact analysis of each of them.

As perspectives of this work and in the case of an application including EV as a temporary storage system, some parameters would enable the micro-grid to charge the EV before the use in priority. Moreover, the ESS is fully described with its level of energy but some other parameters like the State of Charge (SoC) could also be taken into account in further developments.

We could imagine splitting p bal t between the ESS and the grid in a specific amount determined by an optimal balance parameter. In this case, the self-consumption wouldn't be to be maximized.

We also plan to implement a complete aging model coupling calendar and cycling effects based on the Eyring laws and the works [START_REF] Redondo-Iglesias | Modelling Lithium-Ion Battery Ageing in Electric Vehicle Applications-Calendar and Cycling Ageing Combination Effects[END_REF], [START_REF] Houbbadi | Smart charging of electric bus fleet minimizing battery degradation at extreme temperature conditions[END_REF].

VI. CONCLUSIONS

The work presented in this paper aims at improving integration of renewable energy systems and optimising usage of ESS in MG. An optimal EMS based on RL is proposed. Considering the degradation of the storage system both as a cost and as a decision variable is one of the key points of our work.

Fig. 1 .

 1 Fig. 1. Micro-grid structure and power flows

Fig. 2 .

 2 Fig. 2. Schematic and power flow for the considered MG

Fig. 3 .

 3 Fig. 3. Loss of energy characteristic and determination of ∆E B,loss t(carac)

Fig. 4 .

 4 Fig. 4. Flowchart -allowed actions

Fig. 5 .

 5 Fig. 5. Evolution of the return over the episodes on a specific day -moving average (250) and occurrences of the maximum Return

TABLE I INPUT

 I DATA : PV PRODUCTION AND LOAD CONSUMPTION

		p PV t	p load t
	count	1036800	1051200
	mean	0.0838 kW	0,006 kW
	std	0.144 kW	0,017 kW
	min	0.0 kW	0,099 kW
	max	0.578 kW	0.0 kW

TABLE II ON

 II AND OFF -PEAK RATES FOR DAILY TIME INTERVALS

	Time interval	Rate (period 1, period 2, period 3) [C/kWh]
	00:00 -06:00	off-peak rate (0.111, 0.127, 0.125)
	06:00 -12:00	on-peak rate (0.160, 0.156, 0.159)
	12:00 -13:00	off-peak rate (0.111, 0.127, 0.125)
	13:00 -23:00	on-peak rate (0.160, 0.156, 0.159)
	23:00 -00:00	off-peak rate (0.111, 0.127, 0.125)

VII. ACKNOWLEDGEMENTS

We would like to express our sincere thanks and gratitude to the members of the ISGE team at the LAAS-CNRS lab for letting us use their readings and data. We would also like to thank "Collegium de Lyon for their technical and financial support.