Barnabé Potel
email: barnabe.potel@g2elab.grenoble-inp.fr

Florent Cadoux

Vincent Debusschere

Leticia De

Alvaro Garcia

Impact of the Periodicity of Feeder Re-Allocation on the Efficiency of Under-Frequency Load Shedding

Keywords: Defense plan, Europe, Frequency relays, UFLS

When a large generation-consumption unbalance occurs, under-frequency load shedding (UFLS) is the last resort to restore this balance by disconnecting loads from the electrical power system. In France, the UFLS is realized at the level of primary substations by allocating groups of MV feeders to underfrequency relays. Choosing and implementing this allocation requires substantial operator work, and is currently done once a year in France. In the near future, however, it may become both necessary and technically feasible to update the UFLS scheme more frequently; this is the investigated topic of this paper. Firstly, several re-allocation strategies are defined, whose periodicity ranges from months to hours. Then their applications are simulated on a large real-world dataset based on the actual consumption of individual MV feeders from an entire French region over two years. The performance of the various proposed strategies are compared and recommendations regarding the implementation of future UFLS re-allocation methods are provided.

I. INTRODUCTION

The under-frequency load shedding (UFLS) is the last resort to avoid a black-out due to a generation-consumption unbalance. The frequency of the electrical power system (EPS) is directly linked to this unbalance: when consumption is higher than generation, frequency decreases, and conversely. This is mainly due to the swing equation, characteristic of the synchronous generators. The UFLS is an automatic load shedding mechanism which is designed to stop large frequency drops -the purpose of UFLS is thus not to precisely control frequency, only to react to large deviations that could not be prevented by finer controls such as primary frequency control. Once frequency is stabilized, the secondary reserve and the manual actions of system operators (such as startingup additional generation or shedding selected loads) bring it back to its nominal value.

In Europe, the latest grid code dealing with the UFLS is the Network Code on Emergency and Restoration (NC-ER) [START_REF] Entso-E | Network Code on Emergency and Restoration[END_REF]. It requires Transmissions System Operators (TSO) and Distribution System Operators (DSO) to implement a new scheme for the UFLS. A UFLS scheme is currently composed of steps, each of these step being designed to disconnect a specific amount of the load when a given frequency threshold is reached [START_REF] Lokay | Application of Underfrequency Relays for Automatic Load Shedding[END_REF], [START_REF] Delfino | Implementation and comparison of different under frequency loadshedding schemes[END_REF]. Other, more elaborate UFLS methods have been proposed in the literature, typically relying on sophisticated measurements such as the rate of change of frequency (ROCOF) [START_REF] Anderson | An adaptive method for setting underfrequency load shedding relays[END_REF]- [START_REF] Rudez | Monitoring the First Frequency Derivative to Improve Adaptive Underfrequency Load-Shedding Schemes[END_REF] to shed the appropriate amount of load corresponding to the initial unbalance and sometimes even to ensure correct post-UFLS values of power flows and voltage [START_REF] Albrecht | Selective and decentralized underfrequency protection schemes in the distribution grid[END_REF], [START_REF] Das | Improved Load-Shedding Scheme Considering Distributed Generation[END_REF]. To the best of our knowledge, these elaborate methods have however not been implemented so far. The latest version of the NC-ER mostly follows the traditional logic based on frequency relays, but also introduces a new option: system operators may implement an additional load shedding mechanism based on ROCOF measurements. This new scheme can however only be applied before frequency reaches 49 Hz; then the traditional UFLS scheme based on frequency-triggered steps must be used.

In France, steps are composed of thousands of underfrequency relays located in primary substations. Each relay disconnects a few MV feeders -whose typical consumption is in the order of a few MW -all over the French EPS. Each relay is set to trigger at one of four pre-defined frequency thresholds, thus constituting four UFLS steps (plus a fifth "step" for the feeders that are not associated with any relay, and thus cannot be shed by the UFLS scheme).

In France, DSOs such as Enedis are in charge of selecting the UFLS step to which each MV feeder will belong; this is what is called the allocation of the feeder. The allocation is currently chosen by first measuring the load of every feeder at a specific date and time, which provides a "weight" for each feeder; and then by manually populating the five UFLS steps with feeders:

• in such a way that the total weight of each step is adequate; • and according to the nature of their loads (residential, industrial. . . ) in order to reduce the impact of loadshedding on the industry [START_REF]fixant les consignes générales de délestages sur les réseaux électriques[END_REF].

In practice, a coefficient -defined in Section III-A -based on the number of MV and LV customers is used to determine the more or less "industrial" nature of each MV feeder. The allocation process is realized manually by operators, once a year, in every region of France. In this paper, the traditional UFLS logic based on frequency relays is kept, but it is assumed that updating the allocation can be automated and thus be made more frequently; this assumption is true for modern primary substations that use digital supervision and control technologies, and false for older substations that use legacy technologies.

In addition, this paper considers alternative allocation methods where the "weight" of a feeder is not defined as its consumption at a specific date and time, but as another indicator: the mean consumption of the feeder over a certain period of time, as defined in Section III-C, is used.

The core of this paper is thus the proposition and characterization of simple methods to realize the feeder allocation using a shorter periodicity, from few hours to one year, possibly using alternative ways to define the notion of "weight" for a feeder. These allocations are tested with the data provided by Enedis: the consumption of all the feeders of a French region over two years.

II. SCHEME TO IMPLEMENT

A. NC-ER requirements

The NC-ER imposes new requirements to design the UFLS scheme. The scheme proposed in Table I is one of the compliant schemes regarding the national system. The most restrictive requirement is the total load to shed which has to remain close to 45 % of the national consumption [START_REF] Entso-E | Network Code on Emergency and Restoration[END_REF] .

The second part of Table I represents the objective scaled at the distribution system. In the French EPS, load shedding is realized only in the distribution system [START_REF]fixant les consignes générales de délestages sur les réseaux électriques[END_REF]. The latter comprises around 75 % of the total national load. Thus, the TSO, responsible for the load shedding, imposes a ratio to shed to the DSOs. In this study, the ratio that the DSO has to shed is considered to be 60 % of its system -which is 45 % divided by 75 %.

In practice, the UFLS scheme is subject -like any protection plan -to non-idealities such as tripping delays, measurement inaccuracy of the under-frequency relays, faulted relays, etc. As a consequence, when the UFLS activates, not all the relays that were theoretically allocated to a given step actually trigger; a fraction of the relays do, and the other do not [START_REF] Potel | Underfrequency load shedding schemes characteristics and performance criteria[END_REF], [START_REF]Rapport d'enquête de la commission de régulation de l'énergie sur la panne d'électricité du samedi 4 novembre[END_REF]. Because of these possible variations in the shedding of the steps, in the next sections the results are shown mainly using the performances of the total allocated consumption.

The objective to shed can be written such as:

R obj t = α × c Rt (1)
c Rt is the consumption of the considered region R at time t. α is the ratio to shed, here 60 % for the cumulative allocated consumption of the distribution system.

allocated consumption = f ∈F R c ft c Rt (2)
F R are the allocated feeders in region R. c ft is the consumption of the feeder f at instant t. The goal is that the allocated consumption, here in p.u. is the closest possible to the objective to shed at every instant of 60 % = 0.6 p.u.

B. Data and objective

The objective defined in (1) is the real objective at each time. Nevertheless, when allocating the feeders, their consumption is not known in advance. Thus, feeders are allocated according to their historical consumption.

The data which is used correspond to the consumption of 858 feeders over years 2014 and 2015, with a time-step of 30 minutes. These 858 feeders compose a French administrative region whose the total mean consumption is ∼ 1.3 GW.

As example, the consumption of feeders over the year 2014 are used to determine their allocations for the year 2015, then performance over the year 2015 is checked.

III. METHODS OF ALLOCATION

A. Principle of re-allocation

Historically, the allocation of the feeders is done once a year in the French EPS. This allocation has two purposes:

1) To reassess the weight of load that has to be shed; 2) To add or remove specific feeders about how critical they are (composed of loads such as hospitals, government places, street lighting, etc). In term of the weight of the load, the use of the mean consumption of the feeder over the allocation period is considered in this paper. Nevertheless, it gives a lot of possibilities of feeders combinations to reach the 60 % of consumption to shed. To limit these possibilities, what the French DSO Enedis calls the K coefficient is implemented.

This coefficient is defined in order to arrange the feeders according to the type of loads they are composed. The idea is to shed, firstly, the feeders composed mainly by residential consumers (LV customers) instead of industrial customers present on the medium voltage. This coefficient was defined with the aim to reduce the impact of load-shedding on the industry, which has a direct economic impact.

Other authors proposed the use of another coefficient based on the insertion of the photovoltaic generation [START_REF] Boeck | Integration of Distributed PV in Existing and Future UFLS Schemes[END_REF] to avoid the shedding of mixed feeders whose power flow comes back to the substation and will worsen the unbalance.

The steps are created using this coefficient: the first steps are composed by the feeders with the less priorities as the last steps are composed by the feeders with higher priorities.

B. Motivation for shortening the periodicity of re-allocations

The allocated consumption, represented by the sum of the consumption of the allocated feeders, may have large variations. This is due to the consumption profile of the load of which they are composed: some will have higher consumption during winter (residential with electrical heating), others may have important consumption during working days (businesses, industries), etc. The purpose of shortening the periodicity of re-allocation is to mitigate these variations. Fig. 1 illustrates the principle of lower allocation periods: to "force" the mean value of the allocated consumption over the allocation period to be closer to the objective to shed (60 % = 0.6 p.u.).

Whether increasing the allocation frequency will reduce the deviation of the allocated consumption or not is directly linked to the "temporal repeatability" of the consumption used to realize the allocation. Forecast models to improve the performances of the allocation are not exploited. But, in the next part, two simple methods which are straightforward to implement and based on historical data are used.

C. Methods

Two methods are compared. The difference between them is the forecast consumption. The first method is based on the assumption that the whole consumption of the previous year will repeat; while the second method uses the consumption of the previous allocation period only -which can vary from one year to few hours.

For example, considering an allocation period of 1 month:

• Method 1 determines the allocation for a certain month of 2015 considering the data of the same month of 2014.

← ------------------484 ------------------→ 4 months ← ----461----→ ← ----480----→ ← ----482----→ 1 month
453 445 444 469 477 458 468 477 478 487 482 466 E.g. the consumption over April 2014 is used to do the allocation for April 2015. • Method 2 determines the allocation for one month considering the data of the previous month. E.g. consumption of March 2015 is used to do the allocation for April 2015. The idea of using these two methods is to take into account specific variations of the consumption over the time: if the allocation period is one day, taking the consumption of the same day but one year before is not relevant (method 1), it is more relevant to take the consumption of the day before the considered day (method 2). On the contrary, for an allocation period of a few months, it could be interesting to take into account seasonal variations of the allocated consumption of the previous year (method 1).

It allows to determine what is this critical allocation period where the seasonal variations are as important as the changes in the consumption.

IV. PERFORMANCE ASSESSMENT

In this part, the two methods are computed with the data set of the consumption of the 858 feeders. First, various allocations periods are tested with the first method to give an insight of the performances. Then, the two methods are compared to determine the benefits of each methods and in which case it is better to implement one instead of the other.

A. Focus on the first method

For the first method, three allocation periods are compared: one-year, four-months and one-month.

1) Allocated feeders: the number of allocated feeders varies according to the allocation to respect the mean objective to shed. Table II shows the variations of the number of allocated feeders over a year with the three allocation periods considered.

As the K coefficient is used to determine which feeders has to be allocated as a priority, a difference in the allocated feeders numbers means an addition of new feeders or a subtract of the less priority feeders but it also means the subtract of mixed feeders whose power flow is now feeding back to the substations.

For example, for the 4-months allocation period, the change between 461 to 480 allocated feeders corresponds to:

• 26 feeders added • 7 feeders removed • 80 feeders with change in the step The seven feeders removed correspond to mixed feeders whose weight becomes negative. Fig. 2 shows the -1-week averaged -consumption of two feeders over the year 2014. Feeder 200 is only consuming power and keep a mean consumption value comprised between 1.5 MW and 4 MW. The feeder 757 is a mixed feeder whose distributed generation (DG) insertion is such that its consumption reaches negative values: the power-flow is going back to the substation. Its power consumption is comprised between -2 MW and 4 MW depending on the considered period. When this feeder weight is negative, it is not allocated while it may be allocated when this weight is positive.

Thus, the DG has a real impact on the allocated consumption and the weight chosen -here the mean value.

It can be noted that the smaller the allocation period is, the smaller, in this case, the number of allocated feeders are: using a smaller period limits the impact of DG and selects feeders whose power flow is, a priori, not feeding back to the substations. Using larger allocation periods increases the risk to allocated mixed feeders whose weights are lower than non-mixed feeders.

2) Example of the one-month allocation period: Fig. 3. shows the allocated consumption of the one-month allocation period for the year 2014 where the various allocations are calculated and for the year 2015 where these allocations are tested. Due to plotting purposes (high intra-days variations), the allocated consumption are averaged considering a oneweek period.

As expected, for the non-causal allocation over the year 2014 the mean value of the allocated consumption between two allocation instants are equals to the objective of 0.6 p.u. to shed. Moreover, this consumption is varying between 0.59 and 0.61 p.u., which can be considered as small variations. When these allocations are applied during the next year, it shows larger variations: from less than 0.56 p.u. to almost 0.62 p.u. (these values are calculated with the one-week average, thus, they differ from the extreme values shown in Fig. 4).

Two aspects of the allocated consumption need to be considered: the spread of the allocated consumption, i.e. its variations, and the mean value of the allocated consumption. The goal is to have a mean value close to the objective (0.6 p.u.) with the smaller variations possible around this value.

3) Comparison between one-month, four-months and oneyear periods: density probabilities of allocated consumption on the year 2015 for these periods are shown on Fig. 4. On this figure it may be seen the two aspects defined previously: for a one-year allocation period, the mean value of the allocated consumption is 0.604 p.u., which is close to the objective of 0.6 p.u. but its variations are important. On the contrary, for the one-month allocation period, the variations are smaller but the mean value is now 0.589 p.u. Fig. 5 shows the decomposition of this allocated consumption by the sixsteps (which objective to shed is 0.1 p.u.).

The relative variations of the steps are higher than the relative variations of the total allocated consumption -the aggregation of the steps compensate some of their internal variations. As in the Fig. 4, density probabilities can be determine by their variations and by their mean consumption. Here, the first step has large variations but its mean consumption is close to the objective of 10 % while the step 3 has lower variations but a mean consumption close to 9.5 %.

What compromise system operators would prefer, large vari- ations of the allocated consumption or smaller variations with an offset? The offset is due to higher or lower consumption over the considered year compared to the previous year. It is not predictable and is mainly due to weather and industrial activity. As for the variations, they are due to the type of loads within the allocated feeders. In this way they are more likely to be compensated between the feeders themselves (more industrial activity in one feeder with less in another one).

ŖǯŖŝ

B. Comparison of the two methods

In this section, the performances of the two methods are compared to determine which one is better regarding the period of allocation. To compare the performances, the following key performance indicators (KPIs) are used:

• The standard deviation provides an indication of the internal variations of the allocated consumption. • The mean error provides an indication of the offset to the objective to shed. • The root mean square error (RMSE) is a mix indicator of the two previous ones: the quadratic error to the objective.

These indicators are computed for both methods and for allocation periods ranging from 1 hour to 1 year. The results are shown in Fig. 6. An allocation period of one year gives the same results for both methods, as they are identical in this particular case.

The standard deviation and the RMSE are similar for the second method. The difference for the first method is due to the mean error increasing while the allocation period is decreasing. Actually, the mean error evolves in a significant way for the first method compared to the second done.

1) The mean error: is decreasing, for the first method, while reducing the allocation period. This is due to an overfitting, which is realized over a shorter period of the previous year. Thus, internal compensations of load variations are less likely to happen. As in Table II, a mean number of allocated feeders for one-month allocation period is lower than the mean number for the four-months period which is lower than the one-year period. The fact that the mean error is negative in this case is due to the differences between the consumption of the more priority feeders from year 2014 to year 2015. In other cases, the mean error could be positive according to the residential and industrial evolution of the consumption.

For the second method, the allocation period has no real impact on the mean error: as the previous period is used to determine this one, this forecast is quite robust. Nevertheless, the standard deviation is more impacted for this method for the important allocation periods (∼months).

2) The standard deviation: for the first method is lower than the one of the second method if the allocation period is higher than a month. It means, if the DSO plans to reallocate every month, the use of the data of the previous year (first method) is a better estimation of the variations of the allocated consumption, due to a seasonal variations. On the contrary, if the allocation period is lower than a month, it is more interesting to take the data of the previous allocation period (second method) for the allocation, due to the consumption cycle of different customers.

For the second method and allocation periods of several months, the mean error is close to zero while the standard deviation reaches high values. It is due to the computation of the KPIs over the entire year. For example, with a 6-months allocation period, it shows large mean errors when considering only the first six months of the year or the last six months of the year. These mean errors are compensated regarding the whole year but they lead to an important standard deviation.

With this second method, using an allocation period between two and six months gives reduced performances compared to a one-year allocation period: the seasonal variations are worsen variations of the allocated consumption. As example, if the allocated feeders are mainly composed of residential consumers -due to the K coefficient -their weights will be significantly higher during winter season (due to heating): is it not relevant to use this period to realize an allocation for the summer season.

3) The RMSE: this indicator mixes the two previous indicators to take into account these means errors and variations. Considering the RMSE, the first method is more relevant than the second one for allocation periods greater or equal to twomonths (and not one-month considering only the standard deviation).

Moreover, both methods show a limited interest for a small allocation period:

• The first method shows a minimum in the RMSE for a three-months allocation period. Afterward, the RMSE is increasing while the allocation period is decreasing due to the mean error. • The second method has a minimum standard deviation matching with the minimum allocation period: the smaller the allocation period, the better the forecast consumption. Nevertheless, a one-week allocation period shows the same performances than a one-day allocation period, and for the considered study, the four-hour allocation period worsen the performances.

Finally, both methods allow reducing the variations of the allocated consumption by reducing the initial allocation period of one year. The performances reach a maximum for the first method considering few months of allocation period, reducing this period is not relevant for shorter allocation periods as there is an over-fitting of the allocation on the variations of the previous year. The second method shows its performances increasing while the allocation period is decreasing. Nevertheless, as the intra-day variations are substantial, it is not relevant to consider an allocation period lower than a week with this method. Overall, for a period higher than one month, the first method gets better results, while for a period lower than one month, the second method is better.

V. CONCLUSIONS

In this paper, the benefits of updating the UFLS scheme more frequently -namely, of shortening the periodicity with which MV feeders are re-allocated to UFLS steps -are studied. This improvement would reduce the gap between the theoretical and the actual weight of an UFLS step, and thus help meet the stringent weight targets that are imposed by the NC-ER. Two simple methods are introduced to realize these frequent reallocations and the application of these methods is simulated on a large real-world dataset. Relevant KPIs are also introduced to compare the methods, such as the RMSE of the gap between the objective and the actual weight of an UFLS step. The first method consists in taking the consumption data of the previous year to realize the allocation. This method is most efficient with an allocation period comprised between 2 and 4 months. The RMSE is then reduced by 20 % compared to the one-year allocation period, which is a substantial improvement of the performance of the allocation. This method is, additionally, straightforward to implement by the operators as the allocation periods of a few months are manually achievable by technicians with the current technology of under-frequency relaysfor the considered region only about a hundred operations on the feeders is needed to adapt the allocation.

The second method consists in taking the data of the previous considered allocation period. This method is proved more efficient with shorter re-allocation periods, although under a month, a change in the technology of the relays is required. The use of this second method, using a one-week allocation period, made it possible to improve performance even further over the first method: the RMSE was shown to decrease by 30 % compared with the current standard of a oneyear period and even 40 % considering a few-hours allocation period.

In conclusion, to improve the performances of the allocated consumption, the first thing to do would be to choose which feeders should be allocated considering the consumption profile of each feeder. To go further, the technology of the underfrequency relays may be upgraded in specific parts of the EPS, where the power-fluctuations are the most important, to reallocate them more frequently. In these critical parts of the EPS, the use of more sophisticated algorithms to control new technology relays may help mitigating other issues such as voltage or power-flows limitations when load shedding occurs.

Fig. 1 .

 1 Fig. 1. Impact of a reduction of the allocation period.

Fig. 2 .

 2 Fig. 2. Consumption of two feeders over the year 2014.

Fig. 3 .Fig. 4 .

 34 Fig. 3. Example of the allocated consumption for the one-month periodconsumption is averaged over one week.

Fig. 5 .

 5 Fig. 5. Density probabilities of the steps consumption over the year 2015 for the one-month allocation period.

Fig. 6 .

 6 Fig. 6. Performance indicators of the allocated consumption over the year 2015 for the two methods.

TABLE II NUMBER

 II OF ALLOCATED FEEDERS FOR VARIOUS ALLOCATION PERIODS.

	Month	1	2	3	4	5	6	7	8	9	10 11 12
	1 year


Barnabé Potel
email: barnabe.potel@g2elab.grenoble-inp.fr

Vincent Debusschere
email: vincent.debusschere@g2elab.grenoble-inp.fr

Florent Cadoux
email: florent.cadoux@g2elab.grenoble-inp.fr

Leticia De

Alvaro Garcia

Leticia De Alvaro
email: leticia.de-alvaro@enedis.fr

Garcia Scales

Scales and Objectives for Under-frequency Load Shedding

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

To prevent total black-outs due to large generationconsumption unbalance, the defence plan of the electrical power systems (EPS) usually implements an underfrequency load shedding (UFLS) mechanism. The UFLS automatically disconnects loads from the EPS in order to restore the generation-consumption balance. This unbalance determines the evolution of frequency. To realise this automatic disconnection, in France, the substations are equipped with under-frequency relays which are programmed to disconnect pre-chosen MV feeders when frequency reaches specific thresholds. These feeders are allocated to a step of the UFLS scheme once a year by the DSO. The purpose of this re-allocation is:

-To update the scheme according to new local constraints (e.g. not to shed priority users as hospital, government facilities, etc. [START_REF]fixant les consignes générales de délestages sur les réseaux électriques[END_REF]). -To reassess whether the total "weight" of each UFLS step does match the objective to shed. This objective was historically defined as a ratio of the distribution system load as the under-frequency relays are located only there: each step of the UFLS had to shed 20% of load of the DSO. The Network Code on Emergency and Restoration (NC-ER) [START_REF] Entso-E | Network Code on Emergency and Restoration[END_REF] which entered into force in December 2017 redefines the framework of the UFLS schemes across Europe. Especially, the load to shed (i.e. the objective) is now defined as a ratio of the total load at the national level. This total load is defined as a net demand seen from a given point of the EPS. With the increase of the distributed generation (DG), the choice of a given point has an impact on the amount of load which can be shedthe proposition of using the feeder as the given point is proposed in [START_REF] Vde | Technische Anforderungen an die automatische Frequenzentlastung[END_REF]. The purpose of this paper is to quantify this impact by calculating the evolution of the net demand at different levels from two different given points: the feeder and the substation. The considered levels are varying from feeder to a 6-regions aggregation via substation and "department" (French administrative subdivision). This paper is organized as follows. The first section deals with the principle of the UFLS, the structure of the French EPS and the various notions of consumption which can be considered in the objective of the UFLS. The second part investigates the influence of the point of calculation on the objective. The two objectives are calculated at different levels to confirm the proper functioning of the UFLS in the case of a large insertion of DG.

UFLS PRINCIPLE AND OBJECTIVE CALCULATION

The "swing equation"

The aim of the UFLS scheme is to shed consumption in order to restore the power balance. The power imbalance is proportional to the first time derivative of frequency, commonly named "rate of change of frequency":

-If generation is higher than consumption, frequency increases. -If consumption is higher than generation, frequency decreasesthe UFLS can trigger under such a case. This evolution is ruled by the swing equation (1):

f
is the frequency of the EPS (Hz). f0

is the nominal frequency (Hz). Pm is the mechanical power from the synchronous machines (W). Pem is the electromagnetic power from the synchronous machines (W). S is the rated power of the machines (VA).

H

is the inertia of the EPS (s). The swing equation can be normalized:

Paper n° 1093 CIRED 2019 2/5
With f the frequency and p the unbalance between production and consumptionboth in p.u. According to (2), shedding a share of the national consumption allows to control the evolution of frequency.

Commonly, dynamic simulations are realized using a model of the EPS or by using directly this equation (which is easier to implement) [START_REF] Entso-E | Technical background for the Low Frequency Demand Disconnection requirements[END_REF]. Various scenarios with different unbalances and different UFLS schemes are simulated to determine which scheme leads to the most robust response.

Rated power and net demand

When the NC-ER defines a load to shed as a ratio of a national consumption, this national consumption is represented by the rated apparent power S. This rated power can be defined as the total generation of the EPS. Its value depends on the considered inertia. For example, it is possible to make the distinction between the synchronous generation which increase the stored energy H*S (kinetic energy) and the non-synchronous generation which just increase S without adding kinetic energy.

Adding power generation from non-synchronous machine like photovoltaic generationcan be seen as an increase of the rated power S and a corresponding decrease of the inertia value H, in order to keep constant the product H*S which represents the kinetic energy stored into the rotating machines connected to the EPS. We do not introduce and use the notion of virtual inertia which could be created from non-synchronous generation in this paper. This definition of rated power includes every types of generators. It can be quantified by retrieving data from meters and simulating the ones without (typically smallpower DGs). Table 1 presents a NC-ER compliant UFLS scheme for Continental Europe, composed of six steps shedding each 7.5% of the national load to stop the unbalance.

However, these percentage values are expressed as a ratio of a reference of power, and the definition of the latter is important and not obvious. According to the definition provided in the Article 3 from the NC-ER, the load is specified as a 'net demand': "'net demand' means the net value of active power seen from a given point of the system, computed as (loadgeneration), generally expressed in kilowatts (kW) or megawatts (MW), at a given instant or averaged over any designated interval of time"

The choice of using a net demand from a given point of the system is a different definition from the rated apparent power S used in the swing equation.

The French implementation of the UFLS

To determine which given point of the system should be used, the overlapping cells constituting the French EPS is described in Fig. 2. These cells will be used to study the method of calculation of the new rated power Pr.

The national cell is composed of the distribution system and large industrial electricity consumers (transportation, industrial chemistry, etc.) which are directly fed through the transmission system and do not take part into the UFLS scheme.

The distribution system is divided into regions. In France, the load shedding is technically realized only at this level of granularity (where under-frequency relays are deployed). The regions are divided into departments (typically 3 to 5). A region is an operated management cell

Paper n° 1093 CIRED 2019 3/5

for the DSO. The departments contain primary substations.

The UFLS is implemented at this level and includes local constraints such as priority users that shall not take part into the UFLS scheme. The primary substations are the operated cell of the DSO, where the under-frequency relays are located. It is at this scale that the DSO decided which MV feeders should be disconnected in order to fit with the local objective of the department.

The under-frequency relays are automated. Their parameters are set once a year. It consists in choosing for each step, which MV feeders are disconnected when the frequency will reach the specific thresholds.

From this structure, there are two levels which could be used to consider the net demand to shed: -the substation; -the feeder. Indirectly, they correspond to the TSO's point of view (substation) and the DSO's point of view (feeder), respectively. Historically, the consumption of a substation corresponded to the sum of the consumptions of the feeders.

The consumption of a MV feeder

The consumption of a MV feeder is composed of typical loads, such as residential or industrial customers. But, as the penetration rate of DG is increasing, the feeders are not composed only of loads but also of generation units. Thus, three types of feeders are differentiated: -Feeders only composed of consuming loads: net consumption = gross consumption. -Feeders only composed of generation units, called dedicated feeders. They are excluded from the French UFLS. -Feeders composed of loads and generation units: net consumption = gross consumption -local production, these are mixed feeders. Currently, it is considered that around 65% of the DG capacity is deployed into dedicated feedersand 35% is deployed into mixed feeders [START_REF] Arguence | Modélisation des ilots nonintentionnels et caractérisation des méthodes passives de détection d'ilotage[END_REF].

Rated powers

To calculate the two rated powers from the substation's point Pr,s and the feeder's point Pr,f, a new distinction of the rating power is done: not as synchronous and nonsynchronous generations, but from the transmission and distribution systems generations as shown in Fig. 3.

The TSO generation corresponds to the national generation (local generation), excluding exports but including imports. The DSO generation corresponds to the DG deployed in dedicated and mixed feeders.

The corresponding rated powers, as shown in Fig. 3, are:

These rated powers may be used to calculate the corresponding power-unbalance:

Impact on the inertia and the rated power

Considering that the DG from the distribution system is not taking part into the rated power is equivalent as seeing this DG as a negative consumption. Shedding such DG increases the inertia constant of the system. As they are mainly composed of inverters, they do not provide power in the form of kinetic energy or another energy source to create virtual inertia. Thus, the equivalent inertia of the system would increase. Nevertheless, the simulated frequency response of the system may be different depending on the consideration of this DG in the rated power in the studies.

APPLICATION TO A PART OF THE FRENCH SYSTEM

Example of feeder's consumption

Table 2 shows the power-flows of the feeders of a substation at a chosen instant. This substation is composed of 17 MV feeders. From these 17 feeders:

-2 are dedicated for production -3 are composed of priority users that cannot be shed -2, not dedicated, have a negative power-flow (for the chosen instant) The active powers of the substation are:

-21.5 MW gross consumption -19.9 MW consumption Pr,f of the substation, which is calculated by subtracting dedicated feeders to the gross consumption -18.2 MW available load to be shed (from the nondedicated and non-priority feeders): Pallocable -10.1 MW "net" consumption which is the Pr,s of the substation. -6 substations are sending back power to the transmission system at the considered instant (their corresponding Pr,s are negative); one of this substation has a negative Pr,f: DG on mixed feeders is producing more than the consumption on the only-consuming and mixed feeders. -7 substations have the same Pr,s=Pr,f (no observable feeder with a negative power-flow at this instant). -The rated powers of the department are Pr,f =220. 2MW and Pr,s =139 MW-48.9 MW = 90.1 MW. From this example, the DSO can see that there is at least 130MW more consumption that can be observable in its system, by choosing the feeder level to calculate the net demand. This consumption does not appear in the injection from the TSO at the substation level, as it is compensated by local DG.

Example of substation's consumption

Evolution in time of the rated powers

Fig. 4 shows the time-evolution of the rated powers Pr,s and Pr,f and the allocable consumption Pallocable. These evolutions are shown at different levels of the EPS: the department initially considered, the corresponding region and the aggregation of the six regions (data provided by Enedis, the French main DSO). Table 4 shows the average values of these powers. At the department scale, the rated power Pr,s, calculated from the net demand at the substation point, reaches negative values: it means that during some period of the year, the distribution system, in this department area, is feeding back power to the transmission system. With the definition of the rated power Pr,f which excludes the dedicated MV feeders, the objective remains positive. Averagely, during the whole year, the rated power calculated from the substation point is 30% less important than the one calculated from the feeder point. At the region scale, the rated power Pr,s has still large variations but it remains positive all along the year. The average difference between the two rated powers is 13%. For the six-aggregated regions, the impact of the DG is negligible: the two rated powers are equivalent (3% difference). Thus, considering a large EPS with DG implemented only in few parts, the proper functioning of UFLS is not challenged. Nevertheless, as the DG is expected to increase in a significant way for the next years/decades, it is essential to choose the appropriate point of calculation for the UFLS to avoid a negative rated power.

Paper n° 1093 CIRED 2019 5/5

For the three considered levels, the allocable consumption is representing ~76-79% of the rated power Pr,f with similar variations. It is because of the priority users which are deployed uniformly in mixed and only-consuming feeders.

Limits of the study

As presented in Fig. 2, the consumption within the distribution system (through the feeders) represents only a part (~75% [START_REF]Rapport d'enquête de la commission de régulation de l'énergie sur la panne d'électricité du samedi 4 novembre[END_REF]) of the whole French EPS consumption. The net demand fom the large consumers directly fed by the transmission system should be added into the calculation of the rated power.

As there are no under-frequency relays deployed for these large consumers, it is necessary to insure that the allocable consumption from the distribution system is still enough to shed the cumulative ratio of net demand from Table 1: 45% of the chosen objective (rated power). Another possibility would be to implement load shedding also to the consumers directly fed by the transmission system.

CONCLUSION

The evolution of the UFLS schemes in Europe, driven by the NC-ER, gives an objective of shedding a ratio of the national load for every country. Such an objective derives from the swing equation commonly used to study the UFLS performances. This equation involves the equivalent inertia and a rated power (corresponding to the total load). The values of these quantities are not explicit, even more considering the DG. The NC-ER defines the total load as a net demand from a given point of the EPS.

In this paper we quantify this net demand at the substation: Pr,s and at the feeder excluding the DG which is located into generation-dedicated feeders: Pr,f. These two definitions are applied to different parts of the French EPS: from a substation to an aggregation of six-regions.

The calculations show that, some power-flows of the initially considered substations may be negative: the DG is such that the distribution system feeds power back to the transmission system. Thus, for the corresponding department, the rated power using the substation as the calculation point -Pr,sreaches negative values during some periods of the year. As a part of the DG is excluded from the rating power Pr,f, the latter remains positive. The impact of the DG is more limited when considering a larger scale. The difference between the two defined objectives Pr,f and Pr,s is reduced from 30% at the department level to few % at the 6-regions aggregation.

If a significant number of DG would be installed into pre-existing MV feeders with consumption, an optimised choice of the feeders taking part into the UFLS scheme should be done to keep a proper functioning of the UFLS by insuring to have a positive objective to shed and enough allocable consumption to reach the cumulative ratio of it.

For extreme cases of high-insertion of DG into preexisting feeders, others solutions need to be investigated as changing the operated cell to a downstream part of the feeder (e.g. at the MV/LV transformer level).

Table 3

 3

	Paper n° 1093

shows the rated powers Pr,f and Pr,s at a specific time instant of the year 2016 for the substations of a chosen department. The chosen department includes a significant amount of DG and is not representative of the average

ACKNOWLEDGMENTS

The work reported in the paper has been developed in the framework of the Enedis Industrial Research Chair on Smart Grids.


Sylvain Girbal
email: sylvain.girbal@thalesgroup.com

Jimmy Le Rhun
email: jimmy.lerhun@thalesgroup.com

Hadi Saoud
email: hadi.saoud@external.thalesgroup.com

METRICS: A MEASUREMENT ENVIRONMENT FOR MULTI-CORE TIME CRITICAL SYSTEMS

With the upcoming shift from single-core to multicore COTS processors for safety critical products such as avionics, railway or space computer subsystems, the safety critical industry is facing a trade-off in terms of performance versus predictability.

In multi-core processors, concurrent accesses to shared hardware resources are generating inter-task or inter-application timing interference, breaking the timing isolation principles required by the standards for such critical software. Several solutions have been proposed in the literature to control or regulate these timing interference, but most of these solutions require to perform some level of profiling, monitoring or dimensioning.

As time-critical software is running on top of Real Time Operating Systems (ROTS), classical profiling techniques relying on interrupts, multi-threading, or OS modules are either not available or prohibited for predictability, safety or security reasons.

In this paper we present METrICS, a measurement environment for multi-core time-critical systems running on top of the industry-standard PikeOS RTOS. Our framework proposes an accurate real-time runtime and resource usage measurement while having a negligible impact on timing behaviour, allowing us to fully observe and characterize timing interference.

Beyond being able to characterize timing interference, we evaluated METrICS in term of accuracy of the timing and resource usage measurements, intrusiveness both in term of timing and impact on the legacy code. We also present a portfolio of the kind of measurements METrICS provides.

I. INTRODUCTION

For the last decades, industries from the safety-critical domain have been using Commercial Off-The-Shelf (COTS) processors despite their inherent runtime variability. To guarantee hard real-time constraints in such systems, designers massively relied on resource over-provisioning, time and memory space partitioning, and disabling the features responsible for runtime variability.

The demand for cheaper equipment and more stringent SWaP (Size Weight and Power) constraints [START_REF] Thomas | Lessons learned integrating COTS into systems[END_REF] makes the shift from from single-core to multi-core COTS processor for safety critical products appealing. But, as a consequence, the industry is facing an even larger trade-off in term of performance versus predictability [START_REF] Kirner | Obstacles in worst-case execution time analysis[END_REF], [START_REF] Mezzetti | On the industrial fitness of wcet analysis[END_REF].

On a multi-core processor, different pieces of software will be executed on different cores at the same time. Such software will, even if they are completely independent, compete electronically to use the shared hardware resources of the processor architecture, causing concurrent accesses to the same hardware.

On the hardware resources side, concurrent accesses are arbitrated, introducing inter-task or inter-application jitter defined as timing interference [START_REF] Girbal | A complete toolchain for an interference-free deployment of avionic applications on multicore systems[END_REF]. These interference are breaking the timing isolation principles required by the standards [START_REF]Functional safety of electrical, electronic, or programmable electronic safetyrelated systems[END_REF], [20], [START_REF]297: Software, electronic, integrated modular avionics (IMA) development guidance and certification considerations[END_REF] of time-critical software.

The literature [START_REF] Girbal | Deterministic Platform Software for hard real-time systems using multi-core COTS[END_REF] proposes several Deterministic Platform Solutions to tackle this problem, including control solutions [START_REF] Durrieu | Predictable flight management system implementation on a multicore processor[END_REF], [START_REF] Girbal | A complete toolchain for an interference-free deployment of avionic applications on multicore systems[END_REF], [START_REF] Fisher | Certifying Applications in a Multi-Core Environment: The World's First Multi-Core Certification to SIL 4[END_REF], [START_REF] Jean | Ensuring robust partitioning in multicore platforms for ima systems[END_REF], [START_REF] Jean | Hypervisor control of COTS multi-cores processors in order to enforce determinism for future avionics equipment[END_REF] aiming at completely preventing such timing interference and regulation solutions [START_REF] Sha | Single Core Equivalent Virtual Machines for Hard Real-Time Computing on Multicore Processors[END_REF], [START_REF] Yun | Memguard: Memory bandwidth reservation system for efficient performance isolation in multi-core platforms[END_REF], [START_REF] Kritikakou | Distributed run-time WCET controller for concurrent critical tasks in mixed-critical systems[END_REF] reducing the amount of interference below a harmful level.

However, most of these solutions (especially regulation solutions) require to accurately measure either task runtime or some particular hardware resource loads using performance hardware counters [START_REF] Sprunt | The basics of performance-monitoring hardware[END_REF].

On one hand, some of these solutions are only prototyped in bare-metal, neglecting the timing interference that would have been caused by the operating system itself. However, in a safety critical context, the applications are usually driven by data coming from sensor devices issuing some I/O accesses. To be able to deal with concurrent I/O accesses, the RTOS is introducing some software locks. As a consequence, these locks take a significant part of the timing interference and neglecting them might not be relevant in such a context. Running without an operating system also means the absence of a global scheduler that would allow to easily run concurrent applications. Therefore, these bare-metal models are restricted to single applications able to exploit intra-partition parallelism with a single task running in parallel on several cores. A different inter-partition parallelism scheme with different tasks running concurrently on different cores would imply embedding some scheduling facility inside the application itself.

On the other hand some solutions are relying on the RTOS to perform these controls/regulations. Doing so introduces a bias in time measurement caused by the associated system call that might itself introduce a lock (especially when several tasks try to use the same service concurrently). It also prevents us from performing the timing measurement on the operating system itself: how could one rely on a system call to measure the time taken by such a call, or the absence of locks within it?

In the classical Linux world, several techniques or solutions exists to measure timing and perform resource profiling such as gprof [START_REF] Fenlason | GNU gprof[END_REF], valgrind [START_REF] Nethercote | Valgrind: A framework for heavyweight dynamic binary instrumentation[END_REF], atom [START_REF] Eustace | Atom: A flexible interface for building high performance program analysis tools[END_REF] or oprofile [START_REF] Levon | OProfile -A System Profiler for Linux[END_REF]. However, as detailed in the next section, these solutions rely on features that are not available or prohibited for hard real-time systems.

As a consequence, the challenge is to provide a way to 1) perform an accurate real-time runtime and resource usage measurement, 2) with a negligible impact on timing behavior, 3) running outside of the operating system (avoiding system calls) to be able to profile the RTOS as well as the running applications.

Ideally, the toolset should be able to provide some of the information previously available only with a JTAG probe [START_REF] Gmbh | Trace32: RTOS debugger for linux[END_REF], but without requiring the hardware devices, nor the associated skills, making the information available to every software programmer.

In this paper we introduce METrICS: a Measurement Environment for Multi-Core Time Critical Systems running on top of the PikeOS [START_REF] Sysgo | PikeOS 4.2: RTOS with hypervisor-functionality[END_REF] RTOS from SYSGO. This tool is intended to help safety-critical software and system developers to evaluate their design choices in terms of performance and predictability. We evaluate the impact of METrICS in term of accuracy and intrusiveness and provide some examples of usage to extract timing-interference related information.

The paper is organized as follows: In Section II we detail the challenges of runtime and activity measurements for time critical software. In Section III we present the software architecture of METrICS, our proposed measurement environment and in Section IV we furthermore detail how it operates.

Evaluation starts with Section V, analyzing the accuracy and intrusiveness of METrICS both in terms of source code and timing. In Section VI, we provide an example of MET-rICS usage to evaluate different deployments of a multi-core application, measuring communication overhead due to timing interference. In Section VII we explain how we deal with the large number of experiments required to explore all possible configuration of previous use-case, justifying the need for automatic instrumentation and statistical processing. Finally, in Section VIII, we detail the associated GUI we developed as a support for analysis and interpretation of the results obtained with METrICS.

II. CHALLENGES TO REAL-TIME MEASUREMENT

Performance monitoring and profiling tools have been existing for a long time to help the programmers with debugging their systems, optimizing their applications, or identifying bottlenecks. A wide variety of generic tools exists for non-RTOS systems [START_REF] Wun | Survey of software monitoring and profiling tools[END_REF] such as gprof [START_REF] Fenlason | GNU gprof[END_REF], valgrind [START_REF] Nethercote | Valgrind: A framework for heavyweight dynamic binary instrumentation[END_REF], or atom [START_REF] Eustace | Atom: A flexible interface for building high performance program analysis tools[END_REF]. These tools rely on either OS features such as multi-threading, interrupts or timers, or either on pseudoautomatic code instrumentation to collect the required timing information.

In a real-time operating system, such features are either not available (with enforced static scheduling), restricted or prohibited due to their impacts on time determinism (such as the impact of interrupts on WCET). This is especially true for safety critical software that is constrained by drastic limitations due to the safety standards [START_REF]Functional safety of electrical, electronic, or programmable electronic safetyrelated systems[END_REF], [20], [START_REF]297: Software, electronic, integrated modular avionics (IMA) development guidance and certification considerations[END_REF].

Beyond this limitation, if collecting timing information is enough to observe timing interference, it is not sufficient to regulate the shared resource usage that causes interference due to resource contention. As a consequence collecting resource usage information is as critical as collecting timing information.

Generic tools such as oprofile [START_REF] Levon | OProfile -A System Profiler for Linux[END_REF] specialize in collecting such information by gathering the Performance Monitor Counters that are usually only available in privileged mode. The claim is that oprofile is low-overhead and non-obtrusive, and it is true from a non-RTOS point of view: Both the monitored application and the kernel remain untouched thanks to a dedicated kernel module. Also, the overhead mainly depends on the interrupt-based sampling frequency.

In RTOS systems, features like modular kernels do not exist, and using interrupt-based sampling is not an option for systems based on static scheduling. Such systems are relying on micro-kernels and modularity is even prohibited for safety and security reasons. Also "low-overhead" does not have the same meaning for large scale systems running minutes to hour-long applications where a cost of tens of milliseconds is negligible and for periodic safety critical systems that are likely to have tasks deadlines in the order of 10 millisecond or less. Furthermore, dealing with timing interference forces us to perform measurement at function-call or system-call level, where even a cost of tens of microseconds might not be acceptable.

Also, resource contentions (the main sources for timing interference) only occur at specific moments in time, during the cycles when an arbitration occurs. As a consequence, measurement and overheads have to be evaluated at cycle level.

Finally, if sampling techniques are very efficient for best effort applications, such techniques can be very troublesome for safety critical applications that focuses on how the worst case should behave. The sampling just acts as a filter that could filter out the worst case.

III. METRICS SOFTWARE ARCHITECTURE

METrICS consists of several components appearing in Figure 1. The brown parts in the figure correspond to, from bottom to the top, the selected embedded computer architecture and the PikeOS operating system above the platform support package (PSP) corresponding to the board. The blue parts correspond to the running applications we wish to monitor, each in their own partition.

Finally the green parts in Figure 1 are the core components of the METrICS environment and are described below:

A. The METrICS Library

The METrICS library is meant to be linked with the running applications to provide them with an access to the measurement probes API, allowing the collection of time and resource access information. The library contains: 1) the instrumented system call layer; 2) the application instrumentation interface; 3) the user-level interface to the instrumentation kernel driver; and 4) the user-level interface to the collector.

Syscall instrumentation layer: the instrumentation of system calls of some PikeOS personalities (e.g. the APEX

Time-critical application

Time-critical application Collector

Instrumented System-Call Layer

PikeOS µKernel

Hardware Monitor Driver

Hardware-Specific PSP Target Hardware Board

METrICS lib METrICS lib

Fig. 1. Architecture of the METrICS measurement tool system calls in case of ARINC-653 applications) automatically inserts measurement probes before and after every system call. It especially allows us to determine communication times for the intra-partition and inter-partition communications that relies on such system calls, and that can be a significant part of the application running time.

Application instrumentation: besides the syscall-level instrumentation, we provide the ability to manually insert measurement probes directly in applications. This is achieved by adding a pair of functions (metrics_probe_begin() and metrics_probe_end()) around the section of the code to be monitored. Upon execution, these functions collect the highest precision time-base counter available (usually corresponding to the number of elapsed CPU cycles since booting) and the PMC registers of the current core. The later function is also responsible for sending the monitored data to the collector.

B. The Hardware Monitor Kernel Driver

On most hardware architectures, the access to hardware performance monitor counters (PMC) requires supervisor-level privileges. Also on all supported hardware targets, the configuration of these registers to select the events that should be counted does require these privileges. As PikeOS legitimately prevents applications from getting such privileges, it was necessary to develop a kernel driver (a new feature since PikeOS 4.0) allowing us to select the PMC we wish to collect on the target.

The services provided by our driver are: 1) selecting the hardware events monitored by the hardware performance counters of local processor core; 2) starting the counters; 3) stopping the counters.

The user interface for these services, implemented in the library, uses the regular ioctl calls provided by PikeOS for drivers. For each core of the ARMv8 Juno board, about 50 events can be selected for 6 different counters. For each e6500 PowerPC core of the NXP QorIQ, about 250 events can be selected for 6 counters.

C. The Collector

The collector is implemented as a native PikeOS partition which role is: 1) to define a shared memory space where each instrumented application will save its collected measurements;

2) to configure specific measurement scenarios (like selection of events via the driver); 3) to launch the measurement campaign (relying on PikeOS scheduling schemes); 4) to transfer the content of the shared memory to the host computer, either at the end of the measurement campaign (preferred to keep time intrusiveness level low) or during the run (to allow huge data collection).

In case of a failure (e.g. a deadline miss, or an unexpected timeout that is causing the application to stop), the collector also transfer the current memory content, allowing us to perform a post-mortem analysis and debugging.

IV. METRICS INTERNAL OPERATION

As introduced in Section I, our measurement environment has to provide the user with accurate runtime and resource usage measurements while minimizing both the intrusiveness and the adherence to the hardware.

A. Intrusiveness Trade-off

A major challenge in performance monitoring tools is its intrusiveness in the system it monitors. We distinguish execution time intrusiveness and code intrusiveness. The former limits the accuracy of the measurement due to the monitoring overhead, whereas the latter requires an effort from the developer to instrument the code of either the application or the RTOS.

Automated instrumentation tools commonly suffer from a trade-off between measurement granularity (from process level down to instruction level), time intrusiveness, and code intrusiveness. Enhancing one usually have a detrimental impact on the other two. In METrICS we chose to focus on minimizing timing intrusiveness, due to the fact that 1) we focus on timecritical and safety-critical applications where time determinism is of prime concern and 2) our major objective is the ability to characterize timing interference, and correlate them to shared hardware usage.

To minimize time intrusiveness, we used several development techniques that make METrICS as light as possible during the execution of the application under test.

Firstly, all initialization and external communications were implemented in the Collector and run outside of the operational scheduling. We configured the time-base and hardware performance counter registers to be used in user-space, avoiding time-consuming context switches associated with protected mode. Time-base and PMC accesses are performed in inline assembly code to minimize latency. Finally, the shared memory containing the sample collection is mapped into the memory of the processes to avoid accesses trough system calls.

B. Scheduling Policies minimizing Time Intrusiveness

In order to minimize the runtime intrusiveness on application code, the collector is executed outside of the operational scheduling. We distinguish three scheduling phases in the course of a measurement campaign implemented with "scheduling schemes" in PikeOS:

• SCHED BOOT during which only PikeOS and the collector are running in time partition 0 (always schedulable) of PikeOS. During this scheduling phase, the collector performs the initialization of the measurement campaign, preparing the shared memory and the PMC of each core. Once the initialization is completed, the collector shifts to the MONITORING scheme.

• MONITORING during which the time-critical application partitions are scheduled according to their unmodified deployment scheme, using the collector's shared memory to store their collected measurements. During this scheme, the collector remains blocked on a semaphore from the application to notify for the end of the execution, and is therefore not schedulable. Upon receiving this notification, the collector shifts back to the SCHED BOOT scheme.

• SCHED BOOT during which again only PikeOS and the collector are running. During this phase, the collector performs the transfer of the collected measurements to the host using the MUXA protocol of PikeOS.

Using such a set of scheduling schemes allows us to minimize the time intrusiveness of the measurement environment regarding the applications. Indeed, all MUXA communications, system calls to the driver, and the collector itself are not running at the same time as the time-critical applications. It is therefore not necessary to dedicate a time slot for the collector, and the original static schedule of the application remains unchanged.

C. Transmission of collected measurements to the host

At the end of the execution the collector is responsible for transmitting the collected data to the host. This transmission is performed using the MUXA service provided by PikeOS [START_REF] Sysgo | PikeOS 4.2: RTOS with hypervisor-functionality[END_REF]. This service offers multiplexed channels of communication between the target and host computers, for debug and monitoring purposes. It offers abstraction of the hardware communication link.

To avoid the regular issues with transmitting binary data through this channel (endianess issue, control/command issues), we transmit this data directly in ASCII format as the content of a comma-separated-value (CSV) file. Each sample contains information about timestamps, hardware performance counters, as well as task ID, core ID and probe ID.

We dedicated a MUXA channel for this communication (channel 4, port 1506) initially doing MUXA over UART. However we quickly reached the maximum throughput of the UART controller, so we had to switch to MUXA over Ethernet. The implication is that PikeOS has to support the Ethernet controller of the target board, and that one of the Ethernet ports has to be dedicated to MUXA (which itself is able to multiplex).

On the host side, a driving script is performing a telnet connection to the MUXA server to dump the collected data directly in a corresponding CSV file upon reception.

V. EVALUATING METRICS ACCURACY AND

INTRUSIVENESS

The METrICS environment allows us to collect various measurements during the execution of safety critical applications, including execution time distribution and shared hardware resource access information. Rather than only extracting minimum, average and maximum values, the METrICS tool suite extracts the whole distribution of each measured data, allowing us to study the correlations between runtime and hardware resource usage.

The METrICS environment has currently been ported to the PikeOS 4.0, 4.1 and 4.2 RTOS, to the 32-bit NXP QorIQ P4080 [START_REF] Bartlett | QorIQ P4080 Communications Processor design in 45nm SOI[END_REF] based on PowerPC e500mc, to the 64-bit NXP QorIQ T2080 [START_REF]T Series -QorIQ T2080 and T2081 communication processors[END_REF] based on PowerPC e6500, to the 64-bit ARMv8 Juno Board [START_REF] Limited | Versatile Express: 64 Bit Juno r2 ARM Development Platform[END_REF] based on ARM Cortex A72 and ARM Cortex A53, as well as to simulation environments such as QEMU [START_REF] Bellard | QEMU, a fast and portable dynamic translator[END_REF] or ARM FVP [START_REF] Limited | Fixed Virtual Platforms: FVP Reference Guide[END_REF]. This Section will evaluate METrICS in terms of accuracy, precision and intrusiveness. Next sections will present an example of METrICS usage and focus more on the ability to correlate timings and resource usage, as well as its ability to perform timing interference characterization.

A. Selection of time measurement mediums

To be able to perform fine-grain timing, we need to rely on some kind of time measuring instrument. This time measurement medium could be either external, provided by any of the layers of the operating system or directly provided by the target processor as part of the instruction set.

The kind of events we wish to accurately measure includes complex functional chains (up to several seconds in avionics), runtime of individual tasks (with deadlines typically in the order of a few hundreds of milliseconds) or time spend in system calls (typically in the order of microseconds).

Each measuring medium relies on a software or a hardware mechanism that itself has a working period, thus limiting the obtainable precision to no less than this period. Also, each of these mediums actually consumes time to perform a measurement, making it impossible to accurately measure time below this measurement time overhead.

Additionally, for multi-core processors, clocks are not necessarily synchronous between all cores, introducing the concept of inter-core clock offset. If this offset does not remain constant (which could be the case if not connected to the same quartz oscillator (or clock PLL) or if the core is subject to dynamic frequency scaling), then it additionally introduce the concept of clock drift.

Using the APEX (avionic) personality of PikeOS, the operating system provides two system calls allowing us to measure time: p4_get_time() is provided directly by the PikeOS kernel, and returns the system time since boot in nanoseconds. GET_TIME() is provided by the APEX personality, and returns the system clock time, that is common to all processors. However, being system calls, these time measurement mediums involve at least a context switch from the task to the operating system, and may involve switch(es) to privilege mode(s), depending on how the OS is handling system calls.

The expected overhead for such switches is more than 1000 CPU cycles, and relying on such calls for time measurement will simply prevent us to measure short events such as context switches and system calls themselves.

On the other hand, both the ARM-v8 and PowerPC ISA are providing low-level time measurement mediums. For instance, the e500mc/e6500 PowerPC provide two special registers that can be read with the mfspr assembly instruction: The time base register is a 64-bit register, set to 0 at board reset and incremented at the Platform Clock frequency, which is provided by a different PLL than the core clock frequency. The time base is thus 16 to 64 times slower than the Core clock frequency, 48 being a common prescaler ratio. An advantage of the time base is that it corresponds to a global system clock, synchronized on all cores. The alternate time base register is a 64-bit register, also set at 0 upon reset, that increments at every core clock cycle. No specific guarantees are provided in the documentation about it being synchronous in all the cores.

B. Evaluation of time measurement mediums

The resolution of time measurement mediums are provided in their respective documentation. To evaluate the overhead of the above-mentioned mediums, we set up experiments using each medium twice in a row. The time offset between the two measurements is an upper bound of the time overhead. Each measurement pair was performed 180000 times to ensure that each overhead is not subject to variability. The results evaluated on a 1.8 GHz e6500-based NXP T2080 with PikeOS 4.1 are summarized in Table I. As expected, system-call-based mediums have a much higher overhead than special-register-based mediums. The APEX version is clocked with the Time Partition tick, used to define application time windows. Such a low resolution medium has a huge impact on overhead, making it impractical for fine grain timing.

Both special register mediums exhibit a 3 cycles (1.67ns) overhead, the alternate time base version providing a much better precision. For this reason, we chose this time measurement medium as the preferred method of measurement for METrICS.

With regards to timing offset between cores, only the alternate time base does not provide a null offset guarantee. We measured this offset against the synchronized time base and evaluated it being below 200ns for core 0 with respect to the other cores, and within the measurement precision between cores other than core 0. If a very high precision for inter-core measurements is necessary, the method we used for this evaluation can also be used for calibration at boot.

Finally, none of the mediums showed a measurable drift among the 180000 runs.

C. Evaluation of a complete METrICS probe

A METrICS probe involves: 1) retrieving the timing information thanks to the core-dedicated special registers; 2) retrieving the performance monitor counters, again through direct register access; 3) retrieving thread-specific information from the OS (thread and partition identifiers, core number); and 4) storing the collected information into the shared memory.

The intrusiveness of a METrICS probe in the source code is quite low, just adding a function call at the begin and the end of the code sequence to be monitored. Also, all the APEX system calls are automatically instrumented, requiring no further code modification. To do so, we overloaded the APEX function definitions with identical functions surrounded with our probes.

We also measured the intrusiveness in term of timing of a complete METrICS probe by performing successive calls to metrics probe the same way we did in previous section. Figure 2 presents the completion time results of such a probe, sorted over 180000 runs. The probing time varies from 85ns up to 392ns. For 97% of the runs the overhead is below 110ns. For 2.998% of the runs it is between 110ns and 191ns. And for 0.002% of the cases, it is above 191ns and up to 392ns.

More precisely, the first three steps of a METrICS probe are quite stable with ∼25ns to retrieve both the timing and the performance monitor counters, and ∼20ns to retrieve the thread-specific information. The dominant part corresponds to the storage in the shared memory, varying from 40ns to 347ns, probably because of cache effects.

As a consequence, the proposed measurement mechanisms presents a very limited overhead, and the highest achievable precision. Indeed the latency added by the full measurement probe remains in most cases shorter than the sole time measurement provided by the RTOS, and we used the fastest clock available in the target processor.

The next evaluation section will provide an example of usage of these METrICS probes for an example application.

VI. EXAMPLE OF METRICS USAGE

In this section, we provide an example of METrICS usage to evaluate how different multi-core deployments of a simple avionic-like application react with regards to timing interference. The goal would be to evaluate which of these deployments is less sensitive to timing interference.

The experiments conducted in this section were performed with METrICS running on top of PikeOS 4.1 on a 4-core PowerPC-based NXP T2080 hardware target. The application is running with the PikeOS ARINC-653 personality commonly used in the context of avionic applications.

A. Evaluated Application

We evaluated an in-house application: eDRON (embedded Directed Rotodrone Operated Network), that is guiding a fleet composed of four quadricopter drones along a preset flight route. The purpose of this application is to mimic a representative behaviour of an avionic application, while exercising classical ARINC-653 communication services and proposing several multi-core deployment options. The software architecture of eDRON is presented in Figure 3. This application is composed of six ARINC-653 partitions. The first one sets up the preset flight route for all the drones, the last one displays all the drone positions, and each of the four remaining partitions is dedicated to pilot a particular drone. These later partitions are composed of 7 tasks, with most of the computation being performed in the four engine control tasks, each being dedicated to control the velocity for one of the four engine of a drone, so that it follows the preset route.

When mapping such an application on a multi-core processor, we first need to decide what will run in parallel. The application offers two obvious parallelization schemes: inter-partition parallelism where each core will deal with a single drone, running the velocity control tasks sequentially for each drone; and the intra-partition parallelism where each core will focus on one particular engine, dealing with each drone sequentially and running all sequential tasks on core 0. Some other parallelization options are available: for example parallelizing along the pipeline, or performing looplevel parallelization of the tasks, but those are beyond the scope of this paper as they require deeper modifications of the application. These two deployments have advantages and drawbacks: The Amdahl's law [START_REF] Amdahl | Validity of the single processor approach to achieving large scale computing capabilities[END_REF] may limit the performance of the intrapartition version, while the inter-partition version will benefit from the Gustafson's law [START_REF] Gustafson | Reevaluating amdahl's law[END_REF], running independent applications in parallel. However, with regards to timing interference, the intra-partition version offers a white-box context where the partition scheduling can limit the level of interference between known tasks. The inter-partition parallelism on the other hand corresponds to a black-box context where no easy control is possible to limit the interference level of another independent application.

B. Deployment evaluation

To evaluate the eDRON application with METrICS, we instrumented each task appearing in Figure 3 by adding a pair of begin / end probes around each task and partition. We then measured task runtimes while executing the application on one of the cores (single-core version), and later compared the results with both parallel deployments.

Note that while the full application is expected to be faster on a multi-core deployment, at the level of each task / partition, the runtime is only expected to vary due to possible timing interference. As a consequence, we expect to observe a slowdown at task level for the parallel versions.

Table II compares the runtime variability of a single Drone partition for three different deployments: a sequential / singlecore deployment, and the deployments presented above with inter-partition or intra-partition parallelism. As expected, the single-core version is the deployment exhibiting the less variability with runtimes between 16 and 20ms, with a total execution time of 80ms to sequentially run the 4 drones. The multi-core deployment with inter-partition parallelism has similar lower bound and quartiles, but a much larger (x1.7) upper bound around 34ms. The deployment with intra-partition parallelism exhibits close to no variability, but with much larger runtimes of 55ms (x2.8).

We furthermore studied this later version as the increased minimum runtime was suspicious. We figured out that the extra runtime was spent during the system calls performing intertask communications.

C. Communication evaluation

Within the eDRON application, data communication is performed through the ARINC-653 system call layer. This API, corresponding to the PikeOS APEX personality, allows us to perform both intra-partition communication and interpartition communication using either buffer-based or fifo-based communication services as illustrated in Table III. METrICS allowed us, thanks to the instrumented system call layer to automatically collect runtime information relative to these communication services during the experiments we ran to build Table II.

The results corresponding to the deployment with interpartition parallelism, that will serve as a reference, are presented as boxplots [START_REF] Tukey | Exploratory Data Analysis[END_REF] in Figure 4 for both inter and intra partition communication results. The results corresponding to the deployment with intra-partition parallelism will later appear in Figure 5. Intra-partition communications are much more costly than inter-partition communication for the deployment with interpartition parallelism, especially for the receiving functions. This is expected, as running tasks sequentially introduces waiting time for communication receivers. Measurements of inter-partition communication (top) and intrapartition communication services (bottom) for the deployment with intrapartition parallelism Figure 5 presents the communication time for the deployment with intra-partition parallelism. However, to be able to represent intra-partition communication times, we had to distinguish some outlier results. We therefore added a specific inset for both RECEIVER BUFFER and READ BLACKBOARD outliers with communication time above 36.9ms (36 965 107 ns).

Such high communication times occurs in 25% of the runs for READ BLACKBOARD and in 49% of the runs for RECEIVE BUFFER. With such a high occurrence rate, they are clearly responsible for the low performance of the deployment with intra-partition parallelism.

Further studying this strange timing behaviour, we figured out than the runtime of these system calls was mostly equal to their timeout value (The ARINC-653 layer specifies communication with timeouts). Strangely, the communication that reached their timeout were successful as well. Increasing the timeout value or reducing the data size did not decrease the phenomenon, but setting infinite timeout fixed the issue.

We reported this strange behaviour to SYSGO and this issue is now fixed in the latest 4.2 version of the RTOS.

D. Conclusion

Even tough the identified bug did not really allowed us to figure out which of the multi-core deployment is best with regards to interference, we have a proof of concept that METrICS can be used for such a study, and further can be used to identify software bugs with regards to timing.

In the next section, we will evaluate the number of total experiments that would be required to perform a full characterization of several deployments of the eDRON application while varying both the hardware target configuration and the application memory footprint.

VII. DEALING WITH THE DESIGN SPACE AND AUTOMATION

A full characterization of a selection of multicore deployment of the eDRON application would require a number of experiments described in Table IV. It tests four deployment options including the three presented in previous section; three different options for the application memory footprint (fitting in L1 cache, in L2 cache or not fitting in caches); and whether or not to flush all caches between time partitions (a common practice in avionics to reduce variability). The target PowerPC architecture provides a selection of about 250 hardware events that could be measured with performance monitor registers. Among those events, we identified a set of 41 events related to the shared hardware resource the cores may compete on. Both the e6500 PowerPC architecture and the ARM-v8 architecture propose 6 performance monitor counters, meaning that testing all possible pairs of counters would require C 2 41 /6 different configurations. As shown in Table IV, this represents a large number of experiments to run, and some form of automation is desirable. Therefore we developed a series of Python scripts running on the host computer to automate all aspects of running a series of experiments: selecting the right executable file for booting the target by tftp, rebooting the board using a debug probe, providing the collector with the right set of hardware counters to use, receiving the results with a telnet connection to the MUXA server, and storing the CSV file in a directory named after the experiment configuration and date. This automation infrastructure allows us to perform large measurement campaigns by iterating on the steps presented in figure 6 without frequent user supervision. Such experimental campaigns generate a rather large amount of raw data, making the direct analysis quite difficult. In the next section, we present the visualization tools we developed to assist the analysis.

VIII. SUPPORT FOR ANALYSIS

Considering the large amount of data collected in the experiments of previous sections (over 110GB of data collected over 14 cumulative days of runtime), it would be great to also fully automatize the data mining, allowing us to analyze the collected results. This is however far beyond the scope of this paper.

With METrICS we aim at providing means to perform an expert-driven analysis. In such a context, we developed a GUI providing different ways of visualizing the collected data. In this section, we will present a portfolio of the available visualizations.

A. Underlying visualization technology

As the purpose of METrICS is to collect the full distribution of events (rather than only minimum / maximum values) to study correlations between timing and shared hardware resource contention, we need the same kind of visualizations as the ones used in the statistics domain.

As we are manipulating gigabyte-large datasets, a major concern for data mining is scalability. Some other important features will be rendering speed and interactivity, to ease the expert analysis.

In the domain of data science and big data, academic research usually often on Python language coupled with numpy [START_REF] Stéfan Van Der Walt | The numpy array: a structure for efficient numerical computation[END_REF] and pandas [START_REF] Mckinney | pandas: a foundational python library for data analysis and statistics[END_REF] for data analysis, coupled with matplotlib [START_REF] Hunter | Matplotlib: A 2d graphics environment[END_REF] and jupyter [START_REF] Thomas | Jupyter notebooks-a publishing format for reproducible computational workflows[END_REF] for visualization.

An alternative for online and interactive data visualization is to rely on dedicated javascript libraries such as d3.js [START_REF] Bostock | D3 datadriven documents[END_REF] or google charts [START_REF] Google | Google charts: Interactive charts for browsers and mobile devices[END_REF]. These libraries render charts as SVG (Scalable Vector Graphics) which enable the user to interact with each element of the chart, typically with zooming or filtering ability. Data visualization in METrICS involves both charts with millions of points (usually scatterplots) as well as charts with much fewer points (e.g. boxplots) but many filtering options. As a consequence, we opted for two different rendering options: pandas coupled with matplotlib for rendering static large-scale charts, and pandas coupled with d3.js for rendering interactively filterable data. All these visualization are bundled into a single custom Qt-based GUI using the pyside Python binding: xTRACT visualizer (expert Timing and Resource Access Counting Trace visualizer).

B. Visualization related to user probes

User probes are typically used to monitor task runtime and resource usage. Such information allow us to build up classical Gantt charts, effectively showing what is running in parallel, but it does not help to focus on the runtime variation caused by timing interference.

To better visualize runtime variability, we build for each user probe an histogram showing the distribution of observed runtimes during the successive runs, as depicted in Figure 7. The x-axis corresponds to the observed duration while the y-axis indicates how many time each runtime has been observed. The best (shortest) runtime appears on the left, the worst (longest) observed execution time on the right, the median value being identified with a black dot.

We also added colored vertical bar markers, corresponding from left to right on Figure 7 to the best case in terms of runtime, to the median, to the first iteration of the application (that frequently behave quite differently) and finally to the worst case execution time observed.

To figure out correlations between runtime and hardware resource access, we also build histograms with the collected Performance Monitor Counter data, as shown in Figures 8 and9.

In these two figures, the colored vertical bar markers still correspond to the best, worst, median and first iteration case with regards to runtime. In these figures, the x-axis corresponds to the number of accesses to a particular hardware resource. In 8, accesses seems to somewhat correlate with execution times, whereas it is not the case for 9. Scatterplots allow to easily identify linear correlations. Each point of the scatterplot indicates that a particular run has been observed with a number of resource accesses equal to the value on the x-axis, and an observed runtime equal to the value on the y-axis.

If the points approximate a straight line, there is a linear correlation. In Figure 10 most of the points are on a line except the one corresponding to the first iteration that we highlighted with a specific symbol, confirming the correlation.

Another option could be to identify correlation among performance monitor counters to eliminate redundant information provided by correlated hardware resources (like the obvious redundant information of L1 cache misses versus L2 cache accesses) to reduce the experimental design space.

C. Visualization related to the instrumented syscalls

We also rendered various charts related to the probes automatically inserted around system calls.

These renderings allow the expert user to split the runtime into the classical user time (time really spent in the application) and the system time (time spent in the operating system to deal with the application I/O). Alternatively it can be used to observe the usage of kernel locks in system calls. For instance, the top charts of Figure 11 shows the repartition of APEX system calls in the ENGINE R1 task of the eDRON application running on the 6-core Juno board ARM-v8 architecture. The x-axis corresponds to the time, and a gap can be observed between the SEND BUFFER and the SET EVENT system calls despite being called sequentially. This is due to the fact that the application is no more schedulable during system calls, and therefore at least two timeconsuming context switches occur between the two function calls.

The bottom part of Figure 11 shows with boxplots the variability of the execution time of APEX system calls for different runs of the ENGINE R1 task, actually showing that the RTOS is also affected by timing interference.

IX. CONCLUSION AND FUTURE WORKS

With the foreseen shift to mutli-core architectures in the time-and safety-critical domain, dealing with the timing interference issue inherent to multi-core processors becomes of prime concern.

Several solutions have been proposed in the literature [START_REF] Girbal | Deterministic Platform Software for hard real-time systems using multi-core COTS[END_REF], but quantitative evaluations are mostly missing. In this paper, we have presented METrICS, a toolsuite dedicated to perform fine-grain time and resource access measurements in safety critical systems, allowing to actually measure timing interference and search for the causes of these interference.

We evaluated METrICS in terms of timing intrusiveness, code intrusiveness and accuracy, and obtained cycle-level precision with an overall overhead similar to regular RTOS system calls to obtain just the timing information.

Beyond the layer running on top of PikeOS on the target system, METrICS is also a host-side server than can automatically drive large-scale test campaigns, and xTRACT visualizer, a GUI that can help the expert to analyze collected results. A next step for us would be to use METrICS to perform a quantitative evaluation of the Deterministic Platform Solutions presented in the DPS survey [START_REF] Girbal | Deterministic Platform Software for hard real-time systems using multi-core COTS[END_REF], that aim at either controlling or regulating interference.

Fig. 2 .

 2 Fig. 2. Completion time of a METrICS probe over 180000 runs

Fig. 3 .

 3 Fig. 3. Software architecture of the eDRON application

Fig. 4 .

 4 Fig. 4. Measurements of inter-partition communication (top) and intrapartition communication services (bottom) for the deployment with interpartition parallelism

  Fig. 5. Measurements of inter-partition communication (top) and intrapartition communication services (bottom) for the deployment with intrapartition parallelism

Fig. 6 .

 6 Fig.6.Host-side automation server, performing 1) selection of target executable and test configuration 2) configuration of hardware counters to use 3) collection of measurements and 4) storage of result files.

Fig. 7 .

 7 Fig. 7. Histogram of the drone partition runtime as appearing in xTRACT visualizer

Fig. 8 .

 8 Fig. 8. Histogram of correlating resource accesses (L2 read cache accesses) as appearing in xTRACT visualizer

Fig. 10 .

 10 Fig. 10. Scatterplot showing linear correlation between runtime and L2 read cache accesses as appearing in xTRACT visualizer

Fig. 11 .

 11 Fig. 11. Visualizing ARINC-653 syscalls in ENGINE R1 task with xTRACT visualizer

TABLE I

 I

	.	RESOLUTION (PERIOD) AND OVERHEAD
	medium	layer	period	frequency	overhead
	p4 get time() kernel	1 ns	n/a	240 ns
	GET TIME() APEX	10 ms	n/a	10 ms
	time base	register 48 cycles 37.5 MHz	1.67 ns
	alt. time base register 1 cycle	1.8 GHz	1.67 ns

TABLE II

 II

	.	EVALUATING DEPLOYMENT IMPACT ON RUNTIME AND
		VARIABILITY OF ONE DRONE PARTITION
	parallelism			runtime (ms)
		min	25%	median	75%	max
	none	16.75 16.76	16.76	16.76 19.56
	inter-partition 16.91 16.98	16.99	17.02 33.92
	intra-partition 55.17 55.18	55.18	55.19 55.31

TABLE III

 III

	.	ARINC-653 COMMUNICATION SERVICES
	level type	write / read
	intra buffer DISPLAY BLACKBOARD()
		READ BLACKBOARD()
	intra fifo	SEND BUFFER()
		RECEIVE BUFFER()
	inter buffer WRITE SAMPLING MESSAGE()
		READ SAMPLING MESSAGE()
	inter fifo	SEND QUEUING MESSAGE()
		RECEIVE QUEUING MESSAGE()

TABLE IV

 IV

	.	NUMBER OF RUNS FOR A FULL CHARACTERIZATION
	Application deployments	4
	Buffer size	3
	Cache Policies	2
	Hardware Counter Selection	C 2 41
	Number of iterations	1000
	TOTAL RUNS	19.68 million

ACKNOWLEDGMENT

The research leading to this work has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 644080 (SA-FURE).