Pierre Boulet

Chantal

Keywords: implementations, 4, 4 Benchmarking server configuration

I dedicate this thesis to my mother, who has never ceased supporting me since my birth and will continue to do so in the future, and to my father, who has always believed in me and encouraged me to challenge myself constantly. I also want to convey my deepest gratitude to my advisors, Romain Rouvoy and Lionel Seinturier, for

Abstract

The ICT sector is claimed to account for 2% of global emissions. Even though this might seem like a small number, the success of ICT technologies will always lead to more greenhouse gas emissions. In 2019, the number of data centers worldwide was 1.5 million, which is expected to reach 2.5 million by 2025. In this situation, lowering the emissions from the ICT sector depends on reducing the amount of energy used by data centers. There are three main methods to achieve this goal: improving the hardware's efficiency, lowering the cooling systems' energy consumption, or decreasing the energy consumption of the servers themselves. This thesis focuses on the last approach, which I believe is the most affordable one, as it does not require any physical changes to data centers. My goal is to assist developers in making more eco-friendly software services by providing them with tools and guidelines to create software that runs on servers while consuming less energy. To do so, I decided to pursue an empirical approach consisting of three steps: test, measure, and optimize.

The reason for such a decision is to follow the rapid pace of the software industry. In fact, the software industry has one of the fastest growth rates, which makes it challenging to keep up with the newest technologies. So, instead of just reporting my insights, I gave practitioners the means and protocols to allow them to test their hypotheses. I believe that some of the insights shared as part of this thesis might already have become obsolete when published.

Due to the urgency of the climate change issue, I decided first to harness the most popular yet energy-hungry programming language, Python. Therefore, I started by analyzing Python's code's energy behavior during its most commonly used cases. Then, I provided a non-intrusive technique to reduce its energy consumption. After that, I extended this strategy to another programming language famous for its legacy code base, Java, to show that we can still reduce the energy consumption of already running applications without paying a considerable price.

Finally, I adopted a more systemic approach. Instead of optimizing one single application, can one reduce the energy consumption of the data center as a whole entity? Thanks to the micro-services architecture, one application can be constructed using many services, each independent of the other. This type of architecture releases us from the bond of adopting a single programming language as the monolithic application does. And with this, one can use multiple programming languages and take advantage of the strengths of each one for a specific scenario. The last chapter analyzed the energy behavior of several programming languages regarding web services while opening a new path toward sustainability within timeless applications.

Résumé

Le secteur des TIC serait responsable de 2% des émissions mondiales. Même si ce chiffre peut paraître faible, le succès des technologies TIC entraînera toujours une augmentation des émissions de gaz à effet de serre. En 2019, le nombre de centres de données dans le monde était de 1,5 million, et devrait atteindre 2,5 millions d'ici 2025. Dans cette situation, la diminution des émissions du secteur des TIC dépend de la réduction de la consommation énergétique des centres de données. Il existe trois méthodes principales pour atteindre cet objectif : améliorer l'efficacité du matériel, faire baisser la consommation énergétique des systèmes de refroidissement, ou diminuer la consommation énergétique des serveurs euxmêmes. Cette thèse se focalise sur la dernière approche, qui me semble être la plus abordable, car elle ne nécessite aucun changement physique dans les centres de données.

Mon objectif est d'aider les développeurs à créer des logiciels plus écologiques en leur fournissant des outils et des directives pour créer des applications qui tournent sur des serveurs tout en consommant moins d'énergie. Pour ce faire, j'ai décidé d'adopter une approche empirique en trois étapes : tester, mesurer et optimiser.

La raison d'une telle décision est de suivre le rythme rapide de l'industrie du logiciel. En fait, le secteur du logiciel connaît l'un des taux de croissance les plus rapides, ce qui rend difficile de suivre les nouvelles technologies. Ainsi, au lieu de me contenter de rapporter juste mes réflexions, j'ai fourni aux praticiens les moyens et les protocoles nécessaires pour leur permettre de tester leurs hypothèses. Je suis convaincu que certaines des conclusions partagées dans le cadre de cette thèse pourraient déjà être obsolètes au moment de leur publication.

Vue l'urgence de la question du changement climatique, j'ai décidé d'abord d'exploiter le langage de programmation le plus populaire et pourtant le plus gourmand en énergie, Python. J'ai donc commencé par analyser le comportement énergétique du code Python dans ses utilisations les plus courantes. J'ai ensuite proposé une technique non intrusive pour réduire sa consommation énergétique. Ensuite, j'ai étendu cette stratégie à un autre langage de programmation célèbre pour sa base de code ancienne, Java, pour montrer que nous pouvons encore réduire la consommation d'énergie d'applications déjà en cours d'exécution sans payer un prix considérable.

Chapter 1 Introduction

Nowadays, computers are invading our daily lives, from work to leisure, from fancy smartphones to embedded peacemakers that regulate the heartbeat of people. As human beings, we are known to use tools to empower our bodies. Moreover, thanks to computers, we pushed that step even further, to the point where now we are using machines to extend our brains, from equation solvers to tools to recommend to us where we should invest our money, what we should eat, and even who fits best as our partner. One significant aspect pushing computers to become omnipresent in our lives is the Internet, a network connecting millions of computers worldwide. According to Internet World Stats, 1 the number of people connected to the Internet has increased by 4.4 billion in 2019, reaching 4.54 billion worldwide, or 59.2% of the world population.

The Internet has evolved from a place where government researchers share information in the 60s to Communication support at the beginning of the century. Now it is where we can find almost anything we want, from information to entertainment, social media to e-commerce.

At present, a large chunk of the global economy and most governments have shifted their operations to the Internet, at least partially and sometimes wholly; this includes online shops, banking, advertising, video and music consumption, and even public functions.

emissions by 2050 to avoid the worst effects of climate change. The IPCC has also warned that the world has to reduce its emissions by 45% by 2030 to reach the 1.5°C target [START_REF] Pörtner | Climate change 2022: Impacts, adaptation and vulnerability[END_REF].

To survive, we have come up with three solutions. The first one includes finding a new planet that we can populate and live on, 2 which is known as the Planetary Migration [START_REF] Mapstone | Cyanobacteria and microalgae in supporting human habitation on mars[END_REF]. Meanwhile, the second solution is to provide new energy sources, such as nuclear energy, wind energy, and even fusion energy [START_REF] Gross | Fusion energy[END_REF]. The third solution is to reduce our emissions, which is the main focus of this thesis.

While there are many fields where one can optimize energy consumption. Our focus is on the Information and Communication Technology (ICT) sector, which is expected to account for around 4% of global GreenHouse Gas (GHG) emissions in 2020, with an alarming 8% growth rate, according to the French think tank The Shift Project. 3 According to Statista, the energy consumption of ICT increased from 4.3 exajoules in 2018 to 5.8 exajoules in 2025. 4Figure 1.1: Final energy consumption of digital technologies by item in 2019(The Shift Project -Forecast Model 2021 [START_REF] Ferreboeuf | Impact environnemental du numérique et gouvernance de la 5g[END_REF]) Figure 1.1 shows the distribution of ICT consumption in 2018, where the largest chunk of the energy consumption is due to data usage, aka 63% of the energy consumed where 22% of this energy is used by data centers. Reducing energy consumption means reducing the impact 14% of the ICT energy consumption has on the environment.

In 2020, the market for data center services was worth 48.9 billion$. It is thought that this number will go up to 105.6 billion$ by 2026 [START_REF] Inshakova | Data centers: Market trends and contribution to the world economy development during the covid-19 pandemic[END_REF]. This growth is caused mainly by:

• shift to remote lifestyle: work, education, and entertainment,

• increase in the number of connected devices (IoT),

• development of data-hungry technologies such as Machine learning, AI, Big data, and so on,

• edge computing and 5G.

With this increase in the number of data centers comes an increase in energy consumption, which is a significant environmental problem. In 2018, data centers consumed around 205 terawatt-hours (TWh) [START_REF] Schneider | The world nuclear industry status report 2019[END_REF], equivalent to the energy consumption of 1% of the entire world's electricity. This ratio increased up to 1.5% in 2020 according to the Journal of Science [START_REF] Mytton | Data centre water consumption[END_REF]. In Figure 1.2, one can see that 40% of the energy consumed by data centers is used for cooling, while another 40% is used by the servers themselves. Therefore, optimizing these two aspects can significantly impact the energy consumption of data centers. Figure 1.2: Distribution of power consumption in a data center [START_REF] Rong | Optimizing energy consumption for data centers[END_REF] Researchers are trying to reduce the energy consumption of data centers through different angles. Some of the works are focused on the hardware side, such as using new hardware architectures that are more energy-friendly, such as using GPUs instead o ARM processors instead of CPUs [START_REF] Aroca | Towards green data centers: A comparison of x86 and arm architectures power efficiency[END_REF]. Others are trying to optimize the cooling system; this can be achieved by using more efficient cooling systems, putting data centers in cold locations or under water [START_REF] Simon | Project natick-microsoft's self-sufficient underwater datacenters[END_REF], or even using the waste heat for other purposes, such as heating buildings [START_REF] Bouzel | Distributed grid computing manager covering waste heat reuse constraints[END_REF][START_REF] Cao | Carbon neutrality in data center[END_REF].

A third approach is to optimize the software by making software more energy-efficient. In this thesis, we focus on this approach and try to optimize the software by reducing the number of computations done by the software.

The best way to do so is to formulate a theory behind the energy consumption of algorithms, such as the complexity and the o notation. Unfortunately, this is not possible in the current state of the art. Due to the lack of knowledge about the energy consumption of the algorithms and the strong correlation between this consumption and the hardware configuration. Unlike algorithm optimization in the field of performance, which is agnostic toward the platform, the energy consumption of the algorithms is dependent on the execution environment. Therefore, for the moment, we start by formulating some hypotheses and exploring them using empirical analysis. Figure 1.3 highlights this thesis's position on the sustainability of ICT; while this thesis only addresses a small portion of ICT's energy usage, we feel it is a step in the right direction for additional solutions to mature to preserve humanity.

Objectives

The purpose of this thesis is to help developers build more energy-efficient software. Unfortunately, when it comes to the energy consumption of programs, there is a lack of awareness and knowledge among software developers [START_REF] Ournani | On reducing the energy consumption of software: From hurdles to requirements[END_REF][START_REF] Pang | What do programmers know about software energy consumption[END_REF][START_REF] Pinto | Mining questions about software energy consumption[END_REF]. This is mainly due to the lack of tools that can help developers understand the energy consumption of their programs. Therefore, we aim to provide clear, understandable, and easy-to-use tools and guidelines that can help developers reduce the energy consumption of their programs.

To reduce the energy consumption of software, we can use three strategies. The first one consists of reducing energy consumption by reducing the execution time of the program. The most intuitive way for developers, as the optimization of software's performance, is a key metric for most programs. Therefore, in this case, reducing the energy consumption of software becomes a side effect of optimizing the performance of the software.

The second strategy is optimizing the energy while trying to keep performance. In this strategy, we aim to reduce the energy consumption of software without impacting its performance of the software. This is a more challenging task, as it requires a deeper understanding of the software's energy consumption. The main focus of this thesis is to provide tools and guidelines that can help developers in this task.

The final strategy consists of deliberately reducing the performance of the software to reduce its energy consumption. The goal of this method is to reduce energy usage by sacrificing execution time. This strategy is helpful while dealing with services since the task has an unlimited execution time. The second part is to use some opportunistic schedulers, which will stop the software if it is estimated to spend more energy. This strategy is discussed first in Chapter 6, then we continue this strategy in the perspectives section.

We are here !!! .4 summarizes the work completed for this thesis As mentioned in the previous section, our goal is to help developers reduce the energy consumption of their programs during execution. We use an empirical approach to accomplish this. As a result, we require some tools to assist us in this task. To begin, we provide a tool to assist developers in measuring the energy consumption of their programs. The rest of the manuscript is outlined below. • Chapter 2 provides a literature review of the energy consumption of software. First, it introduces the challenges that one can encounter when running an empirical study, reproducibility, accuracy, and representativeness, the ways in the state of the art overcome these challenges in the field of computer sciences. Then, it narrows down to the energy consumption of software and the challenges that one can encounter when measuring the energy consumption of software. In this section, we provide the literature approaches used to measure energy consumption and classify them, compare them and discuss their pros and cons when used for our research. After that, we tackle the challenge of accuracy within the field of energy consumption and discuss how the literature tried to overcome these challenges. Finally, we present some of the most recent works in the field of software energy consumption optimization;

• Chapter 3 discusses two facets of empirical tests and how we adapt them to energy consumption measurements. First, we discuss the reproducibility challenge and how to overcome it with the usage of containers; then we push this approach further by proposing and new protocol to make tests not only reproducible but able to be extended with other features to overcome the rapid pace of software evolution. The second part of this chapter discusses the aspect of accuracy regarding energy consumption measurements. This section demonstrates how energy measurements might differ and produce conflicting results for the same work when run on identical machines or even the same machine. Then, it provides some solutions to reduce this energy variation and improve the accuracy of the measurements;

• After setting the ground for energy benchmarks in the previous chapter, we discuss the energy consumption of Python, one of the world's most popular programming languages. Chapter 4 reports on how much python costs in terms of energy consumption compared to other languages. Then, it provides some statistics about its popularity and typical use cases and studies the impact of Python on energy consumption in three typical use cases, machine learning, web servers, and data manipulation. For each use case, we compare the energy consumption of several approaches and provide guidelines on optimizing the energy consumption of python programs in these use cases. Finally, we provide a non-intrusive way to optimize energy consumption without altering the program's code. We achieve this by using a different implementation of the Python interpreter. This not only allows developers to spend less time optimizing their code but also allows them to use it on the legacy code that they cannot modify;

• In contrast, Chapter 6 uses the flexibility provided by the micro-services architecture [START_REF] Dmitry | On micro-services architecture[END_REF] to analyze each programming language's energy behavior in light of various web scenarios to optimize the energy use of web services. We first examine the effects of the various programming languages when dealing with the Remote Procedure Call (RPC) Protocol. In this instance, we use two scaling factors: the number of concurrent clients and the size of the requests. Then, we compare 261 web frameworks, each implementing the same website using seven use cases. This analysis aims to look at how each technology uses energy in different web situations.

• Finally, we conclude our work in Chapter 7 by summarizing the work done in this thesis and discussing the future work that can be done to reduce the energy consumption of software further.

Contributions

The contributions of this thesis are summarized as follows:

Conferences 1. Ournani, Z., Belgaid, M. C., Rouvoy, R., Rust, P., Penhoat, J., and Seinturier, L. (2020a).

Taming energy consumption variations in systems benchmarking. In Proceedings of the ACM/SPEC International Conference on Performance Engineering, pages 36-47, 2. Ournani, Z., Belgaid, M. C., Rouvoy, R., Rust, P., and Penhoat, J. (2021). Evaluating the impact of java virtual machines on energy consumption. In Proceedings of the 15th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pages 1-11.

Tools

• Jouleit (github.com/powerapi-ng/jouleit): a tool that can be used to monitor energy consumption for any Linux program, this tool was used to compare the energy consumption of different JVMs; Belgaid, M. c., Rouvoy, R., and Seinturier, L. (2020). Jouleit : a tool that can be used to monitor energy consumption for any linux program,

• JRefferal (github.com/chakib-belgaid/jreferral): a tool that allows the user to explore the JVM settings and their impact on the energy consumption of a given Java program.

This tool was the result of the second article of this thesis; Belgaid, M. c., Ournani, Z., Rouvoy, R., and Seinturier, L. (2021b). JRefferal: Which JVM suits your project ?,

• PyJoules (pypi.org/project/pyJoules) is a software toolkit to measure the energy footprint of a host machine along the execution of a piece of Python code. It can measure the energy consumption on the level of script, function, and bloc of code; Belgaid, M. c., Rouvoy, R., and Seinturier, L. (2019). Pyjoules: Python library that measures python code snippets,

• JouleHunter (pypi.org/project/joulehunter): an energy profiler for python applications.

It can be used to highlight the functions that consume the most energy in a given Python program. Its main usage is to help developers do an exploratory analysis of their application to scope the functions that should be optimized to be then targeted by PyJoules; Belgaid, M. c., Kaminetzky, A., Rouvoy, R., and Seinturier, L. (2021a). JouleHunter : an energy profiler for python applications,

• GreenBoard (github.com/chakib-belgaid/greenboard): a dashboard designed to help developers choose the best stack for their web application. It is based on the results of the third article of the last chapter.

Chapter 2

State of the Art 2.1 Introduction

Efficiency in energy usage is a well-known topic. In most fields, the purpose is to minimize the energy consumption of electrical devices and components. Modern times even see energy classification (A, B. . . F) for homes, cars, and electronic products to provide the consumer an indication of the energy consumption of their devices, which will reflect on their power bill.

This criterion is extended even to the hardware components of a computer. Figure 2.1 compares Intel CPUs i9-12900KS and i9-12900KF. 1 The difference between these two CPUs is that the KS has an unlocked multiplier, allowing it to be overclocked. As a result, the basic consumption is less than the KF. This statistic also estimates the average power consumption of these two CPUs each day, as well as the monetary equivalent, to make people more aware of the values of energy consumption rather than the raw data.

In computer science, the objective is essentially the same. Numerous studies have been conducted on energy optimization. Some of these studies concentrate on minimizing energy consumption at the hardware level, while others optimize energy consumption via software.

As an example, Avgerinou et al. evaluated the development of power use effectiveness (PUE) in data centers that belongs to various organizations participating in the European code of conduct for energy efficiency program [START_REF] Avgerinou | Trends in Data Centre Energy Consumption under the European Code of Conduct for Data Centre Energy Efficiency[END_REF]. The research found a gradual decline in the PUE of data centers, which measures the ratio of the overall energy supplied to the energy used by IT equipment. A low PUE implies that the majority of energy is utilized to power the data center's IT equipment, while just a small amount is needed for cooling and lighting. We will place a greater emphasis on the software level to decrease the amount of energy that is used, more particularly in the execution phase of the program cycle. We will be proceeding through an empirical analysis of the energy consumption of the software while changing some components of the source code without impacting its behavior. To do this, we will elaborate on a benchmarking process and a set of tools intended to assist practitioners in better comprehending and optimizing the energy usage of their applications. Thus, we will begin by examining the state of the art in empirical analysis and retrieving the best empirical experimentation methodologies in the research field. Then, we will narrow these practices down to computer science so we can finally adapt them to energy consumption. Section 2.2 will discuss the pitfalls and best practices associated with empirical research before applying them to our field of interest. After that Section 2.3 describes software energy measurements. It provides examples of hardware and software measuring instruments and describes their differences, benefits, and drawbacks. It also examines the sources of energy measurement variations, representing a significant obstacle to achieving precise readings and higher accuracy. Then, in Section 2.5, we will go through some of the previous work on improving the energy consumption of software.

Benchmarking

This section will go through the flaws and best practices of empirical research before applying them to our topic of study.

Threats & Challenges

A successful benchmark must meet three criteria. First, it must be reproducible for others to imitate it. Second, the findings should be accurate, which implies that we should expect the same results each time we run the benchmark. Finally, it should represent reality. In other words, the experiment's findings should also be applicable outside the research lab. The aim of representativeness in this thesis is the manufacturing environment. As a result, the experiments should reflect what happens in production contexts.

Reproducibility

Experiment reproducibility is frequently listed as one of the most difficult challenges researchers face. Reproducing an experiment has been one of the significant tools science has used to help establish the validity and importance of scientific findings since the Philosophical Transactions of the Royal Society were established in 1665 [START_REF] Hankins | A debate over experiment: Leviathan and the air-pump. hobbes, boyle, and the experimental life. steven shapin and simon schaffer. including a translation of hobbes's dialogus physicus de natura aeris by simon schaffer[END_REF]. Many of the outcomes are not reproducible, 2 which led to a replication crisis. As a result of the crisis affecting the majority of empirical studies, most reviews now include reproducibility as a minimum standard for judging scientific merit [START_REF] Peng | Reproducible research in computational science[END_REF]. One of the criteria for supporting reproducibility is the publication of the dataset and the algorithms run on the raw data to derive the results. There is even some disagreement about what the terms "reproducibility" or "replicability" by themselves mean [START_REF] Goodman | What does research reproducibility mean?[END_REF]. According to [START_REF] Echtler | Open source, open science, and the replication crisis in hci. association for computing machinery[END_REF], replicability extends reproducibility with the ability to collect a new raw dataset comparable to the original one by re-executing the experiment under similar conditions, instead of just the ability to get the same results by running the statistical analyses on the original data set.

Accuracy

According to Oxford, accuracy means "technical The degree to which the result of a measurement, calculation, or specification conforms to the correct value or a standard". In our case, this means the ability to run the benchmark multiple times with low variation. This can be achieved by controlling the experiment environment, allowing less room for random factors.

In biology, chemistry, and electronics, they use clean rooms, which are environments where pollutants like dust, airborne microbes, and aerosol particles are filtered out and factors like humidity, airflow, and temperature can be regulated. As for empirical analysis, the accuracy can be measured by numeric metrics such as the variance, the standard deviation (STD), and the interval inter quartile (IRQ). Section 2.4 will go over the accuracy in the subject of energy optimization.

Representativeness

As obvious as it seems, the reason for executing benchmarks is to validate ideas so we can use them in real life. However, this means that those benchmarks have to represent reality somehow. In Social sciences, this can be achieved by selecting a representative sample size. Omair et al. presents a guideline on achieving such representativeness. As for computer science, the field of benchmarking is still in its infancy, and there is no consensus on how to achieve representativeness. However, many attempts have been made to provide a set of benchmarks for a specific purpose. For example, the Standard Performance Evaluation Corporation (SPEC) provides a set of benchmarks for CPU performance evaluation. This sets covers a wide range of use cases such as CPU17 for testing the CPU 3 , SPECviewperf 4for Graphic usage and one can cite StressNg when it comes to benchmark the hardware components, the SPEC benchmark when it comes to benchmark the software performance, and SPECPower 5 . NASA Parallel Benchmarks (NPB) 6 and HPCchallenge7 are two other examples of benchmarking sets that are created to represent the high performance computing. As for programming languages we can cite Computer Language Benchmarks Game8 , which is a collection of benchmarks for various programming languages. The benchmarks are designed to be minor, self-contained, and easy to implement in any language. The benchmarks are also made to represent most of the typical real-world workloads in an isolated manner. Dacapo [18] and renaissance [123] are other examples of benchmarking sets that are created to represent the Java Virtual Machine (JVM) performance. On the other hand, a new sort of test has arisen within the software development life cycle. This type is known as stress testing, and it is used to assess the software's robustness and reliability before releasing it to the public. gatling 9 and TCPCopy10 are great examples of stress testing tools that are used to test the performance of server applications.

Impact of these Challenges in the Empirical Research

In their work [START_REF] Van Der Kouwe | Benchmarking Crimes: An Emerging Threat in Systems Security[END_REF], Van-der-Kouwe et al. investigated 50 papers published in top venues to find out that Tier-1 papers commit an average of five benchmarking crimes. To analyze the magnitude of the phenomenon, they have identified a set of 22 "benchmarking crimes" that threaten the system's validity.

Proposed Solutions

Researchers have proposed several solutions to overcome these challenges in the computer science field. We will discuss some of them below. First, we start with the work of Mytkowicz et al., where they evaluated 133 studies from ASPLOS, PACT, PLDI, and CGO, to find out that none of the experimental findings papers appropriately considered measurement bias. Which can lead derive incorrect results from an experiment if a seemingly insignificant feature of the experimental design is altered. They treated this problem by proposing two strategies for detecting measurement bias by using causal analysis and preventing it with setup randomization [START_REF] Mytkowicz | Producing wrong data without doing anything obviously wrong![END_REF]. Another study that was published in the book "Measuring computer performance: a practitioner's guide" [START_REF] Lilja | Measuring computer performance: a practitioner's guide[END_REF],Lilja examined performance indicators and gave an in-depth treatment of benchmark program tactics. They clearly explained the basic statistical methods required for interpreting measured performance data. They also outlined the overall design of the experimental method and demonstrated how to collect the most information with the least amount of work. This practical book will appeal to anybody seeking a comprehensive yet intuitive grasp of computer system performance analysis.

Bukh wrote a book about computer performance analysis, where they discussed some familiar topics that are relevant to statistical analysis, such as null hypotheses, chi-squared tests, regression, discrete event simulation, Bayes' theorem, how and when to use them for experimental design, measurement, simulation, and modeling for computer systems. The article [START_REF] Brown | Issues with data and analyses: Errors, underlying themes, and potential solutions[END_REF] provides an intellectual framework for understanding the pervasiveness of mistakes in the scientific discovery process and presents methodological, cultural, and system-level techniques for minimizing the frequency of often-seen errors.

Another article [START_REF] Patil | Training replicable predictors in multiple studies[END_REF] expands on this line of thought by evaluating the uncertainty caused by replications in the new research. They offered some strategies to capture uncertainty in inferential investigations, such as cross-study validation and ensemble models.

In their paper [START_REF] Stodden | An empirical analysis of journal policy effectiveness for computational reproducibility[END_REF], the authors found that, even though it's a big step in the right direction, journal policies that require authors to give back digital scholarly objects after publication, like the data and code that back up the claims, do not get more than half of these objects back. Then, using these artifacts, about a fourth of the published computational claims in the study could be made. They suggested putting out the claims in the literature and the digital scholarly objects that back them up simultaneously.

Figure 2.2: The proposed decomposition of an experiment [16] Finally, to unify the benchmarking methodology across different research works in the field of computer science, we can cite the paper [16]. In their approach, they divided any empirical experiment into four components. Figure 2.2 presents these components, which are:

1. measurement contexts indicates the software and hardware components that will alter or remain constant during the experiment.

2. workloads identify the benchmarks to use in the experiment, as well as their inputs;

3. metrics specify the attributes to be measured and how to assess them.

4.

Data analysis shows how to examine data and evaluate the outcomes of the analysis to offer insight into the assertions that arise from the study.

The work of this thesis will be based on this approach since it provides a unified methodology for benchmarking and evaluating the performance of different systems.

Energy Measurement

Now that we have discussed the importance of benchmarking and the different approaches that can be used to evaluate the performance of a system, we will focus on energy measurement, which will be the main metric in this thesis. Therefore, in this section, we will discuss the different approaches that can be used to measure the energy consumption of a system. Many studies have been conducted to estimate such energy consumption that varies from static analysis of the source code to infer its energy consumption like Pereira et al. where they provided a tool to highlight the most energy consuming parts of the code [START_REF] Pereira | Helping Programmers Improve the Energy Efficiency of Source Code[END_REF]. The essential advantage of this approach is that it allows practitioners to estimate a program's energy usage without executing it. Unlike program complexity, energy consumption is strongly dependent on the execution environment. As a result, static analysis may not accurately represent the behavior of the same program when run in a production context. To address the issue of representativeness, many researchers measure the energy consumption of programs as they run. As a result, we will get more accurate results. There are various tools for measuring energy, and they cover a wide range of applications depending on how accurate and precise the results must be on the one hand and the price that practitioners are prepared to pay for such accuracy and precision on the other.

According to Hackenberg et al., there are four main criteria to evaluate an energy measurement tool [START_REF] Hackenberg | Hdeem: high definition energy efficiency monitoring[END_REF]:

• Spatial granularity: the more specific the target of monitoring we can measure, the more efficient we can do optimization since we will know what causes the pitfalls of the energy consumption,

• Temporal granularity: same as spatial granularity, temporal granularity helps us to identify the sequence of code that need to be optimized,

• Scalability: this is mainly related to the cost of the tools and the ease of their integration into our system,

• Precision: to eliminate extra hazards and get a more representative measurement.

We believe that precision can be extracted from these criteria by combining the first two results. Therefore, we will focus more on the first three criteria later on. Below, we will discuss some of the well-known tools used in literature.

Hardware Tools

Nowadays, most high-performance computing systems (HPC) implement a tool to report the nodes' energy consumption for monitoring and administration. Those tools are mainly integrated within the power supply units (PSU) or the power distribution units (PDU). Then, they provide an interface and a log to follow the energy consumption history. Despite their scalability and ease of integration, such tools lack both spatial and temporal granularity since they monitor the whole energy of the nodes, and most of the time, they have a shallow sampling frequency. Most of those tools are provided directly by the manufacturers. Such as IBM EnergyScale technology [START_REF] Mccreary | Energyscale for ibm power6 microprocessor-based systems[END_REF][START_REF] Caldeira | Ibm power systems s814 and s824 technical overview and introduction[END_REF]Caldeira et al.] or Dell poweredge [START_REF] Lovicott | Thermal design of the dell™ poweredge™ t610™, r610™, and r710™ servers[END_REF], MEGware Cluststafe [START_REF] Breitbart | Case study on co-scheduling for hpc applications[END_REF]. As said earlier, the true purpose of those tools is more monitoring than analyzing energy consumption. WattsUp Pro is a device that can be installed between the power source of the machine and the system under test. It allows a sampling frequency of up to 1 Hz. It has an internal memory to store a wide variety of data, such as the maximum voltage and current, that can later be exported via a USB port for personal usage or lined to graph programs like Logger Pro or LabQuest. The main advantage of this tool is the ability to monitor the energy consumption from a different device which will reduce the risk of interference with the energy consumption of the test [START_REF] Hirst | Watts up? pro ac power meter for automated energy recording[END_REF] Despite its high temporal sampling, WattsUp Pro lacks spatial granularity since it monitors the energy consumption of the whole system. To have finer granularity, we need to isolate the energy consumption of each component.

PowerMon and its upgraded version powerMon2 [10] are based on a micro-controller chip that can simultaneously monitor up to 6 channels (8 for powermon2). Therefore, we can monitor the power consumption of 4 devices at the same time. The frequency sample of this tool is up to 50 Hz, with an accuracy of 1.2%. Powermore2 is smaller and can fit within 3.5 inches rack drive.

PowerInsight [START_REF] Laros | Powerinsight-a commodity power measurement capability[END_REF] is another fine-grained measurement tool that is based on an ARM BeagleBone processor [START_REF] Coley | Beaglebone rev a6 system reference manual[END_REF], which can measure up to 30 channels simultaneously with a frequency of 1KHz per channel. powerpack [START_REF] Ge | Powerpack: Energy profiling and analysis of high-performance systems and applications[END_REF] in the other hand, is an API that synchronizes the data gathered from other monitoring tools such as Watt's Up Pro, NI and RadioShack pro and the lines of code.

Other monitor tools have been realized by the manufacturers, such as IBM Power executive [START_REF] Koomey | Growth in data center electricity use 2005 to 2010. A report by Analytical Press[END_REF], which allows their customers to monitor the power consumption and thermal behavior of the BladeCenter systems in the data center.

Accoring to the work of Vasques et al. andWang et al. The CPU is the part responsible for the most energy consumption in a data center [142,[START_REF] Wang | Modelling and developing conflict-aware scheduling on large-scale data centres[END_REF]. Hence, the finer we go to measure this energy consumption, the better it is for our work. Fortunately, Intel and later AMD proposed a tool that estimates the power consumption of different parts of the CPI based on counter performances. RAPL (RUnning Average Power Limit) [START_REF] Hackenberg | Power measurement techniques on standard compute nodes: A quantitative comparison[END_REF][START_REF] Hackenberg | An energy efficiency feature survey of the intel haswell processor[END_REF] is a set of registers that Intel introduced in their CPU since Sandy bridge generation, and later it was followed by AMD since Family 17h Zen. 2.3 shows the different scopes that can be monitored using RAPL. The CPU package can be monitored for both server and desktop processors. However, DRAM is only available on server CPUs, while the integrated GPU is only available on desktop processors. The advantage of such an approach is the absence of intrusive measurement tools. Furthermore, they have a high temporal granularity with a sampling frequency of up to 1 KHz [START_REF] Ilsche | Power measurements for compute nodes: Improving sampling rates, granularity and accuracy[END_REF].

With a similar approach, we can find NVIDIA reporting tools, such as GPU TESLA [START_REF] Burtscher | Measuring gpu power with the k20 built-in sensor[END_REF] and the NVML library [START_REF] Fahad | A comparative study of methods for measurement of energy of computing[END_REF].

Software Tools

Software-based measurement tools are based on other hardware tools to monitor energy use. Granularity is the core value of these technologies, unlike hardware tools, which only provide the total energy usage of the system/component (computer, server, motherboard, etc.) in most cases. Because they are frequently constructed on empirical estimations and data learning methods, they drop in accuracy.

Many software measuring tools understand a power model's behavior and estimate energy usage. This model is then used to allocate the observed energy consumption among different execution entities, such as processes, control groups, threads, or code lines.

The first examples of software measurement tools are PowerAPI [START_REF] Colmant | The next 700 cpu power models[END_REF], SmartWatts formula [START_REF] Fieni | Smartwatts: Self-calibrating softwaredefined power meter for containers[END_REF] and SelfWatts [START_REF] Fieni | Selfwatts: On-the-fly selection of performance events to optimize software-defined power meters[END_REF]. These tools collect global energy consumption measurements from RAPL and use other system events such as cache misses/hits and CPU frequency scaling (DVFS) via a sensor to construct a power model of the control groups (system control groups, docker containers, Kubernetes pods, etc.) using a Ridge regression. The model continuously learns and improves its real-time energy usage data precision with a maximum frequency of 100 Hz. The instrument has a decentralized, lightweight design. Only the lightest sensors required for data collection and transmission are put into the monitored devices. The SmartWatts formula is then executed on the primary server to construct the model that permits assigning the energy usage for each functional control group. PowerAPI is only compatible with Linux on a bare-metal physical computer. WattWatcher [START_REF] Lebeane | Watt watcher: fine-grained power estimation for emerging workloads[END_REF] is a multi-core power measuring framework that provides process-level energy measurements. This program uses power models to predict process energy usage. It uses CPU events passed from the measured node to a model generator node to construct the power model. It works by combining a description of the CPU with a list of the hardware events through multiple calibration phases to build a robust model.

Joulemeter [START_REF] Kothari | Joulemeter: Virtual machine power measurement and management[END_REF][START_REF] Jagroep | Energy efficiency on the product roadmap: an empirical study across releases of a software product[END_REF] is a Microsoft software that estimates the energy usage of Windows running applications down to the process level by using power models (for CPU, memory, and drives). It employs low-overhead power models to infer power consumption from resource utilization during runtime and provides power-limiting features for virtual machines. Previous Joulemeter tests [START_REF] Jagroep | Profiling energy profilers[END_REF] demonstrated that the instrument provides a less accurate estimation of energy use that differs greatly from the real one. To adjust its models to the hardware on which it operates, Joulemeter must first go through a calibration step. It can only monitor one process at a time with a frequency of 1 Hz.

JRAPL is another example of an energy measurement tool estimating tool that has been utilized in a variety of publications [START_REF] Guimarães | Some heuristic approaches for reducing energy consumption on database systems[END_REF][START_REF] Liu | Data-Oriented Characterization of Application-Level Energy Optimization[END_REF]. This software enables the energy usage of Java programs, functions, or even a block of code lines to be profiled and measured. The measurements are heavily reliant on the data supplied by RAPL. As a result, the global energy consumption collected by RAPL between two timestamps (the start and finish of the code to measure) is used to calculate the energy consumption of the Java code. Tests using jRAPL should be done on a well-configured machine to minimize the impact of the operating system and user operations on the overall energy consumption of jRAPL.

Another process-level energy usage measuring tool is Jolinar [START_REF] Islam | Measuring energy footprint of software features[END_REF][START_REF] Noureddine | Jolinar: analysing the energy footprint of software applications[END_REF]. The tool does not need a calibration phase and relies on pre-established power models based on hardware metrics like TDP, disk I/O rate, and CPU frequency. These settings must be identified and supplied by the user for his machine. Jolinar can only measure one application's energy consumption at a time. At the end of the execution, the tool provides the process's CPU, DRAM, and disk energy usage. Jalen [START_REF] Noureddine | Monitoring energy hotspots in software[END_REF] is another tool that profiles and monitors the energy usage of a Java program. Unlike jRAPL, Jalen can cover the scope down to the function level. It gathers data using code instrumentation and statistical sampling at a predetermined pace. Because of the overhead that code instrumentation may incur, the authors recommend utilizing the second option. Every 10 ms, Jalen records the JVM's stack trace together with the CPU time of threads and computes statistics about method calls. These statistics are then utilized to calculate each method's energy usage.

Energy Variations

In theory, using an identical CPU, the same memory configuration, and similar storage and networking capabilities should increase the accuracy of physical measurements. Unfortunately, this is not possible when it comes to measuring the energy consumption of a system. Applying the benchmarking guidelines and repeating the same experiment within the same configuration is insufficient to reproduce the exact energy measurements between identical machines and even within the same machine. This difference-also called energy variation (EV)-has seriously threatened the accuracy of experimental evaluations. Figure 2.4 illustrates this variation problem as a violin plot of 20 executions of the benchmark Conjugate Gradient (CG) taken from the NAS Parallel Benchmarks (NBP) suite [START_REF] Bailey | The nas parallel benchmarks—summary and preliminary results[END_REF], on 4 nodes of an homogeneous cluster (the cluster Dahu described in Table 3.1) at 50 % workload. We can observe a considerable variation of energy consumption, not only among homogeneous nodes but also at the scale of a single node, reaching up to 25 % in this example. Some researchers started investigating the hardware impact of the energy variation of power consumption. As an example, one can cite [19,[START_REF] Tschanz | Adaptive body bias for reducing impacts of die-to-die and within-die parameter variations on microprocessor frequency and leakage[END_REF] who reported that the leading cause of the variation of the power consumption between different machines is due to the CMOS manufacturing process of transistors in a chip. [START_REF] Heinrich | Predicting the Performance and the Power Consumption of MPI Applications With SimGrid[END_REF] described this variation as a set of parameters, such as CPU Frequency and the thermal effect.

Studying Hardware Factors

This variation has often been related to the manufacturing process [START_REF] Coles | Comparing Server Energy Use and Efficiency Using Small Sample Sizes[END_REF]. However, it has also been the subject of many studies, considering several aspects that could impact and vary the energy consumption across executions and on different chips. On the one hand, the correlation between processor temperature and energy consumption was one of the most explored paths. Kistowski et al. showed in [START_REF] Joakim V Kisroski | Variations in CPU Power Consumption[END_REF] that identical processors could exhibit significant energy consumption variation with no close correlation with the processor temperature and performance. On the other hand, the authors of [START_REF] Wang | Potential effects on server power metering and modeling[END_REF] claimed that the processor thermal effect is one of the most contributing factors to the energy variation, and the CPU temperature and the energy consumption variation are tightly coupled (up to 16% increase in the variation when the temperature changed from 37.7°C to 74.5°C ).

This exposes the processor temperature as a delicate factor when comparing energy consumption variations across homogeneous processors.

The ambient temperature was also discussed in many papers as an essential candidate factor for the energy variation of a processor. In [140], the authors claimed that energy consumption might vary due to fluctuations caused by the external environment. These fluctuations may alter the processor's temperature and energy consumption. However, the temperature inside a data center does not show significant variations from one node to another.

In [START_REF] El Mehdi Diouri | Your cluster is not power homogeneous: Take care when designing green schedulers![END_REF], El Mehdi Dirouri et al. showed that switching the spot of two servers does not affect their energy consumption. Moreover, changing hardware components, such as the hard drive, the memory, or even the power supply, does not affect the energy variation of a node, making it mainly related to the processor. This result was recently assessed by [START_REF] Wang | Potential effects on server power metering and modeling[END_REF], where the rack placement and power supply introduced a maximum of 2.8 % variation in the observed energy consumption.

Beyond hardware components, the accuracy of power meters has also been questioned. Inadomi et al. [START_REF] Inadomi | Analyzing and mitigating the impact of manufacturing variability in power-constrained supercomputing[END_REF] used three different power measurement tools: RAPL, Power Insight11 and BGQ EMON. All three tools recorded the same 10 % of energy variation that was supposedly related to the manufacturing process.

Mitigating Energy Variations

Acknowledging the energy variation problem on processors, many papers proposed contributions to reduce and mitigate this variation. In [START_REF] Inadomi | Analyzing and mitigating the impact of manufacturing variability in power-constrained supercomputing[END_REF], the authors introduced a variation-aware algorithm that improves application performance under a power constraint by determining module-level (individual processor and associated DRAM) power allocation, with up to 5.4× speedup. The authors of [START_REF] Hammouda | Noise-Tolerant Explicit Stencil Computations for Nonuniform Process Execution Rates[END_REF] proposed parallel algorithms that tolerate the variability and the non-uniformity by decoupling per process communication over the available CPU. Acun et al. [4] found a way to reduce the energy variation on Ivy Bridge and Sandy Bridge processors by disabling the Turbo Boost feature to stabilize the execution time over a set of processors. They also proposed some guidelines to reduce this variation by replacing the old-slower-chips, load balancing the workload on the CPU cores, and leaving one core idle. They claimed that the variation between the processor cores is insignificant. In [START_REF] Chasapis | Runtime-Guided Mitigation of Manufacturing Variability in Power-Constrained Multi-Socket NUMA Nodes[END_REF], the researchers showed how a parallel system could deal with the energy variation by compensating for the uneven effects of power capping.

In [START_REF] Marathe | An empirical survey of performance and energy efficiency variation on Intel processors[END_REF], the authors highlight the increase of energy variation across the latest Intel micro-architectures by a factor of 4 from Sandy Bridge to Broadwell, a 15 % of run-to-run variation within the same processor and the increase of the inter-cores variation from 2.5 % to 5 % due to hardware-enforced constraints, concluding with some recommendations for Broadwell usage, such as running one hyper-thread per core.

Energy Optimizations

,

We will now focus on the energy optimization challenge after considering the various ways of measuring energy consumption in computers and understanding the energy variation problem. Over the previous decade, there has been considerable interest in this field, with many papers proposing different approaches to reduce the energy consumption of software applications. This section will pass through the main contributions in this field and focus on the following two parts.

Energy Optimization in the Design Phase

The first part of this section will focus on energy optimizations in the conception phase. The goal is to make the final product use less energy by choosing the best programming languages, tools, libraries, etc. It also includes all the work and optimizations that developers can do to the source code to make the software use less energy when running.

We start with the work of Pereira et al. where they did an energy consumption comparison analysis of the most popular programming languages. The paper recommends combining some of these languages to enhance code quality while considering execution time, memory utilization, and energy consumption using the Pareto optimum [START_REF] Hochman | Pareto optimal redistribution[END_REF]. Some of the study's findings demonstrate that interpreted languages, such as Python, have lower energy efficiency than compiled languages, such as C or Rust. The research also offers language combinations that developers might use to improve energy economy, execution speed, and memory utilization. This paper's findings were based on the game benchmark 12 , the most famous benchmark comparing several programming languages.

Couto et al. investigated the influence of programming language choice on the energy consumption of software during execution. In their paper [START_REF] Couto | Towards a green ranking for programming languages[END_REF], they examined a set of computing problems written in ten well-known programming languages while observing the energy required when running each language. They also found exciting situations where slower languages use less energy than faster ones, even though fast languages usually use the least energy. Finally, they produced an energy efficiency rating of programming languages. The paper [START_REF] Bujnowski | Java and kotlin code performance in selected web frameworks[END_REF] compared the energy, performance, and database response time of web applications written in Java versus those written in Kotlin. They discovered no statistically significant difference in CPU load between individual measurements( less than 2%) 2. However, Kotlin implementation has never earned the best results in any collection of measurements.

Other works have studied the impact of website technology on energy consumption, [118,[START_REF] Manotas | Investigating the impacts of web servers on web application energy usage[END_REF]. In their work [118],Philippot et al. measured the computer resources used during the loading of a website in a browser, such as memory utilization and energy consumption, for over 500 websites and proposed some best practices for developers.

As for the impact of the source code on energy consumption, we can cite [START_REF] Pinto | A Comprehensive Study on the Energy Efficiency of Java's Thread-Safe Collections[END_REF][START_REF] Fernandes | Assisting Non-Specialist Developers to Build Energy-Efficient Software[END_REF] where the authors investigated the impact of Java collections on energy usage based on collection size and its usage (insertion, removal, search) and They provide some insights into the energy efficiency of specific collections under various scenarios.

Hasan et al. examined the energy consumption of multiple Java data structures, analyzing the bytecode and evaluating the change in energy consumption in various circumstances(research, insertion, deletion, etc.). They also simulated best-and worst-case energy usage scenarios in real-world production systems by replacing the LinkedList and ArrayList and discovered that incorrect collection could result in a 300% increase in energy consumption [START_REF] Hasan | Energy Profiles of Java Collections Classes[END_REF].

Other studies [START_REF] Longo | Reducing energy usage in resource-intensive Java-based scientific applications via micro-benchmark based code refactorings[END_REF][START_REF] Kumar | Energy consumption in Java: An early experience[END_REF] looked into the energy use of Java primitive types, string operations, and the use of exceptions, loops, and arrays. Kumar et al., examined the energy usage of code snippets and micro-benchmarks and provided several conclusions, such as string concatenation would use less energy than StringBuilder and StringBuffer and static variables tends to consume 60% more energy than instance variables. Pereira et al. [START_REF] Pereira | Helping Programmers Improve the Energy Efficiency of Source Code[END_REF] presented SPELL, a tool that helps developers spot energy leaks in their source code. Using a statistical spectrum-based analysis and JRAPL [START_REF] Guimarães | Some heuristic approaches for reducing energy consumption on database systems[END_REF][START_REF] Liu | Data-Oriented Characterization of Application-Level Energy Optimization[END_REF], the tool locates energy-inefficient code fragments. According to the authors, it is language and context-independent.

Energy Optimizations in the Execution Phase

The second part of this section will focus on energy optimization in the execution phase. The goal is to optimize this energy consumption for already developed software without changing the source code. The goal is to set up and create an environment where software can run with the least amount of energy. This could involve process scheduling, system tunning, and so on.

We begin with Aequitas [START_REF] Ribic | Aequitas: Coordinated energy management across parallel applications[END_REF], a system that allows parallel applications to live on comanaged power domains (sharing the same CPU). The technology is founded on the premise that coexisting programs can regard power-management hardware as a shared resource and collaborate on power-management decisions. As a result, it accomplishes its purpose by scheduling these applications with a time-slicing technique (for example, round robin). The authors claim that their strategy achieves a 12.9% improvement while incurring only a 2.5% performance cost.

As for virtual machines, Kurpicz et al. analyzed the total energy consumption of a VM in a data center while emphasizing the fixed cost versus the dynamic one. In this paper [START_REF] Kurpicz | How much does a vm cost? energyproportional accounting in vm-based environments[END_REF], the authors introduced the transparent, reproducible, and predictive cost calculator model EPAVE for VM-based environments. The purpose of EPAVE is to provide the static cost of each VM on the server, including the air conditioning, power distribution, and the dynamic cost related to the VM activities.

The energy usage of virtual machine allocation and task placement has also been investigated [START_REF] Mishra | Energy-efficient vm-placement in cloud data center[END_REF].In this article, The authors propose a method for mapping workloads to virtual machines and virtual machines to the physical ones (PMs) in an energy-efficient manner. In order to solve the problem of high heterogeneity of activities and resources, the jobs are categorized based on their resource requirements. Then the relevant VM is found, followed by the appropriate PM where the selected VM can be deployed. Using a cloud simulator, the authors claimed that The suggested technique saves energy by reducing the number of active PMs while minimizing the makespan and task rejection rate.

Besides the impact of virtual machine orchestration, some works have been targeting the runtime of specific programming languages. Using the TPC-DS benchmark, Chiba et al. investigated the influence of HOTSPOT 13 and J9 14 on the performance of SQL-on-Hadoop systems (SPARK and TEZ) to reveal a three-fold disadvantage that one JVM can have over the other [START_REF] Chiba | Towards selecting best combination of sql-on-hadoop systems and jvms[END_REF]. Oi also compared the performance of HOTSPOT and J9. They demonstrated in their research [START_REF] Oi | Power-performance analysis of jvm implementations[END_REF] that the workload affects the relative performance of a JVM. They discovered that HOTSPOT's performance ranged from 44% to 289% of J9, while its dynamic power consumption ranged from 2.7W to 7.2W, using the SPECjvm2008 [START_REF] Shiv | Specjvm2008 performance characterization[END_REF] benchmarks.

As for python, Redondo and Ortin compared the performance and memory usage of various Python implementations (CPython, Jython, IronPython, PyPy, and so on) using a 215 set of benchmarks to discover that Python2 performed better with short applications. At the same time, Python3 versions covered more tests due to compatibility and the fact that Python2 became obsolete. [START_REF] Redondo | A Comprehensive Evaluation of Common Python Implementations[END_REF].

Conclusion

As we have seen in the state of the art, many methods have been proposed to reduce the ICT's energy footprint, which can be applied in different parts of the program's lifecycle, from consumption to execution. Furthermore, the execution phase took the attention of many researchers because it is the part where the most energy is consumed. This thesis will focus on that aspect as well. However, unlike most work done on the hardware aspect, we will target the software impact on this energy, starting from the choice of the programming language to how to tune some features of a framework to make the software consumes less energy. To do so, we use the empirical approach due to its consistency for the moment. Unlike the performance, which is essentially related to the algorithm's complexity, the energy consumption is more impacted by the hardware. Therefore, to optimize energy consumption, we choose a spiral method based on 3 phases: This work aims to present a set of guidelines to create a benchmarking system to measure different programs' energy consumption. Following that, we will investigate the energy consumption of two well-known programming languages: Java and Python. Finally, we extend this analysis to cover various programming languages In this study, We will extend the work of Pereira et al. to a closer distance to the production environment.

Chapter 3 Benchmarking Protocol to Measure Software Energy Consumption

This chapter covers ways to overcome empirical analysis challenges in energy consumption studies. First, we go through the three components of a successful benchmark when performing energy-related experiments. Section 3.1 focuses on the "reproducibility" challenge-to deliver reproducible experiments without interfering with the energy measurement-while Section 3.2 discusses the accuracy of software energy consumption.

Reproducibility within the context of energy

Introduction

Empirical measurements are critical to capture the effect of developers' choices on software energy consumption. To accomplish this, one should not overlook the benchmarking pitfalls highlighted in the state-of-the-art [START_REF] Van Der Kouwe | Benchmarking Crimes: An Emerging Threat in Systems Security[END_REF].

Second, one should not ignore the tremendous progress in computer science, which has led to a rise in the number of obsolete results. Furthermore, when it comes to comparative research, the execution environment may impact the study itself. Finally, between the exploratory experiment and the publication of the results, new candidates may have emerged, and others may have changed.

As a result, it is critical to ensure that the results can be reproduced, so one can test their hypothesis in different environments and provide room for new candidates. In this section, we will address these issues and investigate several ways of encapsulating the systems-undertest to ensure experiment reproducibility in the context of energy consumption tests while providing the benefits and drawbacks of each strategy. Later in this section, we will propose a protocol to ensure that the results are reproducible and extensible.

Virtual Machines

The first choice should be to use Virtual Machines (VM). This technology enables researchers to choose the most appropriate tools, software, and operating system for their needs without incurring the cost of changing the working environment, giving them more control over dependencies and the execution environment. Furthermore, adopting a VM addresses the replication crisis since virtual images allow even the most sophisticated architecture to be replicated by instantiating a copy of the image. This option, however, comes at a cost. Because of the hypervisor, the software will be built on two kernels: one for the virtual machine (guest) and one for the host machine, resulting in a visible overhead and a negative influence on the performance of the system-under-test. As a result, we cannot use VM for performance-related tests. Isolation is another drawback of VM: while this feature protects the experimental setting from unwanted interference from the outside world, this interaction may be required--especially if the experiment is dependent on an external source such as energy monitors.

Containers

Another option would be to use something that allows us to benefit from the host OS's isolation while simultaneously simplifying replication as proposed by VM and the direct interface with the hardware provided by the traditional techniques.

Containers provide such an advantage by ensuring application separation and ease of replication. Figure 3.1 depicts the architectural differences between virtualization and container technologies. There are three main types of virtualization.

Docker Vs. Virtual Machine

Even though Type 1 is more performant than Type 2, the latter is the most used in research, as most researchers tend to conduct their experiments on their machines. Docker, on the other hand, is the most well-known container technology. In our case, we are more likely to promote Docker for two reasons:

1. As previously stated in literature [START_REF] Van Kessel | Power efficiency of hypervisor-based virtualization versus container-based virtualization[END_REF][START_REF] Morabito | Power Consumption of Virtualization Technologies: An Empirical Investigation[END_REF], we require a lightweight orchestrator to limit the overhead on the energy usage of our experiments 2. We need to communicate with the host OS because we are using hardware sensors to measure energy consumption.

Docker & Energy

Now that we have decided to use container technology to enclose our tests, what effect will this have on the amount of energy consumed by our tests? Using research from [START_REF] Antonio | How does docker affect energy consumption? Evaluating workloads in and out of Docker containers[END_REF] who examined how adding the Docker layer affected energy consumption, Eddie Antonio Santos et al. conducted their experiment by running numerous benchmarks both with and without Docker. They contrasted the energy usage and execution Figure 3.2: energy consumption of Idle system with and without docker [START_REF] Antonio | How does docker affect energy consumption? Evaluating workloads in and out of Docker containers[END_REF] time that resulted. The first step was to observe the effects of the orchestrator and the Docker daemon while there was no work to be done. Then, they used three benchmarks in their experiments: WordPress, Redis, and PostgreSQL. The values below show the system under test's energy consumption while it is idle. Docker has an overhead of roughly 1, 000 joules, as seen in Figure 3.2.

However, as seen in Figure 3.3, Docker increased the execution duration of the benchmark by 50 seconds, which led to a significant rise in energy usage. According to the authors, the Docker daemon is primarily responsible for this overhead, not the fact that the application is running in a container.

Furthermore, they calculated the cost of this extra energy, which was less than 0.15$ in the worst-case scenario, which is insignificant compared to the benefits of Docker for isolation and reproducibility.

To summarize, Docker-based software consumes more energy since it takes longer to execute. The execution of the Docker daemon causes an increase in average power consumption of only 2 Watts. This overhead can reach up to 5% in IO-intensive applications, while it is barely visible in CPU-or DRAM-intensive workloads.

As can be seen, the addition of the supervisor has increased Docker's impact on energy, which will be distributed equitably across all experiments. Therefore, when it comes to comparison analysis, it will mitigate its impact automatically. Furthermore, because we have access to the host hardware, we do not need to worry about capturing the SUT's energy use. As a result, we will use Docker to keep all of our tests separate, ensuring that they can be repeated clearly and simply.

Definition of Extention

With the rapid evolution of the software industry, Even ensuring the reproducibility of the same research will not be enough. Each day, new software versions are released, and new features are added. Even more, the goal of the research might evolve. As a result, nowadays, especially in comparative studies, it is essential to leave room for expansion.

One can expand their experiment through three axes :

• proposed solutions: Where one will expand their research by including additional solutions and comparing them to earlier ones

• evaluation criteria: This axe's objective is to broaden the evaluation criteria to incorporate additional measures like performance, cost, and security, among others.

• benchmarks: This axe aims to enlarge the benchmarks to include others to broaden the research scope.

The architecture of the extension

To be able to extend the empirical experiments through these axes, We propose to enhance the benchmarking framework suggested by the Collaboratory on Experimental Evaluation of Software and Systems in Computer Science1 . Instead of only presenting the four primary aspects of their guidelines that were mentioned previously in Section 3. The model is composed of different components that are described below,

Context

The hardware and the software configuration for the actual experiment, the purpose of this part is to provide extra information to help readers better judge the results and reproduce the experiment while diminishing the impact of external factors.

orchestrator The core design of the experiment is responsible for running the experiment regardless of the context. In the experiment, it is the only component that is allowed to interact with the SUT. The orchestrator provides three interfaces:

• Workload interface: provides a set of functions that the workload should implement to be called by each candidate in the experiment. This interface is responsible for extending the experiment with new benchmarks.

• Observer interface: provides a set of metrics that the orchestrator collects during the experiment. Implementing the observer interface allows the user to extend the experiment with new metrics.

• The Candidate interface: It provides a set of functions that the candidate should implement to be called by the orchestrator. This interface is responsible for extending the experiment with new solutions.

Data The row data collected by the experiment. In our case, it will be provided by the observator. One or more observators can be used to collect different metrics. This data aims to ensure the replication, allowing the reader to perform extra analysis without executing the experiment a second time.

Analysis The final part should provide the set of methods and functions used to do the empirical analysis to answer the research questions.

Conclusion

In this section, we addressed the first challenge of empirical research, which is the reproducibility of the experiments. We started by listing the different options for encapsulating the system under test. Then, we have shown that using VMs is unsuitable for performance or energy-related tests, while containers are a good alternative. Later, we discussed the benefits and drawbacks of using Docker for energy consumption experiments. We have demonstrated that it has a constant overhead that is self-mitigated when compared. Lastly, we have proposed an addition to the benchmarking framework that would allow the experiment to be extended along the three axes already mentioned.

Improving Accuracy by Taming Energy Variations

introduction

While the previous section aimed to ensure the reproducibility of our software energy consumption experiments, this section provides a collection of tips and tools to help increase the accuracy of these experiments. We are well aware of the effect that hardware has on energy fluctuations. However, we feel that there is still an opportunity for practitioners to minimize this energy variance by employing solely tuneable factors. To that end, we conducted a series of empirical studies utilizing state-of-the-art recommendations to discover which controllable elements can limit the variations of benchmark energy usage.

Research questions

This study will focus on the following research questions:

RQ 1: Does the benchmarking protocol affect the energy variation?

RQ 2: How important is the impact of the processor features on the energy variation?

RQ 3: What effect does the operating system have on energy variation? RQ 4: Does the choice of processor make a difference in reducing the energy variation?

Experimental Setup

This section describes our detailed experimental environment, covering the cluster configuration and the benchmarks we used to justify our experimental methodology.

Measurement Context

There are three main contexts.

• different machines with different settings;

• different machines with the same settings ;

• the same machine.

We used the platform Grid5000 (G5K) [START_REF] Balouek | Adding virtualization capabilities to the Grid'5000 testbed[END_REF][START_REF] Margery | Resources Description, Selection, Reservation and Verification on a Large-scale Testbed[END_REF], a large, flexible testbed for experimentdriven research spread across France, to meet these needs. Grid5000 has several clusters comprising 4 to 124 identical machines with different configurations for each cluster. We looked at four groups for our experiment, and our main criterion was the CPU configuration. Table 3.1 below presents a description of the four clusters that have been chosen for our experiments. As most of the nodes are equipped with two sockets (physical processors), we use the acronym CPU or socket to designate one of the two sockets and PU for the processing unit.

Our study considers hyper-threads as distinct PUs.

Workload

We picked the benchmarks based on two criteria. First, scalability: We wanted to learn as much as possible about the experiment in the time we had, so we needed some benchmarks that could grow or shrink with the number of PUs and work in different situations. The second criterion is whether or not the workload is representative. As stated in the challenges, a workload must represent the production environment or the experiment would be inconsistent [16].

To meet these criteria, we looked at the "state of the art" and found the most common benchmarks used to test hardware performance. We then chose the ones that could be scaled up. Our candidate is NAS Parallel Benchmark (NPB v3.3.1) [START_REF] Bailey | The nas parallel benchmarks—summary and preliminary results[END_REF]: one of the most used benchmarks for HPC. We used the applications (LU), the Conjugate Gradient (CG) and Embarrassingly Parallel (EP) computation-intensive benchmarks in our experiments, with the C data class. Furthermore we have used other applications to validate our results using more applications such as Stress-ng v0.10.0,2 pbzip2 v1.1.9,3 linpack4 and sha256 v8.26. 5

Metrics & Measurement Tools

Our metric for the accuracy of the test is the Standard deviation aka STD of the energy consumption. Therefore whether the tests consume more or less energy is out of our scope. We first need a tool to measure energy consumption to study this variation. For this we used POWERAPI [START_REF] Colmant | The next 700 CPU power models[END_REF], which is a power monitoring toolkit that is based on Intel Running Average Power Limit (RAPL) [START_REF] Khan | Rapl in action: Experiences in using rapl for power measurements[END_REF]. The advantage of PowerAPI is that it reports the Energy consumption of CPU and DRAM at a socket level.

Our testbeds are run with a minimal version of Debian 9 (4.9.0 kernel version) 6 where we install Docker (version 18.09.5), which will be used to run the RAPL sensor and the benchmark itself. The energy sensor collects RAPL reports and stores them in a remote MON-GODB instance, allowing us to perform post-mortem analysis in a dedicated environment. Using Docker makes the deployment process easier on the one hand and provides us with a built-in control group encapsulation of the conducted tests on the other hand. Docker allows POWERAPI to measure all the running containers, even the RAPL sensor consumption, as it is isolated in a container.

Every experiment is conducted on 100 iterations, on multiple nodes, and using the 3 NPB benchmarks we mentioned, with a warmup phase of 10 iterations for each experiment. In most cases, we sought to evaluate the Standard Deviation (STD), which is the most representative factor of the energy variation. While running our experiments, we tried not to fall into the most common benchmarking "crimes" [START_REF] Van Der Kouwe | Benchmarking Crimes: An Emerging Threat in Systems Security[END_REF]. As we study the STD difference of measurements we observed from empirical experiments, we use the bootstrap method [START_REF] Efron | The bootstrap and modern statistics[END_REF] to randomly build multiple subsets of data from the original dataset, and we draw the STD density of those sets, as illustrated in Figure 3.6. Given the space constraints, we report on aggregated results for nodes, benchmarks, and workloads. However, the raw data we collected is available through the public repository we published. 7 We believe this can help to achieve better and more reliable comparisons. We mainly consider three different workloads in our experiments: single process, 50 %, and 100 %, to cover the low, medium, and high CPU usage when analyzing the effect of the studied parameters, respectively. These workloads reflect the ratio of used PU count to the total available PU.

Analysis

In this part, we aim to establish experimental guidelines to reduce CPU energy variation. We, therefore, explore many potential factors and parameters that could have a considerable effect on the energy variation.

Docker & Accuracy

As the state-of-the-art assesses the impact of Docker on energy consumption, one can also consider its impact on accuracy. In other words: RQ: does Docker affect the energy variation of the experiments?

To answer this question, we conducted a preliminary experiment by running the same benchmarks LU, CG, and EP in a Docker container and a flat binary format on three nodes of the cluster Dahu to assess if Docker induces an additional variation. Figure 3.6 reports that this is not the case, as the energy consumption variation does not get noticeably affected by Docker while running the same compiled version of the benchmarks at 5 %, 50 %, and 100 % workloads. While Docker increases the energy consumption due to the extra layer it implements [START_REF] Antonio | How does docker affect energy consumption? Evaluating workloads in and out of Docker containers[END_REF], it does not noticeably affect the energy variation. The standard deviation (STD) is even slightly smaller (ST D Docker = 192mJ,ST D Binary = 207mJ), taking into account the measurements errors and the OS activity. To achieve a robust and reproducible experiment, practitioners often repeat their tests multiple times to analyze the related performance indicators, such as execution time, memory consumption, or energy consumption. We, therefore, aim to study the benchmarking protocol to identify how to efficiently iterate the tests to capture a trustable energy consumption evaluation.

In this first experiment, we investigate if changing the testing protocol affects the energy variation. To achieve this, we considered three execution modes: In the "normal" mode, we iteratively run the benchmark 100 times without any extra command, while the "sleep" mode suspends the execution script for 60 seconds between iterations. Finally, after each iteration, the "reboot" mode automatically reboots the machine. The difference between the normal and sleep modes highlights that the CPU needs rest before starting another iteration, especially for an intense workload. Putting the CPU into sleep for several seconds could give it time to reach a lower frequency state or/and reduce its temperature, which could impact the energy variation. On the other hand, the reboot mode is the most straightforward way to reset the machine state after every iteration. It could also be beneficial to reset the CPU frequency and temperature, the stored data, the cache, or the CPU registries. However, the reboot task takes a considerable amount of time, so rebooting the node after every single operation is not the fastest nor the most eco-friendly solution. However, it deserves to be checked to investigate if it effectively enhances the overall energy variation or not.

Figure 3.7 reports on 300 aggregated executions of the benchmarks LU, CG and EP, on 4 machines of the cluster Dahu (cf. Table 3.1) for different workloads. We note that the results have been executed with different datasets sizes (B, C and D for a single process, 50 % and 100 % respectively) to remedy the brief execution times at high workloads for small datasets. This justifies the scale differences in reported energy consumption between the three modes in Figure 3.7. One can observe that picking one of these strategies does not strongly impact the energy variation for most workloads. All the strategies exhibit the same variation with all the workloads we considered-i.e., the STD is tightly close between the three modes. The only exception is the reboot mode at 100 % load, where the STD is 150 % times worst due to a substantial amount of outliers. This goes against our expectations, even when setting a warm-up time after reboot to stabilize the OS.

In Figure 3.8, we study the standard deviation of the three modes by constituting 5, 000 random 30-iterations sets from the previous executions set, and we compute the STD in each case, considering mainly the 100 % workload as the STD was 150 % higher for the reboot mode with that load. We can observe that the considerable amount of outliers in the reboot mode is not negligible, as the STD density is higher than the two other modes. This makes the reboot mode less appropriate for the energy variation at high workloads.

To answer RQ 1, we conclude that the benchmarking protocol partially affects the energy variation, as highlighted by the reboot mode, results in high workloads. The C-states allow switching the CPU between more or less consuming states upon activities. Turning the C-states on or off have been the subject of many discussions [START_REF] Vasan | Worth their watts? -an empirical study of datacenter servers[END_REF] because of its dynamic frequency mechanism, but, to the best of our knowledge, there has been no entirely conducted C-states behavior analysis on CPU energy variation.

We intend to investigate how much the energy consumption varies when disabling the C-states (thus, keeping the CPU in the C0 state) and at which workload. Figure 3.9 depicts the results of the experiments we executed on three nodes of the cluster Dahu. On each node, we ran the same benchmarks with two modes: C-states on, which is the default mode, and C-states off. Each iteration includes 100 executions of the same benchmark at a given workload, with three workload levels. We note that our results have been confirmed with the benchmarks LU, CG and EP.

We can see the effect that has the C-states off mode when running a single-process application/benchmark. The energy consumption varies 5 times less than the default mode. In this case, only one CPU core is used among 2 × 16 physical cores. The other cores are switched to a low-consumption state when C-states are on. The switching operation causes a vital energy consumption difference between the cores and could be affected by other activities, such as the kernel activity, causing a notable energy consumption variation. On the other hand, switching off the C-states would keep all the cores-even the unused ones-at a At a 100 % workload, disabling the C-states seems not to affect the total energy consumption nor its variation. All the cores are used at 100 %, and the C-states module would have no effect, as the cores are not idle. The same reason would apply for the 50 % load, as the hyper-threading is active on all cores, thus causing the usage of most of them. For single process workloads, disabling the C-states causes the process to consume 50 % more energy as reported in Figure 3.9, but reduces the variation by 5 times compared to the C-states on mode. This leads to two questions: Can a process pinning method reduce/increase the energy variation? Moreover, how does the energy consumption variation evolve at different PU usage levels?

Cores Pinning To answer the first question, we repeated the previous test at 50 % workload. In this experiment, we considered three cores usage strategies, the first one (S1) would pin the processes on all the PU of one of the two sockets (including hyper-threads), so it will be used at 100 %, and leave the other CPU idle. The second strategy (S2) splits the workload on the two sockets so each CPU will handle 50 % of the load. In this strategy, we only use the core PU and not the hyper-threads PU, so every process would not share its core usage (all the cores are being used). The third strategy (S3) also involves splitting the workload between the two sockets, but with the hyper-threads on each core used, i.e., half of the cores are used across the two CPUs.

Figure 3.10 reports on the energy consumption of the three strategies when running the benchmark CG on the cluster Dahu. We can notice the big difference between these three execution modes that we obtained only by changing the PU pinning method (that we acknowledged with more than 100 additional runs over more than 30 machines and with the benchmarks LU and EP). For example, S2 is the least power-consuming strategy. We argue that the reason is related to the isolation of every process on a single physical core, reducing the context switch operations. In the first and third strategies, 32 processes are being scheduled on 16 physical cores using the hyper-threads PU, which will introduce more context switching, thus, more energy consumption.

Even though the first and third strategies are very similar (both use hyper-threads, but only on one CPU for the first and two CPUs for the third), the variation gap between them is significant, as the variation in the first strategy is 30 times lower (ST D S1 = 116mJ,ST D S3 = 3, 452mJ). This shows that the hyper-threads technology is not the main reason behind the variation; the first strategy has even less variation than the second one and still uses hyper-threading.

The reason for the S1 low energy consumption is that one of the two sockets is idle and will likely be in a lower power P-state, even with the disabled C-states. The S2 case reports low energy consumption because distributing the threads across all the cores completes the task faster than in the other cases. Hence, it consumes less energy. The S3 is a highly consuming strategy because both sockets are used, but only half the cores are active. This means that we pay the energy cost for both sockets being operational and for the experiments taking longer to run because of the recurrent context switching. Our hypothesis for the worst results observed when using the third strategy is that recurrent context switching, which is added to the OS scheduling and can reschedule processes from one socket to another, invalidates cache usage because a process cannot benefit from the socket local L3 cache when it moves from one CPU to another. (cf. Figure 3.5).

Moreover, the fact that the variation is 4-5 times higher when using the strategy S2 compared to S1 (ST D S1 = 116mJ, ST D S3 = 575mJ), gives another reason to believe that swapping a process from a CPU to another increases the variation due to CPU micro differences, cache misses and cache coherency. While the mean execution time for the strategy S3 is very high (MeanTime S3 = 46s) compared to the two other strategies (MeanTime S1 = 11s, MeanTime S2 = 7s), we see no correlation between the execution time and the energy variation, as the S1 still give less variations than S2 even if it takes 36 % more time to run.

Table 3.2 reports on additional aggregated results for the STD comparison on four other nodes of the cluster Dahu at 50 %, with the benchmarks LU, CG and EP. The CPU usage strategy S1 is by far the experimentation mode that gave the least variation. The STD is almost 5 times better than the strategy S2, but is up to 10 % more energy consuming (Mean S1 = 4469mJ, Mean S2 = 4016mJ). On the other hand, the strategy S3 is the worst, where the energy consumption can be up to 5 times higher than the strategy S2 (Mean S2 = 4016mJ, Mean S3 = 21645mJ) and the variation is much worst (30 times compared to the first strategy). These results allow us to understand better the different processes-to-PU pinning strategies, where isolating the workload on a single CPU is the best strategy. Using the hyper-threads PU on multiple sockets seems to be a bad recommendation. Keeping the hyper-threading enabled on the machine is not problematic as long as the processes are correctly pinned on the PU. Our experiments show that running one hyper-thread per core is not always the best to do, opposite the claims of [START_REF] Marathe | An empirical survey of performance and energy efficiency variation on Intel processors[END_REF].

Processes Threshold To answer the second question regarding the evolution of the energy variation at different levels of CPU usage, we varied the used PU's count to track the EV evolution. Figure 3.11 compares the aggregated energy variation when the C-states are on and off using 2, 4 and 8 processes for the benchmarks LU, CG and EP. This figure confirms that disabling the CPU C-states does not decrease the variation for all the workloads; as we can ). This variation is 4 times lower when running four processes and almost equal to the C-states on mode when running eight processes. In fact, running more processes implies using more CPU cores, which reduces the idle cores count, so the cores will more likely stay at a higher consumption state even if the C-states mechanism is on.

In our case, using 4 PU reduces the variation by four times and consumes almost the same energy as keeping the C-states mechanism on (Mean C-states-on = 7, 048mJ, Mean C-states-o f f = 7, 119mJ). This case would be the closest to reality as we do not want to increase energy consumption while reducing the variation. However, using a lower number of PU still results in less variation, even if it increases the overall energy consumption.

We note that disabling the C-states is not recommended in production environments, as it introduces extra energy consumption for low workloads (around 50 % in our case for a single process job). However, our goal is not to optimize energy consumption but to minimize the energy variation. Thus, disabling the C-states is very important to stabilize the measurements in some cases when the variation matters the most. Comparing the energy consumption of two algorithms or two software system versions is an example of a use case benefiting from this recommendation.

Turbo Boost The Turbo Boost-also known as Dynamic Overclocking-is a feature that has been incorporated in Intel CPU since the Sandy Bridge micro-architecture, and is now widely available on all of the Core i5, Core i7, Core i9, and Xeon series. It automatically raises some of the CPU cores operating frequency for short periods, thus boosting performance under specific constraints. When demanding tasks run, the operating system decides to use the processor's highest performance state.

Disabling or enabling the Turbo Boost impacts the CPU frequency behavior directly, as enabling it allows the CPU to reach higher frequencies to execute some tasks for a short period. However, its usage does not significantly impact the energy variation. Acun et al. [4] tried to track the Turbo Boost impact on the Ivy Bridge and the Sandy Bridge architectures. They concluded that it is one of the main responsible for the energy variation, as it increases the variation from 1 % to 16 %. In our study, we included a Turbo Boost experiment in our testbed to check this property on the recent Xeon Gold processors, covering various workloads.

The experiment showed that disabling the Turbo Boost does not exhibit any considerable positive or negative effect on the energy variation. Table 3.3 compares the STD when enabling/disabling the Turbo Boost, where the columns combine workload and benchmark. We only had minor measurement differences when switching on and off the Turbo Boost. We favored or against using the Turbo Boost while repeating tests, considering multiple nodes and benchmarks. This behavior is mainly related to the thermal design power (TDP), especially at high workloads. When a CPU is used at its maximum capacity, the cores would heat up very fast and hit the maximum TDP limit. In this case, the Turbo Boost cannot offer more power to the CPU because of the CPU thermal restrictions. At lower workloads, the tests we conducted proved that the Turbo Boost is not one of the main reasons for the energy variation. The variation difference is barely noticeable when disabling the Turbo Boost, which cannot be considered due to the OS activity and the measurement error margin. We cannot affirm that the Turbo Boost does not have an impact on all the CPUs, as we only tested on two recent Xeon CPUs (clusters Chetemi and Dahu). We confirmed our experiments on these machines 100 times at 5 %, 25 %, 50 % and 100 % workloads.

We conclude that CPU features highly impact the energy variation as an answer for RQ 2.

RQ 3: Operating System

The operating system (OS) is the layer that exploits the hardware capabilities efficiently. It has been designed to ease the execution of most tasks with multitasking and resource sharing. In some delicate tests and measurements, the OS activity and processes can cause significant overhead and therefore a potential threat to the validity. The purpose of this experiment is to determine if the sampled consumption can be reliably related to the tested application, especially for low-workload applications where CPU resources are not heavily used by the application.

The first way to do this is to evaluate the OS idle activity consumption and compare it to a low-workload running job. Therefore, we ran 100 iterations of a single process benchmark EP, LU, and CG on multiple nodes from the cluster Dahu, and compared the energy behavior of the node with its idle state on the same duration. The aggregated results, illustrated in Figure 3.12, depict that the idle energy variation is up to 140 % worst than when running a job, even if it consumes 120 % less energy (Mean Job = 8, 746mJ, Mean Idle = 3, 927mJ). In fact, for the three nodes randomly picked from the cluster Dahu, the idle variation is way more important than when a test was running, even if it is a single process test on a 32-cores node. This result shows that OS idle consumption varies widely due to the lack of activity and the different CPU frequency states. However, it does not mean that this variation is responsible for the overall energy variation. The OS behaves differently when a job is running, even if the amount of available cores is more than enough for the OS to keep its idle behavior when running a single process.

Inspecting the OS idle energy variation is insufficient to relate the energy variation to the active job. The OS can behave differently regarding resource usage when running a task. To evaluate the OS and the job energy consumption separately, we used the POWERAPI toolkit. This fine-grained power meter allows the distribution of the RAPL global energy across all the Cgroups of the OS using a power model. Thus, it is possible to isolate the job energy consumption instead of the global energy consumption delivered by RAPL. To do so, we ran tests with a single process workload on the cluster Dahu and used the POWERAPI toolkit to measure the energy consumption. Then, we compared the job energy consumption to the global RAPL data. We calculated the Pearson correlation [START_REF]Pearson's Correlation Coefficient[END_REF] of the energy consumption and variation between global RAPL and POWERAPI, as illustrated in Figure 3.13. The job energy consumption and variation are strongly correlated with the global energy consumption and variation with the coefficients 93.6 %, and 85.3 %, respectively. However, this does not entirely exclude the OS activity, especially if the jobs have tight interaction with the OS through the signals and system calls. This brings a new question on whether applying extra-tuning on a minimal OS would reduce the variation. As well as the Meltdown security .12: OS consumption between idle and when running a single process job patch -known to be causing some performance degradation [START_REF] Kocher | Spectre attacks: Exploiting speculative execution[END_REF][START_REF] Lipp | Meltdown: Reading kernel memory from user space[END_REF]-affects the energy variation?

OS Tuning An OS is a pack of running processes and services that might or not be required for its execution. Even using a minimal version of Debian Linux, we could list many OS running services and processes that could be disabled/stopped without impacting the test execution. This extra-tuning may not be the same, depending on the nature of the test or the OS. Thus, we conducted a test with a deeply-tuned OS version. We disabled all the services/processes that are not essential to the OS/test running, including the OS networking interfaces and logging modules. We only kept the strict minimum required for the experiment's execution. Table 3.4 reports on the aggregated results for running single process measurements with the benchmarks CG, LU and EP, on three servers of the cluster Dahu, before and after tuning the OS. Every cell contains the STD value before the tuning, plus/minus a ratio of the energy variation after the tuning. We notice that the energy variation varies less than 10 % after the extra-tuning. We argue that this variation is not substantial, as it is not stable from one node to another. Moreover, 10 % of the variation is not a representative difference due to many factors that can affect it as CPU temperature or measurement errors. Speculative Executions Meltdown and Spectre are two of the most famous hardware vulnerabilities discovered in 2018, and exploiting them allows a malicious process to access other processes' data that is supposed to be private [START_REF] Kocher | Spectre attacks: Exploiting speculative execution[END_REF][START_REF] Lipp | Meltdown: Reading kernel memory from user space[END_REF]. They both exploit the speculative execution technique where a process anticipates some upcoming tasks, which are not guaranteed to be executed when extra resources are available, and revert those changes if not. Some OS-level patches have been applied to prevent/reduce the criticality of these vulnerabilities.

Energy Consumption

On the Linux kernel, the patch has been automatically applied since version 4.14.12. It mitigates the risk by isolating the kernel and the user space and preventing the mapping of most of the kernel memory in the user space. Nikolay et al. have studied in [START_REF] Simakov | Effect of meltdown and spectre patches on the performance of HPC applications[END_REF] the impact of patching the OS on the performance. The results showed that the overall performance decrease is around 2-3 % for most of the benchmarks and real-world applications; only some specific functions can meet a high-performance decrease. In our study, we are interested in the applied patch's impact on the energy variation, as the performance decrease could mean an energy consumption increase. Thus, we ran the same benchmarks LU, CG ad EP on the cluster Dahu with different workloads, using the same OS, with and without the security patch. Table 3.5 reports on the STD values before disabling the security patch. A minus means that the energy varies less without the patch being applied, while a plus means that it varies more. These results help us to conclude that the security patch's effect on the energy variation is not substantial and can be absorbed through the error margin for the tested benchmarks. The best case to consider is the benchmark LU where the energy variation is less than 10 % when we disable the security patch, but this difference is still moderate. The little performance difference discussed in [START_REF] Kocher | Spectre attacks: Exploiting speculative execution[END_REF][START_REF] Lipp | Meltdown: Reading kernel memory from user space[END_REF] might only be responsible for a slight variation, which will be absorbed through the measurement tools and external noise error margin in most cases. To answer RQ 3, we conclude that the OS should not be the main focus of the energy variation taming efforts. , which makes measurements stability even worse. In this experiment, we, therefore, compare four different generations of CPU to evaluate the energy variation for each CPU and its correlation with the generation. Table 3.6 indicates the characteristics of each tested CPU.

Table 3.6 also shows the aggregated energy variation of the different generations of nodes for the benchmarks LU, CG and EP. The results attest that the latest versions of CPU do not necessarily cause more variation. In the experiments we ran, the nodes from the cluster Paranoia tend to cause more variation at high workloads, even if they are from the latest generation. While the Skylake CPU of the cluster Dahu cause often more energy variation than Chetemi and the Ecotype Broadwell CPU. We argue that the hypothesis "the energy consumption on newer CPU varies more" could be true or not depending on the compared generations, but most importantly, the chip's energy behaviors. On the other hand, our experiments showed the lowest energy variation when using the Ecotype CPU, these CPUs are not the oldest nor the latest, but are tagged with "L" for their low power/TDP. This result rises another hypothesis when considering CPU choice, which implies selecting the CPU with a low TDP. This hypothesis has been confirmed on all the Ecotype cluster nodes, especially at low and medium workloads. reports the worst variation in most cases and that Ecotype is the best cluster to consider to get the least variations, as it has a higher density for small variation values.

We conclude on affirming RQ 4, as selecting the right CPU can help get fewer variations.

Experimental Guidelines

To summarize our experiments, we provide some practical guidelines in Table 3.7, based on the multiple experiments and analyses we did. These guidelines constitute a set of minimal requirements or best practices, depending on the workload and the criticality of the energy measurement precision. Therefore, it intends to help practitioners tame the energy variation on the selected CPU and conduct experiments with the tiniest variations.

Table 3.7 gives a proper understanding of known factors, like the C-states and their variation reduction at low workloads. However, it also lists some new factors we identified along the analysis we conducted in the previous Section, such as the results related to the OS or the reboot mode. Some guidelines are more useful/efficient for specific workloads, as shown in our experiments. Thus, qualifying the workload before conducting the experiments can help choose the proper guidelines. Other studied factors, such as Turbo Boost or Speculative execution, are not mentioned in the guidelines due to the small effect observed in our study. To validate the accuracy of our guidelines among a varied set of benchmarks on the one hand and their effect on the variation between identical machines, on the other hand, we ran seven experiments with benchmarks and real applications on a set of four identical nodes from the cluster Dahu, before (normal mode where everything is left to default and the charge of the OS) and after (optimized) applying our guidelines. Half of these experiments have been performed at a 50 % workload, and the other half on single process jobs. The choice of these two workloads is related to the optimization guidelines that are mainly effective at low and medium workloads. We note that we used the cluster Dahu over Ecotype to highlight the effect of the guidelines on the nodes where the variation is susceptible to be higher.

Figure 3.15 and 3.16 highlight the improvement brought by the adoption of our guidelines. They demonstrate the intra-node STD reduction at low and medium workloads for all the benchmarks used at different levels. Concretely, for low workloads, the energy variation is 2-6 times lower after applying the optimization guidelines for the benchmarks LU and EP, as well as LINPACK, while it is 1.2-1.8 times better for Sha256. For this workload, the overall energy consumption after optimization can be up to 80 % higher due to disabling the C-states to keep all the unused cores at a high power consumption state (Mean LU-normal-Dahu2 = 11, 500mJ, Mean LU-optimized-Dahu2 = 20, 508mJ). For medium workloads, the STD, and thus variation, is up to 100 % better for the benchmark CG, 20-150 % better for the pbzip2 application and up to 100% for STRESS-NG. We note that the optimized version consumes fewer energy thanks to an appropriate core pinning method.

Figures 3.15 and 3.16 also highlight that applying the guidelines does not reduce the internodes variation in all the cases. This variation can be up to 30 % in modern CPU [Wang et al.]. However, taming the intra-node variation is a good strategy for identifying more relevant mediums and medians and then accurately comparing the nodes' variation. Even though using the same node is always better, it avoids the extra inter-nodes variation and thus improves the stability of measurements.

Threats to Validity

Several issues affect the validity of our work. For most of our experiments, we used the Intel RAPL tool, which has evolved along Intel CPU generations to be known as one of the most accurate tools for modern CPUs but still adds a significant overhead if we adopt a sampling at high frequency. The other fine-grained measurement tool we used is POWERAPI. It allows measuring the energy consumption at the granularity of a process or a Cgroup by dividing the RAPL global energy over the running processes using a power model. The usage of POWERAPI adds an error margin because of the power model built over RAPL. The RAPL tool mainly measures CPU and DRAM energy consumption. However, even running CPU/RAM intensive benchmarks would keep a degree of uncertainty concerning the hard disk and networking energy consumption. In addition, the operating system adds a layer of confusion and uncertainty.

The Intel CPU chip manufacturing process and the material's micro-heterogeneity are among the most significant issues, as we cannot track or justify some energy variations between identical CPUs or cores. These CPU/cores might handle frequencies and temperature differently and behave consequently. This hardware heterogeneity also makes reproduction complex and requires the usage of the same nodes on the cluster with the same OS.

Summary

To increase the accuracy in comparative experiments, we conducted an empirical study of controllable factors that may increase the energy variations on platforms with some of the latest CPUs and for several benchmarks. This study is not intended to nullify the variability of the CPU, as some of this variability is related to the chip manufacturing process and its thermal behavior. Instead, it aims to tame and mitigate this variability through controlled experiments. In this study, We investigated some previously discussed aspects of some recent CPUs, considered new factors that have not been deeply analyzed to the best of our knowledge, and constituted a set of guidelines to increase this accuracy for energy-related experiments. Some of these factors, like the C-states usage, can reduce the energy variation up to 500 % at low workloads, while choosing the wrong cores/PU strategy can cause up to 30× more variability.

Conclusion

As seen in Section 2.2 of state of art, a successful benchmark faces three challenges: reproducibility, accuracy, and representativeness. This chapter has covered two of the three criteria.

The first section covered the reproducibility challenge by studying the existing reproducibility techniques. We have observed that there are two primary methods for encapsulating experimental systems, the first is to utilize virtual machines, and the second is based on containers. We established that Docker is more suitable for energy-related studies for three reasons.

1. It is more lightweight than virtual machines.

2. It offers interactivity with the hardware of the host machine, which will enable us to gather more metrics.

3. It has a constant overhead, a key factor to nullify the encapsulation impact on the energy consumption when performing empirical analyses.

After settling on the most suitable choice to encapsulate experiments, we highlighted the need to enhance the reproducibility of empirical tests along several axes (benchmarks, metrics, and candidates) to keep up with the rapid pace of software development. Furthermore, we have provided a model that enables comparative experiment extension. The foundation of this model is separating the experiment into multiple independent components: an orchestrator that executes the experiment, one or more observers that collect metrics, the candidates being tested, and the benchmark against which these candidates are being compared.

As for the second section, we addressed the accuracy challenge in energy-based experiments. We started by analyzing the impact of the chosen encapsulation method from the previous section on this challenge to determine that the impact is negligible. Then, we conducted an empirical study utilizing some of the well-known benchmarks in literature, Stress-ng 8 and NAS Parallel Benchmark [START_REF] Bailey | The nas parallel benchmarks—summary and preliminary results[END_REF], on a variety of machines with diverse hardware combinations and operating system setups and tunings. We have shown that optimizing the operating system can significantly affect how accurately energy is used. This effect can affect the accuracy of the experiments from an increase of 5× to a penalty of 30×.

We have also shown in this section the harm that this taming of the variation can cause to the representativeness of the results since some aspects that help increase the accuracy of the results in the research environment cannot be applied in the production environment, such as turning off the C-states. Another impact of increasing this accuracy was the increase in the overall energy consumption of the tests. Even if the overall was for all the candidates, this remains an issue to be addressed.

This chapter aims to establish a reproducible and accurate protocol for conducting energyrelated experiments. From now on, we shall use this protocol in our studies to reduce software energy usage. In particular, Noureddine et al. [START_REF] Noureddine | A preliminary study of the impact of software engineering on GreenIT[END_REF] in 2012, and then Pereira et al. [START_REF] Pereira | Energy Efficiency Across Programming Languages: How Do Energy, Time, and Memory Relate[END_REF] in 2017, conducted empirical power measurements on this topic: both concluded that compiled programming languages overcome dynamic ones when it comes to power consumption. According to their experiments, an interpreted programming language, like Python, can impose up to a 7, 588 % energy overhead compared to C [START_REF] Pereira | Energy Efficiency Across Programming Languages: How Do Energy, Time, and Memory Relate[END_REF] (cf. In this chapter, we aim to reduce the energy consumption of Python code. We first start by presenting the motivation behind our work in Section 4.2. Then, we share some insights on the impact of programmer choices. To do this, we report on a series of experiments to examine the energy consumption of Python code across various use cases. Section 4.3 will then end by investigating several features of Python structure and their impact on energy consumption. After that, in Section 4.4.1, we investigate other non-intrusive approaches to optimizing the energy consumption of applications by comparing multiple Python runtime implementations, which include alternative interpreters and libraries that are dedicated to optimizing the code without changing its structures, such as ahead-of-time (AOT) compilation and just-in-time (JIT) libraries that are maintained by the community.

Motivation

Python Popularity

Nowadays, Python attracts a large community of developers who are interested in data analysis, web development, system administration, and machine learning. According to a survey conducted in 2018 by JetBrains, 1 one can fear that the wide adoption of dynamic programming languages in production, like Python, may critically hamper the power consumption of ICT. As the popularity of such dynamic programming languages partly builds on the wealth and the diversity of their ecosystem (e.g., the NumPY, SciKit Learn, and Panda libraries in Python), one cannot reasonably expect that developers will likely move to an alternative programming language mainly for energy considerations. Rather, we believe that a better option consists of leveraging the strength of this rich ecosystem to promote energy-efficient solutions to improve the power consumption of legacy software systems.

Python Gluttony

According to [START_REF] Pinto | Energy Efficiency: A New Concern for Application Software Developers[END_REF] and [START_REF] Noureddine | A preliminary study of the impact of software engineering on GreenIT[END_REF], Python tends to be one more energy hungry programming language. As one can notice in Figure 4.2, Python consumes 30 times more than C or C++. The benchmark was implemented with the Tower of Hanoi2 of 30 disks.

As shown in Table 4.1, one can observe that, for most of the applications taken for the Computer Language Benchmark Game (CLBG), Python takes more time to execute-the only case that he was not the worst one was in the benchmark regx-redux where he beats Go-and in some cases the gap was huge, such as in n-body where Python took around 100 times more than C++. 3 Python consumes energy mainly because it is slow in execution. Its flexibility and simplicity caused it to drop off in performance because Python gains its flexibility from being a dynamic language. Therefore, it requires a faster interpreter to execute its programs to compete against alternatives written in compiled programming languages, such as C and C++, or semi-compiled languages, like Java.

Python Use Cases

To reduce the energy consumption of Python, we started by targeting the main usage of this programming language, which is revealed to be data science and web development. Figure 4.3 illustrates a study published by the JetBrain company on Python developers. 4 57% of the respondents reported that they use Python for data science, and 51% said they use it for web development. Around 40% are using it for system administration.

Optimizing the Energy Consumption of Python Applications

In this section, we study the energy footprint of Python in its most popular domains of adoption. We first explore the data and control structures, aiming to reveal some fundamental guidelines, as Hasan et al. did in [START_REF] Hasan | Energy Profiles of Java Collections Classes[END_REF]. Then, we measure the energy consumption of several Python implementations to propose a non-intrusive technique to improve energy efficiency. Overall, this section will address the following research questions:

RQ 1: What is the energy footprint of Python when used in data science? RQ 2: Are the Python guidelines energy-efficient by construction? RQ 3: Can we reduce the energy consumption of Python programs without altering the source code?

To answer these questions, we report on 4 case studies that intend to answer these research questions. First, we study the energy behavior of Python in two application contexts: machine learning (cf. Section 4.3.2) and web applications (cf. Section 4.3.1). Then, we dive deeper into the energy consumption of Python core structures before concluding with the impact of parallelism on energy consumption.

Python for Machine Learning

Machine learning is becoming an integral part of our daily lives, growing more potent and energy-hungry each year. As machine learning can significantly impact climate change, it is vital to investigate mitigation techniques. This part highlights the work of Kaminetzky [73] during his internship at inria

Experimental Protocol

Measurement Context

Hardware settings: Chifflot 8 from Grid 5000's Lille site was used for all of the trials. The machine is outfitted with two Intel Xeon Gold 6126 CPUs, each having 12 physical cores, 192 GB of RAM, and two 32 GB Tesla V100 GPUs.

Software settings: Each experiment is done within a Docker container using Jupyter lab to ensure reproducibility. These tests are run atop a minimal version of Debian-10 to increase the tests' accuracy by eliminating unnecessary processes.

Input Workload

Models: Several models were developed. However, only two were used in the final trials because they achieved 94% accuracy with an acceptable training time. David Page's cifar10-fast 6 and Woonhyuk Baek's torch skeleton 7 are reported here.

Datasets:

The CIFAR-10 dataset was the major source of data for the studies. It is made up of 60, 000 32x32 color images grouped into 10 categories. Some experiments were done using the MNIST dataset of handwritten digits to validate the results acquired from the first dataset. The model did not need to be updated because the 60, 000 28x28 grayscale photos were padded.

Candidates:

The experiments were run with several different CPU and GPU configurations:

• with and without GPU,

• with and without CPU hyper-threading,

• different number of CPU physical cores.

Key Performance Metrics:

We used Pytorch 1.10.0 to train those models and pyJoules to measure the energy consumption of the GPU and CPU. The key performance metrics are, therefore:

• accuracy: in % • execution time: in seconds for both the duration of each epoch and the total duration to achieve a certain accuracy

• total energy consumption: in joules, including the CPU and the GPU.

Results & Findings

As the model's accuracy increases, so do the energy required for the next accuracy increment. This figure shows that there is a non-significant impact on the choice of the strategy on the accuracy; it is only a matter of the number of epochs. However, the increase in accuracy is not linear, as can be observed. It took only 10 epochs to reach an accuracy of 84%, but it required more than twice the number of epochs to add an extra 6% accuracy. This highlights the price one should pay to increase the model's accuracy. Figure 4.5 confirm this observation. As one can see, the training of the model up to 90% accuracy requires three times the energy required for training to 80% accuracy. Moreover, one can notice that this price is paid mainly by the CPU and GPU, while the memory is not that impacted.

Interestingly there were no significant differences between different strategies-around 1.3%-as shown in Figure 4.6. However, this gap may increase as the number of epochs increases.

On the other hand, as the number of epochs increases, the average power consumption decreases until it reaches a steady state after 10 epochs. This could be connected to the caching strategies adopted by the CPU and GPU. However, this evolution is still insignificant compared to the baseline value. This is shown in Figure 4.7. As one can see in Figures 4.7 and 4.8, the average power consumption and the average duration of an epoch do not have a significant variation, as most of them last around 3.7 seconds. However, these two values have an intermediate correlation if we exclude the strategy based on one core and its hyper-thread. The last strategy exhibits more power-even when its duration is not optimal-because the context switches between the core and its hyper-thread.

Synthesis

We discovered that when the model's accuracy improves, so does the energy required for a subsequent accuracy gain. This begs the question of when we should discontinue training. Is a 10% increase in accuracy worth it if we have to spend three times the energy? While using the GPU, the CPU has no significant impact on the energy consumption of the training. Therefore, one can limit the number of CPU cores used during the execution without sacrificing the execution time and accuracy.

Python for Web Development

Django 8 and Flask9 are the most popular Pyhton frameworks for web development. According to a Jetbrains poll, 10 Django is used in 40% of the cases, while Flask is used in 41% of the cases. In contrast to Flask, which is a micro-framework, Django is a high-level web framework that provides a standard method for creating and maintaining complex and scalable databasedriven websites quickly and effectively. Therefore, this study will focus on the latter one.

Life-cycle of a request in Django Django is an MVT (Model-View-Template) framework, which means that it follows the MVC (Model-View-Controller) pattern. Figure 4.9 describes the life-cycle of a request in this framework. Whenever a request arrives in Django, it is processed by the middleware layers, one at a time. These middleware layers are in charge of authentication, security, and so on. Once these layers process the request, it is passed to the URL router, which extracts the URL from the request and tries to match it to the defined URLs. After getting the matching URL, the corresponding view function is called. This function is responsible for treating the request, gathering the data, and then generating the response that will be put inside a template to be returned as a response. As Django adopts an MVT model, it also offers an automatic way to retrieve data from the database to the view using the Object Relational Mapping (ORM) [START_REF] O'neil | Object/relational mapping 2008: hibernate and the entity data model (edm)[END_REF]. First, we investigate the energy consumption of the request-response life cycle in Django to determine which layer consumes the most energy. To do so, we created a sample Django application that returns the response to the request. We tracked its energy consumption using JouleHunter. 11 JouleHunter is an open-source library that we developed to help practitioners identify energy hotspots in their applications using statistical profilers. In the case of Django, JouleHunter is included as a middleware with no additional setup or change to the source code. The energy consumption of the request-response life-cycle in Django is shown in Figure 4.10. As we can notice, 91.4% of the total energy consumption is spent on resolving the request by retrieving the data, while only 5.27% is consumed on rendering the response.

Client

Therefore, we chose to study if the choice of the database and the ORM impact energy consumption and web application performance. To accomplish so, we examined the cost of a single request from the prior website using various ways to extract data from the database. We considered using two different databases, POSTGRESQL and SQLITE3, that store the same records and three different ways to fetch the records:

1. Vanilla relies only on the ORM to retrieve the data, 2. Prefetch queries the data before being requested, 3. Optimized leverages SQL without passing by the ORM. As one can observe in Figure 4.11, the strategy to query stored data greatly impacts energy consumption. As the Vanilla strategy can consume up to 10× more energy than the Optimized one. Conversely, the choice of the database does not exhibit a key impact on the total energy, despite their different behavior regarding the execution time and the average power. This can be useful to support developers in choosing which database engine they can adopt, guided by the number of expected requests and the targeted performance.

Another interesting observation is the impact of the interpreter, as Figure 4.11 highlights. For example, using the PyPy interpreter reduces energy consumption, even when adopting the Vanilla strategy. We will further discuss the choice of Python runtime implementation in the next section.

Finally, we run the same experiment with the Optimized strategy using JouleHunter. Figure 4.12 depicts the resulting energy consumption. As one can notice, while the rendering method consumed the same amount of energy as in the Vanilla strategy (around 1.3 kJ), the resolve part dropped by 20×.

Synthesis We can conclude that the database and ORM selection significantly impact the web application performance and energy consumption. Moreover, unlike rendering responses, many alternatives can be considered to store and query the application database.

Python for Data Processing

Another field of investigation is the type of data and control structures that might impact energy consumption. In this study, we will run some common algorithms with different data structures and control structures to study if there is any difference in energy consumption. This part displays the work of Randrianaina [START_REF] Randrianaina | OPTIMISATION DE L'EMPREINTE ÉNERGÉTIQUE DE PYTHON[END_REF] during his internship at Inria.

Experimental Protocol

All of the tests in this study are done on the Grid5000 cluster using the same machine, Python interpreter, and library version. Due to PowerAPI's frequency constraint, we measure the energy of 1, 000 algorithm iterations for each test. Furthermore, we perform each test 20 times for completeness, deleting the cache between each run. The data is then averaged and presented in graphs.

Python Loops

In the first experiment, we study the impact of the type of loop on energy consumption. To do so, we execute the following code snippet:

1: sum ← 0 2: for i ← 0 to N do 3:

sum ← sum + 1 4: end for 5: return sum First, the classical for(i in range(len(n))). However, as we can see here, unlike other programming languages, it requires extra operations, such as determining the collection length and then using the iterator range. So, we tried an alternative version: for(element in collection). Moreover, in most programming languages, the for loop is translated into a while loop, so we wanted to compare this with a while version. Therefore, our candidate loops are defined as:

• for (i in range(len(n))),

• for (element in collection),

• while. Figure 4.13 shows the results of the experiment. As one can notice, the data type has a negligible impact on energy consumption. However, the way one iterates over the collection has a huge factor. Interestingly, the for in range(...) loop was by far the optimal one, followed by the regular for in collection, and the while part was the last one with an overhead of 400% compared to the first option.

The reason behind such behavior is mainly related to how the Python interpreter is implemented. 12 , 13 Most of the built-in functions and operations are written in C, to reduce the latency of Python applications, and the same goes for the function range. 14 Furthermore, the function len has a complexity of O(1), as it is based on the function Py_SIZE of C, which stores the length in a field for the object 15 . Therefore, the for in range(...) is creating a new iterator that has the same length as the first one and, for each iteration, requires second access (l[i]) instead of one-explaining the doubled time. The while is even slower due to the implicit increment of the variable, which causes an extra operation during the loop.

To confirm this hypothesis, we tried to construct a new list by editing the elements of the previous one (cf. Figure 4.14). We used four different methods to do so: comprehension list, while loop, a map with predefined function, and a map with an anonymous function. A comprehension list is a Pythonic way to create a new list by applying a As one can notice in Figure 4.14, the built-in methods are the most energy-saving ones, while the customize while loop is the heaviest. Moreover, when increasing the collection size, the map with lambda functions tends to consume less energy than the predefined one. The reason behind such behavior is that Python treats these functions as local variables, unlike the predefined ones, which are global in our case. Therefore, they are faster and consume less energy.

Synthesis This study demonstrates that the optimal way to reduce the energy consumption of Python applications is to follow the guidelines and privilege the built-in functions.

Python & Multiprocessing

The purpose of this part is to study the impact of concurrency on energy consumption for Python applications. To do so, we run a simple code snippet that computes the sum of the first N numbers using the standard concurrency libraries multiprocessing 16 and multihtreading 17 . We run the following strategies using a Desktop machine with Intel(R) Core(TM) i7-4800MQ CPU @ 2.70GHz with four cores and four hyper-threads.

• Sequential: we compute the sum of the first N numbers with no concurrency;

• Multithreading: we use the ThreadPoolExecutor library to compute the sum of the first N numbers. We divide N by the number of threads, and each thread will compute the sum of the numbers in its range. For our case, we used four threads;

• Multiprocessing: we use the ProcessPoolExecutor library to compute the sum of the first N numbers. We divide N by the number of processes, and each process sums the numbers in its range. In this situation, we considered the 2, 4, 8, and 16 processes iteratively.

Figure 4.15 shows the results of the experiment regarding four metrics: power, execution time, DRAM energy, and CPU energy. As one can notice, CPU energy is ten folds higher than DRAM one. Therefore, we will focus on the CPU for this study.

In general, the MULTIPROCESSING strategy is the most energy efficient one, and the fastest one. However, unexpectedly, the Multithreading option took more time to execute the code snippet than the Sequential one. Therefore, we will divide our analysis into two parts: the first one will focus on Multithreading versus the Sequential strategies, while the second one will study the behavior of the MULTIPROCESSING strategy.

Multithreading vs Sequential

The unexpected behavior of the MULTIPROCESSING strategy is because the Python interpreter is not thread-safe. Because of the global lock system (GLS), Python cannot run multiple threads at the same time. Therefore, the Multithreading strategy is slower than the Sequential one. On the other hand, a side effect of this is the context switching between threads, which will put the CPU in a lower power state. The Multithreading strategy depicts an average power of 13Watts compared to the Sequential one, which is around 22Watts. This difference in power consumption has overcome the lack of performance of the Multithreading strategy, which leads to better energy efficiency.

Multiprocessing Unlike the Multithreading strategy, the MULTIPROCESSING strategy is based on forking the program into multiple processes, which are independent of each other. This allows the Python interpreter to run multiple processes simultaneously, hence explaining the better performance of the multi-processing strategy. However, this will increase the average power consumption of the CPU, which is around 40Watts, compared to the Sequential one, which is around 22Watts. Nevertheless, the MULTIPROCESSING strategy is still the most energy-efficient one.

Although increasing the number of processes will reduce the execution time because we split the tasks on the number of processes, there is a point where the execution time sequential [START_REF]Pearson's Correlation Coefficient[END_REF] multithreading ( 4) multiprocessing ( 2) multiprocessing ( 4) multiprocessing ( 8 increases. As one can see in Figure 4.15, the energy consumption correlates with the number of processes until the limit of physical cores, when concurrent processes compete for the CPU.

Before reaching the limit of physical cores, we also observe that the operating system's scheduler tends to favor the execution of processes on the same physical core by taking advantage of the hyper-thread feature. While this strategy aims to save energy by leveraging the ACPI P-states and C-states of unallocated cores, this leads to increased execution times.

Synthesis This study demonstrates that the optimal way to reduce the energy consumption of Python applications is to adopt the MULTIPROCESSING strategy, which is the most energyefficient one. However, this strategy should be used carefully as it can lead to increase power consumption. This study demonstrates that the optimal number of processes is equal to the number of physical cores, which is four in our case. When using the Multithreading strategy, it was also found that sometimes the performance of the application has to be given up to save energy.

Python Interpreters

We used the official version of Python for the first studies because the goal was mainly to highlight the code structure's impact on energy consumption. One main drawback of the previous method is the work to be done to update the existing code base to reduce energy consumption. Despite the availability of numerous tools for code refactoring and automatic code optimization, the optimization process remains a manual effort. This is because code optimization is a task that requires an extensive understanding of both the codebase and the language itself.

To avoid a such hustle, we tried to find a non-intrusive approach to make the Python code more eco-friendly without altering its structure. Python is an interpreted language, which led to many initiatives to implement their interpreter to improve one or many aspects of the Python code. Therefore, in this section, we will examine how different Python interpreters affect the energy consumption of software, and in which case, one should use a non-conventional interpreter to save the energy consumption of their application.

Python Runtimes

To do so, we gathered a list of interpreters, transpilers, and other optimization libraries that can contribute to reducing the energy consumption of legacy Python applications:

1. CPython: 18 This Python interpreter, written in C, is the reference interpreter of Python.

CPython compiles the source code into byte-code and then interprets it. The CPython project supports both versions of Python 2 and 3;

2. PyPy: 19 An alternative implementation of the Python interpreter. It is written using RPython to use the JIT. It compiles the most used portions of the Python code into a binary code for better performance. To benefit from these optimizations, the program has to be executed for at least for few seconds, so the JIT has enough time to warm up; the JIT optimization is only applied to the code written by the developer and not to external libraries;

3. Cython:

Runtime Classification

Before further proceeding with the list of candidate runtime for Python applications, we propose a classification according to several criteria:

Type refers to the runtime infrastructure category that supports a Python application's execution. In particular, we consider 3 types of environments: Interpreter, Compiler and Library; Interpreter refers to the class of environment that does not require any preprocessing of Python source code; Compiler introduces a compilation phase before the execution of the application. Finally, Library requires some modification of the source code;

Runtime refers to the technology supporting the execution of a Python application. This technology can refer to the programming language used to program the interpreter, the target language for a compiler or a library;

JIT optimization refers to the support of just-in-time compilation in the runtime infrastructure supporting the execution of the application;

GC optimization refers to the support of garbage collection in the runtime infrastructure supporting the execution of the application;

Python version(s) refers to the list of Python source code versions supported by the runtime environment.

We did not consider other implementations because either the project was aborted many years ago or it has insufficient Python features. After the inventory of those implementations, we filtered them. To keep only the versions that are still maintained and support most Python features. We then classified them into three categories depending on their integration with the Python code. In Table 4.3, we describe the implementations we kept, each implementation's version, and its category.

Experimental Protocol

As discussed in the previous chapter, our idea is to design a benchmarking solution that allows practitioners to reproduce and extends our benchmarks. This benchmarking solution is also used to answer the research questions addressed in this manuscript. Software settings. For the sake of reproducibility, each experiment runs within a Docker container.

Key Performance Metrics

Our focus will be mainly on CPU energy consumption as it is ten folds more than the DRAM one; since it is a task-based benchmarking, time correlates highly with the energy, and it will be only useful to explain some specific energy behaviors. Thus we do not put much focus on this metric.

Energy measurement. As we know, a program's energy is integral to its power over time.

For our case, we used Intel Running Power Average Limit (RAPL) [START_REF] Khan | Rapl in action: Experiences in using rapl for power measurements[END_REF] to collect the power samples of the running tests. We run POWERAPI [START_REF] Colmant | The next 700 CPU power models[END_REF], to report on measurements collected by Intel RAPL and upload them to a so-called computing machine, then we calculate the Energy using the trapezoidal rule: The motivation for separating measurement collection from energy computations is to reduce any interference with the benchmark, our sensor being a lightweight C program running as a Docker container.

Benchmark Preparation

Input workload. We employed the TOMMTI microbenchmarks suite to benchmark our implementations. 39 TOMMTI is a set of 13 microbenchmarks that examine common language features, such as arithmetic operations, data structures, and input/output manipulations, among others. In addition to these microbenchmarks, we implemented some binary operations to investigate the behavior of the previous implementations when it comes to low-level operations that only work with registries.

To study the energy behavior of the Python implementations, we have to focus on the effect of the implementations and mitigate any side effects, such as the organization of the code or any extra consumption due to the operating system or third-party libraries. Therefore, for each benchmark, we took the implementation written in Python2 as a reference and tried to use it in other implementations as it is. If Python3 does not support it, we converted the code using the official library 2to3. 40 In the case of the libraries using JIT, adding a decorator to the function we want to optimize was enough; if there are other changes, we assume that they alter the original code, which is against our purpose. Each benchmark is isolated in a Docker container for several reasons:

• Isolation: each container has only the benchmark program implemented with a single python runtime to remove any interference between different implementations,

• Deployment: to use the benchmarking machine without extra configurations that may alter the behavior of the operating system toward energy consumption,

• Reproducibility: one of the most frequent benchmark crimes [START_REF] Van Der Kouwe | Benchmarking Crimes: An Emerging Threat in Systems Security[END_REF] in research is the lack of reproducibility-by using Docker, we ensure that each benchmark has an image that will be publicly accessible.

Despite the presence of the official docker images for most of the runtimes, we preferred to build our own using the same reference image to remove any bias due to the OS used in the official images. We used ArchLinux with kernel version 4.9.184 as a base image for all our benchmarks. Benchmark extension. As we have done with the previous chapters, we provide a tool that allows extending the benchmarks with new workloads and new candidates. In the repository listing the Python implementations under study, 41 we propose a dedicated tool to generate new workloads and new candidates. The script generator.py allows practitioners to create new benchmarks by implementing Python code within different interpreters. Then, it generates launcher-benchmark.sh that can be executed to run the associated benchmark. Furthermore, all the successful implementations are stored in a separate directory, and the ones that failed (e.g., mostly because of compatibility issues) are put in a recap file called benchmarkTest.md, where benchmark is the name of the new workload. To add new candidate runtimes, one should add a base Docker file that contains the new implementation. Suppose extra manipulation should be performed on the workload files, such as adding a new decorator or changing some parameters. In that case, they should be added as an extra function in the script generator.py. Finally, they should be included in the Python candidates.

Results & Findings

This part will be dedicated to the results of the experiments and statistical analysis of these results. First and foremost, we want to know whether our samples have a normal distribution. As a result, we ran a Shapiro-Wilk [START_REF] Shapiro | A comparative study of various tests for normality[END_REF] test on our data. Even though some implementations had a p-value higher than a = 0.05 (such as CPython and ActivePython), other implementations like PyPy and Numba presented p-values smaller than 0.01, which leads to rejecting the hypothesis H0 that all the distributions follow a normal distribution. Therefore, to compare the different implementations, we performed the non-parametric Mann-Whitney U test [START_REF] Zimmerman | Comparative power of student t test and mann-whitney u test for unequal sample sizes and variances[END_REF], with CPython2 and CPython3 as base references.

The Mann-Whitney U test results for each implementation are shown in Table 4.5. The first column shows the average energy usage of the implementation, while the second shows the p-value of the test when compared to the reference Python implementation. If the p-value is less than 0.05, the test result is significant, indicating that the energy consumption of the given implementation differs significantly from that of the reference implementation. As can be seen, the majority of the implementations deviate significantly from the reference implementation.

As we want to study the position of these implementations with regards to the reference implementation (aka Cpython2 and Cpython3), we proceed with agglomerative hierarchical cluster analysis (HCA) [START_REF] Johnson | Hierarchical clustering schemes[END_REF]. HCA is a method that groups the different implementations based on their energy consumption. This technique is a bottom-up approach as it starts with each implementation being a single cluster and then merges the two most similar clusters until all the clusters are merged into a single cluster. The similarity between two clusters is measured by the maximum distance between their furthest points. The distance between two points is calculated as the Euclidean distance between the two points. Except for GraalPython, one can notice three main clusters. The first cluster contains the reference implementations (CPython2 and CPython3) and the implementations that are based on interpreters (IronPython, IntelPython, ActivePython, Jython, and Nuitka). Moreover, each implementation is the closest to its reference Python version. The interpreters that are based on other virtual machines, such as Jython with JVM and IronPython with .NET, behave slightly differently from the reference implementation. As for the MicroPython, It is the furthest from the references due to his behavior toward exceptions. The second cluster groups the interpreter implementations that are based on a JIT compiler (PyPy2 and PyPy3).

Closer to them, we find Shedskin, which is a translator that converts Python code into C++ code to be later compiled into binary code. Table 4.5 shows that this cluster is by far the most energy-efficient one. While Numba2 and Numba3 are also JIT libraries, they differ from PyPy due to their manual optimization. Unlike PyPy, which decides the code parts to optimize, Numba gives this responsibility to the developer.

To help practitioners to choose the best implementation for their use case, we have created a chart that describes the behavior of each implementation toward a specific aspect of programming. Figure 4.19 introduces a radar plot for all the implementations that have been used in this experimentation.

This graph summarizes each implementation energy score when executing our benchmark. The lower the energy consumption, the higher the score. We adopted a LOGARITHMIC SCALE to help the practitioners compare numerous implementations due to their large differences.

As one can notice in Figure 4.19, there is no evolution between CPython2 and CPython3. Moreover, IntelPython, and ActivePython all behave similarly to the reference implementation. Therefore, one can conclude that the work done on those interpreters is primarily to improve a specific purpose, not the core interpreter. ActivePython states that its version is focused on security and prepackaged libraries, which explains why it is slower than other versions due to the addition of this security support. 42 As for IntelPython, it is designed for machine learning. Unfortunately, the TOMMTI benchmark is geared for general-purpose programming. Although Nuitka is a compiler, there was no discernible difference in energy consumption. It was even more similar to the reference Python implementation than other interpreters. However, looking at the Nuitka techniques, we can see that they simply encapsulate the Python code with an interpreter into a single executable. Finally, Shedskin reports on the best energy consumption pattern in arithmetic operations. One can conclude it is due to the native type of the variables, unlike in the JIT, where they are treated as objects initially.

Regarding the other VM-based interpreters, Jython and Ipy lacked in terms of energy optimization, which was expected as they were at the beginning of the development stage. The primary purpose of such implementation is to link the bytecode generated by Jython and IronPython with their respective virtual machines.

Unlike the previous interpreters, GraalPython exhibits a specific promise regarding complex algorithms (nested loops in particular). However, it is still in its early stages of development. It is not yet ready for production since some benchmarks, such as array manipulation and sorting, require an abnormally lengthy time.

Threats to validity

Due to the lack of support for most non-conventional Python interpreters, we mainly focus on micro-benchmarks. Except for PYPY, most of the Python implementations do not support extra Python libraries, despite those different implementations being developed to optimize a specific library, such as Numba with Numpy, or IntelPython with machine learning algorithms. Furthermore, we excluded from this study runtime that could not complete our benchmarks because we judged that this was the bare minimum for an implementation to be employed in a real-world application. Besides this, since the goal of this study was to provide a non-intrusive technique to decrease the energy consumption of software, we opted to omit some of the implementations that required extra work on the source code to run on them, such as Cython.

Summary

One can conclude that the Python interpreter used considerably impacts the programs' energy consumption. Furthermore, the absence of a universal solution makes this investigation more intriguing. While most of the candidates employed Python2, Python3 provided better compatibility coverage. Except for Pypy, a standout interpreter, the optimization strategy substantially impacted energy usage because each class was grouped regardless of the Python version. Finally, JIT, a technology that converts code into machine code at runtime, was responsible for the significant gain in energy efficiency. The usage of translators, such as Shedskin confirmed the effect of compiling Python code into machine code. Hence, PyPy, an interpreter with a built-in JIT, is within the same class as the complied ones, such as Shedskin.

Conclusion

As many software services use Python in public and private cloud infrastructures, making Python-based apps more energy efficient will lower ICT carbon emissions. This chapter discusses various ways of optimizing the energy consumption of Python-based applications. It first explains why such a choice is relevant. Then, it studies the energy behavior of Python within its most commonly employed use cases, which are revealed to be web development and machine learning. First, we studied the energy consumption of a machine learning algorithm using a benchmark of 60, 000 entries to train the cifar10-fast model. We discovered that this energy consumption increases exponentially while the model accuracy increases. As a result, a reduction in accuracy can result in considerable energy savings. We investigated how data structures, parallelism, and iterative methods affect energy use.

As for web development, we study a website built with Django, which is one of the most popular web frameworks in the Python community. We found that fetching the data from the database consumes most of the energy during the request processing phase. Therefore, Django-based web servers can use up to ten times less energy if the developer chooses the write strategy to fetch the data.

After that, we analyzed the impact of several iteration mechanisms in Python and their impact on energy consumption. We found that the built-in functions and the best practices for writing Python code are the most energy-efficient ones. This energy efficiency is because most of these built-in functions are written in a lower programming language, C.

Finally, we studied the impact of concurrency on energy consumption for Python applications. We found that the multi-processing strategy is the most energy-efficient one. However, this strategy should be used cautiously. This study demonstrates that the optimal number of processes is equal to the number of physical cores, and exceeding this number will cause the application to lose its efficiency in both performance and energy consumption. As for the multithreading strategy, we found that the Python interpreter is not thread-safe, which leads to a slower performance than the sequential one. However, this also leads to lower power consumption, which sometimes overcomes the lack of performance to make the application more energy-efficient.

In the second section, we presented a non-intrusive technique to optimize the energy consumption of Python-based apps without making substantial changes to the code. This technique consists in guiding the choice of the Python runtime implementation. We started by categorizing and filtering these implementations into three significant classes (compiler, interpreter, and extra libraries). Then, we conducted a series of experiments to compare the energy consumption of these alternatives. Our findings indicate the lack of a general solution and the importance of tuning the Python runtime depending on the application. We found that most interpreters had a similar or worse energy consumption than the official implementation of CPython. The reason behind such behavior was that some implementations focused on specific case studies, such as machine learning or security. In contrast, others focused on the compatibility of the Python code with their platforms, like Jython and IronPython. Finally, regardless of implementation, we demonstrated that using JIT is the most efficient technique to boost the energy efficiency of Python-based programs.

We believe that our contributions will benefit a broad spectrum of legacy systems, reducing ICT's carbon footprint and lowering cloud bills for these services' resources.

Chapter 5

Impact of Java Virtual Machine Configurations on Energy Consumption

Introduction

As reported in the state of the art, Java is one of the most popular programming languages adopted by practitioners. In fact, in 2022, Java will be second only to Python, according to PYPY 1 . Furthermore, if we take into consideration legacy applications, Java becomes the most used programming language. According to the TIOBE index, Java was the most frequently used language from 2002 until 2017, and it remained in the top 5 after that 2 . In addition to its popularity, Java exhibits an interesting behavior when it comes to energy consumption and performance, Java applications can be at the same time one of the most energy-efficient or hungry solutions. As we have seen in the previous chapter, an inappropriate combination of parameters can drive Java applications from the top language to the bottom just by setting the wrong parameters. in this chapter, we want to dig deeper into this aspect of Java and study its runtime. This chapter thus focuses on the impact of the runtime of Java applications on energy consumption.

Characteristics of JVM

Java's portability is a core design goal, which means that Java applications will work the same on any operating system and any hardware. As we see in figure 5.1, Instead of machine code, this is accomplished by compiling Java language code to an intermediate representation known as Java bytecode. Java bytecode instructions are similar to machine code, but they are executed by a JVM rather than directly on the machine. As a result, Java programs will run slower and consume more memory than C++ programs. However, the Just-in-Time (JIT) compilation, which is built into the JVM, improves performance by converting bytecode to machine code at runtime. Furthermore, an automatic garbage collector (GC) is used by Java to manage memory throughout the object lifecycle and to recover memory when objects are no longer in use.

Research questions

In this section, we will investigate the following research questions: RQ 1: What is the impact of existing JVM distributions on the energy consumption of Java-based software services? RQ 2: What are the relevant JVM settings that can reduce the energy consumption of a given software service?

To answer those research questions, we conduct an empirical study to highlight the impact of this runtime.

Experimental Protocol

To investigate the effect that can have the JVM distribution choice and/or parameters on software energy consumption, we conducted a wide set of experiments on a cluster of machines and used several established Java benchmarks and JVM configurations.

Measurement Contexts

Software Settings. For the sake of reproducibility, each experiment runs within a Docker container based on SDKMAN3 image and Alpine docker. 4Hardware Settings. To report on reproducible measurements, we used the cluster Dahu from the G5K platform [START_REF] Balouek | Adding virtualization capabilities to the Grid'5000 testbed[END_REF] for most of our experiments. This cluster is composed of 32 identical compute nodes, which are equipped with 2 Intel Xeon Gold 6130 and 192 GB of RAM. Our experimental protocol enforces that the software under test is the only process executed on the node configured with a very minimal Linux Debian 9 (4.9.0 kernel version). The minimal OS configuration ensures that only mandatory services and daemons are kept active

Workload

We ran our experiments across 12 Java benchmarks we picked from OpenBenchmarking.org. 5This includes 5 acknowledged benchmarks from the DACAPO benchmark suite v. 9.12 [START_REF] Blackburn | The DaCapo benchmarks: Java benchmarking development and analysis[END_REF], namely Avrora, H2, Lusearch, Sunflow and PMD, that have been widely used in previous studies and proven to be accurate for memory management and computer architecture [START_REF] Lengauer | A comprehensive java benchmark study on memory and garbage collection behavior of dacapo, dacapo scala, and specjvm2008[END_REF][START_REF] Kalibera | A black-box approach to understanding concurrency in dacapo[END_REF]. It consists of open-source and real-world applications with non-trivial memory loads. Then, we also considered 7 additional benchmarks from the RENAISSANCE benchmark suite [123,[START_REF] Prokopec | Renaissance: benchmarking suite for parallel applications on the JVM[END_REF], namely ALS, Dotty, Fj-kmeans, Neo4j, Philosophers, Reaction and Scrabble, which offer a diversified set of benchmarks aimed at testing JIT, GC, profilers, analyzers, and other tools. The benchmarks we picked from both suites exercise a broad range of programming paradigms, including concurrent, parallel, functional, and object-oriented programming. Table 5.2 summarizes the selected benchmarks with a short description. Meanwhile, Figure 5.2 highlights the scope of each benchmark from the test suite.

Metrics and Measurement

Since the goal of this study is the green aspect of JVM, our key metric will be the energy consumed by the job completed for each JVM configuration. In addition to the energy consumption, we collected additional metrics to explain the reasons behind the behavior of each experiment. Those additional metrics are:

• execution time,

• number of threads.

Energy Measurements. We used Intel RAPL as a physical power meter to analyze the energy consumption of the CPU package and the DRAM. RAPL is one of the most accurate tools to report on the global energy consumption of a processor [START_REF] Khan | Rapl in action: Experiences in using rapl for power measurements[END_REF][START_REF] Desrochers | A validation of dram rapl power measurements[END_REF]. We note that, due to CPU energy consumption variations issues [START_REF] Ournani | Taming energy consumption variations in systems benchmarking[END_REF], we used the same node for all our experiments. Moreover, we tried to be very careful, while running our experiments, not to fall in the most common benchmarking "crimes" [START_REF] Van Der Kouwe | Benchmarking Crimes: An Emerging Threat in Systems Security[END_REF]. Every single experiment, therefore, reports on energy metrics obtained from at least 20 executions of 50 iterations per benchmark. All of our experiments are available for use/reproducibility from our anonymous repository. 6Number of threads. To collect the number of active threads used by the experiment, we use the command top and record at fixed intervals.

Extension

We also added an extension to the protocol to allow the user to run the same experiment with different configurations. The package is available in the GitHub repository. 7 To add extra jvm candidates for the benchmark applications, we added a new configuration file jvm.sh in the root directory of the repository, where we put the name and the version of the JVM to be used. For the input workload,the benchmarks should be provided in the benchmarks directory. As for extra metrics, one can create a new script file that monitors the experiment and record the metrics. We provide some examples, such as recordpower.sh to measure the instantaneous power and recordthreads.sh to measure the number of active threads during the experiment.

For faster experiments, we propose JREFERRAL 8 , an open-source tool that automatically compares the JVM energy consumption and recommends the most energy-efficient configuration for a given Java application. We further discuss this tool in Section 7.1.

Experiments & Results

Energy Impact of JVM Distributions

Job-oriented applications. To answer our first research question, we executed 62, 400 experiments by combining the 52 JVM distributions with the 12 Java benchmarks, thus reasoning on 100 energy samples acquired for each of these combinations. Figure 5.3 first depicts the accumulated energy consumption of the 12 Java benchmarks per JVM distribution and major versions (or LTS when unavailable). Concretely, We measure the energy consumption of each of the benchmarks and compute the ratio of energy consumption compared to HOTSPOT-8, which we consider as the baseline in this experiment. Then, we sum the ratios of the 12 benchmarks and depict them as percentages in Figure 5.3.

One can observe that, along with time and versions, the energy efficiency of JVM distributions tends to improve (10% savings), thus demonstrating the benefits of optimizations delivered by the communities. Yet, one can also observe that energy consumption may differ from one distribution to another, thus showing that the choice of a JVM distribution may have a substantial impact on the energy consumption of the deployed software services. For example, one can note that J9 can exhibit up to 15% of energy consumption overhead, while other distributions seem to converge towards a lower energy footprint for the latest version of Java. As GRAALVM adopts a different strategy focused on LTS support, one can observe that its recent releases provide the best energy efficiency for Java 11, but recent releases of other distributions seem to reach similar efficiency for Java 13 and above, which are recent versions not supported by GRAALVM yet.

Interestingly, this convergence of distributions has been observed since Java 11 and coincides with the adoption of DCE VM by HOTSPOT. Ultimately, 3 clusters of JVMs that encompass JVMs with similar energy consumption can be seen through Figure 5.3: J9, the HOTSPOT and its variants, and GRAALVM. Additional detailed figures to illustrate the evolution of energy consumption per benchmark/JVM are made available from the online repository. 9Then, Figure 5.4 depicts the evolution of the energy consumption of the 12 benchmarks, when executed on the HOTSPOT JVM. Figure 5.4 reports on the energy consumption variation of individual benchmarks, using HOTSPOT-8 as the baseline. Our results show that the JVM version can severely impact the energy consumption of the application. However, unlike Figure 5.3, one can observe that, depending on applications, the latest JVM versions can consume less energy (60% less energy for Scrabble) or more energy (25% more energy for the Neo4J). It is worth noticing that the energy consumption of some benchmarks, such as Reactors, exhibit large variations across JVM versions due to experimental features and changes that are not always kept when releasing LTS versions (version 11 here). For example, the introduction of VarHandle to allow low-level access to the memory order modes available in JDK 9 10 and work along Unsafe Class was removed from JVM 11. 11 Given that the wide set of distributions and versions seems to highlight 3 classes of energy behaviors, the remainder of this chapter considers the following distributions as relevant samples of JVM to be further evaluated: 20.2.0.r11-grl (GRAALVM), 15.0.1-open (HOTSPOT-15), 15.0.21.j9 (J9). We also decided to keep the 8.0.275-open (HOTSPOT-8) as a baseline JVM for some figures to highlight the evolution of energy consumption over time/versions. Figure 5.5 further explores the comparison of energy efficiency of the JVM distributions per benchmark. One can observe that, depending on the benchmark's focus, the energy efficiency of JVM distributions may strongly vary. When considering individual benchmarks, J9 performs the worst for at least 6 out of 12 benchmarks-i.e., the worst ratio among the 4 tested distributions. Even though, J9 can still exhibit a significant energy saving for some benchmarks, such as Avrora, where it consumes 38% less energy than HOTSPOT and others.

Interestingly, GRAALVM delivers good results overall, being among the distributions with low energy consumption for all benchmarks, except for Reactors and Avrora. Yet, some differences still can be observed with HOTSPOT depending on applications. The newer version of HOTSPOT-15 was averagely good and, compared to HOTSPOT-8, it significantly enhances energy consumption for most scenarios. Finally, Neo4J is the only selected benchmark where HOTSPOT-8 is more energy efficient than HOTSPOT-15.

Service-oriented applications. In this section, instead of considering bounded execution of benchmarks, we run the same benchmarks as services for 20 minutes, and we compare the average power and total requests processed by each of the 3 JVM distributions. Globally, the results showed that the average power when using GRAALVM, HOTSPOT, and OPENJ9 is often equivalent and stable over time. This means that the energy efficiency observed for some JVM distributions with job-oriented applications is mainly related to shorter execution times, which incidentally results in energy savings. Nonetheless, we can highlight two interesting observations for two benchmarks whose behaviors differ from others. First, the analysis of the Scrabble benchmark experiments showed that, in some scenarios, some JVMs can exhibit different power consumptions. Figure 5.6 depicts the power consumed by the 3 JVM distributions for the Scrabble benchmark. One can clearly see that GRAALVM requires an average power of 109 W, which is 9 W higher than HOTSPOT-15 and 15 W higher than J9. When it comes to the number of requests processed by Scrabbles during that same amount of time, GRAALVM completes 5, 336 requests, against 3, 595 for HOTSPOT and 2, 603 for J9, as shown in Table 5.3. The higher power usage for GRAALVM helped in achieving a high amount of requests, but also the fastest execution of every request, which was 40% faster on GRAALVM. Thus, GRAALVM was more energy efficient, even if it uses more power, which confirms the results observed in Figure 5.5 for this benchmark. The second interesting situation was observed on the Dotty benchmark. More specifically, during the first 100 seconds of the execution of the Dotty benchmark on all evaluated JVMs. At the beginning of the execution, GRAALVM has a slightly lower power consumption, is faster, and consumes 10% less energy. After about 150 seconds, the power differences between the 3 JVMs is barely noticeable. One can, however, notice the effect of the JIT, as HOTSPOT takes advantage over GRAALVM and becomes more energy efficient. In total, HOTSPOT completes 597 requests against 510 for GRAALVM and 381 for J9, as shown in Table 5.3. HOTSPOT was thus the best choice in the long term, which explains why it is always necessary to consider a warm-up phase and wait for the JIT to be triggered before evaluating the effect of the JVM or the performance of an application. This is precisely what we did in our experiments, and it is why HOTSPOt was more energy efficient than GRAALVM in Figure 5.5; therefore, ignoring the warm-up phase would have been misleading.

To answer RQ 1, we conclude that-while most of the JVM platforms perform similarlywe can cluster JVMs in 3 classes: HOTSPOT, J9, and GRAALVM. The choice of one JVM of these classes can have a major impact on software energy consumption, which strongly depends on the application context. When it comes to the JVM version, the latest releases tend to offer the lowest power consumption, but experimental features should be carefully configured, thus further questioning the impact of JVM parameters.

Energy Impact of JVM Settings

The purpose of our study is not only to investigate the impact of the JVM platform on energy consumption, but also the different JVM parameters and configurations that might have a positive or negative effect, with a focus on 3 available settings: multi-threading, JIT, and GC.

Multithreading

The purpose of this phase is to investigate the impact of JVM thread management strategies on energy consumption. This encompasses exploring if the management strategies of applicationlevel parallelism (so-called threads) result in different energy efficiencies, depending on JVM distributions.

Investigating such a hypothesis requires a selection of highly parallel and CPU-intensive benchmarks, which is one of the main criteria for our benchmark selection. As no tool can accurately monitor the energy consumption at a thread level, we monitor the global power consumption and CPU utilization during the execution using RAPL for the energy, and several Linux tools for the CPU usage (htop, cpufreq). Knowing that most of the benchmarks are multi-threaded jobs that use multiple cores, further analysis of thread management is required to understand the results of our previous experiments. We thus selected the benchmarks that highlighted the highest differences along JVM distributions from Figure 5.5, namely Avrora and Reactors. We studied their multi-threaded behavior to optimize their energy efficiency. Figure 5.8 delivers a closer look at the thread allocation strategies adopted by JVM. First, Figure 5.8a illustrates the active threads count evolution over time (excluding the JVM-related threads, usually 1 or 2 extra threads depending on the execution phase) for Avrora. One can notice through the figure that J9 exploits the CPU more intensively by running much more parallel threads compared to other JVMs (an average of 5.1 threads per second for J9 while the other JVMs do not exceed 1.5 thread per second). Furthermore, the number of context switches is twice bigger for J9, while the number of soft page faults is twice smaller. The efficient J9 thread management explains why running the Avrora benchmark took much less time and consumed less energy, given that no other difference for the JIT or GC configuration was spotted between the JVMs. Another key reason for the J9's efficiency for the Avrora benchmark is memory allocation, as OpenJ9 adopts a different policy for the heap allocation. It creates a non-collectible thread local heap (TLH) within the main heap for each active thread. The benefit of cloning a dedicated TLH is the fast memory access for independent threads: each thread has its heap and no deadlock can occur.

The second example in Figure 5.8b depicts the active threads evolution over time of the Reactors benchmark. In this case, all the JVMs have a close average of threads per second. Nevertheless, one can still observe that HOTSPOT-15 and J9 keep running faster, which confirms the results of Figure 5.5, where both JVMs consume much less energy compared to GRAALVM and HOTSPOT-8. This difference in energy consumption between benchmarks can be less likely caused by thread management for the Reactors benchmark, as HOTSPOT-8 reports on a higher average of active threads. However, the TLH mechanism was not as efficient as for the Avrora benchmark, as dedicating a heap for each thread can also cause some extra memory usage for data duplication and synchronization, especially if a lot of data is shared between threads.

In conclusion, JVM thread management can sometimes constitute a key factor that impacts software energy consumption. However, we suggest checking and comparing JVMs before deploying software, especially if the target application is parallel and multi-threaded.

Just-in-Time Compilation

The purpose of experiments on JIT is to highlight the different strategies that can impact software energy consumption within a JVM and between JVMs. We identified a set of JIT compiler parameters for every JVM platform.

For J9, we considered fixing the intensity of the JIT compiler at multiple levels (cold, warm, hot, veryhot, and scorching). 12 The hotter the JIT, the more code optimization to be triggered. We also varied the minimum count method calls before a JIT compilation occurs (10,[START_REF] Gajewski | Analysis and Comparison of the Spring Framework and Play Framework Performance, Used to Create Web Applications in Java[END_REF][START_REF] Naresh | Education after covid-19 crisis based on ict tools[END_REF], and the number of JIT instances threads (from 1 to 7). For HOTSPOT-15, we conducted experiments while disabling the tiered compilation (that generates compiled versions of methods that collect profiling information about themselves), and we also varied the JIT maximum compilation level from 0 to 4, we also tried out HOTSPOT with a basic GRAALVM JIT. We note that level 0 of JIT compilation only uses the interpreter, with no real JIT compilation. Levels 1, 2, and 3 use the C1 compiler (called client-side) with different amounts of extra tuning. The JIT C2 (also called server-side JIT) compiler only kicks in at level 4.

For GRAALVM, we conducted experiments with and without the JVMCI (a Java-based JVM compiler interface enabling a compiler written in Java to be used by the JVM as a dynamic compiler). We also considered both the community and economy configurations (no enterprise). A JIT+AOT (Ahead Of Time) disabling experiment has also been considered for all the 3 JVM platforms. Table 5.4 reports on the energy consumption of the experiments we conducted for most of the benchmarks and JIT configurations under study.

The p-values are computed with the Mann-Whitney test, with a null hypothesis of the energy consumption being equal to the default configuration. The p-values in bold show the values that are significantly different from the default configuration with a 95% confidence, where the values in green highlight the strategies that consumed significantly less energy than the default (less energy and significant p-value).

For J9, we noticed that adopting the default JIT configuration is always better than specifying a custom JIT intensity. The warm configuration delivers the closest results to the best results observed with the default configuration. Moreover, choosing a low minimum count of method calls seems to have a negative effect on execution time and energy consumption. The only parameter that can give better performance than the default configuration in some cases is the number of parallel JIT threads-using 3 and 7 parallel threads-but is not statistically significant.

For GRAALVM, the default community configuration is often the one that consumes the least energy. Disabling the JVMCI can-in some cases-have a benefit (16% of energy consumption reduction for the H2 benchmark), but still gave overall worst results (80% more energy consumption for the Neo4J benchmark). In addition, switching the economy version of the GRAALVM JIT often results in consuming more energy and delaying the execution.

For HOTSPOT, keeping the default configuration of the JIT is also mostly good. The usage of the C2 JIT is often beneficial (JIT level 4) in most cases while using the GRAALVM JIT reported similar energy efficiency. Yet, some benchmarks showed that using only the C1 JIT (JIT level 1) is more efficient and even outperforms the usage of the C2 compiler. 10% on Avrora and 30% on Pmd are examples of energy savings observed by using the C1 compiler. However, being limited to the C1 compiler can also cause a huge degradation in energy consumption, such as 32% and 34% of additional energy consumed for the Dotty and FJ-kmeans benchmarks, respectively. Hence, if it is a matter of not using the C2 JIT, the experiments have shown that the level 1 JIT is always the best, compared to levels 2 or 3 that also use the C1 JIT, but with more options, such as code profiling that impacts negatively the performance and the energy efficiency. Level 0 JIT compilation should never be an option to consider. No p-value has been computed for Level 0, due to the limited amount of iterations executed with this mode (very high execution time, clearly much more consumed energy).

Globally, we conclude through these experiments that keeping the default JIT configuration was more energy efficient in 80% of our experiments and for the 3 classes of JVMs. This advocates using the default JIT configuration that can often deliver near-optimal energy efficiency. Although, some other configurations, such as using only the C1 JIT or disabling the JVMCI could be advantageous in some cases.

Garbage Collection

Changing or tuning the GC strategy has been acknowledged to impact the JVM performances [START_REF] Libič | On the limits of modeling generational garbage collector performance[END_REF]. To investigate if this impact also benefits energy consumption, we conducted a set of experiments on the selected JVMs. We considered different garbage collector strategies with a limited memory quantity of 2 GB, and recorded the execution time and the energy consumption. The tested GC strategies options mainly vary between J9 and the other 2 JVMs, as detailed in Table 5.5.

For HOTSPOT and GRAALVM, we also considered many GC policies, as described in Table 5.6. Furthermore, other GC settings have also been tested for all JVM platforms, such as the pause time, the number of parallel threads and concurrent threads and tenure age.

Table 5.7 summarizes the results of all the tested GC strategies with our selected benchmarks and the p-values of the Mann-Whitney test, with a null hypothesis of the energy consumption being equal to the default configuration with a 95% confidence. The p-values in bold show the values that are significantly different from the default configuration, whereas the values in green highlight the strategies that consumed significantly less energy than the default. For GRAALVM, one can see that the GC default configuration is efficient in most experiments, compared to other strategies. The main noticeable impact is related to the ParallelGC and ParallelOldGC. In fact, the ParallelGC can be 13% more energy efficient in some applications with a significant p-value, such as Reactors, compared to default. However, the same GC strategy can cause the software to consume twice times more, as for the Neo4j benchmark, due to the high communications between the GC threads, and the fragmentation of the memory.

For J9, the default Gencon GC causes the software to report an overall good energy efficiency among the tested benchmarks. However, other GC can cause better or worse energy consumption than Gencon depending on workloads. Using the Metronome GC consumes 35% less energy for the ALS benchmark and 17% less energy for the Sunflow benchmark, but it also consumes 100% more energy for the Neo4j benchmark and 28% more energy for Reactors. The reason is that Metronome occurs in small preemptible steps to reduce the GC cycles composed of many GC quanta. This suits well for real-time applications and can be very beneficial when long GC pauses are not desired, as observed for ALS. However, if the heap space is insufficient after a GC cycle, another cycle will be triggered with the same ID. As Metronome supports class unloading in the standard way, there might be pause time outliers during GC activities, inducing a negative impact on the Neo4j execution time and energy consumption.

The same goes for the Balanced GC that tries to reduce the maximum pause time on the heap by dividing it into individually managed regions. The Balanced strategy is preferred to reduce the pause times that are caused by global GC, but can also be disadvantageous due to the separate management of the heap regions, such as for ALS where it consumed about three times the energy consumption, compared to the default Gencon GC. On the other hand, the Optthruput GC, which stops the application longer and less frequently, gave very good overall results and sometimes even outperformed the Gencon GC by a small margin. Other JVM parameters, such as the ConcurrentScavenge or noAdaptiveTenure did not have a substantial impact during our experiments.

Finally, the results of HOTSPOT shared similarities with GRAALVM. The ParallelGC happened to give better (6% for Dotty) or worst (10% for Neo4j) energy efficiency compared to the default GC. On the other hand, ParallelOldGC and Serial GC gave better results than the default G1 GC. More specifically, the second one consumed 30% and 6% less energy than the default GC for the Neo4j and Dotty benchmarks, respectively. The most interesting result for HOTSPOT is the 30% energy reduction obtained with the Serial GC. This last was also more efficient on ALS (6% less energy), compared to the default G1 GC, due to its single-threaded GC that only uses one CPU core.

Unfortunately, we cannot convey predictive patterns on how to configure the GC to optimize energy efficiency. However, some considerations should be taken into account when choosing the GC, such as the garbage collection time, the throughput, etc. Other settings are less trivial to determine, such as tenure age, memory size, and GC thread count. Experiments should thus be conducted on the software to tune the most convenient GC configuration to achieve better energy efficiency in production.

Therefore, we noticed during our experiments that, even if using the default GC configuration ensures an overall steady and correct energy consumption, we still found other settings that reduce that energy consumption in 50% of our experiments. Tuning the GC according to the hosted app/benchmark is thus critical to reducing energy consumption.

To answer RQ 2, we conclude that users should be careful while choosing and configuring the garbage collector as substantial energy enhancements can be recorded from one configuration to another. The default GC consumes more energy than other strategies in most situations. However, keeping the default JIT parameters often delivers near-optimal energy efficiency. In addition, the JVM platforms can handle differently multi-threaded applications and thus consume a different amount of time/energy. Dedicated performance tuning evaluations should therefore be conducted on such software to identify the most energy-efficient platform and settings.

Threats to Validity

Several issues may affect the validity of our work. First, we have the use of the Intel RAPL, one of the most accurate available tools to measure the energy consumption of software [START_REF] Khan | Rapl in action: Experiences in using rapl for power measurements[END_REF][START_REF] Desrochers | A validation of dram rapl power measurements[END_REF]. However, RAPL only gives the global energy consumption and no fine-grained measures at process or thread levels. We used bare-metal hardware with a minimal OS and turned off all the non-essential services and daemons to limit the overhead that the OS may add to the execution, even if it is not substantial [START_REF] Ournani | Taming energy consumption variations in systems benchmarking[END_REF].

Another measurement issue is the CPU energy variation within machines (cf. Chapter 3), thus we executed all the comparable tests on the same node and with the recommended settings to mitigate this threat.

Benchmarks' execution time could also constitute a more subtle threat to the validity of our work, especially for some benchmarks that run fast, such as the Pmd benchmark. We thus gave a lot of attention to how long the benchmark is running for the hardware we used, and we tuned the input data workloads to execute benchmarks for at least many (from 10 to hundreds) seconds. Experiments ran at least 30 times to compute the average consumption and the associated standard deviation, therefore reasoning over reasonable dispersion around the average. How generalizable are our results? We believe that our study conclusions and guideline remain empirical, as we do not intend to generalize any result we obtained for some JVM or benchmark. We provide practitioners with some prerequisites to check before software deployment to reduce the software energy footprint by considering the JVM and its settings.

Tools and contributions

J-Referral is an open-source tool designed to assist developers and practitioners in selecting the most energy-efficient JVM configuration for their software. Table 5.8 illustrates an example of the final report returned by J-Referral. The tool was tested for 2 Java projects: Zip4J13 and K-nucleotide. 14 Zip4J runs a large file compression, while K-nucleotide extracts a DNA sequence, and updates a hashtable of k-nucleotide keys to count specific values. The short report presented in Table 5.8 shows the ratio of potential energy saving between the most and least energy-consuming tested JVM (40% and 70% energy savings for Zip4J and K-nucleotide, respectively). Options are available for J-Referral to obtain much more detailed reports including execution time, DRAM usage, split DRAM vs. CPU consumption, etc. The tool is available as open-source software (OSS) from our GitHub repository. 15

conclusion

This chapter describes an empirical investigation of contrast in energy consumption that some of the most well-known and widely supported JVM platforms can exhibit as well as the important factors that can influence this energy consumption. During our experiments, we investigated 12 well-known and diverse-purpose Java benchmarks, as well as 52 JVMs, including many versions of 11 different distributions. Our findings revealed that various JVMs share energy efficiency and may be classified into three major classes: HOTSPOT, J9, and GRAALVM. However, the three JVM classes can show substantially different energy efficiency depending on the workloads and/or application. While there was no clear champion in terms of energy consumption,GRAALVM recorded the highest energy efficiency for the most of benchmarks. Nevertheless, depending on the workload, each JVM can achieve higher or worse efficiency. One explanation could be thread management policy, as seen with J9 when running Avrora.

Additionally, numerous JVM settings can cause changes in energy consumption. Our experiments revealed that the JVM's default JIT compiler is frequently near-optimal in at least 80% of the experiments.

On the other hand, the default GC outperformed the other strategies in just half of our experiments, with some significant benefits reported when employing different GCs based on the application characteristics. Our primary observations and recommendations can be stated as follows: i) testing software on the 3 classes of JVM and identifying the one that consumes the least is a good practice, especially for multi-threading purposes, ii) while the JVM default JIT give often good energy consumption results, some settings may improve the energy consumption and could be tested, iii) the choice of the GC may lead to a large impact on the energy consumption in many situations, thus encouraging a careful tuning of this parameter before deployment.

To make the above principles easier to implement, we propose J-Referral, a tool that recommends the best energy-efficient JVM distribution and configuration from among more than a hundred options. It generates a comprehensive report on the energy consumption of both CPU and DRAM components for each JVM distribution and/or configuration to assist the user in selecting the one with the lowest consumption for Java applications.

Introduction

Nowadays, web applications dominate online systems. From Google to Facebook, web applications are widely deployed across organizations and continuously accessed by endusers for personal and professional daily tasks. In practice, the development of these web applications heavily relies on a vast ecosystem of web frameworks, intended to ease and foster the development process.

However, once deployed, the applications developed with such web frameworks do not exhibit the same performance as reported by the Web Framework Benchmarks periodically published by the TECHEMPOWER company 1 .

Thanks to such benchmarks, developers can make informed decisions on the most efficient technology to adopt to implement their web applications.

Unfortunately, when selecting a web framework, developers and benchmark providers focus primarily on popularity and performance criteria, with little regard for the resource consumption implications of their choice. This is especially unfortunate, given that developers increasingly turn to cloud providers to host their web applications. In such a context, the energy consumption of web applications is a critical factor, as it directly impacts the cost of hosting the applications. While cloud providers provide a convenient elastic provision of resources to scale based on application requirements, this convenience may come at a high cost to their business. Indeed, the energy consumption of cloud data centers is a primary concern for cloud providers, as it represents a significant portion of their operational costs.

Beyond the economic cost of web applications, one can also question the global impact of web applications on worldwide carbon emissions.

Given the tremendous success of web applications, their deployment has severely increased over the last few years, thus causing a rebound effect on the power consumption of server infrastructure hosted or supported by cloud providers.

While one can challenge the relevance of features that developers continuously deploy to keep engaging end-users, reconciling economic and environmental concerns remains an open challenge to address.

Given this context, this chapter intends to address this challenge by investigating the energy footprint of web frameworks. In particular, we aim to support the developers of web applications with relevant guidelines that can help them to choose the web framework that is not only the most popular or provides the best performance but also exhibits a low energy footprint.

By minimizing the energy consumed to process user requests with no service quality penalty, developers can reduce the operational cost of their web applications and contribute to reducing worldwide carbon emissions from ICT.

To achieve this objective, we consider two study cases. First, we study the impact of programming languages while handling requests from the Remote Procedure Call protocol (RPC). RPC is a protocol that allows a computer to call a procedure stored in another address space (usually on another computer on a shared network) without the user having to explicitly code the details for this remote interaction. We compare multiple implementations of GRPC library, a new RPC library developed by Google and based on PROTOCOL BUFFERS, a popular serialization protocol. As for section 6.2, we study the impact of web framework stacks on energy consumption. To do so, we implement a simple web application using the most popular web frameworks and compare their performance, latency, and energy consumption. We leverage the TECHEMPOWER Web Framework Benchmarks to incorporate server-side energy measurements obtained from a software-defined power meter, named POWERAPI [START_REF] Fieni | Smartwatts: Self-calibrating softwaredefined power meter for containers[END_REF]. These measurements are then analyzed in depth to understand the critical criteria that can impact the power consumption of web frameworks and derive guidelines for supporting developers in picking the most energy-efficient web frameworks according to their requirements.

Investigating Remote Procedure Call Frameworks

With the success of the Internet and the emergence of cloud technologies, many communication protocols compete to take the lead. In particular, most software architectures are now based on multi-services and micro-service technologies. Moreover, to cope with the increased versatility of developers, multiple companies choose to open their services to different programming languages. This approach is interesting because it wants to use the best parts of each programming language to meet different needs. However, the challenge nowadays is to make the bridge between these platforms. We have many initiatives, such as OpenAPI, that try to create a taxonomy for RESTful APIs, while other approaches implement all the different interfaces of the protocol by themselves, such as The Remote Procedure Call (aka RPC) protocol. In order to provide a universal implementation of this protocol, Google developed GRPC, a contemporary open-source RPC framework that may run in any environment. Pluggable load balancing, tracing, health monitoring, and authentication support may efficiently link services within and across data centers. It is also valuable for the final mile of distributed computing, connecting devices, mobile apps, and browsers to backend services.

Research Questions

In this section, we first explore the ease of implementation of this protocol, and then we will try to answer the following research questions: RQ 1: How do RPC implementations consume energy regarding the number of concurrent clients?

RQ 2: How do RPC implementations consume energy concerning the size of the incoming request?

Experimental Protocol

To answer these questions, we will define an interface using the Proto22 language, and then we will implement it in multiple servers, each one using a different programming language. We will then compare the energy consumption of each implementation using a custom version of GHZ3 , a tool that allows us to stress test our servers. This fork of GHZ allows us to measure the server's energy consumption during the test's execution and synchronize it with other performance statistics such as Tail latency and the number of requests per second the server can reach. Figure 6.1 show the experimental protocol. We will use different messages inserted in the file helloworld.proto. The rest of this part will be dedicated to explaining each aspect of the experiment. Client & server environments. To limit networks' impact on the experiments, we run both the client and the server on the same machine. However, we isolate each part on a separate CPU socket to reduce the noise that the client might have on the server and vice-versa. To do so, for each iteration, we always run the same client on socket 0 and the server that we want to test on socket 1. Both the server and the client use the whole socket for their experiment. In addition, all the additional services, such as the kernel and HwPC sensor, are run on socket 0. Therefore, the only process being executed in socket 1 is the server that we benchmark and monitor.

Key Performance Metrics

Energy measurements. To report on energy consumption, we used the HWPC sensor 4 , based on Intel RAPL technology, one of the most accurate tools to measure the CPU's and DRAM's energy consumption [START_REF] Hackenberg | Power measurement techniques on standard compute nodes: A quantitative comparison[END_REF][START_REF] Hackenberg | An energy efficiency feature survey of the intel haswell processor[END_REF]. For better accuracy, we ran the HwPC sensor with a frequency of 1 Hz, and we used the same machine for all the experiments to reduce the variability [START_REF] Ilsche | Power measurements for compute nodes: Improving sampling rates, granularity and accuracy[END_REF].

Performances. For better accuracy and more details, We updated the open-source RPC benchmarking tool GHZ (https://ghz.sh/). The modified version allows us to monitor the average power for each request from both the server and the client sides. The new version is available in the repository. 5Input Workload

The purpose of the experiment is to analyze the behavior of different GRPC implementations. Therefore, we have two kinds of workloads: number of clients is the number of concurrent clients that we want to benchmark (handled by the GHZ client), and the payload reflects the size of each request (varies from 50 Bytes up to 10 MB). The client consumes the protocol description found in the file helloworld.proto to generate an implementation for the message and then forks multiple instances that send the same request to the server.

Candidates

The server implementations are based on the official implementation by Google for most of the languages. Each server uses 16 cores and is limited to 512 MB of RAM. Each implementation is packaged as a Docker image, allowing us to add new implementations easily.

Extension

For the sake of experimental extensibility, we provide a GitHub repository that contains the implementation of all our experiments. 6 Adding a new RPC candidate can be achieved by creating a new Docker image and putting it in a new folder named after the language. As for the workload, it can be extended easily by adding new files in the folder payload and by changing the number of clients in the benchmarking tool.

Results & Findings

Preparing the Data Before we start our Analysis, we will first clean the data from the outliers using the interquartile range (IQR) method [START_REF] Leys | Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median[END_REF]. For each implementation, we will calculate the Q3 and Q1 of the population; then, we will exclude all the values outside the range [Q1 -1.5 × IQR, Q3 + 1.5 × IQR].

[RQ 1:] How do RPC implementations react to the number of clients?

In this part, we will analyze the behavior of the different implementations regarding the number of clients. Power behavior First, we start with the overall power consumption for each server implementation. Figure 6.2 shows a heatmap of the average power consumption of each implementation. As one can notice, there are two main scenarios. The first one is when the number of clients is lower than 100, which we will call lite scenario, and the second one, the stressed scenario, is when the number of clients is more exceeds 100.

Lite scenario

The benchmarked implementations can be grouped into two classes:

1. Energy-efficient frameworks where most of the framework's power consumption is around 33 Watts.

2. Energy-greedy frameworks where the average power consumption is higher than 45 Watts.

In each programming category, we observe both energy-efficient and energy-greedy behaviors. Therefore, we conclude that it depends more on the library's implementation than the programming language category. Scala and Kotlin are excellent examples to support this hypothesis, as both run on the same virtual machine as Java (OpenJDK 16.1). However, their average power is 130% higher than the Java implementation.

Stressed scenario Although the same classes remained the same, not all languages had the same evolution. Here we can observe a correlation with the category of the programming language rather than the implementation itself. We can quickly point out that the average amount of power used by VM-based languages doubles when they have more than 100 clients simultaneously. Except for PHP, all the interpreted and compiled languages preserved their energetic behavior. Our hypothesis points to the JIT since it compiles the code and makes it run faster, hence stressing the CPU. An interesting behavior has been noticed for the GraalVM: decreasing energy consumption when increasing the number of clients. This is related to the performance drop, probably due to the bottleneck situation where the GraalVM could not handle more than 100 clients simultaneously.

Performance Behaviour this paragraph will cover the number of requests per second processed per server without looking at its energy. Here, we consider three observable variables:

• Satisfaction ratio: how many requests have been satisfied among the total requests,

• Request Per Seconds: The number of requests that have been answered from the server,

• Tail Latency at 99%: one of the best metrics to evaluate the performances of a server.

Satisfaction ratio Most of the considered frameworks satisfy all the requests by reducing the number of requests per second or increasing the processing time. However, some frameworks, such as Dart or Scala, have taken a different approach, maintaining a specific latency limit even if not all requests are answered. Furthermore, we tend to observe this behavior among other frameworks, such as Python or Asynchronous NodeJS, when the number of clients exceeds 800. To dive more into this behavior, the reader can refer to the following GitHub repository https://github.com/chakib-belgaid/grpc_analysis 3 presents the number of requests per second for each implementation. The heatmap is colored with a logarithmic scale to visualize the results better. As one can observe, most of the servers hit their RPS limit after five clients and 100 clients for VM-based servers, and after this, the number keeps constant, decreasing the average RPS per client. .Net server is the most performant, followed by Java and Go, while Python and Ruby are the least performant.

Tail Latency Even if the number of requests per second increases, that does not mean the latency will go down. As one can notice in figure 6.4, until the 1000 clients, Go provides the least latency besides.Net. GraalVM provides the highest average latency. However, Dart tends to become slower when we increase the number of clients until we pass the 600 simultaneous clients, and then it changes its behavior. Instead of satisfying most of the requests, it notifies the clients directly that the server is saturated, resulting in a drop in satisfaction ratio and an amelioration in average latency. Energy Per Request After separating the energy and the performances, we have seen that most performance servers tend to be energy-greedy, so we propose investigating this trade-off between energy and performance. To do so, we report the average cost of a single request in millijoules in Figure 6.5.

Except for GraalVM (aka Java AOT), the cost of the single request decreases when we add more clients.Java, .Net, and go are the most energy efficient, while Python and Ruby may cost up to 10x more. Despite its low power, textsfCrystal exhibits greedy behavior when dealing with requests from fewer clients. Such behavior is caused by the low number of requests per second, which also results in high latency. Therefore, we conclude that the The purpose of this question is to study the energy consumption of the RPC server when transferring large objects. To do so, we send 80, 000 requests to the server whose size scales from 10 bytes up to 10 Megabytes, resulting in 10, 000 requests per size per server. To eliminate extra factors, we let the server handle the rate at which it can answer each request. However, we put a 20 seconds timeout deadline for each category of requests. Therefore, our boundary condition is only the number of requests received by the server. For this experiment, we investigate four observable variables:

1. the average power consumption during the scenario indicates the overall behavior of the server when working for long durations, Figure 6.6, 2. the tail latency for the 99th percentile, which indicates how efficient the server is, Figure 6.7 For each framework, we can distinguish three modes, and they all depend on the payload size:

1. Stress-free mode when the server has enough resources to satisfy the requests, as they require a memory less than a certain threshold (depends on the language and the platform), 2. Escalation mode when the requests tend to be bigger, but the server can still manage to handle them, at the price of a change in the energy and performance behaviors, 3. Broken state mode when the size of requests exceeds 1MB, most of the servers cannot handle them, and they tend to crash. After 10MB, except for Elixir, no server could handle those requests.

Stress-Free Mode

The compiled languages consume fewer resources (average power) in this mode. JVM-based languages tend to consume more energy, especially Scala. However, we do not observe the same behavior regarding efficiency. Unlike the other interpreted programming languages, PHP's performance can be compared to the compiled ones, such as CPP or GO, and even better to others, such as Swift. JVM-based languages tend to have better performance than interpreted ones. Furthermore, OpenJDK has shown more efficiency than GraalVM. Overall, we can have three groups when it comes to the cost of each request:

• Energy-efficient class: C++, GO, RUST, ELIXIR, and PHP,

• Middle class: Most of the interpreted languages and VM-based ones,

• Energy-greedy class: Crystal and Scala.

Escalation Mode

In this mode, the behavior of the server depends on the payload. We observe three cases:

1. drop in performances without an increased power source, such as Net core, Java Micronaut, Crystal, and Dart. In this case, the server uses the same resources, sometimes less, because it takes more time to handle fewer requests. This class of languages tends to be the most energy-consuming when it comes to the cost per request; 2. increase in power without affecting the performance, such as Go or.Net. The energy consumption of a single request is affected slightly but still increases.

3. Increase in power while dropping in performance. Despite the increase in power consumption, the server becomes slightly slower, which increases the energy cost per request. This cost is still better than the first case, which concludes that the servers in the first category are on the verge of breaking.

We can mention the case of Elixir that keeps scaling despite the lack of performance compared to other compiled languages (Go, CPP).

Broken state mode

Only four of the 25 configurations could parse the 10 MB files, and only one of those could achieve a 76% acceptance rate, which is Elixir. The other 3 had less than a 3% success rate (Rust, Swift, and Dart). The rest could be divided into two categories:

• Timeout where requests took too much time that the client canceled them in this category, we find most of the dynamic codes, such as OpenJDK and Kotlin,

• Size of request exceeded the maximum size when the implementation could not handle requests with a large size, as observed with .Net, Go, .Net core, CPP, PHP, Scala, Nodejs, Ruby, and Python.

Summary

This section reports on a study on the impact of programming languages on the energy for request handling. To do that, we used an implementation of 25 frameworks based on 17 programming languages. These servers use the official RPC protocol, reducing programmers' bias toward specific languages.

The results show that the number of clients had a binary effect on the servers. With a small number of clients, there was no typical behavior between different categories. On the other hand, when dealing with a more significant number of clients, interpreted languages and compiled ones dropped in terms of performance while keeping the same average power, unlike the VM-based ones that increased their average power while keeping the same performance. Overall, the second strategy had a better impact on the single request cost than the first one.

As for the evolution of the query size, we have seen three modes of behavior for each framework, depending on the size of the requests: the stress-free mode, the escalation mode, and the broken state mode. We start with the stress-free mode. The leader class was the compiled languages, while the VM-based ones tended to consume more energy. Finally, with the escalation mode, one could observe three strategies to deal with the increase in request size. The first strategy tends to drop performances while consuming the same power; the second was to increase the average power consumption without impacting the performances; the third was a combination of both previous strategies with an increase in power and a drop in performance. Interestingly, the third strategy had a lower cost per request than the first one. Finally, most implementations could not handle requests heavier than 10 MB for the broken mode. Elixir was an exception to this limit despite its low performance compared to other languages.

This study shows the absence of a universal language when dealing with RPC requests. Moreover, each implementation had a different way of dealing with scalability. While some chose to increase the average power consumption, others dropped in performance. Both of these strategies led to a higher cost of a single request.

Investigating Web Application Frameworks

After studying the impact of the programming languages on energy consumption while handling the RPC requests, we found that the programing language and the web framework significantly impact energy consumption. Therefore, we want to delve deeper into this way. This section will study not only the impact of programming languages but also the web frameworks on the energy consumption of the server. Many studies have been conducted to compare the performance of web frameworks. One can cite [START_REF] Gajewski | Analysis and Comparison of the Spring Framework and Play Framework Performance, Used to Create Web Applications in Java[END_REF] who compared two of the most famous Java frameworks-Play and Spring-or the work of [START_REF] Benmoussa | A new model for the selection of web development frameworks: Application to PHP frameworks[END_REF] who compared different PHP frameworks using six criteria: intrinsic durability, industrialized solution, technical adaptability, strategy, technical architecture, and speed. In our context, we push a 7 th criterion that impacts the economic outcome of the project.

We study the impact of web framework stacks on energy consumption. To do so, we implement a simple web application using the most popular web frameworks and compare their performance, latency, and energy consumption. We leverage the TECHEMPOWER Web Framework Benchmarks to incorporate server-side energy measurements obtained from a software-defined power meter, named POWERAPI [START_REF] Fieni | Smartwatts: Self-calibrating softwaredefined power meter for containers[END_REF]. These measurements are then analyzed in depth to understand the critical criteria that can impact the power consumption of web frameworks and derive guidelines for supporting developers in picking the most energy-efficient web frameworks according to their requirements.

Experimental Protocol

This section describes the environment we used during the experiments. Using the framework presented in chapter 3, we prepared a framework to measure the energy consumption of the web frameworks. The architecture of the experiments is presented in fig. 6.10. As one can see, the system is composed of five key components:

• Orchestrator is responsible for creating Docker images for each server, starting the server, and launching the benchmarks that the client should run,

• web server, or the system-under-test (SUT), It is the server that we want to measure its energy consumption. The server is encapsulated in a Docker container to ease the deployment of the machine and ensure the reproducibility of the experiments,

• database server offers the database that all the frameworks will use during the benchmarks. It is separated from the application server to avoid the impact of the database on the energy consumption of the server, • client machine It is responsible for stressing the web server using the scenarios provided by the orchestrator. To avoid the bottleneck on the client's side, we use WRK7 , a modern HTTP benchmarking tool, capable of creating a large load, thanks to its multithreaded design and scalable event notification technologies such Epoll8 and Kqueue9 .

• recorder collects the power measurements from the SUT and the critical performance metrics collected by the clients, so later can be used for our empirical analysis.

Each component runs on a different machine to limit the outside impact on the server and the client. Such as the overhead of the database, creating and launching the servers, and storing the records of the experiments. Moreover, all the machines belong to the same cluster to avoid network overhead.

Measurement Context

This part describes the hardware and software components of the experiment.

Hardware Settings All the tests have been executed in machines from the cluster chetemi10 of the grid5000 [START_REF] Cappello | Grid[END_REF] platform. This cluster comprises 15 machines with 2 Intel Xeon E5-2630 v4 processors, 256 GB of RAM, 600 GB HDD + 300 GB HDD hard disk. For our experiment, we consider using four machines from this cluster, one for the orchestrator, one for the database server, one for the client, and one for the web server. As for the recorder part, the measures will be stored in a MongoDB Database that runs on a separate virtual machine.

Software Settings Except for the recorder, all machines run a minimal version of Debian 9 (4.9.0 kernel version), which enforces the core processes required for our experiment. Equipped with Docker, we can quickly deploy the web and database servers in a container.

Measurement Context

This experiment aims to highlight the energy impact of the technology stack used to develop web applications once in production. To do so, we will use a web application composed of several URLs; each one will be used to simulate a test scenario. Therefore the clients will be agnostic to the technology stack used by the server. Moreover, spilling the database on a remote server simulates a real-world scenario where the database is not hosted on the same machine as the web application. It allows us to isolate the energy consumption of the server, on the other hand.

Candidate Frameworks Overall, we selected 210 web frameworks to be evaluated in this study. Each framework may have several configurations depending on the database, alternative interpreters, and so on. Table 6.1 highlights the number of frameworks used in the experiment per benchmark category. As we see in this table, some of the frameworks worked on certain conditions, while they failed on other benchmarks, such as Nickel (based on Rust). While it might be one of the most energy-efficient Rust frameworks, Nickel does not work with databases. Therefore, it cannot be used for any situation, but if a (stateless) web application does not interact with a database, it might be the best choice. Many reasons are behind the observed failures; there was no implementation, some errors were raised when handling the request, or simply the framework does not support such a feature.

Input Workload

In order to compare the energy consumption and performance efficiency of various frameworks, each framework is used to implement the same web application-i.e., replying to the same HTTP endpoints and requesting the same database. Then, we run the same sequence for all the SUT: 1. lunch the web application, The following sections describe each type of experiment and its purpose by giving some examples of the expected responses.

Test Scenarios We have seven categories of benchmarks:

Idle In this benchmark, we measure the web framework's idle energy consumption; this reflects an application's average energy consumption during periods without connections, for example, a company website beyond working hours or an online shop at night. Single Query Each request in this scenario is handled by retrieving a single row from a simple database table. This row is then serialized and returned to the client as a JSON response. In a web application, this is the most common sort of request. We employ a variable number of clients to measure the energy usage of the web framework when it is under load for this benchmark.

Multiple Queries This benchmark aims to observe the behavior of a web framework when it processes multiple entries from the database. Therefore, as a result, each request is handled by retrieving numerous rows from a simple database table,then serializing them as a JSON response. This benchmark's purpose is to observe a web framework's behavior while increasing the number of rows. In this case, we will use 512 clients.

Fortunes The purpose of this scenario is to study the behavior of a web framework when it processes a request that requires a database query and a template rendering. To do so, we ask the framework to retrieve all the rows from our database (in our case, 12 rows) and then send the response as a generated HTML page. The stress level varies according to the number of parallel clients.

Update Queries This scenario puts the database write to the test. Each request is handled by retrieving numerous rows from a simple database table, turning the rows into in-memory objects, altering one attribute of each object in memory, individually updating each related row in the database, and serializing the list of objects as a JSON response. Like the previous benchmarks, we will use 512 clients and vary the number of rows retrieved from the database according to the stress level.

Plain Text In this scenario, the server responds with a simple message, "Hello, World!" in plain text. Thanks to the small message size, we can enable the HTTP pipelining on the client side, which leads to greater client-side concurrency levels.

JSON Serialization In this scenario, the server returns a new instance of a JSON object with the following structure. {"message": "Hello, World!"}.

Recap Table 6.2 recaps different stress levels of each scenario above. Whenever the stress level is based on the size request, we use the same size of clients (512); otherwise, we use the number of clients as the stress level. To include additional scenarios, one might implement a Python class that handles the metadata of the workload, such as the query route, the query parameters, and the expected results.

Key Performance Metrics

We focus on comparing the energy behavior of different frameworks in multiple scenarios. To measure the energy consumption of those frameworks, we launch each one for a fixed duration, and then all the clients send multiple requests simultaneously. We compute the number of satisfied responses, which reflects the performance of the framework, the average latency, and the global energy consumed during the whole period, to deduce the energy cost of each request.

Runtime Measurements

• energy measurement : We use PowerAPI [20], a software power meter, to gather the power consumption of the SUT after we project the timestamps of each experiment phase to calculate the energy consumption of the framework during this phase. Energy is an integral of power over time, so we use a numerical approach to isolate the energy consumption.After this, we divide the calculated energy by the number of responses.

E = a b P(t)dt ≃ n ∑ k=1 P(t k -1) + P(t k ) 2 (6.1)
• Total cost of the energy during each period,

• Total number of requests,

• latency of the responses,

• Average energy cost per request.

Notes Due to technical problems, not all the web frameworks returned the tail latency (99%). Therefore we substitute it with the average latency during this study. However, for further details, the reader can always check the available values in our public repository. 11It has been proven in the work of Eddie Antonio Santos et al. that Docker does not impact energy consumption. Thus, using containers and isolation avoids any noise of the operating system after executing one benchmark and contributes to the reproducibility of our results.

Bias Analysis

We are aware of potential bias analysis regarding estimating the total energy cost, the interference of other system processes during the execution, and some external events. Thus, we run experiments multiple times and compute the average values.

Extension

To follow the guidelines that we presented in Chapter 3, we provide a GitHub repository 12where one can add extra candidates by creating a new project using the option -new. Then, interested practitioners must fill out the template and provide the Docker image file. Additionally, to configure the workload we provide the option -concurrency-levels and -duration. The choice of database is included in the Docker image.

Results & Findings

, and all can be found in the online repository. 13 Due to the extensive number of results (261 frameworks), we could not compare frameworks individually. Instead, we will aggregate our results based on the language family. However, in the end, we provide a dashboard that allows the reader to explore the results in more detail and compare several frameworks. We will group the frameworks not by language but by family so that we will have five families:

• compiled languages: Rust, Go, Haskell, C++, C, Vala, Nim, Swift, Prolog, V, Ocaml, Crystal, and D;

• interpreted languages: Javascript, Python, Lua, Php, Cfml, Racket, Ruby, Common lisp, Perl, Julia, Groovy, and Typescript;

• JVM-based languages: Java, Kotlin, Scala, and Clojure;

• .Net-based languages: C#, F# and VB;

• Other VM-based languages: Dart and Elixir and Erlang.

Overall Statistics

To determine which web framework/stack is performing well, we need to establish some general idea about the average energy consumption and latency of the frameworks under study. Instead of reporting the raw energy consumption of those frameworks, we will provide some green factors to determine which one is eco-friendly and which is greedy. In this part, we will discuss the average behavior of the frameworks, highlight some trends, and eliminate the outliers. A gentle reminder that being an outlier, in this case, does not mean that the web framework is not performing well; it means that the web framework is not performing well in the same way as the others within the context of this experiment. First, we will perform a correlation test between the metrics to narrow the research space.

Because the Shapiro-Wilk [START_REF] Shapiro | A comparative study of various tests for normality[END_REF] test yielded a p-value of 0.0 for all metrics, our data do not follow a normal distribution. Therefore, we use The non-parametric test known as Pearson correlation coefficient [START_REF] Zar | Spearman rank correlation[END_REF] Figure 6.11 depicts the correlation between the metrics for the 6 scenarios. The Pearson correlation coefficient quantifies the linear correlation between two variables, X and Y. It ranges from -1 to +1, with 1 being total positive linear correlation, 0 representing no linear correlation, and -1 representing whole negative linear correlation. The stronger the correlation, the closer the value is to 1 or -1. The weaker the correlation, the closer the value is to zero. The Pearson product-moment correlation coefficient is another name for the correlation coefficient. This correlation coefficient is also known as the Pearson productmoment correlation coefficient. One can notice that there is a strong correlation between the energy consumption of the CPU and the DRAM for most of the scenarios. Moreover, the average energy consumption of DRAM is one-sixth of the CPU energy consumption. Therefore, this study will focus more on CPU energy consumption. For more insights about DRAM consumption, we refer the reader to our GitHub repository. 14Another strong correlation is between the number of requests per second and the average cost of a single request. This strong correlation happens because the cost of a single request is proportionate to the number of requests per second since the Average Power consumption remains constant after a certain threshold of clients; one should remember that this limit is different for each framework.

Unlike the multiple queries and update queries, the single query scenario depicts a weak correlation between latency and the number of requests per second. The reason behind such an anomaly is the fact that we summarized the data when we had multiple numbers of clients. If we calculate the correlation when we have a fixed number of clients, like in the case of update queries (512 clients), one can notice a strong correlation. Figure 6.12 demonstrates such a behavior. As one can notice, for each level, we observe a linear clustering. Therefore, we can safely focus our analysis on two variables, number of requests per second and average energy consumption. The first one will indicate the performance of the solution. Meanwhile, the second one will measure how green a framework is.

Scenario-based Analysis

After establishing the general idea about the average behavior of the frameworks, we will now focus on the analysis of each scenario.

Idle behavior This part will treat average power behavior when the framework is in a rest mode. Figure 6.13 presents, a density plot for each family.

As one can notice at rest, most families consume between 20 and 40 Watts, and 6% of the compiled language frameworks consume less than 15 Watts. However, 50% of Java solutions tend to consume around 50 Watts, which makes it the most greedy family. If we look at each of the programming languages from Java separately in Figure 6.14, we find that Java-based Single query As mentioned before, the purpose of this scenario is to benchmark the framework efficiency to handle a single entry. We will start with an average power consumption histogram to determine the frameworks' general behavior. Figure 6.15 reports the density plot of average power consumptions for all the experiments depending on the number of concurrent clients. As one can observe, there are three main states:

1. relaxed state where the number of clients is less than 16: most of the frameworks consume around 70 Watts;

2. average state where the number of clients is between 16 and 64: most of the frameworks consume around 100 Watts;

3. Finally, the stress state beyond 128 concurrent clients: most frameworks have a stable power consumption regardless of the number of clients. This increase in power consumption is due to the database server, which reached its maximum capacity. Now that we have seen the overall distribution of the power within the single query scenario. We analyze each family separately. In addition, we include the number of requests per second (RPS) as a performance metric of interest. In Figure 6.16, each run is represented by a circle, and the size of the circle indicates the number of concurrent clients: The smaller the circle, the fewer clients. On the one hand, one can notice that compiled languages are the most efficient in terms of performance, despite their average energy consumption. Moreover, there is no significant change in the average power consumption when we increase the number of concurrent clients. On the other hand, the JVM-based frameworks tend to consume the most energy while reporting the same performance as the .Net-based ones. Finally, the interpreted languages lack performance while keeping low power, except for PHP, as it has one of the highest RPS with a half million RPS which got beaten only by C++ and Rust. Multiple queries This scenario benchmarks the framework's efficiency in handling multiplerow queries. As mentioned before, this study focuses on a fixed number of concurrent clients while we increase the request size per level. Figure 6.17 reports on the distribution average power consumption for each level. As one can notice, the query size has no substantial impact on the average power consumption. Furthermore, one can notice a slight decrease in the average power consumption (from 110 watts to 90 watts) when the size is more significant than ten rows. This unexpected behavior might be related to the time the database takes to process the query; therefore, the client is resting while waiting for the database answer. Therefore, one can conclude that the query size has more impact on the database than the framework itself. To verify this hypothesis, we compare the average power consumption of the database by combining all the frameworks for each database. Table 6.3 details the average power consumption per level for each database. One can see that for MySQL, there were no changes regardless of the query size, while for Postgres and MongoDB, there is a slight decrease in the average power consumption when the query size is more significant than ten rows. Therefore we can conclude that the type of database had more impact on the average power consumption than the size of the request. After looking at the power distribution within the multiple query scenario, we will analyze each family separately.

We consider the number of requests per second (aka RPS) as a related performance metric. Figure 6.18 presents the total RPS per level for each framework on a logarithmic scale due to the significant performance scale between multiple frameworks. As one can notice, the difference between the best performing framework, aka Lithium, 15 and the worst one, aka Hapi-Nginx, 16 is 4, 000 times. At the same time, the average power consumption Moreover, one can notice that Java-based frameworks tend to consume more power compared to other languages, with a slight increase in performance.

Finally, PHP remains one of the most efficient frameworks in terms of performance while keeping low average power consumption.

Update This scenario tests the framework's efficiency in handling update queries. As mentioned before, for this study, we will focus on the fixed number of parallel clients while increasing the request size per each level. Figure 6.19 presents the distribution of average power consumption for each level. As one can notice, the query size does not strongly impact the average power consumption. Moreover, the overall average power consumption decreased by 20 Watts. This decrease might be again related to the database processing time. Figure 6.20 reports on the number of RPS per level for each framework. Swoole dropped in terms of performance while reducing the average power, unlike compiled languages- based frameworks, such as lithium and actise.net, which kept the same performance ranking and average power consumption. Another interesting observation is that for the compiled languages, both the performance and the average power consumption decrease when the size of the requests increase. This drop comes with a slight decrease in average power. Fortunes The primary purpose of this test is to show the behavior of frameworks when rendering webpages with dynamic content. As one can see in Figure 6.21, Unlike the single query scenario. the more clients are connected, the more the performance performances increase. Moreover, we notice a drop in the performance of all interpreted languages, including PHP, that exceeded many compiled frameworks in the other scenarios.

Plain Text and JSON Serialization Unforntantly, In these scenario, the client hits its limit before servers, as highlighted in Figures 6.22,6.23. The ceiling is almost linear for the compiled frameworks and the JVM-based ones. This is also explained by the fact that, unlike in other scenarios, the high-stress level is on top.

Tools

GreenBoardhttps://github.com/chakib-belgaid/greenboard is an open-source dashboard designed to assist developers and practitioners in selecting the most energy-efficient web framework based on their requirements. It enables the user to compare the performance and energy consumption of different frameworks. As well as filtering the frameworks based on their programming language, database type, and other criteria. This dashboard is designed based on the latest results obtained from our experiments and we will keep updating it with the latest results. fig. 6.24 presents an example of the raw obtained after selecting 7 frameworks. For the sake of versatility, we will consider the single query scenario with two levels of clients (64 and 256). As one can see, although Spark depicts the highest amount of powers (235W), it is ranked 6th when it comes to the energy consumed per request. On the other hand, Flask-pypy consumed half of its power (118W) but it pays in terms of performance with ten times slower latency, and its requests cost seven times more than Spark's requests. In terms of performance and energy consumption, PHP, on the other hand, contained both, the best and the worst frameworks. Therefore one should be careful when choosing a framework and its configurations rather than just the language.

Summary

This section presents a study on several web frameworks' performance and power consumption. To do so, we used the TechEmpower17 benchmark suite, which is a set of tests that measure the performance of several web frameworks. We tested 261 frameworks using 7 scenarios. We found that Java-based frameworks are the most power-consuming, while the complied ones are the most efficient in terms of performance. Moreover, PHP remains one of the most efficient frameworks in terms of performance and energy despite being an interpreted language. This behavior is mainly due to the optimization done on the Zend engine 18 to fit the website's requirements. Furthermore, we found that the database impacts the energy consumption and the average power of the servers even when they are not on the same machine. To summarize our findings, we present in fig. 6.25 a comparison of the energy consumption per request for each scenario. While we have seen in the previous results that Java-based frameworks depict the highest energy; they still provide greener requests compared to the interpreted frameworks; this is mainly because their interpreters report lower performances.

Conclusion

This chapter aims to extend the work of Pereira et al. [START_REF] Pereira | Energy Efficiency Across Programming Languages: How Do Energy, Time, and Memory Relate[END_REF] by studying the energy footprint on other aspects than micro-benchmarking. The main focus of this chapter is the behavior of programming languages within the context of web applications. To do so, we have covered two case studies. The first experiment is the impact of programming languages when developing RPC services. We used the official implementation of GRPC library to build 25 frameworks using 17 programming languages. In the second experiment, we look at how web frameworks affect energy consumption by creating a basic web application and comparing its performance, latency, and energy usage using the most common web frameworks. Overall we had 261 implementation using 35 distinct programming languages and three different databases. For each implementation, we considered 7 representative scenarios. Each scenario is tested with several stress levels to study the scalability of the web frameworks. Therefore, the final results cover 8, 750 use cases.

Our studies revealed the absence of a clear winner in terms of energy efficiency. However, when we take into consideration the cost of a single request. C++ and Rust were the topranked, hence assessing the work done by Pereira et al.. On the other hand, Unlike the results shown in the paper [START_REF] Pereira | Energy Efficiency Across Programming Languages: How Do Energy, Time, and Memory Relate[END_REF], PHP exhibits a higher performance while keeping a high power consumption on average. This result is due to the nature of PHP language, which is dedicated to the web and not general-purpose programming. While most of the frameworks consumed around 100watts, Java-based frameworks consumed around 220watts. This made Java one of the most energy-consuming in the context of online services, especially when considering the periods with low traffic.

Finally, this study revealed that each framework had different behaviors toward scalability. Some of them increased the average power consumption while trying to keep up with the performance. Others, on the other hand, could keep the average power consumption while decreasing the performance. Furthermore, there was a case where there was a mix between both strategies.

As there is no final answer to the question of which framework is the most energyefficient, we provide developers with a tool to help them compare between frameworks based on their needs. The criteria we used to compare frameworks are the scenario type, the stress level, and the database type.

Chapter 7 Conclusion & Perspectives

This manuscript reports on several contributions to measuring and reducing software energy consumption. We used a three-step strategy to lower software energy: benchmarking, measuring, and optimizing. We started with the benchmarking phase. Chapter 2 discussed the challenges of a successful benchmarking strategy: reproducibility, accuracy, and representativeness. We concluded that software containerization technologies, like "Docker", are the best fit to ensure that energy studies can be reproduced. We then extended this reproducibility to an evolving protocol that helps researchers keep up with the rapid pace of software development. Then, we targeted the accuracy by studying the hardware and software factors that can impact the energy variation and how practitioners can tune them to harness this energy variation.

After establishing a robust benchmarking protocol to create energy-based experiments, we shifted our focus to the optimization side. We opted to start with Python, the most popular yet energy-hungry programming language. As a result, we began by examining the energy behavior of Python code in its most common usage. Then, we presented a non-intrusive method to lessen its energy use. Following that, we applied the same strategy to another programming language known for its legacy code base, Java, to prove that we can still cut the energy usage of existing running applications without incurring high costs.

Lastly, we used the flexibility of the micro-services architecture to look at how each programming language uses energy in different web scenarios. We first examined the effects of the various programming languages when dealing with the Remote Procedure Call (RPC) protocol. Then, we extended this study to a more practical application by comparing 261 web frameworks, each implementing the same website using seven use cases. Then, we provided practitioners with a dashboard to determine which stack is best for a given situation.

Summary of Contributions

The contributions reported in this thesis are covered in this section. The following is a summary of them:

Benchmarking Protocol to Measure Software Energy Consumption During this chapter, we tackled two of the main challenges faced by researchers when using empirical analysis to measure software energy consumption: reproducibility, accuracy. First, we discussed the challenge of reproducibility by looking at the existing techniques. We have seen two main methods to encapsulate experimental systems. The first is to use virtual machines, and the second is to use containers. We established that Docker is more suitable for energy-related studies. Then we extended this reproducibility to an evolving protocol that helps researchers keep up with the rapid pace of software development.

Finally, We examined the phenomenon of variation when measuring the energy consumption of experiments to increase their accuracy. To do so, we investigated the following research questions: RQ 1: Does the benchmarking protocol affect the energy variation? RQ 2: How important is the impact of the processor features on the energy variation? RQ 3: What effect does the operating system have on energy variation? RQ 4: Does the choice of processor make a difference in reducing the energy variation?

We discovered that processor features and the operating system can have a significant impact on the variation in energy consumption between the benchmarking protocol.

Finally, we presented several guidelines for controllable parameters that practitioners could easily modify to improve the precision of their experiments.

The results of this work have been published in the following paper: Ournani, Z., Belgaid, M. C., Rouvoy, R., Rust, P., Penhoat, J., and Seinturier, L. (2020a). Taming energy consumption variations in systems benchmarking. In Proceedings of the ACM/SPEC International Conference on Performance Engineering, pages 36-47 Impact of Energy-saving Strategies in the Python Ecosystem During this chapter, we tackled the challenge of reducing the energy consumption of Python code. To do so, we investigated the energy behavior of the python during its most common usage:

• machine learning: we studied the energy consumption of a machine learning algorithm using a benchmark of 60, 000 entries to train the cifar10-fast model. To confirm that the energy consumption of a model is highly impacted by its accuracy, increasing the accuracy from 85% to 95% can lead to double the energy consumption.

• web development: We examined the Django-based application and discovered that fetching data from the database consumes most of the energy. As a result, we compared three methods for retrieving data from two different databases. We discovered that using an optimized strategy can reduce energy consumption by 20×.

• data processing: In this section, we studied the energy consumption of data structures, iteration algorithms, and concurrency. following the guidelines and prioritizing built-in functions is the best way to reduce the energy consumption of Python applications. As for the concurrency, we found out that the Multithreading strategy can lead to a decrease in energy consumption despite the increase in execution time; the Multiprocessing was the optimal option, however, it is greatly affected by the number of processes.

After that, We presented a non-intrusive technique for optimizing the energy consumption of Python-based apps without requiring significant code changes. This technique involves using a different Python runtime implementation. We began by categorizing and filtering these implementations into three major classes (compiler, interpreter, and extra libraries). Then we ran a series of tests to compare the energy consumption of these options. Our findings point to the lack of a general solution and the importance of customizing the Python runtime for each application. We discovered that while most interpreters consumed as much or more energy than the official CPython implementation, the JIT-based ones tend to consume significantly less energy.

This work resulted in the development of two tools to measure the energy consumption of Python code:

• PyJoules (pypi.org/project/pyJoules) is a software toolkit to measure the energy footprint of a host machine along the execution of a piece of Python code. It can measure the energy consumption on the level of script, function, and bloc of code;

• JouleHunter (pypi.org/project/joulehunter): an energy profiler for python applications. It can be used to highlight the functions that consume the most energy in a given Python program. Its main usage is to help developers do an exploratory analysis of their application to scope the functions that should be optimized to be then targeted by PyJoules.

Impact of Java Virtual Machine Configurations on Energy Consumption

In this chapter, we thoroughly explored how JVMs affect software energy consumption by investing in the following research questions:

RQ 1: What is the impact of existing JVM distributions on the energy consumption of Java-based software services?

RQ 1: What are the relevant JVM settings that can reduce the energy consumption of a given software service?

To do so, we compared the energy consumption of 12 benchmarks using 52 implementation, each benchmark is dedicated to a typical use case. After that, we studied two of the JVM features, JIT and GC, and show their impact on the energy consumption of the Java code; The findings demonstrate that choosing the right JVM platform can significantly reduce energy usage depending on the software and use case, and this optimization can also be achieved by properly configuring the JIT and GC parameters. This work resulted in the development of software called JReferral (github.com/chakib-belgaid/ jreferral) that allows the user to explore the JVM settings and their impact on the energy consumption of a given Java program and a publication of the following paper: Ournani, Z., Belgaid, M. C., Rouvoy, R., Rust, P., and Penhoat, J. (2021). Evaluating the impact of java virtual machines on energy consumption. In Proceedings of the 15th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pages 1-11 Energy Footprint of Distributed Programming Frameworks Finally, in Chapter 6, we extended the work of Pereira et al. [START_REF] Pereira | Energy Efficiency Across Programming Languages: How Do Energy, Time, and Memory Relate[END_REF] by studying the energy footprint on other aspects than micro-benchmarking by investigating the impact of the programming language on the energy consumption of web services. In the first study, we used the official implementation of GRPC library to build 25 frameworks using 17 programming languages. As for the second study, we looked at how web frameworks affect energy consumption by creating a basic web application and comparing its performance, latency, and energy usage using the most common web frameworks. We employed 261 implementation using 35 distinct programming languages and three different databases. For each implementation, we considered 7 representative scenarios. Each scenario is tested with several stress levels to study the scalability of the web frameworks. Therefore, the final results cover 8, 750 use cases.

Both studies revealed that each framework had different behaviors toward scalability. While the absence of a clear winner, we found out that most compiled languages are more energy-efficient because of their performance. Java tends to depict more power, and the interpreted languages lacked performance but maintained low power consumption. PHP was an exception, as its performance was compared to the compiled languages. This work resulted in creating a dashboard to help developers choose the correct programming language and framework for their application based on the context. The dashboard can be found at github.com/chakib-belgaid/greenboard.

Future Work

While our contributions are a good start to the energy-aware software engineering field, there are still many challenges to overcome. This is just the tip of the iceberg. The following perspectives are some of the challenges that we would like to address in the future.

Short-Term Challenges

Before starting a journey looking for a new mine, one should first look at the available resources. In the same way, we need to first look at the resources available to us before we start looking for new challenges. The following are some of the challenges that we can address in the short term.

Studying the evolution of Python interpreters. In Chapter 4, we compared several python interpreters. However, most of these alternative solutions were based on Python 2, which is now deprecated. On the other hand, the default python interpreter (aka CPython) has included many features and optimization since our last study, such as the introduction of Python introduced the dataclasses in version 3.7 (PEP 557)1 that can be used to reduce the memory footprint of Python objects, the new parser in Python 3.9,2 the user type alias in Python 3.10. 3 And, the most interesting changes for us occurred in CPython 3.11, which is claimed to be 25% faster than Python 3.10. 4 . We need to re-evaluate the impact of these changes on the energy consumption of Python programs.

Studying the impact of ORMs. We continue with our work with Python, and this time we will dive deeper into the impact of the ORMs. As shown in Section 4.3.2, the ORMs are the most energy-consuming part of a web application. We intend to widen this analysis to other Python ORMs, such as SQLALCHEMY 5 and PEEWEE. 6 We will also explore the real relationship between the ORM, the database, and the web framework.

Studying the impact of Machine learning. In Chapter 4, we showed that training accuracy has a huge cost on the energy using a single model. However, in practice, we often use multiple models to address a given problem. We will explore the impact of using multiple models on the energy consumption of a machine learning application, as well as compare the energy consumption of different machine learning frameworks, such as PYTORCH, 7 SCIKIT-LEARN, 8 and TENSORFLOW. 9 Studying the impact of JVM bytecode. As we have seen in Chapter 5, the choice of the JVM can greatly impact the energy consumption of hosted applications. On the other hand, we have seen in Chapter 6 that other JVM-based languages depicted half the power consumption of the Java code. Was it only because of the adopted web framework? Or is there a real difference between the JVMs and the bytecode handled by JVM-based languages? We will explore this question by comparing the energy consumption of different JVMs and JVM-based languages, such as Scala, Kotlin, Groovy, and Clojure.

Revisiting the impact of programming languages. One of the hardest challenges when comparing the energy consumption of multiple programming languages was the bias of the expertise of the programmer. One solution was to create a reference benchmark that allows one to compare several programming languages. 10 This benchmark has been adopted by many researchers, such as the work of Couto et al. [START_REF] Couto | Towards a green ranking for programming languages[END_REF], while others were to use some basic algorithms, like [START_REF] Noureddine | A preliminary study of the impact of software engineering on GreenIT[END_REF] who compare the energy consumption using the Hanoi tower problem 11 using different programming languages, or even simpler benchmarks, like the Rosettacode base. 12 However, this scope is limited to single algorithms and does not help to cover production-like conditions, this is why we shifted to the web frameworks. Nonetheless, semantic analyzers issued from the OpenAI project 13 like the automatic test generator PONICODE 14 and AI code generator, 15 an interesting new feature provided by the GitHub copilot is GitHub copilot labs 16 is the ability to automatically translate the code from one language to another. We will explore the impact of this feature on the energy consumption of a program.

Raising awareness about the energy consumption of software As the core focus of this Ph.D. thesis was to optimize energy consumption using comparative studies, it was straightforward to claim that approach X is greener than Y, no matter the metrics we were using. However, this cannot be the case for developers when measuring the energy consumption of their programs. Some tried to give labels, such as (A,B,C. . . ), while others translated these raw metrics in an equivalent of fuel consumption (e.g., 1 liter of fuel per 1, 000 lines of code). Most of the approaches were to use the carbon emission, such as [START_REF] Patterson | Carbon emissions and large neural network training[END_REF] or ecograder, 17 as a pivot metric to quantify the environmental cost of the software. We intend to explore these approaches and study how they can be used to increase awareness of the energy consumption of their code.

Representativeness

As reported in the state of the art, a successful benchmark meets three criteria: reproducibility, accuracy, and representativeness. This Ph.D. thesis mostly discussed two of these criteria: reproducibility, and accuracy. However, we did not discuss the representativeness of our benchmarks. We relied on state-of-the-art benchmarks to represent real-world applications. Nevertheless, the gap between these benchmarks and the industry is getting bigger and bigger, due to the extreme pace of software development. In the future, we would like to tackle the issue of representativeness and discuss how we can improve our protocol to fit real-world applications. After all, what is the purpose of doing optimizations if they cannot be applied to real-world applications?

First, we will consider taking advantage of the popularity of CI/CD among developers to provide some insights into the energy consumption of their code. We did prototype a sonar tool that can report on the energy consumption of a Java-based code repository. Figure 7.1 shows the work of our intern who used a Sonar plugin to highlight the energy evolution of Java applications over time. This prototype got even more sophisticated to become Joulediff, 18 which is a tool to be included in a CI/CD workflow to highlight the energy evolution of the Java programs. We intend to push this even further to detect the commits responsible for an increase or the optimization within the source code.

Energy impact Vs. Energy consumption

To develop more eco-friendly software, both state-of-the-art and this thesis focused on reducing software energy consumption. To accomplish this, we sought to isolate the energy consumption of the software under test as much as possible, primarily by running the tests on an isolated machine that runs only a few services. Others tried to infer the energy consumption of a single process using additional metrics that vary from the CPU usage, such as [START_REF] Noureddine | Jolinar: Analysing the energy footprint of software applications (demo)[END_REF] to more sophisticated models like [START_REF] Fieni | Smartwatts: Self-calibrating softwaredefined power meter for containers[END_REF][START_REF] Fieni | Selfwatts: On-the-fly selection of performance events to optimize software-defined power meters[END_REF].

However, by the end of this thesis, I realized that isolating the energy consumption by running the tests on an isolated machine does not represent real-world conditions; many processes are running in parallel. On the other hand, our benchmark can compete with other processes for resources such as CPU, memory, and network. Therefore, this program can increase the energy consumption of other processes running on the same machine, although its energy consumption is low. This is why I have started to work on a new project called ENERGY IMPACT that aims to measure the energy impact of software.

Synthesis I believe that the context can have a significant impact on the energy consumption of the system, and sometimes measuring only the energy consumption of an application does not represent its real impact on the energy consumption of the whole system, especially when we are dealing with timeless Software aka services. In the future, we tend to explore this behavior from different angles, such as the energy consumption of the application itself instead of the whole system using [START_REF] Fieni | Smartwatts: Self-calibrating softwaredefined power meter for containers[END_REF], comparing the strategies that were mentioned in this thesis using multiple contexts instead of just a minimal version of the operating system, and studying the impact of deliberately putting the application on hold when we have a lower CPU frequency to reduce its impact using some opportunistic schedulers.

Figure 1 . 3 :

 13 Figure 1.3: Our position in the IT sustainability research

Figure 1

 1 Figure 1.4 summarizes the work completed for this thesis As mentioned in the previous section, our goal is to help developers reduce the energy consumption of their programs during execution. We use an empirical approach to accomplish this. As a result, we require some tools to assist us in this task. To begin, we provide a tool to assist developers in measuring the energy consumption of their programs. The rest of the manuscript is outlined below.

Figure 1 . 4 :

 14 Figure 1.4: Contributions reported in this thesis

Figure 2 . 1 :

 21 Figure 2.1: Electrical cost comparison between two CPUs.

Figure 2 . 3 :

 23 Figure 2.3: INTEL RAPL scopes

Figure

  Figure2.3 shows the different scopes that can be monitored using RAPL. The CPU package can be monitored for both server and desktop processors. However, DRAM is only available on server CPUs, while the integrated GPU is only available on desktop processors. The advantage of such an approach is the absence of intrusive measurement tools. Furthermore, they have a high temporal granularity with a sampling frequency of up to 1 KHz[START_REF] Ilsche | Power measurements for compute nodes: Improving sampling rates, granularity and accuracy[END_REF].

Figure 2 . 4 :

 24 Figure 2.4: CPU energy variation for the benchmark CG

1 .

 1 execute the code, 2. measure program's energy consumption, 3. infer the guidelines.

Figure 2 . 5 :

 25 Figure 2.5: the spiral method of energy optimization

Figure 3 . 1 :

 31 Figure 3.1: Different Methods of Virtualization

Figure 3 . 3 :

 33 Figure 3.3: Execution time & energy consumption of Redis with and without Docker [41]

Figure 3 . 4 :

 34 Figure 3.4: Benchmarking protocol

Figure 3 .Figure 3 . 5 :

 335 Figure 3.5: Topology of the nodes of the cluster Dahu

Figure 3 . 6 :

 36 Figure 3.6: Comparing the variation of binary and Docker versions of aggregated LU, CG and EP benchmarks

Figure 3 . 7 :

 37 Figure 3.7: Energy variation with the normal, sleep and reboot modes

Figure 3 . 8 :

 38 Figure 3.8: STD analysis of the normal, sleep and reboot modes

Figure 3 . 9 :

 39 Figure 3.9: Energy variation when disabling the C-states

Figure 3 . 10 :

 310 Figure 3.10: Energy variation considering the three cores pinning strategies at 50 % workload

Figure 3 .

 3 Figure 3.11: C-states effect on the energy variation, regarding the application processes count

Figure 3

 3 Figure 3.12: OS consumption between idle and when running a single process job

Figure 3 . 13 :

 313 Figure 3.13: The correlation between the RAPL and the job consumption and variation

Figure 3 .Figure 3 . 14 :

 3314 Figure 3.14 is an illustration of the aggregated STD density of more than 5, 000-random value sets taken from all the conducted experiments. This shows that the cluster Paranoia

Figure 3 . 15 :

 315 Figure 3.15: Energy variation comparison with/without applying our guidelines

Figure 3 . 16 :

 316 Figure 3.16: Energy variation comparison with/without applying our guidelines for STRESS-NG

Chapter 4 Impact

 4 of Energy-saving Strategies in the Python Ecosystem 4.1 Introduction Dynamic programming languages, except Perl, have surpassed compiled programming languages in terms of popularity among software system developers over the past decade (cf.

Figure 4 .

 4 1). However, it remains unclear if this category of dynamic programming languages can truly compete with compiled ones in terms of energy consumption.

Figure 4 . 1 :

 41 Figure 4.1: PYPL popularity of programming languages [2].

  Figure 4.2).

Figure 4 . 2 :

 42 Figure 4.2: Energy consumption of a recursive implementation of Tower of Hanoi program in different languages [101].

5

 5

Figure 4 . 3 :

 43 Figure 4.3: Use cases of Python (source: JetBrain).

  4 https://www.jetbrains.com/lp/python-developers-survey-2020

Figure 4 .

 4 5 depicts how the curve steepens as training progresses. For example, training to 90% accuracy requires three times the energy required for training to 80% accuracy. gpu-core1,3,5,7-nohyper gpu-core1-hyper gpu-core1-nohyper 94

Figure 4 . 4 :

 44 Figure 4.4: Accuracy along epochs [73].

Figure 4 .

 4 Figure 4.4 shows the model's accuracy based on the number of epochs. This figure shows that there is a non-significant impact on the choice of the strategy on the accuracy; it is only a matter of the number of epochs. However, the increase in accuracy is not linear, as can be observed. It took only 10 epochs to reach an accuracy of 84%, but it required more than twice the number of epochs to add an extra 6% accuracy. This highlights the price one should pay to increase the model's accuracy.

Figure 4 . 5 :Figure 4 . 6 :

 4546 Figure 4.5: Cumulative energy consumption vs. accuracy [73].

Figure 4 . 7 :

 47 Figure 4.7: Evolution of average GPU power along epochs [73].

Figure 4 . 8 :

 48 Figure 4.8: Distribution of the durations of each epoch [73].

Figure 4 . 9 :

 49 Figure 4.9: Request-Response life cycle in Django

Figure 4 . 10 :

 410 Figure 4.10: Tree representation of the energy consumption of a single request in Django (naive version).

Figure 4 . 11 :

 411 Figure 4.11: Key performance indicators observed for different data access strategies.

Figure 4 . 12 :

 412 Figure 4.12: Tree representation of the energy consumption of a request in Django (optimized version)

Furthermore, we

  run the same code snippet for each of these implementations with different primitive data types: Int, Float, and String.

Figure 4 . 13 :

 413 Figure 4.13: Comparison of the energy consumption of different Python loops [125].

Figure 4 . 14 :

 414 Figure 4.14: Comparison of the energy consumption of different methods to convert a list [125].

Figure 4 . 15 :

 415 Figure 4.15: Energy consumption resulting from different Python concurrency strategies (number of workers).

Figure 4 .

 4 Figure 4.17 overviews the architecture of our benchmarking infrastructure.

Figure 4 . 17 :

 417 Figure 4.17: Benchmarking architecture deployed with POWERAPI.

Figure 4 .

 4 18 displays the Dendrogram produced by the HCA.

Figure 4 . 18 :

 418 Figure 4.18: Dendogram of the different Python runtime implementations.

Figure 4 . 19 :

 419 Figure 4.19: Different Python runtime energy scores.

Figure 5 . 1 :

 51 Figure 5.1: JVM architecture

Figure 5 . 2 :

 52 Figure 5.2: Target scope of DACAPO and RENAISSANCE benchmarks.

Figure 5 . 3 :

 53 Figure 5.3: Energy consumption evolution of selected JVM distributions along versions.

Figure 5 . 4 :

 54 Figure 5.4: Energy consumption of the HotSpot JVM along versions.

Figure 5 . 5 :

 55 Figure 5.5: Energy consumption comparison across Java benchmarks for HOTSPOT, GRAALVM & J9.

Figure 5 . 6 :Figure 5 . 7 :

 5657 Figure 5.6: Power consumption of Scrabble as a service for HOTSPOT, GRAALVM & J9.

  Active threads of Avrora when using HOTSPOT, GRAALVM, or J9. Active threads of Reactors when using HOTSPOT, GRAALVM, or J9.

Figure 5 . 8 :

 58 Figure 5.8: Active threads evolution when using HOTSPOT, GRAALVM, or J9.

Figure 6 . 2 :

 62 Figure 6.2: Average power consumption based on the number of the clients

Figure 6 . 3 :

 63 Figure 6.3: Number of requests per second based on the number of clients

Figure 6 . 4 :

 64 Figure 6.4: Tail latency (99%) based on the number of clients

Figure 6 . 5 :RQ 2 :

 652 Figure 6.5: The cost of a single request based on the number of clients (mJ)

3 .Figure 6 . 6 :

 366 Figure 6.6: Average power consumption based on the request size

Figure 6 . 9 :

 69 Figure 6.9: Energy consumption based on the request size

Figure 6 . 10 :

 610 Figure 6.10: Architecture of the experiments.

Figure 6 . 11 :

 611 Figure 6.11: Spearman Rank Correlation between different metrics

Figure 6 . 12 :

 612 Figure 6.12: Correlation of latency and number of requests per second for a single query

Figure 6 . 13 :

 613 Figure 6.13: Average power consumption for the idle scenario

Figure 6 . 14 :

 614 Figure 6.14: Average power consumption for Java-based languages in the idle scenario case

Figure 6 . 15 :

 615 Figure 6.15: Distribution of the average power consumption for single query scenario

Figure 6 . 16 :

 616 Figure 6.16: Total request vs. average power consumption for the single query scenario (size of circles represents the number of clients)

Figure 6 . 17 :

 617 Figure 6.17: Distribution of average power consumption for the multiple queries scenario

Figure 6 . 18 :

 618 Figure 6.18: Total request vs. average power consumption for multiple queries scenario (size of circles represents the number of clients)

Figure 6 . 19 :Figure 6 . 20 :

 619620 Figure 6.19: Distribution of the average power consumption for the update scenario

Figure 6 . 21 :

 621 Figure 6.21: Total request vs. average power consumption for Fortunes scenario (size of circles represents the number of clients)

Figure 6 . 22 :

 622 Figure 6.22: Total request vs. average power consumption for plainText scenario (size of circles represents the number of clients)

Figure 6 .

 6 Figure 6.23: total request vs. average power consumption for JSON Serialization scenario (size of circles represents the number of clients)

Figure 6 . 24 :

 624 Figure 6.24: Example of measurement for the single query scenario

FFigure 6 . 25 :

 625 Figure 6.25: Energy consumption per request for each family of programming languages

Figure 7 . 1 :

 71 Figure 7.1: Sonar Energy plugin for JUnit

VM Type 1 VM Type 2 Containers

				Application	Application
				1	n
	Application	Application	Application
	1	2	n	Libs	Libs
	Libs	Libs	Libs			Application 1	Application 2
				Guest OS	Guest OS
					other applications	Libs	Libs	other applications
	Guest OS 1	Guest OS 2	Guest OS n
					Hypervisor	Conainter Engine
		Hypervisor			Host Operating System	Host Operating System
		Hardware Infrastructure			Hardware Infrastructure	Hardware Infrastructure
	• Type 1: runs on the hardware directly. It is primarily utilized by cloud providers
	with no host OS and only VMs that run on the open-source Xen or VMware ESX
	hypervisors.
	• Type 2: runs on top of the host operating system and is most commonly found on
	personal computers. VMware servers and VirtualBox are notable examples of this
	category, and most researchers' experiments use them. However, the applications are
	typically slower because of the two operating systems.

  1, we suggest an

		Experiment	Data	Analysis
	Extension	Reproducibility	Replication
		Experimental
	Workload	Orchestrator
	Interface	Metrics	Data	Analysis
	Candidates
	Experimental	Observator
	Protocol

Table 3 .

 3 1: Description of clusters included in the study

	Cluster	Processor	Nodes	RAM
	Dahu	2× Intel Xeon Gold 6130	32 192 GiB
	Chetemi 2× Intel Xeon E5-2630v4	15 768 GiB
	Ecotype 2× Intel Xeon E5-2630Lv4	48 128 GiB
	Paranoia 2× Intel Xeon E5-2660v2	8 128 GiB

Table 3 .

 3 2: STD (mJ) comparison for 3 pinning strategies

	Strategy S1 S2	S3
	Node 1	88 270 1,654
	Node 2	79 283 2,096
	Node 3	58 287 1,725
	Node 4	51 229 1,334

Table 3 .

 3 3: STD (mJ) comparison when enabling/disabling the Turbo Boost

	Turbo Boost Enabled Disabled
	EP / 5 %	310	308
	CG / 25 %	95	140
	LU / 25 %	204	240
	EP / 50 %	84	79
	EP / 100 %	125	110

Table 3 .

 3

	Node	EP	CG	LU
	N1	1370 -9 % 78 +7 % 128 +2 %
	N2	1278 -7 % 64 -1 % 120 +9 %
	N3	1118 +1 % 83 +2 % 93 +7 %

4: STD (mJ) comparison before/after tuning the OS

Table 3 .

 3

	Node	EP	CG	LU
	N1	269 +2 % 83 +1 % 108 -6 %
	N2	195 +1 %	84 -5 % 121 -9 %
	N3	223 +/-1 % 72 -4 % 117 +8 %
	N4	276 +3 % 60 +0 % 113 -3 %

5: STD (mJ) comparison with/without the security patch

Table 3 .

 3 6: STD (mJ) comparison of experiments from 4 clusters

	Cluster	Dahu	Chetemi	Ecotype	Paranoia
	Arch	Skylake Broadwell Broadwell Ivy Bridge
	Freq	3.7 GHz	3.1 GHz	2.9 GHz	3.0 GHz
	TDP	125 W	85 W	55 W	95 W
	5%	364	210	75	76
	50%	98	86	49	244
	100%	119	116	106	240
	RQ 4: Processor Generation

Intel microprocessors have noticeably evolved during these last 20 years. Most new CPUs have enhancements to the chip density, the maximum Frequency, or some optimization features like the C-states or the Turbo Boost. This dynamic evolution caused different generations of CPUs can handle a task differently. This experiment aims not to justify the evolution of the variation across CPU versions/generations but to observe if the user can choose the best node to execute her experiments. Previous papers have discussed the evolution of the energy consumption variation across CPU generations and concluded that the variation is getting higher with the latest CPU generations

[Wang et al.,[START_REF] Marathe | An empirical survey of performance and energy efficiency variation on Intel processors[END_REF]

Table 3 .

 3 7: Experimental Guidelines for Energy Variations

	Guideline	Load	Gain
	Use a low TDP CPU	Low & medium	Up to 3×
	Disable the CPU C-states	Low	Up to 6×
	Use the least of sockets in a	Medium	Up to 30×
	case of multiple CPU
	Avoid the usage of hyper-	Medium	Up to 5×
	threading whenever possible
	Avoid rebooting the machine	High	Up to 1.5×
	between tests
	Do not relate to the machine	Any	-
	idle variation to isolate a test
	EC, the CPU/OS changes its
	behavior when a test is running
	and can exhibit less variation
	than idle
	Rather focus the optimization	Any	-
	efforts on the system under test
	than the OS
	Execute all the similar and	Any	Up to 1.3×
	comparable experiments on the
	same machine. Identical ma-
	chines can exhibit many differ-
	ences regarding their energy be-
	havior

Table 4 .

 4 1: Comparison of CLBG execution times (in seconds) depending on programming languages.

		C C++ Java Python	Go
	pidigits	1.75 1.89 3.13	3.51 2.04
	reverse-complement 1.75 2.95 3.31	16.76 4.00
	regx-redux	1.45 1.66 10.5	15.56 28.69
	k-nucleotide	5.07 3.66 8.66	79.79 15.36
	binary-trees	2.55 2.63 8.28	92.72 28.90
	fasta	1.32 1.33 2.32	62.88 2.07
	Fannkuch-redux	8.72 10.62 17.9 547.23 17.82
	n-body	9.17 8.24 22.0 882.00 21.00
	spectral-norm	1.99 1.98 4.27 193.86 3.95
	Mandelbort	1.64 1.51 6.96 279.68 5.47

  20 A static compiler for Python. It translates the Python code into C and then compiles it using a C compiler. It also supports an extended version of the Python language that allows programmers to call C functions, declare C types, and use static types, which will help translate Python objects into native types, such as integers, and float. This often means better performances since native C libraries are almost all the time faster than the Python written once[START_REF] Pereira | Energy Efficiency Across Programming Languages: How Do Energy, Time, and Memory Relate[END_REF]; 4. Intel Python:21 A customized interpreter developed by Intel to enhance the performance of Python programs. It is dedicated to data sciences and high-performance computing. It uses some Intel kernel libraries, such as Math Kernel Library (Intel MKL22 ) and data analytics acceleration library (Intel DAAL23 ). It supports both versions of Python;5. Active Python:24 It is developed by the Activestates company and provides a standardized Python distribution to ensure license compliance, security, compatibility and performance. Therefore, ActivePython implements its built-in packages (more than 300 packages) and supports both versions of Python;

6. IronPython:

25

A .Net-based Python interpretation platform is written in C# and used with the .Net virtual machine or Mono. It benefits from all the optimizations of .Net virtual machines, such as the JIT and garbage collector mechanisms; 7. GraalPython: 26 A Python interpreter that is based on GraalVM 27 (a universal virtual machine developed by oracle for running applications written in different programming languages). For the time being, it only supports Python 3, and it is still in the experimental stage; 8. Jython: 28 An implementation of Python programming language written in Java for the Java Virtual Machine (JVM). Similar to IronPython and GraalPython, it leverages the optimization mechanisms provided by the JVM to enhance Python performances; 9. MicroPython: 29 A lightweight Python version dedicated to embedded systems and micro-controllers;

Table 4 .

 4 2: Classification of Python implementations All our benchmarks have been executed on a Dell PowerEdge C6420 server, whose hardware features are summarized in Table4.4. The server uses a minimal version of Debian 9 (4.9.0 kernel version) where we install Docker (version 18.09.5).

	Name	Type	Runtime	Optimisations Python JIT GC 2 3
	CPython	Interpreter	C	n o	n o	✓ ✓
	Intel Python	Interpreter	C	n o	n o	✓ ✓
	ActivePython	Interpreter	C	n o	✓	✓ ✓
	PyPy	Interpreter	Python	✓	✓	✓ ✓
	IronPython	Interpreter	.Net	✓	✓	✓ ✓
	GraalPython	Interpreter GraalVM	✓	✓	n o ✓
	Jython	Interpreter	Java	✓	✓	✓ n o
	Stackless Python Interpreter	Python	n o	n o	✓ n o
	MicroPython	Interpreter	c	n o	n o	n o ✓
	Pyston	Interpreter	LLVM	✓	n o	✓ n o
	Unladen Swallow Interpreter	LLVM	✓	n o	✓ n o
	Cython	Compiler	C	n o	n o	✓ ✓
	Nuitka	Compiler	C	n o	n o	✓ ✓
	Shedskin	Compiler	C++	n o	n o	✓ ✓
	Grumpy	Compiler	Go	n o	n o	✓ ✓
	Numba	Library	C	✓	n o	✓ ✓
	Hope	Library	Python	✓	n o	✓ ✓
	Psyco	Library	Python	✓	n o	✓ ✓
	Pyjion	Library	.NET Core ✓	n o	✓ ✓
	Parakeet	Library	C	-	n o	✓ n o

Table 4 .

 4

4: Benchmarking server configuration.

Table 4 .

 4 5: Energy consumption of Python runtimes when executing our benchmark.

	Implementation	array		intArithmetic	doubleArithmetic	hashes		heapsort		trig
		energy (J) p-values energy (J) p-values energy (J) p-values energy (J) p-values energy (J) p-values energy (J) p-values
	ActivePython	402.76	0.008	678.90	0.002	668.16	0.002	977.59	0.008	290.43	0.008	518.32	0.008
	CPython2	361.29	base	560.96	base	548.63	base	646.84	base	275.31	base	411.83	base
	CPython3	323.90	base	743.64	base	740.50	base	797.60	base	243.32	base	413.60	base
	GraalPython	148859.73	0.008	24.63	0.002	24.98	0.002	641.53	0.008 135834.95	0.008	75.21	0.008
	IntelPython2	367.50	0.690	579.62	0.015	561.35	0.485	710.23	0.008	268.70	0.690	439.83	0.151
	IntelPython3	352.09	0.008	767.40	0.002	765.04	0.026	958.77	0.008	265.97	0.008	479.19	0.008
	ipy	305.35	0.008	437.96	0.002	467.42	0.004	1255.44	0.008	256.25	0.016	453.67	0.008
	Jython	517.46	0.008	133.04	0.002	160.65	0.002	635.71	0.056	450.76	0.008	630.22	0.008
	Micropython	307.76	0.008	821.59	0.002	836.10	0.004	9367.15	0.008	335.25	0.008	532.15	0.008
	Nuitka	292.83	0.008	541.95	0.002	543.88	0.004	946.67	0.008	218.31	0.008	390.98	0.008
	Numba2	185.59	0.008	27.72	0.002	35.92	0.004	681.30	0.008	2065.55	0.008	444.84	0.008
	Numba3	12.39	0.008	11.04	0.002	10.76	0.004	920.99	0.008	720.72	0.008	440.99	0.008
	PyPy2	17.43	0.008	29.46	0.002	30.34	0.002	115.53	0.008	20.58	0.008	64.94	0.008
	PyPy3	14.53	0.008	17.95	0.002	18.69	0.004	191.64	0.008	20.47	0.008	65.27	0.008
	Shedskin	47.75	0.008	7.41	0.002	7.37	0.004	1125.24	0.008	44.97	0.008	7.92	0.008
	Implementation	longArithmetic	matrixMultiply	io		stringConcat	nestedLoop	except
	Activepython	661.49	0.008	430.10	0.008	206.48	0.016	15.11	0.310	414.83	0.008	256.64	0.008
	CPython2	550.78	base	395.68	base	192.16	base	13.05	base	416.58	base	433.33	base
	CPython3	735.87	base	440.09	base	197.98	base	13.95	base	380.24	base	226.73	base
	GraalPython	25.66	0.008	45.26	0.008	742.27	0.008	25.18	0.008	11.71	0.008	158.20	0.008
	IntelPython2	566.94	0.016	479.78	0.008	200.37	0.421	14.64	0.151	447.08	0.008	469.17	0.008
	IntelPython3	778.78	0.008	498.81	0.008	213.12	0.008	14.41	0.310	435.29	0.008	275.28	0.008
	ipy	469.11	0.008	416.34	0.008	379.20	0.008	50.13	0.008	336.01	0.008	722.65	0.008
	Jython	162.90	0.008	187.38	0.008	198.02	0.310	21.45	0.008	196.06	0.008	733.46	0.008
	MicroPython	837.34	0.008	536.38	0.008	5353.12	0.008	43.86	0.008	429.79	0.008	360.71	0.008
	Nuitka	542.77	0.008	441.12	0.421	209.21	0.151	12.74	0.310	360.67	0.008	224.42	0.095
	Numba2	35.13	0.008	401.02	0.310	218.04	0.008	20.95	0.008	14.05	0.008	445.62	0.008
	Numba3	10.75	0.008	432.12	0.222	212.23	0.032	19.59	0.008	10.63	0.008	233.97	0.008
	PyPy2	30.64	0.008	24.31	0.008	177.10	0.008	8.73	0.008	26.78	0.008	14.56	0.008
	PyPy3	18.09	0.008	23.58	0.008	256.08	0.008	8.50	0.008	23.16	0.008	14.55	0.008
	Shedskin	8.00	0.008			380.28	0.008	45.01	0.008	7.54	0.008	564.85	0.008

energy (J) p-values energy (J) p-values energy (J) p-values energy (J) p-values energy (J) p-values energy (J) p-values

Table 5 .

 5 1: List of selected JVM distributions.

	Distribution	Provider	Support Selected versions
	HOTSPOT	Adopt OpenJDK	ALL	8.0.275, 11.0.9, 12.0.2, 13.0.2, 14.0.2, 15.0.1
	HOTSPOT	Oracle	ALL	8.0.265, 9.0.4, 10.0.2, 11.0.2, 12.0.2, 13.0.2, 14.0.2, 15.0.1, 16.ea.24
	ZULU	Azul Systems	ALL	8.0.272, 9.0.7, 10.0.2, 11.0.9, 12.0.2, 13.0.5, 14.0.2, 15.0.1
	SAPMACHINE	SAP	ALL	11.0.9, 12.0.2, 13.0.2, 14.0.2, 15.0.1
	LIBRCA	BellSoft	ALL	8.0.275, 11.0.9, 12.0.2, 13.0.2, 14.0.2, 15.0.1
	CORRETTO	Amazon	MJR	8.0.275, 11.0.9, 15.0.1
	HOTSPOT	Trava OpenJDK	LTS	8.0.232, 11.0.9
	DRAGONWELL	Alibaba	LTS	8.0.272, 11.0.8
	OPENJ9	Eclipse	ALL	8.0.275, 11.0.9, 12.0.2, 13.0.2, 14.0.2, 15.0.1
	GRAALVM	Oracle	LTS	19.3.4.r8, 19.3.4.r11, 20.2.0.r8, 20.2.0.r11
	MANDREL	Redhat	LTS	20.2.0.0
	to conduct robust experiments and reduce the factors that can affect the energy consumption
	measurements during our experiments [110].

Java Virtual Machines Candidates. We considered a set of 52 JVM distributions taken from 8 different providers/packagers mostly obtained from SDKMAN, as listed in Table

5

.1. Depending on providers, either all the versions, majors, or LTS are made available by SDKMAN.

Table 5 .

 5 2: List of selected open-source Java benchmarks taken from DACAPO and RENAIS-

	SANCE.
	BenchmarkDescription	Focus
	ALS	Factorize a matrix using the alternating least square	Data-parallel,	compute-
		algorithm on spark	bound
	Avrora	Simulates and analyses for AVR microcontrollers	Fine-grained	multi-
			threading, events queue
	Dotty	Uses the dotty Scala compiler to compile a Scala code-	Data structure, synchroniza-
		base	tion
	Fj-	Runs K-means algorithm using a fork-join framework Concurrent data structure,
	Kmeans		task parallel
	H2	Simulates an SQL database by executing a TPC-C like	Query processing, transac-
		benchmark written by Apache	tions
	Lusearch Searches keywords over a corpus of data comprising the	Externally multi-threaded
		works of Shakespeare and the King James bible
	Neo4j	Runs analytical queries and transactions on the Neo4j	Query Processing, Transac-
		database	tions
	Philosophers Solves dining philosophers problem	Atomic, guarded blocks
	PMD	Analyzes a list of Java classes for a range of source code	Internally multi-threaded
		problems
	Reactors Runs a set of message-passing workloads based on the	Message-passing, critical-
		reactors framework	sections
	Scrabble Solves a scrabble puzzle using Java streams	Data-parallel,	memory-
			bound
	Sunflow Renders a classic Cornell box; a simple scene com-	Compute-bound
		prising two teapots and two glass spheres within an
		illuminated box

Table 5 .

 5 3: Power per request for HOTSPOT, GRAALVM & J9.

	Benchmark	JVM GRAALVM	Power (P) Requests (R) P/R × 10 -3 109 W 5,336 req 20 mW
	Scrabble	HOTSPOT	98 W	3,595 req	27 mW
		J9	92 W	2,603 req	35 mW
		GRAALVM	45 W	510 req	88 mW
	Dotty	HOTSPOT	45 W	597 req	75 mW
		J9	46 W	381 req	120 mW

Table 5 .

 5

	4: Energy consumption when tuning JIT settings on HOTSPOT, GRAALVM & J9
	JVM	Mode	ALS		Avrora	Dotty		Fj-kmeans	H2
		Default	2848 p-values	3861 p-values	2271 p-values	948 p-values	p-values
	GRAALVM	DisableJVMCI	3099	0.001	4012	0.041	2694	0.001	934	0.011	0.005
		Economy	4503	0.001	3895	0.793	3466	0.001	1306	0.002	0.001
		Default	3792 p-values	2122 p-values	3515 p-values	1271 p-values	p-values
		Thread 1	4157	0.001	2121	0.875	4749	0.001	1297	0.097	0.066
		Thread 3	3849	0.018	2105	0.713	3574	0.104	1259	0.371	0.637
		Thread 7	3843	0.041	2386	0.372	3511	0.875	1259	0.25	0.637
		Count 0	8461	0.001	2425	0.001	4877	0.001	2289	0.002	0.001
	J9	Count 1 Count 10	4281 3980	0.001 0.001	2150 2431	0.431 0.713	3164 3771	0.001 0.001	1841 1312	0.002 0.011	0.431 0.003
		Count 100	3878	0.007	2141	0.713	3469	0.227	1363	0.523	0.128
		Cold	6788	0.001	2134	0.637	4855	0.001	1636	0.002	0.001
		Warm	4594	0.001	2112	0.713	4253	0.001	1244	0.055	0.128
		Hot	7553	0.001	2310	0.001 12749	0.001	1452	0.002	0.001
		VeryHot	15113	0.001	3300	0.001 18235	0.001	2430	0.002	0.001
		Schorching	18316	0.001	3541	0.001 21686	0.001	2514	0.002	0.001
		Default	2997 p-values	4014 p-values	2516 p-values	934 p-values	p-values
		Graal	2999	0.637	3971	0.318	2512	0.318	929	0.609	0.007
	HOTSPOT	Lvl 0	491443	/ 14484	/ 84395	/	/	/	/
		Lvl 1	/	/	3731	0.001	3302	0.001	1256	0.002	0.001
		Lvl 2	3079	0.004	4110	0.189	3723	0.001	22547	0.002	0.001
		Lvl 3	16375	0.001	7729	0.001	6789	0.001 144914	0.002	0.001
		NotTired	3254	0.001	3901	0.189	3110	0.001	912	0.021	0.227
	JVM	Mode	Neo4j	Pmd		Reactors	Scrabble	Sunflow
		Default	3313 p-values	297 p-values	23452 p-values	452 p-values	p-values
	GRAALVM	DisableJVMCI	5086	0.001	353	0.001	25007	0.007	503	0.002	0.227
		Economy	9525	0.001	270	0.001	30317	0.001	649	0.002	0.002
		Default	4336 p-values	277 p-values	12705 p-values	734 p-values	p-values
		Thread 1	4906	0.001	350	0.001	12800	0.713	948	0.002	0.005
		Thread 3	4477	0.005	294	0.004	12647	0.875	795	0.021	0.27
		Thread 7	4431	0.104	273	0.372	12600	0.875	808	0.055	0.372
		Count 0	10565	0.001	744	0.001	18084	0.001 1476	0.002	0.001
	J9	Count 1 Count 10	7166 4979	0.001 0.001	272 299	0.128 0.001	14715 12000	0.001 1005 0.104 860	0.002 0.005	0.052 0.001
		Count 100	4547	0.001	262	0.031	12313	0.024	768	0.16	0.004
		Cold	7250	0.001	275	0.372	20380	0.001	870	0.005	0.001
		Warm	5305	0.001	411	0.001	13726	0.001	913	0.002	0.001
		Hot	8979	0.001	857	0.001	36534	0.001 1180	0.002	0.128
		VeryHot	19359	0.001	793	0.001	38303	0.001 5420	0.002	0.001
		Schorching	26409	0.014	808	0.001	43929	0.001 5583	0.002	0.001
		Default	4787 p-values	323 p-values	11685 p-values	530 p-values	p-values
		Graal	4750	0.372	327	0.189	11548	0.523	537	0.701	0.564
	HOTSPOT	Lvl 0	356287	/ 1073	/ 148381	/	/	/	/
		Lvl 1	8304	0.001	222	0.001	22410	0.002	735	0.002	0.007
		Lvl 2	19058	0.001	226	0.001	40701	0.002 2291	0.002	0.001
		Lvl 3	44594	0.001	330	0.005 190124	0.002 9070	0.002	0.001
		NotTired	3844	0.001	933	0.001	11256	0.041	588	0.003	0.001

Table 5 .

 5 5: The different J9 GC policies

	Policy	Description
	Balanced	Evens out pause times & reduces the overhead of the costlier operations
		associated with GC
	Metronome	GC occurs in small interruptible steps to avoid stop-the-world pauses
	Nogc	Handles only memory allocation & heap expansion, with no memory
		reclaim
	Gencon (default)	Minimizes GC pause times without compromising throughput, best for
		short-lived objects
	Concurrent Scav-	Minimizes the time spent in stop-the-world pauses by collecting nursery
	enge	garbage in parallel with running application threads
	optthruput	Optimized for throughput, stopping applications for long pauses while
		GC takes place
	Optavgpause	Sacrifices performance throughput to reduce pause times compared to
		optthruput

Table 5 .

 5 6: The different HOTSPOT/GRAALVM GC policies

	Policy	Description
	G1GC (default)	Uses concurrent & parallel phases to achieve low-pauses GC and main-
		tain good throughput
	SerialGC	Uses a single thread to perform all garbage collection work (no threads
		communication overhead)
	ParallelGC	Known as throughput collector: similar to SerialGC, but uses multiple
		threads to speed up garbage collections for scavenges
	parallelOldGC	Use parallel garbage collection for the full collections, enabling it
		automatically enables the ParallelGC

Table 5 .

 5 7: Energy consumption when tuning GC settings on HOTSPOT, GRAALVM & J9

	JVM	Mode		ALS	Avrora	Dotty	H2	Neo4j
		Default	2570 p-values 4153 p-values 2223 p-values 1870 p-values	5256 p-values
		1Concurent	2567	0.403 4007	0.023 2220	1.000 1883	0.982	5368	1.000
		1Parallel	2668	0.012 3904	0.008 2228	0.835 2022	0.000	5836	0.012
		5Concurent	2570	0.676 4117	0.161 2215	0.210 1862	0.505	5259	1.000
	GRAALVM	5Parallel	2561	0.676 3863	0.012 2237	1.000 1910	0.103	5223	0.403
		DisableExplicitGC	2559	0.210 3911	0.003 2215	1.000 1978	0.018	5106	0.210
		ParallelCG	2720	0.012 4016	0.206 2237	0.531 1945	0.000 13172	0.037
		ParallelOldGC	2715	0.012 4032	0.103 2221	1.000 1925	0.002 13362	/
		Default	3371 p-values 2243 p-values 3237 p-values 2107 p-values	6277 p-values
		Balanced	9012	0.012 2232	0.597 3429	0.012 2247	0.002	8853	0.012
		ConcurrentScavenge	3487	0.012 2270	0.280 3388	0.012 2319	0.001	6857	0.012
		Metronome	2098	0.012 2265	0.505 3815	0.012 2717	0.000 12103	0.012
	J9	Nogc	3454	0.022 2239	0.872 3259	0.144 2207	0.031 61781	0.012
		Optavgpause	3601	0.012 2431	0.370 3425	0.012 2169	0.297	7495	0.012
		Optthruput	3357	1.000 2432	0.241 3178	0.403 2194	0.139	6324	0.835
		ScvNoAdaptiveTenure 3494	0.012 2253	0.800 3248	0.835 2161	0.103	8442	0.012
		Default	2765 p-values 4115 p-values 2492 p-values 1673 p-values	8152 p-values
		1Concurent	2775	0.060 4137	0.346 2493	0.676 1675	0.918	8062	0.531
		1Parallel	2863	0.012 4142	0.800 2526	0.037 1853	0.001	8270	0.676
		5Concurent	2758	0.676 4091	0.872 2485	0.296 1681	0.608	8087	0.835
		5Parallel	2767	0.144 4176	0.077 2473	0.060 1654	0.720	8046	0.835
	HOTSPOT	DisableExplicitGC	2734	0.012 4062	0.448 2483	0.835 1702	0.248	7710	0.037
		ParallelCG	2653	0.012 4064	0.629 2356	0.012 1602	0.008	8953	0.060
		ParallelOldGC	2764	0.531 4070	0.872 2525	0.802 1675	0.959	7963	0.403
		SerialGC	2593	0.012 4083	0.395 2378	0.012 1620	0.046	5745	0.012
	JVM	Mode		Pmd	Reactors	Scrabble	Sunflow
		Default	281 p-values 2611 p-values	410 p-values 353 p-values
		1Concurent	286	0.182 2664	1.000	413	0.885 347	0.573
		1Parallel	298	0.000 2869	0.144	561	0.030 317	0.000
		5Concurent	282	0.980 2611	0.531	414	0.885 362	0.356
	GRAALVM	5Parallel	282	0.538 2682	0.531	424	0.112 353	0.758
		DisableExplicitGC	281	0.758 2704	0.676	400	0.312 332	0.036
		ParallelCG	282	0.878 2267	0.022	545	0.030 329	0.003
		ParallelOldGC	282	0.918 2514	0.012	535	0.030 329	0.008
		Default	232 p-values 1644 p-values	589 p-values 510 p-values
		Balanced	235	0.412 1902	0.020	661	0.061 519	0.505
		ConcurrentScavenge	233	0.878 1705	0.903	639	0.194 546	0.018
		Metronome	239	0.022 2089	0.020	758	0.030 422	0.000
	J9	Nogc	227	0.151 1505	0.066	711	0.030 499	0.720
		Optavgpause	253	0.000 1772	0.391 1089	0.030 478	0.046
		Optthruput	232	0.878 1554	0.111	640	0.194 429	0.000
		ScvNoAdaptiveTenure 228	0.137 1908	0.020	618	0.665 528	0.218
		Default	316 p-values 1546 p-values	484 p-values 347 p-values
		1Concurent	316	0.383 1533	0.665	478	0.470 334	0.218
		1Parallel	334	0.000 1747	0.030	592	0.030 320	0.002
		5Concurent	314	0.330 1497	0.665	469	0.030 336	0.259
		5Parallel	316	0.573 1546	0.470	489	0.470 342	0.573
	HOTSPOT	DisableExplicitGC	312	0.200 1545	0.470	470	0.061 325	0.014
		ParallelCG	300	0.000 1476	0.885	579	0.030 336	0.081
		ParallelOldGC	314	0.720 1582	0.194	475	0.470 333	0.151
		SerialGC	307	0.002 1672	0.061	601	0.030 352	0.473

Table 5 .

 5 8: J-Referral recommendations.

	Project	Metric	Energy	JVM	Execution flags
	Zip4J	Least energy 2210 J 16-sapmchn Most energy 3680 J 8.0.292-J9	default default
	K-nucl	Least energy 1296 J 21.1.r16-grl Most energy 4433 J 15.0.1-J9	default -Xjit:optlevel=cold

Table 6 .

 6 1: Number of available web frameworks per programming language.

	Language	Bb Query Update Plaintext Fortune Json Total
	c	1	1	1	6	1	5	15
	c#	21	20	14	12	14	17	98
	c++	27	16	14	20	13	25	115
	cfml	2	1	1	1	1	2	8
	clojure	8

Table 6 . 2 :

 62 Stress levels for each scenario.

	Scenario	type of stress	level 1 level 2 level 3 level 4 level 5 level 6 level 7
	Single Query	Number of parallel clients	16	32	64	128	256	512	/
	Multiple Queries	Number of rows to read from the database	1	5	10	15	20	30	50
	Update Queries	Number of rows to update in the database	1	5	10	15	20	30	50
	Fortunes	Number of parallel clients	16	32	64	128	256	512	/
	JSON Serialization Number of parallel clients	16	32	64	128	256	512	/
	Plain Text	Number of parallel clients	256	1024	4096 16384 32384	/	/

Table 6 .

 6 3: Average power consumption of frameworks based on the database type

	Query size	1	5	10	15	20	30	50
	MongoDB	97.17	96.93	93.38	92.58	91.61 92.585 91.17
	MySQL	113.86 112.92 112.74 113.05 112.13 112.62 112.16
	PostgresSQL 113.86 108.25 106.19 102.97 103.41 101.95 102.96

https://www.internetworldstats.com/stats.htm

https://en.wikipedia.org/wiki/Interstellar_(film)

https://www.theshiftproject.org/article/ict-environmental-impact/

https://www.statista.com/statistics/271139/energy-consumption-of-ict-worldwide

https://www.cpubenchmark.net/compare/Intel-i9-12900KS-vs-Intel-i9-12900KF/ 4813vs4611

Trouble at the lab, The Economist, 19 October 2013; www.economist.com/news/briefing/ 21588057scientists-think-science-self-correcting-alarming-degree-it-not-trouble.

https://www.spec.org/cpu2017/

https://gwpg.spec.org/benchmarks/benchmark/specviewperf-2020-v3-1/

https://www.spec.org/power_ssj2008

https://www.nas.nasa.gov/publications/npb.html

https://www.hpcchallenge.org/

https://benchmarksgame-team.pages.debian.net/benchmarksgame/index.html

https://gatling.io

https://github.com/session-replay-tools/tcpcopy

https://www.itssolution.com/products/trellis-power-insight-application

https://benchmarksgame-team.pages.debian.net/benchmarksgame/index.html

https://openjdk.org/groups/hotspot/

https://www.eclipse.org/openj9/

http://evaluate.inf.usi.ch/

https://kernel.ubuntu.com/~cking/stress-ng

https://launchpad.net/pbzip2/

http://www.netlib.org/linpack

https://linux.die.net/man/1/sha256sum

https://github.com/grid5000/environments-recipes/blob/master/debian9-x64-min.yaml

https://github.com/anonymous-data/Energy-Variation

https://kernel.ubuntu.com/~cking/stress-ng

https://www.jetbrains.com/research/python-developers-survey-2018/

https://en.wikipedia.org/wiki/Tower_of_Hanoi

https://benchmarksgame-team.pages.debian.net/benchmarksgame/index.html

The options in this survey were not mutually exclusive. As a result, the total of the percentages is greater than 100%.

https://github.com/davidcpage/cifar10-fast

https://github.com/wbaek/torchskeleton

https://www.djangoproject.com/

https://flask.palletsprojects.com/

https://lp.jetbrains.com/python-developers-survey-2021/

https://github.com/powerapi-ng/joulehunter

https://www.python.org/doc/essays/list2str

https://www.pythonpool.com/for-vs-while-loop-python/

http://python-history.blogspot.com/2010/06/from-list-comprehensions-to-generator. html?m=1

https://wiki.python.org/moin/TimeComplexity

https://docs.python.org/3/library/multiprocessing.html

https://docs.python.org/3/library/threading.html

https://www.python.org/

http://pypy.org

https://github.com/cython/cython

http://www.tommti-systems.de/main-Dateien/reviews/languages/benchmarks.html

https://docs.python.org/3.7/library/2to3.html

https://www.activestate.com/solutions/why-activestate/

https://pypl.github.io/PYPL.html

https://www.tiobe.com/tiobe-index/

https://sdkman.io

https://github.com/alpinelinux/docker-alpine

https://openbenchmarking.org

https://anonymous.4open.science/r/jvm-comparaison-213E/Readme.md

https://github.com/chakib-belgaid/jvm-comparaison

https://github.com/chakib-belgaid/jreferral

https://github.com/chakib-belgaid/jvm-comparaison

https://gee.cs.oswego.edu/dl/html/j9mm.html

https://blogs.oracle.com/javamagazine/the-unsafe-class-unsafe-at-any-speed

[https://www.eclipse.org/openj9/docs/jit/]

https://github.com/srikanth-lingala/zip4j

https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/ knucleotide.html

https://github.com/chakib-belgaid/jreferral

https://www.techempower.com/benchmarks

https://developers.google.com/protocol-buffers

https://github.com/chakib-belgaid/energy_ghz

https://github.com/powerapi-ng/hwpc-sensor

https://github.com/chakib-belgaid/energy_ghz

https://github.com/chakib-belgaid/energy_benchmarking_grpc2

https://github.com/wg/wrk

https://man7.org/linux/man-pages/man7/epoll.7.html

https://man.openbsd.org/kqueue.2

https://www.grid5000.fr/w/Hardware

[START_REF]Pearson's Correlation Coefficient[END_REF]203

https://github.com/chakib-belgaid/

https://github.com/chakib-belgaid/FrameworkBenchmarks

https://github.com/chakib-belgaid/frameworks-benchmarks-results

https://github.com/chakib-belgaid/frameworks-benchmarks-results

https://matt-42.github.io/lithium/

https://github.com/hapijs/hapi

https://www.techempower.com/benchmarks

https://www.zend.com/

https://peps.python.org/pep-0557/

https://docs.python.org/3/whatsnew/3.9.html

https://peps.python.org/pep-0613/

https://github.com/faster-cpython/ideas

https://www.ponicode.com/

https://github.com/features/copilot

https://githubnext.com/projects/copilot-labs/

https://ecograder.com/

https://github.com/davidson-consulting/diff-jjoules

Acknowledgements

Chapter 6

Energy Footprint of Distributed Programming Frameworks