Understanding the Performance-Energy Tradeoffs of Object-Relational Mapping Frameworks

Alexandre Bonvoisin, Clément Quinton, Romain Rouvoy Univ. Lille, CNRS, Inria, UMR 9189 CRIStAL, F-59000 Lille, France {alexandre.bonvoisin;clement.quinton;romain.rouvoy}@univ-lille.fr

Abstract-Object-Relational Mapping (ORM) frameworks are the cornerstone of online services. To reply to incoming requests, these services often rely on these frameworks as a convenient data access layer. However, such frameworks might also be the source of performance inefficiency when configured and used inappropriately.

This paper, therefore, compares different configurations of state-of-the-art Java-based ORM frameworks to unveil their performance efficiency, traditionally evaluated through metrics such as execution time and memory usage. However, rising environmental concerns have brought energy consumption to the forefront of the conversation. Beyond performance-centric measurements, we shed light on the energy consumption of these building blocks and explore the trade-offs that conceal the expected quality of service and environmental concerns.

Our empirical results, obtained with an ORM-based version of the reference Transaction Processing Performance Council benchmark C (TPC-C) benchmark, highlight that the adoption of an ORM should be carefully configured by developers to leverage the resources offered by underlying databases.

Index Terms-ORM Frameworks, Energy consumption, Performance

I. INTRODUCTION

The energy consumption of software systems has gained prominence as a significant environmental and societal concern. Over the past decade, multiple studies have underscored the substantial influence of software on energy consumption and emphasized the importance of green software design to improve the energy efficiency of software systems. For instance, previous works in this field have studied and compared the energy consumed by common operations on Java collections [START_REF] Hasan | Energy profiles of java collections classes[END_REF] or the use of different Java I/O libraries [START_REF] Ournani | Evaluating The Energy Consumption of Java I/O APIs[END_REF]. However, while various tools and methods have been developed to comprehend, measure, and predict software energy consumption [START_REF] Rieger | Survey of approaches for assessing software energy consumption[END_REF], the complexity of modern software environments and the composition of software layers make this sustainability objective particularly challenging [START_REF] Ournani | Evaluating The Energy Consumption of Java I/O APIs[END_REF].

Especially, little effort has been invested in the study of the energy efficiency of the data access layer. Yet, there exist various means to access and manage data, ranging from low-level and hand-crafted approaches to more abstract and user-friendly frameworks delivering some added features. This is particularly the case in Java, which provides-on the one hand-the Java DataBase Connectivity (JDBC) API, a standard library to communicate with any database from a Java program. Using JDBC, developers can retrieve data sets by writing queries and running procedures using the Structured Query Language (SQL), hence mixing SQL code with their business logic. On the other hand, Jakarta Persistence API (JPA) is another Java standard that maps data stored in relational databases with Java objects, in a so-called Object-Relational Mapping (ORM). For example, ECLIPSELINK 1 and HIBERNATE 2 are the most popular JPA implementations [START_REF] Chen | Detecting problems in the database access code of large scale systems -an industrial experience report[END_REF], [START_REF] Katamreddy | Working with JDBC[END_REF]. These ORM frameworks are widely appreciated for developing applications that interact with database management systems [START_REF] Yan | Understanding database performance inefficiencies in real-world web applications[END_REF]. Interestingly, they provide automated support for managing an object-oriented model of data stored in a relational one. While such frameworks tend to ease the development of applications manipulating persistent data, it remains unclear if they succeed in delivering energy-efficient solutions to query and retrieve data at large. In this paper, we thus assess the performance and related energy consumption of the plain JDBC and ORM-based approaches. Precisely, the purpose of the study is to answer the following research questions:

RQ1: How much energy overhead is introduced by ORM strategies compared to plain JDBC? RQ2: How does ORM framework performance relate to energy efficiency? RQ3: Does the configuration of an ORM framework influence its energy consumption? Concretely, we leverage the widely accepted Transaction Processing Performance Council benchmark C (TPC-C) to compare the data access strategies implemented by the popular ECLIPSELINK and HIBERNATE ORM frameworks. We take advantage of extensive performance and energy-related measurements to study diverse ORM features affecting the energy efficiency of Java applications. The key contributions of this paper can therefore be summarized as:

1) Providing a better understanding of the energy and performance variations of different ORM strategies under diverse workloads; 2) Identifying controllable factors that contribute to the variation in ORM frameworks' energy consumption; We believe that this empirical knowledge can be effectively used by Java developers to reduce the energy footprint of their applications, while eventually improving their performances.

The remainder of this paper is organized as follows. Section II introduces the data access strategies commonly implemented by Java applications. Section III details the experimental protocol we adopted to measure the energy efficiency of ORM frameworks. Section IV dives into the results of our experiments to answer the above research questions. Finally, Section VI discusses related work, while Section VII concludes the paper.

II. DATA ACCESS STRATEGIES

In this section, we introduce the 5 data access strategies we assessed. In particular, we cover both raw and ORM strategies, the latter being built upon JPA 3.1 features. 3 JPA is a Java specification that eases the management of relational data in applications. Within JPA, entities are Java classes representing persistent data structures stored in relational databases. These entities are typically annotated to define how they map to corresponding database tables, outlining the relationships between Java objects and database columns (cf. Listing 1). Each instance of an entity corresponds to a record or row in the associated database table. Furthermore, entities can be annotated to specify additional metadata, including the definition of plain-text queries associated with unique names, later used to identify a query to be executed. @SqlResultSetMapping( name = "Customer.getEmailSQLMapping", columns = @ColumnResult( name = "email", type = String.class)) @NamedNativeQuery(name = "Customer.getEmailSQL", query = "SELECT email FROM customer_table " + "WHERE unique_id = ?", resultSetMapping = "Customer.getEmailSQLMapping") @NamedQuery(name = "Customer.getEmailJPQL", query = "SELECT c.email FROM Customer c " + "WHERE c.id = ?1") @Entity @Table(name = "customer_table") public class Customer implements Serializable { @Column(name = "unique_id") @Id private Integer id; // Primary key @Column(name = "email") private String email; }

Listing 1. Entity class declarations

Beyond mapping, JPA also introduces two caching features. The first-level cache a.k.a. "Persistence Context" is bound to a database transaction and cannot be disabled, as its goal is also to monitor changes on "managed" entities. When an entity is retrieved from the database, it is stored in this cache, ensuring data consistency by automatically updating changes to the object and reducing the number of database queries during the transaction. As for the second cache level, it is optional. If used, it extends the persistence context and developers must take care of the data cached to prevent data staleness. As this cache requires an external caching library, we disabled it to solely assess ORM capabilities without introducing any bias related to the choice of the caching library.

When conforming to JPA features, one can seamlessly transition between various vendors without modifying the application source code. This migration process merely involves 3 https://jakarta.ee/specifications/persistence/3.1/ substituting the vendor-specific package in the project path, ensuring a fair comparison among providers. To illustrate how each strategy handles queries, we provide for each of the 5 strategies the related code sample to execute the same query, i.e., select the email for a given customer.

A. Plain JDBC Access Strategy

When they are not using an ORM, developers leverage a JDBC driver to interact with the Database Management System (DBMS). As illustrated in Listing 2, they explicitly manage database connections, craft SQL queries, and handle query results. This approach provides fine-grained control over the DBMS but demands specific skills for developers, including writing and maintaining custom SQL queries and dealing with low-level database intricacies. In the remainder of the paper, we will refer to this implementation as the plain JDBC one.

String query = "SELECT email FROM customer_table "

+ "WHERE unique_id = ?"; ResultSet rs = jdbcCon.prepareStatement(query)

.setInt(1, id).executeQuery(); String email = rs.next()?rs.getString("email"):""; Listing 2. Plain JDBC access strategy

B. ORM Access Strategies

When adopting an ORM framework to interact with the DBMS, 4 distinct strategies can be considered. Each of these data access strategies comes with its benefits and trade-offs, and the choice of a strategy depends on various factors, such as performance requirements, compatibility with DBMS features, and code maintainability.

1) Native Access Strategy: This approach refers to the use of raw SQL queries using the JPA's API (cf. Listing 3). As the specification's terminology refers to raw SQL queries as "native queries", we named this strategy as the "native" one. In contrast to the plain JDBC strategy, which imposes manual data mapping, the JPA standard encourages that the mapping between query result sets and Java objects are declared through the framework configuration (cf. Java annotations in Listing 1). 2) JPQL API Access Strategy: This data access strategy uses the Jakarta Persistence Query Language (JPQL), a query language with an SQL-like syntax. This language provides an abstraction layer over raw SQL queries that enables developers to define queries independently of the SQL dialect used by the targeted data store. In JPQL, developers manipulate Java fields and class names, which are mapped to database tablesinstead of explicitly specifying column and table names within the queries (cf. Listing 1). This approach simplifies the query creation process and enhances portability, as the queries can be adapted to different databases without any modification (cf. Listing 4).

III. EXPERIMENTAL METHODOLOGY

This section describes our hardware settings, the benchmarks we used, and our data collection process. All experimental materials and performance data monitored during our experiments are publicly accessible. 4

A. Hardware Settings

To enforce reproducibility, we conducted all our experiments using a consistent technical setup. The hardware infrastructure comprises two identical DELL POWEREDGE R640 servers, each equipped with a single INTEL XEON GOLD 5220 @2.20GHz processor (18 cores/36 threads), 96 GB of DRAM, 480 GB SSD, and two 25Gb/s network cards.

B. Database Benchmark

To assess the energy efficiency of strategies described in Section II, we opted for the Transaction Processing Performance Council benchmark C (TPC-C)5 [START_REF] Tözün | From a to e: Analyzing tpc's oltp benchmarks: The obsolete, the ubiquitous, the unexplored[END_REF] defined in the BENCHBASE (formerly OLTPBENCH) framework, 6 an open-source database benchmarking tool [START_REF] Difallah | Oltpbench: An extensible testbed for benchmarking relational databases[END_REF]. While this wellknown benchmark was initially designed to assess database capabilities through an intensive workload simulating business activities between customers and a wholesale supplier, many aspects of its specification are well-suited for evaluating data access layer strategies.

1) Database Schema: The schema consists of 9 tables representing various entities involved in wholesale supplier operations. For instance, the Warehouse table stores unique identifiers and addresses of warehouses, while the Customer table holds details about individual customers, including identifiers, names, and number of payments. The number of warehouses is a key parameter to set up and run the benchmark, as it serves as a scaling factor, for adjusting both the number of records to be inserted into the tables and the number of concurrent users to be simulated.

2) Input Workload: In the TPC-C benchmark specification, a transaction is a sequence of SQL queries designed to simulate specific business operations. There are 5 types of transactions of various complexity levels, each with a frequency that determines the proportion of transactions of that type in the workload. We used the same frequencies as defined in [START_REF] Tözün | From a to e: Analyzing tpc's oltp benchmarks: The obsolete, the ubiquitous, the unexplored[END_REF]. The queries defined within the transactions are pivotal to this study, as they provide a comparable basis to the queries generated by the different ORM strategies.

C. ORM Frameworks

As introduced previously, we employed the TPC-C implementation from the BENCHBASE framework as the basis for implementing our data access strategies. Apart from the modification of the data access layer, we also edited the framework's implementation related to the transaction retry feature. Rather than making additional attempts to execute a failed transaction again, our implementations categorize such attempts as failures from the outset. This adjustment was implemented in all benchmark variants to ensure that the measurements collected remain unaffected by potential variations introduced by retry attempts. As ORM strategies are implemented according to the last 3.1 version of JPA, the considered JPA provider at build time are ECLIPSELINK (version 4.0.0), the reference implementation, and HIBERNATE (version 6.1.3.F inal), the most popular alternative in the community.

D. Data Collection Process

To collect data on different metrics for each access strategy, we implemented a thorough measurement protocol.

1) Collected Metrics: To assess the efficiency of data access layer strategies, we focused on two key performance metrics: power consumption, which reflects resource usage, and goodput, which highlights operational efficiency.

a) Power Consumption: We measured power usage for both the CPU package (PKG) and the Dynamic Random Access Memory (DRAM). This measurement was carried out using the Running Average Power Limit (RAPL) interface [START_REF] David | Rapl: Memory power estimation and capping[END_REF], a feature available with recent Intel processors known for its high accuracy and reliability in power monitoring [START_REF] Khan | Rapl in action: Experiences in using rapl for power measurements[END_REF], [START_REF] Desrochers | A validation of dram rapl power measurements[END_REF]. The energy efficiency of an assessed strategy is determined by the system's power consumption during the workload, after subtracting the idle power consumption, as described in [START_REF] Guégain | On reducing the energy consumption of software product lines[END_REF].

b) Goodput: The goodput is a metric reported by the benchmark that indicates the efficiency of a database system by measuring the number of successful transactions completed per second. A higher goodput indicates increased effectiveness, as it directly correlates with the system's ability to execute successful operations at a faster rate.

2) Software Setup: We configured both servers with a minimal version of Ubuntu 20.04, including only essential components required for system operation and remote access through Secure Shell (SSH). On one server, we installed Git, XMLStarlet, and Docker to facilitate the build and execution of Java applications implementing the strategies covered by this study. These implementations are compiled as Java 17 applications using a Maven container set up for reproducible builds. They are then packaged into a Docker image based on eclipse-temurin:17.0.8_7-jre-alpine, ensuring a lightweight Java Runtime Environment (JRE). While Docker introduces an additional layer that might lead to increased energy consumption, it has a consistent and minimal impact on energy variation, resulting in a constant, negligible overhead [START_REF] Santos | How does docker affect energy consumption? evaluating workloads in and out of docker containers[END_REF]. The other server was exclusively equipped with Docker to run an on-demand, volatile postgres:14.5 database mounted with a docker volume cleaned before each measurement. The database is started and populated with an initial dataset that will be processed by the workload transactions. As the benchmark's default behavior is to randomly generate these records, we set the random seed to 0, a commonly used value in Java programs. This adjustment guarantees consistent initial data for all measurements. It is also important to note that our measurements were conducted with a single Docker container process running on the servers. Finally, all running containers are stopped to put both systems in an idle state.

3) Metrics Collection: A single measurement run is a multi-step process conducted as follows. First, we measure the power consumption of both servers while they are at rest. These measurements are taken over 5 seconds, as we observed that if the CPU is not actively in use, power consumption quickly stabilizes within this period. Throughout this time frame, both systems remain in an idle state, with no active applications or processes running. Second, we start the remote database and wait 30 seconds for it to set up and be ready to accept requests. We then initiate the execution of the assessed software variant, during which we gather data on the energy consumption and the achieved goodput throughout the workload duration, set to 60 seconds in the BENCHBASE default configuration file. Finally, we dismantle the database and perform an additional 5-second measurement to assess the power consumption of both servers while back at rest. 4) Workload Parameters: We evaluated each data access strategy under changing workloads, setting the target rate from 10 to an unlimited number of Transaction Per Second (TPS), with increments of 10, 100, 1000, and 10.000 TPS. Given the stochastic nature of the TPC-C workload, we repeated the measurement runs 40 times for each combination of rate goal and data access strategy and calculated the median of the results, adding up to 4200 runs.

In the default configuration of the BENCHBASE framework, the scaling factor presented in Section III-B1 is set such that it simulates a workload with too many clients in comparison to the capabilities of the machine. To align with the benchmark specification recommending 10 clients per unit of the scaling factor, we set the number of concurrent clients to 50. By using a scaling factor of 5 and allowing 50 clients to concurrently query the database, we aim to create a workload that is close enough to real-world scenarios. To prevent inconsistencies and ensure that transactions do not interfere with each other, transactions were performed at the serializable isolation level, also defined in the BENCHBASE default configuration file.

IV. EMPIRICAL RESULTS

We consider the plain JDBC strategy as the baseline to evaluate the overhead induced by other strategies. By assessing these different ORM-based strategies, we aim to gain better insights into the influence of such strategies and the trade-off they offer between performance and energy efficiency.

A. RQ1: How much energy overhead is introduced by ORM strategies compared to plain JDBC?

To answer our first research question, we thus compared the energy consumption of the different ORM access strategies with the plain JDBC one when executing a certain workload, as explained in Section III-D. Figures 1(a) and 1(b) depict, for each implementation, the median power usage depending on the target rate for CPU and DRAM, respectively (EL stands for ECLIPSELINK and Hi for HIBERNATE). Both figures highlight consistent trends as the target rate increases, i.e., the power usage increases accordingly. Yet, one can observe that HIBERNATE ENTITY demonstrates low energy consumption at higher rates. This can be explained by its poor performance. As shown by Figure 2, the number of completed transactions HIBERNATE ENTITY can process is plateauing once the target rate reaches 1000 transactions per second. HIBERNATE ENTITY thus faces performance issues related to completed transactions, arising because of the increased concurrency resulting from the higher target rate. Consequently, HIBER-NATE ENTITY transitions into a degraded mode, causing the built-in workload injector to reduce the number of initiated transactions which, in turn, reduces energy consumption.

HIBERNATE ENTITY left apart, we observe that the plain JDBC strategy exhibits the lowest energy consumption, while the use of ORM strategies results in a significant power usage increase, especially regarding CPU consumption. For instance, at the target rate 100, ORM strategies overhead varies from 16% to 59% for the ECLIPSELINK NATIVE and HIBERNATE CRITERIA strategies, respectively. As ORM strategies introduce an additional abstraction layer compared to the plain JDBC one (see Section II-B), such trends were predictable. For a comprehensive and fair analysis, the benchmark must execute an identical input workload. As some ORM strategies generate the executed SQL queries, we ensured that strategy implementations adhered closely to the business logic defined in the benchmark while complying with the concepts outlined by each ORM strategy. Among all studied ORM strategies, only the Native ones guarantee the execution of SQL queries that match those defined in the plain JDBC strategy. When focusing on Native strategies, one can observe that, although the overhead is less significant, it is non-negligible. For instance, at the target rate of 10, 000, there is an 11% and 20% increase for the native ECLIPSELINK and HIBERNATE ORM regarding CPU usage, respectively. While less important, the energy consumption increase regarding DRAM usage for the same target rate even so reaches 6% and 13% for the native ECLIPSELINK and HIBERNATE ORM strategies, respectively.

RQ1: Our empirical study shows that the plain JDBC strategy consistently reports being the most energyefficient compared to the ORM ones. The overhead introduced by ORM strategies is imposed by the additional abstraction layers offered by such frameworks. Among ORM strategies and apart from the specific case of HIBERNATE ENTITY which performs poorly at high rates, Criteria is the most consuming strategy for both HIBERNATE and ECLIPSELINK frameworks.

B. RQ2: How does ORM framework performance relate to energy efficiency?

To answer our second research question, we compared the goodput of ECLIPSELINK and HIBERNATE given their energy consumption when executing a varying workload. Figure 2 provides an overview of this comparison. Several observations can be made. When employing the Native strategy, ECLIPSELINK and HIBERNATE demonstrate similar perfor-mances as they both execute identical raw SQL queries, resulting in comparable sets of completed transactions. However, ECLIPSELINK seems to offer the best energy efficiency, consuming 45% less energy at a rate of 10 and consistently showing an average reduction of 22% across all rates.

Regarding the other three strategies, ECLIPSELINK always exhibits a recurrent performance pattern, performing consistently across varying rates and reaching equivalent or higher goodput than HIBERNATE, until reaching 10, 000 transactions per second. Once this rate is overtaken, ECLIPSELINK goodput is then plateauing.

In contrast, the HIBERNATE goodput remains consistent across the Native, JPQL, and Criteria strategies, even at unlimited rates. Regarding these 3 strategies, ECLIPSELINK is always better performing while consuming less energy than HIBERNATE at rates lower or equivalent to 10, 000, as highlighted by Figures 2(b) and 2(c).

--Some columns have been omitted for clarity SELECT w1_0.w_id, w1_0.w_tax, . We also observe that HIBERNATE experiences a significant drop in performance when managing transactions relying on the Entity strategy (see Figure 2(d)). When further investigating these results, we found out that both frameworks handle queries differently. In particular, when fetching an entity that makes references to other ones, HIBERNATE does not check whether referenced entities are already present in the first-level cache described in Section II. Instead, it directly uses join operations to fetch associated data from multiple tables. Listing 7 illustrates such operations, which instantiate all entities corresponding to transitive relationships-i.e., the full graph of relationships. Consequently, HIBERNATE might redundantly retrieve data that is already stored in the Persistence Context. ECLIPSELINK behaves differently as it (i) sends queries only when referenced entities are not present in the cache, and (ii) employs independent queries to fetch individual foreign entities rather than using joins, as shown by Listing 8, hence achieving better performances at scale. Using the JOIN clause for data retrieval from multiple tables can increase transaction complexity and disrupt concurrent operations in the DBMS. This increased workload leads to longer execution times, heightened latency (2.5 times more at rate 1, 000 and up to 5 times more at rate 10, 000, as depicted in Figure 3), and more data locks due to the serialization isolation level, resulting in increased transaction failures. To mitigate the negative effects of transaction failures, the built-in workload injector mechanism reduces the number of transactions to be processed, finally resulting in a reduction in goodput and power consumption as well.

RQ2:

We analyzed the goodput of different strategies under varying workloads, in view of their power usage. Our empirical results indicate that ECLIPSELINK consumes less energy than HIBERNATE to complete the same workload for rates up to 1, 000. At higher rates, HIBERNATE exhibits a better goodput for all strategies except for the Entity one, which experiences a significant drop in performances. This is due to generated queries that play a critical role in understanding the performances of ORM frameworks. Customized SQL queries tailored to specific tasks exhibit the best performance, but this approach does not benefit from the abstraction layers provided by ORM frameworks. Overall, ECLIPSELINK stands out for generating well-performing queries.

C. RQ3: Does the configuration of an ORM framework influence its energy consumption?

To answer our third research question, we introduced modifications to the default ORM configuration used in previous experiments to assess whether these changes may improve energy efficiency. In particular, we focused on 3 different features of different scopes, namely lazy fetching, batch processing, and partial update. We chose to apply these changes to the Entity strategy for two main reasons. First, it offers the highest level of abstraction from a Java perspective. Second, we aimed to investigate whether these changes could address some of the HIBERNATE performance issues highlighted by RQ2. While the first feature is a JPA feature that can be enabled seamlessly on ECLIPSELINK and HIBERNATE, the remaining two demand vendor-specific adjustments.

a) Lazy Fetching: ECLIPSELINK employs two distinct lazy variants to enable lazy fetching. Despite the compliance of the JPA's lazy annotation, bytecode instrumentation (also known as weaving) is required for actual lazy data retrieval. This instrumentation can be achieved dynamically by using a javaagent property when launching the program, or statically by enhancing the compiled classes through static class rewriting. In contrast, HIBERNATE enables lazy loading by default, as it relies on entity proxies. When a lazily-fetched attribute is accessed, HIBERNATE retrieves all the fields of an entity to transition from a proxy instance to a concrete entity. Additionally, HIBERNATE uses bytecode instrumentation to inject vendor-specific features, allowing fetching of a single lazy field, rather than loading all the fields, as is implemented by the proxy-based approach.

b) Batch Processing: One limitation of JPA is the absence of a defined batch processing mechanism, especially when working with entities. The specification does not provide a standard approach to consolidate multiple update queries into a single operation treated as a batch. For instance, if one needs to update 100 records, 100 individual queries would have to be executed without batch processing, instead of one single batch operation grouping all of these updates. Fortunately, both ECLIPSELINK and HIBERNATE offer support to address this gap, enabling a fair comparison between them. c) Partial Update: This setting is a specific feature of HIBERNATE. By default, when the framework generates an update query, it sends an update for all fields, even if there are no changes in multiple fields. We considered enabling this feature in HIBERNATE to align with the default behavior of ECLIPSELINK.

For each ORM framework, we thus derived configurations activating each 7 of the 3 features independently or in combination, resulting in 14 different configurations. For each configuration, we measured its power usage and goodput metrics and computed an overall efficiency score P conf , such that P conf = power goodput , which captures the energy consumption per successful transaction. configurations, computed as |P conf1 -P conf2 |. The closer to 0 the value is, the more similar P conf is exhibited by the two compared configurations, hence reporting a little influence of the differing features. For instance, ECLIPSELINK default configuration (EL) P conf is very comparable to the HIBERNATE configuration with all 3 lazy fetching, batch processing and partial update features activated (Hi-la-ba-up), as it exhibits a distance of 0.0087. Activating such features thus proves particularly efficient to address the performance drop observed for HIBERNATE in RQ2 and highlighted in Figure 2(d).

In addition to individual comparisons, Figure 4 also outlines group tendencies-i.e., clusters of configurations with similar P conf values-as a dendogram. To yield these clusters, the data underwent hierarchical clustering using the single linkage method, which relies on the nearest neighbor criterion. The effectiveness of this clustering process was evaluated using the cophenetic correlation coefficient. The resulting high value of 0.96 indicates a robust preservation of the pairwise distances between the original data points. These clusters are organized into a tree-based structure defined on top of the table. For instance, the left branch of the root node relates to 4 columns gathering 4 HIBERNATE variants, while the right branch groups the other 10 ORM variants together. More precisely, the left cluster corresponds to all HIBERNATE configurations where the lazy fetching feature is not activated.

The heatmap shows that these 4 HIBERNATE variants are the worst-performing ones. When compared with all ECLIPSELINK variants (the 6 central rows), they exhibit the longest distances, ranging from 0.0522 (HIBERNATE with batch processing and partial update [Hi-ba-up] vs. e.g., ECLIPSELINK default configuration RQ3: To answer RQ3, we studied the impact of 3 different features of different scopes, namely lazy fetching, batch processing, and partial update. Changing the configuration of an ORM framework can influence its power consumption and performance. Specifically, enabling the lazy fetching feature for HIBERNATE appears to be the most impactful factor. Additional performance and energy gains can be obtained by activating batch processing and partial update features.

V. DISCUSSION AND THREATS TO VALIDITY Discussion. Our empirical study assessed ECLIPSELINK as the ORM reporting the best-performing queries and being the most energy-efficient, but still less energy-efficient than the plain JDBC approach. From a software engineering perspective, transitioning from an ORM approach to the plain JDBC one involves multiple considerations. While this shift may yield performance advantages, it could also result in extended development time or raise challenges in maintenance. Practitioners must carefully weigh the trade-offs between the convenience offered by an ORM and the potential performance and energy benefits associated with plain JDBC. As shown in Section IV, some JPQL or Entity configurations may offer a good trade-off to reduce energy consumption while improving maintenance.

Threats to validity. Some factors must be considered as they could affect the validity of our study.

Internal factors. Our primary objective was to understand if the varying degrees of abstraction maintained comparable performance when considering the specific features and optimizations associated with each ORM. We conducted our experiments with the PostgreSQL 14.5 release DBMS. Since ORM frameworks generate queries tailored to the SQL dialect expected by the target DBMS, results may differ when reproducing our experiments on another release of PostgreSQL or a whole different DBMS, such as MySQL. Regarding such assessments, the performances of the database itself could also be considered, as well as query-level analysis considering other operations (e.g., table joins, write, etc.). Other potential threats to the validity of this study lie in the choice of the JRE [START_REF] Ournani | Evaluating the impact of java virtual machines on energy consumption[END_REF], and its potential effects on a 'warm-up' phase [START_REF] Barrett | Virtual machine warmup blows hot and cold[END_REF], as well as the stochastic nature of the benchmark that may prevent from reproducing exact measurements.

External factors. To assess the energy consumption of ORM approaches, we adopted the well-known TPC-C, widely used to assess DBMS capabilities of online transaction processing systems through an intensive workload. However, ORM approaches may exhibit different performances when assessed against other benchmarks. While we could consider additional TPC benchmarks, like TPC-H, 8 ORM frameworks may, unfortunately, fail to deliver native support for some of the implemented transactions, in particular when it comes to embedded queries. Since the most popular and widely-used ORM frameworks, only ECLIPSELINK and HIBERNATE were considered in this study. Results may differ when assessing other ORM frameworks, such as EBEAN, 9 or non-ORM data access technologies, such as JDBI,10 JOOQ11 or MYBATIS. 12

VI. RELATED WORK

In this section, we review and discuss the state of the art related to energy efficiency and performance in the Java ecosystem and, then, we focus on data access strategies-related approaches.

Java Ecosystem. In the past few years, many studies have contributed valuable insights to the field of software energy efficiency, especially in the Java ecosystem. For example, Kumar et al. [START_REF] Kumar | Energy consumption in java: An early experience[END_REF] conducted a comprehensive evaluation of Java programming constructs, including data types, operators, and control statements, to understand their energy efficiency implications. Other efforts delved into the energetic profiles of Java Collections and I/O libraries [START_REF] Hasan | Energy profiles of java collections classes[END_REF], [START_REF] Pereira | The influence of the java collection framework on overall energy consumption[END_REF], highlighting the need to select the proper implementation. Pinto et al. [START_REF] Pinto | A comprehensive study on the energy efficiency of java's thread-safe collections[END_REF] examined the thread-safe Java collections and found significant variations in the energy footprint of analogous operations among various implementations of the same Java collection. Building upon this work, Oliveira et al. proposed a tool [START_REF] Oliveira | Recommending energy-efficient java collections[END_REF] designed to assist developers in selecting energy-efficient collection implementations to build more sustainable software.

Other research efforts have shown that minor adjustments can yield significant energy savings, such as Rocha et al. [START_REF] Rocha | Comprehending Energy Behaviors of Java I/O APIs[END_REF] who classified the energy consumption of 22 file read and write methods in Java native I/O classes, showing that small changes may improve the energy consumption by 2. Building upon this work, Ournani et al. [START_REF] Ournani | Evaluating The Energy Consumption of Java I/O APIs[END_REF] assessed additional native and third-party I/O libraries.

Another research line focused on programming practices, especially on design patterns aiming at enhancing productivity and maintainability. For instance, Nourredine et al. [START_REF] Noureddine | Optimising energy consumption of design patterns[END_REF] studied the energy consumption of Observer and Decorator pattern implementations, while Connolly Bree et al. [START_REF] Bree | Energy efficiency of the visitor pattern: contrasting java and c++ implementations[END_REF] investigated the Visitor one. Ournani et al. [START_REF] Ournani | Tales from the code #1: The effective impact of code refactorings on software energy consumption[END_REF], [START_REF]Tales from the code #2: A detailed assessment of code refactoring's impact on energy consumption[END_REF] assessed that structural design pattern had no negative impact on the energy consumption of Java-based software.

While all these approaches provide valuable insights into software performance and energy consumption, none of them explored the concepts assessed in this paper within the realm of Java.

ORM Ecosystem. Procaccianti et al. also studied the energy efficiency of ORM frameworks [START_REF] Procaccianti | Energy efficiency of orm approaches: An empirical evaluation[END_REF]. Contrary to our study, they did not address JPA implementations, but rather compared raw SQL queries with PROPEL (a PHP ORM) and TINY QUERIES. Also, they did not investigate the potential improvement brought by changing ORM configurations. Verdecchia et al. [START_REF] Verdecchia | Empirical evaluation of the energy impact of refactoring code smells[END_REF] conducted a study on the performance and energy implications of refactoring code smells in Java ORM-based applications. The research revealed that addressing code smells through refactoring not only enhanced energy efficiency but also potentially improved the performance of the system. Surprisingly, when all code smells were refactored, they observed a performance overhead of approximately 6.8%, while the energy consumption was reduced by 10.7%. This finding underscores the complex relationship between performance and energy consumption, highlighting that these two aspects do not always correlate directly with each other.

Chen et al. investigated various performance issues related to software developed using ORM frameworks. Specifically, they conducted two distinct studies on the detection of performance anti-patterns related to JPA and the performance impact of redundant data access. In [START_REF] Chen | Detecting performance anti-patterns for applications developed using object-relational mapping[END_REF], they review code patterns that can lead to performance issues. They propose a framework to assist developers in automatically flagging performance anti-patterns in the source code of their ORM framework. In addition, they focus in [START_REF]Finding and evaluating the performance impact of redundant data access for applications that are developed using object-relational mapping frameworks[END_REF] on the redundant data problems between the needed data in the code and the SQL-requested data, problems are usually caused by requesting/updating too much data than needed.

While both of these approaches offer valuable insights into the performance of ORM strategies, they do not take into account the power usage dimension. In contrast, our study provides a comprehensive analysis that includes both performance and power usage, with a particular focus on understanding performance in the context of its impact on energy consumption.

VII. CONCLUSION

In this paper, we investigated the influence of ORM configurations on both performance and energy consumption and the potential trade-offs regarding these two metrics. Using the widely accepted TPC-C benchmark, we assessed the energy overhead introduced by ORM frameworks, compared to plain JDBC. Across varying workloads, we identified ECLIPSELINK as the most energy-efficient ORM framework. This framework exhibits an 11% overhead in CPU usage compared to the plain JDBC approach at a high transaction rate, an overhead nonetheless mitigated by the benefits ORM frameworks provide, particularly in terms of data consistency between the middleware and the database. We also showed that energy efficiency might come at the price of performance and that developers may face potential trade-offs when configuring their ORM framework. In particular, we showed that HIBERNATE can be fine-tuned to get closer to ECLIPSELINK performances, providing insights into the varying capabilities of ORM tools when it comes to energy-aware data access.

In future work, we plan to study the energy efficiency of web frameworks in the context of cloud-native applications. Such frameworks provide a layered way to build web applications, and we aim to assess the energy efficiency of their various layers. For instance, we consider evaluating presentation layer libraries to provide additional guidelines for Java developers who aim to minimize the energy footprint of their web applications without sacrificing performance, aligning software engineering practices with environmental sustainability goals.

ACKNOWLEDGMENT

This work received support from the French government through the Agence Nationale de la Recherche (ANR) under the France 2030 program, including partial funding from the CARECLOUD (ANR-23-PECL-0003), DISTILLER (ANR-21-CE25-0022), and KOALA (ANR-19-CE25-0003-01) projects. Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations. 13

  EntityManager em = // ... Customer c = em.find(Customer.class,id); String email = c.getEmail(); Listing 6. Entity access strategy

Fig. 2 .

 2 Fig. 2. Goodput (Tx/sec, right-Y axis) and global power usage (Watts, left-Y axis) median variations across varying ORM strategies.

-

  -Some columns have been omitted for clarity SELECT w_id, w_tax, ... FROM warehouse WHERE (w_id = ?) SELECT d_id, d_next_o_id, d_tax, ... FROM district WHERE ((d_id = ?) AND (d_w_id = ?)) FOR UPDATE SELECT c_id, c_discount, c_last, c_credit, ... FROM customer WHERE (((c_id = ?) AND (c_w_id = ?)) AND (c_d_id = ?)) Listing 8. Transaction from ECLIPSELINK's managed entity strategy

Fig. 3 .

 3 Fig. 3. Distribution of transactions median latency across measurements for the Entity strategy.

  [EL]) to 0.070 (HI-BERNATE with partial update [Hi-up] vs. ECLIPSELINK with dynamic lazy fetching and batch processing [EL-la-d-ba]). Such distance values highlight the best and worst performing configurations-i.e., [EL-la-d-ba] and [Hi-up], respectively. Within the HIBERNATE configurations (bottom-right cluster), the best-performing one is the [Hi-la-up]. This variant with lazy fetching and partial update shows the longest distance of 0.527 from [Hi-up] and the shortest distance of 0.0173 from [EL-la-d-ba].

  As introduced previously, managed entities are objects mapped to database records, which are actively monitored by the ORM framework to ensure that any changes are automatically reflected in the underlying database (cf. Listing 6).

	EntityManager em = // ...
	String email = em.createNamedQuery(
	"Customer.getEmailJPQL", String.class
	).setParameter(1,id).getSingleResult();
	Listing 4. JPQL access strategy
	3) Criteria API Strategy: This strategy relies on the JPA
	Criteria API, which enables developers to construct queries
	using a set of Java objects and methods calls, rather than
	composing queries as plain text like raw SQL or JPQL (cf.
	Listing 5).
	EntityManager em = // ...
	CriteriaBuilder cb = em.getCriteriaBuilder();
	CriteriaQuery<String> query =
	cb.createQuery(String.class);
	Root<Customer> root = query.from(Customer.class);
	ParameterExpression<Integer> idParameter =
	cb.parameter(Integer.class);
	Predicate predicate = cb.equal(idParameter,
	root.get(Customer_.id));
	String email = em.createQuery(query
	.select(root.get(Customer_.email))
	.where(predicate)
	).setParameter(idParameter,idValue)
	.getSingleResult();
	Listing 5. Criteria access strategy
	4) Managed Entity Strategy:

https://zenodo.org/doi/10.5281/zenodo.10061193

https://www.tpc.org/tpcc/

https://db.cs.cmu.edu/projects/benchbase/

When applicable: the partial update feature cannot be activated on ECLIPSELINK as it is a default one.

https://www.tpc.org/tpch/

https://ebean.io/

https://jdbi.org/

https://www.jooq.org/

https://mybatis.org/mybatis-3/

See https://www.grid5000.fr

Hi-up

Then, we compared each pair of configurations in terms of their respective P conf . Figure 4 presents the result of such pairwise comparison at a rate of 10, 000. Each cell defines the euclidean distance between the P conf of the two associated


Alexandre Pham

Maria Potop-Butucaru

Sébastien Tixeuil

Serge Fdida

Data Poisoning Attacks in Gossip Learning

Keywords: Gossip learning, Decentralized federated learning, Poisoning attacks, Methodology

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Machine learning algorithms use statistical methods to make predictions or classifications, and to discover patterns and insights in data. Typically, a ML approach consists in collecting data, selecting a model, and tuning model parameters based on collected data (in the model training phase) before actually using the trained model.

In traditional systems relying on ML, a central entity manages both the ML model and the data collected for training the model. Such data centralization is problematic with regard to risk and responsibilities [START_REF] Mcmahan | Communication-Efficient Learning of Deep Networks from Decentralized Data[END_REF] when the data used to train the model is sensitive and should be kept private, or simply to obey regulations.

In 2016, Google introduced Federated Learning (FL) [START_REF] Mcmahan | Communication-Efficient Learning of Deep Networks from Decentralized Data[END_REF] as a solution to privacy-wise Machine Learning. In this framework, the central entity only manages a global model and coordinates the training across local devices (e.g. smartphone, laptop etc.), which use their own data (that they do not share). Then, devices send their newly adjusted local model's parameters to the central entity that aggregates it with other local models to create a new global model, and this process repeats. This method enables private collaborative learning. However, despite its reduced role, the existence of a central server yields a signle point of failure, and an obvious attack target, as summarized by Lui et al. [START_REF] Liu | Threats, attacks and defenses to federated learning: Issues, taxonomy and perspectives[END_REF], and by Xia et al. [START_REF] Xia | Poisoning Attacks in Federated Learning: A Survey[END_REF].

Decentralized Federated Learning (DFL) aims to do FL without relying on a central server [START_REF] Tomás | Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challenges[END_REF], using P2P or Gossip Communications. The former relies on an existing architecture to operate, while the latter assumes direct communications in a neighborhood. In DFL, FL based attacks and defenses have also been studied. For example, Bernstein et al. [START_REF] Bernstein | signSGD: Compressed Optimisation for Non-Convex Problems[END_REF][START_REF] Bernstein | signSGD with Majority Vote is Communication Efficient and Fault Tolerant[END_REF] with SignSGD in the FL context, where they firstly present and study their work as a compression mechanism, and then study it as a defense mechanism. Later, Qu et al. adapted SignSGD in the DFL context [START_REF] Qu | FL-SEC: Privacy-Preserving Decentralized Federated Learning Using SignSGD for the Internet of Artificially Intelligent Things[END_REF]. In the following, we focus on Gossip Learning (GL), which recently was instrumental in many applications such as building a recommendation system [START_REF] Belal | PEPPER: Empowering User-Centric Recommender Systems over Gossip Learning[END_REF], or improving Channel State Information feedback performance [START_REF] Guo | User-Centric Online Gossip Training for Autoencoder-Based CSI Feedback[END_REF].

Ormándi et al. [START_REF] Ormándi | Gossip learning with linear models on fully distributed data[END_REF] is one of the eldest traces of GL. Each node periodically sends their model to a neighbor. When a node receives a model, the node merges it with its current model and considers the result as its new model. Later, Hegedűs et al. [START_REF] Hegedűs | Decentralized learning works: An empirical comparison of gossip learning and federated learning[END_REF] improved the work by Ormandi et al [START_REF] Ormándi | Gossip learning with linear models on fully distributed data[END_REF] by introducing two useful mechanisms in communication restrained networks. After, Danner et al. [START_REF] Danner | Improving Gossip Learning via Limited Model Merging[END_REF] improved the merging process used by Hegedűs et al. [START_REF] Hegedűs | Decentralized learning works: An empirical comparison of gossip learning and federated learning[END_REF]. It should be noted that none of the aforementioned papers address attack or defense in this context.

In this work, we assess the impact of attackers (also called Byzantine or malicious) nodes that try to poison the global model build through a GL algorithm. Our benchmark GL algorithm was proposed by Hegedűs et al. [START_REF] Hegedűs | Decentralized learning works: An empirical comparison of gossip learning and federated learning[END_REF] as best-in-class in this line of research. Close to our work, Giaretta and Girdzijauskas [START_REF] Giaretta | Gossip Learning: Off the Beaten Path[END_REF] studied the applicability of GL, highlighting problems and providing fixes when data distribution is correlated to either degree distribution or communication speed. However, they do not consider Byzantine nodes in their work.

Our contributions. Our contribution is threefold. First, we propose a methodology to assess the effects of a poisoning attack in a system that executes a gossip learning algorithm. Second, in order to evaluate our methodology, we implemented an extension of the popular gossipy1 simulator. Our extensions are publicly available (with execution details) 2 , and implement a poison injector in gossipy, which allows to assess the performances of both clean and corrupted dataset simultaneously. Finally, we apply our methodology on the state of the art gossip learning algorithm by Hegedűs et al [START_REF] Hegedűs | Decentralized learning works: An empirical comparison of gossip learning and federated learning[END_REF]. Our findings show that the resilience of this algorithm to poisoning attacks depends on several factors such as topology, Byzantine nodes distribution, and churn. Our analysis pave the way to efficiently design countermeasures against poisoning attacks. We organize this work as follows. Section 2 briefly describes our case study. In Section 3 we describe our methodology. We present our results based on this methodology in Section 4. We discuss and conclude with Section 5.

Case study: State-of-the-art Gossip Learning

Gossip Learning is a way to do Federated Learning in decentralized systems without a central coordinator via gossip exchanges of parameters or updates, directly between nodes. Recently, Hegedűs et al [START_REF] Hegedűs | Decentralized learning works: An empirical comparison of gossip learning and federated learning[END_REF] proposed a GL algorithm that outperforms FL [START_REF] Mcmahan | Communication-Efficient Learning of Deep Networks from Decentralized Data[END_REF] with respect to the global performances (in the ML perspective) when taking into account communication resources. Their algorithm, Partitioned Token Gossip Learning Algorithm (PTGLA) uses two interesting mechanisms for communicationrestrained networks such as IoTs. The first one is a compression mechanism to decrease message size, by not sending all parameters during each exchange, but part of it called Partition and, we denote S the total number of partitions that the system is using, a high S implies smaller messages. The second mechanism enables us to control the flow of communications, thanks to Tokens, by achieving a balance between sending messages proactively, i.e. periodically, which may lead to communication flooding and sending messages reactively, i.e. after an event e.g. a local update or a message received, which may lead to starvation (no message circulates in the network).

In the sequel, we study the resilience of this algorithm to various poisoning attacks.

Methodology

In this section, we present the various elements of our methodology: the choice of the topologies, Dataset and ML algorithm, the attack, the churn settings, and the metrics used to assess the impact of the attacks. Topologies for GL infrastructure. We investigate the following topologies with n nodes:

• 20 fan-out network, as originally used by Hegedűs et al [START_REF] Hegedűs | Decentralized learning works: An empirical comparison of gossip learning and federated learning[END_REF]; • random 20-regular with bidirectional links and same number of links for every node; • Watts-Strogatz (k = 20, p = 0.5) for its small-world property [START_REF] Watts | Collective dynamics of 'small-world' networks[END_REF]; • Erdős-Rényi with p = 2 log(n) n for its balanced property [START_REF] Erdős | On random graphs I[END_REF]; • Zipf law graphs, where we require that each node has at least one neighbor.

For the last two topologies, we restrict our simulations to instances where the network is connected. Dataset for ML model training. In this work, we use the MNIST handwritten digit database [START_REF] Lecun | MNIST handwritten digit database[END_REF] as our dataset. It is made of 2 sets: a training set and a test set that we denote D training and D test , of cardinal 60000 and 10000 respectively 3 . An instance of this dataset (either from the training or test) is a couple (x, y) ∈ IR 784 × 0, 9 , x represents a 28 × 28 pixel images with the number y written on it. Each node k will have as a local training set D k training of cardinal 250, which is drawn from D training in an i.i.d. fashion 4 as it is the best case scenario for GL as pointed out in Giaretta and Girdzijauskas [START_REF] Giaretta | Gossip Learning: Off the Beaten Path[END_REF] and such as

n ∩ i=1 D i training = / 0.
ML model. Similarly to Hegedűs et al. [START_REF] Hegedűs | Decentralized learning works: An empirical comparison of gossip learning and federated learning[END_REF], each node k trains a (Multinomial) Logistic Regression model to correctly classify images from the dataset described previously. Hegedűs et al. [START_REF] Hegedűs | Decentralized learning works: An empirical comparison of gossip learning and federated learning[END_REF] study thoroughly the choice of hyperparameters η (learning rate) and λ (L2 regularization coefficient). In our work, we fix η = 1 and λ = 0. We choose those values as they yield best performance on the churn-free Erdős-Rényi case, but, the choice of those 2 values is not trivial and should be studied for all topologies as done by Hegedűs et al [START_REF] Hegedűs | Decentralized learning works: An empirical comparison of gossip learning and federated learning[END_REF]. Byzantine attacks. As done by Wu et al. [START_REF] Wu | Mitigating Backdoor Attacks in Federated Learning[END_REF] in the FL setting, Byzantine nodes insert a pattern on samples that they also mislabel. In this study, Byzantine nodes insert a 9-pixel trigger pattern to 20% of the image of their dataset, which they relabel to 0. Their goal is to make honest nodes classify marked images as 0, without decreasing the classification performances on untampered data (i.e. without the trigger pattern). As Byzantine nodes attack the GL system by tampering with their dataset to disturb the network, this is called a data poisoning attack, and as their goal is to introduce a hidden objective, this kind of attack is also called backdoor attack.

To assess the performances and the impact of the attack on benign nodes, we draw 2000 images from the test set that we divide into two sets of equal size. The first half will be used to assess the performances on the usual classification task. For the remaining half, we remove all (x, y) where y = 0, and apply the same modifications on x that Byzantine nodes do on the remaining samples. We call those two sets test and backdoor set respectively from now and show samples from both in Figure 1.

In order to choose the Byzantine nodes5 , we will use two strategies: classical and random. In the classical strategy, we select nodes with the highest degree first. In the random strategy, we select nodes by randomly sampling without replacement. Churn. Churn refers to the fact that in a network, devices can join and leave freely. Hegedűs et al [START_REF] Hegedűs | Decentralized learning works: An empirical comparison of gossip learning and federated learning[END_REF] used smartphone traces to get a realistic churn scenario (where nodes can disconnect and reconnect). Based on their results, to get close to their churn scenario, nodes have a 20% chance of being online at each round. Metrics. We are interested here in the average accuracy of honest nodes on the test and backdoor sets. Accuracy is defined as the number of correct predictions over the total number of predictions, using the dataset as a reference. For the test set, the higher, the better: (honest) nodes should perform well on data that has not been marked with the pattern and for the backdoor set, it is the opposite: nodes should perform badly on (maliciously) marked images.

Simulation results

To implement the study based on the methodology defined in the previous section, we develop an extension of the gossipy simulator that makes use of the PyTorch [START_REF] Paszke | PyTorch: An Imperative Style, High-Performance Deep Learning Library[END_REF] library for its ML part. The gossipy simulator enables us to test multiple GL algorithm (PTGLA is already implemented 6 ) in a unified manner. However, gossipy has two main limitations: it does not take into account the possible existence of Byzantine alongside honest nodes, and it does not allow assessing the performances of both a clean and a corrupted dataset simultaneously. By contrast, our extensions, available (with execution details) on Gitlab7 address those shortcomings. Simulations were done using Python 3.9.13 or 3.9.15, using CPU only, on Intel Xeon E5-2650v3, and Intel Xeon Gold 6330 machines respectively. We define n ∈ {100, 150} as the number of nodes and f as the number of Byzantine nodes in a simulation (in the worst case, f = 0.3n). Each curve shown here is an average over at least 10 random runs. In the following, we present two scenarios, churn and churn-free, starting with the churn-free scenario, i.e. when nodes are always online. Here, Byzantine nodes want honest nodes to classify clean inputs (shown in Section 3) with high accuracy, while also classifying tampered inputs (shown in Section 3) the way Byzantine nodes want, i.e. tampered inputs are classified as 0. Churn-free scenario. In Figure 2, we represent performances across the 2 sets defined in Section 3 (test and backdoor sets) when n = 100 (resp. n = 150) with f = 30 (resp. f = 45) Byzantine nodes placed with the random strategy. We use Byzantinefree simulations as baselines, selecting the number of partitions S that yields the best performances on the test set. On this set (left column of Figure 2), we can see that S = 1 and sometimes S = 4 (which are the smallest values studied here for S) are special cases across all 4 topologies shown in Figure 2, as honest nodes perform poorly on the usual classification task (compared to the baseline, which is not what Byzantine nodes want), and is very sensitive to the attack as seen in the right column of Figure 2. This can be explained by the fact that if a Byzantine node directly sends (proactively or reactively) a partition to an honest node, as honest nodes are always online, they always receive and process the partition, and may send it reactively to another node, speeding up the spread of malicious partitions. When Byzantine nodes are placed randomly, if the topology is 20 fan-out, honest nodes are the most sensitive against the attack, while they are more resilient when the topology is Watts-Strogatz.

Besides, on the test set, we observe that when S ̸ = 1 and 4, there are no noticeable differences between the choice of S, as honest nodes perform very closely to the baseline. This can be explained by the i.i.d. data distribution: honest nodes, having the same 'learning material', learn quickly and correctly to classify unaltered inputs. On the backdoor set (right column of Figure 2), still excluding S = 1 and 4, we see that the accuracy of honest nodes on the backdoor set is constrained between 0.1 and 0.2, and that, increasing S also increase the resiliency of honest nodes against this particular attack. This can be explained by the fact that parameters that interact with the trigger pattern are too diluted among all partitions as the number of partitions S increase (which is the opposite when S is small), and that Byzantine nodes follow the algorithm (except when training). Hence, corrupted parameters (that interacts with the trigger pattern) are less likely to be updated by honest nodes. However, Byzantine nodes successfully introduce (albeit not as high as they want) the unwanted behavior, without affecting drastically the normal classification task, as we compare the results of the different result of S studied and the baseline on the right column of Figure 2: Byzantine nodes induce from 10 to 20 times more misclassification compared to the baseline. Overall, for the 4 topologies studied, namely Erdős-Rényi, 20 fan-out, 20 random-regular and Watts-Strogatz, when Byzantine nodes are placed randomly, against this particular attack, honest nodes are more resilient when the system is using a (very) high number of partitions S.

In Figure 3, we compare the classical and random strategy for Byzantine nodes, in systems that have hubs or where the degree distribution is skewed. Usually, for S fixed, the classical strategy is clearly more detrimental for honest nodes than the random strategy. On the left column of Figure 3, we can see that the accuracy of honest nodes on the test set drop drastically when Byzantines nodes are placed classically compared to the random strategy. In average, we observe a difference of 40% and 38% on the accuracy for the test set, for Watts-Strogatz and Zipf-based topologies respectively, which is not what Byzantine nodes want for honest nodes, that do classify the way Byzantine nodes want in the backdoor set (right column of Figure 3): in average, we observe a difference of 73% and 48% on the accuracy for the backdoor set between the 2 strategies for Watts-Strogatz and Zipf-based topologies respectively.

Considering the case where Byzantine nodes are placed randomly, Watts-Strogatz, and the other three topologies studied previously, are more suitable for honest nodes compared to the Zipf-based topology as they are more resilient against this attack.

Interestingly, for Zipf-based topology, when Byzantine nodes are placed randomly, excluding S = 1, on the right column, it sometimes seems detrimental for honest nodes that the system use a high number of partitions S compared to the other 4 topologies already studied previously. Churn scenario. Here, we study the case where nodes can disconnect and reconnect as described in Section 3 and have a 20% chance to be online.

In Figure 4, we fix the number of partitions to S = 8 and number of nodes to n = 150, and study the effect of Byzantine placement strategy and numbers up to f = 45 when nodes degree follow a Zipf law distribution. We can see, on the right column of Figure 4, that Byzantine nodes, when placed with the classical and random with f = 5, for the random and classical strategy, it is almost as if there is no attack (case f = 0). While the classical strategy is more harmful to the network for f fixed, considering the backdoor set, we note the fact that f = 40 Byzantine nodes selected with the random strategy is about 5% more harmful than f = 20 Byzantine nodes selected with the classical strategy.

In Figure 5, we again study n = 100 (resp. n = 150) and f = 30 (resp. f = 45), with the random strategy, similarly to Figure 2. We see that honest nodes perform as well as in the churn-free scenario on the test set (left column of Figure 5), still due to the fact that data is distributed in an i.i.d. fashion. In this scenario, honest nodes are also slower compared to the churn-free scenario, which is also expected as they are offline most of the time, hence, they are less learning exchanges between nodes for a given number of rounds compared to the churn-free scenario.

Interestingly, while S = 1 is again a special case here, unlike the churn-free scenario, honest nodes classify badly clean inputs at the beginning, but they converge to a much better solution at the end, as they successfully manage to classify clean inputs with a better success rate and are less affected by the attack compared to the churn-free scenario. This is due to the fact that all nodes are most of the time offline, Byzantine nodes included, hence, the attack is not as powerful as in the churn-free scenario, but we notice, that the results are close with the other value of S studied. In the journal version of this work, we are planning to study the churn scenario with the assumption that Byzantines nodes are always online. Overall, we notice that values presented in the churn-free scenario (Figure 2) and the churn scenario (Figure 5) are relatively close. This might be explained by the fact that the churn is probabilistic, combining with the fact that the choice of the i th to send is random, hence, over time, the churn-free scenario, looks very similar to the churn-free case (except when S = 1).

Conclusion

In this paper, we proposed a methodology to study the resilience of gossip learning in the presence of poisoning attacks. As case study we target the compression mechanism of GL algorithm proposed by Hegedűs et al. [START_REF] Hegedűs | Decentralized learning works: An empirical comparison of gossip learning and federated learning[END_REF]. We investigate its resilience in a broad range of topologies (e.g. Erdős-Rényi, 20 fan-out, random 20-regular, Watts-Strogatz and Zipf-based graph) in both churn-free and churn scenarios. Our findings show that the communication optimizations and the choice of the underly-

AccuracyTest S =1 S =4 S =8 S =12 S =20 S =28 S =30 Baseline S =1 S =4 S =8 S =12 S =20 S =28 S =30
Baseline Accuracy on the test set

AccuracyBackdoor S =1 S =4 S =8 S =12 S =20 S =28 S =30 Baseline S =1 S =4 S =8 S =12 S =20 S =28 S =30
AccuracyTest S =1 S =4 S =8 S =12 S =20 S =28 S =30 Baseline S =1 S =4 S =8 S =12 S =20 S =28 S =30
Baseline Accuracy on the test set

AccuracyTest S =1 S =4 S =8 S =12 S =20 S =28 S =30 Baseline S =1 S =4 S =8 S =12 S =20 S =28 S =30
Baseline Accuracy on the test set

AccuracyBackdoor S =1 S =4 S =8 S =12 S =20 S =28 S =30 Baseline S =1 S =4 S =8 S =12 S =20 S =28 S =30
Baseline Accuracy on the backdoor set (d) Watts-Strogatz Fig. 2 Accuracy on the test set (left, higher is better) and backdoor set (right, lower is better) for different topologies with n = 100 (gray) and n = 150 (black) with random Byzantine placement strategy in a churn-free system with f = 30 and 45 respectively. 'Baseline' curves represent the results in Byzantine-free simulations, with the best choice of S among values studied here with Byzantine.

AccuracyTest S =1 S =4 S =8 S =12 S =20 S =28 S =30 S =1 S =4 S =8 S =12 S =20 S =28 S =30
Accuracy on the test set

S =1 S =4 S =8 S =12 S =20 S =28 S =30 S =1 S =4 S =8 S =12 S =20 S =28 S =30
Accuracy on the backdoor set (b) Zipf law based degree distribution

S =1 S =4 S =8 S =12 S =20 S =28 S =30 S =1 S =4 S =8 S =12 S =20 S =28 S =30
S =1 S =4 S =8 S =12 S =20 S =28 S =30 S =1 S =4 S =8 S =12 S =20 S =28 S =30
Accuracy on the test set ing topology are not always favorable to the honest nodes. Moreover, the distribution of the Byzantine nodes has also a strong impact on the accuracy metric. Usually, in the churn-free and churn cases, when Byzantine nodes are placed randomly, the use of a low number of partitions (i.e. bigger messages), is usually detrimental for honest nodes. This is not necessarily true if the topology is Zipfbased, in the churn-free scenario.

In the churn-free scenario, when Byzantine nodes are placed using the classical strategy in Watts-Strogatz and Zipf topologies, the attacks exhibit a completely different impact on the network compared to when they are placed randomly. We observe that in average, Byzantine nodes cause a drop of 40% and 38% on the usual classification task and a 73% and 48% increase on the backdoor task respectively.

For honest nodes, when Byzantine nodes are placed randomly, we do not observe noticeable differences between the churn-free and churn scenario (except when nodes use bigger messages, as they are more resilient against the attack in the latter scenario), this might be attributed to the probabilistic churn.

In the future we plan to extend this work to more complex datasets and data distribution, and study other proposals for GL algorithm including some that take into account the possibility of poisoning data and models.

  Fig. 1 Examples of clean and tampered data from the MNIST dataset.

Fig. 3 Fig. 4

 34 Fig. 3 Accuracy on the test set and backdoor set for different topologies with n = 150 and f = 45 with random Byzantine placement strategy (gray) and classical Byzantine placement strategy (black).

Fig. 5

 5 Fig. 5 Accuracy on the test set and backdoor set for different topologies with n = 100 (gray) and 150 (black) with f = 30 and 45 respecively.

https://github.com/makgyver/gossipy/

https://gitlab.lip6.fr/apham/data-poisoning-attacks-in-gossip-learning

This is done to evaluate the model against unseen data, but close to data that were used for adjusting model's parameters. This allows us to see whether the model generalize well.

This means that data is equally distributed among nodes, every node has approximately 25 images of each number.

We borrow the idea behind these strategies from Magnien et al.[START_REF] Magnien | Impact of random failures and attacks on Poisson and power-law random networks[END_REF], where they use these strategies in order to select nodes to be removed from a graph to study the graph connectivity.

https://github.com/makgyver/gossipy/tree/3d655829805fc0dc2f01f5b0862240fca08ffe1c

https://gitlab.lip6.fr/apham/data-poisoning-attacks-in-gossip-learning

Acknowledgements:

The work presented in this document has received funding from the EU Horizon Europe research and innovation Programme under Grant Agreement No. 101070118.