Jean-Luc Bertrand-Krajewski

Francois Clemens-Meyer

Mathieu Lepot

Water level and discharge measurements

come


Yasmine Djebrouni

Isabelly Rocha

Sara Bouchenak

Lydia Y Chen

Pascal Felber

Vania Marangozova-Martin

Valerio Schiavoni

Char- Acterizing

Characterizing Distributed Machine Learning and Deep Learning Workloads

Keywords: Distributed Machine Learning, Distributed Deep Learning, Workload Characterization

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Machine learning (ML) and deep learning (DL) models are widely used in many application domains to analyze real-life datasets and help in the decision making process. Search engines [START_REF] Mccallumzy | Building domain-specific search engines with machine learning techniques[END_REF], recommendation systems [START_REF] Aher | Combination of machine learning algorithms for recommendation of courses in e-learning system based on historical data[END_REF] and medical science for disease diagnoses [START_REF] Kourou | Machine learning applications in cancer prognosis and prediction[END_REF] are some of the many examples of ML applications. As for DL models, they bring an important breakthrough for vision and natural language applications [START_REF] Deng | Deep learning: methods and applications[END_REF]. In order to guarantee a high prediction quality and make ML/DL a viable solution, a massive amount of training data is necessary. However, the size of data makes the training phase impractical on centralized platforms. This has lead to the emergence of distributed machine learning (DML) and distributed deep learning (DDL) where distributed systems are used to leverage advantages such as increased task parallelization or total amount of storage disks bandwidth. To simplify the development of DML/DDL solutions, several DML/DDL libraries have been proposed including MLlib [START_REF] Meng | Mllib: Machine learning in apache spark[END_REF], BigDL [START_REF] Dai | Bigdl: A distributed deep learning framework for big data[END_REF] and TensorFlow [START_REF] Abadi | Tensorflow: A system for large-scale machine learning[END_REF]. As DML/DDL's performance is highly dependent on their workloads, workload characterization is a major issue [START_REF] Calzarossa | Workload characterization: A survey revisited[END_REF]. Workload characterization captures the execution characteristics of workloads and identifies the factors for performance degradation or optimization. In the context of DML/DDL methods, workload characterization can provide a better understanding of resource utilization and guide performance optimization at both the hardware and the software levels. Moreover, it may provide insights into the actual effectiveness of configuration strategies of the underlying distributed computing platform, especially for data scientists and ML/DL users who are not familiar with distributed settings. In this paper, we present our work on DML/DDL workload characterization based on realworld collected traces [START_REF] Djebrouni | Collecting and characterizing distributed machine learning workloads[END_REF], resulting from extensive experiments conducted on two popular libraries, MLlib [START_REF] Meng | Mllib: Machine learning in apache spark[END_REF] and BigDL [START_REF] Dai | Bigdl: A distributed deep learning framework for big data[END_REF], running on a Spark cluster [START_REF] Spark | Apache spark[END_REF]. We present an analysis of the statistical distributions of the DML/DDL workloads, and discuss the common patterns and characteristics that we observe in these workloads. We derive key takeaways, among which the main aspects that characterize DML/DDL workloads, and unexpected behavior of workloads involving ensemble learning methods under configuration tuning. The rest of the paper is organized as follows. In §2 we present the necessary background on DML/DDL systems and workloads, as well as the main aspects considered when tuning those systems. Section §3 presents an overview of the traces we rely on in our analysis. Section §4 presents a detailed characterization of DML/DDL workloads. We survey related work in §5, and conclude in §6 with key takeways derived from our detailed characterization.

Background

DML and DDL Architecture. Figure 1 depicts the infrastructure we use to perform the DML and DDL experiments. This architecture represents the typical deployment scenario, either on premises or outsourced to third-party cloud deployments. The computing architecture consists of a physical cluster, a distributed computing platform (i.e., Spark [START_REF]Spark configuration[END_REF]), and a distributed data storage (i.e., the Hadoop distributed file system HDFS [START_REF] Shvachko | The hadoop distributed file system[END_REF]). The Spark platform is composed of a set of nodes, called workers. During training, an interactive process iteratively executes Spark jobs which define computation steps. Each of these jobs is scheduled and processed on a dedicated executor which is a process on a Spark worker. Jobs are composed of a sequence of stages running several parallel and independent tasks on data partitions. A DML/DDL workload is a tuple consisting of a ML or DL method and a dataset. There are several types of ML methods, such as classification (e.g., naive Bayes [START_REF] Rish | An empirical study of the naive bayes classifier[END_REF], decision trees [START_REF] Quinlan | Induction of decision trees[END_REF]), regression (e.g., linear models [START_REF] Nelder | Generalized linear models[END_REF]) or clustering (e.g., K-means [START_REF] Likas | The global k-means clustering algorithm[END_REF]).

When it comes to DL methods, they mostly consist of different types of artificial neural networks which layers and overall structure best fit the domain of interest (e.g., CNNs [START_REF] Sainath | Deep convolutional neural networks for lvcsr[END_REF], LSTMs [START_REF] Hochreiter | Long short-term memory[END_REF], GRU [START_REF] Zhang | Detecting hate speech on twitter using a convolution-gru based deep neural network[END_REF]). There exist several libraries that provide open source implementations for DL and ML methods on distributed paltforms, for example, MLlib [START_REF] Meng | Mllib: Machine learning in apache spark[END_REF] and BigDL [START_REF] Dai | Bigdl: A distributed deep learning framework for big data[END_REF] are respectively the ML and DL libraries running on top of Spark. Regarding datasets, there exist multiple realworld datasets that are publicly available and widely used in ML and DL research [START_REF]News 20 dataset[END_REF][START_REF] Diaz-Chito | A reduced feature set for driver head pose estimation[END_REF][START_REF] Deng | The mnist database of handwritten digit images for machine learning research [best of the web[END_REF][START_REF] Baldi | Searching for exotic particles in high-energy physics with deep learning[END_REF].

Their structure and content vary and depend on the type of learning we are interested in. For

ComPAS'2021 : Parallélisme/ Architecture/ Système / Temps Réel MILC -Lyon, France, 5-9 juillet 2021 example, News20 is a collection of 20,000 messages [START_REF]News 20 dataset[END_REF], collected from 20 different newsgroups, and DrivFace is a popular dataset of 606 images of car drivers [START_REF] Diaz-Chito | A reduced feature set for driver head pose estimation[END_REF]. Configuration of DML and DDL.There are many configuration parameters which can be tuned in distributed platforms. For instance, Spark [START_REF]Spark configuration[END_REF] and Hadoop [START_REF]Hadoop commands guide[END_REF] have over 200 and 50 configurable platform parameters, respectively. These parameters usually fall in one of the following types: (i) parameters related to memory management, (ii) parameters related to data management and compression, (iii) parameters related to job scheduling and (iv) parameters related to parallel execution.

DML and DDL Workload Traces and Setup

We use DML and DDL traces. The DML traces have been obtained from the execution of 26 DML workloads on Spark under different configurations, as described in [START_REF] Djebrouni | Collecting and characterizing distributed machine learning workloads[END_REF]. We consider 9 popular ML methods from MLlib [START_REF] Meng | Mllib: Machine learning in apache spark[END_REF], the Spark ML library, including methods such as Kmeans, decision trees and linear models. The methods have been applied on three datasets, namely DDF, DGS and DHG, that differ in terms of features and records characteristics. DDF corresponds to the DrivFace dataset [START_REF] Diaz-Chito | A reduced feature set for driver head pose estimation[END_REF] that contains a small number of records (606 images) with a big number of features (6,400 pixels). DGS is the Drift dataset [START_REF] Vergara | Chemical gas sensor drift compensation using classifier ensembles[END_REF] with several thousands of records, each corresponding to 128 chemical sensors measurements (features). Finally, DHG references the Higgs dataset [START_REF] Baldi | Searching for exotic particles in high-energy physics with deep learning[END_REF] which is a collection of 11 millions records of a small number of kinematic measures (features). The DDL traces come from the executions of 9 DDL workloads. We have used 4 popular DDL methods from BigDL [START_REF] Dai | Bigdl: A distributed deep learning framework for big data[END_REF], the Spark DL library, including CNN [START_REF] Sainath | Deep convolutional neural networks for lvcsr[END_REF], GRU [START_REF] Zhang | Detecting hate speech on twitter using a convolution-gru based deep neural network[END_REF], LENET [START_REF] Lecun | Lenet-5, convolutional neural networks[END_REF] and LSTM [START_REF] Greff | Lstm: A search space odyssey[END_REF]. The used datasets are Mnist [START_REF] Deng | The mnist database of handwritten digit images for machine learning research [best of the web[END_REF],

Fashion Mnist [START_REF] Xiao | Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms[END_REF] and News20 [START_REF]News 20 dataset[END_REF]. They respectively contain 70,000 images of written digits, 70,000 images of fashion articles and 20,000 newsgroup messages. The traces contain measures collected from system level (CPU, Memory, Energy, etc.), Spark level (task duration, data size per task, etc.) and application level (training time, inference time and accuracy). In the following, we investigate how the application-level metrics are impacted by configuration tuning. More details about the considered datasets and methods, the collected metrics and the considered Spark parameters can be found in the Appendix. All traces have been collected on a Spark cluster of 4 identical nodes with a quad-socket Intel E3-1275 CPU processor, 8 cores per CPU, 64 GiB. The machines run Spark 2.4.0 as the distributed learning platform, MLlib 2.4.0 as the ML library and BigDL 2.4.0 as the DL library. Each workload execution was replicated 3 times.

Characterization of DML and DDL Workloads

In the following, we investigate the impact of 13 Spark configuration parameters (cf Appendix Table 6) on the training time, the inference throughput (i.e., inference responses per time unit) and the accuracy of our workloads. The parameters of interest cover the main aspects of distributed execution. Regarding parallelization, EXEC_NUM and EXEC_COR control respectively the number of executors running concurrently on the cluster, and the number of cores assigned to each executor. Regarding memory management, EXEC_MEM helps setting the memory used by each executor. Other parameters include SHF_COMPR which determines whether the data resulting from shuffle1 should be compressed, and LOC_WAIT which defines how long Spark should wait before giving up on a task and rescheduling it. More parameters related to serialization and data compression are also considered . We start by investigating whether varying a single platform parameter impacts the performance of the underlying system. We execute several runs for each workload where we individually vary each platform parameter. The variations of training time and inference throughput are shown in Figure 2. We use a box-and-whiskers representation to conveniently show the data distribution through their quartiles, as well as outliers. On the horizontal axis we indicate the different workloads 2 , while on the vertical axis we plot the variations in training time and inference throughputs of all the runs grouped by workload. The variation of a run is computed as the difference between the run's time and the mean execution time of the workload, divided by the mean time. We can observe important training time variations spanning between -54% and +247%. For two workloads, namely DDF-GBT and DGS-GMM, configuration tuning result in significant training speedups (respectively 54% and 43%). Regarding inference throughput, we observe that its variation ranges between -78% and +61%. In most scenarios there are configurations resulting in lower throughputs. These observations support our assumption that both training and inference are strongly affected by the platform parameters' configurations. Individual Parameter Impact. To clarify the impact of each configuration parameter on training time and inference throughput, we use a heatmap plot as shown in Figure 3. For each of the considered parameters (vertical axis) and workloads (horizontal axis), we use a three-color scheme to represent the parameters impact, from highest (over 20%, dark grey) to negligible (less than 10%, white). The impacts of a parameter is equal to the relative variation obtained in performance, i.e., the difference between the lowest and highest performance, when different values of that parameter are considered while the other parameters have fixed values. What we can directly observe from this heatmap is that the two parameters (number of executors EXEC_NUM and number of cores per executor EXEC_COR) that configure the parallelization of the execution, highly affect the large majority of the workloads. To further explore the impact of each parameter on both the training time and inference throughput, we calculate the coefficient of variation of these two measures inside each workload when testing several values for each parameter (cf Appendix Table 6). The coefficient of variation is defined as the ratio of the standard deviation σ to the mean µ, i.e., c v = σ µ . We highlight in the following our key observations. Observation 1 : Most of DML and DDL workloads significantly benefit from parallelization.

This observation comes from figures 4a and 4b which present respectively the coefficients of variation of training time and inference throughput for each workload when varying EXEC_NUM and EXEC_COR. The results report variations beyond 40%. Observation 2 : Training times of large datasets workloads are highly impacted by task re-scheduling. 2 Workload names are tuples composed of the dataset name followed by the learning method name (cf. As shown in Figure 5a, when working with large datasets, tasks may take arbitrarily long time to complete. This is due to task rescheduling or preemption variations, which are impacted by LOC_WAIT parameter. Observation 3 : Training of workloads using ensemble learning methods on datasets with a large number of features are affected by shuffle data size reductions.

We observe that, for ensemble learning methods used on datasets with many features (DDF-RFR, DDF-GBT, DGS-RFR and DGS-GBT, Figure 5b), the training phase triggers frequent data transfer (shuffle) operations. One can use the SHF_COMPR parameter to reduce the amount of data during shuffle and reduce the training time. Observation 4 : Inference throughput of workloads involving datasets with a relatively high number of features is impacted by memory, scheduling and shuffle operations.

The number of features presented by the dataset is highly correlated with the resulting inference throughput. Figure 3b shows that there are several parameters interwined that are impacting the inference throughput of workloads involving datasets with high number of features (DDF and DGS). The concerned parameters are EXEC_MEM, MAX_SIZ_INFLIGHT, STR_MEM, LOC_WAIT and SFL_BUF. Figure 5c illustrates the impact of one of these parameters (EXEC_MEM) on DDF-BLR, DDF-GBT, DGS-KM and DGS-LR. Observation 5 : Model accuracy of workloads involving ensemble learning methods on datasets with a relatively large number of features is surprisingly impacted by parallel computing configurations.

We conclude the analysis by studying the effect of configuration tuning on the R2 score of our regression workloads. R2 is a metric that evaluates the precision of regression methods. Figure 5d presents the coefficient of variation of R2 for DDF-RFR and DDF-GBT. The Figure shows an unexpected behavior of (GBT and RFR) that are ensemble learning methods, when executed on the dataset that has the highest number of features (DDF). R2 for these two workloads surprisingly vary when EXEC_NUM and EXEC_COR are tuned. In fact, both RFR and GBT are based on multiple decision trees. Before splitting a node of a decision tree, and in order to determine : Coefficient of variation for different performance measures the best split segment for features, a typical approach is to sample each partition of the dataset, attempting to reduce data transmission operations. On the other hand, when dealing with many features, splitting usually affects the accuracy of the model, and data partitioning comes with a cost. Consequently, for such datasets and learning methods, the impact of parallelization on model quality should be carefully analyzed, and platform configuration and the model hyperparameters should be jointly configured.

Related Work

While there are numerous studies on the usage and optimisation of Spark, few studies tackle the performance of ML workloads. SparkBench [START_REF] Li | Parallélisme/ Architecture/ Système / Temps Réel MILC -Lyon, France, 5-9 juillet 2021 benchmarking suite for in memory data analytic platform spark[END_REF] is a benchmarking suite in which three ML methods are considered. It characterizes the workloads in terms of data access patterns, job execution time and system resources consumption and briefly explores the variation of one Spark parameter. In [START_REF] Mostafaeipour | Investigating the performance of Hadoop and Spark platforms on machine learning algorithms[END_REF], the authors compare ML performances between Spark and Hadoop [START_REF]Apache Hadoop[END_REF] considering resource consumption and variations of dataset sizes and Spark cluster size. However, the work focuses only on the KNN [START_REF] Cover | Nearest neighbor pattern classification[END_REF] method working on a single dataset. Some characterization works describe their traces and share them with the community. The authors of [START_REF] Wang | Characterizing deep learning training workloads on alibaba-pai[END_REF] trace the execution of DL workloads using TensorFlow. They use metrics on job scheduling, resource allocation, DL computation features and data I/O. However, they focus on DL methods, while we analyze both ML and DL methods. The authors of [START_REF] Jeon | Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads[END_REF] have published two-month traces about resource consumption, job scheduling and workload results. They focus on GPU cluster resource usage and scheduling issues of DL workloads.

Key Takeaways

From our DML/DDL workload characterization, we derive the following key takeaways:

(1) Most of DML and DDL workloads significantly benefit from parallelization. For DML/DDL service operators, we report how platform parameters related to parallel computing, should be systematically considered to efficiently handle model training requests.

(2) Shuffle and scheduling related parameters are important in the tuning process. Shuffle operations usually involve data transfer and memory access. Reducing shuffle data size helps improving the execution times. Tasks scheduling parameters also show an important impact on big datasets where tasks are relatively longer due to the big amounts of processed data.

(3) Model accuracy of workloads involving ensemble learning methods on datasets with relatively large numbers of features is impacted by parallel computing tuning. This is counterintuitive, since model accuracy is usually impacted by model hyperparameters, and not by the underlying distributed computing platform parameters. Thus, improved training times of such workloads thanks to parallel computing can come at the cost of lower model accuracy. This trade-off should be carefully analyzed by data scientists and DML/DDL service operators.

As future work, we intend to explore the impact of the joint configuration of platform parameters and hyper-parameters on DML/DDL workloads execution time and model quality.

Figure 1 :

 1 Figure 1: Infrastructure of a typical distributed machine learning or deep learning environment

ComPAS' 2021 :Figure 2 :

 20212 Figure 2: Performance variability per DML/DDL workload Global Impact of Platform Configurations.We start by investigating whether varying a single platform parameter impacts the performance of the underlying system. We execute several runs for each workload where we individually vary each platform parameter. The variations of training time and inference throughput are shown in Figure2. We use a box-and-whiskers representation to conveniently show the data distribution through their quartiles, as well as outliers. On the horizontal axis we indicate the different workloads 2 , while on the vertical axis we plot the variations in training time and inference throughputs of all the runs grouped by workload. The variation of a run is computed as the difference between the run's time and the mean execution time of the workload, divided by the mean time. We can observe important training time variations spanning between -54% and +247%. For two workloads, namely DDF-GBT and DGS-GMM, configuration tuning result in significant training speedups (respectively 54% and 43%). Regarding inference throughput, we observe that its variation ranges between -78% and +61%. In most scenarios there are configurations resulting in lower throughputs. These observations support our assumption that both training and inference are strongly affected by the platform parameters' configurations. Individual Parameter Impact. To clarify the impact of each configuration parameter on training time and inference throughput, we use a heatmap plot as shown in Figure3. For each of the considered parameters (vertical axis) and workloads (horizontal axis), we use a three-color scheme to represent the parameters impact, from highest (over 20%, dark grey) to negligible (less than 10%, white). The impacts of a parameter is equal to the relative variation obtained in performance, i.e., the difference between the lowest and highest performance, when different values of that parameter are considered while the other parameters have fixed values. What we can directly observe from this heatmap is that the two parameters (number of executors EXEC_NUM and number of cores per executor EXEC_COR) that configure the parallelization of the execution, highly affect the large majority of the workloads. To further explore the impact of each parameter on both the training time and inference throughput, we calculate the coefficient of variation of these two measures inside each workload when testing several values for each parameter (cf Appendix Table6). The coefficient of variation is defined as the ratio of the standard deviation σ to the mean µ, i.e., c v = σ µ . We highlight in the following our key observations. Observation 1 : Most of DML and DDL workloads significantly benefit from parallelization.

ComPAS' 2021 :Figure 3 :

 20213 Figure 3: Platform parameters impact on performance

Figure 4 :DFigure 5

 45 Figure 4: Coefficient of performance variation when varying EXEC_NUM and EXEC_COR

Table 1

 1

,2)

Table 1 :

 1 Datasets

		ComPAS'2021 : Parallélisme/ Architecture/ Système / Temps Réel
						MILC -Lyon, France, 5-9 juillet 2021
	7. Appendix
	Dataset	Description	#Records #Features	Size
		Images sequences of sub-
	DDF (Driveface)	jects while driving in real	606	6 400 19.9 MB
		scenarios [12].
		Measurements from 16
		chemical sensors utilized				Library Category	ML / DL Method
	DGS (Drift) DHG (Higgs)	in a discrimination task of 6 gases at various lev-els of concentrations [31]. A collection of kinematic measures to detect signal processes which produce	13 910 11 000 000	128 40.3 MB 28 7.5 GB	MLlib	Clustering Classification	KM (K-Means) BKM (Bisecting K-Means) GMM (Gaussian Mixture Model) DT (Decision Tree) MLP (Multilayer Perceptron) BLR (Binomial Logistic Re-
		Higgs bosons [6].						gression)
		Messages collected from						LR (Linear Regression)
	DN (News20)	20 different newsgroups	20 000	2	15 MB		Regression	RFR (Random Forest Regres-
	DFM (fashion MNIST)	[2]. 28x28 grayscale images of fashion articles, asso-ciated with labels from 10 classes [33]. Handwritten digit im-	70 000	784	31 MB	BigDL	Classification	sor) GBT (Gradient-Boosted Tree) CNN (Convolutional Neural Network) GRU (Gated Recurrent Unit) LENET5 (Convolutional Neu-ral Network)
	DM (MNIST)	ages for ML research	70 000	784	12 MB			LSTM (Long Short-Term
		[10].						Memory)

Table 2 :

 2 Learning Methods

	Application-level metrics	Description
	Training accuracy	Percentage of predicted values that match the actual values in the train-ing dataset.
	Inference accuracy	Percentage of predicted values that match the actual values in the testing dataset.
	Inference throughput	Number of records inferred by unit of time.
	Silouhette	Evaluation metric for clustering methods. It measures how similar an object is to its own cluster.
	R2 Score	TProportion of the variance in the dependent variable that is predictable from the independent variable(s).
	MAE (Mean Absolute Error)	Average of the differences between original values and predicted values.
	MSE (Mean Square Error)	Average of the square of the difference between the original values and the predicted values.
	RMSE (Root Mean Square Error) The square root of MSE.
	F1 Score	Harmonic mean between precision and recall.

Table 3 :

 3 Application-level Metrics

	Platform-level metrics	Description
	Task duration	Total elapsed time.
	Task deserialization time	Elapsed time spent to deserialize this task.
	Garbage collection time	Total JVM garbage collection time.
	Result Serialization Time	Elapsed time spent serializing the task result.
	Shuffle wait time	Time that tasks spent blocked waiting for shuffle data to be read from remote machines.
	Shuffle read size	Total shuffle bytes read, includes both data read locally and data read from remote executors.
	Shuffle write size	Total shuffle bytes written.

Table 4 :

 4 Platform-level Metrics Amount of bytes read from and written to the network. Disk usage Amount of bytes read from and written to disk.Energy consumptionTrapezoidal integral of power measurements collected per second.

	ComPAS'2021 : Parallélisme/ Architecture/ Système / Temps Réel
	MILC -Lyon, France, 5-9 juillet 2021
	Infrastructure-level metrics	Description
	CPU usage	Percentage of used CPU.
	Memory usage	Percentage of memory utilization
	Network traffic

Table 5 :

 5 Infrastructure-level Metrics -From /proc/stat, /proc/meminfo, etc.

	Configuration aspect	Examples of Spark platform pa-rameter	Description	Values (default value in brackets)
	Parallel computing	EXEC_NUM (executor.instances) EXEC_COR (executor.cores)	Number of executors Number of cores per execu-tor	MLlib: 1, 2, 3, 5, (4), 6, 7, 8; BigDL: 1, 2, (4) MLlib: 1, 2, 3, 5, (4), 6, BigDL: 1, 2, 4, (8)
					MLlib: (5 GB), 10 GB, 15 GB,
	Memory management	EXEC_MEM (executor.memory) MAX_SIZ_INF (re-ducer.maxSizeInFlight)	Amount of memory per ex-ecutor Maximum size of map out-puts to fetch simultaneously from reduce tasks	20 GB, 25 GB, 30 GB., BigDL: 1 GB, 4 GB, 8 GB, 16 GB, 24 GB, (32 GB) 12 MB, (48 MB), 72 MB, 128 MB, 256 MB, 512 MB
				Must use off-heap buffers
		PD_BUFS fle.io.preferDirectBufs)	(shuf-	to reduce garbage collection during shuffle and cache	(true), false
				block transfer
		STR_MEM age.memoryFraction)	(stor-	Fraction of Java heap to use for Spark's memory cache	10%, 20%, 40%, (60%), 80%
	Data compression	COMP_CODEC (io.compression.codec) RDD_COMP (rdd.compress)	Codec to compress internal data such as RDD partitions, shuffle outputs, etc. Must to compress serialized RDD partitions	(snappy), lz4 true, (false)
		SHF_SPL_COMP fle.spill.compress)	(shuf-	Must compress data spilled during shuffles	(true), false
				How long to wait to launch
	Job scheduling	LOC_WAIT (locality.wait)		less-local node a data-local task before giv-ing up and launching it on a	10ms, 100ms, 500ms, 1s, (3s), 10s
	Serialization	SER (serializer)		Data serialization mecha-nism	(Java), Kryo
	Shuffle	SHF_COMPR (shuffle.compress)	Must compress map output files	(true), false
		SFL_BUF (shuffle.file.buffer)	Size of in-memory buffer of shuffle file output stream	8 KB, (32 KB), 64 KB, 128 KB, 256 KB, 512 KB

Table 6 :

 6 Considered Spark Parameters

Shuffle is an operation that involves redistributing data across multiple partitions. It occurs whenever data transfers are needed between stages.


Alejandra Reyes-Ruiz

Jordan D Dimitrov
email: jordan.dimitrov@sorbonne-universite.fr

How can polyreactive antibodies conquer rapidly evolving viruses?

Keywords: antibodies, viruses, influenza virus, HIV-1, polyreactivity, broadly neutralizing antibodies

Broadly neutralizing antibodies against rapidly-evolving viruses (e.g. HIV-1 and influenza virus), often manifest antigen-binding promiscuity. Based on a recent study, we hypothesize on the significance of antibody polyreactivity in neutralization of rapidly-evolving viruses.

We propose that polyreactivity contributes to tolerate viral variants and shortens time for appearance of neutralizing antibodies.

Polyreactive antibodies can enable the neutralization of rapidly-evolving viruses

HIV-1 and influenza A virus (IAV) mutate at a fast pace; this translates into the rapid evolution of envelope proteins (Env) sequences, representing among the principal targets for neutralizing antibody (Ab) responses. The sequence heterogeneity of Env, however, makes the neutralization of rapidly-evolving viruses a daunting task. Typically, Ab responses are limited to viral variants to which the host has already been exposed. In this regard, the ingenious property of the adaptive immune system to differentiate closely-related molecular features might be disadvantageous for the neutralization of rapidly-mutating viruses.

Fortunately, the human immune system has preserved a primordial attribute -that of antigenbinding promiscuity, which might be a determining factor for efficient immune responses against pathogens with high genetic variability. Indeed, considerable numbers of Abs capable of neutralizing a broad range of distinct strains of HIV-1 or IAV, broadly neutralizing Abs (bNAbs), display antigen-binding polyreactivity, i.e., the capacity to bind to many virusunrelated antigens in addition to their primary targets [START_REF] Liu | Polyreactivity and autoreactivity among HIV-1 antibodies[END_REF][START_REF] Andrews | Immune history profoundly affects broadly protective B cell responses to influenza[END_REF][START_REF] Prigent | Conformational Plasticity in Broadly Neutralizing HIV-1 Antibodies Triggers Polyreactivity[END_REF][START_REF] Bajic | Autoreactivity profiles of influenza hemagglutinin broadly neutralizing antibodies[END_REF]. However, the question of how polyreactivity can contribute to the broad neutralization of rapidly-evolving viruses remains largely unanswered.

Recently, one study [START_REF] Guthmiller | Polyreactive Broadly Neutralizing B cells Are Selected to Provide Defense against Pandemic Threat Influenza[END_REF] investigated human Ab responses against IAV after natural infection or vaccination. The authors cloned genes encoding variable regions from circulating IAVspecific memory-B cells and plasmablasts to produce panels of recombinant IgG1 Abs. The largest fraction (>80%) of these Abs recognized virus Env protein hemagglutinin (HA, BOX 1), and among HA-binding Abs, a significant percentage (42%) was polyreactive. Of note, 88% of Abs against the stalk region of HA were polyreactive, as were 95% of those Abs targeting a broadly neutralizing epitope on this region. In contrast, the fraction of polyreactive Abs binding to the head domain was much lower (28%) than to the stalk region, implying that Abs directed to the stalk domain, which are frequently endowed with broadly neutralization potential, present also higher percentage of polyreactivity, contrary to the Abs directed to head domain and featuring a limited neutralization breadth. The higher preponderance of polyreactivity among IAV stalk-specific Abs agreed with results from other studies [START_REF] Andrews | Immune history profoundly affects broadly protective B cell responses to influenza[END_REF][START_REF] Bajic | Autoreactivity profiles of influenza hemagglutinin broadly neutralizing antibodies[END_REF].

Collectively, these data suggested that Abs binding to conservative sites on HA, i.e., sites vulnerable to neutralization, were predominantly polyreactive [START_REF] Guthmiller | Polyreactive Broadly Neutralizing B cells Are Selected to Provide Defense against Pandemic Threat Influenza[END_REF].

Furthermore, the increased binding of polyreactive Abs to conservative sites on HA resulted in the ability of many of these Abs to bind to distinct strains of IAV H1N1 [START_REF] Guthmiller | Polyreactive Broadly Neutralizing B cells Are Selected to Provide Defense against Pandemic Threat Influenza[END_REF]. The diversity among these strains reflects an evolutionary history of more than 100 years, highlighting the ability of polyreactive Abs to bind to HA, despite marked changes in the sequence. Moreover, polyreactive Abs induced by exposure to human H1 viruses also recognize zoonotic strains and IAVs belonging to phylogenetically distinct groups, such as type B viruses [START_REF] Guthmiller | Polyreactive Broadly Neutralizing B cells Are Selected to Provide Defense against Pandemic Threat Influenza[END_REF]. In the study, more than half of the polyreactive Abs binding to HA were endowed with neutralization activity; in contrast, only a small fraction of monoreactive Abs exhibited neutralization capacity [START_REF] Guthmiller | Polyreactive Broadly Neutralizing B cells Are Selected to Provide Defense against Pandemic Threat Influenza[END_REF], implying that polyreactivity has functional repercussion for the neutralization of IAV. Also, polyreactive Abs against HA arose predominantly after immunization or infection with novel IAV strains for which the studied individuals were not immune [START_REF] Guthmiller | Polyreactive Broadly Neutralizing B cells Are Selected to Provide Defense against Pandemic Threat Influenza[END_REF], suggesting that the polyreactive Abs developed before the viral-exposure contribute in the immune fight against IAV in nonantigen-experience individuals. Similarly, as in the case of IAV Abs, Abs with broadly neutralizing potential against HIV-1 are frequently (>50 % of Abs) polyreactive [START_REF] Liu | Polyreactivity and autoreactivity among HIV-1 antibodies[END_REF], and a direct relationship between polyreactivity and HIV-1-neutralization potency has recently been demonstrated [START_REF] Prigent | Conformational Plasticity in Broadly Neutralizing HIV-1 Antibodies Triggers Polyreactivity[END_REF]. Taken together these findings imply that polyreactivity directly contributes to broad neutralization of viruses.

In the study discussed above [START_REF] Guthmiller | Polyreactive Broadly Neutralizing B cells Are Selected to Provide Defense against Pandemic Threat Influenza[END_REF], the authors investigated how polyreactive Abs interacted with diverse strains of IAV. As evidenced by molecular dynamics simulations, the antigenbinding sites of polyreactive HA-specific Abs demonstrated increased flexibility relative to monoreactive antibodies. This is in accordance with previous reports about increased flexibility in the antigen-binding sites of polyreactive bNAbs against HIV-1 and against other targets compared with binding sites of monoreactive Abs [START_REF] Prigent | Conformational Plasticity in Broadly Neutralizing HIV-1 Antibodies Triggers Polyreactivity[END_REF][START_REF] Finton | Ontogeny of recognition specificity and functionality for the broadly neutralizing anti-HIV antibody 4E10[END_REF][START_REF] Adhikary | Adaptive mutations alter antibody structure and dynamics during affinity maturation[END_REF]. The variable regions of polyreactive Abs recognizing the HA stalk tend to harbor lower numbers of somatic mutations compared with the variable regions of monoractive HA-binding Abs [START_REF] Guthmiller | Polyreactive Broadly Neutralizing B cells Are Selected to Provide Defense against Pandemic Threat Influenza[END_REF]. Similarly as other polyreactive Abs isolated from healthy individuals or HIV-infected patients, the polyreactive Abs recognizing IAV used with elevated frequency the V H 1-69 gene, encoding exceptionally hydrophobic CDR H2 loop [START_REF] Guthmiller | Polyreactive Broadly Neutralizing B cells Are Selected to Provide Defense against Pandemic Threat Influenza[END_REF][START_REF] Boughter | Biochemical patterns of antibody polyreactivity revealed through a bioinformatics-based analysis of CDR loops[END_REF]. Of note, polyreactive Abs achieved broad IAV-recognition potential with a lower number of somatic mutations in comparison to monoreactive Abs, which suggested that these Abs might need shorter times of affinity maturation process for achievement of efficient neutralization of the virus [START_REF] Guthmiller | Polyreactive Broadly Neutralizing B cells Are Selected to Provide Defense against Pandemic Threat Influenza[END_REF]. Similarly, a low number of somatic mutations have been reported for human polyreactive Abs obtained from healthy donors and screened by high-throughput assays [START_REF] Shehata | Affinity Maturation Enhances Antibody Specificity but Compromises Conformational Stability[END_REF][START_REF] Boughter | Biochemical patterns of antibody polyreactivity revealed through a bioinformatics-based analysis of CDR loops[END_REF]. Compelling evidence from X-ray structural and other biophysical analyses shows that the absence or low number of somatic mutations is associated with increased structural dynamics of the Ab antigen-binding sites [START_REF] Shehata | Affinity Maturation Enhances Antibody Specificity but Compromises Conformational Stability[END_REF][START_REF] Manivel | Maturation of an antibody response is governed by modulations in flexibility of the antigen-combining site[END_REF][START_REF] Fernández-Quintero | Local and Global Rigidification Upon Antibody Affinity Maturation Front[END_REF]. In the case of HIV-1 bNAbs however, the elevated conformational flexibility of the paratope, as well as the polyreactivity status, are not explained by the low numbers of somatic mutations [START_REF] Klein | Somatic mutations of the immunoglobulin framework are generally required for broad and potent HIV-1 neutralization[END_REF]. Indeed, these Abs usually harbor abnormally large numbers of mutations and simultaneously manifest polyreactivity. Also, substantial mutation loads, especially in the framework regions of the variable region are known to compromise the protein stability of the Ab, which might result in elevated flexibility of the antigen-binding site relative to the monoreactive Abs specific for the same antigen [START_REF] Klein | Somatic mutations of the immunoglobulin framework are generally required for broad and potent HIV-1 neutralization[END_REF]. Collectively, these findings demonstrate that there are alternative molecular pathways for achievement of enhanced conformational dynamics and polyreactivity of Abs.

How might Ab polyreactivity contribute to virus neutralization?

Polyreactive bNAbs against IAV or HIV-1 recognize conservative Env epitopes with high affinity that is commensurate to the affinity of monoreactive bNAbs [START_REF] Andrews | Immune history profoundly affects broadly protective B cell responses to influenza[END_REF][START_REF] Prigent | Conformational Plasticity in Broadly Neutralizing HIV-1 Antibodies Triggers Polyreactivity[END_REF][START_REF] Guthmiller | Polyreactive Broadly Neutralizing B cells Are Selected to Provide Defense against Pandemic Threat Influenza[END_REF]. Thus, polyreactive bNAbs incongruously incorporate two opposite qualities -stringent binding specificity and binding promiscuity. They may be optimized to bind with high affinity to their target epitope but might also auxiliary interact with other lower affinity antigens because of the enhanced flexibility and better adaptability of their antigen-binding sites to various molecular features on distinct antigens. The findings on bNAbs clearly showed that polyreactivity can be a consequence of the enhanced structural dynamics of antigen-binding site [START_REF] Guthmiller | Polyreactive Broadly Neutralizing B cells Are Selected to Provide Defense against Pandemic Threat Influenza[END_REF][START_REF] Finton | Ontogeny of recognition specificity and functionality for the broadly neutralizing anti-HIV antibody 4E10[END_REF][START_REF] Adhikary | Adaptive mutations alter antibody structure and dynamics during affinity maturation[END_REF][START_REF] Manivel | Maturation of an antibody response is governed by modulations in flexibility of the antigen-combining site[END_REF]. Hence, the precise formulation of the question about the functional role of polyreactivity in broad neutralization of rapidly-evolving viruses should be on how the increased structural dynamics would contribute to the potential of Abs to neutralize a broad range of virus variants.

Considering the structural organization of vulnerable sites on Env -mostly bearing an invariant sequence, and surrounded by regions of high sequence heterogeneity, we proposed the following scenario: if the Ab-binding footprint overlap with the conserved vulnerability site in the antigen without steric effects that can obstruct the access of Abs to its epitope, the broad virus neutralization potential might be achieved by Abs with rigid antigen-binding sites.

However, the cognate epitope of the neutralizing Ab might not likely perfectly match the vulnerability site, potentially overlapping molecular surfaces with high variability. Here, certain mutations in the viral protein might compromise Ab binding at the vulnerability site, perhaps due to steric clashes or incompatibility of electrostatic charges. Also, the large surface area buried by Abs, while binding to protein antigens (ca. 1000 Å 2 ), might favor simultaneous interactions with both conservative and variable parts of the Env. Thus, the presence of intrinsic conformational dynamics of antigen-binding site might be advantageous by giving capacity of Abs to accommodate the disruptive effects of mutations near the site of vulnerability. Of note, analyses of the effect of somatic mutations on the flexibility of some HIV-1 bNAbs suggest that central parts of antigen-binding sites (including CDR H3) are configured early in the affinity maturation process and characterized by structural rigidity [START_REF] Klein | Somatic mutations of the immunoglobulin framework are generally required for broad and potent HIV-1 neutralization[END_REF][START_REF] Davenport | Somatic Hypermutation-Induced Changes in the Structure and Dynamics of HIV-1 Broadly Neutralizing[END_REF]. In contrast, somatic mutations might affect the flexibility of polypeptide chains in the periphery of the paratope [START_REF] Klein | Somatic mutations of the immunoglobulin framework are generally required for broad and potent HIV-1 neutralization[END_REF][START_REF] Davenport | Somatic Hypermutation-Induced Changes in the Structure and Dynamics of HIV-1 Broadly Neutralizing[END_REF]. Thus, HIV-1 and IAV bNAbs might use molecular flexibility to overcome the detrimental effect of mutations in regions of little significance for neutralization but which might influence key amino acid residue contacts at sites of vulnerability, although this remains conjectural (Fig. 1a).

Aside from the presumed molecular mechanisms of polyreactive bNAbs, the data from the highlighted study [START_REF] Guthmiller | Polyreactive Broadly Neutralizing B cells Are Selected to Provide Defense against Pandemic Threat Influenza[END_REF] incite speculations regarding the physiological significance of polyreactive Abs in the neutralization of rapidly-evolving viruses; the striking findings that novel strains of IAV can induce broadly reactive polyreactive Abs requiring lower number of somatic mutations to achieve neutralization and broad reactivity might have important repercussions. For instance, the rapid clinical course of IAV infection imposes a requirement for rapid development of neutralizing Abs. Indeed, refined Ab responses with multiple iterations in mutagenesis and B cell selection might be time-and resource-consuming for the host. Hence, Abs polyreactivity can be used as an immune system strategy for 'gaining time': accelerating the evolution of Ab responses in the race between a virus with ever-evolving antigenic features and the immune system (Fig. 1b). Further investigations regarding the mechanisms leading to selection and production of polyreactive bNAbs might have significant importance in the development of vaccines against viruses with high genomic diversity. Suggested role of antibody polyreactivity in immune responses against influenza virus. B cells expressing BCR with rigid antigen-binding site would require higher number of iterations of somatic mutations and selection during affinity maturation process before producing an antibody capable to neutralize the virus. A B cell clone expressing BCR with flexible antigen-binding site will reach a state of efficient neutralization of broader range of influenza virus strains after lower number of iterations during affinity maturation process.

Figure 1 .

 1 Figure 1. Human polyreactive antibodies and virus neutralization. a. A virus-neutralizing

Acknowledgments

This work was supported by Institut National de la Santé et de la Recherche Médicale (INSERM, France). A.R.R. is recipient of fellowship from an Innovative Training Network (ITN) funded by the European Union's Horizon 2020 Programme under grant agreement No 859974, project EDUC8. J.D.D. is recipient of a grant from the European Research Council (Project CoBABATI ERC-StG-678905). We are grateful to Dr Sebastien Lacroix-Desmazes (Centre de Recherche des Cordeliers, Paris) for reading and fruitful discussion about the manuscript.