Pierre Matri
email: pmatri@fi.upm.es

María S Pérez
email: mperez@fi.upm.es

Alexandru Costan
email: alexandru.costan@irisa.fr

Gabriel Antoniu
email: gabriel.antoniu@inria.fr

TýrFS: Increasing Small Files Access Performance with Dynamic Metadata Replication

Small files are known to pose major performance challenges for file systems. Yet, such workloads are increasingly common in a number of Big Data Analytics workflows or largescale HPC simulations. These challenges are mainly caused by the common architecture of most state-of-the-art file systems needing one or multiple metadata requests before being able to read from a file. Small input file size causes the overhead of this metadata management to gain relative importance as the size of each file decreases. In this paper we propose a set of techniques leveraging consistent hashing and dynamic metadata replication to significantly reduce this metadata overhead. We implement such techniques inside a new file system named TýrFS, built as a thin layer above the Týr object store. We prove that TýrFS increases small file access performance up to one order of magnitude compared to other state-of-the-art file systems, while only causing a minimal impact on file write throughput.

I. INTRODUCTION

A large portion of research in data storage, management and retrieval focuses on optimizing access performance for large files [START_REF] Wu | Data mining with big data[END_REF]- [START_REF] Cutting | Scatter/Gather: A cluster-based approach to browsing large document collections[END_REF]. Yet, handling a large number of small files raises other difficult challenges, that are partly related to the very architecture of current file systems. Such small files, with a size inferior to a few megabytes, are very common in large-scale facilities, as shown by multiple studies [START_REF] Juve | Characterizing and profiling scientific workflows[END_REF], [START_REF]NERSC storage trends and summaries[END_REF]. They can be generated by data-intensive applications such as CM1 [START_REF] Bryan | A benchmark simulation for moist nonhydrostatic numerical models[END_REF] or HACC [START_REF] Habib | The universe at extreme scale: multi-petaflop sky simulation on the BG/Q[END_REF], Internet of Things or Stream Processing applications, as well as large scale workflows such as Montage [START_REF] Berriman | Montage: An on-demand image mosaic service for the nvo[END_REF], CyberShake [START_REF] Graves | CyberShake: A physics-based seismic hazard model for southern california[END_REF] or LIGO [START_REF] Abramovici | LIGO: The laser interferometer gravitational-wave observatory[END_REF]. Improving file access performance for these applications is critical for scalability in order to handle ever-growing data sets on largescale systems [START_REF] Dorier | Damaris: How to efficiently leverage multicore parallelism to achieve scalable, jitterfree I/O[END_REF].

As the amount of data to be transferred for storage operations on any single small file is intuitively small, the key to optimizing access performance for such files lies in improving the efficiency of the associated metadata management. Actually, as the data size for each file decreases, the relative overhead of opening a file is increasingly significant. In our experiments, with small enough files, opening a file may take up to an order of magnitude more time than reading the data it contains. One key cause of this behavior is the separation of data and metadata inherent to the architecture of current file systems. Indeed, to read a file, a client must first retrieve the metadata for all folders in its access path, that may be located on one or more metadata servers, to check that the user has the correct access rights or to pinpoint the location of the data in the system. The high cost of network communication significantly exceeds the cost of reading the data itself.

We advocate that a different file system architecture is necessary to reduce the cost of metadata management for such workloads involving many small files. While one could think of co-locating data and metadata, a naive implementation would pose significant performance or consistency issues [START_REF] Pineda-Morales | Towards multi-site metadata management for geographically distributed cloud workflows[END_REF]. For example, the metadata for a directory containing large numbers of files would potentially be replicated to an important number of servers, both wasting precious memory resources and causing low metadata update performance. While others worked on optimizing metadata management [START_REF] Dong | An optimized approach for storing and accessing small files on cloud storage[END_REF]- [START_REF] Mackey | Improving metadata management for small files in HDFS[END_REF], their contributions iterate on existing architectures, improving the internal handling of operations at the metadata server by means of caching or compression, without fixing the root cause of the issue, that is, the metadata distribution.

In this paper we propose a completely different approach and design a file system from the bottom up for small files without sacrificing performance for other workloads. This enables us to leverage some design principles that address the metadata distribution issues: consistent hashing [START_REF] Karger | Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on the world wide web[END_REF] and dynamic data replication [START_REF] Lamehamedi | Simulation of dynamic data replication strategies in data grids[END_REF]. Consistent hashing enables a client to locate the data it seeks without requiring access to a metadata server, while dynamic replication adapts to the workload and replicates the metadata on the nodes from which the associated data is accessed. The former is often found in key-value stores [START_REF] Stoica | Chord: A scalable peer-to-peer lookup service for internet applications[END_REF], [START_REF] Decandia | Dynamo: Amazon's highly available key-value store[END_REF], while the latter is mostly used in geo-distributed systems [START_REF] Matri | Towards efficient location and placement of dynamic replicas for geo-distributed data stores[END_REF]- [START_REF] Jayalakshmi | Dynamic data replication across geo-distributed cloud data centres[END_REF].

We briefly motivate this research leveraging a short experiment (Section II) and review the related work (Section III). Our contributions are threefold:

• We propose a novel file system architecture optimized for small files based on consistent hashing and dynamic metadata replication while not sacrificing performance for other workloads (Section IV). • We detail the design and implementation of TýrFS, a file system implementing the aforementioned principles (Section V), built atop the Týr object storage system [START_REF] Matri | Týr: blob storage meets built-in transactions[END_REF]. • After introducing our experimental platform (Section VI), we leverage TýrFS in various applicative contexts and factually prove its applicability to both HPC and Big Data workloads (Section VII). We prove that it significantly outperforms state-of-the-art file systems, increasing throughput by up to one order of magnitude.

We finally discuss the limitations of our approach (Section VIII) and conclude on the future work that could further enhance our proposal (Section IX).

II. MOTIVATION: OVERHEAD OF READING SMALL FILES

In this section we seek to describe in more detail the issue by studying the concrete performance effects of reducing file size with a large data set. For that matter, we plot the time taken and file system CPU usage when reading a constantsize data set while decreasing the size of the files it contains.

We perform these experiments using both the Lustre [START_REF] Schwan | Lustre: Building a file system for 1000-node clusters[END_REF] (2.10.0) and OrangeFS [START_REF] Moore | OrangeFS: Advancing PVFS[END_REF] (2.9.6) distributed file systems, that we setup on 24 nodes of the Paravance cluster in Rennes (France) of the Grid'5000 testbed [START_REF]Grid'5000 -Rennes Hardware (Paravance)[END_REF]. On 24 other machines of the same cluster (384 cores) we launch a simple ad-hoc MPI benchmark reading in parallel the contents of a pre-loaded set of files. The set size varies between 38,400 × 32 MB files (100 files per rank) to 3.84 × 10 7 × 32 KB files (100,000 files per rank), for a total combined size of 1.22 TB.

In Figure 1 we plot the aggregate throughput achieved by Lustre and OrangeFS. For both systems we note a decrease of the read throughput as we reduce the size of the files. The decrease is as significant as one order of magnitude when replacing 32 MB by 32 KB files.

In Figure 2 we detail these results. We calculate the total execution time of the benchmark with Lustre and OrangeFS. We highlight the proportion of this time spent on the open operation fetching the metadata (M) and on the read operation actually reading the data (D). Interestingly, with smaller file sizes, the open operation takes a significant portion of the total I/O time. This proportion is above 90% for 32 KB files for both systems, while it is less than 35% for larger file sizes.

These results clearly demonstrate that reducing the size of the files puts a very high pressure on the file system metadata. This is caused by the very architecture of most file systems, in which metadata and data are separated from each other. This implies that metadata needs to be accessed before performing any read operation. Consequently, we argue that reducing the cost of metadata management without changing the file system architecture cannot possibly address the root cause of the performance drop.

These observations guide the design and implementation of TýrFS, which we detail in Sections IV and V.

III. RELATED WORK: METADATA OPTIMISATIONS

While improving I/O performance when accessing small files in a distributed file system is a known problem, most of the work to date focuses on optimizing metadata management and assumes constant file system architecture. Two main solutions appear: reducing resource consumption required for metadata management, or completely eluding the problem with combining a set of small files into a smaller set of larger files.

Optimizing metadata management can be done in a number of ways. In [START_REF] Carns | Small-file access in parallel file systems[END_REF], Carns et al. study with PVFS [START_REF] Ross | PVFS: A parallel file system for linux clusters[END_REF] the impact of various strategies such as metadata commit coalescing, storing data right inside inodes or pre-creating files. While highly effective, those optimizations are orientated towards improving small file write performance and have little to no effect on read operations. Creating archives from a large number of small files is the second frequent option found in the literature. For instance in HDFS [START_REF] Shvachko | The Hadoop distributed file system[END_REF], the HAR (Hadoop Archive) format [START_REF]Hadoop Documentation -Archives[END_REF] is leveraged by [START_REF] Mackey | Improving metadata management for small files in HDFS[END_REF], [START_REF] Korat | Reduction of data at namenode in HDFS using harballing technique[END_REF] to transform an input dataset composed of many small files into a set of large files that can be handled efficiently by the metadata node. Building on this work, Vorapongkitipun et al. [START_REF] Vorapongkitipun | Improving performance of smallfile accessing in Hadoop[END_REF] propose a new archive format for Hadoop that further improves performance. While this approach is arguably extremely effective, it requires a pre-processing step to transform the input data set. Furthermore, this requires modifying the application so it is able to take such archives as input. As such, this approach is not applicable for workloads where small files are generated as intermediate data between processing steps. These challenges are partly addressed by Zhang et al. in [START_REF] Zhang | Improving the efficiency of storing for small files in HDFS[END_REF], by delegating the merge task to a middleware process. While highly interesting, this approach retains the main drawback of archiving, that is, complexifying operations other than simple reads.

Overall, we note that the optimizations proposed in the literature having a significant impact on read performance rely on archiving, and optionally indexing files. While such approaches are indisputably effective for analyzing large sets of data, they invariably result in relaxing or dropping part of the POSIX I/O API. Notably, unlike this paper, no related work considers modifying a file after it has been written. While this may not be an issue for Big Data Analytics, this certainly is for HPC use cases relying on I/O frameworks such as HDF5 [START_REF] Folk | HDF5: A file format and I/O library for high performance computing applications[END_REF].

Client

Metadata node

Storage nodes

s1 s2 s3 md(/foo/file) [(id, s2)] r(id) (data)
Metadata Data Fig. 3. Typical read protocol for a small file with centralized metadata management (e.g. Lustre or HDFS).

IV. DESIGNING A FILE SYSTEM FOR SMALL FILES

In this section we detail the main architectural and design goals that drive the design of a file system optimized for small files. We consider as small files those for which it does not make sense to split them into multiple pieces (chunking). Since this notion is variable and cannot be defined precisely, we will consider as small -files with a size under a few megabytes.

A. Detailing the metadata distribution issue

First, let us consider the behavior of a typical distributed file system for reading a file, whose path is /foo/file. We consider the case of centralized metadata management, which is the base architecture of Lustre [START_REF] Schwan | Lustre: Building a file system for 1000-node clusters[END_REF] or HDFS [START_REF] Shvachko | The Hadoop distributed file system[END_REF]. As shown in Figure 3, the request is performed in two steps. First, the client interrogates the metadata server to get the layout information of the file to read, including the actual node on which the data is located. Only with this information can the client access the data, in a second step. While the cost of such metadata fetching is amortized when reading large files, it cannot be with small ones. Should the user seek to read many files, the cost of this communication with the metadata node is to be paid for each file access. This largely explains the results observed in Section II.

The previous results also noticeably show a further performance degradation with OrangeFS [START_REF] Moore | OrangeFS: Advancing PVFS[END_REF] compared to Lustre. This is caused by the distribution of the metadata across the entire cluster, which brings better horizontal scalability properties at the cost of additional metadata requests, as illustrated in Figure 4. Note that the location of the metadata for any given file in the cluster is independent from the location of its data.

Intuitively, we understand that two problems are to be tackled when dealing with small files at scale:

• First, the client should be able to locate by itself the pieces of data to be read without requiring communication with a dedicated metadata server. • Second, all the metadata required to read a file should be colocated with the file data whenever possible.

Those two steps would allow the client to bypass metadata requests entirely, consequently offering significantly better performance when dealing with many small files. While these principles are seemingly easy, they are challenging to implement for distributed file systems because of the hierarchical nature of their namespace.

Client s 1 s 2 s 3 s1 s2 s3 md(/) (metadata) md(/foo) (metadata) md(/foo/file) [(id, s2)] r(id) (data)
Metadata Data Fig. 4. Typical read protocol for a small file with decentralized metadata management on three storage nodes s 1 , s 2 and s 3 (e.g. OrangeFS).

B. Predictably distributing data across the cluster

In typical file systems, the data is distributed across the cluster in a non-predictable fashion. A metadata process is responsible for assigning a set of nodes to each file based on some system-specific constraints [START_REF] Laitala | Metadata management in distributed file systems[END_REF]. This location is saved in the file metadata, which therefore needs to be queried in order to later pinpoint the location of the data in the cluster. While this allows a great latitude in finely tweaking data location, it inherently imposes at least two successive requests in order to retrieve data in the cluster. As the data is small, this request cost is non-amortizable in the case of small files. This challenge has long been solved for state-of-the-art distributed object stores such as Dynamo [START_REF] Decandia | Dynamo: Amazon's highly available key-value store[END_REF] or Riak [START_REF] Klophaus | Riak core: Building distributed applications without shared state[END_REF]. Indeed, instead of relying on metadata to locate information in a cluster, these systems distribute the data in a predictable fashion, using a common algorithm shared across the cluster and the clients. This effectively allows the clients to infer the location of any data they want to access without prior communication with any metadata node.

Such predictable data distribution is however challenging for file systems due to their hierarchical nature. For accessing any single file, a client needs to read the metadata of all the folder hierarchy for this file in order to check, for example, that the user has the correct permissions for reading it. Let us illustrate this with a simple example. A folder at /foo contains many files to be processed by a data analytics tool. To distribute the load across the storage nodes, these files should arguably be spread across the cluster. As such, while clients would be able to locate each file in the cluster because of the predictable location algorithm, they would still need to access the metadata for /, /foo and for the file to be read. All these metadata pieces are potentially located on a different set of servers, and would hence require additional communication that we are precisely seeking to eliminate.

Although such predictably data distribution is a step towards the solution, it cannot be considered independently of metadata management. Some file systems such as Ceph [START_REF] Weil | Ceph: A scalable, high-performance distributed file system[END_REF] partially leverage predictable distribution for reducing the metadata size without addressing the issue of its distribution.

C. Towards data and metadata colocation

To significantly reduce the overhead of metadata fetching we propose to collocate it with the data it concerns. Let us keep our example of the /foo folder containing many files. This implies that the node storing the data for the file /foo/a.txt would also hold the metadata for /, /foo and /foo/a.txt. We do so by leveraging an architecture roughly similar to that of OrangeFS. Each node composing the file system cluster integrates two services. First, a metadata service manages meta-information relative to folders and files. Second, a data service persists the data relative to the objects (files or folders) stored on this node.

As a first principle, we propose to always store the metadata for any given file on the same nodes the first chunk of the file is stored. As small files are by definition composed of a single chunk, this effectively eliminates the need for remotely fetching the file metadata before accessing its data. This contrasts with the design of OrangeFS, which does not guarantee such collocation. Such design is conceptually similar to the decision of embedding the contents of small-enough file right inside inodes on the Ext4 local file system.

Collocating the file data with the metadata of its folder hierarchy is a significantly harder problem. Indeed, a folder may contain many other folders and files potentially stored on different nodes. Specifically, the root folder itself is by definition ancestor of all folders and files. One could think of statically replicating this data to every relevant node. Yet, this solution would have huge implications in terms of storage overhead because of the potentially large amount of metadata replicated, and in terms of write performance because of the need to maintain consistency across all metadata replicas.

To alleviate this overhead, we replicate the metadata dynamically. The metadata agent on each node determines which of the metadata to replicate locally based on the current workload it serves. This solution is largely tackled in geo-distributed storage systems, in which dynamic replication is used to selectively replicate data objects to various discrete regions while minimizing write and storage overhead by adapting this replication based on user requests [START_REF] Matri | Towards efficient location and placement of dynamic replicas for geo-distributed data stores[END_REF]- [START_REF] Jayalakshmi | Dynamic data replication across geo-distributed cloud data centres[END_REF]. Our solution is largely inspired by their proved design and best practices.

We further reduce the storage overhead by replicating the metadata that is strictly necessary for accessing a file. Indeed, only a tiny subset of the whole metadata for the folder hierarchy of a file has to be accessed before reading it: permissions. In addition, we argue that permission changes are a very infrequent operation that is performed either when a folder is first created or as a manual management task which is unlikely to be time-sensitive ; we could not find any application in which permission change is in the critical path. As such, replicating the permissions is unlikely to have a significant negative effect on the performance of real-world applications. Intuitively, this solution implies that the closer a folder is to the root of the hierarchy, the more likely its permissions are to be replicated to a large number of nodes. As an extreme case, the permissions for the root folder itself would be replicated to every node in the cluster.

In Figure 5 we show a high-level overview of the proposed architecture, highlighting the different modules on each node composing the file system cluster.

V. T ÝRFS: A FILE SYSTEM OPTIMIZED FOR SMALL FILES

In this section we detail the implementation of the above principles in a prototype file system optimized for small files, that we name TýrFS. We implement it as a thin layer atop the Týr transactional, distributed object storage system [START_REF] Matri | Týr: blob storage meets built-in transactions[END_REF]. The system is composed of a data service described in Section V-A, directly mapped to Týr native semantics. A lightweight metadata management module runs on each server node. We describe in detail its implementation in Section V-B. Finally, a client library exposes both a mountable and a native interface optimized for TýrFS. We detail it in Section V-C.

A. Implementation of the data service

The data service directly leverages the capabilities natively offered by Týr to efficiently distribute and store data in the cluster. A file in TýrFS is directly mapped to a blob in Týr, with the blob key being the absolute file path.

The predictable distribution of the data across the cluster that we describe in Section IV-B is inherited from Týr. The base system leverages consistent hashing of an object absolute path to distribute data across the cluster. Such method has been demonstrated to fairly distribute the load across the whole cluster in distributed systems, and to be applicable to local file systems [START_REF] Lensing | Direct lookup and hashbased metadata placement for local file systems[END_REF]. The stored data can thus be located and accessed by TýrFS clients directly. Read and write operations as per the POSIX I/O standard are mapped directly to the blob read and write operations offered by Týr. Folders are also stored as a blob holding a list of all objects contained in it. A folder list operation only requires reading the blob ; adding a file to a folder is implemented as a blob append. A background process compacts the file list when objects are deleted from a folder.

A side effect of using consistent hashing for data distribution is the need to move the physical location of the data when a file is moved in the folder hierarchy. This operation is implemented by writing "staple" records at the new location of the object and of its descendents, referencing their previous location. The data transfer occurs in the background. Moving a folder containing many descendents is consequently a costly operation. We argue that this trade-off is justified by the scarcity of such operations compared to the shorter critical path and thus higher performance of file reads and writes.

B. Popularity-based metadata service

The metadata service is implemented as a lightweight plugin running inside each Týr server process. Its responsibilities are threefold. First, it serves as a persistence layer for the files whose first chunk of data is stored on this process. Second, it integrates a dynamic replication agent, which analyzes the received requests to determine for which folders to replicate and store the metadata information locally. Finally, it serves as a resolver agent for the metadata requests, dispatching the requested information directly if available locally, or forwarding the request to the appropriate node determined using consistent hashing if not. The most interesting part of this metadata service is arguably the dynamic replication agent, which is the focus of this section.

We use popularity as the main metric for dynamic replication to determine which folders to locally replicate metadata for. We define popularity as the number of metadata requests for any given folder in a defined time period. These popularity measurements are performed independently for each node. Consequently, on any given node, the more frequently metadata for a specific folder is requested the more likely it is to be considered for dynamic replication to that node. This metric is particularly suited for workloads accessing vast numbers of small files. Considering that at least a metadata request is needed for reading each file, a data set composed of many files will be significantly more likely to have the metadata for its ancestor folders replicated locally than a data set composed of a single large file.

To measure the popularity for each folder, we record the number of times its metadata is requested to the local resolver agent. Keeping exact counters for each possible folder is memory-intensive, so we only focus on the k most frequently requested folders, k being user-configurable. This problem is known as top-k counting. A variety of memory-efficient, approximate solutions to this problem exist in the literature. For this use-case, such an approximate algorithm is tolerable as popularity estimation errors cannot jeopardize the correctness of the system. Our implementation uses Space-Saving [START_REF] Metwally | Efficient computation of frequent and top-k elements in data streams[END_REF] as top-k estimator. It guarantees strict error bounds for approximate request counts, while exhibiting a small, predictable memory complexity of O(k). The output of the algorithm is an approximate list of the k folders for which the metadata is the most frequently requested since the data structure has first been initialized. We perform these measurements over a sliding time window in order to favor immediate popularity over historical measurements. Whenever a metadata request arrives at the metadata resolver for a folder whose metadata is not locally replicated, the resolver agent determines whether or not it should be replicated. It does so by adding the requested folder path in the current time window for the top-k estimator. If the folder path is present in the output of the estimator, this path should be considered as popular and the associated metadata should be replicated locally. The agent determines which node holds the metadata for this folder, and remotely fetches the metadata from this node. Should the metadata be kept locally, it is stored in a temporary local store. In any case, the fetched metadata is returned to the client. This process is highlighted by the pseudo-code of Algorithm 1. Although folder permission modifications are a very rare occurrence in real-world applications, such metadata updates should nonetheless be consistently forwarded to metadata replicas. We integrate inside the TýrFS metadata agent a synchronous metadata notification service. When the permissions for any given folder are modified, this service synchronously notifies all nodes locally storing a dynamic replica of this metadata. Subscription to updates is performed atomically by a remote metadata agent during the metadata fetch if the remote agent has determined that this metadata should be dynamically replicated to it. Acknowledgement is sent to the client requesting the metadata change only after the update has been applied to all dynamic replicas.

A background bookkeeping service monitors changes in the current time window for the top-k estimator in order to identify the metadata for previously-popular objects that are not popular anymore. The locally available metadata is marked for deletion. It will be kept locally as long as the locally available storage permits. When space needs to be reclaimed, the node deletes the locally-replicated metadata marked for deletion. It also synchronously cancels its subscription for metadata update notifications.

C. TýrFS client

The TýrFS client provides two different interfaces. A native interface exposes all POSIX I/O functions. In addition, a mountable interface, itself based on the native library, allows TýrFS to be mounted as any other file system. The mountable interface leverages the FUSE library. The native interface is implemented as a thin wrapper around the Týr native client library, which handles all the message serialization and communication with the storage cluster.

As an optimization for the TýrFS design, the native interface adds a function to those of the POSIX I/O standard. In common file systems, reading a file is split between the metadata fetching (open) and the data read operation (read). Withi the design of TýrFS embracing collocation of data and metadata, this can be further optimized by combining the two operations into a single one. Doing so saves the cost of an additional client request to the file system cluster, therefore reducing read latency and increasing throughput. The TýrFS client proposes a native method exposing such behavior, named fetch, which functionally opens and reads a file given by its absolute path.

VI. EXPERIMENTAL CONFIGURATION

We execute all following experiments on the Grid'5000 testbed we introduced in Section II. The machines we use are part of the Paravance cluster [START_REF]Grid'5000 -Rennes Hardware (Paravance)[END_REF], located in Rennes. Each node embeds 16 cores (2 x 8-core Intel Xeon E5-2630v3 processor) as well as 128 GB RAM. Connectivity between nodes is done using low-latency 10 Gigabit ethernet. All nodes are connected to the same switch. Node-local storage is provided by two 600 GB spinning drives.

We compare the performance of TýrFS with that of multiple file systems, from both the HPC and the Big Data communities. Lustre 2.10.0 [START_REF] Schwan | Lustre: Building a file system for 1000-node clusters[END_REF] and OrangeFS 2.9.6 [START_REF] Schwan | Lustre: Building a file system for 1000-node clusters[END_REF] are deployed according to best practices. With Lustre we dedicate one node for metadata. For Big Data applications we compare TýrFS with Ceph 12.2.0 [START_REF] Weil | Ceph: A scalable, high-performance distributed file system[END_REF] and HDFS 2.8.0 [START_REF] Shvachko | The Hadoop distributed file system[END_REF]. All systems are configured not to replicate data, consequently not providing fault-tolerance which is beyond the scope of this paper. We discuss fault tolerance in Section VIII-C.

We benchmark the performance of the file system using two different application profiles. For HPC file systems we leverage the same benchmark as in Section II, which accesses the file system through the TýrFS native client and leverages the fetch operation it provides. The benchmark reads through the whole pre-loaded random dataset from all ranks, evenly distributing the last across all ranks. No actual computation is performed on the data. Big Data Analytics applications are running atop Spark 2.2.0 [START_REF] Zaharia | Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing[END_REF] and are extracted from SparkBench [START_REF] Li | SparkBench: a comprehensive benchmarking suite for in memory data analytic platform spark[END_REF]. All these applications have the common characteristic to be mostly read-intensive. We connect Spark applications to Ceph by modifying Hadoop to use the POSIX I/O interface and to TýrFS using the native interface. We expose to Spark the data layout of both systems to ensure data locality. TýrFS is configured with a measurement time window of 1 minute and a k of 20 that we determined to be optimal for our configuration. We discuss this setup in Section VIII-D.

Unless specified otherwise, for Lustre and OrangeFS we use 24 machines to deploy the storage systems, plus 8 machines to deploy the client applications. With Lustre we dedicate one node for metadata management. Big Data applications are deployed alongside storage on the whole 48 nodes of the cluster. All results are the average of 50 experimental runs.

VII. EXPERIMENTAL VALIDATION

In this section we prove that TýrFS delivers its promise of drastically improving read performance for small files. We also demonstrate that this benefit applies to applicative contexts as diverse as HPC frameworks and Big Data applications compared to state-of-the-art, industry standard file systems.

A. Drawing the baseline performance in HPC contexts

We first compare the performance results obtained with TýrFS to those of Lustre and OrangeFS in the exact same conditions as in Section II. The curves for Lustre and GPFS are reproduced for reference. The size of the input data set is constant. We vary the number of input files between 3,840 × 32 MB files (1 file / rank) to 3.84×10 6 × 32 KB files (10.000 files / rank). The data is generated randomly.

In Figure 6 we plot the total execution time of the benchmark for TýrFS, Lustre and OrangeFS. To evaluate the baseline performance of the metadata replication system, we also plot the performance achieved by TýrFS with dynamic replication disabled. The performance for large files is on par with that of Lustre or OrangeFS due to the possible amortization of the metadata requests. However, TýrFS shows a significant performance improvement over those two systems for small files. Disabling metadata replication causes TýrFS performance to drop at the level of other distributed file systems. The similarity of the performance of TýrFS without metadata replication with that of OrangeFS is due to the fact that both embrace akin distributed metadata management principles. This unequivocally demonstrates the effectiveness of dynamic metadata replication with small files.

Figure 7 shows the relative breakdown of the I/O time between the metadata (M) and data (D) access inside the fetch operation. We note that the relative time spent on metadata access as the data file increases tends to decrease with TýrFS. This clearly contrasts with other file systems, in which the throughput drop is mainly due to the high cost of metadata management. In TýrFS, the main factor impacting read time is the added cost of issuing additional read requests. The efficiency of the dynamic metadata replication is highlighted by the performance of TýrFS with this service disabled, which here also is smilar to that of other file systems.

D. Experimenting with Big Data applications

For these experiments we setup a configuration according to the best practices for Big Data Analytics. We deploy both the storage system and the computation on all 48 nodes of the cluster. We compare the performance of Týr with that of Ceph and HDFS. The experiments are based on a series of different Spark applications.

We leverage SparkBench [START_REF] Li | SparkBench: a comprehensive benchmarking suite for in memory data analytic platform spark[END_REF] to confirm that the design of TýrFS yields significant performance improvements with small files for applications showing a wide range of I/O profiles. We use exactly the same setup as in the previous experiment, and execute three read-intensive benchmarks extracted from the benchmark suite: Connected Component (CC), Grep (GR) and Decision Tree (DT). These benchmarks are intended to evaluate the I/O performance of TýrFS when faced with different workloads types, respectively graph processing, text processing and machine learning. In Figure 10 we plot the aggregate throughput obtained across all nodes when varying the size of the input data set files. Overall, the results are very similar to those obtained with Lustre and OrangeFS. This is not surprising, and is mostly due to the very similar architecture of all these file systems, in which the relative cost of metadata management increases as the file size decreases. For these experiments as well, the throughput drop we observe with TýrFS when using 32 KB files instead of 32 MB files is only of about 25%. In similar conditions, other systems show a much more significant throughput drop of more than 90%. This results in a 3.8× increase of total computation time with OrangeFS, wasting computing resources. Not all applications are read-intensive. Although write performance is not a main target of this paper, it is critical to understand precisely the impact of dynamic metadata replication for balanced and write-intensive workloads as well. To do so, we add two more applications to the experiment: Sort (SO) from SparkBench, which illustrates a balanced I/O workload, and Tokenizer (TZ), which provides a write-intensive one. Tokenizer reads a text file, tokenizes each line into words, and calculates the NGrams=2 for each line. Ngrams are saved as text. This is a common preprocessing step in topic modeling for documents where word patterns are extracted for feeding a machine learning model. In Figure 11 we plot the measured write throughput for Sort and Tokenizer with TýrFS, Ceph and HDFS. The performance of the write operations are similar across the two experiments. In terms of write performance, TýrFS exhibits a write performance consistent with that of Ceph. Ceph outperforms both TýrFS and HDFS for write throughput by a margin or respectively 8% and 35%.

Considering the application as a whole, Figure 12 summarizes our results for Big Data applications. We measure the application completion time difference for 32 MB files relative to that obtained with 32 KB files. The throughput advantage of TýrFS over Ceph and HDFS with small files is clearly visible on the application completion time, allowing it to retain at least 73% of its performance when using 32KB instead of 32MB files. Ceph and HDFS in similar conditions retain less than 5% of their performance, hence wasting costly resources. Overall, the impact of small input files is significantly lower for all three systems with Sort and Tokenizer applications because of the higer influence on writes, for which all three systems show comparable performance. The write-intensive Tokenizer application gives a 4% advantage to Ceph because of its slightly higher write throughput compared to TýrFS.

VIII. DISCUSSION

In this section we discuss the main features and design choices of TýrFS. We detail which parts of its design impose limitations that makes it unfit for several use cases.

A. Is TýrFS also suitable for large files?

TýrFS design is specifically designed to benefit use cases involving large numbers of small files. Yet, while not the scope of this paper, no fundamental part of its design make TýrFS unfit for large files. It inherits from Týr the ability to handle very large files efficiently, by leveraging chunking. The metadata distribution in TýrFS will however be less effective with large files. The main reason is that the metadata for any given file is co-located only with the first chunk of the file. Reading a different chunk will trigger a remote metadata fetch for the file, which is unnecessary for small files.

B. Which use-cases is TýrFS unfit for?

Centralized metadata management offers many possibilities for managing the data saved in the system. For example, data reduction using deduplication is a well known technique to reduce storage resources. Efficient file copy at the metadata level is another technique often used to substantially increase performance of copy-on-write data sets compared to traditional file systems. Automatic data tiering could allow to balance data over multiple storage mediums depending on a given set of rules. The predictable data distribution based on consistent hashing makes all these techniques significantly more challenging to implement in TýrFS compared to other file systems, in which the metadata fetch happens prior to the data fetch.

C. What about fault tolerance?

None of the experiments presented in this paper did leverage fault-tolerance, which is outside the scope of this paper. TýrFS has been designed with this requirement in mind. In particular, data is fault-tolerant natively thanks to the data replication built inside Týr. Although not implemented in our prototype, the replication of the file metadata could be performed easily. In that context, we would replicate the metadata for any given file on all nodes storing a replica of the first chunk for that file. The replicated metadata do not need to be fault-tolerant.

D. How does one configure TýrFS?

The two configuration parameters specific to TýrFS are relative to the metadata replication module.

The size of the measurement sliding window needs to be carefully chosen. Indeed, it poses a tradeoff between responsiveness and accuracy of the measurement system. Setting this value too high would cause a relatively high replication latency because of the higher number of requests needed for a value to be part of the top-k values on each node. Setting it too low causes the popularity estimation to be performed on little data, hence reducing accuracy of the measurement. In practice, this value should be low compared to the average lifetime of applications, but high enough to capture a sufficient amount of requests. In our experiments, we measured 1 minute to provide a correct balance.

The second important parameter is the size of the output top-k estimator. Specifically, a larger k will enable more values to be monitored at the cost of memory. A smaller k would reduce the amount of monitored values at the expense of accuracy. In our experiments and in all traces we could study, most of the I/O is concentrated on a very small portion of a file system at a time. This benefits the metadata replication system, which is able to provide high metadata replication efficiency with relatively low values of k. In all our experiments, a value superior to 20 did never yield significant performance improvements. For larger deployments shared between many users, setting a higher k value could lead to increased efficiency of the metadata replication. This should be determined empirically based on the specifics and estimated load of each deployment.

IX. CONCLUSION

Small files are known to be a significant challenge for most distributed file systems. Indeed, the hierarchical nature of this storage model often commands separating the metadata from data in the cluster. With small files, the cost of fetching metadata easily dominates the cost of actually fetching the data. This is especially problematic when dealing with very large amounts of small files, in which over 90% of the I/O time can be dedicated to fetching metadata. For use-cases reading many small files at large scale, it is critical to ensure efficient metadata management. In this paper we argue that this cannot be done without rethinking data and metadata management in a distributed file system.

We propose TýrFS as a solution to the problem. TýrFS enables clients to locate any piece of data in the cluster independently of any metadata server. A dynamic metadata replication module adapts to the workload to efficiently replicate the necessary metadata on nodes from which the associated data is frequently accessed. Clients can then read frequently accessed data directly, without involving additional servers. We did implement a prototype of TýrFS over the Týr storage system. Experiments on the Grid'5000 testbed showed a performance improvement of more than an order of magnitude with smaller file sizes compared to state-of-theart storage systems. These improvements are applicable both to HPC and Big Data Analytics use cases.

Future work includes further improving the design of TýrFS for enabling writes to small files at high speed to provide a complete solution for managing small files at scale. We plan to evaluate TýrFS on large-scale platforms, facing real-world users and applications.

Fig. 1 .Fig. 2 .

 12 Fig. 1. Lustre and OrangeFS read throughput for a fixed-size data set when varying the size of the individual files it is composed of.

Fig. 5 .

 5 Fig.5. High-level overview of the proposed architecture. Each node integrates both a data and a metadata service. The metadata service itself is composed of a local metadata store and the dynamic metadata replication service.

Fig. 6 .Fig. 7 .

 67 Fig. 6. TýrFS, Lustre and OrangeFS read throughput for a fixed-size data set, varying the size of the individual files it is composed of.

Fig. 10 .Fig. 11 .

 1011 Fig. 10. TýrFS, Ceph and HDFS read throughput for a fixed-size data set, varying the size of the individual files it is composed of.

Fig. 12 .

 12 Fig. 12. completion time difference with 32 MB input files relative to that obtained with 32 KB input files, with TýrFS, Ceph and HDFS. 100% represents the baseline performance for 32 MB files.

  Algorithm 1 High-level algorithm of the metadata resolver.

	function RESOLVEFOLDERMETADATA(path)
	let current window be the current top-k window
	INSERT(path, current window)
	if path is available locally then
	return the locally-available metadata for path
	end if
	md ← REMOTEFETCHMETADATA(path)
	if path in output of current window then
	SUBSCRIBEMETADATAUPDATES(path)
	keep md in the local metadata store
	end if
	return md
	end function

ACKNOWLEDGEMENTS

This work is part of the "BigStorage: Storage-based Convergence between HPC and Cloud to handle Big Data" project (H2020-MSCA-ITN-2014-642963), funded by the European Commission within the Marie Skłodowska-Curie Actions framework. It is also supported by the ANR OverFlow project (ANR-15-CE25-0003). The experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER, several universities as well as other organizations. The authors would like to thank Álvaro Brandon (Universidad Politécnica de Madrid) for his precious expertise with Spark experiments.

B. Evaluating performance in a multi-user, shared context

In the previous experiment we did simulate a single-user use-case, where a single application is free to use all system resources. Accessing the file system in these conditions is especially favorable to TýrFS, as the metadata replication is dedicated to a single user and is able to quickly adapt to this application. Yet, it is typical for HPC platforms to offer a single, centralized file system in which all users share a common pool of resources. This is a significantly more complex problem.

We simulate a multi-user setup by replaying on the file system real-world I/O traces from the Second Animation Dataset [START_REF] Anderson | Capture, conversion, and analysis of an intense NFS workload[END_REF], representative of a data-intensive I/O workload from multiple clients. We dedicate 24 more clients to this task, each replaying a share of the 3.2 TB trace file at maximum rate. We concurrently execute on 24 other nodes the same benchmark as in the previous experiment, and measure the achieved throughput across all systems varying the size of the input files between 32 KB and 32 MB each. We limit the dynamic replication storage on each node to a low number: 20 replication slots shared across all concurrent users.

In Figure 8 we plot the achieved throughput of our benchmark in these conditions. Similarly to what we observe with single-user experiments, TýrFS significantly outperforms Lustre and OrangeFS as the size of the input files is reduced. Compared to the previous experiment, the shared nature of the file system results in a lower achieved throughput across all systems. TýrFS in particular shows a higher performance reduction at smallest file sizes due to the saturation of the file system nodes. Interestingly, we see that reducing the file size of the input data set from 32 MB to 8 MB in Týr causes a noticeable 8% throughput increase. This is because smaller file sizes involve a higher number of metadata requests for the input file set, which in turn increase the likelihood of the metadata for the data set access path to be dynamically replicated across the whole cluster. This effect is amplified by the low limit we put on the replication store size for each node. These results demonstrate the applicability of TýrFS for shared, highly-concurrent file systems, while additionally confirming that the dynamic metadata replication we propose especially benefits small files use-cases.

C. Ensuring horizontal scalability

Horizontal scalability is a critical feature of large-scale systems targeted at data-intensive applications. It allows the system to scale out, increasing its total performance by adding commodity machines to the cluster. Týr, the base building block of TýrFS, has a demonstrated near-linear horizontal scalability that makes it particularly suitable for large-scale use-cases. TýrFS does not introduce any centralization that could jeopardize these features. In particular, it inherits from OrangeFS the distributed metadata management principles, which support the horizontal scalability of this system. As such, we argue that TýrFS should be able to inherit this desirable feature.

In this section we formally demonstrate the horizontal scalability of TýrFS. We adopt the same setup as in Section VII-A. We vary the number of storage nodes from 2 to 64 on the Paravance cluster. We setup clients on up to 24 nodes of a second Grid'5000 cluster: Parasilo. Both clusters are colocated in the same facility, and are interconnected with lowlatency 2 x 40G Ethernet links aggregated using the Link Aggregaton Control Protocol (LACP). Each client node runs 64 ranks of the MPI read benchmark that concurrently reads files from our random, pre-loaded 1.22 TB data set. We run the benchmark for 5 minutes, reading the dataset from within an endless loop. The dataset we use in this experiment is composed of 3.84 × 10 7 × 32 KB files.

In Figure 9 we plot the measured aggregated throughput of TýrFS across all clients, varying the number of nodes from 2 to 64 and the number of concurrent clients from 64 to 1,536. The results demonstrate that TýrFS shows a near-linear horizontal scalability when adding nodes to the storage cluster, being able to serve more clients with higher aggregate throughput. The decrease in throughput observed with the smallest cluster sizes when increasing the number of clients is due to the saturation of the storage servers. In this configuration, and despite working on a data set composed of very small files, we measured the throughput of TýrFS as high as 36 GB / s across all clients with 64 storage servers and 1,536 concurrent client ranks.


Nathanaël Cheriere
email: nathanael.cheriere@irisa.fr

Matthieu Dorier
email: mdorier@anl.gov

Gabriel Antoniu
email: gabriel.antoniu@inria.fr

Pufferbench: Evaluating and Optimizing Malleability of Distributed Storage

Keywords: Distributed Storage System Malleability, Benchmark, Pufferbench

Malleability is the property of an application to be dynamically rescaled at run time. It requires the possibility to dynamically add or remove resources to the infrastructure without interruption. Yet, many Big Data applications cannot benefit from their inherent malleability, since their colocated distributed storage system is not malleable in practice. Commissioning or decommissioning storage nodes is generally assumed to be slow, as such operations have typically been designed for maintenance only. New technologies, however, enable faster data transfers. Still, evaluating the performance of rescaling operations on a given platform is a challenge in itself: no tool currently exists for this purpose.

We introduce Pufferbench, a benchmark for evaluating how fast one can scale up and down a distributed storage system on a given infrastructure and, thereby, how viably can one implement storage malleability on it. Besides, it can serve to quickly prototype and evaluate mechanisms for malleability in existing distributed storage systems. We validate Pufferbench against theoretical lower bounds for commission and decommission: it can achieve performance within 16% of them. We use Pufferbench to evaluate in practice these operations in HDFS: commission in HDFS could be accelerated by as much as 14 times! Our results show that: (1) the lower bounds for commission and decommission times we previously established are sound and can be approached in practice; (2) HDFS could handle these operations much more efficiently; most importantly, (3) malleability in distributed storage systems is viable and should be further leveraged for Big Data applications.

I. INTRODUCTION

Malleable applications or services, supporting the possibility to be dynamically rescaled at runtime, are ideal for an execution on cloud platforms. The core principle of cloud computing is to enable users to quickly get as many resources as they need and to release some when they are not needed. Thus, users have the opportunity of constantly optimizing performance and cost with respect to changing workloads. This is typically refered to as cloud elasticity. When taking a closer look at how it is applied in practice today, it can be noticed that in most cases cloud users have the possibility to rent a fixed number of resources for a certain of time specified in advance, then they run their service or application; in practice, when they need more resources, they often have to stop their service, rent more resources, then redeploy the service on the new configuration. That is, there is still a gap to fill to adequately support malleable applications and services through dynamic resource rescaling without service interruption.

To illustrate this limitation, let us focus on Big Data frameworks such as Hadoop or Spark, which are malleable by design. They often however rely on a distributed storage system (DSS) that is not malleable. This lack of malleability leaves the user with two options: 1) deploy the DSS in a static set of resources that is separate from those running the data processing tasks, which prevents the application from performing any local data accesses; 2) deploy the DSS within the application's resources on a static set of resources, which prevents any kind of malleability altogether.

The reason why malleability has not been widely adopted for DSS is that it involves migrating large amounts of data, which is potentially long. A DSS generally has to ensure three properties when malleability is used: 1) no data can be lost, 2) fault tolerance must be guaranteed, and 3) the placement of data across the storage nodes should be balanced. Thus, in general, DSS malleability is assumed to be too slow for practical use and is traditionally reserved for maintenance purposes. Yet, today, thanks to many technology improvements, data transfers are not as slow as they used to be.

To know whether DSS malleability would be useful for a given workload or on a given platform, one should be able to estimate the duration of both commission (adding nodes) and decommission (releasing nodes) operations. Indeed, having quick operations for the malleability is essential to benefit from the malleability. Quickly decommissioning unused nodes reduces the overall cost of running an application, while quickly commissioning nodes allows the amount of resources to dynamically adjust to a volatile workload.

In previous work [START_REF] Cheriere | How Fast Can One Scale Down a Distributed File System?[END_REF], [START_REF] Cheriere | A Lower Bound for the Commission Times in Replication-Based Distributed Storage Systems[END_REF], 1 we have provided theoretical lower bounds for the duration of rescaling operations (the commission and decommission operations). These lower bounds are useful because they indicate whether the malleability would be too slow on a given platform (i.e., in the case these lower bounds are too high from a practical perspective). However, if the lower bounds are acceptable, it is still difficult to know in practice how fast malleability can really be supported on a specific platform. For this, one must deploy an actual DSS, determine an efficient configuration for the rescaling operations, generate data, and record the duration of the operations. Moreover, the obtained results may not be accurate since rescaling operations implemented in current DSS are often not optimized for speed but to limit their impact on application execution.

In this paper, we address the problem of evaluating malleability by introducing Pufferbench,2 a modular benchmark developed to efficiently measure the duration of commission and decommission operations on a given platform. To this purpose, Pufferbench emulates a DSS, executing only the inputs and outputs needed for a rescaling operation.

Pufferbench has been designed with two goals in mind:

1) Evaluate the viability of DSS malleability on a given platform. Pufferbench provides the duration of rescaling operations on a platform, regardless of the DSS that would use these operations. 2) Help optimize migration mechanisms in order to improve the malleability of a specific DSS. Pufferbench is independent from any DSS and thus can be used to quickly prototype and test custom data migration mechanisms (algorithms, network transfers, storage management) on a simpler code before implementing them into a real DSS.

In the second case, Pufferbench also verifies the correctness of the custom migration algorithms by checking that the postconditions (number of replicas, data distribution, etc.) are satisfied.

First, we validate Pufferbench against the lower bounds established in our previous works. According to our evaluation, Pufferbench achieves performances that are on average only 16% slower than the lower bound for the decommission and 7% for the commission. From these results, we also conclude that the previously determined lower bounds are sound.

In a second set of experiments we illustrate the usefulness of Pufferbench by using it to measure how fast the rescaling operations of Hadoop's distributed storage system (HDFS) could be if they were optimized for speed. Experiments show that even if the performance of Pufferbench varies greatly depending on the operation (commission or decommission) and the type of storage (in RAM or on disk), it is always faster than HDFS. This suggests that the algorithms used for commission and decommission in HDFS can substantially be improved. In particular, for the commission operation, HDFS can be sped up by as much as a factor 7.

The remainder of this paper is organized as follows. In Section II we introduce the context and the related work. Pufferbench is presented in Section III and validated in Section IV. In Section V we detail how we used Pufferbench to measure how fast the rescaling operations of HDFS could be, and we present the experimental results in Section VI. We discuss the results in Section VII and conclude in Section VIII.

II. CONTEXT

Malleability has gained interest because it improves resource utilization on shared cluster-based platforms while saving energy and money. Many frameworks provide support for malleability of computing resources [START_REF] Vadhiyar | SRS: A Framework for Developing Malleable and Migratable Parallel Applications For Distributed Systems[END_REF], [START_REF] Kale | A Malleable-Job System for Timeshared Parallel Machines[END_REF], [START_REF] Buisson | A Framework for Dynamic Adaptation of Parallel Components[END_REF]. Big Data processing frameworks such as Hadoop [START_REF] White | Hadoop: The Definitive Guide[END_REF] and Spark [START_REF] Zaharia | Spark: Cluster Computing with Working Sets[END_REF] are malleable: new computation nodes can be added and removed without having to shut down the framework. However, the lack of malleability of the underlying storage system greatly limits the overall malleability of the application. In Hadoop's case, the underlying HDFS storage system is not malleable; thus the malleability of Hadoop can hardly be used in practice. The reserved cluster cannot be smaller than the size of the HDFS cluster; thus, shrinking the Hadoop deployment less than this size is pointless since HDFS and Hadoop are sharing the same nodes. Conversely, the deployment cannot grow too large without incurring a high load on HDFS and without losing the benefits of data locality. This limitation is not specific to HDFS. DSS need to be able scale to up and down to ensure consistent performance especially when the applications using them are rescaled.

A. Related Works

Most distributed and parallel file systems such as Ceph [START_REF] Weil | Ceph: A Scalable, High-Performance Distributed File System[END_REF] or HDFS [START_REF] Shvachko | The Hadoop Distributed File System[END_REF] include both commission and decommission operations, mainly for maintenance purposes. Their rescaling operations are optimized mostly to reduce their impact on the performance of the applications, rather than to reduce the duration of the operation (which can be understood).

A few DSS are built around a restricted form of malleability: they have a pool of machines available but can shut some of them down to save energy. Three systems have been developed with this strategy: Rabbit [START_REF] Amur | Robust and Flexible Power-Proportional Storage[END_REF], Sierra [START_REF] Thereska | Sierra: Practical Power-Proportionality for Data center Storage[END_REF], and SpringFS [START_REF] Lianghong | SpringFS: Bridging Agility and Performance in Elastic Distributed Storage[END_REF]. They have a common limitation: the shutdown nodes are not fully released. They host data on their permanent storage drive that will be updated when they rejoin the storage cluster, and they must be available for fault tolerance.

The SCADS Director [START_REF] Trushkowsky | The SCADS Director: Scaling a Distributed Storage System under Stringent Performance Requirements[END_REF] is a resource manager designed to ensure service-level objectives. It chooses to add or remove nodes, where and when to move data, and the number of replicas each file needs. The SCADS Director adds malleability to the SCADS file system [START_REF] Armbrust | SCADS: Scale-Independent Storage for Social Computing Applications[END_REF]. Its authors have focused their evaluation on the point of view of applications that use their storage system. They evaluate whether their system is able to follow the workload to scale up and down and whether performing data migration has an impact on the service-level objectives. They do not, however, evaluate the performance of the rescaling operations themselves. Their goal is to be the least impactful to users, not necessarily the fastest.

Lim, Babu, and Chase [START_REF] Lim | Automated Control for Elastic Storage[END_REF] propose a resource manager based on HDFS. This resource manager chooses when to add and remove nodes and the parameters of the rebalancing operations. However, it simply uses HDFS as it is and does not focus on its efficiency. Both [START_REF] Trushkowsky | The SCADS Director: Scaling a Distributed Storage System under Stringent Performance Requirements[END_REF] and [START_REF] Lim | Automated Control for Elastic Storage[END_REF] focus on ways to leverage malleability rather that on improving it. They are therefore orthogonal and complementary to this work.

B. Challenge

The first step towards implementing real, efficient malleability in existing or in future DSS is to demonstrate that it will be fast enough to be used in practice. It is difficult, however, to evaluate the performance of data migration operations on a given platform. In our previous work [START_REF] Cheriere | How Fast Can One Scale Down a Distributed File System?[END_REF], [START_REF] Cheriere | A Lower Bound for the Commission Times in Replication-Based Distributed Storage Systems[END_REF], we provided lower bounds for the duration of these operations; however, these lower bounds are based on strong hypotheses: uniformity of the hardware, perfect load balancing, absence of latency, etc. These hypotheses are rarely met in practice. To accurately measure the duration of the rescaling operations, an evaluation on the actual target platform is needed.

The process of evaluating the performance of malleability with a real DSS is time consuming and not necessarily accurate: the rescaling operations of existing DSS may not be optimized for speed nor for the needs of the workload and the platform.

To address these needs, we introduce Pufferbench, a benchmark specifically designed to improve the process of evaluating storage malleability on cluster-based platforms. Pufferbench is not a distributed file system, nor does it rely on a specific one.

III. PUFFERBENCH

A. Overview of Pufferbench

Pufferbench is implemented as an MPI application that emulates the rescaling operations of a DSS by doing all I/O operations that are needed during such a rescaling operation: data accesses to/from a local storage device (which may be local memory) and across the network. It is a master/workers application with MPI rank 0 acting both as master and worker and all other ranks acting as workers.

Its execution involves four steps. 1) Migration planning: Pufferbench's master node applies the migration algorithm chosen in its configuration file to compute the sequence of I/O operations (writing/reading to/from local device, sending/receiving to/from other nodes) that each node needs to execute to complete the migration. The trace of I/Os is then sent to worker nodes to be replayed. 2) Data generation: All nodes running Pufferbench generate data on their local storage device to be able to read from their local storage. 3) Execution: All Pufferbench nodes execute their respective sequence of I/O, recording timing and statistics. In particular, the overall duration of the rescaling operation is measured. 4) Statistics aggregation: Statistics collected by each node are gathered by the master node and output to the user. While executing the data migration, Pufferbench makes sure that the following properties are met:

1) No data is ever lost.

2) The data replication factor is the same before and after the migration.

3) The distribution of data across nodes remains balanced. Hence Pufferbench not only evaluates the performance of rescaling operations, it also assesses the correctness of the migration algorithm.

B. Customizable Components

In order to match as many platforms and systems as possible, Pufferbench has been designed with modularity in mind. With a simple change in its configuration file, the main components of the system can be switched for custom ones.

1) Master node components: Three of these components are used exclusively by the master node (Fig. 1). In order of action, they are the following.

1) The MetadataGenerator generates the basic metadata of the dataset initially present on the emulated storage system. This set of metadata takes the form of a set of pairs (object id, size). Tuning this component enables choosing between various data sizes (e.g., many small objects, few large objects, random size uniformly distributed across a range, gaussian). By changing this component for a custom one, users can plug in data sizes that best match their workload.

2) The DataDistributionGenerator takes this metadata set as input and assigns each object to as many virtual storage nodes as necessary to meet the required replication factor. The output of this component is a data distribution map associating each virtual node id with the list of (object id, size) that this virtual node manages. Two implementations of this component are provided by default: the first one places data randomly across the nodes, ensuring only the replication factor; the second one balances the load across the nodes. Changing this component enables matching a particular DSS's placement policy or evaluating new ones.

3) The DataTransferScheduler is the core of Pufferbench.

It takes as input the previously generated placement map as well as the desired migration (e.g. "commissioning 3 nodes") and produces a sequence of I/O operations (read, write, send, receive) that each virtual node has to replay in order to accomplish this migration. The default DataTransferScheduler redistributes the data randomly but maintains a level of load balancing and carefully chooses the nodes reading and sending the data in order to mitigate the bottlenecks of the operations: receiving and writing the data for the decommission, and reading the data for the commission. Customizing the DataTransferScheduler allows the user to test new migration algorithms and evaluate their performance before implementing them in a real DSS. Those components are responsible for the simulation of various DSS. For example, one can simulate HDFS with a MetadataGenerator that generates mostly chunks of 128 MiB and a DataDistributionGenerator that will replicate the chunks three times and place them onto random nodes, as HDFS does.

In addition to these three components, the master node includes a DataDistributionValidator component that cannot be customized. This component takes as input a data distribution map that represents the placement of objects on the nodes. From this map, it checks that the data migration is valid, that is, that it respects the requirements listed in Section III-A. This validation is done twice in order to control the data distribution maps: before and after the execution of the DataTransferScheduler. Running the DataDistributionValidator twice ensures that the customized DataDistributionGenerator and DataTransferScheduler are both satisfying their requirements.

To evaluate the validity of a particular migration algorithm, Pufferbench can be executed on a single node and stop at the validation step instead of effectively replaying the I/O operations.

2) Worker nodes components: Three components are used by all nodes (including the master) to replay the I/Os.

1

C. Using Pufferbench to Evaluate a Platform

Pufferbench can be used with its default components to evaluate the potential for storage malleability on a given platform. This can be done with a simple default configuration, which uses the migration algorithms described in Section IV. Pufferbench replays all I/Os needed to commission or decommission nodes on the target platform. The duration of each operation is recorded along with various other metrics.

Since the measurements are done on a real platform, the recorded performance is reachable by any DSS available on that platform, provided that DSS is optimized for the rescaling operations.

Some platforms feature heterogeneous hardware. While the current version of Pufferbench does not handle such a case, we plan to add it in the future.

D. Using Pufferbench to Evaluate Data Migration Algorithms

The modularity of Pufferbench allows any user to evaluate and optimize the algorithms used for the rescaling operations (commission and decommission). Thus, Pufferbench can easily be used to optimize and evaluate data migration mechanisms in an existing DSS without modifying it.

Users can plug in custom algorithms for data migration, which can replace the default ones provided by Pufferbench. To this purpose, Pufferbench embeds a component that checks that the plugged-in algorithm yield a valid migration plan. In particular, it checks the replication factor of all objects and evaluates the load balancing of the final distribution by computing the average, minimum, maximum, median, and standard deviation of the amount of data held by each node and the same set of statistics of their number of objects. These statistics enable the user to check whether the final data distribution is more, less, or equally load balanced than the original one.

Moreover, writing commission and decommission algorithms can be done in significantly fewer lines of codes than in an actual DSS because of the abstraction. For instance, the commission and decommission algorithms used in the following section are written in 350 lines of C++ overall.

IV. VALIDATION AGAINST THE LOWER BOUNDS

In this section, we evaluate the performance of Pufferbench against the lower bounds established for the rescaling operations in previous works [START_REF] Cheriere | How Fast Can One Scale Down a Distributed File System?[END_REF], [START_REF] Cheriere | A Lower Bound for the Commission Times in Replication-Based Distributed Storage Systems[END_REF].

A. Experimental Setup

All measurements were done on the French Grid'5000 [START_REF] Balouek | Adding Virtualization Capabilities to the Grid'5000 Testbed[END_REF] experimental testbed. Experiments on decommission were done on the paravance cluster in Rennes, while experiments on commission were done on the grisou cluster in Nancy. Both clusters feature the same type of node: Dell PowerEdge R630 with Intel Xeon E5-2630 v3 Haswell 2.40 GHz (2 CPUs/node, 8 cores/CPU), 128 GiB of RAM, and two 558 GiB HDD. They are all connected with a 10 Gb/s Ethernet network to a common Cisco Nexus 6000 switch (for paravance) or a Cisco Nexus 9508 (for grisou).

Pufferbench emulates a DSS that initially hosts 50 GiB per node. Ten measurements per configuration of Pufferbench were done. The results are represented by box plots showing the minimum, the first quartile, the median, the third quartile, and the maximum duration of the rescaling operation.

B. Evaluation of Pufferbench against the Lower Bounds

The lower bounds [START_REF] Cheriere | How Fast Can One Scale Down a Distributed File System?[END_REF], [START_REF] Cheriere | A Lower Bound for the Commission Times in Replication-Based Distributed Storage Systems[END_REF] for the duration of the rescaling operations have been built on strong hypotheses, such as an all-to-all network topology and the absence of latency in disk accesses. In practice, these hypotheses are not met. The following sections describe to what extent our experimental setup respects the hypotheses. In order to safely evaluate any result against theoretical lower bounds, the experimental conditions should be such that practical constraints only increase (and never decrease) the duration of the rescaling operations.

1) Hypotheses on the hardware:

• Cluster homogeneity: The cluster should be composed of identical nodes. In practice, although the clusters we used are homogeneous, performance variations can be observed across nodes. For instance, the maximum bandwidth of the drives while reading varies between 195 MiB/s and 207 MiB/s on the cluster used for the experiments. The parameters of the lower bound (maximum disk read speed, maximum disk write speed, and maximum network bandwidth) have been set with the maximum measured values prior to the experiments. • Ideal network: The theoretical lower bounds ignore the network latency and the potential interference that can happen. An all-to-all topology is assumed, with identical bandwidth any two nodes. In practice, all the nodes of our clusters are connected to a common switch through a 10 Gb/s Ethernet link. Hence this switch may become a bottleneck. • Ideal storage backend: The theoretical lower bounds ignore the latency (seek times of the drives) and assume that the drives always read and write at their maximum speed. In practice, seek times and read/write contention add to the duration of the execution. Although the theoretical hypotheses are not met, the differences between the experimental setup and the hypotheses only increase the duration of the commission and decommission operations. This ensures that the lower bounds keep their property of lower bounds even with the relaxed hypotheses.

2) Hypotheses on the components: With the components, we can make sure that the other hypotheses and objectives of the lower bounds are met.

The DataDistributionGenerator ensures the other three hypotheses of the lower bounds:

• Load balancing: The nodes host the same (or similar) amount of data. • Data replication: Each object stored in the cluster is replicated r times. • Uniform data distributions: Each set of r distinct nodes has some data that is replicated only on these r nodes.

The amount of such data should be the same for every such set. The DataTransferScheduler, which implements the migration algorithms, ensures that the following objectives are met at the end of the rescaling operation:

• No data loss: No data is lost during the operations. • Maintained data replication: The replication factor is the same as the initial one. • Load balancing: The nodes host the same amount of data. The last objective, which is to have uniform data distribution at the end of the rescaling operation, is relaxed since it is achieved with random data placement. This does not impact the lower bound in most cases except when, during the commission, reading from disk becomes the bottleneck (commission of more than 22 nodes in the following measurements). However, the randomness ensures that the data distribution generated is close to the objective.

Overall, the experimental setup is such that the lower bounds are valid for the average duration of the commission and decommission. The lower bounds can safely be used to evaluate the performance of Pufferbench configured as detailed in this section.

C. Results

Figure 2 shows the performance of Pufferbench against the lower bounds when decommissioning nodes from a cluster of 20 nodes. Each node initially hosts 50 GiB of data in memory. On average, the decommission times of Pufferbench are 16% longer than the lower bound.

Figure 3 presents the decommission with storage on disk. On average, Pufferbench is 11% slower than the lower bounds.

In both cases, the difference between the lower bounds and Pufferbench is due to the fact that Pufferbench runs on real hardware. The lower bounds consider only the maximum bandwidth of the network and the storage. They ignore the latency and any interference. Pufferbench takes all this into account as it replays all I/Os needed to decommission nodes.

Figure 4 shows the performance of Pufferbench when commissioning nodes into a cluster of initially 10 nodes with storage in memory. On average, Pufferbench is 7% slower than the lower bounds.

Figure 5 presents the results of the commission when the storage is on disk. In this case, Pufferbench is 16% slower than the lower bounds.

The main reason for the difference in performance between the storage in memory and the storage on disk drives is the latency to access data on disks, as well as the disks not exhibiting uniform performance across nodes and across requests (peak read speed varied between 195 MiB/s and 207 MiB/s across nodes).

In both cases, we observe that when 20 nodes are added, the difference between the lower bound and the results of Pufferbench is the largest. This is due to the fact that stragglers appear. In this case, the effect of the stragglers is clear because all the nodes become bottlenecks: the 20 new nodes must finish writing their data in about 15 s, but the old nodes must also finish reading and sending their data in 15 s. Thus, any node straggling has an impact on the overall performance. Stragglers appear because of the variability of hardware performance.

We show that Pufferbench is able to emulate rescaling operations that are close to the theoretical lower bounds (at most 16% of difference on average). Such a difference with the theoretical lower bounds can be attributed to the fact that the hardware does not match the hypotheses of the model. Moreover, these results also show that the lower bounds themselves are realistic and that the optimal duration of the rescaling operations are within 16% of the lower bounds.

V. USE CASE: HDFS

In the following subsections we evaluate the performance of commission and decommission operations in HDFS and compare the results with Pufferbench in order to assess whether HDFS's rescaling algorithms could be optimized.

A. Goals

We focus on HDFS [START_REF] Shvachko | The Hadoop Distributed File System[END_REF] because it is a DSS that is widely used and that has both the commission and decommission operations already implemented. These operations are used in practice mainly for maintenance. In our previous works, we showed that the rescaling operations of HDFS could be greatly accelerated, provided that the estimated lower bound is correct. For example, in the case of decommission, with the storage done on disk, HDFS is three times slower than the lower bound. With Pufferbench, we now have a tool to confirm this in practice.

In this section and the following, our goal is to evaluate how fast rescaling operations could be under the constraints of HDFS. In particular we show that, with a different migration mechanism (algorithms, data transfers, and disk management), HDFS could be much faster at commissioning and decommissioning nodes. Thus, we implement in Pufferbench rescaling mechanisms moving the same objects (chunks of 128 MiB) and aiming for the same final data distribution. Doing so, we can compare the performance of the rescaling operations of HDFS and those of Pufferbench as they produce the same results from the same initial situation.

Note that we did not recreate the commission and decommission mechanisms of HDFS in Pufferbench for a simple reason: the algorithms used by HDFS are dynamic. Data transfers are rescheduled every few seconds to match the actual progress, while Pufferbench assumes that all transfers are scheduled at the beginning of the rescaling operation.

B. How Pufferbench Emulates HDFS

Because we want to create a data distribution similar to that of HDFS, the MetadataGenerator and DataDistributionGenerator components in Pufferbench are configured to generate 128 MiB chunks of data that are replicated three times across randomly selected nodes.

1) New decommission algorithm: Since the data migration mechanism used by HDFS is not optimized for speed, we implemented our own DataTransferScheduler in Pufferbench to try to achieve the best performance rather than imitating HDFS's algorithm.

As shown in [START_REF] Cheriere | How Fast Can One Scale Down a Distributed File System?[END_REF], the bottleneck in the decommission is receiving and writing the data to storage. Indeed, thanks to data replication, not only do the nodes being decommissioned have the data, some other nodes have them as well.

The amount of data written on each new node is determined by the data placement, which is random but with some load balancing. Nodes are classified as either overloaded or underloaded. Nodes being decommissioned are overloaded since they should host no data. Replicas are randomly moved from overloaded nodes to underloaded ones, provided that no two replicas of the same chunk end up on a same node.

When using in-memory storage, no optimization can be made: each node has data to write and can read data simultaneously without interference. In the case of on-drive storage, each replica must be read once and then forwarded to nodes on which it should be written. The replica can be read from any node hosting it, not only in-decommission nodes. Thus, some drive load-balancing is done to choose which node has to read data: all nodes should have balanced amounts of disk I/Os (taking into account the fact that disks from in-decommission nodes will only read, while others can read and write).

2) New commission algorithm: As shown in [START_REF] Cheriere | A Lower Bound for the Commission Times in Replication-Based Distributed Storage Systems[END_REF], two bottlenecks arise during the commission operation: reading the data from the old nodes and writing it to the new nodes.

The data placement is the same as for the decommission.

The main bottleneck that can be mitigated during the commission is when reading data from the nodes initially present. Since existing nodes initially have all the data, each replica to be moved is read once from an existing node and then exclusively forwarded between new nodes.

3) Replay components: The Storage and Network components are the ones provided by default in Pufferbench (inmemory and disk-based for Storage, MPI-based for Network). We disabled caching in the underlying local file system in order to match the configuration used by HDFS.

Compared with HDFS, there is a large improvement in the disk I/Os done by the Storage component. HDFS reads and writes data onto the drives by blocks of 4 KiB and lets the file system cache optimize the operations. The Storage component in Pufferbench buffers the data until the data for the whole object is receive and then writes it. Overall, Pufferbench's storage component writes and reads larger chunks to/from the drives, optimizing the drive bandwidth usage.

VI. EXPERIMENTAL RESULTS

In this section, we compare the performance of HDFS and Pufferbench emulating an optimized HDFS during rescaling operations as presented in Section V.

A. Experimental Setup

The experimental platform is the same as the one presented in Section IV-A.

HDFS and Hadoop 2.7.3 were deployed on the nodes. The replication factor is left unchanged to 3. The configuration of HDFS was adjusted in order to remove the limits on the bandwidth usage of the rescaling operations. The commission is divided in two steps. First, new HDFS workers join the cluster; then a rebalancing operation is started. For the decommission, the built-in mechanism is used. The initial size of the cluster is 20 for the decommission operations and 10 for the commission operation. All nodes in the initial clusters host 50 GiB of data before each rescaling operation.

The lower bounds are different depending on whether the network or the storage is the bottleneck for the data transfers. In order to create a situation in which HDFS has a storage bottleneck, the data is simply stored on the disks with the file system's cache limited to 64 MiB. For the situation in which the network is the bottleneck, the data of HDFS was stored on a RAMDisk, effectively storing all the data in memory.

Each measurement was repeated 10 times except for the commission with storage on drive, for which time constraints limited the number of repetitions to 5.

B. Potential Speed of Decommission in HDFS

Figure 6 presents the results of the decommission when the data is stored in memory. Pufferbench is faster than HDFS. HDFS is on average 23% slower than Pufferbench and up to 40% slower in some cases. The difference with the lower bound is due mainly to the initial load-balancing hypothesis (the nodes should all host exactly 50 GiB) not being met. Because HDFS and Pufferbench randomly distribute the data on the nodes, initial load balancing is indeed not guaranteed. Thus, some nodes receive more data than determined by the lower bound. For instance, a node can initially host 45 GiB of data instead of 50 GiB and have to host 60 GiB at the end of the decommission. Because of the initial load imbalance, this node will have to receive 15 GiB instead of 10 GiB, and the decommission will take longer.

In the case where data is stored on drive, Pufferbench is able to decommission nodes faster than can HDFS (Fig. 7). On average, HDFS decommissions nodes 2.8 times more slowly than does Pufferbench. The problem of initial load imbalance observed in Figure 6 is still present, but the migration scheduler of Pufferbench configured for these experiments mitigates it when balancing the read load across drives. The drives that have more data to write will spend less time reading.

C. Potential Speed of Commission in HDFS

In the case of the commission with storage in memory (Fig. 8), Pufferbench is on average 8 times faster than HDFS and up to 14 times when a large number of nodes are added at the same time.

Note that the commission mechanism in HDFS is not optimized for speed but is made to minimize the impact of the commission on the overall cluster.

Pufferbench also exhibits good performance when commissioning nodes with storage on drive (Fig. 9). On average, Pufferbench is 2.79% faster than HDFS.

D. Low overhead of Pufferbench

Experiments with both HDFS and Pufferbench highlight another aspect about Pufferbench: it provides results quickly. Its speed is related not only to the improved rescaling operations but also to the reduced overhead.

To measure the duration of the commission with storage in memory in HDFS, one has to start HDFS, start Hadoop, generate enough data, start the commission, and stop the system. The whole operation took 39 h for the results presented in Fig. 8, with only 13 h spent commissioning nodes; it is an overhead of 26 h.

In contrast, the measurements with Pufferbench lasted for 2 h and 53 min with 2 h commissioning nodes. The overhead of Pufferbench was only 53 min, about 30 times lower than the overhead of HDFS. Indeed, Pufferbench only has to allocate some memory in order to be able to replay a commission, and it is able to quickly switch between measurements.

With the help of Pufferbench, we showed that the rescaling operations of HDFS could realistically be sped up by 23% on average in the case of the decommission with in-memory storage and by up to 8 times on average and up to 14 times in the case of in-memory commission.

VII. DISCUSSION

In this section, we discuss various points about Pufferbench and its usage.

A. Ideal Setup for the Validation of Pufferbench

The experimental to validate Pufferbench is favorable to fast rescaling operations. In particular, all objects had a size of 128 MiB in order to read, send, receive, and write large sequential chunks of data. This optimized the I/Os for both the network and the local storage.

Thus, the performance should degrade with smaller objects, and Pufferbench should then be used to optimize algorithms, storage, and network transfers to efficiently migrate small objects.

B. HDFS's Case

In the case of HDFS, we observe that the optimized algorithms for the migration are not the only factor of improvement. Pufferbench's network usage is better than HDFS, but the most important part is the backend drive usage: Pufferbench reads and writes full chunks (128 MiB) sequentially, and this approach improves the read and write bandwidth during the decommission operations by at least a factor of 2 compared to HDFS.

Moreover, we show that HDFS's migration mechanism can be sped up by a factor of as much as 14, in particular when commissioning many nodes at once.

The drive management of HDFS can be improved by taking advantage of the amount of memory available on the nodes. HDFS writes and reads data by chunks of 4 KiB. Using larger values (e.g, 128 MiB in the case of Pufferbench) is enough to greatly speed up the decommission of nodes.

Two main aspects of the commission could be improved. First, the algorithm used by HDFS easily accumulates delays: the rebalancing is scheduled by waves of data transfers, and each wave must be completed before the next one starts. This should be replaced with an algorithm that maintains a constant transfer of data between nodes. Second, some buffering should be used in order to read only once each chunk of data from the drives when sending the data to multiple destinations.

The modularity of Pufferbench allows users to test and improve easily all the relevant components used during the rescaling operations: scheduler, network, and storage.

C. Pufferbench and Lower Bounds

Compared against the lower bounds, the results obtained with Pufferbench have the advantage of providing more accurate commission and decommission times. First, it actually replays I/Os so the characteristics of the hardware (network latency, network interferences, disk seek times, disk throughput) are taken into account, but it also evaluates an implementation of the operations.

The lower bounds have the advantage of fixing what is theoretically possible: provided that the hypotheses are met (in particular the initial load balancing), the operations cannot be faster than the lower bound. The lower bound can be a good comparison point when the platform is not available, whereas Pufferbench gives more accurate results when the platform is available.

However, the results show that the lower bounds previously determined are realistic and that performance close to these lower bounds can be reached.

D. Limiting the Impact of Rescaling Operations on Application Performance

The rescaling operations of DSS are used primarily as maintenance operations: permanently increasing the size of a cluster and safely removing faulty nodes. Thus, the rescaling operations are rarely optimized for speed. In HDFS, for example, both the rebalancing (for the commission) and decommission operations have options in the configuration to limit the bandwidth they can use in order to reduce their impact on concurrently running applications.

One can implement such a limitation in Pufferbench simply by implementing a custom Network component that limits the bandwidth usage. The modularity of Pufferbench allows experimenting with multiple limitations (global disk bandwidth, disk read and/or write bandwidth, network bandwidth, network send and/or receive bandwidth, etc.).

Thus, Pufferbench can also be used to optimize other aspects of the rescaling operations. VIII. CONCLUSION Efficient rescaling operations are needed in order to use the malleability of distributed storage systems. Thus, in this work, we introduced Pufferbench, a modular benchmark with two goals. First, it can measure how fast the rescaling operations can be done in practice on a given platform. With this, the administrators of the platform can decide whether using DSS malleability worth it. Second, it can be used to fine-tune all components involved in data migration (scheduler, storage, and network). Modifying an existing DSS is a strenuous task. Pufferbench enables an easy prototyping and testing of the data migration mechanisms before implementing them in any DSS.

By validating Pufferbench against the lower bounds, we show that one can implement rescaling operations with performance within 16% of the lower bound, in practice. Moreover, this result highlights the fact that the lower bounds are realistic and that the optimal duration for rescaling operations is within 16% of the lower bound.

With Pufferbench, we show that the rescaling operations of HDFS can be greatly sped up, by as much as a factor 14 in some cases.

These results strengthen the idea that malleability in distributed storage systems is viable and should further be studied in order to benefit data-intensive applications.

Implementing efficient commission and decommission into a real distributed storage system and using the prototype to evaluate the benefits of malleability with real use cases is a challenge left for future work.

Fig. 1 :

 1 Fig. 1: Components of Pufferbench and their interactions.

Fig. 2 :

 2 Fig. 2: Time needed to decommission nodes, with storage in memory. Nodes initially host 50 GiB of data on average. Comparison with lower bound.

Fig. 3 :

 3 Fig. 3: Time needed to decommission nodes, with storage on drive. Nodes initially host 50 GiB of data on average. Comparison with lower bound.

Fig. 4 :

 4 Fig. 4: Time needed to commission nodes, with a storage in memory. Nodes initially host 50 GiB of data on average. Comparison with lower bound.

Fig. 5 :

 5 Fig. 5: Time needed to commission nodes, with a storage on drive. Nodes initially host 50 GiB of data on average. Comparison with lower bound.

Fig. 6 :

 6 Fig.6: Time needed to decommission nodes, with a storage in memory. Nodes initially host 50 GiB of data on average.

Fig. 7 :

 7 Fig. 7: Time needed to decommission nodes, with storage on disk. Nodes initially host 50 GiB of data on average.

Fig. 8 :

 8 Fig. 8: Time needed to commission nodes, with storage in memory. Nodes initially host 50 GiB of data on average.

Fig. 9 :

 9 Fig.9: Time needed to commission nodes, with storage on disk. Nodes initially host 50 GiB of data on average.

The lower bound for the duration of the decommission has been published and is available in[START_REF] Cheriere | How Fast Can One Scale Down a Distributed File System?[END_REF]. A research report concerning the lower bound of the commission is available[START_REF] Cheriere | A Lower Bound for the Commission Times in Replication-Based Distributed Storage Systems[END_REF]. A journal paper with all results has been submitted and is under review.

Pufferbench is available at https://gitlab.inria.fr/Puffertools/Pufferbench

ACKNOWLEDGMENT

We would like to thank Kevin Harm (ANL) for providing valuable feedback on this paper, and Gail Pieper (ANL) for proofreading it.

The work presented in this paper is the result of a collaboration between the KerData project team at Inria, and Argonne National Laboratory, in the framework of the Data@Exascale Associate team, within the Joint Laboratory for Extreme-Scale Computing (JLESC, https://jlesc.github.io).

Experiments presented in this paper were carried out on the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER, and several universities as well as other organizations (see https://www. grid5000.fr).

This material is based upon work supported by the U.S. Department of Energy, Office of Science under contract DE-AC02-06CH11357.