Alexis Bandet
email: alexis.bandet@inria.fr

Francieli Boito
email: francieli.zanon-boito@inria.fr

Guillaume Pallez
email: guillaume.pallez@inria.fr

Scheduling distributed I/O resources in HPC systems

Keywords: HPC, parallel I/O, parallel file system, object storage targets, I/O forwarding, scheduling, resource allocation

This paper presents a comprehensive investigation on optimizing I/O performance in the access to distributed I/O resources in high-performance computing (HPC) environments. I/O resources, such as the I/O forwarding nodes and object storage targets (OST), are shared between a subset of applications. Each application has access to a subset of them and multiple applications can access the same resources. We propose heuristics to schedule these distributed I/O resources in two steps: for a set of applications, determining the number of I/O resources each will use (allocation) and which resources they will use (placement). We discuss a wide range of required information about applications' characteristics that can be used by the scheduling algorithms. Despite the fact that a higher level of application knowledge is associated with enhanced performance, our comprehensive analysis indicates that strategic decision-making with limited information can still yield significant enhancements in most scenarios. This research provides insights into the trade-offs between the depth of application characterization and the practicality of scheduling I/O resources.

I. INTRODUCTION

In large high-performance computing (HPC) platforms, applications access persistent data by performing I/O operations to a remote shared parallel file system (PFS), such as Lustre or BeeGFS. These file systems cut files into fixed-size stripes and distribute them across a number of storage targets (OSTs) for parallel access. Depending on the files that they access, all compute nodes may require access to the same OSTs at the same time. Thus, to mitigate contention, a layer of I/O forwarding nodes (or simply "I/O nodes") is often placed between compute nodes and the PFS [START_REF] Almási | An overview of the blue gene/l system software organization[END_REF]. In this context, it is important to notice both OST and I/O nodes are potentially shared by running applications. Other shared resources include burst-buffer nodes, present in some systems [START_REF] Bez | Access patterns and performance behaviors of multi-layer supercomputer i/o subsystems under production load[END_REF], [START_REF] Lopez | Ephemeral data access environment: Concept and architecture[END_REF].

When applications access the same I/O resources concurrently, their I/O performance can be slowed down [START_REF] Yang | End-to-end I/O monitoring on a leading supercomputer[END_REF], [START_REF] Yildiz | On the root causes of cross-application i/o interference in hpc storage systems[END_REF], [START_REF] Wang | A zoom-in analysis of i/o logs to detect root causes of i/o performance bottlenecks[END_REF], and hence they occupy compute resources for longer. In addition to wasting resources, the fact that I/O performance depends on what others are doing in the system leads to higher performance variability [START_REF] Costa | Systematically inferring i/o performance variability by examining repetitive job behavior[END_REF], [START_REF] Oral | Best practices and lessons learned from deploying and operating large-scale datacentric parallel file systems[END_REF], which makes execution time less predictable and, consequently, complicates resource management [START_REF] Gainaru | Profiles of upcoming hpc applications and their impact on reservation strategies[END_REF]. Many techniques have been proposed to mitigate contention, mainly on scheduling the accesses to the PFS [START_REF] Gainaru | Scheduling the I/O of HPC Applications Under Congestion[END_REF], [START_REF] Dorier | Calciom: Mitigating i/o interference in hpc systems through cross-application coordination[END_REF], [START_REF] Boito | IO-SETS: Simple and efficient approaches for I/O bandwidth management[END_REF], burst-buffers [START_REF] Aupy | Sizing and partitioning strategies for burst-buffers to reduce io contention[END_REF] and I/O-aware batch scheduling [START_REF] Liu | Server-Side Log Data Analytics for I/O Workload Characterization and Coordination on Large Shared Storage Systems[END_REF], [START_REF] Bleuse | Interference-aware scheduling using geometric constraints[END_REF].

However, these efforts usually see the shared I/O infrastructure as a single resource capable of a certain bandwidth, whereas in practice it is a distributed set of resources from which each application can use a subset. In addition, using X% of the OSTs, for example, does not grant a job X% of the PFS' peak performance [START_REF] Chowdhury | I/o characterization and performance evaluation of beegfs for deep learning[END_REF], [START_REF] Boito | The role of storage target allocation in applications' I/O performance with BeeGFS[END_REF]. Indeed, as we discuss in Section II, depending on their characteristics, each application will be impacted differently by the number of used I/O resources [START_REF] Bez | Towards On-Demand I/O Forwarding in HPC Platforms[END_REF], [START_REF] Yu | On the load imbalance problem of i/o forwarding layer in hpc systems[END_REF], [START_REF] Devarajan | Extracting and characterizing i/o behavior of hpc workloads[END_REF].

In this paper, we present a comprehensive study of the problem of scheduling shared I/O resources -I/O nodes, OSTs, etc -to HPC applications with the goal of mitigating contention and improving I/O and system performance. We tackle this problem by proposing heuristics to answer two questions: 1) how many resources should we give each application (allocation heuristics), and 2) which resources should be given to each application (placement heuristics). These questions are not independent, as using more resources often means sharing them. Nonetheless, our two-step approach allows for simpler heuristics that would be usable in practice. Moreover, it allows for studying the impact of these two steps separately.

In addition to overhead, an important aspect that impacts how "implementable" algorithms are is their input regarding applications' characteristics, since this information is often not available or at least imprecise. Therefore, we propose heuristics that use different input and study their robustness to inaccurate information. Our main contributions are:

• We propose allocation and placement algorithms, which work in two steps to schedule I/O resources among concurrent applications. • We extensively evaluate our approach and compare it to the state of the art using two case studies: scheduling of I/O nodes and of OSTs. • We discuss the quality of various input parameters and study their impact with the goal of answering what information is important from applications, and how robust the heuristics are to inaccurate information. The rest of this paper is organized as follows: Section II further motivates this work by providing background on the relationship between the number of I/O resources and per-Fig. 1: Normalized bandwidth as a function of the number of I/O nodes for 189 different applications, grouped by behavior. Data from [START_REF] Bez | Towards On-Demand I/O Forwarding in HPC Platforms[END_REF]. Fig. 2: Normalized bandwidth as a function of the number of OSTs for 301 benchmark configurations, grouped by behavior. Section V-A details how these results were obtained.

formance. Section III formally states the studied problem, and then Section IV discusses the proposed heuristics. The evaluation methodology is detailed in Section V, and results in Section VI. Section VII discusses related work, and Section VIII concludes this paper.

II. MOTIVATION

For both I/O nodes and OSTs, I/O performance depends on the number of I/O resources, and the impact of the latter on the former depends on their access pattern. In the case of I/O nodes, that was shown to be the case by Bez et al. [START_REF] Bez | Towards On-Demand I/O Forwarding in HPC Platforms[END_REF]. We plotted data from their work in Figure 1, where we see four different behaviors: increasing the number of I/O nodes can have no impact on performance, improve it, decrease it, or increase it until a point from where having more I/O nodes starts to harm it.

Boito et al. [START_REF] Boito | The role of storage target allocation in applications' I/O performance with BeeGFS[END_REF] studied the impact of the number of OSTs on performance and concluded that the more, the better. However, their analysis was limited to a single access pattern. A previous study by Chowdhury et al. [START_REF] Chowdhury | I/o characterization and performance evaluation of beegfs for deep learning[END_REF] with the same file system, but using other applications, concluded the number of OSTs had little or no impact. On the other hand, Xu et al. [START_REF] Xu | Lioprof: exposing lustre file system behavior for i/o middleware[END_REF] concluded that for collective reads using more OSTs actually degraded performance. Figure 2 shows our own results, described in Section V-A.

In practice, file systems are configured with a fixed number of OSTs per file, which is typically rather small to minimize sharing [START_REF] Wang | Diving into petascale production file systems through large scale profiling and analysis[END_REF], [START_REF] Chowdhury | I/o characterization and performance evaluation of beegfs for deep learning[END_REF]. Moreover, in most systems the number of I/O nodes assigned to a job, either depends on its number of compute nodes, which is not necessarily linked to its I/O behavior, or in some cases is a fixed number whatever the size of the job (it used to be seven on Theta). In all cases, sharing I/O resources is generally considered something to be avoided, as it may harm performance. For example, Bez et al. [START_REF] Bez | Arbitration Policies for On-Demand User-Level I/O Forwarding on HPC Platforms[END_REF] proposed an algorithm to schedule I/O nodes to jobs which aims at providing exclusive access to the applications with higher I/O performance. Nonetheless, over the years studies of the HPC I/O workload have consistently pointed to a bursty behavior, with most applications actually spending only a small portion of their time on I/O operations [START_REF] Wang | File system workload analysis for large scale scientific computing applications[END_REF], [START_REF] Carns | Understanding and Improving Computational Science Storage Access through Continuous Characterization[END_REF], [START_REF] Yang | End-to-end I/O monitoring on a leading supercomputer[END_REF], [START_REF] Liu | Characterization and identification of HPC applications at leadership computing facility[END_REF], [START_REF] Yang | A quantitative study of the spatiotemporal i/o burstiness of hpc application[END_REF]. That means an exclusive allocation of I/O resources to applications, if made for their whole execution, would not be the most efficient approach. The alternative would be to dynamically change the scheduling of I/O resources whenever application behavior changes. However, that would require a quickly available characterization of current behavior, which is in general not available, and fast implementation of the calculated schedule. Since the behavior may change very frequently for each job, the desired granularity may be only a few milliseconds. For these reasons, in this paper we focus on algorithms (presented in Section IV) that make decisions for entire executions and that do not focus on exclusive access, as we believe this approach is more suitable for use in practice.

III. MODEL

This section describes the platform (shown in Fig. 3) and application models we use in this study.

A. Platform model

We assume that we have a parallel platform composed of compute nodes and remote shared storage. To access this storage, each application must communicate first through distributed I/O resources. There are N I/O resources. We consider that access to compute nodes are exclusive (congestion-free), hence in this work we focus on a performance model for I/O.

Capacity sharing: When multiple applications access concurrently an I/O resource, they share equally its capacity. In other words, the bandwidth of each I/O resource is divided by the number of applications using it at this time.

B. Application model and I/O behavior

K applications run concurrently on the platform. Each application is a series of phases that alternate between computation and I/O subphases [START_REF] Carns | 24/7 Characterization of petascale I/O workloads[END_REF], [START_REF] Dorier | Using Formal Grammars to Predict I/O Behaviors in HPC: The Omnisc'IO Approach[END_REF], [START_REF] Gainaru | Scheduling the I/O of HPC Applications Under Congestion[END_REF]. We study the synchronous I/O case where compute and I/O subphases cannot overlap. The key parameters describing an application and used in the rest of this paper are summarized in Table I.

The length of each I/O subphase depends on the number of I/O resources allocated to application A j . The bandwidth of A j as a function of I/O resources is given by: n → b j (n). Thus, if A j uses n j I/O resources, when there is no congestion, its aggregated I/O time (for amount of data V io ) is:

T cf io (n) = V io b j (n)
In the rest of this paper, for clarity and when there is no ambiguity, we remove the index j when talking about an application variable.

When there is no congestion, during a total time T j cpu + T cf j io (n), A j occupies n I/O resources during a time T cf j io (n), and we have:

I/O-Stress(i, n) = n • T cf i io (n) T i cpu + T cf i io (n)
1) Characteristic values: For application A j , we define two characteristic values n j perf and n j sys as:

n j perf = argmin n T cf j io (n) = argmax n b j (n) (1)
n j sys = argmin n I/O-Stress(j, n) (2)
n j perf (Eq. ( 1)) corresponds to the number of I/O resources that minimizes the I/O transfer time of A j , and n j sys (Eq. ( 2)) corresponds to the number of I/O resources that minimizes the stress (I/O-Stress) on the system due to A j . Lemma 1. For all applications, n sys ≤ n perf .

(The proof is omitted for space limitations.)

2) Independence of I/O transfers: An I/O subphase over n I/O resources can be seen as n independent I/O transfers. This means that if an application is using multiple I/O resources, and its performance is slowed-down on one of those, then the other transfers are not slowed down. However, an I/O subphase only ends when all its I/O transfers are over (See Fig. 4).

C. Measuring performance

In this Section, we start by describing a solution (Section III-C1), before presenting how the performance of various solutions are measured (Section III-C2).

1) Defining a schedule: A solution has two elements:

• the number of I/O resources each application uses (π = (n 1 , . . . , n K )); • a mapping of the applications over the I/O resources. At a given time t, and given a schedule π = (n 1 , . . . , n K ), we define its I/O load as:

I/O-load(π) = 1 N K i=1 I/O-Stress(i, π(i)) (3)
Intuitively, this is a lower bound on the expectation of I/O time occupied by π per unit of time. By definition, I/O-load is minimized for the schedule π sys = (n 1 sys , . . . , n K sys ). If I/O-load > 1, then the system is saturated. In such a scenario, typical I/O time needed by applications exceeds system capabilities.

2) Measuring a schedule performance: Given a schedule π, we define several optimization criteria to evaluate it.

Mean-I/O-SlowDown: Assume application A j transferred an amount of data V in total I/O time T using n j I/O resources, then we call its I/O-SlowDown ρ j the ratio of time taken to perform its I/O operations compared to the time they take when running in isolation using the number of I/O resources that minimizes this time (n j perf ).

ρ j = T / V /b j (n j perf )
The Mean-I/O-SlowDown is the average of these values:

Mean-I/O-SlowDown = j≤K ρ j K (4)
To have a better qualitative understanding of the I/O-SlowDown and Mean-I/O-SlowDown, we can separate them into two components: one for the slowdown caused by not using the number of I/O resources that minimizes I/O time (ρ io j ), and another for the slowdown due to congestion (ρ con j ):

ρ j = ρ io j + ρ con j where ρ io j = b j (n j perf )/b j (n j )
I/O occupancy: Given a schedule π, the I/O-res-occ O i of an I/O resource i is the measure of the proportion of time this resource is occupied performing I/O operations. Given this value, we can define the spread of I/O-res-occ, as it represents the load imbalance over the different I/O resources:

I/O-spread = max i O i -min i O i (5)
Machine utilization: The previous metrics focus on I/O performance. Nonetheless, from a system administrator perspective, it may be interesting to favor I/O of applications that use more compute nodes/cores in order to improve the utilization of these resources. We define hence Machine-Idletime to represent the proportion of time when the compute nodes are not being use for computing. Let L i be the total I/O time of application A i , in a time interval [0, t] and exclusively using Q i of the Q cpu available compute resources. Then:

Machine-Idletime = K i=1 L i .Q i t • Q cpu IV. ALGORITHMS FOR SCHEDULING I/O RESOURCES
As previously mentioned (Section III-C1), a solution is defined by answers to two questions: the number of I/O resources each application will use and the mapping of applications over the multiple I/O resources. In this section we present heuristics to provide the answers -respectively allocation (Section IV-A) and placement (Section IV-B) algorithms.

While an application is accurately described by the elements in Table I, in practice this data can be hard to collect and inaccurate. Thus, in Section IV-C we further discuss the input required by the different heuristics.

A. Allocation algorithms

We compare five allocation policies, detailed below.

• Random: each application receives a randomly picked number of I/O resources. This serves as a baseline.

• Static: application A j receives Qj •N Q cpu I/O
resources, rounded to the closest positive integer. In other words, a number that depends only on the number of used compute resources. For the case of I/O nodes, this policy represents what happens in HPC systems where the mapping from compute nodes to them is static.

• BestBdw allocates n j perf (Eq. ( 1)) to each A j , i.e., the number of I/O resources that minimizes its I/O time. On the one hand, this policy minimizes ρ io . However, in some cases it may increase I/O-load (Eq. ( 3)) and make applications share more I/O resources, which leads to more congestion and hence to an increased ρ con .

• Nsys-Allocator gives π sys , i.e., it allocates n j sys (Eq. ( 2)) to each A j to minimize the I/O-load.

• TCPU-Allocator, detailed in Algorithm 1, starts at π sys and then repeatedly increases the number of I/O resources of the application that maximizes the utilization of compute resources (the sum of CPULoad) while respecting the constraint that I/O-load must be smaller or equal to 1 (so the I/O system is not saturated).

CPULoad(i, n) = Q i • T i cpu T i cpu + T cf i io (
n) TCPU-Allocator aims at being a compromise between BestBdw and Nsys-Allocator. Note than when the I/O-load of π perf is below 1, then it behaves as BestBdw.

B. Placement algorithms

Three placement algorithms are considered.

7 for i from 1 to K do 8 n ← π(i); 9 while n ̸ = n i perf & loadDiff(i) < 0 do 10 n ← n + 1; 11 if initIOLoad + (I/O-Stress(i, n)/N - I/O-Stress(i, π(i))/N ) ≤ 1 then 12 loadDiff(i) ← CPULoad(i, n) -CPULoad(i, π(i)) ; 13 π(i) ← n;

C. On the difficulty of instantiating an algorithm

The eight described algorithms use different information about applications, as summarized in Table II. The colors represent how easy to obtain we consider these values to be: TABLE II: Heuristics and their input

Q j V j io T j cpu argmax b j b j Random Static x BestBdw x Nsys-Allocator x x x TCPU-Allocator x x x x Random-Placement Greedy-Non-Clairvoyant Greedy-Clairvoyant x x x
• Easy (green): the number of compute resources used by each application can be obtained from the resource man-ager. Similarly to the total number of available compute and I/O resources, it is easily obtained and reliable. • Medium (orange): aggregated information such as the total amount of transferred data and compute time of an application can be obtained from previous runs, for example with profiling tools such as Darshan [START_REF] Carns | Understanding and Improving Computational Science Storage Access through Continuous Characterization[END_REF], or provided by the user. In both cases, the actual observed values could vary and this data is only semi-reliable. Note that n perf , used by BestBdw, is considered in this category because it does not require the whole curve and bandwidth values, only the number of I/O resources that maximizes performance. • Hard (red): for an application A j , obtaining the bandwidth as a function of the number of I/O resources (b j ) requires multiple previous runs and is naturally sensitive to variability. The system could accumulate this information over time (so it would only be available to some of the running applications), or the user could provide it (less reliable). Still, rather than using the exact b j for each application, we believe a more realistic alternative is to approximate it by a general behavior (e.g. the profiles seen in Figure 1). Given an application general I/O characteristics, such as read/write ratio, request size, etc. (medium difficulty in our classification above), it could be matched to a benchmark for which the profile is known [START_REF] Boito | Estimation of the impact of I/O forwarding on application performance[END_REF]. We explore this approach, and the robustness of the studied heuristics, in Section VI-B.

V. EVALUATION METHODOLOGY

The evaluation in this paper relies on data from real executions, described in Section V-A, and simulation of HPC systems, detailed in Section V-B. Sections V-C and V-D describe the workload generation and evaluation protocols.

A. Datasets

We used two sets of data, each covering multiple applications and, for each A j its b j , i.e., bandwidth measurements for different numbers of I/O resources. The first, obtained for 189 applications at the MareNostrum supercomputer and shown in Fig. 1, was made publicly available by Bez et al. [START_REF] Bez | Towards On-Demand I/O Forwarding in HPC Platforms[END_REF] and is used to cover the use case of scheduling I/O nodes. For the second use case (OST scheduling), we generated a similar dataset by running the IOR benchmark with different configurations in the PlaFRIM experimental platform, using numbers of OSTs for BeeGFS varying from 1 to 8. The I/O infrastructure of this platform has been detailed in [START_REF] Boito | The role of storage target allocation in applications' I/O performance with BeeGFS[END_REF]. The 301 configurations all write to a shared file, while covering various values for numbers of nodes, processes per node, request size, contiguous vs. 1D-strided file layout, and total amount of data. These results were shown in Fig. 2.

B. Simulator design

We developed a time-based simulator, available at https: //gitlab.inria.fr/abandet/shared IO simulator along with all the data to reproduce our results. It consists of three phases:

1) the workload generation, described in Section V-C;

2) the execution of the algorithms presented in Section IV. Note that we also implemented MCKP from the state of the art in I/O nodes scheduling [START_REF] Bez | Arbitration Policies for On-Demand User-Level I/O Forwarding on HPC Platforms[END_REF]. It performs both steps (allocation + placement).

3) The evaluation of the results according to the objectives presented in Section III-C2. We detail how these objectives are measured in Section V-D. At each unit of time, the simulator examines the status of each application (compute or I/O). When there is a collision on an I/O resource (i.e. two or more applications try to do I/O), the colliding applications will alternate in performing I/O for one unit of time each until their I/O subphases are over. This approximates the fair-share I/O scheduling algorithm because all simulations are performed for thousands of units of time.

C. Workload generation protocol

We make the hypothesis that the algorithms' behavior depends on the I/O stress on the system. Hence in our generation of the application sets, we cover various I/O-load.

In all the experiments we consider that we have N = 20 I/O resources, and Q cpu = 480 compute resources. The number of applications K depends on the experiment, so does the target I/O-load: Θ. Then, given K and Θ, for each application A j we generate it as follows:

• we pick an I/O bandwidth profile uniformly at random in the used data set (Section V-A); • the number of phases p j is picked uniformly at random in {2, 3, 4, • • • , 20}; • Q j is computed so that 10% (resp. 30%, 60%) of applications use 75% (resp. 20%, 5%) of the compute resources (large, medium, and small applications); • for all applications, we set T j cpu +T cf j io (1) = 5000s (common horizon). Then, we set T j cpu /T cf j io (1) = X, where X is picked uniformly at random in [0, b]. The bound b is computed so that E (I/O-Stress(j, 1)) = ΘN K : b is the solution to log(1+b) = b ΘN K (computed analytically). Consequently, we have

V j io = 5000 • b j (1)/(1 + X) T j cpu = 5000 (1 -1/(1 + X))
By construction, this generation has the following property:

I/O-load(π 1 ) = 1 N K j=1 T cf j io (1) T j cpu + T cf j io (1) = Θ
In a second step and for the evaluation, we categorize the sets of application that we have generated by their minimum I/O-load, i.e., I/O-load(π sys ) ≤ Θ.

D. Evaluation protocol

Each studied scenario is evaluated with over 100 different application sets. The metrics are measured on an interval where the state of the system is constant (i.e. no application finishes), but of a sufficient length (i.e. each application should have performed at least a complete phase compute+I/O).

VI. RESULTS

In this Section we provide various elements to compare the different heuristics. We focus the discussion on the impact of the input that each algorithm considers. Specifically:

• in Section VI-A, we compare the algorithms in a setup where we trust their inputs; • then, in Section VI-B, we loosen the quality of the input information to study the robustness of our algorithms; • finally, after using the I/O nodes case study in all previous sections, in Section VI-C we show that the results hold with another bandwidth model (the OST case study).

A. Evaluation with accurate input data

In this Section, we evaluate the different solutions based on the optimization criteria presented in Section III-C. As already discussed, the comparison between the algorithms is performed as a function of I/O-load(π sys ), which corresponds to the level of stress on the I/O system. 1) User observed performance: In Fig. 5, we present Mean-I/O-SlowDown (Eq. ( 4)) for all combinations of placement and allocation algorithms when I/O-load(π sys ) varies. We can make several key observations:

• In general, the naive algorithm that allocates the number of I/O resources that maximizes I/O bandwidth to each application (BestBdw) is the best allocation strategy. • On the other hand, when I/O-load is large Nsys-Allocator performs better than BestBdw because the latter tends to increase the stress on the system. • Using the Greedy-Clairvoyant placement strategy has negligible gain on Mean-I/O-SlowDown compared to Greedy-Non-Clairvoyant, which requires much less information. These observations show that choosing either Greedy-Clairvoyant or Greedy-Non-Clairvoyant for allocation has little impact on performance (but they do have an impact compared to Random-Placement). That shows that efficient I/O scheduling can be done with limited information.

Note that Greedy-Clairvoyant and Greedy-Non-Clairvoyant have a different behavior. We can verify this by studying the In the rest of this section, we only use Greedy-Non-Clairvoyant as a placement algorithm.

To compare the allocation heuristics, we decompose the I/O-SlowDown into their I/O and congestion components (as discussed in Section III-C2) in Fig. 8 and9. In the first we show two ranges of I/O-load(π sys ) (low and high), and in the second we show the tendencies when it varies.

In Fig. 8, we can observe that, as by design, BestBdw decreases the portion of I/O time due to I/O resources allocation to its bare minimum, at the cost of a much higher congestion overhead (ρ con ) than that of Nsys-Allocator or Static. As I/O-load(π sys ) increases (Fig. 9), this cost increases linearly and may become a problem at very high I/O-load values. On the contrary, the congestion overhead with allocation algorithms that take into account the system I/O-load (Nsys-Allocator and TCPU-Allocator) do not vary as much when I/O-load(π sys ) increases. Nonetheless, that comes at the Finally, MCKP performs worse than all the other studied policies for these objectives. Indeed, it seeks to optimize the sum of applications' bandwidths due to I/O resources allocation, and has hence a tendency of favoring a few applications (the highest bandwidth ones) in detriment of others.

2) Machine utilization: Up to now we focused on the average I/O behavior, a user-oriented objective. This disadvantages more unfair algorithms such as Static, which gives more I/O resources to applications that occupy a larger fraction of the compute resources, possibly in detriment of smaller applications that are more numerous (and hence have a higher impact in a quantitative objective like Mean-I/O-SlowDown). From a system administrator perspective, it may be more interesting to have an application that occupies a large part of the machine to perform its I/O fast, even if it at the cost of worse performance for smaller applications.

To evaluate this, we plot the Machine-Idletime of the compute nodes (i.e. the time when applications are performing I/O) in Fig. 10. We can make two observations:

• At low values of I/O-load(π sys ), the relative allocation algorithm performance in terms of Machine-Idletime are the same as those for Mean-I/O-SlowDown (BestBdw and TCPU-Allocator perform the best). This is not surprising: I/O has less impact.

• However, at larger I/O-load(π sys ), we start to see a real difference of performance between TCPU-Allocator and BestBdw, even though their respective I/O performance were similar, giving an advantage to the heuristic that considers more information. Static is an interesting heuristic: it uses little information about the applications and still can get good machine utilization results when the I/O-load is high.

B. Robustness to the quality of the input

In Section VI-A, we were able to demonstrate the fact that under low I/O-load, BestBdw was the most efficient allocation algorithm, whereas under a heavier I/O-load, one should rather use TCPU-Allocator which provides the same I/O performance from an application perspective but improves on system utilization.

As shown in Section IV-C, there is however an important difference between these two algorithms: their input. In particular, TCPU-Allocator requires a full I/O profile of the applications whereas BestBdw only requires the number of I/O nodes that provides the maximum bandwidth.

In this section, we run the same simulations as before, but giving algorithms partial (and potentially wrong) information to study how robust they are. Specifically, all applications from a given I/O performance profile (Ascent, Descent, Peak, Neutral) is said to have the same bandwidth function b j . This function is obtained by interpolating all curves within a profile to the same function (see Fig. 11).

In Fig. 12, we study the performance of three combinations of algorithms (TCPU-Allocator +Greedy-Non-Clairvoyant; BestBdw +Greedy-Non-Clairvoyant; BestBdw +Greedy-Clairvoyant) with poor input accuracy. To do so, we use all generated scenarios, and plot them as a function of I/O-load(π sys ). Note that Greedy-Non-Clairvoyant is not impacted by the lack of accuracy: its behavior does not change. Similarly, BestBdw is impacted by the lack of accuracy only for some of the applications with a bandwidth profile Peak and Neutral.

Specifically, in Fig. 12a, we compare their performance to what was observed with accurate information: if the value is 2%, for example, it means that the algorithm with inaccurate information performed 2% worse than with accurate information. We can observe that they have almost identical performance (except in scenarios with extremely low I/O-load). As expected, the allocation algorithm where the behavior differs the most is TCPU-Allocator which is the algorithm that requires the most information. Yet this difference is still negligible (up to 4%). With respect to the mapping algorithm, Greedy-Clairvoyant performs almost identically except in cases with extremely low I/O-load, i.e. when the Mean-I/O-SlowDown is close to 1, the error in prediction is most impactful. We confirm their absolute impact by plotting their Machine-Idletime (Fig. 12b) which confirms that even with inaccurate information TCPU-Allocator is the best solution. More generally, we observe that the various algorithms are quite robust to some inaccuracy in I/O behavior, and that the main claims of our work with respect to choosing

C. Scheduling of OSTs

In this final set of experiments, we evaluate our algorithms on a different context: OST scheduling. The main difference with I/O nodes lies on the performance obtained depending on the number of resources allocated to the application (see Fig. 1 and2).

Due to lack of space, we only show plots for the conclusions that differ from the previous section, i.e. we do not show the comparison between placement strategies and focus solely on the Greedy-Non-Clairvoyant placement algorithm. These results are presented in Figure 13.

Overall the main observation is the same as previously: when there is little I/O stress on the system, then BestBdw performs better than I/O stress aware algorithms such as Nsys-Allocator. Then as the stress increases, its performance quickly degrades. The main difference here is that the cut-off point is much earlier. These results confirm that TCPU-Allocator is an excellent alternative that is able to match the performance of BestBdw when needed, while being I/O stress aware.

There is another important difference with the previous behavior: Static's performance are excellent for this profile. This is interesting because i) for OSTs, Static is not what is typically used in practice; and ii) it is quite natural and easy to implement. Unfortunately, the fact that this behavior is highly dependent on application profile (i.e. we did not see the same behavior with the I/O nodes dataset) makes the Static allocation algorithm unreliable.

VII. RELATED WORK

A. Scheduling of I/O nodes

Yu et al. [START_REF] Yu | On the load imbalance problem of I/O forwarding layer in HPC systems[END_REF] document the load imbalance problem of I/O nodes and propose a strategy where they are statically assigned to applications in an exclusive way. Whenever the load of an application's I/O nodes it too high, it is allowed to temporarily use the I/O nodes assigned to others. Differently from us, they do not study the selection of which I/O nodes should be shared, hence our work is complementary to theirs.

In the work by Ji et al. [START_REF] Ji | Automatic, application-aware i/o forwarding resource allocation[END_REF], less than half of the I/O nodes are statically assigned to applications, and then historical data is used to decide if more nodes, from a pool of available ones, should be used. This decision on the number of I/O nodes each application should use is taken only based on the number of compute nodes that perform I/O operations, while it has since been shown (see Section II) that other characteristics impact b j in a more complicated way. Their approach also allocates more I/O nodes to applications to avoid sharing them with others of incompatible access patterns. Differently, we focus on more generic placement strategies, that require less information, and our results for Greedy-Non-Clairvoyant prove how effective they can be. Our techniques can be used in the absence of detailed application information and interference models.

Bez et al. [START_REF] Bez | Towards On-Demand I/O Forwarding in HPC Platforms[END_REF] propose MCKP, which does allocation and placement by optimizing the sum of applications' bandwidths and favoring exclusive access as much as possible. We argue that exclusive access may be wasteful as many applications are not I/O intensive, and instead further explore the placement aspect. In Section VI, we compare our heuristics to MCKP.

B. Scheduling of OSTs

Yang et al. [START_REF] Yang | An end-to-end and adaptive i/o optimization tool for modern hpc storage systems[END_REF] reduce the problem of placing applications on I/O nodes and OSTs to the maximum flow problem, using their I/O load and the current monitored load of the resources. This is similar to what Greedy-Clairvoyant does, with the difference that we do not place applications on all layers of I/O resources at the same time. The strategy by Wang et al. [START_REF] Wang | Improving large-scale storage system performance via topology-aware and balanced data placement[END_REF] also considers all layers at once. The OST with the lowestcost path is selected for each of the application's compute nodes. The cost of a path depends on manually-set weights given to layers according to their importance for performance. Moreover, while they focus on improving load balance in the context of each application, our approach considers a global view of all concurrent applications.

The challenge in determining the best number of OSTs for an application is often tackled in the literature by having systems try different configurations over multiple runs. For example, Kim et al. [START_REF] Kim | Dca-io: A dynamic i/o control scheme for parallel and distributed file systems[END_REF] propose DCA-IO to automatically tune PFS parameters. If no information is available for an application, it receives the number of OSTs historically observed to be the best for the number of compute resources it uses. On the other hand, if the application is known, it will receive more OSTs at each execution until a local maximum is found.

Another way of obtaining such information is to train models on aggregated application metrics (more easily obtained). The auto-tuning framework proposed by Behzad et al. [START_REF] Behzad | Optimizing i/o performance of hpc applications with autotuning[END_REF] adapts the number of OSTs by keeping a database of I/O patterns, extracted from applications, and using non-linear regression models to find the best values. Agarwal et al. [START_REF] Agarwal | Active learningbased automatic tuning and prediction of parallel i/o performance[END_REF] use Bayesian optimization and a pre-trained I/O performance model to recommend the best number of OSTs to each application. These approaches could be used together with our proposed heuristics to provide the required information.

VIII. CONCLUSION

In this work we have investigated the problem of allocating subsets of distributed I/O resources to applications in order to optimize their I/O performance and the platform utilization.

Our contributions include both allocation and placement algorithms. In their design, we have taken into account a trade-off between simplicity and efficiency. To this regard we have shown that the placement algorithm can be quite naive: balancing the absolute number of applications per I/O resource, without considering their I/O load, leads to results as good as I/O-aware placement. This naive algorithm gives more leeway to optimize placement based on other reasons (such as proximity to the applications).

On the contrary, we have shown that the allocation algorithm is more important for I/O performance, and one should use a more fine-tuned algorithm rather than a naive approach such as peak bandwidth or a static approach that allocates a number of I/O resources proportional to the number of compute resources.

An important contribution of our work is the robustness study: indeed, I/O behavior has been shown to be quite volatile and hard to predict. Hence a very efficient heuristic that is not robust to volatility in its input can become quite useless. In this work we have studied different types of input that algorithms could use, and the limits of each algorithm based on these inputs. In addition, we have shown that our presented heuristics are robust to inaccuracy in input information. We believe that this opens avenues in terms of I/O behavior prediction which is a hard problem: indeed one may not need exact prediction for some of the I/O scheduling algorithms. This should considerably simplify the design of I/O analysis tools.

Fig. 3 :

 3 Fig. 3: Representation of the architecture: three applications A 1 (resp. A 2 , A 3 ) use two (resp. three, one) I/O resources. The number of I/O resources is not necessarily correlated to the size (number of compute nodes) of the application.

Fig. 4 :

 4 Fig. 4: Independence of I/O transfers with n = 2

14 idx

 14 ← argmax(loadDiff); 15 if loadDiff(idx) < 0 then 16 done ← True; 17 else 18 π(idx) ← π(idx); 19 return π; Algorithm 1: TCPU-Allocator • Greedy-Non-Clairvoyant aims at providing a balanced number of applications per I/O resource. It sorts applications by decreasing number of I/O resources (computed by the allocation algorithm), then it places each of them going over the I/O resources in a round-robin fashion. • Greedy-Clairvoyant strives for a balanced load across the I/O resources. It sorts applications by decreasing congestion free I/O ratio T cf io /(T cpu + T cf io ), then it greedily places them on the I/O resources with the lowest I/O-res-occ.

Fig. 5 :

 5 Fig. 5: Mean-I/O-SlowDown for different combinations of algorithms at increasing I/O-load(π sys ). Lines show the mean value, and the area around them is the percentile interval (10 th -90 th ).

Fig. 6 :

 6 Fig. 6: I/O-spread for different placement and allocation algorithms when I/O-load(π sys ) = 0.5 ± 0.05.

Fig. 7 :

 7 Fig. 7: I/O-spread for different placement algorithms using BestBdw. Lines connect the mean values, and the area around each line shows the 95% confidence interval.

Fig. 8 :

 8 Fig. 8: Mean-I/O-SlowDown and Max-I/O-SlowDown for allocation algorithms with two ranges for I/O-load(π sys ). The y axes are in log scale and start at 1 (minimum for ρ j ).

Fig. 9 :

 9 Fig. 9: I/O-SlowDown separated into its two causes: I/O resources allocation ρ io and congestion ρ con . The lines show the polynomial regression, and the area around them the 95% confidence intervals.

Fig. 10 :

 10 Fig. 10: Machine-Idletime for allocation algorithms at increasing I/O-load(π sys ). The y axes do not start at 0. The lower the better.

Fig. 11 :Fig. 12 :

 1112 Fig. 11: Polynomial regression over all application profiles.

Fig. 13 :

 13 Fig. 13: Results for the OST case study. Lines show the mean values, and the area around them the interval between the 10 th and 90 th percentiles.

TABLE I :

 I Key parameters for application A j .

  An allocation π 1 π ← initialized as π sys ; 2 done ← False; 3 while not done do an array of size K, filled with -1;

	6	initIOLoad ← I/O-load(π);

• Random-Placement randomly assigns I/O resources to applications. This policy reflects what happens in practice in many HPC systems, where I/O behavior is not taken into consideration for placement. Data: K applications Result: 4 π ← π; 5 loadDiff ←