Avinash Maurya

M Mustafa Rafique
email: mrafique@cs.rit.edu

Thierry Tonellot
email: thierrylaurent.tonellot@aramco.com

Hussain J Alsalem
email: hussain.salim.2@aramco.com

Franck Cappello
email: cappello@anl.gov

Bogdan Nicolae
email: bnicolae@anl.gov

GPU-Enabled Asynchronous Multi-level Checkpoint Caching and Prefetching

Keywords: Computing methodologies → Distributed computing methodologies, Massively parallel and high-performance simulations, • Information systems → Data management systems, • Applied computing → Physical sciences and engineering High-Performance Computing (HPC), Graphics Processing Unit (GPU), asynchronous multi-level checkpointing, hierarchical cache management, prefetching

Checkpointing is an I/O intensive operation increasingly used by High-Performance Computing (HPC) applications to revisit previous intermediate datasets at scale. Unlike the case of resilience, where only the last checkpoint is needed for application restart and rarely accessed to recover from failures, in this scenario, it is important to optimize frequent reads and writes of an entire history of checkpoints. State-of-the-art checkpointing approaches often rely on asynchronous multi-level techniques to hide I/O overheads by writing to fast local tiers (e.g. an SSD) and asynchronously flushing to slower, potentially remote tiers (e.g. a parallel file system) in the background, while the application keeps running. However, such approaches have two limitations. First, despite the fact that HPC infrastructures routinely rely on accelerators (e.g. GPUs), and therefore a majority of the checkpoints involve GPU memory, efficient asynchronous data movement between the GPU memory and host memory is lagging behind. Second, revisiting previous data often involves predictable access patterns, which are not exploited to accelerate read operations. In this paper, we address these limitations by proposing a scalable and asynchronous multi-level checkpointing approach optimized for both reading and writing of an arbitrarily long history of checkpoints. Our approach exploits GPU memory as a first-class citizen in the multi-level storage hierarchy to enable informed caching and prefetching of checkpoints by leveraging foreknowledge about the access order passed by the application as hints. Our evaluation using a variety of scenarios under I/O concurrency shows up to 74× faster checkpoint and restore throughput as compared to the state-of-art runtime and optimized unified virtual memory (UVM) based prefetching strategies and at

INTRODUCTION

Motivation. High-Performance Computing (HPC) applications produce massive amounts of distributed intermediate data during their execution that needs to be checkpointed concurrently by a large number of processes in real-time at scale. This is a fundamental I/O pattern used in a wide range of scenarios [START_REF] Ross | Storage Systems and Input/Output: Organizing, Storing, and Accessing Data for Scientific Discovery[END_REF]: resilience based on checkpoint-restart, producer-consumer patterns in workflows (e.g., simulations coupled with analytics of intermediate checkpoints), reproducibility efforts (e.g., validation of intermediate checkpoints in addition to the end results), revisiting previous states to advance a computation, etc. Some scenarios like checkpoint-restart involve frequent writes of checkpoints, but only need to retain a few latest checkpoints, which are only occasionally read back for restoring workloads in case of failures. However, many other scenarios require retaining the entire (or most of the) history of checkpoints produced during runtime and need to read them back with high frequency, sometimes even higher than the write frequency. For example, reproducibility efforts need to write checkpoints that represent intermediate results and/or performance measurements during the runtime of an application, then read them back in the same order in which they were produced to check for invariants and/or compare multiple runs to identify where they begin to diverge [START_REF] Line | A (meta)data framework for reproducing hybrid workflows with FAIR[END_REF]. As another example, the adjoint state method used for automated differentiation (AD) [START_REF] Paszke | Automatic Differentiation in PyTorch[END_REF], involves a forward pass that produces a large number of checkpoints and a backward pass that reads them back in reverse order. This is popular in a variety of scientific applications, such as climate and ocean modeling, multi-physics, and seismic imaging in the oil industry [START_REF] Plessix | A review of the adjoint-state method for computing the gradient of a functional with geophysical applications[END_REF][START_REF] Zhou | Reverse time migration: A prospect of seismic imaging methodology[END_REF] and deep learning [START_REF] Wan | Deep Learning:Neural Network, Optimizing Method and Libraries Review[END_REF]. Binomial checkpointing [START_REF] Walther | Extending the Binomial Checkpointing Technique for Resilience[END_REF][START_REF] Wang | Memory-Efficient Differentiable Programming for Quantum Optimal Control of Discrete Lattices[END_REF] is another popular approach extensively used by applications such as quantum optimal control to enable execution of memory-bound AD applications [START_REF] Hari | Reducing Memory Requirements of Quantum Optimal Control[END_REF] where the forward pass generates only a subset of checkpoints, while the backward pass recomputes missing checkpoints by triggering smaller forward passes, which themselves may generate new checkpoints. Such interleavings introduce the need to write and read checkpoints in any predefined order. Similarly, coupled workflows that involve producer-consumer patterns rely on intermediate checkpoints to exchange data (e.g., simulations that produce intermediate checkpoints consumed by analytics, data batches produced in real-time and consumed by ML/DL models, etc.) also often need to access the checkpoint history in a predefined order (e.g., based on priority).

These scenarios have two important aspects in common. First, the frequency of writing and reading back checkpoints is much higher than in the case of resilience. Second, there is a large number of processes, each of which needs to read and write large checkpoint sizes. For example, RTM [START_REF] Plessix | A review of the adjoint-state method for computing the gradient of a functional with geophysical applications[END_REF] (reverse time migration) scenarios used in the oil industry need to run an entire ensemble of adjoint computation instances that run concurrently and cover a diverse spectrum of wavelength frequencies. There can be hundreds of instances, each of which needs to access a history of checkpoints that reaches up to hundreds of TBs, with a frequency of reads and writes in the order of milliseconds. For example, an instance corresponding to a wavelength frequency of 100 Hz runs on eight Nvidia V100 GPUs (one process per GPU) and generates checkpointing data at 20 GB/s per GPU, amounting to ∼200 TB of checkpointing data over 1300 seconds for all eight processes.

Limitations of State-of-the-Art. Checkpointing and I/O runtimes (e.g., VELOC [START_REF] Nicolae | VeloC: Towards High Performance Adaptive Asynchronous Checkpointing at Large Scale[END_REF][START_REF] Nicolae | VELOC: VEry Low Overhead Checkpointing in the Age of Exascale[END_REF], ADIOS [START_REF] Godoy | ADIOS 2: The Adaptable Input Output System. A framework for high-performance data management[END_REF], SCR [START_REF] Moody | Design, Modeling, and Evaluation of a Scalable Multi-level Checkpointing System[END_REF], and FTI [START_REF] Parasyris | Checkpoint Restart Support for Heterogeneous HPC Applications[END_REF]) often make use of multi-level strategies that leverage hierarchic node-local storage tiers (e.g., GPU memory, host memory, nonvolatile memory, and SSDs) to flush the checkpoints to the slower tiers asynchronously. Specifically, the application is blocked only for the duration of the writes to the fastest tier, while other cascading transfers to the slower tier proceed concurrently in the background. However, multi-level strategies that retain high performance and scalability for both writing and reading checkpoints have received comparatively less attention.

One solution to address this gap is to complement multi-level asynchronous flushing with multi-level asynchronous prefetching, which solves the opposite problem: read the checkpoints in advance from the slowest tiers to the fastest tiers concurrently in the background to reduce the duration of read operations when the application issues them. Although prefetching strategies have been extensively studied in the context of I/O subsystems, in our case there are several important challenges. First, writing and reading checkpoints concurrently creates a complex producer-consumer interleaving, which means that independent strategies specifically optimized for checkpointing and prefetching may perform suboptimally when combined together. Second, for a majority of HPC applications, the read order of checkpoints is deterministic for long periods of the runtime and is often fully known in advance, which can be leveraged to optimize both the prefetching decisions and eviction policies. Third, the rise in the popularity of high bandwidth memory (HBM) [START_REF] Foley | Ultra-Performance Pascal GPU and NVLink Interconnect[END_REF] available in accelerators, such as GPUs, opens new opportunities to leverage the spare capacity for caching but complicates the interactions with the other node-local storage tiers.

Key Insights and Contributions. To address these challenges, we propose an asynchronous multi-level strategy specifically optimized to leverage synergies between checkpointing and prefetching to minimize the I/O overheads of reading and writing checkpoints under concurrency. The novelty of our proposal is to manage the entire life cycle of the checkpoints by leveraging foreknowledge about the order in which the checkpoints will be consumed in the future using application-level hints. This enables an optimized design of caching, prefetching, and eviction strategies across multiple tiers, including the high bandwidth memory available in GPUs. We summarize our contributions as follows:

(1) We formulate the problem of asynchronous multi-level checkpointing and prefetching on hierarchic storage tiers for scenarios in which checkpoints are frequently read and written in a producer-consumer fashion under concurrency with deterministic patterns (Section 2). (2) We propose a series of design principles and algorithms for managing a limited cache space on both individual and shared heterogeneous storage tiers of the compute nodes. Specifically, we introduce key ideas, such as checkpoint life cycle based on a state machine that combines flushing and prefetching, score-based eviction of consecutive checkpoints that are the least likely to cause I/O bottlenecks as a group in the future, management of fragmentation (Section 4.1). (3) We illustrate these design principles and proposed algorithms as an extension to VELOC [START_REF] Nicolae | VeloC: Towards High Performance Adaptive Asynchronous Checkpointing at Large Scale[END_REF][START_REF] Nicolae | VELOC: VEry Low Overhead Checkpointing in the Age of Exascale[END_REF], a production-ready HPC checkpoint-restart library. In this context, we take advantage of CUDA-enabled GPUs to provide fast node-local caching (Section 4.3). (4) We evaluate our proposed approach in a series of experiments conducted on the Nvidia DGX-A100 [START_REF] Nvidia | NVIDIA DGX A100 System Architecture[END_REF] platform.

Our experiments use two workloads consisting of uniform and variable-sized checkpoints obtain from traces of RTM [START_REF] Tariq Alturhkestani | Maximizing I/O Bandwidth for Reverse Time Migration on Heterogeneous Large-Scale Systems[END_REF] (reverse time migration), a real-life application used in the oil and gas industry (Section 5). These experiments show an order of magnitude higher read and write throughput compared with state-of-art approaches.

Limitations of the Proposed Approach. Our approach assumes that the checkpoints are monolithic objects for which partial writes and reads are not allowed. Furthermore, we assume that each checkpoint remains immutable after a write request at least until it is read back. While these assumptions limit the scope of our proposal compared with generic, system-level caching and prefetching solutions, they also enable significant optimization opportunities.

The rest of the paper is organized as follows. In Section 2, we describe the problem of checkpoint and restore of GPU-resident data structures using asynchronous multi-tier I/O operations. In Section 3, we summarize existing efforts and approaches that are related to the scope of this paper. In Section 4, we explain our proposed system design, describe our design principles and gapaware scoring-based eviction algorithm for efficient checkpoint and prefetch overlaps, and provide the implementation details. In Section 5, we describe our evaluations using diverse applications and system configurations and show a significant improvement in the application performance as compared to the state-of-the-art techniques. Finally, in Section 6, we conclude this paper.

PROBLEM FORMULATION

Consider an HPC application that runs on 𝑁 compute nodes, each of which is equipped with multiple GPUs and features a hierarchy of node-local storage (high-bandwidth GPU memory, host memory, SSDs) and access to remote storage (e.g., parallel file systems [START_REF] Schwan | Lustre: Building a file system for 1000-node clusters[END_REF], object stores [START_REF] Lofstead | DAOS and Friends: A Proposal for an Exascale Storage System[END_REF]). The HPC application is composed of 𝑃 processes, each of which is assigned to a single GPU of each compute node and is allocated a fixed cache size on each of the node-local storage tiers. While the GPU caches are used separately by the processes, the rest of the caches are shared, competing for the I/O bandwidth at either node level (host memory, SSDs) or globally (remote storage). We assume that the SSD has enough capacity to hold all checkpoints of a single node, while the remote storage has enough capacity to hold all checkpoints of all nodes.

The processes perform computations either independently or in a tightly coupled fashion. Each process produces a history of 𝐾 intermediate datasets in the GPU memory assigned to it. These intermediate datasets need to be checkpointed and revisited later by the same process, by using the node-local storage hierarchy to swap in and out the checkpoint data as needed. To revisit an intermediate dataset, we need to "restore" it from a checkpoint. The restore order is known to a large extent in advance and specified at runtime by the application. However, the order is simply considered as a hint and the application is allowed to deviate from it at the expense of a potential performance penalty. Once a checkpoint is restored, it is considered consumed by the application and will either be discarded or will not be accessed again before any other unconsumed checkpoint in the history.

The following conditions apply: (1) during a checkpoint request, a process blocks until the checkpoint has been fully copied to the GPU cache, after which control is returned back to the application. Meanwhile, the checkpoint is asynchronously flushed to the slower tiers; (2) a process is allowed to read a checkpoint it has previously written even if the asynchronous flushes to slower tiers are still pending; (3) to speed up restore requests, the checkpointing runtime is allowed to prefetch checkpoints based on the restore order specified by each process; (4) to avoid trashing, once a checkpoint has been prefetched on the GPU cache, it can only be evicted after it was consumed; and (5) if a checkpoint was consumed and can be discarded, any of its pending flushes to other tiers are not required to complete successfully.

We aim to design and implement an asynchronous multi-level strategy that minimizes the I/O overheads of both reading and writing checkpoints from the application perspective while satisfying the aforementioned conditions.

RELATED WORK 3.1 HPC Checkpoint-Restart

Checkpoint-Restart techniques are traditionally used to provide resilience for HPC applications. Transparent fault tolerance approaches (BLCR [START_REF] Paul | Berkeley lab checkpoint/restart (BLCR) for Linux clusters[END_REF]) automatically capture the full state of a group of processes, at the expense of generating large checkpoint sizes that cannot be used for any other purpose than resuming the process execution. Application-level fault tolerance approaches (FTI [START_REF] Parasyris | Checkpoint Restart Support for Heterogeneous HPC Applications[END_REF], SCR [START_REF] Moody | Design, Modeling, and Evaluation of a Scalable Multi-level Checkpointing System[END_REF], VELOC [START_REF] Nicolae | VeloC: Towards High Performance Adaptive Asynchronous Checkpointing at Large Scale[END_REF], Canary [START_REF] Arif | Canary: Fault-Tolerant FaaS for Stateful Time-Sensitive Applications[END_REF]) require the user to define the critical data structures and reconstruct a consistent state from them on restart. This enables many use cases beyond resilience, including the scenario targeted in this paper. However, such approaches typically focus on checkpointing performance, with comparatively less attention dedicated to restart performance. Both approaches can be complemented by asynchronous multi-level strategies to ensure checkpoint durability. They include optimized flushes from faster to slower tiers [START_REF] Nicolae | VeloC: Towards High Performance Adaptive Asynchronous Checkpointing at Large Scale[END_REF], but also complementary strategies such as partner replication and erasure coding.

Workflow Engines

To enable efficient coupling between different HPC application components, several workflow engines have been extensively explored [START_REF] Dreher | Bredala: Semantic Data Redistribution for In Situ Applications[END_REF][START_REF] Monozov | Henson v1.0[END_REF][START_REF] Morozov | Block-parallel data analysis with DIY2[END_REF]. In addition to addressing the placement and scheduling of application tasks, the increasing data-centric nature of HPC workloads has brought attention to several aspects: complex data redistribution and block-level parallelism (e.g. DIY [START_REF] Morozov | Block-parallel data analysis with DIY2[END_REF] and Decaf [START_REF] Dreher | Bredala: Semantic Data Redistribution for In Situ Applications[END_REF]), data staging and publish-subscribe (e.g. DataSpaces [START_REF] Tong | Exploring Data Staging Across Deep Memory Hierarchies for Coupled Data Intensive Simulation Workflows[END_REF]), in-situ engines (e.g. ADIOS2 [START_REF] Godoy | ADIOS 2: The Adaptable Input Output System. A framework for high-performance data management[END_REF]), coroutines (e.g., Henson [START_REF] Monozov | Henson v1.0[END_REF]). While they provide flexible data pipelines, they are typically optimized for coarse-grained I/O between different tasks with different access patterns and caching requirements.

GPU Checkpointing

Checkpoint-restore techniques have been extended to GPUs, both for fault-tolerance [START_REF] Nukada | NVCR: A Transparent Checkpoint-Restart Library for NVIDIA CUDA[END_REF][START_REF] Parasyris | Checkpoint Restart Support for Heterogeneous HPC Applications[END_REF] and workload migration [START_REF] Nukada | NVCR: A Transparent Checkpoint-Restart Library for NVIDIA CUDA[END_REF][START_REF] Xiao | Transparent Accelerator Migration in a Virtualized GPU Environment[END_REF]. Systemlevel checkpointing libraries such as NVCR [START_REF] Nukada | NVCR: A Transparent Checkpoint-Restart Library for NVIDIA CUDA[END_REF], transparently record and replay the memory-based CUDA APIs for checkpointing and restoring. Approaches such as CheckFreq [START_REF] Mohan | CheckFreq: Frequent, Fine-Grained DNN Checkpointing[END_REF] and Multilayered Buffered System [START_REF] Tariq Alturkestani | MLBS: Transparent Data Caching in Hierarchical Storage for Out-of-Core HPC Applications[END_REF] also exploit heterogeneous storage tiers. However, none of these efforts consider the case of dedicated cache regions on the GPU and host memory. Furthermore, they typically focus on caching and prefetching of single snapshots rather than collections of snapshots.

Caching and Prefetching

Cache management for many-core processors and deep hierarchies has been explored in various contexts, e.g., location awareness [START_REF] Maurya | Towards Efficient I/O Scheduling for Collaborative Multi-Level Checkpointing[END_REF][START_REF] Park | Location-aware cache management for many-core processors with deep cache hierarchy[END_REF], locality-aware data access control [START_REF] Shi | LDAC: Locality-Aware Data Access Control for Large-Scale Multicore Cache Hierarchies[END_REF], software-defined cache hierarchies [START_REF] Tsai | Jenga: Software-defined cache hierarchies[END_REF], and urgency-based prioritization of I/O across caching hierarchies [START_REF] Kandemir | On Urgency of I/O Operations[END_REF]. Several approaches specifically focus on the problem of prefetching for HPC using multi-tiered storage, notably eliminating cache pollution and redundancy [START_REF] Devarajan | HFetch: Hierarchical Data Prefetching for Scientific Workflows in Multi-Tiered Storage Environments[END_REF] and userlevel access pattern modeling [START_REF] Qin | Leveraging user access patterns and advanced cyberinfrastructure to accelerate data delivery from shared-use scientific observatories[END_REF]. Of particular relevance are data pipelines in the context of machine learning. These abstractions (e.g. Nvidia DALI [START_REF] Zolnouri | Importance of data loading pipeline in training DNNs[END_REF]) are used by AI runtimes (e.g., Tensorflow, PyTorch) during model training for sampling the training data. This is often achieved using pseudo-random number generators, whose seed can be exploited to enable foreknowledge about the access order, and, subsequently, optimized prefetching based on reuse distance [START_REF] Dryden | Clairvoyant Prefetching for Distributed Machine Learning I/O[END_REF][START_REF] Liu | Lobster: Load Balance-Aware I/O for Distributed DNN Training[END_REF]. Such approaches treat the problem of caching and prefetching separately, without exploiting synergies between the interleaving of producer-consumer checkpointing patterns. While this limits the performance and scalability of such approaches in the scenarios we consider, several ideas are complementary to our proposal and could be used to enhance our proposal.

To our best knowledge, we are the first to consider the case of combined asynchronous multi-level strategies for flushing and prefetching of checkpoints that exploit foreknowledge about the access order.

SYSTEM DESIGN 4.1 Design Principles

4.1.1 Dynamic Foreknowledge of Read Order using Prefetching Hints. As mentioned in Section 2, we assume that a large part of the checkpoint read order is known in advance, which enables optimized prefetching. In the ideal case, the order is fixed from the beginning and remains static (e.g., adjoints such as RTM [START_REF] Plessix | A review of the adjoint-state method for computing the gradient of a functional with geophysical applications[END_REF] and quantum optimal control [START_REF] Hari | Reducing Memory Requirements of Quantum Optimal Control[END_REF] produce a fixed number of checkpoints revisited in reverse order). However, many applications have partial foreknowledge of the checkpoint read order, which may evolve based on various runtime conditions. To accommodate the need for dynamic foreknowledge, we propose to maintain a separate restore order queue for each process. The queue can be used by each process to programmatically enqueue as many hints about the future read requests as it desires (potentially covering the whole checkpoint history from the beginning) at runtime. These application hints can be enqueued at any time and can be interleaved with checkpoint and restore requests. For simplicity, we assume that once enqueued, hints cannot be revoked, i.e., the application is not allowed to "change its mind". Note that the application does not need to provide the hints directly; they can also be provided by higher-level I/O middleware, e.g., by using predictors [START_REF] Devarajan | HFetch: Hierarchical Data Prefetching for Scientific Workflows in Multi-Tiered Storage Environments[END_REF]. Furthermore, since the hints have an advisory role, i.e., the read requests can deviate from the hints, our proposal is not limited in its applicability to the general case.

Shared

Multi-level Cache Hierarchy for Flush/Prefetch. Asynchronous flushing of checkpoints over a multi-level cache hierarchy involves cascading data transfers from the fastest (GPU memory) tier to the slowest (SSD or parallel file system) tier, while asynchronous prefetching of checkpoints follows the opposite path. A naive strategy could simply manage a separate space on each tier to handle the flushing and prefetching strategies separately. Such an approach is simple to design and implement because it requires minimal coordination between flushing and prefetching. However, it has two major disadvantages. First, it leads to high under-utilization of the scarce cache space (especially GPU memory). Second, it leads to uncontrolled competition for shared I/O bandwidth between the flushing and prefetching. Both of these aspects result in significant degradation of I/O performance and scalability. Therefore, a key design choice in our approach is to maximize the cache sharing and control the competition between flushing and prefetching across as many processes as possible, allowing the application to run prefetches in parallel with the flushes.

4.1.3 Unified Flush/Prefetch Support using Finite-State Machine Life Cycle. Even with improved cache utilization due to maximized sharing across the cache tiers, the available capacity on each tier is typically limited and therefore evictions to slower tiers are unavoidable. However, unlike the case of a traditional cache where the benefits of not evicting an object are well-understood (e.g., evict the least frequently used object), the interleaving of flushes and prefetches mandates a new model to uniquely identify the validity of evictions across the cache tiers for a producer-consumer pattern.

To this end, we introduce a life cycle for every checkpoint instance on all cache tiers. The life cycle is illustrated in Figure 1 and is based on a finite-state machine. Specifically, every new checkpoint is born into the INIT state and transitions into either the Checkpointing path or the Prefetching path, depending on whether it serves a checkpoint or restore request. In the Checkpointing path, the checkpoint waits in the Write in Progress state until the pending transfers from the faster (or the application buffer) to the slower cache tier have finished. Then, it transitions into the Write Complete state. If no pending restore or prefetch request exists for the checkpoint, then it transitions further into the Flushed state, which makes it eligible for eviction. If the checkpoint was successfully flushed and evicted from the cache, then a subsequent prefetch hint will trigger the Prefetching path. In this case, the checkpoint waits in the Read in Progress state until the pending transfers from the slower to the faster cache tier are complete, which transitions it into the Read Complete state. Otherwise, the checkpoint is either in the process of being flushed or was already flushed but not evicted yet. In this case, the checkpoint transitions directly into the Read Complete state. Regardless of how this state was reached, the checkpoint will not be evicted until the restore request has finished copying the data to the application buffer, which triggers a final transition into the Consumed state, at which point the checkpoint is again eligible for eviction. Using this approach, individual checkpoints are aware of any concurrent flushes and prefetches that affect them across all cache tiers, thereby mitigating the competition and improving the caching utilization for the combined checkpoint and restore request interleaving.

Contiguous Cache Buffer

Pre-allocation and Pinning. Allocating memory on the cache tiers on-demand to hold the checkpoints is expensive. For some types of cache tiers, the allocation cost is higher than the transfer cost. For example, on Nvidia A100 GPU HBM, memory allocation speed and GPU-to-GPU transfer speed are about 1 TB/s. However, pinned memory can be allocated on the host cache at about 4 GB/s, whereas the GPU cache-to-host cache transfer speed is about 25 GB/s. Therefore, in our design, we opt to pay the expensive allocation cost (and, where applicable, the pinning cost) upfront at the initialization time by using a single contiguous cache buffer on each node-local storage tier per process, which is reused for the whole application runtime. It is worthwhile to note that such a reuse strategy can also be complemented by techniques to hide the overhead of memory pinning asynchronously, as described in [START_REF] Maurya | Towards Efficient Cache Allocation for High-Frequency Checkpointing[END_REF]. However, this is outside the scope of this paper.

Fragmentation Mitigation of Cache Buffers.

When the checkpoint sizes are identical, the management of the cache buffer is straightforward: each eviction creates a gap that is large enough to accommodate a new checkpoint. However, when the checkpoint sizes are not identical (e.g., because they are compressed) it is difficult to avoid fragmentation. For example, compressed checkpoint sizes are difficult to predict in advance, and even if it were possible to predict them, the interleaving of producing and consuming checkpoints under concurrency will lead to many small contiguous gaps in the cache buffer that cannot be individually used to hold a new checkpoint, even if their combined size would be sufficient to do so. To solve this problem, we propose a fragmentation mitigation approach that is closely coupled with the eviction strategy: we take into consideration not only what checkpoints are the best candidates for eviction, but also how these checkpoints are bordering the gaps around them. This leads to interesting trade-offs. For example, a small checkpoint may not be a good candidate for eviction by itself but becomes so if it is surrounded by large gaps. Furthermore, evictions of single checkpoints may not be sufficient to accommodate new writes of large checkpoints. Thus, we have to determine the value of evicting an entire contiguous region in the cache tier that is potentially made of interleaving of gaps and checkpoints. To this end, we introduce a gap-aware eviction strategy that coalesces gaps and checkpoints during evictions to form a single contiguous gap where we can fit a new checkpoint.

Gap-aware Eviction Policy based on Sliding-window Scoring.

We propose a strategy to identify the value of contiguous regions considered for evictions that works as follows: we start by assigning each checkpoint a score that indicates the estimated overhead on the future restore requests it causes if it were allowed to be evicted. This score is derived from several aspects. First, we obviously prefer consumed checkpoints, followed by flushed checkpoints (which are immediately evictable). If there is a flush pending on a checkpoint and no prefetch in progress, then we prefer the checkpoint whose estimated flush completion time is the smallest based on its size and the bandwidth between the cache tiers. This minimizes the wait time until the transition into the Flushed state makes the checkpoint evictable. Third, we prefer to evict a checkpoint for which there is either no prefetching hint available, or otherwise, its prefetching hint is the farthest away from the head of the restore order queue, i.e., it will be restored last. This excludes any checkpoint for which a prefetch has already started and thus, according to its life cycle, it is not evictable. For the purpose of scoring, gaps are treated like regular checkpoints and have the highest eviction priority. Once we obtained a score for each checkpoint, we combine the scores of consecutive gaps and checkpoints until a window is obtained that is large enough to accommodate a new checkpoint. Then, we slide this window of variable size over the entire cache buffer and retain the best performer. This process is detailed in Section 4.2.

Gap-aware Eviction Policy

In this section, we propose an algorithm for the gap-aware eviction strategy introduced in Section 4.1. As outlined in Algorithm 1, this strategy aims to identify the set of consecutive fragments (represented either by checkpoints or gaps in the cache buffer) such that: (1) they form a gap in the cache buffer that is large enough to write a new checkpoint 𝑐𝑘𝑝𝑡_𝑛𝑒𝑤 of size 𝑠𝑖𝑧𝑒_𝑛𝑒𝑤;

(2) the impact, measured as blocking duration, on subsequent restore requests is minimized. Note that writing a checkpoint into the cache buffer can be triggered either by a checkpoint request or a prefetch operation. We assume that the cache buffer keeps an allocation table 𝐴 for its checkpoints with tuples of the form ⟨𝑐𝑘𝑝𝑡_𝑖𝑑, 𝑜 𝑓 𝑓 𝑠𝑒𝑡, 𝑠𝑖𝑧𝑒, 𝑠𝑡𝑎𝑡𝑒_𝑡𝑠⟩, where 𝑐𝑘𝑝𝑡_𝑖𝑑 denotes its unique id, 𝑜 𝑓 𝑓 𝑠𝑒𝑡 and 𝑠𝑖𝑧𝑒 mark segments of the cache buffer occupied by the checkpoint, and 𝑠𝑡𝑎𝑡𝑒_𝑡𝑠 is the estimated time left at which the checkpoint will reach an evictable state in its life cycle (either Flushed or Completed). We also assume that we have access to the restore order queue, denoted 𝑃. The algorithm uses a sliding window approach to iterate over all fragments of the cache buffer. Specifically, a 𝑤𝑖𝑛𝑑𝑜𝑤 is a sequence of consecutive fragments in 𝐴, with the first denoted 𝑖 and the last denoted 𝑗. Each window is assigned two scores: (1) the 𝑝_𝑠𝑐𝑜𝑟𝑒, which measures the estimated total blocking duration until all fragments in the window are evictable; (2) the 𝑠_𝑠𝑐𝑜𝑟𝑒, which measures the sum of the prefetch distances of each checkpoint belonging to the window. The prefetch distance is the number of elements in 𝑃 between the position of a checkpoint 𝑗 in 𝑃 and the head of 𝑃 and measures how far a checkpoint is from being prefetched. Ideally, we want to minimize (1) and maximize (2) simultaneously. However, if this is not possible, we prioritize (1) with the assumption that waiting and doing nothing while evictions become eligible causes a more negative impact than suboptimal decisions for [START_REF] Tariq Alturhkestani | Maximizing I/O Bandwidth for Reverse Time Migration on Heterogeneous Large-Scale Systems[END_REF].

Once we identify the window with the best score (delimited by 𝑟 _𝑠𝑡𝑎𝑟𝑡 and 𝑟 _𝑒𝑛𝑑), we wait for its checkpoints to become evictable, By adjusting both scores every time we increment 𝑖 and 𝑗 (rather than recalculating the score for each window), we can achieve an optimal O (N ) complexity, where 𝑁 is the number of checkpoints cached on the given memory tier. This aspect is important, because a long response time may delay the data transfer and thus lead to lower overall throughput, especially when fast tiers are involved.

Implementation

We implemented our approach in VELOC [START_REF] Nicolae | VeloC: Towards High Performance Adaptive Asynchronous Checkpointing at Large Scale[END_REF], a production-ready, multi-level checkpointing framework. We added support for GPUaware multi-level flushing and prefetching with the design principles noted in Section 4.1. The default VELOC API exposes basic primitives to declare the memory regions that must be checkpointed (VELOC_Mem_protect), write the defined memory regions into a new checkpoint ID (VELOC_Checkpoint) and read the memory regions from a given checkpoint ID (VELOC_Restart). We extend the VELOC API with two new primitives: (1) VELOC_Prefetch_enqueue, which enables a process to provide a hint about the next checkpoint ID it intends to restore in the future; and (2) VELOC_Prefetch_start, which allows the application to indicate when to start prefetching. The latter is helpful (but optional) in scenarios when it is desirable to delay the asynchronous prefetches in order to avoid unnecessary interference with the asynchronous flushes. For example, this would be the case when the application writes the checkpoints in a forward pass, then reads them back in a backward pass, as illustrated in Listing 1.

Asynchronous Flushing and Prefetching.

We developed a multithreaded approach that serves checkpoint and restore requests issued in the main application thread (𝑇 𝑚𝑎𝑖𝑛 ) using dedicated background threads for asynchronous I/O (GPU to host cache flushes: 𝑇 𝐷2𝐻 , host cache to local SSD flushes: 𝑇 𝐻 2𝐹 , prefetches on all tiers: 𝑇 𝑃 𝐹 ). Additionally, since application processes may use the default CUDA stream, our implementation creates a separate set of dedicated CUDA streams, which are then used by the background threads. Using this approach, transfers between different memory layers, such as GPU-to-GPU, GPU-to-host, and host-to-GPU can be efficiently overlapped thanks to dedicated copy engines employed by the Nvidia GPUs and the CUDA driver.

Workflows.

Next, we detail the workflows of the checkpoint/flush and prefetch/restore requests in Figure 2 and Figure 3, respectively. We start by enqueuing the prefetching hints (Line 3 of Listing 1) after completing the initialization phase. Then, the memory regions that need to be checkpointed as protected using VELOC_Mem_protect (Line 4). During the checkpointing phase (Lines 5-8), VELOC_Checkpoint (Line 7) blocks until the protected memory regions are copied to the GPU cache using the main thread 𝑇 𝑚𝑎𝑖𝑛 , at which point 𝑇 𝐷2𝐻 is notified to begin flushing. In turn, immediately after flushing finished to the host cache, 𝑇 𝐻 2𝐹 is notified to flush to the local SSD (and further to a parallel file system if persistence is required). Prefetching is initiated by the 𝑇 𝑃𝐹 thread, which respects the finite-state machine described in Figure 1 for each tier. Specifically, if the requested checkpoint is present on the GPU cache, VELOC_Restart (Line 13) restores the declared memory regions from the checkpoint and marks its state as Read Consumed. Otherwise, VELOC_Restart blocks until 𝑇 𝑃 𝐹 promotes the checkpoint to the faster tiers as follows: space is reserved for it on the faster tier (by waiting for evictions if necessary) and marked as Read in Progress, then the checkpoint is copied to the reserved space on the faster tier and marked as Read Completed, while the original is marked as Read Consumed.

EVALUATION 5.1 Experimental Setup

We evaluated our approach on the ThetaGPU HPC testbed [START_REF]Theta GPU[END_REF], which consists of 24 Nvidia DGX A100 nodes [START_REF] Nvidia | NVIDIA DGX A100 System Architecture[END_REF]. Each node is equipped with 1 TB DDR4 memory (20 GB/s, 8 NUMA domains), two 64-core AMD Rome CPUs (256 threads), four 3.84 TB Gen 4 NVMe drives (4 GB/s per drive), and eight Nvidia A100 Tensor core GPUs (for a total of 320 GB HBM2e memory). The nodes have access to a 10 PB Lustre parallel file system [START_REF] Schwan | Lustre: Building a file system for 1000-node clusters[END_REF] that is accessible through a POSIX mount point. On each node, the eight A100 GPUs are interconnected using 6 NVSwitches. Furthermore, they can access the host memory through a PCIe Gen 4 interface. The peak unidirectional device-to-device and pinned device-to-host (and vice versa) bandwidths on each GPU are 1 TB/s and 25 GB/s, respectively. Two GPUs share the same PCIe 4 link to the host memory, which has two implications. First, only four of the eight NUMA domains are directly accessible from the GPUs. Second, there is a contention for the device-to-host memory bandwidth. We use up to four nodes, i.e., 32 GPUs, for our experiments.

Compared Approaches

Adaptable Input Output System version 2 (ADIOS2

). This is a data management runtime for HPC applications that aims to provide scalable I/O on HPC infrastructures [START_REF] Godoy | ADIOS 2: The Adaptable Input Output System. A framework for high-performance data management[END_REF]. It provides a set of reusable and extendable components for managing data presentation and transport engines. In our comparison, we use the BP5 transport engine, which enables deferred (asynchronous) I/O to the fast Gen 4 NVMe storage by first buffering in the main memory. We use the adios2::MemorySpace::CUDA feature, which instructs ADIOS2 that the data to be checkpointed resides on the GPU memory. Unlike our approach, ADIOS2 does not use the spare GPU capacity for caching and prefetching and instead performs the ondemand movement of data between the GPU memory and host memory. We choose ADIOS2 because it is representative of a multilevel asynchronous I/O framework that handles checkpointing of GPU data structures on-demand, which is a widely used approach in the production settings of the HPC domain.

Nvidia Unified Virtual Memory (UVM).

The UVM is a popular memory management technique introduced by Nvidia to transparently manage data movement across GPUs and host memory tiers in a cache-coherent fashion. Data movement across these tiers is governed by various heuristics, such as hardware-based counters, memory access pattern, amount of available device memory, prefetching and eviction policies, page-fault replay policy, and memory device configured by the application [START_REF] Allen | Demystifying GPU UVM Cost with Deep Runtime and Workload Analysis[END_REF][START_REF] Ganguly | Interplay between Hardware Prefetcher and Page Eviction Policy in CPU-GPU Unified Virtual Memory[END_REF]. In addition to transparent data movement, UVM provides several other benefits, such as on-demand memory allocation, dynamic expansion and reduction of UVM memory allocated on the device HBM, and zero-copy data access to achieve low latency while eliminating the overheads of copying data between memory tiers. We choose UVM because it is representative of a system-level transparent caching and prefetching solution that adopts a combination of best practices introduced by state-of-the-art over time.

Note that just like our proposal, UVM enables the application to provide hints about the memory access pattern. We provide these hints by exploiting the cudaMemAdvise and cudaMemPrefetchAsync primitives. Specifically, we make use of two flags that are relevant for cudaMemAdvise: (1) cudaMemAdviseSetPreferredLocation, which advises the CUDA driver to move the checkpoint to either the host memory (in case of flushing) or to the GPU memory (in case of prefetching). Once a checkpoint is read by the application, we again set its preferred location to the host memory that enables the CUDA driver to immediately evict consumed checkpoints, which otherwise would have been retained due to the default LRU eviction policy; (2) cudaMemAdviseSetAccessedBy, which enables faster access for specific processes. In addition, we initiate asynchronous prefetching using cudaMemPrefetchAsync in order to make sure that the checkpoints that are already present on the device memory but not yet consumed by the application are not evicted prematurely. Since UVM does not expose a mechanism to block further prefetch operations until the previously prefetched memory regions on the device are consumed by the application, we explicitly control the prefetch request by tracking the amount of device memory consumed (when prefetching operations load checkpoint data for application consumption) and released (when application consumes the prefetched checkpoints). During the restore phase, if the amount of memory consumed by the prefetch operations exceeds the amount of the UVM device cache, the prefetching thread waits until the application consumes checkpoints from the UVM device cache. In addition to avoiding page thrashing, this also allows efficient use of the limited host-to-device interconnect bandwidth. Thanks to these optimizations, we aim for a fair comparison that informs the CUDA driver about the foreknowledge of the access pattern of the checkpoint data.

Our Proposal (Score).

We compare the state-of-the-art approaches mentioned above with our proposal, whose key design principles and implementation details are outlined in Section 4.1 and Section 4.3, respectively. We denote our proposal Score, after the gap-aware eviction policy based on the sliding-window scoring algorithm introduced in Section 4.2.

Number of Prefetching

Hints. We vary the degree of foreknowledge by varying the amount of prefetching hints provided by the application as follows: unlimited (the restore order is fully known in advance); one prefetching hint at a time (the application specifies what checkpoint to prefetch for the next iteration at the beginning of the current iteration) and no prefetching hints (corresponding to direct read requests where foreknowledge is not available). By varying the degree of foreknowledge, we study the impact of our design principles even in the case of the scarcity or absence of prefetching hints. The combination of all the compared approaches is summarized in Table 1.

Evaluation Methodology

We evaluate each approach used in our comparison using the scenarios and metrics described below.

Application Benchmark -RTM.

As an application, we consider the case of Reverse Time Migration (RTM) [START_REF] Suha | GeoDRIVE -a high performance computing flexible platform for seismic applications[END_REF], an HPC adjoint computation commonly used in the oil and gas industry. Specifically, RTM is used to generate subsurface images from seismic data. This process involves wave fields generated in the forward pass, which are checkpointed and consumed in a predefined order during the backward pass to cross-correlate and form the subsurface image. RTM can run either in embarrassingly parallel mode (many shots in parallel, each on a different GPU) or tightly coupled mode (one shot across multiple GPUs with synchronization steps at each iteration).

In this work, due to a large number of compared configurations, we designed a series of benchmarks that emulate the behavior of RTM based on traces obtained during large-scale production runs that record the checkpoint sizes. Specifically, our benchmarks run trivial iterations, by sleeping to simulate computations between consecutive checkpoints or restore operations, but generate the exact same checkpoint sizes as in the RTM traces.

Restore Order

Patterns. We consider a set of distributed processes, each of which performs a single forward pass that writes a checkpoint at each iteration, then starts the prefetching, and then performs a backward pass that reads a checkpoint at each iteration. This corresponds to the example used in Listing 1. We refer to such a run as a shot. In this context, we evaluate all three restore order patterns discussed in Section 1: Sequential (the backward pass consumes the checkpoints in the same order they were generated during the forward pass), Reverse (the backward pass consumes the checkpoints in the reverse order than they were generated during the forward pass), and Irregular (the backward pass consumes the checkpoints in a random but predetermined order).

Traces and Distribution of Checkpoint

Sizes. In a typical shot, RTM compresses the image checkpoints by default during the forward pass, which leads to variable checkpoint sizes not only across different iterations but within the same iteration among different processes. The average compression ratio of these checkpoints is ∼30×. From over 1600 shot traces, 32 representative shots were chosen for benchmarks on up to 32 GPUs (4 compute nodes). Figure 4 shows the minimum, maximum, and average sizes per snapshot for 384 snapshots. The aggregated size of checkpoints on each shot Figure 5: Average checkpoint+restore throughput across 8 GPUs for uniform (Fig. 5a) and variable (Fig. 5b) sized checkpoints when we WAIT for checkpoints to be flushed from the caches before the restore phase. Higher is better. Y-axis is in log scale.

ranges between 38 GB to 50 GB. From a caching perspective, variable checkpoint sizes are difficult to handle because they introduce fragmentation. We complement this scenario with another that uses uniform checkpoint sizes: 128 MB per checkpoint, which roughly corresponds to the 50 percentile distribution of the 1600 RTM shot traces, leading to 384 checkpoints amounting to 48 GB per GPU. This enables us to study the effectiveness of our approach in the absence of cache fragmentation as well.

Multi-level Cache Tiers.

Unless noted otherwise, we use the following cache configurations: for each process, we reserve 4 GB GPU memory (i.e., 10% of the capacity) as a device-level cache and 32 GB main memory (i.e., 25% of the total capacity consumed by 8 processes) as a pinned host cache. Additionally, we use a nodelocal SSD as the slowest tier, which is shared by all processes colocated on the same compute node. The SSD can accommodate all checkpoints produced on the same compute node during the entire forward pass of RTM. If this is not enough for other applications, an external storage tier (e.g., parallel file system) shared by all compute nodes can be used to store the checkpoint data.

Performance

Metrics. We use the following performance metrics in our evaluation:

(1) Write and read throughput observed by the application for two cases. First, wait for checkpoints to be flushed from the caches before starting the restore phase. Second, the restore phase starts immediately after the checkpointing phase. (2) Read throughput observed by the application at different iterations.

(3) Prefetch distance, i.e., the number of successive checkpoints successfully prefetched in advance at the moment when a given checkpoint is restored to measure the effectiveness of the prefetch strategy.

These metrics are obtained in a variety of settings, i.e., single node vs. multiple nodes, tightly coupled (barrier after each iteration) and embarrassingly parallel (no synchronization after each iteration) scenarios, and variable compute complexity (simulated sleep duration) of each iteration and variable GPU cache sizes.

Performance Results

5.4.1 Checkpointing Throughput. Our first set of experiments measures the average checkpointing and restoring throughput as observed by the application (total checkpoint size divided by blocking time of checkpoint and restore operations respectively) over the entire shot. Note that the data movement (flush and prefetch) across slower tiers is asynchronous, therefore blocking time only includes the duration of writes or reads from the GPU/UVM cache and any delays due to evictions or prefetches, which is most relevant from the application perspective. We run our benchmarks on a single node (8 GPUs) in embarrassingly parallel mode and fix the checkpoint frequency to 10 ms to be consistent with the RTM behavior.

Restore

Phase Waits for Checkpoint Phase. We first consider the case when the checkpoint and restore phases are sequential, i.e., the restore phase waits for the checkpoint phase to finish flushing to the slowest tier. This scenario captures the cases where checkpoints from the forward pass are persisted for reproducibility, post-hoc, or in-transit analysis. Figure 5a and Figure 5b show the average checkpoint and restore overheads associated with uniform and variable-sized checkpoints respectively. As expected, the performance of the ADIOS2 approach across all restore patterns for uniform and variable checkpoint sizes is the slowest due to the lack of a dedicated device cache tier.

As compared to the optimized UVM approach, our proposed score-based approach shows 2×-74× and 1.6×-52× faster check-point+restore rates for uniform and variable-sized checkpoints, respectively. Note that even without any restore-order hints, our approach outperforms the optimized UVM approach by at least 2×, thanks to directly evicting the consumed checkpoints from the device cache (if already consistently captured on slower memory tiers), as opposed to UVM's approach of migrating the checkpoints before eviction. We observe similar checkpointing throughput for UVM and score-based approaches due to-(1) the problem of slow host cache initialization (mapping and registration of host cache for faster direct-memory transfers) as studied in [START_REF] Maurya | Towards Efficient Cache Allocation for High-Frequency Checkpointing[END_REF]; and (2) the shot contains 48 GB worth of checkpoints, which does not fit in 4 GB Figure 6: Average checkpoint+restore throughput across 8 GPUs for uniform (Fig. 6a) and variable (Fig. 6b) sized checkpoints when we DO NOT WAIT for checkpoints to be flushed before the restore phase. Higher is better. Y-axis is in log scale.

Sequential

GPU cache and 32 GB host cache, triggering waits for evictions to the SSD towards the end of the shot. An interesting observation when waiting for checkpoints to get flushed before restoring is that all checkpoints residing in either of the cache tiers transition to the Flushed or the Read Complete state, thereby, minimizing waits due to evictions. However, this approach incurs an additional overhead of waiting for cache flushes, which amounts to 70 seconds per rank (effective flush rate per rank = 48 GB/70s ≈ 685 MB/s). Since the cache flush rate is uniform for all approaches, we do not show this additional flush time in the graphs to better highlight the impact of our approach on checkpoint and restore operations exclusively.

We also observe that the average checkpoint rate of variablesized checkpoints (Figure 5b) is higher than the corresponding uniform-sized checkpoints (Figure 5a) for all read-orders in the score-based approach. This is because smaller-sized checkpoints at the beginning of the shot (Figure 4) allow for faster evictions by flushing smaller checkpoints between consecutive iterations (of 10ms), and the disparity in checkpoint sizes across multiple processes reduces competition on shared resources such as PCIe lane, host memory allocated on the same NUMA domain, and the file system. On the contrary, we observe that the restore rates of uniform-sized checkpoints are faster compared to the variable-sized scenario. This is because uniform-sized checkpoints do not cause fragmentation on either of the cache tiers, but the variable-sized checkpoints undergo fragmentation and adopt our proposed gapaware eviction policy based on sliding-window scoring (Section 4.1) which may block the prefetch/restore operation until a contiguous region consisting of multiple checkpoints becomes evictable. Nonetheless, we note that the restore rate of our proposed scorebased approach is at least 2× faster than the optimized UVM-based approach and state-of-art ADIOS2 framework (note the representation of the y-axis in log scale).

5.4.3

Restore Phase Immediately Follows Checkpoint Phase. Next, we consider the case when the restore phase begins immediately after the checkpointing phase. This illustrates scenarios such as adjoint computations where the overall job runtime needs to be minimized and the checkpoints need not be persisted. Similar to the case of waiting for flushes after the checkpoint phase, the ADIOS2 is the slowest across all approaches for all read orders and checkpoint sizes.

As observed in Figure 6, our score-based approach outperforms the optimized UVM-based approach by 1.8×-3.6× and 2.9×-22.8× for uniform (Figure 6a) and variable (Figure 6b) checkpoint sizes, respectively. Again, we observe consistent checkpoint throughput across all approaches for various restore order patterns because of slow cache initialization and waiting for evictions (flush to the file system) when both cache tiers have been fully utilized. Note that in this case, we only wait for flushes to the file system to capture the checkpoints that exceed the combined memory capacity of the device and host cache tiers.

Unlike the previous case of waiting for checkpoints to get flushed before starting the restore phase where checkpoints residing on the cache belong to either Flushed or Read Complete states, in this case, the checkpoints residing on the cache tiers can belong to any state of the checkpoint life cycle (Figure 1). Transitioning through the state life cycle leads to longer cache eviction times in the restore phase, due to which the restore rate across various read orders and Figure 8: Impact of varying compute interval (Fig. 8a) and GPU cache size (Fig. 8b) on I/O throughput for variable-sized checkpoints and irregular read order on 8 GPUs. Higher is better. Y-axis is in log scale.

a varying number of hints are magnitudes of order slower than the corresponding restore rates observed for the case of waiting for checkpoints to be flushed before starting the restore phase. However, this approach achieves faster overall execution time by eliminating the additional wait time (∼70s) to flush checkpoints in between the checkpoint and restore phases.

Experiments with variable-sized checkpoints (Figure 6b) show higher restore rates as compared to uniform-sized checkpoints (Figure 6a) due to faster evictions for smaller checkpoint sizes and lower congestion on shared resources because of differences in checkpoint sizes across multiple processes. The case of irregular read order is the most challenging scenario across all executions because of-(1) large fragmentation management overheads due to variable sizes of checkpoints, (2) complex transitions in the state life cycle due to overlapping checkpoint and restore operations across various cache tiers, and (3) non spatio-temporal read-access due to irregular read order. However, even with these complexities, our proposed score-based approach achieves 3.7× faster checkpoint and 38× faster restore rates as compared to the UVM approach.

Since this case of starting the restore phase immediately after the checkpoint phase introduces performance challenges due to fragmentation and complex transitions in the state life cycle, we run the remainder of the experiments without waiting for checkpoints to complete before starting the restore cycle.

Prefetch Distance and

Restore Rate. To explain the observed restore throughput better, we study the effectiveness of the prefetches by measuring the prefetch distance, i.e., the number of successor checkpoints successfully prefetched based on the hints at the moment of a read request. The intuition behind this metric is as follows: the more successor checkpoints are available on the GPU cache, the higher the likelihood of a cache hit in the near future, which translates to a higher restore throughput. Figure 7 shows the average prefetch distance and corresponding restore throughput achieved by our score-based approach at every restore operation for uniformsized checkpoints for the case of sequential read order. As can be observed, the number of checkpoints prefetched in advance is significant (up to 4 checkpoints per process), which provides enough reserve to amortize unexpected or slow prefetches that delay the restore operations. We also note that irrespective of zero consecutive prefetches completed for the case of Single hint, our proposed approach is able to achieve ∼6.3 GB/s restore rate. For the case of No hints, the score-based approach performs the best towards the end of the restore phase because it retains all checkpoints written in the last iterations on the cache. For All hints, the number of next prefetches completed grows steeply until the maximum GPU cache size (4 GB accommodates 32 uniform-sized checkpoints of 128 MB) after which it restores checkpoints up to iteration 96 from the host cache. Once all the checkpoints from the GPU and host cache are consumed, we observe low prefetching distance for further iterations due to slow reads from the file system.

Impact of Variable Checkpoint Interval and GPU Cache Size.

Next, we evaluate the impact of variable checkpoint interval and cache sizes on the compared approaches. Again, we run 8 processes (one per GPU) in embarrassingly parallel mode on a single compute node. We focus on the irregular read order since it is the most problematic to handle. The Single hint scenario has a similar speedup as in Figure 5b, thus we omit it from further evaluations and focus on the extreme cases of No hints and All hints. Figure 8a shows similar checkpoint throughput for all approaches, except for ADIOS2, which exhibits a significantly lower checkpoint throughput (consistent with the previous experiments) regardless of the checkpoint interval. Meanwhile, the restore throughput increases gradually with an increasing checkpoint interval, reaching the peak deviceto-device bandwidth at a 30 ms checkpoint interval. This can be explained by the fact that a larger checkpoint interval allows more and/or larger prefetches to finish in time until the next read request is issued. Next, we study the impact of varying GPU cache sizes for a checkpoint interval of 10 ms. As expected, the checkpointing throughput increases with larger GPU cache sizes, because evictions are delayed. Figure 8b shows up to 2.6× faster checkpoint and restore throughput of our approach as compared to the optimized UVM approach. Since ADIOS2 does not feature a GPU cache, its throughput remains unchanged, but we include it for reference. Avg. ckpt+restore rate (GB/s) Figure 9: Scalability of checkpoint and restore throughput for variable checkpoint sizes and irregular restore order. Higher is better. Note that Y-axis is log scale.

Scalability Study.

We study the scalability of the compared approaches using variable-sized checkpoints. Figure 9 shows the stacked checkpoint and restore throughputs per process for an increasing number of GPUs. At scale, synchronization can introduce significant overheads, therefore we study both the case of embarrassingly parallel and tightly coupled runs (as explained in Section 5.3). In the former case, the competition for shared resources (host memory, links between GPUs and host memory, SSD) due to asynchronous flushes and prefetches is increasing at scale. In the latter case, the competition for resources is reduced due to synchronization at each iteration. Despite increasing competition for resources, both the embarrassingly parallel and tightly coupled cases maintain a relatively stable throughput for an increasing number of GPUs regardless of the considered approach. This is an important observation because it shows that our approach can maintain its advantage over the other approaches at scale. Even at scale, our approach still maintains 22.8× and 13× advantage over the optimized UVM approach for the No hints and Single hint cases, respectively. It is interesting to note that ADIOS2 suffers a drop in checkpointing throughput for an increasing number of processes and thus demonstrates limited scalability, while the opposite effect can be observed for other approaches.

CONCLUSIONS

In this paper, we address the problem of managing large-scale intermediate data generated by the HPC applications that must be checkpointed and restored with high frequency, typically in the order of milliseconds. State-of-the-art solutions have limited support for such scenarios in which the access pattern of restoring data from the saved checkpoint is often known in advance. Specifically, checkpointing frameworks focus on minimizing I/O overhead through asynchronous multi-level flushing techniques, while transparent system-level memory management techniques, such as UVM, can only make limited use of hints about the access pattern.

To address this gap, we designed and developed a checkpointing strategy that co-optimizes asynchronous multi-level strategies for flushing and prefetching of checkpoints by exploiting foreknowledge about the access order. We conduct extensive evaluation with different settings, such as checkpoint sizes, restore order, number of prefetch hints, varying compute interval and cache sizes, all of which are applicable to a wide range of application scenarios. Our proposed approach shows up to an order of magnitude speedup compared with state-of-the-art I/O runtimes, such as ADIOS2, as well as transparent system-level memory management, such as UVM. We demonstrated these benefits both for synchronized and interleaved checkpoint and restore phase at scale. In the future, we plan to explore how to share the host cache across different processes and nodes to load balance variable-sized checkpoints and incorporate support for Nvidia GPUDirect storage.

Figure 1 :

 1 Figure 1: Life cycle of each checkpoint represented as a finitestate machine.

2 𝑛= 3 𝑝_𝑠𝑐𝑜𝑟𝑒 = 0 ; 𝑠_𝑠𝑐𝑜𝑟𝑒 = 0 4 𝑚𝑖𝑛_𝑝_𝑠𝑐𝑜𝑟𝑒 = ∞ ; 𝑚𝑎𝑥_𝑠_𝑠𝑐𝑜𝑟𝑒 = 0 5 𝑟 7 if 𝑖 > 0 then 8 𝑝_𝑠𝑐𝑜𝑟𝑒 17 𝑚𝑖𝑛_𝑝_𝑠𝑐𝑜𝑟𝑒 = 𝑝_𝑠𝑐𝑜𝑟𝑒 18 𝑚𝑎𝑥_𝑠_𝑠𝑐𝑜𝑟𝑒 = 𝑠_𝑠𝑐𝑜𝑟𝑒 19 𝑟 _𝑠𝑡𝑎𝑟𝑡 = 𝑖 20 𝑟 _𝑒𝑛𝑑 = 𝑗 21 𝑠𝑡𝑎𝑟𝑡 22 𝑓 𝑟𝑒𝑒_𝑠𝑖𝑧𝑒 = 0 23 for

 230457817181920212223 |𝐴|; 𝑗 = 0; 𝑤𝑖𝑛𝑑𝑜𝑤 = 0 _𝑠𝑡𝑎𝑟𝑡 ← 0; 𝑟 _𝑒𝑛𝑑 ← 0 6 for 𝑖 ← 0 to 𝑛 and 𝑗 < 𝑛 do -= 𝐴[𝑖 -1].𝑠𝑡𝑎𝑡𝑒_𝑡𝑠 9 𝑠_𝑠𝑐𝑜𝑟𝑒 -= 𝑃 [𝐴[𝑖 -1].𝑐𝑘𝑝𝑡 _𝑖𝑑 ] 10 𝑤𝑖𝑛𝑑𝑜𝑤-= 𝐴[𝑖 -1].𝑠𝑖𝑧𝑒 11 while 𝑤𝑖𝑛𝑑𝑜𝑤 < 𝑠𝑖𝑧𝑒_𝑡𝑎𝑟𝑔𝑒𝑡 and 𝑗 < 𝑛 do 12 𝑝_𝑠𝑐𝑜𝑟𝑒+ = 𝐴[ 𝑗 ].𝑠𝑡𝑎𝑡𝑒_𝑡𝑠 13 𝑠_𝑠𝑐𝑜𝑟𝑒+ = 𝑃 [𝐴[ 𝑗 ].𝑐𝑘𝑝𝑡 _𝑖𝑑 ] 14 𝑤𝑖𝑛𝑑𝑜𝑤+ = 𝐴[ 𝑗 ].𝑠𝑖𝑧𝑒 15 𝑗 + +; 16 if 𝑤𝑖𝑛𝑑𝑜𝑤 ≥ 𝑠𝑖𝑧𝑒_𝑛𝑒𝑤 and (𝑝_𝑠𝑐𝑜𝑟𝑒 < 𝑚𝑖𝑛_𝑝_𝑠𝑐𝑜𝑟𝑒 or (𝑝_𝑠𝑐𝑜𝑟𝑒 == 𝑚𝑖𝑛_𝑝_𝑠𝑐𝑜𝑟𝑒 and 𝑠_𝑠𝑐𝑜𝑟𝑒 > 𝑚𝑎𝑥_𝑠_𝑠𝑐𝑜𝑟𝑒 ) ) then = 𝐴[𝑟 _𝑠𝑡𝑎𝑟𝑡 ].𝑜 𝑓 𝑓 𝑠𝑒𝑡 𝑖 ← 𝑟 _𝑠𝑡𝑎𝑟𝑡 to 𝑟 _𝑒𝑛𝑑 do 24 while 𝐴[𝑖 ].𝑠𝑡𝑎𝑡𝑒_𝑡𝑠 ≠ 0 do 25 𝑤𝑎𝑖𝑡 until A[i] evictable 26 𝑓 𝑟𝑒𝑒_𝑠𝑖𝑧𝑒+ = 𝐴.𝑒𝑟𝑎𝑠𝑒 (𝑖 ) 27 if 𝑓 𝑟𝑒𝑒_𝑠𝑖𝑧𝑒 > 𝑠𝑖𝑧𝑒_𝑛𝑒𝑤 then 28 insert_gap(start+size_new, free_size-size_new) 29 𝑠𝑡𝑎𝑡𝑒_𝑡𝑠 = 𝑝𝑟𝑒𝑑𝑖𝑐𝑡 _𝑒𝑣𝑖𝑐𝑡𝑎𝑏𝑙𝑒 (𝐴, 𝑠𝑖𝑧𝑒_𝑛𝑒𝑤, 𝐵) 30 return 𝐴.𝑖𝑛𝑠𝑒𝑟𝑡 ( ⟨𝑐𝑘𝑝𝑡 _𝑛𝑒𝑤, 𝑠𝑡𝑎𝑟𝑡, 𝑠𝑖𝑧𝑒_𝑛𝑒𝑤, 𝑠𝑡𝑎𝑡𝑒_𝑡𝑠 ⟩ )

Figure 3 :

 3 Figure 3: Workflow of prefetch and restore operations.

Figure 4 :

 4 Figure 4: Size distribution of 32 RTM snapshots.

Figure 7 :

 7 Figure 7: Restore rate and number of next prefetches completed for different timesteps for uniform sizes checkpoints using score-based approach.

Algorithm 1 :

 1 Score-based look-ahead cache eviction. ⟨𝑐𝑘𝑝𝑡 _𝑛𝑒𝑤, 𝑠𝑖𝑧𝑒_𝑛𝑒𝑤 ⟩: checkpoint to written to the cache buffer and its size; 𝐴: the allocation table of the cache buffer, holding tuples ⟨𝑐𝑘𝑝𝑡 _𝑖𝑑, 𝑜 𝑓 𝑓 𝑠𝑒𝑡 , 𝑠𝑖𝑧𝑒, 𝑠𝑡𝑎𝑡𝑒_𝑡𝑠 ⟩; 𝑃 : restore order Output : index of 𝑐𝑘𝑝𝑡 _𝑡𝑎𝑟𝑔𝑒𝑡 in 𝐴 1 Function score_eviction(⟨𝑐𝑘𝑝𝑡 _𝑛𝑒𝑤, 𝑠𝑖𝑧𝑒_𝑛𝑒𝑤 ⟩, 𝐴, 𝑃 ):

Input :

  Figure 2: Workflow of checkpoint operations.then we remove them from 𝐴, and we insert a new tuple in 𝐴 corresponding to the new checkpoint (which overwrites the window). The estimated time until the new checkpoint becomes evictable in the future is based on its state in the life cycle: (1) if the checkpoint is being flushed, then we estimate how long this will take; (2) if the checkpoint is being prefetched, then we estimate how long it will take until the checkpoint will be consumed. The estimation is based on several factors: the size of the checkpoint 𝑠𝑖𝑧𝑒_𝑛𝑒𝑤, bandwidth 𝐵 between the caching tiers, and other enqueued flushes and prefetches in 𝐴 that compete for bandwidth. It is computed by 𝑝𝑟𝑒𝑑𝑖𝑐𝑡_𝑒𝑣𝑖𝑐𝑡𝑎𝑏𝑙𝑒.

			Dotted arrow means wait until destination
	Application		Main app thread memory tier has enough space
				GPU to Host trf thread
				Host to filesystem trf thread
	1	Compute
				D2D copy: App buffer
		2	Checkpoint	to GPU cache
	App wait	Application buffer	GPU cache	GPU HBM
	4	Return control to app	D2H: Checkpoint from
				GPU cache to host cache
			Host cache
	Restore phase			H2F: Checkpoint from
				host cache to filesystem
	Managed by VeloC -GPU	Optional transfer to
				remote storage

Table 1 :

 1 Compared approaches using different number of prefetching hints.

	Notation	Prefetch Hints
	No hints, ADIOS2	0
	No hints, UVM	0
	No hints, Score	0
	Single hint, UVM	1
	Single hint, Score	1
	All hints, UVM	All
	All hints, Score	All

ms) 2 2 2 6 2 10 2 14 2 18 Avg. ckpt+restore rate (GB/s)

	5 Compute interval (2.4 10 15 5.3 8.1 10.5 3.6 8.4 13.9 20.6 20 19.7 36.8 68.4 129.1 4 8.4 14.1 18.6 84.8 315.4 838.8 1203.6 No hints, ADIOS2 No hints, UVM No hints, Score All hints, UVM All hints, Score Checkpoint 30 16.1 28.8 148 37.8 1777.6 Restore	Avg. ckpt+restore rate (GB/s)	2 1 2 2 2 5 2 8 2 11 2 14	2 1.3 1 1.4 8 24.7 No hints, ADIOS2 4 GPU cache size (GB) 8 12 2 4 6.4 3.2 7.8 3.5 13.2 15.1 31.9 187.5 16.2 92.5 142.4 315.5 190.7 No hints, UVM No hints, Score All hints, UVM All hints, Score Checkpoint 16 8.7 19.4 406 137.8 652.6 Restore
	(a) Increasing compute time [Cache: GPU=4 GB, Host=32 GB]	(b) Increasing GPU cache [Compute time=10 ms, Host=32 GB]

ACKNOWLEDGMENTS

This work is supported in part by the ARAMCO Services Company, the U.S. Department of Energy (DOE), Office of Science, Office of Advanced Scientific Computing Research (under contract DEAC02-06CH11357/0F-60169) and the National Science Foundation (award no. 2106634/2106635). Results presented in this paper are obtained using Argonne's ALCF HPC systems.


Daniel Estevez

Photographie, « Deep Play », Harun Farocki Le concepteur émancipé dissensus et conception en architecture

Keywords: Dissensus, conception architecturale, théories la conception, représentation de l'architecture, expérimentation représentationnelle

L'efficacité de la représentation est paradoxale en matière de conception architecturale. Elle repose en effet sur la possibilité pour le concepteur de mettre en oeuvre des procédures de représentation lui permettant de multiplier et de faire diverger les descriptions de la réalité. Notre texte cherche à promouvoir un tel principe de représentation dissensuelle dans le travail de conception en architecture informé également de certaines pratiques artistiques. Cette approche problématise sur plusieurs aspects la notion de représentation unifiée (constitution d'un modèle unique partageable) et se distingue également de la représentation augmentée (dispositifs d'ajout de couches d'informations à de l'espace physique). La représentation dissensuelle s'appuie sur un principe de "séparation critique" radicale des supports de représentation et on peut la relier à plusieurs écrits du philosophe Jacques Rancière sur ce thème. Nous évoquerons ainsi quelques outils de projet proposant de telles séparations affirmées des modes de représentation de la réalité physique.

V i s i o n t e c h n i q u e e t v i s i o n p r a g m a t i q u e d e l a c o n c e p t i o n .

Jürgen Habermas a défini la technique par la formule "activité rationnelle par rapport à une fin et contrôlée par son succès" [START_REF] Habermas | La technique et la science comme idéologie[END_REF] . Si l'on adopte une vision technique de la conception alors se dégage un schéma d'interprétation tacite de l'activité du concepteur. Dans ce schéma, on devra admettre que la conception, en tant que tâche de fabrication, possède un commencement précis et une fin précise et prévisible, c'est à dire que le travail de conception est toujours finalisé par une production concrète, elle converge vers un objet, un bâtiment. Cette vision convergente est dominante dans l'univers de l'aide à la conception dans la mesure où cette dernière est le plus souvent rabattue implicitement sur le champ général de la fabrication. Pour donner une image on pourrait dire que les outils de CAO, par exemple, ne constituent que des éléments particuliers dans la chaîne, plus vaste, des procédures de FAO. Dans ce cadre, deux caractéristiques de la conception technique apparaissent. D'une part tous les acteurs de la fabrication d'un objet ou d'un édifice engagent un travail de planification, de coordination et de communication tendu vers un but unique et non ambigu et d'autre part toute fabrication s'achève quand l'objet est fabriqué.

A l'opposé de la vision technique efficace qui structure la collaboration productive comme un travail convergent guidé par une fin, il est possible d'envisager une vision pragmatique de l'activité de conception et celle-ci reposerait sur une toute autre hypothèse. Dans celle-ci la conception est appréhendée d'abord comme un processus continu qui ne s'achève pas dans la fabrication et la livraison d'un objet ou d'un édifice. Du point de vue de cette conception-processus en effet, la production d'un objet physique donné ne constitue qu'une étape, un arrêt sur image, à l'intérieur d'un développement continu et plus ample qui concerne les cycles de vie et l'évolution de l'objet conçu. La vision pragmatique de la conception cherche donc d'abord à intégrer la réalité temporelle évolutive des édifices. On peut citer à l'appui de cette thèse par exemple Anton Ehrenzweig lorsqu'il écrit : "[...] L'architecte, en dessinant un bâtiment fonctionnel, doit aussi anticiper un certain nombre d'usages possibles qui sont en partie déterminés par des facteurs futurs inconnaissables. En somme, ces formes ouvertes peuvent absorber des aléas purement accidentels qui échappent entièrement à tout type de planning rationnel. Seul leur emploi ultérieur définit [...] les bons édifices." 2 Le sociologue Bruno Latour et l'architecte Albena Yaneva 3 ont donné de leur côté un certain écho à cette réalité instable de l'architecture dans un texte intéressant intitulé, en référence au fusil chrono-photographique de Jules Etienne Marey : « Donnez-moi un fusil et je ferai bouger tous les bâtiments : le point de vue d'une fourmi sur l'architecture ». De même l'architecte hollandais Herman Hertzberger défend-t-il lui aussi une appréhension évolutive de l'architecture, et hisse ce faisant la question de la durabilité des objets au centre des méthodes de conception. Cela le conduit notamment à proposer une vision de la conception comme activité de création fondée prioritairement sur le concept de recyclage permanent : « En Hollande aujourd'hui des écoles deviennent des logements, des bâtiments d'habitation deviennent des bureaux, des bureaux sont transformés en école, etc. Les discussions portent sur la deuxième vie, la troisième vie de chaque édifice. » [START_REF] Hertzberger | conférence à la SFA Paris[END_REF] Cependant nous estimons que c'est certainement le philosophe Jacques Rancière qui a donné à cette vision pragmatique de la conception son assise théorique la plus claire en la structurant autour de la notion générale de dissensus représentationnel [START_REF] Ranciere | Le spectateur émancipé[END_REF] . Ici la conception s'éloigne du paradigme de la fabrication, elle n'est plus convergente, elle introduit des possibilités de choix imprévus, de jugement et d'action continue. Nous voudrions nous attacher ici à illustrer comment le concept de dissensus, lorsqu'on l'applique au champ de la représentation pour la conception architecturale, peut faire surgir un certain nombre de pistes de réflexion nouvelles pour l'aide à la conception des objets de l'architecture. Ces analyses sont également référées à plusieurs expériences représentationnelles et artistiques que nous avons menées dans le cadre de recherches récentes sur les espaces de mobilité [START_REF] Estevez | Aéroports. Représentations et expérimentations en architecture[END_REF] .

U n e a p p r o c h e c r i t i q u e e t o p é r a t i o n n e l l e d e l a c o n c e p t i o n e n a r c h i t e c t u r e . la représentation pour l'architecture ? Si on l'applique au domaine de l'instrumentation de la conception en architecture, cette approche dissensuelle semble s'opposer sur certains aspects au paradigme de la représentation systémique (ou unifiée) qui vise à constituer un modèle informationnel unique et partageable des édifices, et qu'elle se distingue également de la représentation augmentée qui propose des dispositifs ajoutant des couches d'informations à l'espace physique. La représentation dissensuelle fragmente ses outils et discrétise ses procédures. À l'instar par exemple du géométral de l'architecte [START_REF] Estevez | Dessin d'architecture et infographie, nouvelles pratiques graphiques[END_REF] , elle s'appuie sur un principe de séparation critique des supports de représentation. Mais quelle sorte d'efficacité descriptive ces séparations affirmées des modes de représentation de l'espace et de l'architecture peuvent-elles atteindre ? Quels sont les principes opérationnels sur lesquels peuvent s'appuyer ces conduites de projet dissensuel ? Pour aller plus loin, nous devons expliciter un peu les présupposés de deux des principales démarches de représentation informatisée sur lesquelles s'appuient largement aujourd'hui les outils courants de représentation de l'espace bâti. L a r e p r é s e n t a t i o n u n i f i é e . qui peuvent en être faites et parallèlement est nourrie par ces exploitations. Il est donc également loisible de ranger les Systèmes d'Information Géographiques utilisés par l'urbanisme sous le même paradigme de la représentation unifiée. Car dans ces outils chaque carte produite correspond à une extraction de données issues de bases de données urbaines ou géographiques contenant, tel des modèles complets, toutes les informations disponibles. La carte résulte d'une sélection d'informations répondant à des requêtes d'interrogation de sa base, et dans ce sens, quelle que soit la complexité des informations qu'elle transmet, elle est toujours entièrement prédéterminée dans son contenu. Prise dans un système de représentation technique, toute carte correspond donc ici à une donnée quantitative partielle prélevée dans un ensemble informationnel plus vaste. C'est une découpe qui correspond à une réduction d'informations. De nombreux travaux artistiques [START_REF]Else/Where, Mapping New Cartographies of Networks and Territories[END_REF] ont montré les limites de cette approche, car une carte ne peut être considérée de la sorte sans appauvrir considérablement sa fonction interprétative. En tant qu'outil de conception en effet, chaque carte est une représentation partiale c'est-à-dire qu'elle est certes partielle, parcellaire, mais aussi cohérente dans sa signification, tout comme peut l'être un mot dans le langage. Dans la démarche technique, à l'exemple des SIG, la carte est une mesure objective, elle est information. Mais sur le plan artistique la carte résulte d'une lecture cohérente, elle agit non comme une réduction d'informations, mais comme un opérateur capable de déclencher pour le créateur un surplus de signification. L a r e p r é s e n t a t i o n a u g m e n t é e . D'autres démarches de représentation tentent une plus grande prise en compte des aspects non modélisables de la réalité urbaine et architecturale dans le domaine de la conception. On peut regrouper ces approches sous la catégorie de représentation augmentée. Les dispositifs de réalité mixte ou d'espace augmenté (Réalité Virtuelle, informatique ubiquitaire, Réalité Augmentée, etc.) [START_REF] Chalon R | On trouvera une synthèse claire des définitions de chacune de ces notions dans[END_REF] répondent à un objectif d'équipement des situations réelles de telle sorte à "augmenter les propriétés des objets de notre entourage de capacités de traitement d'information" [START_REF] Baudel | Réalité Augmentée : pour une définition et une nouvelle dénomination du domaine[END_REF] . Ici on ne s'appuie pas sur une modélisation unique et hiérarchisée de la réalité urbaine mais on propose des combinaisons ponctuelles entre pratiques réelles et outils numériques dans la représentation. Cependant, sur le plan opérationnel, les objectifs de ces démarches concernent moins la production directe du cadre bâti que la négociation entre acteurs de la production, et d'ailleurs ces dispositifs visent surtout l'observation, la communication et la concertation : "le débat, la négociation jouent un rôle essentiel. Comment le projet urbain devient-il un espace de partage d'informations ou de données, de partage de mémoire et d'expérience mais aussi de controverse pour devenir un espace d'innovation […] avant d'être le support d'un processus de conception ?" [START_REF] Basile | Le projet IP City, Une recherche sur la place des technologies de réalité mixte dans les représentations du projet urbain[END_REF] Nous retiendrons de ce type d'approche en représentation augmentée le fait que les outils, instruments ou équipements technologiques produits sont des microsystèmes cohérents au niveau élémentaire mais sans coordination globale et qui visent à constituer des environnements d'équipements pour la conception. Ce sont donc des dispositifs communicants, compatibles entre eux, interopérables et pouvant regrouper indifféremment des objets physiques et électroniques, ou des logiciels, des appareils mixtes (blackboards numériques, stylo digital, tables à dessin mixtes, interfaces tangibles, etc.) L a r e p r é s e n t a t i o n d i s s e n s u e l l e .

Par rapport aux démarches de représentation qui viennent d'être résumées très brièvement, le principe de représentation dissensuelle, tel que nous le défendons, propose un approche que nous pourrions qualifier d'anti-systémique. Car la dissociation des éléments de représentation vise ici en premier lieu, et paradoxalement, à accentuer les divergences descriptives de la réalité. Dans le domaine de la recherche sur les dispositifs instrumentaux de conception pour l'architecture, cette réalité pragmatique a déjà été soulignée. Ainsi, le chercheur Michel Léglise, prenant pour référence le champ du Cased-Based Reasoning écrit-il : « C'est pourquoi, si les architectes se réfèrent à des oeuvres-cas, de manière consciente ou inconsciente, ils font des emprunts fragmentés de sens dans plusieurs oeuvres précédentes telles qu'ils les ont interprétées, et extraient donc des morceaux de connaissance, pour les ré-articuler dans le contexte de leur problème. Cette opération ressemble plutôt à une déconstruction (une interprétation des précédents) suivie d'une reconstruction. » [START_REF] Léglise | Precedents in Architecture : a Pragmatic Approach[END_REF] Les fragmentations et déconstructions créatives que décrit ici Léglise contredisent les principes qui président à l'élaboration des systèmes de représentation opérationnels courants dans les environnements numériques de conception et de collaboration. Ces outils en effet ont pour objectif de faciliter les coordinations, les communications voire les planifications des actions de la conception. Rendre la circulation plus fluide, plus aisée, plus commode entre dispositifs de représentation, acteurs de la production, concepteurs, décideurs, etc. telle est la vocation des approches consensuelles et coordonnées. Dans ce jeu, qui est celui de la vision technique, les technologies informatiques fonctionnent donc le plus souvent comme des moyens de faire converger les actions soit par un accent mis sur la planification (représentation système ou unifiée) soit par une primauté accordée à la communication (réalité mixte, réalité augmentée). Pour ce dernier cas, typique en effet de la représentation augmentée, la notion d'interface devient évidement centrale car c'est elle qui assure la fluidité des interconnexions entre systèmes, outils, utilisateurs et appareils. Ici encore, les représentations de consensus organisent le travail de conception autour de l'illusion de la transparence et de la fluidité des échanges sans équivoque ni ambiguïté. Mais dans la représentation de dissensus, ce sont au contraire les conflits, les collisions et les dissociations qui opèrent. On y propose de circuler de représentations en représentations, comme de mondes en mondes, sans liens préétablis, sans ressemblances prévues. Plus ces mondes sont étrangers et plus la collision promet d'être productive. Jacques Rancière exprime parfaitement cette démarche dans le domaine du spectacle et de sa signification pour un spectateur : "On ne passe pas de la vision d'un spectacle à une compréhension du monde, puis d'une compréhension intellectuelle à une décision d'action [du spectateur]. On passe d'un monde sensible à un autre monde sensible avec ses nouvelles capacités et incapacités." En architecture non plus, on ne passe pas de façon linéaire de l'énoncé d'un problème à sa résolution et « la conception ne consiste pas, à partir d'une idée globale, à développer une idée formelle, puis une idée fonctionnelle, pour enfin régler des détails. » [START_REF] Léglise | Conception assistée : Modélisation et interprétation[END_REF] Comme l'affirmait Donald A. Shön, la conception fonctionne au contraire exactement sur le modèle d'une conversation avec la situation, il lui faut des obstacles, des détours, des relances, des répliques, des controverses. La représentation de consensus cherche à diminuer les désaccords, à unifier les langages, travaillant à rendre plus facile la communication fonctionnelle. Pour atteindre une efficacité d'action elle construit "un accord entre un mode de présentation sensible et un régime d'interprétation de ses données." [START_REF] Ranciere | Les mésaventures de la pensée critique[END_REF] Tels ne sont pas les principes de l'approche dissensuelle, car celle-ci vise à des transpositions et non à des traductions. O u t i l s p o u r u n e c o n c e p t i o n d i s s e n s s u e l l e . La représentation dissensuelle privilégie donc la performance du concepteur, c'est à dire son action d'appropriation des compétences que possèdent les multiples dispositifs qui entourent son travail. Comme l'écrit Michel Léglise à propos du domaine de l'aide informatisée à la conception : « [il faut] examiner un logiciel non pas sous l'angle de ce qu'il 'sait' faire, mais sous l'angle de ce qu'un acteur peut en faire » [START_REF] Léglise | [END_REF] . Mais le principe de dissensus permet aussi de dégager des outils précis d'expérimentation et des méthodes effectives pour les tâches de conception. Nous avons décrit plusieurs de ces éléments dans l'ouvrage « Aéroports. Représentations et expérimentations en architecture » où l'on trouvera plusieurs développements sur ce point [START_REF] Estevez | [END_REF] . Nous voudrions énoncer ici en guise de conclusion à ce texte quelques uns de ces éléments de stimulation d'une représentation dissensuelle pour la conception en architecture :

Les inventaires (penser/classer) : De nombreux architectes adoptent aujourd'hui des méthodes de représentation fragmentée mettant en oeuvre de véritable travaux d'inventaires actifs. Il s'agit d'engager une représentation raisonnée de la réalité physique destinée affronter la complexité des contextes de projet instables. Le groupe d'architectes japonais Bow-Wow a notamment illustré cette approche dans plusieurs ouvrages dont le plus connu s'intitule "Made in Tokyo". On y voit à l'oeuvre une lecture systématique des architectures spontanées de la capitale japonaise et cette lecture construit une vraie stratégie opératoire d'architecte. Ce genre d'action n'a rien de commun avec une analyse urbaine. On sépare, on distingue pour agir, et cette séparation est ellemême une action projectuelle, le classement est conception.

Les diagrammes : Le terme diagramme possède une double origine étymologique, comme représentation échantillonnée d'abord et puis comme tracé graphique. Cette étymologie permet de saisir l'intérêt du diagramme pour la description de l'architecture sur le registre à la fois intelligible de la mesure (échantillonnages) et sensible de la perception (figurations). De ce point du vue le diagramme fonctionne d'une part comme une carte c'est-à-dire comme un outil de description visible et fragmentaire du territoire et d'autre part comme un outil de manipulation des schèmes d'action du projet. Le prélèvement d'échantillons et leur mise en relation permettent aux diagrammes, comme l'affirment les architectes Reiser et Umemoto, de « produire un modèle abstrait de la matérialité » [START_REF] Reiser + Umemoto | Atlas of novel tectonics[END_REF] . Abstrait et concrets, indéterminés mais précis, les diagrammes donnent une forme accessible aux architectes des stratégies d'intervention possibles sur l'architecture.

Les objets sériels : La répétition des représentations est une condition de la production des connaissances, c'est ce qu'affirme par exemple la chercheuse et géographe Valérie November : « Non seulement les cartes, mais toutes les représentations scientifiques peuvent être réparties selon deux orientations orthogonales : l'orientation mimétique […] et l'orientation navigationnelle (une connexion entre un ensemble de signes [...]). Il n'y a aucun doute sur le fait qu'en termes de gain d'information, seule la seconde orientation peut produire de la connaissance objective. » [START_REF] November | The Territory is the Map -Space in the Age of Digital Navigation[END_REF] Les représentations sérielles, au premier rang desquelles on trouve par exemple les maquettes d'étude chez les architectes contemporains, permettent d'explorer des possibilités multiples, d'expliciter les données tacites ou imprévues dans chaque situation. En ce sens, ces représentations par répétition, par variation et par combinatoire, jouent un rôle réellement productif. Elles multiplient les hypothèses, ouvrent la voie au projet coopératif hétérogène et surtout à l'expérimentation collective.

On perçoit alors l'une des caractéristiques fondamentales des procédures dissensuelles de représentation : elles ne sont pas substitutives des acteurs humains. Leur efficacité paradoxale est plutôt une sorte de productivité subjective. C'est-à-dire que, pour fonctionner, elles accordent toute sa liberté d'action, d'initiative et d'interprétation au concepteur, à l'usager, au spectateur. Ainsi leur efficience est celle de la discontinuité, de la séparation, et leur fonctionnement même dépend entièrement des capacités d'interprétation du concepteur. C'est pourquoi, étant délibérément tournées vers l'action du sujet-concepteur sur son environnement de conception, elles peuvent dans un certain sens être qualifiées d'émancipatrices.

Opérer sur les dimensions qualitatives de l'architecture ou de la ville suppose d'assumer avec précision le caractère intersubjectif des représentations qui sont produites, manipulées et partagées au cours du processus de conception[START_REF] Perez-Gomez A | Architecture and the Crisis of Modern Science[END_REF] . Un tel objectif doit certainement engager les concepteurs à adopter une attitude de conception critique dans leur maniement effectif des outils d'aide à la conception spatiale. Ici le principe critique doit être pris dans un sens méthodologique où interviennent notamment la subjectivité du concepteur, la controverse entre acteurs ou bien la délibération collective. Dans sa définition de la notion de représentation critique pour le domaine artistique, Jacques Rancière introduit un concept clef, celui de segmentation : "En son sens originel, critique veut dire : qui concerne la séparation, la discrimination. Critique est l'art qui déplace les lignes de séparation, qui met de la séparation dans le tissu consensuel du réel, et, pour cela même, brouille les lignes de séparation qui configurent le champ consensuel du donné […] » 8 . Pour Rancière l'efficacité des représentations et la dynamique conception-représentation sont fondées sur l'idée de dissensus représentationnel : « Ce que dissensus veut dire, c'est une organisation du sensible où il n'y a ni réalité cachée sous les apparences, ni régime unique de présentation et d'interprétation du donné imposant à tous son évidence. C'est que toute situation est susceptible d'être fendue en son intérieur, reconfigurée sous un autre régime de perception et de signification. »[START_REF] Ranciere | Les mésaventures de la pensée critique[END_REF] En quoi cette réflexion, élaborée d'abord dans le cadre de la création artistique, peut-elle concerner le champ plus étroit de la conception et de

La première approche, que nous avons qualifiée de représentation unifiée, s'insère étroitement dans la vision technique et systémique de la conception décrite plus haut. Dans une telle démarche, les éléments de représentation sont unifiés et fonctionnent de façon coordonnée, voire synchrone. Les outils de la CAO professionnelle en architecture, ou bien des SIG en urbanisme, sont des exemples de tels systèmes intégrés. Plusieurs travaux ont décrit ce phénomène[START_REF] Léglise | Plaidoyer pour des dispositifs de conception, Contribution1 aux Ateliers Coordonneś[END_REF] . Aujourd'hui, les outils opérationnels "Building Information Models" (BIM) associés aux standards internationaux de communication dits "Informations For Construction" (IFC) correspondent précisément à l'expression la plus récente et la plus intéressante de ces techniques d'intégration et de partage de données. La représentation unifiée repose ici sur un principe de modélisation informationnelle préalable et structurée de l'espace bâti. Dans les approches BIM notamment, le modèle informationnel est susceptible de décrire de façon exhaustive tout bâtiment en terme de sémantique de ses éléments, de relations entre ceux-ci et de propriétés associées (attributs). Ce modèle est unique pour chaque édifice, il peut être exploité et manipulé selon des points de vue spécifiques correspondant à des acteurs différents dans le processus de construction et de conception. La cohésion de ce genre de système peut être alors améliorée par l'augmentation de "l'interopérabilité" de ses composants, c'est-à-dire leur compatibilité logique et d'utilisation. On pourrait affirmer sans schématiser à outrance, que la représentation unifiée existe lorsqu'une base de donnée centralisée ou son équivalent, formant un modèle consistant de la réalité, nourrit toutes les exploitations