João Carlos

Aquino Carvalho

Isabelle Maurin

Athanasios Laliotis

Daniel Bloch
email: daniel.bloch@univ-paris13.fr

Domingos De

Sousa Meneses

Domingos De Sousa

Meneses Near-Field

J C De

Near-field Thermal emission of sapphire at high-T: precisely locating the T dependent polariton resonances in view of energy conversion

Keywords: Near-field, Surface polariton, Casimir-Polder interaction birefringence

niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

The central topic of this presentation concerns information on the spectral location and features of surface polaritons (sapphire in our case) which are at the origin of the well-known enhancement of thermal exchanges in the near-field [1]. A surface polariton is the evanescent electromagnetic mode which describes the thermal field at an interface between material and vacuum in the extreme near-field, i.e. distances much shorter than the wavelength. Considering the interface between vacuum and a (homogeneous) plate, described by a (relative) dielectric permittivity, the key information for the relevant surface mode is contained in the complex "(planar) surface response".

S() = [() -1] / [() + 1]

(1) The dissipative part of this surface response, governed by mS, allows a selective transfer of energy to passing-by particles. This transfer occurs if the particle can resonantly absorb the polariton mode. This scheme may convert thermal radiation, from a classical source for broadband lamps for far-field, into a selective pumping lamp in the near-field. With such an optical pumping induced by near-field thermal radiation, conversion into electrical energy can be the next step, through the thermally induced chemical or optical excitation.

Among standard materials easily operated at high-temperatures, SiC, used for the first demonstrations of the near-field properties of the thermal radiation, exhibits a (near-field) narrow peak around 10.5 µm (28.5 THz, quality factor  100). Sapphire also accepts very high temperatures, with its (near-field) peak around 12.2 µm (~ 24.5 THz) [2]. Note that the precise features of the surface thermal emission vary with temperatures, because the phonons, and hence the surface polariton of the materials, evolve phenomenologically with temperature [3,4], aside from the thermal statistics governing the overall thermal field.

The evaluation of the surface response S() is usually indirect, as derived from the knowledge of the permittivity () [becoming (,T) if the temperature-dependence is available]. Moreover, the permittivity () is most often evaluated from far-field emittance (or reflectance [3]), through a fitting procedure based upon a more or less elaborate model. Alternately, direct measurements for () are sometimes feasible (e.g. with ellipsometry), but for a limited range of frequency (usually requiring partial transparency), and temperature.

This indirect method was used for a sapphire plate in experiments performed at CEMHTI, where equipment specifically devoted to high-temperature emittance is available [5], and with an elaborate multi-parameter model [3]. A principle difficulty in the evaluation of the surface response is that its resonances are located in a range where the permittivity spectrum is very unspecific, and that the salient features of the permittivity itself do not exhibit any resemblance with the major features of the original reflectance or emittance.

On a very similar sapphire plate (same crystal bowl, same super-polishing and annealing), an original Atomic Physics method was used at LPL as a diagnostics. It essentially involves an ultra-narrow atomic filtering of the near-field thermal emission.

The team at LPL is familiar with the probing of long-range atom-surface interaction in the (van der Waals) dipole-dipole interaction limit of the "Casimir-Polder" interaction [6]. Through a detailed analysis of a spectral lineshape -through selective refection spectroscopy [2,6], ensuring the probing of the vicinity of the surface-, we can evaluate, for a given atomic level, the strength of the potential -C 3 z -3 exerted onto the atom by the surface. In particular, C 3 had been shown to depend on temperature owing to thermal statistics [2], and on the specific resonant coupling between a (virtual) atomic emission and a surface polariton mode [7]. Here, the excited Cs level 7P 1/2 is coupled in absorption a 24.687 THz transition to Cs(6D 3/2 ), and the measured temperature evolution of C 3 (T) is essentially dependent on eS() [for =24.687 (2).THz]. Thanks to our lineshape analysis, we extract a rapid temperature evolution of C 3 (T). Its growth slows down at high-temperature, because of a broadening of the sapphire polariton. As a first result (fig. 1), LPL measurements appear more compatible with CEMHTI measurements, and other T-dependent evaluations, than with predictions derived from the standard thermal statistics, and considering for sapphire only the room-temperature properties.

Figure 1: C 3 (T) as extracted from atomic spectroscopy (points) on the Cs transitions 6S 1/2 -7P 1/2 and 6S 1/2 -7P 3/2 (see inset), along with various predictions (lines) derived assuming various models for the sapphire permittivity (). The dashed [8] and dashed-dotted [9] lines use sapphire permittivity as determined solely at room temperature, with C 3 (T) only governed by a thermal statistics factor; the dotted line [4] and the thick line -our measurements at CEMHTI-involve a temperature-dependence for sapphire permittivity.

Similar spectroscopic measurements were performed on the other fine-structure component Cs(7P 3/2 ), whose dominant couplings to Cs(6D 3/2 ) and Cs(6D 5/2 ) (respectively at 19.3 THz and 20.6 THz) fall only on the wings of the sapphire polariton resonance; This makes C 3 (T) predictions rather insensitive to the detailed modeling of the sapphire polariton. Aside from the atom-surface potential -C 3 z -3 , the resonant atom-surface polariton coupling can induce a real aborption of the surface polariton into an atomic excitation towards Cs(6D 3/2 ); no equivalent exists for Cs (7P 3/2 ). The transfer rate, naturally depending on temperature, is governed by a mS().z -3 dependence [with = 24.687 (2).THz]. This effect induces a change in the spectroscopic lineshape, which is nearly equivalent to a decrease of the reflected amplitude -by a factor ~2 for our temperature range. Because of the linearity of our spectroscopic method, the amplitude comparison, between a study for Cs(7P 1/2 ) and for Cs(7P 3/2 ) is significant. The expected decrease is actually not observed, pointing out a disagreement with the predictions derived from far-field measurements.

This discrepancy for mS() has to be considered in line with the fact that even for a moderately sharp surface resonance S(), the complex response is locally (i.e. around resonance) a complex Lorentzian [10]. The acceptable comparison between our two methods for eS, along with the strong divergence for mS, suggest that the exact location of the surface polariton is shifted, with respect to predictions from far-field measurements. In particular, the already sharp value for eS() [at = 24.687 (2).THz] should not be considered as found in the "abnormal dispersion" region of the dispersive response .

As a specific difficulty of sapphire, it is birefringent. This imposes to average the surface response of a c // and of a c  window. Indeed, for the atomic physics experiment, the window is c  -cut (to ensure the cylindrical symmetry of the surface interaction) while field fluctuations have to be integrated over all directions. In the far IR, the optical differences associated to the choice of axis orientation are much more considerable than in the visible range. Moreover, sapphire is known to exhibit several phonon modes (4 or 2, depending on the orientation). This induces uncertainty when evaluating S eff (), as derived from eq.( 1) with

 eff = ( o . e ) 1/2 .
In conclusion, we show that accurate spectral predictions of the surface polariton exhibit various difficulties, possibly of a high-impact to realize an efficient energy transfer in the near-field, as it can be considered with hot SiC for the selective vibrational excitation of molecules such as NH 3 or SF 6 [NB: for our Cs experiment, Cs(7P 1/2 ) is populated only after a prior excitation from the ground state, and the best energy transfer would be a kind of upconversion [11]]. Accurate evaluation of the surface resonances can be also critical for various near-field systems involving thermal diodes. Finally, thermal exchanges between two different crystalline cuts of a single material may exhibit interesting properties, whose predictions remain delicate.


Dominik Huber
email: domi.huber@tum.demaximilian.streubel@tum.deisaias.compres@tum.deschulzm@in.tum.de

Maximilian Streubel

Isaías Comprés

Martin Schulz

Martin Schreiber
email: martin.schreiber@univ-grenoble-alpes.fr

Howard Pritchard
email: howardp@lanl.gov

Towards Dynamic Resource Management with MPI Sessions and PMIx

Keywords: MPI Sessions, dynamic resources, malleability, PMIx

Job management software on peta-and exascale supercomputers continues to provide static resource allocations, from a program's start until its end. Dynamic resource allocation and management is a research direction that has the potential to improve the efficiency of HPC systems and applications by dynamically adapting the resources of an application during its runtime. Resources can be adapted based on past, current or even future system conditions and matching optimization targets. However, the implementation of dynamic resource management is challenging as it requires support across many layers of the software stack, including the programming model.

In this paper, we focus on the latter and present our approach to extend MPI Sessions to support dynamic resource allocations within MPI applications. While some forms of dynamicity already exist in MPI, it is currently limited by requiring global synchronization, being application or application-domain specific, or by suffering from limited support in current HPC system software stacks.

We overcome these limitations with a simple, yet powerful abstraction: resources as process sets, and changes of resources as set operations leading to a graph-based perspective on resource changes. As the main contribution of this work, we provide an implementation of this approach based on MPI Sessions and PMIx. In addition, an illustration of its usage is provided, as well as a discussion about the required extensions of the PMIx standard. We report results based on a prototype implementation with Open MPI using a synthetic application, as well as a PDE solver benchmark on up to four nodes and a total of 112 cores. Overall, our results show the feasibility of our approach, which has only very moderate overheads. We see this first proof-of-concept as an important step towards resource adaptivity based on MPI Sessions.

Introduction

In the last decade the performance of HPC systems has increased significantly, with first systems (officially) reaching exascale in 2022. This is driven by heterogeneous and accelerated architectures, which require a careful resource management and resource allocation to application types and kernels. However, traditionally, resources are assigned statically at the start of a job and are then fixed for the duration of the job, which is often a limitation for the efficiency of resource usage on HPC systems due to variations in the scalability of applications, algorithmic kernels, intermediate idle times, or different suitability to accelerators, to name just a few issues.

As a consequence, resource management must become more dynamic and adaptive in order to be able to counteract these challenges. We will refer to this in the remainder of this paper as dynamicity, which has been shown to improve overall system efficiency with respect to various optimization goals, such as throughput, energy or job queue latency [START_REF] Hungershfer | Efficient resource management for malleable applications[END_REF][START_REF] Suraj Prabhakaran | A Batch System with Efficient Adaptive Scheduling for Malleable and Evolving Applications[END_REF][START_REF] Schreiber | Cluster-Based Parallelization of Simulations on Dynamically Adaptive Grids and Dynamic Resource Management[END_REF][START_REF] Sudarsan | Scheduling resizable parallel applications[END_REF]. Dynamicity can also be advantageous for individual applications: Avoiding a-priori resource fixation and using the global scheduler perspective can improve individual resource efficiency and support urgent computing [START_REF] Maghraoui | Dynamic Malleability in Iterative MPI Applications[END_REF][START_REF] Marchal | Malleable task-graph scheduling with a practical speed-up model[END_REF][START_REF] Martn | Enhancing the performance of malleable MPI applications by using performance-aware dynamic reconfiguration[END_REF][START_REF] Schreiber | Cluster-Based Parallelization of Simulations on Dynamically Adaptive Grids and Dynamic Resource Management[END_REF].

However, achieving dynamicity is challenging, since it not only increases the complexity of the system scheduler and This is a preprint version of our paper. Please find the official one at https://doi.org/10. 1145

MPI Processes

Figure 1: Illustration of resource dynamicity of MPI jobs on an PMIx-based HPC system. The HPC system consists of PMIx based components: a login node, scheduler, resource manager (left) and compute nodes executing the (dynamic) MPI job. All components are connected via a high-speed interconnect. To provide dynamicity, it is necessary to add/remove MPI processes (or, in most cases, full nodes with multiple MPI processes) to/from the MPI job (red box). As the figure shows, this requires not only an interface for writing resource dynamic MPI applications, but also a dynamic MPI runtime environment.

resource manager, but also requires support from the programming models used on the platforms, so that applications can request and react to resource changes accordingly. In this paper we focus on MPI [START_REF]Message Passing Interface Forum[END_REF], as the currently dominating programming model in HPC, and introduce support for resource-adapting MPI jobs, i.e., MPI jobs with a dynamically varying number of MPI processes.

Fig. 1 illustrates a dynamic MPI job running on a typical HPC system. To give a concrete example, consider an MPI application (e.g., a parallel CFD solver on a structured regular grid) running on 𝑛 nodes (job1 ). Now consider the situation where another job (job2 ) is waiting in the job queue because it requires 𝑚 ≪ 𝑛 more nodes than currently available. With dynamic resource management, a scheduler could initiate a resource change to remove 𝑚 nodes from job1 and to assign them to job2, which would allow job2 to start its execution immediately. However, this requires the RTE of job1 to interact with that job, to ensure that MPI processes running on the 𝑚 nodes to be terminated and removed, and that the application can continue its execution correctly without these processes after the resource change.

Many different approaches to introduce dynamicity to MPI have been proposed in the literature using techniques such as checkpointing and process migration [START_REF] Maghraoui | An Architecture for Reconfigurable Iterative MPI Applications in Dynamic Environments[END_REF][START_REF] Utrera | Implementing malleability on MPI jobs[END_REF][START_REF] Vadhiyar | SRS -A Framework for Developing Malleable and Migratable Parallel Applications for Distributed Systems[END_REF], process virtualization [START_REF] Gupta | Towards realizing the potential of malleable jobs[END_REF][START_REF] Huang | Adaptive MPI[END_REF], MPI-2 Dynamic Process Management [START_REF] Sudarsan | ReSHAPE: A Framework for Dynamic Resizing and Scheduling of Homogeneous Applications in a Parallel Environment[END_REF] or dynamic MPI_COMM_WORLD communicators [START_REF] Comprés | Infrastructure and API Extensions for Elastic Execution of MPI Applications[END_REF].

The approach we present in this paper is a continuation of previous work [START_REF] Fecht | An Emulation Layer for Dynamic Resources with MPI Sessions[END_REF], which emulated dynamicity with MPI Sessions for a single application on-top of an MPI version lacking support for MPI Sessions. In this previous work, we introduced our approach to describing dynamicity with resource sets and set operations for the first time and we extend these concepts in this work. So far, MPI Sessions is mainly a new model for MPI initialization that avoids the necessity of a global communicator (MPI_COMM_WORLD). Instead, communication can be derived from sets of processes, which are organized as a dictionary of URIs. Our first contribution is to use these MPI Sessions as a foundation to our dynamicity approach, which will be presented in Sec. 2.

Further, we describe the implementation of a resource dynamic MPI runtime environment based on PMIx. The Process Management Interface -Exascale (PMIx) [START_REF] Castain | PMIx: Process Management for Exascale Environments[END_REF] defines a standardized interface for portable and scalable interaction between components of the Software Management Stack (SMS). It is frequently used by MPI implementations for an MPI runtime environment (RTE) as well as in other system components, such as resource managers and schedulers (see Fig. 1). Thus, a dynamic MPI implementation based on PMIx can facilitate the integration and interactions of dynamic MPI on HPC systems. We use and extend PMIx to realize a dynamic MPI RTE allowing for generic integration with different SMSs, as shown in Sec. [START_REF] Burstedde | p4est: Scalable Algorithms for Parallel Adaptive Mesh Refinement on Forests of Octrees[END_REF].

Finally, we provide a performance evaluation in Sec. 4 and an assessment of our approach in Sec. 5 followed by a summary in Sec. 6.

MPI application perspective on dynamic resources

In this section we propose a new approach for resource dynamicity for MPI applications. We start by describing our design goals in Sec. 2.1. Subsequently, we describe the set and graph-based abstraction we use to approach these design goals. Finally, we present possible MPI interfaces that realize this approach using the MPI Sessions model. // PSet to group M P I _ C o m m _ c r e a t e _ f r o m _ g r o u p ( main_group , NULL , NULL , & main_comm ) // group to c o m m u n i c a t o r M P I _ S e s s i o n _ f i n a l i z e ( session ) ; Listing 1: Illustrative example of the proposed interface for a resource dynamic MPI application in pseudo code. In this example resource changes target the whole application. Arguments followed by IN/OUT are read/written, respectively. Although this code shows blocking API calls, all of them can be realized as non-blocking API calls.

Design Goals

Our approach follows three core design goals:

• Generality: MPI applications are diverse, from linear algebra solvers and graph workloads to adaptive meshes and large scale machine learning. It is not possible to foresee all different application requirements. Our primary goal, therefore, is the creation of a general approach for dynamic resources, which then provides a maximum of flexibility for MPI application developers.

• Separation-of-concerns: The focus of this work is to target the MPI aspects of resource dynamicity, with the goal of providing a separation-of-concerns. By separating the MPI part from other parts, such as resource scheduling (or optimization) and application data reorganization, we enable clean differentiation and composability of solutions. • Performance: Our goal is to provide a low-overhead mechanism to adapt to changing resources. To this end a) unnecessary, global synchronization between processes should be avoided and b) non-blocking interfaces should be provided, e.g., to avoid blocking during MPI process creation such as when using the MPI_Spawn API.

Underlying Abstractions

To achieve our goal of generality, we build our approach on top of a set of abstractions which are again based on concepts of MPI Sessions for extending them.

• Sets: We use sets to denote an unordered grouping of resources. Such sets can be potentially overlapping. For this definition, we refer to resources in an abstract sense describing a logical or physical component to be used as part of an HPC system. In the context of this paper we focus on the specific case of sets including (MPI) processes, with one of these sets referred to as process set (aka. PSet). This concept of PSets is already defined in the MPI standard [START_REF]Message Passing Interface Forum[END_REF] and is an important building block of the MPI Sessions model. • Set operations: Set operations (union, difference, split, join, ...) can be executed on sets to create one or more new sets. A set operation thus can be used to express subsets or changes of sets, which can be used to indicate the scope of resources (re-)assignments. • Graphs: A graph-based perspective on resources and resource changes can be taken by associating vertices with PSets and directed hyperedges with set operations. Having such a graph-theoretical description of resources and resource changes as an abstract building block, will enable us (and others) to layer dynamicity approaches on top of it, like, e.g., scheduling heuristics.

Dynamic MPI Interfaces and an Example

In this section we describe a first design of an interface for a dynamic MPI Sessions model, following the abstractions and design goals explained in the last sections. While the underlying model applies to all kinds of resources, for now, we only consider MPI processes as resources. This design introduces three additional MPI procedures, all related to the MPI Sessions model : MPI_Session_dyn_recv_res_change, MPI_Session_pset_create_op and MPI_Session_dyn_integrate_res_change. We also implemented asynchronous variants of these procedures which is trivial as these procedures can be executed in the background and MPI processes can use the MPI_Request, MPI_Test and MPI_Wait procedures accordingly. Thus, we omit the details of the asynchronous variants.

In the following we describe the interfaces in the course of which we also refer to the provided pseudo code example for a loop-based MPI Sessions application in Lst. 1. A brief description of the new function signatures is included in this code example in lines 2, 11 and 32 respectively. We use the terminology "root process" to denote the one particular process that takes the role of being in charge for processing resource changes. Note that this interface will very likely undergo further changes in the future to approach an even higher level of generality and performance.

MPI Session dyn recv res change: This procedure allows an MPI process to query the RTE for resource change information.

Following the graph-like description from Sec. 2.2, resource change information is always associated with an existing set of resources (a vertex), thus allowing for fine-grained, independent resource management of different application parts. This information includes the type of resource change and the set including resources to be changed, i.e., the delta between the resources before and after the resource change (which we call "delta set").

In this work we only deal with MPI processes as resources. Accordingly, MPI_Session_dyn_recv_res_change queries the RTE for resource change information associated with the PSet name specified by the application (parameter 2). From hereon, we refer to this PSet as the associated PSet. Here, the PSet name mpi://self can be specified as a special means to query information about the most recent resource change where the calling MPI process is included in the delta set.

The procedure returns the type of the resource change (parameter 3) where we currently support • MPI_RC_TYPE_NONE: No resource change information available, • MPI_RC_ADD: Resources (MPI processes) are added, • MPI_RC_SUB: Resources (MPI processes) are removed with more operations such as replacement of MPI processes possibly defined in future versions.

Moreover, in case of available resource change information, it returns the name of a PSet representing the delta between the MPI processes before and after the resource change, i.e., the MPI processes to be added or removed. Hence, from hereon we refer to this PSet as the delta PSet. Additionally, the function returns a flag to indicate if the calling MPI process is included in the delta PSet.

The interface deals with PSets (resource sets including MPI processes), however, similar interfaces could be used to support other types of resource sets. Also, this interface does not enforce or interfere with how the application adapts to the resource change, as it only provides the name of the delta PSet. Thus, it provides the application with the flexibility to use this delta PSet in set operations according to the specific needs of the application.

In the example, this interface is used at two different places: First, by all processes at program start to allow MPI processes to determine their dynamic nature (line 2). Second, by the root process after each iteration of the main loop to check if a new resource change needs to be processed (line 27).

MPI Session dyn pset create op:

This procedure allows applications to request the creation of new PSets based on a specified set operation.

Using our abstraction, a resource change is represented as a delta set. However, the delta set provided by the query operation is not the only option; the set can be further manipulated based on application specific needs. This procedure enables such further manipulations of PSets, resulting in new PSets. Contrary to local set operations performed on MPI Groups, set operations on PSets result in new PSets visible beyond the local MPI process in all MPI processes that could be affected by the process change. Thus, other processes can query these Sets and then create groups from the discovered PSets without having to have the knowledge about the operations that created them.

In particular, this function enables applications to create a new PSet pset result from a set operation of two existing PSets pset1 and pset2, i.e., pset result = pset1 ∘ pset2. Currently possible PSet operations specified by the pset op ∘ parameter are

• MPI_PSETOP_UNION (pset result = pset1 ∪ pset2), • MPI_PSETOP_DIFF (pset result = pset1 ∖ pset2),
• MPI_PSETOP_INTERSECT (pset result = pset1 ∩ pset2) and further operations could be defined to broaden the generality of our approach.

In Lst. 1 the root MPI process uses set operations to create a new_main_pset based on the old main_pset and the queried delta_pset to reflect the addition or removal of resources to/from the application respectively (lines 32 and 39).

MPI Session dyn integrate res change: This procedure is specific to MPI processes as it targets issues related to communication channels and synchronization. On one hand, when adding processes the concrete usage of dynamically added processes might not be known a-priori. Thus, these processes lack knowledge of the PSet name to be used for creating an initial communicator. On the other hand, when removing processes it might be necessary to have a synchronization barrier before processes terminate. Due to the flexibility provided to applications by the procedures defined above, it is not guaranteed that communication channels (MPI communicators) between processes in the delta PSet and/or the associated PSet exist or are desired.

To this end, the MPI_Session_dyn_integrate_res_change procedure provides an out-of-band mechanism for processes to synchronize and exchange a PSet name, without creation of a common communicator. The procedure is collective over the union of the MPI processes in the delta PSet and the associated PSet.

All processes participating in the collective need to specify the same delta PSet name (parameter 2). A synchronization barrier is performed across the participating processes.

Moreover, one process from the associated PSet (referred to as the provider from hereon) can indicate its intention to act as provider via an input flag (parameter 3). Usually, this would be the root process. The PSet name provided by the provider (parameter 4) is published to a range accessible to all MPI processes involved in the collective.

The behavior of the procedure is undefined in the case that multiple MPI processes set the provider flag.

All other MPI processes can lookup the provided name, which will be stored in the buffer (parameter 4). Here, a NULL pointer as buffer is valid and results in no lookup by this process. If no MPI process acts as provider, all MPI processes involved in the collective are required to specify a NULL pointer as buffer.

An additional output flag indicates if the calling process is included in the delta PSet and that the delta PSet is related to a resource change of type MPI_RC_SUB. Thus this flag indicates if processes are required to finalize and terminate their execution.

An optional info object can be used to influence the behavior of the function, e.g., to indicate if the function should wait until the processes are synchronized before returning.

In our example, the original MPI processes call this procedure (line 46), with the root process providing the name of the new_main_pset. The dynamically added MPI processes call this procedure (line 11) to receive the name of the new main PSet. Subsequently both, original and dynamically created processes, create a common communicator from this PSet.

Implementation using Open MPI and PMIx

Next, we illustrate a possible implementation of the MPI procedures described in the previous section. We build on top of the MPI Sessions prototype in Open MPI [START_REF] Gabriel | Open MPI: Goals, Concept, and Design of a Next Generation MPI Implementation[END_REF] as well as its already existing usage of the PMIx interface [START_REF] Castain | PMIx: Process Management for Exascale Environments[END_REF]. The latter provides mechanisms for runtime environments to interact with resource managers and is used by a range of common software components, including Open MPI, as well as by other system software such as SLURM [START_REF] Yoo | SLURM: Simple Linux Utility for Resource Management[END_REF], for bootstraping MPI processes and to realize MPI runtime functionalities. As PMIx provides standardized APIs for portable access to typical system software services, it naturally simplifies the handling of resource dynamicity. Moreover, PMIx already supports the concept of PSets, which is a central component of our approach. Thus, in Sec. 3.1 we discuss the usage of PMIx to implement the new, dynamic MPI interface. Based on this, Sec. 3.2 describes our prototype implementation based on Open MPI, OpenPMIx [START_REF] Castain | PMIx: Process Management for Exascale Environments[END_REF] and the PMIx Reference RunTime Environment (PRRTE). Our particular contribution is to provide insight into the additional work and challenges one has in realizing these particular dynamicity interfaces also on the runtime levels.

PMIx

Even though PMIx already includes some support for dynamicity, we need a few more extensions to support the new MPI functionalities. PMIx acts as a messenger between SMS components and is based on the storage and transfer of key-value pairs, with some keys being predefined. Another important concept of PMIx is the PMIx server abstraction which gives well defined access to PMIx services via function Listing 2: Proposed PMIx functions related to PSet operations. PMIx Pset Op request requests the execution of the specified process set operation. PMIx passes on this request to the host environment, for which the latter provides a pointer to its implementation of the pmix server pset operation fn t function at initialization of the PMIx server. The host environment provides the results of the operation back to the PMIx server using the provided pmix psetop cbfunc t callback.

pointers provided by the host environment. Here, we use the term host environment according to the PMIx definition "to refer to the process hosting the PMIx server library" [18, p. 7], which is the MPI RTE (daemon) in the context of this paper. For the sake of brevity we refer the interested reader to the PMIx 4.0 standard document [START_REF]Process Management Interface for Exascale (PMIx) Standard Version 4[END_REF] for further information.

MPI Session dyn recv res change:

This procedure interacts with the RTE to receive information about resource changes. PMIx supports queries for runtime/system information via the PMIx_Query_info function, allowing the user to provide query keys and qualifiers to specify a query request.

Therefore, for the delta PSet, resource change type and associated PSet parameters of the MPI procedure, we define the corresponding PMIX_RC_PSET, PMIX_RC_TYPE and PMIX_RC_ASSOC_PSET query/qualifier keys respectively.

The PMIx_Query_info function returns key-value pairs, with the values for each query key provided by the host environment. To facilitate the provision of a value for the type of the resource change we define the new enum data type pmix_rc_type_t. So far, this type is one of the three new PMIx constants: PMIX_RC_NONE/ADD/SUB. MPI Session pset create op: In order to support the needed PSet operations introduced by this proposed MPI procedure, we require three new functions in PMIx. We show their signatures in Lst.

MPI Session dyn integrate res change:

The implementation of this MPI procedure does not require any extensions to the PMIx standard.

For the out-of-band transmission of the PSet name, the existing PMIx_Publish/Lookup interface can be used. This interface provides access to a global key-value datastore, where data can be retrieved solely by referral to its key. Thus, in the MPI procedure the provider publishes the PSet name as key-value pair with PMIx_Publish. The used key is required to be well-defined and unique for each resource Figure 2: Overview of the prototype implementation of our approach with Open MPI. This example shows two nodes running a (dynamic) MPI job. We differentiate between four layers (see labels on the left), which are each accessible from the layer above via the interfaces indicated on the right side. The interior of the nodes delineate the required interactions of our dynamicity extensions to MPI with PRRTE using PMIx.

change, e.g. by deriving it from the corresponding delta PSet name. This way, the published PSet name can be retrieved by a corresponding PMIx_Lookup by other processes in the MPI_Session_dyn_integrate_res_change procedure. The PMIx_WAIT attribute has to be provided in PMIx_Lookup to request that the PMIx server should wait until the value is found, e.g., in case it was not yet published.

Subsequently, the PMIx_Fence function is entered, which performs a PMIx barrier across all the processes specified in the pmix_proc_t array, i.e. the processes in the union of the delta PSet and the associated PSet).

After the barrier, the provider uses PMIx_Unpublish to unpublish the corresponding PSet name, which avoids an unnecessary accumulation of key-value pairs in long-running PMIx environments and hence prevents memory leaks.

Implementation with Open MPI, OpenPMIx and PRRTE

In this section, we illustrate how the extended interfaces of MPI and PMIx achieve resource dynamicity with Open MPI. Fig. 2 provides an high-level overview of our prototype. We differentiate between the application layer, the MPI (Open MPI) layer, the PMIx layer (OpenPMIx) and the runtime layer (PRRTE). In the following we describe the necessary actions and interactions of these layers to implement the proposed MPI extensions.

PRRTE master layer: To launch the MPI runtime, an initial process starts PRRTE daemon on each node of the allocation, excepting the one on which it is executing. Here, we call this process PRRTE master, as it is responsible for the orchestration of the runtime services and the control of the resource change procedure. To this end, the PRRTE master provides an entry point for triggering resource changes via PMIx event notifications. The PMIx event handler performs the following actions required for the initiation of a resource change: a) it defines the corresponding delta PSet b) it sends all necessary meta and query data to the daemons and c) it sends the launch command for the corresponding processes in case of a resource addition. In our design, also the (centralized) PSet operation module for executing PSet operations, as well as the PMIx data-server for the global dictionary (publish/lookup) are also hosted by the PRRTE master.

PRRTE daemon layer: On each node of the allocation a PRRTE daemon is running, providing runtime services for the MPI job. The PRRTE master also acts as PRRTE daemon on the node on which it is executing. Each daemon operates a local PMIx server, thus providing function pointers for the PMIx services during the initialization of the server. Particularly relevant for resource changes are the function pointers for five PMIx services: queries, PSet operations, publish/lookup requests and fence operations. We indicate these services in Fig. 2 at the interface between the OpenPMIx and PRRTE layers.

MPI and OpenPMIx layers: In Fig. 2, we denote the three new MPI procedures for resource changes (Sec. 2) at the MPI API level. The implementation of these procedures makes use of the five PMIx services that we list at the PMIx server / PRRTE daemon interface. The lines between the MPI interface and the PRRTE layer illustrate how these services are used by the MPI procedures via PMIx as discussed in Sec. 3.1. To give one example, the implementation of the MPI_Session_pset_create_op procedure uses the PMIx client interface (PMIx_Pset_Op_request) to send the request to the local PMIx server. The PMIx server passes on the request to the local PRRTE daemon (pmix_server_psetop_request_fn_t), which further transmits the request to the PRRTE master. Here, the PSet operation is executed in the PSet operation module and the result is broadcasted to all daemons. The daemon from which the request originated uses the pmix_pset_op_cbfunc_t callback function to call back into its local PMIx server to provide the results to the MPI process.

Performance Evaluation

Next, we provide performance results for a synthetic benchmark as well as for a use case of a Partial Differential Equation (PDE) solver, which has been enhanced to exploit resource dynamicity. For sake of reproducibility we provide the code used for the benchmarks [START_REF] Huber | DynMPI Prototype (version used for benchmarks[END_REF].

System Setup

We use the CoolMuc2 Linux cluster at the Leibniz Supercomputing Centre [START_REF]LRZ Linux Cluster[END_REF] for all experiments , which features 28 cores (56 hardware threads) per node (2 sockets per node each containing Intel Xeon E5-2697V3 processors with 14 cores and 28 hardware threads) and a 64 GB DDR4 memory per node. The interconnect between the nodes is based on FDR14 Infiniband (bandwidth to interconnect per node up to 13.64 GB/s). The cluster runs an instance of the SLURM workload manager for resource scheduling and management. We create an interactive SLURM shell with the salloc command, allocating four nodes and a total of 112 cores. Following that, we launch a daemon job (runtime layer) spanning these four nodes using the prterun command. Our benchmarks (user jobs) run on top of this runtime layer with their size being grown and shrunk dynamically. Thus, the presented performance results do not include any resource allocation and management of the resource management system, but rather provide insight into the overheads of the adaption process of dynamic MPI jobs using our approach.

Benchmarks

We implement a synthetic application similar to the example in Lst. 1 to evaluate the performance characteristics of the suggested interface. As workload we use a problem size of 10 10 elements and ≈ 10 FLOPs per element. The elements are equally distributed over the available processes, however, no communication is required for redistributing elements after a resource change. Resource changes are triggered by the application every 10-th iteration. We report performance results for adding/removing 𝑛 𝑑𝑒𝑙𝑡𝑎 ∈ {28, 56, 84} MPI processes.

We further use the proposed interface to provide dynamic resources to a PDE solver with fixed workload and loadbalancing to evaluate the impact on application performance in a realistic setting. For this we build on top of prior work [START_REF] Streubel | Dynamic Resource Management Using MPI Sessions on p4est[END_REF], which combined an emulation layer for dynamic MPI Sessions [START_REF] Fecht | An Emulation Layer for Dynamic Resources with MPI Sessions[END_REF] with a Shallow-Water-Equation solver [START_REF]Shallow Water Equations Teaching Code[END_REF] and the P4est library [START_REF] Burstedde | p4est: Scalable Algorithms for Parallel Adaptive Mesh Refinement on Forests of Octrees[END_REF]. We replace the emulation layer with our prototype implementation for dynamic MPI Sessions. We use a uniform grid of size 𝑆 = 4 10 , a simulation time of 𝑡 = 10 seconds which is split in 𝑐 = 10 phases, and the application triggering resource changes each phase. We report the results for a simulation run starting with 𝑛𝑠𝑡𝑎𝑟𝑡 = 28 processes and adding 𝑛 𝑑𝑒𝑙𝑡𝑎 = 28 processes per resource change up to a maximum of 𝑛 𝑒𝑛𝑑 = 112 processes.

Performance Results

Synthetic Benchmark: We first discuss the performance results of the synthetic benchmark, which we list in Tab. 1.

For resource addition, we measure a total time of up to 1.69 seconds on the RTE level, which is dominated by the MPI initialization (MPI_Session_init) of the added processes (up to 1.42 seconds). The sole overhead introduced on the RTE level for initiating the resource change was 11.56 -18.49 ms. We use an asynchronous implementation of MPI_Session_dyn_integrate_res_change, which allows the application to resume its work (i.e., perform further iterations) until the dynamically added processes have been initialized. Based on this, the overhead of resource changes on the application level, including the creation of the new MPI communicator, is 109.39 -181.1 ms. Both overheads slightly increase with increasing 𝑛 𝑑𝑒𝑙𝑡𝑎 .

For resource subtraction no new processes need to be initialized, which naturally leads to a lower total time required for the resource change (88.27 -125.86 ms). However, also the overheads are slightly smaller with 7.29 -14.85 ms on the RTE level and 71.4 -112.78 ms on the application level. Again, increasing 𝑛 𝑑𝑒𝑙𝑡𝑎 leads to slightly larger overheads.

In Fig. 3 we show the implications of resource subtraction on the performance of the synthetic loop-based benchmark. This synthetic benchmark represents a near optimal application, which has almost no load-balancing overhead and shows a straight-forward relationship between the number of processes and the application performance. This simplifies the assessment of the performance impact of our approach. As expected, the time required for the workload (black bars) increases after resources are removed in iterations 11, 22 and 33 respectively. In these iterations, overheads of around 100 ms related to processing the resource change are incurred (blue bars). These overheads are moderate relative to the overall time required for one iteration and the interval in which resource changes occur.

PDE solver: Fig. 4 shows the performance results of the resource-adaptive PDE solver run. Similar to the synthetic benchmark the application overhead for resource changes including the creation of the new communicator (blue bars) is between 100 and 125 ms. Additional overheads arise from the adaption of the grid with p4est (red and grey bars).

Table 1: Performance results of the synthetic benchmark for resource addition and subtraction. We denote as Total time of resource change the time measured on the RTE level from receiving the notification of a resource change, until the application has adapted to the resource change. The initiation of the resource change is the time required by the RTE to process the resource change and prepare the data for the interaction with the application. The application overhead is the time the application spends with processing the resource change using the proposed MPI interface and creating a corresponding new communicator. For resource addition we also provide the maximum time the dynamic processes required for the MPI initialization (MPI Session init). However, due to additional resources, the time required for one phase after the resource changes decreases significantly by 𝐺𝑟𝑐1 = 50.37%, 𝐺𝑟𝑐2 = 25.33% and 𝐺𝑟𝑐3 = 10.68% respectively. Thus, the total overhead of the resource change, can be amortized due to the improved performance with additional resources. In our benchmark, the total overhead of the resource change including repartitioning with p4est, would be amortized after 𝑛𝑟𝑐1 > 0.37, 𝑛𝑟𝑐2 > 1.13 and 𝑛𝑟𝑐3 > 3.11 subsequent phases with additional resources, pointing towards the practicability of our approach.

Discussion of Proposed Interfaces

The interface we provide in this work is the result of one of the first attempts to use the set and graph-based approach and is intended to help drive and support the discussions in the MPI Forum. As part of this discussion, in the following we focus on how our proposal is in alignment with our stated design goals, which we describe in Sec. 2.1 and infer possible changes and extensions.

Data redistribution: Changing resources of an application might require corresponding redistribution/reorganization of application data. Our approach does not provide implicit data redistribution, which follows the goal of separation-of-concerns. Instead it gives application programmers full control over the data reorganization in response to EuroMPI/USA'22, September 26-28, 2022, Chattanooga, TN, USA changing resources to maintain generality. However, our approach provides applications with the necessary information about resource changes and mechanisms to create appropriate communicators, which can be used for standard MPI communication to redistribute any application data.

Resource Optimization: Again following the design goal of separation-of-concerns, our proposed interface is independent from the resource optimization process. In particular, our approach does not make any assumptions about the entity initiating the resource changes. Instead, it defines a general mechanism for MPI applications and RTEs to adapt to resource changes. Such a resource change could be triggered by any SMS component, e.g., by the scheduler to improve overall system utilization or by an evolving application to adapt to dynamic workloads.

Moreover, to achieve the goal of high performance, a sophisticated, potentially cooperative scheduling mechanism needs to be added on top of our graph-based approach. As our approach uses PMIx, resource changes and PSets could be made transparent to entities in the SMS (such as schedulers and tools) if they attach to the corresponding PMIx server. Exploring this potential could be valuable future work.

Application-driven set operations: In our current design, the application drives the set operations related to resource changes based on the information received via the MPI_Session_dyn_recv_res_change, such as the name of the delta process set. This gives applications the flexibility to control the resource usage in response to a resource change. For performance reasons, it might be preferable to perform set operations representing resource changes on a higher level of the SMS, e.g., the system scheduler, which would require minor changes of MPI_Session_dyn_recv_res_change.

Granularity of interfaces & fault tolerance: The current state of the suggested interface requires only three additional MPI procedures (and three asynchronous variants), which eases the complexity for application programmers to adopt the approach. However, we plan to achieve further generality by dividing the interface into smaller, but more general functions. To give one example, in our current design, MPI_Session_dyn_integrate_res_change performs the transmission of a PSet name as well as synchronization. More flexibility could be achieved by implementing these functionalities in separate procedures. Such a more fine-granular interface could also facilitate support for fault tolerance.

Root process: So far, our approach uses a root process, which performs the query for resource change information and the potentially required set operations. A consensus-based approach involving all processes of one PSets could be an alternative in particular due to fault tolerant aspects. We do not see both approaches excluding each other, but rather complementary to further improve generality.

Summary and Future Work

Dynamic resource management is an active research area which can improve the efficiency of HPC systems, as well as HPC applications. In this context, we proposed and evaluated a new approach towards dynamic resource management of MPI jobs based on MPI Sessions and PMIx.

In our design, resources are represented by (potentially overlapping) sets, which can be modified using set operations. This allows for a graph-based perspective on resource changes where sets of resources are represented by vertices and set operations by directed hyperedges. Our expectations are that this will cover various requirements of different HPC applications.

We showed how our concept can be implemented on top of the standardized PMIx interface, and what extensions are necessary to support dynamicity. The latter are minimal, as PMIx already provides many of the needed features to support dynamicity and, hence, forms an excellent basis for our work. It is used by most MPI implementations as well as many software components for resource management on HPC systems, and it provides portable access to common system services. Thus, our approach can easily be integrated into such PMIx-based software management stacks.

We developed a prototype of our approach extending Open MPI, OpenPMIx and PRRTE and demonstrated its functionality to dynamically add/remove resources to/from MPI jobs on a modern HPC system on up to 4 nodes and a total of 112 cores. The application overheads were modest, with 109-181 ms for resource addition and 71-113 ms for resource subtraction.

Overall, this work is a proof of concept, providing a first demonstration of a novel set-and graph-based approach towards resource management based on MPI Sessions and PMIx.

In future work, we plan to continue working on the proposed interfaces to further improve on its generality and performance. To this end, we will also need to extend our performance evaluations in terms of scale, resource types and application use cases. Moreover, integration of our approach for resource dynamic MPI Sessions with HPC resource management systems is a natural next step to assess its applicability on modern HPC systems. Here, we will particularly focus on usage and extensions of PMIx to facilitate cooperation between HPC applications and other system components for improved resource optimization.

  2. • PMIx_Pset_Op_request: This function allows to request a PSet operation from the host environment. The operation can be specified as one of the three new PMIx constants PMIX_PSETOP_UNION/DIFF/INTERSECT. The pset1/pset2 arguments of the set operation are provided in the info parameter using the new PMIX_PSETOP_P1/P2 keys. The name of the resulting PSet is returned in the result parameter with the new PMIX_PSETOP_PRESULT key. • pmix_server_pset_operation_fn_t: This function definition allows to pass on the request from the PMIx server to the host environment. The host environment provides a pointer to a valid implementation of this function definition during the initialization of the PMIx server. The implementation is expected a) to perform the specified PSet operation; b) to make the information about the resulting PSet available in the given scope; and c) to call back into the local PMIx server via a callback function. • pmix_psetop_cbfunc_t: This callback function specification allows the host environment to pass the operation results to the PMIx server.

Figure 3 :

 3 Figure 3: Performance implications of dynamic resource subtraction in the synthetic benchmark. The bars indicate the timings for each of the 70 iterations. The colored areas in the background are labeled with the current number of processes 𝑛. The black bars denote the time required for the synthetic workload in each iteration. The hatched areas mark the iterations where the resource change is processed by the application. The blue bars indicate the application overhead related to processing resource changes.

Figure 4 :

 4 Figure 4: Performance implications of dynamic resource addition in a PDE solver for a hyperbolic problem. The total simulation time frame is split into ten phases, in which a certain time interval is simulated with fixed resources (black bars). The colored areas in the background are labeled with the current number of processes 𝑛. The blue bars indicate the application overhead related to processing resource changes. As we use a non-blocking implementation of the dynamic MPI interface, overheads related to resource changes are distributed over multiple iterations. The red and gray bars denote the overhead related to the grid adaptions with p4est.

  /3555819.3555856 EuroMPI/USA'22, September 26-28, 2022, Chattanooga, TN, USA

								Dynamic Resources
	Resource Manager	Node 1	Node 2	Node N
	PMIx Server	MPI Processes	MPI Processes
	Scheduler PMIx	PMIx Client	PMIx Client	PMIx Client	PMIx Client	PMIx Client	PMIx Client	PMIx Client	PMIx Client	PMIx Client
	Server
	Login Node		PMIx			PMIx			PMIx
	Tool PMIx	MPI Runtime Environment Server Server	Server
				Interconnect

  Towards Dynamic Resource Management with MPI Sessions and PMIx EuroMPI/USA'22, September 26-28, 2022, Chattanooga, TN, USA MP I_S essi on_ i n i t (& session ) // Create a session handle M P I _ S e s s i o n _ d y n _ r e c v _ r e s _ c h a n g e ( // Check if process was added d y n a m i c a l l y ( in c l u d e d in delta pset ) session , // IN: Session handle ' ' mpi : // self '', // IN: The name of the associated PSet, for which resource change info is queried IN: Name of the delta PSet of the resource change false , // IN: Flag indicating if this process provides the name of the new PSet main_pset , // IN/OUT: Name of the new PSet related to the resource change & terminate // OUT: Flag indicating that this process needs to terminate

	& rc_type ,	// OUT: The type of the resource change
	delta_pset ,	// OUT: Name of delta PSet, describing the changing resources
	& incl	// OUT: Flag indicating if this process is included in the delta PSet
	)
	if incl : // D y n a m i c a l l y added r e s o u r c e ? -> Receive name of new PSet for c o m m u n i c a t i o n with o r i g i n al procs
	M P I _ S e s s i o n _ d y n _ i n t e g r a t e _ r e s _ c h a n g e (
	NULL ,	// IN: mpi info (optional)
	delta_pset , // )
	M P I _ G r o u p _ f r o m _ s e s s i o n _ p s e t ( session , main_pset , & main_group )	// PSet to group
	M P I _ C o m m _ c r e a t e _ f r o m _ g r o u p ( main_group , " tag " , NULL , NULL , & main_comm )	// group to c o m m u n i c a t o r
	while work_to_do : // MAIN LOOP
	if my_rank == root_rank :
	M P I _ S e s s i o n _ d y n _ r e c v _ r e s _ c h a n g e ( session , main_pset , & rc_type , delta_pset , & incl )
	r e s o u r ce s _ c h a n g i n g = rc_type != MPI_RC_NONE
	if re s ou r c e s _ c h a n g i n g : // For each r e s o u r c e change , root process e x e c u t e s the set o p e r a t i o n
	if rc_type == MPI_RC_ADD :
	M P I _ S e s s i o n _ p s e t _ c r e a t e _ o p (
	MPI_PSETOP_UNION , // IN: Requested set operation
	main_pset ,	// IN: First argument to the set operation (PSet name)
	delta_pset ,	// IN: Second argument to the set operation (PSet name)
	new_ma in_ pse t	// OUT: Result of the set operation (PSet name)
	)
	else if rc_type == MPI_RC_SUB :
	M P I _ S e s s i o n _ p s e t _ c r e a t e _ o p ( session , MPI_PSETOP_DIFF , main_pset , delta_pset , n ew_ main_ps et )
	// Provide i n f o r m a t i o n whether there is a r e s o u r c e change
	MPI_Bcast (& resources_changing , ... , root_rank , main_comm )
	if re s ou rc e s _ c h a n g i n g : // r e s o u r c e change o p e r a t i o n s e x e c u t e d by all e x i s t i n g p r o c e s s e s
	MPI_Bcast ( delta_pset , ... , root_rank , main_comm ) // Provide the name of the delta PSet
	M P I _ S e s s i o n _ d y n _ i n t e g r a t e _ r e s _ c h a n g e ( session , NULL , delta_pset , new_main_pset , my_rank == root_rank , &
	terminate )
	M P I _ C o m m _ d i s c o n n e c t (& main_comm ) // Destroy old c o m m u n i c a t o r
	if terminate :
	break // This process is not part of the new pset = > t e r m i n a t e
	main_pset = ne w_main_pse t	// new main pset
	M P I _ G r o u p _ f r o m _ s e s s i o n _ p s e t ( session , main_pset , & main_group )

... // A p p l i c a t i o n d e p e n d e n t work , p o t e n t i a l l y i n c l u d i n g data m i g r a t i o n to new r e s o u r c e s // Rest of while loop copes with r e s o u r c e change r e q u e s t s

  EuroMPI/USA'22, September 26-28, 2022, Chattanooga, TN, USA

	pmix status PMIx_Pset_Op_request(
	psetop_directive,	// IN pmix psetop directive t specifying the requested PSet operation
	info,	// IN Array of pmix info t structures (array of handles)
	ninfo,	// IN Number of elements in the info array (integer)
	results,	// OUT Array of pmix info t structures (array of handles)
	nresults	// OUT Number of elements in the results array (integer)
	)
	typedef pmix_status (*pmix_server_pset_operation_fn_t)(
	proc,	// IN pmix proc t structure identifying the process requesting the PSet operation (handle)
	psetop_directive,	// IN pmix psetop directive t specifying the requested PSet operation
	info,	// IN Array of pmix info t structures (array of handles)
	ninfo,	// IN Number of elements in the info array (integer)
	cbfunc,	// IN Callback function pmix op cbfunc t (function reference)
	cbdata	// IN Data to be passed to the callback function (memory reference)
	)
	typedef void pmix_psetop_cbfunc_t(
	status,	// IN pmix status t describing the status of the request
	psetop_directive,	// IN pmix psetop directive t specifying the requested PSet operation
	info,	// IN Array of pmix info t returned by the operation(pointer)
	ninfo,	// IN Number of elements in the info array (size t)
	cbdata,	// IN Callback data passed to original API call (memory reference)
	release_fn,	// IN Function to be called when done with the info data (function pointer)
	release_cbdata	// IN Callback data to be passed to release fn (memory reference)
	)
	Attributes required to be included in the passed info array:
	-PMIX PSETOP PRESULT	"pmix.psetop.p1" (char*)	// Name of the resulting PSet
	-PMIX PSET MEMBERS	"pmix.pset.mems" (pmix_data_array_t*)	// The members of the resulting PSet

EuroMPI/USA'22, September 26-28, 2022, Chattanooga, TN, USA

Acknowledgments

We like to gratefully acknowledge in particular Dan Holmes for his valuable feedback and discussions as part of the meetings of the MPI Session workgroup. We like to thank the anonymous reviewers and in particular Jeffrey M. Squyres for the valuable feedback. This project has received funding from the Federal Ministry of Education and Research and the European HPC Joint Undertaking (JU) under grant agreement No 955701, Time-X and No 955606, DEEP-SEA. The JU receives support from the European Unions Horizon 2020 research and innovation programme and Belgium, France, Germany, Switzerland. Towards Dynamic Resource Management with MPI Sessions and PMIx EuroMPI/USA'22, September 26-28, 2022, Chattanooga, TN, USA


Jan Fecht

Martin Schreiber

Martin Schulz

Howard Pritchard

Daniel J Holmes

An Emulation Layer for Dynamic Resources with MPI Sessions

Keywords: MPI, MPI Sessions, Dynamic resources, Resource management

The current static job scheduling on supercomputers for MPIbased applications is well known to be a limiting factor for the exploitation of a system's top performance in terms of application throughput. Hence, allowing fully flexible and dynamically varying job sizes would provide multiple advantages compared to the current approach, e.g., by prioritizing jobs dynamically and optimizing resource usage by transferring resources economically. A critical step in achieving dynamic resource management with MPI on supercomputers is the development of sound and robust interfaces between MPI applications and the runtime system. Our approach extends the concept of MPI Sessions, a new concept introduced with MPI 4.0, by adding new features to support varying computing resources via the MPI process set abstraction. We then show how these features can be used, as a proof of concept, to request (active) and cope with (passive) varying resources from an application's perspective. To validate of our approach, we develop libmpidynres, a C library providing an emulated MPI Sessions environment on top of existing MPI implementations without MPI Sessions support, which we then use to integrate our proposed extensions to the interface specification. Using this proof-of-concept environment, we show how an MPI Sessions enabled application can use process sets to handle dynamically varying resources.

Introduction

Motivation

Job scheduling systems for MPI-based applications allocate a fixed amount of resources (cores, nodes, GPUs, FPGAs, . . . ) for the job's runtime. This is a strong constraint on resource usage, leading to various inefficiencies, e.g., idling cores, lack of taking runtime-changing resource requirements into account, to name just a few.

Preprint of Paper accepted at HPCMALL 2022

To solve this issue, dynamic resources, meaning that the number of available resources can change during an application's execution, need to be introduced to and supported by applications as well as the runtime. For example, with dynamic resources, the job scheduler can withdraw resources from an application and transfer them to another application. This could potentially lead to a higher throughput of application on the entire system, hence an overall better parallel efficiency. Furthermore, this approach allows the job scheduler to prioritize certain jobs dynamically by having more flexibility and scheduling abilities than in the static resource allocation case.

To use dynamic resources in high-performance scenarios, the following components need to be carefully designed and realized:

1. API: a flexible, robust interface for dynamic resources that can be used by MPI applications. 2. Runtime: dynamic resource support in the runtime (MPI library, job scheduler, . . . ). 3. Applications: MPI-based software using this interface to handle dynamic resource changes including all required changes in the software. Having all three components working together in the right way is a very long lasting process and this work is on the first component.

Our contributions are a proposal of an API for robust and flexible dynamic resource changes based on the new MPI Sessions concept introduced with MPI 4 [START_REF]The MPI Forum: MPI: A Message-Passing Interface Standard Version 4[END_REF]. In addition to this, we evaluate our proposal based on a library called libmpidynres which emulates a dynamic resource environment on top of an existing MPI communicator.

Related Work MPI 2's Dynamic Process Model:

The MPI Forum already addressed the need for a more dynamic process management approach with the introduction of the MPI dynamic process model in the MPI 2 standard [START_REF]MPI: A Message-Passing Interface Standard Ver[END_REF]. However, the number of running MPI processes is still limited to MPI UNIVERSE SIZE, which is typically equal to the number of resources reserved by the job scheduler.

Task-based parallelization models: Another set of applications are taskbased parallel programs. E.g. DucTeip is a framework for creating task-based MPI programs [START_REF] Zafari | Advances in task-based parallel programming for distributed memory architectures[END_REF]. That could be exploited rather in a straight-forward manner by, e.g., executing different applications using the same MPI context in parallel. In such a scenario, having a separate MPI context with dynamic resources could be used not only to avoid idling MPI processes, but also to avoid applications influencing each other, e.g., due to a bug.

Invasive Computing (IC): The IC paradigm suggests varying resource utilization for embedded systems [START_REF] Teich | Invasive Computing: An Overview[END_REF] with certain progress to adopt this also in HPC. As a first step, the OpenMP and Threading Building Blocks parallelization models have been modified to allow for varying resources, see e.g. [START_REF] Schreiber | Invasive Comp. Balancing for Appl. with Shared and Hybrid Par[END_REF].

Here, the underlying idea has been to allow a resource manager to improve the system-wide efficiency for concurrently running applications and to start applications at arbitrary points in time where the present work also borrows this idea of a resource manager. Based on the aforementioned work, an extension for distributed-memory systems with MPI was developed [START_REF] Comprés | Infrastructure and API Ext. for Elastic Execution of MPI Applications[END_REF]. However, this led to several drawbacks of this approach, such as that resource changes are solely based on MPI COMM WORLD (which is obviously a serious problem for, e.g., coupled simulations) and that only specialized cases have been taken into account.

MPI fault tolerance: Work on MPI fault tolerance approaches, including MPI global restart [START_REF] Laguna | Evaluating and Extending User-Level Fault Tolerance in MPI Applications[END_REF] and the Fenix project [START_REF] Gamell | Exploring Automatic, Online Failure Recovery for Scient[END_REF], share some of the features of libmpidynres. Both provide mechanisms for an application to recover from an initial loss of compute resources and utilize replacing resources when available, but their functionality is limited in scope to resilience. Our proposed approach, on the other hand, covers a much wider field, but can be used to implement the recovery models supported by these approaches to fault tolerance.

Malleability in MPI: Dynamic resource management in MPI has been studied intensively over the last years, usually under the umbrella term of "malleability". Since then, multiple frameworks have been created to support malleability in MPI applications [START_REF] El Maghraoui | Dynamic Malleability in Iterative MPI Applications[END_REF][START_REF] Sudarsan | ReSHAPE: A Framework for Dynamic Resizing and Scheduling of Homogeneous Applications in a Parallel Environment[END_REF][START_REF] Martín | FLEX-MPI: An MPI Extension for Supporting Dynamic Load Balancing on Heterogeneous Non-dedicated Systems[END_REF][START_REF] Gupta | Towards realizing the potential of malleable jobs[END_REF][START_REF] Iserte | DMRlib: Easy-Coding and Eff[END_REF]. These frameworks used different techniques and APIs to achieve malleability. For example, some authors propose process splitting and merging for expansion and shrinking of the application [START_REF] El Maghraoui | Dynamic Malleability in Iterative MPI Applications[END_REF][START_REF] Iserte | DMRlib: Easy-Coding and Eff[END_REF]. Other authors start new processes while keeping the old, existing processes running [START_REF] Sudarsan | ReSHAPE: A Framework for Dynamic Resizing and Scheduling of Homogeneous Applications in a Parallel Environment[END_REF]. Another approach is to use checkpointing systems and restart the actual MPI application for resizing it [START_REF] Lemarinier | Architecting Malleable MPI Applications for Priority-Driven Adaptive Scheduling[END_REF][START_REF] Prabhakaran | A Batch System with Efficient Adaptive Scheduling for Malleable and Evolving Applications[END_REF].

Although there has been much research around malleability, there is still a lack of a highly flexible, efficient and future proof API. The work presented in this paper takes the attempt to propose such an API and further differs from previous approaches by its use of MPI Sessions for malleability.

MPI Sessions

We start with a brief introduction to MPI Sessions since this is at the core of our proposal. The concept of MPI Sessions was first introduced in 2016 [START_REF] Holmes | MPI Sessions: Leveraging runtime infrastructure to increase scalability of appl. at exascale[END_REF] and was later included into the MPI standard with the release of MPI 4.0 in June 2021 [START_REF] Holmes | MPI Sessions: Leveraging runtime infrastructure to increase scalability of appl. at exascale[END_REF][START_REF]The MPI Forum: MPI: A Message-Passing Interface Standard Version 4[END_REF]. It defines a new object, the MPI Session, which is a lightweight handle to the MPI runtime. Using a session, MPI can be initialized without the MPI COMM WORLD communicator. Further, an application and its libraries can allocate multiple independent sessions allowing for better isolation and a higher degree of composability compared to traditional global MPI intitialization. MPI Sessions also offer a tighter runtime integration by allowing the runtime to expose available resources via process sets. A process set groups together multiple potential MPI processes and is identified by a name in a URI-like format (e.g., "mpi://world", "mpi://self"). Using process sets, an application can create local MPI groups, which can be further used to create communicators that connect MPI processes contained in the respective process sets.

Process sets allow the runtime to expose available resources to the application. Fig. 1 shows an example view of an application's process sets. A process set can represent something of a static nature (e.g., a NUMA node, "mpi://numa/0" in Fig. 1), but it can also represent more dynamic groups of resources (e.g., a specific task in an application: "app://atmos/task/1" in Fig. 1). There is still ongoing discussion about the exact nature of process sets, their lifetime, scope and dynamic behavior. Our approach takes a look at process sets from a more dynamic perspective, which leads to the following assumptions for the remainder of the paper:

Immutability: A process set identified by a unique name will always represent the same resources, even if resources are not actually available to an MPI application, e.g., because the job scheduler removed the resource during a resource change. This property avoids race conditions in cases where an MPI group is created from the process sets by different MPI processes.

Change of process sets: We expect the available process sets to change frequently during an application run. At the same time, the number of available process sets at any point in time is expected to remain small as process sets that become invalid (due to an MPI process exiting a process set) are removed in our model (compare with Sec. 4.2).

Dynamic Resources with MPI Sessions

MPI Sessions Advantages Compared to MPI COMM WORLD

Dynamic resources are non-trivial to implement in the traditional global MPI architecture with MPI COMM WORLD. This is because MPI COMM WORLD needs to be mutated or invalidated when resources are added or removed. As a consequence, communicators that originate from MPI COMM WORLD would need to adapt or get invalidated together with associated rank and size information, which is hard to do in a consistent fashion that is transparent to an application.

MPI Sessions provide one way to tackle the aforementioned problem by allowing MPI COMM WORLD to be avoided entirely. Besides various other benefits, we briefly discuss the main advantages in the context of varying resources.

In our work, process sets are used to globally express resource changes. Once an MPI object is invalidated, new MPI objects from process sets can be created without the need for complex application coordination. Also, there is no need to change the mechanism and semantics of MPI groups and communicators like we would may need to do in the mutable MPI COMM WORLD case. Another advantage of MPI Sessions is given by the current interfaces that already permit the dynamic modification of an applications point of view on available MPI processes by changing the process sets that are exposed to the application.

Resource Changes with Process Sets

Next, we discuss our strategy to realize dynamic resource changes. We would like to point out, though, that our approach focuses on loop-based applications similar to the application shown in Sec. 6.

Resource changes happen when the runtime removes or adds new resources from/to an application. For our dynamic resource model we assume that the runtime does not implicitly add new resources in the form of a new process set, but an explicit function call needs to be made by the application. From the application point of view, implicit adding of resources is problematic due to assumptions on a particular number of resources in a communicator, e.g. the number of ranks. This explicit approach is also useful as the application might need to do load balancing/process coordination work after each resource change. Once a resource change arrives, the application has a time window to do cleanup/load balancing and then accepts the resource change. This is especially important when MPI processes are being removed because the data from these MPI processes needs to be transmitted to avoid data loss.

A resource change consists of a resource change type and a resource change process set:

The resource change type indicates how the application's resources are modified. In the present work, we only investigate two resource change types: addition and removal of processes. To migrate resources, both operations have to be applied sequentially. Obviously, a replace resource change type could be also implemented that both removes and adds processes, as well as a split/join change that (de-)partitions existing process sets. However, these type of changes are not the focus of this work.

The resource change process set, on the other hand, describes the difference between the current set of processes and the set of processes after the resource change. In the case of MPI process addition, the resource change process set will contain all to-be-started MPI processes and in the case of MPI process removal all to-be-removed MPI processes.

Interface Design

MPI Sessions Interface

libmipdynres's MPI Sessions interface was developed along the lines of the Sessions interface in the MPI 4.0 draft from November 2020 [START_REF]The MPI Forum: MPI: A Message-Passing Interface Std[END_REF]. The draft's inter-face description matches the one that was finally published with the official MPI 4.0 standard in June 2021 [START_REF]The MPI Forum: MPI: A Message-Passing Interface Standard Version 4[END_REF].

The MPI 4.0 standard document defines multiple C signatures of MPI Sessions functions and explains the semantics of these functions. However, the document does not fully define all concepts of MPI Sessions. Many questions remain open in regard to process sets. Because of that, we have modified and extended the MPI Sessions interface to fit the way process sets are viewed in this work (see Sec. 2). The MPI Sessions interface that is included in our library, libmpidynres, contains the following functions:

-MPI Session init -initialize an MPI Session -MPI Session finalize -finalize an MPI Session -MPI Session get info -query information about an MPI Session -MPI Session get psets -query for available process sets -MPI Session get pset info -query information about a process set -MPI Group from session pset -create an MPI group from a process set -MPI Comm create from group -create an MPI communicator from an MPI group without a parent communicator

These functions match the functionality and semantics described in MPI 4.0 [START_REF]The MPI Forum: MPI: A Message-Passing Interface Standard Version 4[END_REF], except for MPI Session get psets, which we discuss in the next section.

MPI Session get psets

The signature of MPI Session get psets is shown in Fig. 2. The function replaces two functions in MPI 4.0: MPI Session get num psets and MPI Session get nth pset. These two functions assume a more static behavior of process sets, as they use a virtual array model for querying process sets. With the MPI Session get num psets function one can query the length of the virtual array and with the MPI Session get nth pset one can query a process set at a specific index. The runtime can only append new process sets to the array, an index can become invalid if the process set does not exist anymore. This approach has multiple disadvantages with our assumed process set properties (see Sec. 2):

1. The frequent change of process sets leads to an ever-growing array that will lead to increasing memory usage and increasing access times. 2. The frequent change will also lead to most indices being invalidated at some point in time. This in return increases the chance of invalid requests and creates an additional management overhead on the application side.

To adapt the API to our model, we replace the two function calls with one. Instead of querying each process set on its own, the application queries the names and sizes of available process sets at once. The result is returned in the psets argument of MPI Session get psets. It consists of an MPI Info object with process set names as the keys and the respective process set size as the value, basically representing a snapshot of the current process set state. While this leads to more data being transferred, we expect the number of active process sets at any point in time to remain low. However, to make the API more futureproof and allow for more complex process set situations, an MPI Info object can be passed to the function. This object could be used to filter the results and only return a subset of available process sets.

Process Set Management Interface

When dealing with resource changes, an application must be able to establish communication with new resources. In our work, new resources are expressed via process sets. To establish communication, an application can create an MPI Group from the new process set and use MPI group operations to create a group that both contains the new processes and old application processes. However, this approach has to be made by each process in the new group. This can become quite complex with increasingly more resource changes and is hard to coordinate. This is especially a problem for the newly created processes as they need to know which process sets they need to use to create MPI groups containing old application processes. To avoid these problems, we allow the application to create new process sets.

In our design, only one MPI process, which we will refer to as "main process", is responsible for operations on process sets. Note that these operations could be executed by all involved MPI processes, but leave this to future work and here strictly follow the "main process" approach.

To create new process sets, the application must call the MPIDYNRES pset create op function, whose signature is shown in Fig. 3. The names of two existing process sets need to be given in the pset1 and pset2 arguments. Additionally, a set operation to be applied needs to be passed in the op argument. Calling the function has the effect that, if the arguments are valid, the runtime will create a new process set containing the result of the set operation on the process sets. Currently, three set operations are supported, see also Fig. 4: -Union: The result contains all processes from both psets. This can be used to add new processes from a resource change set to the application's main process set. -Difference: The result contains all processes from pset1 that are not in pset2. This is useful to remove a resource change set (if the resource change takes away resources) from the application's main process set.

-Intersection: The result contains all processes that are both in pset1 and pset2. Allowing these fundamental set operations has multiple advantages which we like to summarize as follows:

-Process set changes do not rely on collective operations involving, e.g., processes which have not yet been started. Therefore, process sets including

processes not yet available to the application (see Fig. 8) can be created. -Since resource changes can be abstractly described as a directed acyclic graph as transitions on resource sets, designing an interface supporting such resource changes should also cover the typical requirements of such resource changes without taking application-specifics into account. -A "main process" driven change of resources makes the process coordination easier since all management can happen in a single process. The only information that needs to be shared with other processes are process set names which are available globally. An extension to a consensus-based management should still be possible. -Process set operations fit nicely into applications that rely on a single communicator during their runtime. There, the operations can be used to derive a new "main process set" from the previous "main process set" and the resource change process set (see Fig. 5).

Our main goal is the creation of rather generic interfaces to cover various requirements on resource change patterns. As usual, there are always optimizations possible by providing specialized interfaces or more feature-rich interfaces, however this is left for future work.

Resource Change Management Interface

To implement the mechanisms described in Sec. 3.2, the resource change API has to provide a way to a) query for pending resource changes and b) accept and apply these resource changes.

For a), libmpidynres offers the MPIDYNRES RC get function. Its signature is shown in Fig. 6. If there is a resource change, the type of resource change and the resource change process set are returned in the rc type and delta pset respectively. Furthermore, a handle to the resource change is returned in the tag argument. For b), libmpidynres offers the MPIDYNRES RC accept function. Using this function, the application can tell the runtime to apply the resource change referenced by the tag argument. The info argument can be used to pass information new processes. Once this function is called with valid arguments, the runtime will start new processes in the case of resource addition. If the resource change removes resources, the application has to shutdown the relevant processes itself. If possible, the runtime can try to enforce the shutdown by forcefully shutting down running processes after a specific amount of time. In the case of libmpidynres, due to its architecture, the shutdown cannot be enforced.

An example application execution with both resource changes and process set operations in shown in Fig. 5. The application shown constructs a new "main pset" after each resource change. Note that the "main process" main thread is not highlighted, as it is application dependent to choose a main rank. One possible way to choose a "main process" is to use rank 0 of the communicator based on the "main pset".

libmpidynres

In order to evaluate our proposal, we implement the runtime component in the form of a C library, called libmpidynres, that emulates a dynamic resource environment on top of an existing MPI communicator. The different components of an application using libmpidynres. Both the application and libmpidynres access the same MPI library. However, the application should derive most of its MPI objects from libmpidynres.

The library uses a communicator of fixed size to emulate an MPI Sessions environment with dynamic resources by using subsets of the fixed-size communicator. This is achieved by hiding and exposing the processes of the communicator to the application. Using this emulated environment, applications can use the MPI Sessions and dynamic resource management API described in Sec. 4, hence already explore and test these features even if the underlying MPI implementation and job scheduler do not support MPI Sessions and dynamic resource management. For sake of reproducibility and open science, the source code of libmpidynres is available on GitHub. 5 .

libmpidynres as an Emulation Layer on top of MPI

libmpidynres is implemented as a C library that is used on top of an existing MPI library. This means that libmpidynres uses MPI calls internally for communication and management. From the application's point of view, it extends the available MPI API with additional functions.

Before the MPI Sessions environment becomes active, the application has to configure libmpidynres and initialize MPI. This part of the user application is called the application wrapper. The application wrapper then passes an entry point and a communicator for the emulated application to libmpidynres.

From there, libmpidynres manages the communicator's processes and runs the emulated application from the given entrypoint. The emulated application should only use MPI communicators and groups that are returned by libmpidynres or were derived from these. This ensures that libmpidynres has full control over the available processes. This architecture is illustrated in Fig. 7.

Emulated Process States

libmpidynres emulates dynamic resources on a communicator of fixed size (typically MPI COMM WORLD). This is achieved by selectively exposing a subset of the communicator's processes to the application as its world process set. Consequently, the maximum number of processes that can be scheduled is limited by the size of the communicator used for emulation. Inactive processes are idling (in an MPI Recv operation) and are waiting to be requested and then made available to the MPI application. However, unlike a full implementation inside a runtime, such processes cannot be made available to other, separate applications. Therefore, it is again important to stress that libmpidynres is only a proof-of-concept library for testing the interface and real support for dynamic resources has to be included in the various software components of the MPI stack.

The process of starting and stopping resources is quite complex and involves multiple temporary states a process can be in. These states are illustrated in Fig. 8. Note that these are the states from the library's point of view.

Resource Manager

libmpidynres uses a server-client model for managing process sets and resource change states. For that, the MPI process with rank zero of the libmpidynres MPI communicator acts as a dedicated resource manager.

This server-client approach avoids race conditions and assures a consistent state across all MPI ranks. However, the additional communication overhead leads to decreased performance especially with redundant requests from multiple ranks and to increased latency when doing API calls. However, the proof-ofconcept, emulating nature of libmpidynres justifies this trade-off.

Case Study

To evaluate the proposed interface and libmpidynres, we implement an application example that is based on a loop where work is distributed among all processes in each iteration. In the following, we describe this example application.

Let us first look at the initialization part of the application given in lines 1-10 in Fig. 9. When an MPI process is started, it needs to set up and gain information about its environment. For that, the application initializes an MPI Session using MPI Session init. Furthermore, the application queries its process sets using MPI Session get psets. If the process is part of the "mpi://world" process set, the process was started together with the start of the application. If it is not part of the process set, the process was started because of a resource change and has to query some information to successfully join the application. In this example, it queries the current loop iteration and the process set that should be used for communication from its MPI session (the information was passed with an Info object when the resource change was accepted).

Once a communicator is created from the main pset, the main loop is started. This is shown in lines 11-25 of Fig. 9. The application queries for resource changes at the beginning of each loop iteration. When dealing with resource changes, the application follows the strategy from Fig. 5. This means that the application tries to have all of its available resources grouped together in one process set, the "main process set". If a resource change adds new resources, the union process set operation is used to create a new "main process set". If a resource change removes existing resources, the difference process set operation is used instead. When the application accepts a resource change using MPIDYNRES RC accept, some information (the main process set name and the current loop iteration) are passed to newly started processes.

Using this system, the application is able to handle and adapt to resource changes while constantly having a valid MPI communicator. A concrete C implementation of this application was tested and evaluated using different scheduling algorithms and different communicator sizes. The application could successfully finish all of its loop iterations without any crashes or race conditions in the application or libmpidynres. More libmpidynres examples can be found on GitHub. 6 .

Conclusion

In this work, we presented an interface that uses new MPI Sessions concepts to handle dynamically varying resources. The interface uses process sets to express resource differences that will be applied to the application. We implemented an emulation layer that allows applications to use the new interface. This makes prototyping of malleable applications with the proposed interface possible, even without MPI providing support for this, yet. Furthermore, using an example application built on top of this emulation layer, we have validated that using this interface, applications are capable of dealing with resource changes.

Regarding future work, one of the next steps is an extension of the prototype with the implementation of different parallel programming patterns (beside the loop pattern) and combine them with the interface proposed in this work. While the current interface is quite general and therefore may be useful for other programming patterns, it still provides some global changes that may affect all processes of the application. For more distributed programming patterns, a less global approach is needed where the application can group its own processes and tell the runtime that only certain groups should be affected by resource changes.

Another interesting area to apply this new interface to are existing tools for dynamic computing. For example, tools like p4est and PETSc can help with automating parts of the load balancing process in dynamic mesh refinement applications [START_REF] Burstedde | p4est: Scalable Alg. for Parallel Adaptive Mesh Ref. on Forests of Octrees[END_REF][START_REF] Balay | PETSc Web page[END_REF]. Integrating dynamic resources into these tools is currently work-in-progress and could abstract the dynamic resources away from the library user and ease the creation of scalable parallel applications.

Besides many other future research aspects, we like to finally point out the problem of scheduling, which will require disruptive algorithms to cope with runtime-varying resources.

Fig. 3 .

 3 Fig.3. Proposed API for creating process sets by applying a set operation on existing process sets.

Fig. 4 .

 4 Fig. 4. Venn diagrams of the process set operations. Process set A contains the processes P1,P2,P3 and P4; Process set B contains the processes P3,P4,P5 and P6. The operation's result is written underneath each Venn diagram.

Fig. 6 .

 6 Fig. 6. Proposed API for managing resource changes.

Fig. 7 .

 7 Fig.7. The different components of an application using libmpidynres. Both the application and libmpidynres access the same MPI library. However, the application should derive most of its MPI objects from libmpidynres.

Fig. 8 .

 8 Fig.8. The different states that an emulated process can be in, from libmpidynres' point of view. States where the application has control over the (OS-)process are highlighted in green.

Fig. 9 .

 9 Fig. 9. Pseudo code showing using the proposed interface to successfully query and adapt to dynamic resources changes.

  i n t M P I S e s s i o n g e t p s e t s ( M P I S e s s i o n s e s s i o n , MPI Info i n f o , MPI Info * p s e t s ) ; IN info Info object containing runtime hints OUT psets Info object containing process set names as keys and process set sizes as decimal values Fig. 2. libmpidynres API for querying available process sets.

	i n t MPIDYNRES pset create op (
		M P I S e s s i o n s e s s i o n , MPI Info h i n t s , const char p s e t 1 [ ] ,
		const char p s e t 2 [ ] , MPIDYNRES pset op op , char p s e t r e s u l t [ ] ) ;
	IN	hints	Hints passed to runtime
	IN	pset1	Name of first process set argument
	IN	pset2	Name of second process set argument
	IN	op	Operation type to apply
	OUT pset result Name or resulting process set

  Diagram showing how to use process set operations for resource changes. IN tag Identifier of the resource change to accept IN info Runtime hints and hints for newly created processes

				inactive	running	running, marked for removal	inactive, marked for start
	P1
	P2
	P3
	P4
	P5
	P6
	P7
	main process set is started: {P2,P3,P4,P5}	resource change: type: removal set: {P2}	set operation: difference {P2,P3,P4,P5}, {P2} -> New main set: {P3,P4,P5}	change accepted: {P2} is removed	resource change: type: addition set: {P2,P6,P7}	set operation: union {P3,P4,P5}, {P2,P6,P7} -> New main set: {P2,P3,P4,P5,P6,P7}	change accepted: {P2,P6,P7} are started	Application is done, processes shut down
		Fig. 5. i n t MPIDYNRES RC get ( M P I S e s s i o n s e s s i o n ,
		MPIDYNRES RC type * r c t y p e , char d e l t a p s e t [ ] ,
		MPIDYNRES RC tag * tag , MPI Info * i n f o ) ;
		OUT rc type	Type of resource change
		OUT delta pset Name of the new resource change process set
		OUT tag		Identifier for the resource change
		OUT info		Optional additional information about the resource change

i n t MPIDYNRES RC accept ( M P I S e s s i o n s e s s i o n , MPIDYNRES RC tag tag , MPI Info i n f o ) ;

https://github.com/boi4/libmpidynres

https://github.com/boi4/libmpidynres/tree/master/examples

Acknowledgments: This project has received funding from the Federal Ministry of Education and Research and the European HPC Joint Undertaking (JU) under grant agreement No 955701, Time-X and No 955606, DEEP-SEA. The JU receives support from the European Union's Horizon 2020 research and innovation programme and Belgium, France, Germany, Switzerland.