I Pelevanyuk

A Tsaregorodtsev

DIRAC Interware as a service for high-thoughput computing in JINR

Keywords: 07, 05, -t;

DIRAC Interware as a Service for High-Thoughput

Computing in JINR I. Pelevanyuk, A. Tsaregorodtsev

Introduction

The Joint Institute for Nuclear Research (JINR) is an international intergovernmental organization. It is developing as a large multidisciplinary international scientific center incorporating basic research in the field of modern nuclear physics, the development, and application of high technologies, as well as university education in the relevant fields of knowledge. Currently, JINR has 18 Member States and 6 countries participating in JINR's activities based on bilateral agreements signed at the governmental level.

There is a wide range of different studies which require massive computations performed with the help of high-throughput computing techniques. Various computing resources may be used to perform required computations in JINR: Tier1 and Tier2 clusters, cloud, NICA cluster. In some cases, it is possible and effective to use Govorun supercomputer not only for highperformance computing jobs but also for high-throughput jobs. All described resources are unique from the point of authentication, authorizations, data access, resource quota, job submission mode, and some other aspects. That does not allow effective use of all available resources for a single big set of tasks. The users have to decide what resource to use and tune their workload for that particular resource. That reduces the efficiency of resource usage.

DIRAC Interware is an open-source development platform for the integration of heterogeneous computing and storage resources. It provides a central job queue and basic tools for data management alongside with accounting system, user management system, workflow management system, and other systems. The decision to use DIRAC was taken based on three main factors. First, DIRAC is a general-purpose framework for building distributed infrastructures and it provides all tools required for distributed data processing and data management. Second, it demonstrates good performance up to at least a hundred thousand concurrently running jobs. Third, around DIRAC gathered an active community of developers and institutes. Every year users' workshop is organized to bring together users and developers. DIRAC was chosen for the integration of heterogeneous computing resources for the high-throughput workload.

DIRAC development in JINR

The work related to DIRAC integration was started in 2016 [START_REF] Korenkov | Accessing Distributed Computing Resources by Scientific Communities using DIRAC Services[END_REF]. The standard incremental process of system enhancement went through three basic steps:

1. DIRAC setup, configuration, development, and tuning. That is the first step. Its completion does not increase usefulness by itself but provides us with new possible features of the system and new perspectives for further development.

2. Integration of computing and storage resources to DIRAC. On that step real resources were integrated into the existing system. Basic tests performed and correctness of basic operations ensured.

3. Elaboration of approaches for effective use of resources. That is the final step in the process of resource integration. After that, it is possible to propose to users access to distributed resources through the DIRAC system.

Historically integration and development of DIRAC in JINR started back in 2016. As a first step, the DIRAC Interware system was installed, the basic configuration is done. It was used for educational purposes only to demonstrate the concept of distributed high-throughput computing for students. dCache became the first storage system integrated, and Tier2 cluster was the first integrated computing element [START_REF] Gergel | Hybrid Distributed Computing Service Based on the DIRAC Interware[END_REF].

DIRAC platform showed good potential to become a general-purpose system that can unite all kinds of heterogeneous resources. So, the idea was born to unite all clouds of JINR member-states. All member-states' clouds and JINR cloud use OpenNebula as a platform for cloud computing. DIRAC Interware had no support for that platform. The new module was developed to allow OpenNebula cloud integration to DIRAC [START_REF] Balashov | Cloud Integration Within the DIRAC Interware // CEUR Workshop Proceedings[END_REF]. Basic tests of the full cycle of Monte-Carlo for BM@N were performed on the integrated resources to prove the usability of the system.

Since 2019 Multi-Purpose Detector (MPD) collaboration uses DIRAC for centralized massive Monte-Carlo production. Tier1 cluster, Govorun supercomputer, and EOS storage system were integrated into DIRAC to support MPD workflow and provide enough computing performance. In 2020 Folding@Home jobs were submitted to clouds via DIRAC. That became a good test for cloud infrastructures. After that Baikal-GigaVolumeDetector(GVD) experiment started to submit jobs to JINR and Plekhanov Russian Economic University(PRUE) clouds using DIRAC.

Right now integration of distributed heterogeneous computing resources proved to be efficient and useful for the high-throughput workload.

DIRAC usage in JINR

Right now there are several use-cases of DIRAC in JINR: MPD, Barionuc Matter at Nuclotron (BM@N), Baikal-GVD, Folding@home, educational purposes. Most of them are related to the Monte-Carlo simulation.

The biggest consumer of computing and storage resources is MPD experiment [START_REF] Kutovskiy | Integration of Distributed Heterogeneous Computing Resources for the MPD Experiment with DIRAC Interware[END_REF]. Right now it generates around 90% of the computing workload. MPD uses all major computing resources: Tier1, Tier2, Govorun, and NICA cluster. Most of the time software for user jobs was compiled on computing resources manually. The maximal amount of simultaneously running jobs is around 2800: Govorun -876, Tier1 -917, Tier2 -997. It was successfully executed 970 thousand jobs. The total amount of consumed CPU is 5.12 MHS06 days, which corresponds to 783 years of walltime on an average CPU core. The total amount of generated data was 278 TB.

The second big consumer of resources is Baikal-GVD. This experiment also used DIRAC for Monte-Carlo simulation. This experiment was using DIRAC primarily for calculations on the clouds. That required placing all software in the CVMFS system to save network bandwidth on remote clouds. In the end, only two cloud resources were used: JINR cloud and PRUE cloud. The maximum amount of simultaneously running jobs is 105: JINR -95, PRUE -10. The total amount of executed jobs is 68 thousand. The consumed CPU is 285 KHS06 days or 47 years of walltime. The total amount of generated data is 21 TB.

The third use case of DIRAC usage is running Folding@Home jobs on free cloud computing resources in JINR and member-states. The original idea was to use the Folding@Home workload for the full-scale test of cloud resources. Jobs were submitted from May till October 2020. The success of that test allowed proposing cloud for other experiments including Baikal-GVD. Only clouds were used for Folding@Home workloads. The total amount of completed jobs is 13 thousand. The maximal amount of concurrently running jobs was around 80. The consumed CPU is KHS06 days or 23 years of walltime. There was no data transfer through DIRAC since Folding@Home uses its client for that.

Forth user of distributed infrastructure via DIRAC is the BM@N experiment. It was the first experiment that tried to use DIRAC for raw data processing in 2018. But at that time existing integration did not allow the processing of big files with sizes ranging from 15 to 100 GB. Up to the current moment raw data processing workload was successfully tested, but only Monte-Carlo jobs were executed for production. BM@N uses the same resources as MPD since both of them are related to Nuclotron based Ion Collider fAcility (NICA) project. The maximal amount of simultaneously running jobs was 1686: Tier2 -831, Tier2 -855. The total amount of executed jobs is 12.5 thousand. The consumed CPU is 124 KHS06 days or 22 years of walltime. The amount of generated data is 7 TB.

The The use of DIRAC for educational purposes does not generate much load, so it is neglected in this review. The summary of all numbers mentioned previously is in the Table 1. The normalized CPU usage ratio is in the Fig. 1. It is visible that the use of any single resource will be significantly slower than the use of all of them together over DIRAC. Even the single use of the most powerful component: the Govorun supercomputer, will be 2.7 times slower.

Conclusion

High-throughput workload appears often in the scope of modern scientific research. A relatively simple approach is to choose one of the available computing resources and tune all workflows to be executed on that particular resource. It will be slower and generate an excess of load on one resource while others may be free. Finding a way to use all available computing resources may solve these issues. DIRAC Interware is a platform for building custom heterogeneous distributed computing infrastructures. It was used in JINR to unite together all major computing resources. That process required to develop expertise in the DIRAC platform, collaborate with administrators of resources and potential users of the united infrastructure. Three years passed since the beginning of DIRAC development in JINR until the first user job was submitted and successfully executed. Today DIRAC system in JINR is a tool that helps users to execute their tasks faster than on any available single computing resource. The use of DIRAC also increases the efficiency of computing resource usage.

Fig. 1 .

 1 Fig. 1. Normalized CPU usage ratio between main computing resources

Table 1 .

 1 most recent example of DIRAC usage by a new experiment is Spin-Physics Detector (SPD). It has just submitted the first block of Monte-Carlo jobs. The number of jobs was 20 thousand. Maximal jobs running in a particular moment are around 1900: Tier1 -909, Tier2 -986. The normalized CPU consumed by SPD is 78 KHS06 days or 18 years of walltime. The amount of generated data is around 43 TB. Overview of all DIRAC users

	Experiment	First usage Completed	Consumed	Data gen-
			jobs	CPU,	erated,
				HS06 days	TB
	MPD	Aug 2019	970000	5.12 M	278
	Baikal-GVD	Oct 2020	68000	285 k	21
	Folding@Home May 2020	13000	137 k	n/a
	BM@N	Jul 2021	12500	124 k	7
	SPD	Nov 2021	20000	78 k	43

E-mail: pelevanyuk@jinr.ru

E-mail: atsareg@gmail.com