José Armando Hernández González
email: jose.hernandezgonzalez@ens-paris-saclay.fr

Miguel Colom
email: mcolomba@ens-paris-saclay.fr

Repeatability, Reproducibility, Replicability, Reusability (4R) in Journals' Policies and Software/Data Management in Scientific Publications: A Survey, Discussion, and Perspectives

Keywords: Repeatability, Reproducibility, Replicability, Reusability, Data Science AI/ML, RaaS, Scientific journal, Trustworthy, Data Citation, Rewarding Research, Reproducible Research

With the recognized crisis of credibility in scientific research, there is a growth of reproducibility studies in computer science, and although existing surveys have reviewed reproducibility from various perspectives, especially very specific technological issues, they do not address the author-publisher relationship in the publication of reproducible computational scientific articles. This aspect requires significant attention because it is the basis for reliable research. We have found a large gap between the reproducibility-oriented practices, journal policies, recommendations, publisher artifact Description/Evaluation guidelines, submission guides, technological reproducibility evolution, and its effective adoption to contribute to tackling the crisis.

We conducted a narrative survey, a comprehensive overview and discussion identifying the mutual efforts required from Authors, Journals, and Technological actors to achieve reproducibility research. The relationship between authors and scientific journals in their mutual efforts to jointly improve the reproducibility of scientific results is analyzed. Eventually, we propose recommendations for the journal policies, as well as a unified and standardized Reproducibility Guide for the submission of scientific articles for authors.

The main objective of this work is to analyze the implementation and experiences of reproducibility policies, techniques and technologies, standards, methodologies, software, and data management tools required for scientific reproducible publications. Also, the benefits and drawbacks of such an adoption, as well as open challenges and promising trends, to propose possible strategies and efforts to mitigate the identified gaps. To this purpose, we analyzed 200 scientific articles, surveyed 16 Computer Science journals, and systematically classified them according to reproducibility strategies, technologies, policies, code citation, and editorial business.

We conclude there is still a reproducibility gap in scientific publications, although at the same time also the opportunity to reduce this gap with the joint effort of authors, publishers, and technological providers.

Introduction

Reproducibility is a broad and complex topic strongly related to the history of science and knowledge [START_REF] Ivie | Reproducibility in Scientific Computing[END_REF] reflected in the cumulative technological and scientific development of humanity [START_REF] Hughes | History of Technology[END_REF]. Such development has been based on the evolutionary capacity of human beings to build new knowledge from previous discoveries and achievements, transmitting this knowledge to new generations in a continuous cycle of improvement. The evolution of Science through the reproducibility of knowledge could be metaphorically compared to the natural mechanisms of DNA replication [START_REF] Hu | Evolution of DNA replication origin specification and gene silencing mechanisms[END_REF] transmitted from generation to generation in a continuous cycle of refinement. Within these reproducible mechanisms, scientific journals play a significant role in the communication, divulgation, corroboration, validation, and acceptance of reliable and trustworthy knowledge.

The reproducibility of knowledge has recently become relevant to the scientific community given that there is a growing concern for ethics and transparency in the research results in scientific publications in the so-called reproducibility crisis [START_REF] Plavén-Sigray | The readability of scientific texts is decreasing over time[END_REF][START_REF] Odd | The Reproducibility Crisis Is Real[END_REF]. In addition, with the boom in artificial intelligence/machine learning (AI/ML), publications have evolved towards data-centric and modelcentric developments that force journals to adapt their publishing business models to new dynamics accelerated by technological changes [START_REF] Hutson | Artificial intelligence faces reproducibility crisis[END_REF].

In response to these developments, several recent articles have hypothesized what the future of academic publishing will be like [START_REF] Dodds | The future of academic publishing: Revolution or evolution revisited[END_REF][START_REF] Baillieul | Reflections on the future of research curation and research reproducibility[END_REF] [9], analyzing important changes, proposing technological tools [START_REF] Kitchenham | Meta-analysis for families of experiments in software engineering: a systematic review and reproducibility and validity assessment[END_REF][START_REF] Anchundia | Resources for reproducibility of experiments in empirical software engineering: Topics derived from a secondary study[END_REF] and identifying significant gaps in publishing policies [START_REF] Stoddart | Is there a reproducibility crisis in science? Nature[END_REF][START_REF] Kapoor | Leakage and the Reproducibility Crisis in ML-based Science[END_REF][START_REF] Lucic | Reproducibility as a Mechanism for Teaching Fairness, Accountability, Confidentiality, and Transparency in Artificial Intelligence[END_REF][START_REF] Ahmed | Measuring Reproduciblity of Machine Learning Methods for Medical Diagnosis[END_REF][START_REF] Moradi | Reproducible Model Sharing for AI Practitioners[END_REF][START_REF] Haibe-Kains | Transparency and reproducibility in artificial intelligence[END_REF][START_REF] Mayer Mckinney | Reply to: Transparency and reproducibility in artificial intelligence[END_REF][START_REF] Kitamura | Reproducible Artificial Intelligence Research Requires Open Communication of Complete Source Code[END_REF][START_REF] Gibney | This AI researcher is trying to ward off a reproducibility crisis[END_REF][START_REF] Ghanta | Interpretability and Reproducability in Production Machine Learning Applications[END_REF][START_REF] Aviyente | Explainability in Graph Data Science: Interpretability, replicability, and reproducibility of community detection[END_REF][START_REF] Lopresti | Reproducibility: Evaluating the Evaluations[END_REF][START_REF] Raff | Research Reproducibility as a Survival Analysis[END_REF]. These articles account for policies implemented by publishers and their evolution, which are crucial for understanding their evaluation processes oriented to the Reproducibility of knowledge.

This paper analyzes the journal policies concerning the reproducibility of knowledge addressed to trustworthiness and transparency through a survey of computer science journals indexed in SCO-PUS& WoS. This survey identifies recent advancements in computational reproducibility and policies, develop strategies that engage with the reproducibility crisis, and analyze future perspectives in computer base scientific publications. In particular, the role of stakeholders [START_REF] Diaba | Reproducibility and research integrity: the role of scientists and institutions[END_REF] in the reproducibility of computational scientific articles, especially ML and AI-related projects, is explored to understand better the relationship between authors and scientific journals driven by advanced cloud-computing technologies. By reviewing the most promising trends, this paper broadens the landscape [START_REF] Baillieul | Reflections on the future of research curation and research reproducibility[END_REF] of possibilities for advances in reproducibility from both technological and methodological points of view, addressing the gap in effective journal policies that guarantee the reproducibility of computational works.

A wide variety of scientific journals deal with different topics or specialize in various disciplines and fields of knowledge. In several cases, these are subjects based completely on theory or physical experiments. Therefore, this article delimits its scope to specifically analyze scientific journals of computer science, especially the management and reproducibility of scientific articles based on software and data, where the objects to be replicated are information (bits), highlighting that many scientific disciplines converge in the computational field to solve their problems and build up their knowledge base.

Figure 1 summarizes the structure of this paper, where three aspects are analyzed: efforts required by authors (Section 5.1) and by publishers (Section 5.2), and the technological evolution required to reproduce results (Section 2).

The combined efforts of these three actors are required to close the reproducibility gap. Our systematic literature is a PRISMA-based review (see Appendix A.1). We discuss the corresponding definitions, difficulties, and measures related to Reproducibility, which allows us to define the technological evolution as necessary to reproduce results and the fundamental strategies of reproducibility in Section 2. Section 2.4 presents a landscape of existing tools, data management platforms, and techniques that are helpful in reproducible research. Section 2.6 introduces the role of publishers in reproducible research, including new types of publications with code, and discusses the problem of evaluating research artifacts. Section 3 surveys 16 computer science journals and provides insights about experiences implementing data-code sharing policies based on the reviewed reproducibility platforms and technologies. The methodology that we followed is described in Appendix A.2. Section 4 includes our technological discussion of the topics presented before. Section 4.2 focuses on and highlights the shared responsibility between authors and publishers supported by technological evolution. Section 5 asses and analyzes the combined efforts of these three actors required to close the reproducibility gap. The efforts required by authors in Section 5.1 and by publishers in section5.2. Important dilemmas that emerge are addressed in sections 4.1 5.3. Under the possibility of regarding reproducibility as a service provided by trusted third party, considering software as valuable research artifacts, and how to reward authors. Section 6 concludes the paper.

Definitions

Several works [START_REF] Sculley | Hidden technical debt in machine learning systems[END_REF][START_REF] Kitzes | The practice of reproducible research: case studies and lessons from the data-intensive sciences[END_REF][START_REF] Baker | Workshop Report on Basic Research Needs for Scientific Machine Learning: Core Technologies for Artificial Intelligence[END_REF][START_REF] Parashar | Heroux, and Victoria Stodden[END_REF][START_REF] Thompson | Reproducible research[END_REF][START_REF] Raghupathi | Reproducibility in Computing Research: An Empirical Study[END_REF] have addressed reproducibility from different points of view, as [START_REF] Odd | The Fundamental Principles of Reproducibility[END_REF] reproducibility is considered a fundamental part of the scientific method. However, to our knowledge, no works have holistically reviewed the different dimensions and strategies of reproducibility in computer science, i.e., to consider their essential participation within an end-to-end data science project/experiment life cycle. This life cycle begins from scientific research and ends in mass industrial production for final customers. The life cycle also incorporates the responsibilities of the main stakeholders [START_REF] Diaba | Reproducibility and research integrity: the role of scientists and institutions[END_REF][START_REF] Sebastian | Reproducibility: A Researcher-Centered Definition[END_REF][START_REF] Macleod | Improving the reproducibility and integrity of research: what can different stakeholders contribute[END_REF] in this process (e.g., journals, authors, industry, and the scientific community).

The report [START_REF]Committee on Reproducibility and Replicability in Science[END_REF] from the National Academies of Sciences, Engineering, and Medicine (NASEM) is a reference reproducibility study that gathers contributions from relevant specialized researchers. It focuses on strategies for obtaining consistent computational results using the same input data, computational steps, methods, code, analysis conditions, and replicability to get consistent results across studies. In NASEM's definitions, reproducibility involves the original data and code, whereas replicability is related to the collection of new data and similar methods used in previous studies.

The simplest definition of reproducibility extended and used in the different works is the one proposed by ACM in version 1.1 of their Artifact Review and Badging report 1 , as shown in Figure 2.

Figure 4 shows the four definitions concerning the team, method, code, metadata, and setup elements.

Reproducibility (different team, same experimental setup): the experiment is done with different equipment, different environment, same code/algorithm. Repeatability (same team, same experimental setup): the experiment is done by the same team, same environment (software/hardware), same code/algorithm. Replicability (different team, different experimental setup): the experiment is done with different equipment, different environment, different code, same algorithm. Reusability (different equipment, different and partial experimental configuration): the experiment is carried out with different equipment, different environments, different codes, and the algorithm partially implemented.

There is still some discussion, in some cases even confusion, about the definitions [START_REF] Plesser | Reproducibility vs. Replicability: A Brief History of a Confused Terminology[END_REF] even from a taxonomic point of view [START_REF] Bakinam | A taxonomy for reproducible and replicable research in environmental modelling[END_REF][START_REF] Michael | Toward a Compatible Reproducibility Taxonomy for Computational and Computing Sciences[END_REF]. A very different interpretetion of reproducibility is presented in [START_REF] Lin | Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments[END_REF] where it is a continuous improvement process rather than an achievable objective. However, following the discussions with the National Information Standards Organization (NISO), ACM accepted the recommendation to harmonize its terminology and definitions with those widely used in the community of scientific research. In this way, it interchanged the terms reproducibility and replicability with the existing definitions proposed by ACM to ensure consistency.

In this article, we specifically discuss the reproducibility of complex ML/AI data science projects. A scientific publication in ML/AI can range from effectively a model developed in an experiment by a single researcher for a tiny device to large implementations of Distributed Big Data supercomputing developed by large consortiums of universities, governmental, or research institutions.

Types of Reproducibility

Defining what reproducibility is as important as determining the types of reproducibility, considering the nuances that conceptually appear when studying the various cases and possibilities. The term reproducibility is acceptable in the case where the same input can lead to statically equivalent same results. It is also important to note that reproducibility in data science does not necessarily imply obtaining the same numerical result from the same numerical input.

Previous works [START_REF] Odd | State of the Art: Reproducibility in Artificial Intelligence[END_REF][START_REF] Raghupathi | Reproducibility in Computing Research: An Empirical Study[END_REF] have defined three degrees of reproducibility: R1 (Experiment, Data, Method), R2 (Data, Method), and R3 (Method). It is only sometimes possible to obtain the same numerical result from different realizations of an experiment. In that case, one can consider the following definitions [START_REF] Impagliazzo | Reproducibility in learning[END_REF] :

• Experimental reproducibility: similar input (data) + similar experimental protocol → similar results

• Statistical reproducibility: same input (data) + same analysis → same conclusions (independently from (random) sampling variability)

• Computational reproducibility: similar input (data) + same code/software + same software environment → exact same bit-wise results

Side difficulties to achieve reproducibility

Several difficulties identified in our review are not related to the actual shared source code used to reproduce an experiment. Rather, they involve external considerations such as dependency on 3rd-party libraries, complex and uncontrolled software dependencies, the quality of the writing in scientific articles, the documentation of the software, sustainability for the long term [START_REF] Akhlaghi | Toward Long-Term and Archivable Reproducibility[END_REF], reproducibility, and the reward for the career advancement of the researchers [START_REF] Baker | Why scientists must share their research code[END_REF].

Complex and uncontrolled software dependencies With the increasing demand for functionalities, the code has become more complex, relying on several 3rd party software dependencies. These include libraries, packages, or complete frameworks. Changes from different releases (or even from version to version) and irregular maintenance or support lead to what has been described as dependency hell. This irregular maintenance represents one significant contemporary challenge for obtaining reproducible results from software made from different components that are sometimes obsolete. Using package managers, virtual environments, and container tools, keep an up-to-date list of dependencies, and perform continuous testing [START_REF] Fan | Escaping dependency hell: finding build dependency errors with the unified dependency graph[END_REF], are recommended to mitigate this problem. With these strategies, the management of dependencies can be simplified, and projects can then be run in a reproducible manner across different configurations and systems.

Low writing quality

The impact of the writing quality in scientific articles and the associated documentation of the software has already been studied [START_REF] Mack | How to write a good scientific paper[END_REF]. In practice, however, these aspects are sometimes overlooked. One can easily identify articles with confusing writing that is unnecessarily overloaded with complex academic jargon. Such writing is very difficult to interpret and, consequently, very hard or even impossible to reproduce.

Compilation and infrastructure setup. In applied computer science, significant time is generally spent compiling source code, debugging it, and configuring the running platform. The time spent configuring is comparable to the effort to solve the scientific problem. If building and running the program is too time-consuming, the software could be de facto considered as nonreproducible.

Float point operations. Given their deterministic nature, computer systems should theoretically reproduce any numerical result per se. However, in practice, point floating operations can give slightly different numerical results in different systems [START_REF] David | Reproducibility and variable precision computing[END_REF][START_REF] Faraji | Improving Numerical Reproducibility of Scientific Software in Parallel Systems[END_REF]. This aspect needs to be taken into account in the reproducibility assessment. Float point operations errors are also analyzed by Jezequel [START_REF] Jézéquel | First steps towards more numerical reproducibility[END_REF] on numerical reproducibility with the IEEE 754 encoding.

Adapted operating systems. Replicated environments require installing the exact version and its respective dependencies to obtain the equivalent result, and doing so is very difficult with no isolated systems. Conda resolves this difficulty at the Python level by creating isolated environments (env); Equally, the same mechanism is applied at the operating system level in "NixOS/Guix:a Purely Functional Linux Distribution [START_REF] Dolstra | NixOS: A purely functional Linux distribution[END_REF]. One benefit to reproducibility is that Nix creates packages that are isolated from each other. These packages ensure that the environments are reproducible and have no undeclared dependencies. For example, BioNix [START_REF] Vallet | Toward practical transparent verifiable and long-term reproducible research using Guix[END_REF] is based on the characteristics of Guix.

Lack of academic reward. One major problem in reproducibility concerns the career advancement [START_REF] Liu | Promoting and Enabling Reproducible Data Science Through a Reproducibility Challenge[END_REF] of researchers in academia and how their work is acknowledged and eventually rewarded [START_REF] Ghimpau | Incentives, rewards, and recognition -what really motivates a researcher[END_REF][START_REF] Desmond | Incentivizing replication is insufficient to safeguard default trust[END_REF]. Traditionally, recognition and prestige have been associated with the number of publications and citations [START_REF] Dag | Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories[END_REF] in high-impact factor publications and have been decided according to metrics such as the h-index-h, Altmetrics, CiteScore, the Clarivate Analytic Journal Impact Factor (JIF), the Source Normalized Impact per Paper (SNIP), the SCImago Journal Rank (SJR), and the proposed Scientific Impact Factor (SIF) [START_REF] Lippi | Scientist impact factor (sif): a new metric for improving scientists' evaluation[END_REF]. The abuse of these metrics to decide career advancement promotes behaviors in scientists and journals that are detrimental to reproducibility, such as reticence or reluctance to publish code and data in articles or to split a single research into several non-significant articles because of the high pressure to publish. Much of the effort required to perform quality and reproducible research is not usually reflected in traditional metrics [START_REF] Raff | Does the Market of Citations Reward Reproducible Work?[END_REF]. In particular, the rewarding of computer science publications is limited to granting reproducibility badges [START_REF] Frery | A Badging System for Reproducibility and Replicability in Remote Sensing Research[END_REF][START_REF] Mauerer | Beyond the badge: reproducibility engineering as a lifetime skill[END_REF]. Currently, at least 138 computer science journals award reproducibility badges2 .

Figure 1: This diagram summarizes the structure of this paper, where three aspects are analyzed: efforts required by authors and publishers and the technological evolution required to reproduce results. The combined efforts of these three actors are required to close the reproducibility gap.

Moreover, some of the current incentives produce perverse behaviors in a hyper-competitive environment [START_REF] Edwards | Academic Research in the 21st Century: Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition[END_REF] which certainly goes against ethics and scientific transparency. They have also promoted the rise of predatory journals [START_REF] Cukier | Checklists to detect potential predatory biomedical journals: a systematic review[END_REF] based solely on Article Processing Charges (APC), articles of low-quality review standards, uncorroborated, or even false claims.

Measure and Evaluate Reproducibility

Reproducibility can be measured from different points of view, including the type of reproducibility which is evaluated (bitwise or statistical), the type of data [START_REF] Rosenblatt | Epistemic Parity: Reproducibility as an Evaluation Metric for Differential Privacy[END_REF] and field of the research [START_REF] Ahmed | Measuring Reproduciblity of Machine Learning Methods for Medical Diagnosis[END_REF]. It has already been shown in multiple works [START_REF] Raff | Research Reproducibility as a Survival Analysis[END_REF] that executing the same code on a different machine might not necessarily produce the same numerical results, but one can establish that a result is statistically equivalent to other [START_REF] Raff | A Step Toward Quantifying Independently Reproducible Machine Learning Research[END_REF][START_REF] Nordling | A literature review of methods for assessment of reproducibility in science[END_REF].

The survival analysis proposed in [START_REF] Raff | Research Reproducibility as a Survival Analysis[END_REF] permits to extract new insights that better explain past longitudinal data and extend a recent data set with reproduction times, taking into account the number of days it took to reproduce an article [START_REF] Collberg | Measuring reproducibility in computer systems research[END_REF].

This point is certainly important because it is imperative to measure reproducibility to evaluate the degree and percentage of reproducibility of an article. As will be seen in the artifact evaluation section 2.8, there is a wide disparity among journals/conferences in the criteria and policies for describing and evaluating artifacts partly as a result of the difficulty in measuring reproducibility.

Reproducibility Strategies and Technological evolution

Motivated by the great reproducibility challenges [START_REF] Schelter | On challenges in machine learning model management[END_REF][START_REF] Freire | Computational reproducibility: stateof-the-art, challenges, and database research opportunities[END_REF], there is extensive literature on data science projects, including current approaches for executing big data science projects [START_REF] Saltz | Current approaches for executing big data science projects-a systematic literature review[END_REF], and the Figure 2: Definition of Reproducibility, Replicability, Repeatability, and Reusability (4R) [START_REF] Fabien | Re-run, repeat, reproduce, reuse, replicate: Transforming code into scientific contributions[END_REF]. Different degrees of reproducibility can be considered according to the characteristics of the particular experiment or project. Figure 4: A classification of Repeatability, Reproducibility, Replicability, and Reusability (4R) [START_REF] Fabien | Re-run, repeat, reproduce, reuse, replicate: Transforming code into scientific contributions[END_REF] according to the characteristics a project or experiment.

Figure 5: General description of an editorial process for the publication of a scientific article with code and the involved actors: authors, editors, reviewers, and readers.

best coding practices to ensure reproducibility [START_REF] Chen | Best coding practices to ensure reproducibility[END_REF]. Different strategies have been proposed [START_REF] Leveque | Reproducible research for scientific computing: Tools and strategies for changing the culture[END_REF] to tackle the problem of reproducibility of scientific works, specifically in ML/AI.

Given that the size and scope of data science projects that can range from small projects of the Internet of Things (IoT) [START_REF] Pratim | A review on TinyML: State-of-the-art and prospects[END_REF] to very large high-end distributed HPC [START_REF] Pouchard | Computational reproducibility of scientific workflows at extreme scales[END_REF] as complex infrastructure required for example for the recently popular Large Language Models (LLM), different strategies are required to address the complexity of each project/experiment specifically.

In this section, we survey the most relevant characteristics that can be considered as general reproducibility strategies [START_REF] Odd | The Fundamental Principles of Reproducibility[END_REF], such as the use of open source software, open repositories, and open data formats, the use of well-established methodologies and following good practices, or system architectures which are typically used in systems dedicated to run ML/AI applications.

We classify the strategies identified in the literature into four main classes: software and data, environment, system data management and workflows, and methods. Each strategy can be part of a more complex one. For example Workflows can be themselves made of Containers, and Code publications strategies require the Open code/data repositories strategy.

1. Software and data reproducibility

• Adoption of free and open source software

• Tools with the potential to be used as reproducibility tools. For example, notebooks.

• Standardized automation benchmarks, open dataset formats, state of the art model baselines.

Environment reproducibility

• Software reproducibility: containers and virtualization

• System architecture: monolithic, microservices, server-less functions

• Hardware reproducibility, including ambient configuration. For example, Infrastructure as Code.

System and workflow reproducibility

• Data science project-life cycle management tools

• Metadata and provenance (lineage and traceability)

• Reproducibility as a Service. This includes 3rd-party specialized and trusted entities that certify reproducibility. They typically also offer services for the execution of algorithms on the infrastructure they provide.

Methodological reproducibility

• Adaption of good practices and methodologies

• Teaching and reproducibility culture

• Performing evaluation specifically for research code and data artifacts

• Publications with code (journals and conferences)

Open Source software and open repositories

One could think that uploading code and data to a public repository and labeling it as open-source software could be a sufficient guarantee of reproducibility and transparency in research [START_REF] Macleod | Improving the reproducibility and integrity of research: what can different stakeholders contribute[END_REF][START_REF] Barba | Defining the Role of Open Source Software in Research Reproducibility[END_REF].

There have been objections to this approach [START_REF] Abernathey | Cloud-native repositories for big scientific data[END_REF], as well as proposals for evaluating the reproducibility level [START_REF] Gonzalez-Barahona | Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositories[END_REF]. Others have proposed concrete solutions to the problem is reproducibility and transparency of scientific software [START_REF] Haim | How to Open Science: Promoting Principles and Reproducibility Practices Within the Artificial Intelligence in Education Community[END_REF][START_REF] Haim | How to Open Science: A Principle and Reproducibility Review of the Learning Analytics and Knowledge Conference[END_REF][START_REF] Stodden | Beyond Open Data: A Model for Linking Digital Artifacts to Enable Reproducibility of Scientific Claims[END_REF]. It cannot be ensured that code will not be modified after publication 3 , or that the code is executed in exactly the same environment, dependencies, and parameters. In many cases, the full reproduction of the work cannot be achieved and often requires contacting the authors to obtain detailed information. It might happen that even the authors themselves cannot replicate the experiment due to changes in their own research infrastructure, lack of documentation, or being outdated as the project evolves [START_REF] Stodden | An empirical analysis of journal policy effectiveness for computational reproducibility[END_REF].

As significant examples of these repositories and open source communities, we could cite, for example Github, Bitbucket, Gitlab, Zenodo, the Open Science Framework Findable: Metadata is assigned a globally unique and persistent identifier. For example, the Minimal Viable Identifiers (minids), or the Software Heritage SWHIDs; Accessible: The metadata is retrievable by its identifier using a standardized communication protocol; Interoperable: Metadata uses a formal, accessible, shared, and widely applicable specification for knowledge representation; Reusable: metadata is described in detail with a plurality of precise and relevant attributes.

The FAIR principles [START_REF] Parland-Von Essen | Supporting FAIR data: categorization of research data as a tool in data management[END_REF] applied to data allow promoting the reproducibility of the scientific publications focusing specifically on its R (Reuse) aspect. However, each research work is a particular case.

Compliance with the FAIR principles is a real challenge that is usually only partially achieved [START_REF] Albertoni | Reproducibility of Machine Learning: Terminology, Recommendations and Open Issues[END_REF].

These principles aim at categorizing the data in a more extensive and systematic way [START_REF] Parland-Von Essen | Supporting FAIR data: categorization of research data as a tool in data management[END_REF], as a mean to improve research data services. Also, they promote a convenient tripartite categorization of research data artifacts.

Many data science projects and research labs have started to adopt the FAIR principles, but there are discrepancies in how to implement it, from considering how to handle big data and using cloudnative repositories [START_REF] Abernathey | Cloud-native repositories for big scientific data[END_REF], as well as smaller scale data science projects that require an affordable means of sharing [START_REF] Vanschoren | Open science in machine learning[END_REF]. Each team or laboratory ends up establishing its own means to comply with the guiding principles. Therefore, it becomes a challenging task to determine the degree of FAIR compliance and, to a certain extent, audit it.

Open data formats and benchmarking

There are cases in which it is not viable to publish datasets and codes because they contain sensitive data (for example, medical data corresponding to individuals) or simply because it is under industrial secret. In these cases, the assessment of the reproducibility of the methods is compromised [START_REF] Rosenblatt | Epistemic Parity: Reproducibility as an Evaluation Metric for Differential Privacy[END_REF]. However, is in developing the concept of Federated Learning [START_REF] Baracaldo | Towards an Accountable and Reproducible Federated Learning: A FactSheets Approach[END_REF][START_REF] José | Towards a Metadata Management System for Provenance, Reproducibility and Accountability in Federated Machine Learning[END_REF] as a novel paradigm based on decentralized and private data for the shared training of models.

However, these exceptions are scarce and, in general, it is possible to approach open science by the use of open datasets, standardized formats, baselines, and benchmarks [START_REF] Vitek | Repeatability, reproducibility, and rigor in systems research[END_REF], allowing the scientific community to check the results published methods reliably.

Even when data can not be made available because of confidentiality reasons, one can rely on benchmarking and comparing results without necessarily accessing the data itself. There are several recent tools for this purpose [START_REF] Vitek | Repeatability, reproducibility, and rigor in systems research[END_REF], such as DataPerf, Mlperf, Collective Mind [START_REF] Fursin | Collective mind: Towards practical and collaborative auto-tuning[END_REF], ReQuEST [START_REF] Fursin | Invited Talk Abstract: Introducing ReQuEST: An Open Platform for Reproducible and Quality-Efficient Systems-ML Tournaments[END_REF] or MLCommons 7 with MLcube8 among others. They attempt to determine the state of the art in certain disciplines by comparing the performances. Competitions such as Kaggle or BRATS (brain tumor segmentation) [START_REF] Fathi Kazerooni | The Brain Tumor Segmentation (BraTS) Challenge 2023: Focus on Pediatrics[END_REF] challenge and others publish open datasets and have become a reference for the industry to evaluate and compare models.

System architecture

In terms of architecture for ML/AI systems, one can find two major trends: the deployment of microservices and the use of serverless functions.

Microservices allow the building of scalable and flexible software systems for which each component works independently and can be reused in different contexts. Because many applications of ML/AI require large resources in terms of computations and storage, they are usually deployed as a distributed system. The fact that different modules can work autonomously contributes to the reproducibility and understanding of the system, compared to monolithic ones [START_REF] Fritzsch | Adopting Microservices and DevOps in the Cyber-Physical Systems Domain: A Rapid Review and Case Study[END_REF].

In particular, to reproduce scientific experiments, microservices can help improve the portability and reusability of the software. By dividing the components of an experiment into microservices, one can increase the flexibility and modularity of the software, making it easier to adapt the code for new tests or experiments and lessening the dependency on software specific to a development environment.

Also related to isolation, Serverless Computing is a popular cloud-based computing model [START_REF] Jonas | Cloud Programming Simplified: A Berkeley View on Serverless Computing[END_REF] where the cloud provider manages the server infrastructure and platform resources, allowing developers to focus on application logic. Depending on the provider, they can also be referred to as lambdafunctions 9 .

The use of these serverless functions is beneficial for reproducibility in computer science as it reduces the complexity and variability of the underlying infrastructure, and it enables greater modularity and automation when developing applications and services.

Tools and platforms

This section surveys tools and platforms that are commonly used in ML/AI applications and how they contribute to reproducibility. Specifically, we focus on containers and cloud computing and the Infrastructure as Code (IaC) technique.

Notebooks

In data science, the use of notebooks has been popularized because of the opportunity to incorporate executable code, rich visualization, and documentation in the same document. It has become a common practice to publish and share work and a step forward for reproducibility. However, it has been shown [START_REF] João | A Large-Scale Study About Quality and Reproducibility of Jupyter Notebooks[END_REF] that this approach has some deficiencies, such as the lack of version control. Very recent studies [START_REF] Samuel | Computational reproducibility of jupyter notebooks from biomedical publications[END_REF] have also studied the low degree of reproducibility of Jupyter notebooks in biomedical publications.

Several solutions have been proposed to address these challenges [START_REF] João | Understanding and improving the quality and reproducibility of Jupyter notebooks[END_REF], including the use of Python scripts and the adoption of best practices for documentation, version control or additional packages as ReproduceMeGit tool to analyze the reproducibility of ML pipelines in Notebooks [START_REF] Samuel | ReproduceMeGit: A Visualization Tool for Analyzing Reproducibility of Jupyter Notebooks[END_REF] and Osiris [START_REF] Wang | Restoring reproducibility of Jupyter notebooks[END_REF].

Containers and cloud computing

Advances in cloud computing and containerization have undoubtedly contributed to the reproducibility of large distributed systems.

These systems are complex, made of several interacting components [START_REF] Wolke | Reproducible experiments on dynamic resource allocation in cloud data centers[END_REF][START_REF] Congo | Building a Cloud Service for Reproducible Simulation Management[END_REF] along a pipeline. It is required to have control over the execution environment in order to not only reproduce the experiments but even trust them at all.

Given a code, the associated data, and the execution pipeline, one should be able to obtain the same results over and over again. To achieve this, the pipelines, dependencies of the software, and the environment need to be perfectly defined. Virtual machines and lightweight containers such as Docker help by defining and fixing the execution environment [START_REF] Howe | Virtual appliances, cloud computing, and reproducible research[END_REF]. One could summarize these two concepts as

• Virtualization = data + code + environment,

• Cloudcomputing = data + code + environment + resources + services.

We address the topics on lightweight containers such as Docker, the MLOps methodology, the management of scientific workflows, and techniques such as IaC in the following.

Docker containers

Since its appearance in 2007, Docker containers have quickly become popular in computer systems and have become a fundamental tool for reproducibility. Its lightweight nature allows having several containers dedicated to small microservices on the same machine, with limited consumption and sharing of resources. This is a major advantage with respect to full virtual machines such as VMWare or Hyper-V. The light containers eventually allow for better reuse, and many infrastructures are nowadays migrating to containers, e.g. RE3 [START_REF] Bahaidarah | Toward Reusable Science with Readable Code and Reproducibility[END_REF].

Docker is one of the most efficient and widely used tools with applications for reproducibility nowadays, but, at the same time, we identify some of its limitations [START_REF] Shane Canon | The Role of Containers in Reproducibility[END_REF] to this purpose compared to container alternatives such as for example, Singularity Containers for HPC.

The emergence of containerization technologies such as Docker and orchestrators such as Kubernetes [START_REF] Micha L Orzechowski | Reproducibility of Computational Experiments on Kubernetes-Managed Container Clouds with HyperFlow[END_REF] has allowed the rapid development and automation [START_REF] Bahaidarah | Toward Reusable Science with Readable Code and Reproducibility[END_REF][START_REF] Vasyukov | Using Computing Containers and Continuous Integration to Improve Numerical Research Reproducibility[END_REF] of pipelines of experiments, thus making the reproduction of complex and computationally intensive experiments possible. Indeed, they can be divided into different functional blocks.

We can cite Repo2Docker [START_REF] Forde | Reproducible Research Environments with Repo2Docker[END_REF], which with Binder, can fetch a notebook for a given repository, create a proper execution environment, and run it inside a container. This makes the code publicly available for anyone to reproduce the results.

Specifically for HPC, there are initiatives such as The Extreme-Scale Scientific Software Stack (E4S), a community effort to provide open-source software packages for developing, deploying, and running scientific applications on High-Performance Computing (HPC) platforms. As an important contribution to the reproducibility of such a complex, E4S builds from source and provides containers of a wide collection of HPC software packages.

Many scientific experiments are made up of pipelines that concatenate several processes [START_REF] Sugimura | Building a Reproducible Machine Learning Pipeline[END_REF]. In terms of reproducibility over time, highly specialized platforms have been developed to manage these complex workflow management systems [START_REF] Steidl | The pipeline for the continuous development of artificial intelligence models-Current state of research and practice[END_REF] (e.g. watchdog [START_REF] Kluge | Watchdog 2.0: New developments for reusability, reproducibility, and workflow execution[END_REF]), tools [START_REF] Samuel | Machine Learning Pipelines: Provenance, Reproducibility and FAIR Data Principles[END_REF], roadmaps [START_REF] Ferreira | A Community Roadmap for Scientific Workflows Research and Development[END_REF] and general frameworks are proposed [START_REF] Melchor | A Model-Driven Approach for Systematic Reproducibility and Replicability of Data Science Projects[END_REF]. They allow researchers can focus on solving their specific scientific problems rather than the underlying infrastructure, networking, or other technical specifics [START_REF] Françoise | Marcelle: Composing Interactive Machine Learning Workflows and Interfaces[END_REF]. Despite the great step forward, many interoperability and reproducibility difficulties still persist [START_REF] Prabhu | Reproducible Workflow[END_REF][START_REF] Ghoshal | Experiences with Reproducibility: Case Studies from Scientific Workflows[END_REF] considering the immense possibility of languages, open or private infrastructures that are currently available or under development in the ecosystem of ML/AI data science technologies.

Workflow management systems

Scientific workflow management systems are useful for managing complex, cloud-distributed workflows [START_REF] Rosendo | KheOps: Cost-effective Repeatability, Reproducibility, and Replicability of Edge-to-Cloud Experiments[END_REF] and automating repetitive processes [START_REF] Cohen-Boulakia | Scientific workflows for computational reproducibility in the life sciences: Status, challenges and opportunities[END_REF]. They also enable detailed documentation and workflow sharing with other researchers, thus helping improve the reproducibility of results and speeding up scientific research [START_REF] Plale | Reproducibility Practice in High-Performance Computing: Community Survey Results[END_REF].

Formally, the workflows are represented as Direct Acyclic Graphs (DAG) [START_REF] Santana | Towards Reproducibility in Scientific Workflows: An Infrastructure-Based Approach[END_REF], where a task starts in a particular node to be processed, and then transferred to the next one in the chain until the final result is available in the last node. As pointed out in Section 2, it is required that the pipeline of the workshop and the node themselves follow well-established reproducibility principles to obtain reliable results, including access to the source code running the computations, along with an accurate description of the environment, the use of FAIR data, and the use of open data formats for interoperability, among others.

Each scientific community has developed its own workflow managers. We can cite some well-known ones, such as Taverna (bioinformatics, cheminformatics, and ever social sciences) or the Galaxy project (bioinformatics), OpenAlea (Botanics), Chimera (cheminformatics), or Pegasus (physics and bioinformatics), Knime(semantic workflow), Wings(graphical workbench). Pegasus was the workflow management system used by LIGO for the first detection of gravitational waves, certainly a paramount hit in physics.

The criteria to establish the reproducibility of a given pipeline can vary much from community to community. Although the basic principles remain (see Section 2), there are specificities depending on the field. We invite the reader to check the work of Cohen-Boulakia and co-authors, who conducted a study [START_REF] Cohen-Boulakia | Scientific workflows for computational reproducibility in the life sciences: Status, challenges and opportunities[END_REF] analyzing three cases of use of in-silico experiments in the domain of biological sciences with Taverna, Galaxy, OpenAlea, VisTrails, and Nextflow, proposing different criteria and discussing about these reproducible environments based on docker, Vagrant, Conda, and ReproZip.

The significant increase in articles on AI/ML inevitably forces an adaptation towards the management of both data and software because both are a source and contribution to knowledge. Therefore it is necessary to analyze tools, infrastructures, and technologies that have evolved to support these requirements. In this sense, AIOps/MLops evolves from the DevOps/DevSecOps (Development -Operations) concept to cover several of these aspects of reproducibility management infrastructures for computer-based scientific articles.

Transferring knowledge and prototypes from the academy to the industry is, most of the time, challenging [START_REF] Breck | The ml test score: A rubric for ml production readiness and technical debt reduction[END_REF]. There are very well-specified methodologies for the development of software in the industry, such as DevOps, which include CI/CD (Continuous Integration/Continuous Delivery). However, in the academic environment, these practices are not necessarily followed. In part, this is explained by the lack of career reward pointed out in Section 1.3.

MLOps [START_REF] Gift | Practical MLOps: operationalizing machine learning models[END_REF] can be considered the natural evolution of the DevOps best practices components adapted to the particular needs of ML-based software development [START_REF] Amershi | Software engineering for machine learning: A case study[END_REF]. In general terms, within data science projects MLOps tries to harmonize the practices of two environments with very different characteristics, such as academic/research environments with ML production environments for a final client, where reproducibility plays a very important role. It is an end-to-end process from the research model to the final model, exploited by the end customer or reproducibility reviewer.

There are few works that deal with MLops from the point of view of reproducibility. Among these, [START_REF] Odd | Do machine learning platforms provide out-of-the-box reproducibility?[END_REF] does an excellent analysis of the reproducibility of various MLops tools.

Other articles made a benchmark for different MLops features [START_REF] Schlegel | Management of Machine Learning Lifecycle Artifacts: A Survey[END_REF] and products available in the open source such as private code [122], which is equally important when data and software management is required by a journal. Let us mention here the most relevant ones, from our review of the literature:

• Neptune. A metadata store for any MLOps workflow. It was built for both research and production teams that run a lot of experiments 10 .

• Weights&Biases. A machine learning platform built for experiment tracking, dataset versioning, and model management 11 .

• Comet. An ML platform that helps data scientists track, compare, explain and optimize experiments and models across the model's entire lifecycle 12 .

• Sacred + Omniboard. Open-source software that allows machine learning researchers to configure, organize, log, and reproduce experiments 13 .

• Tensorboard. A visualization toolkit for TensorFlow 14 .

• Polyaxon. A platform for reproducible and scalable machine learning and deep learning applications 15 .

• ClearML. An open-source platform, a suite of tools to streamline your ML workflow 16 . [START_REF] Fursin | Collective Knowledge: organizing research projects as a database of reusable components and portable workflows with common APIs[END_REF] is an initiative that, based on its experience trying to reproduce hundreds of experiments, came to identify common patterns that are repeated from project to project. In a certain sense, it is a unifying proposal within the wide variety of existing MLOps solutions and seeks to resolve persistent integration issues.

With the emerging Internet of Things (IoT) technology and the advances in smaller devices with significant computing power, simplified ML models at the edge are possible with TinyMLOps [START_REF] Pratim | A review on TinyML: State-of-the-art and prospects[END_REF]. Significant reproducibility challenges appear considering the strong restrictions of energy consumption, limited computing capacity, and heterogeneity between different devices and technologies. Also considering that you can no longer containerize/virtualize with Docker.

Workflow languages

Despite the efforts to unify existing workflows, each community has kept its own particularities, including the language to define the pipelines [START_REF] Cohen-Boulakia | Scientific workflows for computational reproducibility in the life sciences: Status, challenges and opportunities[END_REF]. This fragmentation [START_REF] Adams | YAWL: An open source Business Process Management System from science for science[END_REF] makes it harder for integration and interoperability between different academic groups. Indeed, some of the groups use a very particular language for their workflows.

There are initiatives such as SHIWA (SHaring Interoperable Workflows for Large-Scale Scientific Simulations on Available DCIs) [START_REF] Korkhov | Shiwa workflow interoperability solutions for neuroimaging data analysis[END_REF] which try to provide a solution to this problem of interoperability. Multiple organizations and providers of workflow systems have also jointly worked to propose the Common Workflow Language (CWL) [START_REF] Demchenko | Experimental Research Reproducibility and Experiment Workflow Management[END_REF] with the aim of standardizing the pipelines around a common language.

Those specifications propose a conceptual workflow language to describe high-level scientific tasks, with the aim of promoting workflow specification portability and reusability and addressing the heterogeneity of workflow languages.

Infrastructure as code (IaC)

Much attention is paid to source code and containerization in order to address reproducibility, but unfortunately, not that much to hardware [START_REF] Bowman | Improving instrument reproducibility with open source hardware[END_REF]. With the rise of cloud computing technologies, the possibility of replicating the exact execution environment for an experiment is viable. Indeed, for reproducibility purposes, it is a requirement to define the characteristics of the hardware, such as the type of CPU, TPU, GPU, memory amount, or network architecture. This is especially important for a large distributed system as, for example, HPC applications.

In this respect, IaC provides several advantages towards reproducibility in computer science. One of the main benefits is that IaC allows researchers to accurately define and control their infrastructure in a format that can be easily stored, versioned, and shared, making it easy to reproduce experiments and obtain the same results at each execution. Defining infrastructure as code discharges from manually configuring infrastructure resources and allows researchers to easily version and share the infrastructure configuration with other colleagues. According to the Octave 2022 report [128], the Hashicorp Configuration Language (HCL) programming Terraform languages were widely used by developers in 2022, indicating that IaC practices are becoming quite popular for Github projects.

Additionally, IaC can help improve consistency and accuracy by ensuring that all instances of the infrastructure are created and configured identically. This helps ensure that the test conditions are the same each time an experiment is performed. IaC in the academic environment can significantly help in many aspects, such as the quality of the software developed, and is a step forward in the reproducibility of scientific research. As a recent example, Adorno-Gomes and Serodio [START_REF] Mestre | Infrastructure-as-code for scientific computing environments[END_REF] managed to define a complete experiment with IaC from a unique high-level code with Pulumi [130].

Provenance and Metadata Traceability of Artifacts

Provenance refers to the way in which the origin [START_REF] Silva | Provenance-and machine learning-based recommendation of parameter values in scientific workflows[END_REF] of the artifacts of an experiment is documented in metadata. Provenance documentation is a commonly used technique to improve the reproducibility of scientific workflows and research artifacts. There are numerous articles proposing tools such as ProvStore [START_REF] Huynh | ProvStore: A Public Provenance Repository[END_REF], ReproZip [START_REF] Chirigati | Reprozip: Using provenance to support computational reproducibility[END_REF], MERIT [START_REF] Wonsil | Integrated Reproducibility with Self-describing Machine Learning Models[END_REF], CAESAR [START_REF] Samuel | A collaborative semantic-based provenance management platform for reproducibility[END_REF], Provbook [START_REF] Samuel | Machine Learning Pipelines: Provenance, Reproducibility and FAIR Data Principles[END_REF] in several different disciplines and research areas [START_REF] Samuel | End-to-End provenance representation for the understandability and reproducibility of scientific experiments using a semantic approach[END_REF], demonstrating how it can help improve traceability, linage and transparency of results.

The PROV standards allow the task to be carried out (see Openprovenance 21 , for example. However, it is not yet complete and does not allow it to be generalized to multiple cases and languages. The foregoing requires the use of permanent, Unique Identifiers and tools that manage this aspect in order to have correct traceability of data sources and artifacts, even using new technologies such as blockchain [START_REF] Wittek | A Blockchain-Based Approach to Provenance and Reproducibility in Research Workflows[END_REF] InterPlanetary File System (IPFS) [START_REF] Kawamoto | AI pedigree verification platform using blockchain[END_REF] to achieve traceability and lineage of Software or code snippets.

Reproducibility as a Service (RaaS)

The Reproducibility as a Service (RaaS) concept was proposed in 2021 by Wolsin [START_REF] Wonsil | Reproducibility as a service[END_REF]. An strategy based on RaaS takes advantage of the availability of cloud computing technology to offer reproducibility services. This include the reproduction and research artifacts after the execution of the software in the controlled environment and its evaluation, validation, and certification (related to this, see Section 2.8 about code review). Also, granting reproducibility badges, tracking the provenance of software, or assigning persistent identifiers to the software at different granularity levels.

Another responsibility of RaaS is to manage the underlying architecture if a way that makes it easier for authors to share and execute their code depending on the chosen complexity, from baremetal infrastructure to fully managed services. Figure 6 shows how a SaaS architecture is organized in a complex system.

Crick and co-authors proposed [START_REF] Crick | Reproducibility as a Technical Specification[END_REF] to make a first approach to the offer of reproducibility services for journals/conferences from an empirical and quantitative point of view. They presented a cyberinfrastructure and the associated workflow for a reproducibility service as a high-level technical specification without delving into technical details. On the other hand, the work of Demchenko [START_REF] Demchenko | Experimental Research Reproducibility and Experiment Workflow Management[END_REF] addresses the topic of provisioning on demand of research environments and introduces the concept Equally, among other existing reference platforms, we could mention CodeOcean, Chameleon, and Whole Tale. They allow code to be executed in a wide range of languages, but they are still maintained at the demo level with certain technical restrictions to offer the mentioned features. They start to be actively taken into account by publishers.

Good practices and data management methodologies

Agility and security are among the many quality attributes of software [START_REF] Milewicz | Towards Evidence-Based Software Quality Practices for Reproducibility: Preliminary Results and Research Directions[END_REF], even though the majority of them are not specifically designed for reproducibility in computer science. However, data project management methodologies and well-known best practice guides are applied widely across the AI/ML industry to improve reproducibility.

Several studies [START_REF] Schelter | On challenges in machine learning model management[END_REF] and best practices guides [START_REF] Stodden | Best practices for computational science: Software infrastructure and environments for reproducible and extensible research[END_REF] have proposed different tools for the management of data science project artifacts [START_REF] Schlegel | Management of Machine Learning Lifecycle Artifacts: A Survey[END_REF] as well as methodologies. For example, Goodman et al. propose ten simple rules to achieve reproducibility, and The Turing Way handbook also provides a relevant compilation of good practices [START_REF]Illustrations from The Turing Way: Shared under CC-BY 4.0 for reuse[END_REF] to reproducible, ethical, and collaborative data science projects.

There is a consensus that one of the main factors limiting the success of data science projects is the lack of reproducibility in the management platforms [START_REF] Baillieul | Reflections on the future of research curation and research reproducibility[END_REF][START_REF] Alnoamany | Towards computational reproducibility: researcher perspectives on the use and sharing of software[END_REF][START_REF] Martinez | A survey study of success factors in data science projects[END_REF]. From the many methodologies available, the most popular are CRISP-DM [START_REF] Schröer | A systematic literature review on applying crisp-dm process model[END_REF], KDD, SEMMA, Microsoft TDSP, Agile DS Lifecycle, Domino, DS Lifecycle, IBM FMDS, RAMSYS, and MIDST, among others. These are widely used in the industry, especially CRISP-DM.

Certainly, as observed in the scarce literature, there is not a standard or unified methodology that is focused on reproducibility for data management itself. So far, only good practices, recommendations [START_REF] Turkyilmaz-Van Der Velden | Reproducibility Starts from You Today[END_REF][START_REF] Kenneth | Editorial: Method and data sharing and reproducibility of scientific results[END_REF][START_REF] Samuel | Understanding experiments and research practices for reproducibility: an exploratory study[END_REF][START_REF] Nichols | A better approach for dealing with reproducibility and replicability in science[END_REF], and guides from different fields of computer science of different needs are available.

Scientific publishers and reproducible research

Historically one of the main forms of communication, recognition, socialization, and validation before the scientific community are the articles published in journals and conferences [START_REF] Stodden | An empirical analysis of journal policy effectiveness for computational reproducibility[END_REF]. Publishers de facto become auditors of the scientific activity, and indeed, the metrics (Impact Factor and others) that they have established are the typical indicators that are used to evaluate researchers in their career and their advancement. Publishers have, therefore, a responsibility to assure the scientific integrity of the work they make public, along with their own interest in maintaining their own reputation. This includes not only avoiding fraud but also establishing clear quality criteria. In scientific publications, reproducibility is fundamental since it allows others to verify if the same or equivalent results are obtained when repeating the experiment, thus allowing them to potentially refuse a paper containing wrong or inaccurate claims. As pointed out by Heesen [START_REF] Heesen | Communism and the incentive to share in science[END_REF], the work that is not widely shared is not really scientific work.

In the following, we discuss two significant initiatives which have been put into practice by publishers: the possibility of associating code with the publications and the proper evaluation of software artifacts.

Publications with code

Associating source code with a particular publication is gaining great popularity in the scientific and technical community [START_REF] Bonsignorio | A New Kind of Article for Reproducible Research in Intelligent Robotics [From the Field[END_REF]. It allows for greater transparency and reproducibility, essential to guarantee the quality and reliability of the results [START_REF] De Sterck | Enhancing Reproducibility of Research Papers in SISC, JSC and JCP[END_REF]. However, the reproducibility aspects of this practice are evaluated in the Dataverse repository [START_REF] Trisovic | Advancing Computational Reproducibility in the Dataverse Data Repository Platform[END_REF], Figure 5 describes a typical editorial structure for publishing articles with code, which, as will be seen later, is not so easy to implement by the Journals considering technical and economic aspects.

Many conferences have started to request that the source code be given and made public. Others go one step further and perform an exhaustive evaluation of the artifacts. For example, Checklist NeuroIPS [START_REF] Pineau | Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)[END_REF]: It is a widely recognized checklist for the reproducibility assessment of conference papers.

From many examples, one might include here Code Ocean, used by IEEE's publishers after the integration of the CodeOcean's platform as a Computational Research Platform, Whole Tale [START_REF] Chard | Toward enabling reproducibility for data-intensive research using the whole tale platform[END_REF] allowing researchers to create and share scientific narratives that include data, code, and runtime environments [START_REF] Brinckman | Computing environments for reproducibility: Capturing the "Whole Tale[END_REF][START_REF] Chard | Implementing Computational Reproducibility in the Whole Tale Environment[END_REF], Binder as a platform that allows users to create and share code execution environments online, making it easy to reproduce and distribute results, PapersWithCode with open resources on ML, ReproducedPapers with open teaching and structuring machine learning reproducibility [START_REF] Yildiz | ReproducedPapers.org: Openly teaching and structuring machine learning reproducibility[END_REF], or the ReScience Journal [START_REF] Rougier | ReScience C: A Journal for Reproducible Replications in Computational Science[END_REF] which replicates computations from independent open-source implementations of original research and the advanced Chameleon22 large-scale edge to cloud tool [START_REF] Keahey | Chameleon: A Scalable Production Testbed for Computer Science Research[END_REF].

Unfortunately in many cases, this is limited to providing a non-persistent link [START_REF] Salsabil | A Study of Computational Reproducibility using URLs Linking to Open Access Datasets and Software[END_REF][START_REF] Idrissou | Documenting the Creation, Manipulation and Evaluation of Links for Reuse and Reproducibility[END_REF] to the source code repository in public platforms (see Section 2.1). Moreover, each journal sets its own strict criteria, formats, and procedures for authors. Aspects such as consistency, reproducibility, and reusability cannot be properly tracked or audited by other teams and research over time, thus limiting their impact [START_REF] Raff | A Siren Song of Open Source Reproducibility[END_REF].

Review of Research Artifacts

To begin with, it must be understood that for different reasons [START_REF] Dylan | Why don't we share data and code? Perceived barriers and benefits to public archiving practices[END_REF] an article is not 100% reproducible, but rather certain elements (e.g. computational artifacts, pseudocode, algorithms, demos) that the author decides to share and considers sufficient grounds to legitimize his results.

The evaluation criteria to accept articles for publication is traditionally well defined for scientific journals. They are based typically on originality, novelty, or overall scientific interest. However, when considering a publication not only as the article but also all major research artifacts, including source code, the criteria is relaxed, if considered at all. When the evaluation takes into account the associated source code, it is required to establish the proper evaluation criteria for peer review [167].

Conferences have started to publish guides containing checklists for the evaluation of artifacts and to grant the so-called reproducibility badges [START_REF] Frery | A Badging System for Reproducibility and Replicability in Remote Sensing Research[END_REF][START_REF] Athanassoulis | Artifacts Availability & Reproducibility (VLDB 2021 Round Table)[END_REF] if the conditions are met. Among the most important conferences, we can cite the checklist of NeurIPS 2019 23 , the ACM reproducibility badges 24 , as well as other initiatives such as the Unified Artifact Appendix and the Reproducibility Checklist 25 , the CTuning artifact evaluate 26 or the Empirical Evaluation Guidelines SIGPLAN NISO RP-31-202127 , among others.

Following several of the published guides, recently, the SC23 supercomputing conference (one of the most important conferences in HPC) [START_REF] Plale | Reproducibility Practice in High-Performance Computing: Community Survey Results[END_REF] adopted the Reproducibility Initiative where accepted papers with available artifacts were acknowledged with the corresponding ACM badges. The use of blockchain technology for artifact traceability has also been proposed [START_REF] Swapna Krishnakumar Radha | Verifiable Badging System for scientific data reproducibility[END_REF][START_REF] Kawamoto | AI pedigree verification platform using blockchain[END_REF].

CTuning has participated in the artifact evaluation task for different ACM conferences [START_REF] Fursin | Enabling reproducible ML and Systems research: the good, the bad, and the ugly[END_REF] and has defined a more detailed Unified Artifact Appendix and the Reproducibility Checklist based on the previous evaluation experience in ACM ASPLOS, MLSys, MICRO, and SCC'23 conferences.

Other specialized scientific journals have already implemented specific criteria to a greater or lesser degree. For example, Table 6 in the Appendix summarizes the checklist for Artifacts Description/Artifacts Evaluation (AD/AE) reproducibility [START_REF] Fostiropoulos | Reproducibility Requires Consolidated Artifacts[END_REF][START_REF] Malik | Artifact Description/Artifact Evaluation: A Reproducibility Bane or a Boon[END_REF] for a data science experiments and projects of different publishers.

Finally, reproducibility-certifying agencies are starting to offer their evaluation as a service in different disciplines working with sensitive or confidential data, outsourcing this function as a trusted third party. Recently Cascad [START_REF] Pérignon | Certify reproducibility with confidential data[END_REF] has been proposed in the field of Economics and Management [START_REF] Swapna Krishnakumar Radha | Verifiable Badging System for scientific data reproducibility[END_REF].

From our review of the data above, we observe that the existing criteria are still quite varied, not standardized, complex for the authors to fulfill, and time-consuming on the reviewer's side.

Table 4 is an extensive summary of the reproducibility strategies and technologies that have been reviewed in this work. However, it needs to be analyzed how they are implemented according to the reproducibility policies of the different scientific journals. Our survey tries, from an empirical point of view, to provide insights on the application of these strategies directly from participating journals.

Survey in Computer Science Journals

The increasing number of published articles in computer sciences, as well as and the fast development of new and innovative AI/ML methods, pose several challenges to publishers. The reproducibility, legitimacy of the works, and adapting the policy of the journal and the procedures of evaluation of the works is challenging [START_REF] Cousijn | A data citation roadmap for scientific publishers[END_REF].

Several papers [START_REF] Seibold | A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses[END_REF] have explored the effectiveness of journal policies regarding open source code and data sharing to validate the research procedures in an attempt to mitigate the reproducibility crisis.

Likewise, other studies address solutions, platforms, technologies, mechanisms, and procedures for the reproducibility of scientific articles that have been proposed to deal with the problem from the perspectives of the different actors involved: authors, publishers, industry, and scientific community. For example, the work of Gomes et al. [START_REF] Dylan | Why don't we share data and code? Perceived barriers and benefits to public archiving practices[END_REF], as well as Baker et al. [START_REF] Baker | Why scientists must share their research code[END_REF], focus on the barriers why authors might reluctant to share code and data in their publications and why that would be pertinent.

From another point of view, the reproducibility culture has also been analyzed in previous works [START_REF] Karathanasis | Reproducibility efforts as a teaching tool: A pilot study[END_REF][START_REF] Mauerer | Beyond the badge: reproducibility engineering as a lifetime skill[END_REF][START_REF] Hofman | Expanding the Scope of Reproducibility Research Through Data Analysis Replications[END_REF][START_REF] Fund | We Need More Reproducibility Content Across the Computer Science Curriculum[END_REF][START_REF] Lin | Building a Culture of Reproducibility in Academic Research[END_REF], and along with the culture, it has been discussed how to teach reproducibility in academic environments to young students and as seen in previous sections incentives [START_REF] Edwards | Academic Research in the 21st Century: Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition[END_REF], Massive Open Online Courses (MOOC) 2829 , good practices in the way of measuring and rewarding reproducibility, such as novel badging mechanisms, new measurement indices of valuation and code/data citation 30 [START_REF] Parsons | The History and Future of Data Citation in Practice[END_REF], the Scientist Impact Factor (SIF) for reputation and impact of researchers [START_REF] Lippi | Scientist impact factor (sif): a new metric for improving scientists' evaluation[END_REF], or applying statistical methods a mean to measure impact [START_REF] Mikhail | GitHub Statistics as a Measure of the Impact of Open-Source Bioinformatics Software[END_REF]. However, all these elements have not been analyzed as a whole as part of an articulated, agreed-upon Reproducibility policy in publishers.

Moreover, the experience, opinions, and results of journals implementing and adapting their policies with a strong focus on reproducibility have not yet been surveyed. Therefore, we surveyed SCOPUS-indexed journals specialized in computer science to know from them as a primary source of information about their experience in the application of reproducibility policies, insights, and the difficulties and successes derived from their policies.

In subsection 3.1 we explore which aspects were previously surveyed by other authors, including especially relevant questions. Subsection 3.2 presents our survey, along with the answers, which eventually leverages the discussion at Section 4.

Previous work

From the existing literature one can conclude that there is still incipient and timid progress toward implementing sharing and open science policies in scientific works [START_REF] Stodden | Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals[END_REF][START_REF] Stodden | An empirical analysis of journal policy effectiveness for computational reproducibility[END_REF]. The traditional peer review scheme is maintained, with slight variations, and it is, in general, limited to encouraging the publication of the source code and data in software repositories [START_REF] Lewis | From policy to practice: How journal-based data policies encourage scientists' adoption of reproducible research practices[END_REF][START_REF] Stodden | How journals are adopting open data and code policies[END_REF] 28 https://www.fun-mooc.fr/en/courses/reproducible-research-methodological-principles-transparent-scie/ Question 54 asked "Does the journal require linking to data, code, and other research results? ".

Although there is not much information available from journals about requiring links to data, code, and other research outputs in DOAJ, from the survey data the study found that nearly half of respondents reported not requiring this, against 24.8% who do. For more than 25% the answer was "No" or "Unknown".

The above questions are certainly limited to code-sharing policies in journals, but do not delve into actual reproducibility policies through article automation, evaluation, and preservation of reproducibility technologies. This represents a dilemma that is discussed in Section 5.3.

In the article [START_REF] Vasilevsky | Reproducible and reusable research: are journal data sharing policies meeting the mark?[END_REF], 318 biomedical journals were manually reviewed to analyze the journal's data sharing requirements and characteristics A total of 11.9% of journals analyzed explicitly stated that data share a total of 11.9% of journals analyzed explicitly stated that data sharing was required as a condition of publication. A total of 9.1% of journals required data sharing but did not state that it would affect publication decisions. 23.3% of journals had a statement encouraging authors to share their data but did not require it. A total of 9.1% of journals mentioned data sharing indirectly, and only 14.8% addressed protein, proteomic, or genomic data sharing. There was no mention of data sharing in 31.8% of journals. Impact factors were significantly higher for journals with the strongest data-sharing policies compared to all other data-sharing criteria. Open-access journals were not more likely to require data sharing than subscription journals.

Another contribution by Konkol et al. from the point of view of the analysis of reproducibility technologies for Publishing computational research [START_REF] Konkol | Publishing computational research -a review of infrastructures for reproducible and transparent scholarly communication[END_REF] concludes that still, publishing reproducible articles is a demanding task and not achieved simply by providing access to code scripts and data files. Several platforms were analyzed, including Whole Tale, ReproZip, REANA, o2r, Manuscripts, Gigantum, Galaxy, eLife RDS, Code Ocean, Binder and its limitations as well, the facilities it offers for authors. The previous article is complemented by the work of Willis [START_REF] Willis | Trust but Verify: How to Leverage Policies, Workflows, and Infrastructure to Ensure Computational Reproducibility in Publication[END_REF], who made an analysis of technical aspects and use of some of these technological infrastructures and repositories around seven reproducibility initiatives designed by journals to improve computational reproducibility.

In the work of Malik [START_REF] Malik | Artifact Description/Artifact Evaluation: A Reproducibility Bane or a Boon[END_REF], the technical difficulties are discussed, but also the benefits of implementing Artifact Description and evaluation policies for presenting scientific articles to journals and conferences.

The percentages of implementation of concrete reproducibility policies remain low. However, there is ongoing open discussion on the efforts and contributions that can be made by each of the actors in the reproducibility research ecosystem. In our case, this work analyzes the problem from the point of view of practical implementation of policies by publishers, based on your opinion and experiences with the following research questions: What is the best way reproducibility policy mandatory, or instead an incentive policy for authors and reviewers, to allow publishers improve the quality and impact of their publications? What type of technological infrastructure best supports these types of reproducibility policies?

Survey and Results

Following the methodology described in section A.2 and considering the literature mentioned above and the technologies discussed in Section 2.4, a series of reproducibility-oriented questions were carefully designed for the evaluation of reproducibility policies and their implementation. This is the base of the brief gap analysis we do in Section 5.4, where the answers are analyzed and discussed.

In the following we present the questions of the survey and the results.

Question 1. Do you want to be mentioned in the acknowledgment section as a Survey participant?

Despite having a policy of sharing and publishing code and data implemented at some level, some publishers refrained from being mentioned, probably due to not being able to match several items. Indeed, the survey asked for very specific questions about the implementation of infrastructures and technical details. Some publishers requested to be considered as anonymous in this question. Figure 7 shows the results.

Question 2. Respondent's role in the scientific journal

The answers came from a variety of different roles, with a slight predominance of Editors in Chief, Figure 8. A large majority (75 %) of the respondents indicate that they do have a reproducibility policy, as shows in Figure 9.

Related to this, in Figure 10 we can observe that there is a significant percentage (41.7%) of journals which request reproducibility as an essential condition for publication and thus make it mandatory. This decision has important consequences and, in general, it is counterproductive for almost all journals to add extra requirements for the publication because it reduced the publication rate 31 ). On the other hand, it improves the overall quality of the publications. Nine journals provided a link to their reproducibility policy, Table 1.

Question 5. How do you think the reproducibility policy requirements should be?

In this question we asked about what should be the most significant requirements for a reproducibility policy, regardless whether the journal actually implemented them or not. The results are given in Figure 11, with a variety of different preferences and showing, in any case, gradual interest towards making them mandatory. If so, which one?

The responses were very varied, which shows the lack of standardization in this matter. The problem of the evaluation fo the research artifacts has been extensively studied, yet without much agreement or formalization. See Figure 12.

Question 7. Journal access modality

Most of the journals answered that their publication modality was open-access, Figure 13.

Question 8. What is the range of your APC (Article Publication Charges)?

The ACP are very relevant for the discussion about how the reproducibility costs are shared between authors, publishers, and technology providers (see Section 5.1). Free publication costs predominate in the responses. In addition to question #7, it is an indicator that the business model of these journals is based on open platforms and repositories.

Question 9. Preferred sharing method

This question confirms that free open platforms are used to share the code, and the number of journals that owe third parties, or have their own technological storage infrastructure, is very low. See Figure 15 for the results.

Question 10. How compliant is your publication of software and data policy with FAIR-TLC?

The answers indicate the increasing level of implementation of the reproducibility policies, considering that most of the articles are accessible and reusable, but still low in the other attributes. This could be explained because the use of open repositories limits the journals to offer the other attributes satisfactorily. Figure 16 provides the results.

Question 11. Reproducibility validation method

The results (Figure 17) show that the traditional peer review model for article validation and acceptance is maintained, compared to other more automated forms of reproducibility validation. Therefore, validating the legitimacy of an article rests on one or two experts as well as their own available testing resources. The results (Figure 18) describe show that Github if the preferred specialized platform, although more for developers than for publishing research results. Zenodo, on the other hand, allows the citation of code and data through its identifiers, but remains a simple non peer-reviewed repository. There is therefore still a significant lack of automation in the policies of code and data for reproducibility purposes, to validate the legitimacy and quality of the articles.

Question 13. In the case you request reproducible research artifacts, which format is preferred?

The majority of the journals indicated that indeed they request the software and data artifacts, but as supplementary material (58.3%), with a large majority (50%) that request a link to the source code. Figure 19 shows the results.

Question 14. Which do you think would be the best way to reward the authors to submit reproducible articles?

Most of the answers (Figure 20) indicate that for most of the journals (46.2%) the preferred way to reward authors is to grant reproducibility badges, followed by a 38.5% of the answers which claim that the best way would be to offer free access to the journal or discounts.

Question 15. Which do you think would be the best way to reward the reviewers of reproducible articles?

The answers to this question (Figure 21) do not show a strong preference for any of the options, being the most preferred to offer free access to the journal or discounts, offering the reviewers being part of the editorial board of the journals, or even considering that it should be a voluntary task.

Question 16. Would you like to share briefly your view on the impact of implementing a reproducibility policy in the journal? For example, in terms of APC or other costs, the quality, citation impact, credibility of articles, or any other topic you would like to address.

This was an open questions to obtain from direct insights from the publishers. In the following we present their answers: • Observed average 15% increase in citations after one year, 33% after two years.

• It's a great selling point and definitely has increased our visibility and content reuse.

• We strongly encourage the authors to use Zenodo/Github as the repository for the code used in their experiments. We feel that both the reproducibility of the work and the ability of other to build upon it has a positive impact on science as a whole.

• Credibility of the results is the main impact. However, software evaluation adds time and complexity to our reviewing procedure.

• It has a deep impact on the credibility and citation of any article. I consider that our reproducibility initiative should be encouraged by most of journals. However, the most important issue is to encourage the authors to adopt a reproducibility-centered research methodology from the very beginning of their research.

• As an author, I always submit my articles with a reproducibility appendix, and I always design my experiments to be fully automatic and reproducible from scratch, including the automatic generation of all data tables and figures in the articles. On the other hand, as editor, I am encouraging this same approach among our authors, and most authors who have published reproducible papers in our reproducibility section are adopting this approach. However, it demands extra effort and a lot of discipline, and for this reason, not all authors are probed to do it. For this reason, our reproducibility initiative rewards the authors with a second article (reproducible paper).

Finally, there is a long walk to achieve most of the authors adopt a reproducibility-centered approach. It is absolutely necessary that PhD supervisors adopt and encourage this approach among their students. A very good way of achieving it is teaching all MSc and PhD students to design and implement reproducibility protocols and submitting reproducibility appendixes with all their papers.

• We do not have APC. In most cases, as much of the generated information has normally already been paid by public institutions.

• Higher Impact Factor since the policy has been in place

• Unfortunately, we do not implement a reproducibility policy in the BSR journal, but I would like to know more about it. On the other hand, I am not fully for the transparent publication of the data and software since it may violate privacy and licensing.

• APC for original software publications is less than half of the APC for research papers. Original Software Publications are currently the strategic focus of our journal.

• Surprisingly, authors are fine with our repository and requirements. We have not had any pushback. It does add editorial work and slows down the final acceptance. See the Github IJOC platform for how it works. https://github.com/INFORMSJoC".

• This is very new to us -we have only introduced it this year -to start on Sept 1st. We expect it to improve the reputation of the Journal and thus its attractiveness to researchers.

From this answers and insights, we shall in the following discuss on the technological possibilities available to address the reproducibility problem, and about the shared responsibility between authors, publishers and technology providers, including a brief gap analysis.

Reproducibility technological Discussion

As pointed out by several of the works we have reviewed here, reproducibility comes with great benefits for both authors and journals. Let us briefly summarize them.

• Greater credibility and recognition [START_REF] Ghimpau | Incentives, rewards, and recognition -what really motivates a researcher[END_REF]: reproducibility increases the credibility of the research and, therefore, can increase the recognition of the authors and the influence of their work in the scientific community.

• Research results are accurate and reliable to the extent that reproducibility is guaranteed. This is fundamental in the Scientific Method.

• Increased visibility and impact [START_REF] Boulbes | A Survey on Data Reproducibility and the Effect of Publication Process on the Ethical Reporting of Laboratory Research[END_REF]: the attention of the scientific community is attracted when articles are reproducible, thus increasing the visibility and the impact of the research.

• Facilitate collaboration and reuse [START_REF] Alnoamany | Towards computational reproducibility: researcher perspectives on the use and sharing of software[END_REF]: results obtained from reproducible experiments allow other researchers to build on them, which promotes collaboration and reuse of findings. Eventually, it allows for faster scientific advancement.

• Increase credibility and confidence in the results [START_REF] Samuel | Understanding experiments and research practices for reproducibility: an exploratory study[END_REF]: reproducibility allows the results obtained in an investigation to be verified and validated, increasing the confidence in its validity, following ethic directives [190], and transparency.

In the following, we shall discuss from the point of view of technological evolution based on the strategies presented as a reproducibility fundamental lever and support, how these interrelate into different challenges, problems, and solutions that have been proposed, and how they relate to these benefits. In particular, we discuss the problem of the responsibility of authors and publishers, such as their efforts towards reproducibility, the possibility of understanding reproducibility as a service, and finally, the impact of considering software as an important research artifact and the reward to researchers.

Dilemma: virtualization solution or dependency

As described in the technological evolution subsection 2.4, many of the reproducibility tools and platforms (eg, Workflows) proposed so far are completely based on container technology; this strong trend leads us to discuss and draw attention to the benefits and drawbacks of relying exclusively on this technology.

It could be stated that Docker or, in general, lightweight virtualization, is the holy grail of reproducibility, and, as will be seen, many solutions are based on this technology. With the popularization of agile methodologies and as shown by the landscape and the containerization strategy, many of the reproducibility problems try to be solved with docker [START_REF] Moreau | Containers for computational reproducibility[END_REF][START_REF] Shane Canon | The Role of Containers in Reproducibility[END_REF], which leads us to question if there is an abuse of lightweight virtualization. As suggested, Docker is practical light and facilitates many processes that in the past were tedious; however, it is not a tool specifically designed for reproducibility, and it cannot be used indiscriminately to hide bad practices.

The possibility of packaging, freezing, and porting a code to any infrastructure and maintaining stable functionality over time make it attractive in the scientific world; however, as stated in [START_REF] Fursin | The Collective Knowledge project: making ML models more portable and reproducible with open APIs, reusable best practices and MLOps[END_REF], this indiscriminate use brings great inconveniences, we will discuss in the following.

At this point, it is necessary to analyze in depth the problem of reproducibility, repeatability, containers, and development. The problem is that two characteristics are desirable in systems, but actually, they are antagonists. On the one hand, we want robust systems that will not break after an update. The classic example is a Python program that uses PyPI packages that, even if the user sets the versions in a virtual environment, the libraries may not be available in a particular version of Python. In that case, many system designers opt for virtualization.

The containers ensure reproducibility given that the complete environment is fixed. However, if proper attention is not paid to the maintenance of the container, it might end up facing security problems given that if the environment is simply fixed and not updated, the libraries will stop receiving bug fixes and security updates.

Docker is certainly a useful tool that allows to fix the execution environment, but still maintenance is required. Regular automatic testing is recommended.

The shared responsibility between authors and publishers

As presented in Sections 2.4 and 2.6, complying with the criteria for reproducibility implies some costs, as well as the shared responsibility and pooling between authors and scientific journals. It also needs a commitment to transparency and reproducibility [START_REF] Haibe-Kains | Transparency and reproducibility in artificial intelligence[END_REF]. Despite the analysis of the reproducibility stakeholders in [START_REF] Diaba | Reproducibility and research integrity: the role of scientists and institutions[END_REF][START_REF] Sebastian | Reproducibility: A Researcher-Centered Definition[END_REF][START_REF] Macleod | Improving the reproducibility and integrity of research: what can different stakeholders contribute[END_REF], the roles and relationship between authors and publishers are still diffuse and at least questionable: indeed much of the reproducibility burden relies on the authors. Authors have a responsibility to provide detailed information about the methods and techniques used in their research, as well as to make public the data and codes that were used to generate the results. They must also ensure that their results are replicable and thus they can be verified by peers.

On the other hand, publishers have the responsibility to establish clear policies and guidelines [START_REF] Stodden | An empirical analysis of journal policy effectiveness for computational reproducibility[END_REF] for the submission of scientific articles, as well as looking for the transparency and reproducibility of the results.

Indeed, guaranteeing and legitimizing the reproducibility of scientific work in ML/AI implies assuming significant economic and time costs [START_REF] Poldrack | The Costs of Reproducibility[END_REF] depending on the size and complexity of the research project. These cannot be assumed only by the researcher.

Reproducibility Cost

Estimating the cost of reproducibility is not easy because it can be considered from the execution of a simple container on a personal laptop to a distributed execution of software in the cloud, with the market costs per hour of CPU, GPUs and storage depending on each provider and their business model (for example, GCP, Amazon, Azure, Oracle, and others).

Existing virtualization and containerization techniques and cloud computing infrastructure are key elements in this problem [START_REF] Howe | Virtual appliances, cloud computing, and reproducible research[END_REF]. Therefore, the costs associated with cloud computing become relevant concerning reproducibility.

It is essential to highlight these associated costs [START_REF] Armbrust | Above the clouds: A berkeley view of cloud computing[END_REF] and the implications for the scientific parties that have a role in the reproducibility of the scientific work. One can describe the main technological costs for the reproducibility of computational projects/experiments as follows:

C R = C HD + C HC + C RC ,
where C R is the total reproducibility cost, C HD the cost of hosting data, C HC the cost of hosting code, and C RC the cost of running code.

The problem of increasingly complex research projects is not specific to computer science but common to other disciplines, especially when they combine different fields. Let's mention briefly the case of bioinformatics as an example. The researchers need to have not only knowledge of Biology but also the skills to operate the software and the data formats of the research artifacts, as well as the running environment. Typically, specialized expertise is required in Python, R [START_REF] Gandrud | Reproducible research with R and RStudio. The R series[END_REF], diverse operating systems, and database management and complex platforms such as, for example, Galaxy [START_REF] Enis Afgan | The galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2018 update[END_REF].

IaC, virtualization and containerization, and cloud computing approach help address this divisional responsibility in a simplified manner. They allow to track the steps followed by the author so other researchers can repeat and reproduce the experiment in the same environment. However, this still requires a high level of computing skills, which should not necessarily be assumed only by the authors. In Section 5.2.1, we discuss the Reproducibility as a Service (RaaS) strategy, which could be applied to manage this shared responsibility.

5 Reproducibility efforts of authors and publishers

Effort of authors

In the case of the authors, it should be understood that for several of the reasons discussed, in most cases, a scientific article cannot be reproduced in its entirety (100%). The authors generally choose to reproduce only parts of the algorithms, demos, or data, which is essential to support the conclusions.

Therefore, it is the authors' effort to seek 100% reproducibility of their work or to fully clarify the reasons that prevented reaching this objective, complying with the policies and requirements of the journals/conferences.

Articles Submission Reproducibility Guide for Authors

From our analysis of the shared responsibility between authors and journals and the most recent technological advances in computing, we shall discuss the efforts required by each of these two actors.

It is still very difficult for journals and authors to close the gap in a mutual effort, and it is even harder when the authors must comply with article submission guidelines between different journals.

Figure 22: Proposal for a future editorial process on which the article and code are published as a whole, and third-parties certify reproducibility.

An article composed of theoretical and computational parts can only be reproduced in a certain percentage and certain components that only the author is responsible for defining and specifying with the greatest of details and following a standard guide that avoids reprocessing between publishers.

MICRO2023 is a recent experience towards unified EA (artifact evaluation) guides and procedures 32 , which allow speed up the AE process. A conference where artifacts can be complex and timeconsuming to evaluate and 25% of the submitted artifacts were awarded the artifact reusable badge. In this context, practices were developed such as Reviewers performed an initial 'smoke-test' (for example, installing the artifact, or resolving access/environment/setup issues) and also reviewed the key claims of the paper and the artifact. Likewise, two surveys were carried out consulting authors and evaluators to seek feedback on the AE process. Important insights are derived from this survey, especially in enabling authors and reviewers to faster iterate on artifacts efficiently, seamlessly, in reasonable time. For example Reviewers provided suggest that requesting authors to prepare a subset of simulations (and/or representative checkpoints) would be a good practice. Results "will appear in the ACM/IEEE MICRO 2023 conference front-matter " 33 and support a trend towards improvements to the process and clearer and standardized instructions preferable to most subjective assessment of other experiences.

Therefore, in addition to standardizing the different evaluation and description guides of Artifacts (see Table 6), we propose to incorporate a mandatory and standardized unified guide between journals where the author contributes the effort to comply and assess the level of reproducibility of their scientific article (see 2: Proposal of a reproducibility checklist guide for authors. We propose to incorporate a mandatory and standardized unified guide between journals where the author contributes the effort to comply and assess the level of reproducibility of the scientific article.

Efforts of publishers

In the case of journals, given the wide typology of submitted articles and the reproducibility costs, it is economically unfeasible that they have their own reproducibility infrastructure, which explains the current tendency to rely on third trusted parties. In the following, we shall discuss how the Reproducibility as a Service (RaaS) methodology could help discharge authors from the burden that implies running code and maintaining a complex reproducibility infrastructure, as well as the opportunity of considering a software more valued research artifacts and thus properly rewarding authors. Finally, we provide a brief gap analysis from the results of our survey in Section 3.2.

Reproducibility as a Service

As pointed out in Section 4.2, reaching reproducibility might require for some projects a large technological investment, which should not be assumed only by the authors but shared with publishers and offered by specialized third parties. Here, we will focus on a particular strategy, Reproducibility as a Service (RaaS) [START_REF] Wonsil | Reproducibility as a service[END_REF], which might be helpful to this purpose.

As introduced in Section 2, RaaS is an approach to address non-reproducibility in scientific research by providing access to tools and resources that researchers and industrial actors to replicate experiments and data science projects. Also, to facilitate, manage, or overcome many of the limitations and barriers that we have identified in our review of the literature.

According to Brundage and co-authors [START_REF] Brundage | Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims[END_REF], one could label as RaaS any 3rd-party service made of tools that allow the reproducibility of scientific work. Their proposal is to use the existing cloud computing tools to offer a service that fills the gap between two major requirements to achieve reproducibility. On one hand, the actions taken by researchers who want to facilitate reproducibility. They provide a detailed procedure that allows to obtain the result artifacts, as well as the exact execution environment. On the other hand, the actions taken by publishers or the industry validate reproducibility.

Moreover, there are trusted third parties that deal with big data projects and confidentiality issues of sensitive datasets. They aim to reduce the need for the strong computing skills typically required to work in complex AI/ML data science projects. Cloud Native-based RaaS [START_REF] Wonsil | Reproducibility as a service[END_REF] adds an additional standardized layer with simplified interfaces for IaC, virtualization, and cloud computing (see Figure 6). As examples of these tools, we can cite Invenio34 , Eprints 35 , or DSpace36 , among others.

Although not particularly adapted to complex workflow systems and user interactivity, a good example of the relationship between a journal and a third party that offers reproducibility services in the cloud is the partnership between IEEE and Code Ocean37 . The code from IEEE articles can be browsed, discovered (assigned a DOI), run, modified, and eventually built the researcher's work on the cloud without any complex setup.

Software as a valuable research artifact and reward to authors

As presented in Section 1.3, traditionally, the published article has been considered the most important and rewarded [START_REF] Parsons | The History and Future of Data Citation in Practice[END_REF] research artifact, leaving aside Software production. Universities, research centers, and evaluation committees usually consider the number of articles published in high-impact factor journals and the number of citations and sometimes as the major criterion to hire a researcher for increasing the salary and career evolution, among other incentives. Consequently, researchers typically do not invest large resources in the reproducibility of the results, the quality of the produced software, or even the possibility of publishing the software itself.

Many research projects are based on software contributed by others, including libraries, applications, or complete frameworks, and in many cases, there is no explicit recognition of the authors of the third-party software. The recent incident about the vulnerability in the log4j library [START_REF] Hiesgen | The race to the vulnerable: Measuring the log4j shell incident[END_REF] is a good example of a widely-spread software used by hundreds of companies, not necessarily acknowledging the library's authors.

The lack of incentives for researchers and software developers to produce quality and reproducible software has a clear negative impact [START_REF] Ke | A network-based normalized impact measure reveals successful periods of scientific discovery across disciplines[END_REF] on the development of Open Science. Fortunately, the criteria to evaluate researchers are evolving in parallel in the right direction. For example, CNRS (the Center for National Scientific Research) in France announced in 2022 changes in their evaluation policy to include SW projects as an equally valid element to evaluate scientific production. The reproducibility tools provided by journals and conferences are fundamental for traceability, thus allowing the proposal of new metrics specific to software.

Different works [START_REF] Parsons | The History and Future of Data Citation in Practice[END_REF][START_REF] Arfon | FORCE11 Software Citation Working Group. Software citation principles[END_REF] have focused on analyzing the citation of scientific software and data publishers [START_REF] Cousijn | A data citation roadmap for scientific publishers[END_REF] as a natural need to implement FAIR and paper-with-code strategies. To this purpose, FORCE11 (The Future of Research Communications and e-Scholarship) [START_REF] Arfon | FORCE11 Software Citation Working Group. Software citation principles[END_REF] provides guidelines for the citation of software and data.

Software needs to be properly cited and preserved. These two requirements are certainly not easy to fulfill, given their dynamic and changing nature. Indeed, millions of software repositories are constantly being updated at every instant in Github and other repositories.

The Software Heritage project, supported by UNESCO, is one important step forward in both citation and perpetual preservation of software via proper identifiers, such as the SWHID [START_REF] Di | 4 Identifiers for Digital Objects: The case of software source code preservation[END_REF].

Zenodo also provides a Digital Object Identifier (DOI) and Chameleon a QR-code to reference the code. It is also a great source of information to determine the provenance of software contributions.

Regarding using badges as an incentive for authors reproducibility, it must be observed that they really impact the researcher's reputation in the same way as the popularized and mature badge system awarded in e-learning by important companies, academies, to certify technical skills [START_REF] Stefaniak | Instilling purpose and value in the implementation of digital badges in higher education[END_REF] published on reputable platforms such as Credly 38 and easily shared on Linkedin39 , which allow the candidate to reinforce their CV and demonstrate to the employers in a competitive labor market.

As pointed out by Dozmorov at al. [START_REF] Mikhail | GitHub Statistics as a Measure of the Impact of Open-Source Bioinformatics Software[END_REF], Github is, at the moment, the most complete database to measure the impact of software. Interestingly, they concluded that the number of forks as a measure of software impact is not correlated with the number of citations associated with a scientific paper. This finding, at first counter-intuitive, shows that citation indices (such as the h-index and others) do not fully explain the true impact of the scientific work and the associated software. The consequence is, therefore, that producing quality software is neither properly promoted nor taken into account for the career advancement of researchers.

There is, therefore, a need for metrics that are specific to software, beyond indirect measures such as the number of forks or stars in public repositories. Strategies such as RaaS and more adapted metrics such as the Scientific Impact Factor (SIF) [START_REF] Lippi | Scientist impact factor (sif): a new metric for improving scientists' evaluation[END_REF] could be of great help rather than the H-index or impact factor of the journal (FIJ) discussed in Section 1.3.

Finally, our recommendation is depicted in Figure 22, where there are incentives for all the actors, including 3rd-parties that implement permanent and long-term reproducibility infrastructures that support the publishing business.

Dilemma: reproducibility sharing policies

At this point an important clarification must be made, journal reproducibility policies should not be confused with traditional open access and open science initiatives. It could even be considered as an open topic that requires standardization. In [START_REF] Stodden | An empirical analysis of journal policy effectiveness for computational reproducibility[END_REF] Stodden makes a first approximation from the analysis of Data and Code Policy Adoption by Journals, and then in [START_REF] Stodden | Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals[END_REF] [184] analysis of journal policy implementation and effectiveness for computational reproducibility, however a clear concept of "reproducibility policies" is not consolidated. This leads us to consider that journals face an important dilemma in defining their internal policy of just limiting themselves to a code and data sharing policy or going further in defining veritable and strict automation tools and reproducibility evaluation article reproducibility policy.

Brief gap analysis

We provide a small gap analysis of the level of implementation of reproducibility policies that we observed from our survey. We intended to bring together all the elements of analysis. We include technological aspects, as well as the efforts required by both authors and publishers to help close or, at least, reduce the reproducibility gap. Aspects such as the standardization and implementation of reproducibility policies, adaptation of business models, and association with specialized third parties are considered. These recommendations come from analyzing the answers in our survey (Sec. 3.2).

Table 3 shows the journal policy evaluation gap in identified key aspects, indicating the survey question that helps evaluate the percentage of implementation of the reproducibility policies. With this table, each journal is evaluated in terms of its reproducibility policies and the effort it must make in the key aspects identified.

It can observed from the answers that there is a low percentage of implementation of reproducibility policies, as well as the low use of technological tools for automation, validation and sustainability of reproducibility in the long term (longevity of reproducibility). This is explained because there is still no consensus and standardization on what should be a good reproducibility policy for journals, as well as the lack of a developed and mature market of trusted specialized RaaS services.

To determine the gap, we used the following qualitative ranking:

• High: when there is a complete lack of accomplishment or implementation of the criterion

• Intermediate: when there is the presence of an initiative with immature development of the criterion

• Low: when there is a complete and functional implementation of the criterion Unfortunately there is a large gap in the implementation of some aspects, mainly those related to automation, establishing reproducibility policies, management of repositories, and the use of Table 3: The gap in the implementation of the journal policies policies, along with the related survey's questions.

persistent identifiers for the research artifacts, including software. Other aspects such as orienting the business model towards reproducibility itself or the use of FAIR data seem to be more developed.

Despite the observed gap, there is an opportunity to reduce the reproducibility gap with the common efforts of authors, publishers, and technological providers. See tables 2, 3 and 4 for more details.

Conclusion

We have presented a PRISMA-based systematic review of the existing literature on techniques and platforms which can be used in computer science and machine learning projects, putting the focus on reproducible research. In order to clarify what reproducibility is, we have also reviewed the different definitions in the literature, which after the NASEM report have been de facto standardized.

The main difficulties reported by researchers when trying to reproduce the work of their peers have been enumerated. We analyzed also the problem of how to measure objectively the reproducibility, especially in the case where the experiment not necessarily gives the same results each time it is repeated, but statistically equivalent.

From our discussion we have identified what we consider are the most important reproducibility strategies, such as the use of open source repositories or the use of FAIR data, or following methodologies which have been proven to be relevant to achieve reproducibility.

In this work, we have intended to address the problem of the credibility crisis specifically in computer science including ML/AI projects, from diverse reproducibility stakeholder points of view. We establish insights from the best practices, frameworks, methodologies, and technologies available at the moment.

In computer science, the variety of languages, new developments, platforms, frameworks, hardware, and architectures on which the code of scientific articles can be run are vast. In some cases, it requires third-party proprietary software and data, which is why means a significant challenge for publishers. Implementing reproducibility policies supported by RaaS or similar methodologies can certainly help reduce the reproducibility gap on publications. We conclude that the high cost of guaranteeing the reproducibility of a software project is not properly rewarded at this moment to the reproducibility stakeholders. Considering the costs in own infrastructure that would be required, one still needs to take into consideration business models that encourage investment by third parties in infrastructure and thus guarantee longevity and perennity in the reproducibility of scientific publications.

It is, therefore, necessary to share the efforts among the different actors. A mutually-beneficial relationship must be established between authors, reviewers, and publishers to balance the benefits and costs that authors and scientific publishers eventually assume. There are not standardized Description/Evaluation standards in this regard and, in many cases, authors reported that it is very tedious to meet the requirements of each different publisher. some authors are discouraged from improving the quality of their papers in the so-called publish or perish race. Moreover, some publishers (excluding those under the diamond model) are in a position of power over the authors who are charged significant article processing charges.

It is convenient to define a new metric equivalent of the Impact Factor, which could be used specifically for software. This could help properly reward the effort of software developers by acknowledging them clearly as co-authors of the scientific work and measuring the real impact of their contributions in reproducible computer science projects.

Our survey shows a growing concern about the reproducibility and how their policies can get into it. This reflected in the comments provided by the journals and the initiatives they want to push forward. Therefore, clearly defined and long-term plans are required to achieve a sustainable reproducibility model where each stakeholder obtains a benefit. For the moment, the traditional peer review evaluation methodologies are preferred. The responsibility of the validation is mainly on the expertise of the reviewers chosen by the editors and the few functional tests to the artifacts that they can do with their limited testing infrastructure. It also indicates that, in computer science journals indexed in SCOPUS, there is a low level of formal implementation of reproducibility policies supported by their own reproducibility platforms.

We conclude that it is imperative to bring together coordinated efforts to agree on standardized guides for authors for the submission of articles, unified reproducibility policies and artifact evaluation criteria from editors, supported by the reproducibility strategies and technological evolution discussed in this article. Consequently, there is a promising future with opportunities and potential to reduce the reproducibility gap identified with the joint effort of all actors involved to ensure reliability and trustworthiness in the knowledge conveyed by computer science-based publications. The data must be accompanied by information that makes it possible to trace its origin and the way in which it has been modified or processed. This includes provenance information Data

( a )

 a Basic block Software/Data and Hardware/Environment Reproducibility. (b) System/Flow Reproducibility.

Figure 3 :

 3 Figure 3: Generalization of an architecture allowing for reproducible projects or experiments. It is made of basic blocks, interconnected to build complex systems, applications, and workflows.

Figure 6 :

 6 Figure6: Representation of a RaaS-managed cloud infrastructure. The description layers, microservices, the serverless approach, and taking care of the granularity of the software help the reproducibility of a complex system.

Figure 7 :Figure 8 :

 78 Figure 7: Do you want to be mentioned in the acknowledgment section as a Survey participant?

Question 3 .Figure 9 :Figure 10 :

 3910 Figure 9: Do you have a Reproducibility policy or similar in your guidelines for authors?

Question 4 .

 4 If you wish, you can indicate the link to the policy of the scientific journal or guides for authors.

Question 6 .Figure 11 :

 611 Figure 11: How do you think the reproducibility policy requirements should be?

Figure 12 :Figure 13 :

 1213 Figure 12: Do you follow any guide or checklist for the evaluation of research artifacts? If so, which one?

Figure 14 :

 14 Figure 14: Which is the range of your APC (Article Publication Charges)?

Figure 15 :

 15 Figure 15: Preferred method to share research artifacts.

Figure 16 :

 16 Figure 16: How compliant is your publication of software and data policy with FAIR-TLC?

Figure 17 :

 17 Figure 17: Reproducibility validation method.

Figure 18 :Figure 19 :Figure 20 :

 181920 Figure 18: If you request to share source code, what platforms or repositories do you recommend for sharing code?

Figure 21 :

 21 Figure 21: Which do you think would be the best way to reward the Reviewers of reproducible articles?

  • Pachyderm. An enterprise-grade, open-source data science platform that makes it possible for its users to control an end-to-end machine learning cycle 17 . • MLflow. An open-source platform that helps manage the whole machine learning lifecycle. This includes experimentation but also model storage, reproducibility, and deployment 18 . • DVC (Data Version Control). It is a very popular tool in MLops and in the data science environment because it allows the versioning of training, testing and validation datasets in a very simple format 19 . • NextFlow. In terms of reproducibility, it allows Docker and Singularity containers technology for the creation of workflows. 20 • Collective Knowledge

	10 https://neptune.ai/
	11 https://wandb.ai/
	12 https://www.comet.com/
	13 https://github.com/IDSIA/sacred
	14 https://www.tensorflow.org/tensorboard
	15 https://polyaxon.com/
	16 https://clear.ml/

  For example, The Diamond OA Journals Study[START_REF] Bosman | OA Diamond Journals Study. Part 1: Findings[END_REF] makes a general survey; in our case, the results of its question 41 are highlighted. To the question "Do you have any policy or practice to stimulate open sharing of research data? " 42% of the respondents declared to have policy or practice to stimulate open sharing of research data. The study finds that an equal number of respondents who did not have an established policy and an additional 15% answered "Unknown". However, the factors that explain the adoption of open-data policies are not analyzed and it only focuses on other aspects of the publishing business.

	29 https://www.coursera.org/learn/reproducible-research
	30 https://datacite.org/

Table 2

 2 ).

	ITEM			OPTIONS
	Article based on Software/Data?		yes/no
	Programming languages used		(e.g., Python, C++)
	Contains instructions for reproducibil-	(e.g., complete, verified)
	ity
	Badges, Certified third-party Repro-	(e.g., ACM badge, Ctuning)
	ducibility Evaluators
	Infrastructure Reproducibility Re-	(e.g., Docker containers, MLflow,
	quired/Trusted	third-party	RaaS	CodeOcean, Chameleon)
	Operator
	Repository			(e.g., Zenodo, Software Heritage)
	Unique persistent citable identifiers of	(DOI, SWID, BlockchainID)
	Software/Data Artifacts
	Percentage of reproducibility of the Ar-	(%)
	ticle
	Reproducible components		(e.g. DEMO,virtual infrastructure, fig-
				ures, tables, Backend, Frontend, Mi-
				croservices, Lambda functions)
	Component Reproducibility degree	(R1,R2,R3,R4)
	Non-reproducible components (Why)	(e.g., proprietary software, sensitive
				data, distributed project)
	Table

Table 4 :

 4 Strategies for Reproducibility.

			Summary of strategies for reproducibility
	Type	Strategy			Papers		Examples
		Open Source Software, Open sci-	[80, 165, 34,	Github, Gitlab, bitbucket, Zen-
	(1)Sof	ence, repositories, FAIR	74, 76, 77,	odo, softwareheritage, dataverse,
					75, 73, 78]	Huggingface
		CSharing/Documentation tools	[91, 93, 94,	Reprozip, Notebooks, CRAN,
					95]		Rmarkdown,
		Open data formats, Baselines,	[203, 88, 86]	JSON, XML, MLperf, Data-
		SOTA Benchmarks				perf, Kaggel, Brats,CM, ML-
							cube, MLdev
		Container/	virtualization/	[98, 100, 191,	Docker, Vmware, singularity,
	(2)Env	Cloud			204]		AWS, GCP, AZURE, ORACLE,
							BioNix/ Guix
		Architectures			[89, 90]		monolithic/	microservice/
							serverless/cloud/hybrid
		IaC-Infrastructure as a Code	[127,	128,	Terraform, pulumi, kubernetes
					129,	130,	CloudFormation, Ansible, pup-
					101]		pet
		Cientific Workflows and MLOps	[118, 126,	Taverna, Galaxy, VisTrails,
	(3)Sys	tools BPML CWL languages	114,	113,	Nextflow, Neptune, Weight,
					120, 99, 112,	Comet, Omniboard, Mlflow,
					106, 125]		TensorBoard,	Polyaxon,
							ClearML, Valohai, Pachyderm,
							Kubeflow, Verta.ai, SageMaker,
							DVC, kheOps, RE3, Hyperflow,
							watchdog ,SHIWA
		Metadata	and	Provenance	[132,	131,	MERIT,	ROVPY,	PROV-
		(Traceability Lineage Logging	134,	136,	NEO4J,	PROV-DB,	CON-
		Monitoring)			135, 84, 138,	NECTOR,		NOWORK-
					137, 107]		FLOW, GIT2PROV,Provbook,
							blockchain, SWHID, DOI
		RaaS-Reproducibility as a Ser-	[139,	126,	Whole Tale, chameleon, CodeO-
		vice			157]		cean, IPOL
		AE/AD Peer Code Reviews	[167,	171,	reviewcommons, ArVix, Peer
	(4)Met				172,	156,	Community In (PCI), SIG-
					168, 115, 23]	PLAN, Ctuning,	NeuroIPS,
							Badging
		Publications with code	[154,	153,	Some Journals(nature), Confer-
					163, 155]		ences(ACM,IEEE), runmycode
		Policies, Good Practices, Gen-	[125,	147,	NASEM, DevSecOps, AIOps,
		eral frameworks, Recommenda-	142,	144,	MLops,	CRISP-DM, KDD,
		tions, Methodologies Life Cycle	144, 109, 58,	SEMMA, Turing way, Teaching
		Management			179, 42, 149,	Culture Reproducibility, Journal
					148,	177,	Reproducibility Policies
					150, 151]

  The results of the article have been obtained independently in a study by a team or reviewer other than the authors, without the use of artifacts provided by the authors.Method Results Replicated (RER)ROR + ORO + without the use of artifacts provided by the authors, the main results of the article were obtained independently in a subsequent evaluation by a reviewer or team other than the authors,MethodResults Reproduced (ROR-R) ROR + ORO + the main results of the article have been obtained in a subsequent evaluation carried out by a reviewer or team other than the authors, using, in part, artifacts provided by the author.

	Experiment customization	Does the system provide special instructions to cus-	Method
				tomize and tune the experiments?
	Notes
	NISO
	Artifact Available		A DOI or URL link to the repository along with a	Data
				unique identifier for the object is provided
	Hypothesis Artifacts Evaluated-Functional	Documented hypothesis the artifacts are documented, consistent, complete,	Method Method
	Prediction			Documented predictions exercisable, and include appropriate evidence of ver-	Method
	Setup			Documented parameters and conditions, statistical ification and validation.	Method
	Artifacts Evaluate-Reusable	significance of results. Artifacts associated with the paper are of a quality,	Method
	Problem description		Clearly described problem documented and well-structured that significantly	Method
	Outline			Conceptually described method exceeds minimal functionality to the extent that	Method
	Pseudo code			Documented pseudo code reuse and re-purposing is	Method
	Data repository Open Research Objects (ORO)	Data shared in accessible repository A DOI or URL link to the repository along with	Data Method
	Data metadata		Metadata included in the datasets a unique identifier for the object is provided for	Data
	Data license			Licensed data Functional + placed on a publicly accessible archival	Data
	Data citeable		DOI or P-URL of data assigned repository.	Data
	Research	Object	Reviewed
	NEUROips Checklist [156] (ROR)
	Model and algorithms		Clarified mathematical models, algorithms, settings	Experiment
				-assumptions explained -algorithm complexity an-
				alyzed
	Theoretical claim		Clarified claim statements -fully proven claims	Method
	Datasets			Relevant statistics, details of train/validation/test
				split, explanation of excluded data and preprocess-ing, link to downloadable version of environment and	Method
				dataset, description of quality Control methods
	FAIR-TLC[80]
	Findable			Data must have rich and accurate metadata that al-	Data
				lows for easy discovery and identification
	Accessible			Data must be available and accessible for free or	Data
				through clear and well-defined mechanisms
	Interoperable		Data must be structured and organized in a coherent	Data
				way so that it can be combined, integrated, and used
				in conjunction with other data
	Reusable			The data must be made available under a clearly	Data
				specified open license that allows its use and reuse
				by other users
	Traceable

https://www.acm.org/publications/policies/artifact-review-and-badging-current

https://cos.io/our-services/open-science-badges/

Indeed, the history of a repository in Github can be altered with a hard push command or using the corresponding tools provided by GitHub.

https://osf.io

https://www.dlhub.org/

https://dataverse.org/

https://mlcommons.org

https://mlcommons.org/en/mlcube/

Note that, despite their name, they are totally unrelated to lambda calculus!

https://www.pachyderm.com/

https://mlflow.org/

https://dvc.org/

https://www.nextflow.io/

https://openprovenance.org/store/)

https://www.chameleoncloud.org/

https://nips.cc/Conferences/2019/CallForPapers

https://www.acm.org/publications/policies/artifact-review-badging

https://ctuning.org/ae/checklist.html

https://ctuning.org/ae/reviewing.html

https://www.sigplan.org/Resources/EmpiricalEvaluation/

https://ctuning.org/ae/micro2023.html

https://www.linkedin.com/pulse/micro-2023-artifact-evaluation-report-56th-ieeeacm-symposium-fursin-bsgwe/

http://invenio-software.org

http://www.eprints.org

http://www.dspace.org

https://codeocean.com/signup/ieee

https://info.credly.com/

https://www.linkedin.com/

Acknowledgements

The authors would like to thank the financial support to the SESAME's OVD-SaaS project from Région Île de France and BPI France. Also, to the Ministry of Science, Technology and Innovation of Colombia (Minciencias), within the framework of Call 885 of 2020 for the financing of the Doctorates of Excellence Program.

Code repository Shared code in repository Experiment Code metadata Code metadata included Experiment Code license Code license included Experiment Code citeable Code (DOI) or (PURL) assigned. Experiment Data Code Specified dependencies, Training code, evaluation code, README file with results table, pre trained

A Research Methods

A.1 PRISMA Literature Review

To perform our review two strategies [START_REF] Antonio | reproducibilidad de las estrategias de búsqueda en revisiones sistemáticas publicadas en revistas españolas de Biblioteconomía y Documentación[END_REF] were adopted. On one hand, active search for articles highly cited reports, and follow-up of citation threads on computational reproducibility. On the other hand, we applied the PRISMA 2020 [START_REF] Matthew J Page | The PRISMA 2020 statement: an updated guideline for reporting systematic reviews[END_REF] methodology with the SCOPUS & WoS databases between 2020 to the present day (Table 5). Eventually, we reduced our analysis to [START_REF] Shane Canon | The Role of Containers in Reproducibility[END_REF]

A.2 Journals Survey

To avoid bias in the research and not only address journals that are known to apply reproducibility policies, e.g. (IEEE with CodeOcean 40 , ACM with reproducibility Badges 41 , Journal Nature 42 ), a request for participation was sent to several journals specialized in computer science.

Information was voluntarily requested through a 16-question form from a list of 500 journals specialized in computer science, scopus indexed from Q1 to Q4 in the period from July 1, 2023 to September 1, 2023. Although the question was very precise, they were left open to comments.

A.3 Classification of reproducibility criteria.

Table 6 shows the classification of reproducibility criteria for Artifacts Description (AD) and Artifacts Evaluation (AE).

Criterion Description Type

Results

Documented result and analysis