Robert Jäschke
email: robert.jaeschke@hu-berlin.de]

Yoann Moranville
email: yoann.moranville@dariah.eu

Keywords: DARIAH, research infrastructure, sustainability, technology, technical reference, software quality

Executive Summary

A fundamental basis of a successfully operating digital infrastructure such as DARIAH is formed by the services it provides to its users. In the particular case of the distributed setup DARIAH is using, the integration of new services requires support and guidelines that can be agreed to by all current and future service providers. Such generic guidelines can support individual research as well as new research projects just starting out, and -ideallylater enable the infrastructure to sustain their products.

Introduction

Through DESIR (DARIAH ERIC Sustainability Refined), DARIAH strives to define a roadmap for the sustainable operation of its services. This consists of the administrative sustainability through a business plan as well as guidelines and best practices for the actual development and operation of the services. The latter is addressed by work package 4 "Technology", aiming to technologically enhance the DARIAH research infrastructure and its services. The implementation of new services into the DARIAH infrastructure has been already investigated within the Humanities at Scale1 project (see [START_REF] Matoni | Basic DARIAH Services and Demonstrators[END_REF]. One of the specific challenges within DARIAH is its distributed nature: [...] the DARIAH infrastructure does not have a central coordinating authority that decides on its portfolio and strategic plans to extend and improve its overall possibilities. Rather, the DARIAH member countries contribute the services their partner institutions operate to form the DARIAH infrastructure. Therefore, the decision to integrate a service is always up to random volunteers addressing national requirements. (Matoni, Schmunk, Thiel 2017, Sec. 4) Building on this previous work, DESIR has organised a workshop with service providers and representatives of DARIAH and related infrastructures in Berlin in October 2017 (cf. Thiel 2017) to discuss these matters. Amongst the participating infrastructures CESSDA 2 , CLARIAH 3 , CLARIN 4 and DARIAH, the problems are similar and a number of commonalities were identified. These range from the requirements of service providers to use supported and supportable technologies to the user's requirement for continued availability of the services they need. These contrast with the individual research engineer's goal of developing innovative technology and building a demonstrator that can be used for new academic publications. At the same time, as technology and research questions develop and progress, new requirements can emerge to continue development of individual solutions. Thus, one of the central tasks for DESIR identified at the workshop was the formation of developer guidelines as basis for reusability: [These needs] can possibly be addressed through guidelines, recommendations, and criteria catalogues at varying levels of detail. Elementary guidelines for software developers just starting out, which typically only involve modest scripting and ad-hoc solutions implemented for highly specialised problems, can support the process from the start. ( Thiel

The Technical Reference

The DARIAH Technical Reference (TR), is available online [START_REF] Thiel | DARIAH Technical Reference[END_REF] as well as an annex to this document. It is being developed within DESIR and under its coordination and includes external contributions. The TR builds on a number of previous guidelines and best practices developed by several of the workshop attendants, in particular the CLARIAH Software Quality Guidelines (van Gompel et al 2016), the Netherlands eScience Center Guide (NLeSC 2018) and the CESSDA Software Maturity Levels (Shepherdson et al 2016).

The purpose of the reference is to collect best practices and software development guidelines as well as quality checklists. It is a collection of best practices and guidelines for developers and maintainers of infrastructure components, as well as quality recommendations. They can be used to either gauge the quality of ongoing developments or as a starting point for new research projects, in particular when the components are planned to be attached to the infrastructures.

The TR is a reference in that it lists general requirements and considerations, but it does not always specify choices. In particular, it does not define licences, technology stacks or hosting services. These are part of the implementation of the TR for a research infrastructure or a data centre. This approach was chosen, because it allows an increased compatibility with existing (internal) requirements on specific choices and it reduces the effect of the 'not invented here' syndrome5 .

Further Work

Over the course of the remaining project time, the applicability of the TR to software development will be tested, in particular at the DESIR code sprint6 . Applying the guidelines and best practices to the software developed by DARIAH and its partners in projects will ultimately allow to create new DARIAH services. Ideally, this will enable DESIR to create new services addressing the gaps identified previously, see Buddenbohm, Barthauer 2017, who provided an overview of the existing DARIAH infrastructure and services.

At the same time, quality criteria play a vital role in DARIAH's assessment of its partners' contributions as well as in the upcoming SSH Open Marketplace 7 . The Marketplace will be a registry of services and resources and will be based on certain quality standards for the DESIR INFRADEV-03-2016-2017 -Individual support to ESFRI and other world-class research infrastructures, Grant Agreement no. 731081. listing of services and resources. But it is also clear that the quality standards for the registry will likely be oriented to a large part at end users of the DARIAH infrastructure and services and whereas the TR addresses the development phase and operational aspects of services.

Beyond the immediate work within DESIR, the collaboration among CESSDA, CLARIN and DARIAH continues on the technical level as well as other common areas that have been identified by the governing bodies. This work explicitly relies on the Netherlands eScience Center Guide and the CLARIAH Software Quality Guidelines.

Technical Reference

To make use of these guides, implementing an institutional manual based on this is recommended. In many places, possible or required choices are pointed out and should be made for a specific manual.

It is particularly recommended to adopt the Recommendations to encourage best practices in research software whenever starting a new project. Also, consider using the Software Sustainability Institute's Software Management Plan.

This work is licensed under Creative Commons Attribution 4.0 International. • If in doubt, use Git.

CONTENTS

• Implement a Code Hosting Policy.

• Use meaningful commit messages, cf.

[ProGit] Sec. 5.2:

-Capitalised summary with a maximum of 50 characters followed by a blank line.

-Detailed but concise explanations in paragraphs or bullet points at 72 characters line length.

2. Use an appropriate OSI approved license.

• Decide on an appropriate license before you first commit.

• Ensure the license is compatible with all dependencies.

• If in doubt, choose APACHE-2.0 or EUPL-1.2.

• Add the text in a LICENSE.txt.

Maintain a README.

4. Document your software properly.

5. Use existing tooling to support development workflows.

6. Ensure maximal interoperability.

7. Ensure your software is usable and accessible.

8. Implement a release policy and keep a changelog.

9. Add a code of conduct in a CODE_OF_CONDUCT.md, like we do.

10. Specify contribution policies in a CONTRIBUTING.md, like we do. A legitimate policy can be that external contributions are not accepted and merged.

The README

The first thing users and other developers look at when first making contact with a repository is the README.md in the repository's root directory. Its purpose is to give a concise but comprehensive introduction to the project. It should 4. Static code analysis to reduce common errors and improve overall quality by adhering to standards and best practices.

5. A Continuous Integration solution that runs on every code commit to verify the test suite, lint code, perform static analysis and more.

Popular choices for CI are:

• Travis CI

• GitLab CI

• Jenkins

Interoperability

1. Internationalisation principles must be applied to ensure future localisation.

2. Localisation of the software must be available in English and should be available in other relevant languages.

3. Provide an API.

4. Use federated authentication.

Provide full configuration examples.

6. Provide endpoints for monitoring (service status) and statistics (user request and hardware utilisation).

Changelog

1. Adopt a release policy.

2. Maintain a manually curated Changelog.

• Use a CHANGELOG.md, like ours.

• Structure in a descending order by version with release date.

• Add all changes applied in that version.

• Have an unreleased section at the top for the next release.

6

Chapter 1. Developer Guidelines

CHAPTER

TWO OPERATIONAL GUIDELINES

For a generic collection of IT Service Management documents, see e.g. [FitSM].

Basic Operational Guidelines

1. Maintain a proper up-to-date documentation.

2. Establish processes for recurring actions.

3. Automate as much as possible.

4. Implement and verify backups.

5. Monitor the infrastructure.

6. Consider software configuration management.

Infrastructure Documentation

1. Keep an up to date inventory of all infrastructure components used, including

• Actual physical hardware (servers, switches, racks, tapes, appliances etc.)

• Virtual machines

• Cloud and container infrastructure

• Storage solutions and instances

• Backup systems 2. Keep an inventory of all services that are offered and their dependencies with each other and the underlying infrastructure.

3. Implement security.

4. Keep a record of incidents.

Documentation for (virtual) servers

Documentation for virtual servers should include the following information:

• Where is it?

• When was the incident mitigated?

• Who was informed and how?

• Has this ever happened before?

• Has sensitive data, such as user data or secrets, been compromised? Particular importance should be applied to record all steps taken to mitigate the incident. These should include the person, time and specifics of any action taken.

Security breaches and vulnerability exploits may need to be reported to authorities, in particular if sensitive and/or (legally) protected data was (potentially) affected. Users must be informed appropriately, responsibly and quickly.

Finally, decide upon and implement measures to prevent repetitions.

Incidents and Postmortems Chapter 2. Operational Guidelines

CHAPTER

THREE

POLICY RECOMMENDATIONS

Code Hosting Policies

General Rules

Make sure to publish your code in a version control repository.

• There are a number of well-known commercial solutions, such as

-GitHub -GitLab -Bitbucket
They all offer some free options and using them has a number of advantages, e.g.

-Good and established usability -High visibility of you code -Low barrier for find-ability and re-use -Good integration with other services and solutions

When using commercial and in particular external services, you must have a backup and data extraction strategy in place, which ensures that you can always move to another solution.

• There are a number of possibilities to host your own solution -The commercial solutions above.

-GitLab Community Edition -Gogs -Gitea gitolite

Specific Solutions

On Site

• Be sure to implement all relevant operational procedures.

GitHub

When using GitHub, implementing a suitable policy is recommended. This should include the following considerations.

Organisations

GitHub organisations can be used to group all repositories of an institution or a research project. It should be clearly stated who is responsible for an organisation, ideally stated on the organisation's GitHub Page.

Always ensure rights are managed by a sufficient number of people:

• At least two people should be owner of the organisation. It may be appropriate to have two owners from each institution involved in the project.

• Create an institutional account. This account becomes owner of all organisations the institution is involved in and makes sure access is granted to all appropriate individual employees.

Organisations should have a policy on who can be a member as well as how repository maintainers and maintenance status are defined and communicated.

Backup

As GitHub is owned by a commercial company (the Microsoft Corporation), the service provided through GitHub is subject to change through corporate development. It is highly recommended to set up an automated backup system, in order to ensure that a copy of all code and metadata (including issues, wikis etc.) exists.

Features

Use the features provided by GitHub, such as issue labels, issue and pull-request templates etc.

Release policy

• Use semantic versioning.

• Provide releases as downloads.

• If applicable, provide binaries for all supported platforms.

• Have a roadmap.

• Consider providing releases via research repositories with citable references.

12 Chapter 3. Policy Recommendations CHAPTER FOUR SOFTWARE QUALITY CHECKLIST

General

• [ ] Does the software have a descriptive name?

• [ ] Is there a short high-level description of the software?

• [ ] Is the purpose of the software clear?

• [ ] Is the targeted audience of the software clear?

• [ ] Does it (and its dependencies) use OSI approved licenses?

• [ ] Is the software under version control?

• [ ] Is there a website for the software?

• [ ] Does the software have a release mechanism?

• [ ] Is the software available in packaged format or only sources?

• [ ] Are maintainer and development status clear, including contact information?

• [ ] Are the requirements listed and up to date?

• [ ] Is the interface responsive and accessible?

• [ ] Is copyright and authorship clear?

• [ ] Is there a contribution guide?

Documentation

• [ ] Is there an accessible getting started guide?

• [ ] Is there an accessible user guide?

• [ ] Is there a full user documentation?

• [ ] Does the user interface link to held references?

• [ ] Are there examples, FAQs and tutorials?

• [ ] Are known issues documented?

Development

• [ ] Is the development setup documented?

• [ ] Is the build mechanism documented?

• [ ] Does the build mechanism use a common single-command system (i.e. Maven)?

• [ ] Is the software API documented?

• [ ] Are all appropriate config options externalised and documented?

• [ ] Does the code allow internationalisation (i18n)?

• [ ] Is the software localised (l10n)? English is mandatory.

• [ ] Is there a test suite?

• [ ] Is test coverage above 80%?

Interoperability

• [ ] Are file formats standard compliant and documented?

• [ ] Is the API standard compliant?

• [ ] Does it provide a monitoring endpoint?

• [ ] Does it adhere to an interface style guide?

• [ ] Does it use existing authentication systems (OAuth2/eduGain)?

Administration

• [ ] Are software requirements such as operating system, required libraries and dependencies specified including versions?

• [ ] Are hardware requirements for CPU, RAM, HDD, Network specified?

• [ ] Are there deployment instructions?

• [ ] Is there a comprehensive and fully documented example configuration?

• [ ] Is a start-up script provided?

• [ ] Are there troubleshooting guides?

14 Chapter 4. Software Quality Checklist

GLOSSARY

These are some of the most important terms used throughout. See also GitHub Glossary.

• Collaborator -Someone with write access to the repository.

• Contributor -Someone who submits code to the repository, either directly or via pull-/merge-request.

• Developer -Anyone who writes code.

• Maintainer -The person responsible for feature development and deciding about contributions.

  DESIR INFRADEV-03-2016-2017 -Individual support to ESFRI and other world-class research infrastructures, Grant Agreement no. 731081.

  members of the consortium (including the Commission Services) EU-RES Classified Information: RESTREINT UE (Commission Decision 2005/444/EC) EU-CON Classified Information: CONFIDENTIEL UE (Commission Decision 2005/444/EC) EU-SEC Classified Information: SECRET UE (Commission Decision 2005/444/EC) Disclaimer DESIR has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 731081. This publication reflects the views only of the author, and the Commission cannot be held responsible for any use which may be made of the information contained therein.

  2017) DESIR INFRADEV-03-2016-2017 -Individual support to ESFRI and other world-class research infrastructures, Grant Agreement no. 731081.

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2 The README . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.3 Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.4 Tooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.5 Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.6 Changelog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Operational Guidelines 7 2.1 Basic Operational Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2 Infrastructure Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.3 Documentation for (virtual) servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.4 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.5 Incidents and Postmortems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Policy Recommendations 11 3.1 Code Hosting Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.2 Release policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Software Quality Checklist 13 4.1 General . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 4.2 Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 4.3 Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 4.4 Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 4.5 Administration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Glossary 15 Bibliography This is the technical reference for the DARIAH Infrastructure. It is a collection of basic guidelines and references for development and maintenance of infrastructure services within DARIAH and beyond. This work relies heavily on work by the community and in particular CESSDA ERIC, CLARIAH-NL, DARIAH-DE as well as the Netherlands eScience Center and the UK Software Sustainability Institute.

  All development should be made available publicly under open source licences. 1. Use version control right from the beginning of a new project.

Humanities at Scale, Horizon

2020, Project ID 675570 2 https://www.cessda.eu/ 3 https://www.clariah.nl/en/ 4 https://www.clarin.eu/

https://en.wikipedia.org/wiki/Not_invented_here

To take place in Berlin, 31.07.-02.08.2018, https://desircodesprint.sciencesconf.org/

The Marketplace has been identified as high priority in the DARIAH Strategic Action Plan. DESIR WP5 will release a mockup of the Marketplace, work is planned for future DARIAH activities.

provide links to further (more detailed) documentation, websites or other background information. Depending on the relevant ecosystem, specific guides or templates for READMEs exist.

The following basic information must be provided:

1. The project name 2. A short but meaningful descriptive summary of the repository

The maintenance status

The following questions must be answered:

1. What does the software do? 2. Who will use the software? 9. Who has to be acknowledged? I.e. who has played a significant role for the creating of the software? This can e.g. include funding, research communities, or co-workers that are not part of the project but have given advice and/or input to the development process.

Documentation

Documentation is fundamental to ensure usability and usefulness of the software. It must be stored along the code, ideally in the repository's docs folder. Basic documentation should also be included in the README.

Documentation is relevant in many forms, each of which should be addressed for different audiences with varying degree of experience and knowledge.

4

Chapter 1. Developer Guidelines Technical Reference, Release 0.1 1. User documentation: Include a documentation for end users, including e.g.

• Examples

• Tutorials

• How-Tos

• FAQs

• Screen-casts

• API documentation 2. Developer documentation: Provide instructions for developers.

• How to set up the environment.

• Dependencies, including -Supported operating systems

• Requirements, e.g. hardware, architecture, CPU, RAM, disk space and network bandwidth.

• How to build the code.

• How to package the code.

Additionally, inline code documentation should be used as appropriate.

• Always adhere to the language's standard or well established once such as the Google Style Guides.

• Document the why and not the what, cf. [CleanCode].

3. Administration documentation: Provide instructions for installation, configuration and maintenance, in particular when as a daemon (e.g. a micro service).

• Configuration instructions

• Start-up script (e.g. init or systemd)

• Monitoring setup, ideally through a monitoring endpoint

Tooling

In order to support an efficient development workflow and ensure a high degree of software quality the use of appropriate tooling is strongly recommended.

This should include:

1. Using an appropriate code editor or IDE.

While the individual person should be free to choose the solution that best fits his or her need, the editor/IDE should provide standard features such as syntax highlighting and code completion for all relevant languages.

Use EditorConfig to ensure consistency of code submitted by all developers. In case others can contribute to the software as well, it is recommended to add the respective .editorconfig to the software's repository.

2. Code linting within the editor, as pre-commit hooks and part of further automation should be used.

3. Unit testing to improve code quality and simplify future development.

Tooling

• Location, e.g. rack number or visualisation infrastructure

• IP address(es)

• DNS name(s)

• What is it?

• Operating system

• Human readable purpose

• Who knows about it?

• Responsible system administrator

• Secondary system administrators

• Project manager

• Which services does it run?

For each service, list the following:

• Data directories

• Dependencies

• How can it be fixed?

List possible procedures for emergencies, such as

• Whether to try rebooting or remounting something

• How to restore from backup

Security

Implement a good AAI and adhere to standards such as [Sirtfi] and [Snctfi]

Incidents and Postmortems

Record all outages including • Which service was disrupted?

• What else was affected?

• Who was in charge of the recovery?

• When was the incident discovered?

• How and by whom?

• When has the incident begun? 8 Chapter 2. Operational Guidelines