Will Simm
email: w.simm@lancaster.ac.uk

Gordon Blair

Richard Bassett

Faiza Samreen

Paul Young

Models in the Cloud: Exploring Next Generation Environmental Software Systems

Keywords: cloud computing, environmental modelling, data science

There is growing interest in the application of the latest trends in computing and data science methods to improve environmental science. However we found the penetration of best practice from computing domains such as software engineering and cloud computing into supporting every day environmental science to be poor. We take from this work a real need to re-evaluate the complexity of software tools and bring these to the right level of abstraction for environmental scientists to be able to leverage the latest developments in computing. In the Models in the Cloud project, we look at the role of model driven engineering, software frameworks and cloud computing in achieving this abstraction. As a case study we deployed a complex weather model to the cloud and developed a collaborative notebook interface for orchestrating the deployment and analysis of results. We navigate relatively poor support for complex high performance computing in the cloud to develop abstractions from complexity in cloud deployment and model configuration. We found great potential in cloud computing to transform science by enabling models to leverage elastic, flexible computing infrastructure and support new ways to deliver collaborative and open science.

Introduction

Models in the Cloud is a three year project which sets out to explore the opportunity for a paradigm shift in the support offered by cloud computing for the execution of complex environmental models. Cloud computing is having a major and transformative impact on many areas of society, including in smart cities, eCommerce and eGovernment. Significant advances are being made in these areas and cloud computing is proving to be of significant benefit in minimising up-front investment, achieving economies of scale, supporting elasticity in the underlying computational/ storage capacity and out-sourcing of the infrastructure management [START_REF] Johnson | Frameworks = (components + patterns)[END_REF].

These advances present a new opportunity for configuring computing on demand, and provide a fabric of services that can be configured and combined to support new modes of working. In visioning the future, consider that the code for a software model of the environment is constrained by the computer system it is to be deployed upon, traditionally a desktop computer or a high performance parallel computing environment. Here, the experiment is designed to fit the available computer. Now consider that cloud computing effectively allows you to design a computer around your experiment, removing constraints in a flexible, on demand, dynamic environment, then the potential opportunities for science are vast.

In his call to action for software engineers, Easterbrook [START_REF] Easterbrook | Climate change: A grand software challenge[END_REF] specifically identifies 'Computer-Supported Collaborative Science' as a fundamental way in which software engineering can contribute to addressing the grand challenge of climate change. He identified that through supporting earth system models with software engineering tools and techniques we can accelerate the process of getting scientific ideas into working code.

Blair et al. [START_REF] Blair | Data science of the natural environment: A research roadmap[END_REF] laid out a roadmap of 10 challenges for research in data science of the natural environment. These research challenges cross-cut the themes of data acquisition, infrastructure, methods and policy making and are summarised here: 1) supporting a cultural shift towards more open and more collaborative science; 2) build on cloud computing, extending the levels of abstraction for the domain of science; 3) address complexity more fundamentally and explicitly, e.g. data science techniques to address extremes and emergent behaviours; 4) provide the tools to reify uncertainty and reason about cascading uncertainty; 5) seek adaptive techniques such as adaptive sampling or adaptive modelling driven by uncertainty considerations; 6) seek approaches that deal with epistemic uncertainty in environmental modeling and links with dealing with emergent behavior in complex and irreducible phenomena; 7) seek novel data science techniques, especially those that can make sense of the increasing complexity, variety and veracity of underlying environmental data; 8) seek innovations in modeling by combining process models with data-driven or stochastic modeling techniques; 9) incorporate sophisticated spatial and temporal reasoning, including reasoning across scales; and 10) discover new modes of working, methods and means of organization that enable new levels of cross-disciplinary collaboration.

In this work we primarily address challenge 2) and see running models in a collaborative, elastic cloud environment is fundamental to underpin many of the other challenges. In supporting new modes of science, it is critical to provide tools at the right level of abstraction for uptake by scientists in their work, and it is through the lens of abstraction that this project focuses to ascertain what is in place and what is required to leverage cloud computing.

Abstracting from underlying compute infrastructure and defining interfaces and datastores for environmental models will allow models to be connected and run more efficiently, and to allow scientists to concentrate on the science, resulting in a better understanding of phenomena and uncertainties which would hopefully be reflected in better policy informed by results. This work forms a core pillar of technology enabling the concept of 'models of everywhere' -a vi-sion to have models of everywhere, models of everything and models at all times, being constantly re-evaluated against the most current evidence [START_REF] Blair | Models of everywhere revisited: A technological perspective[END_REF].

In prior work we published a qualitative study with environmental modellers, and an implementation and associated feedback from modellers on a cloud deployment of a complex weather model [9]. This paper summarises that work, adds further analysis and describes the next steps taken.

The aim of this research is to determine if the principles and approaches exist to leverage cloud computing at a usable level of abstraction, and to inform the development of new tools and practices. This paper contributes the approach we took, the lessons learned from qualitative and experimental work, and adds reflections on lessons learnt to inform taking this work forward.

Methodology

As part of an interdisciplinary team of researchers3 , we undertake agile research. This involves relatively short cycles of "Plan -Act -Reflect", with each cycle informing the next. Supporting this work is an evolution of the Speedplay research methodology [START_REF] Ferrario | Software engineering for 'social good': Integrating action research, participatory design, and agile development[END_REF], rooted in participatory design that sees the computer scientists and environmental scientists in equal partnership, developing together. In this research we will go full circle, not just unravelling and understanding the opportunity, but designing and building technologies with end users embedded in the process.

We had end users as collaborators embedded throughout the project, the research team was comprised of experienced computer scientists and environmental modellers. We first did a qualitative study phase with environmental modellers from across the spectrum of the modelling community, this allowed us to up skill in understanding, shared language and get a handle on the challenges faced by modellers. It also helped focus the resources of the study.

We went onto an experimental phase where we began 'learning through doing' -exploring the application of abstraction technologies to this domain, putting models into cloud architectures and learning about the challenges in doing this both from computing and environmental science perspectives. This is the 'Act' part of the cycles in which feedback and reflection is embedded in directing the research.

Qualitative Phase

We undertook a study consisting of semi-structured interviews and demonstrations with a diverse group of five environmental scientists engaged in writing software to model a variety of environmental systems and processes. The purpose of the study was to gain an understanding of how environmental models are developed and deployed, and the computing tools and architectures which are used. These sessions also up skilled both computer and environmental scientists in their domain languages (e.g. cloud in the sky vs cloud computing, environmental model vs software model) and understanding of each other's domain. Further detail is in Simm et al. [9] but the findings are summarised and analysis extended here.

We selected our 5x participants from those working with a range of model complexities, from small scale statistical models of insect population to community developed global climate models. This was to enable us to gain an understanding of the challenges and opportunities at each scale. We grouped findings under Technical, Scientific and Human headings, for brevity just the main technical findings are summarised here [9]:

Computational Demands and Resources required by models varied from desktop machines to institutional high performance computing (HPC) facilities. When running models on HPC, in depth systems administration skills are required such as file system preparation, shell script writing and good command line familiarity. Data is input and output in flat files not data stores, and sorted and downloaded by FTP and shared by email. Projects are not costed to include projected compute costs or support form experienced software engineers.

In terms of Computational Skills and Expertise each of the participants had self-taught programming and systems admin without a formal education in computing. Code is frequently written as a monolith with little thought to code reuse or defining interfaces or structure.

Code Understanding -models are written in scientific languages such as Fortran, Matlab and R, often models are configured and run with Bash scripts and results are analysed using Python and R. Code is often not well commented and is difficult for others to understand, and Integrated Development Environments (IDEs) are not used often except for Matlab and R.

Version Control systems are not widely used, code (and data) being shared by email amongst collaborators rather than using version controlled repositories which leads to confusion about the 'latest version' and why changes are made and by whom.

Fault Tolerance and Resilience is good in large community models, allowing them to restart after a crash. However many models are difficult to debug and understand why crashes have occurred.

(In)efficiency arises from needing to download large data files (sometimes 10's of GiB) to extract the small amount of data for focus as data is stored in flat file format rather than query-able data stores. Code reuse is poor, languages like R do not enforce object orientated styles, and the choice of language is sometimes sub optimal, e.g. R is good at statistical modelling but often used for many other things.

Compatibility -the interfaces for running models are not abstracted from the models -in depth knowledge of the model is required to configure, parameterise and run models. Data input and output is usually flat file CSV, netCDF and Excel spreadsheet format.

This phase of embedding computer scientists in the environmental modelling domain allowed us to grasp the challenges faced by environmental modellers and understand what appropriate abstractions over computing complexity might be. The main finding of this phase of work from the perspective of this project is that the code of environmental models is deeply entwined with the architecture of the conventional computing systems and the working practices of environmental scientists. There is a great desire to move the science on from this complexity, which we believe could be achieved through abstraction. Indeed, to foster greater understanding of uncertainty in models of environmental systems, models runs may need to be more efficient and run many more times in many more places with approaches such as models of everywhere [START_REF] Blair | Models of everywhere revisited: A technological perspective[END_REF]. Software engineering achieves the separation of concerns through abstraction using modularity, frameworks and defined interfaces; these practices are not often seen in day to day environmental modelling, possibly due to a lack of SE training, but mainly due to the absence of tools at a usable level of abstraction for this domain.

The offline mode of working makes sharing and collaborating difficult, we found our participants indicated closer collaboration would be desirable for science. Code is not released to the community because it is not considered robust or efficient; usually this is because of concerns about input data, misunderstood bugs or the specifics of a system such as configuration of dependencies. A software engineer can see much of this could be resolved through abstraction (e.g. by using pre-built data cleaning libraries rather than unique data cleaning code), changes to code writing style to make debugging easier, and using dependency management tools.

Opportunities for Abstraction

Computational abstraction alone will not address the problems identified in the qualitative work, social factors are important as well, but beyond the scope of this paper. However we believe abstracting from computational complexity can play a significant role at multiple levels:

1. At the model code level: Software frameworks and robust software engineering practices such as defined interfaces and modular code architectures would allow code reuse, reduce dev time and reduce the need for debugging. 2. At the infrastructure level: By removing the reliance on the desktop and HPC there is the opportunity to deploy models to new and flexible architectures provided by cloud computing. There are multiple additional benefits here of opening access to experiments for collaborative science, and access to data stores and data science tools already present in cloud offerings. 3. At the model configuration level: Complex models have interdependence across a huge volume of configurations, the interaction of which needs to be understood, and sometimes the same value needs to be specified in multiple places. 4. At the model parameterisation level: There is a range of methods to parameterise models, often involving multiple model runs before settling on a suitable parameter set.

The first opportunity is perhaps symptomatic of a lack of core software engineering skills within the community writing software models. It cannot be addressed without significant re-engineering of models; our participants pledged to learn about best practice but software frameworks for developing and running models would support this.

The second is one area in which we chose to focus our efforts by deploying models to the cloud to explore the opportunity, and look for tools at the right level of abstraction to enable an environmental modeller to configure and deploy a suitable infrastructure for their experiment.

The third is another area we looked at, and felt their was an opportunity for techniques such as Model Driven Engineering (MDE) to build a software model of the complexities in configuration of environmental models.

The fourth we have considered and propose that techniques from AI and machine learning can reduce the parameter space and help the scientist to select the most suitable parameters for their experiment, but only if models exist in an on demand, flexible cloud infrastructure supported by fit for purpose data stores and powerful data science machines.

Experimental Phase

This phase describes approaches to abstracting over i) the complexity in the configuration and deployment of an environmental model (abstraction 3 above) and ii) the complexity in cloud deployment of the model (abstraction 2 above). We created a tool for the configuration experiments, and a cloud deployment of the Weather Research and Forecasting (WRF) model. Here, we raise the level of abstraction through reducing the complexity of configuration and deployment, leveraging the on demand scale-ability of cloud architecture.

WRF

The Weather Research and Forecasting (WRF) model [START_REF] Skamarock | A description of the advanced research wrf version 3[END_REF] is a large communitybased endeavour (around 40,000 users), supported by the National Center for Atmospheric Research (NCAR). The model is primarily used for atmospheric research and forecasting across a wide range of scales (thousands of kilometres to meters). The diverse range of extensively validated science WRF can simulate includes regional climate, air quality, urban heat islands, hurricanes, forest fires, and flooding through coupling with hydrological models.

WRF is chosen as a case study here for the following reasons: i) WRF installation is viewed as a barrier to use; ii) cloud resources will enable WRF users to conduct simulations beyond current capability [START_REF] Powers | The weather research and forecasting model: Overview, system efforts, and future directions[END_REF]; iii) WRFs open-source nature and portability; and iv) benefits will impact WRFs large community user base.

Configuration and Collaboration

Model Driven Engineering (MDE) is an approach to managing complexity in software systems and to capture domain knowledge effectively [8]. Domain knowledge is captured in a software model of the system, this model is configured using Domain Specific Languages (DSLs) that relate to the application domain and potentially the underlying platform features. Code is then generated by the model using transformation approaches to configure and deploy the system being modelled. This approach has been applied successfully in a variety of areas [8] including in industry settings [START_REF] Hutchinson | Empirical assessment of mde in industry[END_REF].

In this work we investigated using this approach to develop DSLs to allow scientists to describe an experiment, with the underlying software model managing the generation of code to configure the environmental model appropriately, deploying the model to appropriate flexible cloud infrastructure and returning results. In exploring the tools available, we found they are not able to support such a vision at the current state of readiness. There are so many complexities and flexibility required in configuring models like WRF for the many different uses, that to hard code a set of rules into a DSL was not an approach that was likely to be successful.

Instead we decided to use a general purpose language to manage the configuration and deployment, and proposed embedding a future 'learning' approach that would match experimental configurations to infrastructures, and would be able to recommend an appropriate architecture for an experimental configuration. In our qualitative phase we found our participants were familiar with languages such as R and Python, often used to process data and produce visualisations.

In taking this MDE-lite approach, we produced a Python object based model of a WRF experiment configuration, that allows configuration using standard Python constructs. The Python package f90nml4 allows the object to generate Fortran 90 namelist files that are used to configure WRF, these are autouploaded to the WRF instance for deployment. This provides a layer of abstraction over the skills required to login to an instance, navigate the file system, and edit the Fortran namelist file without introducing errors. No configuration dependencies are managed in this first iteration, but this can be added to the Python model in future.

Code notebooks such as Jupyter5 are now widely used in the data science community to collaborate and annotate data analysis. They are online environments, often hosted in the cloud (but not necessarily publicly available) that allow code to be edited and run with multiple collaborators. They allow mixed mode documentation, with in line code and output visualisation from many modern programming languages, including Python and R. They are able to pull in data from outside sources and write out data to attached cloud native stores or via API.

For our purposes a Jupyter code notebook provides an ideal space for environmental scientists to control modelling experiments. Whilst it would not be possible to run a complex model such as WRF in the notebook, it is possible to configure a run experiments via system APIs. The model can be configured us-ing our Python object based configuration tool, and the experiment run outside of the notebook (either on the same machine or an external 'cloud burst') with output data returned to the notebook for collaborative analysis. The notebook allows the experiment to be documented (supporting scientific reproducibility) and shared, extending the reach of the experiment.

Cloud Deployment of WRF

We built a scripted cloud deployment of WRF, and evaluated usability, performance and cost metrics. This installation script dealt with dependencies and the configuration and compilation required to run WRF on the Microsoft Azure cloud platform. The script is automated as far as possible, however in a number of places unavoidable user input is requested from the WRF installer.

The architecture of the WRF model itself is complex, described by [START_REF] Skamarock | A description of the advanced research wrf version 3[END_REF]. Our standard cloud configuration consisted of a Message Passing Interface (MPI) supported cluster of 9 standard compute nodes from the Microsoft Azure Dsv3series each having a 3.2 GHz Intel Xeon E5-2673 v4 (Broadwell) processor. One node is a master node taking care of all the compilation and providing a means of sharing the storage and computation with all the nodes.

We used a predefined image of Ubuntu Server 16.06 LTS for each of the cluster machines and each node has 8 processors with 32 GiB RAM and temporary storage of 64 GiB that is considered a secondary storage for each compute node. We used the GNU Fortran and GCC compilers. The cluster provides primary storage of 100 GiB shared amongst nodes via the Network File System (NFS). The shared location contains all the simulation related input/output data and files required for WRF configuration as well as compilation. All the cluster nodes and storage are deployed in Western Europe under one secure virtual network and have friction-less access to enable data sharing and execution of MPI jobs.

An expert user group of 6 regular WRF users agreed that our automated WRF deployment successfully abstracted over the major hurdle of initially installing the model. They helped to develop the use cases that we used in developing the work: A) removing barriers to entry for new users, allowing them to immediately run experiments; B) users wanting to run the model in a standard way to feed results into other models; and C) power users wishing to deploy massively in parallel and without waiting for institutional HPC queue times [9].

Mechanisms for Cloud Computing Configuration

To build on this scripted installation, we investigated key mechanisms to abstract over the complexities of configuring compute architecture, by exploring different modes of cloud deployment. We wanted to retain as much flexibility in the deployment whilst retaining usability for our use cases.

Portal configuration. The scripted installation described above requires the manual configuration of the cluster from the Azure web portal, or defined in code using their Infrastructure As Code (IAC) offering, which are general purpose interfaces and require a deep understanding of the desired infrastructure.

Because of this knowledge required, it is perhaps not a suitable interface for our use cases except perhaps (C) the power user who may have an understanding of the relative merits of different cluster configurations and be able to fine tune to their experiment manually. Since the WRF installation is not fully automated, it still requires user interaction.

Containerisation is a mechanism whereby the software environment including dependencies for a particular application is defined in code. These containers are infrastructure agnostic, so can be deployed to any suitable provider. However the MPI architecture of the WRF model makes it unsuitable for containerisation -MPI is the mechanism by which messages are transferred between nodes, and we found little support for this in existing technologies. This situation is changing, with a number of providers beginning to offer support for MPI 6 .

LibCloud is a Python library for interacting with many cloud providers, abstracting from their specific IAC offering. This allows a provider agnostic configuration of a cloud infrastructure from a notebook, however it is again general purpose and does not encompass every offering from every provider. It is designed for the computer scientist, and does not really abstract from the complexity of deploying a cloud computing system, just from the differences that individual cloud providers might have for their standard machines. It also does not get around the user interaction required for installing WRF so perhaps only suitable for use case (C).

Infrastructure As A Service (IAAS) allows the replication of predefined and configured machine images, allowing standard node with WRF installed to be instantiated on demand. In Azure this can be in their machine image library for anyone to use, and other providers have similar facilities.

In summary, the IAAS approach was selected as being most suitable for our WRF cloud deployment as it means deployment is instantaneous with no user interaction required in getting a system running WRF running, so good for use cases (A) and (B). However this reduces flexibility for (C) in terms of virtual machine specification as it is tied to a specific standard machine type offered by our cloud provider. Without re-engineering the WRF installer, or WRF itself it was not possible at this time to create a cloud-native, provider agnostic system without user input in the installation.

Experimental System

The IAAS approach whilst reducing ultimate flexibility in infrastructure, allowed us to explore how WRF might be configured and deployed from a notebook environment. We built a demonstrator using the Azure SDK for Python to configure and deploy a WRF cluster from within a Jupyter notebook running on an instance of Azure data science machine and so far a number of test experiments have been run. Ongoing Work: The next stage in developing this system is to integrate the WRF namelist configuration tool to the same notebook, and return results as an attached data store to the machine. The same Jupyter notebook can then be used to sort output data and prepare data visualizations. In this way the whole experiment can be orchestrated through a single collaborative notebook interface, which is version controlled and can be archived easily. Figure 1 shows this visually.

Reflections

The key successes of this project were in using the WRF case study to navigate available technologies to create useful abstractions not only to enable the scientist to focus on science, but in support of Easterbrook's vision of the next generation of computer supported collaborative science [START_REF] Easterbrook | Climate change: A grand software challenge[END_REF]. A vision where models form part of a service fabric of technologies that can be recombined and run an unlimited number of times to explore the specifics of place and rationalise about uncertainty. The work underpins Blair's ten challenges to environmental data science community [START_REF] Blair | Models of everywhere revisited: A technological perspective[END_REF] and leverages the elasticity and flexibility of cloud computing to provide on demand, scale-able computing.

Adapting WRF to a cloud computing infrastructure and attempting to leverage the latest advances was bound to face some challenges. WRF has been developed over decades and consists of about half a million lines of Fortran code. Fortran is especially good at large scale numerical simulation, however it is not a typically supported language in the modern software industry and hence cloud computing platforms. The MPI libraries for parallelising code were written for Fortran and C++, and support for these is low in cloud native technologies such as containerisation, although this is changing as more cloud providers consider supporting scientific applications. Containerisation allows complete abstraction from underlying compute architecture, and would make deployment on new generations of machines possible. As such it was not possible during the project to leverage these cutting-edge technologies to run WRF, and we had to step back to an IAAS approach of using machine images to explore our ideas.

Installing and compiling WRF on new machines is a chore, and requires an advanced computing skill set. Our demonstrator of an auto-install of WRF to the cloud was an eye-opener for many of our participant scientists. The ability to simply spin up a virtual machine pre-installed with WRF is beginning to be used in teaching labs, instead of requiring students to go through the long error-prone process. This was a successful exercise in abstraction from complexity.

We found the tools developed by the MDE research community to be unable to capture the complexity of WRF deployment without reducing the flexibility required by the wide range of use cases -they required hard coding a set of configurations. However we designed a Python tool to allow the scientist to retain full control of the experimental configuration, without risking errors in the generation of a Fortran namelist configuration file, that in future could manage configuration dependencies and abstract from configuration complexities.

We added further abstraction from the cloud provider's interface by bringing these tools together into a now becoming familiar Jupyter notebook, writing code to deploy a WRF system from within the notebook, which can then also be used to analyse results in a collaborative environment that describes the whole experiment. An obstacle we are facing in completing this vision is the poor support of WRF's netCDF output files, which are designed for flat file stores. Other groups are working on this, for example Unidata at UCAR7 .

Conclusion

In this project we aimed to find out if the principles and approaches exist to leverage cloud computing at a usable level of abstraction in environmental modelling, and we found we were able to make usable abstractions from complexity in deployment of the WRF model to cloud, and the configuration of WRF. We found a notebook based approach supports new practices in collaborative and open science through the ability to describe and run an experiment in a single notebook document that can be shared and examined by others. Our findings can be generalised to other complex models, specifically MPI based models but learning is applicable to many others.

The abstractions proposed and explored through this project mean that WRF could form part of a cloud service fabric consisting of many models, data stores and data science services that could run as many times and in as many combinations as desired. Add in cloud based machine learning facilities and a system could propose its own infrastructures based on the experiment to be run, as well as help to reduce the workload of scientists in terms of reducing parameter spaces when parameterising models. These abstractions also have potential to be significant when operationalising code which needs to run continuously and effectively, allowing code to be run on diverse and shared architectures.

Future work will involve 'closing the loop' by returning outputs from WRF to the notebook by connecting output data to a cloud native data store, thus enabling the analysis of output within the same notebook, but also powerfully interfacing output to other models as part of the aforementioned service fabric. This work primarily focused on technical factors, but equally there are many social factors at play in defining the software and hardware architectures in use in environmental science. Future work could include exploring the barriers these present to the take up of new computing technologies.

Fig. 1 .

 1 Fig. 1. WRF-Cloud System Architecture

https://www.ensembleprojects.org/

https://pypi.org/project/f90nml/

https://jupyter.org/

https://www.stackhpc.com/the-state-of-hpc-containers.html

https://www.unidata.ucar.edu/blogs/news/entry/netcdf-and-native-cloud-storage

Acknowledgements

Thanks to the wider Ensemble team (https://www.ensembleprojects.org/) for their support. This work is supported by "Models in the Cloud: Generative Software Frameworks to Support the Execution of Environmental Models in the Cloud" EPSRC: EP/N027736/1 and a Microsoft Azure AI for Earth grant.


M Knapen

Rob M Lokers

Leonardo Candela

Sander Janssen

Rob Knapen

AGINFRA PLUS: Running Crop Simulations on the D4Science Distributed e-Infrastructure

Keywords: Distributed Computing, e-Infrastructure, Virtual Research Environment, Crop Simulation Model, WOFOST

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

In [START_REF] Lokers | Analysis of Big Data Technologies for Use in Agro-Environmental Science[END_REF], it is argued that in Agronomy the major Big Data challenges are with variety and veracity, and that tackling the issues with volume and velocity of the data is more generic and to be solved with common industrial Information Technology solutions. Yet, such solutions still have to be created, adapted to, and tried in the field of Agro Informatics. Besides, agronomists and researchers of related domains have to be introduced to them and start making use of these new technologies. Meanwhile the range of options is getting broader with the rise of container technologies such as Docker (https://www.docker.com), Kubernetes (https://kubernetes.io), and Singularity (https://sylabs.io), and Cloud computing. Platforms familiar to computer scientists and software engineers such as Google Cloud Platform (https://cloud.google.com), Amazon Web Services (https://aws.amazon.com), Microsoft Azure (https://azure.microsoft.com/), are extending their reach by adding tools interesting to other researchers, targeting various domains. While other vendors make more basic compute and storage resources still easier accessible at low prices. However, the high-level platforms have the risk of vendor lock-in and unknowingly handing over of data, while the low-level solutions (including direct use of High Performance Computing) require IT proficiency to make proper use of them.

The European Open Science Cloud (EOSC) is an initiative that addresses such issues and promotes a more open, federated, research infrastructure that gives access to compute and storage resources, and support the ideas of Open Science and FAIR data sharing (https://www.force11.org/group/fairgroup/fairprinciples). It is in line with those thoughts that the D4Science platform [START_REF] Assante | Enacting Open Science By D4science[END_REF] is being developed.

In the European H2020 AGINFRA PLUS project [START_REF] Assante | Realising a Science Gateway for the Agri-Food: The Aginfra Plus Experience[END_REF] the D4Science technology is used to create a number of Virtual Research Environments (VRE) specifically targeting research communities in the food and agriculture domain, with the goal to accelerate user-driven innovations of the existing e-Infrastructure. A number of key use cases were selected based on typical work and requirements within each community. This paper will further focus on one of those use cases, being the ability to run crop simulation models at scale on a compute cluster hosted by D4Science, allowing for horizontal scalability to handle large workloads such as running crop simulations for all the crop parcels in the Netherlands.

Methods

D4Science Cloud Computing e-Infrastructure

D4Science [5] promotes open science through the operation of an innovative data and compute infrastructure service, build using the gCube framework [8]. The gCube technology allows easy construction and development of VREs. A VRE in general is a webbased working environment tailored to support the needs of their designated communities, each working on specific research questions. The VRE offers users with domainspecific facilities, e.g. certain computational algorithms, and typically needed datasets, integrated with more common services that support collaboration and cooperation amongst users, e.g. a shared Workspace to store and organize versions of research artifacts, a social networking area, a data analytics platform, and a catalogue-based publishing platform (see [START_REF] Assante | Enacting Open Science By D4science[END_REF]).

The data analytics platform (gCube DataMiner, see Fig. 1) is of special interest, since this is the service used to run the crop simulation model. There are two ways to interact with it, one is at a low level where algorithms have to be written using specific Java interfaces that are set up to do map-reduce types of processing [START_REF] Dean | Mapreduce: Simplified Data Processing on Large Clusters[END_REF]. Such algorithms gain access to the 'Worker' cluster, a set of thin nodes ('slow' CPU and limited memory) specifically targeted at this type of data processing. The other option is to write algorithms that can be deployed using the DataMiner importer, which wraps it and deploys it as a Web Processing Service (WPS, a standard from the Open Geospatial Consortium (OGC)) on the 'Master' cluster. These are fat nodes (fast CPU and lots of memory) in the system that handle the queuing, load balancing, and execution of WPS requests.

For both approaches DataMiner automatically provides recording of provenance information using the W3C Prov-O standard [START_REF] Prov | The PROV Ontology specification[END_REF], so that any run of an algorithm can easily be repeated.

2.2

The Web Processing Service standard

The Web Processing Service interface standard is one of the standards of the Open Geospatial Consortium. It is lightweight and XML-based, and easy to use to publish, discover, and execute processes as a service. Processes (e.g. calculations, algorithms, simulation models) can be both simple and complex. Input data for the processes can be included in the HTTP POST request to the WPS service or be made accessible via HTTP URLs. WPS services can also be integrated into workflows by workflow management systems such as Galaxy (galaxyproject.org), Knime (www.knime.com), and Apache Taverna (taverna.apache.org). WPS has been designed to work with spatially referenced data, but it can be used with any kind of data. It also was not designed to work with distributed computing systems. The main operations defined by the interface are: GetCapabilities, Describe-Process, and Execute. Allowing to get the list of available services on a system, get a description of the required inputs and the produced outputs, and to execute the process. These do not allow for very dynamic use of a compute cluster. E.g. all outputs of a process have to be specified at process description time, and it is hard to track how busy the system is and how many new requests it would be able to process quickly. In D4Science some of these issues have been addressed.

WOFOST-WISS

WOFOST-WISS is a new implementation of the well-known and widely used WOFOST crop growth model [START_REF] De Wit | 25 Years of the Wofost Cropping Systems Model[END_REF]. It is built on top of the Wageningen Integrated Systems Simulator (WISS), a Java-based, lightweight simulation model framework targeting the agro-economical modelling domain. WOFOST-WISS ensures high numerical performance and robustness, both required for large scale operational application of crop models. It is a mechanistic, dynamic model that explains daily crop growth on the basis of the underlying processes, such as photosynthesis, respiration, and how these processes are influenced by environmental conditions. Since it is written in Java (openjdk.java.net), WOFOST-WISS needs a Java Runtime Environment (JRE) to run it. This makes it easy portable to any computing platform for which a JRE is available (almost all). The code is executed by a Java Virtual Machine (JVM), which allows for runtime code optimizations, as well as remote debugging, and real-time performance monitoring. It also makes the model easy to integrate into the Java eco-systems and use it from other JVM-based programming languages.

An essential design principle of WOFOST-WISS is that the model components are stateless, and all state is securely kept into an object called SimXChange. All input data is provided to the model in a ParXChange object. The model itself has no sideeffects, hence, from a Function Programming (FP) perspective it can be regarded as a pure function. This makes it well suited for use in distributed computing.

AgroDataCube

Given that for many applications in the agri-food domain the same basic, large, datasets are always needed, at Wageningen Environmental Research these are being made available as harmonized open data in the AgroDataCube [START_REF] Agrodatacube | A Big Open Data collection for Agri-Food Applications[END_REF], including information about crop fields, crop growth (as indexes derived from remote sensing data), observed weather, terrain height, and soil conditions. Using an access token all data can be retrieved by HTTP GET and POST requests in GeoJSON format (geojson.org) from a REST (https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm) based Application Programming Interface (API). It is also possible to retrieve sub-field gridded data such as the detailed 10m x 10m Normalized Difference Vegetation Index (NDVI).

For the Netherlands the AgroDataCube contains in principle all the data for multiple years needed to run crop simulations at the crop field level. It does however require some further pre-processing of the data, including the mapping of crop codes into the needed crop specific parameters, completing available weather data into a full daily time series of all required weather variables, and calculation of essential soil characteristics from the available soil information.

Functional Programming and the Actor Framework

The technologies available for building applications continue to evolve at a rapid pace. Systems such as D4Science make it possible to effectively utilize clusters of cores on individual servers and clusters of servers that work together as a single application platform. Costs for memory and disk storage have dropped, and network speeds have grown significantly. This allows for large volumes of data to be collected and processed. However, timely processing typically cannot be done any more on a single traditional computer with limited (vertical) scalability. It requires distributed computing on multiple computers that allow better (horizontal) scalability. And distributed computing is difficult to program without using new programming paradigms such as functional programming. Functional Programming (FP) is a style of writing computer programs that treats computation as the evaluation of mathematical functions and avoids changing state and mutable data, and side-effects in general. Programming is done by writing expressions or declarations, instead of statements. FP stems from lambda calculus, a formal system developed in the 1930s to investigate computability, amongst others. Lisp (lisplang.org), Clojure (clojure.org), Erlang (erlang.org), Haskell (haskell.org), and Scala (scala-lang.org) are some well-known functional programming languages. The latter is a JVM based programming language, hence offers good integration with Java.

Because it avoids side-effects FP fits well for writing programs that are run in distributed computing environments. It is much easier to reason about the execution of (pure) functions, than about multiple computers and threads processing the objects of a program written in an Object-Oriented programming language. Combining FP with a message passing architecture such as the Akka framework [1] allows even further abstraction and isolation, away from the low-level wiring and very technical programming techniques otherwise needed to make efficient use of computational resources.

Results

As described, one of the use cases in the AGINFRA PLUS project was to pilot running crop simulations for the crop fields of the Netherlands, using the D4Science distributed infrastructure. Within the project a small cluster was available for testing purposes, with 6 fat nodes, and a slightly larger cluster for operational deployments. A number of VREs have been created, among which one for the Agro-Environmental modelling use cases. This VRE contained the generic components such as the social messaging, the shared Workspace, and DataMiner, as well as more domain specific tools such as access to SoilGrids (soilgrids.org), Jupyter notebooks (jupyter.org), and customized visualizations.

It would certainly have been possible to use the map-reduce approach of DataMiner for running many crop simulations on the system. However, for the long term, this appeared to be a less flexible solution, binding the software directly to the D4Science platform. Being a research organization, future projects might involve running crop simulations in other environments than D4Science, and not in a strictly map-reduce way, making it more appealing to use the WPS based processing of DataMiner.

DataMiner runs the WPS processes on available fat nodes in the cluster. For all incoming WPS requests of the same process it handles the queuing and load balancing. Running the crop simulation models efficiently therefore required implementing two types of algorithms for DataMiner. One that can run as many crop simulations simultaneously per fat node once it gets assigned to one, and another type of algorithm that can divide a total workload of crop simulations to be done for a set of selected crop fields, e.g. all maize fields in the Dutch province Limburg (approx. 15.000 for a single year), into smaller batches, start the crop simulation jobs on DataMiner for each batch, and collect and process all results. In essence the two types of algorithms perform the mapreduce processing via the WPS interfaces.

Both algorithms are implemented using the Akka actor framework, so that they make the best possible use of the machines they get assigned to (within limitations set by the system). The 'worker' algorithm (see Fig. 2) contains WOFOST-WISS and accesses the AgroDataCube in an optimized way, running as many crop simulations in parallel as possible. Retrieving data from the AgroDataCube is a clear bottleneck that technically should be solved. For now, data retrieved is cached and reused when possible, so after warmup the algorithm optimizes for running similar crop simulations (e.g. in the same region with little variation in weather and soils). The output of the algorithm is either a CSV file with daily states of all calculated variables in the simulation, in case a single simulation is run, or a JSON file with a summary of the main variables for each simulation run. This keeps the total output manageable and should the detailed data for a specific simulation be needed it can easily be produced.

The 'scheduler' algorithm (Fig. 3) retrieves all IDs of crop fields from the Agro-DataCube that match criteria specified by the user of the system, i.e. crop to process, year to simulate, spatial region the crop fields should be in. And some more technical parameters as well such as the preferred batch size, the maximum number of batches to process, and a timeout value for the total processing. These might be hidden after the system has been tuned and runs in a more operational state. Given the list of IDs the algorithm creates the batches, and for each one sends a WPS request to DataMiner to run the crop simulations algorithm. The scheduler then tracks the status of all algorithm executions in progress. When they all have finished all produced output is analyzed by the algorithm and an overall summary is created and made available in HTML format, giving the research a quick overview of the state of all crop simulations performed. Although improvements still need to be made and many optimizations are possible, it is clear that technically the solution works. Even in this stage and with a limited system of 6 fat nodes in the cluster, round-trip processing of crop simulations runs at about 50 simulations per second. Large workloads can easily be handled by (temporarily) adding more nodes to the cluster. And via WPS both types of algorithms can be called from other applications than the VRE. E.g. a custom web user interface, a mobile application, another WPS client such as QGIS (qgis.org), and so on. To showcase this within the AGINFRA PLUS project a dashboard visualization (see Fig. 4) is also being build, bringing together the input data from AgroDataCube and the output data from crop simulations.

Conclusions

The described implementation allows running crop simulations in a standardized environment, with horizontal scalability. For larger workloads required computational time can be decreased by adding more computers to the cluster. Besides, the processes are available via the standardized WPS interfaces and can thus be used in many applications. Or within VREs, tailored to specific user communities and usages. Further finetuning of the system is of course needed to turn the current proof-of-concept into an operational solution. With similar data as the contents of the AgroDataCube and proper calibration of the WOFOST-WISS model the same system can also be used for other regions. Parts of the system will also be used in the Cybele EU H2020 project [4], which looks at the convergence of Cloud Computing services, such as D4Science, and High Performance Computing (HPC). This should make available even more compute power for running crop simulations, opening the door for doing even more advanced studies requiring ever increasing numbers of crop field level simulations to be run, e.g. yield forecasting at the field level based on detailed weather forecasts.

Naturally the real value in the end is to be able to better advice farmers and achieve a more efficient and future-proof agri-food system.

Fig. 1 .

 1 Fig. 1. Architecture of the gCube DataMiner system. With on the right side the two compute clusters and the distributed storage system, and on the left side several e-Infrastructure resources which are indexed on the D4Science Information System (IS).

Fig. 2 .

 2 Fig. 2. Actors in the Worker algorithm. The user provides a description of a study in JSON format. The Librarians (n=5) smartly retrieves all required data from AgroDataCube and other data sources. Multiple Researchers (n=500) run WOFOST crop simulations in parallel via the Simulator. A single Reporter (n=1) collects and summarizes all results.

Fig. 3 .

 3 Fig. 3. Actors in the Scheduler algorithm. The user provides a description of a study in JSON format. A number of JobStarters send WPS requests to DataMiner to run the worker algorithm.Once started their execution progress is tracked by JobMonitors. Processing results (all the output files that DataMiner writes to the storage system) are then analyzed by LogFileProcessors and SummaryProcessors, that provide input to a Reporter that creates an overview.

Fig. 4 .

 4 Fig. 4. The Crop Modelling Dashboard. It displays a map with crop fields (top left)from which a parcel can be selected. Data from AgroDataCube is then retrieved and field character-istics such as the crop, previous crops, soil types, etc. are displayed top ri right. In the bottom left weather and NDVI timeseries can be displayed, while the view on the bottom right side allows running a crop simulation for the selected field and visualization of the outputs, such as leaf area index, above ground biomass, soil moisture, harvest index, etc.

Acknowledgment

This work has received funding from the European Union's Horizon 2020 research and innovation programme under the AGINFRA PLUS project (grant agreement No 731001).


Herman Mollenhorst
email: erwin.mollenhorst@wur.nlyamine.bouzembrak@wur.nl

Michel De Haan
email: michel.dehaan@wur.nl

Hans J P Marvin
email: hans.marvin@wur.nlroel.veerkamp@wur.nlclaudia.kamphuis@wur.nl

Roel F Veerkamp

Claudia Kamp- Huis

Predicting Nitrogen Excretion of Dairy Cattle with Machine Learning

Keywords: Bayesian Networks, Boosted regression trees, nitrogen excretion, dairy cows

Several tools were developed during the past decades to support farmers in nutrient management and to meet legal requirements such as the farm specific excretion tool. This tool is used by dairy farmers to estimate the farm specific nitrogen (N) excretion of their animals, which is calculated from farm specific data and some normative values. Some variables, like intake of grazed grass or roughage, are hard to measure. A data driven approach could help finding structures in data, and identifying key factors determining N excretion. The aim of this study was to benchmark machine learning methods such as Bayesian Network (BN) and boosted regression trees (BRT) in predicting N excretion, and to assess how sensitive both approaches are on the absence of hard-to-measure input variables. Data were collected from 25 Dutch dairy farms. In the period 2006-2018, detailed recordings of N intake and output were made during 6-10 weeks distributed over each year. Variables included milk production, feed intake and their composition. Calculated N excretion was categorized as low, medium, and high, with limits of 300 and 450 g/day/animal. Accuracy of prediction of the farm specific N excretion, and distinguishing the low and high cases from the medium ones, was slightly better with BRT than with BN. Leaving out information on intake during grazing did not negatively influence validation performance of both models, which opens opportunities to diminish data collection efforts on this aspect. Further analyses are required to confirm these results, such as cross-validation.

Introduction

Agricultural policy, at least in the Netherlands, is in a transition. Since the second world war, the focus has been on producing as much food as possible in a highly efficient way. Favorable conditions combined with high-level knowledge and expertise, have put Dutch agriculture in a world leading position. However, drawbacks of this policy, become clear in regards to environmental impacts of agriculture and mineral surpluses due to factors such as high imports of animal feed. In 2018, the Dutch minister of agriculture presented a new policy [START_REF] Minlnv | natuur en voedsel: waardevol en verbonden -Nederland als koploper in kringlooplandbouw[END_REF], positioning Dutch agriculture as the front runner in circular agriculture, with minimal losses and minimal inputs of artificial inputs (fertilizer, pesticides) and scarce resources (e.g., phosphate, potassium).

Wageningen UR researchers have developed several tools and collected several datasets during the past decades to support farmers in nutrient management to meet the legal requirements. Examples of tools are the Annual Nutrient Cycling Assessment [START_REF] Aarts | Quantifying the environmental performance of individual dairy farms -the Annual Nutrient Cycling Assessment (ANCA)[END_REF], which includes the whole nutrient cycle, or the Excretion Assessment [START_REF] Rvo | Handreiking bedrijfsspecifieke excretie melkvee[END_REF], focusing on nutrients excreted in the manure by the animals. The Excretion Assessment is a tool that farmers can use to proof the actual excretion of their animals, in comparison to the excretion standards. The government accepts the results of this tool in order to calculate the manure export of a farm. This tool is based on farm specific data and some normative values modelled in a mechanistic way. However, some variables including intake of grazed grass or roughage, are hard to measure directly, and therefore are measured indirectly or estimated. A data driven approach could help finding structures in data, and identifying key factors determining N excretion. This could lead to possible exclusion of estimates and normative values for hard-to-measure input variables, or to focus on more detailed assessments, on field, group, or individual animal level. This would make the process of monitoring nutrient management more efficient and could be used to elaborate tools with prediction of the effects of possible management interventions. Better monitoring and prediction could support farmers and advisors, in making informed decisions on nutrient management, such as changes in feed ration or manure application.

An earlier study has proven that machine learning can help to predict grass and crop yields at field level, to support decisions on manure application rates [START_REF] Mollenhorst | Field and crop specific manure application on a dairy farm based on historical data and machine learning[END_REF]. However, it considered only one part of the nutrient circle, but decisions have to be made in more parts of the circle, such as animal nutrition and management. Therefore, in this study we will focus on the excretion of N from the animals, predicted from feed intake and production variables. To make models more explainable, a Bayesian network (BN) approach was applied, which delivers a graphical model that presents probabilistic relationships among a set of variables [START_REF] Marvin | Application of Bayesian networks for hazard ranking of nanomaterials to support human health risk assessment[END_REF][START_REF] Cheng | Learning Bayesian networks from data: An information-theory based approach[END_REF][START_REF] Bouzembrak | Application of Bayesian Networks in the development of herbs and spices sampling monitoring system[END_REF][START_REF] Bouzembrak | Impact of drivers of change, including climatic factors, on the occurrence of chemical food safety hazards in fruits and vegetables: A Bayesian Network approach[END_REF]. Next to BN, a less explainable model, boosted regression trees (BRT), was applied, which was also used in the previous study [START_REF] Mollenhorst | Field and crop specific manure application on a dairy farm based on historical data and machine learning[END_REF].

The aim of this study was to compare BN and BRT in predicting N excretion of dairy cows, and to assess how sensitive both approaches are on the absence of hard-to-measure input variables.

2

Materials and Methods

Dataset

Data used for this study originated from the project called 'Cows and Opportunities'.

In this project, a group of 16 Dutch dairy farmers, together with researchers, searched for opportunities for sustainable and socially accepted agriculture [START_REF]Cows and Opportunities homepage[END_REF]. The farms varied in intensity, scale, soil type, and style of farming. The main aim was to implement expected environmental legislation to monitor environmental, technical, and economic effects at farm level. In this way, each of these farms are a kind of pilot farm. Data from years 2006 to 2018 were used, and originated from 25 different farms that were part of the project for different periods of time. On each farm detailed recordings were made during 6 to 10 weeks, distributed over the year. Between 14 to 16 farms were present in the dataset per year, which resulted in 108 to 151 weeks with detailed recordings per year. The final dataset contained 1,640 records.

Variables used concerned; milk production, milk composition, feed intake, and feed composition, for milk producing animals, on an average per cow basis. Milk production and milk composition were measured as part of the national milk production registration, according to ICAR regulations. Feed intake was measured through weighing of offered feed and refusals. Feed composition was determined by routine laboratory analyses of feed composition of all feeds. These variables are also used for calculating the N excretion based on formulas described by Oenema et al. [START_REF] Oenema | Toetsing van de Kringloopwijzer -Gemeten en voorspelde stikstofen fosfaatproducties van mest en gewas[END_REF], which were used as 'observed' values to be predicted with machine learning models. N excretion was categorized as low, medium, or high with limits of 300 and 450 g N per day per animal.

Machine Learning Models

In this study, two machine learning techniques were applied and benchmarked, namely BN and BRT. Decision tree induction is one of the basic machine learning techniques, is robust against irrelevant input variables, and is able to handle missing values [START_REF] Friedman | Greedy Function Approximation: A Gradient Boosting Machine[END_REF][START_REF] Witten | Data mining : practical machine learning tools and techniques[END_REF]. To alleviate the main disadvantage of decision trees, namely its inaccuracy in prediction, we used the iterative method called boosting. Generalized boosted regression using the Gradient Boosting Machine (h2o.gbm function (h2o version 3.20.0.2)), was applied in this study. BRT, lacked the possibility to clearly show relationships between variables in the model and, subsequently some interpretability. Therefore, BN was applied, which is a class of probabilistic model originating from Bayesian statistics and decision theory combined with graph theory. BN has the ability to integrate different data sources and types such as expert knowledge, measurement data, and feedback experience via Bayes theorem [START_REF] Bouzembrak | Application of Bayesian Networks in the development of herbs and spices sampling monitoring system[END_REF][START_REF] Bouzembrak | Impact of drivers of change, including climatic factors, on the occurrence of chemical food safety hazards in fruits and vegetables: A Bayesian Network approach[END_REF]. Hugin software (version 8.7), was used to develop the BN model. Both models were applied on the same training and validation datasets.

The validation dataset was a 20% random sample from the original dataset, and the remaining 80% was used for training. Model performance was evaluated by reporting confusion matrices and accuracy indicators. Accuracy was calculated in the conventional method, as well as for the low and high categories only (LHacc). Furthermore, a false alert rate (FAR) was calculated, representing the number of false alerts where an actual low case was predicted to be high or vice versa, over the total number of low and high predictions. All data processing and analyses, except for the BN development, were performed in RStudio (version 1.1.463 running R version 3.6.1).

Results and discussion

Categorization of records concerning N excretion was rather imbalanced with over 88 % of cases being medium. Randomization of records over training and validation sets, resulted in a prevalence of 89 % medium in the training set and just 85% in the validation set. Especially notable was records categorized high, were over-represented in the validation dataset (Table 1). Performance of BRT models was better than for BN models, with an overall accuracy for BRT models, just above the prevalence of medium in the validation dataset. The accuracy of predicting for high and low was 8.3 % and 10.4 %, for BN and BRT respectively (Table 2), both with 0 % false alerts for low and high. For both models, this means that they have difficulty distinguishing the low and high cases from the medium ones, and predicting some of these cases corresponds with classifying slightly more (BN) or fewer (BRT) true medium cases incorrectly. When the first hard-to-measure variable, dry matter (DM) intake from grazing, was excluded from the validation set, performance improved slightly for BRT and stayed equal for BN, which means that there was no additional value in this variable for the model (Table 2). Additionally leaving out DM intake from silages and hay, decreased the performance of BRT compared to leaving out only DM intake from grazing, but improved the overall accuracy of the BN model. However, for both BRT and BN, the accuracy of predicting high and low decreased, which means that models tend to predict all records as medium (overall accuracy approaching prevalence of medium in validation dataset of 85.4%, see Table 1), when information on DM intake is left out. Table 2. Overall accuracy (Accur.), and accuracy for low and high categories (LHacc) of Bayesian Networks (BN) and Boosted Regression Tree (BRT) models, for predicting N excretion with all variables present in the validation dataset or when excluding variables; dry matter intake (DMint) information of grazing, (grass and maize) silages, and hay. Protein content of grazed grass consistently appeared to be the most important variable in all models, often closely followed by variables concerning DM intake of grazed grass, grass or maize silage or their protein content, and milk production. For BRT models, only variable importance could be reported, whereas for BN models also relationships could be made visible (Fig. 1). A remarkable point that should be investigated further is that the DM intake from mineral supplements, which constitutes a minor part of the diet, was rather important in the BN model, with many connections to other variables. Additional analyses were performed to check the consistency of the results. We checked with BRT the effect of leaving out DM intake from both the training and validation dataset. When leaving out DM intake from grazing, this resulted in an increased overall accuracy (87.8%), as well as an increased accuracy for the low and high classes (18.8%), probably indicating that data on DM intake from grazing were noisy data and hampered the development of the prediction model. When also DM intake from grass and maize silage were excluded, accuracy dropped below that of the model containing all variables.

Results in this paper were based on a single split of the dataset and only two machine learning techniques were applied, with their default settings. These results, therefore, need to be confirmed with additional analyses, for example by applying cross-validation or different machine learning techniques.

Conclusions

In this study, we developed and benchmarked two machine learning methods (BN and BRT), to predict N excretion of dairy cows. Based on the initial evaluation of models for predicting N excretion from dairy cattle, BRT performed slightly better than BN, both with respect to overall accuracy as well as distinguishing high and low cases from medium ones. We conclude that both BN and BRT have difficulty distinguishing the low and high cases from the medium ones. Further analyses of the relationship between variables in the model, is better facilitated by the graphical representation of the model for BN, whereas for BRT only variable importance is available. Leaving out information on DM intake of grazed grass did not negatively influence the validation performance of both models, which opens opportunities to diminish data collection efforts on this aspect.

Fig. 1 .

 1 Fig. 1. Graphical representation of Bayesian Network model predicting N excretion (vBrtEx-crNc…). (vDMInt = DM intake; vRE = protein content; DMcont = dry matter content; GH = grass hay; GK = grass silage; MI = minerals; MK = maize silage; KV_EA = low-protein concentrates; KV_ER = high-protein concentrates; OV = other feeds; WG = grazing; KGMELKD… = milk production level; PCTEIW = protein content of milk)

Table 1 .

 1 Number of low, medium, and high cases, and percentage of medium in training and validation datasets

	Dataset	# low	# medium	# high	Percentage
					medium
	Training	80	1171	61	89.3
	Validation	22	280	26	85.4

Acknowledgements

This research was conducted by Wageningen Livestock Research, commissioned and funded by the Ministry of Agriculture, Nature and Food Quality, within the framework of Policy Support Research theme "Data driven & High Tech" (project number KB-38-001-002 AI in animal and arable systems). Data used were part of the project 'Cows and Opportunities', funded by the Dutch ministries of Agriculture, Nature and Food Quality, and Infrastructure and Water Management, and funded by ZuivelNL (the organization of the Dutch dairy sector), and due to the efforts of the mentioned 25 dairy farmers.


Tom Hardy
email: tom.hardy@wur.nl

Marston Domingues Franceschini
email: marston.franceschini@wur.nl

Lammert Kooistra
email: lammert.kooistra@wur.nl

Marcello Novani
email: marcello.novani@wur.nl

Sebastiaan Richter
email: sebastiaan.richter@lwk.nrw.de

Automated processing of Sentinel-2 products for timeseries analysis in grassland monitoring

Keywords: Grassland monitoring, Open-source system, Sentinel-2, Cloud cover, Time-series analysis

Effective grassland management practices require a good understanding of soil and vegetation properties, that can be quantified by farmers' knowledge and remote sensing techniques. Many systems have been proposed in the past for grassland monitoring, but open-source alternatives are increasingly being preferred. In this paper, a system is proposed to process data in an open-source and automated way. This system made use of Sentinel-2 data to support grassland management at Haus Riswick in the region around Kleve, Germany, retrieved with help of a platform called Sentinelsat that was developed by ESA. Consecutive processing steps consisted of atmospheric correction, cloud masking, clipping the raster data, and calculation of vegetation indices. First results from 2018 resembled the mowing regime of the area with four growing cycles, although outliers were detected due to a lack of data caused by cloud cover. Moreover, that year's extremely dry summer was visible in the time-series pattern as well. The proposed script is a primary version of a processing chain, which is suitable to be further expanded for more advanced data pre-processing and data analysis in the future.

Introduction

Efficient grassland management is beneficial to the cultivation of crops in a sustainable and cost-effective way. It is therefore important to quantify the spatial and temporal variation of such lands, in order to get a better understanding of soil and vegetation properties such as nitrogen (N) uptake and the health status of grasslands [START_REF] Capolupo | Estimating plant traits of grasslands from UAV-acquired hyperspectral images: A comparison of statistical approaches[END_REF]. One way to quantify such features is based on farmers' experience, but this is increasingly being complemented by technologies such as remote sensing, which makes use of data derived from satellite and airborne based platforms [START_REF] Mulla | Twenty five years of remote sensing in precision agriculture: Key advances and remaining knowledge gaps[END_REF].

In recent years, many systems have been proposed and developed for the purpose of (grass)land monitoring, such as the groenmonitor (green monitor) initiated by Wageningen Environmental Research, aimed at providing the most recent vegetation information that was extracted from satellite imagery [3]. Another platform is Descartes Labs, which collects and processes open satellite data on a daily basis to prepare it for further analysis such as time-series to determine land cover change over time [START_REF]DescartesLabs DescartesLabs[END_REF]. However, these initiatives are on a commercial basis for which it is hard to gain insight in the underlying processing chains, making open-source based platforms more and more attractive. One such platform is Google Earth engine aimed at environmental data-analysis on a planetary scale, making use of open-source data from satellite platforms such as Landsat and Sentinel [START_REF] Gorelick | Google Earth Engine: Planetary-scale geospatial analysis for everyone[END_REF]. Although it is a very promising platform to perform big data analysis in remote sensing, it is required to have programming skills to some extent, for instance in JavaScript, in order to perform such analyses.

This paper describes an alternative for developing a grassland monitoring system in an automated way, aimed at providing stakeholders with up-to-date information about the condition and cultivation status of grasslands throughout a growing season in an open-source platform. Stakeholders include for instance farmers, plant breeders, or companies providing agricultural services to those farmers and breeders. Moreover, this application could also be relevant for researchers at universities or research companies, to be able to use the obtained data and information for further research in relation to crop monitoring or environmental protection. The proposed information consists of time-series graphs and output maps of vegetation indices throughout a growing season for any chosen grassland (or land with other agricultural purposes) worldwide. For this paper, this information is derived from images obtained by Sentinel-2 satellites, which are part of an online and open source platform provided by the European Space Agency (ESA).

Methodology

For the study area described in section 2.1, the proposed processing chain started with querying and harvesting Sentinel-2 satellite data (explained in sections 2.2 and 2.3). Next processing steps consisted of an atmospheric correction and extraction of these data for each individual parcel, which is further explained in sections 2.4 and 2.5. The final part of the workflow involved calculation and visualization of vegetation indices and cloud cover percentages for each given date (described in section 2.6).

Study area

The study area for this case study was Haus Riswick, located just east of the city of Kleve, Germany (Fig. 1). Haus Riswick is a teaching and research institution facilitated by the Landwirtschaftskammer (chamber of agriculture) from Nordrhein-Westfalen. The soil in that area consists of sandy, clay loam [6]. Data provided by Haus Riswick were geoJSON files representing the field boundaries from (experimental) grassland parcels located at Haus Riswick, as well as harvest data obtained during the days of mowing including its grassland yield expressed in dry matter content per hectare. A geoJSON is a data exchange format based on JavaScript Object Notation (JSON), which is able to contain a variety of geospatial attributes [START_REF] Butler | The GeoJSON Format[END_REF]. For this case study, the data from the field Lenzen Groβe Weide (the blue polygon in Fig. 1) were used.

Fig. 1. The region where Haus Riswick is located (east of the city of Kleve), including three of its grassland parcels. Only data from the field Lenzen Groβe Weide were used in this paper.

Sentinel-2 satellite data

Besides the field data from Haus Riswick, earth observation data recorded with Sentinel satellites were addressed for this application. Sentinel satellites are being developed especially for the Copernicus program, one of ESA's earth observation programs aimed at improving environmental management and monitoring the impact of climate change worldwide [8]. For this program, a number of satellites are launched, such as weather satellites and satellites for land-and ocean monitoring. This project made use of data obtained from the two Sentinel-2 satellites (2A and 2B, which both sense the earth in parallel), designed to obtain high resolution (currently 60m, 20m and 10m) optical images for land monitoring worldwide [START_REF]ESA ESA Copernicus data products[END_REF]. One way to obtain Sentinel-2 imagery would be by means of the Open Access Hub, which is part of the ESA Copernicus program. This platform has been developed with the aim to offer free and open access to data recorded by the different Sentinel satellites. Those data can be viewed and retrieved one by one via an interactive web viewer [START_REF]ESA Copernicus Scihub[END_REF], based on a number of search criteria such as sensing period, sensing mode, satellite platform, and product type. This may be useful for anyone who is occasionally interested in only one or two images for a specific location. However, when a larger number of images is needed, this process becomes laborious and timeconsuming. In that case, a platform such as the Sentinelsat Application Programming Interface (API) [START_REF]ESA Copernicus Scihub API Hub[END_REF], which is aimed at being used in a scripting interface to query and harvest Sentinel data, could be a more effective approach.

Harvesting Sentinel-2 data with Sentinelsat API

For the application in this case study, the data were accessed with help of the Sentinelsat API used in Jupyter notebooks, which is an open-source platform for interactive computing, coding and data analysis in programming languages such as R and Python [START_REF] Wille | [END_REF]. The API was used to harvest, (pre-)process and analyze all input data in an automated way. Based on parameters and input data such as a given range of dates, the field's geoJSON file providing the geographical extents, cloud cover percentage, and other (meta)data, large batches of Sentinel-2 products could be downloaded at a time. Those products consisted of images with a size of 100×100 km 2 for a region in the world matching the location of the input geoJSON file. For most areas in the world, two or three images each week are recorded by either of the two Sentinel-2 satellites. For each of those days, a check was made to see whether a level-2A or a level-1C product was available for the location. A level-1C product contains top-ofatmosphere (TOA) reflectance, while level-2A products contain bottom-ofatmosphere (BOA) reflectance. It was preferred to make use of level-2A imagery, since level-1C products gave inconsistent analysis results because of distortions in atmospheric reflectance. However, if a level-2A image was not available for a given date, a level-1C product was downloaded nonetheless. Consecutively, such product was processed to level-2A with help of the Sen2cor tool.

Atmospheric correction with sen2cor processing tool

The sen2cor tool has been developed as part of the ESA STEP science toolbox and has the purpose to perform pre-processing on level-1C (TOA) images based on a scene classification, and atmospheric, terrain and cirrus correction, leading to Level-2A (BOA) products [START_REF]ESA S2A processing user guide[END_REF]. The sen2cor tool is a standalone package and can be freely obtained from the ESA STEP website [14]. The package contains a batch file that was accessed through a python script to automatically process level-1C products to level-2A images.

Extracting data based on field boundaries

After downloading and levelling up of Sentinel-2 products, all raster bands for each date with a resolution of 20m were clipped according to the Lenzen Groβe Weide field's geoJSON file for from Haus Riswick. For a resolution of 20m, the RGB bands, three Red-edge bands, one NIR band, two SWIR bands and a cloud mask were available. The function GDALWarp, part of the GDAL library for fast processing of raster and vector data [START_REF] Warmerdam | [END_REF] was used to clip all raster bands according to the input field boundary. Besides raster bands and the geoJSON representing this boundary, other input parameters for GDALWarp were for instance input and output coordinate reference system, output format (such as a GeoTIFF file), output resolution and resampling method of the pixels.

Calculation and visualization of vegetation indices

From the different Sentinel-2 bands, vegetation indices were calculated. For the initial version of the processing chain, three indices were included: NDVI, WDVI, and a Red Edge index that made use of the sharp increase in reflectance between Red and NIR, useful to quantify leaf chlorophyll content in crops [START_REF] Gitelson | Remote estimation of canopy chlorophyll content in crops 17. ESA S2A processing algorithm[END_REF]. Its equation is as follows, where REband7 and REband5 represent Sentinel-2 bands with wavelengths in respectively the higher and lower regions of the Red Edge slope:

𝐶𝐼 𝑟𝑒𝑑 𝑒𝑑𝑔𝑒 = 𝑅𝐸 𝑏𝑎𝑛𝑑7 𝑅𝐸 𝑏𝑎𝑛𝑑5 -1 (1)
For each Sentinel-2 product, an extra masking band was available. This band was a classification raster with classes representing cloud probability, cloud shadow, vegetation, unvegetated pixels, water, cirrus, and snow. The vegetation index raster was masked according to this masking band [17]. This information was visualized by raster maps for each date in range of dates on which either of the two Sentinel-2 satellites recorded data (given that the maximum cloud cover was not exceeded). For presentation purposes, these maps were compiled in GIF animations showing the crop growth over time in a more dynamic way.

For each date in the input date range, the vegetation index mean and standard deviation were calculated based on the Sentinel-2 pixels located within the field boundary. The cloud cover percentage was calculated by dividing the number of cloud-masked pixels by the total number of pixels within the field boundary. Raster datasets containing more cloud-masked pixels than the selected cloud cover threshold were discarded. For the selected range of dates, vegetation index means and standard deviations, as well as cloud cover were visualized by means of time-series charts.

Results

The aim of the proposed application is to generate relevant information from opensource data, based on a limited number of input parameters. Ideally, those parameters should include start-and end date, which vegetation index to calculate as described in section 2.4, and maximum cloud cover percentage for visualization of results. However, in order to guarantee a smoother data processing workflow, additional parameters such as main working directory, folder paths to input and output locations, and the location of the field boundary shape-or geoJSON file should be provided in the Python script as well. Based on these input parameters and the workflow in the script, information as depicted in Fig. 2 and Fig. 3 was created. Fig. 2 shows the Red Edge index calculated according to eq. 1, for nine selected days during the growing season of 2018. For this specific case study, that index ranged roughly between values of 0 and 5, where red colors represent values closer to zero and green colors imply values closer to 5. For most maps depicted in Fig. 2, 100 percent (so zero percent cloud cover) of pixels were available, but Fig. 2b and Fig. 2g were greatly hindered by cloud cover. In addition, almost in the north-east of this field a divergent patch of pixels is visible in some of the maps, which is a location with experimental plots having a different mowing regime than the rest of the field. The mowing dates for the field Lenzen Groβe Weide in 2018 were April 26, May 26, July 12, and October 9, indicating a season with four growing cycles. On the one hand, the data for the map in for instance Fig. 2c was recorded just before harvesting of the field, clearly depicted by high values in Red Edge index. On the other hand, the data for Fig. 2d was taken not long after mowing, which is represented by lower Red Edge values. Similar patterns were also observed for the dates close to the other three moments of harvesting. Lastly, the map in Fig. 2i shows very low values in Red Edge, which was caused by the extremely dry summer of 2018. Fig. 3a shows time-series containing the mean and standard deviation of the selected vegetation index, and Fig. 3b depicts the cloud cover percentage for the given range of dates. The mowing dates described earlier are also resembled in Fig. 3a, since large drops in Red Edge are visible for those moments in the growing season. However, also large decreases in Red Edge are visible on for instance April 16, as well as a pattern containing very low values for the months July and August. One explanation for the first issue is the large influence of cloud cover, while the second issue was caused by the very dry summer of 2018. Fig. 3b shows the cloud cover range for each date in the growing season, with grey values exceeding the maximum cloud cover percentage and blue values falling below this threshold. For this case study, the cloud cover threshold was set to 90 percent. Only data recorded on days whose cloud cover did not exceed this threshold were used to produce a time-series output as shown in Fig. 3a.

Discussion

In this paper, an alternative was proposed with a first version of a processing chain for extracting, processing and analyzing satellite-based data for grassland monitoring in an automated and open-source way, aimed at stakeholders resembling agribusiness companies and researchers to support their services and research practices. The development and primary testing of this processing chain was mainly conducted on a university's notebook. However, a notebook's or PC's processing memory and storage capacity are limited, which is one reason to upscale the processing workflow to a cloud computing environment for more efficient data processing and analysis in the future. Moreover, this system could still be enhanced to perform more advanced data (pre)processing and analysis, for instance by creating a more elaborated atmospheric correction algorithm and the inclusion of additional vegetation indices. For instance, atmospheric correction was merely performed with help of the sen2cor tool developed by ESA. A great variability in radiometry was observed comparing Sentinel images of multiple days. Therefore, many processing parameters could have been changed in configuration files (xml files as part of a Sentinel product's metadata) to be used in the sen2cor tool for each day separately. These options were considered, but were not included in the pre-processing chain, since this was tough to implement in an automated way. An alternative to the sen2cor tool could have been MACCS/MAJA, which has a more elaborated workflow in comparison to the sen2cor tool, since it includes more advanced weather analysis, cloud detection and atmospheric correction in its algorithm [START_REF]CESBIO multitemp MACCS/MAJA tool[END_REF]. In addition, ESA has been started to process all Sentinel L1C data to L2A in retrospect, so the sen2cor tool might become obsolete in the future. The next step was to discard vegetation images with a cloud cover more than a given threshold (90% was used for this case study, see Fig. 3b). One way to deal with this issue would be to lower the cloud cover threshold, which would lead to more accurate vegetation index output for the remaining data, but also to a loss of data not meeting the cloud cover threshold requirement. For instance, the maps in Fig. 2b and Fig. 2d were greatly influenced by cloud cover, which is resembled by the outliers in Fig. 3a for the dates on April 16 and June 20. In conclusion, the cloud cover percentage threshold is a trade-off that the end-user of this monitoring system should decide upon. Another way to improve this process would be to take a sequence of images in time as input, and systematically replace disturbed pixels from one image by clean pixels from a subsequent or previous image. Clean pixels are pixels that do not containing cloud cover or NA values. This replacing process could for example be performed by the sen2three tool, which was developed as part of the ESA STEP program as well [START_REF]ESA Sen2Three tool[END_REF]. However, this tool takes a sequence of Sentinel L2A images and merges them into one clean composite image. Since data was limited already, this could have had a disadvantageous effect on time series output such as in Fig. 3a.

Conclusions

The main goal of this paper was to develop and provide insight in the processing workflow of a grassland monitoring system in an open-source and automated way. In such workflow, processing steps are required to get from input data to output representing useful information for grassland management practices, and to compare that output to information derived from airborne or ground based data. The higher the cloud cover threshold, the more data is available to create a timeseries, but also the lower the accuracy becomes. Lowering this threshold leads to a loss of data but also to an increase in accuracy. This is a trade-off that should be decided upon by the end-user of the monitoring system.

Despite the outliers caused by a lack of pixels due to cloud cover, first results gave insights about the mowing regime over time, which could support optimizing and managing a grassland system in a more effective way. These results and data could be useful as well to elaborate on in further research.

The proposed script was a preliminary version of a processing change, which is ready to be further enhanced and expanded with coding for more advanced data preprocessing and data analysis in the future.

6

Further research

• Further develop processing chain for more elaborate pre-processing and analysis steps such as atmospheric correction and the inclusion of extra vegetation indices. • Further develop processing chain to make it suitable to include airborne based data.

• Translate time-series information into crop biomass information for the different cycles in a growing season. • Upscale the processing chain to a cloud computing environment for more efficient data processing and analysis.

Fig. 2 .

 2 Fig. 2. Red Edge on March 4 (a), April 16 (b), April 26 (c), May 3 (d), May 21 (e), June 7 (f), June 20 (g), June 27 (h), August 6 (i), 2018

Fig. 3 .

 3 Fig. 3. Red Edge index (a) and cloud cover percentage (b) from March 1 till October 31, 2018

Acknowledgements

This work was supported by the SPECTORS project (143081), which is funded by the European cooperation program INTERREG Deutschland-Nederland.