Dale Roberts
email: ds.roberts@anu.edu.au

Mark Cheeseman
email: mark.cheeseman@anu.edu.au

A Performance Assessment of the Unified Model

Keywords: unified model, numerical weather prediction, performance analysis, high performance computing

The Unified Model (UM) is a model produced by the UK MetOffice for Numerical Weather Prediction (NWP) and climate simulation. It is used extensively by various university, government and other research organizations on the large supercomputer hosted at the National Computing Infrastructure (NCI). A 3-year collaboration between NCI, the Australian Bureau of Meteorology and Fujitsu is underway to address performance and scalability issues in the UM on NCI's supercomputer, Raijin.

IO performance in the UM is the most dominant factor in its overall performance. The IO server approach employed is sophisticated and requires proper calibration to achieve acceptable performance. Global synchronization and file lock contention is a problem that can be remedied with simple MPI global collective calls. Complimentary IO strategies, such as MPI-IO and directed IO, are being investigated for implementation.

The OpenMP implementation employed in the UM is investigated, and is found to have inefficiencies that are detrimental to the load balance of the model. Only loop-wise parallelism is employed. Due to the inherently imbalanced nature of the model, a task-wise approach could yield improved threading efficiency.

Introduction

The Unified Model, produced by the UK MetOffice, is a model used for atmospheric simulation over time scales ranging from a few hours for numerical weather prediction (NWP) to decades for climate modeling. An overview of its development can be found in [START_REF] Brown | Unified Modeling and Prediction of Weather and Climate A 25-Year Journey[END_REF]. It is used in the Australian Bureau of Meteorology's (BoM) operational NWP forecasts at both a global and regional resolution. It also serves as the atmosphere component of the Australian Community Climate and Earth System Simulator (ACCESS) coupled climate model. Its complexity and scale demand the use of supercomputers. For this paper, we focus on a global configuration of the UM on the National Computational Infrastructure's supercomputer, Raijin. Raijin is a Fujitsu Primergy cluster consisting of 3592 nodes. Each node possesses two Intel Xeon E5-2670 ble of 56GBit/s between any two nodes. Raijin also provides a fast lustre-based [START_REF] Braam | The Lustre Storage Architecture[END_REF] file system with approximately 10PB of storage capacity and a peak write performance in excess of 150GB/s. For this paper Intel FORTRAN and C compilers are used along with OpenMPI [START_REF] Gabriel | Open MPI: Goals, Concept and Design of a Next Generation MPI Implementation[END_REF] for MPI (Message Passing Interface) support.

The UM configuration of interest for the presented performance assessment and optimization work is the global N512L70 configuration. It possesses an approximately 25km horizontal resolution (making for a horizontal grid of dimensions 1024 by 769) at mid latitudes and contains 70 model vertical levels. Optimization of configuration is important to BoM's future operational weather forecasting suite, as it will lead to greater accuracy in both weather and climate prediction. BoM has mandated a strict performance target of 10 model days per wallclock hour if this global configuration is to be included in its operational forecast suite. We suspect that performance issues observed at N512L70 (such as I/O and efficient OpenMP use) may be more severe at higher resolutions, such as N768L85, the global resolution of the next BoM operational forecast suite. Thus any solutions implemented at the N512L70 configuration should be applicable and valuable for them as well.

Initial Scaling Tests

Using version 8.4 of the Unified Model [START_REF] Wood | An inherently mass-conserving semi-implicit semi-Lagrangian discretization of the deepatmosphere global non-hydrostatic equations[END_REF] a strong scaling analysis of the global N512L70 configuration is performed. Each run lasts 24 model hours with a time step of 10 minutes. The UM is capable of using a multi-threaded model, in that each MPI task is able to spawn multiple OpenMP threads. When this multi-threaded mode is enabled, the IO can be performed using the IO Server asynchronous dump feature. In this feature a number of MPI tasks are set aside to perform the expensive process of writing model output to disk as the remaining MPI tasks perform the normal computational work in the atmosphere model run. If this feature is not activated, all output is passed to and written by a single MPI task while all other MPI tasks sit idle -thereby interrupting the normal model calculations.

Figure [START_REF] Brown | Unified Modeling and Prediction of Weather and Climate A 25-Year Journey[END_REF] shows observed strong scaling of the model with the IO Server feature disabled and with all output switched off. It's clear that the single task writing approach severely limits scalability. There is no observed benefit in using more than 512 CPU cores with output enabled in this case. With an averaged runtime of 984 seconds at 2048 CPU cores (over 5 runs), it will not be possible to meet BoM's operational performance requirement with this configuration.

With the IO Server feature enabled, observed strong scaling improves substantially as illustrated in Figure [START_REF] Braam | The Lustre Storage Architecture[END_REF]. The UM must be run in multithreaded mode here. (I.e. at least 2 OpenMP threads need to be spawned and available to each MPI task). This requirement comes from the IO Server design that uses two threads per IO Server MPI task. Thus, the use of IO servers will be an integral part in meeting operational performance requirement.

The two sets of points at 512 cores in Fig. [START_REF] Braam | The Lustre Storage Architecture[END_REF] denote two different runs performed on Raijin, one with 2 OpenMP threads per MPI task, and one with 4 OpenMP threads per MPI task. It is clear that the best utilization of 512 cores is to have 256 MPI tasks each with 2 OpenMP threads. This type of observation leads us to believe that improved performance in the UM could be achieved through additional OpenMP optimization. In spite of the IO now residing on separate MPI tasks, Figure [START_REF] Braam | The Lustre Storage Architecture[END_REF] shows that the performance with IO disabled altogether is still better than when IO is enabled on the IO servers. Fig. 1. Observed strong scaling of the global N512L70 configuration using the single MPI task I/O approach. We define 'Speedup' as the ratio of the runtime at 256 cores compared to that at larger stated core counts. Ideal scaling would see a job run on twice as many cores take half the time, a speedup factor of 2.

It is possible for an IO server to possess more than 1 MPI task. For the 'IO enabled' runs shown in Figure (2), we used IO servers containing 8 MPI tasks each. Write performance can be enhanced by fine-tuning the layout and structure of the IO servers used -including the number of MPI tasks allocated to each. As an example, in Figure (2), we display the difference in observed strong scaling at 4096 cores using 8 and 16 MPI tasks per IO Server (e.g. the green cross). While IO Server tuning does improve observed runtimes, it is not enough to significantly change the overall scalability of the global configuration.

Threading Performance

Investigations showed that, while the UM can be run with up to 16 OpenMP threads per MPI task on Raijin, performance decreases significantly when more than two threads per MPI task are employed. A detailed analysis of the OpenMP performance was completed using the Score-P [START_REF] Schlutter | Profiling Hybrid HMPP Applications with Score-P on Heterogeneous Hardware[END_REF] profiling and tracing tool, in conjunction with the Vampir trace viewer [START_REF] Knüpfer | The Vampir Performance Analysis Tool-Set[END_REF]. For this particular analysis, the UM N512L70 configuration was run in a 16×30 decomposition, with 4 IO servers, each comprising 8 tasks; giving a total of 512 MPI tasks, each with 2 OpenMP threads. Score-P adds approximately 20% the model run time for a full performance trace. Vampir categorizes all function calls, and displays each category in with different colors according to the legend in Figure [START_REF] Gabriel | Open MPI: Goals, Concept and Design of a Next Generation MPI Implementation[END_REF]. The vast majority of time between the start of the atmosphere step and the first 'MPI_WaitAll' call (the circled section of the trace) is spent in the 'OMP_SYNC' category of calls. This category denotes every time an OpenMP thread needs to wait for another thread to complete. In this case, the large scale precipitation calculation in the microphysics section of the atmosphere step contains an !$OMP PARALLEL declaration, which then proceeds to be executed in serial, whilst the newly created OpenMP thread waits for the completion of the main thread. This section of the code is potentially taking twice as long as necessary, thereby increasing the load imbalance among the tasks. Load imbalance comes about when each MPI task performs a different amount of work on the data it has. When this occurs, the tasks that complete that section first need to wait for the final tasks to complete that section. This can be seen in the circled region of Fig. [START_REF] Wood | An inherently mass-conserving semi-implicit semi-Lagrangian discretization of the deepatmosphere global non-hydrostatic equations[END_REF]. During this wait period, they will not be doing any useful work, and are considered wasted. One of key activities in improving the performance of this UM configuration will be eliminating this type of observed load imbalance. For the entire atmosphere step, OMP thread 1 is active (i.e. the program is inside an!$OMP PARALLEL region) for approximately one third of the time and spends over a quarter of its time in in the OMP_SYNC category. Though we have selected an arbitrary task for this comparison, this behavior is observed in all non-IO MPI tasks.

Consequently, one OpenMP thread on each atmosphere MPI task spends approximately 30% of its time in MPI_WaitAll calls. This load imbalance is further exacerbated as the halo exchange immediately follows, resulting in more idle CPU time as neighbor tasks wait for threads to synchronize.

OpenMP use in the UM atmosphere code is overwhelmingly data-based parallelization, in which separate iterations of a 'DO' loop are executed on individual OpenMP threads. For a model as inherently load imbalanced as the UM, this may not be sufficient. An alternative approach could be to employing work-share OpenMP constructs where separate independent tasks are allocated to different OpenMP threads. A potential target for this approach would be the convection control subroutines. Every atmosphere tile may undergo one of deep, shallow or congestus convection, as well as mid-level convection. This routine is particularly load imbalanced, as the entire model area on a particular MPI task may only undergo mid-level convection, whereas other MPI tasks may spend a significant amount of time in one of the other convection routines as well as mid-level convection. One could separate the deep, shallow and congestus convection routines on to different OpenMP tasks, such that they are calculated simultaneously, and then revert to data-based parallelism for the mid-level convection calculation.

IO Performance

Jobs ran at the global N512L70 configuration run for 1 model day and generate approximately 151GB of output data to disk. If one ignores the UM code completely and just performs a single low-level sequential write to Raijin's high-speed filesystem, then it takes about 220 seconds to perform the write. BoM's current operational forecast suite requires the global UM configuration to run for 10 model days and generate an accordingly ten-fold increase in output. If one ignores the UM code and gain just performs a single low-level sequential write, the time required for generating the output jumps to approximately 36 minutes. BoM's operational forecast suite cannot last longer than one wallclock hour. Thus, one has (at best) 24 minutes to complete all computations in a N521L70 run which is not feasible currently. (Both I/O approaches, IO Server and single MPI task output, use sequential writes for data output.)

In the IO server implementation, model output can be written concurrently with normal atmospheric calculations. At each write, each MPI task performing atmosphere computations sends relevant output data to the nearest IO task (by MPI rank) of each IO server. The first IO server to receive all necessary output data commences the writing of the first output/dump file. The remaining IO servers will flush out all data pertaining to that output file and focus on the next output file. The 1 model-day runs of the N512L70 configuration produces fifteen dump files in total. Testing showed that the optimal IO Server setting for this configuration varies depending on the number of MPI tasks assigned to the atmosphere model. Fig. [START_REF] Knüpfer | The Vampir Performance Analysis Tool-Set[END_REF] shows that for fewer than 1024 or fewer MPI tasks (2048 total cores), 8 MPI tasks per IO server and up to 4 IO servers. Moving to 4096 total cores, gives 8 IO servers in total, which is seen to harm performance. In the 8 IO server setup, several servers seem to remain idle as fewer than 8 files are being written to simultaneously for the majority of the time. Furthermore, the overhead associated with synchronizing all files across 8 IO servers will ultimately increase the time spent in IO, to the point where the IO will take longer than the model run when a large number of MPI tasks are utilized. Reducing the number of IO servers by increasing the number of tasks per server but retaining the total number of MPI tasks for IO is also seen to be detrimental to performance for 1024 or fewer MPI tasks, as this limits the ability of the IO servers to write multiple files in parallel. Furthermore, increasing the number of IO tasks and retaining the same number of IO servers is also seen to reduce performance, as in this case, fewer MPI tasks are being allocated to the model itself.

Even when IO servers are employed, there is still a significant overhead due to IO. At 4096 cores, it takes approximately 185 seconds to complete the 24-model hour run without IO, and 305 seconds to complete it with IO enabled. Fig. [START_REF] Thakur | Data sieving and collective I/O in ROMIO[END_REF] shows a timesampled profile of the speed of UM output writes during a run with IO servers enabled. Raijin's high-speed file system is capable of sequentially writing to a single file at around 700MB/s. The observed average write speed for a UM model dump is around 300MB/s. This is due to the overhead of the IO servers moving all data across to the main IO server MPI task in 4MB segments, and writing each segment as it is received. This was performed with a 32×62 decomposition using 4 IO servers each with 16 MPI tasks, for a total of 4096 cores.

Over half of the total run time was spent in the twelve longest atmosphere steps. The long duration of these steps is caused by IO servers being unable to complete some given request. There are a variety of reasons for this, for instance, an insufficient amount of memory has been allocated to the IO servers, which, when full, will cause Cumulative Time Spent in Atmosphere Steps the model to stop whilst the data in memory on the IO servers is written to disk. Many of these issues could be alleviated with tuning of the IO server parameters. However, as one increase the model resolution, this type of fine-tuning will not prevent such stalling. The number of output files will not increase as resolution increase. Thus one will need to either a) increase the amount of memory to each IO server task so that it can hold more output data, or b) increase the number of MPI tasks dedicated to IO server activity (eg. Increase number of IO servers and/or increase number of MPI tasks belonging to each IO server). Each alternative will suffer from the same stalling behavior. Thus alternate I/O methods should be explored. One possible approach is MPI-IO [START_REF] Thakur | Data sieving and collective I/O in ROMIO[END_REF] where multiple IO server tasks can write to files concurrently thereby achieving greater write speeds. The overhead of sending entire files across to a single MPI task would also be eliminated.

Conclusions and Future Work

We have investigated the overall performance of the UM MetOffice's Unified Model focusing on the OpenMP and IO server implementation. We found inefficiency in the threading, with the second OpenMP thread of each MPI task spending approximately 75% of the time idle. OpenMP parallelism has been predominantly loop/data-wise. A more task-wise OpenMP parallelism strategy could improve the treading efficiency significantly. In particular, the sections of code identified in Section 3, namely, the different convection subroutines, will be the first target for optimization. IO is a significant factor in the run time of the UM, even with the IO server approach. Tuning of available IO server parameters can improve the performance at N512L70, but it is unlikely that parameter tuning will have sufficient performance for higher resolution configurations. One possible way to provide such speeds is to make use of Raijin's high-speed parallel file systems by employing a parallel IO standard, such as MPI-IO. In particular, the large dump and post-processing files produced, at several gigabytes each, rely on accumulating data from each IO server process before writing the data from a single MPI task. This accumulation step can be replaced by MPI-IO calls that allow each IO process to write to the file in lustre-aware manner.

Fig. 3 .

 3 Fig.3. The function group legend used by Vampir. An ideal trace will spend the majority of its time in the 'Application', 'OMP_LOOP' or 'OMP_WORKSHARE' states. These categories denote function calls in which useful computational work is occurring. Time spent in 'OMP_SYNC' and 'MPI' denotes points where one OpenMP thread is waiting for another to compete a task, or waiting for data to be received from another process, and no useful computation is occurring.

Fig. ( 4

 4 Fig. (4) shows a selection of atmosphere tasks in the 'Master Timeline' view from Vampir version 8.3.0. The view has been zoomed in on a single atmosphere time step, without radiation calculations, that is performed every 6 time steps.

Fig. 4 .

 4 Fig. 4. The 'Master Timeline' view of 5 MPI processes performing a single UM atmosphere step. Function groups are colored as per Fig. (3). Note the significant load imbalance in the circled region.

  Fig. (5) illustrates a similar scenario. Here we display the work done by two OpenMP threads assigned to a single MPI task running an atmosphere step.

Fig. 5 .

 5 Fig. 5. Vampir's view of the call stack of 2 OpenMP threads allocated to an MPI task running a portion of one atmosphere step. Function groups are colored as per Fig (3).

Fig. 6 .

 6 Fig. 6. Comparison of different IO server configurations with the same number of total MPI tasks.

  Fig. (8) shows the effect this has on atmosphere tasks.

Fig. 8 .

 8 Fig. 8. Cumulative time spent in UM atmosphere steps, ordered by longest time step to shortest.This was performed with a 32×62 decomposition using 4 IO servers each with 16 MPI tasks, for a total of 4096 cores.

  Observed strong scaling of global N512L70 configuration with IO Servers enabled.

	Fig. 2.
			N512L70 scaling, single threaded
		8
	Speedup	2 4					Ideal Scaling IO enabled
							IO disabled
		1
		256	512	1024 # of CPU cores	2048
			N512L70 scaling test, multi threaded
	16					Ideal Scaling
	Speedup	2 4 8					IO disabled, 2 threads per MPI task IO enabled, 2 threads per MPI task
							4 threads per MPI
		1					task
		256	512	1024	2048	4096	Alternate IO server
				Cores			configuration

  Observed write speeds for a model day run of the N512L70 configuration. The model produces output at approximately 200, 400, 600 and 800 seconds.

				IO sever configuration comparison
		4
								8 IO tasks per IO server,
	Speed up	2						default configuration 16 IO tasks per IO server, same number of IO tasks as
								default configuration
								16 tasks per IO server, twice
	Fig. 7.	1	256	512	1024	2048	4096	as many IO tasks as default configuration
					Cores

core CPUs giving Raijin a total of 57,472 CPU cores and a theoretical peak performance of 1.2 Petaflops. The nodes are connected by a fast Infiniband network, capaadfa, p. 1, 2011. © Springer-Verlag Berlin Heidelberg 2011


Ben Evans

Lesley Wyborn

Tim Pugh
email: t.pugh@bom.gov.au

Chris Allen

Joseph Antony

Kashif Gohar

David Porter

Jon Smillie

Claire Trenham

Jingbo Wang
email: jingbo.wang@anu.edu.au

Kashif Go- Har

Alex Ip
email: alex.ip@ga.gov.au

Gavin Bell

{ben Evans

The NCI High Performance Computing and High Performance Data Platform to Support the Analysis of Petascale Environmental Data Collections

Keywords: high performance computing, high performance data, cloud Computing, data-intensive science, scalable data services, data cube, virtual laboratories adfa, p. 1, 2011

scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

The National Computational Infrastructure (NCI) at the Australian National University (ANU) has organised a priority set of large volume national environmental data assets on a High Performance Data (HPD) Node within a High Performance Compu-

Data Collections

There are 31 (and growing) data collections in the initial ingestion at NCI [START_REF]National Computational Infrastructure: National Collections[END_REF] requiring over 10 Petabytes (PBytes) in storage volume (Table 1). They are currently categorised into six major fields all related to the environmental sciences: 1) earth system sciences, climate and weather model data assets and products; 2) earth and marine observations and products; 3) geosciences; 4) terrestrial ecosystem; 5) water management and hydrology; and 6) astronomy, social science and biosciences. These data collections are predicted to grow exponentially over time. For example, the datasets for the international Coupled Model Inter-comparison Project (CMIP5) [START_REF] Taylor | An Overview of CMIP5 and the experiment design[END_REF] experiment completed in 2013 are well over 3 PBytes; the data for the next CMIP6 experiment will be an order of magnitude larger again by 2020 due to higher resolution spatial grids. Current satellite remote sensing data collections, along with their higher products derived from large time-series of Australian Landsat collection, Moderate Resolution Imaging Spectroradiometer (MODIS), Advanced Very High Resolution Radiometer (AVHRR) and Visible Infrared Imaging Radiometer Suite (VIIRS) are in the order of 2 PBytes. The next generation of satellites such as Japanese Himawari-8/9 mission [START_REF]Himarwari-8/9 of the Meteorological Satellite Center (MSC) of the JMA[END_REF] and the European Space Agency's Copernicus programme [START_REF]European Space Agency: The Copernicus programme[END_REF] are about to come online, with higher spatial resolution sensors and data volumes that in time will dwarf the earlier satellite data collections.

To facilitate effective and efficient use of large volume data collections and to support their use in high performance computing environments, attention must be placed Table 1. Data Collections hosted at the NCI RDSI node as of 27 November, 2014. The fields are 1) earth system sciences, climate and weather model data assets and products, 2) earth and marine observations and products, 3) geosciences, 4) terrestrial ecosystem, 5) water management and hydrology, and 6) astronomy, social science and biosciences. on the whole data workflow from creation to publication, including data management plans, provenance capture, and unique identification of the data through Digital Object Identifiers (DOIs) and other forms of data discovery and access. As there is also a move towards enabling HPC analytics directly on the data content, increasingly attributes will need to be self-describing and conform to international standards for vocabularies and ontologies.

Field Collection

Because of the scale of the data collections at NCI, some aspects of data management are now active areas of ongoing research and development collaboration. One example is provenance systems, where specialized databases will be created to allow use-dependent provenance graphs to be generated on-the-fly.

High Performance Data (HPD)

With both the increased data volumes, and the increased computing power required to effectively process and analyse them, the coupled computational environments are increasingly too large or complicated to be architected separately: they must be codesigned at inception. This has given rise to High Performance Data (HPD), as a complementary approach to the traditional High Performance Computing (HPC). We define High Performance Data (HPD) as data that is carefully prepared, standardised and structured so that it can be used in Data-Intensive Science on HPC.

To achieve this, the use of the data needs to be considered as part of the production and management of the data, rather than an after-thought for data curation.

Many of the data sets that have been assembled on the NCI are ones that were considered too large to process and access by conventional means; bandwidth limits the capacity to move them, data transfers are too slow; and latencies to access them are too high. Even if the data sets can be moved, few can afford to store them locally, the energy costs are substantial, and provenance and versioning of the large data collections at multiple sites are too complex to manage. We predict the increasing scale of data, its associated processing, and complexity of software environments will be too difficult or affordable to host beyond a few trusted international centres.

Properly structured, the HPD collections provide new opportunities to integrate and process large volumes of data at resolutions and at scales never before possible. Once created, HPD enables in-situ processing of the data and requires users changing their processing to use trusted online applications and services on the data in-situ.

4

High Performance Data Infrastructure

Design Considerations

The infrastructure at NCI has been designed to support Data-Intensive Science within HPC environments. The new infrastructure was designed to reduce the time taken to process the data and required a balance between the available processing power and the ability to access data.

More specifically the infrastructure to support HPD at NCI was designed and programmed, and is operated in a way that enables multiple users to rapidly invoke dynamic high performance computation on these large volume data collections from a variety of use-cases. Processing can take place in a variety of modes: batch-style deep processing; interactive environments for preparing model and analysis computation; web-time data processing including the Open Geospatial Consortium (OGC) Web Processing Service (WPS) [START_REF]Open Geospatial Consortium Web Processing Service[END_REF], Virtual Desktop Interfaces (VDIs) for fully assembled software environments; and various server-side visualisation capabilities.

Computer Platforms for HPD

To ensure that HPD techniques can scale out for the demands of Data-Intensive Science, all parts of the data and services need to be considered for their ability to scale out and for high bandwidth and low latency access to data. This includes batch-mode HPC platforms, cloud infrastructure, and high performance filesystems. NCI provides a 1.2 PetaFLOP supercomputer (Raijin) with over 57,000 processing cores and an approximately 10 PBytes Lustre filesystem. The system includes Fujitsu compute nodes consisting of Intel SandyBridge processors, a QDR InfiniBand network, and a Lustre filesystem for a cluster-wide namespace and a PBSPro batch system.

For its private cloud, (Tenjin), NCI selected OpenStack [9] as its underlying cloud technology on a HPC-class 3000 processor core system using the same generation of technology as the Raijin supercomputer, including Intel's SandyBridge processors and a Mellanox QDR InfiniBand interconnect. In addition, each node of Tenjin has local Solid State Drives (SSDs) to allow high IOPs workloads. When using virtual machines, the performance of the hypervisor is critical for the performance of both computational and IO systems. Over time, the performance of these hypervisors has improved. Future work on OpenStack will see bare-metal nodes used as part of the node structure where virtualisation is the limiting factor for operations.

Both the supercomputer and cloud can mount several facility wide Lustre [START_REF]Lustre file system[END_REF] file systems, which maintain the data collections and other persistent reference data. The Lustre systems use a mixture of NetApp and DDN storage arrays, which are managed by Object Storage Servers (OSS) within scalable units. There are currently over 50 OSSes serving approximately 20 PBytes of filesystem space.

To provide operational robustness, each layer of the infrastructure has its own management domain. The connection to both the supercomputer and the cloud are managed with a scalable set of data serving bridging infrastructure using Lustre LNET and NFS that manages both the performance and security to each of the systems over the InfiniBand network fabric.

Software Stack Management

The variety of both environments and services for Data-Intensive HPD environmental science has required a deeper consideration of the software environment including management and deployment practices and the supporting software infrastructure.

This involves the two key concepts of the Development-to-Operations (DevOps) process which includes firstly testing and integration, and secondly re-usable trusted software stacks.

To manage the software systems, NCI has developed a system of Puppet Bundles based on Puppet [11] and Git [12]. The NCI Puppet Bundles are run in a masterless mode, where the coordination of nodes and deployment is achieved through standard Git management. All operational stacks are maintained to a high degree of traceability and reproducibility through this system.

All shareable services of the NCI environment have been written to make use of the underlying Puppet core framework. This provides easy integration, simplified sharing and re-use, and the ability to more easily scale-out the infrastructure. Other technologies (e.g., Docker [START_REF] Docker | [END_REF]) can be deployed in a layered ecosystem of software deployments.

Scalable Data Access

Modern Data-Intensive environments require much more than just discovery and access to filesystems. To find data, a research problem requires the ability to programmatically search for attributes and extract data using defined services like the Data Access Protocol (DAP), and other geospatial services such as those defined by the OGC including Web Map Service (WMS) [START_REF]Open Geospatial Consortium: Web Map Service[END_REF], Web Coverage Service (WCS) [START_REF]Open Geospatial Consortium: Web Coverage Service, OGC[END_REF] and Web Feature Service (WFS) [START_REF]Open Geospatial Consortium: Web Feature Service, OGC[END_REF]. NCI have developed a scalable catalogue framework using a combination of GeoNetwork [17] and ElasticSearch [START_REF] Elasticsearch | [END_REF] to record the metadata, and allows for rapid search across key attributes and respond with the relevant data service URI or filesystem location for access.

All data is registered in the catalogue framework including the services that are meaningful to the data, e.g., THREDDS [START_REF] Thredds | [END_REF], Hyrax [20], and GeoServer [START_REF]Open Source Geospatial Foundation: GeoServer[END_REF]. To find the data, programs have begun to standardize on JSON [START_REF]JSON[END_REF] objects for interacting with the search infrastructure.

Datacube Technologies

For analysis in high performance environments, those collections that comprise multiple data sets and data series, even if they conform to agreed international standards, need to be aggregated into seamless data arrays. The datasets being assembled are often of different spatial and temporal resolution. To analyse these data as coherent units, the data will increasingly need to be stored, aligned and indexed using datacube methodologies.

A number of collaborations have now been established to address this issue. For example, GA, CSIRO and NCI are collaborating on a project called the Australian Geoscience Data Cube (AGDC) [START_REF]Australian Geoscience Data Cube (AGDC[END_REF]. For this project, the data is prepared to allow rapid analysis of the data in several dimensions (e.g., spectral, spatial and temporal) by converting to consistent grids, each with a fixed and consistent geospatial footprint, forming a space-time 'data cube'. In the development phase of the AGDC, some 636,000 Landsat product scenes were converted to 4 million spatially-regular, time-stamped, band-aggregated tiles. This consistent gridding minimised the geospatial processing necessary to locate individual pixels of data across multiple scenes (or files) of Earth observation data. Tile-based analyses could then be distributed across thousands of compute nodes for parallel computation underpinned by the highperformance Lustre filesystems of the NCI facility. As a result of this successful development phase, the AGDC project is now being expanded to incorporate other data Earth Observation imagery including MODIS data and elevation data.

Interdisciplinary and Collaborative Virtual Laboratories

Virtual Laboratories are a set of integrated software stacks that provides workflow software, specialised tools, web services, and visualisation services. The nature of the Virtual Laboratories is that they consist of a mixture of software stacks that are sourced from independent, trusted software and data repositories and are integrated as an interoperable stack.

Common software stack components may also be reused, and re-integrated by the individual Virtual Laboratory that requires them. This is a key way where software (and ultimately data) can be reused and/or repurposed for multiple Virtual Laboratories and will ultimately assist in achieving interdisciplinary outcomes.

The development of these trusted software stacks usually involves large-scale engineering efforts, often entailing global collaborations. Some well-known examples used at NCI include the Earth Systems Grid Federation (ESGF) [START_REF]Earth Systems Grid Federation: About the Earth System Grid[END_REF] (of which NCI is a core member), and model management environments such as Cylc [START_REF] Oliver | The Cylc Suite Engine[END_REF] and Rose [START_REF]Rose[END_REF] and visualisation software such as UV-CDAT [START_REF] Williams | UV-CDAT[END_REF] and the Oceans Data Interoperability Platform (ODIP) [START_REF]Oceans Data Interoperability Platform[END_REF].

The release and integration of the software environments at NCI are managed using our Puppet Bundles approach, which ensures that software is both upgradable and maintainable. By carefully managing the software stack to track the upstream software repository, the trustworthiness of individual Virtual Laboratory stacks can be maintained.

Opportunities and Challenges

The new infrastructure developed at the NCI has enabled:

• A dramatic improvement in the scale, resolution and reach of Australian environmental research; • High performance access to nationally significant research collections using new Data-Intensive capabilities to support cutting-edge research methods; and • The realisation of synergies with related international research infrastructure programs.

Further, by co-locating major diverse environmental data collections with HPC infrastructures an interdisciplinary approach to solving key global Environmental Science issues such as impacts of climate change; sustainably exploiting scarce water, mineral and energy resources; and protecting our communities through better prediction of the behaviour of natural hazards is now feasible.

However, these opportunities cannot be easily realised unless some key remaining challenges are addressed, including common licensing framework, authentication and authorisation, semantic enablement of data using controlled vocabularies, and provenance tracking of workflows. There is also a growing recognition of the need for the development of new specialised skills, in particular in data science. There is also a need to deploy these skills in multi-faceted interdisciplinary teams.

Conclusions

We have reached the following conclusions about the future directions of computational and Data-Intensive methods in environmental and earth systems science through the course of this work:

1. The need to access the large and exponentially increasing data volumes in Environmental Science is now acute; traditional data access and HPC technologies no longer scale. 2. We define High Performance Data (HPD) as data that is carefully prepared, standardised and structured so that it can be used in Data-Intensive Science on HPC. 3. HPC can now be combined with HPD to support the more complex workflows of Data-Intensive Science. 4. The coupled HPC-HPD environments and the data are increasingly too large or complicated to be architected separately: they must be co-designed at inception. 5. HPC and HPD are now integral to Earth and Environmental Research. Combined they offer new opportunities to collaboratively address problems such as climate change; sustainable exploitation of our water, mineral and energy resources; and better prediction of natural hazards.

Acknowledgements

The authors wish to acknowledge funding from the Australian Government Department of Education, through the National Collaboration Research Infrastructure Strategy (NCRIS) and Education Investment Fund (EIF) Superscience fund through the NCI, RDSI and National eResearch Collaboration Tools and Resources (NeCTAR) projects.

ting (HPC) facility, as a special node under the Australian Government's National Collaborative Research Infrastructure


Ritaban Dutta
email: ritaban.dutta@csiro.au

Cecil Li

Daniel Smith

Aruneema Das

Jagannath Aryal

Big Data Architecture for Environmental Analytics

Keywords: big data, architecture, machine learning, semantics

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

The ultimate challenge in agricultural decision support systems is to overcome the data unavailability and uncertainty to improve the natural resource management efficiency and achieve better business objectives. Uncertainty factors in the agricultural and environmental monitoring processes are more evident than before due to current technological transparency achieved by most recent advanced communication technologies. Poor data quality and uncertainties make most agricultural decision support systems unreliable and inefficient. This inefficiency leads to failure of agricultural and environmental resource management [START_REF] Dutta | Recommending Environmental Big Data Using Semantic Machine Learning[END_REF][START_REF] Plaisant | Interface and data architecture for query preview in networked information systems[END_REF][START_REF] Marchal | A Big Data Architecture for Large Scale Security Monitoring[END_REF]. It is evident that there is a serious need to capture and integrate environmental and agricultural knowledge from various heterogeneous data sources including sensor networks, individual sensory systems, large scale simulated models, patched historical weather data, satellite imagery, domain knowledge and contextual user experience for better decision support with high reliability and confidence. This approach would be able to provide a much wider framework for complementary knowledge validation and meaningful utilization of the acquired knowledge. In agricultural domain knowledge is very volatile in nature, which can only be passed from generation to generation, can vary on a daily basis based on requirements. The farmers can understand and only tell the story of the field, just by looking at the colour of the soil or by looking at the sky. One of the key motivations of this research was to develop an architecture, which can incorporate unstructured, undocumented, ad-hoc knowledge into a structure rule base to be used directly in the big data analytics for better decision support system. Recent development of heterogeneous big data analytics and architectural development for big data integration for agricultural solution was the main motivation behind this work.

Big Data Centric Architecture

In the intelligent environmental knowledgebase (i-EKBase) project we aimed to integrate multi scale heterogeneous data from 'Australian Water Availability Project (AWAP)', 'Australian National Cosmic Ray Soil Moisture Monitoring Facility (Cos-mOz)', 'SILO Data', 'ASRIS Soil data', '250m resolution \ NASA MODIS data', '30m resolution NASA LANDSAT data', 'Australian Digital Elevation Data', and finally 'Domain knowledge is available (i.e. farmer's long experience of daily decision making based on generation wise climate adaptation knowledge). Then on top of this large data integration, our novelty was to apply data driven spatio-temporal artificial intelligence (machine learning) analytics to learn and establish new environmental correlations to make the decision support system more sustainable and scientifically justified to improve business profitability and productivity [START_REF] Zhong | On mixing high-speed updates and in-memory queries: big-data architecture for real-time analytics[END_REF][START_REF] Luo | A Component-based Software Development Method Combined with Enterprise Architecture[END_REF][START_REF] Dutta | Deep cognitive imaging systems enable estimation of continentalscale fire incidence from climate data[END_REF]. Fig. 1 shows the UML diagram of the system architecture that has been proposed and implemented in this study. In the proposed mechanism we process the available metadata separately from the data observations. Feature extraction process was applied to the metadata to create a unique feature base from the individual data source. This processing was done using data mining and text mining techniques. It also involved unsupervised machine learning techniques (i.e. unsupervised clustering tech-niques). Main application area of machine learning technique was in the processing of actual observations from various sensors and systems. Various supervised machinelearning techniques (i.e. neural networks) were used to extract feature base from the observation database. Uniqueness of the architecture in this context was in the selection of machine learning methodology to extract targeted feature base from the observations. The targeted feature base was determined on the basis of the actual application and also based on the available domain expert knowledge usually undocumented and belonging to individuals (i.e. farmers) as long-term field experience. We used the domain knowledge to provide guidance to the machine learning processes to extract the desired feature space. This was done to make the feature extraction process more meaningful and unique. This approach was able to make the machine learning process (so called black box approach) more usable in terms of bio-physically explanations for the domain people. Domain guided extraction using machine learning was named as semantic extraction hence it produced semantic feature base. Meta-feature base and semantic feature base were integrated to form an enriched feature space, which was a most significant representation of the heterogeneous big data. The dimension reduction is an important step in any big data related architecture, especially when extremely large number of highly correlated observations is present. In this architecture we show that dimension reduction could be done in a domain specific meaningful way to increase the efficiency of the system, and also to increase the accuracy of the system by enriching the data semantically [START_REF] Dutta | Recommending Environmental Big Data Using Semantic Machine Learning[END_REF][START_REF] Plaisant | Interface and data architecture for query preview in networked information systems[END_REF][START_REF] Marchal | A Big Data Architecture for Large Scale Security Monitoring[END_REF][START_REF] Zhong | On mixing high-speed updates and in-memory queries: big-data architecture for real-time analytics[END_REF][START_REF] Luo | A Component-based Software Development Method Combined with Enterprise Architecture[END_REF]. In the second phase of the architecture, various supervised and unsupervised algorithms were used to analyze the extracted and enriched spatio-temporal feature space. Temporality of the feature space was determined based on the required decision support frequency of the targeted domain and the application. Fig. 2 shows the concept diagram of the proposed architecture for agricultural decision support system based on true heterogeneous data integration.

Architecture for Domain Knowledge Capturing

In this part of the work we investigated how domain knowledge can be efficiently represented in Linked Open Data (LOD) and how this representation can be incorporated for use with machine learning models. Further, we will investigate how to perform learning, inference and prediction tasks with LOD. The work package aims to demonstrate the value of combining semantic and machine learning representations for environmental modeling. Primary objective of this part of the architecture was to establish a standard protocol for capturing essential domain knowledge in a way that could directly be used in automated reasoning and machine learning based processes. Agricultural domain experts and farmers could potentially define the complete feature space required to optimize the harvesting. Often that could be in a casual format or unstructured know-ledge that makes it inaccessible from the system point of view. It is also true that domain knowledge often get lost in translation from generation to generation due to lack of standard mechanism to capture. In all kind of agricultural decision support system there are some essential environ-mental attributes and prior defined thresholds for those variables. The farmers based on their best experience and business profitability in the past usually predefine these thresholds. They also have some special features (i.e. soil colour to determine soil moisture) that they use in the daily decision making process. Decision making process is always a classification problem with two or more possible solutions. It is also possible to find some historical ground truth data (scattered over long time frame) from the farming companies that could be used for cross validations. Fig. 3 shows this part of the architecture. In this architecture we have applied data driven autonomous hybrid unsupervised clustering techniques (combining Principal Component Analysis (PCA), Self Organizing Map (SOM) and Fuzzy C Means (FCM) to automatically create and match class labels for possible decisions those are defined and stored in farmers mind. Ad-vantage of this autonomous clustering was that this architecture was able to establish few more class definitions. We have defined this part as semantic class identifier. Any of these class labels defining a kind of decision should be related to a set of essential attributes defined by the farmers. We have defined these labels as semantic categories of the decision-making system. Attributes, thresholds and associated decision class could be modeled as a complete association set that was captured as rule in our architecture. This was captured within the relationship constructor module. Four different rule constructors were used to formulate these relationship, namely, fuzzy rule constructor (using fuzzy rule base creation), conditional probabilistic rule constructor (using conditional probability rule generation), order logic constructor (using order logic to formulate rules) and threshold based significant event rule constructor (where threshold of few environmental variables, directly defined by the farmers in conjunction with an event which led to make an unusual decision). In this architecture we proposed to use all or some of these rule constructors based on requirements and availability of ground truth data.

Analytics Driven Big Data Architecture

The i-EKbase" knowledgebase architecture represents semantically enriched spatiotemporal features from the environment in a unified manner. The purpose of this work was to learn spatio-temporal patterns acquired from different knowledge sources, translating the learned knowledge into a more efficient decision support system. In this part we describe the detailed architecture behind the semantic analysis. Data driven machine-learning methodologies were employed for this purpose. Machine learning could be used for autonomous knowledge discovery and online inference of knowledge for the intended application scenarios. This approach included "Unsupervised Learning", "Ensemble Learning", "Deep Learning", and "State Based Learning". The aim was to incorporate various learning algorithms on the cloud based computing infrastructure to provide the analytical power required to capture and integrate interesting patterns from the heterogeneous data sources. Captured knowledge from this framework was also integrated with the domain knowledge to be used in future prediction, knowledge recommendation, and decision support systems.

Design of knowledge integration architecture was motivated by the fact that none of the existing data model integration architectures were capable of handling, processing and analyzing multiple large environmental data sources simultaneously. Database on its own does not carry any weightage unless data is converted into knowledge. Based on the domain specific rule or relationship structure an ontology translator was created for automatic reasoning from the dynamic time series data from the environmental sensor and sensor networks. A task of this translator was to convert the domain knowledge into usable format that could be used in functional block called 'Semantic Signal Translator' (as shown in the Fig. 4).

The block called 'Data Conditioner' was a time series integrator for the semantic signal selector function. Pre-processed time series data were batch processed and represented as daily averaged data for this study. Data from the different sources measuring the same environmental attribute were harmonized. Again different measured attributes from the same node were also harmonized according to the daily average. Simultaneously a generic time window selector function was developed to define the required temporal frequency defined by the domain ontology. Combination of this kind formed a pool of similar variables, which should be able to validate or complement each other in case of missing values in the time series. Complementary method identified the missing value segments of a time series and replaced those segments with an average segment based on other available time series in the same pool. Next a 'cross-correlation technique' was used to measure the similarities between two complemented time series signals representing similar scenarios (in terms of location and time period). Other purpose of this step was to cross validate similar time series data in the same pool to find a representative time series regarding that particular pool. If the two signals being compared were completely identical then the cross-correlation coefficient should be equal to 1 and if there are no similarities between the signals it should be equal to 0. A scoring protocol was designed on cross correlation results. The time series with highest score were selected from each sub group as best representative of the associated environmental variable for that time period. Semantics representations are usually intended as a medium for conveying meaning about some world or environment. One of the significant novelties of this architecture was to implement a generic mechanism to parameterize all possible available metadata into static attributes (i.e. Boolean flag for each of the metadata or a numeric representation with a library of definition). Metadata is "data about the data" and it can provide the description of the what, where, who and how about data. For example, a sensor node metadata could describe when and where the sensor node was deployed, who deployed that node, which environmental attributes are being measured, what are the features or characteristics of that particular sensory system, and finally the valid range of measurement that could be expected. However, metadata are generally used to describe the principal aspect of data with the aim of sharing, reusing, and understanding heterogeneous data sets. In fact, different types of sensor or sensor-simulation model metadata may be considered, namely, static and dynamic sensor metadata and associated sensing information. Other important aspect of the metadata was the inclusion of domain specific decision support class categories and experience based expected outcomes, extracted in the previous domain knowledge extraction block. Parametric representation of metadata provided us with a unique functionality to reason with metadata programmatically in an analytical process. In this process already existing vocabularies (i.e. Geoname, XKOS etc.) and user defined lookup tables were consulted programmatically to refine the parameterization of the metadata [START_REF] Dutta | Recommending Environmental Big Data Using Semantic Machine Learning[END_REF]. The functional block for this part was called 'Metadata Parameterization'. Data normalization was used in the functional block called 'Signal Normalization' to prepare the integrated sensor response matrix for the subsequent signal pre-processing paradigms on a local or global level. Wide range feature selection using machine learning and signal processing techniques, in a domain specific manner was a key element of this architecture.

The next level of functionality was called 'Semantic Space integrator' where integration of newly extracted features from the semantic enriched block was combined with the Meta features. From the domain perspective this was a pivotal point where wide range of observation-based features was meaningfully integrated with domain specific Meta knowledge. At this point the architecture had enough information to formulate a learning process and design a decision support system. The primary focus moved towards the usage of various supervised machine-learning techniques along with unsupervised techniques to learn about semantically enriched data. Four essential functional blocks were implemented for this purpose, namely, 'Training Data processor', 'Objective Identifier', 'Prior Extractor' and finally 'Semantic Structure Learner'. In the hybrid learning process a training set is always required against a ground truth data set as training target. The 'Semantic Structural Learner' functional block was the core body for learning and spatio-temporal analytics. Four different learners were used which were based on 'Deep Network Learning (for low dimensional network weights representing semantic structure)', 'Principal Component Analysis (PCA) -Guided Self Organizing Map (G-SOM) network learning (to train a grid structure to represent the semantic structure)', 'Dynamic Bayesian Network Learning (for learning the structure behind the semantic relationship)' and 'Fuzzy Subtractive Cluster Learner (for learning the natural grouping among the feature space using autonomous clustering)'. The parallelization was implemented to establish a wide range of crossvalidation and complementary learning. An autonomous unsupervised learner algorithm was implemented to select randomized training sets for the learning phase. The 'Training Data Processor' was responsible for this step. The other two functional blocks, the Objective Identifier' and the 'Prior Extractor' were responsible to generate the ground truth based training target for the training phase and also for the testing phase which is called 'Objective Evaluation and Fusion'. Testing of the learning phase was based on evaluation parameters i.e. accuracy, specificity and sensitivity. To process the evaluation and accuracy estimates from the testing of the trained structure learner models a functional block called 'Network Rule Extractor' was implemented. These rules could be structured representation of some existing rules or they could also be new associations derived as part of the learning. The functionality of this block was to identify new rules those are being generated during semantic structure learning processes. The new rules were represented as new knowledge into the original domain ontology marked as 'Input A'. This was done as part of the implemented enrichment block called 'Ontology Enrichment Adaptor'.

The functional block called 'Optimum Structure Exchanger' was implemented to perform this stage of the architecture. This block was constructed based on Resource description framework (RDF), uniform resource identifier (URI) and triple store technologies. Extracted rules were converted into RDF format to be represented on the LOD. This i-EKbase architecture was implemented on the CSIRO's Bowen research cloud infrastructure. In order to establish the effectiveness of this newly proposed heterogeneous big data analytical framework, this paper presents few real life case studies and associated performance factors to highlight the principal achievements of this architecture [START_REF] Dutta | Recommending Environmental Big Data Using Semantic Machine Learning[END_REF][START_REF] Plaisant | Interface and data architecture for query preview in networked information systems[END_REF][START_REF] Marchal | A Big Data Architecture for Large Scale Security Monitoring[END_REF][START_REF] Zhong | On mixing high-speed updates and in-memory queries: big-data architecture for real-time analytics[END_REF][START_REF] Luo | A Component-based Software Development Method Combined with Enterprise Architecture[END_REF][START_REF] Dutta | Deep cognitive imaging systems enable estimation of continentalscale fire incidence from climate data[END_REF][START_REF] Dutta | Performance Evaluation of South Esk Hydrological Sensor Web: Using Machine Learning and Semantic Linked Data Approach[END_REF][START_REF] Ce | Area Wise High Resolution Water Availability Estimation Using Heterogeneous Remote Sensing and Ensemble Machine Learning[END_REF].

Big Data Architecture Based Environmental Case Studies

This case study has three main achievements. Firstly a multi-source environmental knowledge framework (i-EKbase) was developed to provide large-scale availability of relevant sensor-model databases for any environmental application. We have developed lightweight ontologies based on extracted metadata from heterogeneous data sources. Next historic surface water balance for one location in Tasmania, Australia was estimated using unsupervised machine learning knowledge recommendation [START_REF] Ce | Area Wise High Resolution Water Availability Estimation Using Heterogeneous Remote Sensing and Ensemble Machine Learning[END_REF]. The traditional water balance model based method does not include several other environmental parameters (i.e. solar radiation, temperature and humidity), which might have significant influence on the water balance. This issue has been addressed by using multi dimensional and heterogeneous data in i-EKbase system. The machine learning based data driven approach behind i-EKbase system was demonstrated as an effective method to estimate water balance with potentially higher accuracy. Supervised machine learning paradigms were experimented to explore generalization capability and prediction accuracy of this proposed water resource management solution based on multi sensor -model integration [START_REF] Dutta | Performance Evaluation of South Esk Hydrological Sensor Web: Using Machine Learning and Semantic Linked Data Approach[END_REF]. The Radial Basis Function Network based 91.3% accuracy performance proved that newly proposed predictive water resource estimation method based on large multi scale knowledge integration could potentially make the irrigation decision support systems more robust and efficient. Fig. 5(a) shows the area wise water balance estimated using newly developed big data analytical architecture [START_REF] Dutta | Recommending Environmental Big Data Using Semantic Machine Learning[END_REF][START_REF] Dutta | Performance Evaluation of South Esk Hydrological Sensor Web: Using Machine Learning and Semantic Linked Data Approach[END_REF][START_REF] Ce | Area Wise High Resolution Water Availability Estimation Using Heterogeneous Remote Sensing and Ensemble Machine Learning[END_REF]. In a different case study we focused on automatic sensor data annotation and visualisation of dynamic weather data acquired from a large sensor network using this newly proposed big data analytics platform. Aim was to develop a data visualisation method for CSIRO's South Esk hydrological sensor web to evaluate the overall network performance and visual data quality assessment. This visual data quality technique developed from this study could be used for quality assurance of any sensor network (See Fig 5(b)). In another case study, Main aspect of this study was to establish a methodology to predict wild fire prone locations as spots on the Australian map based on publicly available gridded maps of Australian weather variables and hydrological variables. Integrated data sets were created from the gridded maps available from Australian Water Availability Project (AWAP) and Bureau of Meteorology (BOM). On the other hand, NASA-MODIS historical active fire image archives for Australia were used as ground truth. 70% of the data (2008)(2009) were used to train all the neural networks where as the monthly image data from 2010 (30% of the data set) were used to test the networks. Independent training and testing were critical for this study to prove the generalization capability of the hybrid architecture based on the neural networks. 94% overall prediction accuracy was achieved from this approach, with 93% sensitivity and 95.3% specificity. Maximum false positive rate was 0.7% whereas overall precision was 96% (Fig. 6) [START_REF] Dutta | Deep cognitive imaging systems enable estimation of continentalscale fire incidence from climate data[END_REF].

Conclusions

Our understanding of the environment is greatly associated with the interlinked knowledge of the phenomena surrounding to us. Such knowledge is a result of data and extracted information. With the availability of very high and even ultra-high resolution sensor data there is a greater need of managing data, information and essentially the knowledge. With the advent of technological novelties and their wider applications the generated data is surpassing our capacities to store it. There is an urgent need for improved methods and advancement in data-intensive science to retrieve, filter, integrate, and share data. Data and meaningful information are key for the actors in every walk of life, however, how to conceive, perceive, recognize and interpret such data in space and time is a big question and a big challenge. Taking this challenge into the perspective, we have presented an opportunity of recommending environmental big data using machine learning approaches. We have a firm belief that our simple approach will contribute to the body of knowledge in big data study and big knowledge management in this era of data intensive science.

Fig. 1 .

 1 Fig. 1. UML diagram of the proposed architecture.

Fig. 2

 2 Fig. 2 Concept diagram of the proposed architectural overview to integrate big heterogeneous knowledge.

Fig. 3 .

 3 Fig. 3. UML diagram of the domain knowledge capturing architecture.

Fig. 4 .

 4 Fig. 4. UML diagram of the semantic machine learning based big data architecture.

Fig. 5 (

 5 Fig. 5(a). Area wise water balance estimation using multi source heterogeneous data architecture; (b) Dynamic Annotation and Recommendation about the sensor network's node based data quality.

Fig. 6 .

 6 Fig. 6. Bush-fire hot-spot estimation and prediction based on the proposed analytical architecture.


Mark Cheeseman
email: mark.cheeseman@anu.edu.au

Ben Evans
email: ben.evans@anu.edu.au

Dale Roberts
email: ds.roberts@anu.edu.au

Marshall Ward
email: marshall.ward@anu.edu.au

A Performance Study of Applications in the Australian Community Climate and Earth System Simulator

Keywords: climate simulation, Unified Model, performance evaluation

A 3-year investigation is underway into the performance of applications used in the Australian Community Climate and Earth System Simulator on the petascale supercomputer Raijin hosted at the National Computational Infrastructure. Several applications have been identified as candidates for this investigation including the UK MetOffice's Unified Model (UM) atmospheric model and Princeton University's Modular Ocean Model (MOM). In this paper we present initial results of the investigation of the performance and scalability of UM and MOM on Raijin. We also present initial results of a performance study on the data assimilation package (VAR) developed by the UK MetOffice and used by the Australian Bureau of Meteorology in its operational weather forecasting suite. Further investigation and optimization is envisioned for each application investigated and will be discussed.

Introduction

Australian Community Climate and Earth System Simulator Optimization Project

The Australian Community Climate and Earth System Simulator, or ACCESS [START_REF] Keenan | Next Generation Australian Community Climate and Earth-System Simulator (NGACCESS) -A Roadmap 2014-2019[END_REF], is an integrated software system for coupled climate and earth system simulations. It is a joint initiative of the Australian Bureau of Meteorology (Bureau) and the Commonwealth Scientific and Industrial Research Organization (CSIRO) in cooperation with the university community in Australia. It is used extensively by the Australian academic community through the Australian Research Council Centre of Excellence for Climate System Science (ARCCSS) (ARC Centre of Excellence for Climate System Science, 2011). The simulation work performed within ACCESS is so computationally intensive that supercomputers are needed to complete this work at a reasonable rate. One such supercomputer heavily used for ACCESS-related work is Raijin -a petascale supercomputer installed at the National Computational Infrastructure (NCI) in Canberra, Australia. With ACCESS-related research being a core activity, knowing (and improving) the performance of will translate into additional and/or larger simulations that achieve greater scientific output.

To date, no coordinated effort has been done to gauge the current performance of the ACCESS framework at NCI and what could be done to increase it. The ACCESS Optimization project (referred to as ACCESS-Opt henceforth) was created to address the issue. This three-year collaboration between NCI, Bureau and Fujitsu has clear performance-driven goals: 1) determine strategically important ACCESS simulations being run on Raijin (or to be run in the very near future), 2) assess the performance of these configurations on Raijin, and 3) implement upgrades for identified performance issues in these simulations.

ACCESS-Opt provides an opportunity for high performance computing experts to engage with Australian weather/climate researchers. It is hoped that this collaboration will further their scientific deliverables and aspirations with ACCESS.

Raijin Supercomputer

ACCESS simulations performed at NCI use Raijin -an x86 based cluster comprised of 3592 nodes connected by Infiniband FDR interconnect. Each node contains two 8core Intel Sandybridge CPUs running at 2.6 MHz clock speed. At least 32 GB of physical memory is available and shared between the two CPUs on a node. A highspeed Lustre-based filesystem containing approximately ten petabytes of space is available for high-speed storage. Intel FORTRAN and C compilers are used for source code compilation. OpenMPI [START_REF] Gabriel | Open MPI: Goals, Concept and Design of a Next Generation MPI Implementation[END_REF]) is used for messagepassing support in the parallel applications.

Project Methodology

Organizational Collaboration

Cooperation and collaboration between the member organizations is key to ACCESS-Opt's success and methodology. NCI specialists first interact with climate and NWP researchers at Bureau, CSIRO and ARCCSS with the goal of determining which ACCESS applications (and corresponding configurations) should be investigated. The same specialists, with input and assistance from staff at the before-mentioned organizations, then conduct performance assessments and possible optimizations on the selected configurations. The results are then relayed back to the same researchers for dissemination. Wider distribution of the results is made through publications and presentations.

Software Benchmarking

Figure 1 outlines the methodology followed when assessing an application's performance. The first step is to perform what is known as a strong scaling analysis. Here, a particular configuration of an ACCESS simulation is run multiple times. All runs

Strong Scaling Analysis

Profille Summary Trace Analysis Fig. 1. Performance assessment methodology are identical except for the number of CPU cores being used -which is increased at a constant factor (usually two). Ideally, as the number of cores is doubled, the simulation's runtime should be halved. So, if one were to plot the ratio of the observed runtimes at increasing numbers of cores to the observed runtime at the smallest number of cores used, one would get a straight line if the vertical axis is logarithmic with a base of 2. There are multiple examples of such strong scaling plots in the paper stating with Figure 3.

Strong scaling plots generally will indicate the point where adding additional CPUs to a given simulation is no longer prudent. At this point, one needs to take a more detailed look at the application's performance by creating a performance profile. A performance profile summarizes where an application's runtime is being spent. The profile's specificity is up to the investigator -one can record the time spent in general types of activity observed (I/O, communication, computational, etc) or in individual subroutines and functions. Figure 2 shows a basic profile for a data assimilation code ran on Raijin using 384 CPU cores -with each core running a single MPI task. One sees that communication-related activity clearly dominates the application's runtime. Profiling was performed with the open-sourced tool, Score-P [START_REF] Schlutter | Profiling Hybrid HMPP Applications with Score-P on Heterogeneous Hardware[END_REF].

The final step is to generate traces for selected configurations on ACCESS-related jobs. In a trace, snapshots of activity within a running job are gathered at a specified frequency for the duration of the entire job. While a performance profile can indicate which parts of an application take up significant portions of its runtime, a trace can show how and when these "hot-spots" accumulate during a run. Visualization of trace logs is performed using the Vampir software (Overview: Vampir 8.3, 2014).

Preliminary Results

Unified Model

The Unified Model (UM) is a general atmospheric circulation model developed by the United Kingdom MetOffice (UKMO). An overview of its development can be found

MPI calls 41%

Commsrelated 47% compute 12%

Fig. 2. Profile of the data assimilation application at 384 CPU cores

in [START_REF] Brown | Unified Modeling and Prediction of Weather and Climate A 25-Year Journey[END_REF]. UM is used operationally for numerical weather prediction in Australia, New Zealand and other countries. Under ACCESS, it is used for atmosphere-alone simulations and serves as the atmospheric component in coupled climate model configurations.

Initial UM optimization work uses version 8.4 of the UM and has focused on a global N512L70 configuration (eg. ~25km horizontal resolution with 70 model vertical levels) -which we will refer to as UM-N512L70 from this point onwards. This configuration due for use by the Bureau for operational weather forecasting duties and is also well known by a number of supercomputer vendors (including Fujitsu). It uses distributed memory parallelism almost exclusively with the addition of OpenMP threading in certain subroutines and areas such as I/O. I/O is typically a dominant factor in the performance of any earth-system application. UM has adopted an asynchronous IO server approach to improve its I/O performance -particularly in the output of large datasets. In this approach, a subset of MPI tasks being used for the UM job are dedicated to IO activity only -thereby freeing the remaining MPI tasks to concentrate on "normal" computational work. This approach has been successfully adopted in other codes [START_REF] Edwards | Using I/O Servers to Improve Application Performance on Cray XT Technology[END_REF] but requires proper configuration of the IO servers for optimal performance. This finetuning involves manipulating a number of parameters such as the number of active IO servers, number of MPI tasks dedicated to each IO server, frequency at which data is passed to IO servers and so forth. A strong scaling analysis was performed on UM-N512L70 with and without the use of IO servers as shown in Figure 3. All runs lasted one model day. We define strong scaling factor as the logarithmic base 2 of the ratio of the recorded run time at each number of CPU cores to the lowest number of CPU cores used (256 in this case). Strong scaling significantly improves up to 2048 cores with IO servers. At 4096 cores, observed runtime increases again due to the design of the IO server implementation in UM. Output data is replicated and passed between MPI tasks acting as IO servers in order to balance the workload among them. The "least-busy" IO server will output the next datafield to be written while the remaining IO server MPI tasks will flush data belonging to that field from their memory buffer and concentrate on other output datafields. However, this data replication/sharing can become excessive and can stall the overall writing process. A potential remedy would be incorporating MPI-IO into the IO server design. Instead of passing data between the individual IO servers, all the MPI tasks holding out the parts of the output datafield(s) could concurrently write to the same file.

The existing OpenMP implementation in UM is inefficient in a number of areas in the code -particularly in the convection routines. In such areas, 1 or more OpenMP threads are idle for long periods of time. Most existing OpenMP was added for loop and small-scale data parallelism only. Task-wise parallelism would be a more beneficial use of OpenMP and future work will focus on this. Also, some existing OpenMP use is architecture dependent (for the IBM Power architecture). Removing this dependence should yield significant performance benefits for Raijin.

The CPUs used in Raijin are capable of running more than one thread per physical core by using hyper-threading (Intel Corporation, 2014). In certain circumstances, enabling hyper-threading can increase application performance however this was not the case with the UM. Because of the low utilization of present OpenMP threads, hyper-threading is not necessary to effectively deal with extra processes spun up by OpenMPI and other system software.

Additional UM configurations to be assessed include the N768L85 global configuration that is scheduled to be part of the Bureau's next generation operational forecast suite in 2016/17. Performance issues flagged in UM-N512L70 (I/O, OpenMP) are expected to be more severe at this higher resolution. In 2006, the UK MetOffice developed a 4D variational data assimilation (known simply as VAR) scheme for use in its operational weather forecasting suites [START_REF] Rawlins | The Met Office global four-dimensional variational data assimilation scheme[END_REF]. This scheme has since been adopted by other organizations worldwide including the Bureau. In VAR, time series of observational data for certain parameters (such as temperature and pressure) are fitted to data from a Unified Model weather forecast. Fitting the observations involves repeated running of a simplified low resolution of the UM to assess the impact small changes to the model states has on the fit to the observations and to determine new perturbations to improve this fit. These simplified versions of the model are currently run at the horizontal resolutions of N108 and N216 (approximately 120km and 60km respectively) with the same number of vertical levels (70). Performance tests are conducted on version 30.0 of VAR.

Observed strong scaling is persistently good up to 384 MPI tasks as seen in Figure 5. After this point, performance depended on the number of MPI tasks assigned to each multicore CPU on Raijin. Observed strong scaling and associated runtimes continued favorably up to 1536 MPI tasks (the limit for domain decomposition) when only 4 MPI tasks were assigned to each 8-corPU (while still reserving the entire CPU). However, when the CPUs are fully populated with 8 MPI tasks, observed runtimes increased significantly at all MPI task counts as seen in Figure 6. At lower counts (eg, 192 and less), memory bandwidth contention is the main culprit for the increased runtimes. With large computational sub-domains and the use of three and four-dimensional data arrays, cache stalling is prevalent. At larger MPI task counts (over 384), OS jitter becomes apparent. The main control daemon for PBS (the job scheduling software used on Raijin) frequently interrupts the execution on a single CPU core on each node as it checks resource usage. It was observed that OpenMPI would spawn 4 hardware threads per MPI task it created. These threads are used to enable OpenMPI's asynchronous non-blocking communication calls as explained in [START_REF] Wittmann | Asynchronous MPI for the Masses[END_REF]. While they do overload physical CPU cores, we do not believe these additional OpenMPI threads contribute significantly to the observed OS jitter. Enabling hyper-threading (Intel Corporation, 2014) on the Intel CPUs alleviated the OS jitter issue by efficiently scheduling the non-compute processes onto logical threads. Observed runtimes dropped dramatically -at 1536 MPI tasks, there is now only a 12% difference in runtime between using full and half populated CPUs. Table 2 shows the observed run times for VAR as the number of MPI tasks is increased. It also highlights the OS jitter effect apparent when fullyallocated CPUs are utilized on the Raijin supercomputer (and how hyperthreading and under-allocating CPUs are effective workarounds).

Future VAR work includes the assessment of a higher resolution configuration using N320L70 of the outermost loop. No OpenMP was used in the current assessment. Addressing its effective use in VAR will be the target of upcoming work.

Modular Ocean Model

The Modular Ocean Model (MOM) [START_REF] Griffies | A Technical Guide to MOM4[END_REF]) is a general ocean circulation model developed at the Geophysical Fluids Dynamics Laboratory (GFDL) at Princeton University. It is a finite difference application written in FORTRAN90/95 using MPI distributed parallelization. Domain decomposition is performed along the horizontal dimensions only (eg. longitude and latitude). MOM is one of the main earth science models used by the ACCESS user community. It is employed for standalone ocean simulations and serves as the ocean component for coupled climate model configurations. Performance assessment work has targeted version 5.1 of MOM using a global configuration with a 0.25 o horizontal resolution and 50 model levels. Sea-ice representation is done by GFDL's Sea Ice Simulator model [START_REF] Winton | FMS Sea Ice Simulator[END_REF] that is coupled to MOM. Observed strong scaling of the 0.25 o global configuration is shown in Figure 6. The issue of an observed difference in performance between using fully and partially populated CPUs on Raijin is present just like with VAR. The cause of the inflated runtimes (see Figure 8) when using 8 MPI tasks per CPU is the same as with VAR. The observed differences in performance at lower core counts is smaller than that observed with VAR. We be- Profiling showed that a significant number of calls to the MPI_Allreduce collective operation. Code modifications have reduced the frequency of these calls leading to lower runtimes. Introducing land masking (eg. ignoring land-filled grid cells during the integration steps) dropped observed run times another 20%. The final observed run times are given in Table 3 below. Additional profiling on Raijin revealed that approximately 50% of all observed runtime is concentrated in three areas of the model: a) tracer advection, b) horizontal viscosity determination, and c) vertical mixing. These areas will be the focus of renewed optimization efforts in the upcoming year.

Future Work

Optimization work will continue on the applications and configurations previously discussed. NCI, Bureau, ARCCSS and Fujitsu staff frequently meet to discuss the project's status and to explore new goals. Higher resolution configurations are to be assessed for UM (N768L85), MOM (0.1 o horizontal resolution) and VAR (N320L70).

Emphasis will be placed on the performance assessment and optimization of version 1.4 of the coupled climate model ACCESS-CM (Bi, et al., 2013). This model couples UM 9.1, MOM 5.1 the sea ice mode, CICE 5.0 (Los Alamos National Laboratory, 2013) and the land surface model, CABLE 2.0 [START_REF] Kowalczyk | The CSIRO Atmosphere Biosphere Land Exchange (CABLE) model for use in climate models and as an offline model[END_REF] via the OASIS3-MCT library [START_REF] Valcke | OASIS3-MCT User Guide[END_REF].

Fig. 3 .Fig. 4 .

 34 Fig. 3. Strong scaling for the N512L70 global UM configuration The use of IO servers reduces observed runtimes while also improving scalability as shown in Figure 4.

Fig. 5 .

 5 Fig. 5. Observed strong scaling for the N216/108L70 global VAR configuration

  is due to less memory bandwidth contention in MOM. At 1920 CPU cores, OS jitter becomes a serious impediment to performance. The same interrupting behavior from the PBS scheduler is still present. Enabling hyper-threading on the CPUs alleviated the problem just like with VAR.

Fig. 6 .

 6 Fig. 6. Strong scaling results for the 0.25 o global MOM configuration

Table 2 : Measured run times for the N216/108L70 configuration of VAR

 2

			Run Time (seconds)
	# of MPI	4 MPI tasks / CPU	8 MPI tasks / CPU	8 MPI tasks / CPU
	Tasks	(no hyperthreading)	(no hyperthreading)	(hyperthreading)
	24	8751	15033	15315
	48	4342	8057	7629
	96	2180	4002	3828
	192	1158	2000	1821
	384	705	1425	1013
	768	491	1586	941
	1536	471	2440	539

Table 3 Observed run times for the 0.25o global MOM configuration

 3

			Run Time (seconds)
	# of MPI	6 MPI tasks / CPU	8 MPI tasks / CPU	8 MPI tasks / CPU
	Tasks	(no hyperthreading)	(no hyperthreading)	(hyperthreading)
	120	4112	5298	5220
	240	1970	2558	2393
	480	1013	1469	1221
	960	583	1152	707
	1920	419	1810	480

ACCESS-related simulations on Raijin is of great interest. Improved performance adfa, p. 1, 2011. © Springer-Verlag Berlin Heidelberg 2011

Acknowledgements

We wish to thank all the Bureau and CSIRO staff who have assisted us -in particular, Dr. Ilia Bermous, Dr Justin Freeman, Dr Martin Dix, Dr. Simon Marsland, Dr. Vicki Steinle and Dr Peter Steinle. We would also like to acknowledge several individuals from ARCCSS: Dr. Scott Wales, Prof. Andy Hogg and Dr. Nic Hannah.

Finally we extend a special thanks to ACCESS-Opt's steering committee for their continued guidance and support (Ben Evans and Mark Cheeseman of NCI, Robin Bowen and Tim Pugh of the Bureau, Ross Nobes and Tomohiro Yamada of Fujitsu).