Stuart R Mead
email: stuart.mead@mq.edu.au

Mahesh Prakash

Christina Magill

Matt Bolger

Jean- Claude Thouret

A Distributed Computing Workflow for Modelling Environmental Flows in Complex Terrain

Keywords: Structure-from-Motion, photogrammetry, numerical modelling, rapid mass flow, natural hazards

Numerical modelling of extreme environmental flows such as flash floods, avalanches and mudflows can be used to understand fundamental processes, predict outcomes and assess the loss potential of future events. These extreme flows can produce complicated and dynamic free surfaces as a result of interactions with the terrain and built environment. In order to resolve these features that may affect flows, high resolution, accurate terrain models are required. However, terrain models can be difficult and costly to acquire, and often lack detail of important flow steering structures such as bridges or debris. To overcome these issues we have developed a photogrammetry workflow for reconstructing high spatial resolution three dimensional terrain models. The workflow utilises parallel and distributed computing to provide inexpensive terrain models that can then be used in numerical simulations of environmental flows. A section of Quebrada San Lazaro within the city of Arequipa, Peru is used as a case study to demonstrate the construction and usage of the terrain models and applicability of the workflow for a flash flood scenario.

Introduction

Extreme environmental flows and mass movements such as floods, landslides, avalanches and debris flows pose significant risk to exposed populations and can cause substantial damage to buildings, infrastructure and the environment. Due to safety concerns and difficulty in predicting the occurrence of these events, field measurements are rare and, when available, are generally limited to depth and point velocity measurements (e.g. [START_REF] Manville | Breakout Lahar from New Zealand's Crater Lake[END_REF]). As a consequence, computational flow models are commonly employed in an attempt to predict the outcomes of specific events, understand fundamental processes and to provide a greater understanding and delineation of the hazard. Two-dimensional (2D) depth-averaged numerical models such as the shallow-water (SW) method for fluids or Savage-Hutter method for granular mass movements are widely used to simulate environmental flows. These depth-averaged approaches are suitable for predicting large scale flow features and inundation; however the shallowness assumption can be limiting within complex environments (such as urban areas) where there are large and sudden changes in the terrain gradient. Complex and varying topography causes the flow to have three dimensional features and varying vertical velocity profiles which cannot be captured by depth-averaged approaches. For these circumstances, three dimensional (3D) particle based flow modelling methods, such as smoothed particle hydrodynamics (SPH), may be more suitable as they have the ability to predict and track the motion of objects or debris within the flow, model complex flooding scenarios including interaction with buildings, and predict forces on structures [START_REF] Cleary | Discrete-element modelling and smoothed particle hydrodynamics: potential in the environmental sciences[END_REF]. Use of SPH and other 3D methods has traditionally been limited due to their high computational requirements. However, advances in processing power and parallel computing solutions such as OpenMP and Message Passing Interface (MPI) have created opportunities to simulate larger areas at higher resolutions.

In both depth-averaged 2D and fully 3D flow modelling approaches, topographic information (in the form of terrain models) is required as a primary input and modelled outcomes are highly sensitive to the accuracy of this topographic data [START_REF] Williams | Hydraulic validation of two-dimensional simulations of braided river flow with spatially continuous aDcp data[END_REF]. For example, Legleiter, Kyriakidis, McDonald and Nelson [START_REF] Legleiter | Effects of uncertain topographic input data on two-dimensional flow modeling in a gravel-bed river[END_REF] examined the effects of uncertain topographic data on a typical 2D depth-integrated (i.e. Shallow Water) method, finding that the method was sensitive to morphological features such as point bars. Small-scale bed features were also found to affect results, albeit at a reduced level as flow depth increased. The applicability and accuracy of simulation predictions is therefore reliant on accurate reproduction of small scale flow-steering features and topographic obstacles in the terrain model.

While advances in computational power and remote sensing methods have generally increased the dimensionality and resolution of terrain models [START_REF] Javernick | Modeling the topography of shallow braided rivers using Structure-from-Motion photogrammetry[END_REF], quality and accuracy has not necessarily increased at the same pace. Measurements of the terrain are commonly acquired from satellite or aerial platforms using techniques such as Li-DAR, stereo photogrammetry or radar interferometry. This information is then used to create gridded digital terrain models (DTM), a representation of the topography at a single point in time. Frequent data acquisition is critical to ensuring currency of the DTM, as sediment transport processes and human interaction continually modify the terrain and create small scale features that may affect the behavior of flows. The frequency of data acquisition is, however, limited by high computational requirements, costly equipment and lengthy data processing times [START_REF] Javernick | Modeling the topography of shallow braided rivers using Structure-from-Motion photogrammetry[END_REF], resulting in an inaccurate representation of the current shape and features of the terrain. In addition, terrain models can be limited by the acquisition platform. High altitude aerial platform methods are not suitable when the area of interest has large amounts of cloud cover, which is often the case before and after weather based flow events. Tall trees, buildings and bridges may also cause occlusion, leading to inadequate representations of key terrain features.

For high resolution modelling, accurate reproduction of small-scale terrain features and the 3D terrain structure is particularly important, although grid based terrain models are limited in their ability to reproduce these 3D features. Reasons for this have been described by Kreylos, Oskin, Cowgill, Gold, Elliott and Kellogg [START_REF] Kreylos | Point-based computing on scanned terrain with LidarViewer[END_REF] and include:

• Constraints of a rectangular grid system limits the gradient of the slope, and therefore flow direction, to eight cardinal directions. • Re-projection of steep topographic features, such as dips, overhangs or bridges, onto a horizontal plane degrades the level of detail captured, implicitly reducing the resolution. • Gridding can have a directional dependence, and features such as channels that do not align with the primary grid direction can be degraded. Three dimensional terrain representations, such as triangulated irregular networks (TIN) or point based methods [START_REF] Kreylos | Point-based computing on scanned terrain with LidarViewer[END_REF] present a feasible alternative to reduce the limitations of grid-based terrain models as well as more accurately reproducing terrain features.

Here, we demonstrate a workflow to acquire and process detailed 3D terrain models for use in 3D flow modelling and visualisation. The process utilises and integrates several core technologies to provide a fast, reliable and inexpensive method for terrain generation and numerical modelling that eliminates or reduces many of the common limitations of traditional terrain models. Open source libraries and data structures are used to enhance the flexibility of the process and allow users to extend this methodology. The workflow utilises parallel and distributed computing to allow for rapid turnaround time from terrain acquisition to output of numerical simulation results. The key components of the workflow are (1) acquisition of ground images and low altitude aerial images from a light remote control quadcopter, (2) image feature detection, matching and 3D scene reproduction using Structure-from-Motion (SfM) photogrammetry, (3) point cloud processing, including reprojection and meshing; and (4) numerical modelling using SPH. The low cost and high speed of this method allows for faster acquisition of terrain models, increasing their currency, while the point based approach enables features to be more accurately represented. The application of this workflow is demonstrated as a case study by simulating flash flood scenarios in complex terrain for the city of Arequipa, Peru.

Methods

Workflow Outline and Software Integration

The four stages of the terrain generation and modelling process are shown in figure 1.

The first stage, image acquisition, involves capturing images of the terrain from multiple angles, taken either aerially and from ground level. The main objective of this stage is to collect as many images of the area of interest as possible. The Structurefrom-Motion (SfM) stage uses these images to generate a point cloud representation of the area using photogrammetry and computer vision methods including feature identification and matching, camera pose estimation and point cloud reconstruction using multi-view stereo (MVS). The dense point cloud produced by the SfM stage is then processed using algorithms to filter out erroneous points, reduce the size of the point cloud and reconstruct a manifold, watertight surface mesh representing the ter-rain. In the final numerical modelling stage, the terrain mesh model is converted into an input for simulation and subsequent visualisation.

Fig. 1. Terrain generation and modelling workflow outline

The modelling and terrain generation process, shown in figure 1, is sequential with each step relying on inputs from the previous. Aside from the requirements for inputs, each component of the process is independent from each other. This makes the process well suited for implementation into a workflow tool, where each software component can be developed and modified independently. This is particularly important for image feature detection (step 1 in figure 1), surface reconstruction (step 6) and numerical modelling (steps 7-9), as these are all active areas of research where a range of approaches (e.g. fully 3d or 2d depth-averaged numerical models) could be employed and may be more appropriate depending on user requirements. We chose to implement this process (referred to hereafter as the modelling workflow) into Workspace, a cross-platform workflow framework with a plug-in architecture [7]. The plug-in architecture exposes data types and operations, enabling the development of new workflows and operations. Other advantages of Workspace for this application are:

• 3D visualisation capabilities allows for integrated model checking, without the need for additional software packages, • the distributed/parallel execution engine, which allows components of the workflow to be processed on multiple CPU's, both locally and distributed through TCP and cluster based systems, and, • the engine is built upon permissive free software licenses, allowing for extensions and applications to be developed without third party licensing requirements.

Currently image feature matching, point cloud processing (steps 4-6), input generation (step 7) and visualisation (step 9) are implemented within Workspace. The triangulation, bundle adjustment and multi-view stereo processes are run separately, but are executed within the workspace from a command line operation. The SPH simulations (step 8) are run externally using the terrain model generated within the workflow.

2.2

Structure-from-Motion (SfM)

Structure-from-Motion (SfM) is a photogrammetry method that reconstructs sparse 3D points and camera viewpoints from image collections of a scene. Dense 3D point clouds can be reconstructed by coupling with MVS methods, which construct dense 3D point clouds from overlapping images and camera viewpoints [START_REF] Furukawa | Towards Internet-scale multi-view stereo[END_REF]. The accuracy of SfM-MVS approaches is said to be nearly on par with laser scanners [START_REF] Seitz | A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms[END_REF].

To obtain camera viewpoints, the SfM process is as follows: (1) within each image, find and compute descriptors for unique features such as building corners (keypoints);

(2) find matching keypoints in images taken from different angles; (3) obtain an estimate of camera parameters and location through triangulation of the image matches; and, (4) optimise the matrix of camera parameters and locations (bundle adjustment). For a detailed description of the SfM process, see [START_REF] Snavely | Photo tourism: exploring photo collections in 3D[END_REF].

In the workflow, our approach differs slightly from the commonly used methods presented by Snavely, Seitz and Szeliski [START_REF] Snavely | Photo tourism: exploring photo collections in 3D[END_REF] in that we utilise the open source library OpenCV (www.opencv.org) to compute and match features. OpenCV is a BSDlicensed computer vision and machine learning software library with a feature detection and matching framework that allows for a variety of different keypoint extraction and descriptor methods. Currently implemented matching methods are SIFT [START_REF] Lowe | Object recognition from local scale-invariant features[END_REF], SURF [START_REF] Bay | SURF: Speeded Up Robust Features[END_REF] and BRISK [START_REF] Leutenegger | BRISK: Binary Robust invariant scalable keypoints[END_REF]. These provide a range of keypoint descriptors that can be utilised depending on requirements for speed, robustness or commercial purposes (SIFT and SURF are patented methods, freely available for non-commercial use only).

Following the triangulation and bundle adjustment stage of the workflow, the camera pose and sparse point cloud are used to create a dense reconstruction of the scene using MVS. The MVS method used here implements view-clustering to group images in order to reduce memory requirements and reconstruction speed. For details on the view-clustering MVS method used here refer to [START_REF] Furukawa | Accurate, Dense, and Robust Multiview Stereopsis[END_REF].

Point Cloud Processing and Surface Generation

Following dense reconstruction, the point cloud is converted into a structure that can be used in the Point Cloud Library (PCL), a large scale open project for point cloud processing [START_REF] Rusu | 3D is here: Point Cloud Library (PCL)[END_REF]. The PCL libraries contains algorithms for filtering, feature and object detection, processing and smoothing, model fitting and surface reconstruction. Currently, the following PCL operations are implemented in Workspace: (1) normal and curvature estimation using OpenMP; (2) statistical processing and filtering; (3) moving least squares smoothing and upsampling (useful for DTM generation); and, (4) iterative closest point registration. These operations, and others within the libraries can be used interchangeably in the workflow to filter, smooth and transform the point clouds depending on output requirements. As a result, the workflow can be easily modified or extended to have uses beyond the creation of surfaces for numerical modelling.

For simulation of environmental flows, a watertight reconstruction of the surface is required, with minimal artificial perturbations or mesh artefacts such as non-manifold geometry. The main challenge with creating surfaces from the SfM-MVS process is handling non-uniform point density, noise and misaligned points; while being scalable to large point clouds. There are several surface reconstruction methods currently available and research in this area is active. In this workflow, the Smoothed Signed Distance Colored (SSD-C) surface reconstruction method of Calakli and Taubin [START_REF] Calakli | SSD-C: Smooth Signed Distance Colored Surface Reconstruction[END_REF] is used. This method was chosen for its speed and durability, however additional methods may be easily substituted if required.

Numerical Modelling

The reconstructed surface created from the previous steps has a number of potential applications, including visualisation and as an input for numerical modelling. Here, we demonstrate an application for high-resolution numerical modelling within a complicated urban environment using 3D smoothed particle hydrodynamics (SPH). SPH is a versatile, meshless, Lagrangian particle method that has been used to model environmental phenomena such as tsunami, dam breaks, landslide initiation, lava flows and mudflows [START_REF] Cleary | Discrete-element modelling and smoothed particle hydrodynamics: potential in the environmental sciences[END_REF]. SPH is particularly advantageous for modelling complicated flows within complex environments due to the natural handling of complex topography and free surface flows, and the ability to include additional physics such as non-Newtonian rheology and entrainment of dynamic objects.

In the workflow, the reconstructed surface is converted into a data structure that can be read by the SPH software, which is run externally to the workflow due to the long computational time (in the order of weeks to months) required for high resolution simulations. Once completed, the data can be analysed and visualised within Workspace using separate data analysis workflows.

Case Study: Arequipa, Peru

To demonstrate the applicability of this method, we present a case study of the workflow for Arequipa City, Peru. Arequipa's central business district is located approximately 17 km south-west of El Misti, a steep potentially active volcano. The city is exposed to flash floods and lahars (volcanic debris flows) due to climatic conditions and the city's proximity to El Misti. Areas along the main quebradas (ravines) of San Lazaro and Huarangal are at risk from flash floods and lahars during the rainy season or after eruptions [START_REF] Thouret | Combining criteria for delineating lahar-and flash-flood-prone hazard and risk zones for the city of Arequipa, Peru[END_REF]. For example, flash floods in February 2011 destroyed 20 houses and caused damage to another 400, mostly along the quebradas and steep slopes. Much of this damage was attributed to heavy rainfall or breakout floods caused by infill obstacles such as makeshift crossings [START_REF] Thouret | Combining criteria for delineating lahar-and flash-flood-prone hazard and risk zones for the city of Arequipa, Peru[END_REF]. High resolution terrain models are required to represent the small scale of these features, while the rapidly changing environment requires frequent updating to accurately represent the topography for hazard simulations. This area was therefore chosen as a case study to demonstrate the value of the modelling workflow.

Image Collection

An approximately 200 m long and 200 m wide area of Quebrada San Lazaro was studied and is shown in figure 2a, looking upstream from a road bridge. The area in question contained the bridge and an upstream bend in the channel, which are thought to create overbank flow and strong 3D flow features that are ideal to test the use of SPH. In total, 2000 images of the area were acquired in September 2013; 600 images were taken from the ground and 1400 aerially. The aerial images were obtained from a DJI Phantom, a small (35 × 35 × 19 cm), lightweight (take off mass < 1 kg) quadcopter with an attached camera.

The objective of the image collection stage was to acquire as many images as possible from multiple angles to ensure all features were reproduced correctly and that there was minimal occlusion. A quadcopter is ideal in these circumstances, as its light weight and maneuverability means that it can be flown in close proximity to buildings to acquire images from multiple angles. In other areas (e.g. larger regions will less changes to terrain) a higher, more stable, but less maneuverable, aerial platform such as a helium balloon may be more suitable.

Feature Matching, SfM and MVS

Image features for the case study were detected and matched using SIFT feature descriptors. In an unordered image set, such as the one used in this case study, each image can potentially match each other image, meaning that image matching has O(n 2 ) complexity. To reduce the time taken to match images, we parallelised this portion of the workflow, enabling matches to be determined on local and external CPU cores using the Workspace parallel execution engine. A speedup of 600% was achieved using parallel processing on 12 CPU cores (at 2.93 GHz), the speedup of the parallel implementation in this case is limited by the file I/O. Following image matching, the triangulation, bundle adjustment and multi-view stereo processes are run separately using VisualSFM [START_REF] Wu | Towards Linear-Time Incremental Structure from Motion[END_REF], which utilises the CMVS-PMVS method of Furukawa and Ponce [START_REF] Furukawa | Accurate, Dense, and Robust Multiview Stereopsis[END_REF].

Mesh Generation

The SfM-MVS process created a dense point cloud containing ~8.5 million nodes. An image of the point cloud taken from a similar view angle to figure 2a is shown in figure 2b, where the point cloud model appears to correspond well with the real topography. The density of the raw point cloud varied due to occlusion and the number of images taken of a particular area. The maximum point cloud density was approximately 250 points in a 0.25 m 2 area. However, for this application point densities of this magnitude are much higher than required and the point cloud was reduced by 10% to have a maximum of 1 point within a 1 cm radius (7.6 million points over the entire case study area). The filtered point cloud was then reconstructed into a watertight, manifold mesh using SSD reconstruction, shown in figure 3. For this example, the entire point cloud processing workflow took 10 minutes; however the amount of required memory (~10Gb) may limit the scalability of SSD reconstruction to much larger areas.

SPH Simulation

Multiple SPH simulations examining the effect of flash flood flow rates on inundation, flow structures and velocities were run, using the terrain model generated from the previous steps. Computational time for the simulations increases by the cube of particle spacing and can therefore be reduced significantly through increasing particle spacing and simplification of the terrain. Terrain resolution and particle spacing was chosen here to be the largest resolution size of the smallest features that might affect the flow. Input flow rates of 25, 50 and 150 m 3 s -1 were considered, with a particle spacing of 12.5 cm. In the highest flow rate scenario, this required 2.8 million fluid particles to be simulated, which took 2 weeks of runtime to simulate 150 seconds of inundation on a 12 core, 2.93 GHz processor. Figure 4 displays the velocities and inundation patterns 40 seconds after the flash flood was initiated for each flow rate. The fluid is shaded by velocity, with red being 10 ms -1 and blue being 0 ms -1 . The inundated area and velocities increase with flow rate, which is expected given the larger volume of water in the flow. The shape and magnitude of some flow structures are also different for the three flow rates. The effect of small features in the terrain is largely invisible in the highest flow rate scenario, with the flow mainly being guided by the bend in the channel. The effects of smaller features on the flow are noticeable for the 50 and 25 m 3 s -1 scenarios. For example, the small access road into the channel (circled in figure 4) causes hydraulic jumps to form in the lower flow rate scenarios, but the high flow rate scenario is largely unaffected by this feature. This demonstrates the necessity of detailed 3D representations of terrain and modelling, particularly when considering smaller, high frequency flooding scenarios.

Conclusion

The workflow presented here describes a process for generating 3D terrain models by utilising advances in structure-from-motion, multi-view stereo, point cloud processing and surface generation. The process is inexpensive, rapid, captures fine scale features and accurately represents 3D structures within complex environments. The parallel processing of image matching ensures the workflow is scalable to a large number of images. By reducing the cost and acquisition time, terrain models can be generated at a higher frequency, better capturing transient small-scale features such as flow blockages and obstacles that can affect flow behaviour. Data processing is achieved through the use of a modular workflow engine, which allows for new methods to be implemented as research into particular areas progresses. The flexibility of the workflow engine, in addition to the usage of open source libraries and data structures provides opportunities beyond the use case demonstrated here, as the process and data generated may be used in various visualisation, terrain analysis and risk assessment fields of study. The use and an application of this workflow was demonstrated using the case study area of Arequipa, Peru, where the resulting terrain model was used to predict the impacts of flash flood events using smoothed particle hydrodynamics. In addition to the presented numerical modelling application, the resulting terrain models can be used in a variety of applications such as visualisation, building classification and vulnerability estimation.

Fig. 2 .

 2 Fig. 2. (a) Case study area in Quebrada San Lazaro and, (b) dense point cloud reconstruction from the SfM-MVS.

Fig. 3 .

 3 Fig. 3. Watertight surface reconstruction of the case study area using SSD (looking downstream towards bridge).

Fig. 4 .

 4 Fig. 4. SPH simulations of a flash flood in Quebrada San Lazaro with flow rates of (a) 150, (b) 50 and, (c) 25 m 3 s -1 . White circle highlights small road seen to affect flow structures in lower flow rate simulations.

Mead and, p. 1, 2011. © Springer-Verlag Berlin Heidelberg 2011


James Hilton
email: flagshipjames.hilton@csiro.au

Claire Miller

Matt Bolger

Lachlan Hetherton

Mahesh Prakash

An Integrated Workflow Architecture for Natural Hazards, Analytics and Decision Support

Keywords: simulation, workflow, natural hazard

We present a modular workflow platform for the simulation and analysis of natural hazards. The system is based on the CSIRO Workspace architecture, allowing transparent data interoperability between input, simulation and analysis components. The system is currently in use for flooding and inundation modelling, as well as research into bushfire propagation. The modularity of the architecture allows tools from multiple development groups to be easily combined into complex workflows for any given scenario. Examples of platform usage are demonstrated for saline intrusion into an environmentally sensitive area and image analysis inputs into a fire propagation model.

Introduction

Computational simulations of natural hazards such as floods and fires can now be performed with high accuracy at speeds greatly exceeding real-time. Analytics from such models can give valuable prediction information to decision makers in the event of a crisis, aiding decision support, or allow detailed risk assessments to be performed on vulnerable regions. Such natural hazard models have specific architectural needs, such as the ability to handle large data sets, run complex computational methods and perform detailed ensemble analysis over the results of these models. A typical workflow in a natural hazard modelling process is shown in Fig. 1. This can be subdivided into a series of stages, which are:

1. Determination of the requirements for the given scenario, such as prediction of risk or impact assessment. The requirements for the scenario decide the input data requirements for the model, as well as the parameter space over which the model will be run. This input data may need to be gathered from local or remote data repositories and may be a static or real-time data source.

For natural hazards this data may include items such as digital elevation models, wind predictions or measurements, precipitation levels, bathymetry measurements or land usage maps.

2. Generation of formatted data suitable for the model. The data is quality checked, cleaned, processed and converted into a format required for the model. For geophysical data this stage may involve processing such as filling missing values in data sets or geodetic re-projection into a common datum or format. The required parameters or sample space for the model are also generated during this stage. Examples of such parameter spaces include ranges of probable tidal extents for inundation scenarios, or the degree of variation in fuel load in bushfire scenarios.

3. Simulation of the given scenario over the parameter space using the cleaned, formatted data sets. One instance of an underlying computational solver is usually run for each element in the parameter space. These instances may be run sequentially or in parallel on local or remote compute nodes. Each instance requires the ability to read and write large data sets to storage which may be local or remote relative to the instance.

4. Synthesis and analysis of the multiple raw data items produced by the computation. This can be an automated or manual processes depending on the scenario, such as risk identification from spatial maps, statistical ensemble analysis or visualization of the results. In some applications, such as realtime scenarios with rapidly changing input conditions, the results from the final stage can be used to inform the choice of parameter space for subsequent simulations of the scenario.

The requirement for an end-to-end solution for these stages has led to the development of an integrated platform for modelling natural hazards. The aim of this platform is to provide all of this functionality in a user-friendly and rapidly configurable environment for different types of natural hazard. Once a particular scenario is defined, and the corresponding workflow has been constructed, the workflow can be run on multiple operating systems and hardware types, ranging from desktop systems to GPU based supercomputers. If required, interaction with the workflow can be exposed through a custom graphical user interface. The architecture behind this modular framework for natural hazards is outlined in the following section, followed by examples of two current use cases.

Software Architecture

A schematic diagram of the architecture behind the framework is shown in Fig. 2. The natural hazard solvers sit within the CSIRO Workspace environment, which is based on the Qt framework. There are currently two operational natural hazard solvers: SWIFT, which is a hydrodynamic modelling tool for inundation and flooding and SPARK, which is a fire perimeter propagation tool. CSIRO Workspace is a cross-platform, extensible workflow and application development framework which handles three main elements; data objects, operations, and user interface elements or widgets [1]. As well as natural hazards, Workspace has been applied to a number of other scientific areas including quantitative imaging, additive manufacturing and industrial process modelling.

In Workspace, workflow stages are encapsulated into individual Workspace operations with multiple input and output data objects connected together into a dependency graph. Data is passed in-memory between operations, allowing efficient, transparent communication of data through the system. This data can be inspected and interacted with at any stage, including during execution, with various data-type specific widgets. The plugin architecture exposes data types, operations and widgets to the framework through shared libraries. Workspace plugins are shared libraries (.dll, .so or .dylib) which act as containers, exposing components to the framework. The Workspace editor includes code generation wizards to create stub code for new components and utilizes CMake for cross platform compilation. Development of complex user interfaces is supported with easy drag and drop designing within Qt Designer allowing any input or output throughout a workflow to be visualized or controlled from a simplified, abstracted user interface. Workspace provides a large library of useful data types, operations and widgets out-of-the-box, such as image manipulation, visualization, mesh processing, database support and plot generation. Workspace itself is written predominantly in C++ for performance and scalability and leverages the full power of the Qt toolkit providing a library of underlying capabilities for developers to leverage such as threading, file IO (XML, JSON etc.), networking, image processing, and 3D visualization using OpenGL.

A number of open-source modules have been exposed within the framework. These include OpenCV for image analysis, NetCDF for large data file support and GDAL for geospatial datasets (including remote OGC WMS and WCS web-services). Scripting is included in the form of ECMAScript and Python for non-C++ developers and calls to the statistical package 'R' are also available. Parallel APIs, such as OpenMP, are fully supported and the current bushfire and flood solvers both use OpenCL for GPU computation. Any other language, toolset or API with C bindings can also be incorporated. Workspace also features a distributed execution capability allowing it to farm out work to local CPU cores, remote machines via TCP and cluster job systems. This feature allows easy distribution of simulation jobs to PBS-based clusters and can be easily extended to support other job systems.

Workspace shares many similarities with other commercially available and open source workflow tools such as Taverna [START_REF]Taverna Workflow Management System[END_REF], Trident [START_REF]Project Trident: A scientific workflow workbench[END_REF], KNIME [START_REF] Knime | [END_REF], Kepler [START_REF]The Kepler project[END_REF][START_REF] Altintas | Kepler: an extensible system for design and execution of scientific workflows[END_REF], Simulink/Matlab [7] and LabView [START_REF]LabVIEW System Design Software[END_REF]. However, the Workspace framework has a stronger focus on being an application development platform rather than simply a workflow editor. It offers a natural path from early code development as part of a research workflow right through to the development of standalone applications for deployment to a collaborator or external client. Inbuilt support for automated testing, cross-platform installer generation and native user-interface development are some distinguishing features supporting this focus. Workspace is designed to be generically applicable to any kind of dependency-based workflow problem, and is easily extendable. It is not restricted to a particular domain, or particular type of workflow, and developers can easily extend its functionality by writing plug-ins. Some workflow engines (such as KNIME and Kepler) are similar in this way, while others have more domain specific focus (such as LabView for measurement and control systems and Simulink for simulation and model-based design). The system also transfers data inmemory between operations as opposed to using local storage or a database as the intermediary between operations, as implemented in workflow engines that focus on web-service connectivity such as Taverna and Trident. In-memory operation is very suited to the computationally intensive needs of natural hazard modelling, in contrast to local storage which can severely impede performance. Workspace is also the only workflow tool which can execute in continuous mode. In this mode, any modification to an operation input triggers an immediate downstream update of the workflow, in-cluding all relevant visual outputs. This event-driven approach enables users to immediately visualize factors such as changes to their input conditions or post processing steps.

Workspace is built on free software license tools, such as the Qt framework. This allows researchers and developers to freely build and distribute applications based on their workflows, or integrate them into existing software packages. Workflow-based application development like this is not well supported in open source available alternatives while commercial packages such as Simulink/Matlab and LabView require paid licenses. Furthermore, although packages such as Simulink/Matlab also provide user interface customization features, Workspace's tight integration with the Qt framework enables developers to quickly and easily develop user interfaces for their workflows. A number of specialized Qt widgets have been developed specifically for natural hazard operations, such as interactive time-series inputs and two-dimensional layer views. These can be directly built into a user interface using the Qt Designer tool and linked with an underlying workflow. Furthermore, a design goal of Workspace was for computational fluid dynamics applications, and it therefore offers far greater interactive 2D and 3D scientific visualization capabilities than alternative workflow engines. An example workflow is shown in Fig. 3 within the Workspace editor GUI. Individual workflow elements, or operations, are represented as colored triangular blocks which are logically joined in the GUI by dragging output data from an operation to the input of another operation. The joins, and the direction of data flow, are shown as black connection arrows in the GUI. In this simple example some text data is created ('Text' block) and passed into an operation to output the text to a log ('Log Text' block). A final block acts as an operation representing the end of the workflow ('End' block). This example passes string data, but new data types can easily be defined and added. For the natural hazard solvers both two-dimensional gridded data sets and onedimensional time series data sets were added as new data types within Workspace.

The workflow is saved as xml, and once a Workspace workflow has been developed it can be easily shared between collaborators. Workspace supports execution both from the GUI shown in Fig. 3, execution in batch mode from the command-line, or an API for stand-alone Qt-based applications.

The computational solver elements are implemented as individual operations. Each solver operation requires a set of configuration parameters specific to the particular operation. These parameters consist of items such as the total time for the solver to be run, how often to output simulation data and global physical variables for the solver. The solver inputs also include two dimensional input data sets for the starting conditions as well as two dimensional layers such as topography and land usage. The solver executes for a specified period before pausing and returning control back to Workspace. Once control is returned, Workspace operations outside the solver have access to internal data from the solver and can be used to analyze, output, display or collate the data. Once these steps are complete, control is returned to the solver which resumes, runs again for the specified period, then returns control back to Workspace. This loop is carried out until the solver reaches the specified total simulation time, at which point Workspace ends execution.

The solver operations incorporate an additional feature, called processor modules, which are exposed as a subclass of Workspace operations. These processors are subsolvers with full access to the solver data and are used to provide specific capabilities for any given scenario. The processor modules are separate operations which plug into a corresponding solver. Multiple processors can be connected to a solver, which transparently controls execution and data management within each processor. Examples of these processors for the hydrodynamic solver, SWIFT, include providing rainfall conditions, applying tidal boundary conditions, infiltration effects, evapotranspiration and coupling with a one-dimensional pipe network model. These processors can be developed separately from the solver, which was a key requirement in the design of the solvers as open platforms for collaboration.

Workflow for Saline Intrusion

The hydrodynamic model SWIFT is based on a finite-volume shallow water formulation, which is both conservative, positivity preserving and well-balanced [START_REF] Kurganov | A Second-Order Well-Balanced Positivity Preserving Central-Upwind Scheme for the Saint-Venant System[END_REF]. The project in this example was designed to examine the effects of saline intrusion from potential sea level rise on environmentally and culturally sensitive areas of Kakadu National Park in the Northern Territory, Australia. Multiple tidal cycles were required to ensure adequate saline mixing with the freshwater from the catchments, so the simulations were run for a period of 30 days. The simulation domain was an area of approximately 150 km × 120 km, encompassing the major catchments of the park, at a resolution of 60 m. The Workspace GUI in Fig. 4 has been arranged with input operations on the lefthand side, the solver at the center and a 3D output of Kadadu national park on the right hand side. The inputs for this model consisted of a digital elevation model (DEM) for the bathymetry and topography as well as the initial salinity state. These were supplied as files in a standard ESRI ASCII format. The 3D output was configured for this example to show the textured digital elevation model with a water layer colored by saline concentration. The processor modules are shown as the set of upper red boxes. These included a processor module to apply the tides to the northern boundary, a module for rainfall based on a time series from measured data, a module for evapotranspiration and a module for advective-diffusive saline transport and mixing.

The actual simulations for the project were run in batch mode using PBS on the CSIRO Accelerator Cluster, rather than through the interactive GUI shown in Fig. 4. The simulations took 42 hours to complete on a Tesla K20 GPU, with an average simulation time-step of 0.7 s. Data from the simulation was written to disk in a binary format and an analysis step was performed on this data using a second workflow to provide various time-averaged metrics such as the maximum, minimum and median water levels and salinity concentrations over the 30 day simulation. These metrics are currently being used for an impact assessment of sea level rise on freshwater species and cultural sites in the region for the NERP Northern Australia Hub.

Workflow for Bushfires

The scenario in this second example is a comparison between a controlled experimental fire in a plot of cured grass and a level-set based bushfire computational model, SPARK [START_REF] Hilton | Incorporation of variation into wildfire spread models using a level set approach[END_REF]. The level set method is a computational model for the propagation of an interface [START_REF] Sethian | Evolution, implementation, and application of level set and fast marching methods for advancing fronts[END_REF], and can be applied to numerous systems such as crystal growth, droplet dynamics and bubbles. SPARK uses the level set method to track the interface between regions in a fire that are burnt and un-burnt. The level set method has several advantages over other computational methods for interface propagation; it is raster based and therefore highly suited to GPU acceleration, it naturally handles the merging of interfaces and the interface can be given an arbitrary outward speed at any point.

The outward speed of the interface in SPARK is equal to the rate of spread of the fire. The functional form of rates of spread in real fires are known to depend on several factors relating to fire propagation including the wind speed, wind direction, fuel and moisture levels and slope of the ground. Many empirical expressions have been given for various fuel types [START_REF] Gould | Project Vesta -Fire in Dry Eucalypt Forest: Fuel Structure, Dynamics and Fire Behaviour[END_REF]13], but the functional form for rates of spread in two dimensions is currently an active area of research. SPARK is currently being used as a research tool for investigating new two-dimensional rate of spread empirical models based on recorded fire data. The experiments shown in this example are number of large-scale fire experiments were carried out in Ballarat, Victoria in Jan 2014. These experiments measured the speed and shape of fire propagation over flat fields of cured grass. The wind speed and direction during the fire were recorded during the experiment, and the progression of the fire perimeter was filmed using camera mounted on a quad-copter. Each of the stages in this example consists of a number of self-contained processes which operated on similar data types. Using our architecture, all stages can be directly processed from start to end within the same framework. These were logically chained together in the workflow environment for each image processing step. An operation could then be used to convert the output image directly into the initial conditions required for the bushfire computational solver after the image processing steps were complete. Finally, data from the solver was visualized for prediction of the fire perimeter. As well as a research tool, SPARK is currently being built into an operational predictive model using published empirical expressions for fire rates of spread over grassland and Eucalypt forest.

Conclusion

The combination of computational solvers and the CSIRO Workspace framework gives a flexible, modular platform for modelling natural hazards. All of the steps in a typical natural hazard simulation, from reading and cleaning input data, simulation, analysis and visualization, have been implemented as individual operations on common data types. These operation can be either be contained within a single end-to-end workflow or, dependent on requirements, split into sub-workflows for pre-processing, execution and post processing. The Workspace framework has a number of advantages over existing workflow engines. These include in-memory data transfer between operations, which is advantageous for the complex, real-time applications and computationally intensive simulations found in natural hazard modelling. The framework also allows workflows to be run in an interactive continuous mode. In fast natural hazard models, such as SPARK, this interactivity enables conditions to be changed on-the fly. Alternatively, workflows for detailed simulations, such as the SWIFT application in this paper, can be executed in a batch mode on high performance computing nodes.

The architecture has been demonstrated in this paper for two very different applications, environmental impact analysis from saline intrusion and image processing of real fire events as input into a fire model. These applications represent only a small subset of the potential applications for the framework. Environmental modelling workflows can be rapidly tailored for any natural hazard application, such as operational usage from data streams or detailed simulations for environmental modeling. These workflows can, if required, be packaged and run behind a Qt user interface and freely distributed by virtue of the permissive software licenses on which Workspace is built. Future work includes adding further solver components to the system, including models for dust advection, granular dynamics and mudflows. However, the open nature of the architecture allows other users and research groups to add any new computational models to the system. Alternatively, processing units can be constructed and freely added to run sub-models on any of the existing natural hazard models.

We have demonstrated the potential of this system within this paper, and expect many applications for this system in the future within the environmental modelling space.

Fig. 1 .

 1 Fig. 1. Stages in modelling a natural hazard event.

Fig. 2 .

 2 Fig. 2. Architecture of the CSIRO Workspace and natural hazard solvers.

Fig. 3 .

 3 Fig. 3. A simple example of a workflow in the Workspace GUI.

Fig. 4 .

 4 Fig. 4. Example workflow configuration for environmental impact assessment of saline intrusion in Kakadu National Park, Australia.

Fig. 5 .

 5 Fig. 5. Stages for comparison of experimental data to simulation results.


Mauno Rönkkö
email: mauno.ronkko@uef.fi

Okko Kauhanen
email: okko.kauhanen@uef.fi

Markus Stocker
email: markus.stocker@uef.fi

Harri Hytönen
email: harri.hytonen@vaisala.com

Ville Kotovirta
email: ville.kotovirta@vtt.fi

Esko Juuso
email: esko.juuso@oulu.fi

Mikko Kolehmainen
email: mikko.kolehmainen@uef.fi

Quality Control of Environmental Measurement Data with Quality Flagging

Keywords: quality control, quality flagging, enterprise service bus, environmental data, built environment

We discuss quality control of environmental measurement data. Typically, environmental data is used to compute some specific indicators based on models, historical data, and the most recent measurement data. For such a computation to produce reliable results, the data must be of sufficient quality. The reality is, however, that environmental measurement data has a huge variation in quality. Therefore, we study the use of quality flagging as a means to perform both real-time and off-line quality control of environmental measurement data. We propose the adoption of the quality flagging scheme introduced by the Nordic meteorological institutes. As the main contribution, we present both a uniform interpretation for the quality flag values and a scalable Enterprise Service Bus based architecture for implementing the quality flagging. We exemplify the use of the quality flagging and the architecture with a case study for monitoring of built environment.

Introduction

Environmental measurement and monitoring has been a growing trend for the past decade [START_REF] Messer | Environmental Monitoring by Wireless Communication Networks[END_REF]. It is needed for instance for assessing the negative impact of human activities to the environment [START_REF] Cyranoski | Fukushima impact is still hazy[END_REF][START_REF] Gilbert | Drug waste harms fish[END_REF]. Environmental measurements, however, are prone to external variation and even disruptions. Therefore, raw measurement data must always be somehow preprocessed before it can be used in computations as an input.

There exist standards for the representation of environmental data. For instance, the Open Geospatial Consortium provides standards for the representation and access of the spatial data. However, the standards do not address the issue of data quality comprehensively. For instance, UncertML [START_REF] Williams | Uncertainty markup language (Un-certML)[END_REF] was proposed as an extension to OGC, to address uncertainty representation. With UncertML, one can attach probabilistic uncertainties to environmental data sets. Still, standards do not provide sufficient support, for instance for real-time quality control of environmental data, as discussed in Section 2.

Quality flagging is a means to provide quality information on the level of individual measurement data points both in real-time and off-line. Most importantly, quality flagging is also a reversible activity, as it preserves all original measurement values. As discussed in Section 2, we focus on one specific quality flagging scheme. It is the scheme presented by Vejen et al. [START_REF] Vejen ; ) | Quality Control of Meteorological Observations[END_REF] that is recommended and used by the Nordic meteorological institutes. Since the quality flagging scheme by Vejen et al. is tailored for weather measurement data, we propose as part of the main contribution a uniform interpretation for the quality flag values to be used for the flagging of any kind of environmental measurement data. It should be noted that the quality flagging scheme by Vejen et al. is not what World Meteorological Organization (WMO) refers to when speaking of a Quality Management Framework. In particular, WMO strives after an ISO certification, whereas the quality flagging scheme is a technical implementation of a real-time and off-line computational quality procedure.

As the other part of the main contribution, in Section 3, we present an Enterprise Service Bus (ESB) [START_REF] Chappell | Enterprise Service Bus: Theory in Practice[END_REF] based architecture to perform quality flagging in a scalable and measurement device independent manner. In Section 4, we illustrate the use of the ESB based architecture to perform quality flagging of data for built environment, including room temperature and water consumption measurements. As the research work is still ongoing, we present here our complete plans. We conclude in Section 5.

Quality Control of Environmental Measurement Data

When considering quality control of environmental measurement data, there is practically one well-known proposed standard, UncertML [START_REF] Williams | Uncertainty markup language (Un-certML)[END_REF]. With UncertML, one can attach probabilistic uncertainties to environmental data sets to support statistical preprocessing. For instance, Williams et al. [START_REF] Williams | Automatic processing, quality assurance and serving of real-time weather data[END_REF] used UncertML to attach uncertainty information to raw weather data as provided by Weather Underground1 . By using UncertML and INTAMAP [START_REF] Williams | Supporting interoperable interpolation: the INTAMAP approach[END_REF], they were able to estimate the bias and residual variance, to adjust, merge, and interpolate temperature data from independent data sources. As a result, they were able to produce an interpolated temperature map for the whole UK based on the Weather Underground data with statistical corrections.

Although UncertML does provide means to improve the quality of environmental data, it operates on the level of measurement data sets. Such level of data quality, however, is not sufficient for all applications. A complementary approach is to per-form quality control on the level of individual measurement data points. For this purpose quality flagging is used. Table 1. Quality flag values and their original interpretation [START_REF] Vejen ; ) | Quality Control of Meteorological Observations[END_REF] The Nordic meteorological institutes have developed a fully functioning quality flagging scheme as discussed by Vejen et al. [START_REF] Vejen ; ) | Quality Control of Meteorological Observations[END_REF]. It provides both real-time and offline quality flagging. Vejen et al. [START_REF] Vejen ; ) | Quality Control of Meteorological Observations[END_REF] distinguish between four quality control levels. QC0 is a real-time quality control performed by the measurement devices or stations. QC1 is a real-time quality control performed by the data acquisition system prior to storing the data. QC2 is an off-line quality control performed by the data management system based on the stored data. Lastly, HQC is the final off-line quality control check performed by a human operator. Each of these levels use the same quality flag values as indicated in Table 1. Thus, the quality flag is a number with four digits: C=E QC0 +10×E QC1 +100×E QC2 +1000×E HQC , where each E QC0 , E QC1 , E QC2 and E HQC are quality flag values for the corresponding quality control levels.

Because the quality flagging scheme by Vejen et al. is designed for weather measurements, it does not apply to generic environmental measurement data. In particular, the original interpretation can be non-informative or misleading in a generic case. Also, the original flag values do not support observations of a malfunctioning measurement device that produces constant, "frozen", or clearly erroneous measurement values. Therefore, we propose a generic interpretation for the quality flag values, as indicated in Table 1. The proposed interpretation is downwards compatible with the original interpretation, so that it could also be used for weather measurements. In particular, in the generic interpretation "suspicious value" and "anomalous value" are used instead of "small difference" and "big difference". Also, "imputation" is used instead of "interpolation", as interpolation may not be applicable in a generic case. Similarly, the generic interpretation replaces "calculated value" with "corrected value" to emphasize the difference between value correction and missing value imputation. Lastly, the generic interpretation uses the two originally unused flag values for diagnostics, to indicate a clear measurement error or a "frozen" measurement value.

It should be noted that quality flagging is a complementary approach with respect to use of UncertML. In particular, quality flagging operates on the level of individual measurement points, whereas UncertML operates on the level of data sets. Thus, both can be applied to the same data set at the same time to provide detailed information about the data quality. What makes the quality flagging an attractive approach is that it supports quality restrictions during the data queries. For instance, one can query only such data, where the final quality check has been performed. Similarly, one can query data, where there are no QC0 or QC1 failures or corrections. Implementing such queries requires no extra work, as they can be constructed based on the quality flag values. Such a query style is supported, for instance, by all SQL databases. Furthermore, queries about failures, suspicions, and corrections provide valuable information to be used with UncertML. In particular, information about bad quality can be used to select and fine tune appropriate statistical and probabilistic model for Un-certML, to match the observed data quality.

Quality flagging provides also valuable information to systems diagnostics and maintenance. The frequency and trend of quality failures function as indicators for device failures or model inadequacies. Thus, the bigger the measurement network is, the more useful and valuable quality flagging becomes. This is something that is not currently addressed by methods such as UncertML that focus on interoperability at the level of datasets.

3

The Enterprise Service Bus Based Architecture

As depicted in Fig. 1, the role of the ESB is to pass measurement data as messages between services. More specifically, we use here WSO2 ESB 2 . The ESB is extended and configured so that it has a dedicated port and a mediator for each sensor. Thus, a sensor performs QC0 on the measurement data after each measurement and sends the data in its native format, such as JSON or XML, to the dedicated ESB port. The ESB then redirects the received measurement data to a dedicated mediator that performs QC1 and passes the checked measurement data back to the ESB. The ESB then redirects the checked measurement data to a data storage. The ESB is configured to trigger QC2 on stored data on regular intervals. The actual QC2 is then performed by a computational service, for which we use Octave3 . For this purpose, a predetermined subset of the stored data is retrieved for the computational service. After QC2, the checked data is stored back to the data storage. Lastly, HQC is performed by a human operator on the data that is already checked by QC2. HQC is initiated by the human operator through a dedicated client application. The client application accesses the stored data requested by the operator through the ESB. Similarly, after HQC, the client application stores the checked data back to the data storage through the ESB. The advantage of the ESB architecture is that it can be reconfigured by an administrator while the system is running. Therefore, it is possible to add new sensors and algorithms, as well as expand and refactor a data storage while the system is in use. The reception ports of the ESB can also be configured to receive measurement data as messages virtually in any format. For instance, the WSO2 ESB supports by default messages that are passed in HTTP and SOAP format. It should be noted that the ESB architecture is scalable by using multiple ESB instances to improve performance by passing the messages in between them. Several concurrent instances of the ESB architecture can also be used to make the overall system more robust and fault-tolerant.

The ESB architecture also supports the use of OGC SWE Standards. The ESB architecture can be extended with ports configured, for instance, to receive and pass OGC O&M compliant data. Similarly, the architecture can be extended to support the OGC SOS standard for sensor data management.

Case Study

As our case study, we consider the monitoring of residential buildings. The buildings sector is the largest user of energy and CO2 emitter in the EU, estimated at approximately 40% of the total consumption. In particular, we study a specific home monitoring system called AsTEKa [START_REF] Skön | Energy Consumption and Air Quality Monitoring System[END_REF][START_REF] Skön | Wireless Building Monitoring and Control System[END_REF]. For simplicity, we consider here only two variables: room temperature and water consumption. The sensors for these two variables are both physically and technologically different. As the research work is still ongoing, we present here our complete plans. We have already implemented the ESB architecture and we have studied various statistical methods to be used in the quality flagging. We have also implemented example mediators for quality control, but we

have not yet implemented in full the quality flagging scheme that we discus next. We decided not to consider all quality flag values for all quality control checks. Instead, only the most critical quality checks are considered, as indicated in Table 2. In particular, QC0 was not used, as AsTEKa uses low-cost sensors that do not support real-time computations. Instead, QC1 is extended to consider also the checks usually performed by QC0. Room temperature. Fig. 2 depicts the whole chain of quality control for room temperature data. QC1 decides coarsely if the data points are approved, suspicious, erroneous, or missing. As QC1 runs once a minute, quality controlled data points are available one minute after the measurement. Hence, such approved data points can be used for near real-time control of heating and cooling. By avoiding using suspicious and erroneous data points we can also avoid unnecessary heating and cooling.

Since QC2 runs every 2 hours, data after QC2 can be used for alerting occupants and maintenance personnel of anomalies and potential malfunctions. When considering heating and cooling, a 2 hours window is sufficient to prevent systemic failures that could cause damage to devices or structures. Thus, data after QC2 is particularly suited for diagnostic purposes and detecting occupant behavior or system settings causing to waste energy.

As HQC aims at resolving frozen, erroneous, or missing data, it is useful for analyzing structural changes in the residential building. Because structures weaken over time and the performance of heating or cooling devices also deteriorates over time, one can expect an increasing trend in use of energy over time. This trend can be computed by comparing quality controlled room temperature values with use of heating and cooling energy. As such a change is not abrupt, it is sufficient to perform HQC once a month. The frequency and number of performed corrections by HQC acts also as an indicator for the condition of the home monitoring system as a whole. Water consumption. Fig. 3 depicts the whole chain of quality control for water consumption data. QC1 decides coarsely if the data points are approved, suspicious, erroneous, or missing. As QC1 runs once a minute, only the erroneous and missing data points are of interest. In such a case, maintenance personnel could be notified and the measurement devices could be repaired quickly.

Since QC2 runs every 2 hours, data after QC2 can be used for alerting occupants and maintenance personnel of anomalies and potential malfunctions. In particular, the data after QC2 can be used to spot leaks and malfunction of valves and appliances that use water. When considering water consumption, a 2 hours window is generally sufficient to prevent systemic failures that could cause damage to appliances or structures. Thus, data after QC2 is particularly suited for diagnostic purposes and detecting occupant behavior or system malfunctions causing to waste water or causing structural damage. The frequency and number of performed corrections by QC2 indicates also the condition of the home monitoring system as a whole.

As HQC aims at resolving remaining frozen data, it is useful for analyzing the condition of appliances as well as occupant behavior that leads to wasting water. As such conditions do not evolve fast over time, it is sufficient to perform HQC once a month.

Conclusion

We studied the use of quality flagging as a means to perform quality control of environmental measurement data. We proposed the adoption of the quality flagging scheme introduced by the Nordic meteorological institutes to be used with any kind of environmental measurement data. We presented both a uniform, generalized interpretation for the quality flag values and a scalable Enterprise Service Bus based architecture for implementing the quality flagging. We exemplified the use of the quality flagging a case study for the monitoring of built environment. Our research is ongoing. We presented our design and approach to quality control by quality flagging. We have implemented the core ESB based architecture and we are currently implementing the quality flagging algorithms.

As for future work, we plan on studying occupant profile-based imputation of missing or erroneous values. Such profiling based models could also be used earlier in quality control, for instance, by having QC1 flagging suspicious values with respect to profile based reference values. We also plan on including different kinds of measurement variables, such as CO 2 and humidity. This would enable monitoring indoor air quality and automated notifications on degraded air quality. We are also simultaneously investigating the use of quality flagging in a sensor network monitoring water quality of lakes in Finland together with the Finnish Environment Institute.

Fig. 1 .

 1 Fig. 1. An Enterprise Service Bus based architecture for quality flagging.

Fig. 2 .

 2 Fig. 2. Quality control of room temperature data.

Fig. 3 .

 3 Fig. 3. Quality control of water consumption data.

Table 2 .

 2 Quality flag values used in AsTEKa quality control; an applicable flag value is indicated by "yes".

	Flag	Interpretation	QC1 QC2	QC3	HQC
	0	Value not checked	no	no	yes	yes
	1	Approved value	no	yes	yes	yes
	2	Suspicious value	no	yes	yes	yes
	3	Anomalous value	no	no	yes	yes
	4	Corrected value	no	no	yes	yes
	5	Imputed value	no	no	yes	yes
	6	Erroneous value	no	yes	yes	no
	7	Frozen value	no	no	yes	no
	8	Missing value	no	yes	yes	no
	9	Deleted value	no	no	yes	yes

http://www.wunderground.com

http://wso2.com/products/enterprise-service-bus/

https://www.gnu.org/software/octave/

Acknowledgements

We wish to thank the anonymous reviewers for valuable comments. This research is funded by the Tekes project "MMEA: Measurement, Monitoring, and Environmental Assessment" (Decision number 427/10).


Thorsten Schlachter
email: thorsten.schlachter@kit.edu

Clemens Düpmeier
email: clemens.duepmeier@kit.edu

Oliver Kusche
email: oliver.kusche@kit.edu

Christian Schmitt
email: christian.schmitt@kit.edu

Wolfgang Schillinger
email: wolfgang.schillinger@lubw.bwl.de

Towards a Search Driven System Architecture for Environmental Information Portals

Keywords:

In order to merge data from different information systems in web portals, querying of this data has to be simple and with good performance. If no direct, high-performance query services are available, data access can be provided (and often accelerated) using external search indexes, which is wellproven for unstructured data by means of classical full text search engines. This article describes how structured data can be provided through search engines, too, and how this data then can be re-used by other applications, e.g., mobile apps or business applications, incidentally reducing their complexity and the number of required interfaces. Users of environmental portals and applications can benefit from an integrated view on unstructured as well as on structured data.

Motivation

Environmental portals offer a quick and comprehensive overview on available environmental information and data, which are often distributed over a variety of individual databases, systems, and business applications [START_REF] Schlachter | Umweltinformationssystem Baden-Württemberg F+E-Vorhaben MAF-UIS Moderne anwendungsorientierte Forschung und Entwicklung für Umweltinformationssysteme, Phase II 2012/14[END_REF]. The information are of manifold types and ranges, i.e., structured measurement data, meta data, individual structured or semi-structured (environmental) objects, or reports on the state of the environment without an explicitly modelled information structure.

In web portals such a mix of structured, semi-structured 1 and unstructured data is typically made accessible via hierarchical navigation paths and/or a full text search.

For example, a search for 'nature reserves' in the environmental portal of the federal state of Baden-Württemberg in its present form links to about 3,000 relevant documents and also provides a direct entry into the map view of a business application. However, neither information on individual nature reserves nor a localized entry point into the map view are displayed -although all necessary data is present in databases as well as in a web-based business application, it is not available for the portal via services and interfaces.

From a user's perspective, the portal application should already provide an overview of all relevant nature reserves (in a desired area). Access to a complex business application should only be necessary for very specific tasks or advanced search requests. However, for this purpose all objects, e.g. nature reserves, must be made directly available for the portal application, which basically also applies for all other kinds of environmental information. This means that a vast number of information and environmental objects has to be made searchable in the portal. For the user, this search should be easy to use and as fast as possible.

Querying an Exploding Number of Data

With the aim to present fine-grained environmental information, the number of objects increases dramatically. Some traditional navigation approaches are therefore no longer appropriate, and the design of search interfaces becomes more important.

While full text search interfaces are usually instrumented to provide easy access to unstructured textual data, access to structured data is often implemented by querying the available data via complex search forms, data selectors, or map-based access mechanisms backed up by dedicated application logic mapped to underlying (relational) databases.

An advanced search approach based on simple full text search paradigms has to be implemented which integrates the classical (single-slot full text) search functionality with a search for structured data, and includes specialized approaches for displaying structured data such as map presentations (for the querying and representation of spatial data) or graphical and tabular presentations of measurement data.

Furthermore, search by voice input should be supported on mobile devices, which is easy as search terms are ultimately converted into textual queries anyway.

User Expectations and Objectives

A new search driven approach will change user behavior and raise expectations with respect to how information is accessed using mobile devices. For many users, accessing data via complex navigation paths is no longer acceptable. Especially on mobile devices, a context-based, e.g. location-based, representation of suitable information with a minimum number of user interactions is expected. Any further navigation is often done in an exploratory manner, based on the information already displayed. Every subsequent interaction can trigger a change in the search, i.e., the displayed information has to be updated if necessary. This generates great demands both on the flexibility of query interfaces and on their performance. Noticeable latencies in updating the display are unreasonable for reasons of ergonomics.

As many systems and services do not meet these requirements in practice (yet), it is desirable to prepare structured data in an optimized way for flexible and highperformance querying -an approach proven for unstructured data by means of full text search indexing. At this point, search indices maintaining the structure of data and providing advanced access functionalities come into play.

The principle of search driven websites and portals certainly also applies for structured data. Background services are used to maintain data in their original systems, e.g., relational databases, while keeping a structured search index up to date in parallel. The portal uses the query interface of the search engine for primary data access via search and takes care of data presentation.

Within a search driven portal, the primary information access is driven by search. All navigation items (including menus) as well as the contents are displayed as a result of one or several search queries. In this regard, search driven websites and portals can be characterized as being highly dynamic environments.

The Search Engine as Glue for Distributed Information

The described use of search engines can also contribute to solve further problems of the classical, service-oriented portal approach: as the landscape of environmental information systems is highly heterogeneous, the same applies for the nature of services and interfaces used. It is desired to significantly reduce the number of required interfaces within the portal software -however, for the synchronization of data with the search engine, they have to be implemented elsewhere. In turn, other applications, e.g. mobile apps, can also make use of the search engine and benefit from the availability of data by means of a single query interface. By adding them to a structured search index, even such data can be made available for real time access that aren't available via an (accessible) service, or that are hidden behind a complex navigation or query form. Yet, even in the case of available data services, the performance they offer may often not be sufficient in order for them to be the basis for a search implementation with satisfactory response time. For example, in the context of environmental data, response times in the range of several seconds have been observed from some systems.

While importing or indexing data, these can be filtered or enriched by additional attributes, e.g., place names can be converted into explicit geographical coordinates using gazetteer services, which then can be used to implement a proximity search or bounding box queries. By integrating machine learning algorithms, even a more complex semantic enrichment of data is possible.

In summary, by using a search driven architecture for environmental information portals, the following objectives will be pursued:

• Use of data from their original source and concentration in a powerful search engine • Comprehensive view of all available, relevant information, accessed via simple single-slot full text search queries • Provision of additional aids for an exploratory search experience

• Both the discovery of object classes and of individual objects has to be possible • All data is supposed to contain references to their original systems, as only the most important information is displayed in the portal or mobile application • Reduction of the number and complexity of data interfaces (for portals and applications) • Use of the search index by a variety of applications • No or little extra load on original systems through additional search requests • Querying large numbers of objects has to be as performant as a "simple" full text search • The current state of the environment (measurements) has to be presented • Context information (mobile devices, settings for personalization) has to be included into search

A Basic Architecture for Search Driven Environmental Information Portals

The presented architecture for a search driven environmental information portal is based on a JEE (Java Platform, Enterprise Edition) portal server (Liferay Portal [3]) as both a web frontend and an integration platform for underlying data services, but it also applies for a variety of other applications. The portal software provides a rich set of standard functionalities needed for an environmental portal, e.g., user authentication and authorization, support for corporate web design with responsive layouts [START_REF] Zillgens | Responsive Webdesign: Reaktionsfähige Websites gestalten und umsetzen[END_REF], content and asset management, etc. In particular, however, its modular extensibility is important. New content and data presentation functionality can be easily added to a JEE portal using concepts like portlets and web widgets. The application logic of these extensions uses two search engines for primary data access.

Use of Specialized Search Engines

Since different search engines exist which each have their strengths in specific areas, several search engines are used in the portal. Unstructured data is indexed by a traditional full text search engine (Google Search Appliance [5]) using web crawlers and special connectors. Structured data items are stored and indexed as JSON objects in a separate structured index server based on NoSQL technology (Elasticsearch [START_REF]Elasticsearch[END_REF]). This second search engine can be easily accessed and managed through a REST interface. The necessary tools for importing structured data into the Elasticsearch server as well as for ongoing/periodic synchronization of data are running outside the portal software and should be part of a workflow which also includes quality management for the data.

When data are not available by means of services or when access is limited, these data can be made available using generic cloud services, e.g., data from georeferenced files can be uploaded to the Google Maps Engine, and subsequently can be used via data services or as map layers.

Creating the Search Experience

In addition to editorial content and navigation structures, the search engine is forming the core of the portal application for data access. Queries entered in the search slot will be preprocessed and semantically enriched, e.g., by use of gazetteer services for the identification of geographic coordinates and other location context, or by identifying a time context, and then forwarded to the search engines.

In particular, the enriched queries can drive a more precise search -not only on the basis of mere string comparisons, but allowing location-aware queries finding nearby objects by radius search. Using the enhanced query terms, the search engine returns weighted results according to (configurable) relevance criteria.

The presentation of search results can be performed by various components within the portal software. Currently this is realized via web widgets, i.e., encapsulated Javascript components, each implementing a certain type of data visualization. As soon as supported by relevant web browsers, these can be replaced by standardized "Web Components" [10]. Web widgets are independent from their container application, and therefore can easily be re-used in other applications, e.g., in hybrid mobile apps [START_REF]Multi-channel app development[END_REF].

For data access they use REST APIs, and for the seamless integration with the portal software every web widget is managed within a wrapper portlet. The individual widgets are kept as generic as possible, i.e., they can be configured using Mustache templates [START_REF]Mustache[END_REF] for display, thus being adapted to the respective search results. For the re-use of generic components, it may be helpful to generate additional general attributes such as "title", "abstract" and "link".

Data Mashup: Beyond Simple Result Lists

Various presentation widgets can create or show menus, navigation structures, or individual objects. Each widget again can be a (complex) web application, e.g., a component to display objects on a map. Several portlets (including web widgets) can be placed on the result page, and thus form an information mashup, assembling information from different sources to a single integrated view (Fig. 3).

Fig. 2. Event bus connecting a set of components (widgets and portlets) on the result page.

Other communication systems can be connected using bridge interfaces, e.g., the Liferay Inter-Portlet Communication (IPC).

In order to interconnect components (widgets) and to offer a rich user interface, these have to be able to communicate. An event bus implements the loose coupling of individual components (Fig. 2). Each component can do both listen to events and send own events to the bus, and is solely responsible for the events to which it responds or not. In order to achieve a coherent interaction of components, only the set of event types has to be defined for the portal application.

Additional mechanisms are integrated in the event concept, e.g., for gathering personalization information from cookies, browser storage, or a web server, or to query sensor data (GPS or coarse-grained location) on a mobile device. URL parameters or search slot queries may be sent as events as well. If required, further adapters can be connected to the event bus for handling of external or system events, e.g., the portal software.

In addition to external data from various environmental information systems, data from the portal software itself can be made available via the (structured) search engine. For large datasets or complex requests, the load on the portal software and the underlying database can be reduced significantly by intelligent preprocessing and querying in the search engine. The next major release of the Liferay portal software (Liferay 7) will provide this option by default by means of an embedded Elasticsearch engine.

Case Study "State Environmental Portals" and Environmental App

The described search engine-based approach has been used to perform a prototype reimplementation of the state environmental portal (LUPO) [START_REF] Schlachter | Umweltinformationssystem Baden-Württemberg F+E-Vorhaben MAF-UIS Moderne anwendungsorientierte Forschung und Entwicklung für Umweltinformationssysteme, Phase II 2012/14[END_REF] (Fig. 3). In addition to the weak structured and unstructured content (about 2,000,000 documents from more than 2,000 sites), many structured data, e.g., various types of protected areas, and locations of energy systems such as windmills, biogas plants, solar panels, and hydropower plants have been stored in the Elasticsearch index. Where possible, geographical coordinates were acquired or produced for all objects. This index forms the core of a significantly expanded functionality of the portal which now integrates the display of location based search results on maps and provides a location-based search. Each query, whether manually entered to the search slot or clicked on using the tag cloud, first generates an empty result page. After preprocessing and semantic enrichment of keywords, both search engines and additional backend systems are queried in parallel and asynchronously. While the results for the full text search engine are shown in a classic hit list, the structured hits can trigger a variety of representations. For example, the discovery of objects of certain classes can force the display of corresponding map layers within the map component. Classes and individual objects can be shown for information and further navigation. Clicking on an object class can force the display of matching instances and switches the corresponding map layers on or off. The results already contain quantitative information, e.g., how many objects of a certain class were found for the current query. Clicking on a particular object can, for example, trigger the centering of the map to this object. Coming back to the example of the "motivation" section, a search for "nature reserves" and a place name now displays the closest reserve objects in the left column.

The corresponding heading with the concept name "nature reserve" serves for further navigation and allows switching on and off the respective map layer. Clicking on a particular object (a single nature reserve) centers the map to the appropriate place. Another click on that area in the map then displays additional details and links to business applications. Many components of the state environmental portals (data, services, search engines, and user interface) are also used in the mobile application "Meine Umwelt" ("My Environment") [START_REF] Schlachter | My environment -a dashboard for environmental information on mobile devices[END_REF]. It is a hybrid app whose core consists of a HTML5 application used in apps for Android, iOS and Windows Phone. The HTML5 application is organized into web widgets as well and utilizes the same services and search engines. This saves resources for both the development and the maintenance of components.

Outlook

Putting unstructured, semi-structured, and structured information into search engines can be considered as a first step towards a unified information access platform (UIA) [START_REF] Probstein | Five Advantages of Unified Information Access (UIA)[END_REF], even if central aspects of UIA like advanced data processing or data analysis remain outside yet. However, the basic technologies create a huge potential for the generation of added value.

Most planned extensions do not affect the search-driven architecture as such, but rather the applications it supports. Especially with regard to user-experience and ergonomics, quite a number of improvements can be achieved. Due to the loose coupling of components and the use of an event-bus in the portal application, the integration of additional navigation aids and further functionality is very easy.

An improved linking of data in the search index is probably the most promising approach. Currently, many object classes in the index are data islands that are not or hardly connected to other classes. The use of shared concepts, such as in the thematic and spatial classification of data, promises a great potential, even for the generation of completely new applications. Corresponding relation or graph-oriented concepts based on Elasticsearch already exist and are waiting to be used.

  External search engines may complement the portal search by delivering special data, e.g., statistical data or descriptions [7] of administrative services [8]. The use of standardized interfaces like OpenSearch [9] can simplify the integration of external search engines.

Fig. 1 .

 1 Fig. 1. Main components of a search driven architecture for web portals and (mobile) applications including two search engines and cloud services to complement original data services.

Fig. 3 .

 3 Fig. 3. Result page of a state environmental portal. Object classes and individual objects as well as measurement values are presented by multiple widgets on the left. They may trigger events toggling the display of map layers, centering of the map, or refining of search queries. Full text search results are presented below the map widget. Current measured values are displayed either from the search engine or from separate cloud services to match the context.Many components of the state environmental portals (data, services, search engines, and user interface) are also used in the mobile application "Meine Umwelt" ("My Environment")[START_REF] Schlachter | My environment -a dashboard for environmental information on mobile devices[END_REF]. It is a hybrid app whose core consists of a HTML5 application used in apps for Android, iOS and Windows Phone. The HTML5 application is organized into web widgets as well and utilizes the same services and search engines. This saves resources for both the development and the maintenance of components.

As semi-structured data can be transformed into a structured form[START_REF] Abiteboul | Data on the Web: From Relations to Semistructured Data and XML[END_REF], and both are treated analogously in the portal application, in the following only structured (as opposed to unstructured) data will be mentioned. adfa, p. 1

, 2011. © Springer-Verlag Berlin Heidelberg 2011

Conclusion

The procedure described can be summarized as follows: Modern web applications, and in particular mobile solutions, dynamically load data. In portal-like applications, a great variety of different data interfaces and data sources is used. In order to simplify the application and to reduce the load on the primary backend systems, data can therefore be administered in a separate search index:

• Let the engine do the hard work (preprocessing, indexing, querying)

• Existing data can be enriched • Use structured data types to obtain individual ways to query and to present object classes • Use generic formats and mechanisms, e.g., templates for presentation, so a variety of applications can be covered "out of the box"

What distinguishes the search driven approach in comparison to previous/alternative concepts?

• Reduction of interface diversity in the consuming applications • Easy integration of data sets. However, it requires an operating concept for the update of the index • (Quick!) retrieval of object classes and individual objects • Fine-grained local search on a single object-level (as opposed to entire map layers) • Combination of structured/semi-structured and unstructured data in the search • Reduction of redundantly managed metadata • Reusability of web widgets, e.g., for mobile and web applications • Reusability of services and search indexes • More interactivity for the user • Prevention of a bottleneck on the server side (portal) • Less load on the portal server by offloading and indexing content The concept of search-driven portals and websites will stand or fall related to the relevance of their search index. Automatic updates of the search index, which have been only briefly mentioned in this paper, must not be underestimated. Basically, the operation of a search engine for information integration in principle means an added expense and also raises questions about data consistency, redundancy avoidance and operating cost.

However, the advantages of the search-driven approach clearly outweigh these concerns for the existing applications. The objectives are therefore fully met by the prototype. User feedback has been very positive and expectations are even being surpassed in many ways.