Maria-Helena Ramos
email: maria-helena.ramos@irstea.fr

Andrea Castelletti

Manuel Pulido- Velasquez

Matteo Giuliani

Hector Macian-Sorribes

David Gustafsson

Manon Cassagnole

Guillaume Thirel

Louise Crochemore

Ilias Pechlivanidis

This report presents the hydropower policy-brief: "The benefits of informing hydropower reservoir operations with climate and inflow forecasts". This policy brief summarizes the key results and recommendations from research carried out during the IMPREX project on four case studies representative of typical hydropower systems encountered in Europe. Based on strong collaborations with stakeholders and operational managers from European energy production companies, proof-of-concept demonstrators were developed to show the value of hydro-meteorological forecasts for the hydropower sector.

D8.2 Hydropower Policy Brief:

The benefits of informing hydropower reservoir operations with climate and inflow forecasts

POLICY BRIEF

The benefits of informing hydropower reservoir operations with climate and inflow forecasts ~Background and challenges

The hydropower sector is sensitive to climate (rainfall, snowfall and melt, temperature), hydrologic conditions (river runoff variations) and river basin conditions (land use and occupation). Water and climate not only affect hydropower production capacity (i.e., electricity supply), but also energy consumption (e.g., electricity demand for heating or cooling).

The quantification of water resources availability and the anticipation of future climate conditions, considering the uncertainties associated with future scenarios, represent a challenge to the sector in both its daily operations and long-term strategic planning.

Daily operations include managing hydroelectricity production to effectively balance demand with supply, and avoid unstable prices and disruptions in the access to electricity. Long-term strategic planning ensures that adequate investments are considered to satisfy projected energy demands from population growth, increased electricity consumption, economic development and the global energy transition to renewables.

Hydropower strategic planning on medium-(weekly) to long-(monthly to seasonal) time scales, which are mostly driven by climate and hydrology at the local scale, needs to be aligned with daily and sub-daily decisions, which are mostly primarily driven by the energy market. Climate-oriented and market-oriented decisions have to cohabit in decision-making in hydropower reservoir operations.

Hydropower is the leading renewable energy resource globally. It is key to achieving the EU targets for reduction of greenhouse gas emissions and transition to a carbon-neutral electricity system. A major characteristic of hydropower systems is their flexible and reliable production, which allows hydropower to be used for meeting base-load electricity requirements, as well as providing peak-load supply during peaks of high or unexpected demand. This is greatly due to the capacity of storing water in reservoirs, which translates into storing energy at one time for its use at a later time. This is an asset for daily operations but also to assist in building a climate-resilient energy system.

Hydropower reservoirs provide flexibility and reliability to energy systems, but challenges remain on how optimal operations can be improved given our current capacity to predict future climate and water conditions and the associated uncertainties. There are technical, economic and environmental challenges related to the operation of multipurpose hydropower reservoirs. Changes in hydropower operation affect other water uses and, reversely, operations from other uses affect hydropower production. This raises the challenge of evaluating the benefits of a joint optimal operation, supported by tailored forecast information on future climate and water conditions.

The optimization of hydropower production is a water allocation problem. It relies on making the best decision between using water today (i.e., releasing water through the turbines to produce electricity and deliver it to the power grid) or storing water to be used tomorrow, when, based on projected demand and future value of water, electricity prices and demand are expected to be higher.

Climate variability and change affect water availability and the future value of water. Consequently, it also affects the decisions to be taken.

Although hydropower is largely considered a mature sector in terms of innovation, progress remains to be made in terms of enhancing in-house capacity to integrate state-of-the-art European climate services in daily operations and planning. To better inform decisions and operations, climate and hydro-meteorological information on future water inflows to reservoirs can be used. Challenges remain on quantifying the economic value of using this information efficiently in hydropower operations.

~Relevance of informing hydropower operations with climate and inflow forecasts  The research project IMPREX explored the use of climate and inflow forecasts to inform hydropower reservoir operations. It focused on forecasts for several days to months ahead, and on projections of water availability under climate change.

 Based on strong collaborations with stakeholders and operational managers from European energy companies, it addressed the challenges of matching available forecast information with local hydrological-energy contexts and needs.

 Proof-of-concept demonstrators were developed in four case study areas in France, Italy, Spain and Sweden. They assessed the operational value of forecasts at short to seasonal time scales, and the impacts of climate predictions on the adaptability of reservoir operation rules.

 Altogether, the IMPREX case studies provide a strong body of evidence on the economic benefits of hydro-meteorological predictions and climate services to the hydropower sector in Europe.

Key Messages:

 There is evidence of the link between the quality of inflow forecasts and their economic value to hydropower reservoir operations. In general, more accurate forecasts bring additional benefits to the hydropower sector in terms of increased hydropower production and revenues. The quantification of these benefits depends however on how climate-driven decisions interact with price-driven decisions in the hydropower system.

 Hydropower systems that are already dimensioned to capture the seasonal hydro-climatic variability of the inflows in their operations are less impacted by imperfect climate predictions. The room for improving hydropower benefits with better forecasts and hydroclimatic-informed operations is small in terms of yearly percentages of total electricity production (on average, up to around 3% per year). However, it corresponds to potential extra gains of thousands or millions of euros per year that can be used to invest in the sector and in integrated water management in complex river basin water systems.

 When there are competitive water uses, the main source of improved benefit for the hydropower sector may not be associated with the use of a particular forecasting system, but rather with the fact that the additional forecast information enables an improved operation that benefits all water uses.

 Under climate change scenarios, particular attention should be paid to transition periods of projected changes in hydrological regimes. The higher variability of projected hydro-climatic conditions will make the optimal operations of hydropower reservoirs more challenging. It tends to amplify the potential value of forecast information.  Benefits are expected from including better hydro-meteorological forecasts in the chain of forecasting and reservoir management systems. It is therefore important to continue increasing the understanding of climate-water-energy systems and their interdependencies and responses to extreme events. The continued exploration of weather, climate and water services' capability of anticipating extreme events over different temporal scales is to be encouraged and fostered: when hydropower systems are at critical states (e.g., reservoir water levels too low or too high), knowledge of future conditions facilitates interactions between water users.

 Multi-sectoral co-benefit analysis can shed light on how consumptive water uses can also benefit from better-informed hydropower operations. There is therefore a potential for winwin situations, and open opportunities to promote further integrated water management frameworks. Multi-sectoral contexts can also foster commitments to align policies and practices among sectors by taking into account interlinkages and trade-offs on short-term operations and long-term strategic planning.

 Research and operations move at a different pace. However, the involvement of hydropower operational forecasters and service developers in joint research and innovation projects has proven to be effective: this can be an asset to optimize the efforts needed to adapt hydropower operations to state-of-the-art forecast products made available by service providers.

IMPREX has received funding from the European Union Horizon 2020 Research and Innovation Programme under Grant Agreement N° 641811

  Expanding power generation from renewable energy sources has been a policy focus in Europe since the 2001 EU Sustainable Development Strategy (Commission Communication COM/2001/0264 final), followed by the 2020 climate and energy package, including the Renewable Energy Directive (DIRECTIVE 2009/28/EC of 23 April 2009), the 2030 climate and energy framework, reflecting EU commitments under the Paris Agreement, and the EU long-term decarbonisation scenarios up to 2050. Although the share of hydropower has remained largely constant over time (10-13%), while other renewable sources (e.g., bioenergy, wind and solar) grow rapidly, hydropower is predicted to remain an important source of renewable electricity generation in the total electricity mix. IMPREX has received funding from the European Union Horizon 2020 Research and Innovation Programme under Grant agreement N° 641811 4

  At long-term, there are challenges in determining the viability of major investments or dam operation procedures under changing climate conditions. Hydropower reservoir operations affect river flows, and hydropower water use can compete with other water users, such as agriculture and nature protection. The 2007 EU Directive on the assessment and management of flood risks (DIRECTIVE 2007/60/EC) acknowledges the cases of multi-purpose use of water bodies and calls attention to their impact on requirements for the achievement of the environmental objectives established under the EU Water Framework Directive on water policy (DIRECTIVE 2000/60/EC). While it is crucial for hydropower to deliver electricity when demands (and prices) are high, generating economic benefits, the sector must also take into account minimum environmental flows to maintain ecosystem services, downstream minimum flows during low flow and drought periods for other water uses, and flood risk regulations.

  ~Recommendations IMPREX addressed the challenges of quantitatively measuring the economic value of hydrometeorological predictions to the hydropower sector. The following recommendations can be derived from the work carried out:  The increased availability of climate services prompts questions about how these services can meet the sector's needs and expectations, and be incorporated in current hydropower practices. Given the presence of multiple time dynamics in hydropower systems, tailoring information to the required level of scale and detail is crucial to move from the stage of having predictions available to having predictions fully integrated in decision-making and economically impactful.

Deliverable n° D8.2

X PU Public PP Restricted to other programme participants (including the Commission Services) RE Restricted to a group specified by the consortium (including the European Commission Services) CO Confidential, only for members of the consortium (including the European Commission Services)


Table of Contents

List of Figures ... TABLE 1: Forecast value: how a forecast helps the user to make a better decision.

List of Figures

List of Tables

Forecast skill: the relative accuracy of the forecast over some reference forecast.

Score: a quantitative measure of forecast quality.

Skill score: a relative measure (or scaled representation) of forecast quality that relates the forecast accuracy of a particular forecast to some reference forecast. Skill scores range from negative infinity to positive one. A perfect categorical forecast yields skill values of 1. A forecast with similar skill to the reference forecast will have a skill score of zero, and a forecast that is less skilful than the reference forecast will have negative skill score values.

Scoreboard utility: a graphical interface, connected to a score database, to support comparisons between numerical scores of different forecasts or forecast systems.

Score data provider: a partner of the IMPREX project who contributes data to the scoreboard.

Scoreboard user: any person (partner of the IMPREX project or not) who wishes to visualize the quality of a forecast or a forecast system investigated in one of the case-studies of the IMPREX project (under the condition that the score of this forecast or a forecast system is made available by a score data provider).

Introduction 1

One of the main aims of the IMPREX project is to enhance the forecast quality of extreme hydro-meteorological conditions and, as a result, their impacts. Impacts are measured for six targeted types of users involved in the following socio-economic sectors: Floods, Hydropower, Agriculture and droughts, Navigation, Water supply, and Water economy. This wide range of users also represents a wide range of needs concerning the performance of a forecast system. For instance, while a user in the flood protection sector may be more interested in the moment the level of a river or stream will exceed a critical flood alert level, a user in the hydropower sector may search to improve the quality of the weekly or monthly averages of river inflows to a water reservoir to plan its operation.

It was considered useful to the IMPREX project to have a common framework to evaluate and inter-compare the performance of different forecast systems over the case study applications of the project. This report presents a prototype of a verification scoreboard for hydrological reforecasts, developed to be used during the IMPREX project. Our focus is placed on the main technical aspects of the implementation of the scoreboard utility and its capabilities. The aim is to introduce the technical concepts behind the tool in order to better satisfy the needs of its potential score data providers (namely, the project's partners implementing the case studies). This report is organized as follows: In Section 2, we present an overview of the main aspects involved in the verification of hydrological forecasts. The aim is to highlight the main issues one has to have in mind when developing a scoreboard. Section 3 focuses on the utility that was developed, which combines a score database and a web-based display to facilitate the visualization and the interpretation of the quality of the forecasts produced during the project. An initial testing on reforecast data is presented to exemplify the approach adopted. Section 4 provides a brief conclusion and proposes ways forward for the use and maintenance of the utility during the lifetime of the project and beyond.

Verification of hydrological forecasts 2

A brief overview of forecast verification 2.1 A large part of the performance of a forecast system is measured by the quality of the forecasts delivered by the system. The quality of a forecast measures how well a forecast is compared against a corresponding observation of what actually occurred, or some good estimate of the true outcome. Forecast verification is the process of assessing the quality of a forecast. In hydrology, forecast verification is also called "forecast evaluation". In meteorology, the "value of a prediction" specifically refers to how a forecast helps the user make better decisions.

The quality of a forecast can be assessed through a wide range of metrics (or verification scores) 1 (see details in [START_REF] Wilks | Statistical Methods in the Atmospheric Sciences : An Introduction[END_REF][START_REF] Jolliffe | Forecast Verification: A Practitioner's Guide in Atmospheric Science[END_REF]. Many forecasters and practitioners recommend the use of several scores to better assess the attributes of a forecast (such as reliability, resolution, discrimination and sharpness): "any set of forecasts can then be ranked as best, second best,…, worst, according to a chosen score, though the ranking need not be the same for different choices of score." 2 What scores to choose depends on the user's objectives as well as on the characteristics of the forecasts being evaluated. In several situations, it may be interesting to evaluate the relative quality of a forecast with regard to a reference system or a baseline (e.g., to decide which is better between system A and system B). Here, again, there exist several baselines that can be considered, varying from simple approaches such as climatology (always forecasting an average value) or persistence (the last observation is forecast to persist into the future) to more sophisticated benchmarks that take into account analogue-based features (see, for instance, the twenty-three benchmarks that were designed and used for the assessment of hydrological forecasts in the study proposed by [START_REF] Pappenberger | How do I know if my forecasts are better? Using benchmarks in Hydrological Ensemble Prediction[END_REF]. 1 See, for instance, http://www.cawcr.gov.au/projects/verification/

Deliverable n° 4.1

Relative measures of quality are known as skill scores. They measure the improvement of a given forecast relative to a reference forecast, or how much better one forecasting system is in comparison to another forecasting system given the same variables and location.

There are several reasons for assessing the quality of a forecast dataset. Measures of forecast quality can be used, for instance, to guide decisions on the additional human and/or financial resources needed to further develop a forecasting system; to judge how changes in modelling approaches affect the quality of the forecasts; to improve our understanding of uncertainties and biases involved in the forecast processes; and to understand the drivers of forecast performance in a modelling system as well as the strengths and weakness of different forecasting approaches.

Forecast quality may depend on the forecast situation or conditions (e.g., setup of the forecast verification framework). It also varies with human and modelling factors, forecast lead time, forecast location, as well as temporal and spatial scales considered3 . Providing information on these factors and on the calculations of the verification scores is therefore important in the communication of forecast quality assessments.

Main aspects of the verification of hydrological forecasts 2.2

Verification of hydrological model outputs (river flows or water levels) is a continuing and necessary effort for modellers and operational forecasters to provide guidance on model development, implementation and on the operational use of hydrological forecasts. Many of the metrics used in the verification of hydrological forecasts come from the meteorological community or have been adapted to answer to the specific needs encountered in hydrological analyses [START_REF] Casati | Forecast verification: current status and future directions[END_REF][START_REF] Pappenberger | Hydrological aspects of meteorological verification[END_REF].

Most often, typical constraints of the modelling of hydrological processes require metrics that target a given variable or focus on a specific aspect of quality searched by an advanced forecast user [START_REF] Demargne | Application of Forecast Verification Science to Operational River Forecasting in the U.S. National Weather Service[END_REF][START_REF] Brown | The Ensemble Verification System (EVS): a software tool for verifying ensemble forecasts of hydrometeorological and hydrologic variables at discrete locations[END_REF]: for instance, metrics have been developed to evaluate how accurate predictions of flood peaks are in magnitude and timing (e.g., [START_REF] Liu | A wavelet-based approach to assessing timing errors in hydrologic predictions[END_REF][START_REF] Zappa | A 'Peak-Box' approach for supporting interpretation and verification of operational ensemble peak-flow forecasts[END_REF] or to evaluate highly temporally correlated seasonal low flows (e.g., [START_REF] Wood | A retrospective assessment of National Centers for Environmental Prediction climate model-based ensemble hydrologic forecasting in the western United States[END_REF][START_REF] Nicolle | Benchmarking hydrological models for low-flow simulation and forecasting on French catchments[END_REF][START_REF] Trambauer | Hydrological drought forecasting and skill assessment for the Limpopo River basin, southern Africa[END_REF].

Hydrological forecast verification focuses more and more on quantifying the quality of probabilistic or ensemble-based predictions [START_REF] Cloke | Ensemble flood forecasting: a review[END_REF], although metrics for deterministic forecasts are also commonly used (e.g., correlation coefficient, Nash-Sutcliffe model efficiency coefficient or Persistence Index), as well as traditional metrics for categorical events associated to contingency tables (e.g. probability of detection, false alarm rate or Critical success index). A variety of metrics exist for a variety of forecast goals, variables to be evaluated, associated space and time steps, etc. Examples of this diversity of applications of verification metrics to evaluate forecasts in hydrological studies are presented in Table A1.1 in Annex 1.

Particularities within the IMPREX project 2.3

The IMPREX project focuses on enhancing forecast quality of extreme hydro-meteorological conditions and their impacts. Its partners work to develop methods and tools to improve the forecasting of meteorological and hydrological extremes. It is then important to quantitatively measure if (and how) improved forecasts perform better than the reference forecasts or the benchmark system that is being improved. Additionally, it is also of interest to measure how meteorological forecast improvements impact hydrological forecasts, as well as how hydrological forecast improvements impact risk assessment and management variables, following the flow of information and models that is typically seen in an end-toend hydro-meteorological forecasting chain. In IMPREX, the value of the improved forecast information is demonstrated in a set of case studies, which evaluate hydrological risks and impacts that are relevant to stakeholders at the regional and European scales. These case studies cover six water sectors, as shown below. The scoreboard should allow partners to easily store their score data, computed for the many configurations of improved forecasts they may want to test, and visualize the evolution of scores with lead time, across locations within a case study, and using different metrics of forecast quality. In IMPREX, a verification scoreboard should include short, medium, and long ranges. It should also be flexible enough to accept metrics that are traditionally used in forecast verification as well as new metrics, specifically developed to focus on hydrological variables of special interest (e.g., hydrologic threshold exceedances, hydrological extremes of high and low flows) and to quantify the added value of improved forecasts for specific applications (e.g., gain in lead time from anticipating drought conditions, gain in energy production from the use of better forecasts in models of hydropower reservoir optimization, evaluation of transport costs by using probabilistic forecasts in a simulation model for river navigation, etc.).

Design of a verification scoreboard utility for IMPREX 3

In order to contribute to the forecast verification goals of the IMPREX project, a prototype It is out of scope of this deliverable to achieve the same level of display and generality of the scoreboards examples presented here. Our goal is to propose a utility that can be used during the lifetime of the project to centralize forecast verification scores on hydrological reforecasts for the water sectors studied in the IMPREX project. Scoreboards for hydrological forecasts are rare and this prototype is an attempt to face the challenge of adapting graphic and display options for forecast verification focusing on hydrological variables. The design of the score database and the setup of the display interface are presented in details in the next sections.

Overview of the concept behind the IMPREX scoreboard utility

3.2

Sharing a scoreboard -a tool which allows comparison and collaboration -is an excellent opportunity for a web page, specifically if it is a combination of a back-end data repository with a graphical, shared, interface. This allows analyses and updates to be performed by several users, potentially at the same time. For the IMPREX project, we developed a database (which can reside on a centralized server) as the repository, and an HTML5 graphical user interface (GUI) front-end to query and display score data. Figure 1 illustrates the verification scoreboard concept set up for the IMPREX project.

A user can connect to the scoreboard utility (App) and start a query for a given case (pertaining to a case study of IMPREX), selecting the choices related to the score the user wants to visualize. This will make the interface display R plots and tables, which the user can export as a PDF file. The interface is connected to the IMPREX score database, which will be fed with scores computed by the IMPREX partners involved in the evaluation of the forecasts for the IMPREX case studies. PostgreSQL is a robust database system based on SQL, structured query language. We have incorporated an R package developed by RStudio called "Shiny" to produce the web-based interactive GUI.

In the future, it can be envisaged that the scoreboard may be accessed two ways: Local installers for the PostgreSQL database are available for Linux, Mac and Windows platforms (see Table 1 for software specifications). A GUI interface (e.g. "pgAdmin") is typically installed along with the instance of the database to help manipulate the database, set password information, and backup data. To clone the schema of the IMPREX database, a DDL (Data Definition Language) script is available, along with more detailed instructions for first-time users.

Adding or connecting to multiple databases is facilitated by an environment text-or ini-file, ".Renviron". User information (username, password) is pulled from this file for local database instances. A template for this file is included in Annex 3.

Format of data files 3.3.2

In the schema proposed in Figure 1, the "IMPREX database" has score data that need to be provided by the project partners in a pre-defined way. In order to match project requirements, we have identified the RDS R data file format 5 as the optimal data file format supported.

Compared to the more commonly-used RData file format, RDS prevents from possibly erasing internal variables of the application if a variable name is identical to an object name in an imported file. From a user perspective, producing an RDS file is very similar to producing an RData file. The scoreboard utility also supports text file imports, although it is important to note that file size limitations may play a role on uncompressed files. Data files support the following common data elements on any user-imported score file: 1. Metadata; 2. Data. The metadata contains contact information for the individual most responsible for the data submittal (an IMPREX partner), as well as details such as model verification start and end dates, score type, case study and forecast system details (see 5 R has its own data file format and usually uses the .rds extension. The .rds file format is usually smaller than its text file counterpart and will therefore take up less storage space. The .rds file will also preserve data types such as factors and dates, eliminating the need to redefine data types after loading the file.

Deliverable n° 4.1

Design of the scoreboard interface 3.4

The layout of the scoreboard features typical web page navigation tools; though R software is used to create the webpages, they are standard HTML5 with JavaScript and readable by any platform and browser.

HTML display 3.4.1

The web page was developed using R, and specifically RStudio's Shiny package (see Table 2 for software specifications). Figure 3 illustrates the main components of the HTML display.

We have added the IMPREX logo on the top left corner. In the space below, the left column is dedicated to allow the user to select the IMPREX case study and the system for which scores are to be visualized. A number of filter choices are available, so that the user can choose the location to display (e.g. sub-basins of a catchment included in a case study or location of point stations included in the European-wide case studies), the variable (e.g. streamflow or precipitation), the forecast system (e.g., the benchmark system, a system where bias correction was applied), and the score to plot.

The graphical plot is displayed in real-time, once all the choices have been made. The tabs in the upper part of the plot area allow the user to visualize panel plots (i.e., several score plots for inter-comparison purposes), compare skill scores, and access a summary table of the database, the scores definition and an upload tool. Plots and table can be saved in a PDF file and downloaded. This selection updates the remaining navigation filters, and presents the user with a narrower set of choices. The list of items available in the filter criteria is thus adapted according to the available items in the Score database, which are automatically detected by the interface once the selection of Case Study and System has been done. The user can:

1. select one (or more) location(s), a list of locations appear and update the plots, 2. select one score to view the associated plot (in the main plot area).

At any point during the use of the Scoreboard the user may view a panel plot with all scores available for the selected locations, download a PDF of the last plot drawn, as well as view a summary table of the database.

An alternative workflow may suit more experienced R users, or those testing their data import format. If the user has installed RStudio (and necessary libraries) locally, they may clone the Scoreboard project to their disk and run it with the following command: shiny::runApp. While this Scoreboard cannot connect to the IMPREX database nor make submissions to it, it can accept and verify a user's RDS or txt file uploads, as well as working with and visualizing the data locally.

Example with a test data from IMPREX partner 3.5

The Scoreboard utility was tested based on a test dataset provided by partner SMHI. Data comes from streamflow simulations based on the E-HYPE seasonal forecasting system in Europe, which were verified by SMHI with a variety of scores. These E-HYPE score data were provided in RData file format. It contained scores and skill scores for several stations in Europe. The file contained scores for each month of the year, 6 lead months and each station.

Conclusions and way forward 4

We have developed and tested a prototype of verification scoreboard utility for the IMPREX project. It is based on a database as the repository for score data collected during the lifetime of the project and an HTML5 graphical user interface (GUI) front-end to query the database and display plots of the scores that were uploaded to it. Following the development and first test phase, priorities and needs were discussed with IMPREX partners, allowing to enhance the utility.

We proposed a RDS format file for the score data, with possibility to a txt file format, after data import process had been agreed among partners. We have written the specifications for the data format and a loader utility was included to facilitate IMPREX partners to upload their score data files.

Some issues remaining include:

 A variety of types of plots can be found in the literature. We have included some of the more common plots. IMPREX partners may have other suggestions and these could eventually be implemented, if technically feasible, including some more integrated and easy-to-understand plots for a general public of users of the scoreboard.

 We have left the possibility in the format file to collect location data (geographic coordinates). The display can thus be enhanced with maps of locations and/or scores.

 We have worked with typical statistical scores, but other types of scores (including those measuring economic gains in the water sectors investigated in IMPREX) could also be included.

 An important step is to identify the host for the database and the web application (Shiny server). This issue was discussed during the 2nd Imprex General Assembly and a decision was taken together with IMPREX partners, the website administrator and coordinator to explore a URL link between partners ECMWF and ARCTIK.

 User manuals with detailed descriptions of the scoreboard utility can be provided to introduce the score data providers and the users to the tool.

Files formats and needed data and metadata

The scoreboard works with RDS and text files.

One single data file can only contain data from one experiment evaluated on one score, with one initialisation, but it can include several stations.

NA values must be entered as either:

-NA --9999.

I invite you to open these files to better know what to put in. I provide here some important elements of explanation:

-The RDS files are R data files. They contain a list of elements containing the metadata and data you want to upload.

-The text file contains several lines with semi-column separated metadata (8 fields needed) and semi-column separated data.

-In both formats, the metadata are read as characters. -Important notice: some fields must be very carefully filled. Indeed, the database uses some of them to link the different tables of the database. The fields are all the metadata fields (except VerificationPeriod for now) and the LocationID. To illustrate what I am saying, if provider 1 gives CRPSS and provider 2 provides CRPSkillScore, which are basically the same scores, they will not be comparable in the 3 rd panel of the scoreboard.

Annex 1 -

 1 ............................................................................................................................................................... List of Tables.................................................................................................................................................................... Glossary .............................................................................................................................................................................. Introduction ............................................................................................................................................................ 1 Verification of hydrological forecasts .......................................................................................................... 2 A brief overview of forecast verification ........................................................................................... 2.1 Main aspects of the verification of hydrological forecasts ...................................................... 2.2 Particularities within the IMPREX project ....................................................................................... 2.3 Design of a verification scoreboard utility for IMPREX..................................................................... 3 Examples of existing verification scoreboards ............................................................................. 3.1 Overview of the concept behind the IMPREX scoreboard utility........................................ 3.2 Design of the score database ............................................................................................................. 3.3 SQL database: PostgreSQL .......................................................................................................... 3.3.1 Format of data files......................................................................................................................... 3.3.2 Design of the scoreboard interface .................................................................................................. 3.4 HTML display ..................................................................................................................................... 3.4.1 Workflow .............................................................................................................................................. 3.4.2 Example with a test data from IMPREX partner ......................................................................... 3.5 Conclusions and way forward ...................................................................................................................... 4 References ............................................................................................................................................................. 5 Annexes .................................................................................................................................................................. 6 Examples of applications of forecast verification metrics in hydrology .... 6.1 Annex 2 -Examples of existing verification scoreboards ...................................................... 6.2 Annex 3 -Example of .Renviron file ................................................................................................ 6.3 Annex 4 -Format specifications for input files to the IMPREX score database .........

FIGURE 1 :FIGURE 5 :

 15 FIGURE 1: WORKFLOW DIAGRAM OF THE VERIFICATION SCOREBOARD ........................................................................................14 FIGURE 2: EXAMPLE OF THE HEADER OF A SCORE DATA FILE (METADATA) ..................................................................................16 FIGURE 3: EXAMPLE OF THE HTML DISPLAY OF THE SCOREBOARD UTILITY ................................................................................18 FIGURE 4: SCREENSHOTS OF THE HTML DISPLAY INTERFACE OF THE SCOREBOARD UTILITY FOR 8 LOCATIONS IN THE E-HYPE TEST SCORE DATASET, THE CRPSS SCORE (TOP) AND CORRELATION COEFFICIENT (BOTTOM) .........................20 FIGURE 5: SCREENSHOTS OF THE TAB OPTIONS OF THE HTML DISPLAY INTERFACE OF THE SCOREBOARD UTILITY: PANEL PLOTS (TOP) WITH 3 SCORES (CRPS SKILL SCORE, CORRELATION COEFFICIENT AND KGE) AND 8 LOCATIONS, AND SUMMARY TABLE OF THE E-HYPE TEST SCORE DATASET (BOTTOM) ..................................................................................21

  SOFTWARE SPECIFICATION FOR POSTGRESQL ..................................................................................................................15 TABLE 2: SOFTWARE SPECIFICATION FOR R AND RSTUDIO ............................................................................................................18 Glossary Forecast verification: the process of comparing forecasts to relevant observations, or assessing the quality of a forecast product or a forecast system. Forecast quality: how well a forecast compares against a corresponding observation of what actually occurred, or some good estimate of the true outcome.

  2 from: Jolliffe, I. T., and D. B. Stephenson (2012), 2 nd Edition, Chapter 1 Introduction, page 5.

  1. Central European Rivers: floods, transport, agriculture and droughts Deliverable n° 4.1 2. Bisagno river basin in Italy: floods 3. Thames River Basin in UK: floods 4. South-East French Catchments: hydropower 5. Júcar River Basin in eastern Spain: hydropower, agriculture and droughts 6. Lake Como basin in the Italian Alpine region: hydropower, agriculture and droughts 7. Upper River Umeälven in Sweden: hydropower 8. Segura River Basin in the Iberian Peninsula: urban water, agriculture and droughts 9. The Llobregat River Basin in north-eastern Spain: urban water 10. The Messara valley in Crete: agriculture and droughts Many evaluations of forecast quality will be carried out along the different case studies and sectors during the lifetime of the IMPREX project. Side by side evaluations would help partners and stakeholders to better understand the impacts of the forecasts on the sectors investigated. To facilitate inter-comparisons, a verification scoreboard for the hydrological reforecasts is proposed.

  Figure A2.2 shows the verification of Quantitative Precipitation Forecasts (QPF) provided for a single event: the Hurricane Sandy, which affected most of the eastern United States (especially the coastal Mid-Atlantic States), in the autumn 2012. Verification is provided through the comparison of the maps of the accumulated precipitation forecasts and the accumulated observed precipitation over the affected areas. Another example can be found in the website of the UK MetOffice for global long-range predictions. Probabilistic skill maps and plots are available and updated monthly for temperature and rainfall predictions up to six months ahead. Figure A2.3 illustrates how the user can change the "skill score type" to display a ROC score map or a Reliability diagram.

Figure 1 :

 1 Figure 1: Workflow diagram of the verification scoreboard

1 .

 1 By navigating to the IMPREX website and clicking on the link to the server URL; 2. By downloading or cloning the source repository to run the application through a local R installation (additional libraries may be needed).While the first option is more anticipated, and the best way to connect to the shared IMPREX score database, a local installation allows users to experiment with file upload formats, visualizing their score input files locally, and creating local databases as needed.Deliverable n° 4.1The user may also use the local scoreboard without an internet connection, for example during a presentation. phase, we used a combination of local database instances 4 at partner IRSTEA (France) and an instance running on Amazon's free AWS service and local database instances.

Figure 2

 2 Figure 2 for an illustration). Scores are included last, and take the biggest part of the file space. Annex 4 presents a file specification and further explanations.

Figure 2 :

 2 Figure 2: Example of the header of a score data file (metadata)

Figure 3 :

 3 Figure 3: Example of the HTML display of the scoreboard utility

Figure 4 and

 4 Figure4and Figure5show examples of the visualization obtained from the scoreboard utility for score plots, panel plots and summary table, and for selected locations and scores in the E-HYPE test score dataset.

Figure 4 :Figure 5 :

 45 Figure 4: Screenshots of the HTML display interface of the scoreboard utility for 8 locations in the E-HYPE test score dataset, the CRPSS score (top) and Correlation coefficient (bottom)

  potential of radar-based ensemble flash-flood forecasts, including evaluation against deterministic discharge forecasts (Liechti et al., 2013): 3 catchments in the southern Swiss Alps. Hourly runoff, lead time up to eight hours Brier skill score, FAR and POD, global approach to producing hydrological ensemble forecasts in river basins where in situ data are sparse (Voisin et al., 2011): 4 outlets at the Ohio River basin, USA. Daily gridded runoff forecasts, be located in the ~/R home directory of RStudio, or the file location must be updated in server.R and ui.R. Annex 4 -Format specifications for input files to the IMPREX score database 6.4 Below is the file specification with individual descriptions (available in a Read_Me directory).



  Provider : your name  CaseStudy: your case study ("Jucar River Basin" for instance)  ForecastSystem: your forecast system  ForecastType: the name you want to give to your experiment, e.g. the name of your model, with its time step if you can have several time steps. It is important to give a different name if you want to compare the new configuration with an older one. The score name should not appear here.  ModelVariable: the name of the variable on which the score is evaluated.  ScoreType: the name of the score  ScoreLeadTimeUnit: "Day", "Week", "Month", etc. are examples. If every 15 days you make a seasonal forecast at a monthly time step for the next 6 months, you must put "Month".  VerificationPeriod: on which period you computed your score  Data:  LocationID: whatever code you use to design your station / grid point / area on which the score is computed. This is read as character  River_names: read as characters  Station_names: read as characters  Latitude: read as numeric  Longitude: read as numeric  Score_Value: your actual score values! Read as numeric  LeadTime: the leadtime that is evaluated. If every 15 days you make a seasonal forecast at a monthly time step for the next 6 months, you must put here values from 1 to 6.

Table 1 :

 1 Software specification for postgreSQL

	Installation source:	https://www.postgresql.org/download/
	Version:	Tested on 9.4 and 9.5

http://hepex.irstea.fr/hepex-science-and-challenges-verification-of-ensemble-forecasts/

A database is a collection of files that reside on a server. A database instance is a set of memory structures that manage database files. The instance can also refer to the allocated memory and collection of processes running on a server.