Introduction

In the 1920s, Lewis Fry Richardson envisioned "human computers" solving the equations of fluid mechanics for describing the atmospheric dynamics [START_REF] Richardson | Weather Prediction by Numerical Process[END_REF]. Shortly thereafter, weather forecasting became in one of the classical computational challenges. In particular, the first operational computer (the Electronic Numerical Integrator and Computer, ENIAC) resolved in 1950 a filtered version of these equations set by John von Neumann, Ragnar Fjørtoft and Jules Charney, which constituted the first problem undertaken by this computer [START_REF] Charney | Numerical Integration of the Barotropic Vorticity Equation[END_REF]. Since then, the description and complexity included in the numerical weather models, as well as their spatial and temporal resolutions have grown rapidly, in a continuous challenge against computing capacity.

Until the last decade, weather forecast has relied in a deterministic paradigm.

Notwithstanding, the chaotic nature of the nonlinear dissipative fluid equations leads to a high sensitivity to initial conditions that makes that such deterministic forecasts are subject to errors and should not be considered as exact predictions (even taken into account the computational power available and years of research). These errors have led to quantify forecasting uncertainties, and have also provided an opportunity for other disciplines (mathematicians, statisticians) for characterizing and correcting systematic errors in weather forecasts.

A modern approach to quantify the uncertainty in weather forecasts is the so-called ensemble forecast, which is constituted by a collection of deterministic forecasts differing in either their initial atmospheric conditions or the numerical parametrizations of the forecast model. Ensemble forecast allows for the estimation of the likelihood of the occurrence of weather events, as their constituent members will typically differ in their forecasted values. This scheme embraces the chaotic nature of the atmospheric behavior and seeks to provide multiple plausible realizations of the development of the weather event. It is worth to mention that several meteorological centers use up to 50 model ensemble members which form a probability density function of the development of the weather. The ensemble approach is very successful for traditional meteorological parameters as pressure levels or temperatures. An example application is forecasting fluctuating renewable energy sources, in particular solar irradiance.

Accurate Direct Normal solar Irradiance (DNI) forecast is of utmost importance for the accurate management of energy markets, for the management and operation of Concentrating Solar Thermal Power (CSTP) plants, and for the power generation control by means of thermal energy storage (if available). This allows maintaining the grid stability in CSTP power management and ultimately facilitates the widespread implementation of CSTP technologies [START_REF] Mathiesen | Evaluation of numerical weather prediction for intra-day solar forecasting in the continental United States[END_REF]. The optimum DNI forecasting method depends strongly on the timescale of interest, which ranges from horizons of a few seconds or minutes to few days-ahead. Numerical models are the best forecasting approach for the 1-5 days [START_REF] Perez | Validation of short and medium term operational solar radiation forecasts in the US[END_REF]. It is worth to highlight that the vast majority of conventional generation is scheduled in the day-ahead market [START_REF] Kleissl | Solar Energy Forecasting and Resource Assessment[END_REF].

Unfortunately, persistent errors in the numerical models and the selection of the initial conditions for initializing them makes that the ensemble forecast approach nowadays does not provide reliable representations of DNI. In the particular case of DNI, in addition, the high variability of atmospheric phenomena as clouds and aerosols, the complex atmospheric modeling on which NWP models rely, the extremely large underlying systems for data capture and assimilation and, in general, the large uncertainty associated with weather behavior make accurate localized and high-frequency forecasting a very difficult task and NWP models still do not resolve the local weather details [START_REF] Lorenz | Prediction of solar irradiance and photovoltaic power[END_REF]. In order to obtain an optimized local prediction, postprocessing techniques should be used to refine the output of NWP models.

NWP derived solar irradiance forecasts have been improved considerably in the last years due to advancements in the post-processing of models. They are worth to mention classical approaches (Model Output Statistics, MOS) and the Dynamic Integrated ForeCast System (DICast) developed in the late 1990s at the National Center for Atmospheric Research (NCAR).

In this report, we describe a statistical post-processing of DNI forecasting of NWP based on Machine Learning and ground measurements.

The rest of the report is organized as follows: Section 2 introduces the Numerical Weather Predictions Models (both deterministic and probabilistic approaches); Section 3 describes Postprocessing of weather forecasting, providing results in both deterministic and probabilistic approaches; Section 4 summarizes the conclusions; Section 5 enlists references.

Numerical Weather Prediction models

In PreFlexMS project, the meteorological forecasts used as input data by CENER postprocessing are provided by AEMET. This section briefly describes these inputs. For a more detailed description of the characteristics of both models and the parameters used in this project, the reader is referred to Section 1 of the deliverable 4.3.

Deterministic Weather forecasts in PreFlexMS

Fundamentals

Numerical Weather Prediction (NWP) models allow predicting the behaviour of the atmosphere by means of the numerical resolution of a set of hydrodynamic equations which describe the different physical processes taking place there. To achieve this, it is necessary to know the current state of the atmosphere, defined by the spatial distribution of physical variables such as wind or temperature, so that numerical models can produce forecasts of the future weather.

The models output consists of predicted values of the aforementioned variables in a grid of geographical points, with a temporal resolution in the range of hours and a spatial resolution in the range of kilometres for current models. These predictions are bound to have an error, due to the different temporal and spatial scales involved, the insufficient knowledge of the current state of the atmosphere and the physical laws which govern it at these scales, the different temporal and spatial scales involved, and the non-linearity of the problem.

These problems are especially severe when forecasting solar radiation, which strongly depends on meteorological variables difficult to predict and with a high variability in space and time, such as cloudiness or aerosol content. Anyway, this problem can be addressed by postprocessing methods, which can take advantage of the knowledge of local conditions to reduce significantly the errors.

Deterministic Weather forecasts in PreFlexMS

Two NWP models, Arome-HARMONIE (Hirlam Aladin Regional/Meso- The following HARMONIE forecasted variables (Table 1) will be used as meteorological input of the post-processing method described in section 3. The IFS is a deterministic global model with a horizontal resolution of about 0.1° and 137 vertical levels. It gives 3-hour outputs up to ten days forecast length, two times a day: at 00 and 12Z. The following IFS forecasted variables (Table 2) will be used as meteorological input of the post-processing method described in section 3.

D4.4 Post-processing for deterministic and probabilistic weather forecasting H2020-PreFlexMS/ Grant no. 654984

Probabilistic weather forecasts 2.2.1 Fundamentals

The traditional deterministic approach gave way to a new paradigm with richer information than a single solution of the future state of the atmosphere. The new paradigm includes quantitative information about the uncertainty (errors) of the predictive process. The atmospheric non-linear behavior, consequently chaotic, must be treated now in a probabilistic way by means of the generation of multiple forecasts starting from slightly different but equally probable initial conditions in order to characterize the uncertainty of the prediction [START_REF] Leith | Theoretical Skill of Monte Carlo Forecasts[END_REF].

To capture these sources of uncertainty, many operational and scientific centres worldwide produce ensemble forecasts (e.g. NCEP, ECMWF, etc.) since the early of 1990s. The basic idea behind ensemble forecasting is to run multiple (ensemble) forecast integrations from slightly perturbed ICs (ICs forecast error source) coming from multiple models and/or perturbing model formulation (model formulation forecast error source).

The ensemble prediction system (EPS) is a tool for estimating the time evolution of the Probability Density Function (PDF) viewed as an ensemble of individual selected atmospheric states. Each of these initial different states is physically plausible. The spread of the states is representative of the prediction error [START_REF] Toth | Ensemble Forecasting at NCEP and the Breeding Method[END_REF].

Not only global model can be used in developing EPS, but also Limited-area Models (LAMs) are involved in designing EPS, normally for the short range and covering fine domains in order to

Probabilistic Weather forecasts in PreFlexMS

The configuration of the 20-member AEMET ensemble, called gamma Short Range Ensemble Prediction System (γ-SREPS) is shown in Fig. 1. Same output variables as described in Table 1 within section Error! Reference source not found. but for each ensemble member will be used as meteorological input of the postprocessing method described in section 3.

According to the guidelines on ensemble prediction systems and forecasting given by the WMO [9], a statistical post-processing is needed in order to correct systematic errors in models and thereby add value to direct NWP model output. These errors are particularly important for surface parameters (for example, 2-m temperature, 2-m humidity, 10-m wind speed, precipitation, and total cloudiness) and are linked to local conditions.

More precisely, statistical post-processing can be used to:

 Remove systematic biases.  Adjust ensemble spread.  Quantify uncertainty not represented directly by the EPS.

 Predict what the model does not represent explicitly.

γ-SREPS will be calibrated using the former prediction's skill to correct the current probabilistic forecast according to different methodologies which were proposed in deliverable D4.1 to build calibrated probabilistic forecasts from ensembles as Bayesian Model Averaging and Extended Logistic Regression (see D4.1).

Post-processing of weather forecasting 3.1 Fundamentals

The solar irradiance forecast has not been treated traditionally as a priority in NWP models. Conversely, its role has been mainly focused in the average surface energy balance. This situation, influenced by the traditional lack of major stakeholders, is changing in recent years in which substantial investments are now being made in this field due to the increasing demand for operational and improved solar forecasts. Verification efforts have recently been undertaken with respect to hourly resolved DNI forecasts as provided by the European Centre for Medium-Range Weather Forecasts (ECMWF) [START_REF]Verification of ECMWF and ECMWF/MACC's global and direct irradiance forecasts with respect to solar electricity production forecasts[END_REF]. Notwithstanding, it is clear that the industry demands a more detailed site dynamic of DNI than those provided by NWP models [START_REF] Göber | Could a perfect model ever satisfy a naïve forecaster? On grid box mean versus point verification[END_REF]. Post processing methods are frequently applied to refine the solar irradiance of NWP models [START_REF] Lauret | A benchmarking of machine learning techniques for solar radiation forecasting in an insular context[END_REF]. In particular, they may be utilized to reduce systematic forecast errors (through the correction of systematic deviations), to account for local effects (e.g. topography) and for combining the output of different models in an optimum way.

The DNI sequence can be seen as a time series and therefore statistical models can be built to capture its underlying random processes and predict the next values. Several statistical techniques can be employed to forecast solar radiation time series. The spectrum of methods can range, for instance, from linear models like the autoregressive (AR) model to nonlinear models like artificial Neural Networks (NNs).

It is worth to highlight the recent use of Machine Learning techniques like the Support Vector Machines (SVM) in classification and regression problems. They present a nonlinear approach to solve the problems, and also the robustness of their algorithm avoids typical problems like overfitting. Also, SVM allow the possibility of integrate explicative variables from different origin and formats. To finish, it is also very important that computational requirements of this Machine Learning are very low and, therefore, it is very attractive to include them in an operational system.

SVM plays an ever more important role in classification and regression analysis. In particular, this technique has shown excellent performance in time series forecasting [START_REF] Müller | Predicting time series with support vector machines[END_REF]. SVM were developed initially for binary classifications [START_REF] Cortes | Support-vector networks[END_REF], by searching for the optimal separating hyperplane between the classes through the maximization of the margin between the classes' closest points. This task can be formulated as a quadratic optimization problem to be solved by well-established techniques. Learning algorithms in SVM use high-dimensional features, and its performance drastically depends on the definition of its model parameters. There are strategies for selecting parameters and also for model parameters alignment [START_REF] Bao | A PSO and pattern search based memetic algorithm for SVMs parameters optimization[END_REF][START_REF] Friedrichs | Evolutionary tuning of multiple SVM parameters[END_REF][START_REF] Lorena | Evolutionary tuning of SVM parameter values in multiclass problems[END_REF].

Fig. 2 shows an optimal separation. The separating hyperplane (green) is defined by the middle of the margin between by the support vectors (points lying on the boundaries). and, therefore, the width of the margin is

𝑑 = 2 ‖𝑤‖
At this point the problem to find the margin can be solved by finding 𝑤 and 𝑏 such as 𝑑 = 2 ‖𝑤‖ is maximized; and for all {(𝑥 𝑦 , 𝑦 𝑖 )} 𝑤 • 𝑥 𝑖 + 𝑏 ≥ 1 𝑖𝑓 𝑦 𝑖 = 1; 𝑤𝑥 𝑖 + 𝑏 ≤ -1 𝑖𝑓 𝑦 𝑖 = -1. That can be formulated in a simpler form as a quadratic optimization problem. This is a well-known class of mathematical programming problems, and many algorithms exist for solving them:

𝑀𝑖𝑛𝑖𝑚𝑖𝑧𝑖𝑛𝑔 1 2 ‖𝑤‖ 2 𝑆𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑦 𝑖 (𝑤 • 𝑥 𝑖 + 𝑏) ≥ 1, 𝑓𝑜𝑟 {𝑖 = 1 … 𝑁}
The solution involves the construction of the dual problem that starts with the Lagrangian L

𝐿(𝑤, 𝑏; ℎ) = 1 2 ‖𝑤‖ 2 -∑ ℎ 𝑖 𝑁 𝑖=1 [𝑦 𝑖 (𝑤 • 𝑥 𝑖 + 𝑏) -1]
Where ℎ = (ℎ 1 , … . ℎ 𝑁 ) is the vector of non-negative Lagrange multipliers. Minimizing L over w and b:

𝜕𝐿 𝜕𝑤 = 𝑤 -∑ ℎ 𝑖 𝑦 𝑖 𝑥 𝑖 𝑁 𝑖=1 = 0 𝜕𝐿 𝜕𝑏 = ∑ ℎ 𝑖 𝑦 𝑖 𝑁 𝑖=1 = 0
Therefore, the optimal value of w is

𝑤 * = ∑ ℎ 𝑖 𝑦 𝑖 𝑥 𝑖 𝑁 𝑖=1
And using the above result we have The solution of the problem depends on the inner-product between data points (i.e., 𝒙 𝒊 • 𝒙 𝒋 ) instead data points themselves. Successive modifications of this first problem have been made to cover the real situations. First, the concept of soft-margin allows the misclassification of set points. Furthermore, the inclusion of kernel functions replacing the inner products allows solving non-linear separating situations.

𝐿(𝒉) = ∑ ℎ 𝒊 - 1 2 ‖𝒘 * ‖ 2 = ∑ ℎ 𝒊 - 1
In this section, a new post-processing scheme based on SVM is presented and applied both to deterministic and probabilistic NWP DNI forecasts. The performance of this approach is compared against raw DNI NWP model forecasts.

Deterministic forecasts post-processing

Information provided by local and validated ground measurements allows the local adaptation of NWP forecasted DNI, providing site-specific DNI behaviour not well reproduced by NWP models. This procedure, usually referred as site-adaptation, has to consider many aspects in operational mode, as data flow management, update of data and operative performance. This section briefly presents the forecast post-processing system designed, its main features, operative working scheme and data flow.

System architecture

The post-process of the NWP model forecasted DNI can be divided into two phases: the first one contains the training of the Machine Learning modules; meanwhile the second one applies these modules on the newly available weather forecast. It is worth to highlight that the new weather forecast is used both as input of phases 1 and 2, but playing different roles depending of the stage.

The procedure is summarized in Fig. 3, where a general system data flow is shown. First, the available historic ground measurements and a simultaneous historic weather forecast dataset (Fig. 3, blue round shapes) provide the inputs to the Machine Learning Modules (Fig. 3, orange rectangle). The goal of the Machine Learning module is to find a relationship among the weather forecasts and the ground data to eliminate systematic deviation of the meteorological predictions. Once these relationships are established, new weather forecasts provided by the NWP model (Fig. 3, green round shape) feed the trained Machine Learning modules (Fig. 3, red rectangle) to generate forecasted DNI values. Even if it is recommended at least one year of coincident forecasted DNI and validated ground measurements as historic dataset, the system is designed to learn continuously from new ground measurements and NWP forecasts.

Validation of ground measured data from historic database

As it is said, ground data play the key role to the local characteristics of the climate. Therefore, accurate time series of solar radiation data (DNI and GHI) are required for the optimum training of forecasting post process. Erroneous data would imply a bad understanding of the DNI behavior and would influence the operative prediction. Hence, it is needed to use a real time treatment of ground data as an online monitoring of the station.

The data base collecting the ground measurements is supervised by quality analysis procedures. Criteria established by the BSRN for measured solar radiation are applied [START_REF] Ohmura | Baseline Surface Radiation Network (BSRN/WCRP): New Precision Radiometry for Climate Research[END_REF]. This step is carried out in real time to ensure that only correct data are included in the system. The BSRN recommendations include the following steps (Fig. 4):  Visual inspection. The first recommended check is a visual inspection of measured solar irradiance data by an experienced technician, for detecting wrong values or inconsistencies which cannot be found by automatic tests.

 Physically Possible test, intended to detect extremely large errors in the measurements and the large random errors introduced during data handling.

 Extremely Rare test, used to evaluate whether the measurements are within the limits known as extremely rare.

 Across Quantities test, based on empirical relations of the different quantities measured.

To carry out the tasks of monitoring and quality check, the prediction system is connected to a tool developed by CENER named ATDR (Adquisición y Tratamiento de Datos de Radiación), shown in Fig. 5. As example the following figures show differences tool screens. (Access to public information CENER BSRN station for real time monitoring: http://it.cener.com/solar). This database has the responsibility to deliver the most recent ground data to the forecasting system.

NWP model predictions management

The periodicity update of weather forecast establishes the final update of the predictions delivery. So, the schedule of the NWP model will determine the schedule of the whole system running. The first point of the post-process system consists of the download and integration of the NWPM output. This output is usually generated in GRIB format (GRIdded Binary or General Regularly-distributed Information in Binary form [9]), a concise data format commonly used in meteorology to store historical and forecast weather data.

The operative system requires a frequent access to the data (for selecting subsets of data, synchronizing or smoothing among other tasks). Consequently, the use of data formats that allow a fast and easy access is desirable. in this work, we have selected NetCDF format (Network Common Data Form [START_REF] Opengeospatial | OGC standard netCDF Classic and 64-bit Offset. 2013. D4.4 Post-processing for deterministic and probabilistic weather forecasting H[END_REF]), a set of software libraries and self-describing, machineindependent data formats that support the creation, access, and sharing of array-oriented scientific data. This format is commonly used in climatology, meteorology and oceanography applications (e.g., weather forecasting, climate change), as well as in GIS (Geographic Information Systems) applications.

One of the netCDF characteristics that present advantages in the post-processing system is the multi-dimensional treatment of the data that allows present each variable as a multi-array. Concretely, each meteorological variable presents the following dimensions:

 Latitude and longitude: identify the grid point  Time: hour when the weather prediction is update  Date: date of each predicted data  Step: the horizon of forecasts.

Machine Learning modules

The first step of the post-process is centered in data treatment as is shown in the previous section. Taking into account the principal source of variability and nature of DNI, it is immediate to suppose that the treatment of clear and cloudy situations should be different. Therefore, the first module of the system has the goal of deciding if the time instant to be post-processed will correspond to clear or cloudy sky conditions, taking into account forecasted meteorological variables and DNI.

After this first decision, two different options are possible (Fig. 6):

 Clear sky conditions. The clear sky module acts to post-process the DNI forecasts. This module consists on a clear sky model which uses forecasted atmospheric parameters.

 Cloudy sky conditions. A Machine Learning module is used in this case, consisting in a two-step model that selects the similar situations in the historic and fits a nonlinear model that post-processes the DNI forecast from the meteorological information. The Machine Learning modules are briefly described below:

The goal of this module is to determine if the time instant to be post-processed will correspond to clear or cloudy sky conditions. To achieve this, it implements a supervised classification problem. The historic ground data are classified as clear or cloudy situation, in the same time resolution that coincident historic NWP model forecasts. The classification is based on the NWP DNI forecast, which is compared with the corresponding DNI under clear sky conditions estimated using the forecasted atmospheric parametrization. Simultaneously, each data point is associated to a set of variables or characteristics offered by the historic NWP forecasts. In case of Harmonie predictions a vector of four features is used as classificatory variables: ambient temperature, relative humidity, surface level pressure and total cloud cover. The total cloud cover prediction is temporally and spatially smoothed: spatially it is smoothed by mean of a 3x3 grid points (7.5 x 7.5 km) meanwhile the temporal feature is smoothed using a four steps (4 h) rolling window. The classification algorithm for the incoming forecast in this module is a classification support vector machine; concretely a C-SVM [9], the general expression and the inputs are described as follow.

( , , , )

p n n n n

C f P T rH Tcc 

Where the sub index p refers to the prediction and the sub index n refers to the NWP forecast.

This second module has the goal of forecast DNI when a cloudy sky situation is identified. It is also based on Machine Learning models fed by historic NWP forecasts with coincident and validated ground measurements. The module is divided into two steps: the first one tries to select the most useful data among all those available in the historical data base. To do that, a version of the k-nearest neighbor's (knn) algorithm [START_REF]Verification of ECMWF and ECMWF/MACC's global and direct irradiance forecasts with respect to solar electricity production forecasts[END_REF] is used. The dissimilarity among data is calculated from the sun elevation angle, the weather forecast of atmosphere surface level pressure, ambient temperature, and relative humidity. The output of this first step is a subset of the whole available data in the historic that will act as the final training set of the nonlinear regression machine learning. Illustrative examples of this step of the methodology can be seen in Fig. 8, in the left plot the whole set of available data are sky blue points meanwhile the selected subset are plotted in red color. The right plot corresponds to the selected training subset and the fitted model (red points), the dark blue and magenta points correspond to the predicted and real data respectively. [START_REF] Gastón | A new adaptative methodology of Global-to-Direct Irradiance based on clustering and kernel machines techniques[END_REF].

The second step of the module consists on a non-linear regression model based on support vector machines, concretely a ε-SVM with Gaussian kernel [START_REF] Göber | Could a perfect model ever satisfy a naïve forecaster? On grid box mean versus point verification[END_REF]. This non-linear regression has shown good generalization ability for various function approximation and time series prediction problems.

Module 2: Cloudy sky model Training set Fitted model

Suppose we are given training data {(𝑥 1 , 𝑦 1 ), … , (𝑥 𝑛 , 𝑦 𝑛 )}  𝑋  𝑅 , where x and y are the input parameters and target values. The aim of the procedure is to find a function g that has the greatest  deviation from actual targets yi for all training data. In our case, the feature variables are the weather forecasts of GHI and DNI meanwhile the goal variable is the measured DNI.

( , )

p n n

DNI g GHI DNI 

Radiative transfer models are used to calculate the radiation field for given atmospheric and surface conditions. However, transfer radiation model forecasts have several shortcomings primarily related to insufficient data to accurately apply the models, and also insufficient information provided by forecasted inputs (i.e., aerosols and water vapor). It is worth to highlight the libRadtran, a software package is a suite of tools for radiative transfer calculations in the Earth's atmosphere [START_REF] Mayer | Technical note: The libRadtran software package for radiative transfer calculations -description and examples of use[END_REF]. It may be used to compute radiances, irradiances and actinic fluxes in the solar and terrestrial part of the spectrum.

Even if a wide variety of aerosol information is allowed to be included in the libRadtran, the closed scheme of this model hinder a site-specific adaptation and consequently the use of flexible and adaptable clear-sky models seems to be more appropriate for their use in our post-processing system.

In the development of the European Solar Radiation Atlas, several models were considered for the computation of the solar irradiance under clear sky conditions. The application of fitting techniques allowed for the parametrization of these models using hourly measurements spanned over several years and for several locations in Europe [START_REF] Rigollier | On the clear sky model of the ESRA -European Solar Radiation Atlas -with respect to the heliosat method[END_REF]. This adaptability is a key aspect in the suitability of these models in local forecasting systems, which are based on the characterization of the Linke turbidity factor, which is a function of the scattering by aerosols and the absorption by gases, mainly water vapor. When combined with the atmosphere molecules scattering, it summarizes the turbidity of the atmosphere, and hence the attenuation of the incoming solar radiation. In this scheme, DNI is given by the following expression:

𝐷𝑁𝐼 = 𝐼 0  𝑠𝑖𝑛(𝛾 𝑠 )𝑒𝑥𝑝[-0.8662 𝑇 𝐿 (𝐴𝑀2) 𝑚 𝛿 𝑅 (𝑚)]
Where 𝐼 0 is the solar constant (extraterrestrial irradiance normal to the solar beam at the mean solar distance), 1,367 W/m 2 ;  accounts for the Sun-Earth distance correction; 𝛾 𝑠 is the solar elevation angle; TL(AM2) is the Linke turbidity factor at an air mass equal to 2; m is the relative optical air mass; 𝛿 𝑅 (𝑚) is the integral Rayleigh optical thickness. As the solar altitude decreases, the relative optical path length increases. Also, this quantity decreases with increasing station height above the sea level (z). A correction procedure is applied, obtained as the ratio of mean atmospheric pressure, p, at the site elevation, to mean atmospheric pressure at sea level, p0, which is particularly significant in mountainous areas.

The relative optical air mass (dimensionless) is given by the following expression which relies in the true solar elevation, 𝛾 𝑠 𝑡𝑟𝑢𝑒 :

𝛾 𝑠 𝑡𝑟𝑢𝑒 = ( 𝑃 𝑃 0 ⁄ )
𝑠𝑖𝑛(𝛾 𝑠 𝑡𝑟𝑢𝑒 ) + 0.50572(𝛾 𝑠 𝑡𝑟𝑢𝑒 + 6.07995) -1.6364 Where the true solar elevation considers the atmospheric refraction The station height correction given by:

𝑃 𝑃 0 ⁄ = 𝑒𝑥𝑝( -𝑧 𝑧 ℎ ⁄ )
Being z the site elevation (in m), and zh the scale height of the Rayleigh atmosphere near the Earth's surface (8,434.5 m).

Finally, the parameter 𝛿 𝑅 represents the optical thickness of a pure Rayleigh-scattering atmosphere per unit of air mass along a specific path length:

𝑖𝑓 𝑚 ≤ 20 (𝛾 𝑠 ≥ 1.9º)

Operative working scheme and data flow.

Up to now, the scheme of the post-processing with its modules is described. However, this post-process needs to be implemented in an operative system that must be able of delivering DNI predictions on time and in the required data format.

The flow of actions presented in the operations can be seen in Fig. 9. It presents three steps: Availability of new information, run of post-process and delivery of DNI predictions. The first step is triggered from the availability of new weather forecasts. So, each new run of the NWP model starts the procedure. Once the NWP output (GRIB files) is available, they are downloaded and integrated in the system. The meteorological variables used in the postprocess are selected and the historical weather predictions database is updated. Ground measurements are also updated and integrated in the database after its quality validation. The availability of new measurements is used both to estimate the Linke Turbidity index according to the ESRA model (see clear sky module) and to train the Machine Learning modules.

Once the new data are downloaded and integrated into the database, the post-processing modules run. To ensure efficiency, the three modules are run independently. Hence, it is generated forecasts of type of data, of clear sky DNI and by using Machine Learning to predict DNI. Finally the system generates predictions in the required data format and uploads files to the agreed ftp server.

It is important to remember that the frequency of NWPM updates can vary depending of the NWP center, but it is usual that it consists of four updates per day. An example schedule is presented in Fig. 10, where a 48 h step-ahead forecast with 1 h temporal resolution is shown. It is considered that the computational requirement of the NWP model employs 5 h to generate the grib files at the NWP center. The post-process and delivery of final forecasts needs 1 h to complete its calculi. This is a usual schedule that could vary if more often updates of the NWPM are available. The design of the post-process allows an automatic adaptation to other situations in terms of update frequency, temporal resolution or horizons to be predicted.

Probabilistic forecasts post-processing

While considerable effort has been devoted to the production of NWP models that accurately describe the physics of the atmosphere, as well as to the development of perturbations to the analysis that accurately represent the forecaster's uncertainty of the atmospheric state at the model initialization time, it remains the case that the evolution of the atmosphere is insufficiently resolved, and that the growth of the perturbations does not accurately reflect the state-dependent predictability of the atmosphere. Often the growth rates of the perturbations are slower than the growth rates resulting from the instabilities of the true atmospheric flow, and therefore many operational ensemble forecasts are under-dispersed.

EPS forecasts post-processing

The Ensemble Prediction System (EPS) of the ECMWF represents the uncertainty in the initial conditions by creating a set of 50 forecasts, starting from slightly different states that are close but not identical to the best estimation of the initial situation of the atmosphere. The divergence of the 50 forecasts offers an estimation of the uncertainty of the prediction. A graphical example of the impact of variation in initial conditions to the final forecast can be seen in Fig. 11.

The perturbation of initial conditions is made by a stochastic procedure. This methodology implies that the 50 members of the EPS are indistinguishable (it is not possible to make an historic of each one of the 50 models because), and consequently each new run generates independent and equiprobable members. The different approaches considered in the framework of this work have the goal of overcome this fact, and achieve the possibility of use the historic forecasts and measurements to post-process the 50 members of the EPS outputs and obtain more accuracy information, delivering finally 50 post-processed predictions of DNI.

The following procedures are being implemented:

The first option consists of the use of Bayesian regression methods to post-process the 50 EPS predictions of DNI. To do that, the EPS predictions will be considered as respective percentiles forecasts. So, thanks to the equiprobable character of the members, we can dispose of an historic percentiles, P2, P4 The second option is the implementation of a Machine Learning based post-processing on a robust statistical parameter that characterizes the EPS predictions, and to extend this treatment to the whole individual members. In this work, the median value of the 50 individual forecasts has been selected as the statistical parameter for applying the post-processing. This allows the possibility of replicate the deterministic scheme, considering the median of the EPS as a deterministic model, and thus following the same procedure described in Section 3.2. This procedure requires the availability of historic EPS forecasts coincident with validated ground measurements (at least 1 year) for training the Machine Learning modules. After that, the 50 predictions available in the EPS will be used as 50 different inputs to this median model. So, , the output of this methodology will be the post-processed full ensemble members, and not a statistical characterization of their probabilities of exceedance.

SREPS forecasts post-processing

The second option for generating probabilistic forecasts is the use of a set of different NWP models. This is the case of the SREPS system developed by AEMET (section 2.2.2), which runs a total of 20 different approaches. Each NWP forecasts comes from a combination of a global weather model and a mesoscale numerical model. Several global models (GFS, ECMWF, GSM, GME, and GEM) are used as inputs for the 4 mesoscale models selected (for more information, see section 2.2.2).

This convection allows the system runs at 2.5 km horizontal resolution using two different dynamical cores, and physical parameterization packages of the Harmonie model (AROME and ALARO), the WRF-ARW model and the NMMB model with boundary conditions from 5 different global deterministic models. Summarizing, this approach provides 20 independent forecasts, being also available their respective historical database. Therefore, the postprocessing scheme presented in previous sections can be replicated and the whole system can be summarized like in Fig. 12.

For each different NWP model, the Machine Learning modules are trained separately according to the procedure described in Section 3.2, providing post-processed DNI forecasts.

The whole system delivers 20 different DNI predictions.

Conclusions

Post-processing techniques are commonly applied to NWP DNI forecasts, correcting biases and accounting for local effects. In this regard, value is added to the raw NWP model output through a number of different methods.

In this report, we describe the application of Machine Learning techniques in the search of a functional scheme for post-processing the outputs of the NWP model achieving local adaptation and removing bias, based on a training data set of historical DNI observations and forecasts.

The NWP models post-processing scheme consists on a complete system of modules that treat the outputs of the NWP model to obtain accuracy predictions of DNI. Some of its main features are summarized below:

 The design of the system comes from the solving problems theory and is based in Machine Learning techniques.

 The modularity of the system allows the application of the post-processing to different NWP models. This is used in the case of a multi-model scheme for probabilistic forecasts.

 It is designed to ease the inclusion of new developments like new classification and regression algorithms.

 The post-processing system is thought to be run in an operational way to deliver forecasts on time and data format as requested by the PreFlexMS project.

 The system covers the whole chain from ground data treatment, training and application of Machine Learning methods, and the delivery of information.

 Ground information and weather forecast are continuously updated and the system learns from the new information available, ensuring the adaptability of the models.

D4. 4

 4 Post-processing for deterministic and probabilistic weather forecasting resolve the internal processes of convection over local areas. LAMs have horizontal and vertical boundaries whereas global models cover the entire Earth having only vertical boundaries. For LAMs, larger-domain models supply the data for the horizontal boundary conditions. Errors in forecasts from larger-domain models will move into the LAMs forecast domain and can amplify. High-resolution LAMs are constrained to limited areas due to high computational cost of running on such a fine mesh.Error sources in LAM EPS are basically the same as in global EPS but LAM require lateral boundary conditions that update the weather situation regularly throughout the integration. Uncertainty (errors) similar to those in the model supplying boundary conditions can spread rapidly across the limited domain of the high-resolution model forecast. Lateral boundary conditions largely control the position and evolution of features that cover the entire forecast domain. In a high-resolution mesoscale model running over a limited area, the placement and timing of synoptic-scale features are determined almost completely by the synoptic-scale model supplying the boundary conditions.

Fig. 1 .

 1 Fig. 1. AEMET-γ-SREPS. This convection permitting system runs at 2.5 km horizontal resolution using two different dynamical cores and physical parameterization packages of the Harmonie model (AROME and ALARO), the WRF-ARW model and the NMMB model with boundary conditions from 5 different global deterministic models (Table3):

Fig. 2 .

 2 Fig. 2. Optimal separation through SVM, highlighting separating hyperplane, margin and support vectors.

  𝑦 𝑖 𝑦 𝑗 𝒙 𝒊 • 𝒙 𝒋 and the dual optimization problem 𝑀𝑎𝑥𝑖𝑚𝑖𝑧𝑖𝑛𝑔 𝐿(𝒉) = ∑ ℎ 𝒊 -

Fig. 3 .

 3 Fig. 3. General scheme of the post-process.

Fig. 4 .

 4 Fig. 4. Quality checks for measured solar radiation.

Fig. 5 .

 5 Fig. 5. Example of data acquisition system user interface.

Fig. 6 .

 6 Fig. 6. DNI post-processing scheme.

  Clear sky conditions are identified as all data points with an absolute difference to the theoretical DNI in clear sky condition less than 10%. Two examples of DNI measurement and the respective DNI of clear sky are shown in Fig.7. In the left example, the first two hours of the day would be classified as cloudy instants meanwhile the rest of the data would correspond to clear situations. On the other hand, all data of the right example would be classified as cloudy instants.

Fig. 7 .

 7 Fig. 7. Example of DNI measurement and Clear Sky DNI to classify each data.

Module 1 :

 1 Data classificationD4.4 Post-processing for deterministic and probabilistic weather forecasting

Fig. 8 .

 8 Fig. 8. Example of data selected by the knn algorithm[START_REF] Gastón | A new adaptative methodology of Global-to-Direct Irradiance based on clustering and kernel machines techniques[END_REF].

Module 3 :

 3 Clear sky-modelThe following expression represents the transmittance of the beam radiation (𝑇 𝐷𝑁𝐼 𝑐𝑙𝑒𝑎𝑟 ) under clear-sky conditions:𝑇 𝐷𝑁𝐼 𝑐𝑙𝑒𝑎𝑟 = 𝑒𝑥𝑝[-0.8662 𝑇 𝐿 (𝐴𝑀2) 𝑚 𝛿 𝑅 (𝑚)]

Fig. 9 .

 9 Fig. 9. Operative forecast procedure.

Fig. 10 .

 10 Fig. 10. Typical schedule of forecast.

Fig. 11 .

 11 Fig. 11. Dispersion of forecasts due to variation in initial conditions. (Source ECMWF).

Fig. 12 .

 12 Fig. 12. Probabilistic multi model scheme.

Table of Contents 1. Introduction 2. Numerical Weather Prediction models

 of

	2.1	Deterministic Weather forecasts in PreFlexMS
	2.1.1 Fundamentals
	2.1.2 Deterministic Weather forecasts in PreFlexMS
	2.2	Probabilistic weather forecasts
	2.2.1 Fundamentals
	2.2.2 Probabilistic Weather forecasts in PreFlexMS
	3. Post-processing of weather forecasting
	3.1	Fundamentals
	3.2	Deterministic forecasts post-processing

3.2.1 System architecture 3.2.2 Operative working scheme and data flow.

Table 1 . Parameters from the HARMONIE model. Parameter definition Inst / Acc Units

 1

	Surface Solar Radiation (SW down global)	Accumulated	Jm -2
	Surface Direct Solar Flux (Surface parallel solar flux)	Accumulated	Jm -2
	Surface Pressure	Instantaneous	Pa
	Mean Sea Level Pressure	Instantaneous	Pa
	Total Cloud Cover	Instantaneous	%
	10 m U Wind Component	Instantaneous	ms -1
	10 m V Wind Component	Instantaneous	ms -1
	200 m U Wind Component	Instantaneous	ms -1
	200 m V Wind Component	Instantaneous	ms -1
	2 m Temperature	Instantaneous	K
	10 m Temperature	Instantaneous	K
	200 m Temperature	Instantaneous	K
	Low Cloud Cover	Instantaneous	%
	Medium Cloud Cover	Instantaneous	%
	High Cloud Cover	Instantaneous	%
	2 metre Relative Humidity	Instantaneous	%

Table 2 . Parameters from the ECMWF model. Parameter definition Inst / Acc Units

 2

	Surface Solar Radiation Downwards	Accumulated	Jm -2
	Total Sky direct Solar Radiation at Surface	Accumulated	Jm -2
	Surface Pressure	Instantaneous	Pa
	Mean Sea Level Pressure	Instantaneous	Pa
	Total Cloud Cover	Instantaneous	(0-1)
	10 m U Wind Component	Instantaneous	ms -1
	10 m V Wind Component	Instantaneous	ms -1
	200 m U Wind Component	Instantaneous	ms -1
	200 m V Wind Component	Instantaneous	ms -1
	2 m Temperature	Instantaneous	K
	10 m Temperature	Instantaneous	K
	200 m Temperature	Instantaneous	K
	Low Cloud Cover	Instantaneous	(0-1)
	Medium Cloud Cover	Instantaneous	(0-1)
	High Cloud Cover	Instantaneous	(0-1)
	2 m Dew Point Temperature	Instantaneous	K

Table 3 . AEMET-γ-SREPS members with corresponding NWP models and BCs.

 3

	Member #	NWP Model	Boundary Conditions
	01	HAR / AROME	ECMWF -IFS
	02	HAR / ALARO	ECMWF -IFS
	03	WRF -ARW	ECMWF -IFS
	04	NMMB	ECMWF -IFS
	05	HAR / AROME	NCEP -GFS
	06	HAR / ALARO	NCEP -GFS
	07	WRF -ARW	NCEP -GFS
	08	NMMB	NCEP -GFS
	09	HAR / AROME	MF -ARPEGE
	10	HAR / ALARO	MF -ARPEGE
	11	WRF -ARW	MF -ARPEGE
	12	NMMB	MF -ARPEGE
	13	HAR / AROME	CMC -GEM
	14	HAR / ALARO	CMC -GEM
	15	WRF -ARW	CMC -GEM
	16	NMMB	CMC -GEM
	17	HAR / AROME	JMA -GSM
	18	HAR / ALARO	JMA -GSM
	19	WRF -ARW	JMA -GSM
	20	NMMB	JMA -GSM

H2020-PreFlexMS/ Grant no. 654984