Josselin Le Gal

La Salle

Mathieu David

Philippe Lauret

A new reference model to benchmark probabilistic solar forecasts

Keywords: Benchmarking, Solar irradiance, Probabilistic forecasting, Climatology, Reference model

Probabilistic solar forecasting is becoming a major topic in the solar research community as it provides more information about the uncertainty of the forecast compared to deterministic forecasting. However, to facilitate the adoption of probabilistic forecasts within solar forecasting communities (industry and academic), the definition and the use of standardized best practices are a prerequisite. Among others, there is a need for benchmark models that are able to properly assess the performance of new probabilistic forecasting methods. In this work, we propose a new benchmark model called "CSD-CLIM" (for Clear-Sky Dependent Climatology). This reference model is evaluated against two other climatology benchmark models namely the naive climatology and a well-referenced model in the literature, the CH-PeEn (for Complete History Persistence Ensemble). The verification of compliance with a set of properties that a climatology benchmark model must follow demonstrates that the new CSD-CLIM model outperforms the naive climatology and that it can be a viable alternative to the CH-PeEn model. It is shown that the better performance of CSD-CLIM is due to a specific binning of the historical irradiance data based on the clear-sky irradiance values.

Fully documented templates are available in the elsarticle package on CTAN.

Introduction

It is now commonly accepted that solar forecasting is a cost-effective way to increase the share of solar energy in the electrical grid [START_REF] Pierro | Residual load probabilistic forecast for reserve assessment: a real case study[END_REF]. Recently, there has been a growing interest in solar probabilistic forecasting [START_REF] Van Der Meer | Review on probabilistic forecasting of photovoltaic power production and electricity consumption[END_REF]. Indeed, contrary to deterministic forecasts, probabilistic forecasts of intrinsically highly variable weather predictands like wind or solar bring more value to the grid operator as demonstrated by [START_REF] Zhu | The economic value of ensemble-based weather forecasts[END_REF] or [START_REF] Buizza | The value of probabilistic prediction[END_REF].

However, as for deterministic forecasts, a wide adoption of probabilistic forecasts in the solar forecasting communities (industry and academic) requires a set of best practices. For instance, one can cite first the existence of a specific framework for verifying the quality of solar probabilistic forecasts. Raising concerns about the verification of probabilistic forecasts and notably the use of improper scores to measure the performance of the probabilistic methods, [START_REF] Lauret | Verification of solar irradiance probabilistic forecasts[END_REF] have recently recommended a set of diagnostic tools and numerical scoring rules like the Continuous Ranked Probability Score (CRPS) to assess the quality of solar probabilistic forecasts.

A second point is related to the use of well-accepted reference models to fairly benchmark any new proposed forecasting methods on preferably standardized datasets (Yang et al., 2020b). In the realm of solar probabilistic forecasts, a reference model called the Persistence ensemble (PeEn) model [START_REF] Alessandrini | An analog ensemble for short-term probabilistic solar power forecast[END_REF] is routinely proposed to benchmark new probabilistic models [START_REF] David | Probabilistic forecasting of the solar irradiance with recursive ARMA and GARCH models[END_REF]. Unfortunately, [START_REF] Doubleday | Benchmark probabilistic solar forecasts: Characteristics and recommendations[END_REF] and [START_REF] Yang | A universal benchmarking method for probabilistic solar irradiance forecasting[END_REF] noted a wide spectrum of implementations of the PeEn model in the literature.

Besides, [START_REF] Yang | A universal benchmarking method for probabilistic solar irradiance forecasting[END_REF] emphasized the need for universal benchmark models whose implementation must depend only on the data at a particular site. This is why [START_REF] Yang | A universal benchmarking method for probabilistic solar irradiance forecasting[END_REF] proposed a universal benchmarking model, the Complete History-Persistence ensemble (CH-PeEn).

As stated by [START_REF] Yang | A universal benchmarking method for probabilistic solar irradiance forecasting[END_REF], the CH-PeEn model constitutes a consistent baseline model for assessing the skill of a forecasting method. The requirement of such benchmarks models has been also highlighted in [START_REF] Doubleday | Benchmark probabilistic solar forecasts: Characteristics and recommendations[END_REF]. In their work, the authors compared ten variants of six reference models, and the class of climatology reference models such as naive (no-skill) classical climatology and the CH-PeEn were implemented.

Similarly to skill scores used in the case of deterministic forecasts (Yang et al., 2020a), skill scores like the CRPS skill score, or "CRPSS" can be used to gauge the performance of a new probabilistic forecasting model against a reference easy-to-implement method. However, as noted by [START_REF] Yang | A universal benchmarking method for probabilistic solar irradiance forecasting[END_REF], different implementations of the benchmark model can hamper the interpretability of the skill score. Therefore, the computation of these skill scores should be done using a universal well-accepted benchmark model. Such a practice will promote a fair evaluation of probabilistic forecasting techniques.

Let us stress here the importance of using skills scores. Indeed, a score like CRPS obtained by a forecasting method is not in itself a measure of the skills of the forecast as the score strongly depends on the sky conditions of the considered location. For example, [START_REF] Alessandrini | An analog ensemble for short-term probabilistic solar power forecast[END_REF] pointed out that meteorological conditions of a site impact the quality of solar probabilistic forecasts. Hence, the CRPS score must be compared with the CRPS score of the reference model. The latter is expected to reflect the difficulty of forecasting at a particular site or equivalently to quantify the predictability of the solar irradiance at that specific location.

Following the need of easy-to-implement (naive) and universal reference models, it has been shown in [START_REF] Murphy | A new vector partition of the probability score[END_REF] that one component of the CRPS called the uncertainty corresponds to the CRPS of the climatology. The score of this naive climatology is only sensitive to the observations variability and therefore, for a given location and temporal resolution of the data, does not depend on any other kind of parameters. Thus, one way to avoid a CRPSS that depends on the implementation of the reference model is to use the uncertainty part of the CRPS as the baseline value. Moreover, it must be noted that, for meteorologists, the baseline model for computing skill scores is usually the climatology -see for instance [START_REF] Cusack | Assessing the usefulness of probabilistic forecasts[END_REF] or [START_REF] Binter | Applied Probabilistic Forecasting[END_REF].

However, while appealing, we will show in this work that the naive climatology is not the best candidate for being a reference model for the particular case of solar irradiance forecasting. Indeed, the raw GHI time series exhibit specific diurnal and seasonal patterns which are not taken into account by the naive climatology model. Consequently, these deterministic patterns increase the CRPS of this benchmark model (denoted hereafter UNC )1

Hence, it would be desirable to design a benchmark model which does not suffer from this issue.

In this work, we propose to take advantage of the clear-sky irradiance in order to compute a new reference model called the Clear-Sky dependent climatology or "CSD-CLIM". Unlike the CH-PeEn reference model which relies on hour-dependent predictive distributions, the CSD-CLIM model makes use of a binning process of the clear-sky irradiance to compute its CRPS. Additionally, it will be shown that, unlike CH-PeEn, the score of the CSD-CLIM can be directly computed from the historical data at hand without needing to first form the predictive distribution and then calculate the score on the historical dataset.

We will show also that the new model improves on the notion of universal benchmarking.

Besides, we will demonstrate that the CRPS of CSD-CLIM and CH-PeEn can reflect the difficulty of forecasting at a particular site.

This paper is structured as follows. Section 2 discusses the required properties that a good benchmark model should exhibit. Section 3 presents the context of the study and in particular, the data and sites used to assess the performance of the different climatology benchmark models. Section 4 details the benchmark models while section 5 verifies the compliance of the reference models with the required properties. A discussion related to the score obtained by the CSD-CLIM model and a detailed comparison of the methodologies pertaining to the CSD-CLIM and CH-PeEn models is conducted in section 6. Finally, section 7 will present our conclusions.

Required properties for a good climatology benchmark model

Particular attention must be paid to the selection of a benchmark model and the amount of information used to feed it. Since a benchmark model is mainly used to calculate a skill score of a new forecasting method2 , choosing a benchmark model requires addressing these questions:

1. What information should be given by a skill score ?

2. What score should correspond to the "0" skill score?

This could be subject to discussion, but in our opinion, the purpose of a skill score should be to indicate which part of the information given by the new forecast is naive (i.e. captured by the benchmark model), and to what extent it provides valuable extra-information (which should be credited to the particular skill of the forecast).

Furthermore, the "0" skill score should be defined by the best possible exploitation of all information derived from historical observations (i.e. the climatology). Thus, all the historical data should be exploitable by the benchmark model. Conversely, all extra-information (from meteorological data, satellite observation, etc.) treated by the new forecasting method and the potentially associated better performance should be credited to the merit of the forecast.

From the answer of these questions, we propose in this section to establish a set of required properties that benchmark models should meet. [START_REF] Doubleday | Benchmark probabilistic solar forecasts: Characteristics and recommendations[END_REF] The benchmark model should be easy-to-implement

P2

The implementation of the model must depend exclusively on the historical data at hand (and not on any other kind of parameters such as number of past measurements, forecast horizon/lead time)

P3

The score (CRPS) of the model (for a specific location and time resolution of the data) must be unique (or near unique) irrespective of the period or length of the period used to compute the score ("Time-invariance property")

P4

The model should verify the statistical consistency of the naive climatology (i.e. a perfect reliability when compared to new observations)

A1

The quality in terms of CRPS of the benchmark model should be as high as possible

A2

The score of the benchmark model should reflect the difficulty of forecasting at a particular location Consequently, this set of properties implies the following exclusions : P2 excludes the model PeEn ("Persistence Ensemble") developed by [START_REF] Alessandrini | An analog ensemble for short-term probabilistic solar power forecast[END_REF] and used for example in [START_REF] David | Probabilistic forecasting of the solar irradiance with recursive ARMA and GARCH models[END_REF][START_REF] Lauret | Probabilistic solar forecasting using quantile regression models[END_REF][START_REF] Pedro | Assessment of machine learning techniques for deterministic and probabilistic intra-hour solar forecasts[END_REF]. P2 and P4 exclude raw ensemble forecasts used in [START_REF] Golestaneh | Very short-term nonparametric probabilistic forecasting of renewable energy generation-with application to solar energy[END_REF] and [START_REF] Thorey | Ensemble forecast of photovoltaic power with online crps learning[END_REF] and other benchmark models based on simple post-processing of raw ensemble forecasts.

Note that, in this study, we focus on the class of climatology reference models which can be used to benchmark either intra-hour or hourly forecasts according the terminology used by [START_REF] Doubleday | Benchmark probabilistic solar forecasts: Characteristics and recommendations[END_REF].

Context of the study

Data

A selection of 20 sites serves as support for the comparison of the different benchmark models. This choice was made by trying to keep the widest possible spectrum in terms of different sky conditions and locations around the world. The vast majority of data was chosen from BSRN (https://bsrn.awi.de/) collection data in order to minimize the differences in data acquisition and data validation between sites. The remaining data comes from previous works dedicated to state-of-the-art of solar forecasting [START_REF] David | Probabilistic forecasting of the solar irradiance with recursive ARMA and GARCH models[END_REF][START_REF] David | Comparison of intraday probabilistic forecasting of solar irradiance using only endogenous data[END_REF]; [START_REF] Gal | Added-value of ensemble prediction system on the quality of solar irradiance probabilistic forecasts[END_REF]). The complete list of sites is given in

Clear-sky model

The building of the CH-PeEn and CSD-CLIM benchmark models (see section 4) requires a clear-sky model. As demonstrated by Yang ( 2019) who compared two clear-sky models in the assessment of the performance of the CH-PeEn model, the choice of the clear-sky model is of primary importance (See also [START_REF] Yang | Choice of clear-sky model in solar forecasting[END_REF]).

Among the different clear-sky models that can be found in the literature, one can cite the Bird Model [START_REF] Bird | Simplified Clear Sky Model for Direct and Diffuse Insolation on Horizontal Surfaces[END_REF], the Ineichen-Perez model [START_REF] Ineichen | A new airmass independent formulation for the Linke turbidity coefficient[END_REF] or the McClear model [START_REF] Lefèvre | Mcclear: a new model estimating downwelling solar radiation at ground level in clear-sky conditions[END_REF]. In this work, and following the work of Yang

Generating the predictive cumalative distrubution function (CDF)

The computation of the CRPS requires the building of the predictive cumulative distribution function (CDF) Ffcst (see Equation 3). In this section, we describe briefly how these forecast CDFs can be generated either for benchmark models or for Ensemble Prediction System (EPS).

Generation of climatology predictive distributions

For instance, the naive climatology forecast is an empirical (CDF) based on long period

of N historical sorted measurements (Y 1 , Y 2 , • • • , Y N ).
Here, to generate the predictive CDF for the benchmark models, we implement the classical approach [START_REF] Lauret | Verification of solar irradiance probabilistic forecasts[END_REF] that consists in building a piecewise constant function with a jump probability of 1 N at each Y i and null probabilities for events outside the set of historical measurements. The predictive CDF is given by

Ffcst (x) = 1 N N i=1 1 {x≥Y i } , (1)
where 1 {u} is the indicator function which has the value of 1 if its argument u is true and 0 otherwise. Note that in the case of the CH-PeEn model, predictive CDFs are built with a training set of historical ordered measurements for each hour of the day.

Generation of the day-ahead LQR probabilistic forecasts

Property A2 states that the CRPS of the benchmark model should be a proxy for judging a priori the quality of a probabilistic forecast. In order to be able to evaluate the proposed benchmark models regarding this property, we generate day-ahead probabilistic forecasts for the different sites listed in Table 2.

The day-ahead GHI ensemble forecast has been provided by the European Centre of Medium-Range Weather Forecasts (ECMWF). This ensemble forecast also called EPS (for ensemble prediction system) is constituted of 51 members : one unperturbed member (control member) and 50 perturbed members. The temporal resolution is of 3 hours and the spatial resolution is of 0.2 • in both longitude and latitude. Consequently, 3h GHI times series recorded on-site are compared with the nearest ECWMF pixel.

For an EPS with M ordered members (E 1 , E 2 , • • • , E M ), we use again the classical approach described in [START_REF] Lauret | Verification of solar irradiance probabilistic forecasts[END_REF] to build the predictive CDF related to the raw ECMWF ensemble forecast which reads as :

Ffcst (x) = 1 M M i=1 1 {x≥E i } . (2)
However, numerous studies (see for instance [START_REF] Vannitsem | Statistical Postprocessing of Ensemble Forecasts[END_REF]) have shown that CDFs drawn from raw EPS ensemble with the classical construction are statistically unreliable, meaning that the probability assigned to an event is not consistent with the observations [START_REF] Hamill | Verification of eta-rsm short-range ensemble forecasts[END_REF]. Hence, the use of calibration models (i.e. techniques which improve the reliability of raw ensemble forecasts) is a common practice. The interested reader is referred to [START_REF] Gneiting | Calibrated probabilistic forecasting using ensemble model output statistics and minimum crps estimation[END_REF] or (Le Gal La Salle et al., 2020) for details regarding the implementation of calibration techniques.

In this study, we propose to use a state-of-the-art non-parametric and very flexible calibration method, the Linear Quantile Regression (LQR) technique. The LQR method is depicted at length in (Le Gal La Salle et al., 2020).

Verification of probabilistic forecasts

In the verification framework proposed by [START_REF] Lauret | Verification of solar irradiance probabilistic forecasts[END_REF], the authors recommend the computation of a proper score like the Continuous Ranked Probability Score (CRPS) to evaluate the overall quality of a probabilistic forecast. We will recall here the mathematical definition of the CRPS.

The CRPS measures the distance between the forecast CDF and the CDF associated with the measurement x obs [START_REF] Hersbach | Decomposition of the Continuous Ranked Probability Score for Ensemble Prediction Systems[END_REF]. The CRPS is defined as

CRP S = 1 N N i=1 +∞ -∞ F i f cst (x) -F i x obs (x) 2 dx. (3)
Ffcst (x) is the predictive CDF and F x obs (x) is the cumulative distribution given by the Heaviside (or step) function H(x -x obs ), which is zero if x < x obs and one if x ≥ x obs . The squared difference between the two CDFs is averaged over the N forecast/observation pairs. The CRPS is negatively oriented (smaller values indicate a better forecast). Like the Brier score (see Appendix for the definition of the Brier Score), the CRPS is a proper score and can be decomposed into the three important attributes detailed in Appendix B. The decomposition is as follows

CRP S = REL -RES + U N C, (4)
where REL, RES and UNC are respectively the reliability part, the resolution part and the uncertainty part of the CRPS.

Reliability is an indication of the statistical consistency between the forecasts and the observations while resolution indicates how far the observations are discriminated from the climatological mean by the forecasts. Finally, the uncertainty term depends on the variability of the observations and will be further developed in section 4.1. As mentioned in the introduction, the uncertainty term corresponds theoretically to the CRPS of the naive climatology. Indeed, if we assume an infinite historical time series, the reliability is perfect (i.e. REL = 0) and the resolution is null (i.e. RES = 0). More details regarding the calculation of the different components of the CRPS can be found in Appendix B, Appendix C and to [START_REF] Lauret | Verification of solar irradiance probabilistic forecasts[END_REF].

Note that, in order to assess the reliability and sharpness properties of the different benchmark models, [START_REF] Doubleday | Benchmark probabilistic solar forecasts: Characteristics and recommendations[END_REF] used visual diagnostic tools like reliability and sharpness diagrams. In this work, we rely on the decomposition of the CRPS in order to obtain a quantitative measure of these two attributes.

Finally, and as mentioned in the introduction, the purpose of the benchmark models is mainly to be used as references in the calculation of skill scores. A skill score is the level of improvement of a forecasting model over the reference model. For example, the CRPS skill score reads as

CRP SS = 1 - CRP S model CRP S ref erence .
(5)

Climatology benchmark models for solar probabilistic forecasting

In this section, three reference models, the naive climatology (CLIM), the Clearskydependent uncertainty (CSD-CLIM) and the Complete-History Persistence Ensemble (CH-PeEn) are presented and their possible pros and cons are discussed.

The naive climatology (CLIM)

The climatology refers to the ensemble of all the observed values of a weather variable over a long period of time. The predictive CDF created from the aggregation of all these past observations forms the climatological predictive distribution. We denote hereafter the corresponding model as the naive climatology model (CLIM). The CRPS of the CLIM model can be computed either by using Equation 3 or by computing the uncertainty part of the CRPS which reads as [START_REF] Todter | Generalization of the ignorance score: Continuous ranked version and its decomposition[END_REF])

U N C = GHI M AX 0 U N C BS (x)dx. ( 6
)
UNC BS (x) is the uncertainty relative to the Brier Score (BS) (see Appendix B for the decomposition of the Brier Score) for a fixed level of irradiance x and GHI MAX is the maximum possible value of irradiance (also called climatological bound) proposed by [START_REF] Yang | A universal benchmarking method for probabilistic solar irradiance forecasting[END_REF] et [START_REF] Long | An automated quality assessment and control algorithm for surface radiation measurements[END_REF].

Let Y be the measurement of the predictand (here the GHI). For a fixed level x of GHI, UNC BS (x) is defined as

U N C BS (x) = o(x) 1 -o(x) , ( 7
)
where o is the frequency with which Y is lower or equal to x i.e.

o(x) = 1 {Y ≤x} N , ( 8
)
where N is the number of historical observations and 1 {u} the indicator function.

Note that Equation 6 allows a graphical representation of the UNC which will be extensively used in this study. For each level of GHI x, a point equal to UNC BS (x) is plotted, and UNC is given by the area under the curve created by this set of points. Such a representation is given in Figure 1.

The clearsky-dependent Climatology (CSD-CLIM)

In the meteorological community, the naive climatology depicted above is used as a reference model for the calculation of skill scores. However, it does not account for the strong time dependence experienced by a variable such as GHI (daily and seasonnal patterns).

It is well-known that the irradiance can be decomposed as

GHI = GHI CS * k * t , (9)
where GHI CS stands for the clear-sky irradiance and represents the irradiance if no cloud cover is observed, and is given by a chosen clear-sky model (see section 3.2). The clear-sky irradiance is fundamentally time-dependent and follows strong temporal patterns.

k * t is the clear-sky index and ranges theoretically between 0 to 13 and represents the share of GHI CS which is lost due to cloud cover. Equation 9shows that forecasting k * t and forecasting irradiance are equivalent tasks. Indeed, GHI CS is deterministically and fully determined by the clear-sky model. However, this is not taken into account by the naive climatology, which only reflects the variability of the GHI.

For example, a site which always experiences clear sky conditions (i.e k * t = 1) will exhibit a high uncertainty score UNC, even though the forecast is not difficult (knowing that k * t = 1).

Also, for various cases (e.g. early morning, late evening), the clear-sky model alone, which is not the uncertain part of the forecast, permits to determine that GHI is limited to lowirradiance values. These considerations lead to the idea that k * t and the clear-sky model should play a key role in the building of a benchmark model. Thus, the score (i.e. the CRPS) of this new benchmark model can be calculated using Equation 3, or equivalently by computing

CSD-UNC = N b i f i * U N C i . ( 10
)
For each bin i of clear-sky irradiance values (see Table 3), we calculate U N C i as defined by Equation 6and the frequency f i represents the relative frequency of each bin in the clear-sky model. The main goal of such a model is to discriminate the situations that the clear-sky model alone permits to separate. An example of the binning process for Desert

Rock is shown in Table 3.

In this study, we chose a number of N b = 30 bins. This choice could be questioned and is discussed in Appendix A.

bin i (W/m As shown by Figure 3, a substantial difference can exist between UNC and CSD-UNC (and by extension between the two reference models CLIM and CSD-CLIM). While for Marshall Island, CSD-UNC represents approximately 50% of UNC, the reduction is more drastic in the case of Desert Rock. This discrepancy can be explained with the histograms of k * t presented in Figure 4 which reveals that k * t is both lower and much more variable in Marshall Island than in Desert Rock. This has two consequences:

1. UNC is more important in Desert Rock, mainly because k * t is in average higher leading to a distribution of higher values of GHI (see Figure 3)

2. The forecasting task, which essentially consists in guessing the most probable value of k * t , is much more difficult in Marshall Island. It must be stressed also that, unlike CSD-UNC whose computation does not require the construction of predictive GHI distributions, the derivation of CH-PeEn implies the generation of forecasts and the assessment of the CRPS of these forecasts.

Results

In this section, we verify the compliance of the different benchmark models with the properties listed in Table 1. It must be noted that, by construction, the 3 climatology benchmark models meet property P1. The goal of this section is then to find whether the models follow P2 and P3, and to evaluate the performance of the benchmark models in the light of properties P4, A1 and A2.

Compliance with property P2

As claimed by P2 in Table given the benefits of using a clear sky model.

CSD-CLIM uses also a clear-sky model to bin the data. The same argument used to justify this usage for CH-PeEn is also valid for CSD-CLIM. Besides, CSD-UNC could also depend on the chosen number of bins N b . We justify in Appendix D that the number of bins does not have a strong impact on the final result, as long as the number is chosen reasonably large.

Finally, apart from the above details of implementation, we can state that the three climatology benchmark models follow property P2.

Compliance with property P3

Property P3 states that all benchmark models must be time-invariant i.e. that their resulting CRPS (for a specific location and time resolution of the data) must be unique or near unique. The verification of the time invariance of the CSD-CLIM model is detailed in Appendix D. As demonstrated in it, we can conclude that the CSD-CLIM produces a near unique score regardless of the period or the length of the historical data used to compute it. The CH-PeEn is also time-invariant, as demonstrated by [START_REF] Yang | A universal benchmarking method for probabilistic solar irradiance forecasting[END_REF]. Furthermore, it is well-known that the naive climatology is time-invariant, as soon as the amount of data considered for its computation is sufficiently large.

Compliance with property P4

Property P4 emphasizes the importance of reliability of the benchmark model. We As shown by Table 5, for any given considered site, the CSD-CLIM model exhibits the highest overall quality in terms of CRPS. The previous decomposition of the CRPS of the CSD-CLIM model shows that the overall better performance of CSD-CLIM originates from its high reliability. This finding strengthens the assumption that knowledge on clear-sky irradiance decreases the uncertainty4 associated with a forecast.

Compliance with property A2

As mentioned in section 3.3.2, in order to verify the compliance of the benchmark models with the additional property A2, we generate day-ahead LQR calibrated forecasts. The assumption of this study is that the score of a benchmark model should be related to the quality of the LQR forecasts (measured here by its CRPS). Figure 5 plots the results of the three benchmark models (x axis) versus the CRPS of the LQR forecasts (y axis). Table 5 also gives the CRPS of the LQR forecasts.

Figure 5: CRPS of the LQR forecasts vs CRPS of the benchmark models computed for each site of Table 2. To proceed further, we built a linear regression for each considered benchmark model and extracted the coefficient of determination R 2 . We found respectively a R 2 coefficient of 0.06, 0.56 and 0.63 for the CLIM, CH-PeEn and CSD-CLIM. Furthermore, it must be noted that a similar ranking has been established for other types of calibrated forecasting models described in [START_REF] Gal | Added-value of ensemble prediction system on the quality of solar irradiance probabilistic forecasts[END_REF]. Put differently, whatever the forecasting model, the best correlation is always obtained by CSD-CLIM. As a conclusion, we can state that CH-PeEn and CSD-CLIM outperform the naive climatology regarding property A2.

Overview

Finally, Table 6 The qualitative results of Table 6 suggest that CSD-CLIM should be preferred as a reference model. It appears to lead to the best trade-off between all the properties required by a climatology benchmark model (see section 2). For high clear-sky irradiances, even low GHI measurements can occur in Fouillole, which is not the case at Desert Rock. This enlarges the area under U N C i=24 BS for Fouillole, leading to a higher U N C i=24 , and consequently to a higher CSD-UNC.

Comparison of the binning approaches used by CH-PeEn and CSD-CLIM

In terms of CRPS, the results obtained with the CH-PeEn turned out to be quite comparable with those of CSD-CLIM. This is not surprising since the general idea behind these two models is very close : giving a time-of-the-day dependent image of the uncertainty.

The main difference between the two models is related to their approach to binning. The CH-PeEn model groups together all observations made at the exact same hour whereas the CSD-CLIM model proposes to bin the GHI data according to the clear-sky irradiance value.

We propose here to use contingency tables in order to better highlight the relative difference in the binning methodology used by each model. Figure 8 shows such contingency tables for 3 specific sites namely Nauru, Desert Rock and Toravere. Note that the numbers of the contingency table are translated to a color scale to ease readability. In practice, the difference between the two approaches varies also according to the seasons of the year. An example of this significant difference is illustrated for the site of Toravere in Figure 9. • is easy-to-implement,

• has an implementation that depends only on the historical data at hand,

• has a performance which is time invariant.

Besides, it was also demonstrated that, unlike the naive climatology, CH-PeEn and CSD-CLIM are able to reflect the difficulty of forecasting at a particular location.

More importantly, it was shown that CSD-CLIM achieves the best trade-off between the two most important attributes of a probabilistic forecast namely reliability and resolution.

In particular, CSD-CLIM can be qualified as more statistically consistent than CH-PeEn.

As such, in terms of overall performance, CSD-CLIM slightly outperforms CH-PeEn. This improved performance is due to a specific binning of the historical irradiance data based on the clear-sky irradiance values.

Finally, and as a conclusion, we can argue that the CSD-CLIM model can be a viable alternative to the CH-PeEn model. As shown by these tables, CSD-UNC is not strongly dependent on the chosen period or length of the dataset used to calculate it. In this work, the data granularity was 3h. However, it must be stressed that the stability in the CRPS results can be improved provided that the time resolution of the data increases. We recall that in a practical case, if the dependency on the input data is found strong, the recommendation should be to extend the length of the input data in order to get closer to the climatological mean.

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Clear-sky model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.3 Generating the predictive cumalative distrubution function (CDF) . . . . . . 7

(

  2019), we have selected the McClear clear-sky model. McClear Clear-sky GHI estimates are publicly available at 1-min resolution from the CAMS (Copernicus Atmosphere Monitoring Service) McClear Service (www.soda-pro.com).

Figure 1 :

 1 Figure 1: Uncertainty of the Brier Score in relation with the level of GHI for the site of Payerne. The uncertainty part of the CRPS (UNC ) is given by the shaded area.

  This is why we propose here a new benchmark model that exploits the periodic variations of the clear-sky model. In order to get rid of this strong temporal pattern, we propose to bin the climatology according the clear-sky irradiance values and to call this baseline model the clearsky-dependent climatology ("CSD-CLIM"). For each bin, CSD-CLIM computes the CDF in a similar manner as the climatology does, using only the historical data belonging to the bin. Considering the close relationship between this new model and the naive climatology, we propose to call the score of this benchmark model the "clearsky-dependent uncertainty" or " CSD-UNC ", just as the score of the naive climatology is called uncertainty (UNC ) in the literature (see for instance[START_REF] Todter | Generalization of the ignorance score: Continuous ranked version and its decomposition[END_REF]).

Figure 2 :

 2 Figure 2: Construction of CSD-UNC for Payerne. The sum of the 30 colored areas gives CSD-UNC

Figure 3 :

 3 Figure 3: Comparison between UNC and CSD-UNC.

Figure 4 :

 4 Figure 4: Histograms of k * t for Marshall Island and Desert Rock

  recall that a climatology benchmark model should possess the same statistical consistency as for the naive climatology and therefore should exhibit a reliability component as close as possible to zero. In addition, while respecting the statistical consistency property, any other benchmark model should beat the naive CLIM model in terms of resolution.

Figure 5

 5 Figure 5 clearly shows that, unlike the score of the naive CLIM, the scores of CSD-CLIM and CH-PeEn can be good proxies for judging a priori the quality of a forecast obtained at a particular site. In other words, just like the RMSE score obtained by the clear-sky persistence reference model in case of deterministic forecasts, the score of these two reference models reflects the difficulty of forecasting at a particular location.

1 .

 1 Figure 6.

Figure 6 :

 6 Figure 6: UNC i in relation with each bin i of clear-sky irradiance for Desert Rock and Fouillole

Figure 7 :

 7 Figure 7: UNC i=24 in Desert Rock and Fouillole for clear-sky irradiances between 900 and 940 W/m 2 (24 th bin)

Figure 8 :

 8 Figure 8: Contingency tables related to the two binning processes for three sites

  Figure 9: Differences in the binning process between CH-PeEn and CSD-CLIM models

  Figure 10: Clear-sky index distributions at Toravere, averaged on the 3-hour window[09h-12h] UTC.

Figure 10

 10 Figure 10 shows a significant difference between the two distributions. This difference cannot be taken into account by CH-PeEn since it groups the GHI data acccording the hour of the day. Thus, the weak point of CH-PeEn is to aggregate together cases that a clearsky model could easily discriminate. This also explains the difference in the CRPS results between CH-PeEn and CSD-CLIM. The mixture of situations that are statistically different made by CH-PeEn increases its CRPS score. Indeed, it is known that the local hour is not directly correlated with the seasonal and daily cycles of the sun irradiance. Consequently, the variability of the k * t bins used for the CH-PeEn includes the variability due to the solar declination, the time equation or the level of Aerosol Optical Depths (AODs). Conversely, the binning made by CSD-CLIM is finer as it is governed by the specification of the number of bins used by the binning process (see Appendix A). Put differently, CH-PeEn only takes

  Figure A.11: Impact of N b on CSD-UNC

Table 1 :

 1 Desired properties of a good climatology benchmark model. "P" refers to properties already mentioned in the literature. We propose in this study to add the properties denoted by "A".

Table 2

 2 Note that the quality checks of all BSRN data are available in BSRN website. For quality checks of data which is not part of BSRN, please refer to[START_REF] Gal | Added-value of ensemble prediction system on the quality of solar irradiance probabilistic forecasts[END_REF].

	. The

Table 2 :

 2 Characteristics of sites used for the study

Table 3 :

 3 Binning process for Desert Rock. The relative frequency f i is the number of clear-sky irradiance values in the bin out of the total number of irradiances values.

	2 )	0-40	40-80	80-120	120-160	160-200	200-240
	relative frequency f i	0.06	0.07	0.06	0.05	0.04	0.04
	bin i (W/m 2 )	240-280	280-320	320-360	360-400	400-440	440-480
	relative frequency f i	0.04	0.04	0.02	0.03	0.02	0.02
	bin i (W/m 2 )	480-520	520-560	560-600	600-640	640-680	680-720
	relative frequency f i	0.02	0.05	0.05	0.04	0.04	0.05
	bin i (W/m 2 )	720-760	760-800	800-840	840-880	880-920	920-960
	relative frequency f i	0.06	0.04	0.04	0.02	0.02	0.02
	bin i (W/m 2 )	960-1000 1000-1040 1040-1080 1080-1120 1120-1160 1160-1200
	relative frequency f i	0.03	0.02	0.01	0.00	0.00	0.00
	Let us stress here that there is no need to form a predictive CDF and compute its CRPS

score with Equation 3. Instead, Equation 10 together with Equation 6 fully determine the score CSD-UNC of the proposed new reference model CSD-CLIM. Hence, based on the

  1, the implementation of a good benchmark model should require a minimum number of parameters, for both simplicity of calculation and universality of results. The naive climatology is the only model which, by construction, indisputably fully respects this criteria. On the contrary, the other models presented here use some additional parameters. CH-PeEn needs the time of the day to bin the data and a clear-sky model to compute the clear-sky indices k * t . The time of the day is not strictly speaking an additional parameter, because it does not depend on a discutable model. The time resolution of the bin could influence the results (note that this is also true for the naive climatology), but should be governed by the available data, and thus should depend only on the data at hand, as stated in P2. The choice of the clear-sky model could be more problematic, because different models could provide different results. However, it seems reasonable to accept this compromise

Table 5 :

 5 CRPS of LQR forecast (grey) and CRPS of the 3 benchmark models for all sites

	Model	SPI HAW DRO FPE FOU
	LQR forecast 51.36 46.82 25.58 39.37 77.89
	CLIM	172.7 157.8 175.0 137.8 140.1
	CSD-CLIM	59.5	50.7	34.6	48.7	76.2
	CH-PeEn	61.2	56.2	35.6	51.2	77.4
	Model	PAY PAL TOR ADE TIR
	LQR forecast 40.61 36.14 30.95 41.40 42.49
	CLIM	143.4 126.4 116.5 162.4 161.0
	CSD-CLIM	58.3	53.5	49.8	48.4	50.8
	CH-PeEn	64.8	58.6	54.6	60.4	52.8
	Model	SXF NAU MAR BAR COC
	LQR forecast 39.65 54.42 61.12 29.62 48.40
	CLIM	137.4 175.7 167.3 88.8 166.8
	CSD-CLIM	62.5	56.0	72.1	36.3	61.5
	CH-PeEn	65.0	58.0	74.4	37.1	63.3
	Model	MAN TEN MIN BER LAN
	LQR forecast 68.25 40.87 33.09 44.16 35.16
	CLIM	162.1 199.6 170.0 152.4 154.4
	CSD-CLIM	79.0	35.2	43.5	57.7	64.8
	CH-PeEn	80.1	35.2	44.0	59.7	67.3

  gives an overview of the performance of each benchmark model in the light of properties P4, A1 and A2. Let us recall that all the climatology benchmark models discussed here meet the required rules P1, P2 and P3.

		P4	A1		A2
	Model	Statistical consistency	Overall quality of the	Proxy indicator for
			model		forecast difficulty
	CLIM
	CH-PeEn
	CSD-CLIM
		Best Model	2nd best Model	Worst Model

Table 6 :

 6 Overall qualitative assessment of the benchmark models.

  Table D.9: Sensitivity of CSD-UNC on data period Table D.10: Sensitivity of CSD-UNC on data length

			CSD-UNC (W/m 2 )
	Site		period 1 period 2 period 3
	Payerne		59.3		60.5	57.9
	Sioux Falls		60.1		61.0	61.2
	Tenerife		35.1		37.6	35.9
	Bermuda Islands	59.0		57.8	56.2
			CSD-UNC (W/m 2 )
	Site	1	2	3	4	5	6	7
	Payerne	55.6 57.6 59.3 59.3 59.3 60.0 59.8
	Sioux Falls	60.4 61.3 60.1 61.5 61.6 61.4 60.8
	Tenerife	38.0 35.6 35.1 36.4 36.2 36.5 36.7
	Bermuda Islands 56.9 58.0 59.0 59.5 59.6 58.6 58.6

Note that UNC is also the uncertainty component of the CRPS.

For a negatively oriented score like the CRPS for instance, the value of the skill score ranges from -∞ for the worst forecast to 1 for a perfect forecast.

In practice, cloud enhancement events (i.e. multi-reflections of the sun beams by the clouds) can produce over-irradiance with clear-sky indices superior to 1.

The word "uncertainty" is not used here to refer to the uncertainty term of the decomposition of the CRPS, but to the expected level of variability of the predictand, which a forecast model has to deal with.

Acknowledgements

The authors acknowledge the financial support given by Region Reunion and European Regional development Fund (FEDER) under the POE-FEDER 2014-2020 PhD scholarship DIRED/20171387.

As mentioned above, contrary to [START_REF] Doubleday | Benchmark probabilistic solar forecasts: Characteristics and recommendations[END_REF] who used visual diagnostic tools like reliability and sharpness diagrams in order to assess the two important attributes of a forecasting scheme i.e reliability and resolution, we prefer here to rely on the quantitative decomposition of the CRPS. As expected, the reliability and resolution components of the naive climatology (CLIM) are zero or near zero. It appears also that while retaining the statistical consistency feature, the CSD-CLIM model improves on the resolution of the basic CLIM model. Conversely, it can be noted that the CH-PeEn exhbits a slighty higher resolution than the CSD-CLIM model but at the expense of a degradation in reliability.

Compliance with property A1

Additional property A1 states that the benchmark models should obtain the best possible quality using only historical data. The overall quality of the reference forecasts is measured here by the CRPS. Table 5 gives the CRPS of the 3 benchmark models for all the sites listed in Table 2.

Appendix A. Sensitivity Analysis related to the number of bins used by the

CSD-CLIM binning process

In this section, an analysis of the sensitivity of CSD-UNC related to N b is conducted.

A number N b = 30 has been chosen in this study. However, this choice is arbitrary. It The Brier Score (BS) is a probabilistic score used for the evaluation of binary forecasts (i.e. forecast for an event that fully realizes or not). Its mathematical definition is

where o is the observation (0 if the event does not realize and 1 if it realizes), f is the probability forecast (that can take any value between 0 and 1) and N is the number of forecast occurences [START_REF] Brier | Verification of forecasts in terms of probability[END_REF].

The Brier Score can be decomposed into the three main attributes of a forecast namely reliability (REL BS ), resolution (RES BS ) and uncertainty (UNC BS ). This decomposition reads as

The reliability measures the difference between the probability forecasts and the observations and is given by

where ( fj , j = 1, ..., N k ) denotes the ensemble of all the different probabilities provided by the forecast, p j is the number of occurrences of fj over the test period, ōj is the mean observation when the forecast is equal to fj The resolution measures to what extent the forecast discriminates the observations from the climatological mean ō. It is given by

The uncertainty term depends on the variability of the observations and is defined by

In the case of a continuous variable like GHI, the Brier score can be used to evaluate the probability that GHI exceeds a threshold x.

Appendix C. CRPS as the integral of the Brier score

In the general case of a continuous variable, the CRPS is also the integral of the Brier Score over all thresholds x, as demonstrated by [START_REF] Todter | Generalization of the ignorance score: Continuous ranked version and its decomposition[END_REF].

and its decomposition comes as :

Since the score CSD-UNC of CSD-CLIM is a climatological indicator theoretically based on all historical data, its stability is necessarily achieved when the length of the input data is sufficiently large. Nonetheless, in this section, we investigate the dependency of CSD-UNC for different cases of input data.

Four sites of the study i.e. Payerne, Sioux Falls, Tenerife and Bermuda island which experience different sky conditions (see Table 2) have been selected. The CSD-UNC was calculated for 3 different periods of 3 years and for 7 historical datasets with different length (from 1 to 7 years). The periods and the lengths of the different datasets are listed respectively in Table D 2010[START_REF] Binter | Applied Probabilistic Forecasting[END_REF][START_REF] Lefèvre | Mcclear: a new model estimating downwelling solar radiation at ground level in clear-sky conditions[END_REF][START_REF] Alessandrini | An analog ensemble for short-term probabilistic solar power forecast[END_REF]2016-2018Tenerife 2012-2014[START_REF] Alessandrini | An analog ensemble for short-term probabilistic solar power forecast[END_REF][START_REF] Lauret | Probabilistic solar forecasting using quantile regression models[END_REF]2018[START_REF] Yang | Choice of clear-sky model in solar forecasting[END_REF]Bermuda Islands 2004-20062007-20092010[START_REF] Binter | Applied Probabilistic Forecasting[END_REF] 20112011[START_REF] Binter | Applied Probabilistic Forecasting[END_REF]2011[START_REF] Lefèvre | Mcclear: a new model estimating downwelling solar radiation at ground level in clear-sky conditions[END_REF]2011-20142011[START_REF] Alessandrini | An analog ensemble for short-term probabilistic solar power forecast[END_REF]2011-20162011[START_REF] Lauret | Probabilistic solar forecasting using quantile regression models[END_REF]Sioux Falls 20102010-20112010[START_REF] Binter | Applied Probabilistic Forecasting[END_REF]2010[START_REF] Lefèvre | Mcclear: a new model estimating downwelling solar radiation at ground level in clear-sky conditions[END_REF]2010-20142010[START_REF] Alessandrini | An analog ensemble for short-term probabilistic solar power forecast[END_REF]2010-2016Tenerife 2012[START_REF] Binter | Applied Probabilistic Forecasting[END_REF][START_REF] Lefèvre | Mcclear: a new model estimating downwelling solar radiation at ground level in clear-sky conditions[END_REF][START_REF] Binter | Applied Probabilistic Forecasting[END_REF]-2014[START_REF] Binter | Applied Probabilistic Forecasting[END_REF][START_REF] Alessandrini | An analog ensemble for short-term probabilistic solar power forecast[END_REF][START_REF] Binter | Applied Probabilistic Forecasting[END_REF]-2016[START_REF] Binter | Applied Probabilistic Forecasting[END_REF][START_REF] Lauret | Probabilistic solar forecasting using quantile regression models[END_REF][START_REF] Binter | Applied Probabilistic Forecasting[END_REF]-2018Bermuda Islands 20042004[START_REF] Gneiting | Calibrated probabilistic forecasting using ensemble model output statistics and minimum crps estimation[END_REF]2004-20062004-20072004[START_REF] Buizza | The value of probabilistic prediction[END_REF]2004-20092004-2010 The resulting CRPS are given respectively in Table D.9 and Table D.10

Declaration of interests

☒ The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

☐The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:

Declaration of Interest Statement

 A new reference model to benchmark probabilistic solar forecasts called "CSD-CLIM" is introduced  CSD-CLIM is easy-to-implement, there is no need to form probability distributions to compute its CRPS  In the light of a baseline for benchmark models, CSD-CLIM is compared with two reference models: the naive climatology and the CH-PeEn  CSD-CLIM meets all prerequisite properties of a good benchmark model  CSD-CLIM achieves the best trade-off between reliability and resolution, and can be a viable alternative to existing models

Highlights (for review)