Jrssb Submission

Revision

Tilmann Gneiting
email: tilmann@stat.washington.edu

Fadoua Balabdaoui
email: fadoua@math.uni-goettingen.de

Adrian E Raftery
email: raftery@stat.washington.edu

Probabilistic Forecasts, Calibration and Sharpness

Keywords: Cross-validation, Density forecast, Ensemble prediction system, Ex post evaluation, Forecast verification, Model diagnostics, Posterior predictive assessment, Predictive distribution, Prequential principle, Probability integral transform, Proper scoring rule

Probabilistic forecasts of a continuous variable take the form of predictive densities or predictive cumulative distribution functions. We propose a diagnostic approach to the evaluation of predictive performance that is based on the paradigm of maximizing the sharpness of the predictive distributions subject to calibration. Calibration refers to the statistical consistency between the distributional forecasts and the observations and is a joint property of the predictions and the events that materialize. Sharpness refers to the concentration of the predictive distributions and is a property of the forecasts only. A simple theoretical framework phrased in terms of a game between nature and forecaster allows us to distinguish probabilistic calibration, exceedance calibration and marginal calibration. We propose and study tools for checking calibration and sharpness, among them the probability integral transform (PIT) histogram, marginal calibration plots, the sharpness diagram and proper scoring rules. The diagnostic approach is illustrated by an assessment and ranking of probabilistic forecasts of wind speed at the Stateline wind energy center in the U.S. Pacific Northwest. In combination with cross-validation or in the time series context, our proposal provides very general, nonparametric alternatives to the use of information criteria for model diagnostics and model selection.

Introduction

A major human desire is to make forecasts for the future. Forecasts characterize and reduce but generally do not eliminate uncertainty. Consequently, forecasts should be probabilistic in nature, taking the form of probability distributions over future events [START_REF] Dawid | Statistical theory: The prequential approach (with discussion)[END_REF]. Indeed, over the past two decades the quest for good probabilistic forecasts has become a driving force in meteorology (Gneiting and Raftery 2005). Major economic forecasts such as the quarterly Bank of England inflation report are issued in terms of predictive distributions, and the rapidly growing area of financial risk management is dedicated to probabilistic forecasts of portfolio values [START_REF] Pan | An overview of value at risk[END_REF][START_REF] Granger | Preface: Some thoughts on the future of forecasting[END_REF]). In the statistical literature, advances in Markov chain Monte Carlo methodology (see, for example, [START_REF] Besag | Bayesian computing and stochastic systems (with discussion and rejoinder)[END_REF] have led to explosive growth in the use of predictive distributions, mostly in the form of Monte Carlo samples from the posterior predictive distribution of quantities of interest.

1 It is often critical to assess the predictive ability of forecasters, or to compare and rank competing forecasting methods. Atmospheric scientists talk of forecast verification when they refer to this process (Jolliffe and Stephenson 2003), and much of the underlying methodology has been developed by meteorologists. There is also a relevant strand of work in the econometrics literature [START_REF] Diebold | Comparing predictive accuracy[END_REF][START_REF] Christoffersen | Evaluating interval forecasts[END_REF][START_REF] Diebold | Evaluating density forecasts with applications to financial risk management[END_REF]. [START_REF] Murphy | A general framework for forecast verification[END_REF] proposed a general framework for the evaluation of point forecasts that uses a diagnostic approach based on graphical displays, summary measures and scoring rules. In this paper, we consider probabilistic forecasts (as opposed to point forecasts) of continuous and mixed discrete-continuous variables, such as temperature, wind speed, precipitation, gross domestic product, inflation rates and portfolio values. In this situation, probabilistic forecasts take the form of predictive densities or predictive cumulative distribution functions (CDFs), and the diagnostic approach faces a challenge, in that the forecasts take the form of probability distributions while the observations are real-valued.

We employ the following, simple theoretical framework to provide guidance in our methodological work. At times or instances t = 1, 2, . . ., nature chooses a distribution, G t , which we think of as the true data generating process, and the forecaster picks a probabilistic forecast in the form of a predictive cumulative distribution function, F t . The outcome, x t , is a random number with distribution G t . Throughout, we assume that nature is omniscient, in the sense that the forecaster's information basis is at most that of nature. Hence, if

F t = G t for all t (1)
we talk of the ideal forecaster. In practice, the true distribution, G t , remains hypothetical, and the predictive distribution, F t , is an expert opinion that may or may not derive from a statistical prediction algorithm. In accordance with [START_REF] Dawid | Statistical theory: The prequential approach (with discussion)[END_REF] prequential principle, the predictive distributions need to be assessed on the basis of the forecast-observation pairs (F t , x t ) only, irrespectively of their origins. [START_REF] Dawid | Statistical theory: The prequential approach (with discussion)[END_REF] and [START_REF] Diebold | Evaluating density forecasts with applications to financial risk management[END_REF] proposed the use of the probability integral transform (PIT) value,

p t = F t (x t ), (2)
for doing this. If the forecasts are ideal and F t is continuous, then p t has a uniform distribution. Hence, the uniformity of the probability integral transform is a necessary condition for the forecaster to be ideal, and checks for its uniformity have formed a cornerstone of forecast evaluation. In the classical time series framework, each F t corresponds to a one-step ahead forecast, and checks for the uniformity of the PIT values have been supplemented by tests for independence [START_REF] Frühwirth-Schnatter | Recursive residuals and model diagnostics for normal and nonnormal state space models[END_REF][START_REF] Diebold | Evaluating density forecasts with applications to financial risk management[END_REF]. [START_REF] Hamill | Interpretation of rank histograms for verifying ensemble forecasts[END_REF] gave a thought-provoking example of a forecaster for whom the histogram of the PIT values is essentially uniform, even though every single probabilistic forecast is biased. His example aimed to show that the uniformity of the PIT values is a necessary but not a sufficient condition for the forecaster to be ideal. To fix the idea, we consider a simulation study based on the scenario described in Table 1. At times or instances t = 1, 2, . . ., nature draws a standard normal random number µ t and selects the data generating distribution, G t = N (µ t , 1). In the context of weather forecasts, we might think of µ t as an accurate description of the latest observable state of the atmosphere, summarizing all information that a forecaster might possibly have access to. The ideal forecaster is an expert meteorologist who conditions on the current state, µ t , and issues an ideal probabilistic forecast, F t = G t . The climatological forecaster takes the unconditional distribution, F t = N (0, 2), as probabilistic forecast. The unfocused forecaster observes the current state, µ t , but adds a mixture component to the forecast, which can be interpreted as distributional Table 1: Scenario for the simulation study. At times t = 1, 2, . . ., nature chooses a distribution, G t , and forecaster chooses a probabilistic forecast, F t . The observations are independent random numbers x t with distribution G t . We write N (µ, σ 2 ) for the normal distribution with mean µ and variance σ 2 , respectively. The sequences (µ t ) t=1,2,... , (τ t ) t=1,2,... and (δ t , σ 2 t ) t=1,2,... are independent identically distributed, and independent of each other and the observations.

Nature

G t = N (µ t , 1) where µ t ∼ N (0, 1)

Ideal forecaster

F t = N (µ t , 1) Climatological forecaster F t = N (0, 2)
Unfocused forecaster

F t = 1 2 ( N (µ t , 1) + N (µ t + τ t , 1
)) where τ t = ±1 with probability 1 2 each

Hamill's forecaster , (-1 2 , 1) or (0, 169 100 ) with probability 1 3 each bias. A similar comment applies to Hamill's forecaster. Clearly, our forecasters are caricatures; yet, climatological reference forecasts and conditional biases are frequently observed in practice. The observation, x t , is a random draw from G t , and we repeat the prediction experiment 10000 times.

F t = N (µ t + δ t , σ 2 t ) where (δ t , σ 2 t ) = ( 1 2 , 1)
Figure 1 shows that the PIT histograms for the four forecasters are essentially uniform.

In view of the reliance on the probability integral transform in the literature, this is a disconcerting result. As [START_REF] Diebold | Evaluating density forecasts with applications to financial risk management[END_REF] pointed out, the ideal forecaster is preferred by all users, regardless of the respective loss function. Nevertheless, the probability integral transform is unable to distinguish between the ideal forecaster and her competitors. To address these limitations, we propose a diagnostic approach to the evaluation of predictive performance that is based on the paradigm of maximizing the sharpness of the predictive distributions subject to calibration. Calibration refers to the statistical consistency between the distributional forecasts and the observations, and is a joint property of the predictions and the observed values. Sharpness refers to the concentration of the predictive distributions and is a property of the forecasts only. The more concentrated the predictive distributions, the sharper the forecasts, and the sharper the better, subject to calibration.

The remainder of the paper is organized as follows. Section 2 develops our theoretical framework for the assessment of predictive performance. We introduce the notions of probabilistic, exceedance and marginal calibration, give examples and counterexamples, and discuss a conjectured sharpness principle. In Section 3, we propose diagnostic tools such as marginal calibration plots and sharpness diagrams that complement the PIT histogram. Proper scoring rules address calibration as well as sharpness and allow to rank competing forecast procedures. Section 4 turns to a case study on probabilistic forecasts at the Stateline wind energy center in the U.S. Pacific Northwest. The diagnostic approach yields a clear-cut ranking of statistical algorithms for forecasts of wind speed, and suggests improvements that can be addressed in future research. Similar approaches hold considerable promise as very general, nonparametric tools for statistical model selection and model diagnostics. The paper closes with a discussion in Section 5 that emphasizes the need for routine assessments of sharpness in the evaluation of predictive performance.

Modes of calibration

In our theoretical framework, we consider probabilistic forecasting as a game played between nature and forecaster. At times or instances t = 1, 2, . . ., nature picks a probability distribution, G t , and forecaster chooses a probabilistic forecast in the form of a predictive distribution, F t . The observation, x t , is a random draw from nature's proposal distribution, G t . Throughout, we assume that nature is omniscient, in the sense that the information basis of the forecaster is at most that of nature. For simplicity, we assume that F t and G t are continuous and strictly increasing on R.

Some critical comments are in order. Evidently, G t is not observed in practice and any operational evaluation must be performed on the basis of the forecasts, F t , and the outcomes, x t , only. The game theoretic framework of [START_REF] Shafer | Probability and Finance: It's Only a Game![END_REF] and [START_REF] Vovk | Good randomized sequential probability forecasting is always possible[END_REF] also views forecasting as a game, with three players: forecaster, sceptic, and reality or nature. Forecaster and sceptic have opposite goals, and one of them wins, while the other loses. There is no goal assigned to nature, who directly chooses and reveals the outcome, x t , without recourse to any underlying data generating distribution. The key question in this deep strand of literature, which culminates in Theorem 3 of [START_REF] Vovk | Good randomized sequential probability forecasting is always possible[END_REF], is that for the existence of certain types of strategies for forecaster. Shafer and Vovk consider probability forecasts for dichotomous events, rather than distributional forecasts of real-valued quantities, and they do not consider the problem tackled here, namely the comparative evaluation of competing forecasters, for which they hint at future work (Shafer and Vovk 2001, p. 50).

In comparing forecasters, we take the pragmatic standpoint of a user, who is to rank and choose among a number of competitors, as exemplified in Section 4. In this type of situation, it is absolute performance that matters, rather than relative performance that may result from the use of possibly distinct information bases.

Our approach seems slightly broader than [START_REF] Dawid | Statistical theory: The prequential approach (with discussion)[END_REF] prequential framework, in that we think of (F t ) t=1,2,... as a general countable sequence of forecasts, with the index referring to time, space or subjects, depending on the prediction problem at hand. The forecasts need not be sequential, and when F t+1 is issued, x t may or may not be available yet.

Probabilistic calibration, exceedance calibration and marginal calibration

Henceforth, (F t ) t=1,2,... and (G t ) t=1,2,... denote sequences of continuous and strictly increasing CDFs, possibly depending on stochastic parameters. We think of (G t ) t=1,2,... as the true data generating process and of (F t ) t=1,2,... as the associated sequence of probabilistic forecasts. The following definition refers to the asymptotic compatibility between the data generating process and the predictive distributions in terms of three major modes of calibration. Given that (F t ) t=1,2,... and (G t ) t=1,2,... might depend on stochastic parameters, convergence is understood as almost sure convergence and is denoted by an arrow. For now, these notions are of theoretical interest only; in Section 3 below, they lend support to our methodological proposals.

Definition 1 (modes of calibration) (a) The sequence (F t ) t=1,2,... is probabilistically calibrated relative to the sequence (G t ) t=1,2,... if

1 T T t=1 G t • F -1 t (p) -→ p for all p ∈ (0, 1). ( 3
) (b) The sequence (F t ) t=1,2,... is exceedance calibrated relative to (G t ) t=1,2,... if 1 T T t=1 G -1 t • F t (x) -→ x for all x ∈ R. ( 4
) (c) The sequence (F t ) t=1,2,... is marginally calibrated relative to (G t ) t=1,2,... if the limits Ḡ(x) = lim T →∞ 1 T T t=1 G t (x) and F (x) = lim T →∞ 1 T T t=1 F t (x)
exist and equal each other for all x ∈ R, and if the common limit distribution places all mass on finite values.

(d) The sequence (F t ) t=1,2,... is strongly calibrated relative to (G t ) t=1,2,... if it is probabilistically calibrated, exceedance calibrated and marginally calibrated.

If each subsequence of (F t ) t=1,2,... is probabilistically calibrated relative to the associated subsequence of (G t ) t=1,2,... , we talk of complete probabilistic calibration. Similarly, we define completeness for exceedance, marginal and strong calibration. Probabilistic calibration is essentially equivalent to the uniformity of the PIT values. Exceedance calibration is defined in terms of thresholds, and marginal calibration requires that the limit distributions Ḡ and F exist and equal each other. The existence of Ḡ is a natural assumption in meteorological problems and corresponds to the existence of a stable climate. Hence, marginal calibration can be interpreted in terms of the equality of observed and forecast climatology.

Various authors have studied calibration in the context of probability forecasts for sequences of binary events (De Groot and Fienberg 1982;[START_REF] Dawid | The well-calibrated Bayesian[END_REF]Dawid , 1985aDawid , 1985b;;[START_REF] Oakes | Self-calibrating priors do not exist[END_REF][START_REF] Schervish | Comment[END_REF][START_REF] Schervish | A general method for comparing probability assessors[END_REF][START_REF] Dawid | Prequential probability: Principles and properties[END_REF][START_REF] Shafer | Probability and Finance: It's Only a Game![END_REF][START_REF] Sandroni | Calibration with many checking rules[END_REF]. The progress is impressive and culminates in the elegant game theoretic approach of [START_REF] Vovk | Good randomized sequential probability forecasting is always possible[END_REF]. [START_REF] Krzysztofowicz | Bayesian theory of probabilistic forecasting via deterministic hydrologic model[END_REF] discussed calibration in the context of Bayesian forecasting systems, and [START_REF] Krzysztofowicz | Calibration of probabilistic quantitative precipitation forecasts[END_REF] studied calibration for quantile forecasts of quantitative precipitation. We are unaware of any prior discussion of notions of calibration for probabilistic forecasts of continuous variables.

Examples

The examples in this section illustrate the aforementioned modes of calibration and discuss some of the forecasters in our initial simulation study. Throughout, (µ t ) t=1,2,... , (σ t ) t=1,2,... and (τ t ) t=1,2,... denote independent sequences of independent identically distributed random variables. We write N (µ, σ 2 ) for the normal distribution with mean µ and variance σ 2 , identify distributions and CDFs, and let Φ denote the standard normal cumulative. In each example, nature draws a standard normal random number, µ t , that corresponds to the information basis at time t and picks the data generating distribution G t = N (µ t , 1).

Example 1 (ideal forecaster) The predictive distribution of the ideal forecaster equals nature's proposal distribution, that is, F t = G t = N (µ t , 1) for all t. This forecaster is strongly calibrated.

Example 2 (climatological forecaster) The climatological forecaster issues the distributional forecast F t = N (0, 2), irrespectively of t. This forecaster is probabilistically calibrated and marginally calibrated. However,

1 T T t=1 G -1 t • F t (x) = 1 T T t=1 Φ -1 Φ x √ 2 + µ t -→ x √ 2 for x ∈ R, in violation of exceedance calibration.
The characteristic property in Example 2 is that each predictive distribution, F t , equals nature's limiting distribution, Ḡ. We call any forecaster with this property a climatological forecaster. For climatological forecasts, probabilistic calibration is essentially equivalent to marginal calibration. Indeed, if Ḡ is continuous and strictly increasing, then putting p = F t (x) = Ḡ(x) in (3) recovers the marginal calibration condition. In practice, climatological forecasts are constructed from historical records of observations, and they are often used as reference forecasts.

Example 3 (unfocused forecaster) The predictive distribution of the unfocused forecaster is the mixture distribution

F t = 1 2 (N (µ t , 1) + N (µ t + τ t , 1)),
Table 2: The three major modes of calibration are logically independent of each other and may occur in any combination. For instance, the unfocused forecaster in Example 3 is probabilistically calibrated (P), but neither exceedance calibrated (E) nor marginally calibrated (P).

Properties

Example

P E M Example 1 (ideal forecaster) P E M G t = F t = N (t,
P E M G t = N (0, 1), F t = N (1, 1)
where τ t is either 1 or -1, with equal probabilities, and independently of µ t . This forecaster is probabilistically calibrated, but neither exceedance calibrated nor marginally calibrated. To prove the claim for probabilistic calibration, put Φ ±

(x) = 1 2 (Φ(x) + Φ(x ∓ 1)) and note that 1 T T t=1 G t • F -1 t (p) -→ 1 2 Φ • Φ -1 + (p) + Φ • Φ -1 -(p) = p,
where the equality follows upon putting p = Φ + (x), substituting and simplifying. Exceedance calibration does not hold, because

1 T T t=1 G -1 t • F t (x) -→ 1 2 Φ -1 • Φ + (x) + Φ -1 • Φ -(x) = x in general. The marginal calibration condition is violated, because nature's limit distribution, Ḡ = N (0, 2), does not equal F = 1 2 N (0, 2) + 1 4 N (-1, 2) + 1 4 N (1, 2).
Example 4 (mean-biased forecaster) This forecaster issues the probabilistic forecast F t = N (µ t + τ t , 1), where, again, τ t is either 1 or -1, with equal probabilities, and independently of µ t . The mean-biased forecaster is exceedance calibrated but neither probabilistically calibrated nor marginally calibrated.

Example 5 (sign-biased forecaster) The predictive distribution of the sign-biased forecaster is F t = N (-µ t , 1). This forecaster is exceedance calibrated and marginally calibrated, but not probabilistically calibrated.

Example 6 (mixed forecaster) The mixed forecaster randomizes between the climatological and the sign-biased forecast, with equal probabilities and independently of µ t . This forecaster is marginally calibrated, but neither probabilistically calibrated nor exceedance calibrated.

The examples in this section show that probabilistic calibration, exceedance calibration and marginal calibration are logically independent of each other and may occur in any combination. Table 2 summarizes the respective results.

Hamill's forecaster

We add a discussion of Hamill's forecaster. As previously, nature picks G t = N (µ t , 1), where µ t is standard normal. Hamill's forecaster is a master forecaster who assigns the prediction task with equal probability to any of three student forecasters, each of whom is biased, as described in Table 1. For Hamill's forecaster,

1 T T t=1 G t • F -1 t (p) -→ 1 3 Φ Φ -1 (p) - 1 2 + Φ 13 10 Φ -1 (p) + Φ Φ -1 (p) + 1 2 = p + (p),
where | (p)| ≤ 0.0032 for all p but (p) = 0 in general. The probabilistic calibration condition (3) is violated, but only slightly so, resulting in deceptively uniform PIT histograms. As for exceedance calibration, we note that

1 T T t=1 G -1 t • F t (p) -→ 1 3 x + 1 2 + 10 13 x + x - 1 2 = 12 13 x for x ∈ R.
Hence, Hamill's forecaster is not exceedance calibrated either, nor marginally calibrated, given that Ḡ = N (0, 2) while

F = 1 3 (N (-1 2 , 2) + N ( 1 2 , 2) + N (0, 269 100 )).

Sharpness principle

In view of our assumption that forecaster's information basis is at most that of nature, the best situation that we can possibly hope for is the equality (1) of F t and G t that characterizes the ideal forecaster. Operationally, we adopt the paradigm of maximizing the sharpness of the predictive distributions subject to calibration. Our conjectured sharpness principle contends that the two goals -ideal forecasts and the maximization of sharpness subject to calibration -are equivalent. This conjectured equivalence, which we deliberately state loosely, could be explained in two ways.

One explanation is that sufficiently stark notions of calibration, such as complete strong calibration across many dynamic subsequences, imply asymptotic equivalence to the ideal forecaster. Strong calibration alone, without the completeness condition, does not seem to impose enough restrictions, but we are unaware of a counterexample and would like to know of one. An alternative and weaker explanation states that any sufficiently calibrated forecaster is at least as spread out as the ideal forecaster.

With respect to this latter explanation, none of probabilistic, exceedance or marginal calibration alone is sufficiently stark. In the examples below it will be convenient to consider a probabilistic calibration condition, 1

T T t=1 G t • F -1 t (p) = p for all p ∈ (0, 1), (5)
for finite sequences (F t ) 1≤t≤T relative to (G t ) 1≤t≤T , and similarly for exceedance calibration and marginal calibration. The examples extend to countable sequences in obvious ways. Now suppose that σ > 0, a > 1, 0 < λ < 1/a and T = 2. Let G 1 and G 2 be continuous and strictly increasing distributions functions with associated densities that are symmetric about zero and have finite variance, var(G 1 ) = σ 2 and var(G 2 ) = λσ 2 . If we define

F 1 (x) = 1 2 G 1 (x) + G 2 x a , F 2 (x) = F 1 (ax), then var(F 1 ) + var(F 2 ) = 1 2 1 + 1 a 2 (1 + a 2 λ 2 ) σ 2 < (1 + λ 2 ) σ 2 = var(G 1 ) + var(G 2 ),
even though the finite probabilistic calibration condition (5) holds. A similar example can be given for exceedance calibration. Suppose that σ > 0, 0 < a < 1 and

0 < λ < a 3 + a 1 + 3a 1/2 .
Let G 1 and G 2 be as above and define

F 1 (x) = G 1 2x 1 + a , F 2 (x) = G 2 2ax 1 + a . Then var(F 1 ) + var(F 2 ) = 1 4 (1 + a) 2 1 + λ 2 a 2 σ 2 < (1 + λ 2 ) σ 2 = var(G 1 ) + var(G 2 ),
even though the finite exceedance calibration condition holds. Evidently, a forecaster can be marginally calibrated yet sharper than the ideal forecaster.

For climatological forecasts, however, finite probabilistic calibration and finite marginal calibration are equivalent, and a weak form of the sharpness principle holds.

Theorem 1 Suppose that G 1 , . . . , G T and F 1 = • • • = F T = F have
if E(G 1 ) = • • • = E(G T ).
The proof of Theorem 1 is given in the appendix. We are unaware of any other results in this direction; in particular, we do not know whether a non-climatological forecaster can be probabilistically calibrated and marginally calibrated yet sharper than the ideal forecaster.

Diagnostic tools

We now discuss diagnostic tools for the evaluation of predictive performance. In accordance with [START_REF] Dawid | Statistical theory: The prequential approach (with discussion)[END_REF] prequential principle, the assessment of probabilistic forecasts needs to be based on the predictive distributions and the observations only. Previously, we defined notions of calibration in terms of the asymptotic consistency between the probabilistic forecasts and the data generating distributions, which are unavailable in practice. Hence, we turn to sample versions, by substituting empirical distribution functions based on the outcomes, resulting in methodological tools that stand in their own right as well. In the following, this program is carried out for probabilistic and marginal calibration. Exceedance calibration does not allow for an obvious sample analogue, and it is not clear whether such exists. We discuss graphical displays of sharpness and propose the use of proper scoring rules, that assign numerical measures of predictive performance and find key applications in the ranking of competing forecast procedures.

Assessing probabilistic calibration

The probability integral transform (PIT) is the value that the predictive CDF attains at the observation. Specifically, if F t is the predictive distribution and x t materializes, the transform is defined as p t = F t (x t ). The literature usually refers to [START_REF] Rosenblatt | Remarks on a multivariate transformation[END_REF], although the probability integral transform can be traced back at least to [START_REF] Pearson | On a method of determining whether a sample of size n supposed to have been drawn from a parent population having a known probability integral has probably been drawn at random[END_REF]. The connection to probabilistic calibration is established by substituting the empirical distribution function 1{x t ≤ x} for the data generating distribution G t (x), x ∈ R in the probabilistic calibration condition (3), and noting that x t ≤ F -1 t (p) if and only if p t ≤ p. The following theorem characterizes the asymptotic uniformity of the empirical sequence of PIT values in terms of probabilistic calibration. We state this result under the assumption of a * -mixing sequence of observations [START_REF] Blum | On the strong law of large numbers for a class of stochastic processes[END_REF]. The proof is deferred to the appendix.

Theorem 2 Let (F t ) t=1,2,... and (G t ) t=1,2,... be sequences of continuous, strictly increasing distribution functions. Suppose that x t has distribution G t and that the x t form a * -mixing sequence of random variables. Then if and only if (F t ) t=1,2,... is probabilistically calibrated with respect to (G t ) t=1,2,... . We emphasize that condition (6) stands in its own right as a criterion for the validity of probabilistic forecasts, independently of our theoretical framework. Indeed, following the lead of [START_REF] Dawid | Statistical theory: The prequential approach (with discussion)[END_REF] and [START_REF] Diebold | Evaluating density forecasts with applications to financial risk management[END_REF], checks for the uniformity of the PIT values have formed a cornerstone of forecast evaluation.

Uniformity is usually assessed in an exploratory sense, and one way of doing this is by plotting the empirical CDF of the PIT values and comparing to the identity function. This approach is adequate for small sample sizes and notable departures from uniformity, and its proponents include [START_REF] Staël Von Holstein | Assessment and Evaluation of Subjective Probability Distributions[END_REF], p. 142), Seillier-Moiseiwitsch (1993), [START_REF] Hoeting | Accounting for model uncertainty in linear regression[END_REF], p. 33), Frühwirth-Schnatter (1996), [START_REF] Raftery | Bayesian model averaging for linear regression models[END_REF], [START_REF] Clements | Evaluating the forecast densities of linear and non-linear models: Applications to output growth and unemployment[END_REF], [START_REF] Moyeed | An empirical comparison of kriging methods for nonlinear spatial point prediction[END_REF], [START_REF] Wallis | Chi-squared tests of interval and density forecasts, and the Bank of England's fan charts[END_REF] and [START_REF] Boero | The performance of SETAR models: A regime conditional evaluation of point, interval and density forecasts[END_REF]. Histograms of the PIT values accentuate departures from uniformity when the sample size is large and the deviations from uniformity are small. This alternative type of display has been used by [START_REF] Diebold | Evaluating density forecasts with applications to financial risk management[END_REF], [START_REF] Weigend | Predicting daily probability distributions of S&P500 returns[END_REF], [START_REF] Bauwens | A comparison of financial duration models via density forecasts[END_REF] and Gneiting et al. (2005), among others, and 10 or 20 histogram bins generally seem adequate. Figure 1 employs 20 bins and shows the PIT histograms for the various forecasters in our initial simulation study. The histograms are essentially uniform. Table 3 shows the empirical coverage of the associated central 50% and 90% prediction intervals. This information is redundant, since the empirical coverage can be read off the PIT histogram, as the area under the 10 and 18 central bins, respectively.

Probabilistic weather forecasts are typically based on ensemble prediction systems, which generate a set of perturbations of the best estimate of the current state of the atmosphere, run each of them forward in time using a numerical weather prediction model, and use the resulting set of forecasts as a sample from the predictive distribution of future weather quantities [START_REF] Palmer | The economic value of ensemble forecasts as a tool for risk assessment: From days to decades[END_REF]Gneiting and Raftery 2005). The principal device for assessing the calibration of ensemble forecasts is the verification rank histogram or Talagrand diagram, proposed independently by [START_REF] Anderson | A method for producing and evaluating probabilistic forecasts from ensemble model integrations[END_REF], [START_REF] Hamill | Verification of Eta-RSM short-range ensemble forecasts[END_REF] and [START_REF] Talagrand | Evaluation of probabilistic prediction systems[END_REF], and extensively used since. To obtain a verification rank histogram, find the rank of the observation when pooled within the ordered Shephard (1994, p. 129) gave an instructive example of how this could be done.

Visual inspection of a PIT or rank histogram can provide hints to the reasons for forecast deficiency. Hump shaped histograms indicate overdispersed predictive distributions with prediction intervals that are too wide on average. U-shaped histograms often correspond to predictive distributions that are too narrow. Triangle-shaped histograms are seen when the predictive distributions are biased. Formal tests of uniformity can be employed and have been studied by [START_REF] Anderson | A method for producing and evaluating probabilistic forecasts from ensemble model integrations[END_REF], [START_REF] Talagrand | Evaluation of probabilistic prediction systems[END_REF], [START_REF] Noceti | An evaluation of tests of distributional forecasts[END_REF], [START_REF] Garratt | Forecast uncertainties in macroeconomic modelling: An application to the UK economy[END_REF], [START_REF] Wallis | Chi-squared tests of interval and density forecasts, and the Bank of England's fan charts[END_REF] and [START_REF] Candille | Evaluation of probabilistic prediction systems for a scalar variable[END_REF], among others. However, the use of formal tests is often hindered by complex dependence structures, particularly in cases in which the PIT values are spatially aggregated. [START_REF] Hamill | Interpretation of rank histograms for verifying ensemble forecasts[END_REF] gave a thoughtful discussion of the associated issues and potential fallacies.

In the time series context, the observations are sequential, and the predictive distributions correspond to sequential k-step ahead forecasts. The probability integral transforms for ideal kstep ahead forecasts are at most (k -1)-dependent, and this assumption can be checked empirically, by plotting the sample autocorrelation function for the PIT values and their moments [START_REF] Diebold | Evaluating density forecasts with applications to financial risk management[END_REF]. [START_REF] Smith | Diagnostic checks of non-standard time series models[END_REF], [START_REF] Frühwirth-Schnatter | Recursive residuals and model diagnostics for normal and nonnormal state space models[END_REF] and [START_REF] Berkowitz | Testing density forecasts, with applications to risk management[END_REF] proposed an assessment of independence based on the transformed PIT values, Φ -1 (p t ), which are Gaussian under the assumption of ideal forecasts. This further transformation has obvious advantages when formal tests of independence are employed, and seems to make little difference otherwise.

Assessing marginal calibration

Marginal calibration concerns the equality of forecast climate and actual climate. To assess marginal calibration, we propose a comparison of the average predictive CDF,

FT (x) = 1 T T t=1 F t (x), x ∈ R, (7)
to the empirical CDF of the observations, ĜT

(x) = 1 T T t=1 1{x t ≤ x}, x ∈ R. (8)
Indeed, if we substitute the indicator function 1{x t ≤ x} for the data generating distribution G t (x),

x ∈ R, in the definition of marginal calibration, we are led to the asymptotic equality of FT and ĜT . Theorem 3 provides a rigorous version of this correspondence. Under mild regularity conditions, marginal calibration is a necessary and sufficient condition for the asymptotic equality of ĜT and FT . The proof of this result is deferred to the appendix.

Theorem 3 Let (F t ) t=1,2,... and (G t ) t=1,2,... be sequences of continuous, strictly increasing distribution functions. Suppose that each x t has distribution G t and that the x t form a * -mixing sequence of random variables. Suppose furthermore that F (x) = lim T →∞ 1 T T t=1 F t (x) exists for all x ∈ R and that the limit function is strictly increasing on R. Then ĜT

(x) = 1 T T t=1 1{x t ≤ x} -→ F (x) almost surely for all x ∈ R (9)
if and only if (F t ) t=1,2,... is marginally calibrated with respect to (G t ) t=1,2,... . We note that (9) stands in its own right as a criterion for the validity of probabilistic forecasts. Still, Theorems 2 and 3 provide reassurance, in that ( 6) and (9) will be satisfied almost surely if the forecaster issues the same sequence of distributions that nature uses to generate the outcomes, assuming mixing conditions. These results are also of interest because they characterize situations under which (6) and ( 9) lead us to accept as valid forecasts that might in fact be far from ideal.

The most obvious graphical device in the assessment of marginal calibration is a plot of ĜT (x) and FT (x) versus x. However, it is often more instructive to plot the difference of the two CDFs, as on the left-hand side of Figure 2, which shows the difference for the various forecasters in our initial simulation study. We call this type of display a marginal calibration plot. Under the hypothesis of marginal calibration, we expect minor fluctuations about zero only, and this is indeed the case for the ideal forecaster and the climatological forecaster. The unfocused forecaster and Hamill's forecaster lack marginal calibration, resulting in major excursions from zero. The same information can be visualized in terms of quantiles, as on the right-hand side of Figure 2, which shows the difference

FT (x) -ĜT (x), x ∈ R, (10)
Q( FT , q) -Q( ĜT , q), q ∈ (0, 1), (11)
of the quantile functions for FT and ĜT , respectively. Under the hypothesis of marginal calibration, we again expect minor fluctuations about zero only, and this is the case for the ideal forecaster and the climatological forecaster. The unfocused forecaster and Hamill's forecaster show quantile difference functions that increase from negative to positive values, indicating forecast climates that are too spread out.

Assessing sharpness

Sharpness refers to the concentration of the predictive distributions and is a property of the forecasts only. The more concentrated the predictive distributions, the sharper the forecasts, and the sharper the better, subject to calibration. To assess sharpness, we use numerical and graphical summaries of the width of prediction intervals. For instance, Table 4 shows the average width of the central 50% and 90% prediction intervals for the forecasters in our simulation study. The ideal forecaster is the sharpest, followed by Hamill's, the unfocused and the climatological forecaster. In our simplistic simulation study, the width of the prediction intervals is fixed, expect for Hamill's forecaster, and the tabulation is perfectly adequate. In real-world applications, conditional heteroscedasticity often leads to considerable variability in the width of the prediction intervals. The average width then is insufficient to characterize sharpness, and we follow [START_REF] Bremnes | Probabilistic forecasts of precipitation in terms of quantiles using NWP model output[END_REF] in proposing boxplots as a more instructive graphical device. We refer to this type of display as a sharpness diagram, and an example thereof is shown in Figure 9 below.

Proper scoring rules

Scoring rules assign numerical scores to probabilistic forecasts and form attractive summary measures of predictive performance, in that they address calibration and sharpness simultaneously. We write s(F, x) for the score assigned when the forecaster issues the predictive distribution F and x materializes, and we take scores to be penalties that the forecaster wishes to minimize. A scoring 5: Average logarithmic score (LogS) and continuous ranked probability score (CRPS). [START_REF] Winkler | Rewarding expertize in probability assessment[END_REF] gave an interesting discussion of the ways in which proper scoring rules encourage honest and sharp forecasts. The logarithmic score is the negative of the logarithm of the predictive density evaluated at the observation [START_REF] Good | Rational decisions[END_REF][START_REF] Bernardo | Expected information as expected utility[END_REF]. This scoring rule is proper and has many desirable properties [START_REF] Roulston | Evaluating probabilistic forecasts using information theory[END_REF], but lacks robustness [START_REF] Selten | Axiomatic characterization of the quadratic scoring rule[END_REF][START_REF] Gneiting | Strictly proper scoring rules, prediction, and estimation[END_REF]). The continuous ranked probability score is defined directly in terms of the predictive cumulative distribution function, F, namely as

LogS CRPS

crps(F, x) = ∞ -∞ (F (y) -1(y ≥ x)) 2 dy, (12)
and provides a more robust alternative. [START_REF] Gneiting | Strictly proper scoring rules, prediction, and estimation[END_REF] gave an alternative representation and showed that crps(F,

x) = E F |X -x| - 1 2 E F X -X , (13)
where X and X are independent copies of a random variable with cumulative distribution function F and finite first moment. The representation (13) shows that the continuous ranked probability score generalizes the absolute error, to which it reduces if F is a point forecast. It is reported in the same unit as the observations. The continuous ranked probability score is proper, and we rank competing forecast procedures based on its average,

CRPS = 1 T T t=1 crps(F t , x t ) = ∞ -∞ BS(y) dy, ( 14
)
where BS(y) = 1 T T t=1 (F t (y) -1{x t ≤ y}) 2 denotes the Brier score [START_REF] Brier | Verification of forecasts expressed in terms of probability[END_REF] for probability forecasts of the binary event at the threshold value y ∈ R. Like all proper scoring rules for binary probability forecasts, the Brier score allows for the distinction of a calibration component and a refinement component [START_REF] Murphy | Scalar and vector partitions of the probability score. Part I: Two-state situation[END_REF][START_REF] Degroot | The comparison and evaluation of probability forecasters[END_REF][START_REF] Dawid | Probability forecasting[END_REF]). [START_REF] Candille | Evaluation of probabilistic prediction systems for a scalar variable[END_REF] discuss calibration-sharpness decompositions of the continuous ranked probability score.

Table 5 shows the logarithmic score and the continuous ranked probability score for the various forecasters in our initial simulation study, averaged over the 10000 replicates of the prediction experiment. As expected, both scoring rules rank the ideal forecaster highest, followed by Hamill's, the unfocused and the climatological forecaster. Figure 3 plots the Brier score for the associated binary forecasts in dependence on the threshold value, illustrating the integral representation on the right-hand side of ( 14). This type of display was proposed by Gerds (2002, Section 2.3) and [START_REF] Schumacher | How to assess prognostic models for survival data: A case study in oncology[END_REF] who called the graphs prediction error curves. 4 Case study: Probabilistic forecasts at the Stateline wind energy center

Wind power is the fastest-growing energy source today. Estimates are that within the next 15 years wind energy will fill about 6% of the electricity supply in the United States. In Denmark, wind energy already meets 20% of the country's total energy needs. However, arguments against the proliferation of wind energy have been put forth, often focusing on the perceived inability to forecast wind resources with any degree of accuracy. The development of advanced probabilistic forecast methodologies helps address these concerns.

The prevalent approach to short-range forecasts of wind speed and wind power at prediction horizons up to a few hours is based on on-site observations and autoregressive time series models [START_REF] Brown | Time series models to simulate and forecast wind speed and wind power[END_REF]. Gneiting et al. (2004) proposed a novel spatio-temporal approach, the regime-switching space-time (RST) method, that merges meteorological and statistical expertise to obtain fully probabilistic forecasts of wind resources. Henceforth, we illustrate our diagnostic approach to the evaluation of predictive performance by a comparison and ranking of three competing methodologies for two-hour ahead forecasts of hourly average wind speed at the Stateline wind energy center. The evaluation period is May through November 2003, resulting in a total of 5136 probabilistic forecasts.

Predictive distributions for hourly average wind speed

We consider three competing statistical prediction algorithms for two-hour ahead probabilistic forecasts of hourly average wind speed, w t , at the Stateline wind energy center. Stateline is located on the Vansycle ridge at the border between the states of Oregon and Washington in the U.S. Pacific Northwest. The data source is described in Gneiting et al. (2004). The first method is the persistence forecast, a naive yet surprisingly skillful, nonparametric reference forecast. The persistence point forecast is simply the most recent observed value of hourly average wind speed at Stateline. To obtain a predictive distribution, we dress the point forecast with the 19 most recent observed values of the persistence error, similarly to the approach proposed by [START_REF] Roulston | Combining dynamical and statistical ensembles[END_REF]. Specifically, the predictive CDF for w t+2 is the empirical distribution function of the set

{max(w t -w t-h + w t-h-2 , 0) : h = 0, . . . , 18}.
The second technique is the autoregressive time series approach, which was proposed by [START_REF] Brown | Time series models to simulate and forecast wind speed and wind power[END_REF] and has found widespread use since. To apply this technique, we fit and extract a diurnal trend component based on a sliding 40-day training period, fit a stationary autoregression to the residual component and find a Gaussian predictive distribution in the customary way. The predictive distribution assigns a typically small positive mass to the negative half-axis, and in view of the nonnegativity of the predictand, we redistribute this mass to wind speed zero. The details are described in Gneiting et al. (2004), where the method is referred to as the AR-D technique. The third method is the regime-switching space-time (RST) approach of Gneiting et al. (2004). The RST model is parsimonious, yet takes account of all the salient features of wind speed: alternating atmospheric regimes, temporal and spatial autocorrelation, diurnal and seasonal nonstationarity, conditional heteroscedasticity and non-Gaussianity. The method uses offsite information from the nearby meteorological towers at Goodnoe Hills and Kennewick, identifies atmospheric regimes and fits conditional predictive models for each regime, based on a sliding 45-day training period. Details can be found in Gneiting et al. (2004), where the method is referred to as the RST-D-CH technique. Any minor discrepancies in the results reported below and in Gneiting et al. (2004) stem from the use of R versus Splus and differences in optimization routines. persistence and RST forecasts appear uniform. The histogram for the autoregressive forecasts is hump shaped, thereby suggesting departures from probabilistic calibration. Table 6 shows the empirical coverage of central prediction intervals.

Assessing calibration

The PIT values for ideal 2-step ahead forecasts are at most 1-dependent, and the sample autocorrelation functions for the RST forecasts seem compatible with this assumption. The sample autocorrelations for the persistence forecasts were nonnegligible at lag 2, and the centered second moment showed notable negative correlations at lags between 15 and 20 hours. These features indicate a lack of fit of the predictive model, even though they seem hard to interpret diagnostically. The respective sample autocorrelations for the autoregressive forecast were positive and nonnegligible at lags up to five hours, suggesting conditional heteroscedasticity in the wind speed series. Indeed, Gneiting et al. (2004) showed that the autoregressive forecasts improve when a conditionally heteroscedastic model is employed. In the current classical autoregressive formulation the predictive variance varies as a result of the sliding training period, but high-frequency changes in predictability are not taken into account.

Figure 7 shows marginal calibration plots for the three forecasts, both in terms of CDFs and in terms of quantiles. The graphs show the differentials ( 10) and ( 11) and point at nonnegligible excursions from zero, particularly at small wind speeds and for the autoregressive forecast. The lack of predictive model finds an explanation in Figure 8, which shows the empirical cumulative distribution function, FT , of hourly average wind speed. Hourly average wind speeds less than 1

Assessing sharpness

The sharpness diagram in Figure 9 shows boxplots that illustrate the width of central prediction intervals for the 5136 predictive distributions in the evaluation period. The prediction intervals for the persistence forecast varied the most in width, followed by the RST and autoregressive forecasts. Table 7 shows the respective average widths. The RST method was by far the sharpest, with prediction intervals that were about 20% shorter on average than those for the autoregressive technique.

Continuous ranked probability score

Table 8 shows the CRPS value ( 14) for the various techniques. We report the scores month by month, which allows for an assessment of seasonal effects and straightforward tests of the null hypothesis of no difference in predictive performance. For instance, the RST method showed lower CRPS than the autoregressive technique in each month during the evaluation period, May through November 2003. Under the null hypothesis of equal predictive performance this happens with probability ( 1 2 ) 7 = 1 128 only. Similarly, the autoregressive technique outperformed the persistence method in May through October, but not in November. Various other tests can be employed, but care needs to be taken to avoid dependencies in the forecast differentials. Here, the results for distinct months can be considered independent for all practical purposes. [START_REF] Diebold | Comparing predictive accuracy[END_REF] gave a thoughtful discussion of these issues, and we refer to their work for a comprehensive account of tests of predictive performance. Figure 10 illustrates the Brier score decomposition (14) of the CRPS value for the entire evaluation period. The RST method outperformed the other techniques at all thresholds. We noted in Section 3.4 that the continuous ranked probability score generalizes the absolute error, and reduces to the latter for point forecasts. Table 9 shows the respective mean absolute error (MAE) for the persistence, autoregressive and RST point forecasts. The persistence point forecast is the most recent observed value of hourly average wind speed at Vansycle. The autoregressive point forecast is the mean of the respective predictive distribution, and similarly for the RST forecast. The RST method had the lowest MAE, followed by the autoregressive and persistence techniques. The MAE and CRPS values are reported in the same unit as the wind speed observations, that is, in m • s -1 , and can be directly compared. The insights that the monthly scores provide are indicative of the potential benefits of thoughtful stratification.

The CRPS and MAE values establish a clear-cut ranking of the forecast methodologies that places the RST method first, followed by the autoregressive and persistence techniques. The RST method also performed best in terms of probabilistic and marginal calibration, and the RST forecasts were by far the sharpest. The diagnostic approach points at forecast deficiencies and suggests potential improvements to the predictive models. In particular, the marginal calibration plots in Figure 7 suggest a modified version of the RST technique that uses truncated normal rather than

Discussion

Our paper addressed the important issue of evaluating predictive performance for probabilistic forecasts of continuous variables. Following the lead of [START_REF] Dawid | Statistical theory: The prequential approach (with discussion)[END_REF] and [START_REF] Diebold | Evaluating density forecasts with applications to financial risk management[END_REF], predictive distributions have traditionally been evaluated by checking the uniformity of the probability integral transform (PIT). We introduced the pragmatic and flexible paradigm of maximizing the sharpness of the predictive distributions subject to calibration. Calibration refers to the statistical consistency between the predictive distributions and the associated observations and is a joint property of the predictions and the values that materialize. Sharpness refers to the concentration of the predictive distributions and is a property of the forecasts only. We interpreted probabilistic forecasting within a simple theoretical framework that allowed us to distinguish probabilistic, exceedance and marginal calibration, and that lends support to the use of diagnostic tools in evaluating and comparing probabilistic forecasters. Probabilistic calibration corresponds to the uniformity of the PIT values, and the PIT histogram remains a key tool in the diagnostic approach to forecast evaluation. In addition, we proposed the use of marginal calibration plots, sharpness diagrams and proper scoring rules, which form powerful tools for learning about forecast deficiencies and ranking competing forecast methodologies. Our own applied work on probabilistic forecasting has benefitted immensely from these tools, as documented in Section 4 and in the partial applications in Gneiting et al. (2004), [START_REF] Raftery | Using Bayesian model averaging to calibrate forecast ensembles[END_REF] and Gneiting et al. (2005). Predictive distributions can be reduced to point forecasts, or to probability forecasts of binary events, and the associated forecasts can be assessed using the diagnostic devices described by [START_REF] Murphy | Diagnostic verification of temperature forecasts[END_REF] and [START_REF] Murphy | Diagnostic verification of probability forecasts[END_REF], among others. If we were to reduce our conclusions to a single recommendation, we would close with a call for the assessment of sharpness, particularly when the goal is that of ranking. Previous comparative studies of the predictive performance of probabilistic forecasts have focused on calibration. For instance, [START_REF] Moyeed | An empirical comparison of kriging methods for nonlinear spatial point prediction[END_REF] compared spatial prediction techniques, [START_REF] Clements | Evaluating the forecast densities of linear and non-linear models: Applications to output growth and unemployment[END_REF] and [START_REF] Boero | The performance of SETAR models: A regime conditional evaluation of point, interval and density forecasts[END_REF] evaluated linear and non-linear time series models, Garrat et al. (2003) assessed macroeconomic forecast models, and [START_REF] Bauwens | A comparison of financial duration models via density forecasts[END_REF] studied the predictive performance of financial duration models. In each of these works, the assessment was best on the predictive performance of the associated point forecasts, and on the uniformity of the PIT values. We contend that comparative studies of these types call for routine assessments of sharpness, in the form of sharpness diagrams and through the use of proper scoring rules.

Despite the frequentist flavor of our diagnostic approach, calibration and sharpness are properties that are relevant to Bayesian forecasters as well. [START_REF] Rubin | Bayesianly justifiable and relevant frequency calculations for the applied statistician[END_REF]Rubin ( , pp. 1161Rubin ( and 1160) ) argued that "the probabilities attached to Bayesian statements do have frequency interpretations that tie the statements to verifiable real world events." Consequently, a "Bayesian is calibrated if his probability statements have their asserted coverage in repeated experience. [START_REF] Gelman | Posterior predictive assessment of model fitness via realized discrepancies[END_REF] developed Rubin's posterior predictive approach, proposed posterior predictive checks as Bayesian counterparts to the classical tests for goodness of fit, and advocated their use in judging the fit of Bayesian models. This relates to our diagnostic approach, which emphasizes the need for understanding the ways in which predictive distributions fail or succeed. Indeed, the diagnostic devices posited herein form powerful tools for Bayesian as well as frequentist model diagnostics and model choice. Tools such as the PIT histogram, marginal calibration plots, sharpness diagrams and proper scoring rules are widely applicable, since they are nonparametric, do not depend on nested models, allow for structural change, and apply to predictive distributions that are represented by samples, as they arise in a rapidly growing number of Markov chain Monte Carlo methodologies and ensemble prediction systems. In the time series context, the predictive framework is natural and model fit can be assessed through the performance of the time-forward predictive distributions [START_REF] Smith | Diagnostic checks of non-standard time series models[END_REF][START_REF] Shephard | Partial non-Gaussian state space[END_REF][START_REF] Frühwirth-Schnatter | Recursive residuals and model diagnostics for normal and nonnormal state space models[END_REF]. In other types of situations, cross-validatory approaches often apply fruitfully (Dawid 1984, p. 288;[START_REF] Gneiting | Strictly proper scoring rules, prediction, and estimation[END_REF].

Appendix

Proof of Theorem 1

Consider the random variable U = F (x 1 ) z 1 F (x 2 ) z 2 • • • F (x T ) z T where x 1 ∼ G 1 , . . . , x T ∼ G T and (z 1 , . . . , z T ) is multinomial with equal probabilities. The finite probabilistic calibration condition implies that U is uniformly distributed. By the variance decomposition formula, var(F ) = var(F -1 (U )) = E var F -1 (U ) | z 1 , . . . , z T + var E F -1 (U ) | z 1 , . . . , z T .

The first term in the decomposition equals 1 T

Proof of Theorem 2

For p ∈ (0, 1) and t = 1, 2, . . ., put Y t = 1{p t < p} -G t • F -1 t (p) and note that E(Y t ) = 0. By Theorem 2 of [START_REF] Blum | On the strong law of large numbers for a class of stochastic processes[END_REF], lim

T →∞ 1 T T t=1 Y t = lim T →∞ 1 T T t=1 1{p t < p} -G t • F -1 t (p) = 0
almost surely. The uniqueness of the limit implies that ( 6) is equivalent to the probabilistic calibration condition (3).

Proof of Theorem 3

For x ∈ R let q = F (x), and for t = 1, 2, . . . put q t = F (x t ). Then By Theorem 2 with F t = F for t = 1, 2, . . ., we have that 1 T T t=1 1{q t ≤ q} → q almost surely if and and only if 1 T T t=1 G t • F -1 (q) → q almost surely. Hence, marginal calibration is equivalent to (9).

Figure 1 :

 1 Figure 1: Probability integral transform (PIT) histograms.

  second moments and satisfy the finite probabilistic calibration condition (5). Then1 T T t=1 var(F t ) = var(F ) ≥ 1 T T t=1 var(G t )with equality if and only

  p} -→ p almost surely for all p (6)

Figure 2 :

 2 Figure 2: Marginal calibration plot for the ideal forecaster (solid line), climatological forecaster (short dashes), unfocused forecaster (dot-dashed line) and Hamill's forecaster (long dashes). The presentation is in terms of CDFs (left) and in terms of quantiles (right), respectively.

Figure 3 :

 3 Figure 3: Brier score plot for the ideal forecaster (solid line), climatological forecaster (short dashes), unfocused forecaster (dot-dashed line), and Hamill's forecaster (long dashes). The graphs show the Brier score as a function of the threshold value. The area under each forecaster's curve equals the CRPS value (14).

Figure 4 :

 4 Figure 4: Probability integral transform (PIT) histogram and sample autocorrelation functions for the first three centered moments of the PIT values, for persistence forecasts of hourly average wind speed at the Stateline wind energy center.

Figure 5 :

 5 Figure 5: Same as Figure 4, but for autoregressive forecasts.

Figures 4 ,Figure 6 :

 46 Figures 4, 5 and 6 show the probability integral transform (PIT) histograms for the three forecast techniques, along with the sample autocorrelation functions for the first three centered moments of the PIT values and the respective Bartlett confidence intervals. The PIT histograms for the

Figure 7 :Figure 8 :

 78 Figure 7: Marginal calibration plot for persistence forecasts (dashed line), autoregressive forecasts (dot-dashed line) and RST forecasts (solid line) of hourly average wind speed at the Stateline wind energy center in terms of CDFs (left) and in terms of quantiles (right), in m • s -1 .

Figure 9 :

 9 Figure 9: Sharpness diagram for persistence (PS), autoregressive (AR) and RST forecasts of hourly average wind speed at the Stateline wind energy center. The boxplots show the 5th, 25th, 50th, 75th and 95th percentile of the width of the central prediction interval, in m • s -1 . The smaller the width, the sharper. The nominal coverage is 50% (left) and 90% (right), respectively.

Figure 10 :

 10 Figure 10: Brier score plot for persistence forecasts (dashed line), autoregressive forecasts (dotdashed line) and RST forecasts (solid line) of hourly average wind speed at the Stateline wind energy center, in m • s -1 . The graphs show the Brier score as a function of the threshold value. The area under each forecast's curve equals the CRPS value (14).

  term is nonnegative and vanishes if and only if E(G1 ) = • • • = E(G T ).

ĜT

Table 3 :

 3 Empirical coverage of central prediction intervals. The nominal coverage is 50% and 90%, respectively. plot the histogram of the ranks. If we identify the predictive distribution with the empirical CDF of the ensemble values, this technique is seen to be equivalent to plotting a PIT histogram. A similar procedure could be drawn on fruitfully to assess samples from posterior predictive distributions obtained by Markov chain Monte Carlo techniques.

	Interval	50%	90%
	Perfect forecaster	51.2% 90.0%
	Climatological forecaster 51.3% 90.7%
	Unfocused forecaster	50.1% 90.1%
	Hamill's forecaster	50.9% 89.5%
	ensemble values and

Table 4 :

 4 Average width of central intervals. The nominal coverage is 50% and 90%, respectively.

	Interval	50% 90%
	Ideal forecaster	1.35 3.29
	Climatological forecaster 1.91 4.65
	Unfocused forecaster	1.52 3.68
	Hamill's forecaster	1.49 3.62

  if the expected value of the penalty s(F, x) for an observation x drawn from G is minimized if F = G. It is strictly proper if the minimum is unique.

	Ideal forecaster	1.41	0.56
	Climatological forecaster 1.75	0.78
	Unfocused forecaster	1.53	0.63
	Hamill's forecaster	1.52	0.61
	rule is proper

Table 6 :

 6 Empirical coverage of central prediction intervals. The nominal coverage is 50% and 90%, respectively.

	Interval	50%	90%
	Persistence forecast	50.9%
	Autoregressive forecast 55.6% 90.4%
	RST forecast	51.2% 88.4%

Table 7 :

 7 Average width of central prediction intervals, in m • s -1 . The nominal coverage is 50% and 90%, respectively. -1 were almost never observed, even though the predictive distributions assign positive point mass to wind speed zero.

	Interval	50% 90%
	Persistence forecast	2.63 7.51
	Autoregressive forecast 2.74 6.55
	RST forecast	2.20 5.31
	m • s

Table 8 :

 8 CRPS value (14) for probabilistic forecasts of hourly average wind speed at the Stateline wind energy center in March through November 2003, month by month and for the entire evaluation period, in m • s -1 .

	CRPS	May Jun Jul Aug Sep Oct Nov Mar-Nov
	Persistence forecast	1.16 1.08 1.29 1.21 1.20 1.29 1.16	1.20
	Autoregressive forecast 1.12 1.02 1.10 1.11 1.11 1.22 1.13	1.12
	RST forecast	0.96 0.85 0.95 0.95 0.97 1.08 1.00	0.97

Table 9 :

 9 Mean absolute error (MAE) for point forecasts of hourly average wind speed at the Stateline wind energy center in March through November 2003, month by month and for the entire evaluation period, in m • s -1 .

	MAE	May Jun Jul Aug Sep Oct Nov Mar-Nov
	Persistence forecast	1.60 1.45 1.74 1.68 1.59 1.68 1.51	1.61
	Autoregressive forecast 1.53 1.38 1.50 1.54 1.53 1.68 1.54	1.53
	RST forecast	1.32 1.18 1.33 1.31 1.36 1.48 1.37	1.34

cut-off normal predictive distributions. This modification yields small but consistent improvements in predictive performance

[START_REF] Gneiting | Calibrated probabilistic forecasting at the Stateline wind energy center: The regime-switching space-time (RST) method[END_REF]

).

Acknowledgements

We thank John B. Bremnes, Barbara G. Brown, Richard E. Chandler, Theo Eicher, Hans R. Künsch, Kristin Larson, Jon A. Wellner, Anton H. Westveld and Kenneth Westrick for discussions, comments and references, and we are grateful to an anonymous reviewer for a wealth of helpful suggestions. Our work has been supported by the DoD Multidisciplinary University Research Initiative (MURI) program administered by the Office of Naval Research under Grant N00014-01-10745. Tilmann Gneiting furthermore acknowledges support by the National Science Foundation under Award no. 0134264 and by the Washington Technology Center, and thanks the Soil Physics group at Universität Bayreuth, Germany, where part of this work was performed.