Gilbert Colletaz
email: gilbert.colletaz@univ-orleans.fr

Christophe Hurlin
email: christophe.hurlin@univ-orleans.fr

Christophe Pérignon
email: perignon@hec.fr.

John Cotter

Jorge Cruz Lopez

Laurent Frésard

Andras Fulop

Thomas Gilbert

Bertrand Maillet

Olivier Scaillet

Stefan Straetmans

The Risk Map: A New Tool for Validating Risk Models

Keywords: JEL classication: G21, G28, G32 Financial Risk Management, Tail Risk, Basel III

This paper presents a new method to validate risk models: the Risk Map. This method jointly accounts for the number and the magnitude of extreme losses and graphically summarizes all information about the performance of a risk model. It relies on the concept of a super exception, which is dened as a situation in which the loss exceeds both the standard Value-at-Risk (VaR) and a VaR dened at an extremely low coverage probability. We then formally test whether the sequences of exceptions and super exceptions are rejected by standard model validation tests. We show that the Risk Map can be used to validate market, credit, operational, or systemic risk estimates (VaR, stressed VaR, expected shortfall, and CoVaR) or to assess the performance of the margin system of a clearing house.

Introduction

The need for sound risk management has never been more essential than in todays nancial environment. Of paramount importance for risk managers and regulators is the ability to detect misspecied risk models as they lead to misrepresentation of actual risk exposures [START_REF] Blöchlinger | Validation of Default Probabilities[END_REF]. In this paper, we focus on a popular family of risk models, namely the tail risk models. These models can generate a variety of risk measures including the Value-at-Risk (VaR), which is dened as an extreme quantile of a return or prot-and-loss (P&L) distribution, as well as stressed VaR, expected shortfall, and CoVaR. In practice, tail risk models are used to measure downside risk [START_REF] Bali | Is There an Intertemporal Relation between Downside Risk and Expected Returns?[END_REF], construct portfolios (Basak and Shapiro, 2001), quantify nancial institutions exposures to, and capital requirements for, market, credit, and operational risks [START_REF] Jorion | Value at Risk: The New Benchmark for Managing Financial Risk[END_REF], set margin requirements for derivatives users [START_REF] Booth | Prudent Margin Levels in the Finnish Stock Index Futures Market[END_REF], and measure the systemic-risk contribution of a nancial institution [START_REF] Adrian | CoVaR[END_REF].

In this paper, we present a new tool, called the Risk Map, for validating risk models. To grasp the intuition of our approach, consider two banks that both have a one-day Valueat-Risk (VaR) of $100 million at the 1% probability level. This means that each bank has a one percent chance of losing more than $100 million over the next day. Assume that, over the past year, each bank has reported three VaR exceptions, or days when the trading loss exceeds its VaR, but the average VaR exceedance is $1 million for bank A and $999 million for bank B. In this case, standard backtesting methodologies would indicate that the performance of both models is equal (since both models lead to the same number of exceptions) and acceptable (since the annual number of exceptions is close enough to its target value of 2.5 = 1% of 250 trading days). The reason is that current backtesting methodologies only focus on the number of VaR exceptions and totally disregard the magnitude of these exceptions [START_REF] Berkowitz | Testing Density Forecasts, with Applications to Risk Management[END_REF][START_REF] Stulz | Risk Management Failures: What Are They and When Do they Happen?[END_REF]. However, in practice, market participants and regulators do care about the magnitude of their losses. 1 It is indeed the severity of the trading losses, and not the exceptions per se, that jeopardize the solvency of nancial institutions. For instance, banking regulators may want to penalize more heavily in terms of capital requirements a bank that experiences extremely large exceptions than a bank that experiences moderate exceptions. Furthermore, it makes a big di¤erence whether the margin of a given derivative market participant is exceeded by a small or by a large amount as it creates a shortfall of the same amount that the clearing house must cover. To the best of our knowledge, there is no general hypothesis-testing framework available in the literature that accounts for both the number and the magnitude of extreme losses. The objective of this paper is to ll this gap.

The Risk Map approach jointly accounts for the number and the magnitude of the VaR exceptions. The basic intuition is that a large loss not only exceeds the regular VaR dened with a probability B (e.g. 1%) but is also likely to exceed a VaR dened with a much lower probability B 0 (e.g. 0.2%). On this ground, we dene a VaR exception as r t < 0V aR t (B), where r t denotes the P&L, and a VaR super exception as r t < 0V aR t (B 0 ), with B 0 much smaller than B. As an illustration, we show in Figure 1 the joint evolution of the daily P&L, V aR (B), and V aR (B 0 ) for a hypothetical portfolio. We see that, as expected, 0V aR (B 0 ) is systematically more negative than 0V aR (B) as V aR (B 0 ) is measured further left in the tail of the P&L distribution. For this portfolio, there are four exceptions and three super exceptions.

< Insert Figure 1 > In practice, the choice of the probability B 0 is key. We show in this paper that this parameter can be either chosen freely by the user or dened endogenously. In the former case, the B 0 parameter must reect the loss aversion of the investor whereas in the latter case, B 0 is set such that V aR (B 0 ) corresponds either to the stressed VaR or to the expected shortfall.

In order to validate the risk model, we formally test whether the sequences of exceptions and super exceptions satisfy standard backtesting conditions. Formally, we need to test the joint null hypothesis that the probability of having an exception is B and that the probability of having a super exception is B 0 . Several statistical strategies can be used to test this null hypothesis (e.g. log-likelihood ratio, hit regression test) and some statistical tests are readily available in the literature. We then report the p-value of the statistical test in a three-dimensional graph, which we call the Risk Map, that graphically summarizes all information about the performance of a risk model. To sum up, what the Risk Map approach allows us to do is to simplify an intricate problem dened over a {loss frequency, loss severity} domain to an easier problem dened over a {loss frequency, loss frequency} domain.

There are several advantages to the Risk Map approach. First, it preserves the simplicity of the standard validation techniques [START_REF] Kupiec | Techniques for Verifying the Accuracy of Risk Measurement Models[END_REF], while still accounting for the magnitude of the losses. Thus, the Risk Map approach is a three-dimensional generalization of the "Tra¢c Light" system (Basel Committee on Banking Supervision, 2006, 2011a) which remains the reference backtest methodology for banking regulators. 2 Second, it is a formal hypothesis testing framework that provides p-values and rejection ranges. Indeed, it allows us to jointly test the null hypothesis that both the numbers of VaR exceptions and super exceptions are accurate. Third, the Risk Map approach is general and can be used with any tail risk model as it only relies on the sequence of exceptions and super exceptions. In particular, no assumptions need to be made regarding the distribution of the P&L. For instance, it can be used to backtest the market VaR of a single asset, portfolio, trading desk, business line, bank, insurance company, mutual fund, or hedge fund [START_REF] Berkowitz | Evaluating Value-at-Risk Models with Desk-Level Data[END_REF]. It also permits to jointly validate the standard and stressed VaRs that banks must compute under Basel III, as well as expected shortfalls. Furthermore, the Risk Map can be used to backtest credit-risk VaRs [START_REF] Lopez | Evaluating Credit Risk Models[END_REF], operational-risk VaRs (Dahen and Dionne, 2010), or VaR-based margins for derivative users (Cruz [START_REF] Lopez | Clearing House, Margin Requirements, and Systemic Risk[END_REF]. Finally, we show that the Risk Map can be used to validate the systemic risk measure recently proposed by Adrian and Brunnermeier (CoVaR, 2011) as it is dened as the conditional quantile of a bank asset return. The Risk Map is, to the best of our knowledge, the rst method allowing one to backtest a systemic risk measure.

The outline of the paper is as follows. In the next section, we describe our model validation framework. In Section 3, we present several applications of the Risk Map methodology that sequentially deal with market risk, systemic risk, and margin requirements. We summarize and conclude our paper in Section 4. 2 Accounting for VaR exceedances is particularly important in the context of the Basel III regulation (Basel Committee on Banking Supervision, 2011a). Indeed, under Basel III, capital requirements for market risk depend on both the VaR and stressed VaR (calibrated to historical data from a continuous 12-month period of signicant nancial stress) of the bank:

c = max 8 V aR tjt0 ; m 1 V aR avg 9 + max 8 sV aR tjt0 ; m s 1 sV aR avg 9 :
As the value of the multiplicative factors m and m s only depends on the backtesting results of the VaR, but not of the stressed VaR, it is particularly important to account for the magnitude of the VaR exceedances.

2 Validation Framework

Background

Let r t denote the return or P&L of a portfolio at time t and V aR tjt01 (B) the ex-ante oneday ahead VaR forecast for an B coverage rate conditionally on an information set F t01 .

In practice, VaR is computed using either non-parametric techniques, such as Historical Simulation [START_REF] Pritsker | The Hidden Dangers of Historical Simulation[END_REF], or parametric techniques, such as Monte Carlo simulation [START_REF] Broadie | E¢cient Risk Estimation via Nested Sequential Simulation[END_REF]. If the VaR model is adequate, then the following relation must hold:

Pr[r t < 0V aR tjt01 (B)] = B. (1)
Let I t (B) be the hit variable associated with the ex-post observation of a VaR(B) violation at time t:

I t (B) = ( 1 if r t < 0V aR tjt01 (B) 0 otherwise : (2)
Backtesting procedures are typically only based on the violation process fI t (B)g T t=1 . As stressed by [START_REF] Christo¤ersen | Evaluating Interval Forecasts[END_REF], VaR forecasts are valid if and only if this violation sequence satises the Unconditional Coverage (UC) hypothesis. 3 Under the UC hypothesis, the probability of an ex-post return exceeding the VaR forecast must be equal to the B coverage rate:

Pr [I t (B) = 1] = -[I t (B)] = B. (3)
A key limitation of this approach is that it is unable to distinguish between a situation in which losses are below but close to the VaR (e.g. bank A in the introduction) and a situation in which losses are considerably below the VaR (e.g. bank B). 4 A solution proposed by Lopez (1999a,b) consists in considering the excess losses over and above the 3 Validation tests are also based on the independence hypothesis (IND), under which VaR violations observed at two di¤erent dates for the same coverage rate must be distributed independently. Formally, the variable I t (B) associated with a VaR violation at time t for a coverage rate B should be independent of the variable I t0k (B), 8k 6 = 0. In other words, past VaR violations should not be informative about current and future violations. When the UC and IND hypotheses are simultaneously valid, VaR forecasts are said to have a correct Conditional Coverage (CC), and the VaR violation process is a martingale di¤erence, with -[I t (B) 0 B F t0 ] = 0. For a test of the CC hypothesis, see Christo¤ersen (1998), [START_REF] Christo¤ersen | Backtesting Value-at-Risk: A Duration-Based Approach[END_REF], [START_REF] Engle | CAViaR: Conditional Autoregressive Value at Risk by Regression Quantiles[END_REF], and Berkowitz, Christo¤ersen and Pelletier (2011). 4 Other standard backtesting methods also disregard the magnitude of the losses beyond the VaR (Christo¤ersen and Pelletier, 2004; [START_REF] Engle | CAViaR: Conditional Autoregressive Value at Risk by Regression Quantiles[END_REF][START_REF] Gaglianone | Evaluating Value-at-Risk Models via Quantile Regression[END_REF][START_REF] Berkowitz | Evaluating Value-at-Risk Models with Desk-Level Data[END_REF].

VaR and dening a loss function:

L t (B) = ( (r t + V aR tjt01 (B)) 2 if r t < 0V aR tjt01 (B) 0 otherwise : (4)
Lopez proposes various heuristic criteria in order to assess the magnitude of these excess losses. However, such criteria convey only limited information since no normative rule can be deduced for the magnitude of these excess losses.

Another possibility is to complement the VaR data with some extra risk estimates. A rst approach is to consider the average loss beyond the VaR using the concept of expected shortfall:

ES t (B) = 1 B Z B 0 F 01 (p) dp (5)
where F (:) denotes the cumulative distribution function of the P&L [START_REF] Artzner | Coherent Measures of Risk[END_REF]. Alternatively, [START_REF] Berkowitz | Testing Density Forecasts, with Applications to Risk Management[END_REF] suggests testing the entire P&L density see [START_REF] Christo¤ersen | Backtesting[END_REF] for a discussion of the backtesting of ES or density forecasts.

Di¤erently, we propose a model validation methodology that is based on the number and the severity of VaR exceptions. 5 Our approach exploits the concept of a super exception, which we dene as a loss greater than V aR(B 0 ), with B 0 much smaller than B (e.g. B = 1% and B 0 = 0:2%). One can similarly dene a hit variable associated with V aR t (B 0 ):

I t (B 0 ) = ( 1 if r t < 0V aR tjt01 (B 0 ) 0 otherwise with B 0 < B: (6)
The dening feature of our approach is to account for both the frequency and the magnitude of trading losses. The intuition of our test is the following. If the frequency of super exceptions is abnormally high, this means that the magnitude of the losses with respect to V aR(B) is too large.

For both VaR exceptions and super exceptions, we propose to use a standard backtesting procedure. Consider a time series of T VaR forecasts for an B (respectively B 0 ) coverage rate and let N (respectively N 0 ) be the number of associated VaR violations:

N = T X t=1 I t (B) N 0 = T X t=1 I t (B 0 ) : (7)
If we assume that the I t (:) variables are i:i:d:; then under the U C hypothesis, the total number of VaR exceptions follows a Binomial distribution:

N B (T; B) (8)
with E (N ) = BT and V (N ) = B (1 0 B) T: Thus, it is possible to test the U C hypothesis for the VaR expectations as:

H 0 : -[I t (B)] = B (9)
H 1 : -[I t (B)] 6 = B: (10)
Under H 0 ; the corresponding log-likelihood ratio statistics is dened as:

LR U C (B) = 02 ln h (1 0 B) T 0N B N i + 2 ln " 1 0 N T T 0N N T N # d 0! T !1 2 (1) (11)
where the LR U C statistic is asymptotically chi-square with one degree of freedom [START_REF] Jorion | Value at Risk: The New Benchmark for Managing Financial Risk[END_REF]. A similar validation test can be dened for super exceptions:

H 0 : -[I t (B 0 )] = B 0 (12)
H 1 : -[I t (B 0 )] 6 = B 0 : (13)
An LR test statistic LR U C (N 0 ) can be dened as in equation [START_REF] Berkowitz | How Accurate Are Value-At-Risk Models at Commercial Banks?[END_REF], except that we now use the coverage rate B 0 and the corresponding number of hits N 0 .

The Risk Map

The goal of the Risk Map is to present the backtesting results for a given risk model in a graphical way. A rst approach is to jointly display the non-rejection zones for the

LR U C (B)
H 0 : -[I t (B)] = B and -[I t (B 0 )] = B 0 : (14)
This joint null hypothesis can be tested using either a multivariate version of the unconditional coverage test (LR U C ) or of the hit regression test of [START_REF] Engle | CAViaR: Conditional Autoregressive Value at Risk by Regression Quantiles[END_REF].

We follow [START_REF] Pérignon | A New Approach to Comparing VaR Estimation Methods[END_REF] and dene several indicator variables for revenues falling in each disjoint interval:

J 1;t = I t (B) 0 I t (B 0 ) = ( 1 if 0 V aR tjt01 (B 0 ) < r t < 0V aR tjt01 (B) 0 otherwise ( 15
)
J 2;t = I t (B 0 ) = ( 1 if r t < 0V aR tjt01 (B 0 ) 0 otherwise ( 16
)
and J 0;t = 1 0 J 1;t 0 J 2;t = 1 0 I t (B). The fJ i;t g 2 i=0 are Bernoulli random variables equal to one with probability 1 0 B, B 0 B 0 , and B 0 , respectively. However, they are clearly not independent since only one J variable may be equal to one at any point in time, P 2 i=0 J i;t = 1. We can test the joint hypothesis ( 14) of the specication of the VaR model using a simple Likelihood Ratio test. Let us denote N i;t = P T t=1 J i;t , for i = 0; 1; 2, the count variable associated with each of the Bernoulli variables. This multivariate unconditional coverage test is a likelihood ratio test LR M U C that the empirical exception frequencies signicantly deviate from the theoretical ones. Formally, it is given by:

LR M U C (B; B 0 ) = 02 ln h (1 0 B) N (B 0 B 0 ) N (B 0 ) N i +2 ln " N 0 T N N 1 T N N 2 T N # d 0! T !1 2 (2) : (17)
Under the null ( 14) of joint conditional coverage for the VaR exceptions and super exceptions, the LR M U C statistic is asymptotically chi-square with two degrees of freedom. 6 The p-values of the test are displayed for di¤erent combinations of number of exceptions N and super exceptions N 0 in Figure 2. Alternatively, a Risk Map can be constructed based on the rejection zones for di¤erent condence levels (Figure 3). Note that the cells below the diagonal are not colored as they correspond to situations in which the number of super exceptions exceeds the number of exceptions, which is of course impossible. If the (N; N 0 ) pair corresponds to a green cell, we conclude that we cannot reject the null hypothesis

-[I t (B)] = B and -[I t (B 0 )] = B 0
at the 95% condence level. If (N; N 0 ) falls in the orange zone, we can reject the null at the 95% but not at the 99% condence level. Finally, a red cell implies that we can reject the null hypothesis at the 99% condence level. Note also that, for technical reasons, the p-values for the rst-column cells (N 0 = 0) and diagonal cells (N 0 = N ) are computed from a univariate LR U C test because the multivariate test cannot be computed in these cases. Similarly, when N 0 = N = 0, none of the LR test can be computed.

< Insert Figures 2 and 3 >

We illustrate the di¤erence between the Risk Map and standard validation tests using a simple example. Consider a sample of 500 daily VaR forecasts generated from a hypothetical VaR model. Out of these 500 days, there are 10 exceptions. According to the LR U C test, such model would be rejected as the 95% non-rejection range for this test is 2 N 9. Di¤erently, with the Risk Map, a model with 10 exceptions would not be rejected at the 95% condence level as long as there are either 1, 2, or 3 three super exceptions. However, the model would be rejected if the number of super exceptions is N 0 = 0 or N 0 > 3.

What the Risk Map does is to test whether both N and N 0 are at an acceptable level. The intuition is as follows: if there are too few super exceptions, it means that even if the model is valid with respect to the B coverage rate, it turns out not to be valid with respect to lower coverage rates. In that sense, the model is considered to be globally invalid. For instance, if among the 10 exceptions, there is no super exception, the VaR model overestimates the risk for the B 0 coverage rate, which casts doubt on the overall performance of the risk model. On the contrary, if there are four or more super exceptions, the model is also rejected as it underestimates V aR(B 0 ). In the latter case, even if the VaR model is validated on the basis of the V aR(B) exceptions, the magnitude of these exceptions is too large.

Extended Risk Map and Finite Sample Properties

The Risk Map can be enriched with information about empirical frequencies of VaR (super) exceptions under the assumption that the VaR model is valid. This additional layer of information complements the p-values and is independent of the statistical test used to build the Risk Map. To do so, we draw T observations from a uniform distribution over [0; 1]. The simulated violation process I t (B) is then dened by a binary variable that takes a value of one if the simulated value is less than B and zero otherwise. The simulated super violation process I t (B 0 ) is dened in the same way with a cut-o¤ of B 0 . Theoretically, these violation processes have the same properties as the processes obtained by comparing valid ex-ante VaR forecasts with ex-post losses. We repeat the experiment 10,000 times and for each simulation we count the number of violations (N ) and super violations (N 0 ).

The upper Risk Map in Figure 4 displays the empirical frequencies of each pair (N; N 0 ) for T = 500, B = 1%, and B 0 = 0:2%. This extended Risk Map provides useful information about the validity of the risk model. Consider for instance a model that yields 13 exceptions and 3 super exceptions. Figure 4 indicates that this case has only 3 chances out of 10,000 to arise with a valid risk model. Di¤erently, an outcome like (N = 5; N 0 = 1) has more than 700 chances out 10,000 to occur with a valid model, which suggests that the analyzed model is more likely to be valid.

< Insert Figure 4 >

This extended Risk Map can also be seen as a graphical representation of the nite sample properties of the statistical test used to build the Risk Map (i.e., the LR M U C test in our case). The statistical test determines the shape of the non-rejection area, which is the green zone if we consider a 5% level of nominal risk. Logically, if the test is well sized, most (N; N 0 ) observations fall inside the non-rejection area. More precisely, the sum of empirical frequencies of pairs of violations/super violations that are outside the non-rejection area corresponds to the empirical size of the test (dened as the risk of rejecting a valid model) for a nite sample size of 500. If the test is well sized, 95% of (N; N 0 ) observations remain inside the non-rejection area.

To further illustrate the size performance of the Risk Map, we report in Panel A of Table 1 the empirical sizes considering successively the cases when [B = 5%; B 0 = 1%], [B = 2%; B 0 = 0:4%], and [B = 1%; B 0 = 0:2%], as well as various sample sizes. The reported empirical sizes correspond to the rejection rates calculated over 10; 000 simulations for a nominal size of 5%. We can observe that the size of the Risk Map is quite close to the nominal size, even for small samples.

< Insert Table 1 >

In the same manner, the Risk Map can be used to illustrate the power of the test (dened as the probability to reject an invalid model). Here, the larger the number of exceptions and super exceptions that fall outside the non-rejection zone, the more powerful the test. In this case, two elements are required: a data generating process for the P&L and a risk model to produce the VaR forecasts. Let us consider the following experiment. We assume that the P&L are generated from a Markov Switching GARCH model [START_REF] Bauwens | Theory and Inference for a Markov Switching GARCH Model[END_REF]). This particular model allows both the conditional mean and the volatility to change from one state to the next:

r t = sJ + t u t ( 18
)
2 t = ! sJ + B sJ 0 r t01 0 s J0 1 2 + C sJ 2 t01 ( 19
)
where u t is i:i:d:N: (0; 1) and s t denotes a two-state hidden Markov Chain with a transition matrix dened by the probabilities

8 ij = Pr ( s t = ij s t01 = j) 9 :
The parameters values are those estimated by Bauwens, Preminger and Rombouts (2010) for the S&P 500 index between 2001 and 2007. 7 To illustrate the power of the test, we assume that the VaR model is misspecied and that VaR forecasts are computed according to the Historical Simulation method. This method, widely used by professionals, simply consists in calculating the empirical quantile of the past P&L over a rolling window (set here to 500 observations). The VaR forecasts are clearly invalid in this case, especially in periods of crisis, in the sense that they do not react fast enough to changes in the P&L dynamics [START_REF] Pritsker | The Hidden Dangers of Historical Simulation[END_REF]. We simulate 10,000 P&L sequences of 1,000 observations each, and for each of them we compute 500 Historical Simulation out-of-sample VaR(1%) and VaR(0.2%) forecasts. Corresponding violations and super violations are then obtained by comparing these forecasts to the expost losses. Similar to what we did in the case of the valid model, we count for each simulation the number of exceptions (N ) and super exceptions (N 0 ).

The lower Risk Map in Figure 4 displays the empirical frequencies of each couple (N; N 0 ) for this invalid model. The Risk Map reveals that most of the (N; N 0 ) couples are outside the non-rejection area (95% condence level). This conguration clearly indicates that the VaR model employed is misspecied. In this case, the sum of empirical frequencies of pairs of violations/super violations that are outside the non-rejection area corresponds to the empirical power of the LR test.

Similar to what we did for size, we report in Panel B of Table 1 the empirical power of the Risk Map and of the LR U C test for various sample sizes and coverage rates (B; B 0 ) : The empirical power corresponds to the rejection rates (for a nominal size equal to 5%) based on 10,000 replications of the experiment previously described (MS GARCH DGP and VaR-HS). For comparison purposes, we report in Panel C the power of the LR U C test for the 1%, 2%, and 5% coverage rates. Two conclusions can be drawn from this experiment. First, the power of the Risk Map generally increases with the sample size and the coverage rate (the latter result being true only for moderately sized samples, as in Berkowitz, Christo¤ersen and Pelletier, 2011). Second, the power of the Risk Map is at least as good as the one of the LR U C test.

Computation of VaR(B 0 )

In most applications, Risk Map users, namely risk managers and regulators, will have both data on V aR(B) and V aR(B 0 ). Nevertheless, we show in this subsection that a Risk Map can still be generated when only V aR(B) is available. This could for instance be the case when a bank only discloses to the public its VaR at the 1% level. To overcome this problem, we propose a calibration procedure allowing us to extract V aR(B 0 ) from V aR(B), with B 0 < B. The main elements of our procedure are (1) the data generating process of the P&L, (2) the internal VaR model used by the bank, and (3) the auxiliary model that we use to generate the V aR(B 0 ) estimates.

We assume that the P&L distribution is a member of the location scale family and, for simplicity, that it is centered, i.e., -(r t ) = 0.8 Under these assumptions, the conditional VaR can be expressed as an a¢ne function of the conditional variance of the P&L, denoted

h t : V aR tjt01 (B; C) = 0 p h t F 01 (B; C) (20)
where F 01 (B; C) denotes the B-quantile of the conditional standardized P&L distribution. We assume that this distribution is parametric and depends on a set of parameters C.

A rst way to compute V aR(B 0 ) is to use the Quasi-Maximum Likelihood (QML) esti-mates of the conditional variance and of the conditional quantile:

V aR tjt01 (B 0 ; b C) = 0 q b h t F 01 B 0 ; b C (21)
where b h and b C are the QML estimates. Alternatively, we propose an approach that does not rely on the QML estimate b h t . Given the VaR forecast reported by the bank, V aR(B), we dene an implied P&L conditional variance as:

q e h t = 0 V aR tjt01 (B) F 01 B; b C : (22)
We here proceed by analogy with the option pricing literature, in which implied volatility is extracted from option prices [START_REF] Taylor | Generating Volatility Forecasts from Value at Risk Estimates[END_REF]. V aR(B 0 ) is then dened as:

V aR tjt01 (B 0 ; b C) = 0 q e h t F 01 B 0 ; b C = V aR tjt01 (B)
F 01 B 0 ; b C F 01 B; b C : (23)
Interestingly, we notice that the auxiliary model is only used to get an estimate of the conditional quantile F 01 B; b C . As a result, this process mitigates as much as possible the impact of a misspecication of the auxiliary model on the VaR estimates. Implementing this calibration method requires two ingredients: (1) an auxiliary model for the conditional volatility h t , such as a GARCH or stochastic volatility model, and (2) a conditional distribution for the P&L, which depends on a set of parameters C.

In the Appendix, we use a Monte Carlo study to assess the empirical performance of our calibration procedure. We use a GARCH model as the auxiliary model and a tdistribution for F . In that case, the set of parameters C simply corresponds to the degree of freedom of the t-distribution, which can be estimated by QML. Overall, we nd that this calibration procedure leads to reliable estimates for V aR(B 0 ). Furthermore, we implement this calibration procedure on real data in Section 3.1.1.

The Risk Map at Work

In this section, we present a variety of applications for the Risk Map. The di¤erent areas of application include market risk modeling, systemic risk measurement, and margins for derivatives users. From one application to the next, it is only the denition of the r variable that is changing. It denotes successively the P&L of a bank trading portfolio, the asset return of a nancial institution and of the nancial system, and the change in a derivatives (portfolio) price, respectively.

Market Risk

Bank VaR

A natural application of the Risk Map is to backtest the VaR of a nancial rm. It is indeed key for both risk managers and banking regulators to check the validity of banks VaR engines as capital requirements depend on banks VaR internal estimates [START_REF] Jorion | Value at Risk: The New Benchmark for Managing Financial Risk[END_REF], and footnote 2). [START_REF] Berkowitz | How Accurate Are Value-At-Risk Models at Commercial Banks?[END_REF] show that VaR estimates of leading US banks tended to be conservative during the late nineties. Using P&L data from four business lines in a large international commercial bank, Berkowitz, Christo¤ersen, and Pelletier (2011) nd evidence of volatility dynamics and clustering in VaR exceptions. Frésard, Pérignon and Wilhelmsson (2011) report that during the global nancial crisis, the number of VaR exceptions increased sharply, with some nancial institutions experiencing more than 50 VaR(1%) exceptions per year. However, prior empirical literature did not consider the magnitude of the VaR exceedances.

We use actual VaR and P&L for a large European bank, namely La Caixa, which is the third largest Spanish bank. 9 We use daily one-day ahead VaR(1%) and daily P&L for that bank over the period 2007-2008. We plot in the upper part of Figure 5, the bank VaR along with the actual P&L. As our sample period includes the beginning of the recent nancial crisis, there is a clear regime shift in the variability of the trading revenues. We see that volatility spiked after the end of the rst semester of 2007. Similarly, the VaR(1%) jumped from around 2 million euros during 2007-Q1-Q2 to around 4 million afterwards. We provide some summary statistics on the P&L and VaR time series in Table 2.

< Insert Table 2 >

VaR forecasts dened at a lower coverage rate are required to implement the Risk Map methodology but are not disclosed by the bank. As a result, we use the calibration procedure presented in Section 2.4 to extract the VaR(0.2%) series from the VaR(1%) series. We use a t-GARCH as the auxiliary model for the P&L and report the estimated parameters in Panel B of Table 2. Out of the four parameters, only the degree of freedom b v is needed to estimate the quantile of the conditional distribution. As shown in equation [START_REF] Dahen | Scaling Models for the Severity and Frequency of External Operational Loss Data[END_REF], the implied conditional variance e h t is given by the ratio of VaR(1%) and the quantile of the conditional distribution. Given the implied variance e h t , we then compute VaR(0.2%) following equation [START_REF] Du¢e | Does a Central Clearing Counterparty Reduce Counterparty Risk?[END_REF]. We display in Panel A of Table 2 some summary statistics on VaR(0.2%). We notice that the ratio of means of VaR(0.2%) and VaR(1%) is 1.73, which corresponds to the ratio of quantiles dened in equation [START_REF] Du¢e | Does a Central Clearing Counterparty Reduce Counterparty Risk?[END_REF]. Interestingly, we see that VaR(0.2%) has the same skewness and kurtosis as VaR(1%) which suggests that our calibration procedure preserves the main statistical properties of the original VaR(B).

< Insert Figure 5 > Over this sample period, there were 13 VaR exceptions (N = 13) and three super exceptions (N 0 = 3) for La Caixa. The number of violations dene the coordinates of the point associated with the risk model used by the bank in our Risk Map representation. The lower part of Figure 5 displays the point associated with La Caixa on the Risk Map (see the cross at (3;13)). We conclude that we can reject the validity of the VaR model of the bank since the observation falls outside the non-rejection zone colored in green. The corresponding p-value of the LR M U C test is 0.0108.

One may wonder whether our conclusion is a¤ected by the value of the probability B 0 . If we assume that there is a continuum of VaR(B 0 ) associated with a continuum of B 0 , the number of super exceptions would remain the same for a range of value for B 0 . For La Caixa, this range is pretty large as it goes from B=10 to B=4:3. Furthermore, we plot in Figure 6 both the number of super exceptions and the p-value of the LR M U C test for di¤erent B 0 coe¢cients. The main take away from this gure is that the p-value remains remarkably stable, which conrms the robustness of our results. Note also that the question about the value of B 0 is not relevant anymore under Basel III. Indeed, as shown in the next two subsections, B 0 becomes in this case endogenous.

< Insert Figure 6 >

In the context of banking regulation, the Risk Map can be used to generalize the "Tra¢c Light" system of the Basel Committee. Under current regulation, part of the capital requirement for banks comes from the market risk exposure of the banks. More specically, the Market Risk Charge depends on the V aR(1%) and a multiplicative factor which is set to 3 plus a penalty term . The value of the applicable penalty only depends on the number of exceptions: = 0 for N < 5, = 0:4 for N = 5, = 0:5 for N = 6, ..., and = 1 for N > 10. We suggest generalizing this approach by accounting jointly for the number and the magnitude of the exceptions within the framework of the Risk Map. We present in Table 3 the value of the penalty term for di¤erent number of exceptions and super exceptions and di¤erent sample sizes (T = 250 and T = 500). The no-penalty zone corresponds to the green zone of the Risk Map and the maximum penalty zone corresponds to the red zone (see Figure 3). Between these two zones, the bank incurs an increasing penalty, which is obtained through linear interpolation. As an illustration, given the backtesting results in Figure 5, the penalty coe¢cient that would apply to La Caixa is 0.7.

< Insert Table 3 >

Bank Stressed VaR

Under Basel III, all nancial institutions with material trading activities must compute both their VaR using recent data and their stressed VaR (sVaR) using data from a particularly volatile period (Basel Committee on Banking Supervision, 2011a). As shown in equation [START_REF] Cotter | Margin Exceedences for European Stock Index Futures using Extreme Value Theory[END_REF], the VaR is an a¢ne function of the conditional variance of the P&L, denoted h t . Di¤erently, the sVaR depends on a conditional variance of the P&L measured over a particularly volatile period, denoted H t :

sV aR tjt01 (B; B) = 0 p H t F 01 (B; B) (24)
where B denotes the parameters of the P&L distribution F in the high-volatility period.

The Risk Map can be used to validate globally the VaR engine of the bank using both VaR and sVaR gures. Note that this is, to the best of our knowledge, the rst attempt to validate a sVaR, which is the preferred market-risk measure under Basel III. To do so, we dene the coverage rate B 0 for which V aR(B 0 ; C) is as close as possible to sV aR(B; B):

b B 0 = arg min fB g ( 1 T T X t=1 2 V aR tjt01 (B 0 ; C) 0 sV aR tjt01 (B; B) 3 2 ) T t=1 : (25)
In this case, we dene a super exception as a loss exceeding the stressed VaR and consider the associated hit variable:

I t (B 0 ) = ( 1 if r t < 0sV aR tjt01 (B; B) 0 otherwise : (26)
The relevant joint null hypothesis remains equation [START_REF] Brownlees | Volatility, Correlation and Tails for Systemic Risk Measurement[END_REF]. A particularly appealing feature of applying the Risk Map to stressed VaRs is that the probability B 0 becomes endogenous.

Expected Shortfall

Similarly to what we did with stressed VaR, the risk model can be validated using both the VaR and the expected shortfall ES. The latter risk measure is dened as the average loss beyond the V aR (see equation 5) and is produced by the same risk model as the VaR. First, we identify the V aR(B 0 ) that corresponds to the ES(B).

As V aR (B 0 ) = F 01 (B 0 ), then B 0 = F (ES (B)).
When the P&L distribution is known, backing out the B 0 probability is straightforward. For instance, if we assume that the unconditional distribution of the P&L is a t-distribution, then the expected shortfall is:

ES (B; v) = 0 v v 0 1 1 + 1 v F 01 (B; v) f (B; v) F (B; v) (27)
where f (x; v) and F (x; v) respectively denote the pdf and the cdf of the t-distribution with v degrees of freedom [START_REF] Zhu | A Generalized Asymmetric Student-t Distribution with Application to Financial Econometrics[END_REF]. For B = 1% and v = 2, we obtain B 0 = F (ES (B; v) ; v) = 0:25%, which is close to our benchmark value of 0.2%.

Alternatively, when the P&L distribution is unknown, B 0 can be estimated by:

b B 0 = arg min fB g ( 1 T T X t=1 2 V aR tjt01 (B 0 ; C) 0 ES tjt01 (B; C) 3 2 ) T t=1 : (28)
Then we dene the ES-based hit variable as:

I t (B 0 ) = ( 1 if r t < 0ES tjt01 (B; C) 0 otherwise (29)
and, again, consider the null hypothesis in equation ( 14).

Systemic Risk

Since the recent nancial crisis, the quest for measuring and forecasting systemic risk has never been more popular. Of particular importance is the quantication of the marginal contribution of systemically important nancial institutions to the overall risk of the system. While many methodologies have been recently proposed to measure systemic risk [START_REF] Acharya | Measuring Systemic Risk[END_REF][START_REF] Adrian | CoVaR[END_REF]Engle and Brownless, 2011), there is to the best of our knowledge no ex-post validation method for systemic risk measures. In this section, we show that the Risk Map approach can be used to backtest systemic risk measures.

We follow [START_REF] Adrian | CoVaR[END_REF] and dene the CoVaR measure as the VaR of the nancial system conditional on institutions being under distress. Formally, CoVaR is dened as the B-quantile of the conditional probability distribution of the nancial system asset returns r j :

Pr 2 r j 0CoV aR jj+(r E ) (B) C C + (r i ) 3 = B ( 30
)
where + (r i ) denotes a conditioning event concerning rm i. One possible conditioning event is a situation in which the loss of rm i exceeds its VaR:

Pr 2 r j 0CoV aR jji (B) C C r i 0V aR i (B) 3 = B: (31)
Given this denition, it is obvious that CoVaR can be backtested within the Risk Map framework. Just like with VaR, we need to analyze the frequency of the conditional probability and the magnitude of the losses in excess of the CoVaR. The latter will provide us with some crucial information about the resiliency of the nancial system when a particular rm is in nancial distress.

The Risk Map framework allows one to validate the CoVaR in two dimensions: number and severity of CoVaR exceptions. We dene a CoVaR exception and super exception as r j < 0CoV aR jji (B) and r j < 0CoV aR jji (B 0 ), respectively. By analogy with VaR, we dene the following hit variables:

I j (q) = (
1 if r j < 0CoV aR jji (q) 0 otherwise ; for q = B or q = B 0 (32) with B 0 < B: By denition of the CoVaR exception (and super exception), we have:

- 2 I j (q) C C r i 0V aR i (q) 3 = q: (33)
It is possible to transform the conditional expectation ( 33) into an unconditional one since the condition [START_REF] Lopez | Methods for Evaluating Value-at-Risk Models[END_REF] implies -[I j (q) 2 I i (q)] = q 2 for q = B or q = B 0 , where I i t (B) is the standard VaR(B) hit variable for institution i:

I i (B) = ( 1 if r i < 0V aR i (B) 0 otherwise : (34)
So, the CoVaR-Risk Map can be dened as the non-rejection area of the joint test:

H 0 : - 2 I j (B) 2 I i (B) 3 = B 2 and - 2 I j (B 0 ) 2 I i (B) 3 = B 02 : (35)
Under the null, conditional on the distress of institution i, the probability to observe a loss in the nancial system larger than the CoVaR(B) is precisely equal to B, and the probability to observe a CoVaR super exception should not exceed B 0 .

The CoVaR Risk Map can be constructed by analogy with the VaR Risk Map. Given a sequence of estimated conditional CoVaR for the system,

n CoV aR jji tjt01 (q) o T t=1
for q = B and q = B 0 , we compute the sequences of hits 8 I j t (q)

9

. The corresponding test statistic can then directly be derived from the LR M U C (B; B 0 ) test shown in [START_REF] Christo¤ersen | Backtesting[END_REF]. In this case,

N 1 = P T t=1 2 I j t (B) 0 I j t (B 0 ) 3 2 I i t (B), N 2 = P T t=1 I j t (B 0 ) 2 I i t (B), and N 0 = T 0 N 1 0 N 2 :
The non-rejection area of the test can be represented as in Figure 3. For a given nancial institution i, the numbers of CoVaR exceptions and CoVaR super exceptions correspond to one particular cell on the Risk Map. It produces a direct diagnostic about the validity of the systemic risk measure, which jointly accounts for the number and the magnitude of the exceptions.

Finally, given the CoVaR denition, it is possible to compute the di¤erence between (1) the VaR of the nancial system conditional on the distress of a particular nancial institution i and (2) the VaR of the nancial system conditional on the median state of institution i: Adrian and Brunnermeier (2011) call this di¤erence 1CoV aR:

1CoV aR jj+(r E ) (B) = CoV aR jjr E 0V aR E (B) (B) 0 CoV aR jjr E M edian E (B): (36)
The measure 1CoV aR jji quanties how much an institution contributes to the overall systemic risk. Backtesting 1CoV aR jji can be achieved by applying the Risk Map methodology successively to CoV aR jjr E 0V aR E (B) and CoV aR jjr E M edian E .

Margin Requirements

Tail risk models are also used to set margin and collateral requirements on derivatives markets. Margins are key to protect derivatives users against the default of their counterparties. The di¢cult trade-o¤ faced by the derivative exchange is to set margins high enough to mitigate default risk but not so high as to shy traders away and damage liquidity.

The initial margin C for one futures contract (long or short position) must be set so that the probability of a futures price F t change, r t = F t 0 F t01 , exceeding the margin is equal to a prespecied level:

Pr[r t < 0C tjt01 (B)] = Pr[r t > C tjt01 (B)] = B: (37)
Depending on the expected volatility, the derivatives exchange frequently adjusts the level of the margin, as shown by Brunnermeier and Pedersen (2009, Figure 1) for the S&P 500 futures. The empirical literature has considered a variety of distributions for futures price changes and volatility dynamics [START_REF] Booth | Prudent Margin Levels in the Finnish Stock Index Futures Market[END_REF][START_REF] Cotter | Margin Exceedences for European Stock Index Futures using Extreme Value Theory[END_REF]. The Risk Map approach can be used to test whether actual margins or optimal margins according to a given modeling technique generate too many margin exceedances and too many margin super exceedances. The analysis would have to be conducted separately for the left and right tails.

A related problem is the determination of the margin requirements for the clearing members of a given clearing house (Cruz [START_REF] Lopez | Clearing House, Margin Requirements, and Systemic Risk[END_REF][START_REF] Hurlin | Margin Backtesting[END_REF][START_REF] Hurlin | Margin Backtesting[END_REF]. In this case again, VaR models are one of the two main techniques used to set margins the other one being the SPAN system. The issue of validating clearing members margins takes nowadays central stage as clearing houses have moved to clear new products that used to be over-the-counter (OTC) [START_REF] Du¢e | Does a Central Clearing Counterparty Reduce Counterparty Risk?[END_REF].

To show how the Risk Map can be used in the context of a clearing house, let ! i;t01 be the vector of positions of clearing member i at the end of day t 0 1:

! i;t01 = 2 6 6 4 ! i;1;t01 . . . ! i;D;t01 3 7 7 5 ( 38
)
where D is the number of derivatives contracts (e.g. futures, options, CDS, and swaps) cleared by this clearing house and i = 1; :::; N . We assume that these contracts are written on U di¤erent underlying assets. To arrive at a margin for this portfolio, the clearing house considers a series of S scenarios representing potential changes in the value and volatility of the underlying assets. For each scenario, the value of the portfolio is recomputed, or marked-to-model, using derivatives pricing formulas, and the associated hypothetical P&L is computed:

e r i;t = 2 6 6 4 e r 1 i;t . . . e r S i;t 3 7 7 5 : (39)
Notice that this simulation-based technique allows the clearing house to account for diversication among underlying assets and maturities, which reduces collateral requirements. From this simulated distribution of P&L, the clearing house can set the margins for clearing member i such that:

Pr[e r s i;t < 0C V aR i;tjt01 (B)] = B (40)
with s = 1; :::; S. 10 The clearing house will proceed in the same way for the N 0 1 other clearing members and only those who will be able to pile up this amount of collateral on their margin accounts will be allowed to trade on the next day. On a regularly basis, the risk-management department of the clearing house and the regulatory agencies check the validity of the margining system. In particular, they need to check whether the hypothetical shocks used in the scenarios are extreme enough or whether the estimation of the derivative prices is reliable. Of particular concern is a situation in which the collateral is set at too low a level. In this case, a default by a clearing member following a big trading loss would lead to a massive shortfall, which may propagate default within the clearing system [START_REF] Eisenberg | Systemic Risk in Financial Systems[END_REF]. The evaluation of the margining system can be conducted using the Risk Map approach. In this particular case, the analysis can be conducted by clearing member or for all the clearing members pooled together.

In addition to the VaR-based margins presented in equation ( 40), clearing houses have recently started using expected shortfall-based margins C ES , which are dened as:11

C ES i;tjt01 (B) = E[e r s i;t je r s i;t < 0C V aR i;tjt01 (B)]: (41)
The margin model of the clearing house can then be tested using both C V aR and C ES by following the procedure described in Section 3.1.3.

Conclusion

In nance, risk models must capture both (1) the frequency of the losses and (2) the magnitude of these losses. Surprisingly, current model validation techniques completely neglect the magnitude dimension of nancial risk. In this paper, we have proposed a validation framework allowing risk managers and regulators to assess the validity of a risk model by accounting for both the number and the magnitude of extreme losses. We have introduced the concept of a super exception and designed a testing procedure that combines information about both exceptions and super exceptions. The main advantages of the Risk Map is that it is as simple to use as standard validation techniques; it is a formal hypothesis testing framework; and it can be applied to any tail risk model. The Risk Map framework can be handy in validating market risk, credit risk, or operational risk estimates, systemic risk measures such as CoVaR, or the margin system of a clearing house.

The Risk Map approach may prove particularly e¤ective in banking. Indeed, as a generalization of the system currently used by banking regulators to validate banks risk models, the Risk Map could help detecting misspecied risk models and penalizing banks that experience VaR exceptions that are too frequent and/or too large. In this case, bank capital requirements would directly be a¤ected by the conclusions of the Risk Map analysis.

In order to ease the implementation of the present methodology, we have created a website that automatically generates Risk Maps: www.RunMyCode.org/CompanionSite/site2 Users only have to upload their risk estimates and P&L data and to indicate the respective coverage rates. All calculations are done in a cloud computer and results are directly displayed to the users. Using the simulated P&L distribution issued from this DGP, it is then necessary to select a method to forecast the VaR. This method represents the internal VaR model used by the nancial institution. We rst consider a VaR calculation method that perfectly matches the P&L distribution. As a result, unconditional coverage is satised for both standard and super exceptions. In a second experiment, we use a method that induces a violation of unconditional coverage for super exceptions: a method that is valid with respect to the current backtesting procedures, but it generates too many extreme losses (see Panel A of Table A1). Then, the third experiment considers the case when both the internal VaR models and the auxiliary model are misspecied.

v 0 2 v z 2 t01 2 t01 + 2 t01 : (

< Insert Table A1 >

Recall that the aim of the experiments is to assess the capacity of our method to estimate the V aR(B 0 ) produced by the internal model of the bank. For that, it is possible to compare directly the true VaR, denoted V aR 0 tjt01 (B 0 ), to the estimated one, denoted V aR tjt01 (B 0 ; b C): However, what is most important is not comparing the VaRs, but the exceptions induced by these VaRs, since backtesting is based on exceptions. Ideally, the timing of the super exceptions should be the same with the true and the estimated V aR(B 0 ).

We denote I 0 t (B 0 ) the true hit process associated to V aR 0 tjt01 (B 0 ) and I t B 0 ; b C the estimated hit process associated to V aR tjt01 (B 0 ; b C): There are two types of errors that can occur: false positive (type 1 error) and false negative (type 2 error). We evaluate the quality of our approach by using three indicators. The True Positive Rate (TPR) denotes the fraction of super exceptions that occur concurrently for the true and for the estimated V aR(B 0 ). More formally, if we denote T the out-of-sample size of the sample, we dene:

T P R = P T t=1 I 0 t (B 0 ) 2 I t B 0 ; b C P T t=1 I 0 t (B 0 ) : (A3)
The False Positive Rate (FPR) represents the frequency of type 1 error. It gives the fraction of estimated super exceptions events that were observed to be non events (false alarms).

F P R = P T t=1 [1 0 I 0 t (B 0 )] 2 I t B 0 ; b C P T t=1 [1 0 I 0 t (B 0 )] : (A4)
Finally, we consider the True Negative Rate (TNR), which measures the fraction of non super VaR exceptions that are correctly identied (complementary of the type 2 error).

T N R =

P T t=1 [1 0 I 0 t (B 0 )] 2 h 1 0 I t B 0 ; b C i P T t=1 [1 0 I 0 t (B 0 )] : (A5) Experiment 1: Valid Internal VaR Model
In the rst experiment, the internal risk model of the bank corresponds to the true DGP. In this context, the true conditional VaR is dened as:

V aR 0 tjt01 (B) = 0 s C 0 2 C t F 01 (B; C) (A6)
where F (:; C) denotes the c.d.f. of the t (v) distribution. Let us denote V aR 0 tjt01 (B 0 ) the true conditional VaR (unobservable) for the coverage rate B 0 . We consider the case where B = 1% and B 0 = 0:2%: Theoretically, these VaR forecasts are deduced from a valid internal model and consequently the unconditional coverage assumption is satised for all coverage rates. Given the simulated returns path and the VaR displayed by the bank,

n V aR 0 tjt01 (B) o T t=1
, we apply our calibration procedure to estimate the VaR for a coverage rate B 0 . In this rst experiment, we consider an auxiliary model also dened as a t (C)-GARCH(1; 1) model:

r t = t z t s C 0 2 C (A7) 2 t = D 0 + D 1 C 0 2 C z 2 t01 2 t01 + D 2 2 t01 : (A8)
The parameters of this auxiliary model are estimated by QML. Let us denote b C the estimator of the distributional parameter (i.e., the degree of freedom of the t-distribution). Conditionally on this estimated parameter and to the V aR 0 tjt01 (B) produced by the bank, we can dene the estimated V aR(B 0 ) as:

V aR tjt01 (B 0 ; b C) = V aR 0 tjt01 (B) F 01 B 0 ; b C F 01 B; b C : (A9)
In Panel B of Table A1, the three indicators of the quality of the calibration process are reported. In the second column, the frequency of super VaR exceptions obtained from our estimated 0.2% VaR is reported for various sample sizes. Recall that in this experiment, the unconditional coverage is valid (since the VaR model corresponds to the true P&L DGP), and the super exception frequency should be equal to 0.2% if the calibration method correctly estimates the VaR. The third, fourth, and fth columns, respectively, report the True Positive Rate, the False Positive Rate, and the True Negative Rate obtained from 10,000 simulations. Overall, Panel B shows that the performance of our estimation method is very good.

Experiment 2: Invalid Internal VaR Model

In the second experiment, we introduce a discrepancy between the internal risk model used by the bank to generate the VaR gures and the auxiliary model used in the calibration procedure. More specically, we assume that the bank uses Historical Simulation whereas the auxiliary model is a t-GARCH(1,1) model. We see in Panel C of Table A1 that even in a situation in which the auxiliary model is wrong, our calibration procedure allows us to extract a reliable estimate of V aR (B 0 ).

Experiment 3: Invalid Internal and Auxiliary Models

In the third experiment, neither the internal model nor the auxiliary model conform with the DGP, which is assumed to be a MS-GARCH model described by equations ( 18) and [START_REF] Christo¤ersen | Backtesting Value-at-Risk: A Duration-Based Approach[END_REF]. We assume that the bank computes its VaR through Historical Simulation and we use a GARCH(1,1) model as the auxiliary model. In that case, V aR(B 0 ) is simply dened as:

V aR tjt01 (B 0 ; b C) = V aR tjt01 (B) 8 01 (B 0 ) 8 01 (B) (A10)
where 8 (:) is the cdf of the standard normal distribution. Results in Panel D indicate that the calibration procedure works ne in this case too. Note that this does not imply that our calibrated VaR(B 0 ) is always a good proxy for the true VaR(B 0 ) in term of bias. It only means that the errors in the VaR(B 0 ) estimates are not su¢ciently important to have any material impact on the number of super exceptions. Even with a MS-GARCH model, the TPR is close to 95% and the TNR close to 99%. Notes: This gure displays the daily P&L, V aR (B), and V aR (B 0 ) for a hypothetical portfolio, with B = 1% and B 0 = 0:2%. Both the P&L and VaR series are simulated using a t-Garch model. A VaR exception is dened as r t < 0V aR tjt01 (B) whereas a super exception is dened as r t < 0V aR tjt01 (B 0 ). Over this 500-day sample period, there are ve exceptions and four super exceptions, which are highlighted with stars.

Table 3: Generalized Tra¢c Light System

Notes: This table displays the increase in the multiplicative factor k that is applied to the average VaR to generate the Market Risk Charge (M RC = 12:5 2 k2V aR) within the Basel framework. Unlike the standard Tra¢c Light system, the Generalized Tra¢c Light system accounts for both the number of exceptions and super exceptions. By analogy to the standard Tra¢c Light system, the base multiplicative factor is set to a value of three and can increase up to four. The no penalty zone corresponds to the green zone of the Risk Map and the maximum penalty zone corresponds to the red zone of the Risk Rap. Between these two zones, the bank incurs an increasing penalty wich is obtained by linear interpolation. The upper panel is for a sample size of T = 250 whereas the lower panel is for a sample size of T = 500. Notes: This table presents the design and the results of the Monte Carlo simulation. In the three experiments, we vary (1) the data generating process (DGP) of the prot-and-loss data, (2) the internal VaR model used by the bank, and (3) the auxiliary model used to generate the VaR(B 0 ) estimates. In experiment 1, (1), (2), and (3) match perfectly. In experiment 2, only the internal model di¤er from the DGP, whereas in experiment 3, both internal and external models di¤er from the DGP. For each experiment, we report the True Positive Rate (TPR; fraction of super exceptions that occur concurrently for the true and for the estimated V aR(B 0 )); the False Positive Rate (FPR; fraction of estimated super exceptions events that were observed to be non events); and True Negative Rate (TNR; fraction of non super exceptions that are correctly identied). In each experiment, we vary the sample size from 250 to 2,000 observations and we use 10,000 simulations.

  A2)Parameterization of the coe¢cients and initial conditions are deduced from maximumlikelihood estimated parameters for the S&P 500 index daily returns over the period 08/07/2010 to 01/07/2011. The parameter values are ! = 4:9453e 07 , D = 0:1012, = 0:8550, and v = 4:1561. The initial condition2 1 is set to the unconditional variance.

Figure 1 :

 1 Figure 1: VaR Exception vs. VaR Super Exception

Figure 2 :

 2 Figure 2: P-values of the Multivariate Unconditional Coverage Test

Figure 3

 3 Figure 3: The Risk Map

Figure 4 :

 4 Figure 4: Valid vs. Invalid Model

Figure 5 :

 5 Figure 5: Backtesting Results for a Large Bank

Figure 6 :

 6 Figure 6: Changing the Value of the B 0 Coverage Rate

  Notes: Panel A presents the empirical size of the Risk Map for various values of (B; B 0 ) and sample sizes. The empirical size corresponds to the rejection rate calculated over 10,000 simulations for a nominal size equal to 5%. Panel B reports the empirical power of the Risk Map when the P&L are generated from a MS-GARCH model and the VaR forecasts are computed through Historical Simulation. Panel C reports the power of the LR U C tests for the 1%, 2%, and 5% coverage rates.

  and LR U C (B 0 ) tests. For instance, with 500 observations, if the number of V aR (1%) exceptions is between two and nine, we cannot reject the risk model at the 95% condence level. Similarly, if the number of super exceptions is strictly greater than two, we reject the validity of the VaR model at a similar condence level for the reason that it leads to too many super exceptions. It is then possible to check whether any risk model lays inside the global non-rejection area, i.e., 2 $ N $ 9 and N 0 $ 3. It is important to notice that in this rst approach, we investigate this double validation process (loss frequency + loss magnitude) in a disjointed way. It boils down to consider the LR U C (B) and LR U C (B 0 ) tests independently, which does not allow us to control for the nominal size of the test, i.e., the probability of rejecting a valid model. An alternative approach is to jointly test the number of VaR exceptions and super exceptions:

Table 1 :

 1 Finite Sample Properties of the Risk Map Panel A: Empirical Size Sample B = 5%; B 0 = 1% B = 2%; B 0 = 0:4% B = 1%; B 0 = 0:2% Panel B: Empirical Power of the Risk Map Sample B = 5%; B 0 = 1% B = 2%; B 0 = 0:4% B = 1%; B 0 = 0:2%

	500	0.047	0.043	0.046
	1,000	0.050	0.041	0.043
	2,000	0.054	0.053	0.039
	500	0.393	0.345	0.305
	1,000	0.335	0.365	0.391
	2,000	0.378	0.518	0.645
		Panel C: Empirical Power of the LR U C Test
	Sample	B = 5%	B = 2%	B = 1%
	500	0.399	0.361	0.326
	1,000	0.275	0.297	0.339
	2,000	0.220	0.378	0.434

Table A1 :

 A1 Monte Carlo ExperimentPanel A: Summary of the Monte Carlo Experiment Experiment 1 Experiment 2 Experiment 3

	P&L Data Generating Process	t-Garch(1,1)	t-Garch(1,1)	MS-Garch(1,1)
	Internal VaR Model	t-Garch(1,1)	HS	HS
	Auxiliary Model	t-Garch(1,1)	t-Garch(1,1)	Garch(1,1)
	Panel B: Experiment 1 -Valid Internal VaR Model
	Sample Size	TPR	FPR	TNR
	250	0.8763	0.0002	0.9998
	500	0.9220	0.0002	0.9998
	1,000	0.9535	0.0001	0.9999
	1,500	0.9562	0.0001	0.9999
	2,000	0.9642	0.0001	0.9999
	Panel C: Experiment 2 -Invalid Internal VaR Model
	Sample Size	TPR	FPR	TNR
	250	0.7785	0.0006	0.9994
	500	0.8134	0.0006	0.9994
	1,000	0.8424	0.0007	0.9993
	1,500	0.8366	0.0006	0.9994
	2,000	0.8160	0.0007	0.9993
	Panel D: Experiment 3 -Invalid Internal and Auxiliary Models
	Sample Size	TPR	FPR	TNR
	250	0.9515	0.0020	0.9980
	500	0.9606	0.0021	0.9979
	1,000	0.9543	0.0021	0.9979
	1,500	0.9514	0.0021	0.9979
	2,000	0.9584	0.0022	0.9978

In a recent survey on trading risk, members of the Basel Committee state that "account for the severity of losses beyond the condence threshold [...] is especially important for regulators, who are [...] concerned about exactly these losses" (Basel Committee on Banking Supervision,

2011b).

We show in Section 3.1.3 that our method can also be used with expected shortfalls.

The p-values of the test can be based on either the asymptotic distribution or the nite sample distribution, which can be generated by simulation. See Section 2.3 for more discussion on nite sample properties.

The parameter values are ! = 0:419; ! = 1:988; B = 0:014; B = 0:115; C = C = 0; = 0:046; = 00:040; = 0:994, and = 0:986:

When -(r t ) 6 = 0, the estimated VaR can simply be deduced from[START_REF] Cotter | Margin Exceedences for European Stock Index Futures using Extreme Value Theory[END_REF] by adding the unconditional average return.

The VaR and P&L gures correspond to the entire trading portfolio of the bank. VaR and P&L data have been extracted directly from the rms annual reports, as in Frésard,[START_REF] Frésard | The Pernicious E¤ects of Contaminated Data in Risk Management[END_REF].

For instance, the European Central Banks Recommendations for Central Counterparties (2009) state that Margin requirements [...] should be su¢cient to cover losses that result from at least 99 % of the price movements over an appropriate time horizon.

See the Options Clearing Corporation (OCC) website for a recent example of expected shortfall-based margins: http://www.optionsclearing.com/risk-management/margins/

t = ! + D

Appendix: Monte Carlo Experiment

We assess the accuracy of the calibration procedure for V aR(B 0 ) presented in Section 2.4 in a series of controlled experiments. The aim of these experiments is to systematically compare the estimated V aR(B 0 ) to the true V aR(B 0 ). To conduct the experiments, we need to specify (1) the data-generating process (DGP) of the P&L, (2) the internal VaR model used by the bank, and (3) the auxiliary model that we use to generate the VaR estimates. We check whether our approach is able to accurately estimate V aR(B 0 ) when ( 1), ( 2), and/or (3) are misspecied.

For the DGP of the P&L, we follow Berkowitz, Christo¤ersen and Pelletier (2011) and assume that returns r t are issued from a t (v)-GARCH(1; 1) model:

where fz t g is an i:i:d: sequence form a Students t-distribution with v degrees of freedom and where conditional variance is: Notes: Panel A presents some descriptive statistics on the daily trading prot-and-loss (P&L),

VaR(B = 1%), and VaR(B 0 = 0:2%) for La Caixa between January 1, 2007 and December 31,

2008. All gures are in thousands of euros. Panel B presents some detailed information about the estimation of VaR(B 0 = 0:2%). We report the estimated parameters and standard errors for the auxiliary t-GARCH model, along with the value of the log-likelihood function. For each time series, the number of observations is 500.


Christophe Hurlin
email: christophe.hurlin@univ-orleans.fr

Christophe Pérignon
email: perignon@hec.fr.

Christophe Pérignon Margin

Backtesting 2012 Halshs-00746274

Margin Backtesting

Keywords: Collateral Requirements, Futures Markets, Tail Risk, Derivatives Clearing

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

What makes derivatives exchanges so special is the extremely low default risk that market participants are exposed to. Collateral requirements or margins are the major tools to protect derivatives users against the default of their counterparties. The challenge faced by derivative exchanges is to set margins high enough to mitigate default risk but not so high as to shy traders away and damage liquidity. The goal of this paper is to design a methodological framework allowing risk managers and regulators to check the validity of the margins charged to derivatives users. It consists of a series of diagnostic tools allowing one to detect misspecied models that lead to margins that are either excessively conservative or lenient. Checking the validity of a margining system is particularly important nowadays as more and more OTC derivatives products are migrating to clearing platforms (Du¢e and Zhu 2010). 1There are two types of margining systems used in practice: the Standard Portfolio Analysis of Risk (hereafter SPAN) system and the Value-at-Risk (hereafter VaR) model. Both margining systems consider a series of scenarios representing potential one-day ahead changes in the underlying assets price and volatility and generate simulated distributions of potential prot-and-loss (hereafter P&L) for derivatives users. Under SPAN, the system selects for each position the largest loss across all considered scenarios, combines nancial instruments within the same underlying asset, and total margin is given by the sum of the risk of all underlying assets less some diversication adjustments (CFTC 2001; Chicago Mercantile Exchange 2009). Di¤erently, VaR margins are set such that the probability of the loss on the entire derivatives portfolio exceeding the margin is equal to a pre-specied level, such as 1% [START_REF] Knott | Modelling Risk in Central Counterparty Clearing Houses: A Review[END_REF]; Cruz Lopez, Harris and Pérignon 2011).

On a regular basis, the risk-management department of the clearing-house and the regulatory agencies check the validity of the margining system. In particular, they make sure that the hypothetical shocks used in the scenarios are extreme enough and that the estimation of the derivative prices is reliable. Of particular concern is a situation in which margins are set at too low a level. In this case, a default by a clearing member following a large trading loss would lead to a massive shortfall, which may propagate default within the clearing system (Jones and Pérignon 2011).

While the performance of the SPAN system has been investigated in a number of papers [START_REF] Kupiec | The Performance of S&P 500 Futures Product Margins under the SPAN Margining System[END_REF][START_REF] Kupiec | Regulatory Competition and the E¢ciency of Alternative Derivative Product Margining Systems[END_REF] White 1996; Eldor, Hauser and Yaari 2011), VaR margins have not to our knowledge been investigated in the academic literature yet. This increasinglypopular modeling approach o¤ers several advantages though. First, as it is based on a quantile, it allows derivatives exchanges to pick the level of tail risk that best ts with their risk tolerance. A second advantage is that quantile-based margins are less sensitive to simulation design than maximum-based margins, such as SPAN margins. Most importantly for this study, quantile-based margins can be validated ex-post using formal backtesting methodologies. For instance, as an B% quantile is by denition exceeded B% of the time, one can check whether in reality B% VaR margins are indeed exceeded B% of the time. A similar ex-post validation is not possible for SPAN margins.

Compared to market risk VaR [START_REF] Jorion | Value at Risk: The New Benchmark for Managing Financial Risk[END_REF]Christo¤ersen 2009a), which is used by banks to monitor their trading risk and compute capital requirements, the estimation of VaR margin is much simpler. In general, the quantile of the return at time t cannot be estimated without making some strong assumptions about the underlying distribution. Specically, since there is only one return observation on each date, it is usually assumed that the returns are independently and identically distributed over time. Under these assumptions, VaR can be estimated from the historical path of past returns. In the context of VaR margin, however, the situation is quite di¤erent because P&L observations are simulated at time t. This is an ideal situation from an econometric point of view because the quantile of the P&L distribution can be directly estimated without making any assumptions regarding its behavior over time.

Our main contribution to the literature on derivatives margins is to present a backtesting framework for derivatives margins. It consists of a series of hypotheses that must be validated by any well-functioning margin model. Then, we propose a series of statistical tests that aim to test these hypotheses in order to detect misspecied margining models. We show that these validation tests can be implemented either at the individual investor level or at the global exchange level. In this framework, not only can we nd out whether a model is misspecied but we can also unmask the reasons of rejection of a misspecied model. Finally, in order to ease the implementation of the backdating methodologies presented in this paper, we created a website on which users can freely upload their margins and P&L data and run the associated computer codes (www.RunMyCode.com/backtest release date is December 2011).

The outline of the paper is the following. In Section II, we discuss how to estimate VaR margins and present the main testable hypotheses. In Section III, we show how to test these hypotheses in order to validate or invalidate a given margining model. We present in Section IV some statistical tests that aim to validate the margining model at the exchange level. Section V summarizes and concludes our paper.

2 Margin Estimation and Testable Hypotheses

Margin Estimation

For retail investors, margins are typically set at the contract level (e.g. $1,000 for any long or short position in a given futures contract). Depending on the expected volatility, the derivatives exchange can adjust the level of the margin, as shown by Brunnermeier and Pedersen (2009, Figure 1) for the S&P 500 futures. Di¤erently, for large market participants such as clearing members, margins are computed at the portfolio level in order to account for diversication e¤ects and are adjusted daily. The VaR margin B i is set such that there is a probability B that the loss on the derivative position exceeds the margin:

Pr 2 V i;t < B i; tjt01 (B) 3 = B ( 1
)
where V i denotes the P&L of investor i and B is called the coverage rate. Let ! i;t01 be the vector of positions of clearing member i at the end of day t 0 1:

! i;t01 = $ $ " ! i;1;t01 . . . ! i;D;t01 ! % % # ( 2
)
where D is the number of derivatives contracts (futures and options) traded on this exchange and i = 1; :::; N . To arrive at a margin for this portfolio, the clearing house considers a series of S scenarios representing potential one-day ahead changes in the level and volatility of the underlying assets. For each scenario, the value of the portfolio is recomputed, or markedto-model, using futures and option pricing formulas, and the associated hypothetical P&L is computed:

v i;t = $ $ " v 1 i;t . . . v S i;t ! % % # : (3) v s i;t 9 S
s=1 , the VaR margin for clearing member i is given by:

> B i;t = percentile 8 v s i;t 9 S s=1 ,100B : (4)
The clearing house will proceed in the same way for the N 0 1 other clearing members and only those who will be able to pile up this amount of collateral on their margin accounts will be allowed to trade on the next day.

Backtesting VaR Margin

Traditionally the quality of the forecast of an economic variable is assessed by comparing its ex-post realization with the ex-ante forecast value. The comparison of the various forecast models is thus generally made by using a criterion such as the Mean Squared Error criterion or standard information criteria (AIC and BIC). However, this approach is not suitable for VaR margin forecasts because the true quantile of the P&L distribution is not observable. That is why VaR assessment is generally based on the concept of margin exceedance (also called hits, violations, or exceptions).

For a given clearing member i, a margin exceedance is said to occur if the ex-post realization of the P&L at time t, V i;t , is more negative than the ex-ante VaR margin forecast. Let I t (B) be a binary variable associated with an B% VaR margin at time t:

I t (B) = 1 if V i;t < 0B i; tjt01 (B) 0 otherwise . ( 5
)
As stressed by [START_REF] Christo¤ersen | Evaluating Interval Forecasts[END_REF]Christo¤ersen ( , 2009b)), VaR forecasts are valid if and only if the violation process I t (B) satises the following two hypotheses:

F The Unconditional Coverage (hereafter UC) hypothesis: The probability of an ex-post return exceeding the VaR forecast must be equal to the B coverage rate:

Pr [I t (B) = 1] = E [I t (B)] = B. (6)
F The Independence (hereafter IND) hypothesis: VaR margin violations observed at two di¤erent dates for the same coverage rate must be distributed independently. Formally, the variable I t (B) associated with a margin exceedance at time t for an B% coverage rate should be independent of the variables I t0k (B), 8k 6 = 0. In other words, past VaR violations should not be informative about current and future violations.

The UC hypothesis is quite intuitive. Indeed, if the frequency of violations observed over T days is signicantly lower (respectively higher) than the coverage rate B, then risk is overestimated (respectively underestimated). However, the UC hypothesis shades no light on the possible dependence of margin exceedances. Therefore, the independence property of violations is an essential one, because it is related to the ability of a VaR margin model to accurately model the higher-order dynamics of the P&L. In fact, a model that does not satisfy the independence property can lead to clusterings of margin exceedances even if it has the correct average number of violations. Consequently, there must be no dependence in the violations variable, whatever the coverage rate considered.

When the UC and IND hypotheses are simultaneously valid, VaR forecasts are said to have a correct Conditional Coverage (hereafter CC), and the VaR violation process is a martingale di¤erence with:

E [I t (B) 0 Bj A t01 ] = 0. (7)
This last property is at the core of most of the validation tests for VaR models (Christoffersen 1998; Engle and Manganelli 2004; Berkowitz, Christo¤ersen and Pelletier 2011). It worth noting that equation [START_REF] Christo¤ersen | Value-at-Risk Models[END_REF] implies that the violation I t (B) has a Bernoulli distribution with a success probability equal to B:

fI t (B)g are i.i.d. Bernoulli (B) . ( 8
)
3 Tests of Margin Accuracy

Frequency of Margin Exceedances

A rst way of testing margin accuracy is to test the number or the frequency of margin exceedances. Thus, the null hypothesis corresponds to equation ( 6):

H 0;U C : E [I t (B)] = B: (9)
A rst statistical test, called the z-test, is based on a normal approximation and the assumption of independence. Let us consider a sequence fI t (B)g T t=1 of T margin exceedances associated to VaR(B%) margins and denote by H the total number of exceedances or hits, H = 2 T t=1 I t (B) : If we assume that the variables I t (B) are i:i:d:, then under the null of U C, the total number of hits has a Binomial distribution:

H B (T; B) (10)
with E (H) = BT and 8 (H) = B (1 0 B) T: For a large T sample, the Binomial distribution can be approximated by a normal distribution and a simple Z-test statistic can be dened as:

Z = H 0 BT F B (1 0 B) T N (0; 1) : (11)
Alternatively, [START_REF] Kupiec | Techniques for Verifying the Accuracy of Risk Measurement Models[END_REF] and [START_REF] Christo¤ersen | Evaluating Interval Forecasts[END_REF] propose a Likelihood Ratio (hereafter LR) test based on the process of V aR margin exceedances I t (B) : Under H 0 , the LR statistic is dened as:

LR U C = 02 ln D (1 0 B) T 0H B H E + 2 ln 1 0 H T T 0H H T H d 0! T !1 2 (1) : (12)
Under the null (9), the LR U C statistic converges to a chi-square distribution with two degrees of freedom. The intuition for the LR test is the same as for the Z statistics. The null of UC is not rejected if the empirical frequency of VaR margin exceedances H=T is close enough to the coverage rate B: Jorion (2007) reports some non-rejection regions for the LR U C test. For a 5% nominal size and sample size T = 250, the UC assumption is not rejected if the total number of VaR(1%) violations is strictly smaller than 7. If the sample size is equal to 500, the total number of exceedances must strictly range between 1 and 11.

Frequency and Severity of Margin Exceedances

A key limitation of the previous approach is that it is unable to distinguish between a situation in which losses are below but close to the margin and a situation in which losses are considerably below the margin. Colletaz, Hurlin and Pérignon (2011) propose a backtesting methodology that is based on the number and the severity of VaR exceptions. Their approach exploits the concept of super exception, which is dened as a loss greater than a super VaR margin B i; tjt01 (B 0 ) whereas the coverage probability B 0 is much smaller than B (e.g. B = 1% and B 0 = 0:2%). As in Section 2.2, we dene a hit variable associated with B i; tjt01 (B 0 ):

I t (B 0 ) = 1 if V i;t < B i; tjt01 (B 0 ) 0 otherwise with B 0 < B: (13)
The dening feature of their approach is to account for both the frequency and the magnitude of VaR margin exceedances. Their intuition is the following. If the frequency of super exceptions is abnormally high, this means that the magnitude of the losses with respect to B i; tjt01 (B) is too large. For both VaR margin exceptions and super exceptions, they propose to use a standard backtesting procedure. Consider a time series of T VaR margin forecasts for an B (respectively B 0 ) coverage rate and let H (respectively H 0 ) be the number of associated VaR margin violations:

H = T : t=1 I t (B) H 0 = T : t=1 I t (B 0 ) : (14)
H 0;M U C : E [I t (B)] = B and E [I t (B 0 )] = B 0 : (15)
The corresponding test statistic is a multivariate unconditional coverage test. This test is based on three indicator variables:

J 0;t = 1 0 J 1;t 0 J 2;t = 1 0 I t (B) (16)
J 1;t = I t (B) 0 I t (B 0 ) = 1 if 0 B i; tjt01 (B 0 ) < V i;t < 0B i; tjt01 (B) 0 otherwise (17)
J 2;t = I t (B 0 ) = 1 if V i;t < 0B i; tjt01 (B 0 ) 0 otherwise : (18)
The fJ i;t g 2 i=0 are Bernoulli random variables equal to one with probability 10B, B0B 0 , and B 0 , respectively. Given these denitions, we can test the joint hypothesis (15) using a LR test. Let us denote H i = 2 T t=1 J i;t , for i = 0; 1; 2, the count variable associated with each of the Bernoulli variables. The multivariate unconditional coverage test is a LR test that the empirical exception frequencies signicantly deviate from the theoretical ones. Formally, it is given by:

LR M U C (B; B 0 ) = 02 ln D (1 0 B) H (B 0 B 0 ) H (B 0 ) H E +2 ln 1 0 H 0 T H H 0 T 0 H 1 T H H 2 T H d 0! T !1 2 (2) : (19)
A Risk Map can be constructed based on the rejection zones for di¤erent condence levels (Figure 1). Note that the cells below the diagonal are not colored as they correspond to situations in which the number of super exceptions exceeds the number of exceptions, which is of course impossible. If the (H; H 0 ) pair corresponds to a green cell, we conclude that we cannot reject the null hypothesis E [I t (B)] = B and E [I t (B 0 )] = B 0 at the 95% condence level. If (H; H 0 ) falls in the orange zone, we can reject the null at the 95% but not at the 99% condence level. Finally, a red cell implies that we can reject the null hypothesis at the 99% condence level.

< Insert Figure 1 >

Independence of Margin Exceedances

The UC property does not give any information about the temporal independence of VaR margin exceedances. However, generating margin exceedances that are temporally independent is an important property for a margining system to have since it suggests that the margin immediately reects new information. A margining system that violates this property leads to clusters of margin exceedances. 2 It is important to note that these two VaR margin properties are independent one from the other. At this point, if a VaR margin does not satisfy either of these two hypotheses, it must be considered as not valid. For example, satisfying the hypothesis of unconditional coverage does not compensate for the possible existence of violations clusters nor the non compliance with the independence hypothesis. On the contrary, there is CC when the VaR margin satises both the UC and IND hypotheses.

LR approach

Christo¤ersen (1998) proposes a LR test based on the assumption that the process of V aR margin exceedances I t (B) is modeled with the following matrix of transition probabilities:

5 = 1 0 01 01 1 0 11 11 (20)
2 [START_REF] Berkowitz | How Accurate Are Value-At-Risk Models at Commercial Banks[END_REF] show that the VaR models used by six large US commercial banks (1) tend to be very conservative, at least when nancial markets are not under stress and (2) lead to clusters of VaR exceedances, which indicates that risk models fail to forecast volatility changes.

where ij = Pr [ I t (B) = j j I t01 (B) = i], i.e., probability of being in state j at time t conditioning on being in state i at time t 0 1. Under the null of independence, we have 01 = 11 = C and:

H 0;IN D : 5 C = 1 0 C C 1 0 C C (21)
where C denotes a margin exceedance probability, which can be di¤erent from the coverage rate B. What these transition probabilities imply is that the probability of experiencing a margin exceedance in the current period depends on the occurrence or not of a margin exceedance in the previous period. The estimated VaR margin exceedance probability is the empirical frequency of violations, H=T: Under the alternative, no restriction is imposed on the 5 matrix. The corresponding LR statistic, denoted LR IN D ; is dened by:

LR IN D = 02 ln 1 0 H T T 0H H T H +2 ln [(1 0 > 01 ) n > n 01 (1 0 > 11 ) n > n 11 ] d 0! T !1 2 (1) (22)
where n ij denotes the number of times we have I t (B) = j and I t01 (B) = i, and:

> 01 = n 01 n 00 + n 01 > 11 = n 11 n 10 + n 11 : (23)
Finally, it is also possible to test the CC assumption for VaR margins. Under CC,

H 0;CC : 5 B = 1 0 B B 1 0 B B (24)
and then:

LR CC = 02 ln D (1 0 B) T 0H (B) H E +2 ln [(1 0 > 01 ) n > n 01 (1 0 > 11 ) n > n 11 ] d 0! T !1 2 (2) (25)
The corresponding LR CC statistic corresponds to the sum of the LR U C and LR IN D statistics. Under the null of CC, it satises:

LR CC = LR U C + LR IN D d 0! T !1 2 (2) : (26)

Regression-based tests

Engle and Manganelli (2004) suggest another approach based on a linear regression model. This model links current margin exceedances to past exceedances and/or past information. Let Hit (B) = I t (B) 0 B be the demeaned process associated with I t (B):

Hit t (B) = 1 0 B if V i;t < 0B i; tjt01 (B) 0B otherwise : (27)
Consider the following linear regression model:

Hit t (B) = E + K : k=1 C k Hit t0k (B) + K : k=1 D k z t0k + " t (28)
where the z t0k variables belong to the information set A t01 . For example, one can use lagged P&L, squared past P&L, past margins, etc. Whatever the chosen specication, the null hypothesis test of conditional e¢ciency corresponds to testing the joint nullity of all the regression coe¢cients:

H 0;CC : E = C k = D k = 0; 8k = 1; :::; K: (29)
The independence hypothesis implies that C k and D k coe¢cients are equal to zero whereas the unconditional coverage hypothesis is veried when E is null. Indeed, under the null hypothesis, E [Hit t (B)] = E (" t ) = 0, which implies by denition that

Pr [I t (B) = 1] = E [I t (B)] = B:
Denote 9 = (E C 1 :::C K D 1 :::D K ) 0 the vector of the 2K +1 parameters in this model and Z the matrix of explanatory variables of model (28), the Wald statistic, denoted DQ CC , in association with the test of CC hypothesis then veries:

DQ CC = > 9 0 Z 0 Z > 9 B (1 0 B) d 0! T !1 2 (2K + 1) (30)
where > 9 is the OLS estimate of 9. Notice that one can also test the UC hypothesis by testing

Autocorrelation test

Rather than using a regression model, Berkowitz, Christo¤ersen and Pelletier (2011) test directly the martingale di¤erence assumption. As under CC the VaR margin exceedance process Hit t (B) is a martingale di¤erence, it should be uncorrelated. A natural test is the univariate Ljung-Box test of H 0;CC : r 1 = ::: = r K = 0 where r k denotes the k th autocorrelation:

LB(K) = T (T + 2) K : k=1 > r 2 k T 0 k d 0! T !1 2 (K) (31)
where > r k is the empirical autocorrelation of order k of the Hit (B) process.

Duration between Margin Exceedances

The UC, IND, and CC hypotheses also have some implications on the time between two consecutive VaR margin exceedances. Following [START_REF] Christo¤ersen | Backtesting Value-at-Risk: A Duration-Based Approach[END_REF], we denote by d v the duration between two consecutive VaR margin violations:

d v = t v 0 t v01 (32)
where t v denotes the date of the v th exceedance. Under CC hypothesis, the duration process d i has a probability density function given by:

f (d v ; B) = B (1 0 B) dv01 d v 2 N 3 . ( 33
)
This distribution characterizes the memory-free property of the VaR margin violation process I t (B), which means that the probability of observing a violation today does not depend on the number of days that have elapsed since the last violation. Note that E (d v ) = 1=B since the CC hypothesis implies an average duration between two margin exceedances equals to 1=B. The general idea of the test consists in specifying a distribution that nests equation (33), so that the memoryless property can be tested through parameter restriction. In this line, [START_REF] Christo¤ersen | Backtesting Value-at-Risk: A Duration-Based Approach[END_REF] use under the null hypothesis the exponential distribution, which is the continuous analogue of the probability density function in equation (33):

g (d v ; B) = B exp (0Bd v ) . ( 34
)
Under the alternative hypothesis, Christo¤ersen and Pelletier (2004) postulate a Weibull distribution for the duration variable:

h (d v ; a; b) = a b bd b01 v exp D 0 (ad v ) b E : (35)
As the exponential distribution corresponds to a Weibull distribution with b = 1, the test for IND is:

H 0;IN D : b = 1 (36)
and for CC is:

H 0;CC : b = 1; a = B (

Tests of Global Validity

To the best of our knowledge, all empirical studies on VaR backtesting consider individual banks in isolation (Berkowitz and OBrien 2002; Pérignon and Smith 2010; Berkowitz, Christo¤ersen and Pelletier 2011). The reason for doing so is that nancial institutions use di¤erent proprietary risk models, which need to be tested separately. Di¤erently on a derivatives exchange, the margins of all market participants are computed using the same model developed by the clearing house. Hence, this model can be tested globally using information from all market participants, which helps in detecting misspecied models.

Denitions

Let us denote I i;t (B) the VaR margin exceedance for clearing member i at time t: We dene the Global Unconditional Coverage (hereafter GUC) hypothesis as a situation where the probability of an ex-post loss exceeds the VaR margin forecast is equal to the B coverage rate for all clearing members:

H 0;GU C : E [I i;t (B)] = B 8i = 1; :::; N . ( 38
)
The GUC means that the frequency of VaR margin exceedances is accurate for all clearing members. Note that it is important not to pool the N margin exceedance processes. Indeed, an under-estimation of the margin for member i could be o¤set by an over-estimation of the margin for another member j. Thus, the GUC hypothesis requires the UC hypothesis to be valid for all clearing members.

We proceed in a similar way for the Global Independence (hereafter GIND) hypothesis. Under GIND, the VaR margin exceedances observed for all the members at two di¤erent dates are independent, i.e., I i;t (B) is independent from I i;t0k (B), 8k 6 = 0. Furthermore, I i;t (B) is also independent from past (and future) VaR margin exceedances of other members I j;t0k (B), 8k 6 = 0 and j 6 = i: Notice that we allow for contemporaneous dependencies between VaR margin exceedances of di¤erent members.

Finally, the global conditionnal coverage (GCC) hypothesis corresponds to a case where the N margin exceedance processes are a martingale di¤erence:

H 0;GCC : E [I i;t (B)j A t01 ] = B 8i = 1; :::; N (39)
where A t01 denotes the information set available at time t01 for all the members, including past values of VaR margin and VaR margin exceedances of other members j:

A natural test for the GUC hypothesis consists in testing the null (38) against the following alternative:

H 1;GU C : E [I i;t (B)] 6 = B for i 2 S (40) E [I i;t (B)] = B for i 2 S (41)
where dim (S) = N 1 satises 1 < N 1 N and dim 0 S 1 = N 2 with N 1 + N 2 = N . Under this alternative, the margin of at least one member does not satisfy the UC hypothesis. Similarly, a natural test of GCC is based on the null (39) against the alternative:

Conclusion

Having a well-functioning margining system is a pre-requesite for any derivatives exchange. It allows the exchange to closely monitor tail risk and make the system resilient. In this paper, we have provided a backtesting framework allowing investors, risk managers and regulators to validate margin models. The statistical tests we have presented capture di¤erent facets of the margin model performance including frequency, timing, and magnitude of margin exceedances. Rather than being substitutes, the di¤erent statistical tests appear to complement each others and can be used to identify the source(s) of model misspecication.

The quest for the ideal margining system is still ongoing. Market participants and regulators want collateral requirements to be less procyclical in order to prevent liquidity spirals [START_REF] Brunnermeier | Market Liquidity and Funding Liquidity[END_REF]. What we show in this paper is a second property that ideal margins should have: their accuracy should be testable ex-post. Indeed, even the most advanced risk measures are of little help if they cannot be systematically validated.

  H 0;U C : E = 0 or test the IND hypothesis with H 0;IN D : C k = D k = 0. A natural extension of the test of Engle and Manganelli (2004) consists in considering a (probit or logit) binary model linking current violations to past ones (Patton 2002; Dumitrescu, Hurlin and Pham 2011).

  37) Christo¤ersen and Pelletier (2004) propose the corresponding LR test (see also Haas 2005) and Candelon et al. (2011) derive a GMM duration-based test.

H 1 ;) 4 . 2

 142 GCC : E [ I i;t (B)j A t01 ] 6 = B for i 2 S (42) E [ I i;t (B)j A t01 ] = B for i 2 S:(43Testing strategies Let us consider an individual test statistic of the UC (or CC) hypothesis, denoted X i ; specic to clearing member i. For instance, for the U C test, this statistic corresponds to the LR U C statistic or the duration-based LR U C statistic. For the CC test, this statistic corresponds to the LR CC statistic, DQ statistic, or duration-based statistic LR CC . Whatever the

Figure 1 :

 1 Figure 1: Backtesting VaR Margins with the Risk Map

  Colletaz, Hurlin and Pérignon (2011) propose a new tool, called the Risk Map, that graphically summarizes all information about the performance of a VaR model. It is based on a joint test of the number of VaR exceptions and VaR super exceptions:

The clearing activity consists in conrming, matching, and settling all trades on an exchange. In order to reduce the risk of non-performance, exchange-traded derivatives are guaranteed against counterparty failure by a central counterparty clearing house. On most derivatives exchanges, only a subset of market participants (i.e., the clearing members) can directly trade with the clearing house whereas all non-clearing member participants have to trade through a designated clearing member.

When the contemporaneous exceedances 1 i;t and 1 j;t are correlated, the distribution of the average statistic : N can be estimated by bootstrap.

chosen test, the individual statistic for member i can be expressed as a non-linear function of the sequence of the margin exceedances of this member, i.e., X i = g (I i;1 (B) ; :::; I i;T (B)) : To test the GUC or GCC null hypotheses, we follow Im, Pesaran and Shin (2003) and use the average of the individual statistics:

If we assume margin exceedances are cross-sectionally independent, i.e., I i;t are independent of I j;s for i 6 = j and all (t; s), the X N statistic converges to a normal distribution when T and N grows large. The intuition is as follows. When T tends to innity, each individual statistic X i converges to the same distribution. For instance, the LR U C statistic converges to a chi-square distribution. Under the cross-sectional independence assumption, the individual statistics X i = g (I i;1 (B) ; :::; I i;T (B)) are also independent. Thus, the individual statistics X i are independently and identically distributed. The central limit theorem is then su¢cient to show that the cross-sectional average mean X N converges to a normal distribution when N tends to innity: 3

An alternative testing strategy consists in combining the p-values associated with the N individual tests. A Fisher type test is then dened by:

For any statistic X i , such as LR U C ; LR CC ; or DQ CC , its p-value is uniformly distributed over [0; 1] : Under the assumption of cross-sectional independence, P X N has a chi-square distribution with 2N degrees of freedom. For large N samples, we can use a standardized statistic: