Beyond Exceedance-Based Backtesting of Value-at-Risk Models

What this chapter covers

Beyond Exceedance-Based Backtesting of Value-at-Risk Models: topics in the order to study them

1. 1Limitations of Exceedance-Based VaR BacktestingStart here because it explains the problems (low power, no severity, ignores clustering) that every later method tries to solve.
2. 2Backtesting with Probability Integral Transform (PIT)PIT is the building block for all the distribution tests, so you need it before them.
3. 3Distribution Tests: Kolmogorov-Smirnov, Anderson-Darling, BerkowitzThese tests are applied to PIT values, so they come after PIT and show how to judge uniformity and independence.
4. 4Expected Shortfall Backtesting and ElicitabilityOnce you can test a full distribution, you can see why ES is harder to backtest directly and what elicitability means.
5. 5Scoring Functions and Comparing VaR ModelsLast, because it builds on elicitability: scoring functions rank models rather than just accept or reject them.

How to prepare Beyond Exceedance-Based Backtesting of Value-at-Risk Models

1. Write down three weaknesses of exceedance counting in your own words, such as ignoring loss size and clustering.
2. Learn PIT as a procedure: forecast distribution, realised return, percentile. State what a correct model implies for the PIT series.
3. Make a one-line comparison of KS, Anderson-Darling and Berkowitz: what each tests and where it is strongest, for example Anderson-Darling giving more weight to the tails.
4. Learn the Berkowitz idea: transform PIT values to a standard normal and test with a likelihood ratio, which can also check independence.
5. Define elicitability simply and note why VaR is elicitable while ES alone is not, and what that means for backtesting and scoring.
6. Practise interpreting scenarios: given a test result or score, say what is wrong with the model and what you would do next.
7. Do timed mixed questions on this chapter and review each wrong answer by the concept it tested.

Common mistakes in Beyond Exceedance-Based Backtesting of Value-at-Risk Models

• Saying a model passes because exceedances match the expected count.

  Fix: Always check whether the question hints at clustering or large tail losses, then name the method that captures it.

• Confusing what the PIT series should look like.

  Fix: Remember: PIT values are uniform; the Berkowitz step converts them to standard normal before testing.

• Mixing up KS and Anderson-Darling.

  Fix: Link KS to the maximum distance and Anderson-Darling to extra weight on the tails.

• Claiming ES cannot be backtested at all because it is not elicitable.

  Fix: State that ES lacks a stand-alone scoring function, yet it can be backtested and jointly scored with VaR.

• Treating a test as proof the model is right.

  Fix: Say that not rejecting means no evidence against the model at that power, and that tests have limited power with small samples.

• Reading a scoring function the wrong way round.

  Fix: Check the definition given in the question; for loss-type scoring functions, lower average score means the better model.

Last-day revision: Beyond Exceedance-Based Backtesting of Value-at-Risk Models

• ✓Exceedance backtests use one binary signal per day and ignore how large the loss was.
• ✓A correct model gives PIT values that are independent and uniform on [0, 1].
• ✓PIT value = forecast CDF evaluated at the realised return.
• ✓Kolmogorov-Smirnov uses the largest gap between empirical and theoretical CDF.
• ✓Anderson-Darling is similar but puts more weight on the tails.
• ✓Berkowitz maps PIT values to standard normal and uses a likelihood ratio test.
• ✓Berkowitz can test for mean, variance and autocorrelation in the transformed series.
• ✓Too many PIT values near 0 or 1 suggest the model understates tail risk.
• ✓ES looks at the average loss beyond VaR, so it captures tail severity.
• ✓A statistic is elicitable if a scoring function exists that the true value minimises in expectation.
• ✓VaR is elicitable; ES on its own is not, but it can be elicited jointly with VaR.
• ✓Scoring functions rank competing models; a lower expected score (for loss-type scores) means a better forecast.

Beyond Exceedance-Based Backtesting of Value-at-Risk Models practice questions

• A bank's VaR model is accurate in the center of the P&L distribution but understates the likelihood of extreme losses. The validation team w…
• An analyst transforms PIT values u_t into z_t = Φ^-1(u_t) to test a VaR model's full distribution. Which statement best describes the advant…
• A risk manager compares two competing 99% VaR models using a quantile (pinball) scoring function applied to realised losses over a long samp…
• Two banks each report 3 exceedances of their 99% one-day VaR over 250 days. Bank A's exceedance losses average 1.1 times VaR; Bank B's avera…
• A risk analyst notes that Value-at-Risk (VaR) is elicitable, whereas Expected Shortfall (ES) is not elicitable on its own. Which statement b…
• A bank's VaR model tends to misstate the extreme tail of losses while fitting the central part of the distribution well. The validation team…
• A validator finds that a bank's VaR model passes the Kolmogorov-Smirnov test on its PIT values, but she suspects the model misstates extreme…
• A bank backtests its 97.5% one-day ES model using a joint approach in which ES is treated as elicitable together with VaR. Which description…

Beyond Exceedance-Based Backtesting of Value-at-Risk Models in other exams

The same ground in other exams, if you are preparing for more than one or want another angle on it.

• FRM Part ICalculating and Applying VaR
• Actuarial Core Principles (IAI)Hypothesis testing and goodness of fit