Speaker(s): Alexander McNeil (University of York)
In this talk we present some recent work on the validation of risk models using reported probability-integral-transform or PIT values. We develop a flexible framework in which backtests are constructed using spectral transformations of PIT values which allows the model validator to impose weights on a set of confidence levels in the tail of a loss distribution. This approach, which nests many existing backtesting approaches including standard VaR exceptions tests at one or more levels, permits an investigation of the trade-off between building tests that are more powerful than conventional VaR-based tests and retaining a focus on the specific extreme confidence levels which are relevant to regulatory capital calculations (such as 99% in Basel market risk applications or 99.5% in Solvency II).
We show how the basic framework may be extended to include so-called tests of conditional coverage, which simultaneously address the accuracy of models in the tail and their ability to model volatility and other aspects of the conditional distribution of losses. We also show how multivariate versions of the tests can be constructed to simultaneously validate models for several sub-portfolios or trading desks.
In the final part of the talk we link backtesting approaches based on PIT values to the literature on comparative testing of forecasting procedures using concepts like elicitability and consistent or proper scaling roles. We show that the two approaches can reveal different deficiencies of risk models and thus provide complementary tools in the model validator’s armoury.