Dirk Tasche.
Stress testing is ubiquitous in today’s banking supervision regime. The stress test results are eagerly anticipated and received by the public and can have serious consequences for banks presenting ‘bad’ numbers. The public discussion of the stress scenarios seems to be focussed on their economic meaning (here is an example). The statistical smallprint relating to stress tests receives much less public attention. I pick up two modelling choices for closer inspection:
- Stress scenarios are meant to be point scenarios.
- Stress test results tend to be presented as single values.
I demonstrate that depending on the understanding of the scenario and the representation of the results, there is a wide range of plausible outcomes of a stress test.
The following analysis is focussed on the most common approach to stress testing in banks. This approach is based on the use of historically observed time series to estimate forecast models for loss-related quantities like default rates or values of specific asset classes. To achieve this, regressions are run on the economic variables that specify the stress scenario. It is at this step that a choice on the statistical meaning of the stress scenario must be made.
What do stress scenarios mean?
Typically, a stress scenario might be phrased like “unemployment rate goes up by 2 percentage points, gross domestic product falls by 0.5%” – which looks pretty clear at first glance. But is the stress scenario really meant to be a precision landing (‘exact’ scenario) or should one rather understand it in the sense of “unemployment rate goes up by 2 percentage points or more, gross domestic product falls by 0.5% or more” (‘equal or worse’ scenario)?
To illustrate the potential impact on stress test results by these different interpretations of the scenario, let us look at the following artificial example:
- A is an asset value (imagine a bank share price) that is given as an exponential function of some economic factor X (for instance a stock index) and a latent factor Y (the unexplained part of A):
- I assumed that X and Y are independent standard normal random variables. Hence A has a lognormal distribution (which, in particular, is not a normal distribution).
- I chose values for the parameters μ = 0, σ = 0.5 and ρ = 0.75 that make the probability distribution of the values of A appear similar to the distribution of the future values of – say – a share price.
- The stress test scenario is specified by a low value outturn (or realisation) xstress of the economic factor X. By choosing xstress as the 10th percentile of the standard normal distribution of X, one obtains xstress = -1.28.
In the exact scenario interpretation, the scenario would be described as “factor X falls to the value xstress = -1.28”. But note that this precise scenario has probability zero such that you cannot expect to ever observe it (but possibly very similar scenarios).
In the equal or worse scenario interpretation, the scenario would be described as “factor X falls to the value xstress = -1.28 or below”. By the definition of xstress as a 10th percentile then it follows that this scenario has a probability of 10% to occur and therefore represents a ‘1 in 10’ event.
In this example, ‘applying a stress scenario’ to the asset value means determining its probability distribution conditional on the event specified by the scenario. The following chart shows the probability densities of the asset value A conditional on the two events corresponding to the two different interpretations of the scenario:
- Exact scenario: xstress = -1.28
- Equal or worse scenario: xstress ≤ -1.28
In addition, the chart also shows the original (i.e. unconditional) density of A. Comparison of either of the conditional densities to the original density demonstrates the fact that the specification of a scenario always reduces the variability of the potential outcomes of a model. But there is still significant statistical uncertainty about the model results.
The chart clearly shows that the likelihood of observing lower asset values is greater under the ‘equal or worse’ scenario than under the ‘exact’ scenario – as should be intuitively expected. As a consequence, stress testing based on an ‘exact’ scenario would underestimate the stress impact if the scenario provider (like a banking supervisor) had in mind an ‘equal or worse’ scenario. Recipients of the stress test results (like the public) might be lulled into a false sense of security if they expected stress results of an ‘equal or worse’ scenario while being delivered ‘exact’ scenario results.
What do stress test results tell us?
Stress tests are seldom reported as probability densities or distributions of the potential outcomes, mainly because the estimation of entire distributions is more difficult than point estimation and requires larger sample sizes to achieve satisfactory accuracy. There are a number of popular statistics that ‘summarise’ a distribution by providing just one of its statistical features:
- Mode: The most likely realisation of the distribution. Formally, this is the point maximising the density.
- Mean: The expected value of the distribution.
- k-th quartile: The smallest number such that the probability to observe a smaller or equal realisation of the distribution is no less than 25*k%.
- Expected shortfall at level k%: The expected value of the distribution conditional on its realisation being in the k% lower tail event. In particular, expected shortfall at level 25% is the expected value of the distribution conditional on its realisation being no greater than the first quartile.
Quite likely, when you see stress test results they are meant to be conditional means. For it is easier to estimate conditional means than any of the other summary statistics listed in the table. In addition, in contrast to the other summary statistics, under the same scenario conditional means are additive across portfolios, i.e. in principle stress results for different portfolios can be easily aggregated. But for good reasons like the desire for more robust estimation or for covering tail risks, authors of stress test reports might choose to present other summary statistics like conditional percentiles or expected shortfall.
Basically, all these choices of summary statistics may make sense in reports of stress test results, depending on criteria like whether or not there is more interest in the centre of the stressed loss distribution or in its tail. The following table gives the values of some summary statistics (mode, mean, 1st quartile and expected shortfall at 25% level) for the three artificial asset value distributions that are shown in the density chart.
| Distribution | Mode | Mean | 1st quartile | 25% expected shortfall |
| Original | 0.78 | 1.13 | 0.71 | 0.54 |
| Exact scenario | 0.55 | 0.65 | 0.49 | 0.41 |
| Equal or worse scenario | 0.46 | 0.55 | 0.41 | 0.33 |
Unsurprisingly, as suggested by the density chart, all the summary statistics are considerably smaller for the equal or worse scenario than for the exact scenario. Hence, whatever summary statistic is chosen to represent the stress test result, in this example the equal or worse scenario gives a much more pessimistic picture of the future asset value than the exact scenario.
Similarly, for both of the scenario interpretations, there is high ‘within’ variation between the values of the summary statistics. Without reaching very far out in the tail of the stressed loss distributions, one already observes differences of more than 50% between the most optimistic (in this case the conditional means) and most pessimistic (the conditional expected shortfall values) stress test results for this artificial example. Hence, if stress test results were calculated as conditional means when conditional expected shortfall was required by the scenario provider then serious underestimation of the stress impact could be the consequence.
Conclusions
Both stress test scenarios and results are subject to choices of statistical meaning and representation. As I have explained, scenarios may be ‘exact’ or ‘equal or worse’. Stress test results may be presented in terms of different summary statistics like means, quartiles or others. Ignorance of the choices made can lead to overoptimistic understanding of the stress test results. For instance, out of a desire for more conservative estimates, the stress test scenario might be meant as an ‘equal or worse’ scenario, with an implicit expectation of the results to be 1st quartiles. As illustrated by my example, then the calculation of the stress test results as means conditional on an ‘exact’ scenario would make the results appear less extreme than they really should be.
Hence in order to avoid misunderstandings between scenario providers (like banking supervisors), stress test performers (like banks) and recipients of stress test results (the public) it is important that the stress tests’ ‘small print’ (for instance the Bank of England’s principles for deriving stress projections) is carefully read by all interested parties.
Dirk Tasche works in the Bank’s Risk Architecture Division.
If you want to get in touch, please email us at bankunderground@bankofengland.co.uk. You are also welcome to leave a comment below. Comments are moderated and will not appear until they have been approved.
Bank Underground is a blog for Bank of England staff to share views that challenge – or support – prevailing policy orthodoxies. The views expressed here are those of the authors, and are not necessarily those of the Bank of England, or its policy committees.
