Seeing into the future is always difficult. But in the world of macroeconomics, just trying to look at the past can be a challenge. Official estimates of economic growth in the UK are regularly revised, so forecasts for growth over the next year have to be made on the basis of an ever-changing report card for the previous year. This post tackles some of the most common questions about UK GDP revisions, a topic close to the heart of many users of the UK’s National Accounts. Are the initial estimates of growth biased? Can you predict revisions? Does UK data get revised more than other countries? And which parts of early estimates of GDP should be approached with caution?
Why is GDP revised?
The UK’s Office for National Statistics (ONS) faces a trade-off: users want statistics to be accurate, but they also want them to be delivered fast. Since it can take up to three years for all the source data to arrive, the ONS appease their impatient users with early estimates based on just a fraction of the eventual source data (about 44%, in the case of the UK’s first estimate of output). Updates to statistical methods can cause revisions even after three years have passed.
The early estimates of GDP are important for the Bank’s Monetary Policy Committee. The more information they have on today’s economy, the better equipped they are to set monetary policy for tomorrow. But since history is subject to change, it is important for anyone making decisions informed by the GDP data to understand the properties of revisions.
Are early estimates of GDP growth biased?
A charge commonly levelled at UK GDP is that early estimates tend to get revised up over time. In other words, early estimates have a downward bias.
It is easy to see how UK GDP gained this reputation. Chart 1 shows the average revision to quarterly growth (the ‘bias’) over time. It was stubbornly positive during the late 1990s and early 2000s, meaning the first estimate of growth was typically too low. After three years the statisticians had, on average, revised up quarterly growth by 0.1pp.
Chart 1 – Bias in quarterly GDP growth over time
But this relationship went out of the window during the financial crisis. The first attempts at quantifying the recession fell short: the true magnitude of the downturn was only apparent three years later (Chart 2). These large downward revisions caused the bias to slip into negative territory, and the myth of the miserly statistician constantly downplaying the UK economy was busted. Recent work by the ONS confirms that when looking over a long time period, there is no statistically significant bias.
Chart 2 – GDP during the Great Recession
Can you predict revisions?
There may not be a simple bias in GDP, but there might be other, more complex patterns in past revisions that can be exploited to predict future revisions. The practice of predicting revisions to published data is known as ‘backcasting’. It’s like forecasting, but backwards.
The Bank of England has been in the backcasting game for many years, and has published a backcast for GDP in the Inflation Report since 2007. The model is supposed to estimate a ‘true’ value of GDP growth. Since it assumes GDP data approach this value over time, trying to estimate the ‘truth’ is equivalent to predicting a sufficiently mature vintage of data.
So, has the backcast helped us predict revisions? It turns out it has – but only marginally. Comparing the ONS’s first estimate of GDP growth and our first backcast with the estimate of growth three years later reveals our average error (RMSE) to be 0.46, compared to the ONS’s 0.48. It’s a technical victory for a very technical model.
The 6 years of available test data might have been a relatively harsh testing ground for the backcasting model. The model was trained over the Great Moderation, and was immediately asked to predict revisions during the Great Recession. If we exclude the extraordinary revisions between 2007 and 2009, the relative performance of the backcast is a little better: it has had an error of 0.21pp, compared to 0.28pp for the ONS’s first estimate.
So revisions might have a small predictable part in them. But as Chart 3 makes clear, the unpredictable part is dominant. The recent history of UK GDP will always be more uncertain than we’d like.
Chart 3 – Actual and predicted revisions to quarterly GDP growth
Do we have bigger revisions than other countries?
Another occasional claim is that the UK’s ONS revise GDP more than other countries. This does not appear to be true in the short term. The average absolute revision (a measure of the size of revisions) to GDP growth after 3 years is 0.25pp in the UK (1998 Q4 – 2010 Q4). This is very slightly lower than in the US (0.27) and Germany (0.29). The OECD reported similar statistics last year, and the ONS, Economist and the Bean review of UK economic statistics have all subsequently held this finding up as evidence that UK revisions are not unusually large.
However, the revisions to a UK GDP estimate over the next three years are larger than in the US or Germany: the average absolute revision between an estimate’s 3rd and 6th birthday is 0.28pp in the UK, 0.20 in the US, and 0.23 in Germany. So while other countries’ GDP estimates settle down after three years, the UK’s actually move about more!
Since revisions after three years are rarely due to new source data, it appears the UK’s estimates are more susceptible to retrospective changes in methodology. It isn’t clear why this should be the case. The drive to make statistics more comparable internationally means many countries have to implement similar changes. But each country will be starting from a slightly different base, and countries still have substantial discretion in exactly how they implement changes to very old data.
Which parts of early GDP estimates are most reliable?
Some parts of early GDP estimates are more ‘reliable’ because the relevant source data arrive sooner. When source data is missing for a component of GDP, the ONS must make an estimate using partial data or forecasts.
Chart 4 shows the signal to noise ratio for the quarterly growth rate of various demand and output components at the time of the Quarterly National Accounts (QNA), published three months after the quarter in question. (Technically this is the third estimate of GDP, but it is the first estimate of some of its components.) The higher a statistic is on this list, the less revised it is between the QNA and the estimate 3 years later, after controlling for the volatility of the series.
Chart 4 – Signal to noise ratio* of quarterly growth at the first QNA
*Signal to noise ratio = 1 – (Variance of revisions/Variance of data)
The components of output (in green) are generally more trustworthy than the components of demand (in blue). Early estimates of GDP growth are mainly based on output data precisely for this reason.
Of the demand components, the trade data are relatively reliable. This is somewhat surprising – imports and exports data are notoriously volatile. It turns out revisions are relatively small when set against this volatility, so early estimates contain a lot of ‘signal’. This is probably because all the data on trade in goods is available at the time of the QNA. Data on trade in services – which is more important for UK exports than imports – arrives a bit later.
Government consumption growth, on the other hand, has a negative signal-to-noise at QNA time. This means subsequent revisions have actually been larger than the eventual volatility in the series. In other words, early estimates are more noise than signal! Cunningham and Jeffrey documented this phenomenon nine years ago, and Chamberlain has also confirmed it remains true today. The obvious explanation is that local government accounts are not incorporated into the National Accounts for at least two years, meaning the early estimates are forecasts.
More reasons to be unsure…
Examining revisions is one way of measuring the uncertainty around GDP figures. But even if there were never any revisions, there would still be uncertainty. Aggregate statistics are almost always based on samples of people or firms, rather than the population. And for some hard-to-measure sectors of the economy – particularly the illegal parts – the ONS can do little better than make an educated guess at the level of activity.
Worse still, many people use GDP for purposes it is ill-suited for. The link between GDP and living standards, for example, is very uncertain, and likely to have changed over time.
The forecasts of economists come in for a lot of criticism. But when just trying to figure out the past is this hard, it’s no surprise we’re sometimes wrong about the future.
Lee Robinson works in the Bank’s Conjunctural Assessment and Projections Division
If you want to get in touch, please email us at firstname.lastname@example.org. You are also welcome to leave a comment below. Comments are moderated and will not appear until they have been approved.
Bank Underground is a blog for Bank of England staff to share views that challenge – or support – prevailing policy orthodoxies. The views expressed here are those of the authors, and are not necessarily those of the Bank of England, or its policy committees.