Can central bankers become Superforecasters?

Aakash Mankodi and Tim Pike

Tetlock and Gardner’s acclaimed work on Superforecasting provides a compelling case for seeing forecasting as a skill that can be improved, and one that is related to the behavioural traits of the forecaster. These so-called Superforecasters have in recent years been pitted against experts ranging from U.S intelligence analysts to participants in the World Economic Forum, and have performed on par or better by accurately predicting the outcomes of a broad range of questions. Sounds like music to a central banker’s ears? In this post, we examine the traits of these individuals, compare them with economic forecasting and draw some related lessons. We conclude that considering the principles and applications of Superforecasting can enhance the work of central bank forecasting.

Setting the scene

It is helpful to begin by considering the purpose of forecasting in central banks, and how the process works in practice.  This speech by Gertjan Vlieghe explains how forecasting is an important tool that helps policymakers diagnose the state and outlook for the economy, and in turn assess – and communicate – the implications for current and future policy. So achieving accuracy is not always the sole aim of the forecast. However, forecasts are also a means to provide public accountability of central bank actions, and the presence of persistent or significant forecast errors may damage the credibility of the policy making institution amongst key stakeholders (individuals, governments and financial and capital markets).

A typical forecast set-up at a central bank is (see here) supported by two pillars: i) statistical frameworks underpinned by specific (for example, New Keynesian General Equilibrium) economic concepts, which can be supported by tools that process a range of economic and financial data, and ii) monetary policymakers’ judgements and deliberations that overlay these strict model-based forecasts – all of which form part of the deliberation process.

The accuracy of such forecasts has come under much scrutiny (see here) since the financial crisis, resulting in a great deal of effort to improve their performance. Several reviews and studies (see Stockton (2012), BoE IEO (2015), FRBNY Staff Report (2014) and ECB WP 1635 (2014)) have evaluated forecast performance across many major central banks and suggested improvements in calibrating economic models (e.g. to reduce bias), challenging prior conventions more and learning more from other central banks/economic forecasters.  The BoE’s MPC for example has also started commenting on its own ‘key judgements’ in its quarterly Inflation Report.

This is all welcome progress. But this iterative process from inside the central banking community over time leaves us with an impression that improving forecast performance could benefit from further considering the successes of forecasting  in other fields (similar to taking an “outside view” when forecasting as described by Kahneman). We may then move forward from this process of gradual evolution… to a potential revolution.

Enter Superforecasting

Superforecasters have been described as “unusually thoughtful humans on a wide spectrum of problems”. They are drawn from necessarily diverse backgrounds, and include amateurs and experts in a given field. They compete in tournaments which test their judgements on a range of questions about economic or geopolitical events. And through making these predictions they are expected to hone a range of forecasting skills. They are judged on several measures (including a daily average ‘Brier score’ – a measure of forecast accuracy originally proposed to test weather forecasting), and they receive their title by consistently outperforming a top percentile of their peers.

Superforecasters were first identified on the back of The Good Judgement Project (now a private enterprise), which was part of a US Intelligence Agency program in 2011. The GJP’s testing team included renowned advisors from psychology, statistics and economics. Their work used personality trait tests and training methods to reduce cognitive biases and improve the forecasting abilities of their volunteer forecasters. They then identified individuals who consistently out-performed their peers. Subsequent studies of this experiment found that when these top forecasters were placed in teams with other such forecasters  (described here as ‘group of average citizens doing Google searches in their suburban town homes’), they performed around 30% better than the average for intelligence community analysts who had access to confidential intercepts and other relevant data. Pretty Super-ising results one might say!

Can central banks become this ‘super’?

The story so far could imply that the answer simply lies in replacing central bank forecasters with these Superforecasters and leaving them to it. However, central bank forecasting is as much about forming a coherent economic narrative (the preserve of economists) as it is about numerical accuracy (for which the traits that make these individuals outperform the ‘experts’ matter). Central bankers may have a comparative advantage in the former, but their forecasting can be enhanced by considering key behavioural traits of those responsible for forecasting.

So how do central bankers fare against these Superforecasters?

The similarities: Superforecasters (most importantly) have a ‘growth mind-set’, which is a real willingness to address why a forecast is different from its eventual outcome, rather than just an ex-post evaluation of whether the prediction was correct. They also demonstrate a good balance of data and judgements when forming conclusions, not placing undue weight on either one.

Central bankers in comparison likely fare favourably against these traits, given most major central banks provide detailed updated assessments (the ‘why’) accompanying changes to their forecasts on a regular (usually quarterly) basis. At the BoE, these follow a substantial consultation process between staff and policymakers.

Some differences:

Using the wisdom of crowds: Another key trait of Superforecasters is that their forecasting abilities are enhanced when working in diverse teams – with people drawn from a range of disciplines, levels and areas of expertise. This enables them to tap into the well-known concept of the wisdom of crowds, and the process reportedly leads to better forecasts by providing a more stimulating environment for debate. Results are further aggregated to give more weight to forecasters who have a better track-record.

The central bank forecasting process does incorporate some elements of this – for example, many central bank policymakers make decisions in committees, after debate and exchanging views.  Moreover, several central banks regularly use surveys of external economic forecasters as an input to the forecasting process, or draw on external views, e.g. the use of the Agency or Market Intelligence networks to gather views in the BoE.

But (we would assert that) the forecasting outputs do not benefit in the same way as the Superforecasting process, where particular behavioural traits, a mix of expert/non-expert opinions or previous track records of those forecasters are considered.  Engaging a wider cohort of participants in forecasting could address this.  One suggestion would be to create Citizen Economists – as suggested by Andy Haldane in this speech, who argued that the wisdom of crowds can be harnessed by regularly canvassing the views of the public on the economy. Central banks could consider creating an online platform that engages the public directly with forecasting – which might also improve public understanding of policy and the economy (the RSA’s Citizen Economic Council and the Bank’s recently announced Citizen Panels for example intend to achieve a similar purpose). Central bankers can use these as an input into their own forecast process (perhaps even publishing the alternative crowd sourced forecast, similar to the way that the Fed publishes a staff forecast alongside the FOMC’s official one), though policymakers would remain accountable for their own forecasts and any policy decisions based on them.

Competing forecasts: A further avenue to explore is whether central banks might engage external Superforecasters (who aren’t constrained by the same institutional challenges as central banks) to produce their own macro-economic forecasts. Superforecasters currently partner with organisations (e.g. humanitarian or policy-making) around the world on topics ranging from geopolitics, future currency movements, and economics. In a similar vein, central banks could use Superforecasters’ macro-economic forecasts alongside their surveys of external economic forecasters as an additional input to the forecasting process.

Central Bank Superforecasters: Central banks could also try to identify and train their own internal economic Superforecasters, by employing the same techniques of cognitive training, team-work and result aggregation as another additional input to the forecasting process. One part of such a programme could be the continual assessment of forecasting performance, as measured by Brier scores.

Conclusions

With continuous forecasting challenges on the horizon in coming years, perhaps it is an opportune time to incorporate these ideas in the central banking sphere. Economic forecasting will always be an imperfect science. So while it is unlikely that a major shock such as the global financial crisis would have been averted by improving the accuracy of forecasting efforts in these ways, we believe the lessons learnt through the experiment of Superforecasting have a lot to offer to take forecasting a step forward in that direction.  Potentially over time, we might be able to create a next generation of central bank Superforecasters.

Aakash Mankodi works in the Bank’s Market Intelligence and Analysis Division and Tim Pike works in the Bank’s Agencies Division.

If you want to get in touch, please email us at bankunderground@bankofengland.co.uk or leave a comment below.

Comments will only appear once approved by a moderator, and are only published where a full name is supplied.

Bank Underground is a blog for Bank of England staff to share views that challenge – or support – prevailing policy orthodoxies. The views expressed here are those of the authors, and are not necessarily those of the Bank of England, or its policy committees.

3 Comments

Filed under Macroeconomics, Monetary Policy, New Methodologies

3 responses to “Can central bankers become Superforecasters?

  1. Louis Lavery

    Things maybe fine if you’re forecasts are correct but if not, then what? I doubt there are any super forecasters that always get it right.

    Perhaps we ought have more input from engineers. The way they deal with complex systems, systems far less complex than an economy, is to build in redundancy so as the whole system doesn’t come crashing down just because one link in the chain breaks. That tends to result in, what appears to be, a less efficient system but as only parts of the system crash they recover more quickly as they are aided by the parts than don’t crash. Trouble is, economists don’t seem inclined to recommend (what appears to be) inefficient systems, probably because if they did they’d not be in a job for long.

  2. But what if the superforecasters are using past BoE etc forecasts to make theirs? Plus the amount of studying that has to be undertaken just to create a decent univariate series forecast is not for the faint hearted, so its hard to see how just guess work could be superior to rigorous techniques.

  3. Laurence Russell Sanders

    A very interesting paper, that focuses on key challenges in economic forecasting. I concur with one conclusion – that “economic forecasting will always be an imperfect science.” A second conclusion, whilst apparently compatible with evidence from the 2007/ 08 financial crisis, is more contentious – “it is unlikely that a major shock such as the global financial crisis could have been avoided”. The origins of the 2007/ 08 financial crisis lay in the US non-prime market, whose problems were well known in 2006. The crisis was aggravated by a wide range of factors, including the excessive level of UK bank lending. As a member of a bank Risk Committee in spring 2007, I recommended a strengthening of credit criteria in the UK mortgage market, a recommendation that was endorsed by senior management.

    The lessons to be learned from the Great Financial Crisis are numerous. They include (1) awareness of economic and financial events in the global economy (2) a rigorous, albeit pragmatic, prudential regime (3) weaknesses in widely used econometric models, such as the New Keynesian dynamic stochastic equilibrium model – weaknesses referred to in Oxford University economic research papers (e.g. Hendry and Muellbauer, Sept 2007 – followed by further research papers, (4) problems in economic management due to the long standing imbalance between UK housing demand and supply, initially quantified in the Barker Report 2004 – and a key factor in the Oxford University critique of UK econometric models in relation to the UK housing market (5) greater use of scientific and technical analysis to supplement mainstream economic analysis (6) given that it is impossible to accurately forecast the economic future, greater use of scenario analysis, an area of Bank of England strength, as evidenced by the long standing use of scenario analysis fan charts. (7) a wholly objective analysis of the UK economy. With respect, too many practising economists allow their personal political views to influence their forecasts. In addition, too many economists produce forecasts that are compatible with their organisation’s sales and marketing strategy. Based on personal experience, all financial organisations should have a powerful Risk Committee, whose composition should include key economic and treasury professionals, with a strong track record

    The paper highlights the need for closer links between academic, public sector and business economists, plus the need for greater use of scientific principals in economic forecasting. One area of research that is essential to economic forecasting is the impact of the increasingly rapid rate of technical change, on both the path of the economy and on prudential risks to financial stability. Whilst I welcome the focus on increasingly rigorous economic forecasting – and a new generation of superforecasters, I conclude on a note of caution, based on experience in transport consultancy, the engineering industry and the financial sector. As the economic profession moves to increasingly sophisticated forecasting models, we must not forget core economic principals, including the importance to the UK of the global economy.

    Laurence Russell Sanders

    Member, Association of Corporate Treasurers; Member, Society of Professional Economists

    ——————–