The U Word: What can text analytics tell us about how far uncertainty has risen during 2016?

Alastair Cunningham, David Bradnum and Alastair Firrell.

Uncertainty is a hot topic for economists at the moment.  Have business leaders become more uncertain as a result of the EU referendum?  If so, has that uncertainty had any effect on their plans?   The Bank’s analysts look at lots of measures of economic uncertainty, from complex financial market metrics to how often newspaper articles mention it.  But few of those measures are sourced directly from the trading businesses up and down the country whose investment and employment plans affect the UK economy.  This blog reports on recent efforts to draw out what the Bank’s wide network of business contacts are telling us about uncertainty – comparing what we’re hearing now to trends seen in recent years.

Business intelligence at the Bank of England

The Bank’s network of 12 regional Agencies does a unique job. Whilst there’s no shortage of business surveys in existence, offering a view on pretty much every aspect of economic activity, the information we get from businesses – known as our ‘contacts’ – has a different quality.  Rather than simply asking business leaders to tick a box to say how confident or otherwise they are, we delve deeper to really try to understand how their thinking is evolving and what’s affecting it.  Each month, our Agents visit more than 400 businesses to explore these issues.

After each meeting, a confidential record of the discussion is produced – a detailed, written description of all the main points raised that really tries to capture the nuances of the conversation. We also produce a range of scores – known as Company Visit Scores – that summarise a handful of key features such as the contact’s investment intentions or turnover.  (Historic scores are publicly available in anonymised form on the Bank of England website as part of the One Bank Research initiative.) Armed with these records, the Agents’ job is to look for common trends across the many businesses they talk with.  The scores have provided the basis for previous Bank Underground posts, but analysing the written reports brings a fresh perspective.

One major challenge is that the sheer volume of visit records makes it hard to compare what we’re hearing now with trends in the past – there’s too much to read!  Recently, we’ve experimented with making use of computational text analytics techniques to help us raise our game – applying an objective test of whether a topic is being cited more often today than was the case in the past.  To do this, we needed to extract a clean set of visit records from the myriad of documents created by the Bank’s Agents – writing an algorithm that strips out meeting agendas, confirmations, planning notes etc.  Having come up with an effective filter, we’re in a position to answer the following key questions:

• Are contacts mentioning “uncertainty” more frequently now than they did in the past?
• What words are most commonly associated with references to “uncertainty”?
• Do those contacts referring to “uncertainty” have weaker investment intentions than those that do not?

How frequently are contacts mentioning uncertainty?

Chart 1 plots a time-series of the proportion of paragraphs in our visit records that contained the words “uncertainty”, “uncertainties” or “uncertain”. (For brevity, we’ll simply refer to this trio as “uncertainty” from here on). References to uncertainty picked up sharply in the 3 months ahead of the referendum and continue to run at high levels.  The modest post-referendum fall contrasts with the picture of far sharper declines in uncertainty painted by some of the other metrics tracked by our colleagues – such as the number of references to uncertainty in broadsheet newspaper articles.

Chart 1

But while the results appear striking, there are caveats.  In particular, one factor driving the recent pick-up may simply be that the Bank’s Agents are asking their contacts about uncertainty more consistently than was the case previously – after all, uncertainty is a hot topic for economists at the moment.  Moreover, this simple metric cannot distinguish false positive conversations such as “no more uncertain than normal” from clear instances of increased uncertainty.  So Chart 1 is indicative, not definitive – we need to delve further into the text analytics to test our results.

What words are most commonly associated with references to “uncertainty”?

As a basic sanity check on Chart 1, we can look at which words appear most frequently in the same paragraph as “uncertainty”.  When we run the tests, we find that positive descriptors (such as “increased” and “more”) have outnumbered negatives (such as “no” and “reduced”) by 7:3 in visits made during 2016 Q2 and Q3.  That gives us some comfort that the measure in Chart 1 is not being completely dominated by false positives.

As well as offering a sanity check on our measure of uncertainty, looking at the words that appear commonly alongside references to “uncertainty” might give us insights into how contacts are reacting to the uncertainty they face.  Chart 2 plots a “word cloud” of the terms that appeared most frequently in the same paragraph as recent references to “uncertainty”.  The darker and larger the text, the more frequently the word in question has co-occurred with “uncertainty”.

Chart 2

Words appearing most commonly in the same paragraph as “uncertain”, “uncertainty”, or “uncertainties” – for visits undertaken during 2016 Q2 and Q3

Perhaps unsurprisingly, “Brexit” and “referendum” appear frequently.  There may be other economic uncertainties at present, but the ones being discussed most frequently at our visits concern the future of the EU.  The word “demand” appears more frequently than pricing or costs – despite sterling’s sharp depreciation since the vote.  And “investment” appears more frequently than “staff”, which might suggest that we look to capital spending for any signs of uncertainty having an impact on our contacts’ strategic positioning.

Once again, there are caveats.  Simply noting that those contacts talking about uncertainty tend to mention it alongside investment does not give a definitive sense of strategic impact.  We cannot tell from this analysis whether the reference is “no impact on investment” or “huge impact on investment”.  So again, the chart is indicative rather than definitive.

Do contacts referring to “uncertainty” have weaker investment intentions than those that do not?

Chart 3 gives one perspective on whether contacts’ talk of “uncertainty” is impacting on their investment plans.  It draws on the Company Visit Scores that Agents assign as part of their visit record to see whether scores for investment intentions have looked materially different for firms referencing uncertainty and those that do not.

Two features stand out: first that proportionally fewer firms mentioning uncertainty expect significant growth in capital spend (the highest scores) than is the case for the rest of the sample; second that other than that the two distributions look similar.  In other words, thus far the high levels of uncertainty identified in Chart 1 appear to have had only a marginal downside impact on contacts’ investment intentions.

Chart 3

Conclusion

During 2016 Q2 and Q3, the Bank’s business contacts have mentioned uncertainty more frequently than in previous years.  And strikingly, references to uncertainty did not fall back far through August and September – in contrast to some of the other uncertainty metrics we are tracking.  Digging deeper into the Agents’ visit records, we see more discussion of uncertainty in investment than in staffing and employment.  But even here, any impact has been relatively small-scale to date.

Of course, text analytics methods may be blunt tools that don’t pick up the context in which points were raised.  But when analysing agency intelligence, they give us an alternative – and in this case, corroborative – angle to the Agents’ regular published summary. So we plan to keep exploring what’s possible if we treat text as data, and are thinking about other extensions and opportunities. Suggestions are very welcome!

Alastair Cunningham works in Inflation Report and Agency Intelligence. David Bradnum and Alastair Firrell work in the Bank’s Advanced Analytics Division.

If you want to get in touch, please email us at bankunderground@bankofengland.co.uk. You are also welcome to leave a comment below. Comments are moderated and will not appear until they have been approved.

Comments will only appear once approved by a moderator, and are only published where a full name is supplied.

Bank Underground is a blog for Bank of England staff to share views that challenge – or support – prevailing policy orthodoxies. The views expressed here are those of the authors, and are not necessarily those of the Bank of England, or its policy committees.