Wrong! The reporting of statistical error
The BBC reported last week that "the number of people out of work fell by 50,000 to 2.53 million in the three months to August”. That sounded like a small change in an estimate from a sample survey, so wondering about the statistical significance of the finding I looked it up on the Office for National Statistics' (ONS) website. Indeed, it says "there were 2.53 million unemployed people, down 50,000 from March to May 2012 and from a year earlier."
The headline did not contain any information about the sampling error associated with deriving this estimate from the Labour Force Survey (LFS), but in one of the underlying tables I found that the quarterly change was -50,000 +/-89,000. To my eyes that looks like a statistically insignificant change, from which I would infer that we can't be sure whether unemployment has gone up, down, or stayed the same. This wasn't the story reported by ONS however, and there was no nuance to the debate at Prime Minister's Questions where government and opposition alike quoted the 50,000 fall with absolute certainty.
Unemployment rate (aged 16+), seasonally adjusted. Source: Labour Force Survey - ONS.
Unemployment rates for the United Kingdom and the European Union, seasonally adjusted. Source: Labour Force Survey - ONS, Eurostat.
It turns out that the unemployment time series is a very noisy one, because the quarterly change is rarely larger than the sampling error. This important fact didn't become clear to me until I engaged in an email conversation with the ONS labour market team, so I'm sure it wasn't obvious to our politicians either. It would be useful for such an important consideration to be communicated in the headlines of such a report. It might make the headline more difficult to communicate but would caution users against becoming over-excited by small changes. It would also make an important contribution to raising the levels of statistical literacy in public debate.
It is a favourite pastime of statisticians and the wider 'numbers community' to complain about statistical gaffes by politicians and the media. The BBC radio programme 'More or Less' is full of entertaining examples. But producers of statistics have a crucial role in improving the understanding of statistics, a role which is sometimes neglected. For various reasons - wanting to provide clear answers, or believing statistical ideas to be too complex to communicate - basic concepts such as the existence, source and size of sampling error are sometimes glossed over. Statisticians should be confident of their ground and be willing to communicate these ideas.
So why are ONS reporting a fall in unemployment which is statistically insignificant? Well, it turns out that my initial inference - we don't know what has happened to unemployment - was wrong. Probably. Here's my stab at explaining the ONS approach. The point estimate for the change in unemployment is -50,000, with a 95% confidence interval of -139,000 to 39,000. Because more of the confidence interval is below zero than above, we can infer that unemployment is more likely to have fallen than risen (you can check the official version of this explanation by following this link http://www.statistics.gov.uk/hub/labour-market/people-in-work/employment/index.html - click 'Technical Data').
I'm not entirely convinced of this, which evidently reflects a gap in my understanding. In any case, the way that ONS are interpreting statistically insignificant changes in an important time series wasn't clear to me from their report. I reckon there are hundreds of users of LFS and other statistics wondering what to do with statistically insignificant changes in their time series. I've always ignored such changes, and maybe I've been wrong to do so. Either way, by communicating their methods ONS have an opportunity to improve statistical understanding and the interpretation of official statistics. An opportunity which I hope they will take advantage of.
Comment on this article
Skip to Main Site Navigation / Login
Comments
Statman
It's not wrong Matt. If employment is estimated to have increased by X, it is more likely that it increased than decreased irrespective of the value of X and irrespective of whether X is statistically significant or not.
reply to this comment
Matt Sperrin
Quote:
I do not say anywhere that ONS statisticians don't understand sampling variability. But the statement "It is more likely that employment increased, rather than decreased" IS 'plain wrong', and that is all I am referring to. ONS statisticians need to communicate basic statistical concepts effectively as well as understanding them!
reply to this comment
Anon
It may be worth noting that the series of interest is the seasonally adjusted (SA) series but the variance estimates are the variance estimates of the non-seasonally adjusted (NSA) series. Applying the NSA variance estimates to the SA series does not take account of several sources of error: forecast/backcast error, rotating group bias, autocorrelation of errors, etc. There are several methods to calculate the variance estimates of the seasonally adjusted series but these methods are not being used by the national statistical organisations.
reply to this comment
X Hays
Matt, I agree. Isn't there a similar misconception that a true value near the c.i. limit is less likely than a value near the center? This may often reflect wishful thinking, too.
reply to this comment
StatMan
I agree with Tony Manners. The sample size of the LFS needs to be improved, but that would cost money that ONS does not have. Alternatively LFS could be made compulsory but that would require legislation through parliament and ONS would have to drop questions about sexual identity and the silly well-being questions about how anxious you were yesterday. Questions like that must surely deter people from responsing to the survey. Matt Sperrin is plain wrong. The ONS may have its faults but their statisticians do understand basic statistical concepts like sampling variability. It's very silly to pretend otherwise.
reply to this comment
StatMan
Quote:
reply to this comment
Tony Manners
I'm not convinced either. The simplest way for ONS to provide greater precision would be to lower the confidence criterion - as is often said, there's nothing sacred about the 95% criterion. It might be educational for politicians to know that, say, they can be 40% confident that unemployment has fallen by x amount. (This is a hypothetical example.) There could be good or bad outcomes to such enlightenment. A good outcome, in my opinion, would be a funding increase to support the sample sizes needed for whatever higher level of confidence is required. But of course ONS would need to be clear about just how much bigger those samples would need to be (a lot, unless politicians are happy to have a lot of uncertainty in their statistics). A common defence is "these is the best estimate for the amount of money people are prepared to pay" - but I believe that defence is only defensible if you come clean about the quality of what you get for the money. I believe good quality statistics are essential for such matters as the labour market. They can't be funded on a "nice to have" basis.
reply to this comment
Matt Sperrin
The relevant sentences from the ONS are
reply to this comment