After Trump and Brexit disasters, pollsters are now getting Scottish independence wrong too

Thursday, December 1, 2016

The most important facts about opinion polling on Scottish independence are that most of it is carried out by polling companies based south of the border, and that the vast majority of it is commissioned by anti-independence clients in the mainstream media. 

For example, this week’s widely-reported St Andrew’s Day poll was commissioned by The Times, and was conducted by London-based YouGov. So why does that matter? In theory, perhaps it shouldn’t. But pollsters are human beings, not robots, and the way they conduct their work is bound to be influenced, either consciously or subconsciously, by their own assumptions and prejudices. 

The former president of YouGov, Peter Kellner, went on the record a number of times during the long independence referendum campaign with his belief that the Yes side could not even conceivably win, and bizarrely claimed at one point that Alex Salmond knew this and was just using the referendum as a cunning tactic to achieve more devolution. Underlying all of these assertions, of course, was the London-centric notion that Scottish independence was a romantic fantasy and an obvious non-starter, and that anyone with a brain who considered the point for five seconds would realise that.

When that’s the distorted lens a polling company is viewing a referendum campaign through, the suspicion is bound to arise that the polling numbers reported in the media are somehow being affected. YouGov were, until a very late stage in the campaign, producing much bigger No leads than any other online firm. 

It may be – I only say ‘may’ – that because they found their results intuitively plausible, they weren’t questioning their own methods to a sufficient extent.

In particular, there were three elements of the YouGov methodology that very obviously ran the risk of artificially inflating the No lead. All of them were eventually rectified, but it took an incomprehensibly long time for that to happen.

The first problem was that YouGov didn’t weight their results by country of birth, even though their own datasets revealed they had far too many English-born respondents in their samples.Country of birth is one of the strongest predictors of a person’s views on independence (one academic study concluded that a narrow majority of the Scottish-born population actually voted Yes in September 2014). So failing to upweight Scottish-born respondents to the correct target figure almost inevitably led to a reported No vote that was slightly too high.

The second problem was that YouGov didn’t interview 16 and 17-year-olds, even though the voting age for the referendum had been reduced to 16. It’s less clear what effect that was having, but it’s at least possible that it was contributing to an underestimation of Yes support. 

Incredibly, YouGov reverted to that bad practice after the referendum was over, and it took them until this week’s poll to finally get their act together again. It’s hard to think of a more open-and-shut case of London-centricity than that – it’s plainly inconceivable that the wrong electorate would ever be interviewed for a Britain-wide poll.

And the third problem was the question that YouGov asked, which in a none-too-subtle way attempted to reframe the issue in hand as being primarily about Scotland leaving the United Kingdom, rather than about Scotland becoming an independent country. This appeared to reflect Peter Kellner’s belief that Scots didn’t understand what independence meant unless it was spelt out for them in a pejorative way. 

Essentially YouGov were attempting to ‘improve’ on the actual referendum question, and in doing so they lost sight of the danger that polling the wrong question will produce the wrong numbers. When they finally switched to a more neutral wording, the No lead dropped sharply – that may have just been a coincidence, but it seems unlikely.

To be fair, the situation is now a lot better than it was. Having been through the bruising experience of a referendum campaign that produced a Yes bandwagon that the likes of Peter Kellner had insisted was impossible, London-based pollsters seem to have quietly confronted their own prejudices and are less likely to make the same mistakes again. However, it’s important to remain vigilant and not assume that the monster of methodological blunders has been permanently vanquished. 

A recent BMG poll marked a dreadful regression – it was widely misrepresented, not least by the firm themselves, as being a poll on ‘independence’, even though it hadn’t actually asked about independence directly. Hopefully that will prove to be a one-off, rather than the start of a new dark age of polling.

And what of the other side of the coin – the fact that most independence polls are commissioned by anti-independence clients? Take a closer look at the new YouGov poll and see if the totality of the information it presents actually tallies with the impression the mainstream media have helped you form about it. 

You might be surprised to learn, for example, that the poll shows a 2-1 margin in favour of Britain remaining in the European Union (higher even than in the June referendum), and a narrow majority in favour of Scotland remaining in the EU when the rest of Britain leaves. If the poll had been commissioned by a pro-independence client, it’s likely you would have heard a lot more about how it uncovers a vitally important contradiction in public opinion. 

Although a majority are opposed to independence at the moment, it is accepted by an almost 3-1 margin that remaining in the EU after Britain leaves will not be possible without independence. How that contradiction resolves itself is obviously the key to understanding the next independence referendum.

A pro-independence client (or even a neutral client) would likely have placed much less weight on the apparent small drop in headline support for independence, which could very easily prove to be an illusory blip caused by normal sampling variation. The standard margin of error in each individual poll is 3%, after all. And it’s unlikely that YouGov would even have been invited to pose their very narrow question about whether there should be another independence referendum before Britain leaves the EU

Instead, respondents would probably have been asked whether they support a second referendum in principle, and then asked for their views on the timescale. Previous polls using that format have produced results that are far more favourable for Nicola Sturgeon.

Alas, we live in an unfair world, and it’s unlikely that we’ll see more balance in the commissioning and reporting of independence polls any time soon.  That being the case, it’s perhaps a fortuitous coincidence that recent events have so severely undermined the public’s faith in the entire polling industry.

James Kelly's blog, Scot goes POP!, is among the most popular political blogs in the UK. He has also contributed to a number of newspapers and magazines.

You can check out the blog here or follow James on Twitter.

James has also written for us about the madness of King Trump, the crisis created by the Brexit legal challenge and why Scottish Labour care more about Corbyn than their country.