The Times shows how not to report an opinion poll

Welcome to the 83rd edition of The Week in Polls, which, once again, takes a look at a poll reported by The Times and finds the full story is rather different from the newspaper’s story.

Then it’s a look at the latest voting intention polls followed by, for paid-for subscribers, 10 insights from the last week’s polling and analysis. (If you’re a free subscriber, sign up for a free trial here to see what you’re missing.)

As ever, if you have any feedback or questions prompted by what follows, or spotted some other recent polling you’d like to see covered, just hit reply. I personally read every response.

---

Been forwarded this email by someone else? Sign up to get your own copy here.

Want to know more about political polling? Get my book Polling UnPacked: the history, uses and abuses of political opinion polling.

Or for more news about the Lib Dems specifically, get the free monthly Liberal Democrat Newswire.

---

The Times shows how not to report an opinion poll

The top story on the front page of The Times after the Autumn Statement was clear and unequivocal:

Front page via the ever helpful @hendopolis.

As was the slightly differently worded website story from the night before:

But were they true?

Well…

In an important, narrow, technical sense - yes.

The poll was from a reputable pollster, YouGov, ¹ carried out after the Autumn Statement and with reasonably worded questions. The four point rise was also - correctly - a like-for-like comparison with the previous similarly conduct poll from YouGov, which was pre-Autumn Statement.

But in a broader sense - no.

Because any story based one opinion poll is, in the journalist lingo, a single-sourced story. There’s a reason why journalists and editors with good standards fret over single-sourced stories (and often decline to run them). It’s because a single-sourced story is also a story that’s at risk of telling people what the source wants the world to think rather than telling people what is reality. Multiple sources guard against this, though of course sometimes the source is so apposite, so relevant or so trusted that a single-sourced story runs, scoops prizes and changes the world.

Which is why any story based on a single poll,² needs to be treated with particular caution. Apply caution and there are two problems with The Times story.

One is that the four point change in Conservative support is not large enough on its own to give confidence that the change was substantive rather than just noise, caused by the random fluctuations in what polls find even if underlying support has not changed.

Look at the level of Conservative support in all the YouGov polls since the start of October: 24%, 24%, 25%, 24%, 23%, 23%, 21%, 25%. That final, post-Autumn Statement, 25% looks pretty much in line with the rest of the previous two months. The four point rise comes not from 25% being a high number but its predecessor, 21%, being a (slightly) low number in the sequence.

Even if you can simply assume that a change in the polls has been driven by one specific political event (an assumption made far more often than is warranted) that isn’t a run of figures which suggests with confidence that something has happened beyond reversion to the mean after the 21%.

If you really want to find a number that’s out of the ordinary in that run and given it a single causal explanation, the King’s Speech and the 21% is the better combination.³

The second problem is that we now have a clutch of other sources - polls - against which to judge The Times’s front page single-sourced top story.

Here are what the three other pollsters who have carried out voting intention polls with fieldwork since the Autumn Statement have found:

• Techne: 21% (-1)

• Opinium: 26% (-1)

• WeThink: 26% (+1)

Those don’t back up The Times’s story.

I do have some sympathy with how the newspaper wrote-up its poll, however. Because the weakness at the heart of modern British political polling is how dependent it is on media outlets paying for polls. It’s very understandable that if you pay for a poll you then want to make a story out of it.

But - as indeed The Times has often done on other occasions⁴ - you can square the circle of having to make a story out a single poll that is both headline-worthy and cognisant of the problems of single-sourced stories.

One very common way of doing this is to find polling results that still tell an interesting story, even if the truth is a few points higher or lower than the figures in your one poll. If a poll finds that 73% of people think the Prime Minister does not tie their shoelaces properly⁵, then it doesn’t really matter if the true figure is 65% or 81%. It’s a big number either way.

Indeed, the full tables of the YouGov poll provide plenty of interesting detail of this sort, some of which made it into The Times’s reporting in a lower profile way.

One difference, though - which is that the detail shows in a variety of questions the public give the government a firm thumbs down on the economy. Which shows why, even if you take the four point rise at face value, it was only a rise to 25%. Still worse than the Duke of Wellington in 1832.

---

Know other people interested in political polling?

Refer friends to sign-up to The Week in Polls too and you can get up to 6 months of free subscription to the paid-for version of this newsletter.

---

National voting intention polls

Yes, once again it was a week without a poll putting the Conservatives on more than 30%, extending the run stretching back to late June (when a Savanta poll gave them 31%). Although Opinium continues to show a lower Labour lead, note that it’s now back to being more in line with other pollsters.

Here then are the latest figures from each currently active pollster:

For more details and updates through the week, see my daily updated table here and for all the historic figures, including Parliamentary by-election polls, see PollBase.

Last week’s edition

What does the polling say about David Cameron?

---

My privacy policy and related legal information is available here. Links to purchase books online are usually affiliate links which pay a commission for each sale. Please note that if you are subscribed to other email lists of mine, unsubscribing from this list will not automatically remove you from the other lists. If you wish to be removed from all lists, simply hit reply and let me know.

---

What the public made of the Autumn Statement, and other polling news

The following 10 findings from the most recent polls and analysis are for paying subscribers only, but you can sign up for a free trial to read them straight away.