The problem with opinion poll averages

Welcome to the 101st edition of The Week in Polls, which ponders how wise it is to stare at polling averages.

Then it’s a look at the latest voting intention polls followed by, for paid-for subscribers, 10 insights from the last week’s polling and analysis. (If you’re a free subscriber, sign up for a free trial here to see what you’re missing.)

But first a quick comment about the latest Survation/Best for Britain MRP that is out this weekend. It’s grabbed the headlines but it also doesn’t tell us anything new, as the overall ‘doom for the Conservatives’ reflects the picture from traditional voting intention polls too. The seat specific details in this MRP reflects the fact that, unlike some other MRPs such as YouGov’s, it suggests relatively muted local variation and tactical voting and hence, for example, Labour winning seats from third where they have no councillors. The real question to ponder is whether that’s revelatory insight, or implausible modelling,¹ and on that note a reminder of my guide to understanding MRPs.

As ever, if you have any feedback or questions prompted by what follows, or spotted other recent polling you’d like to see covered, just hit reply. I personally read every response.

---

Been forwarded this email by someone else? Sign up to get your own copy here.

Already a reader and know others who might enjoy this newsletter? Refer a friend and you can get up to 6 months of free subscription to the paid-for version.

Want to know more about political polling? Get my book Polling UnPacked: the history, uses and abuses of political opinion polling.

---

Are you paying too much attention to polling averages?

I like averages.

I use them, I calculate them and I can bore you with the pros and cons of mean versus median versus mode.

Averages sound like they should be useful for understanding opinion polls. After all, Understanding Opinion Polls 101 is ‘don’t be dazzled by just one poll, look at the trends across multiple polls instead’. That sounds like a job averages can help with.

Averages also sound useful if you understand a bit about the random vagaries of opinion poll sampling, about how a poll can be unlucky in having a sample skewed a bit one way and another poll could have a sample tilted the other way. Even though very few polls these days rely on genuinely random sampling (with quotas and weighting instead playing a bigger role), averaging out different polls sounds a good way to filter out such variations and see the underlying reality. 

Plus of course averages of the latest opinion polls are a staple of coverage of polling. Whether it’s the mainstream media or high profile social media accounts dedicated to polling, it’s common to see averages.

And yet…

You may have noticed that in both my weekly newsletter, The Week in Polls, and in my regularly updated web page of latest polls, I avoid giving averages.

What’s more, although there are monthly averages in my PollBase database of voting intention polls all the way back to 1943, those averages are almost provocatively crude. A simple averaging of all the polls in each month whose fieldwork dates commenced in that month. No adjustments for even basics such as how many polls came out from different companies. Those averages are unashamedly rudimentary.

That’s because the wise take on polling averages is to treat them with a deep ambivalence.

Averaging is sometimes a good way to head towards the truth. Yet on other occasions, it obscures the truth and is purpose-designed to give the wrong answer.

Imagine you are the first time visitor to a building you’ve not been to before and you’re sat in reception. The person on reception is busy at the moment and you need to head to the toilet. You ask a couple of other people sat there if they know where the toilets are. One says, “I think they’re down that corridor and then through the door on the left”. The other says, “Yeah, down that corridor but actually, it’s the door on the right”.

A true believer in averages would average out one left recommendation, one right recommendation and then stride confidently down the corridor and through the door in the middle instead. Neither left nor right but average. And almost certainly wrong.

Sometimes you have to choose, not average.

So it can be with polling.

Take the current mix of pollsters and methodologies that we have. Opinium’s way of treating people who say ‘don’t know’ consistently gives lower Labour leads than other pollsters report.

Now, you might think Opinium is right, or that they’re wrong.² If they’re right, then averaging their figures in with those from other wrong pollsters just drags you away from the truth. And if they’re wrong, polluting the average of those other pollsters with Opinium’s figures makes the average worse.

The point is you have to choose. Averaging just guarantees being wrong.³

So should you swear off all averaging?

Not quite. There is certainly value in very basic averaging that gives a brief overall view of what the polls say as that is also therefore giving you some useful indications about what in general the media or politicians are thinking they are being told by the polls. Averaging does also guard against getting overly excited or depressed by any single poll. (Though I doubt averages are enough to save the Daily Express from thinking it’s appropriate to call a one point rise in support for their favoured party a “surge”.)

Averaging can give you a simple broad brush picture, which is why I take that deliberately simplistic approach in PollBase.

Which is why a table of the latest polls from each pollster plus some super simple averages (or better yet, a graph) is generally my preferred analytical view of the data. Averages obscure, and tables of data reveal, the consistent differences between pollsters which indicate the issues to think about (such as what’s up with the don’t knows at the moment).

Averages are like water wings, useful for getting for incompetence to basic competence in understanding the polls, but then rather annoying encumbrances which get in the way when you’re moving from basic competence to expertise.

But before my colleagues Jack and Luke get too cross with me, I should add that there’s also a role for super-smart averaging, of the sort they used to produce PollBase Pro.

This sort of analysis brings many advantages, such as making use of the points of truth - general election results - and factoring in expected levels of random noise. It does though still have as a weak spot about that Opinium choice as it ducks the choice. If Opinium is the outlier and is also right - as some previous outliers have been in other Parliaments - then even PollBase Pro’s cleverness will not be clever enough.

Which is why the best solution is keep an eye on that table of latest polls, keep an eye on graphs for trends and, yes, do also keep an eye on averages as clever as PollBase Pro.

---

National voting intention polls

As I wrote last week, spare a thought for Iain Duncan Smith. He was ousted for being a disastrously unsuccessful party leader at a point when the Conservatives were on 33%-35% in the polls, with a Labour lead of 3%-5%.⁴ By contrast, here are the latest figures from each currently active pollster:

For more details and updates through the week, see my daily updated table here and for all the historic figures, including Parliamentary by-election polls, see PollBase.

---

Last week’s edition

Rishi Sunak vs Liz Truss: how do the polls compare?

---

My privacy policy and related legal information is available here. Links to purchase books online are usually affiliate links which pay a commission for each sale. Please note that if you are subscribed to other email lists of mine, unsubscribing from this list will not automatically remove you from the other lists. If you wish to be removed from all lists, simply hit reply and let me know.

---

The relative popularity of National Insurance, and other polling news

The following 10 findings from the most recent polls and analysis are for paying subscribers only, but you can sign up for a free trial to read them straight away.