The trouble with issue and policy polls
There are three staples in the diet of the British poll watcher: voting intention, leader ratings, and policy/issue analysis. Spend a few weeks following the outputs of YouGov, Survation and the rest and you are bound to encounter all of these, among of course a smorgasbord of other questions on everything from the nation’s favourite chocolate bars to the perennial question of whether Die Hard is a Christmas movie.
When it comes to interpretation, voting intention and leadership ratings are pretty straightforward for the consumer – they basically do what they say on the tin. But issue and policy polls are a little more tricky. They are usually presented and discussed in media as though they have an obvious meaning: that, if a majority of the public supports a cause or a policy, a party or government ought to back it (or would benefit electorally from backing it). This is far from the case.
You might already be familiar with the problem of a party having policies that “poll well in isolation” while not adding up to an election-winning programme. Tackling the many interlocking challenges of how to build a deliverable policy programme that would make a real positive difference to people’s lives, is credible to voters, differentiates a party from its opponents, represents a set of values that target voters share, and reinforces a party’s brand strengths – in other words, that might actually win an election – would make for an impressive blog post. Sadly that’s not this one: for now I just want to focus on how you should interpret a poll that shows most people back a certain policy.
Agreement bias
Opinion polling has a systematic acquiescence bias, meaning that when asked whether they agree about something, people have a tendency to say “yes.” This is exacerbated by the use of positively-framed statements and the notorious “agree/disagree” scale (“please indicate whether you agree or disagree with each of the following statements”), which can be abused to give the impression that the public agrees with just about anything. Even if the questionnaire isn’t loaded to provoke agreement with a particular policy stance (as brilliantly satirised by Yes, Prime Minister in 1986 and demonstrated with real-life samples by Ipsos just last year), you can easily shift poll results to back a particular point of view simply by phrasing one side of an argument positively.
But it’s somewhat more than this. Policy questions are usually asked in the form of support/oppose, e.g. “would you support or oppose each of the following: a ban on playing music on the bus without headphones; having a conductor on every bus”, etc. Given that policies, whether devised by parties or advocated by campaign groups, tend to be positively framed interventions on things that at least some people care about, there is a bias to support them. Most policies sound good, and most organisations can come up with a range of plausible options to throw into a poll.
This means that absolute support for a particular policy means very little. Be instinctively distrustful of news stories that quote popular support for a single policy measure or political position. Look instead for some way of comparing between positions: out of a list of policies, which did best? What has increased over time? How does the other side’s policy do?
Missing factors: importance and trade-offs
The missing context for a lot of policy and issue polling is how important the issue is to voters, and what the trade-offs are. You may very well have a policy most people say they’d support, but if hardly anyone cares about the issue then backing it is unlikely to make a party or candidate more popular. Watch out for questions that ask “how important” a series of policies or positions are – you’ll notice that, like agree/disagree scales, there’s a fairly strong bias toward saying that anything sounding even vaguely urgent is at least “somewhat important”. Instead, look for measures that force the public to choose between things: “which is the most important to you out of the following?”
But even if lots of people really do care about it, a policy can easily come unstuck under scrutiny if pursuing it would result in some other effect that people don’t like. You very rarely see policies polled in such a way as to take account of the cost (financial or opportunity) of doing them, or any other negative consequences they might have.
To give a very simple example: you might find majority support for ensuring every rural community has a bus service to the nearest town, providing residents access to jobs, entertainment and public services (by the way, another thing to watch out for: when the question wording includes the policy plus all of its supposed benefits). Adopting this policy will almost certainly make no positive difference to a party, because people are actually looking for a government that will help with, say, the cost of living, healthcare and crime. A party adopting the rural bus policy might in fact run into trouble when it gets asked how the extra services, many of which would be uneconomical to run, would be funded: there’s a tax and spending trade-off. Again, polling should be looking to show that the public prefer what you are proposing to the alternative, taking account of the benefits and costs both of doing and not doing it.
Attention
Another less-noticed trade-off (perhaps more relevant to campaign pollsters than to those more focused on policy) is communications bandwidth. However skilful your media and comms team is, there is a limit to how much information a party or candidate can get across to the public. If you want a party to adopt and advocate for your favoured policy proposal, you have to show that it is more worthy of occupying a 10-second broadcast clip than any of the dozens of other things a spokesperson might talk about.
All of this is why, in polling designed to inform what politicians should actually do and say, there will be questions that force respondents to prioritise: for example, to pick the 2 or 3 best or most important policies from a list; or to evaluate two opposing statements and say which they agree with more (even if neither is perfect).
Impact
As noted at the top, the obvious interpretation of an issue or policy poll is that if a majority supports something, a party would benefit electorally from adopting it. Hopefully the above shows why this doesn’t really work. This doesn’t mean that polling is unable to show whether adopting a particular policy stance will help a party, it just requires a lot more effort than a one-line support/oppose question.