Some thoughts on the EU referendum polls
Some thoughts on the EU referendum polls
Discounting yet another Daily Express voodoo poll, the story since David Cameron?s renegotiation deal was announced has been that a YouGov poll showed a sharp move towards leaving the EU after showing a near tie a few weeks ago.But actually we won?t be able to tell if there has been a move until we see what the phone polls are saying and can put together a more holistic picture.The issue with the EU referendum polls for the last few months has been that telephone polls (ComRes and Ipsos MORI) have shown a strong 15-20 point lead for ?Remain? while online polls (YouGov and ICM) have shown a much tighter race, often a dead heat. We haven?t published any polls on this but what work we have done (internally and for clients) has been fairly consistent with other online polls.
So which mode should we trust for the referendum?
It?s a bit strange because the results at last year?s general election found that telephone polls and online polls performed as well or badly as each other. Historically telephone polls performed better in general elections (2010 and 2005 being examples) but last year?s election result shows that this advantage has dissipated as the response rate to telephone polls has continued to diminish.With that in mind, here are two things that have informed how I view the referendum polls:
- t
- The main reason for the polling failure was that both telephone and online samples spoke to, among others, too many overenthusiastic young, liberal Labour leaning voters who all claimed to be 10/10 certain to vote. These were then treated as being representative of an age group which in reality has the lowest turnout of all.The people we surveyed probably did go out and vote exactly as they said they would but we didn?t talk to enough people who were more representative of these groups (politically unengaged and unlikely to vote) and as a result had too many Labour voters in our samples.
- The last major modal difference in polling was after April 2012 when UKIP started to consistently poll higher than the Lib Dems. This was mainly shown in online surveys with telephone polls showing UKIP rising but nowhere near as high as the online polls were showing. Because this was such a new and unexpected phenomenon, polling models didn?t know how to properly correct for it so there was an ongoing debate over how well UKIP were actually doing.
t
In 2012, 2013 and early 2014 the view at Opinium was effectively ?this is what the numbers are showing without any extra adjustments, let?s see how they do in an actual election before we make any judgements?. Then in the 2014 European Parliament elections, we overstated UKIP by about 5 points along with TNS, ComRes and Survation (post mortem here). We took this result into account for our general election methodology and ended up predicting UKIP?s share of the vote pretty accurately.However, the lesson I took from this was that the raw results from online panels were a bit too ?Ukippy? vs. the actual population and that this is related to the polling failure in the 2015 general election. We had two unrepresentative groups who were having too big an impact on the final VI numbers but only corrected for one of them before the election.I suspect something similar may be happening with the referendum polls. The group that is most pro-UKIP are those aged 65+. The age group that is the most difficult to reach online is aged 65+ and therefore we may be over-representing the views of the most politically engaged of the 65+ age group. The difference between the telephone and online results also seems to be coming from this 65+ age group. Where online surveys show them overwhelmingly in favour of leaving the EU, telephone polls show a much closer margin.It is possible that the same thing as happened in the election with young people is happening again with online polls over-representing the easiest over-65s to reach on panels who happen to be more pro-Brexit than the rest of their age group. The difficulty here is that this is happening after all of our corrections for UKIP have been taken into account and is an issue which, in theory, shouldn?t exist.The other thing that?s possible is that social desirability and interviewer effect are having an impact. The giveaway for social desirability is that respondents don?t want to seem uninformed and are reluctant to answer ?don?t know?. ?Don?t know? figures are typically about 10 points higher in online EU referendum polls than telephone so this may be a factor.I?ve been looking for evidence that social desirability bias increases with age and haven?t found any yet (though that is more down to me than anything else) but that?s a possible explanation as well, particularly if telephone polls are reaching more unengaged over 65s and if staying in the EU is the socially desirable position (something which would be enhanced by the respective leaders of each campaign).The other side of the social desirability argument is that if it is causing more people who don?t have an opinion to say that they would vote to stay in the EU then the leads for ?remain? in phone polls are phantom.
What is the upshot of all this?
Basically that I think the ?true? answer is somewhere between what the telephone and online polls are saying at the moment.That?s a bit of a cop-out I admit but I think both methods have different flaws and hopefully looking at both of them gives us a more accurate picture until polling companies find a way to correct those flaws.
