BY NIGEL BEAN & PAUL READ

‘Politicians are only interested in our research when it supports what they have already made up their minds to do….’

Professor Ted Friend

Last year we contacted the British Polling Council after the National Trust vote on whether to ban fox hunting on their land. It was fairly obvious at the time that there was a major failure in polling results with Ipsos Mori claiming 84% of the country wanted a ban on hunting, when compared to the National Trust vote, where just 60,000 could be bothered to cast a vote out of a possible 5 million members. Given that the vote received nationwide coverage in the media, a showing of less than 1% put a mockery on the Ipsos Mori poll result. (And let’s not forget a very important point here, The League Against Cruel Sports have just 3,000 members).

We took this up with the British Polling Council as quite frankly the Ipsos Mori poll was a joke. The BPC were very open and honest being clear that it’s not their job to decide what is or isn’t a bad or skewed poll – their job is purely to make sure the information provided by the polling company on a particular poll to the public is correct and for the public then to make a decision on its reliability. For instance, as long as the polling companies clearly tell the public, they will fit polls to suit finances and objectives – no rules are being broken.

This was from the Ipsos Mori website at the time of writing:

“Our aim when designing studies and responding to briefs is to put forward the approach that is most fit for purpose, taking account of client’s objectives and budget.”

With their poll, Ipsos were being honest and fulfilling the guidelines laid down by the British Polling Council, no matter how loose the wording. What they are saying is they will create a poll to fulfil the client’s objective within budget. So in the case of the League Against Cruel Sports their objective was to show the largest possible majority in favour of a hunting ban. In order to show this large majority they will be advised by a polling company, in this case Ipsos Mori, to use the quota sampling poll technique as opposed to the purest form of random probability sampling polling.

So why use quota sampling polling instead of the far more accurate random probability sampling poll?

A random probability sampling poll is far more accurate (and costly) and will inform the nation the 1,600 respondents required to represent the U.K. population as a whole was only reached after they had asked 30,000 people. This means 28,400 were not interested enough to take part. (The UK government use random probability sampling polls for accuracy).

The quota sampling polling technique used by the League Against Cruel Sports does not report that figure of 30,000. Quite simply put, they don’t want people knowing the vast majority are simply not interested enough to give time to answering questions on hunting. They want you to believe 1,600 were asked and willingly gave an answer.

Another benefit to the League Against Cruel Sports of using the quota sampling polling technique is it wrongfully assumes everyone in the country must have an opinion on the subject being polled. Up to 30,000 can be asked by a polling company to respond to their questionnaire – of that only 1,600 will agree, thus the 28,400 not willing to take part through lack of interest will now be lumped in with the rest of the country and the 1,600 will be giving an opinion on their behalf.

“Disadvantages of quota sampling. In quota sampling, the sample has not been chosen using random selection, which makes it impossible to determine the possible sampling error. … It also means that it is not possible to make statistical inferences from the sample to the population.”

Now we can see how the National Trust vote materialised from just 60,000 voting out of a possible 5 million. We can also see how the ORB survey conducted during the election campaign found (0.39) of the respondents would list hunting in the top three to influence their vote.

‘In total, 2,038 were interviewed and each one was asked to spontaneously raise three issues that would influence their vote. Only eight mentioned hunting (0.39 per cent). Moreover, of these eight people, seven said they were unlikely, or would never, vote Conservative.’

The League Against Cruel Sports also want to show the majority in rural areas are in support of a ban on hunting and this can be included in the quota sampling poll. To obtain their majority in the countryside, Ipsos/Mori offered a choice of four ways to define the size of a rural/urban area. These ways are known as definitions. Which definition you pick will determine how many towns will be included as rural, so now you can live on the outskirts of town next to a busy A-road and overlook an industrial estate and be classed as rural.

Naturally, as the ONS definition includes a lot of town areas and this is the rural definition the League Against Cruel Sports used. Again, Ipsos Mori cover themselves in their rural/urban guide – they do tell you to pick the definition that is most likely to give you the result you want.

“It is vital that if you wish to include a rurality analysis in your tables or a rurality definition into your sample design, you should understand what you are trying to achieve, and ensure that you use an appropriate definition.”

As LACS well know, the people most likely to want to take part in their surveys are the most politically active and opinionated (their own green inker members). On top of that a push-pull polling technique is used – questions involving irrelevant cruelty comparisons specifically designed to elicit a response. LACS ask if dog fighting should be made legal again, before asking if fox hunting should be made legal. So we end up with a well-known bias polling technique used in conjunction with a biased rural polling technique, giving them greater coverage in urban areas and asking politically active people in these areas a loaded question specially designed to elicit a particular response.

So what happened this year?

This year the League Against Cruel Sports (LACS) appear to have broken with tradition and used a polling company called Survation for their annual boxing day hunting poll. Not only that, for the first time in years they changed the questions.

This was by no coincidence as the League were causing Ipsos Mori, and pollsters in general, considerable reputational damage by asking them to collude in their blatant annual push-pull poll. Things appear to have come to a head last year as I (Bean) sat on a windswept hill mounted upon my trusty steed ‘Theo’ discussing rural definitions with Ipsos Mori’s CEO Ben Page. It would have become apparent to Ben that some members of the public do their homework. This was followed up with emails to the British Polling Council who were very honest in their answers and held nothing back. Let’s just hope Ipsos Mori are wise enough to continue to distance themselves from the LACS extremists and their deceitful poll requests in future.

The government use only the purest form of random probability sampling poll and should insist – where a topic has been promoted to a state of national importance – then only this method be used, and results made available to the public. The failure by successive Conservative governments to recognise this rigging has led the country to be directed by tiny minority interest groups. The question that springs to mind is “In what other areas have the public been misled?”