Margin of error explained
Margin of error explained
What does this term really mean?
To make this clearer, I?m going to use a fairly simple example. Imagine we have a bag of marbles, some of them are red and the rest are blue, but we don?t know how many of each. One way to find out would be to tip all the marbles out and count each one; this is like the approach the government takes every 10 years in the national Census. The alternative is to randomly pick out a selection of them, see how many are red and then infer that the proportion of red marbles in the bag is likely to be similar to the proportion in the selection, this is like conducting a survey.
If half the marbles in the bag were red, you can see that it would be pretty unlikely to randomly pick out 20 red marbles and no blue ones. Likewise if 90% are red, how probable is it that we pick out 20 blue marbles? Not very.
In fact the mathematics of probability means we can be very precise about this: if you randomly pick out a number of marbles and a certain proportion turn out to be red, then knowing the number of marbles in the bag, I can calculate the probability that you would get that result for any proportion of red marbles in the bag.
You can also look at this the other way round. Based on the proportion of red marbles in the random selection I can calculate a certain range around that proportion that the proportion of marbles in the bag is likely to be in. Then I could say something like this: ?There is a 95% chance that the proportion of red marbles in the bag, is within 2% of the proportion of red marbles in the sample?. That?s margin of error: how close a property of the whole bag is likely to be to the same property of a randomly selected sample. How likely? Well in this case it was 95%. We call this the confidence level and it?s something we have to stipulate when we calculate the margin of error. We could also calculate the margin of error at the 90% level, or the 99% level.
There are two important points to make about this:
- Margin of error does not make sense without knowing the confidence level – We asked at the start, ?How can researchers be so sure the result can?t be outside the margin of error?? Well, we can?t. A margin of error does not mean that the real property of the population cannot be outside the margin of error. It just means that there is a high probability that property of the population is close to the property of the sample. We calculate how close (the margin of error) after we choose the level of confidence we want to have that the real property is within the margin of error. Confidence levels of 90%, 95% and 99% are pretty standard.
- Margin of error only describes one source of error – Let?s move away from bags of marbles to market research. Our bag of marbles is the bit of the whole country we want to find out about and we look at a sample by phone interview or online questionnaire. Unlike pulling marbles blind from a bag, the means of selecting people is not entirely random. Another problem is that instead of colour of marbles we want to find out something about these people?s behaviour or opinions. We can?t just look at them; we have to ask questions, which is not 100% reliable.
These errors don?t lend themselves to easy quantification, but as a research agency it is our job to minimize them by applying principles of good questionnaire design and techniques like weighting.
So next time someone tells you some results are ?inside margin of error?, you might want to ask them:
“What confidence level have you used to calculate the margin of error, and why do you think that is the appropriate level?”
“What other sources of error are you aware of that the margin of error won?t account for and what has been done to minimize these?”
