Every mainstream media outlet in Canada is reporting that 82 per cent of Canadians support Bill C-51, the Harper government’s proposed “anti-terror” legislation. The reporting makes the 82 per cent figure seem certain even though the Angus Reid poll it’s based on was not a random sample of the whole population and more than half of those who took part in the survey knew little or nothing about the new legislation.
Globe and Mail coverage was typical. The newspaper ran an “analysis” piece under the headline, “New poll finds Harper’s anti-terror bill is a political juggernaut.” The paper’s chief political writer, Campbell Clark, began with a declaration: “There’s rarely been a bill before Parliament that was more popular. The public (sic) Conservatives’ new anti-terror legislation is filling a public demand for tough new measures aimed at a terrorism threat that Canadians believe is serious, and close to home, according to a new poll.”
It took Clark 12 paragraphs to tell readers how the poll was done. “The Internet survey was conducted with a randomly selected sample of 1,509 Canadians drawn from an Angus Reid panel.”
After the piece ends 17 paragraphs down, an editor’s note informs readers that an earlier version of the story mistakenly gave the poll’s margin of error. “Because it was not a random sample of the whole population, the pollster, the Angus Reid Institute, does not cite a margin of error,” the editor’s note continued. It then went on to say that the margin of error the pollster did give was for a “probabilistic sample of the same size.”
Questions about the accuracy of online polls
The Pew Research Center in the U.S. warns that online polls have their limitations. For one thing, participation is limited to people with access to the Internet. For another, online polls like this one are based on panels of people who volunteer to fill out surveys in return for small amounts of money, a chance to win prizes and the promise that their opinions will influence elite decision-makers. (See below for more details on the Angus Reid panels. For more detailed information about the limitations of online polls, see this 2010 report from the American Association for Public Opinion Research.)
Aside from concerns about the limitations of online polling, more than half of those who took part in the survey knew little or nothing about Bill C-51. In fact, only 18 per cent seem to have paid much attention to media reports and what friends and family had to say.
So why attach any significance to this poll?
Well, it’s an election year and for months, journalists have been scrambling to figure out which issues may favour one party or another. It’s part of the media fascination with the political horse race, who’s gaining and who’s losing. And the polls, with their precise numbers, seem to offer answers.
As they say, perception is reality.
How Angus Reid puts his panels together
Angus Reid panellists are volunteers who sign up to complete monthly profiling questionnaires. The links to the online surveys are emailed to them. The surveys ask for information about a range of things such as age, income, family size, lifestyle, consumer preferences and habits as well as opinions on current political issues.
Panellists are told their opinions will influence decision-makers “whether they are developing public policy or moulding new products and services. Your voice will be heard in the media as they report on the changing values and preferences of the Canadian public.”
In return for taking part, panellists are eligible to receive small amounts of money, usually 50 cents for each profiling survey, plus a chance to win monthly prizes. Panellists can redeem their “survey dollars” only when their total reaches $50. They also earn additional survey dollars if they fill out other questionnaires that the company invites them to complete.
From the Globe and Mail, Feb. 19, 2015:
Editor’s Note: The original newspaper version of this story and an earlier digital version mistakenly referred to a margin of error for this poll. The poll was conducted with a randomly selected sample of 1,509 people drawn from an Angus Reid panel of 130,000 people. Because it was not a random sample of the whole population, the pollster, the Angus Reid Institute, does not cite a margin of error. It instead noted the margin of error for a probabilistic sample of the same size. This digital version has been corrected.