By: Ish Theilheimer
The subject always comes up, as it did in Larry Solway's commentary this week, so Straight Goods publisher Ish Theilheimer conducted an email interview with two experienced pollsters, Jim Matsui of Comquest Research and Marc Zwelling of Vector Research.
Straight Goods: Do polling firms have political biases? How are these biases reflected in public polls?
Marc Zwelling: No. Of what possible use would a polling firm be if it had political biases? The firm's clients have political biases. Every poll is biased by the questions it asks - or doesn't ask.
So why choose one firm over the other? Comfort. Big companies like big pollsters. Polls should be strategy-driven, but in reality they are budget-driven. Got a lot of money - use a huge firm. Got a little money, go to a niche player. Pollsters compete on reputation and service - clients see distinctions. Some polling firms specialize in health, technology, autos (Des Rosiers, Polk), labour. Pollsters get associated with clients from inertia (long-term relationships are standard in consulting, polling, training, and other business services). Sometimes a company's research specialist defects to a market research firm and takes the specialist's former employer's business. Sometimes ex-corporate market research people go freelance in their area (packaged goods, cars, etc.).
 |
|
Any firm that leaks a poll for a party/candidate for whom they are working should be held in suspicion |
Jim Matsui: In a general election, a media pollster can seriously damage their reputation by putting out a biased poll. Differences across media polling companies (ComQUEST polled for the Globe in 1993) reflect differences in sampling, sample size and style of questions.
You will see "oddities" when the party/leader preference questions are carried on omnibus surveys. These are the type of surveys where various organizations purchase individual questions and they don't sponsor the entire survey. If an organization wants to influence the party/leader standings, they would buy questions asked ahead of the party/leader questions that provide a favourable or unfavourable context for answering the succeeding questions. There was an episode of "Yes, Minister" that illustrated the context effect in polling. Party standings from omnibus polls are usually found outside of elections. Do you recall an Environics Poll during the Rae government which had some negative labour stuff ahead of the vote preference question?
Straight Goods: Which firms have what biases?
Jim Matsui: Any firm that leaks a poll for a party/candidate for whom they are working should be held in suspicion. You can always find something positive to leak if you have asked enough questions. I have never heard of a legitimate research company that has fabricated polling results. Campaigns sometimes try to release their canvass results under the guise of research.
Straight Goods: How does the client influence the poll and pollster? To what extent does the political outlook of the client influence the poll, and how?
Jim Matsui: Again, it is the context in which subsequent questions are asked. Parties, in most cases, want an accurate reading of how well or poorly they are doing. That is why neutral or even-handed questions should appear before the voting intention questions.
Marc Zwelling: Biases in polls are reflected in question wording. Asking the public which of these eight issues is the most important in deciding how you will vote ... health care, immigration, jobs, the environment, education, day care, gasoline prices, or housing... is biased because a critical ninth issue - crime - isn't on the list. The political outlook of the client influences the poll for the same reason - clients won't let pollsters ask all the questions that could give the public a fair chance to voice its true opinion.
|
|
Generally it's only far-left and far-right folks who think people are stupid and are influenced by ads, polls, and the media |
Straight Goods: What tricks are used in the polling trade to generate desired results?
Jim Matsui: See the "Yes, Minister" example. Sampling and response rate, more as shoddy research standards, are ways that poll results can be inadvertently tilted. For example, certain regions or areas might be undersampled. You would expect to find a higher percentage of older respondents than younger ones when you are interviewing on a hot, sunny weekend.
Marc Zwelling: Not asking certain questions (see above). Just asking people if they agree that ... only the NDP will save medicare (and not offering respondents the chance to disagree). Not giving respondents the other arguments... stating just one argument. Offering only two possible answers... should we raise taxes or invest in public services... while there is a third option that a pollster could ask... re-ordering government spending so that more taxes are spent on the public's favourite priorities. What you ask is what you get. In an experiment I've conducted in a split sample for a Vector National Poll you get little support for more government spending on "welfare" while a majority wants more government spending to "help poor people."
Straight Goods: What examples of such tricks would you point to in the current federal election campaign?
Marc Zwelling: Any National Post headline polls... the CA "surge" and today's "big time" Liberal "slide." In fact, since the campaign began the parties have moved in narrow bands and hardly beyond the 2-3-4 percentage point margin of error for samples of these sizes.
Straight Goods: Can you point to famous or infamous examples of polls being used to shape political outcomes?
Jim Matsui: The Gallup rogue poll in 1988. Whether they intended to or not, this certainly had at least a short-term effect. There are many stories concerning how Gallup botched this poll.
In the 1991 Saskatchewan election, Angus Reid reported a very favourable poll for the NDP. They had oversampled in Regina and Saskatoon (where the local newspapers were located who sponsored the poll), undersampled in the rural areas, them weighted the data to look representative. The net effect was to overrepresent the NDP vote well ahead of the NDP's actual support. Subsequently, the NDP coasted to victory.
Marc Zwelling: Famous polls to shape outcomes? Not likely to find any. Only half the public trust poll findings anyway. The idea that a poll could influence voters is nonsense (sorry, I know some want to believe that "others are stupid but I'm not").
Straight Goods: This federal election has been framed as a two-horse race by media commentators, despite apparent lameness of both horses. To what extent has polling played a part?
Jim Matsui: In fact, the media has relied very heavily on poll results. But horse race numbers are simply horse race numbers. The media should be looking somewhat deeper into why people are voting in certain ways.
Marc Zwelling: Polling helped frame the current election as a two-horse race because polls only measure what people think. Don't blame the polls if it's a two-horse race. Do you blame the thermometer if it's too cold?
Despite the polls (not because of them) the media seem to be allotting more than 9% of their coverage to the NDP and the Tories and nothing like 47% of their space/air time/web site to the Liberals.
Yes polling reports influence voters. It's part of the information they use to reach a decision. But how do polls influence voters? No one knows (believe me, they've tried to know). Remember half the public dismisses polls. Others may use the information to reinforce their inclination; some may use polls to switch their vote. Since voting and polling are both secret, who will ever know? (Generally it's only far-left and far-right folks who think people are stupid and are influenced by ads, polls, the media, etc.) How could voters be "made to appear to back one horse"? Polls have a good track record in elections (see the recent US election polls which were like the results too close to call), but lots of market research the public never sees also has a good track record of anticipating market share, consumer acceptance of new products, likelihood of signing a union card, etc. If people vote for winners to avoid losers how could the PQ or the NDP ever have launched? At one point these were losers and had single-digit support. Remember the Charlottetown Accord, which had massive support initially and steadily lost support (as measured in polls). If you're right then it should have passed becuase every sheep would have said, well, most people think it's OK, so I guess I have to agree.
Every poll is biased. Common sense is the public's best protection. That and reading lots of polls and being critical thinkers, which is not all that hard for most people (despite the dorks CBC radio keeps turning up in their pathetic Dick Gordon spots with "ordinatry voters - no telling what you'll turn up when you drag a microphone through an Edmonton mall).
Marc Zwelling can be reached at marc@vectorresearch.com or on the web at www.vectorresearch.com
Jim Matsui can be reached at jmatsui@comquest.ca or on the web at www.comquest.ca
[ Front Page ]
