The Great Game

Polls can lie, deliberately or by accident. The predicted election win for Dewey against Truman in the 1948 presidential election was a good example of an accidental lie. Today, certain pro-life activists have accused ABC of errors in their poll wich showed a large majority of Americans support the termination of Terri Schiavo's life. A vocal minority does not want to believe that they are far out of step with mainstream political opinion and are attacking anything that disagrees with their viewpoint. The fact that Bush's approval rating fell 6% points since he interfered with the Schiavo case implies that the decision has out of step.

All of this is a preamble to an article on the detail of the ABC poll and polling in general. The purpose is to point out misconceptions about polling and explain why the ABC poll is a reasonable effort at measuring the opinions of Americans.

Jax

Mark, I think that your discussion here implicitly endorses a commonly held error about the best way to interpret polling data about matters of public interest. (And this error underlies the criticism of the ABC poll as well.)

The error is the incorrect belief that there is a "right" or "unbiased" way to ask a question about any given public issue. There is no such thing. Everyone who works within the polling field is well aware that small changes in wording can affect the ways in which respondents answer questions. This approach leads us into tortuous discussions of question wording on which reasonable people can differ. Further, as you have pointed out many times in the past, random variation in the construction of the sample or in response rates can skew the results of any single poll away from the true distribution of opinions in the population.

So how do we look at public opinion on an issue such as the Schiavo case? The answer is NOT to find a single poll with the "best" wording and point to its results as the final word on the subject. Instead, we should look at ALL of the polls conducted on the issue by various different polling organizations. Each scientifically fielded poll presents us with useful information. By comparing the different responses to multiple polls -- each with different wording -- we end up with a far more nuanced picture of where public opinion stands on a particular issue. If we can see through such comparisons that stressing different arguments or pieces of information produces shifts in responses, then we have perhaps learned something. Like our own personal opinions, public opinion is not some sort of simple yes/no set of answers; it is complex, and it can see both sides of complicated issues when presented with enough information.

If we were to lock pollsters of all partisan persuasions in a room and force them to pick the "best" question wording on the Schiavo issue, we might end up with everyone asking the same question, but overall we would end up with less information about public opinion, not more. We are better off having the wide variety of different polls, with questions stressing different points of view on the issues, and then comparing them all to one another. This is precisely what you do in your discussion of the ABC poll, but I think you are asking entirely the wrong question -- not "is the ABC wording defensible?" but rather "what does the ABC poll, when compared to other polls with different wording, add to our overall understanding of public opinion on this issue?"

Of course, this sort of contextualizing of polling results is exceedingly rare in the media. Much more common is the front page story saying "here is our poll, and here is what it found, and it is a true representation of public opinion" -- and by implication, no other poll matters. Intellectual honesty is trumped by competition. The best we usually get are vague generalizations of all of the polls lumped together ("polls have consistently shown disapproval of Congress' actions"), and even those generalizations almost never appear in the initial story trumpeting the "exclusive" poll fielded by the newspaper/network itself.

The end result is that even those who pay close attention to the news media and the chattering classes often have very little real understanding of how to interpret polls in a thoughtful way -- which is one of the reasons your blog is so valuable.

P.S. Polls which attempt to predict election results are a rather different kettle of fish, for two important reasons: (1) Pollsters have been experimenting with questions wording for over 50 years and can keep wording the same regardless of the issues in a race; and (2) There is an actual real-world "check" on pollsters' work in the form of the actual election results. Neither of these characteristics apply to polling about issues of public interest. (Link)

# posted by Jacob @ 3/25/2005 04:14:00 p.m.