Thursday, October 07, 2004
More Fun With Surveys
by Tom Bozzo
This seems like deplorable state of misinformation on the part of the public, but one should never accept such results without first examining how the question was posed. Here's the full question:
This formulation yields two main objections.
First, there is the problem of translating the repsonse categories into (subjective) probabilities. "Very" likely could be interpreted as ">50 percent probability" or "nearly 100% probability;" ditto "somewhat likely" could easily be taken as "likelier than not" or just "not high but not close to zero probability." Without a "nearly certain" category (to mirror "not likely at all"), it's harder to tell. But the lower-probability interpretation of the categories would make the respondents appear to be less at odds with established facts.
Second, the question itself is formulated such that it's difficult to be certain that it's registering fears specifically of Iraqi WMD (as opposed to anyone's WMD, which is not an irrational fear). The question may appear to be straightforward enough assuming good listening comprehension, but real respondents have a tendency to respond to "A and B" statements as "A or B," or perhaps just "B" -- some will forget about crucial modifiers in the question. It would arguably be better to vary the wording to show that a specific form doesn't materially affect the results, or to break the item up into multiple questions.
Note to Readers: Nothing in the foregoing shall be construed as stating that there is not 27 percent or more of the electorate who respond to everything George W. Bush or Richard Cheney state or imply with total credulity.
Overall, the Badger Poll, which states that it strives to use the best available methods, could do better. But at least they are not unaware of the issue. Wisconsin Public Radio quoted Don Ferree, the Badger Poll XVII director, regarding a Harris Interactive poll sponsored by the Wisconsin Policy Research Institute indicating 2/3 support for strict local spending caps similar to (actually, stricter than -- another case of a significant technicality that does not likely register with a lay audience) the Republican-sponsored Taxpayer Bill of "Rights" (or TABOR). Ferree noted that by asking the question without also exploring respondents' attitudes to actual spending items, the formulation is likely to overstate true support for the item.
More significantly, the Wisconsin Policy Research Institute is a partisan group (linking from their home page an article entitled "Lowly Sinner: George W. Bush's Humble Faith") and so polls they sponsor should be regarded with additional skepticism.
Another release of results from Badger Poll XVII finds 27% of surveyed Wisconsinites calling an attack by non-existent Iraqi WMD "very likely" and another 33% considered it "somewhat likely."
This seems like deplorable state of misinformation on the part of the public, but one should never accept such results without first examining how the question was posed. Here's the full question:
IRAQ10X. If the US had decided NOT to go to war, how likely do you think it is that we would eventually have been attacked with Iraqi weapons of mass destruction? Very likely, somewhat likely, not very likely, or not likely at all?
This formulation yields two main objections.
First, there is the problem of translating the repsonse categories into (subjective) probabilities. "Very" likely could be interpreted as ">50 percent probability" or "nearly 100% probability;" ditto "somewhat likely" could easily be taken as "likelier than not" or just "not high but not close to zero probability." Without a "nearly certain" category (to mirror "not likely at all"), it's harder to tell. But the lower-probability interpretation of the categories would make the respondents appear to be less at odds with established facts.
Second, the question itself is formulated such that it's difficult to be certain that it's registering fears specifically of Iraqi WMD (as opposed to anyone's WMD, which is not an irrational fear). The question may appear to be straightforward enough assuming good listening comprehension, but real respondents have a tendency to respond to "A and B" statements as "A or B," or perhaps just "B" -- some will forget about crucial modifiers in the question. It would arguably be better to vary the wording to show that a specific form doesn't materially affect the results, or to break the item up into multiple questions.
Note to Readers: Nothing in the foregoing shall be construed as stating that there is not 27 percent or more of the electorate who respond to everything George W. Bush or Richard Cheney state or imply with total credulity.
Overall, the Badger Poll, which states that it strives to use the best available methods, could do better. But at least they are not unaware of the issue. Wisconsin Public Radio quoted Don Ferree, the Badger Poll XVII director, regarding a Harris Interactive poll sponsored by the Wisconsin Policy Research Institute indicating 2/3 support for strict local spending caps similar to (actually, stricter than -- another case of a significant technicality that does not likely register with a lay audience) the Republican-sponsored Taxpayer Bill of "Rights" (or TABOR). Ferree noted that by asking the question without also exploring respondents' attitudes to actual spending items, the formulation is likely to overstate true support for the item.
More significantly, the Wisconsin Policy Research Institute is a partisan group (linking from their home page an article entitled "Lowly Sinner: George W. Bush's Humble Faith") and so polls they sponsor should be regarded with additional skepticism.
