Friday, April 07, 2006
Republican War on (Social) Science*
by Anonymous
It quickly became evident that the "research organization" is sponsored by one of the Republican challengers who is trying to unseat Maurice Hinchey, the liberal Democrat who represents NY22 in the House of Representatives. Most of the questions were thinly veiled attempts to pursuade, not to reveal preferences. So, the questions about Hinchey were on this order (paraphrased):
These were followed by questions worded to favor the challenger:
What made the whole experience so surreal, though, was not so much the strong-arm wording tactics, but the surveyor herself, a very nice but probably minimally educated woman in her 50s. If someone had trained her how NOT to administer a survey, she couldn't have done a better job of skewing the results.
If social science statistics are like sausages, the results coming out of this survey are dog food. The best part, though, is that the poll didn't even persuade effectively. The surveyor couldn't pronounce the challenger's last name, so she just called him Ray for short. Somehow I don't think that's what the campaign director had in mind.
* Disclaimer: I do not mean to imply that abysmal survey practices are widespread within the Republican party, nor that Democratic politicians are any better at following the basic rules of survey construction and administration.
One of the occupational hazards of being a sociologist is an inability to say "no" to any organization that claims to be doing a survey or opinion poll. Last week, I was called (during dinner, of course) by a representative of "political opinion research organization," and, with visions of the American National Election Survey dancing in my head, I agreed to take part in the survey.
It quickly became evident that the "research organization" is sponsored by one of the Republican challengers who is trying to unseat Maurice Hinchey, the liberal Democrat who represents NY22 in the House of Representatives. Most of the questions were thinly veiled attempts to pursuade, not to reveal preferences. So, the questions about Hinchey were on this order (paraphrased):
- "If you found out that Hinchey tried to block efforts to stop terrorism within our borders, would you be more or less likely to vote for him?"
- "If you found out that Hinchey supported laws that would let rapists and child molesters out of prison, would you be more or less likely to vote for him,"
- "If you found out that Hinchey had used taxpayer money to make over 200 trips to foreign countries, including the South Carribbean, would you be more or less likely to vote for him."
These were followed by questions worded to favor the challenger:
- "[The challenger] plans to reduce taxes for working families to help them make ends meet. Does this make you more or less likely to vote for him?"
- "[The challenger] plans to introduce legislation to hold lazy teachers accountable for failing students. Does this make you more or less likely to vote for him?"
- "[The challenger] plans to visit China to forge connections with businessmen who will bring jobs to Central New York. Does this make you more or less likely to vote for him?"
What made the whole experience so surreal, though, was not so much the strong-arm wording tactics, but the surveyor herself, a very nice but probably minimally educated woman in her 50s. If someone had trained her how NOT to administer a survey, she couldn't have done a better job of skewing the results.
- She stumbled over words, and eventually decided just to skip possible answers that contained words she couldn't pronounce. From what I could tell, this was about half of the possible responses.
- She announced how most other people she had surveyed had answered the question. "Oppose [Bush's decision to invade Iraq]? That's what most people are saying."
- If I paused too long to try to parse the remaining list of possible answers, she would prompt me with what she thought was the best one. "Not many people have said clean air and water [is the most important environmental issue facing Central New York], but that's what I think."
- She kept up a running commentary of how much she was getting paid to administer the survey ($8/hour), the poor state of her health, and the high cost of medications. I appreciated the irony of this in light of the challenger's platform, but didn't ask her if she did.
If social science statistics are like sausages, the results coming out of this survey are dog food. The best part, though, is that the poll didn't even persuade effectively. The surveyor couldn't pronounce the challenger's last name, so she just called him Ray for short. Somehow I don't think that's what the campaign director had in mind.
* Disclaimer: I do not mean to imply that abysmal survey practices are widespread within the Republican party, nor that Democratic politicians are any better at following the basic rules of survey construction and administration.
Comments:
<< Home
When I get surveys like that, I answer the questions contrarywise. For example:
Telephone pollster: "If you found out that Hinchey tried to block efforts to stop terrorism within our borders, would you be more or less likely to vote for him?"
Me: "More likely."
If figure if the survey design is that poor, I might as well confound the results just a tiny bit in protest.
Oh, and I suspect the polltaker was well aware of the irony of her own problems in light of the challenger's positions...but if you're making $8 an hour cold-calling, you're probably not someone with a lot of other employment choices.
Telephone pollster: "If you found out that Hinchey tried to block efforts to stop terrorism within our borders, would you be more or less likely to vote for him?"
Me: "More likely."
If figure if the survey design is that poor, I might as well confound the results just a tiny bit in protest.
Oh, and I suspect the polltaker was well aware of the irony of her own problems in light of the challenger's positions...but if you're making $8 an hour cold-calling, you're probably not someone with a lot of other employment choices.
I've assumed that with their engagement of evil geniuses such as Frank "Death Tax" Luntz, Republicans would have certain structural advantages in opinion manipulation.
I should say, Kim, I have nightmares about waking up to a Government Accountability Office report detailing survey protocol violations of the sort you describe in my favorite ongoing probability sample study...
I should say, Kim, I have nightmares about waking up to a Government Accountability Office report detailing survey protocol violations of the sort you describe in my favorite ongoing probability sample study...
Jeremy, I've wanted this to be a place where all of the social sciences can peacefully coexist.
This is actually Kim's second (and hopefully not last) posting appearance at this site. The first one is here. I am curious about the other blog (deleted or never fully set up?) listed in her profile.
This is actually Kim's second (and hopefully not last) posting appearance at this site. The first one is here. I am curious about the other blog (deleted or never fully set up?) listed in her profile.
"I am curious about the other blog (deleted or never fully set up?) listed in her profile."
I had to (or thought I had to) set up a blogger account of my own to be able to leave comments on someone else's blog. However, after about 20 failed efforts to be a diarist, each lasting a maximum of 2 days each, I have no illusions that I would be able to maintain a viable blog on my own.
Post a Comment
I had to (or thought I had to) set up a blogger account of my own to be able to leave comments on someone else's blog. However, after about 20 failed efforts to be a diarist, each lasting a maximum of 2 days each, I have no illusions that I would be able to maintain a viable blog on my own.
<< Home
