Sunday, September 30, 2012

About those polls.

The anomaly in the polls is that they are so skewed to Democrats during a time when Democrat registration is trailing Republican registration.

Assuming that pollsters are simply reporting what they are finding, then there would seem to be a 13% advantage among Democrats, which doesn't exist according to the information we have about registration.

I haven't been able to figure out how that is possible. If pollsters simply call up 1,000 people and those people report themselves to be 113% more Democrat than Republican, that seems to be a random sample, and it certainly wasn't "rigged" by the pollster.

That was the explanation offered to Hugh Hewitt in this interview:

HH: Joined now by Steven Shepard. Steven is the director of polling for the National Journal. Hello, Steven, welcome to the program, thanks for being on.

SS: Hi, how are you?

HH: Great. Your article today on polling, would you summarize it for the audience?

SS: Sure. I guess there has been a lot of talk in largely conservative circles lately, as you’re aware, and I know that you’ve addressed this on your show. And you talked to some of those same people I did for my story, basically that polls are sampling, or including too many Democrats in their samples. And I spoke to a few public and media pollsters about this, and they responded basically that they’re not, there is no model to which they’re setting the party ID breakdown in their polls. They’re just reporting the results of who they’re speaking with.

HH: Now they do weight their results based upon any number of different factors, correct?

SS: Correct. They weight based on gender, to make sure that they have sort of an accurate reflection of the overall population or the overall electorate with regards to the amount of men and women. They weight according to race, they, some pollsters will weight according to other factors like age, and mostly to age, but some towards also education or income, or something that is generally measured, and is relatively static, or changes over time. It’s something that can be measured.

HH: So why do they weight those factors, but not party affiliation?

SS: Because party affiliation is an attitude. This is their argument. It’s an attitude. This is exactly what the poll is designed to measure – how Americans are feeling about politics. And frequently, for instance, if you conduct a poll about, let’s say, the presidential race, and you ask the party identification at the end of the poll, it will frequently, especially for those sort of soft independents who could move one way or the other, they’ll just end up identifying with the party of the candidate that they chose earlier in the poll. So that tends to fluctuate a lot.


On the face, it seems reasonable to say that if you call up a thousand people and 700 says they are Democrat, then that tells you that the Democrats have a 70% registration rate.

But things are not the simple. We know this because every election, exit polls over-estimate the Democrat edge in voting. Michael Barone offers this insight:

It may be that we're seeing the phenomenon we've seen for years in exit polls, which have consistently skewed Democratic (and toward Barack Obama in the 2008 primaries). Part of that is interviewer error: Exit poll pioneer Warren Mitofsky found the biggest discrepancies between exit polls and actual results were in precincts where the interviewers were female graduate students.

But he also found that Democrats were simply more willing to fill out the exit poll. Which raises the question: Are we seeing the same thing in this month's polls?


Then, there is this interesting bit about cellphones from Hugh Hewitt's interview with Lee Mirengoff:

LM: Well, you have to understand also that there’s a difference between using live interviewers who call cell phones, and polls which use robocalls in terms of their scientific basis and their dependability.

HH: I understand that, and we…

LM: Let me finish the point, please. If you do not use cell phones in your surveys, as a lot of the surveys don’t, you’re missing what is most of the Obama margin, because right now, about 30% of the households, and the likely voters in each of these states, are cell phone only. And in fact, I could give you the numbers, but you don’t want to get bored by the statistics. The truth is that most of the Obama margin is based on the cell phone, which are obviously younger people and people who are minorities. If you don’t have cell phones in the sample, you’re going to find a one or two point difference in the race. We do also among our land line only respondents.


Hewitt has done the best job of explaining these nuances in this article.

Here are the key takeaways from all these conversations.

The pro-Obama pollsters don't have answers as to why their skewed samples are trustworthy beyond the fact that they think their approach to randomness is a guarantee of fairness, and they seem to resent greatly that the questions are even asked. Like Madoff would have resented questions about his stunning rate of return.

Barone notes that percentage turnout by party in a presidential year hasn't been much greater for the president's party than it was in the preceding off-year, which makes samples outstripping even the 2008 model of Democratic participation "inherently suspicious.".

Cost notes that Romney is winning the independent vote in every poll, which also makes big Obama leads suspect.

And my conversation with Mr. Shepard, whose employer National Journal has a reputation for the best non-partisan work inside the Beltway, didn't find any academic, disinterested support for the proposition that party identification cannot be weighted because of the inherent instability of the marker.

The biggest unanswered question of all: If party ID is so subject to change that it should not be weighted according to an estimate of turnout, why ask about it at all? And if it is for the purpose of detecting big moves, as Mr. Shepard argued, why not report that "big move" in the stories that depend upon the polling?

Over the next few weeks, the junk polls will start tweaking their samples and dancing around that fact in order to come closer to reality. Too late. The sample controversy has penetrated into the public's mind, and Quinnipiac and Marist are marked as either rubes or rogues unless the huge democratic turnout advantage materializes on November 6.

In the meantime, if you see a story on a poll that doesn't tell you the partisan make-up of the sample, note to yourself that the reporter is missing the most interesting bit of data to most readers.

Just as Bernie Madoff used to hide the real story.


No comments:

 
Who links to me?