Poor Surveys Get Opposite Poll Results

March 2011

Most national polling organizations are independent and do not have an agenda they want to advance. Independent and accurate polling is the basic requirement for trust in the polling organization. Knowing this, those organizations that do have a political agenda are usually good at hiding it.

But, bias does exist. The possibility of bias warrants scrutiny of the methodology and questions used. In the rest of this article, we provide examples of two polls that asked seemingly similar questions, yet provided quite different answers. The different results occur because the polls each make errors discussed in our primer on creating and using surveys.

A February 28, 2011 New York Times headline read, “Majority in Poll Back Employees in Public Sector Unions”. In contrast, a February 21, 2011 Rasmussen Report headline read, “48% Back GOP Governor in Wisconsin Spat, 38% Side with Unions”. The recent New York Times/CBS poll contained sampling errors. The Rasmussen poll contained question wording bias.

The New York Times/CBS Poll

The New York Times February 28, 2011 article summarized its poll results as follows:

“As labor battles erupt in state capitals around the nation, a majority of Americans say they oppose efforts to weaken the collective bargaining rights of public employee unions and are also against cutting the pay or benefits of public workers to reduce state budget deficits, according to the latest New York Times/CBS News poll.

…the nationwide poll found that embattled public employee unions have the support of most Americans — and most independents — as they fight the efforts of newly elected Republican governors in Wisconsin and Ohio to weaken their bargaining powers, and the attempts of governors from both parties to cut their pay or benefits.

Americans oppose weakening the bargaining rights of public employee unions by a margin of nearly two to one: 60 percent to 33 percent. While a slim majority of Republicans favored taking away some bargaining rights, they were outnumbered by large majorities of Democrats and independents who said they opposed weakening them.

Those surveyed said they opposed, 56 percent to 37 percent, cutting the pay or benefits of public employees to reduce deficits, breaking down along similar party lines…”

These results are questionable for the following three reasons, all of which involve starting with an unrepresentative sample:

1. The poll sample contained a political party split (Republican/Democrat/Independent, aka R/D/I) of 26%/36%/31%, giving Democrats a ten-point advantage. This would be fine if this party split was representative of the entire population. Based on recent Gallup results, general-population surveys show erosion in Democratic affiliation that bears no resemblance on the split provided by The New York Time/CBS. This change is seen in the results of the November 2010 elections. In 2008, President Obama won the popular vote by seven points nationwide; the current poll wrongly assumes that the electorate has grown more Democratic in 2011.
2. One fifth (20%) of the poll’s respondents claim to come from union households. Again, this would be fine if this percentage was representative of the entire population. According to a January 2011 Bureau of Labor Statistics (“BLS”) report, 11.9% of American workers belong to a union, down from 12.3% a year earlier. Interestingly, a January 21, 2011 New York Times article titled, “Union Membership in U.S. Fell to a 70-Year Low Last Year” reported these BLS results, as follows:

   “The number of American workers in unions declined sharply last year, the Bureau of Labor Statistics reported on Friday, with the percentage slipping to 11.9 percent, the lowest rate in more than 70 years.”

   While some of the higher 20% amount is explained because of the more inclusive description of a having a union member in the household, the reported percentages between union households and union membership cannot be entirely explained by this difference.

3. One fourth (25%) of the poll’s respondents claim to be either public employees or share a household with a public employee. Again, this would be fine if this percentage was representative of the entire population. According to the most recent Current Employment Statistics survey by the BLS, the U.S. has approximately 22 million government employees out of an employed workforce of approximately 130 million. Thus, government employees account for approximately 17% of all workers, less than this poll’s sample of 25%.

As noted above, each of these errors provides a sample that is not representative of the larger population, and so provides a conclusion that is suspect.

The Rasmussen Poll

The Rasmussen poll contains survey question flaws. Rasmussen asks the following five questions:

1. How closely have you followed news reports about the Wisconsin governor’s effort to limit collective bargaining rights for most state employees?

   This first question is typical. It isn’t necessarily a bad question; it serves to frame the subject of the discussion.

2. Does the average public employee in your state earn more than the average private sector worker in your state, less than the average private sector worker in your state or do they earn about the same amount?

   The question jumps from Wisconsin to your state. This was a nationwide poll, not restricted to Wisconsin. Putting aside the difficulty of making comparisons for differences in skill levels and other factors that appropriately impact compensation, it is silly to seek an opinion about a factually calculable number – particularly a detailed one like this. An average respondent is not likely to know factually off the top of his/her head about the wages in public versus private sector jobs. The question has the undesired impact of potentially introducing bias into the results.

3. Should teachers, firemen and policemen be allowed to go on strike?

   Public safety employees are exempted from the proposed Wisconsin legislation that was being considered at the time of this poll. Thus, the two most likely emotionally charged of the three examples in the question are irrelevant to the issue. Consequently, the question is not properly focused and may be misleading, resulting in biased results.

4. In the dispute between the governor and the union workers, do you agree more with the governor or the union for teachers and other state employees?

   This is the question that Rasmussen uses for its headline poll result, “48% Back GOP Governor in Wisconsin Spat, 38% Side with Unions”. But, at this point, the poll has already likely been biased by the preceding questions.

5. Would you favor or oppose reducing your state government payroll 1% a year for 10 years, either by reducing the number of state employees or by cutting the pay of state workers?

   This question is too vague to understand whatever results may come from it. First, the question gives multiple options for how to obtain the reduction. Additionally, since respondents undoubtedly favor some programs over others, the question fails by making no mention of what program area might receive the cuts. The question is easy to say “yes” to, since who would not want to save money? But one person may respond “yes” for wage reductions, another person may respond “yes” for cutbacks in department X, and yet another person may respond “yes” for cutbacks in everyplace other than department X (that the second person wanted to cut).

The following video clip from the British Series, “Yes Prime Minister” makes a similar point to this article with more humor.

Fulcrum Inquiry uses statistics and advanced quantitative methods to address disputed issues.