Earlier today I received a phone call from a polling company. They wanted to ask me questions about my local water district authority. For the most part I was ignorant, but one of the questions struck me as extremely odd. There was a section of the survey in which they read me a series of statements and asked me to indicate whether and how strongly I agreed with them. One of the statements was “The water provided by the district authority meets all federal, state and local quality standards.”

Bear in mind that this was a random survey. I’m John Q. Public to these people. Now ask yourself what information you would need to be able to evaluate that statement. Do you know all of the federal, state and local water quality standards that apply where you live? Do you know the details of the contents of the water and the condition of the water infrastructure? I don’t, and I’m pretty sure you don’t either. And without that information, there is no basis to hold any opinion on whether the statement in question is true or false.  The chance of any randomly selected person knowing what they would need to know to give a meaningful answer to that question falls squarely between ‘slim’ and ‘fat’.

If the people who made up these surveys had any grasp of proper cognitive methodology, asking such a question would be simply unthinkable. But, sadly, they do not, and they’re probably no different in that regard than the majority of the people they poll. As a result, these sorts of polls turn into an orgy of mutual subjectivism, in which the emotionally-driven responses of hundreds of people get aggregated together into a pseudo-objective sum. The fact that some percentage of people ‘think’, on no particular basis, that the water district is meeting quality standards winds up being adduced as evidence that the water district is doing its job well.

Collective inter-subjectivity as a substitute for objectivity. I’ve been back from OCON for less than a week, and already I feel like I’m surrounded by monkeys.

Anne comments: I was in a job once where I was tasked with creating a survey for the organization’s membership (approximately 20,000 people at the time). I had no particular experience in creating surveys, but I worked with a consultant who did. One of the things I was very careful about was to ask questions that would produce the data we actually wanted. That’s trickier than you might imagine, and involves taking into account all the possible exceptions that someone might have to a survey question — but without making the question so open-ended that it produces useless results.

4 Responses to “The Epistemology of Pointless Survey Questions”
  1. Oscar Cwajbaum says:

    I think the question makes perfect sense — they wanted to know what percentage of their customers wrongly believe (for whatever reason) that the water provided by the water district does not meet federal, state, and local quality standards. If a substantial percentage of their customers had a mistaken belief, they would likely consider an advertising campaign to correct people’s beliefs.

  2. Kyle Haight says:

    I’d be more inclined to believe that if the survey had any indications of probing into why I did or did not think certain things. There is a big difference, for example, between believing something out of ignorance and believing something based on misinformation, and if you’re interested in education you’d want to distinguish between those cases. This survey did not.

  3. Oscar Cwajbaum says:

    There are certainly ways the question could have been improved, but that still doesn’t make it pointless.

  4. Richard says:

    Sorry, sort of, Oscar. The question is absolutely absurd. There was a time when “Ignorance was no excuse.” but now laws abound that make no moral sense whatsoever. Men cannot possibly grasp every detail of a particular issue controlled by the state. People are so naive that they believe the present financial crisis is a consquence of corporate greed. The fact is, that since 1996 over 53000 new rules have been instigated to control the financial markets. Who could possibly keep track of so much bullshit? Then you want the average person to know? You are fuckin’ NUTS/

Leave a Reply