[Link] "Fewer than X% of Americans know Y"

So, basically every survey has to deal with bothersome subjects, especially if it's a survey of teens or children (for different reasons).

I remember a survey given by some friends of mine for a school project; they had taught a lesson to a classroom of children and wanted to measure how much stuck. The survey answers were all on a 1-5 scale, where 1 was "I disagree strongly" and 5 was "I agree strongly."

One of the questions, put on there as a test to ensure the children understood the format, was "I eat breakfast with Martin Luther King Jr every morning." (The lesson mentioned him, among others.) They were expecting 1s, but the average answer was 2.

At least some public-ignorance surveys use multiple-choice questions, which do not suffer from the problem in the quoted text.

"only 28 percent of Americans are able to name one of the constitutional freedoms, yet 52 percent are able to name at least two Simpsons family members"

My first, almost-like-a-sort-of-trained-reflex reaction to reading something like the above:

"So okay, about 55% of whichever target sub-population this study targeted (default: probably students) watch or hear about The Simpsons often enough to remember at least two names. On the other hand, a lot of that sub-population probably named something as a constitutional freedom after being primed on some unrelated subject but that something wasn't a constitutional freedom, and only about 30% saw the trap and managed to remember an actual good answer."

In general, I'm pretty dubious of conclusions based on polls and questionnaires, and assign lower probability to both the author's and my own interpretation until I see the specifics of the methodology. I have trust issues.

Also, what incentives are there for answering truthfully? The alternative explanations provided a Language Log seem better, but I used to take these sorts of results as primarily being evidence for a high natural frequency of trolls in the sample population.

Nate Silver blew up a public ignorance survey a few years ago, on the 538 blog.

http://www.fivethirtyeight.com/search/label/strategic%20vision

I would not be surprised to find more of that around.

Neat! I'll put less confidence in such surveys now. HOWEVER! Many of the questions in such surveys are plain-ol' 50/50, and I have no idea how they could be very biased.

As an example, here is a scan from Carpini and Keeter's What Americans Know About Politics and Why It Matters. You'll notice that, in table 2.7, only 42% of Americans knew that Soviets suffered more deaths than Americans during World War 2. Seems like a coin flip to me, unless they asked, "Who had the most deaths during World War 2?" and ignored all answers besides US and USSR. I still think Americans are pretty durn ignorant of most political and historical matters. (Myself included, for many of the questions. I have no idea who my state's congressmen are (and I don't really care.))

But then, I've never been one to compare this to modern cultural knowledge. I see that as irrelevant. Asking about fresh memory vs. deep memory doesn't tell you about political knowledge per se. Responses should be compared against questions of similar difficulty.

If, every time you heard a claim of the form "Only X% of Americans know Y" you thought "there's something strange about that", then you get 1 rationality point. If you thought "I don't believe that", then you get 2 rationality points.

Well, if this was indeed a common methodological flaw. I'm not ready to break out the champagne yet.

The Language Log post also emphasizes that mass media reports of such surveys sometimes quote numbers completely different from the actual survey results, presumably to increase the value of the news story. So:

In the passage quoted above, Robin Young states the survey result incorrectly — actually, 73% of respondents, not 28%, were able to name one of the constitutional freedoms – and she spins it in a doubtful direction to boot, because only 65% were able to name one of the Simpsons characters.

In the cited New York Times article, Diane Ravitch is referring to the 2010 NAEP 12th grade U.S. History test, in which 82%, not 2%, of 12th graders correctly identified Brown v. Board of Education.

In addition to discounting "public ignorance" surveys, we should discount surveys and other factual information reported through such media.

I wasn't skeptical enough of these. -1 point to self. Thanks, Nisan. (:

I actually don't find anything strange about that. I am reasonably well-educated and know a lot of things, and I have no idea who the US supreme court chief justice is (though if I needed to know, it would take me about 2 seconds).

I remember when a few years ago, on the news on TV, there was an article about how 40-70% (forgot the exact number) of the interviewed people said that Beethoven is a dog. I was frustrated at how shocked the other people in the room were.