The "Spot the Fakes" Test

post by Scott Alexander (Yvain) · 2009-03-17T00:52:54.095Z · LW · GW · Legacy · 18 comments

Contents

  Footnotes
None
18 comments

Followup to: Are You a Solar Deity?

James McAuley and Harold Stewart were mid-20th century Australian poets, and they were not happy. After having society ignore their poetry in favor of "experimental" styles they considered fashionable nonsense, they wanted to show everyone what they already knew: the Australian literary world was full of empty poseurs.

They began by selecting random phrases from random books. Then they linked them together into something sort of like poetry. Then they invented the most fashionable possible story: Ern Malley, a loner working a thankless job as an insurance salesman, writing sad poetry in his spare time and hiding it away until his death at an early age. Posing as Malley's sister, who had recently discovered the hidden collection, they sent the works to Angry Penguins, one of Australia's top experimental poetry magazines.

You wouldn't be reading this if the magazine hadn't rushed a special issue to print in honor of "a poet in the same class as W.H. Auden or Dylan Thomas".

The hoax was later revealed¹, everyone involved ended up with egg on their faces, and modernism in Australia received a serious blow. But as I am reminded every time I look through a modern poetry anthology, one Ern Malley every fifty years just isn't enough. I daydream about an alternate dimension where people are genuinely interested in keeping literary criticism honest. In this universe, any would-be literary critic would have to distinguish between ten poems generally recognized as brilliant that he'd never seen before, and ten pieces of nonsense invented on the spot by drunk college students, in order to keep his critic's license.

Can we refine this test? And could it help Max Muller with his solar deity problem?

In the Malley hoax, McAuley and Steward suspected that a certain school of modernist poetry was without value. Because its supporters were too biased to admit this directly, they submitted a control poem they knew was without value, and found the modernists couldn't tell the difference. This suggests a powerful technique for determining when something otherwise untestable might be, as Neal Stephenson calls it, bulshytte.

Perhaps Max Muller thinks Hercules is a solar deity. He will write up a argument for this proposition, and submit it for consideration before all the great mythologists of the world. Even if these mythologists want to be unbiased, they will have a difficult time of it: Muller has a prestigious reputation, and they may not have any set conception of what does and doesn't qualify as a solar deity.

What if, instead of submitting one argument, Muller submitted ten? One sincere argument for why Hercules is a solar deity, and other bogus arguments for why Perseus, Bellerophon, Theseus, et cetera are solar myths (which he has nevertheless constructs to the best of his ability). Then he instructs the mythologists "Please independently determine which of these arguments is true, and which ones I have just come up with by writing 'X is a solar deity' as my bottom line and then inventing fake justifications for the fact?" If every mythologist finds the Hercules argument most convincing, then that doesn't prove anything about Hercules but it at least shows Muller has a strong case. On the other hand, if they're all convinced by different arguments, or find none of the arguments convincing, or worst of all they all settle on Bellerophon, then Dr. Muller knows his beliefs about Hercules are quite probably wishful thinking.

This method hinges on Dr. Muller's personal honesty: a dishonest man could simply do a bad job arguing for Theseus and Bellerophon. What if we thought Dr. Muller was dishonest? We might find another mythologist whom independent observers rate as equally persuasive as Dr. Muller, and ask her to come up with the bogus arguments.

The rationalists I know sometimes take a dim view of the humanities as academic disciplines. Part of the problem is the seeming untestability of their conclusions through good, blinded experimental methods. I don't think most humanities professors are really looking all that hard for such methods. But for those who are, I consider this technique a little better than nothing².

Footnotes

1: The Sokal Affair is another related hoax. Wikipedia's Sokal Hoax page has some other excellent examples of this sort of test.

2: One more example where this method could prove useful. I remember debating a very smart Christian on the subject of Biblical atrocities. You know, stuff about death by stoning for minor crimes, or God ordering the Israelites to murder women and enslave children - that sort of thing. My friend, who was quite smart, was always able to come up with a superficially plausible excuse, and it was getting on my nerves. But having just read Your Strength as a Rationalist, I knew that being able to explain anything wasn't always a virtue. I proposed the following experiment: I'd give my friend ten atrocities commanded by random Bronze Age kings generally agreed by historical consensus to be jerks, and ten commanded by God in the Bible. His job would be to determine which ten, for whatever reason, really weren't all that bad. If he identified the ten Bible passages, that would be strong evidence that Biblical commandments only seemed atrocious when misunderstood. But if he couldn't tell the difference between God and Ashurbanipal, that would prove God wasn't really that great. To my disgust, my friend knew his Bible so well that I couldn't find any atrocities he wasn't already familiar with. So much for that technique. I offer it to anyone who debates theists with less comprehensive knowledge of Scripture.

18 comments

Comments sorted by top scores.