Vegetarianism Ideological Turing Test Results

Interesting update/realization I just had:

I was one of the people putting down "40%" or "60%" type answers - despite also thinking "man, I'm really not sure here." But "40%/60%" feels like a number that's "sufficiently" uncertain to represent a rough mental state of "not really sure, but if I had to pick I'd go with meat-eater." When it fact if anything I should have been putting down 49%/51% at best, because seriously it was genuinely hard to tell.

I also remember the essays being very weird and throwing me off a bit. Like, in the normal population, vegetarians say "animals can feel pain and we shouldn't hurt or kill them" and meat eaters say "we're at the top of the food chain and it's the natural order of things for us to eat."

On less wrong, vegetarians say things like "animals feel pain, therefore factory farming is evil but also we probably should destroy all natural habitat so that wild animals can't exist or suffer. Also maybe we should just eradicate all life on earth to prevent suffering." And meat eaters say "I look forward to tasty cost effective meat grown in test tubes but right now I'm too lazy and don't care that much. Also, probably better to have slightly-better farming conditions but then continue creating billions of chickens for a net hedonic gain."

It was pretty clear that any lessons I took on identifying vegetarians/meat-eaters from this test were pretty localized.

every single judge thought themselves decently able to discern genuine writing from fakery. The numbers suggest that every single judge was wrong.

I think the first of these claims is a little too pessimistic, and the second may be too.

Here are some comments made by one of the judges (full disclosure: it was me) at the time. "I found these very difficult [...] I had much the same problem [sc. that pretty much every entry felt >50% credible]. [...] almost all my estimates were 40%-60% [...] I fear that this one [...] is just too difficult." I'm pretty sure (though of course memory is deceptive) that I would not have said that I thought myself "decently able to discern genuine writing from fakery". ("Almost all" was too strong, though, if I've correctly guessed which row in the table is mine. Four of my estimates were 70%. One was 99% but that's OK because that was my own entry, which I recognized. The others were all 40-60%. Incidentally, I got two of my four 70% guesses right and two wrong, and four of my eight 40%/60% guesses right and four wrong.)

On the second, I remark that judge 14 (full disclosure: this was definitely not me) scored better than +450 and got only two of the 13 entries wrong. The probability of any given judge getting 11/13 or better by chance is about 1%. [EDITED to add: As Douglas_Knight points out, it would be better to say 10/12 because judge 14 guessed 50% for one entry.] In a sample of 53 people you'll get someone doing this well just by chance a little over half the time. But wait, the two wrong ones were both 60/40 judgements, and judge 14 had a bunch of 70s and 80s and one 90 as well, all of them correct. With judge 14's probability assignments and random actual results, simulation (I'm too lazy to do it analytically) says that as good a logarithmic score happens only about 0.3% of the time. To figure out exactly what that says about the overall results we'd need some kind of probabilistic model for how people assign their probabilities or something, and I'm way too lazy for that, but my feeling is that judge 14's results are good enough to suggest genuinely better-than-chance performance.

If anyone wants to own up to being judge 14, I'd be extremely interested to hear what they have to say about their mental processes while judging.

Possible confounding variable: people who don't think they can tell the difference might be less likely to do the test. I remember Brienne doing something similar on facebook. I read a couple of entries, spent a while agonizing about whether I thought they were slightly more likely to be genuine or fake, got bored/frustrated and gave up.

Number 'correct' - 1  2  3  4  5  6  7  8  9  10 Grand Total
Omnivore---------- 1  0  1  5  3  8  7  3  0  1      29
Vegetarian-------- 0  0  2  1  5  4  2  0  0  0      14

I think you should distinguish between "average score across judges is close to 50%" and "every single judge is close to 50%". I suspect the latter is not true, as pointed out in one of the other comments.

I really appreciate the effort that this took and I think this is an interesting and valuable result which will influence my actions in the future.

After reading the first part of your post, I decided that I would have a 55% chance of correctly predicting the positions of Less Wrongers but a 90% chance of correctly predicting the position of an average sample of Americans. I hereby pat myself on the back for my correct calibration on the prior claim, and wonder how far off my latter claim might be. A lot of the arguments that I encounter in daily life (e.g. bar-b-ques where I refrain from eating meat) are pretty crummy.

According to the PM I got, I had the most credible vegetarian entry, and it was ranked as much more credible than my actual (meat-eating) beliefs. I'm not sure how I feel about that.

On test this hard you didn't even have an option between 50 and 60%? Weak!

Or rather, allow them to be weaker!

This is basically what I expected with an ITT on this sort of opinion, and is the reason I didn't bother to participate.