2012 Survey Results

Hi Yvain,

please state a definite end date next year. Filling out the survey didn't have a really high priority for me, but knowing that I had "about a month" made me put it off. Had I known that the last possible day was the 26th of November, I probably would have fit it in sometime in between other stuff.

The calibration question is an n=1 sample on one of the two important axes (those axes being who's answering, and what question they're answering). Give a question that's harder than it looks, and people will come out overconfident on average; give a question that's easier than it looks, and they'll come out underconfident on average. Getting rid of this effect requires a pool of questions, so that it'll average out.

I previously mentioned that item non-response might be a good measure of Conscientiousness. Before doing anything fancy with non-response, I first checked that there was a correlation with the questionnaire reports. The correlation is zero:

R> lwc <- subset(lw, !is.na(as.integer(as.character(BigFiveC))))
R> missing_answers <- apply(lwc, 1, function(x) sum(sapply(x, function(y) is.na(y) || as.character(y)==" ")))
R> cor.test(as.integer(as.character(lwc$BigFiveC)), missing_answers)

    Pearson's product-moment correlation

data:  as.integer(as.character(lwc$BigFiveC)) and missing_answers
t = -0.0061, df = 421, p-value = 0.9952
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 -0.09564  0.09505
sample estimates:
       cor
-0.0002954
# visualize to see if we made some mistake somewhere
R> plot(as.integer(as.character(lwc$BigFiveC)), missing_answers)

I am completely surprised. The results in the economics paper looked great and the rationale is very plausible. Yet... The 2 sets of data here have the right ranges, there's plenty of variation in both dimension, I'm sure I'm catching most of the item non-responses or NAs given that there are non-responses as high as 34, there's a lot of datapoints, and it's not that the correlation is the opposite direction which might indicate a coding error but that there's none at all. Yvain questions the Big Five results, but otherwise they look exactly as I would've predicted before seeing the results: low C and E and A, high O, medium N.

There may be something very odd about LWers and Conscientiousness; when I try C vs Income, there's a almost-zero correlation again:

R> cor.test(as.integer(as.character(lwc$BigFiveC)), log1p(as.integer(lwc$Income)))

    Pearson's product-moment correlation

data:  as.integer(as.character(lwc$BigFiveC)) and log1p(as.integer(lwc$Income))
t = 0.2178, df = 421, p-value = 0.8277
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 -0.08482  0.10585
sample estimates:
    cor
0.01061

I guess the next step is a linear model on income vs age, Conscientiousness, and IQ:

lwc <- subset(lw, !is.na(as.integer(as.character(BigFiveC)))))
lwc <- subset(lw, !is.na(as.integer(as.character(Age))))
lwc <- subset(lw, !is.na(as.integer(as.character(IQ))))
lwc <- subset(lw, !is.na(as.integer(as.character(Income))))
c <- as.integer(as.character(lwc$BigFiveC))
age <- as.integer(as.character(lwc$Age))
iq <- as.integer(as.character(lwc$IQ))
income <- log1p(as.integer(as.character(lwc$Income)))
summary(lm(income ~ (age + iq + c)))

Call:
lm(formula = income ~ (age + iq + c))

Residuals:
   Min     1Q Median     3Q    Max
-8.762 -0.849  1.191  2.319  3.644

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  -0.5531     3.5479   -0.16     0.88
age           0.1311     0.0323    4.06  9.5e-05
iq            0.0339     0.0267    1.27     0.21
c             0.0174     0.0121    1.44     0.15

Residual standard error: 3.35 on 106 degrees of freedom
  (489 observations deleted due to missingness)
Multiple R-squared: 0.196,    Adjusted R-squared: 0.173
F-statistic: 8.59 on 3 and 106 DF,  p-value: 3.73e-05

So all of them combined don't explain much and most of the work is being done by the age variable... There's many high-income LWers, supposedly (in this subset of respondents reporting age, income, IQ, and Conscientiousness, the max is 700,000), so I'd expect a cumulative r^2 of more than 0.173 for all 3 variables; if those aren't governing income, what is? Maybe everyone working with computers is rich and the others poor? Let's look at everyone who submitted salary and profession and see whether the practical computer people are making bank:

lwi <- subset(lw, !is.na(as.integer(as.character(Income))))
lwi <- subset(lwi, !is.na(as.character(Profession)))
cs <- as.integer(as.character(lwi[as.character(lwi$Profession)=="Computers (practical: IT, programming, etc.)",]$Income))
others <- as.integer(as.character(lwi[as.character(lwi$Profession)!="Computers (practical: IT, programming, etc.)",]$Income))
# ordinary t-test, but we'll exclude anyone with zero income (unemployed?)
t.test(cs[cs!=0],others[others!=0])

    Welch Two Sample t-test

data:  cs[cs != 0] and others[others != 0]
t = 5.905, df = 309.3, p-value = 9.255e-09
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 22344 44673
sample estimates:
mean of x mean of y
    76458     42950

Wow. Just wow. 76k vs 43k. I mean, maybe this would go away with enough fiddling (eg. cost-of-living) but it's still dramatic. This suggests a new theory to me: maybe Conscientiousness does correlate with income at its usual high rate for everyone but computer people who are simply in so high demand that lack of Conscientiousness doesn't matter:

R> lwi <- subset(lw, !is.na(as.integer(as.character(Income))))
R> lwi <- subset(lwi, !is.na(as.character(Profession)))
R> lwi <- subset(lwi, !is.naBigFiveC)))))
R> cs <- lwi[as.character(lwi$Profession)=="Computers (practical: IT, programming, etc.)",]
R> others <- lwi[as.character(lwi$Profession)!="Computers (practical: IT, programming, etc.)",]
R> cor.test(as.integer(as.character(cs$BigFiveC)), as.integer(as.character(cs$Income)))

    Pearson's product-moment correlation

data:  as.integer(as.character(cs$BigFiveC)) and as.integer(as.character(cs$Income))
t = 0.5361, df = 87, p-value = 0.5933
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 -0.1527  0.2625
sample estimates:
    cor
0.05738
R> cor.test(as.integer(as.character(others$BigFiveC)), as.integer(as.character(others$Income)))

    Pearson's product-moment correlation

data:  as.integer(as.character(others$BigFiveC)) and as.integer(as.character(others$Income))
t = 1.997, df = 200, p-value = 0.04721
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 0.001785 0.272592
sample estimates:
   cor
0.1398

So for the CS people the correlation is small and non-statistically-significant, for non-CS people the correlation is almost 3x larger and statistically-significant.

R> lwc <- subset(lw, !is.na(as.integer(as.character(BigFiveC))))
R> missing_answers1 <- apply(lwc, 1, function(x) sum(sapply(x, function(y) is.na(y) || as.character(y)==" ")))
R> lwnc <- subset(lw, is.na(as.integer(as.character(BigFiveC))))
R> missing_answers2 <- apply(lwnc, 1, function(x) sum(sapply(x, function(y) is.na(y) || as.character(y)==" ")))
R> t.test(missing_answers1, missing_answers2)

    Welch Two Sample t-test

data:  missing_answers1 and missing_answers2
t = -25.19, df = 806.8, p-value < 2.2e-16
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -18.77 -16.05
sample estimates:
mean of x mean of y
    9.719    27.129

I really have no idea what went so wrong [with the question about Bayes' birth year]

Note also that in the last two surveys the mean and median answers were approximately correct, whereas this time even the first quartile answer was too late by almost a decade. So it's not just a matter of overconfidence -- there also was a systematic error. Note that Essay Towards Solving a Problem in the Doctrine of Chances was published posthumously when Bayes would have been 62; if people estimated the year it was published and assumed that he had been approximately in his thirties (as I did), that would explain half of the systematic bias.

On IQ Accuracy:

As Yvain says, "people have been pretty quick to ridicule this survey's intelligence numbers as completely useless and impossible and so on" because if they're true, it means that the average LessWronger is gifted. Yvain added a few questions to the 2012 survey, including the ACT and SAT questions and the Myers-Briggs personality type question that I requested (I'll explain why this is interesting), and that give us a few other things to check against, which has made the figures more believable. The ridicule may be an example of the "virtuous doubt" that Luke warns about in Overconfident Pessimism, so it makes sense to "consider the opposite":

The distribution of Myers-Briggs personality types on LessWrong replicates the Mensa pattern. This is remarkable since the patterns of personality types here are, in many significant ways, the exact opposite of what you'd find in the regular population. For instance, the introverted rationalists and idealists are each about 1% of the population. Here, they are the majority and it's the artisans and guardians who are relegated to 1% or less of our population.

Mensa's personality test results were published in the December 1993 Mensa Bulletin. Their numbers.

So, if you believe that most of the people who took the survey lied about their IQ, you also need to believe all of the following:

• That most of these people also realized they needed to do IQ correlation research and fudge their SAT and ACT scores in order for their IQ lie to be believable.

• Some explanation as to why the average of lurker's IQ scores would come out so close to the average of poster's IQ scores. The lurkers don't have karma to show off, and there's no known incentive good enough to get so many lurkers to lie about their IQ score. Vaniver's figures.

• Some explanation for why the personality type pattern at LessWrong is radically different from the norm and yet very similar to the personality type pattern Mensa published and also matched my predictions. Even if they had knowledge of the Mensa personality test results and decided to fudge their personality type responses, too, they somehow managed to fudge them in such a way that their personality types accidentally matched my predictions.

• That they decided not to cheat when answering the Bayes birthday question even though they were dishonest enough to lie on the IQ question, motivated to look intelligent, and it takes a lot less effort to fudge the Bayes question than the intelligence and personality questions. (This was suggested by ArisKatsaris).

• That both posters and lurkers had some motive strong enough to justify spending 20+ minutes doing the IQ correlation research and fudging personality test questions while probably bored of ticking options after filling out most of a very long survey.

It's easier just to put the real number in the IQ box than do all that work to make it believable, and it's not like the liars are likely to get anything out of boasting anonymously, so the cost-benefit ratio is just not working in favor of the liar explanation.

If you think about it in terms of Occam's razor, what is the better explanation? That most people lied about their IQ, and fudged their SAT, ACT and personality type data to match, or that they're telling the truth?

Summary of criticism:

Possible Motive to Lie: The desire to be associated with a "gifted" group:

In re to this post, it was argued by NonComposMentis that a potential motive to lie is that if the outside world perceives LessWrong as gifted, then anyone having an account on LessWrong will look high-status. In rebuttal:

• I figure that lurkers would not be motivated to fudge their results because they don't have a bunch of karma on their account to show off and anybody can claim to read LessWrong, so fudging your IQ just to claim that the site you read is full of gifted people isn't likely to be motivating. I suggested that we compare the average IQs of lurkers and others. Vaniver did the math and they are very, very close..

• I argued, among other things, that it would be falling for a Pascal's mugging to believe that investing the extra time (probably at least $5 worth of time for most of us) into fudging the various different survey questions is likely to contribute to a secret conspiracy to inflate LessWrong's average IQ.

Did the majority avoid filling out intelligence related questions, letting the gifted skew the results?

Short answer: 74% of people answered at least one intelligence related question and since most people filled out only one or two, the fact that the self-report, ACT and SAT score averages are so similar is remarkable.

I realized, while reading Vaniver's post that if only 1/3 of the survey participants filled out the IQ score, this may have been due to something which could have skewed the results toward the gifted range, for instance, if more gifted people had been given IQ tests for schooling placement (and the others didn't post their IQ score because they did not know it) or if the amount of pride one has in their IQ score has a significant influence on whether one reported it.

So I went through the data and realized that most of the people who filled out the IQ test question did not fill out all the others. That means that 804 people (74% not 33%) answered at least one intelligence related question. As we have seen, the IQ correlations for the IQ, SAT and ACT questions were very close to each other (unsurprisingly, it looks like something's up with the internet test... removing those, it's 63% of survey participants that answered an intelligence related question). It's remarkable in and of itself that each category of test scores generated an average IQ so similar to the others considering that different people filled them out. I mean if 1/3 of the population filled out all of the questions, and the other 2/3 filled out none, we could say "maybe the 1/3 did IQ correlation research and fudged these" but if most of the population fills out one or two, and the averages for each category come out close to the averages for the other categories, why is that? How would that happen if they were fudging?

It does look to me like people gave whatever test scores they had and that not all the people had test scores to give but it does not look to me like a greater proportion of the gifted people provided an intelligence related survey answer. Instead it looks like most people provided an intelligence related survey answer and the average LessWronger is gifted.

Exploration of personality test fudging:

Erratio and I explored how likely it is that people could successfully fudge their personality tests and why they might do that.

• There are a lot of questions on the personality test that have an obvious intelligence component, so it's possible that people chose the answer they thought was most intelligent.

• There are also intelligence related questions where it's not clear which answer is most intelligent. I listed those.

• The intelligence questions would mostly influence the sensing/intuition dichotomy and the thinking/feeling dichotomy. This does not explain why the extraversion/introversion and perceiving/judging results were similar to Mensa's.

 E/I | S/N | T/F | J/P  (Category)

----------------------------------------------

75/25 75/25 55/45 50/50 (Overall population)

27/73 10/90 75/25 65/35 (Mensans)

15/85  03/97 88/12 54/46 (LessWrongers) *

• "Robot god apocalypse cult spinoff from Harry Potter."

That should be on a T-shirt.

But I am skeptical of these numbers. I hang out with some people who are very closely associated with the greater Less Wrong community, and a lot of them didn't know about the survey until I mentioned it to them in person. I know some people who could plausibly be described as focusing their lives around the community who just never took the survey for one reason or another. One lesson of this survey may be that the community is no longer limited to people who check Less Wrong very often, if at all. One friend didn't see the survey because she hangs out on the #lesswrong channel more than the main site. Another mostly just goes to meetups. So I think this represents only a small sample of people who could justly be considered Less Wrongers.

Yeah, this also fits my observations--I suspect that reading LW and hanging out with LW types in real life are substitute goods.

Some of the 'descriptions of LessWrong' can make for a great quote on the back of Yudkowsky's book.

Before even reading the full details, I want to congratulate to you for the impressive amount of work. The survey period is possibly my favorite time of the year on lesswrong!

EDIT: The links for the raw csv/xls data at the bottom don't seem to work for me.

Top 100 Users' Data, aka Karma 1000+

I was thinking about the fact that there is probably a difference between active LWers versus lurkers or newbies. So I looked at the data for the Top 100 users (actually Top 107, because there was a tie). This happily coincided with the nice Schelling point of 1000 karma. (make sense, because people are likely to round to that number.) To me, this reads as "has been actively posting for at least a year".

So, some data on 1000+ karma people:

Slightly more likely to be male:
92.5% Male, 7.4% Female

NOTE: First percentage is for 1000+ users, second number is for all survey respondents

Much more likely to be polyamorous:

Prefer mono 36% v. 54%
Prefer poly 24% v. 13%
Uncertain 33% v. 30%
Other 4% v. 2%

About the same Age:
average 28.6 v. 27.8

About as likely to be single
51% v. 53%

Equally likely to be vegetarian
12%

Much more likely to use modafinil at least once per month:
15% v. 4%

About equal on intelligence tests

SAT out of 1600: 1509 v. 1486
SAT out of 2400: 2260 v. 2319
Self reported IQ: 138.5 v. 138.7
online IQ test: 127 v. 126
ACT score: 33.3 v. 32.7

Similar income
50k

Slightly lower Autism quotient:
average 22 v. 24

More likely to choose torture
Torture: 42% v. 22%
Dust Specks: 29% v. 37%

More likely to cooperate in a Prisoner's Dilemma:
Cooperate: 36% v. 27%
Defect: 20% v. 29%

Some notes: Yes, I realize my data analysis methods are not the best. Namely, that instead of comparing the people with >1000 karma to the people with <100 karma, which would have been more accurate, I just compared them to the overall results (which includes their answers). I did this because it takes much less time.

Also, a hint for other people playing with the data in Excel format: A lot of the numbers are in text format, and a pain to convert to numeric format in a way that allows you to manipulate them. The easiest work around (so long as you don't want to do anything complicated) is to just paste the needed columns either into google spreadsheet, or into another Excel sheet that's been formatted numerically. If you want to do something complicated you probably need to find the "right" way to fix it.

"Eliezer Yudkowsky personality cult."
"The new thing for people who would have been Randian Objectivists 30 years ago."
"A sinister instrument of billionaire Peter Thiel."

Nope, no one guessed whose sinister instrument this site is. Muaha.

So I suggest that we now have pretty good, pretty believable evidence that the average IQ for this site really is somewhere in the 130s, and that self-reported IQ isn't as terrible a measure as one might think.

This still suffers from selection bias - I'd imagine that people with lower IQ are more likely to leave the field blank than people with higher IQ.

R> lwa <- subset(lw, !is.na(as.integer(ACTscoreoutof36)))
R> lwiq <- subset(lwa, !is.na(as.integer(IQ)))
R> lwiqnot <- subset(lwa, is.na(as.integer(IQ)))
R> t.test(lwiq$ACTscoreoutof36, lwiqnot$ACTscoreoutof36, alternative="less")

    Welch Two Sample t-test

data:  lwiq$ACTscoreoutof36 and lwiqnot$ACTscoreoutof36 
t = 0.5088, df = 141.9, p-value = 0.6942
alternative hypothesis: true difference in means is less than 0 
95 percent confidence interval:
   -Inf 0.7507 
sample estimates:
mean of x mean of y 
    32.68     32.50 

R> lwa <- subset(lw, !is.na(as.integer(SATscoresoutof1600)))
R> lwiq <- subset(lwa, !is.na(as.integer(IQ)))
R> lwiqnot <- subset(lwa, is.na(as.integer(IQ)))
R> t.test(lwiq$SATscoresoutof1600, lwiqnot$SATscoresoutof1600, alternative="less")

    Welch Two Sample t-test

data:  lwiq$SATscoresoutof1600 and lwiqnot$SATscoresoutof1600 
t = -1.137, df = 237.4, p-value = 0.1284
alternative hypothesis: true difference in means is less than 0 
95 percent confidence interval:
  -Inf 6.607 
sample estimates:
mean of x mean of y 
     1476      1490 

R> lwa <- subset(lw, !is.na(as.integer(SATscoresoutof2400)))
R> lwiq <- subset(lwa, !is.na(as.integer(IQ)))
R> lwiqnot <- subset(lwa, is.na(as.integer(IQ)))
R> t.test(lwiq$SATscoresoutof2400, lwiqnot$SATscoresoutof2400, alternative="less")
    Welch Two Sample t-test

data:  lwiq$SATscoresoutof2400 and lwiqnot$SATscoresoutof2400 
t = -0.9645, df = 129.9, p-value = 0.1683
alternative hypothesis: true difference in means is less than 0 
95 percent confidence interval:
  -Inf 109.3 
sample estimates:
mean of x mean of y 
     2221      2374 

R> lwa <- subset(lw, !is.na(as.integer(ACTscoreoutof36)))
R> lwsat <- subset(lwa, !is.na(as.integer(SATscoresoutof1600)))
R> cor.test(lwsat$SATscoresoutof1600, lwsat$ACTscoreoutof36)

    Pearson's product-moment correlation

data:  lwsat$SATscoresoutof1600 and lwsat$ACTscoreoutof36 
t = 8.839, df = 66, p-value = 8.415e-13
alternative hypothesis: true correlation is not equal to 0 
95 percent confidence interval:
 0.6038 0.8291 
sample estimates:
   cor 
0.7362 

Or we could just be really horrible. If we haven't even learned to avoid the one bias that we can measure super well and which is most susceptible to training, what are we even doing here?

You're fun to read. Posts explaining things and introducing terms that connect subjects and form patterns trigger reward mechanisms in the brain. This is uncorrelated to actually applying any lessons in daily life.

Two questions you might want to ask next year is "do you think it is practical and advantageous to reduce people's biases via standardized exercises?" and "Has reading LW inspired you to try and reduce your own biases?"

If we haven't even learned to avoid the one bias that we can measure super well and which is most susceptible to training, what are we even doing here?

This sounds like a job for cognitive psychology!

"Well-calibrated" should probably be improved to "well-calibrated about X"-- it's plausible that people have better and worse calibration about different subjects, and the samples in the survey only explored a tiny part of calibration space.

The 2011 survey ran 33 days and collected 1090 responses. This year's survey ran 23 days and collected 1195 responses.

Why did you close it early? That seems entirely unnecessary.

One friend didn't see the survey because she hangs out on the #lesswrong channel more than the main site.

I put a link and exhortation prominently in the #lesswrong topic from the day the survey opened to the day it closed.

M (trans f->m): 3, 0.3% / F (trans m->f): 16, 1.3%

3 vs 16 seems like quite a difference, even allowing for the small sample size. Is this consistent with the larger population?

Prefer polyamorous: 155, 13.1%...NUMBER OF CURRENT PARTNERS:... [>1 partners = 4.5%]

So ~3x more people prefer polyamory than are actually engaged in it...

Referred by HPMOR: 262, 22.1%

Impressive.

gwern.net: 5 people

Woot! And I'm not even trying or linking LW especially often.

(I am also pleased by the nicotine and modafinil results, although you dropped a number in 'Never: 76.5%')

TROLL TOLL POLICY: Disapprove: 194, 16.4% Approve: 178, 15%

So more people are against than for. Not exactly a mandate for its use.

Are people who understand quantum mechanics are more likely to believe in Many Worlds? We perform a t-test, checking whether one's probability of the MWI being true depends on whether or not one can solve the Schrodinger Equation. People who could solve the equation had on average a 54.3% probability of MWI, compared to 51.3% in those who could not. The p-value is 0.26; there is a 26% probability this occurs by chance. Therefore, we fail to establish that people's probability of MWI varies with understanding of quantum mechanics.

Sounds like you did a two-tailed test. shminux's hypothesis, which he has stated several times IIRC, is that people who can solve it will not be taken in by Eliezer's MWI flim-flam, as it were, and would be less likely to accept MWI. So you should've been running a one-tailed t-test to reject the hypothesis that the can-solvers are less MWI'd. The p-value would then be something like 0.13 by symmetry.

R> lwtroll <- lw[!is.na(lw$KarmaScore),]
R> lwtroll <- lwtroll[lwtroll$TrollToll=="Agree with toll" | lwtroll$TrollToll=="Disagree with toll",]
R> # disagree=3, agree=2; so:
R> # if positive correlation, higher karma associates with disagreement
R> # if negative correlation, higher karma associates with agreement
R> # we are testing hypothesis higher karma = lower score/higher agreement
R> cor.test(as.integer(lwtroll$TrollToll), lwtroll$KarmaScore, alternative="less")

    Pearson's product-moment correlation

data:  as.integer(lwtroll$TrollToll) and lwtroll$KarmaScore 
t = 1.362, df = 315, p-value = 0.9129
alternative hypothesis: true correlation is less than 0 
95 percent confidence interval:
 -1.0000  0.1679 
sample estimates:
    cor 
0.07653
R> # a log-transform of the karma scores doesn't help:
R> cor.test(as.integer(lwtroll$TrollToll), log1p(lwtroll$KarmaScore), alternative="less")

    Pearson's product-moment correlation

data:  as.integer(lwtroll$TrollToll) and log1p(lwtroll$KarmaScore) 
t = 2.559, df = 315, p-value = 0.9945
alternative hypothesis: true correlation is less than 0 
95 percent confidence interval:
 -1.0000  0.2322 
sample estimates:
   cor 
0.1427 

R> M <- as.table(rbind(c(120, 1057), c(3,16)))
R> dimnames(M) <- list(status=c("c","t"), gender=c("M","F"))
R> M
      gender
status    M    F
     c  120 1057
     t    3   16

R> chisq.test(M, simulate.p.value = TRUE, B = 20000000)

    Pearson's Chi-squared test with simulated p-value (based on 2e+07 replicates)

data:  M 
X-squared = 0.6342, df = NA, p-value = 0.4346

Thank you for this public service. It seems definitely helpful for the community, and possibly helpful for historians :-)

According to IQ Comparison Site, an SAT score of 1485/1600 corresponds to an IQ of about 144. According to Ivy West, an ACT of 33 corresponds to an SAT of 1470 (and thence to IQ of 143).

Only if you took the SAT before 1994. Here's the percentiles for SATs taken in 2012; someone who was 97th percentile would get ~760 on math and ~730 on critical reading, adding up to 1490 (leaving alone the writing section to keep it within 1600), and 97th percentile corresponds to an IQ of 128.

Here's a classic calibration chart.

An important part of the calibration chart (for people) is the frequency of times that they provide various calibrations. Looking at your table, I would focus on the large frequency between 10% and 30%.

I'll also point out that fixed windows are a pretty bad way to do elicitation. I tend to come from the calibration question from the practical side: how do we get useful probabilities out of subject-matter experts without those people being experts at calibration? Adopting those strategies seems more useful than making people experts at calibration.

Are people who understand quantum mechanics are more likely to believe in Many Worlds? We perform a t-test, checking whether one's probability of the MWI being true depends on whether or not one can solve the Schrodinger Equation. People who could solve the equation had on average a 54.3% probability of MWI, compared to 51.3% in those who could not. The p-value is 0.26; there is a 26% probability this occurs by chance. Therefore, we fail to establish that people's probability of MWI varies with understanding of quantum mechanics.

Some Bayesian analysis using the BEST MCMC library for normal two-group comparisons:

R> lw <- read.csv("lw-2012.csv")
R> 
R> lwm <- subset(lw, !(" " == as.character(SchrodingerEquation)))
R> lwm <- subset(lwm, !is.na(as.integer(as.character(PManyWorlds))))
R> mwiyes <- as.integer(as.character(subset(lwm, SchrodingerEquation == "Yes")$PManyWorlds))
R> mwino <- as.integer(as.character(subset(lwm, SchrodingerEquation == "No")$PManyWorlds))
R> 
R> source("BEST.R")
R> mcmcChain = BESTmcmc(mwino, mwiyes)
R> show(postInfo)
           SUMMARY.INFO
PARAMETER       mean  median    mode   HDIlow   HDIhigh pcgtZero
  mu1        51.1693 51.1675 51.1964  48.8152  53.48281       NA
  mu2        55.5708 55.5647 55.4871  50.2376  61.03888       NA
  muDiff     -4.4016 -4.4010 -4.1635 -10.1932   1.52931    7.154
  sigma1     30.5558 30.5355 30.4243  28.9332  32.24333       NA
  sigma2     32.8187 32.7136 32.5672  29.0621  36.73604       NA
  sigmaDiff  -2.2629 -2.1800 -2.1232  -6.4056   1.97323   14.226
  nu        106.4690 98.6244 84.7466  36.2142 194.12061       NA
  nuLog10     1.9929  1.9940  1.9853   1.6566   2.33430       NA
  effSz      -0.1389 -0.1388 -0.1323  -0.3208   0.04864    7.154

(Full size image.)

The results are interesting and not quite the same as a t-test:

1. we get estimates of standard deviations, among other things, for free - they look pretty different and there's an 85.8% chance the deviations of the Schrodinger-knowers and not-knowers are different on MWI, suggesting to me a polarizing effect where the more you know, the more extreme your view either for or against, which seems reasonable since the more information you have, the less should your uncertainty be.

2. the difference in means estimate is sharper than the t-test: Yvain's t-test gave a p-value of 0.26 if the null hypothesis were true (he makes the classic error when he says "there is a 26% probability this occurs by chance" - no, there's a 26% chance this happened by chance if one assumes the null hypothesis is true, which says absolutely nothing about whether this happened by chance).

   We, however, by using Bayesian techniques can say that given the difference in mean beliefs: there is a 7.2% chance that the null hypothesis (equal belief) or the opposite hypothesis (lower belief) is true in this sample.

   We also get an effect-size for free from the difference in means. -0.132 (mode) isn't too impressive, but it's there.

However, both BEST and the t-test are normal tests. The histograms look like the data may be a bimodal distribution: a hump of skeptics at 10%, a hump of believers in the 70%s - and the weirdly low 40s in both groups is just a low point in both? I don't know how much of an issue this is.

When you discuss the calibration results, could you mention that the surveyors were told what constituted a correct answer? I didn't take the survey and it isn't obvious from reading this post. Also, could you include a plug for PredictionBook around there? You've included lots of other helpful plugs.

Another result: no correlation between autism score and consequentialism endorsement.

R> lw <- read.csv("2012.csv")
R> lwr <- subset(lw, as.character(FamilyReligion) != " ")
R> lwr <- subset(lwr, as.character(MoralViews) != " " & as.character(MoralViews) != "Other / no answer")
R> levels(lwr$FamilyReligion); levels(lwr$MoralViews)
[1] " "                         "Agnostic"                  "Atheist and not spiritual"
[4] "Atheist but spiritual"     "Committed theist"          "Deist/Pantheist/etc"
[7] "Lukewarm theist"           "Mixed / Other"
[1] " "                                     "Accept / lean toward consequentialism"
[3] "Accept / lean toward deontology"       "Accept / lean toward virtue ethics"
[5] "Other / no answer"
R>
R> cor.test(as.integer(lwr$FamilyReligion), as.integer(lwr$MoralViews))

    Pearson's product-moment correlation

data:  as.integer(lwr$FamilyReligion) and as.integer(lwr$MoralViews)
t = -0.6631, df = 858, p-value = 0.5075
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 -0.08935  0.04429
sample estimates:
     cor
-0.02263

R> fr <- sapply(as.integer(lwr$FamilyReligion), function(x) if(x>4) {1} else {0})
R> mv <- sapply(as.integer(lwr$MoralViews), function(x) if(x>2) {1} else {0})
R> ct <- chisq.test(fr,mv); ct

    Pearson's Chi-squared test with Yates' continuity correction

data:  fr and mv
X-squared = 2e-04, df = 1, p-value = 0.9894
R> ct$expected; ct$observed
   mv
fr      0      1
  0 200.6  58.43
  1 465.4 135.57
   mv
fr    0   1
  0 200  59
  1 466 135

Yvain, I rechecked the calibration survey results, and encourage someone to recheck my recheck further:

First, these strata overlap... is 5 in 0-5 or 5-15? The N I doesn't actually match either one get either one when I recheck.

Secondly, I am not sure what program you used to calculate the statistics, but when I checked in excel, some people used percentages that got pulled as numbers less than one. I tried to clean that for these. (also removed someone who answered 150.)

Thirdly, there are 20 people in this N. You can be either 60% correct (12 correct), or 65% correct (13 correct), but 60.2% correct in this line seems weird. 85-95: 60.2% [n = 20]

Here was my attempt at recalculating those figures: N after data cleaning was 998.

0-<5: 9.1% [n = 2/22]

5-<15: 13.7% [n = 25/183]

15-<25: 9.3% [n = 21/226]

25-<35: 10% [n = 20/200]

35-<45: 11.1% [n = 10/90]

45-<55: 17.3% [n = 19/110]

55-<65: 20.8% [n = 11/53]

65-<75: 22.6% [n = 7/31]

75-<85: 36.7% [n = 11/30]

85-<95: 63.2% [n = 12/19]

95-100: 88.2% [n = 30/34]

I express low confidence in these remarks because I haven't rechecked this or gone into detail about data cleaning, but my brief take is:

1: Yes, there were some errors that made it look a bit worse than it was.

2: It's still shows overconfidence. (Edit: see possible caveat below)

Question: Do we have enough data to determine if that hump at near 10% confidence that you are right is significant?

Edit: I'm not a statistician, but I do notice there appears to be substantially more N that answered in the lower confidence ranges. I mean, yes, on average, the people who answered in those high 55-<85 ranges were quite far off, but there were more people than answered in the 15-<25 range then all of those three groups put together.

In the fair coin questions, there were two people answering 49.9, one 49.9999, one 49.999999, and one 51. :-/

Other Christian: 517, 43.6% Catholic: 295, 24.9%

Now that I think about that, lumping Protestants and Orthodoxes together and keeping Catholics separate is about as bizarre as it gets.

Pandemic (bioengineered): 272, 23% Environmental collapse: 171, 14.5% Unfriendly AI: 160, 13.5% Nuclear war: 155, 13.1% Economic/Political collapse: 137, 11.6% Pandemic (natural): 99, 8.4% Nanotech: 49, 4.1% Asteroid: 43, 3.6%

This is one question where the results really suprised me. Combining natural and engendered pandemics, almost a third of respondents picked it as the top near term X risk which was almost twice as many as the next highest risk. I wonder if the x risk discussions we tend to have may be somewhat misallocated.

This survey looks like it was a massive amount of work to analyse. Three cheers for Yvain!

These are the results of the CFAR questions; I have also posted this as its own Discussion section post.

SUMMARY: The CFAR questions were all adapted from the heuristics and biases literature, based on five different cognitive biases or reasoning errors. LWers, on the whole, showed less bias than is typical in the published research (on all 4 questions where this was testable), but did show clear evidence of bias on 2-3 of those 4 questions. Further, those with closer ties to the LW community (e.g., those who had read more of the sequences) showed significantly less bias than those with weaker ties (on 3 out of 4-5 questions where that was testable). These results all held when controlling for measures of intelligence.

METHOD & RESULTS

Being less susceptible to cognitive biases or reasoning errors is one sign of rationality (see the work of Keith Stanovich & his colleagues, for example). You'd hope that a community dedicated to rationality would be less prone to these biases, so I selected 5 cognitive biases and reasoning errors from the heuristics & biases literature to include on the LW survey. There are two possible patterns of results which would point in this direction:

• high scores: LWers show less bias than other populations that have answered these questions (like students at top universities)
• correlation with strength of LW exposure: those who have read the sequences (have been around LW a long time, have high karma, attend meetups, make posts) score better than those who have not.

The 5 biases were selected in part because they can be tested with everyone answering the same questions; I also preferred biases that haven't been discussed in detail on LW. On some questions there is a definitive wrong answer and on others there is reason to believe that a bias will tend to lead people towards one answer (so that, even though there might be good reasons for a person to choose that answer, in the aggregate it is evidence of bias if more people choose that answer).

This is only one quick, rough survey. If the results are as predicted, that could be because LW makes people more rational, or because LW makes people more familiar with the heuristics & biases literature (including how to avoid falling for the standard tricks used to test for biases), or because the people who are attracted to LW are already unusually rational (or just unusually good at avoiding standard biases). Susceptibility to standard biases is just one angle on rationality. Etc.

Here are the question-by-question results, in brief. The comment below contains the exact text of the questions, and more detailed explanations.

Question 1 was a disjunctive reasoning task, which had a definitive correct answer. Only 13% of undergraduates got the answer right in the published paper that I took it from. 46% of LWers got it right, which is much better but still a very high error rate. Accuracy was 58% for those high in LW exposure vs. 31% for those low in LW exposure. So for this question, that's:

1. LWers biased: yes
2. LWers less biased than others: yes
3. Less bias with more LW exposure: yes

Question 2 was a temporal discounting question; in the original paper about half the subjects chose money-now (which reflects a very high discount rate). Only 8% of LWers did; that did not leave much room for differences among LWers (and there was only a weak & nonsignificant trend in the predicted direction). So for this question:

1. LWers biased: not really
2. LWers less biased than others: yes
3. Less bias with more LW exposure: n/a (or no)

Question 3 was about the law of large numbers. Only 22% got it right in Tversky & Kahneman's original paper. 84% of LWers did: 93% of those high in LW exposure, 75% of those low in LW exposure. So:

1. LWers biased: a bit
2. LWers less biased than others: yes
3. Less bias with more LW exposure: yes

Question 4 was based on the decoy effect aka asymmetric dominance aka attraction effect (but missing a control condition). I don't have numbers from the original study (and there is no correct answer) so I can't really answer 1 or 2 for this question, but there was a difference based on LW exposure: 57% vs. 44% selecting the less bias related answer.

1. LWers biased: n/a
2. LWers less biased than others: n/a
3. Less bias with more LW exposure: yes

Question 5 was an anchoring question. The original study found an effect (measured by slope) of 0.55 (though it was less transparent about the randomness of the anchor; transparent studies w. other questions have found effects around 0.3 on average). For LWers there was a significant anchoring effect but it was only 0.14 in magnitude, and it did not vary based on LW exposure (there was a weak & nonsignificant trend in the wrong direction).

1. LWers biased: yes
2. LWers less biased than others: yes
3. Less bias with more LW exposure: no

One thing you might wonder: how much of this is just intelligence? There were several questions on the survey about performance on IQ tests or SATs. Controlling for scores on those tests, all of the results about the effects of LW exposure held up nearly as strongly. Intelligence test scores were also predictive of lower bias, independent of LW exposure, and those two relationships were almost the same in magnitude. If we extrapolate the relationship between IQ scores and the 5 biases to someone with an IQ of 100 (on either of the 2 IQ measures), they are still less biased than the participants in the original study, which suggests that the "LWers less biased than others" effect is not based solely on IQ.

ALTERNATIVE POLITICS QUESTION: Progressive: 429, 36.3% Libertarian: 278, 23.5% Reactionary: 30, 2.5% Conservative: 24, 2% Communist: 22, 1.9% Other: 156, 13.2%

I'd like to note that my suggestion as I offered it didn't include an "Other" option -- you added that one by yourself, and it ended up being selected by more people than "Reactionary" "Conservative" and "Communist" combined. My suggested question would have forced the current "Others" to choose between the five options provided or not answer at all.

Fishing for correlations is a statistically dubious practice, but also fun. Some interesting ones (none were very high, except e.g. Father Age and Mother Age):

• IQ and Hours Writing have correlation 0.26 (75 degrees), which is the only interesting IQ correlation.
• Siblings and Older siblings have correlation 0.48 (61 degrees), which isn't too surprising , but makes me wonder: do we expect this correlation to be 0.5 in general?
• Most of the Big Five answers are slightly correlated (around +/-0.25, or 90+/-15 degrees) with each other, but not with anything else except the Autism Score. Shouldn't well-designed personality traits be orthogonal, ideally?
• CFAR question 7 (guess of height of redwood) was negatively correlated with Height (-0.23, or 103 degrees). No notable correlation with the random number, though.

I was surprised to see that LW has almost as many socialists as libertarians. I had thought due to anecdotal evidence that the site was libertarian-dominated.

I was also surprised that a plurality of people preferred dust specks to torture, given that it appears to be just a classic problem of scope insensitivity, which this site talks about repeatedly.

I was happy to see that we have more vegetarians and fewer smokers than the general population.

R> lw <- read.csv("lw-survey/2012.csv")
R> hpmor <- lw[as.character(lw$Referrals) == "Referred by Harry Potter and the Methods of Rationality",]
R> hpmor <- lw[as.character(lw$Referrals) != "Referred by Harry Potter and the Methods of Rationality",]
R> t.test(hpmor$IQ, hpmor$IQ)

    Welch Two Sample t-test

data:  hpmor$IQ and hpmor$IQ
t = 0.5444, df = 99.28, p-value = 0.5874
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -2.614  4.591
sample estimates:
mean of x mean of y
    139.1     138.1
R> t.test(as.integer(as.character(hpmor$IQTest)), as.integer(as.character(hpmor$IQTest)))

    Welch Two Sample t-test

data:  as.integer(as.character(hpmor$IQTest)) and as.integer(as.character(hpmor$IQTest))
t = -0.0925, df = 264.8, p-value = 0.9264
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -2.802  2.551
sample estimates:
mean of x mean of y
    125.6     125.8
R> t.test(as.numeric(as.character(hpmor$Income)), as.numeric(as.character(hpmor$Income)))

    Welch Two Sample t-test

data:  as.numeric(as.character(hpmor$Income)) and as.numeric(as.character(hpmor$Income))
t = -4.341, df = 314.3, p-value = 1.917e-05
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -29762 -11197
sample estimates:
mean of x mean of y
    33948     54427
R> t.test(hpmor$Age, hpmor$Age)

    Welch Two Sample t-test

data:  hpmor$Age and hpmor$Age
t = -7.033, df = 484.4, p-value = 6.93e-12
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -5.318 -2.995
sample estimates:
mean of x mean of y
    24.51     28.67
R> t.test(as.character(hpmor$WorkStatus) == "Student", as.character(hpmor$WorkStatus) == "Student")

    Welch Two Sample t-test

data:  as.character(hpmor$WorkStatus) == "Student" and as.character(hpmor$WorkStatus) == "Student"
t = 4.154, df = 389.8, p-value = 4.018e-05
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 0.0791 0.2213
sample estimates:
mean of x mean of y
   0.5224    0.3723

R> hpmortime <- hpmor$KarmaScore / as.numeric(as.character(hpmor$TimeinCommunity))
R> hpmortime <-  hpmortime[!is.na(hpmortime) & !is.nan(hpmortime) & !is.infinite(hpmortime) ]
R> hpmortime <- hpmor$KarmaScore / as.numeric(as.character(hpmor$TimeinCommunity))
R> hpmortime <-  hpmortime[!is.na(hpmortime) & !is.nan(hpmortime) & !is.infinite(hpmortime) ]
R> t.test(hpmortime, hpmortime)

    Welch Two Sample t-test

data:  hpmortime and hpmortime
t = 1.05, df = 642.7, p-value = 0.2942
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -4.257 14.036
sample estimates:
mean of x mean of y
    17.69     12.80

R> hpmortime <- log1p(hpmor$KarmaScore / as.numeric(as.character(hpmor$TimeinCommunity)))
R> hpmortime <-  hpmortime[!is.na(hpmortime) & !is.nan(hpmortime) & !is.infinite(hpmortime) ]
R> hpmortime <- log1p(hpmor$KarmaScore / as.numeric(as.character(hpmor$TimeinCommunity)))
R> hpmortime <-  hpmortime[!is.na(hpmortime) & !is.nan(hpmortime) & !is.infinite(hpmortime) ]
R> t.test(hpmortime, hpmortime)
    Welch Two Sample t-test

data:  hpmortime and hpmortime
t = 2.263, df = 396.9, p-value = 0.02416
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 0.03366 0.47878
sample estimates:
mean of x mean of y
   1.1978    0.9415

R> sort(hpmor$KarmaScore, decreasing=T)[1:25]
 [1] 9122 6815 4887 4500 2782 2600 2545 2117 2000 1800 1300 1017 1000 1000  858  771  694  575  560
[20]  443  425  422  350  285  274
R> sort(other$KarmaScore, decreasing=T)[1:83]
 [1] 47384 32394 27418 15000 12200 11094 11000 10000  9000  8799  8000  8000  8000  6164  5000  5000
[17]  5000  5000  4658  4000  4000  4000  3960  3800  3693  3600  3500  3500  3500  3353  3300  3000
[33]  3000  3000  3000  3000  3000  3000  3000  2700  2500  2486  2400  2300  2204  2200  2100  2000
[49]  2000  2000  2000  2000  1977  1975  1900  1800  1800  1800  1750  1700  1653  1650  1648  1600
[65]  1590  1540  1520  1500  1500  1500  1500  1500  1500  1400  1253  1250  1200  1200  1115  1095
[81]  1044  1000  1000

R> lw <- read.csv("lw-survey/2012.csv")
R>
R> hpmor <- lw[as.character(lw$Referrals) == "Referred by Harry Potter and the Methods of Rationality",]
R> other <- lw[as.character(lw$Referrals) != "Referred by Harry Potter and the Methods of Rationality",]
R>
R> hpmor <- hpmor[order(hpmor$KarmaScore, decreasing=TRUE),][1:25,]
R> other <- other[order(other$KarmaScore, decreasing=TRUE),][1:83,]
R>
R> hpmortime <- hpmor$KarmaScore / as.numeric(as.character(hpmor$TimeinCommunity))
R> hpmortime <- hpmortime[!is.na(hpmortime) & !is.nan(hpmortime) & !is.infinite(hpmortime) ]
R> othertime <- other$KarmaScore / as.numeric(as.character(other$TimeinCommunity))
R> othertime <- othertime[!is.na(othertime) & !is.nan(othertime) & !is.infinite(othertime) ]
R>
R> sort(hpmortime, decreasing=TRUE)
 [1] 506.78 300.00 283.96 203.62 138.46 133.95 117.61 115.92  72.22  66.67  59.09  50.00  36.92
[14]  35.05  35.00  33.90  28.60  26.67  24.82  23.96  20.36  19.28  17.71  11.91
R> sort(othertime, decreasing=TRUE)
 [1] 1895.36  647.88  456.97  338.89  263.16  250.00  250.00  235.71  184.90  183.33  173.91  166.67
[13]  165.00  146.93  146.65  145.83  142.86  133.33  133.33  133.33  125.00  125.00  125.00  116.45
[25]  102.73  100.00   97.22   84.38   83.33   83.33   83.33   83.33   75.00   75.00   74.51   72.00
[37]   69.60   68.88   66.67   66.67   63.33   61.11   60.71   60.34   58.33   57.14   55.95   53.43
[49]   52.17   50.00   50.00   50.00   50.00   48.48   46.46   44.12   43.75   41.67   41.43   40.00
[61]   39.66   36.36   35.91   33.33   33.33   31.67   31.32   30.00   30.00   30.00   27.50   27.47
[73]   26.95   25.33   25.00   25.00   24.06   23.33   22.73   18.25   16.67   16.67
R>
R> t.test(hpmortime,othertime)

    Welch Two Sample t-test

data:  hpmortime and othertime
t = -0.544, df = 72.4, p-value = 0.5881
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -87.52  49.99
sample estimates:
mean of x mean of y
    98.44    117.20

                | Torturevs.DustSpecks
----------------+------------------+------+---------+--------+------
MoralViews      | don't understand | dust | torture | unsure | <NA>
----------------+------------------+------+---------+--------+------
  consequential |               10 |  228 |   188   |    134 |  109
  deontology    |                1 |   24 |     3   |      8 |    6
  other/none    |                6 |   68 |    38   |     33 |   47
  virtue        |                1 |   75 |    12   |     28 |   36
  <NA>          |                0 |    2 |     2   |      0 |    8