Participation in the LW Community Associated with Less Bias

Contents

  Summary
None
50 comments

Summary

CFAR included 5 questions on the 2012 LW Survey which were adapted from the heuristics and biases literature, based on five different cognitive biases or reasoning errors.  LWers, on the whole, showed less bias than is typical in the published research (on all 4 questions where this was testable), but did show clear evidence of bias on 2-3 of those 4 questions.  Further, those with closer ties to the LW community (e.g., those who had read more of the sequences) showed significantly less bias than those with weaker ties (on 3 out of 4-5 questions where that was testable).  These results all held when controlling for measures of intelligence.


METHOD & RESULTS

Being less susceptible to cognitive biases or reasoning errors is one sign of rationality (see the work of Keith Stanovich & his colleagues, for example).  You'd hope that a community dedicated to rationality would be less prone to these biases, so I selected 5 cognitive biases and reasoning errors from the heuristics & biases literature to include on the LW survey.  There are two possible patterns of results which would point in this direction:

• high scores: LWers show less bias than other populations that have answered these questions (like students at top universities) 
• correlation with strength of LW exposure: those who have read the sequences (or have been around LW a long time, have high karma, attend meetups, make posts) score better than those who have not. 

The 5 biases were selected in part because they can be tested with everyone answering the same questions; I also preferred biases that haven't been discussed in detail on LW.  On some questions there is a definitive wrong answer and on others there is reason to believe that a bias will tend to lead people towards one answer (so that, even though there might be good reasons for a person to choose that answer, in the aggregate it is evidence of bias if more people choose that answer).

This is only one quick, rough survey.  If the results are as predicted, that could be because LW makes people more rational, or because LW makes people more familiar with the heuristics & biases literature (including how to avoid falling for the standard tricks used to test for biases), or because the people who are attracted to LW are already unusually rational (or just unusually good at avoiding standard biases).  Susceptibility to standard biases is just one angle on rationality.  Etc.

Here are the question-by-question results, in brief.  The next section contains the exact text of the questions, and more detailed explanations.

Question 1 was a disjunctive reasoning task, which had a definitive correct answer.  Only 13% of undergraduates got the answer right in the published paper that I took it from.  46% of LWers got it right, which is much better but still a very high error rate.  Accuracy was 58% for those high in LW exposure vs. 31% for those low in LW exposure.  So for this question, that's: 
1. LWers biased: yes 
2. LWers less biased than others: yes 
3. Less bias with more LW exposure: yes 

Question 2 was a temporal discounting question; in the original paper about half the subjects chose money-now (which reflects a very high discount rate).  Only 8% of LWers did; that did not leave much room for differences among LWers (and there was only a weak & nonsignificant trend in the predicted direction). So for this question: 
1. LWers biased: not really 
2. LWers less biased than others: yes 
3. Less bias with more LW exposure: n/a (or no) 

Question 3 was about the law of large numbers.  Only 22% got it right in Tversky & Kahneman's original paper. 84% of LWers did: 93% of those high in LW exposure, 75% of those low in LW exposure.  So: 
1. LWers biased: a bit 
2. LWers less biased than others: yes 
3. Less bias with more LW exposure: yes 

Question 4 was based on the decoy effect aka asymmetric dominance aka attraction effect (but missing a control condition).  I don't have numbers from the original study (and there is no correct answer) so I can't really answer 1 or 2 for this question, but there was a difference based on LW exposure: 57% vs. 44% selecting the less bias related answer. 
1. LWers biased: n/a 
2. LWers less biased than others: n/a 
3. Less bias with more LW exposure: yes 

Question 5 was an anchoring question.  The original study found an effect (measured by slope) of 0.55 (though it was less transparent about the randomness of the anchor; transparent studies w. other questions have found effects around 0.3 on average).  For LWers there was a significant anchoring effect but it was only 0.14 in magnitude, and it did not vary based on LW exposure (there was a weak & nonsignificant trend in the wrong direction). 
1. LWers biased: yes 
2. LWers less biased than others: yes 
3. Less bias with more LW exposure: no 

One thing you might wonder: how much of this is just intelligence?  There were several questions on the survey about performance on IQ tests or SATs.  Controlling for scores on those tests, all of the results about the effects of LW exposure held up nearly as strongly.  Intelligence test scores were also predictive of lower bias, independent of LW exposure, and those two relationships were almost the same in magnitude.  If we extrapolate the relationship between IQ scores and the 5 biases to someone with an IQ of 100 (on either of the 2 IQ measures), they are still less biased than the participants in the original study, which suggests that the "LWers less biased than others" effect is not based solely on IQ.

MORE DETAILED RESULTS

There were 5 questions related to strength of membership in the LW community which I standardized and combined into a single composite measure of LW exposure (LW use, sequence reading, time in community, karma, meetup attendance); this was the main predictor variable I used (time per day on LW also seems related, but I found out while analyzing last year's survey that it doesn't hang together with the others or associate the same way with other variables).  I analyzed the results using a continuous measure of LW exposure, but to simplify reporting, I'll give the results below by comparing those in the top third on this measure of LW exposure with those in the bottom third.

There were 5 intelligence-related measures which I combined into a single composite measure of Intelligence (SAT out of 2400, SAT out of 1600, ACT, previously-tested IQ, extra credit IQ test); I used this to control for intelligence and to compare the effects of LW exposure with the effects of Intelligence (for the latter, I did a similar split into thirds).  Sample sizes: 1101 people answered at least one of the CFAR questions; 1099 of those answered at least one LW exposure question and 835 of those answered at least one of the Intelligence questions.  Further details about method available on request.

Here are the results, question by question.

Question 1: Jack is looking at Anne, but Anne is looking at George. Jack is married but George is not. Is a married person looking at an unmarried person?

• Yes 
• No 
• Cannot be determined 

This is a "disjunctive reasoning" question, which means that getting the correct answer requires using "or".  That is, it requires considering multiple scenarios.  In this case, either Anne is married or Anne is unmarried.  If Anne is married then married Anne is looking at unmarried George; if Anne is unmarried then married Jack is looking at unmarried Anne.  So the correct answer is "yes".  A study by Toplak & Stanovich (2002) of students at a large Canadian university found that only 13% correctly answered "yes" while 86% answered "cannot be determined" (2% answered "no").

On this LW survey, 46% of participants correctly answered "yes"; 54% chose "cannot be determined" (and 0.4% said"no").  Further, correct answers were much more common among those high in LW exposure: 58% of those in the top third of LW exposure answered "yes", vs. only 31% of those in the bottom third.  The effect remains nearly as big after controlling for Intelligence (the gap between the top third and the bottom third shrinks from 27% to 24% when Intelligence is included as a covariate).  The effect of LW exposure is very close in magnitude to the effect of Intelligence; 60% of those in the top third in Intelligence answered correctly vs. 37% of those in the bottom third.

original study: 13% 
weakly-tied LWers: 31% 
strongly-tied LWers: 58% 


Question 2: Would you prefer to receive $55 today or $75 in 60 days?

This is a temporal discounting question.  Preferring $55 today implies an extremely (and, for most people, implausibly) high discount rate, is often indicative of a pattern of discounting that involves preference reversals, and is correlated with other biases.  The question was used in a study by Kirby (2009) of undergraduates at Williams College (with a delay of 61 days instead of 60; I took it from a secondary source that said "60" without checking the original), and based on the graph of parameter values in that paper it looks like just under half of participants chose the larger later option of $75 in 61 days.

LW survey participants almost uniformly showed a low discount rate: 92% chose $75 in 61 days.  This is near ceiling, which didn't leave much room for differences among LWers.  For LW exposure, top third vs. bottom third was 93% vs. 90%, and this relationship was not statistically significant (p=.15); for Intelligence it was 96% vs. 91% and the relationship was statistically significant (p=.007).  (EDITED: I originally described the Intelligence result as nonsignificant.)

original study: ~47% 
weakly-tied LWers: 90% 
strongly-tied LWers: 93% 


Question 3: A certain town is served by two hospitals. In the larger hospital, about 45 babies are born each day. In the smaller one, about 15 babies are born each day. Although the overall proportion of girls is about 50%, the actual proportion at either hospital may be greater or less on any day. At the end of a year, which hospital will have the greater number of days on which more than 60% of the babies born were girls? 

• The larger hospital 
• The smaller hospital 
• Neither - the number of these days will be about the same 

This is a statistical reasoning question, which requires applying the law of large numbers.  In Tversky & Kahneman's (1974) original paper, only 22% of participants correctly chose the smaller hospital; 57% said "about the same" and 22% chose the larger hospital.

On the LW survey, 84% of people correctly chose the smaller hospital; 15% said "about the same" and only 1% chose the larger hospital.  Further, this was strongly correlated with strength of LW exposure: 93% of those in the top third answered correctly vs. 75% of those in the bottom third.  As with #1, controlling for Intelligence barely changed this gap (shrinking it from 18% to 16%), and the measure of Intelligence produced a similarly sized gap: 90% for the top third vs. 79% for the bottom third.

original study: 22% 
weakly-tied LWers: 75% 
strongly-tied LWers: 93%

Question 4: Imagine that you are a doctor, and one of your patients suffers from migraine headaches that last about 3 hours and involve intense pain, nausea, dizziness, and hyper-sensitivity to bright lights and loud noises. The patient usually needs to lie quietly in a dark room until the headache passes. This patient has a migraine headache about 100 times each year. You are considering three medications that you could prescribe for this patient. The medications have similar side effects, but differ in effectiveness and cost. The patient has a low income and must pay the cost because her insurance plan does not cover any of these medications. Which medication would you be most likely to recommend? 

• Drug A: reduces the number of headaches per year from 100 to 30. It costs $350 per year. 
• Drug B: reduces the number of headaches per year from 100 to 50. It costs $100 per year. 
• Drug C: reduces the number of headaches per year from 100 to 60. It costs $100 per year. 

This question is based on research on the decoy effect (aka "asymmetric dominance" or the "attraction effect").  Drug C is obviously worse than Drug B (it is strictly dominated by it) but it is not obviously worse than Drug A, which tends to make B look more attractive by comparison.  This is normally tested by comparing responses to the three-option question with a control group that gets a two-option question (removing option C), but I cut a corner and only included the three-option question.  The assumption is that more-biased people would make similar choices to unbiased people in the two-option question, and would be more likely to choose Drug B on the three-option question.  The model behind that assumption is that there are various reasons for choosing Drug A and Drug B; the three-option question gives biased people one more reason to choose Drug B but other than that the reasons are the same (on average) for more-biased people and unbiased people (and for the three-option question and the two-option question).

Based on the discussion on the original survey thread, this assumption might not be correct.  Cost-benefit reasoning seems to favor Drug A (and those with more LW exposure or higher intelligence might be more likely to run the numbers).  Part of the problem is that I didn't update the costs for inflation - the original problem appears to be from 1995 which means that the real price difference was over 1.5 times as big then.

I don't know the results from the original study; I found this particular example online (and edited it heavily for length) with a reference to Chapman & Malik (1995), but after looking for that paper I see that it's listed on Chapman's CV as only a "published abstract".

49% of LWers chose Drug A (the one that is more likely for unbiased reasoners), vs. 50% for Drug B (which benefits from the decoy effect) and 1% for Drug C (the decoy).  There was a strong effect of LW exposure: 57% of those in the top third chose Drug A vs. only 44% of those in the bottom third.  Again, this gap remained nearly the same when controlling for Intelligence (shrinking from 14% to 13%), and differences in Intelligence were associated with a similarly sized effect: 59% for the top third vs. 44% for the bottom third.

original study: ?? 
weakly-tied LWers: 44% 
strongly-tied LWers: 57% 


Question 5: Get a random three digit number (000-999) from http://goo.gl/x45un and enter the number here. 

Treat the three digit number that you just wrote down as a length, in feet. Is the height of the tallest redwood tree in the world more or less than the number that you wrote down? 

What is your best guess about the height of the tallest redwood tree in the world (in feet)?

This is an anchoring question; if there are anchoring effects then people's responses will be positively correlated with the random number they were given (and a regression analysis can estimate the size of the effect to compare with published results, which used two groups instead of a random number).

Asking a question with the answer in feet was a mistake which generated a great deal of controversy and discussion.  Dealing with unfamiliar units could interfere with answers in various ways so the safest approach is to look at only the US respondents; I'll also see if there are interaction effects based on country.

The question is from a paper by Jacowitz & Kahneman (1995), who provided anchors of 180 ft. and 1200 ft. to two groups and found mean estimates of 282 ft. and 844 ft., respectively.  One natural way of expressing the strength of an anchoring effect is as a slope (change in estimates divided by change in anchor values), which in this case is 562/1020 = 0.55.  However, that study did not explicitly lead participants through the randomization process like the LW survey did.  The classic Tversky & Kahneman (1974) anchoring question did use an explicit randomization procedure (spinning a wheel of fortune; though it was actually rigged to create two groups) and found a slope of 0.36.  Similarly, several studies by Ariely & colleagues (2003) which used the participant's Social Security number to explicitly randomize the anchor value found slopes averaging about 0.28.

There was a significant anchoring effect among US LWers (n=578), but it was much weaker, with a slope of only 0.14 (p=.0025).  That means that getting a random number that is 100 higher led to estimates that were 14 ft. higher, on average.  LW exposure did not moderate this effect (p=.88); looking at the pattern of results, if anything the anchoring effect was slightly higher among the top third (slope of 0.17) than among the bottom third (slope of 0.09). Intelligence did not moderate the results either (slope of 0.12 for both the top third and bottom third).  It's not relevant to this analysis, but in case you're curious, the median estimate was 350 ft. and the actual answer is 379.3 ft. (115.6 meters).

Among non-US LWers (n=397), the anchoring effect was slightly smaller in magnitude compared with US LWers (slope of 0.08), and not significantly different from the US LWers or from zero.

original study: slope of 0.55 (0.36 and 0.28 in similar studies) 
weakly-tied LWers: slope of 0.09 
strongly-tied LWers: slope of 0.17 


If we break the LW exposure variable down into its 5 components, every one of the five is strongly predictive of lower susceptibility to bias.  We can combine the first four CFAR questions into a composite measure of unbiasedness, by taking the percentage of questions on which a person gave the "correct" answer (the answer suggestive of lower bias).  Each component of LW exposure is correlated with lower bias on that measure, with r ranging from 0.18 (meetup attendance) to 0.23 (LW use), all p < .0001 (time per day on LW is uncorrelated with unbiasedness, r=0.03, p=.39).  For the composite LW exposure variable the correlation is 0.28; another way to express this relationship is that people one standard deviation above average on LW exposure 75% of CFAR questions "correct" while those one standard deviation below average got 61% "correct".  Alternatively, focusing on sequence-reading, the accuracy rates were:

75%    Nearly all of the Sequences (n = 302) 
70%    About 75% of the Sequences (n = 186) 
67%    About 50% of the Sequences (n = 156) 
64%    About 25% of the Sequences (n = 137) 
64%    Some, but less than 25% (n = 210) 
62%    Know they existed, but never looked at them (n = 19) 
57%    Never even knew they existed until this moment (n = 89) 

Another way to summarize is that, on 4 of the 5 questions (all but question 4 on the decoy effect) we can make comparisons to the results of previous research, and in all 4 cases LWers were much less susceptible to the bias or reasoning error.  On 1 of the 5 questions (question 2 on temporal discounting) there was a ceiling effect which made it extremely difficult to find differences within LWers; on 3 of the other 4 LWers with a strong connection to the LW community were much less susceptible to the bias or reasoning error than those with weaker ties.


REFERENCES
Ariely, Loewenstein, & Prelec (2003), "Coherent Arbitrariness: Stable demand curves without stable preferences" 
Chapman & Malik (1995), "The attraction effect in prescribing decisions and consumer choice" 
Jacowitz & Kahneman (1995), "Measures of Anchoring in Estimation Tasks" 
Kirby (2009), "One-year temporal stability of delay-discount rates" 
Toplak & Stanovich (2002), "The Domain Specificity and Generality of Disjunctive Reasoning: Searching for a Generalizable Critical Thinking Skill" 
Tversky & Kahneman's (1974), "Judgment under Uncertainty: Heuristics and Biases"

50 comments

Comments sorted by top scores.

Option A: $350 / 70 migraines avoided = $5 per migraine avoided
Option B: $100 / 50 migraines avoided = $2 per migraine avoided

R> lw <- read.csv("2012.csv")
R> lw$MissingAnswers <- apply(lw, 1, function(x) sum(sapply(x, function(y) is.na(y) || as.character(y)==" ")))
R> right <- lw[as.character(lw$CFARQuestion1) == "Yes",]$MissingAnswers
R> wrong <- lw[as.character(lw$CFARQuestion1) == "no" | as.character(lw$CFARQuestion1) == "Cannot be determined",]$MissingAnswers
R> t.test(right, wrong)
        Welch Two Sample t-test

    data:  right and wrong
    t = -1.542, df = 942.5, p-value = 0.1234
    alternative hypothesis: true difference in means is not equal to 0
    95 percent confidence interval:
     -2.3817  0.2858
    sample estimates:
    mean of x mean of y
        16.69     17.74

R> lw <- read.csv("2012.csv")
R> lw$MissingAnswers <- apply(lw, 1, function(x) sum(sapply(x, function(y) is.na(y) || as.character(y)==" ")))
R> lw <- lw[as.character(lw$CFARQuestion1) != " " & !is.na(as.character(lw$CFARQuestion1)),]
R> lw <- data.frame(lw$CFARQuestion1, lw$MissingAnswers)
R> summary(glm(lw.CFARQuestion1 ~ lw.MissingAnswers, data = lw, family = "binomial"))

Deviance Residuals:
   Min      1Q  Median      3Q     Max
 -1.17   -1.12   -1.05    1.23    1.41

Coefficients:
                  Estimate Std. Error z value Pr(>|z|)
(Intercept)        0.00111    0.12214    0.01     0.99
lw.MissingAnswers -0.00900    0.00607   -1.48     0.14

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 1366.6  on 989  degrees of freedom
Residual deviance: 1364.4  on 988  degrees of freedom
AIC: 1368

Number of Fisher Scoring iterations: 3