Is it true that most studies in Kahneman's 'Thinking fast and slow' are not replicable and the book is largely not supported by evidence?
post by Davy Jones (davy-jones) · 2021-06-30T07:14:39.238Z · LW · GW · 1 commentThis is a question post.
Contents
Answers 11 Mike Listen 5 jp 3 akhil_jalan 1 mikbp 1 Richard Zander 1 Cesar Orlando Pallares Delgado None 1 comment
So I stumbled on this article.
https://replicationindex.com/2020/12/30/a-meta-scientific-perspective-on-thinking-fast-and-slow
'So, replicability [of all studies in this book] is somewhere between 12% and 46%. Even if half of the results are replicable, we do not know which results are replicable and which ones are not.'
'Readers of “Thinking: Fast and Slow” should read the book as a subjective account by an eminent psychologist, rather than an objective summary of scientific evidence.'
I have no background in social sciences or statistics so I don't know if claims and math in this article are correct. Could somebody with more knowledge comment on this? This is HUGE if true.
I love this book to pieces but I don't want to go around spreading outdated science.. are there any similar books on human biases and thinking which are more recent and more robust when it comes to evidence and statistics?
Answers
Kahneman is rare example of a scientist critical to his own ideas,trying to falsify them rather than p-hack his way to fame. I recommend M. Lewis's book about his and Tversky's work.
↑ comment by mingyuan · 2021-07-01T19:39:22.808Z · LW(p) · GW(p)
My impression of The Undoing Project (Michael Lewis's book that you recommend) was that it absolutely and completely ignored the replication crisis. I was hoping for more and ended up super disappointed. It was a fun bit of biography but I don't think it has any bearing on Davy's question.
↑ comment by Davy Jones (davy-jones) · 2021-06-30T19:53:52.659Z · LW(p) · GW(p)
Thanks, this is helpful and in line with that article.
I read the book post-replication crisis, but had to put it down because it seemed like half its content was the type that I wouldn't expect to pass replication. I still think that the message I've gotten from this community around:
- The difference between intuitive vs explicit/analytical reasoning
- There are knowable, common and correctable biases in intuitive reasoning
have stood the test of time, at least in principle, albeit with less of a central role in my worldview.
The results cited in Kahneman's book score poorly on replicability measures. This is not his fault, just a general flaw in the psychology literature that was unknown to experts pre-replication crisis. Some of the claims in the book, like the concept of priming in Chapter 4, have been largely refuted by the literature. Table 1 of the link above gives a more detailed breakdown. Quoting,
"Tversky and Kahneman (1971) themselves warned against studies that provide so little evidence for a hypothesis. A 50% probability of answering multiple choice questions correctly is also used to fail students. So, we decided to give chapters with an R-Index below 50 a failing grade. Other chapters with failing grades are Chapter 3, 6, 711, 14, 16. Chapter 24 has the highest highest score (80, wich is an A- in the Canadian grading scheme), but there are only 8 results."
I don't know how well the studies on which is based replicate, but 'Predictably irrational' from Dan Ariely is also in the Behavioural Economics field, and Dan seems to put a lot of effort into preparing valid experiments.
Try books by Gerd Gigerenzer (search on Amazon). I particularly like his Empire of Chance, about different schools of probability analysis. Bounded Rationality is pretty good, too. The difference between Gigerenzer's and Tversky's attitudes is not particularly impressive (they apparently hate each other, even though one is dead). One is decision making when in a hurry (Gigerenzer) and the other is decision-making with insufficient data (Tversky) (or maybe it's the other way around, tedious stuff). Discussions of biases is fascinating by both authors.
↑ comment by Davy Jones (davy-jones) · 2021-06-30T20:13:52.867Z · LW(p) · GW(p)
Thanks
I agree with jp's comment in the sense that the true value of Thinking Fast & Slow may not be the specific examples but the cautionary message it gives about relying too much on our decisions and overcoming the "homo-economicus
However, I'm also interested now to see the replicability of those studies quoted by the book.."
1 comment
Comments sorted by top scores.
comment by Kenny · 2021-07-01T19:03:45.796Z · LW(p) · GW(p)
This is a good question!
One quibble – "not replicable" seems like a much stronger criticism than "not replicated" – I imagine you might have meant the latter, i.e. attempted replications failed to find similarly 'significant' effects and not 'the effect(s), if real, can not be replicated in principle'.
(And now I'm wondering whether "not replicable" is common and I've never noticed it before.)
Some thoughts about your question itself:
replicability [of all studies in this book] is somewhere between 12% and 46%.
That's a pretty wide range!
It would probably also, significantly, matter as to which specific studies were not be replicated (and whether the attempted replications themselves were well done) and then which parts of the book depend on those studies, or cite them as evidence. Ideally, one would want to consider something like the entire 'graph' of claims/beliefs made or argued for in the book and both the specific and overall updates one would reasonably make based on the (non-)replication of the cited/referenced studies. That would be an interesting, tho costly (if only in time), project itself!