[Link] Why Science Is Not Necessarily Self-Correcting
post by ChristianKl · 2014-10-13T13:51:02.708Z · LW · GW · Legacy · 17 commentsContents
Why Science Is Not Necessarily Self-Correcting John P. A. Ioannidis None 17 comments
Why Science Is Not Necessarily Self-Correcting John P. A. Ioannidis
The ability to self-correct is considered a hallmark of science. However, self-correction does not always happen to scientific evidence by default. The trajectory of scientific credibility can fluctuate over time, both for defined scientific fields and for science at-large. History suggests that major catastrophes in scientific credibility are unfortunately possible and the argument that “it is obvious that progress is made” is weak. Careful evaluation of the current status of credibility of various scientific fields is important in order to understand any credibility deficits and how one could obtain and establish more trustworthy results. Efficient and unbiased replication mechanisms are essential for maintaining high levels of scientific credibility. Depending on the types of results obtained in the discovery and replication phases, there are different paradigms of research: optimal, self-correcting, false nonreplication, and perpetuated fallacy. In the absence of replication efforts, one is left with unconfirmed (genuine) discoveries and unchallenged fallacies. In several fields of investigation, including many areas of psychological science, perpetuated and unchallenged fallacies may comprise the majority of the circulating evidence. I catalogue a number of impediments to self-correction that have been empirically studied in psychological science. Finally, I discuss some proposed solutions to promote sound replication practices enhancing the credibility of scientific results as well as some potential disadvantages of each of them. Any deviation from the principle that seeking the truth has priority over any other goals may be seriously damaging to the self-correcting functions of science.
17 comments
Comments sorted by top scores.
comment by buybuydandavis · 2014-10-13T18:54:56.378Z · LW(p) · GW(p)
Don't know if it was apparent to everyone else, but it wasn't apparent to me that the bolded title was also a link.
comment by shminux · 2014-10-13T22:44:31.433Z · LW(p) · GW(p)
The article seems to be heavily biased towards psychology. I wonder if the "harder" sciences like physics, chemistry and biology suffer from the same issues to a similar degree.
Replies from: Salemicus, Richard_Kennaway, CronoDAS, ChristianKl↑ comment by Salemicus · 2014-10-14T13:11:50.923Z · LW(p) · GW(p)
The author of the article, Ioannidis, has published extensively about the unreliability of reported medical and biochemical results, over a more than 10 year period. The article is not so much "biased" towards psychology, as focusing on that one area.
Replies from: shminux↑ comment by shminux · 2014-10-14T17:07:25.031Z · LW(p) · GW(p)
Right, "focusing" is a better description. But I wonder if this focusing resulted in a generalization which is a bit too sweeping. The "publish or perish" race is certainly everywhere in academia, but its side effects might be better mitigated in some areas than in others.
↑ comment by Richard_Kennaway · 2014-10-14T12:05:28.478Z · LW(p) · GW(p)
I think of the work on blue LEDs that recently got the physics Nobel.
Blue LEDs work. You can buy them off the shelf. Each one works pretty much every time.
Is there anything in sociology or psychology of which the same can be said?
Replies from: satt, army1987↑ comment by satt · 2014-10-15T00:20:31.626Z · LW(p) · GW(p)
Blue LEDs work. You can buy them off the shelf. Each one works pretty much every time.
Is there anything in sociology or psychology of which the same can be said?
Depends on whether "Each one works pretty much every time" means a phenomenon which works on pretty much every individual on pretty much every occasion, or a phenomenon which can simply be replicated reliably given a big enough sample.
I can think of nothing in sociology or psychology satisfying the former criterion. But the latter, weaker criterion seems to be satisfied by anchoring bias, which was replicated by 36 sites out of 36 in the Many Labs project, as indicated by its table of summary statistics.
Replies from: Desrtopa, Richard_Kennaway↑ comment by Desrtopa · 2014-10-15T15:11:04.196Z · LW(p) · GW(p)
Whether one counts anything in psychology as satisfying the former or not, I think depends on where one draws the line between psychology and neurology. There are certainly things we've discovered about how the brain works that tell us things about the thought processes of every human, but one might argue that these fall under the purview of neurology, and not psychology.
↑ comment by Richard_Kennaway · 2014-10-15T06:15:49.126Z · LW(p) · GW(p)
Depends on whether "Each one works pretty much every time" means a phenomenon which works on pretty much every individual on pretty much every occasion, or a phenomenon which can simply be replicated reliably given a big enough sample.
Definitely the former. Each one, every time. The world around us is filled with such things, yet when it comes to the study of anything to do with living organisms, people dismiss the idea as "physics envy", a concept which makes no more sense than "separate magisteria", and serves the same function.
↑ comment by A1987dM (army1987) · 2014-10-14T17:50:09.381Z · LW(p) · GW(p)
CBT?
Replies from: ChristianKl↑ comment by ChristianKl · 2014-10-15T15:02:49.845Z · LW(p) · GW(p)
CBT has a proven chance to help, but it doesn't have a 100% success rate for anything.
↑ comment by ChristianKl · 2014-10-13T22:49:49.965Z · LW(p) · GW(p)
In psychology you have the controversial replication initiative. In physics you have nobody complain about people attempting replications.
Replies from: Richard_Kennaway↑ comment by Richard_Kennaway · 2014-10-14T12:06:29.595Z · LW(p) · GW(p)
In physics you have nobody complain about people attempting replications.
Perhaps that is because the stuff actually replicates.
comment by buybuydandavis · 2014-10-13T19:05:55.994Z · LW(p) · GW(p)
Wasn't there a recent thread exactly on the recent brouhaha in psychology over replication? Maybe even linking to this article?
Replies from: ChristianKl↑ comment by ChristianKl · 2014-10-13T20:53:06.278Z · LW(p) · GW(p)
There was a thread about some psychologists stating that the replication initiative does more harm than good.
comment by TheMajor · 2014-10-13T16:12:45.523Z · LW(p) · GW(p)
Finally, I discuss some proposed solutions to promote sound replication practices enhancing the credibility of scientific results
Which would these be? I skimmed through the article and found nothing beyond the standard 'truth must become more important', and I doubt if that should even be called a solution.
Replies from: satt↑ comment by satt · 2014-10-13T23:02:42.222Z · LW(p) · GW(p)
Which would these be? I skimmed through the article and found nothing beyond the standard 'truth must become more important', and I doubt if that should even be called a solution.
I guess it's these, from the last section of the main text:
Some suggestions for potential amendments that can be tested have been made in previous articles (Ioannidis, 2005; Young, Ioannidis, & Al-Ubaydli, 2008) and additional suggestions are made also by authors in this issue of Perspectives. Nosek et al. (2012) provide the most explicit and extensive list of recommended changes, including promoting paradigm-driven research; use of author, reviewer, editor checklists; challenging the focus on the number of publications and journal impact factor; developing metrics to identify what is worth replicating; crowdsourcing replication efforts; raising the status of journals with peer review standards focused on soundness and not on the perceived significance of research; lowering or removing the standards for publication; and, finally, provision of open data, materials, and workflow. Other authors are struggling with who will perform these much-desired, but seldom performed, independent replications. Frank and Saxe (2012) and Grahe et al. (2012) suggest that students in training could populate the ranks of replicators. Finally, Wagenmakers et al. (2012) repeat the plea for separating exploratory and confirmatory research and demand rigorous a priori registration of the analysis plans for confirmatory research.