"Is Science Broken?" is underspecified
post by NancyLebovitz · 2016-08-12T11:59:39.834Z · LW · GW · Legacy · 11 commentshttp://fivethirtyeight.com/features/science-isnt-broken/
This is an interesting article-- it's got an overview of what's currently seen as the problems with replicability and fraud, and some material I haven't seen before about handing the same question to a bunch of scientists, and looking at how they come up with their divergent answers.
However, while I think it's fair to say that science is really hard, the article gets into claiming that scientists aren't especially awful people (probably true), but doesnn't address the hard question of "Given that there's a lot of inaccurate science, how much should we trust specific scientific claims?"
11 comments
Comments sorted by top scores.
comment by Lumifer · 2016-08-12T14:53:41.235Z · LW(p) · GW(p)
I think this article suffers from aggregating all science into one big bin. In reality, different disciplines have a radically different level of problems with replicability and fraud. Classical hard sciences like physics and chemistry don't have much of a problem. Very soft sciences like psychology or anthropology have a huge problem.
Replies from: NancyLebovitz↑ comment by NancyLebovitz · 2016-08-12T15:29:29.776Z · LW(p) · GW(p)
You're right, though I'm not sure what the best way to phrase it better is.
My question still stands, since the parts of science which are most fucked seems to be the parts that have the most immediate impact on people's choices.
Replies from: Vaniver, Lumifer↑ comment by Vaniver · 2016-08-12T15:43:29.558Z · LW(p) · GW(p)
My question still stands, since the parts of science which are most fucked seems to be the parts that have the most immediate impact on people's choices.
Sure, but the problem here is that the causality probably goes in the opposite direction. That is, the more a scientific endeavor will affect people's choices, the more pressure there is to corrupt that scientific endeavor.
↑ comment by Lumifer · 2016-08-12T15:58:44.963Z · LW(p) · GW(p)
What do you mean by immediate impact on choices? Very few people make choices based on what the psychological theory of the day says they must do.
The most impactful branches are probably medicine and economics. They are medium-fucked, I think, because at the psych/anthro levels of dysfunction your patients just die or your economy implodes and people tend to dislike such things :-/
Replies from: NancyLebovitz↑ comment by NancyLebovitz · 2016-08-12T16:50:13.383Z · LW(p) · GW(p)
Now that I'm thinking about it, psychological papers probably have more effect in the LW-sphere than in the world generally. Are you counting nutrition as part of medicine?
Replies from: Val, Lumifercomment by Daniel_Burfoot · 2016-08-12T13:02:31.228Z · LW(p) · GW(p)
The infographic on that page is amazing.
comment by Gram_Stone · 2016-08-12T22:12:59.309Z · LW(p) · GW(p)
Given the OP and this, I thought that you might like this.
That's not too closely related to the OP in one sense, but I've been collecting what I might call 'stories of broken science,' and thought you might be doing the same thing for different reasons.
comment by Val · 2016-08-12T21:00:54.609Z · LW(p) · GW(p)
Has anyone heard about the book "The egg-laying dog" from Beck-Bornholdt? I don't know about an English translation, I freely translated the title from German. It is a book about fallacies in statistics, research, especially in medicine, written in a style to be comprehensible by the layman.
It discusses at great length the problems plaguing modern research (well, the research of the 1990's when the book was written, but I doubt that very much has changed). For example, the required statistical significance for a publication is much more relaxed than it was a long time ago. Often a p-value of 5% is enough for a publication, so even with perfectly unbiased researchers, without p-fishing or other unethical tricks, there is a huge number of accepted publications around which are utterly rubbish. This is all made much worse by the fact that everyone wants new results, so few researchers can get funding by repeating and verifying already published results (unless the publication in question is on every headline), and also few researchers are inclined (or supported by the system) to publish negative results.
comment by MrMind · 2016-08-16T07:12:56.180Z · LW(p) · GW(p)
I dream of a future where published papers will no more need to be about a 0-1 effect (wether an hypothesis is true or not), but will be like that amazing infographic; a future where a scientist's job will be no more to interpret data, but to construct interactive models and allow anyone to explore them and draw the conclusions they like.