comment by Galap ·
2016-02-23T09:52:47.964Z · LW(p) · GW(p)
This may not be strictly statistical, but I would choose the idea that in order to make any meaningful statement with data, you always have to have something to compare it to.
Like someone will always come in some political thread and say , "X will increase/decrease Y by Z%.) And my first thought in response is always, "Is that a lot?"
For a recent example I saw, someone showed a graph of Japanese student suicides as a function of day of the year. There were pretty high spikes (about double the baseline value) on the days corresponding to the first day of each school semester. The poster was attributing this to Japanese school bullying and other problems with Japan's school system.
My first thought was, "wait. Show me that graph for other countries. For the world, if such data has been reliably gathered." If it looks the same, it's not a uniquely Japanese problem. What if it's worse in other countries, even?
Yeah, I'd really like to see people stop using information where it doesn't mean anything in isolation. A lot of people think that controls in science exist to make sure that the effects you see aren't spurious or adventitious. It's not like that's wrong, but it's deeper and even more fundamental than that.
I'm a scientist, so let me give you an example from my research (grossly simiplified and generalized for brevity).
Substance A was designed such that it manifests an as-of-yet unexplored type of structural situation. We then carried out a reaction on substance A to see what some of the effects of this situation are. Something happened.
So, if we were to leave it at that, what would we have learned? Nothing. We need substance B, which does not have that siutation going on but is otherwise as similar to A as we can make it, to see what IT does, to see if it does anything different than A. See, we need to do the experiments on both B and A not to see whether the results of A are 'real'. We need to do it to see what the results even ARE in the first place.