Follow-up on ESP study: "We don't publish replications"

Yes, this is a standard incentives problem. But one to keep in mind when parsing the literature.

We simply do not have a scientific process any more.

This is both unfair to scientists and inaccurate. In 2011, we've had such novel scientific discoveries as snails that can survive being eaten by birds, we've estimated the body temperature of dinosaurs, we've captured the most detailed picture of a dying star ever taken, and we've made small but significant progress to resolving P ?= NP. These are but a few of the highlights that happened to both be in my recent memory and which I could easily locate links to. I've also not included anything that could be argued to be engineering rather than science. There are many achievements just like this.

Why might it seem like we don't have a scientific process?

First, there's simple nostalgia. As I write this, the space shuttle is on its very last mission. I suspect that almost everyone here either longs for the days of their youth when humans walked on the moon, or wish they had lived then to witness that. Thus, the normal human nostalgia is wrapped up in some actual problems of stagnation and lack of funding. This creates a potential halo effect for the past.

Second, as the number of scientists increases over time, the number of scientists who are putting out poor science will increase. Similarly, the amount of stuff that gets through peer review even when it shouldn't will increase as the number of journals and the number of papers submitted goes up. So the amount of bad science will go up.

Third, the internet, and similar modern communication technologies lets us find out about so-called bad science much faster than we would otherwise. Much of that would get buried in obscure journals but instead we have bloggers commenting and respected scientists responding. So as time goes on, even if the amount of bad science stays constant, the perception would be of an increase.

I would go so far as to venture that we might have a more robust and widespread scientific process than at any other time in history. To put the Bem study in perspective, keep in mind that a hundred years ago, psychology wasn't even trying to use statistical methods; look at how Freud and Jung's ideas were viewed. Areas like sociology and psychology have if anything become more scientific over time. From that standpoint, a paper that uses statistics in a flawed fashion is indicative of how much progress the soft sciences have made in terms of being real sciences in that one needs bad stats to get bad ideas through rather than just anecdotal evidence.

To paraphrase someone speaking on a completely different issue, the arc of history is long, but it bends towards science.

Hearing one probably bad thing and deciding we fell from grace and should shake our heads in bitter nostalgia? That's what the villains do. We, the ingroup of Truth and Right and Apple Pie, are dutifully skeptical of claims of increasing stupidity. You taught me that.

We simply do not have a scientific process any more.

Even if true that wouldn't be simple.

What do you mean by that?

To the point responses = encouraging to me that your busy :)

Not necessarily. Ignoring crap may be a better strategy than picking it apart.

Cooperation is also easier when different groups in the same research area don't try too hard to invalidate each other's claims. If the problem in question is interesting you're much better off writing your own paper on it with your own claims and results. You can dismiss the other paper with a single paragraph: "Contrary to the findings of I.C. Wiener in [2] we observe that..." and leave it at that.

The system is entirely broken but I don't see an easy way to make it better.

If this were true how would anyone ever get the first citation?

Some papers (those written by high status authors) are ones that everyone knows will get citations soon after they are published, and so they feel safe in citing them since others are soon to do so. Self-fulfilling prophecy.

If this were true how would anyone ever get the first citation?

Because the policy wasn't applied until after a cutoff date, so the recursion bottoms out at an author from before the cutoff. Obviously. Edit: Non-obviously. Edit2: HOW AM I SUPPOSED TO END THIS COMMENT FOR YOU MEATBAGS NOT TO VOTE ME DOWN???

If this were true how would anyone ever get the first citation?

Zed didn't say you should never cite a previously uncited paper, only that you shouldn't invest time and effort into work that depends on the assumption that its conclusions are sound. There are many possible reasons why you might nevertheless want to cite it, and perhaps even give it some lip service.

I'm confused. I parsed this as "papers which contain no citations are considered bad sources," but it seems that everyone else is parsing it as "papers which have not been cited are considered bad sources." Am I making a mistake here? The latter doesn't make much sense to me, but Zed hasn't stepped in to correct that interpretation.

If this were true how would anyone ever get the first citation?

I think that you can cite it if you really think that it's good (and perhaps this is what Zed meant by "if you must"), but you'd better be citing something more widely accepted too. Then if lots of people think that it's really good, it will join the canon of widely cited papers.

Also, people outside of the publish-or-perish framework (respected elders, hobbyists outside of the research track, grad students in some fields) can get the ball rolling.