"The Journal of Real Effects"

post by CarlShulman · 2012-03-05T03:07:02.685Z · LW · GW · Legacy · 6 comments

Contents

6 comments

Luke's recent post mentioned that The Lancet has a policy encouraging the advance registration of clinical trials, while mine examined an apparent case study of data-peeking and on-the-fly transformation of studies. But how much variation is there across journals on such dimensions? Are there journals that buck the standards of their fields (demanding registration, p=0.01 rather than p=0.05 where the latter is typical in the field, advance specification of statistical analyses and subject numbers, etc)? What are some of the standouts? Are there fields without any such?

I wonder if there is a niche for a new open-access journal, along the lines of PLoS, with standards strict enough to reliably exclude false-positives. Some possible titles:

 

 

6 comments

Comments sorted by top scores.

comment by Vladimir_Nesov · 2012-03-05T05:59:59.474Z · LW(p) · GW(p)

"Journal" might be a wrong concept, this service looks like something that could be decoupled from publishing.

Replies from: Viliam_Bur
↑ comment by Viliam_Bur · 2012-03-05T11:05:57.784Z · LW(p) · GW(p)

I imagine it would not contain articles, only information about reliability of statements. Such as "result R published in article A was replicated successfully, here are the relevant data". So when someone reads an article, they could come to this service to check its reliability.

How should such thing be organized?

Replies from: CarlShulman
↑ comment by CarlShulman · 2012-03-05T11:48:14.644Z · LW(p) · GW(p)

Wikis, Consumer Reports, Cochrane Reports, etc. The virtue of journals would be to reduce switching costs for academia.

comment by gwern · 2012-03-05T04:45:00.657Z · LW(p) · GW(p)

I know there are already journals which require pre-registration; perhaps one could investigate how successful they have been. (We've known about these issues for a long time, but they haven't been fixed - which suggests incentive problems....)

Replies from: CarlShulman
↑ comment by CarlShulman · 2012-03-05T04:58:50.833Z · LW(p) · GW(p)

Yes, I'm curious about the barriers if anyone knows more.

comment by CasioTheSane · 2012-03-08T04:05:48.910Z · LW(p) · GW(p)

Would one get banned from this journal for registering the testing of 100 only slightly different hypotheses at p=0.01?