If my previous research is wrong, what are my options ?

post by BB6 (barbarabrezna@gmail.com) · 2021-04-07T16:44:28.156Z · LW · GW · 4 comments

This is a question post.

Contents

  Answers
    32 ChristianKl
    2 jacopo
    2 Dustin
None
4 comments

In 2007 I coauthored a scientific paper, which is essentially a method for food analysis. To be honest, we have noticed one sample behaving weirdly already at that time, but we rationalised it away and omitted the weird sample from the results. Still, just in case, we have burried the article in a local journal of our institution, which is low profile. Later I discovered, it is wrong by design (My design :-( ). Knowing more background would have prevented the problem, but I did it as a side job. I discovered the full stupidity of the design in cca 2012 and told my boss. He suggested we correct ourselves by publishing another paper, claiming we did more experiments and found out we were wrong. I agreed, obtained more samples, my colleague worked on it, but I went on a long maternal leave with 2 kids and forgot to follow up on it. It is uncorrected up until this day. What are my options ?

If we publish a new article, as my boss wanted, I fear some people will still find the first paper and not the second one, will keep quoting it, and, god forbid, use that published method. (To my best knowledge the method was not used so far. I hope it really wasn't). Retraction would be the cleanest way to do it, but also the worst looking. Also we could publish erratum. Erratums are paired up with the original articles and search engines show them together. But can erratum contain new samples ? Any other ideas ?

If my boss finds the correction a low priority, as he did so far, can I somehow act alone ?

Can I have any problems, that I did not correct myself fast enough ? I left some traces about knowing the design is wrong in 2012. I wrote the truth to one lady requesting the reprint. I also wrote a book chapter, explaining how important it is to know the backround while designing analysis method, and giving the example of this tricky food. I did not mentioned nor quoted my article...

Please, don't be too judgemental, and tell me, how you would or did handle these situations.

Answers

answer by ChristianKl · 2021-04-07T18:34:39.542Z · LW(p) · GW(p)

A lot of research is wrong for one reason or another. If a scientist finds out that a paper of someone wrote contains errors the usual process is to write a new paper. Science progresses through new papers and everyone accepts that old papers often contain errors and are overtaken by new research.

The only thing that might be research misconduct is to to forge your research results by not changing the data you publish. Given how hard it is to persue people for massive fraud it's however unlikely that anybody will argue in present day academia that leaving out one sample for understandable reasons will constitute fraud. 

An erratum might say: 

"We initually omitted one sample from our results because we believed it to be due to measurement error (or whatever rationalization you had at the time). That sample was X. Given our new research reported in paper Z we now believe that we made an error in omitting that sample and think our here proposed method has systematic flaws that lead to results like the sample."

answer by jacopo · 2021-04-09T17:26:25.570Z · LW(p) · GW(p)

I think this happens to many scientists. I found myself in a similar situation once - we could not have done better at the time, but we could have noticed that the tools we used were not sufficient. Fortunately, by the time we noticed we had better tools and we found that the conclusions were still valid, even if some quantitative results were pretty inaccurate. As you, I wanted to submit an erratum, but my boss insisted to include the results in another related paper instead. I still feel that an erratum would have been better, but I think he was worried that the referee would be someone who disliked him for unrelated reasons.

If I understand correctly, your paper was about a new method and it turns out that the method itself is fatally flawed. However, what you report is what comes out of the approach, and there is a problem with the basic idea which was not trivial to see. The  thing you feel guilty about is omitting the outlier from the results, but that you cannot fix anyway. Is this more or less it?

I don't think you should retract. It understand the impulse if you feel there is no value left in the paper, but it seems to me that retraction is mostly done for misconduct, if the paper contains something that is factually wrong (you wrote you did X but actually you did Y), or if your results come out of not adopting established best practices. 

An erratum would be good -because it's linked to the paper- but in your case it may be difficult to write. On one hand, the whole paper is invalidated. On the other hand, there is no factual error to correct, and a lot of papers are based on ideas which looked good at the time but are found to be wrong by later literature. Most of them I would say :) Scientists are not systematically publishing erratums when they recognize their proposed method was not as good as hoped.

In the end, I think your boss suggestion of making another paper may be the best in your case. You would be discussing why method X, which seems like a good idea, does not work after better analysis, the subject of hundreds of papers every year. The fact that you are the ones that proposed method X is no problem. If someone gets the idea of using that approach, given that it's 15 years old they will check more recent literature I hope. Nowadays it's easy to check the citing papers with google scholar.

I understand if you feel like you are hiding the fact that you had those suspicious results from the beginning, but you didn't figure out they were important until later. Also, the important thing is to correct the mistake in any form. If your boss finds the correction low priority, discuss with your colleague that did some work for for the new paper, and try to find the time to present your boss with a draft. 

answer by Dustin · 2021-04-07T22:30:45.372Z · LW(p) · GW(p)

If we publish a new article, as my boss wanted, I fear some people will still find the first paper and not the second one, will keep quoting it, and, god forbid, use that published method.

 

Can you clarify this for me?  Why would a new article make it more likely that people would find the first paper compared to the current situation wherein the only paper they could find is the first paper?

comment by BB6 (barbarabrezna@gmail.com) · 2021-04-08T08:37:35.868Z · LW(p) · GW(p)

New separate article is certainly better than nothing.

Erratum or retraction are, for my conscience, better than a new separate article.

I wonder if there are other ways people deal with this.

Replies from: jean-nassar
comment by masasin (jean-nassar) · 2021-04-15T04:38:12.138Z · LW(p) · GW(p)

Why not both? Refer to the new paper in the errata.

4 comments

Comments sorted by top scores.

comment by just_browsing · 2021-04-07T22:34:42.222Z · LW(p) · GW(p)

If you want more opinions on your situation than whatever you get on LessWrong, you could try asking this question on https://academia.stackexchange.com/ ). They have an entire tag on errors in published papers. 

comment by Daniel Kokotajlo (daniel-kokotajlo) · 2021-04-08T10:21:40.886Z · LW(p) · GW(p)

No judgment; quite the opposite! You are displaying far above-average concern for the truth and for others by admitting your mistake and seeking to correct it. (Arguably, even noticing your mistake already makes you above-average, since most would rationalize it away.)

Replies from: barbarabrezna@gmail.com
comment by BB6 (barbarabrezna@gmail.com) · 2021-04-13T12:26:23.512Z · LW(p) · GW(p)

Thanks. I am not ashamed about correcting myself. Rather, the time that passed, since I know about the mistake and did not manage to correct it :-( .

If the method were used, e.g. in a food control lab, it could mean health problems to people, who have an intolerance to the food compound, our analyte. Because it would tell them, it is not there and the food is safe.

Fortunately, to my best knowledge, the method is not in use.

Replies from: libero
comment by libero · 2021-04-19T15:47:25.002Z · LW(p) · GW(p)

Kudos for trying to address the issue, late is better than never. If you believe there are possible risks if the methods were used, the first things to do would be a retraction, since an erratum/corrigendum presuppose the consistency of the conclusion. Acting quickly could also prevent legal troubles.