There is a lot of arguing in the comments about what the 'tradeoffs' are for individuals in the scientific community and whether making those tradeoffs is reasonable. I think what's key in the quoted article is that fraudsters are trading so much for so little. They are actively obscuring and destroying scientific progress while contributing to the norm of obscuring and destroying scientific progress. Potentially preventing cures to diseases, time and life-saving technology, etc. This is REALLY BAD. And for what? A few dollars and an ego trip? An 11% instead of a 7% chance at a few dollars and an ego trip? I do not think it is unreasonable to judge this behavior as reprehensible, reguargless if it is the 'norm'.

Using peers in a field as a proxy for good vs. bad behavior doesn't make sense if the entire field is corrupt and destroying value. If 100% of scam artists steal people's money, I don't forgive a scam artist for stealing less money than the average scam artist. They are not 'making things better' by in theory reducing the average amount of money stolen per scam artist. They are still stealing money. DO NOT BECOME A SCAM ARTIST IN THE FIRST PLACE. If academia is all a scam, then that is very sad, but it does not make it ok for people to join in the scam and shrug it off as a norm.

And being fraudulent in science is SO MUCH WORSE than just being an ordinary scam artist who steals money. It's more like being a scam artist who takes money in exchange for poisoning the water and land with radioactive waste. No, it's not ok because other people are doing it.

I do not like this post. I think it gets most of its rhetorical oomph from speaking in a very moralizing tone, with effectively no data, and presenting everything in the worst light possible; I also think many of its claims are flat-out false. Let's go through each point in order.

1. You can excuse anything by appealing to The Incentives

No, seriously—anything. Once you start crying that The System is Broken in order to excuse your actions (or inactions), you can absolve yourself of responsibility for all kinds of behaviors that, on paper, should raise red flags. Consider just a few behaviors that few scientists would condone:

• Fabricating data or results
• Regulary threatening to fire trainees in order to scare them into working harder
• Deliberately sabotaging competitors’ papers or grants by reviewing them negatively

Wow, that would be truly shocking; indeed that would be truly an indictment of academia. What's the evidence?

When Diederik Stapel confessed to fabricating the data used in over 50 publications, he didn’t explain his actions by saying “oh, you know, I’m probably a bit of a psychopath”; instead, he placed much of the blame squarely on The Incentives:

... Did you expect people who were caught doing bad things to say "yup, I'm a terrible person?" Also, a single extreme case does not make the rule. I can only assume you're going to say that all the academics accepted this excuse; that would be an indictment of academia...

Curiously, I’ve never heard any of my peers—including many of the same people who are quick to invoke The Incentives to excuse their own imperfections—seriously endorse The Incentives as an acceptable justification for Stapel’s behavior. In Stapel’s case, the inference we overwhelmingly jump to is that there must be something deeply wrong with Stapel, seeing as the rest of us also face the same perverse incentives on a daily basis, yet we somehow manage to get by without fabricating data.

... I guess not. Well, point 1 is deeply unconvincing. (I also think it is false, just from my own experience in academia.) Let's hope the others do better.

2. It would break the world if everyone did it

I abhor arguments of this form; I don't think I've ever seen one used well. It would also break the world if everyone ate food but didn't farm; are you going to become a farmer now?

The actual content of the section is a bit different from the heading:

To be clear, I’m not saying perverse incentives never induce bad behavior in medicine or other fields. Of course they do. My point is that practitioners in other fields at least appear to have enough sense not to loudly trumpet The Incentives as a reasonable justification for their antisocial behavior—or to pat themselves on the back for being the kind of people who are clever enough to see the fiendish Incentives for exactly what they are.

... So academics are bad because they know that the bad incentives are there, and are willing to talk about them? Do you prefer the world in which they don't know the incentives are there or don't talk about them but still act in accordance with them? What?

we do seem to have collectively developed a rather powerful form of learned helplessness that doesn’t seem to be matched by other communities. Which is a fortunate thing, because if every other community also developed the same attitude, we would be in a world of trouble.

If true, that seems bad. I think it is mostly false; no evidence was given in the post. I feel like the author just sees more of academia than other fields; I don't get the sense that doctors, lawyers, journalists etc. are valiantly fighting off the bad incentives.

Okay, well, onwards to point 3:

3. You are not special

[...]

Well yeah, people actually do suffer. There are many scientists who are willing to do the right things—to preregister their analysis plans, to work hard to falsify rather than confirm their hypotheses, to diligently draw attention to potential confounds that complicate their preferred story, and so on. When you assert your right to opt out of these things because apparently your publications, your promotions, and your students are so much more important than everyone else’s, you’re cheating those people.

This seems to be an argument that most of the academics are truthful, and only a few bad actors are ruining it for everyone else. I certainly agree that if there are a few bad actors violating academic norms, those people are doing something bad / should be punished. But I thought the claim was that nearly all academics are doing bad things? If so, you can't reasonably argue that these people are being "special".

You can't be willfully blind to norms and then claim that people following the norms are bad and think they are "special".

Let's say you want to smoke pot, and you're in an area where there's a law against smoking pot. However, no one enforces it and everyone smokes pot all the time. Perhaps if everyone stopped smoking pot, you could make the case that smoking pot isn't addictive and get the law overturned. It is incorrect to blame everyone else for smoking pot, or to say that they are defecting, even though it is true that the law could be reversed if everyone stopped smoking pot.

If the claim is actually that there are only a few bad actors, then I assume the author must be talking about the especially bad cases like fabricating data. I expect that when caught such cases really are shamed, and you can't get out of it by saying "Oh, The Incentives".

4. You (probably) have no data

It’s telling that appeals to The Incentives are rarely supported by any actual data. It’s simply taken for granted that engaging in the practice in question would be detrimental to one’s career.

This is a clear isolated demand for rigor [LW · GW]. (Ironically, this post itself has no data, making it even clearer than usual.)

Coming by the kind of data you’d need to answer this question is actually not that easy: it’s not enough to reflexively point to, say, the fact that some journals have higher impact factors than others, To identify the utility-maximizing course of action, you’d need to integrate over both benefits and costs, and the costs are not always so obvious.

I can only assume that the author decides when to shower, what to eat, what transit to use, etc by simulating out all the possible consequences of each possible decision several years into the future, aggregating the consequences according to their personal utility function, and then choosing the one that is best. I must applaud the author for their immense brainpower; unfortunately us mere mortals can't do this ourselves and need to rely on heuristics.

Sarcasm aside; I think it is perfectly possible to make good decisions without having all the available data. We do this all the time in lots of different scenarios. Why expect that people can't do it here as well?

5. It (probably) won’t matter anyway

Outcomes in academia are multiply determined and enormously complex. You can tell yourself that getting more papers out faster will get you a job if it makes you feel better, but that doesn’t make it true. If you’re a graduate student on the job market these days, I have sad news for you: you’re probably not getting a tenure-track job no matter what you do. It doesn’t matter how many p-hacked papers you publish, or how thinly you slice your dissertation into different “studies”; there are not nearly enough jobs to go around for everyone who wants one.

Conditions of extreme competition over scarce resources are exactly where you expect the most conformance to the incentives.

Suppose you’re right, and your sustained pattern of corner-cutting is in fact helping you get ahead. How far ahead do you think it’s helping you get? Is it taking you from a 3% chance of getting a tenure-track position at an R1 university to an 80% chance? Almost certainly not. Maybe it’s increasing that probability from 7% to 11%; that would still be a non-trivial relative increase, but it doesn’t change the fact that, for the average grad student, there is no full-time faculty position waiting at the end of the road.

I'll note that these are once again claims with no data. I think it's false for the people who follow The Incentives who do become professors; I would not be surprised if they went from 20% to 80%, which seems like a substantial change. I agree that on average it's probably not very high, because a lot of people are not going to become professors regardless (i.e. maybe they go from 2% to 3%), but it's hard to tell from the inside which of these situations you're in.

6. You’re (probably) not going to “change things from the inside”

Over the years, I’ve talked to quite a few early-career researchers who have told me that while they can’t really stop engaging in questionable research practices right now without hurting their career, they’re definitely going to do better once they’re in a more established position.

[...]

I can think of at least a half-dozen people off-hand who’ve regaled me with me some flavor of “once I’m in a better position” story, and none of them, to my knowledge, have carried through on their stated intentions in a meaningful way.

I totally agree with this point.

7. You’re not thinking long-term

[...] One thing that I think has been largely overlooked in discussions about the current incentive structure of science is what impact the replication crisis will have on the legacies of a huge number of presently famous scientists.

I’ll tell you what impact it will have: many of those legacies will be completely zeroed out. And this isn’t just hypothetical scaremongering. It’s happening right now to many former stars of psychology (and, I imagine, other fields I’m less familiar with).

[...] So if your justification for cutting corners is that you can’t otherwise survive or thrive in the present environment, you should consider the prospect—and I mean, really take some time to think about it—that any success you earn within the next 10 years by playing along with The Incentives could ultimately make your work a professional joke within the 20 years after that.

I am pretty uncertain about this point. Most people do not become famous and have their work scrutinized in detail. I agree that long-term considerations do point in the direction of bucking the incentives; I just don't know how strongly.

8. It achieves nothing and probably makes things worse

(The "It" here is "complaining about incentives", not "following the incentives".)

If your complaints are achieving anything at all, they’re probably actually making things worse by constantly (and incorrectly) reminding everyone around you about just how powerful The Incentives are. Here’s a suggestion: maybe try not talking about The Incentives for a while.

I don't really see the point here. Surely fixing the problem involves first creating common knowledge about the problem? But in any case, this seems to be orthogonal to the main point (that following the incentives is bad), so I'm going to skip it.

9. It’s your job

This last one seems so obvious it should go without saying, but it does need saying, so I’ll say it: a good reason why you should avoid hanging bad behavior on The Incentives is that you’re a scientist, and trying to get closer to the truth, and not just to tenure, is in your fucking job description. Taxpayers don’t fund you because they care about your career; they fund you to learn shit, cure shit, and build shit.

This same complaint can be leveled at most of society. You don't get to choose a job description and then have people faithfully execute that job description; what you get depends on the incentives you give.

Would the world be better if you could just give a job description and have people faithfully do that? Probably. Does the world work that way? No. Just yelling "I want you to do X, therefore you should do X" seems counterproductive and willfully blind about how the world works.

----

I do not know what people get out of this post; the arguments given do not remotely support the conclusions, with a couple of exceptions. If I had to guess some effects that would make people upvote / nominate the post:

1. Powerfully suggested that people should bear responsibility for bucking the incentives
2. Written as a rant, and so was fun to read
3. Anti-academia, which agrees with the reader's beliefs
4. Pro-principles, which agrees with the reader's beliefs / morals

(I am not claiming that people consciously thought about these reasons.)

I'll ignore 2, 3, and 4; I think it's obvious why those are not reasons we want to promote on LW.

I think the first one could plausibly be a reason that we would want to promote this on LW. Unfortunately, I think it is wrong: I do not think that people should usually feel upon themselves the burden of bucking bad incentives. There are many, many bad incentives in the world; you cannot buck them all simultaneously and make the world a better place. Rather, you need to conform with the bad incentives, even though it makes your blood boil, and choose a select few areas in which you are going to change the world, and focus on those.

(Tbc, this is not a recommendation to p-hack; that is not the norm in academia. The norm in academia was to do other questionable things that in effect constitute p-hacking, but it was never to explicitly p-hack. And that norm might be changing at this point, I don't know. Personally, while there are questionable academic practices that I follow when writing papers, such as including all the pros of a proposal in the introduction but ignoring the cons, I try to make my experiments as un-p-hacked as I can, though I have the advantage of not seeking a job in academia, as well as a good network, so incentives affect me less than they do most academics.)

The discussion around It's Not the Incentives, It's You, was pretty gnarly. I think at the time there were some concrete, simple mistakes I was making. I also think there were 4-6 major cruxes of disagreement between me and some other LessWrongers. The 2019 Review seemed like a good time to take stock of that.

I've spent around 12 hours talking with a couple people who thought I was mistaken and/or harmful last time, and then 5-10 writing this up. And I don't feel anywhere near done, but I'm reaching the end of the timebox so here goes.

Core Claims

I think this post and the surrounding commentary (at least on the “pro” side) was making approximately these claims:

A. You are obligated to buck incentives. You might be tempted sometimes to blame The Incentives rather than take personal responsibility for various failures of virtue (epistemic or otherwise). You should take responsibility. 

B. Academia has gotten more dishonest, and academics are (wrongly) blaming “The Incentives” instead of taking responsibility.

C. Epistemics are the most important thing. Epistemic Integrity is the most important virtue. Improving societal epistemics is the top cause area. 

• Possible stronger claim: Lying or manipulating data on the public record is (sometimes? often?) worse than state-sanctioned killing.

D. Special Academic Responsibility: Scientists/Academics have a special responsibility to never be dishonest about their work, and/or to be proactively extremely honest, due to the nature of their profession, independent of the current standards of their profession. If they can’t, they should leave.

E. The previous points are not just correct, but obviously correct.

This is just a short summary, and the claims are necessarily simplified. I think if you add lots of hedging words like “often”, and “mostly”, some of the claims become less controversial. 

But I disagreed a lot with some people on how much to weight them. It seemed to me they were leaning a lot more towards absolutism, in a domain that seemed to me to be legitimately murky and confusing.

–

Quick tl;dr on my takes on those claims: 

Re: Bucking Incentives: You should buck incentives at least some of the time, and you should allocate a lot of attention to "notice when incentives are pressuring you, and think about what you really value." 

I'm not sure about "exactly how much" and "when?".

Re: Academia is more dishonest: I’m mostly agnostic on this. Important if true. Rohin argues with this claim, and I think people who want to debate it should respond to him.

Re: Epistemics are Most Important Thing: The previous discussion actively changed my mind on this. Previously I believed “epistemics are maybe in the top 5 causes”. I now think that Epistemics are plausibly the very top cause and most important virtue. At the very least, they are much more important than they seemed to me at the time. (Note that I still maintain that there are degrees of freedom in "how exactly do you accomplish good epistemics", which truthseekers can disagree on [LW · GW])
 
I am still somewhat confused about "lying on public record vs state sanctioned killing". I now think it is plausible. But it is a pretty intense claim (which I think most society disagrees with). I am still mulling over, and am unclear on some details and not even sure what the people I've argued with believe about it.

RE: Special Academic Responsibility: IMO this depends a lot on the current state of academia, and what its onboarding culture is like. 

RE: "This is all obvious": This is importantly false, and this is what I was mostly arguing against last year, and still argue against now.

–

I have thoughts on individual pieces of this, which I'll write as followup comments as I find time.

I can't upvote this strongly enough. It's the perfect followup to discussion and analysis of Moloch and imperfect equilibria (and Moral Mazes) - goes straight to the heart of "what is altruism?" If you're not taking actions contrary to incentives, choosing to do something you value that "the system" doesn't, you're not making moral choices, only economic ones.

Very nice. Few notes:

1. Wrong incentives are no excuse for bad behaviour, they should rather quit their jobs than engaging in one.

2. World isn't black or white, sometimes there is a gray zone where you contribute enough to be net+, while cut some corners to get your contribution accepted.

3. People tend to overestimate their contribution and underestimate the impact of their behaviour, so 2. is quite dangerous.

4. In an environment with sufficiently strong wrong incentives, the only result is that only those with weak morals survive. Natural selection.

5. There is lot of truth in Taleb's position that research should not be a source of your income, rather a hobby.

I disagree with most of the post and most of the comments here. I think most academics are not explicitly committing fraud, but bad science results anyway. I also think that for the vast majority of (non-tenured) academics, if you don't follow the incentives, you don't make it in academia. If you intervened on ~100 entering PhD students and made them committed to always not following the incentives where they are bad, I predict that < 10% of them will become professors -- maybe an expected 2 of them would. So you can't say "why don't the academics just not follow the incentives"; any such person wouldn't have made it into academia. I think the appropriate worlds to consider are: science as it exists now with academics following incentives or ~no academia at all.

It is probably correct that each individual instance of having to deal with bad incentives doesn't make that much of a difference, but there are many such instances. Probably there's an 80-20 thing to do here where you get 80% of the benefit by not following the worst 20% of bad incentives, but it's actually quite hard to identify these, and it requires you to be able to predict the consequences of not following the bad incentives, which is really hard to do. (I don't think I could do it, and I've been in a PhD program for 5 years now.)

To be clear: if you know that someone explicitly and intentionally committed fraud for personal gain with the knowledge that it would result in bad science, that seems fine to punish. But this is rare, and it's easy to mistake well-intentioned mistakes for intentional fraud.

Related: https://www.lesswrong.com/posts/x5ASTMPKPowLKpLpZ/moloch-s-toolbox-1-2#NStyom7bTek7QiGJv [LW(p) · GW(p)]

I did not follow the Moral Mazes discussion as it unfolded. I came across this article context-less. So I don't know that it adds much to Lesswrong. If that context is relevant, it should get a summary before diving in. From my perspective, its inclusion in the list was a jump sideways.

It's written engagingly. I feel Yarkoni's anger. Frustration bleeds off the page, and he has clearly gotten on a roll. Not performing moral outrage, just *properly, thoroughly livid* that so much has gone wrong in the science world.

We might need that.

What he wrote does not only apply to scientists. It is *written* as though scientists are a unique concern. He makes strong claims about the way things are in academia very casually, as if tacitly assuming anyone with eyes would have seen enough to corroborate it themselves. (Whether they would agree, I don't know. For followup work I would like to see more concrete estimates and hard data about the extent and impact of these issues.) He writes this for his peers, I think, it almost feels intimate and like I ought to feel embarassed for eavesdropping.

But as I said, scientists don't seem like a special case. The loss of integrity in science is what he knows, but I suspect it's a much wider trend and he's rather gelman amnesia'd about the extent to which it exists in other fields. From a quick perusal of the Moral Mazes tag... it sure seems like. In Anna Salamon's question post about institutions, the dissolution of personal integrity is up there in the list of possible causes for modern institutional decay. He writes "if every other community also developed the same attitude, we would be in a world of trouble", but I don't think he should be so confident that they *didn't*.

This article is a kick in the pants. I don't tend to like that approach because I have empathy for people who are afraid. It is exactly when they are making decisions from fear that they don't support their decisions with data or reason, as Yarkoni complains about. I continue to believe that you can't really fix fear with punishment and shaming. But I'm coming around to the idea that the wake-up call needs to come before the support.

As off-audience as this post seems... the message is timely and on brand.

We are not slave to incentives. Constraining your goals to be safely within the bounds of ease and social agreeableness is NOT HOW YOU ACTUALLY WIN, and we are playing to win. Shut up and multiply.

This post substantially updated my thinking about personal responsibility.  While I totally disagree with the one-side framing of the post, the framing of it made me see that the "personal responsibility" vs. "incentives" thing wasn't really about beliefs at all, but was in fact about the framing.

I think it articulates the "personal responsibility" frame particularly well, and helps see how choosing "individuals" as the level of abstraction naturally leads to a personal responsibility framing.

In general, I think this post does a great job of articulatng a single, incomplete frame. Others in the review take umbrage with the moralizing tone, but I think the moralizing tone is actually quite useful to give an inside view of this frame. 

I believe this frame is incomplete, but gives an important perspective that is often ignored in the Lesswrong/Gray tribe.

While I don't think this post is actually eligible for the Best of LW 2019 book (since it's written offsite and is only a linkpost here), I think it's reasonable to nominate the comments here for some kind of "what do we collectively feel about this 1.5 years later?" discussion.

Definitely think this is an important point in the conversation.

I think my take is something like "The incentives are the problem" is a useful frame for how to look at systems and (often but not always) other people, but should throw up a red flag when you use it as an excuse for your own behavior.

I'm not sure I endorse this post precisely as written, because "take ownership of your behavior" is a cause that will be Out To Get You [LW · GW] for everything you've got (while leaving you vulnerable to Asymmetric Justice in the meanwhile). There are lots of things you [generic you] probably do or are complicit in, that have a bad effect on other people. If the norm in academia is to use bad statistics, or fake data (I don't know whether it is or not, or how common it is), and I'm an academic, should I be more worried about avoiding that norm, or avoiding eating meat (which personally seems worse to me), or some third thing I do that causes harm?

The things that actually seem workable to me are:

• Try to be in the top 50% of the population at morality
• In general practice the muscle of defying social norms that push you to do wrong things, and pick at least some areas where you commit to be significantly better than the status quo.
• "don't play the game" may not always be an option, but "try to create situations where you *actually* create opportunities stag hunts [LW · GW] to succeed." i.e. in this case, maybe try to at least make it true that at your department or university or conference, it is unacceptable to fake or exaggerate data. This is a longterm battle that won't work if you halfass it.

Writing posts a certain way to get more karma on lesswrong is an area of application for this stance.