Power Buys You Distance From The Crime

Just thinking loudly about the boundaries of this effect...

Suppose I have a garden, and a robot that can pick apples. I instruct the robot to bring me the "nearest big apple" (where "big" is exactly defined as having a diameter at least X). Coincidentally, all apples in my garden are small, so the robot picks the nearest apple at neighbor's garden and brings it to me, without saying anything.

When the neighbor notices it, he will be angry at me, but I can defend myself that I made an innocent mistake; I didn't mean to steal apples. This may be a good excuse for the first time; but if the thing keeps happening, the excuse no longer works; it was based on me not knowing.

Also, the robot will not even try to be inconspicuous; it will climb straight over the fence, in the middle of a day, even when the neighbor is there and looking at it. If I try to avoid this, by giving instructions like "bring me the nearest apple, but make sure no one notices you", I lose the plausible deniability. (I am assuming here that if my neighbor decides to sue me, the judge will see the robot's programming.)

Now why specifically does the situation change when instead of a robot I use a slave? It seems to me the only difference is that I can (knowingly or not, or anything between that) achieve an outcome where the slave will steal for me, using his intelligence to avoid detection; and if caught anyway, I can afterwards claim that the slave decided to steal on his own will; not because of my commands (either ordering him to steal, or just failing to mention that theft is prohibited), but specifically and intentionally against them. That relieves of all blame.

Okay, so how specifically do I make a slave steal for me, without giving the explicit order? I have to give a command that is impossible (or highly unlikely) to accomplish otherwise. For example, my garden only contains small apples, and I command the slave to bring me a big apple; threatening to whip him if he fails. The only way to avoid whipping is to steal, but hey, I never mentioned that! Also, it is a common knowledge that (unlike the robot) the slave is supposed to know that stealing is wrong.

But this naive strategy fails if the slave checks my garden and then tells me "master, your garden only contains small apples; what am I supposed to do?" Saying "I don't care, just bring the f-ing apple or feel my wrath" will achieve the purpose, but it puts a dent into my deniability. To do things properly, I must be able to claim afterwards that I believed that my garden contains big apples, and therefore I believed that my orders can be fulfilled legitimately. (Bonus points if afterwards, despite the slave's testimony of the contrary, I can insist with indignation that my garden still contains big apples, i.e. the slave is just lying to save his ass.)

Therefore, I need to remove the communication channel where the slave could tell me that there are no big apples in my garden. A simple approach is to give the command, along with the threat, and then leave. The slave knows that he gets whipped if the apple is not on my table by sunset, and I am not there to communicate the fact that the order cannot be achieved legitimately. So he either takes the risk and waits for my return -- betting on my sense of fairness, that I wouldn't punish him for not having accomplished the impossible -- or decides that it is relatively safer to simply steal the apple. This is better, but still too random. I could increase the probability of stealing by cultivating an image of a hot-tempered master who punishes firsts and asks questions later. But there are still ways this could fail (e.g. the slave could say afterwards "master, this was the last big apple in your garden, the remaining ones are all small", which would remove my deniability for giving the same command the next day).

A more complex approach is to make it know that talking about small apples is a taboo, and all violations of this taboo will be punished. Now the slave will not dare mention that the apples in my garden are small, but now I need a plausible excuse for the taboo. Is perhaps my garden a source of my pride, and thus I take any criticism of my garden as a personal offense? That could work. I only need to establish the rule "I am proud of my garden and any kind of criticism will be severely punished" sufficiently long before I start ordering my slaves to steal; to avoid the impression that I established that rule exactly for that purpose.

Summary: a command that is impossible (or unlikely) to be accomplished legitimately; a threat of punishment; and proactively destroying the feedback channel under some pretense.

Now I'd like to give some advice on how to notice when you are in a similar situation -- where the feedback channels are sabotaged, and perhaps it's just a question of time when you receive the impossible command, and will have to use your own will to break the rules and take the responsibility for the decision -- but actually, all situations with power differential are to smaller or greater degree like this. People usually can't communicate with their superiors openly. (Not even when the boss says "actually, I prefer if you communicate with me openly". Seriously, don't. What that statement actually means is that talking about difficulties in communication to superiors is also a taboo. Generally, a company statement "X is true" is usually best translated as "saying 'X is false' will be punished".)

A practical advice would perhaps be to notice the local taboos, and maneuver to avoid coming into contact with them. If a person X cannot be criticized, avoid being on the same team as X. If software Y cannot be criticized, avoid work that involves using software Y. If ideology Z cannot be criticized, avoid projects where Z applies most strongly. Yeah, I know, this is easier said than done, and it does not work reliably (e.g. you can choose a different team, and the next day X switches to that team, too).

ETA 1/12: This review is critical and at times harsh, not because I want to harshly criticize the post or the author, but because I did not consider harshness of criticism when writing. I still think the post is positive-net-value, and might even vote it up in the review. I especially want to emphasize that I do not think it is in any way useful to blame or punish the author for the things I complain about below; this is intended as a "pointing out a problematic habit which a lot of people have and society often encourages" criticism, not a "bad thing must be punished" criticism.

When this post first came out, I said something felt off about it. The same thing still feels off about it, but I no longer endorse my original explanation of what-felt-off. So here's another attempt.

First, what this post does well. There's a core model which says something like "people with the power to structure incentives tend get the appearance of what they ask for, which often means bad behavior is hidden". It's a useful and insightful model, and the post presents it with lots of examples, producing a well-written and engaging explanation. The things which the post does well more than outweigh the problems below; it's a great post.

On to the problem. Let's use the slave labor example, because that's the first spot where the problem comes up:

No company goes “I’m going to go out and enslave people today” (especially not publicly), but not paying people is sometimes cheaper than paying them, so financial pressure will push towards slavery. Public pressure pushes in the opposite direction, so companies try not to visibly use slave labor. But they can’t control what their subcontractors do, and especially not what their subcontractors’ subcontractors’ subcontractors do, and sometimes this results in workers being unpaid and physically blocked from leaving.

... so far, so good. This is generally solid analysis of an interesting phenomenon.

But then we get to the next sentence:

Who’s at fault for the subcontractor(^3)’s slave labor?

... and this where I want to say NO. My instinct says DO NOT EVER ASK THAT QUESTION, it is a WRONG QUESTION, you will be instantly mindkilled every time you ask "who should be blamed for X?".

... on reflection, I do not want to endorse this as an all-the-time heuristic, but I do want to endorse it whenever good epistemic discussion is an objective. Asking "who should we blame?" is always engaging in a status fight. Status fights are generally mindkillers, and should be kept strictly separate from modelling and epistemics.

Now, this does not mean that we shouldn't model status fights. Rather, it means that we should strive to avoid engaging in status fights when modelling them. Concretely: rather than ask "who should we blame?", ask "what incentives do we create by blaming <actor>?". This puts the question in an analytical frame, rather than a "we're having a status fight right now" frame.

Some sections of the post do discuss things in that sort of analytical frame, and those sections are where I see most of the value. Unfortunately, they're mixed in with parts which don't use an analytical frame. For instance, in the "Dating an Artist" example, we see "It wasn’t my fault he felt that pressure..." -  a sentence which is engaging in a status fight. But then two sentences later, there's a great analysis of the incentives created by precommitting to housing provision. If the sentences about how "Time in my spare bedroom was a gift to him I had no obligation to keep giving" were stripped out, and just the analysis were left behind, then the post would be dramatically better.

Now, I'm sure somebody's going to come along and object that we can never fully separate the status fights from the status-fight-models in reality; people will always sneak their status fights into epistemic discussions. My answer is that this is the fallacy of gray [? · GW]:

The Sophisticate: “The world isn’t black and white. No one does pure good or pure bad. It’s all gray. Therefore, no one is better than anyone else.”

The Zetet: “Knowing only gray, you conclude that all grays are the same shade. You mock the simplicity of the two-color view, yet you replace it with a one-color view . . .”

—Marc Stiegler, David’s Sling

Yes, the status fights will sometimes sneak into epistemic discussions. Ideally, we want epistemic discussion norms which are robust to the presence of status fights. "Don't engage in explicit status fights, use an analytic frame instead" is one such norm - it is not sufficient on its own, but it seems to take a large step in the right direction, and in practice it is the norm which seems most important for keeping epistemic discussion sane.

I can still see where there's room to quibble about using this norm all the time, but at the very least it is a norm which would improve overall epistemics if applied to posts like this.

This case is more complicated than the corporate cases because the powerful person (me) was getting merely the appearance of what she wanted (a genuine relationship with a compatible person), not the real thing. And because the exploited party was either me or Connor, not a third party like bank customers. No one thinks the Wells Fargo CEO was a victim the way I arguably was.

I think you have misunderstood the Wells Fargo case. These fake accounts generally didn't bring in any material revenue; they were just about internal 'number of new accounts targets'. It was directly a case of bank employees being incentivised to defraud management and investors, which they then did. If ordinary Wells employees had not behaved fraudulently, all the targets would have been missed, informing management/investors about their mis-calibration, and more appropriate targets would have been set. In this case power didn't buy distance from the crime, but only in the sense that it meant you couldn't tell you were being cheated.

For more on this I recommend the prolific Matt Levine:

There's a standard story in most bank scandals, in which small groups of highly paid traders gleefully and ungrammatically conspire to rip-off customers and make a lot of money for themselves and their bank. This isn't that. This looks more like a vast uprising of low-paid and ill-treated Wells Fargo employees against their bosses.

...

So that's about 2.1 million fake deposit and credit-card accounts, of which about 100,000 -- fewer than 5 percent -- brought in any fee income to Wells Fargo. The total fee income was $2.4 million, or about $1.14 per fake account. And that overstates the profitability: Wells Fargo also enrolled people for debit cards and online banking, but the CFPB doesn't bother to count those incidents, or suggest that any of them led to any fees. Which makes sense: You'd expect online banking and debit cards to be free, if you never use them or even know about them. Meanwhile, all this dumb stuff seems to have occupied huge amounts of employee time that could have been spent on more productive activities. If you divide the $2.4 million among the 5,300 employees fired for setting up fake accounts, you get about $450 per employee. Presumably it cost Wells Fargo way more than that just to replace them.

In the abstract, you can see why Wells Fargo would emphasize cross-selling of multiple "solutions" to customers. It is a good sales practice; it both indicates and encourages customer loyalty. If your customers have a checking account, and a savings account, and a credit card and online banking, all in one place, then they'll probably use each of those products more than if they had only one. And when they want a new, lucrative product -- a mortgage, say, or investment advice -- they're more likely to turn to the bank where they keep the rest of their financial life. 

But obviously no one in senior management wanted this. Signing customers up for online banking without telling them about it doesn't help Wells Fargo at all. No one feels extra loyalty because they have a banking product that they don't use or know about. Even signing them up for a credit card without telling them about it generally doesn't help Wells Fargo, because people don't use credit cards that they don't know about. Cards with an annual fee are a different story -- at least you can charge them the fee! -- but it seems like customers weren't signed up for many of those.  This isn't a case of management pushing for something profitable and getting what they asked for, albeit in a regrettable and illegal way. This is a case of management pushing for something profitable but difficult, and the workers pushing back with something worthless but easy.

Something about this piece felt off to me, like I couldn't see anything specifically wrong with it but still had a strong instinctive prior that lots of things were wrong.

After thinking about it for a bit, I think my main heuristic is: this whole piece sounds like it's built on a conflict-theory worldview. The whole question of the essay is basically "who should we be angry at"? Based on that, I'd expect that many or most of the individual examples are probably inaccurately understood or poorly analyzed. Lark's comment about the Wells Fargo case confirms that instinct for one of the examples.

Then I started thinking about the "conflict theory = predictably wrong" heuristic. We say "politics is the mindkiller [LW · GW]", but I don't think that's quite right - people have plenty of intelligent discussions about policy, even when those discussions inherently involve politics. "Tribalism is the mindkiller" is another obvious formulation, but I'd also propose "conflict theory is the mindkiller". Models like "arguments are soldiers [? · GW]" or "our enemies are evil [? · GW]" are the core of Yudkowsky's original argument [? · GW] for viewing politics as a mind-killer. But these sort of models are essentially synonymous with conflict theory; if we could somehow have a tribalistic or political discussion without those conflict-theoretic elements, I'd expect it wouldn't be so mindkiller-ish.

Looping back to the main topic of the OP: what would be a more mistake-theoretic way to view the same examples? One theme that jumps out to me is principal-agent problems: when something is outsourced, it's hard to align incentives. That topic has a whole literature in game theory, and I imagine more useful insight could be had by thinking about how it applies to the examples above, rather than thinking about "moral culpability" - a.k.a. who to be angry at.

This post makes a fairly straightforward point that has been vary helpful for thinking about power. Having several grounding concrete examples really helped as well. The quote from moral mazes that gave examples of the sorts of wiping-hands-of-knowledge things executives actually say really helped make this more real to me.

Explains a particular social phenomenon that I hadn't previously been aware of.

Thanks for this insightful piece.

It seems to me that there's a third key message, or possibly a reframing of#1, which is that people without power should be considered less morally culpable for their actions -eg the Wells Fargo employees should be judged less harshly.

The concept of "human error" is often invoked to explain system breakdown as resulting from individual deficiencies (eg, early public discussion of the Boeing 737 MAX crashes had an underlying theme of "Ethiopian and Indonesian pilots are just not as skilled as American pilots") - but a human factors / resilient engineering perspective recognises that humans' roles in technical systems can be empowered or constrained by the system design. And of course it was other humans who designed (approved, built, ...) the system in the first place.

Stumbled across a laww concepts that splits "power" in this context into finer distinctions.

There is a concept of "qualified immunity". If a police officer makes an act in the course of official duties lawsuits regarding that should be addressed to the police department not to the individual police officer. Power there does not buy distance to the crime but on the contrary great power leads to great accountability even responcibility for things that do not have rules governing them but maybe should. There it can be clear that a police officer used his judgement and took initiative on his own accord but can still use the organizations blame shield.

It would be silly to say "It was not me who stole, it was my hand". Focusing on which individual effected the blameworthy action within an organization can distract from the framework that the organization is responcible for the result. So saying that "it was not the hand that was at fault, it was the brain" might be in some sense be more accurate but it is important to catch the point that the human stole, not individual organs.

Thanks for writing this, but I don't believe there's broad agreement on either of your main examples.

I don't know anyone who claims taxes should be only proportional to wealth or income. There are those who say it should be super-linear with income or wealth (tax the rich even more than their proportion of income), and those who say not to tax based on wealth or income, but on consumption (my preference) or the hyper-pragmatic "tax everyone, we need the money". And of course many more nuanced variations.

I likewise don't think that most blame goes to people had power but failed to stop a harm - it goes to people who were active in the harm ( ideally; but often people who are active near the harm, even if not causal. https://www.lesswrong.com/posts/YRgMCXMbkKBZgMz4M/asymmetric-justice [LW · GW] ).

Agreed that in both cases, some dimensions of power can distort the straightforward application of principles, but I argue that that's true mostly BECAUSE the principles are not clearly agreed to by most people. Power exploits the disagreements by diverting attention toward the interpretations that favor the power. "Obscures information flow" is a misleading framing. It's closer to "diverts attention toward different models". And mostly on topics complicated enough that it's hard to say what's factually best, only what preferences are prioritized.