Consequentialism Need Not Be Nearsighted

Well, that's the thing: some people do. Even obvious things can require some explanation.

It would have quite a few bad knockon effects:

1) you have handed the government the ability to decide, at any point, to kill anyone they consider undesirable, provided they can find five compatible transplant recipients; this is a massive increase in their power, and a big step towards totalitarian society.

2) You are discouraging people from undergoing genetic screening

3) you are discouraging people from living healthily. If you are unhealthy, your organs are of less use to the government, and hence you are more likely to survive.

4) you are encouraging people to go off the grid; as people who are off the grid are less likely to be found for the purposes of harvesting.

Yes, these logical reasons are not directly the reason people are creeped out; but were you to find a less harmful scenario, you would also likely find the scenario less creepy.

For instance, most people would find it less creepy if the harvesting was limited only to those who are already in prison, on long (20 year+) sentences; and it also seems that that policy would have less indirect harms.

As far as I can tell, this would have no bad effects beyond the obvious one of killing the people involved - it wouldn't make people less likely to go to hospitals or anything

No, but it would make them afraid to go outside, or at least within the vicinity of police. This law might encourage people to walk around with weapons to deter police from nabbing them, and/or to fight back. People would be afraid to get genetic screening lest they make their organs a target. They would be afraid to go to police stations to report crimes lest they come out minus a kidney.

People with good organs would start bribing the police to defer their harvesting, and corruption would become rampant. Law and order would break down.

This sounds like an excellent plot for a science fiction movie about a dystopia, which indicates that it fails on consequentialist grounds unless our utility function is so warped that we are willing to create a police state to give organ transplants.

Fourth reply: people deeply value autonomy.

Fifth reply: While in this case I don't think that the policy is the right consequentialist thing to do, in general I expect consequentialism to endorse some decisions that violate our current commonsense morality. Such decisions are usually seen as moral progress in retrospect.

The probability of being killed in such a way would be tiny and wouldn't significantly alter expected lifespan. However people are bad at intuitive risk evaluation and even if any person would at least twice more likely have their life saved than destroyed because of the policy, people would feel endangered and unhappy, which fact may overweigh the positive benefit. But if this concern didn't apply (e.g. if most people learned to evaluate risks correctly on the intuitive level), I'd bite the bullet and vote for the policy.

By the way, upvoted for correct application of least convenient possible world technique.

Good point. My first objection is the same as prase's, my second is that a government that depends on popular support shouldn't enforce policies that creep out the citizens (either because they'd lose the next election in a landslide to a party with different values or decision theory, or because in a Parfit's Hitchhiker way, if it were clear they would do such things then they'd have lost the previous election).

My third is that the creepiness here comes from our very real (and very understandable in consequentialist terms) fear of allowing the government too much power over citizens' life and death. If instead you asked, should we make it legal for people to voluntarily off themselves when by so doing they can save several other lives, and should we make it an honorable thing to do, then I'm not nearly as disturbed by the idea. (There are myriad variations to try out, but generally whenever it gets creepy I can identify some sort of awful incentive structure being set up.)

I think the first hurdle is whether this a worthwhile policy at all. We'd need to weigh the expected cost in donor lives against the improvement in recipient lives; I would think that the factors which make for the best donors like youth and health would tend to militate against the policy given the profile of recipients, often older and already known to be unhealthy. Secondly, even if the program would be positive sum, we'd need to weigh it against alternatives (like forced donation at death) to ensure that it was actually the best policy possible.

Obviously, you could restate the hypothetical until all the factors which must be weighed demand Policy 145. But this is almost certainly a trivial exercise provided by the rule set governing consequentialism. At that point, however, I think there are a few responses available: (1) the world is not recognizable to me and I cannot, even with difficulty, really imagine what the balance of worlds close to it would be like; (2) while this world sounds great, I think I'm better off in this one and so I can safely say that I do not prefer it for myself; (3) the world is so different from the actual world that it is difficult to say whether such a world would be internally consistent if usefully similar to our own.

I think response (1) allows us to "bite the bullet" on paper, knowing it will never be fired; response (2) seems like it may usefully encapsulate our problems with the hypothetical and generate the correct response "good for them, then"; (3) this response allows a denial or agnosticism about the world and "out there" hypotheticals in general.

I think the proper response to this process is all three: I should agree were it so; I should properly recognize that I don't like it (and don't have to); and I can deny that the hypothetical reveals any important information about theory. I think these responses could be elided, though, simply by noting what was suggested earlier: given a static rule set and a fully malleable world, the generation of repugnant (or euphoric) results is trivial and thus unhelpful.

Why not just label the organ collection a "tax", and say "Even if the tax burdens some people disproportionately, it helps people more than it harms people, and is therefore justified"?

But people still die.

I think a major part of how our instinctive morality works (and a reason humans, as a species, have been so successful) is that we don't go for cheap solutions. The most moral thing is to save everyone. The solution here is a stopgap that just diminishes the urgency of technology to grow organ replacements, and even if short-term consequentially it leaves more people alive, it in fact worsens out long-term life expectancy by not addressing the problem (which is that people's organs get damaged or wear out).

If a train is heading for 5 people, and you can press a switch to make it hit 1 person, the best moral decision is "I will find a way to save them all!" Even if you don't find that solution, at least you were looking!

I think strictly speaking consequentalism is a property of first-order ethical theories. That is they either are consequentialist or are not. But it is not by itself a first-order theory.

Ah, that was a mistake- thanks for catching it.

You've identified a subtle problem with implementing decision theories, but the answer that the commonsense version of TDT "should" give is pretty clear: if the differences between two situations don't massively affect the utilities involved (from the perspective of the deciding agent), then they should belong to the same reference class.

If you shouldn't kill the patient in a house, then you still shouldn't kill him with a mouse, or on a boat, with a goat, on a train, in the rain, here or there, or anywhere.

TDT seems to sometimes fail this, depending on whether the other scenario counts as the 'same computation', whatever that means. UDT postulates the existence of a 'mathematical intuition module' that can tell which computations influence which others, but it is not known how such a module could be created. Developing one would probably constitute a large fraction of the difficulty of creating AGI.

That means CON+TDT doesn't prohibit a decision to carve up a vagrant for organs conditional on some unique feature of the situation.

Provided that the unique feature is relevant, no it does not. For example, if the vagrant's parts were capable of saving 1,000 lives (a very unlikely situation, and not one anyone needs to worry of finding themself in) that would be a relevant unique feature.

However merely noticing that the vagrant is wearing a red baseball cap, made in 1953, and has $1.94 in their left pants pocket; while unique, is irrelevant. And as such it is easily modelled by using the protocol "insert random, irrelevant, unique aspect".

That's a good point, and why I like Gary Drescher's handling of such dilemmas in Good and Real (ch. 7), which allows a kind of consequentialist but acausal (or extracausal) reasoning. His decision theory is the TDT-like "subjunctive reciprocity" which says that "You should do what you self-interestedly wish all similarly-situtated beings would do, because if you would regard it as optimal, so would they". That is, as a matter of the structure of reality, beings that are similar to you have similar input/output relationships -- so make sure you instantiate "good" I/O behavior.

The upshot is that subjunctive reciprocity can justify not harvesting in this case, and similar cases where we can assume no negative causal fallout from the doctor's decision. Why? Because if the doctor reasons that way (kill/harvest the patient "for the greater good") it logically follows that other people in similar situations would do the same thing (and thereby expect and adapt for it), even though there would be (by supposition) no causal influence between the doctor's decision and those other cases. On the contrary: like in Newcomb-like problems, the relevant effect is not one your decision causes; instead, your decision and the effect are common causal descendents of the same parent -- the parent being "the extent to which humans instantiate a decision theory like this".

Similarly, Drescher reasons e.g. that "I should not cheat business partners, even if I could secretly get away with it, because if I regarded this as optimal, it's more likely that my business partner regards it as optimal to do to me." More importantly, "If I make my decision depend on whether I'm likely to get caught, so will my partner's decision, a dependency I do not wish existed."

Consider the case of a 1000x run of Prisoner's Dilemma between 100 "similar" people, where they get randomly paired and play the game many times, but don't know anyone else's decision until it's all over. In that case, if you always cooperate, you are more likely to find yourself with more cooperating partners, even though your decisions can't possibly causally influence the others.

I discussed the merit of incorporating acausal consequences into your reasoning in Morality as Parfitian-Filtered Decision Theory.

As for whether it's a good model of human behavior to assume that people even think about "Does the doctor follow TDT?", look at it this way: people may not think about it in these exact terms, but then, why else are people convinced by arguments like, "What if everyone did that?" or "How would you like it if the roles were reversed?", even when

• everyone doesn't "do that",
• you don't causally influence whether people "do that", and
• the roles aren't likely to be reversed.

It seems they're relying on these "subjunctive acausal means-ends links" (SAMELs in my article) that cannot be justified in consequentialist-causal terms.

On the other hand, other people knowing that you're a TDT agent is the reason TDT agents are able to cooperate. TDT on a large scale kind of depends on other people suspecting it.

This least convenient world basically requires the doctor (or perhaps all doctors) to have a massive epistemic advantage over every other human being, such that the idea won't even cross any patient's mind.

In general, even if you're that much smarter than 99.9% of the world, you need to take into account that other people in the 0.1% can communicate their suspicions to everyone else.

That that, specific, doctor runs TDT perhaps; but it is implausible to the point of irrelevance that no-one would ever suspect that any doctor anywhere runs on a TDT-esque thought process.

And people suspecting that any doctor might run such processes is sufficient harm.

Huh. I took '≥' to mean "better describes human morality than"

Thanks for the critique; the post you were looking for was the post I wanted to write, but I had too much difficulty writing it, and I chose to post what I did rather than risk getting burnt out and posting nothing at all. That's also, in fact, what happened with my qualia sequence.

So I've now had two sequences of posts that were received tepidly despite most readers seeming to agree with them and finding them insightful; the most obvious conclusion is that I really need to improve as a writer. (Further evidence: people get much more excited about my ideas when I explain them face-to-face than when they read my writing.) I'm now publicly committing to relearn writing, with the help of Less Wrong... in two weeks, after the initial rush of the academic year subsides.

Interesting; your phrasing (one severely injured vs 5 moderately injured) seems to pass on both deontological and utilitarian grounds for me, but if it was saving 5 severely injured people by letting one moderately injured person expire of their injuries, it feels like the trolley problem again.

Maybe it is less "unfairness to individuals" and more "unfairness to people better off than the subjects".

I wonder if it's a coincidence that it's currently late at night and I find myself agreeing with both those readings. "Deontological ethics is morally wrong" sounds about accurate.

My late-night reading: "A deontological theory of ethics is not actually right. It is wrong. Morally wrong."

I am not sure what caused me to read it this way, but it cracked me up.

Cracked you up? Rather than just seeming like a straightforward implication of conflicting moral systems?

As a sidenote to my previous comment: I do wonder to what extent deontological concepts of "Acting With Honor" and "Just Vengeance" evolved in human societies as an effective approximation of a TDT, which encouraged both initial cooperation and punishment of defections by making societal members into the sort of being that would instinctively cooperate and punish those people with accurate enough models of themselves.

On the other hand, attitudes of vengeance towards non-intelligent beings (beings that can't model you) are seen as much more... insane. Captain Ahab is perceived as a more insane figure than the Count of Monte Cristo; though both are driven by vengeance, because the former seeks vengeance against a whale, not against people.

Though mind you, even against animals, vengeance is rather useful; because even animals can model humans to some extent. The wolves in The Jungle Book learned to "seven times never kill Man", after learning that to hurt one man, means many other men with guns coming to kill wolves in return.

For an example of acausal consequences: getting a million dollars as a result of one-boxing in Newcomb's. Or getting a hundred dollars as a result of two-boxing.

I would argue that TDT (or UDT) is actually a more consequentialist theory than CDT. The qualitative difference between consequentialism and deontology is that for consequentialists the most important thing is a good outcome, whereas deontology means following the correct rules, regardless of the outcome. But it's casual decision theorists, after all, that continue to adhere to their decision ritual that two-boxes, and loses, in the face of all the empirical evidence (well, hypothetical empirical evidence, anyway :p) that it's the wrong thing to do!

TDT basically takes into consideration the consequences of itself -- not just each particular action it endorses, but the consequences of you following a specific logic towards that action, and the consequences of other people knowing that you would follow such a logic.

It's a consequentialist theory because it seeks to maximize the utility of consequent states of the world -- it doesn't have deontological instructions like "cooperate because it's the nice thing to do" -- it says things like "cooperate if and only if the other guy would be able to predict and punish your non-cooperation, because that leads to an optimal-utility state for you"

All that having been said, I think some people are misusing TDT when they say people would know your non-cooperation. Omega would know your non-cooperation, but other people you may be able to trick. And TDT orders cooperation only in the cases of those you wouldn't be able to trick.

People's moral intuitions are incoherent, and will tend to return different answers for the same dilemma phrased in different terms. Our evolved heuristics have their uses, among them is not turning into a social pariah in a group that relies on the same heuristics, but they're certainly not isomorphic to strict consequentialism.

By carefully setting the free parameters, you can turn one into the other.

How would this be different than inserting epicycles to express geocentrism as heliocentrism? From consequentialist and deontological perspectives, I don't think this is an insight that dissolves the question.

It might be better if doctors could make hard choices like this and keep it absolutely secret, but it's nearly impossible to contrive. As long as most people strongly disapprove of that sort of action, and people who want to become doctors do not have overwhelmingly different inclinations, and the training itself does not explicitly advocate for that sort of action, the vast majority of doctors will not take the organ harvester side in the dilemma, even in circumstances where they think they can get away with it (which will be rare,) and those are the basic minimum requirements to pull it off without people guessing.

A society where the public didn't mind doctors harvesting the few to save the many would probably be considerably better off, but that would require the thoughts and actions of the entire society to be different, not just the doctors within it.

Following consequentialist ethics doesn't mean that you should behave as you would in the highest possible utility world, if that doesn't increase utility in the world in which you actually find yourself.

I don't think that's right. A TDT agent wants people to deduce that TDT would not endorse the action, and therefore TDT would not endorse the action. If it did, it would be the equivalent of defecting in the Prisoner's Dilemma - the other guy would simulate you defecting even if he cooperated, and therefore defect himself, and you end up choosing a sub-optimal option. You can't say "the other guy's going to cooperate so I'll defect" - the other guy's only going to cooperate if he thinks you are (and he thinks you wouldn't if he defects), and if your decision theory is open to the consideration "the other guy's going to cooperate so I'll defect", the other won't think you'll cooperate if he does, and will therefore defect. You can't assume that you've thought it all through one more time than the other guy.