Consequentialism FAQ

OK, I've read the whole FAQ. Clearly, a really detailed critique would have to be given at similar length. Therefore, here is just a sketch of the problems I see with your exposition.

For start, you use several invalid examples, or at least controversial examples that you incorrectly present as clear-cut. For example, the phlogiston theory was nothing like the silly strawman you present. It was a falsifiable scientific theory that was abandoned because it was eventually falsified (when it was discovered that burning stuff adds mass due to oxidation, rather than losing mass due to escaped phlogiston). It was certainly a reductionist theory -- it attempted to reduce fire (which itself has different manifestations) and the human and animal metabolism to the same underlying physical process. (Google "Becher-Stahl theory".) Or, at another place, you present the issue of "opposing condoms" as a clear-cut case of "a horrendous decision" from a consequentialist perspective -- although in reality the question is far less clear.

Otherwise, up to Section 4, your argumentation is passable. But then it goes completely off the rails. I'll list just a few main issues:

• In the discussion of the trolley problem, you present a miserable caricature of the "don't push" arguments. The real reason why pushing the fat main is problematic requires delving into a broader game-theoretic analysis that establishes the Schelling points that hold in interactions between people, including those gravest ones that define unprovoked deadly assault. The reason why any sort of organized society is possible is that you can trust that other people will always respect these Schelling points without regards to any cost-benefit calculations, except perhaps when the alternative to violating them is by orders of magnitude more awful than in the trolley examples. (I have compressed an essay's worth of arguments into a few sentences, but I hope the main point is clear.)

• In Section 5, you don't even mention the key problem of how utilities are supposed to be compared and aggregated interpersonally. If you cannot address this issue convincingly, the whole edifice crumbles.

• In Section 6, at first it seems like you get the important point that even if we agree on some aggregate welfare maximization, we have no hope of getting any practical guidelines for action beyond quasi-deontologist heuristics. But they you boldly declare that "we do have procedures in place for breaking the heuristic when we need to." No, we don't. You may think we have them, but what we actually have are either somewhat more finely tuned heuristics that aren't captured by simple first-order formulations (which is good), or rationalizations and other nonsensical arguments couched in terms of a plausible-sounding consequentialist analysis (which is often a recipe for disaster). The law of unintended consequences often bites even in seemingly clear-cut "what could possibly go wrong?" situations.

• Along similar lines, you note that in any conflict all parties are quick to point out that their natural rights are at stake. Well, guess what. If they just have smart enough advocates, they can also all come up with different consequentialist analyses whose implications favor their interests. Different ways of interpersonal utility comparison are often themselves enough to tilt the scales as you like. Further, these analyses will all by necessity be based on spherical-cow models of the real world, which you can usually engineer to get pretty much any implication you like.

• Section 7 is rather incoherent. You jump from one case study to another arguing that even when it seems like consequentialism might imply something revolting, that's not really so. Well, if you're ready to bite awful consequentialist bullets like Robin Hanson does, then be explicit about it. Otherwise, clarify where exactly you draw the lines.

• Since we're already at biting bullets, your FAQ fails to address another crucial issue: it is normal for humans to value the welfare of some people more than others. You clearly value your own welfare and the welfare of your family and friends more than strangers (and even for strangers there are normally multiple circles of diminishing caring). How to reconcile this with global maximization of aggregate utility? Or do you bite the bullet that it's immoral to care about one's own family and friends more than strangers?

• Question 7.6 is the only one where you give even a passing nod to game-theoretical issues. Considering their fundamental importance in the human social order and all human interactions, and their complex and often counter-intuitive nature, this fact by itself means that most of your discussion is likely to be remote from reality. This is another aspect of the law of unintended consequences that you nonchalantly ignore.

• Finally, your idea that it is possible to employ economists and statisticians and get accurate and objective consequentialist analysis to guide public policy is altogether utopian. If such things were possible, economic central planning would be a path to prosperity, not the disaster that it is. (That particular consequentialist folly was finally abandoned in the mainstream after it had produced utter disaster in a sizable part of the world, but many currently fashionable ideas about "scientific" management of government and society suffer from similar delusions.)

I like it, but stop using "ey". For god's sake, just use "they".

Some "first impressions" feedback: though it has a long tradition in geekdom, and occasionally works well, I for one find the fake-FAQ format extremely offputting. These aren't "frequently" asked questions - they are your questions, and not so much questions as lead-ins to various parts of an essay.

I'd be more interested if you started off with a concise definition of what you mean by "consequentialism". (If you were truly writing a FAQ, that would be the most frequently asked question of all). As it is, the essay starts losing me by the time it gets to "part one" - I skipped ahead to see how long I should expect to spend on preliminaries before getting to the heart of the matter.

(Usually, when you feel obliged to make a self-deprecating excuse such as "Sorry, I fell asleep several pages back" in your own writing, there's a structure issue that you want to correct urgently.)

I think your analysis of the 'gladiatorial objection' misses something:

I hope I'm not caricaturing you too much if I condense your rebuttal as follows: "People wouldn't really enjoy watching gladiators fight to the death. In fact, they'd be sickened and outraged. Therefore, utilitarianism does not endorse gladiatorial games after all."

But there's a problem here: If the negative reaction to gladiatorial games is itself partly due to analyzing those games in utilitarian terms then we have a feedback loop.

Games are outrageous --> decrease utility --> are outrageous --> etc.

But this could just as well be a 'virtuous circle':

Games are noble --> increase utility --> are noble --> etc.

If we started off with a society like that of ancient Rome, couldn't it be that the existence of gladiatorial games is just as 'stable' (with respect to the utilitarian calculus) as their non-existence in our own society?

Couldn't it be that we came to regard such bloodsports as being 'immoral' for independent, non-utilitarian reasons*? And then once this new moral zeitgeist became prevalent, utilitarians could come along and say "Aha! Far from being 'fun', just look at how much outrage the games would generate. If only our predecessors had been utilitarians, we could have had avoided all this ugly carnage."

(Perhaps you will bite the bullet here, and grant that there could be a society where gladiatorial games are 'good' by utilitarian standards. But then there doesn't seem to be much hope for a utilitarian justification of the idea that, insofar as we have outlawed bloodsports, we have 'progressed' to a better state of affairs.

Or perhaps you will say that bloodsports would always be judged 'bad' under ideal rational reflection (that is, they go against our CEV). I think this is a much stronger reply, but it's not clear that CEV actually makes sense (i.e. that the limit is well-defined).)

* Sadly my knowledge of history is too meagre to venture an account of how this actually happened.

I have many other objections to utilitarianism up my sleeve. To give the gist of a few of them:

1. Utilitarian calculations are impossible in practice because the future cannot be predicted sufficiently far.
2. Utilitarian calculations are impossible even if theory because outcomes are incommensurable. The indeterminacies concerning whether 'more people' are preferable to 'happier people', and how far a superbeing's happiness is 'worth more' than a human's, are special cases of this, but incommensurability is ubiquitous. (For instance, just try weighing up all of the effects of the decision to buy a car rather than use public transport. The idea is there is a Right Answer Out There seems to me an article of blind faith.)
3. Utilitarianism holds 'terminal preferences' to be beyond reproach. It does not allow for the possibility that an entire self-contained society ought to change its system of preferences, no matter how 'brutal' and 'destructive' these preferences are. (The point about gladiatorial games is a special case of this). It denies that one can make an objective judgement as to whether a paperclipper is 'wrong' and/or 'stupid' to fill the universe with paperclips. Ultimately, might makes right in the struggle between humanity and clippy.
4. Utilitarianism faces some awkward choices in how it values the lives of 'ordinary people' (people who live reasonably happy lives but do not make lasting 'achievements' e.g. progress in science). If their value is positive then apparently it would be better to fill the universe with them than not, which seems absurd. How is it worthwhile or noble to try to explore the entire 'soap opera of Babel'? Isn't it just a stationary stochastic process? Haven't you seen it all once you've seen the first few billion episodes? But if their value is zero (resp. negative) then it seems that nuking an entire planet full of 'ordinary people', assuming it's not the only such planet, is morally neutral (resp. desirable). The only way of resolving the contradiction of human life being both incredibly precious and utterly worthless is to deny that the premise that we need to assign it some value in order to decide how to act.

I skimmed the FAQ (no time to read it in detail right now, though I have bookmarked it for later). I must say that it doesn't address some of the most crucial problems of consequentialism.

Most notably, as far as I can tell, you don't even address the problem of interpersonal utility comparison, which makes the whole enterprise moot from the beginning. Then, as far as I see, you give the game-theoretic concerns only a cursory passing mention, whereas in reality, the inability to account for those is one reason why attempts to derive useful guidelines for action based on maximizing some measure of aggregate welfare are usually doomed to end up in nonsense. This, in turn, is just a special case of the general law of unintended consequences that consequentialists typically treat with hubristic nonchalance. I don't see any discussion of these essential issues.

On the whole, I would guess that your FAQ will sound convincing to a certain type of people, but it fails to address the most important problems with the views you advocate.

Okay, summary of things I've learned I should change from this feedback:

1. Fix dead links (I think OpenOffice is muddling open quotes and close quotes again)

2. Table of contents/navigation.

3. Stress REALLY REALLY REALLY HARD that this is meant to be an introduction and that there's much more stuff like game theory and decision theory that is necessary for a full understanding of utilitarianism.

4. Change phlogiston metaphor subsequent to response from Vladimir_M

5. Remove reference to Eliezer in "warm fuzzies" section.

6. Rephrase "We have procedures in place for violating heuristics" to mention that it's patchwork and we still don't have elegant rules for this sort of thing.

7. In part about using utilitarianism to set policy, stress involvement of clever tools like prediction markets to lessen the immediate reaction that I'm obviously flaming mad.

8. Rephrase gladiatorial games to be more clear.

9. Possibly change title to "Rarely Asked Questions" or "Never Asked Questions" or just keep the abbreviation the same but expand it to "Formatted Answers and Questions"

10. Remove reference to desire utilitarianism until I'm sure I understand it.

11. Change 'most consequentialists give similar results' to 'most popular consequentialisms...'

12. Change bit about axiology collapsing distinctions.

13. Consequentialism chooses best state of world ---> consequentialism chooses better world-state of two actions.

14. Add to part about fat man that although the example is correct insofar as it correctly teaches consequentialist reasoning, game/decision theory type considerations might change the actual correct answer.

One small thing: you define consequentialism as choosing the best outcome. I think it makes a big difference, at least to our intuitions, if we instead say something like:

Okay. The moral law is that you should take actions that make the world better. Or, put more formally, when asked to select between possible actions A and B, the more moral choice is the one that leads to the better state of the world by whatever standards you judge states of the world by.

In other words, it's not all about the one point at the pinnacle of all the choices you could make - it's about the whole scale. This helps people get over the burdensomeness/ "I'm not going to give all my money to SIAI/GWWC, so I might as well give up" thing.

Some criticism that I hope you will find useful:

First of all, you should mention timeless decision theory, or at least superrationality. These concepts are useful for explaining why people's intuition that they should not steal is not horribly misguided even if the thief cares about himself more and/or would needs it more than the previous owner. You touched on this by pointing out that the economy would collapse if everyone stole all the time, but I would suggest being more explicit.

(3.8) I think the best course of action would be to assign equal value to yourself and other people, which seems nicely in accord with there being no objective reason for a moral difference between you.

I take issue with this simply because it is not even remotely similar to the way anyone acts. I'd prefer it if we could just admit that we cared more about ourselves than about other people. Sure, utilitarianism says that the right thing to do would be to act like everyone, including oneself, is of equal value, and the world would be a better place if people actually acted this way. But no one does, and endorsing utilitarianism does not usually get them closer.

(5.31) Desire utilitarianism replaces preferences with desire. The differences are pretty technical and I don't understand all of them, but desire utilitarians sure seem to think their system is better.

Then I would suggest either doing the research or not mentioning it, since this is not critical to the concept of consequentialism. I'm not entirely clear on it either.

(7.4) For example, in coherent extrapolated volition utilitarianism, instead of respecting a specific racist's current preference, we would abstract out the reflective equilibrium of that racist's preferences if ey was well-informed and in philosophical balance. Presumably, at that point ey would no longer be a racist.

But what if he doesn't? You are right that this situation is a problem for for simple preference utilitarianism that can be rectified by some other form of utilitarianism, but your suggested solution leads to a slippery slope towards justifying anything you want with CEV utilitarianism by claiming that everyone else's moral preferences would be exactly what you want them to be in their CEV. I think the real issue here is that we respect some forms of preferences much more than others. Recall that pleasure utilitarianism (which would be the extreme case of giving 0 weight to all but one form of preference) gives the answer we like in this case.

One problem with the FAQ: The standard metaethics around here, at least EY's metaethics, is not utilitarianism. Utilitarianism says maximize aggregate utility, with "aggregate" defined in some suitable way. EY's metaethics says maximize your own utility (with the caveat that you only have partial information of your utility function), and that all humans have sufficiently similar utility functions.

You get something pretty similar to utilitarianism from that last condition (if everyone has the same utility function and you're maximizing your own utility function, then you're also maximizing aggregate utility in many senses of the term "aggregate"). But note that the interpersonal comparison of utility problem vanishes: you're maximizing your own utility function. Maximization of the aggregate (under some methods of comparisons) is merely a consequence of this, nothing more. Also note that if we relax that last condition and let humans have differing utility function, there is no intrinsic problem for EY's theory. If someone has a legitimate preference for killing people, the utilitarian has to take that into account as positive utility (or add some ad hoc assumptions about which preferences matter). On EY's theory, sans that last condition, you only take into account someone's preference for murder if your utility function tells you to. You may value other humans satisfying their preferences, but that doesn't mean you have to value every single preference every single human has. You could, but it really just depends on what your utility function says.

A contents section, with links to the relivant sections, would aid navigation.

Some thoughts I had while reading (part of) the FAQ:

...our moral intuition...that we should care about other people.

Is it an established fact that this is a natural human intuition, or is it a cultural induced disposition?

If it is a natural human tendency, can we draw the line at caring about other people or do we also care about cute kittens?

Other moral systems are more concerned with looking good than being good...

Signaling is a natural human tendency. Just like caring about other people, humans care how they appear to other people.

Why should a moral theory satisfy one intuition but not the other?

You'll have to differentiate what it means to be good from what it means to look good.

...but in fact people are generally very moral: they feel intense moral outrage at the suffering in the world...

People also feel intense moral outrage about others burning paper books or eating meat.

You can't establish a fact about morality by cherry-picking certain behaviors.

So no particular intuition can be called definitely correct until a person has achieved a reflective equilibrium of their entire morality, which can only be done through careful philosophical consideration.

I am very skeptical of trying to apply game theoretic concepts to human values.

Human values are partly inconsistent. Can you show that forced consistency won't destroy what it means to be human?

People enjoy watching a large moon, even though it is an optical illusion. If you were to argue that we generally assign more importance to not falling prey to optical illusions and therefore shouldn't conclude that it is desireable to see a large moon, you're messing with human nature, you're creating the sort of rational agent that is assumed by economic and game theoretic theories rather than protecting human nature.

It's my moral intuition that if I failed to reflect on my disgust over homosexuality, and ended out denying homosexuals the right to marry based on that disgust, then later when I thought about it more I would wish I had reflected earlier.

You expect hindsight bias and therefore conclude that you should discard all your complex values in favor of...what?

I am not saying it is wrong, but we'll have to decide if we want to replace human nature with alien academic models or swallow the bitter pill and accept that we are inconsistent beings without stable utility functions that are constantly reborn.

...that morality must live in the world, and that morality must weight people equally.

Define "people"? What is it that assigns less weight to a chimpanzee than me?

Why should we assign a nonzero value to other people?

If I only assign weight to certain people then other people with more power might do the same and I will lose. So everyone except those who can overpower all others together would be better off to weigh each other equally.

But just because there is an equilibrium doesn't mean that it is desirable. Humans don't work like that. Humans refuse blackmail and are willing to sacrifice their own life's rather than accepting a compromise.

But guilt is a faulty signal; the course of action which minimizes our guilt is not always the course of action that is morally right. A desire to minimize guilt is no more noble than any other desire to make one's self feel good at the expense of others, and so a morality that follows the principle of according value to other people must worry about more than just feeling guilty.

You are wrapping your argument in terms your are trying to explain. You are begging the question.

But just as guilt is not a perfect signal, neither are warm fuzzies. As Eliezer puts it, you might well get more warm fuzzy feelings from volunteering for an afternoon at the local Shelter For Cute Kittens With Rare Diseasess than you would from developing a new anti-malarial drug, but that doesn't mean that playing with kittens is more important than curing malaria.

You don't explain why warm fuzzies aren't more important than curing malaria.

If you go on and argue that what one really wants by helping cute kittens is to minimize suffering you are introducing and inducing an unwarranted proposition. You need empirical evidence to show that what humans really want isn't warm fuzzies.

Ironically, although these sorts of decisions are meant to prove the signaler is moral, they are not in themselves moral decisions: they demonstrate interest only in a good to the signaler...

So? What is a moral decision? You still didn't show why signaling is less important than saving your friend. All you do is telling people to feel guilty by signaling that it is immoral to be more concerned with signaling...

...there's a big difference between promoting your own happiness by promoting the happiness of others, and promoting your own happiness instead of promoting the happiness of others.

You are signaling again...

On 3.4: The term "warm fuzzies" was invented before EY was born - I remember the story from high school.

With the power of Google, I found the story on the web, it is by Claude Steiner, copyrighted in 1969.

Not really philosophical feedback, but all the links except the one to the LW metaethics sequence seem to be broken for me.

Can this be an article on LW please? This link isn't very pretty and the raikoth link doesn't work. Thanks!

If God made His rules arbitrarily, then there is no reason to follow them except for self-interest (which is hardly a moral motive)

A perfectly moral motive, in the sense you are using the term.

I'm looking forward to live discussion of this topic at the Paris meetup. :)

Meanwhile, I've read through it, more closely. Much of it seems, not necessarily right, but at least unobjectionable - it raises few red flags. On the other hand, I don't think it makes me much the wiser about the advantages of consequentialism.

Paras 7.2 and 7.3 (the slavery and gladiator questions) left me with an odd impression. The "test" you propose in both cases is more or less the same as Rawls' Veil of Ignorance. So at that point I was wondering, if you apply Rawls' procedure to determine what is a preferable social contract, perhaps you're a Rawlsian more than you're a consequentialist. :) BTW, are you familiar with Rawls' objections to (classical) utilitarianism?

Para 8.2 comes across as terribly naive, and "politics has been reduced to math" in particular seems almost designed to cause people to dismiss you. (A nitpick: the links at the end of 8.2 are broken.)

One thing that makes the essay confusing for me is the absence of a clear distinction between the questions "how do I decide what to do next" and "what makes for a desirable set of agreements among a large number of people" - between evaluating the morality of individual actions and choosing a social contract.

Another thing that's left out is the issue of comparing or aggregating happiness, or "utility", across different people. The one place where you touch on it, your response to the "utility monster" argument, does not match my own understanding of how a "utility monster" might be a problem. As I understood it, a "utility monster" isn't someone who is to you as you are to an ant, but someone just like you. They just happen to insist that an ice cream makes them a thousand times happier than they make you, so in all cases where it must be decided which of you should get an ice cream they should always get it.

Your analogy with optical illusions is apt, and it gives a good guideline for evaluating a proposed system of morality: in what cases does the proposed system lead me to change my mind on something that I previously did or avoided doing because of a moral judgement.

Interestingly, though, you give more examples that have to do with the social contract (gun control, birth control, organ donation policy, public funding of art, discrimination, slavery, etc.) than you give examples that have to do with personal decisions (giving to charities, trolley problems).

My own positions are contractarian, much more than they are deontological or consequentialist. I'm generally truthful, not because it is "wrong" to lie or because I have a rule against it (for instance I'm OK with lying in the context of a game, say Diplomacy, where the usual social conventions are known to be suspended - though I'd be careful about hurting others' feelings through my play even in such a context). I don't trust myself to compute the complete consequences of lying vs. not lying in each case, and so a literal consequentialism isn't an option for me.

However, I would prefer to live in a world where people can be relied upon to tell the truth, and for that I am willing to sacrifice the dubious advantage of being able to put a fast one over on other people from time to time. It is "wrong" to lie in the sense that if you didn't know ahead of time what particular position you'd end up occupying in the world (e.g. a politician with power) but only knew some general facts about the world, you would find a contract that banned lying acceptable, and would be willing to let this contract sanction lying with penalties. (At the same time, and for the same reason, I also put some value on privacy: being able to lie by omission about some things.)

I find XiXiDu's remarks interesting. It seems to me that at present something like "might makes right" is descriptively true of us humans: we could describe a morality only in terms of agreements and generally reliable penalties for violating these agreements. "If you injure others, you can expect to be put in prison, because that's the way society is currently set up; so if you're rational, you'll curb your desires to hurt others because your expected utility for doing so is negative".

However this sort of description doesn't help in finding out what the social contract "should" be - it doesn't help us find what agreements we currently have that are wrong because they result from the moral equivalent of optical illusions "fooling us" into believing something that isn't the case.

It also doesn't help us in imagining what the social contract could be if we weren't the sort of beings we are: if the agreements we enter were binding for reasons other than fear of penalties. This is a current limitation of our cognitive architectures but not a necessary one.

(I find this a very exciting question, and at present the only place I've seen where it can even be discussed is LW: what kind of moral philosophy would apply to beings who can "change their own source code".)

EDIT: having read Vladimir_M's reply below, his comments capture much of what I wanted to say, only better.

suppose some mathematician were to prove, using logic, that it was moral to wear green clothing on Saturday. There are no benefits to anyone for wearing green clothing on Saturday, and it won't hurt anyone if you don't. But the math apparently checks out. Do you shrug and start wearing green clothing? Or do you say “It looks like you have done some very strange mathematical trick, but it doesn't seem to have any relevance to real life and I feel no need to comply with it"?

Supposing a consequentialist were to prove using maths that you should be tortured a bit in order to bring about a nett tincrease the sum total of human happiness. Do you shrug and say: "get out the thumbscrews"? Or do you say "this has something to do with the world, but it is not moral because there is a rule that the end does not justify the means"

This isn't so much a critique against consequentialism as the attempt at creating objective moral systems in general. I would love for the world to follow a particular moral order (namely mine). But there are people who, for what I would see as being completely sane reasons, disagree with me. On the edges, I have no problem writing mass murderers off as being insane. Beyond that, though, in the murky middle, there are a number of moral issues (and how is that dividing line drawn? Is it moral to have anything above a sustenance level meal if others are starving i the world, for instance?) that I see as leading only to endless argument. This doesn't indicate one of the sides is being disingenuous, just that they have different values that cannot be simultaneously optimized. The Roman gladiator post by another commenter is an example. I view the Romans as PETA members would view me. I have justifications for my actions, as I'm sure Romans had for their actions. That's just the nature of the human condition. Academic moral philosophizing always comes across to me as trying to unearth a cosmic grading scale, even if there isn't a cosmic grader.

Somewhat cringeworthy:

This term ("warm fuzzies"), invented as far as I know by computer ethicist Eliezer Yudkowsky...

I'm pretty sure not. See here for a reference dating the term itself back to 1969; also, it's been in use in geek culture for quite a while.

Link doesn't seem to work.

Here is what David Pearce has to say about the FAQ (via Facebook):

Lucid. But the FAQ (and lesswrong) would be much stronger if it weren't shot through with anthropocentric bias...

Suggestion: replace "people" with "sentient beings".

For the rational consequentialist, the interests of an adult pig are (other things being equal) as important as a human toddler. Sperm whales are probably more sentient than human toddlers; chickens probably less.

Ethnocentric bias now seems obvious. If the FAQ said "white people" throughout rather than "people", then such bias would leap off the page - though it wouldn't to the Victorians.

Sadly, anthropocentric bias is largely invisible.

As far as I can tell, she means “Better that everyone involved dies as long as you follow some arbitrary condition I just made up, than that most people live but the arbirary condition is not satisfied.” Do you really want to make your moral decisions like this?

The whole problem is that everything is framed in moral terminology.

The trolley problem really isn't any different from the prisoner's dilemma with regard to human nature.

On one side there is the game theoretic pareto-suboptimal solution that a rational agent would choose and on the other side there is human nature.

Human nature and game theory are incompatible.

To cooperate or not to push the fat guy are human values.

Academic models like game theory are memes that discard complex human values and replace them with simple equilibria between agents carrying the same memes.

Here's something that, if it's not a frequently asked question, ought to be one. What do you mean by "state of the world" - more specifically, evaluated at what time, and to what degree of precision?

Some utilitarians argue that we should give money to poorer people because it will help them much more than the same amount will help us. An obvious question is "how do you know"? You don't typically know the consequences that a given action will have over time, and there may be different consequences depending on what "state of the world" you consider.

The immediate difference is that someone has a little more money and you have a little less, but it becomes a much more complicated problem when you start thinking of longer term consequences. If I give that person money, will they use it well or will they spend it on drink; if we send aid to poor countries, will it help the people who need it or be diverted; will these countries become dependent on foreign aid, destroying an autonomy which is more precious than money; etc.

The organ donation example is interesting in this regard. You have worked out one particular subset of the consequences ("the lives of a thousand people a year"), but what guarantees do you have that the proposed opt-out system wouldn't have other consequences, in the short, medium or long term, that eventually turn out to outweigh these positive consequences?

6.4) In the United States at least there are so many laws that it's not possible to live a normal life without breaking many of them.

See: http://www.harveysilverglate.com/Books/tabid/287/Default.aspx

7.1) you could come up with better horribly "seeming" outcomes that consequentialism would result in. For example, consequentialists who believe in heaven and hell and think they have insight into how to get people into heaven would be willing to do lots of nasty things to increase the number of people who go to heaven. Also, dangerous sweatshops in poor countries that employ eight-year-olds become praiseworthy if they provide the children with better outcomes than the children would otherwise receive.

8.1) Does "lack of will" account for failure to solve collective action problems?

As a philosophy student, complicated though the FAQ is I think I could knock down the arguments therein fairly easily. That's kind of irrelevant, however.

More importantly, I'm looking for the serious argument for Consequentialism on Lesswrong. Could somebody help me out here?

Some observations... There is no discussion in your FAQ of the distinctions between utilitarianism and other forms of consequentialism, or between act consequentialism and rule consequentialism, or between "partial" and "full" rule consequentialism. See http://plato.stanford.edu/entries/consequentialism-rule/ for more on this.

Where you discuss using "heuristics" for moral decision making (rather than trying to calculate the consequences of each action), you are heading into the "partial rule consequentialism" camp. To move further into that camp, you might consider whether it is praiseworthy or blameworthy to follow usual moral rules (heuristics) in a case where breaking the rules would lead to higher expected utility. Generally it will lead to better consequences if we praise people for following the heuristics, while blaming them for departing from the heuristics. And then ask yourself whether you really want an ethical theory that says when faced with a choice between action X and Y, act X is strictly the "right" one (while Y is strictly wrong), yet X is blameworthy (while Y is praiseworthy). These are the sorts of considerations that lead to full rule utilitarianism.

Another thing you might want to cover is the objection that consequentialism is extremely demanding. Act consequentialism in particular requires well-off people to give away almost all their money to charities which have a high utility impact per dollar (sanitation imrovements in slums, malaria nets and de-worming of children in poor African countries), rather than spending the marginal dollar on themselves or their friends or family. Most versions of rule utilitarianism can avoid that problem (humans, being human, will never accept such overly demanding moral rules, and it will have very bad consequences to try and make them).

Part One: Methodology: Why think that intuitions are reliable? What is reflective equilibrium, other than reflecting on our intuitions? If it is some process by which we balance first-order intuitions against general principles, why think this process is reliable? Metaethics: Realism vs. Error vs. Expressivism?

Part Two: 2.6 I don't see the collapse - an axiology may be paired with different moralities - e.g. a satisficing morality, or a maximizing morality. Maybe all that is meant by the collapse is that the right is a function of the good? Then 'collapse' is misleading.

Part Four: 4.2 Taking actions that make the world better is different from taking actions that make the world best. Consequentialism says that only consequences matter - a controversial claim that hasn't been addressed.
4.4 Makes a strawman of the deontologist. Deontologists differ from consequentialists in ways other than avoiding dirtying their hands / guilt. They may care about not using others as means, or distinctions like doing/allowing, killing/letting die, etc., which apply to some trolley cases, and (purportedly) justify not producing the best consequences. More argument is needed to show that this precludes morality from 'living in the world'.

Part Five: 5.4 Not obvious that different consequentialisms converge on most practical cases. Some desire pain. Some desire authenticity, achievement, relationships, etc. (no experience machine). Some desire not to be cheated on / have their wills disregarded / etc.

Part Seven: 7.3 Doesn't address the strongest form of the objection. A stronger form is: we know that certain acts or institutions are necessarily immoral (gladiatorial games, slavery); utilitarianism could (whether or not it does) require we promote these; therefore utilitarianism is false. I like the utility monster example of this. The response in 7.5 to the utility monster case is bullet-biting - this should be the response in 7.3. The response that utilitarianism probably won't tell us to promote these is inadequate. The mistake is remade by the three responses in 7.4 (prior to the appeal to ideal rather than actual preferences). 7.6 Similar problem here. The response quibbles with contingent facts, but the force of the objection is that vicious, repugnant, petty, stupid, etc., preferences have no less weight in principle, i.e. in virtue of their status as such. 7.7 Response misses the point. The objection is that it's hard to see how utilitarianism can accommodate the intuitive distinction between higher and lower pleasures. Sure, utilitarians have nothing against symphonies, but would a world with symphonies be best? (Would an FAI-generated world contain symphonies?) 7.9 Rather quick treatment of the demandingness objection. One relevant issue in the vicinity is that of agent-centered permissions - permissions to do less than the best (in consequentialist terms), e.g. to favor those with whom we have special relations. Many philosophers and folk alike believe in such permissions - utilitarianism has a counterintuitive result here.

Suggestions for further content: (1) How are we to conceive of 'better' consequences? Perhaps any of the answers given by the aforementioned systems would suffice - pleasure, preference satisfaction, ideal preference satisfaction. But I'm not convinced these are practically/pragmatically equivalent. For instance, there may be different best methods for investigating what produces the most pleasure vs. what would best satisfy our ideal preferences, and so different practical recommendations. (2) What's our axiology? Is it total utilitarian, egalitarian, prioritarian, maximizing, satisficing, etc.? How do the interests of animals, future time slices, and future individuals weigh against present human interests? A total utilitarian approach seems to be advocated, but that faces its own set of problems (repugnant conclusions, fanaticism, etc.).

I like the idea of a non-technical explanation of consequentialism, but I worry that many important distinctions will be conflated or lost in the process of generating something that reads well and doesn't require that the reader to spend a lot of time thinking about the subject by themselves before it makes sense.

The issue that stands out the most to me is what you write about axiology. The point you seem to want to get across, which is what I would consider to be the core of consequentilalism in one sentence, is that "[...]our idea of “the good” should be equivalent or directly linked to our idea of 'the right'." But that woefully underspecifies a moral theory; all it does is pick out a group of related theories, which we call "consequentialist".

It's important to realize how much possible variation there is among consequentialist theories, and the most straightforward way to do it that I see is to give more serious consideration to the role that axiology plays in a total moral theory. For example, a basic way to taxonomize theories while simplifying away a lot of technical issues that are not important for the kind of overview you're providing is:

1) Does the theory fundamentally concern itself with outcomes or something other than outcomes?

• A theory that fundamentally concerns itself with outcomes (however "outcomes" are defined) is consequentialist. (Other theories have other concerns and other names.)

2) What kinds of outcomes does the theory concern itself with?

• Outcomes concerning the satisfaction of people's preferences.

• Outcomes concerning people's happiness.

• Outcomes concerning non-human happiness.

• Outcomes concerning ecological sustainability.

• Outcomes concerning paperclips.

• etc.

All of these describe consequentialist axiologies which lead to different consequentialist theories.

3) For consequentialist theories, the rightness of an action (i.e. the indicator of whether it should be done or not) depends on its consequences or expected consequences. What actions are right for an agent?

• Any action that leads to a sufficiently good outcome, where "sufficiently good" is somehow defined...

  -...in relation to the current state of the world. E.g. an action is right if it leads to an outcome that is better (according to the theory's axiology) than the current state of things. "Leave things better than you found them."

  -...in relation to the actions that an agent can actually do. E.g. an action is right if it leads to an outcome that is better than 90% of the other outcomes (according to the theory's axiology) that an agent can bring into effect through other actions. "Do enough good; sainthood not required."

• Any action that leads to an outcome for which there is no better outcome according to the theory, among all the other outcomes the agent can bring about.

  -E.g. You have the task of distributing two bars of chocolate to Ann and Bob. Any bar of chocolate you don't distribute immediately disappears. Your theory's complete axiology is "all else being equal, it's better when one person has more chocolate bars than they otherwise would have had." Your actions can distribute chocolate bars like this:

  Action A --> Outcome A: Ann 0 Bob 0

  Action B --> Outcome B:: Ann 1 Bob 0

  Action C --> Outcome C: Ann 0 Bob 1

  Action D --> Outcome D: Ann 2 Bob 0

  Action E --> Outcome E: Ann 0 Bob 2

  Action F --> Outcome F: Ann 1 Bob 1

  According to your theory, every outcome is better than A. Outcomes D and F are better than B. Outcomes E and F are better than C. Outcomes D, E, and F are neither better than nor worse than nor equal to each other. So Actions D, E, and F would be right, and the rest would be wrong. "Act to maximize value; if that's undefined, don't leave any extra value on the table."

• etc.

I've left out a lot of issues concerning expected consequences vs. actual consequences, agent knowledge, value measurement and aggregation, satisficing, etc. which I think are not important given the goals of your FAQ. But I'd say it's important to get across to the non-specialist that the range of consequentialist theories is pretty large, and there are a lot of issues that a consequentialist theory will have to deal with. (In other words, there's no monolithic theory called "consequentialism" that you can subscribe to which will pass judgements on your actions. If you say believe in "consequentialism", you have to say more in order to pin down what actions you believe are right and wrong.) If you don't make this clear, people may fill in the blanks in idiosyncratic ways and then react to the ways they've filled them in, which is likely not to lead to anyone being persuaded, or more importantly, anyone being informed. One easy way to resolve this is to define consequentialist theories as those concerned with outcomes, define consequentialist axiologies as theories of what kinds of outcomes are valuable, describe some common methods for determining right actions, and say that a consequentialist moral theory is a consequentialist axiology + a way of determining right actions based on that axiology.

EDIT FOR CLARITY: My point is not that you don't ever bring up these issues, but that these issues are fundamental (theoretically and pedagogically) and I'd make sure that the structure of the FAQ emphasizes that.

The basic thesis is that consequentialism is the only system which both satisfies our moral intuition that morality should make a difference to the real world, and that we should care about other people.

Why am I supposed to adopt beliefs about ethics and decision theory based on how closely they match my intuitions, but I am not supposed to adopt beliefs about quantum physics or cognitive psychology based on how closely they match my intuitions? What warrants this asymmetry?

Also, your FAQ lacks an answer to the most important question regarding utilitarianism: By what method could we aggregate the utilities of different persons?

Fairly good summary. I don't mind the FAQ structure. The writing style is good, and the subject matter suggests obvious potential to contribute to the upcoming Wiki Felicifia in some way. Now as good as the essay is, I have some specific feedback:

In section 2.2, I wonder if you could at put your point more strongly...

you wrote: if morality is just some kind of metaphysical rule, the magic powers of the Heartstone should be sufficient to cancel that rule and make morality irrelevant. But the Heartstone, for all its legendary powers, is utterly worthless and in fact totally indistinguishable, by any possible or conceivable experiment, from a fake...

I would suggest: Metaphysical rules are like a kind of heartstone that one can wear when they make moral decisions. It is reputed to rule our moral considerations. But despite its reputation, the heartstone is utterly worthless and..."

If you're going to use a metaphor, you might as well get full value from it!

2.61: I understand the point you're making here. I couldn't agree with it more. Still, if you're trying to reduce the amount of words in the way between the reader and the later sections - as you should be - then this section is one you could consider abbreviating or removing. The whole phlogiston analogy is not obvious to a layperson.

Your line of thought seems to get somewhat deraied at 3.5. I don't quite understand why 'signalling' fits in 'assigning value to other people'.

4 is extremely good. The trolley discussions are reminiscent of Peter Unger's Living High and Letting Die. It's a shame it takes so long to get there.

Continued in this Felicifia post

On this site it's probably just me, but I just can't (or won't) bring myself to assigning a non-zero value to people I do not even know. Since the FAQ rest on the assumption that this is not the case it's practically worthless to me. This could be as it should, if people like me just aren't your target audience in which case it would be helpful to have such a statement in the 3.1 answer.

Edit: Thinking about it, this is probably not the case. If all of the people except those I know were to die, I might feel sad about it. So my valuation of them might actually be non-zero.

OT: The format seemed familiar and then I looked back and found out it was because I had read your libertarian FAQ. Best anti-libertarian FAQ I've seen!

Or, put more formally, when asked to select between several possible actions A, B and C, the most moral choice is the one that leads to the best state of the world by whatever standards you judge states of the world by.

This is the key definition, yet it doesn't seem to actually say anything. Moral choice = the choice that makes you happy. This is a rejection of ethics, not an ethical system. If it were, it would be called individual consequentialism, that is, "Forget all this ethics tripe."

Yet after that pretense of doing away with all ethics, you gently slip in the collectivist consequentialism as rule-of-thumb principles, but to my mind these can hardly serve as anything other than a political agenda. From then on you seem to equivocate between individual consequentialism and collective consequentialism. It would be a lot clearer - and shorter - if you made that distinction more explicit throughout.

Isn't the difference between good and right where decision theory lives?