The "Intuitions" Behind "Utilitarianism"

Eliezer, to be clear, do you still think that 3^^^3 people having momentary eye irritations--from dust-specs--is worth torturing a single person for 50 years, or is there a possibility that you did the math incorrectly for that example? A proper utilitarian needs to consider the full range of outcomes--and their probabilities--associated with different alternatives. If the momentary eye irritation leads to a greater than 1/3^^^3 probability that someone will have an accident that leads to an outcome worse than 50 years of torture, then the dust specs are preferable. But if the chance of further negative consequences from momentary eye-irritation is so small as to be negligible, then we can consider the cost of the dust specs to be the linear sum of the hedonic loss across all of the people afflicted. The torture, on the other hand, has a significant probability of leading to further negative consequences that could persist across a life-span and impact those who care about that individual. If the tortured individual has a significant probability of committing suicide, then we need to consider all of the potential experiences and accomplishments that the person would have had over the course of their life-time--which could be indefinitely long, depending on how technology progresses--and the impact that the person would have had on others. And finally, as I think you would agree, we wouldn't want to use an ethical utility function that only considered basic hedonic experience and ignored higher-level meaning. If you merely integrated all of the moments of pleasure/pain across a life-span, you wouldn't have come close to calculating the value of that life. Music is worth more than the sum of the notes that went into the song. While your basic argument is valid and important, you probably--depending on the details of the argument--came to the wrong conclusion with respect to dust-specs and torture.

Eliezer, to be clear, do you still think that 3^^^3 people having momentary eye irritations--from dust-specs--is worth torturing a single person for 50 years, or is there a possibility that you did the math incorrectly for that example?

No. I used a number large enough to make math unnecessary.

I specified the dust specks had no distant consequences (no car crashes etc.) in the original puzzle.

Unless the torture somehow causes Vast consequences larger than the observable universe, or the suicide of someone who otherwise would have been literally immortal, it doesn't matter whether the torture has distant consequences or not.

I confess I didn't think of the suicide one, but I was very careful to choose an example that didn't involve actually killing anyone, because there someone was bound to point out that there was a greater-than-tiny probability that literal immortality is possible and would otherwise be available to that person.

So I will specify only that the torture does not have any lasting consequences larger than a moderately sized galaxy, and then I'm done. Nothing bound by lightspeed limits in our material universe can morally outweigh 3^^^3 of anything noticeable. You'd have to leave our physics to do it.

You know how some people's brains toss out the numbers? Well, when you're dealing with a number like 3^^^3 in a thought experiment, you can toss out the event descriptions. If the thing being multiplied by 3^^^3 is good, it wins. If the thing being multiplied by 3^^^3 is bad, it loses. Period. End of discussion. There are no natural utility differences that large.

Favoring an unconditional social injunction against valuing money over lives is consistent with risking one's own life for money; you could reason that if trading off money and other people's lives is permitted at all, this power will be abused so badly that an unconditional injunction has the best expected consequences. I don't think this is true (because I don't think such an injunction is practical), but it's at least plausible.

A Utilitarian should care about the outcomes of Utilitarianism..... and yes, as soon as ends justify means, you do get Stalin, Mao, Pol Pot, who were all striving for good consequences...... Which is relevant, as your arguments mostly involve saving lives (a single type of outcome, so making options intuitively comparable). I'm afraid the 'rant' doesn't add much in terms of content, that I can see.

So it seems you have two intuitions. One is that you like certain kinds of "feel good" feedback that aren't necessarily mathematically proportional to the quantifiable consequences. Another is that you like mathematical proportionality. The "Shut up and multiply" mantra is simply a statement that your second preference is stronger than the first.

In some ways it seems reasonable to define morality in a way that treats all people equally. If we do so, than our preference for multiplying can be more moral, by definition, than our less rational sympathies. But creating a precise definition generally has the result of creating a gap between that definition and common usage. People use the term "morality" and accompanying concepts in a number of ways. Restricting its usage may make a debate more intelligible, but it tends to obscure the fact that morality is a multi-faceted concept that represents a number of different preferences and beliefs. Even meta-morality can do no more than articulate certain kinds of meta-preferences.

Also, equating utilitarianism with Pol Pot and Stalin is a bit disingenuous. Those people weren't utilitarian in any recognizable sense because the total consequences of their actions (millions dead), didn't justify their intended consequences (whatever those were). Millions dead shouldn't be placed solely in the "means" category.

"Well, when you're dealing with a number like 3^^^3 in a thought experiment, you can toss out the event descriptions. If the thing being multiplied by 3^^^3 is good, it wins. If the thing being multiplied by 3^^^3 is bad, it loses. Period. End of discussion. There are no natural utility differences that large."

Let's assume the eye-irritation lasts 1-second (with no further negative consequences). I would agree that 3^^^3 people suffering this 1-second irritation is 3^^^3-times worse than 1 person suffering thusly. But this irritation should not be considered to be equal to 3^^^3 seconds of wasted lives. In fact, this scenario is so negligibly bad, as to not be worth the mental effort to consider it.

And for the torture option, let's assume that the suffering stops the instant the person finishes their 50 years of pain (the person leaves in exactly the same psychological state they were in before they found out that they would be tortured). However, in this case, 50 years of being tortured is not (50 years 365 days 24 hours * 3600 seconds)-times worse than 1-second of torture. It is much (non-linearly) worse than that. There are other variables to consider. In those 50-years, the person will miss 50 years of life. Unlike the dust-speck irritation distributed across 3^^^3 people, 50 years of torture is worth considering.

Adding experiences across people should linearly impact the estimated utility, but things do not add linearly when considering the experiences of a single person. Even if it doesn't lead to further negative consequences, the one-second of irritation is less than 3^^^3-times as bad as the 50 years of torture.

If you're multiplication has taken you so far afield of your intuitions, re-check the math. If it still comes out the same way, check your assumptions. If it still comes out the same way, go with the calculations.

3^^^3?

http://www.overcomingbias.com/2008/01/protecting-acro.html#comment-97982570

A 2% annual return adds up to a googol (10^100) return over 12,000 years

Well, just to point out the obvious, there aren't nearly that many atoms in a 12,000 lightyear radius.

Robin Hanson didn't get very close to 3^^^3 before you set limits on his use of "very very large numbers".

Secondly, you refuse to put "death" on the same continuum as "mote in the eye", but behave sanctimoniously (example below) when people refuse to put "50 years of torture" on the same continuum as "mote in the eye".

Where music is concerned, I care about the journey.

When lives are at stake, I shut up and multiply.

I assert the use of 3^^^3 in a moral argument is to avoid the effort of multiplying. Demonstration: what is 3^^^3 times 6? What is 3^^^3 times a trillion to the trillionth power?

Where am I going with this? I am very interested in improving my own personal morality and rationality. I am profoundly disinterested in passing judgment on any one else's morality or rationality.

I assert that the use of 3^^^3 in a moral argument has nothing to do with someone improving their own personal morality or rationality. It has everything to do with trying to shame someone else into admitting that they aren't A "good little rational moralist".

My comment is an attempt to steer the thread of your (very interesting and well written) posts towards topics that will help me improve my own personal morality and rationality. (I admit that I perceive no linkage between the "wheel in my hand" and the "rudder of the ship", so I doubt my steering will work.)

The only reason Eliezer didn't put death on the same scale as the dust mote was on account of his condition that the dust specks have no further consequences. In real life, everything has consequences, and so in real life, death is on the same scale with everything else, including dust motes. Eliezer expressed this extremely well: "Whatever value is worth thinking about at all, must be worth trading off against all other values worth thinking about, because thought itself is a limited resource that must be traded off."

So yes, in real life there is some number of dust motes such that it would be better to prevent the dust storm than to save a life.

A dust speck in the eye with no external ill effects was chosen as the largest non-zero negative utility. Torture, absent external effects (e.g., suicide), for any finite time, is a finite amount of negative utility. Death in a world of literal immortality cuts off an infinite amount of utility. There is a break in the continuum here.

If you don't accept that dust specks are negative utility, you didn't follow the rules. Pick a new tiny ill effect (like a stubbed toe) and rethink the problem.

If you still don't like it because for a given utility n, n + n != 2n, there are then issues with circular preferences. Two units of utility are defined as twice as "utilitous" as one unit of utility. (This is not saying that two dollars are twice as good as one dollar.)

I assert the use of 3^^^3 in a moral argument is to avoid the effort of multiplying.

Yes, that's what I said. If the quantities were close enough to have to multiply, the case would be open for debate even to utilitarians.

Demonstration: what is 3^^^3 times 6?

3^^^3, or as close as makes no difference.

What is 3^^^3 times a trillion to the trillionth power?

3^^^3, or as close as makes no difference.

...that's kinda the point.

So it seems you have two intuitions. One is that you like certain kinds of "feel good" feedback that aren't necessarily mathematically proportional to the quantifiable consequences. Another is that you like mathematical proportionality.

Er, no. One intuition is that I like to save lives - in fact, as many lives as possible, as reflected by my always preferring a larger number of lives saved to a smaller number. The other "intuition" is actually a complex compound of intuitions, that is, a rational verbal judgment, which enables me to appreciate that any non-aggregative decision-making will fail to lead to the consequence of saving as many lives as possible given bounded resources to save them.

I'm feeling a bit of despair here... it seems that no matter how I explain that this is how you have to plan if you want the plans to work, people just hear, "You like neat mathematical symmetries." Optimal plans are neat because optimality is governed by laws and the laws are math - it has nothing to do with liking neatness.

50 years of being tortured is not (50 years 365 days 24 hours * 3600 seconds)-times worse than 1-second of torture. It is much (non-linearly) worse than that.

Utilitarianism does not assume that multiple experiences to the same person aggregate linearly.

Yes, I agree that it is non-linearly worse.

It is not infinitely worse. Just non-linearly worse.

The non-linearity factor is nowhere within a trillion to the trillionth power galaxies as large as 3^^^3.

If it were, no human being would ever think about anything except preventing torture or goals of similar importance. You would never take a single moment to think about putting an extra pinch of salt in your soup, if you felt a utility gradient that large. For that matter, your brain would have to be larger than the observable universe to feel a gradient that large.

I do not think people understand the largeness of the Large Number here.

Correction: What I said: "one-second of irritation is less than 3^^^3-times as bad as the 50 years of torture." What I meant: "50 years of torture is more than 3^^^3-times as bad as 1-second of eye-irritation." Apologies for the mis-type (as well as for saying "you're" when I meant "your").

But the point is, if there are no additional consequences to the suffering, then it's irrelevant. I don't care how many people experience the 1-second of suffering. There is no number large enough to make it matter.

Eliezer had a good point. It works if we're considering lives saved. It doesn't work with dust-specks and torture. It's not because torture is a taboo that only hard-headed rationalists are able to consider with a clear-mind. It's because something that's non-consequential is non-consequential, even when you multiply it by unimaginably large numbers. But we can't imagine torture and 50 years of lost time as being non-consequential for good reason. The example was bad. We should move on to more productive endeavors.

It's because something that's non-consequential is non-consequential

The dust specks are consequential; people suffer because of them. The further negative consequences of torture are only finitely bad.

Eliezer: would you torture a person for fifty years, if you lived in a large enough universe to contain 3^^^3 people, and if the omnipotent and omniscient ruler of that universe informed you that if you did not do so, he would carry out the dust-speck operation?

Seriously, would you pick up the blow torch and use it for the rest of your life, for the sake of the dust-specks?

Eliezer: it doesn't matter how big of a number you can write down. You are dealing with an asymptote. There is a limit to how bad momentary eye-irritation can be, no matter how many people it happens to. no matter how many people. That limit is far less than how bad a 50 year torture is.

let f(x) = (5x - 1)/x what is f(3^^^3)? It's 5, or close enough that it doesn't matter.

Eliezer: after wrestling with this for a while, I think I've identified at least one of the reasons for all the fighting. First of all, I agree with you that the people who say, "3^^^3 isn't large enough" are off-base. If there's some N that justifies the tradeoff, 3^^^3 is almost certainly big enough; and even if it isn't, we can change the number to 4^^^4, or 3^^^^3, or Busy Beaver (Busy Beaver (3^^^3)), or something, and we're back to the original problem.

For me, at least, the problem comes down to what 'preference' means. I don't think I have any coherent preferences over the idea of 3^^^3 dust specks. Note, I don't mean that I think my preferences are inconsistent, or poorly-formed, or that my intuition is bad. I don't think that talking about my preferences on that issue has any meaning.

Basically, I don't believe there's any objective standard of value. Even preferences like "I think as many people should die as painfully as possible" aren't wrong, per se; they just put you beyond the bounds of civilized society and make me have no desire to interact with you any more. So asking which of two circumstances is 'really better' doesn't have any meaning; 'better' only makes sense when you ask 'better to whom.' Which leads to two problems.

First is that the question tends to slip over to "which choice would you make." But once I start phrasing it in terms of me making a choice, all my procedural safeguards start kicking in. First, if you're a true deontologist your mental side constraints start jumping in. Even if you're sort-of utilitarian, like I am, the mental rules that say things like "we can't be sure that 3^^^3 people are actually going to suffer" and "helping to forge a society that considers torture acceptable leads to horrifying long-term consequences" kick in. I agree those are outside of the parameters of the original question; but the original question was ill-posed, and this is one of the places it slips to in translation.

But even if you avoid that, you still come to the question of what it means to prefer A over B, when you have no meaningful choice in the matter. I can't imagine a situation in which I could cause 3^^^3 people any coherent result. I'm not sure I believe there are or ever will be 3^^^3 moral agents. And do I have a coherent preference over circumstances that I will never know have occurred? Even if 3^^^3 people suffer, I'm not going to know that they do. It won't affect me, and I won't know that it affected anyone else, either.

Basically, moral questions that involve wildly unlikely or outright impossible scenarios don't tend to be terribly enlightening. If we lived in a world where we could reliably benefit unimaginably large numbers of people by causing vast pain to a few, maybe that would be okay. But since we don't, I think hypotheticals like this are more likely to short-circuit on the bounds of our extremely useful assumptions about the nature of the world than they are to tell us anything interesting.

Stick with your resolution to avoid meta-ethical reasoning, since yours sucks.

I share El's despair. Look at the forest, folks. The point is that you have to recognize that harm aggregates (and not to an asymptote) or you are willing to do terrible things. The idea of torture is introduced precisely to make it hard to see. But it is important, particularly in light of how easily our brains fail to scale harm and benefit. Geez, I don't even have to look at the research El cites - the comments are enough.

Stop saying the specks are "zero harm." This is a thought experiment and they are defined as positive harm.

Stop saying that torture is different. This is a thought experiment and torture is defined to be absolutely terrible, but finite, harm.

Stop saying that torture is infinite harm. That's just silly.

Stop proving the point over and over in the comments!

/rant/

This whole dust vs. torture "dilemma" depends on a couple assumptions: (1) That you can assign a cost to any event and that all such values lie within the same group (allowing multiples of one event to "add up" to another event) and (2) That the function that determines the cost of a certain number of a specific type of events does not have a hard upper limit (such as a logistic function). If either of these assumptions is wrong then the largeness of 3^^^3 or any other "large" number is totally irrelevant. One way to test (1) is to replace "torture" with "kill". If the answer is no then (1) is an invalidate assumption.

Larry D'anna:

You are dealing with an asymptote. There is a limit to how bad momentary eye-irritation can be, no matter how many people it happens to.

By positing that dust-speck irritation aggregates non-linearly with respect to number of persons, and thereby precisely exemplifying the scope-insensitivity that Eliezer is talking about, you are not making an argument against his thesis; instead, you are merely providing an example of what he's warning against.

You are in effect saying that as the number of persons increases, the marginal badness of the suffering of each new victim decreases. But why is it more of an offense to put a speck in the eye of Person #1 than Person #6873?

Isn't there maybe a class insignificant harms where net utility is neutral or even positive (learn to squint or where goggles in a duststorm, learn that motes in ones eye are annoying but nothing really to worry about, increased unerstanding of Christian parables, eg; also consider schools of parenting that allow children to experiment with various behaviors that the parents would prefer they avoid, since directly experiencing the adverse event in a more controlled situation will prevent worse outcomes down the road). I'm not sure you can trust most people's expressed preference on this.

That being said, I don't know where that class begins and ends.

Well, Eliezer has not only failed to recognize that he was wrong, he actively defended his point.

Should the ten credibility points be donated to charity, or dispersed among the readers of Overcoming Bias?

Bob: Sure, if you specify a disutility function that mandates lots-o'-specks to be worse than torture, decision theory will prefer torture. But that is literally begging the question, since you can write down a utility function to come to any conclusion you like. On what basis are you choosing that functional form? That's where the actual moral reasoning goes. For instance, here's a disutility function, without any of your dreaded asymptotes, that strictly prefers specks to torture:

U(T,S) = ST + S

Freaking out about asymptotes reflects a basic misunderstanding of decision theory, though. If you've got a rational preference relation, then you can always give a bounded utility function. (For example, the function I wrote above can be transformed to U(T,S) = (ST + S)/(ST + S + 1), which always gives you a function in [0,1], and gives rise to the same preference relation as the original.) If you absolutely require unbounded utilities in utility functions, then you become subject to a Dutch book (see Vann McGee's "An Airtight Dutch Book"). Attempts to salvage unbounded utility pretty much always end up accepting certain Dutch books as rational, which means you've rejected the whole decision-theoretic justification of Bayesian probability theory. Now, the existence of bounds means that if you have a monotone utility function, then the limits will be well-defined.

So asymptotic reasoning about monotonically increasing harms is entirely legit, and you can't rule it out of bounds without giving up on either Bayesianism or rational preferences.

Care to advance an argument, Caledonian? (Not saying I disagree... or agree, for that matter.)

If harm aggregates less-than-linearly in general, then the difference between the harm caused by 6271 murders and that caused by 6270 is less than the difference between the harm caused by one murder and that caused by zero. That is, it is worse to put a dust mote in someone's eye if no one else has one, than it is if lots of other people have one.

If relative utility is as nonlocal as that, it's entirely incalculable anyway. No one has any idea of how many beings are in the universe. It may be that murdering a few thousand people barely registers as harm, because eight trillion zarquons are murdered every second in Galaxy NQL-1193. However, Coca-Cola is relatively rare in the universe, so a marginal gain of one Coca-Cola is liable to be a far more weighty issue than a marginal loss of a few thousand individuals.

(This example is deliberately ridiculous.)

Neel Krishnaswami has already said pretty much everything I'm thinking.

From a rhetorical and logical perspective, it would have been much better if Eliezer had presented an experiment in which the participants were asked to choose between abstracted types of harm specially constructed to avoid all of the issues people have brought up... and still chose irrationally.

What's 'rational' in the example he did use is far from clear.

Oh, and his "I didn't have to do the math" is absurd, not only on its own grounds, but in the context of his broader points. Eliezer is normally an advocate for doing the math, and how he's brushing it off.

So what exactly do you multiply when you shut up and multiply? Can it be anything else then a function of the consequences? Because if it is a function of the consequences, you do believe or at least act as if believing your #4.

In which case I still want an answer to my previously raised and unanswered point: As Arrow demonstrated a contradiction-free aggregate utility function derived from different individual utility functions is not possible. So either you need to impose uniform utility functions or your "normalization" of intuition leads to a logical contradiction - which is simple, because it is math.

Neel Krishnaswami: you reference an article called "An Airtight Dutch Book," but I can't find it online without a subscription. Can you summarize the argument?

Neel, I think you and I are looking at this as two different questions. I'm fine with bounded utility at the individual level, not so good with bounds on some aggregate utility measure across an unbounded population (but certainly willing to listen to a counter position), which is what we're talking about here. Now, what form an aggregate utility function should take is a legitimate question (although, as Salutator points out, unlikely to be a productive discussion), but I doubt that you would argue it should be bounded.

I have really enjoyed following this discussion. My intuition to the initial post was "specks" but upon reflection I couldn't see how that could be. My intuition couldn't scale the problem correctly - I had to "do the math." Magically diminishing unit harm from the same action across an increasing number of individuals was not convincing. The notion that specks are not harm in the same sense that torture is harm is appealling in practice but was ruled out in the thought experiment, IMHO. Bottom line, if both specks and torture are finite harm (a reasonable premise, although the choice of "specks" was unfortunate because it opens the "zero harm" door), I can't see how there is not some sufficiently large number of specks that have to outweigh the torture. This is very uncomfortable because it implies that there is also (in theory) a sufficiently large number of specks that would outweigh 10^100 people being tortured. Ouch, my brain is trying to reject that conclusion. More evidence that I need to do the math. Luckily, this is all hypothetical. But scaling questions are not always hypothetical and nothing in this discussion has convinced me that intuition will give you the right answer. To the contrary...

Peter DeBlanc: check your email

The issue with a utility function U(T,S) = ST + S is that there is no motivation to have torture's utility depend on dust's utility. They are distinct and independent events, and in no way will additional specks worsen torture. If it is posited that dust specks asymptotically approach a bound lower than torture's bound, order issues present themselves and there should be rational preferences that place certain evils at such order that people should be unable to do anything but act to prevent those evils.

There's additional problems here, like the idea that distributing a dust speck to the group needs calculation in the group's net utility function, rather than in the individual's utility function. That is, if a group of ten people has 600 apples, they do not have 600*U(A), nor U(600A), but U_1(A_1)+ ... + U_10(A_10). Adding an additional apple has a marginal effect on net utility equal to the marginal effect on the net utility of the person receiving the apple. This result is in utils, and utils do sum linearly.

I'll say that again: Utils sum linearly. It's what they do. The rational utilitarian favors n utils gained from a chocolate as much as he favors avoiding -n utils of a stubbed toe. Summing -n utils over m people will have an identical effect on total or average utility as granting -(n*m) utility to one person.

If you reject any of the utilitarian or rational premises of the question, point them out, suggest your fix and defend it.

Caledonian: The idea is to make the math obvious. If you can't get the right answer with the math clean and easy, how can you do it on your own? If you insist there is a natural number greater than the cardinality of the reals, you will run into problems somewhere else. (And on the other hand, if you reject any of the concepts such as cardinality, reals, or "greater than", you probably shouldn't be taking a math class.)

Sean: why is that "what utils do"? To the extent that we view utils as the semi-scientific concept from economics, they don't "just sum linearly." To economists utils don't sum at all; you can't make interpersonal comparisons of utility. So if you claim that utils sum linearly, you're making a claim of moral philosophy, and haven't argued for it terribly strongly.

Sean, one problem is that people can't follow the arguments you suggest without these things being made explicit. So I'll try to do that:

Suppose the badness of distributed dust specks approaches a limit, say 10 disutility units.

On the other hand, let the badness of (a single case of ) 50 years of torture equal 10,000 disutility units. Then no number of dust specks will ever add up to the torture.

But what about 49 years of torture distributed among many? Presumably people will not be willing to say that this approaches a limit less than 10,000; otherwise we would torture a trillion people for 49 years rather than one person for 50.

So for the sake of definiteness, let 49 years of torture, repeatedly given to many, converge to a limit of 1,000,000 disutility units.

48 years of torture, let's say, might converge to 980,000 disutility units, or whatever.

Then since we can continuously decrease the pain until we reach the dust specks, there must be some pain that converges approximately to 10,000. Let's say that this is a stubbed toe.

Three possibilities: it converges exactly to 10,000, to less than 10,000, or more than 10,000. If it converges to less, than if we choose another pain ever so slightly greater than a toe-stubbing, this greater pain will converge to more than 10,000. Likewise, if it converges to more than 10,000, we can choose an ever so slightly less pain that converges to less than 10,000. If it converges to exactly 10,000, again we can choose a slightly greater pain, that will converge to more than 10,000.

Suppose the two pains are a stubbed toe that is noticed for 3.27 seconds, and one that is noticed for 3.28 seconds. Stubbed toes that are noticed for 3.27 seconds converge to 10,000 or less, let's say 9,999.9999. Stubbed toes that are notice for 3.28 seconds converge to 10,000.0001.

Now the problem should be obvious. There is some number of 3.28 second toe stubbings that is worse than torture, while there is no number of 3.27 second toe stubbings that is worse. So there is some number of 3.28 second toe stubbings such that no number of 3.27 second toe stubbings can ever match the 3.28 second toe stubbings.

On the other hand, three 3.27 second toe stubbings are surely worse than one 3.28 second toe stubbings. So as you increase the number of 3.28 second toe stubbings, there must be a magical point where the last 3.28 second toe stubbing crosses a line in the sand: up to that point, multiplied 3.27 second toe stubbings could be worse, but with that last 3.28 second stubbing, we can multiply the 3.27 second stubbings by a googleplex, or by whatever we like, and they will never be worse than that last, infinitely bad, 3.28 second toe stubbing.

So do the asymptote people really accept this? Your position requires it with mathematical necessity.

It seems to me that the dust-specks example depends on the following being true: both dust-specks and 50 years of torture can be precisely quantified.

What is the justification for this belief? I find it hard to see any way of avoiding the conclusion that some harms may be compared, as in A < B (A=1 person/1 dustspeck, B=1 person/torture), but that does not imply that we can assign precise values to A and B and then determine how many A are equivalent to one B.

Why do some people believe that we can precisely say how much worse the torture of 1 individual is than dust specks in the eye of 1 individual? Why are different harms necessarily commensurable? Any pointers to standard arguments or more information about this is much appreciated.

Joseph,

The point of using 3^^^3 is to avoid the need to assign precise values, which I agree seems impossible to do with any confidence. Once you accept the premise that A is less than B (with both being finite and nonzero), you need to accept that there exists some number k where kA is greater than B. The objections have been that A=0, B is infinite, or the operation kA is not only nonlinear, but bounded. The first may be valid for specks but misses the point - just change it to "mild hangnail" or "banged funnybone." I cannot take the second seriously. The third is tempting when disguised as something like "you cannot compare a banged funnybone with torture" but becomes less appealing when you ask "can it really cause (virtually) zero harm to bang a trillion funnybones just because you've already banged 10^100?"

It was the need for a "magical line" as in Unknown's example that convinced me. I'm truly curious why it fails to convince others. I admit I may be missing something but it seems very simple at its core.

"Renormalizing intuition" - that sounds like making sure all the intuitions are consistent and proportional to each other. Which is analogous to a coherence theory of truth as against a correspondence one. But you can make something as internally consistent as you like and maybe it still bears no relation to reality. It is necessary to know where the intuitions came from and what they mean.

Ideas such as good and evil are abstract, and the mind of a newborn can't hold abstract ideas, only simple concretes. So those ideas can't have already been there at birth, simply because of how abstract they are. Therefore the intuitions must be learned after birth, which means they most likely come from society. Which means other people. Other people who have no more magic source of truth than we do.

So all we are doing is sorting through and evening out the opinions of others that we have subconsciously absorbed, which doesn't seem a very scientific way to go.

Bob: "The point of using 3^^^3 is to avoid the need to assign precise values".

But then you are not facing up to the problems of your own ethical hypothesis. I insist that advocates of additive aggregation take seriously the problem of quantifying the exact ratio of badness between torture and speck-of-dust. The argument falls down if there is no such quantity, but how would you arrive at it, even in principle? I do not insist on an impersonally objective ratio of badness; we are talking about an idealized rational completion of one's personal preferences, nothing more. What considerations would allow you to determine what that ratio should be?

Unknown has pointed out that anyone who takes the opposite tack, insisting that 'any amount of X is always preferable to just one case of Y!', faces the problem of boundary cases: keep substituting worse and worse things for X, and eventually one will get into the territory of commensurable evils, and one will start trying to weigh up X' against Y.

However, this is not a knockdown argument for additivism. Let us say that I am clear about my preferences for situations A, D and E, but I am in a quandary with respect to B and C. Then I am presented with an alternative moral philosophy, which offers a clear decision procedure even for B and C, but at the price of violating my original preferences for A, D or E. Should I say, oh well, the desirability of being able to decide in all situations is so great that I should accept the new system, and abandon my original preferences? Or should I just keep thinking about B and C until I find a way to decide there as well? A utility function needs only to be able to rank everything, nothing more. There is absolutely no requirement that the utility (or disutility) associated with n occurrences of some event should scale linearly with n.

This is an important counterargument so I'll repeat it: The existence of problematic boundary cases is not yet a falsification of an ethical heuristic. Give your opponent a chance to think about the boundary cases, and see what they come up with! The same applies to my challenge to additive utilitarians, to say how they would arrive at an exact ratio: I am not asserting, apriori, that it is impossible. I am pointing out that it must be possible for your argument to be valid, and I'm giving you a chance to indicate how this can be done.

This whole thought experiment was, I believe, meant to illustrate a cognitive bias, a preference which, upon reflection, would appear to be mistaken, the mistake deriving from the principle that 'sacred values', such as an aversion to torture, always trump 'nonsacred values', like preventing minor inconveniences. But premises which pass for rational in a given time and culture - which are common sense, and just have to be so - can be wrong. The premise here is what I keep calling additivism, and we have every reason to scrutinize as critically as possible any premise which would endorse an evil of this magnitude (the 50 years of torture) as a necessary evil.

One last thought, I don't think Ben Jones's observation has been adequately answered. What if those 3^^^3 people are individually willing to endure the speck of dust rather than have someone tortured on their behalf? Again boundary cases arise. But if we're seriously going to resolve this question, rather than just all reaffirm our preferred intuitions, we need to keep looking at such details.

Mitchell, if I say an average second of the torture is about equal 10,000 distributed dust specks (notice I said "average second"; there is absolutely no claim that torture adds up linearly or anything like that), then something less than 2 trillion dust specks will be about equal to 50 years of torture. I would arrive at the ratio by some comparison of this sort, trying to guess how bad an average second is, and how many dust specks I would be willing to inflict to save a man from that amount of harm.

Notice that 3^^^3 is completely unnecessary here. That's why I said previously that N doesn't have to be particularly large.

Thanks for the explanations, Bob.

Bob: The point of using 3^^^3 is to avoid the need to assign precise values... Once you accept the premise that A is less than B (with both being finite and nonzero), you need to accept that there exists some number k where kA is greater than B.

This still requires that they are commensurable though, which is what seeking a strong argument for. Saying that 3^^^3 dust specks in 3^^^3 eyes is greater harm than 50 years of torture means that they are commensurable and that whatever the utilities are, 3^^^3 specks divided by 50 years of torture is greater than 1.0. I don't see that they are commensurable. A < B < C < D doesn't imply that there's some k such that kA>D.

Consider: I prefer Bach to Radiohead (though I love both). That doesn't imply that there's some ratio of Bach to Radiohead, or that I think a certain number of Radiohead songs are collectively better than or more desirable than, for example, the d-minor partita. Even if I did in some cases believe that 10 Radiohead songs were worth 1 Bach prelude and fugue, that would just be my subjective feeling. I don't see why there must be an objective ratio, and I can't see grounds for saying what such a ratio would be. Likewise for dust-specks and torture.

Like Mitchell, I would like to see exactly how people propose to assign these ratios such that a certan number of one harm is greater than a radically different harm.

A < B < C < D doesn't imply that there's some k such that kA>D

Yes it does.

Again, we return to the central issue:

Why must we accept an additive model of harm to be rational?

If you don't accept the additivity of harm, you accept that for any harm x, there is some number of people y where 2^y people suffering x harm is the same welfare wise as y people suffering x harm.

(Not to mention that when normalized across people, utils are meant to provide direct and simple mathematical comparisons. In this case, it doesn't really matter how the normalization occurs as the inequality stands for any epsilon of dust-speck harm greater than zero.)

Polling people to find if they will take a dust speck grants an external harm to the torture (e.g., mental distress at the thought of someone being tortured). Since they would prefer the dust speck, this indicates that they find the thought of someone being subject to 50 years of torture (Harm a) to be of more harm than a dust speck (Harm b). Harm a > Harm b, so n Harm a > n Harm b, and it doesn't even matter what Harm a or Harm b is, nor the additional nondistributed harm of the actual torture. How could this be tricked? Replace the dust speck with the least harm greater than the distress at the thought of someone being tortured for 50 years, say, the thought of someone being tortured for 51 years.

There are no natural utility differences that large. (Eliezer, re 3^^^3)

You've measured this with your utility meter, yes?

If you mean that it's not possible for there to be a utility difference that large, because the smallest possible utility shift is the size of a single particle moving a planck distance, and the largest possible utility difference is the creation or destruction of the universe, and the scale between those two is smaller than 3^^^3 ... then you'll have to remind me again where all these 3^^^3 people that are getting dust specks in their eyes live.

If 3^^^3 makes the math unnecessary because utility differences can't be that large, then your example fails to prove anything because it can't take place. For your example to be meaningful, it is necessary to postulate a universe in which 3^^^3 people can suffer a very small harm, which necessarily implies that yes, in fact, it is possible in this hypothetical universe for one thing to have 3^^^3 times the utility of something else. At which point, in order to prove that the dust specks outweigh the torture, you will now have to shut up and multiply. And be sure to show your work.

Your first task in performing this multiplication will be to measure the harm from torture and dust specks.

Good luck.

If you don't accept the additivity of harm, you accept that for any harm x, there is some number of people y where 2^y people suffering x harm is the same welfare wise as y people suffering x harm.

No. I can imagine non-additive harm evaluation systems where that is not the case.

Even in the limited subset of systems where that IS the case, so what?

Although the argument didn't depend on harm adding linearly in any case, it is true that two similar harms to two different people must be exactly twice as bad as one harm to one person.

Many people on this thread have already given the reason: how bad it is that someone is killed, or tortured, or dust specked, obviously does not depend on how many other people this has happened to. Otherwise death couldn't be such a bad thing, since it has already happened to almost everyone.

Sean, say you're one of the 3^^^3 voters. A vote one way potentially has torture on your conscience, as you say, but a vote the other way potentially has 3^^^3 dust specks on your conscience - by your definition a much greater sin. Square one - shut up and vote!

I am aware that a billion people getting punched in the face appears to aggregate to a greater harm than one person being tortured for 50 years. (I should say that when I ask my intuition what it thinks about the number 1,000,000,000, it says 'WTF?', so it's not coming from there....) However, if I was one of the billion and had to cast my vote, I'd ask for the punch, and hope you would too. (I am also confident that it would be because it was the right thing for me to do, and not to alleviate any potential guilt.) What does this say about intuition and additivism?

Ben, that's not about additivism, but indicates that you are a deontologist by nature, as everyone is. A better test: would you flip a lever which would stop the torture if everyone on earth would instantly be automatically punched in the face? I don't think I would.

Joseph et al, I appreciate your thoughts. I think, though, that your objections keep coming back to "it's more complicated." And in reality it would be. But the simple thought experiment suggests that any realistic derivative of the specks question would likely get answered wrong because our (OUR!) intuition is heavily biased toward large (in aggregate) distributed harm. It appears that we personalize individual harm but not group harm.

Ben, I assume that we would all vote that way, if only because the thought of having sentenced someone to torture would be more painful than the punch. But you've changed the question. How would you choose if faced with deciding that either 1) every person on earth receives a painful but nonfatal punch in the face or 2) one random person is chosen to be tortured? That's the specks question.

If politics is the mind killer, morality is at least the mind masher. We should probably only talk about morality in small doses, interspersed with many other topics our minds can more easily manage.

@Sean

If your utility function u was replaced by 3u,there would be no observable difference in your behavior. So which of these functions is declared real and goes on to the interpersonal summing? "The same factor for everyone" isn't an answer, because if u_you doesn't equal u_me "the same factor" is simply meaningless.

@Tomhs2

A < B < C < D doesn't imply that there's some k such that kA>D

Yes it does.

I think you're letting the notation confuse you. It would imply that, if A,B,C,D where e.g. real numbers, and that is the context the "<"-sign is mostly used in. But Orders can exist on sets other then sets of numbers. You can for example sort (order) the telephone book alphabetically, so that Cooper < Smith and still there is no k so that k*Cooper>Smith.

@most people here:

A lot of confusion is caused by the unspoken premise that a moral system should sort outcomes rather then actions, so that it doesn't matter who would do the torturing or speck-placing. Now for Eliezer that assumption is de fide, because otherwise the concept of a friendly AI (sharing our ends and choosing the unimportant-declared means with its superior intelligence) is meaningless. But the assumption contradicts basically everyone's intuition. So why should it convince anyone not following Eliezer's religion?

[Edit: fixed some typos and formating years later]

Ben, that's not about additivism, but indicates that you are a deontologist by nature, as everyone is. A better test: would you flip a lever which would stop the torture if everyone on earth would instantly be automatically punched in the face? I don't think I would.

I'm fairly certain I would pull the lever. And I'm certain that if I had to watch the person be tortured (or do it myself!) I would happily pull the lever.

It was this sort of intuition that motivated my earlier question to Eliezer (which he still hasn't responded to). I'd be interested to hear from any of the people advocating torture over specks, though: would you all be willing to personally commit the torture, knowing that the alternative is either (a) a punch in the face (without permanent injury/pain/risk of death) to every human on the planet, or (b) in a universe with 3^^^3 people in it, a speck in everyone's eye?

I suspect that the most people would refuse to pick up the blowtorch in either case. Which is very important -- only some very exotic, and very implausible, metaethical theories would casually disregard the ethical intuitions of the vast majority of people.

@Unknown
So if everyone is a deontologist by nature, shouldn't a "normalization" of intuitions result in a deontological system of morals? If so, what makes you look for the right utilitarian system?

I think you're letting the notation confuse you. It would imply that, if A,B,C,D where e.g. real numbers, and that is the context the "<"-sign is mostly used in. But Orders can exist on sets other then sets of numbers. You can for example sort (order) the telephone book alphabetically, so that Cooper < Smith and still there is no k so that k*Cooper>Smith.

This is fairly confusing...in the telephone book example, you haven't defined * as an operator. I frankly have no idea what you would mean by it. Using the notation kA > D implies a defined multiplication operation, which any reader should naturally understand as the one we all use everyday (and hence, we must assume that the set contains the sort of objects to which our everyday understanding of multiplication normally applies).

Now, this doesn't mean that you are wrong to say that, on all such sets, kA > D does not follow necessarily from A < B < C < D for some k. It does not obtain, for instance, when A = 0. It also wouldn't obtain on the natural numbers modulo 8, for A=2 and D=5 (just to take one example -- it should be easy to create others). But neither of these have any relation to the context in which you made your claim.

So, the question is, can you find a plausible set of definitions for your set that makes your claim relevant to this conversation?

Eisegates, is there no limit to the number of people you would subject to a punch in the face (very painful but temporary with no risk of death) in order to avoid the torture of one person? What if you personally had to do (at least some of) the punching? I agree that I might not be willing to personally commit the torture despite the terrible (aggregate) harm my refusal would bring, but I'm not proud of that fact - it seems selfish to me. And extrapolating your position seems to justify pretty terrible acts. It seems to me that the punch is equivalent to some very small amount of torture.

Salutator: "...unspoken premise, that a moral system should sort outcomes rather then actions, so that it doesn't matter who would do the torturing or speck-placing."

I fear that I'm missing something. Is this just another way of asking if I would pick up the blowtorch? If it's true, and you seem to agree, that our intuition focuses on actions over outcomes, don't you think that's a problem? Perhaps you're not convinced that our intuition reflects a bias? That we'd make better decisions if we shifted a little bit of our attention to outcomes? Or is the fear that "doing the math" will produce a more biased position than intuition alone? I think that you need to do the math to balance, not (in all cases) replace, your intuition. El's point was that this is particularly important when you scale because your intuition cannot be (blindly) trusted.

Eisegates, is there no limit to the number of people you would subject to a punch in the face (very painful but temporary with no risk of death) in order to avoid the torture of one person? What if you personally had to do (at least some of) the punching? I agree that I might not be willing to personally commit the torture despite the terrible (aggregate) harm my refusal would bring, but I'm not proud of that fact - it seems selfish to me. And extrapolating your position seems to justify pretty terrible acts. It seems to me that the punch is equivalent to some very small amount of torture.

1. My intuition on this point is very insensitve to scale. You could put a googol of persons in the galaxy, and faced with a choice between torturing one of them and causing them all to take one shot, I'd probably choose the punches.

2. Depends how much punching I had to do. I'd happily punch a hundred people, and let others do the rest of the work, to keep one stranger from getting tortured for the rest of his life. Make it one of my loved ones at risk of torture, and I would punch people until the day I die (presumably, I would be given icepacks from time to time for my hand).

3. Extrapolating is risky business with ethical intuitions. Change the facts and my intuition might change, too. I think that, in general, ethical intuitions are highly complex products of social forces that do not reduce well to abstract moral theories -- either of the deontological or utilitarian variety. And to the extent that moral statements are cognitive, I think they are referring to these sort of sociological facts -- meaning that any abstract theory will end up incorporating a lot of error along the way.

Would you really feel selfish in the dust speck scenario? I think, at the end of the day, I'd feel pretty good about the whole thing.

Please let me interrupt this discussion on utilitarianism/humanism with an alternative perspective.

I do not claim to know what the meaning of life is, but I can rule certain answers out. For example, I am highly certain that it is not to maximize the number of paperclips in my vicinity.

I also believe it has nothing to do with how much pain or pleasure the humans experience -- or in fact anything to do with the humans.

More broadly, I believe that although perhaps intelligent or ethical agents are somehow integral to the meaning of life, they are integral for what they do, not because the success or failure of the universe hinges somehow on what the sentients experience or whether their preferences or desires are realized.

Humans, or rather human civilization, which is an amalgam of humans, knowledge and machines, are of course the most potent means anyone knows about for executing plans and achieving goals. Hobble the humans and you probably hobble whatever it is that really is the meaning of life.

But I firmly reject humanity as repository of ultimate moral value.

I looks to me like Eliezer plans to put humanism at the center of the intelligence explosion. I think that is a bad idea. I am horrified. I am appalled.

I think claims like "exactly twice as bad" are ill-defined.

Suppose you have some preference relation on possible states R, so that X is preferred to Y if and only if R(X, Y) holds. Next, suppose we have a utility function U, such that if R(X, Y) holds, then U(X) > U(Y). Now, take any monotone transformation of this utility function. For example, we can take the exponential of U, and define U'(X) = 2^(U(X)). Now, note that U(X) > U(Y) if and only if that U'(X) > U'(Y). Now, even if U is additive along some dimension of X, U' won't be.

But there's no principled reason to believe that U is a "truer" reflection of the agent's preferences than U', since both of them are equally faithful to the underlying preference relation. So if you want to do meaningful comparisons of utility you have to do them in a way that's invariant under monotone transformations. Since "twice as bad" isn't invariant such a transformation, it's not evidently a meaningful claim.

Now, there might be some additional principle you can advance to justify claims like that, but I haven't seen it, or its justification, yet.

Krishnaswami: I think claims like "exactly twice as bad" are ill-defined. Suppose you have some preference relation on possible states R, so that X is preferred to Y if and only if R(X, Y) holds. Next, suppose we have a utility function U, such that if R(X, Y) holds, then U(X) > U(Y). Now, take any monotone transformation of this utility function. For example, we can take the exponential of U, and define U'(X) = 2^(U(X)). Now, note that U(X) > U(Y) if and only if that U'(X) > U'(Y). Now, even if U is additive along some dimension of X, U' won't be.

Utility functions over outcomes have additional structure beyond tehir ordering, because of how utilities interact with scalar probabilities to produce expected utilities that imply preferences over actions (as distinct from preferences over outcomes).

Taking the exponential of a positive utility function will produce the same preference ordering over outcomes but not the same preference ordering over actions (which is itself a quite interesting observation!) given fixed beliefs about conditional probabilities.

So when I say that two punches to two faces are twice as bad as one punch, I mean that if I would be willing to trade off the distance from the status quo to one punch in the face against a billionth (probability) of the distance between the status quo and one person being tortured for one week, then I would be willing to trade off the distance from the status quo to two people being punched in the face against a two-billionths probability of one person being tortured for one week. ("If...then" because I don't necessarily defend this as a good preference - the actual comparison here is controversial even for utilitarians, since there are no overwhelming quantities involved.)

Any positive affine transformation of the utility function preserves the preference ordering over actions. The above statement is invariant under positive affine transformations of the utility function over outcomes, and thus exposes the underlying structure of the utility function. It's not that events have some intrinsic number of utilons attached to them - a utility function invariantly describes the ratios of intervals between outcomes. This is what remains invariant under a positive affine transformation.

(I haven't heard this pointed out anywhere, come to think, but surely it must have been observed before.)

"Polling people to find if they will take a dust speck grants an external harm to the torture (e.g., mental distress at the thought of someone being tortured)."

TYPE ERROR

This is what remains invariant under a positive affine transformation.

(I haven't heard this pointed out anywhere, come to think, but surely it must have been observed before.)

Didn't Marcello point that out to you a couple years ago?

i got to tell you guys, a dust speck just flew in my eye, and man it was torture.

I think I've found one of the factors (besides scope insensivity) involved in the intuitive choice: in real life, a small amount of harm inflicted n times to one person has negative side-effects which don't happen when you inflict it once to n persons. Even though there aren't any in this thought experiment, we are so used to it we probably take it into account (at least I did).

Peter, I'm not sure what the chain of causality was there. (Let me know if I've previously written it down.) I think you or Nick Hay said that utility functions obey positive affine transformations, Marcello said that preserved the ratios of intervals, and I sketched out the interpretation for optimization problems.

I just meant that I haven't seen it elsewhere in the Literature. You're right, I should have credited the Summer of AI group.

Eisegetes, would you pull the lever if it would stop someone from being tortured for 50 years, but inflict one day of torture on each human being in the world? And if so, how about one year? or 10 years, or 25? In other words, the same problem arises as with the specks. Perhaps you can defend one punch per human being, but there must be some number of human beings for whom one punch each would outweigh torture.

Salutator, I never said utilitarianism is completely true.

Also: I wonder if Robin Hanson's comment shows concern about the lack of comments on his posts?

Hmmm... What can we actually agree on?

The disutility of a pain is a function of the Number of people who experience the pain, the Intensity of the pain, and the Time the pain lasts. It also an increasing function of all three: all else being equal, a pain experienced by more people is worse than one experienced by less people, a more intense pain is worse than a less intense pain, and a longer pain is worse than a shorter one. Or, more formally,

U = f(N,I,T)

∂U/∂N > 0 (for I,T > 0)
∂U/∂I > 0 (for N,T > 0)
∂U/∂T > 0 (for N,I > 0)
[In case that symbol doesn't display properly, it's supposed to be a partial derivative sign.]

Furthermore, for all finite N,I,T:
U(0,I,T) = 0
U(N,0,T) = 0
U(N,I,0) = 0

Do we at least agree on that much?

Doug, I do not agree because my utility function depends on the identity of the people involved, not simply on N. Specifically, it might be possible for an agent to become confident that Bob is much more useful to whatever is the real meaning of life than Charlie is, in which case a harm to Bob has greater disutility in my system than a harm to Charlie. In other words, I do not consider egalitarianism to be a moral principle that applies to every situation without exception. So, for me U is not a function of (N,I,T)

There seems to be an unexamined assumption here.

Why should the moral weight of applying a specified harm to someone be independent of who it is?
When making moral decisions, I tend to weight effects on my friends and family most heavily, then acquaintences, then fellow Americans, and so on. I value random strangers to some extent, but this is based more on arguments about the small size of the planet than true concern for their welfare.

I claim that moral obligations must be reciprocal in order to exist. Altruism is never mandatory.

None of Eliezer's 3^^^3 people will
(with the given hypotheses) ever interact with anyone on Earth or any of their descendents.
I think the sum of moral weights I would assign to these 3^^^3 people would be less than
the sum of weights for (e.g.) all inhabitants of Earth from 2000BC to the present. I would happily
subject all of them to dust motes to prevent one American from being tortured for 50 years, and would think less of any fellow citizen who would not do the same.

1: First of all, I want to acknowledge my belief that Eliezer's thought experiment is indeed usefuel, although it is "worse" than hypothetical. This is because it forces us to either face our psychological limitations when it comes to moral intuitions, or succumb to them (by arguing that the thought experiment is fundamentally unsound, in order to preserve harmony among our contradictive intuitions).
2: Once we admit that our patchwork'o'rules'o'thumb moral intuitions are indeed contradictive, the question remains if he is actually right. In another comment I have implied that one must either be an utilitarian or strictly amoral (actually I forgot the third option: one can be neither by being irrational). If this assertion is true then, in my book, Eliezer wins.
3: As I believe 1 to be sound, I'd really like to hear voices about 2. =)

Frank, re: #2: One can also believe option 4: that pleasure and pain have some moral significance, but do not perfectly determine moral outcomes. That is not necessarily irrational, it is not amoral, and it is not utilitarian. Indeed, I would posit that it represents the primary strand of all moral thinking and intuitions, so it is strange that it wasn't on your list.

Unknown: 10 years and I would leave the lever alone, no doubt. 1 day is a very hard call; probably I would pull the lever. Most of us could get over 1 day over torture in a way that is fundamentally different from years of torture, after all.

Perhaps you can defend one punch per human being, but there must be some number of human beings for whom one punch each would outweigh torture.

As I said, I don't have that intuition. A punch is a fairly trivial harm. I doubt I would ever feel worse about a lot of people (even 5^^^^^^5) getting punched than about a single individual being tortured for a lifetime. Sorry -- I am just not very aggregative when it comes to these sorts of attitudes.

Is that "irrational?" Frankly, I'm not sure the word applies in the sense you mean. It is inconsistent with most accounts of strict utilitarianism. But I don't agree that abstract ethical theories have truth values in the sense you probably assume. It is consistent with my attitudes and preferences, and with my society's attitudes and preferences, I think. You assume that we should be able to add those attitudes up and do math with them, but I don't see why that should necessarily be the case.

I think the difference is that you are assuming (at least in a very background sort of way) that there are non-natural, mind-independent, moral facts somehow engrafted onto the structure of reality. You feel like those entities should behave like physical entities, however, in being subject to the sort of mathematical relations we have developed based upon our interactions with real-world entities (even if those relations are now used abstractly). Even if you could make a strong argument for the existence of these sorts of moral rules, however, that is a far sight from saying that they should have an internal structure that behaves in a mathematically-tidy way.

You haven't ever given reasons to think that ethical truths ought to obey mathematical rules; you've just assumed it. It's easy to miss this assumption unless you've spent some time mulling over moral ontology, but it definitely animates most of the arguments made in this thread.

In short: unless you've grappled seriously with what you mean when you talk of moral rules, you have very little basis for assuming that you should be able to do sums with them. Is 6 billion punches for everyone "worse than" 50 years of torture for one person? It certainly involves the firing of more pain neurons. But the fact that a number of pain neurons fire is just a fact about the world; it isn't the answer to a moral question, UNLESS you make a large number of assumptions. I agree that we can count neuron-firings, and do sums with them, and all other sorts of things. I just disagree that the firing of pain and pleasure neurons is the sum total of what we mean when we say "it was wrong of Fred to murder Sally."

Eisegetes: I admit your fourth option did not even enter my mind. I'll try (in a rather ad-hoc way) to dispute this on the grounds of computationalism. To be able to impose an order on conflicting options, it must be possible to reduced the combined expected outcomes (pleasure, displeasure, whatever else) into a single scalar value. Even if they are in some way lexically ordered, we can do this by projecting the lexical options onto non-intersecting intervals. Everything that is morally significant does, by virtue of the definition, enter into this calculus. Everything that doesn't, isn't.
If you feel this does not apply, please help me by elaborating your objection.

@Eisegates
Yes, I was operating on the implicit convention, that true statements must be meaningfull, so I could also say there is no k, so that I have exactly k quobbelwocks.
The nonexistence of a -operator (and of a +-operator) is actually the point. I don't think preferences of different persons can be meaningfully combined, and that includes, that {possible world-states} or {possible actions} don't, in your formulation, contain the sort of objects to which our everyday understanding of multiplication normally applies. Now if you insist on an intuitively defined -operator every bounded utility function is an example. For example my utility for the amount c of chocolate available for consumption in some given timeframe could well be approximately 1- exp(1-(min(c/1kg,1)), so 100g<1kg but there is no k to make k*100g>1kg. That is, of course, nothing new even in this discussion. Also more directly to the point, me doing evil is something I should avoid more then other people doing evil. So when I do the choosing "I kill 1 innocent person" < "someone else kills 1 innocent person", but there is no k so that "I kill 1 innocent person"> "someone else kills k innocent persons". In fact, if a kidnapper plausibly threatened to kill his k hostages unless I kill a random passerby almost nobody would think me justified in doing so for an imaginable value of k. That people may think different for unimaginably large values of k is a much more plausible candidate for failure to be rational whit large numbers then not adding speckles up to torture.

But basically I wasn't making a claim, just trying to give an understandable (or so I thought) formulation for denying Thombs' non-technically stated claim that existence of an order implies the Archimedian axiom.

@Bob
If it's true, and you seem to agree, that our intuition focuses on actions over outcomes, don't you think that's a problem? Perhaps you're not convinced that our intuition reflects a bias? That we'd make better decisions if we shifted a little bit of our attention to outcomes?
You nailed it. Not only am I not convinced, that our intuition on this point reflects a bias, I'm actually convinced, that it doesn't. Utility is irrelevant, rights are relevant. And while I may sacrifice a lesser right for a greater right I can't sacrifice a person for another person. So in the torture example I may not flip the (50a,1 person/49a, 2 persons)switch either way.

@Doug S.
I disagree. An objective U doesn't exist and individual Us can't be meaningfully aggregated. Moreover, if the individual Us are meant to be von-Neumann-Morgenstern-functions they don't exist either.

Eliezer:

no, Nick Hay and I were not involved at all. You mentioned this to us as something you and Marcello had discussed before the summer of AI.

Frank, I think a utility function like that is a mathematical abstraction, and nothing more. People do not, in fact, have scalar-ordered ranked preferences across every possible hypothetical outcome. They are essentially indifferent between a wide range of choices. And anyway, I'm sure that there is sufficient agreement among moral agents to permit the useful aggregation of their varied, and sometimes conflicting, notions of what is preferable into a single useful metric. And even if we could do that, I'm not sure that such a function would correspond with all (or even most) of the standard ways that we use moral language.

The statement that X is wrong can be taken to mean that X has bad consequences according to some metric. It can also mean (or be used to perform the functions of) the following variants:

(1) I do not approve of X.
(2) X makes me squeamish.
(3) Most people in [relevant group] would disapprove of X.
(4) X is not an exemplar of an action that corresponds with what I believe to be appropriate rules to live by.
(5) [Same as 4, but change reference point to social group]
(6) X is not an action that would be performed by a virtuous person operating in similar circumstances.
(7) I do not want X to occur.
(8) Do not do X.

That is probably not even an exhaustive list. Most uses of moral language probably blur the lines between a large number of these statements. Even if you want to limit the discussion to consequences, however, you have to pick a metric; if you are referring only to "bad" or "undesireable" consequences, you have to incorporate some other form of moral reasoning in order to articulate why your particular metric is constitutive or representative of what is wrong.

Hence, I think the problem with you argument is that (a) I'm not sure that there is enough agreement about morality to make a universal scalar ordering meaningful, and (b) a scalar ordering would be meaningless for many plausible variants of what morality means.

Salutator: thanks for clarifying. I would tend to think that physical facts like neural firings can be quite easily multiplied. I think the problem has less to do with the multiplying, than with the assumption that the number of neural firings is constitutive of wrongness.

Eliezer: So when I say that two punches to two faces are twice as bad as one punch, I mean that if I would be willing to trade off the distance from the status quo to one punch in the face against a billionth (probability) of the distance between the status quo and one person being tortured for one week, then I would be willing to trade off the distance from the status quo to two people being punched in the face against a two-billionths probability of one person being tortured for one week.

So alternatives that have twice the probability of some good thing X happening have twice the utility? A sure gain of a dollar has twice the utility of a gaining a dollar on a coin flip? Insurance companies and casinos certainly think so, but their customers certainly don't.

I think you are conflating utility and expected utility. I'm not convinced they are the same thing, although I think you believe they are.

Eisegetes: This is my third posting now, and I hope I will be forgiven by the powers that be...

Your (a): I was not talking about a universal, but of a personal scalar ordering. Somewhere inside everybody's brain there must be a mechanism that decides which of the considered options wins the competition for "most moral option of the moment". Once the existence of this (personal) ordering is acknowledged (rationality), we can either disavow it (amorality) or try our best with what we have [always keeping in mind that the mechanisms at work are imperfect] - including math (utilitarianism).

Your (b): I view morality not as the set of rules postulated by creed X at time T but as the result of a genetically biased social learning process. Morality is expressed through it's influence on every (healthy) individual's personal utility function.

"The statement that X is wrong can be taken to mean that X has bad consequences according to some metric. It can also mean (or be used to perform the functions of) the following variants:"

(1,2,4,6) X makes me feel bad because it triggers one of my morality circuits.
(3,5) X makes me nervous because [relevant group] might retribute.
(7) I do not want X to occur.
(8) ? [Sorry, I don't understand this one.]

me: A < B < C < D doesn't imply that there's some k such that kA>D

Tomh: Yes it does.

As Salutator stated, perhaps I should not have used the notation I did in my example. What I mean by '<' in the context of harms is "is preferred to". What I meant when I said that there was no k such that kA > D is that the notion of multiplication does not make sense when applied to "is preferred to". Perhaps I should not have used the notation I did. Apologies for the confusion.

I looks to me like Eliezer plans to put humanism at the center of the intelligence explosion. I think that is a bad idea. I am horrified. I am appalled.

I wouldn't worry about it if I were you. One of the worst cases of yang excess I've ever seen.

Are you familiar with the concept of a Monkey Trap?

When I write U(N,I,T), I was trying to refer to the preferences of the person being presented with the scenario; if the person being asked the question was a wicked sadist, he might prefer more suffering to less suffering. Specifically, I was trying to come up with a "least common denominator" list of relevant factors that can matter in this kind of scenario. Apparently "how close I am to the person who suffers the pain" is another significant factor in the preferences, at least for Richard.

If we stipulate that, say, the pain is to be experienced by a human living on a planet orbiting Alpha Centauri 100,000,000 years from now, then does it make sense that N, I, and T provide enough information to fully define a preference function for the individual answering the question? [For example, all else being equal, I prefer the world in which you (a stranger) don't stub your toe tomorrow at 11:00 AM to the one in which you do stub your toe but is otherwise identical in every way I care about.] If you literally don't care at all about humans near Alpha Centauri living 100,000,000 years in the future, then your preference function would be constant.

There also seem to be some relevant bounds on N, I, and T. There are only so many humans that exist (or will exist), which bounds N. There is a worst possible pain that a human brain can experience, which provides an upper bound maximum for I. Finally, a human has a finite lifespan, which bounds T. (In the extreme case, T is bounded by the lifetime of the universe.)

The answer is simple. If you accept the bounds of the dust-speck argument where there is no further consequence of the dust-speck beyond the moment of irritation, then the cost of the irritation cannot be distinguished from 0 cost. If I can be assured that an event will have no negative consequences in my life beyond the quality of a moment of experience, then I wouldn't even think that the event is worth my consideration. Utility = 0. Multiply any number by 0, and you get 0. The only way for the dust-speck to have negative utility is if it has some sort of impact on my life beyond that moment. The dust-speck argument can't work without violating its own assumptions. Torture is worse. Case closed.

Adam, by that argument the torture is worth 0 as well, since after 1,000,000 years, no one will remember the torture or any of its consequences. So you should be entirely indifferent between the two, since each is worth zero.

But I guess the utility could be considered to be non-0 and without further impact if some individual would choose for it not to happen to them. All else being equal, I would rather not have my eye irritated (even if I had no further consequences). And even if cost is super-astronomically small, Eliezer could think up a super-duper astronomically large number by which it could be multiplied. I guess he was right.
I'm confused.
I think I'm done.

Richard Hollerith: "It looks to me like Eliezer plans to put humanism at the center of the intelligence explosion."

"Renormalized" humanism, perhaps; the outcome of which need not be anthropocentric in any way. You are a human being, and you have come up with some non-anthropocentric value system for yourself. This more or less demonstrates that you can start with a human utility function and still produce such an outcome. But there is no point in trying to completely ditch human-specific preferences before doing anything else; if you did that, you wouldn't even be able to reject paperclip maximization.

But you've changed the question.

I've added a wildcard, certainly, but I haven't changed the game. Say I'm standing there, lever in hand. While I can't be certain, I can fairly safely assume that if I went person to person and asked, the vast majority of those 3^^^3 would be personally willing to suffer a dust speck to save one person's torture. So I'm not necessarily polling, I'm just conjecturing. With this in mind, I choose specks.

[If I were to poll people, every now and then I would probably come across a Cold Hard Ratinoalist who said, "well, I'm happy to take the speck, but I have to consider the potential disutility to all these people if I said 'dust'...."

And I would reply, "That's not what I asked you! Let them worry about that. Get over yourself, shut up, and vote!"]

Ben: suppose the lever has a continuous scale of values between 1 and 3^^^3. When the lever is set to 1, 1 person is being tortured (and the torture will last for 50 years.). If you set it to 2, two people will be tortured by an amount less the first person by 1/3^^^3 of the difference between the 50 years and a dust speck. If you set it to 3, three people will be tortured by an amount less than the first person by 2/3^^^3 of the difference between the 50 years and the dust speck. Naturally, if you pull the lever all the way to 3^^^3, that number of people will suffer the dust specks.

Will you pull the lever over to 3^^^3? And if so, will you assert that things are getting better during the intermediate stages (for example when you are torturing a google persons by an amount less than the first person by an entirely insignificant quantity?) And if things first get worse and then get better, where does it change?

Will you try to pull the lever over to 3^^^3 if there's a significant chance the lever might get stuck somewhere in the middle?

Unknown, that's a very interesting take indeed, and a good argument for Eliezer's proposition, but it doesn't say much about what to do if you can assume most of the 3^^^3 would ask for dust. Can you tell me what you would do purely in the context of my previous post?

If you set it to 2, two people will be tortured by an amount less the first person by 1/3^^^3 of the difference between the 50 years and a dust speck.

Of course not, this would be a no-brainer ratio for the lever to operate with. You should have said that position 2 on the lever tortures 2 people for something like 25.0001 years. That puts me in far more of a quandary. And intuition (gasp!) leads me to believe that while harm does, of course, aggregate over people, it aggregates slightly less than linearly. In this case, push the lever as far as it can go! Spread that harm as thinly as you can! [Braces self for backlash....]

Say we found the magical (subjective, natch) ratio of harm-to-people (which must exist). This ratio is plugged into the lever - harm decreases with people exactly along this line. If two people getting dusted once is equal to one person getting dusted twice, does this mean you don't care where the lever is placed, since (harm)/(people) = k ?

Will you try to pull the lever over to 3^^^3 if there's a significant chance the lever might get stuck somewhere in the middle?

I would make sure I had an oil can to hand. ;)

To your voting scenario: I vote to torture the terrorist who proposes this choice to everyone. In other words, asking each one personally, "Would you rather be dust specked or have someone randomly tortured?" would be much like a terrorist demanding $1 per person (from the whole world), otherwise he will kill someone. In this case, of course, one would kill the terrorist.

I'm still thinking about the best way to set up the lever to make the point the most obvious.

What if everyone would be willing to individually suffer 10 years of torture to spare the one person? Obviously it's not better to torture 3^^^3 people for 10 years than one person for 50 years.

Obviously it's not better to torture 3^^^3 people for 10 years than one person for 50 years.

Obviously? There's that word again.

If it's really so obvious, please explain and elaborate on why it's not better.

Ben, here's my new and improved lever. It has 7,625,597,484,987 settings. On setting 1, 1 person is tortured for 50 years plus the pain of one dust speck. On setting 2, 3 persons are tortured for 50 years minus the pain of (50-year torture/7,625,597,484,987), i.e. they are tortured for a minute fraction of a second less than 50 years, again plus the pain of one dust speck. On setting 3, 3^3 persons, i.e. 27 persons, are tortured for 50 years minus two such fractions of a second, plus the pain of one dust speck. On setting 4, 3^27, i.e. 7,625,597,484,987 persons are tortured for 50 years minus 3 such fractions, plus the pain of one dust speck....

Once again, on setting 7,625,597,484,987, 3^^^3 persons are dust specked.

Will you still push the lever over?

Your (a): I was not talking about a universal, but of a personal scalar ordering. Somewhere inside everybody's brain there must be a mechanism that decides which of the considered options wins the competition for "most moral option of the moment".

That's a common utilitarian assumption/axiom, but I'm not sure it's true. I think for most people, analysis stops at "this action is not wrong," and potential actions are not ranked much beyond that. Thus, most people would not say that one is behaving immorally by volunteering at a soup kitchen, even if volunteering for MSF in Africa might be a more effective means of increasing the utility of other people. Your scalar ordering might work a bit better for the related, but distinct, concept of "praiseworthiness" -- but even there, I think people's intuitions are much too rough-hewn to admit of a stable scalar ordering.

To conceptualize that for you in a slightly different sense: we probably have far fewer brain states than the set of all possible actions we could hypothetically take in any given situation (once those possible actions are described in enough detail). Thus, it is simply wrong to say that we have ordered preferences over all of those possible actions -- in fact, it would be impossible to have a unique brain state correspond to all possibilities. And remember -- we are dealing here not with all possible brain states, but with all possible states of the portion of the brain which involves itself in ethical judgments.

Your (b): I view morality not as the set of rules postulated by creed X at time T but as the result of a genetically biased social learning process. Morality is expressed through it's influence on every (healthy) individual's personal utility function.

Intersting, but I think also incomplete. To see why: ask yourself whether it makes sense for someone to ask you, following G.E. Moore, the following question:

"Yes, I understand that X is a action that I am disposed to prefer/regard favorably/etc for reasons having to do with evolutionary imperatives. Nevertheless, is it right/proper/moral to do X?"

In other words, there may well be evolutionary imperatives that drive us to engage in infidelity, murder, and even rape. Does that make those actions necessarily moral? If not, your account fails to capture a significant amount of the meaning of moral language.

(8) ? [Sorry, I don't understand this one.]

Some component of ethical language is probably intended to serve prescriptive functions in social interactions. Thus, in some cases, when we say that "X is immoral" or "X is wrong" to someone proposing to engage in X, part of what we mean is simply "Do not do X." I put that one last because I think it is less important as a component of our understanding of ethical language -- typically, I think people don't actually mean (8), but rather, (8) is logically implied as a prudential corrolary of meanings 1-7.

To your voting scenario: I vote to torture the terrorist who proposes this choice to everyone. In other words, asking each one personally, "Would you rather be dust specked or have someone randomly tortured?" would be much like a terrorist demanding $1 per person (from the whole world), otherwise he will kill someone. In this case, of course, one would kill the terrorist.

So, the fact that an immoral person is forcing a choice upon you, means that there is no longer any moral significance to the choice? That makes no sense at all.

---
Unknown: Your example only has bite if you assume that moral preferences must be transitive across examples. I think you need to justify your argument that moral preferences must necessarily be immune to Dutch Books. I can see why it might be desireable for them to not be Dutch-Bookable; but not everything that is pleasant is true.

It has 7,625,597,484,987 settings. On setting 1, 1 person is tortured for 50 years plus the pain of one dust speck. On setting 2, 3 persons are tortured for 50 years minus the pain of (50-year torture/7,625,597,484,987), i.e. they are tortured for a minute fraction of a second less than 50 years, again plus the pain of one dust speck. On setting 3, 3^3 persons, i.e. 27 persons, are tortured for 50 years minus two such fractions of a second, plus the pain of one dust speck. On setting 4, 3^27, i.e. 7,625,597,484,987 persons are tortured for 50 years minus 3 such fractions, plus the pain of one dust speck....

Once again, on setting 7,625,597,484,987, 3^^^3 persons are dust specked.

Any particular reason why the lever scales like that? Given a setting s we have the torture time defined by T(s) = 50-0.0002(s-1) and the number of people being tortured defined by P(s) = 3^P(s-1) where P(1) = 1. I see no reason why the torture time should decrease linearly if the number of people being tortured increases super-exponentially.

Btw, I got the 0.0002 constant by finding the number number of seconds in 50 years and dividing by 7,625,597,484,987 (assuming 365 days per year). It's rounded. The actual number is around 0.00020678.

Ben:
"but a vote the other way potentially has 3^^^3 dust specks on your conscience - by your definition a much greater sin. Square one - shut up and vote!"

When presented with voting, each of the 3^^^3-1 people favored the dust specks (and their larger natural harm) to the torture (and its larger aggregated "mental distress"). The mental distress exists only on the basis of "sacred values". To say that in the face of 3^^^3-1 people preferring specks to torture, you should vote torture on the naive utility construction (no external effects of torture or specks) is paternalistic. If I know from reading the universe's configuration file that an apple is worth 10 utils and a pear is worth 9 utils, I should give the customer what he asks for, not an apple. Maybe an apple fell on him and the irrational fear of apples grants them -1.5 utils. The actors have found that harms outside the event described dictate their choices.

Ben P: the arrangement of the scale is meant to show that the further you move the lever toward 3^^^3 dust specks, the worse things get. The torture decreases linearly simply because there's no reason to decrease it by more; the number of people increases in the way that it does because of the nature of 3^^^3 (i.e. the number is large enough to allow for this). The more we can increase it at each stop, the more obvious it is that we shouldn't move the lever at all, but rather we should leave it at torturing 1 person 50 years.

The torture decreases linearly simply because there's no reason to decrease it by more; the number of people increases in the way that it does because of the nature of 3^^^3 (i.e. the number is large enough to allow for this)

I don't see how that follows. Even the progression from the first setting to the second setting seems arbitrary. You've established a progression from one scenario (torturing a person for 50 years) to another (3^^^3 dust specks) but to me it just seems like one possible progression. I see no reason to set up the intermediate stages like you have.

The more we can increase it at each stop, the more obvious it is that we shouldn't move the lever at all, but rather we should leave it at torturing 1 person 50 years.

That's only true up to a certain point. If I had to make a graph of the harm caused by the settings it would probably look like a parabola with what would look almost like an asymptote near setting 1.

My own anti-preference function seems to have a form something like this:

U(N,I,T) = kI(1 - e^(-NT/a))
where a and k are constants with appropriate units.

Relevant "intuitions" not listed before:
1) For the purposes of this thought experiment, who suffers a pain doesn't matter. Therefore:
1a) Transferring an instant of pain from one person to another, without changing the (subjective) intensity of the pain, doesn't change the "badness" of the situation. Two people suffering torture for 25 years simultaneously equals one person suffering 25 years of torture and then a second 25 years of torture. Therefore:
1b) U is a function of N*T and I; U(N,I,T) = Ū(NT,I) for some Ū.
2) Pains that are sufficiently more intense are qualitatively different from pains that are less intense, but insufficiently more intense pains are not. Therefore:
2a) For a sufficiently large difference in intensity, no amount of less intense pains are worse than a single more intense pain (of sufficient aggregated duration). Therefore:
2b) Ū(NT,I) approaches a finite limit as NT approaches infinity and I is held constant, because there must be a NT,I1,I2 such that Ū(NT,I1) > lim(NT->+∞) Ū(NT,I2) and I2 > 0.
2c) For insufficiently large differences in intensity, there is an amount of less intense pains that are worse than a more intense pain.

3) My mathematical intuition suggests to me that an equation of the form U(N,I,T) = kI(1 - e^(-NT/a)) has the properties I want.

There's a catch here. N and T are fairly well-defined, measurable terms. Intensity of pain doesn't have a well-defined scale, though; the term "I" has to be some function mapping subjective feelings (or, more precisely, brain states) to some subset of real numbers. Depending on how you do this mapping and how you choose the constant a, you can get many different preference orderings.

In the case of nearly infinite dust specks vs. 50 years of torture, the anti-preference function gives U(SPECKS) = kI(speck) and U(TORTURE)= kI(torture)(1-e^(-50 personyears/a)). Using the time-tested technique of "making stuff up", I assign I(speck) = 0.001 (as stipulated, one speck is a very small pain), I(torture)=1 (as torture is understood to be a very large pain), and a = 100 personyears (approximately, a human lifespan). This gives U(SPECKS) = 0.001 and U(TORTURE) = 0.393.

U(SPECKS) < U(TORTURE), and therefore, I prefer SPECKS. It's based on actual math! ;)

Naturally the T(s) function I posted earlier was wrong. It should have been T(s)=1576800000-0.0002(s-1). However, that doesn't change my question.

There is yet another angle on this dilemma which hasn't been raised yet. How bad is the outcome you are willing to prefer, in order to avoid those 3^^^3 dust specks? Are you willing to have the torture victim killed after the 50 years? How about all life on Earth? How about all life in the visible universe? I presume that truly convinced additivists will say yes in every case, because they "know" that 3^^^3 dust specks would still be incomprehensibly worse.

Actually, I see Eliezer raised that issue back here.

Notice that in Doug's function, suffering with intensity less than 0.393 can never add up to 50 years of torture, even when multiplied infinitely, while suffering of 0.394 will be worse than torture if it is sufficiently multiplied. So there is some number of 0.394 intensity pains such that no number of 0.393 intensity pains can ever be worse, despite the fact that these pains differ by 0.001, stipulated by Doug to be the pain of a dust speck. This is the conclusion that I pointed out follows with mathematical necessity from the position of those who prefer the specks.

Doug, do you actually accept this conclusion (about the 0.393 and 0.394 pains), or you just trying to show that the position is not logically impossible?

Yes, mitchell porter, of course there is no method (so far) (that we know of) for moral perception or moral action that does not rely on the human mind. But that does not refute my point, which again is as follows: most of the readers of these words seem to believe that the maximization of happiness or pleasure and the minimization of pain is the ultimate good. Now when you combine that belief with egalitarianism, which can be described as the belief that you yourself have no special moral value relative to any other human, and neither do kings or movie stars or Harvard graduates, you get a value system that is often called utilitarianism. Utilitarianism and egalitarianism have become central features of our moral culture over the last 400 years, and have exerted many beneficial effects. To give one brief example, they have done much to eliminate the waste of human potential that came from having a small groupand their descendants own everything. But the scientific and technological environment we now find ourselves in has become challenging enough that if we continue to use utilitarianism and egalitarianism to guide us, we will go badly astray. (I have believed this since 1992 when I read a very good book on the subject.) I consider utilitarianism particularly inadequate in planning for futures in which humans will no longer be the only ethical intelligences. I refer to those futures in which humans will share the planet and the solar system with AGIs.

You mentioned CEV, which is a complex topic, but I will briefly summarize my two main objections. The author of CEV says that one of his intentions is for everyone's opinion to have weight: he does not wish to disenfranchise anyone. Since most humans care mainly or only about happiness, I worry that that will lead to an intelligence explosion that is mostly or all about maximizing happiness and that that will interfere with my plans, which are to exert a beneficial effect on reality that persists indefinitely but has little to do in the long term with whether the humans were happy or sad. Second, there is much ambiguity in CEV that has to be resolved in the process of putting it into a computer program. In other words, everything that goes into a computer program has to be specified very precisely. The person who currently has the most influence on how the ambiguities will be resolved has a complex and not-easily summarized value system, but utilitarianism and "humanism", which for the sake of this comment will be defined as the idea that humankind is the measure of all things, obviously figure very prominently.

I will keep checking this thread for replies to my comment.

Unknown, I'll bite. While you do point out some extremely counterintuitive consequences of positing that harms aggregate to an asymptote, accepting the dust specks as being worse than the torture is also extremely counterintuitive to most people.

For the moment, I accept the asymptote position, including the mathematical necessity you've pointed out.

So far this discussion has focused on harm to persons. But there are other forms of utility and disutility. Here's the intuition pump I used on myself: the person concept is not so atomic to resist quantification--surely chimpanzees and dogs and fish and such must factor into humane utility calculations, even if they are not full persons. So are we then to prefer a universe with 3^^^3 banana slugs in it and no other life, over our own universe which contains (a much smaller number of) beings capable of greater feelings and thought? Absurd!

Perhaps in most realistic situations, the same experience happening to two different entities should count as almost exactly twice as good or bad as one instance of the experience. But I don't think we should extend that intuition to these extreme cases with numbers like 3^^^3, else we must consider it an improvement when (say) Eliezer's buggy AI decides to replace us with an incomprehensible number of slugs, each of which counts as one hundred-thousandth of a person.

At some point, the same experience repeated over and over again just doesn't count.

So there is some number of 0.394 intensity pains such that no number of 0.393 intensity pains can ever be worse, despite the fact that these pains differ by 0.001, stipulated by Doug to be the pain of a dust speck.

Let's see just what that number is...

0.394(1-e^(-NT/100 personyears) > 0.393
1-e(-NT/100 personyears) > 0.998

e^(-NT/100 personyears) < 0.002538
-NT/100 personyears < -5.976
NT > 597.6 person*years

In terms of the constants, it comes out to NT > -a*ln(1-I1/I2), where I1 is the lesser pain and I2 is the greater pain. This does strike me as somewhat undesirable; I would prefer that the required NT go to infinity when I1 and I2 are sufficiently close but not sufficiently far. Unfortunately, I can't do this and still be consistent; the limit can't depend on the difference between I1 and I2. I either have to accept a preference function in which all pains aggregate to the same limit, or there exits two pains arbitrarily close together such that a finite amount of one is worse than a Nearly Infinite amount of the other.

I'm not confident in my constants or in my ability to calculate I(brain state), but yes, I think I can "bite the bullet" on this one. I hereby declare that, for any two pains, if I1 > I2, then there is an amount of I1 pain that is worse than a Nearly Infinite amount of pain I2.

However, I believe that we live in a finite universe, so hopefully I don't have to deal with Nearly Infinite quantities of anything. ;) You'd best keep me well away from that button that destroys the world, because I find it very, very tempting.

Richard, my understanding is that CEV is not democracy, not by design anyway. Think of any individual human being as a combination of some species-universal traits and some contingent individual properties. CEV, I would think, is about taking the preference-relevant cognitive universals and extrapolating an ideal moral agent relative to those. The contingent idiosyncrasies or limitations of particular human beings should not be a factor.

At your website, you propose that "objective reality" is the locus of intrinsic value, sentient beings have only derivative value as a means whereby objective reality may be known, and that "The more a possible future turns on what you do or how you decide, the more you should focus on it at the expense of other possible futures". That last is the same as saying that you should seek power, but without saying what the power is for. I also see no explanation as to why knowledge of objective reality is of any value, even derivative; objective reality is there, and is what it is, regardless of whether it's known or not.

Z.M. Davis, that's an interesting point about the slugs, I might get to it later. However, I suspect it has little to do with the torture and dust specks.

Doug, here's another problem for your proposed function: according to your function, it doesn't matter whether a single person takes all the pain or if it is distributed, as long as it sums to the same amount according to your function.

So let's suppose that the pain of solitary confinement without anything interesting to do can never add up to the pain of 50 years torture. According to this, would you honestly choose to suffer the solitary confinement for 3^^^3 years, rather than the 50 years torture?

I suspect that most people would prefer to take the torture and get on with their lives, instead of suffering for the confinement for eternity.

But if you modify the function to allow for this, more preference reversals are coming: for we can begin to decrease the length of the solitary confinement by a microsecond while increasing the number of people who suffer it by a large amount.

In order to prevent an extremely short confinement for 3^^^3 people from exceeding the torture, which would presumably imply the same possibility for dust specks, you will have to say that there is some length of solitary confinement for some number of people, such that solitary confinement for Almost Infinite people, for a length of time shorter by the shortest possible noticeable time period, can never be worse.

Would you hold to this too, or would you honestly prefer the 3^^^3 years confinement to 50 years torture?

Utility doesn't aggregate. Neither human lives. You don't use 4, you have to use 1+1+1+1. If you aggregate human lives, you get diminishing marginal value for huma life/ Goverment does it. Millitary does it. You send a squad to suicide missoin to save the division. A la guerre com ala guerre. So I agree with Jadagul. Preference is a tricky subject , in which, there is always marginal utility.
But since you used economic term of utility here is a simple economic question upon aggregate utility:
You are the Goverment. You need to raise 1 Million $ for, let say, the new interchange.
You can get the money by 9 ways: increase taxation of 100M people by 0.01$, 10M people by 0.10$ .... 1 man by 1 million. Can you draw a histogram of disutility for the 9 cases?

Unknown, I think the slugs are relevant. I should think most of us would agree that all other things being equal, a world with less pain is better than one with more, and a world with more intelligent life is better than one with less.

Defenders of SPECKS argue that the quality of pain absolutely matters: that the pain of no amount of dust specks could add up to that of torture. To do this, they must accept the awkward position that the badness of an experience partially depends on how many other people have suffered it. Defenders of TORTURE say, "Shut up and multiply."

Defenders of HUMANS say that the quality of personhood absolutely matters: that the goodness of no amount of existing slugs could add up to that of existing humans. To do this, they must accept the awkward position that the goodness of an entity existing partially depends on what other kinds of entities exist. Hypothetical defenders of SLUGS say, "Shut up and multiply."

Aren't the situations similar?

Still haven't heard from even one proponent of TORTURE who would be willing to pick up the blowtorch themselves. Kind of casts doubt on the degree to which you really believe what you are asserting.

I mean, perhaps it is the case that although picking up the blowtorch is ethically obligatory, you are too squeamish to do what is required. But that should be overrideable by a strong enough ethical imperative. (I don't know if I would pick up the blowtorch to save the life of one stranger, for instance, but I would feel compelled to do it to save the population of New York City). So: that should be solveable, in your system, by using a bigger number of people than 3^^^3. Right? So make it a g64 (= graham's number,) of people getting dust-specked.

Will anyone on this board forthrightly assert that they would pick up the blowtorch to keep specks out of the eyes of g64 people? Not "I might do it," but "yes, I would do it," in the same sense where I can say with a lot of confidence that I would torture one individual if I was certain that doing so would save millions of lives.

And if you wouldn't, would you do it in the New York City example?

Eliezer?

About the slugs, there is nothing strange in asserting that the utility of the existence of something depends partly on what else exists. Consider chapters in a book: one chapter might be useless without the others, and one chapter repeated several times would actually add disutility.

So I agree that a world with human beings in it is better than one with only slugs: but this says nothing about the torture and dust specks.

Eisegetes, we had that discussion previously in regard to the difference between comparing actions and comparing outcomes. I am fairly sure I would not torture someone to save New York (at least not for 50 years), but this doesn't mean I think that the fact of someone being tortured, even for 50 years, outweighs the lives of everyone in New York. I might simply accept Paul's original statement on the matter, "Torture is wicked. Period."

It does matter how it is done, though. In my lever case, if the lever were set to cause the dust specks, I would definitely move it over to the 50 year torture side.

Another factor that no one has yet considered (to make things more realistic). If there were 3^^^3 people, googleplexes of them would certainly be tortured for 50 years (because the probability of someone being tortured for 50 years is certainly high enough to ensure this). So given an asymptote utility function (which I don't accept), it shouldn't matter if one more person is tortured for 50 years.

Unknown,

"So given an asymptote utility function (which I don't accept), it shouldn't matter if one more person is tortured for 50 years."

With such an asymptotic utility function your calculations will be dominated by the possible worlds in which there are few other beings.

I also see no explanation as to why knowledge of objective reality is of any value, even derivative; objective reality is there, and is what it is, regardless of whether it's known or not.

You and I can influence the future course of objective reality, or at least that is what I want you to assume. Why should you assume it, you ask? For the same reason you should assume that reality has a compact algorithmic description (an assumption we might call Occam's Razor): no one knows how to be rational without assuming it: in other words, it is an inductive bias necessary for effectiveness.

It is an open question which future courses are good and which are evil, but IMO neither the difficulty of the question nor the fact that no one so far has advanced a satisfactory answer for futures involving ultratechnologies and intelligence explosions -- neither of those two facts -- removes from you and I the obligation to search for an answer the best we can -- or to contribute in some way to the search. This contribution can take many forms. For example, many contribute by holding down a job in which they make lunches for other people to eat or a job in which they care for other people's elderly or disabled family members.

That last is the same as saying that you should seek power, but without saying what the power is for.

The power is for searching for a goal greater than ourselves and if the search succeeds, the power is for achieving the goal. The goal should follow from the fundamental principles of rationality and from correct knowledge of reality. I do not know what that goal is. I can only hope that someone will recognize the goal when they see it. I do not know what the goal is, but I can rule out paperclip maximization, and I am almost sure I can rule out saving each and every human life. That last goal is not IMO worthwhile enough for a power as large as the power that comes from an explosion of general intelligence. I believe that Eliezer should be free to apply his intelligence and his resources to a goal of his own choosing and that I have no valid moral claim on his resources, time or attention. My big worry is that even if my plans do not rely on his help or cooperation in any way, the intelligence explosion Eliezer plans to use to achieve his goal will prevent me from achieving my goal.

I like extended back-and-forth. Since extended back-and-forth is not common in blog comment sections, let me repeat my intention to continue to check back here. In fact, I will check back till further notice.

This comment section is now 74 hours old. Once a comment section has reached that age, I suggest that it is read mainly by people who have already read it and are checking back to look for replies to particular conversational threads.

I would ask the moderator to allow longer conversations and even longer individual comments once a comment section reaches a certain age.

Mitchell Porter, please consider the possibility that many if not most of the "preference-relevant human cognitive universals" you refer to are a hinderance rather than a help to agents who find themselves in an environment as different from the EEA as our environment is. It is my considered opinion that my main value to the universe derives from the ways my mind is different -- differences which I believe I acquired by undergoing experiences that would have been extremely rare in the EEA. (Actually, they would have been depressingly common: what would have been extremely rare is for an individual to survive them.) So, it does not exactly ease my fear that the really powerful optimizing process will cancel my efforts to affect the far future for you to reply that CEV will factor out the "contigent idiosyncracies . . . of particular human beings".

Unknown, it seems like what you are doing is making a distinction between a particular action being obligatory -- you do not feel like you "ought" to torture someone -- and its outcome being preferable -- you feel like it would be better, all other things being equal, if you did torture the person.

Is that correct? If it isn't, I have trouble seeing why the g64 variant of the problem wouldn't overcome your hesitation to torture. Or are you simply stating a deontological side-constraint -- I will never torture, period, not even to save the lives of my family or the whole human race?

In any event, what a lot of people mean when asked what they "should do" or what they "ought to do" is "what am I obligated to do?" I think this disambiguation helps, because it seems as if you are now making a distinction between TORTURE being morally required (which you do not seem to believe) and its being morally virtuous (which you do seem to believe).

Is that about right?

So let's suppose that the pain of solitary confinement without anything interesting to do can never add up to the pain of 50 years torture. According to this, would you honestly choose to suffer the solitary confinement for 3^^^3 years, rather than the 50 years torture?

You've already defined the answer; "the pain of solitary confinement without anything interesting to do can never add up to the pain of 50 years torture." If that's so, then shouldn't I say yes?

To some extent, my preferences do tell me to work on a "minimize the worst pain I will ever experience," so it doesn't seem that ridiculous to say that there is SOME amount of torture that even a Nearly Infinite duration of "something slightly less bad than torture" doesn't add up to.

Going back to the math, it seems as though at least one of the following must be true for a "reasonable", non-sadistic preference function:
1) For all I1 > 0, lim (NT->+∞) U(NT,I1) = lim (NT->+∞) U(NT, I2) for all I2 such that 0 < I2 < I1
2) There exists an I1 > 0 such that lim (NT->+∞) U(NT,I1) > lim (NT->+∞) U(NT, I1 - ε) for all ε such that 0 < ε < I1
3) There exists an I1 > 0 such that lim (NT->+∞) U(NT,I1) < lim (NT->+∞) U(NT, I1 + ε) for all ε > 0

In the first case, we have the case that, for any NT1, I1, and I2 such that I1 > I2 > 0, there exists a NT2 such that U(NT1,I1) < U(NT2,I2), no matter how small I2 is and how large NT1 and I1 are.
In the second and third cases, we have a situation in which there is an I1 and I2 such that I1 > I2 and are arbitrarily close together, and there is an NT1 such that U(NT1, I1) > lim (NT->+∞) U(NT, I2).

I have to bite the bullet on one of these problematic conclusions, so I'll bite the bullet on #2.

"An Impossibility Theorem for Welfarist Axiologies" is an interesting paper on a similar subject; given the choice of which of the criteria I have to reject, I choose to reject "The Minimal Non-Extreme Priority Principle".

I understand that choosing specks theoretically leads to an overall decrease in happiness in the universe. One (irrational, given my previous conclusion) thought, however, always seems to dominate my interior monologue about specks vrs. torture - if someone were to ask me whether or not I would take a dust speck in the eye to save someone from 50 years of torture, I would do it (as I would expect most people to). I realize that I would have to take 3^^^3 dust specks for the problem to match the original question (and I wouldn't be willing to get 3^^^3 dust specks in my eyes to stop 50 years of torture, as my pain would exceed that of the tortured), but my brain always goes back to thinking "...but I wouldn't mind getting a dust speck in my eye to keep someone from getting tortured..." I can't seem to be able to dismiss that thought as illogical, even though I know that it theoretically is. Substitute "dust speck" for "punch to the face" and I would still be willing, however, substitute one year of torture and I can see that the pain of 3^^^3 would outweigh the pain of one. Should I just force my brain to accept what I see as intuitively illogical but know is not?

Phil, a sufficiently altruistic person would accept 25 years of torture to spare someone else 50, but that doesn't mean it's better to torture 3^^^3 people for 25 years (even if they're all willing) than one person for 50 years.

If you call a utilitarian's utility function T, then you can pick the dust specks over torture if your utility function is -T.

I'm taking the discussion with Richard to email; if it issues in anything I suppose it will end up on his website.

Eisegetes (please excuse the delay):

That's a common utilitarian assumption/axiom, but I'm not sure it's true. I think for most people, analysis stops at "this action is not wrong," and potential actions are not ranked much beyond that. [...] Thus, it is simply wrong to say that we have ordered preferences over all of those possible actions -- in fact, it would be impossible to have a unique brain state correspond to all possibilities. And remember -- we are dealing here not with all possible brain states, but with all possible states of the portion of the brain which involves itself in ethical judgments.

I don't think so. Even if only a few (or even just one) option is actually entertained, a complete ranking of all of them is implicit in your brain. If I asked you if table salt was green, you'd surely answer it wasn't. Where in your brain did you store the information that table salt is not green?
I could make your brain's implicit ordering of moral options explicit with a simple algorithm:
1. Ask for the most moral option.
2. Exclude it from the set of options.
3. While options left, goto 1.

Intersting, but I think also incomplete. To see why: ask yourself whether it makes sense for someone to ask you, following G.E. Moore, the following question:
"Yes, I understand that X is a action that I am disposed to prefer/regard favorably/etc for reasons having to do with evolutionary imperatives. Nevertheless, is it right/proper/moral to do X?"
In other words, there may well be evolutionary imperatives that drive us to engage in infidelity, murder, and even rape. Does that make those actions necessarily moral? If not, your account fails to capture a significant amount of the meaning of moral language.

That's a confusion. I was explicitly talking of "moral" circuits. Not making a distinction between moral and amoral circuits makes moral a non-concept. (Maybe it is one, but that's also beside the point.) The question "is it moral to do X" just makes no sense without this distinction. (Btw. "right/proper" might just be different beasts than "moral".)

That's a confusion. I was explicitly talking of "moral" circuits.

Well, that presupposes that we have some ability to distinguish between moral circuits and other circuits. To do that, you need some other criteria for what morality consists in than evolutionary imperatives, b/c all brain connections are at least partially caused by evolution. Ask yourself: what decision procedure would I articulate to justify to Eisegetes that the circuits responsible for regulating blinking, for creating feelings of hunger, or giving rise to sexual desire are, or are not, "moral circuits."

In other words, you will always be faced with the problem of showing a particular brain circuit X, which you call a "moral circuit," and having someone say, "the behavior that circuit controls/compels/mediates is not something I would describe as moral." In order to justify your claim that there are moral circuits, or that specific circuits relate to morality, you need an exogenous conception of what morality is. Or else your definitions of morality will necessarily encompass a lot of brain circuitry that very few people would call "moral."

It's Euthyphro, all over again, but with brains.

I could make your brain's implicit ordering of moral options explicit with a simple algorithm:
1. Ask for the most moral option.
2. Exclude it from the set of options.
3. While options left, goto 1.

Well, I was trying to say that I don't think we have preferences that finely-grained. To wit:

Rank the following options in order of moral preference:

1. Kill one Ugandan child, at random.
2. Kill one South African child, at random.
3. Kill one Thai child. You have to torture him horribly for three days before he dies, but his death will make the lives of his siblings better.
3.5 Kill two Thai children, in order to get money with which to treat your sick spouse.
4. Rape and murder ten children, but also donate $500 million to a charity which fights AIDS in Africa.
5. Rape 500 children.
6. Sexually molest (short of rape) 2,000 children.
7. Rape 2000 women and men.
8. Rape 4000 convicted criminals.
9. Execute 40,000 convicted criminals per year in a system with a significant, but unknowable, error rate.
10. Start a war that may, or may not, make many millions of people safer, but will certainly cause at least 100,000 excess deaths.

The problem becomes that the devil is in the details. It would be very hard to determine, as between many of these examples, which is "better" or "worse", or which is "more moral" or "less moral." Even strict utilitarians would get into trouble, because they would experience such uncertainty trying to articulate the consequences of each scenario. Honestly, I think many people, if forced, could put them in some order, but they would view that order as very arbitrary, and not necessarily something that expressed any "truth" about morality. Pressed, they would be reluctant to defend it.

Hence, I said above that people are probably indifferent between many choices in terms of whether they are "more moral" or "less moral." They won't necessarily have a preference ordering between many options, viewing them as equivalently heinous or virtuous. This makes sense if you view "moral circuitry" as made up of gradated feelings of shame/disgust/approval/pleasure. Our brain states are quantized and finite, so there are certainly a finite number of "levels" of shame or disgust that I can experience. Thus, necessarily, many states of affairs in the world will trigger those responses to an identical degree. This is the biological basis for ethical equivalence -- if two different actions produce the same response from my ethical circuitry, how can I say meaningfully that I view one or the other as more or less "moral?"

To be sure, we can disagree on how many levels of response there are. I would tend to think the number of ethical responses we can have is quite small -- we can clearly say that murder is usually worse than rape, for instance. But we have great difficulty saying whether raping a 34 year old is better or worse than raping a 35 year old. You might think that enough reflection would produce a stable preference order between those states every time. But if we make the difference between their ages something on the order of a second, I don't see how you could seriously maintain that you experience a moral preference.

Eisegetes:
Well I (or you?) really maneuvered me into a tight spot here.
About those options, you made a goot point.
To the question "Which circuits are moral?", I kind of saw that one coming. If you allow me to mirror it: How do you know which decisions involve moral judgements?
I don't know of any satisfiying definition of morality. I probably must involve actions that are neither taylored for personal nor inclusive fitness. I suppose the best I can come up with is "A moral action is one which you choose (== that makes you feel good) without being likely to benefit your genes.". Morality is the effect of some adaption that's so flexible/plastic that it can be turned against itself. I admit that sounds rather like some kind of accident.
Maybe I should just give up and go back to being a moral nihilist again... there, now! See what you've made me believe! =)

"'A moral action is one which you choose (== that makes you feel good) without being likely to benefit your genes.'"

So using birth control is an inherently moral act? Overeating sweet and fatty foods to the point of damaging your health is an inherently moral act? Please. "Adaptation-executers," &c.

ZMD:
C'mon gimme a break, I said it's not satisfying!
I get your point, but I dare you to come up with a meaningful but unassailable one-line definition of morality yourself!
BTW birth control certainly IS moral, and overeating is just overdoing a beneficial adaption (i.e. eating).

To the question "Which circuits are moral?", I kind of saw that one coming. If you allow me to mirror it: How do you know which decisions involve moral judgements?

Well, I would ask whether the decision in question is one that people (including me) normally refer to as a moral decision. "Moral" is a category of meaning whose content we determine through social negotiations, produced by some combination of each person's inner shame/disgust/disapproval registers, and the views and attitudes expressed more generally throughout their society. (Those two sources of moral judgments have important interrelations, of course!) I tend to think that many decisions have a moral flavor, but certainly not all. Very few people would say that there is an ethical imperative to choose an english muffin instead of a bagel for breakfast, for instance.

"A moral action is one which you choose (== that makes you feel good) without being likely to benefit your genes."

Oh, I think a large subset of moral choices are moral precisely because they do benefit our genes -- we say that someone who is a good parent is moral, not immoral, despite the genetic advantages conferred by being a good parent. I think some common denominators are altruism (favoring tribe over self, with tribe defined at various scales), virtuous motives, prudence, and compassion. Note that these are all features that relate to our role as social animals -- you could say that morality is a conceptual outgrowth of survival strategies that rely on group action (and hence, become a way to avoid collective action problems and other examples of individual rationality that are suboptimal when viewed from the group's perspective).

Eisegetes:
"Moral" is a category of meaning whose content we determine through social negotiations, produced by some combination of each person's inner shame/disgust/disapproval registers, and the views and attitudes expressed more generally throughout their society.

From a practical POV, without any ambitions to look under the hood, we can just draw this "ordinary language defense line", as I'd call it. Where it gets interesting from an Evolutionary Psychology POV is exactly those "inner shame/disgust/disapproval registers". The part about "social negotiations" is just so much noise mixed into the underlying signal.
Unfortunately, as I believe we have shown, there is a circularity trap here: When we try to partition our biases into categories (e.g. "moral" and "amoral"), the partitioning depends on the definition, which depends on the partitioning, etc. etc. ad nauseam. I'll try a resolution further down.

Oh, I think a large subset of moral choices are moral precisely because they do benefit our genes -- we say that someone who is a good parent is moral, not immoral, despite the genetic advantages conferred by being a good parent.

Well, this is where I used to prod people with my personal definition. I'd say that good parenting is just Evolutionary Good Sense (TM), so there's no need to muddy the water by sticking the label "moral" to it. Ordinary language does, but I think it's noise (or rather, in this case, a systematic error; more below).

I think some common denominators are altruism (favoring tribe over self, with tribe defined at various scales), virtuous motives, prudence, and compassion. Note that these are all features that relate to our role as social animals -- you could say that morality is a conceptual outgrowth of survival strategies that rely on group action (and hence, become a way to avoid collective action problems and other examples of individual rationality that are suboptimal when viewed from the group's perspective).

I think the ordinary language definition of moral is useless for Evolutionary Psychology and must either be radically redefined in this context or dropped alltogether and replaced by something new (with the benefit of avoiding a mixup with the ordinary language sense of the word).
If we take for granted that we are the product of evolutionary processes fed by random variations, we can claim that (to a first approximation) everything about us is there because it furthers its own survival. Specifically, our genetic makeup is the way it is because it tends to produce successful survival machines.
1) Personal egoism exists because it is a useful and simple approximation of gene egoism.
2) For important instances of personal egoism going against gene egoism, we have built-in exceptions (e.g. altrusim towards own children and some other social adaptions).
3) But biasing behaviour using evolutionary adaption is slow. Therefore it would be useful to provide a survival machine with a mechanism that is able to override personal egoism using culturally transmitted bias. This proclaimed mechanism is at the core of my definition of morality (and, incidentally, a reasonable source of group selection effects).
4) Traditional definitions of morality are flawed because they confuse/conflate 2 and 3 and oppose them to 1. This duality is deeply mistaken, and must be rooted out if we are to make any headway in understanding ourselves.

Btw, the fun thing about 3 is that it does not only allow us to overcome personal egoism biases (1) but also inclusive fitness biases (2). So morality is exactly that thing that allows us to laugh in the face of our selfish genes and commit truly altrustic acts.
It is an adaption to override adaptions.

Regards, Frank

As noted above, 50 years of torture WITHOUT ANY CONSEQUENCES is a fucking useless, contradictory definition that's part of an overzealous effort to confuse intuition. If, say, the victim's mental state was carefully patched to what it once used to be, 5 years after the experience, so that the enormous utility tax of the experience would disappear, then it wouldn't be so contradictory, and Eliezer would still make his point (which I vaguely agree with, although this doesn't imply agreement with this particular decision).

Or, he could call it "purgatory", or "missing the world's greatest orgy due to lethargic sleep", or whatever. If torture is a loaded definition, you switch to a different definition to describe a different thing, not complain about LW's collective blindness.

Is the disagreement about 4 simply because of timeless decision theory etc?

Using a number big enough not to do the math is just a way of assigning 1 under any other name.

Among other things, if you try to violate "utilitarianism", you run into paradoxes, contradictions, circular preferences, and other things that aren't symptoms of moral wrongness so much as moral incoherence.

It seems to be an unsubstantiated slur on other moral systems :-(

I notice I'm confused here. The morality is a computation. And my computation, when given the TORTURE vs SPECKS problem as input, unambiguously computes SPECKS. If probed about reasons and justifications, it mentions things like "it's unfair to the tortured person", "specks are negligible", "the 3^^^3 people would prefer to get a SPECK than to let the person be tortured if I could ask them", etc.

There is an opposite voice in the mix, saying "but if you multiply, then...", but it is overwhelmingly weaker.

I assume, since we're both human, Eliezer's morality computation is not significantly different from mine. Yet, he says I should SHUT UP AND MULTIPLY. His computation gives the single utilitarian voice the majority vote. Isn't this a Paperclip Maximizer-like morality instead of a human morality?

I'm confused => something is probably wrong with my understanding here. Please help?

When lives are at stake, I shut up and multiply. It is more important that lives be saved, than that we conform to any particular ritual in saving them.

This is inconsistent. Why should you shut up and multiply in this specific case and not in others? Especially, when you (persusively) argued against "human life is of infinite worth" several paragraphs above?

What if the ritual matters, in terms of the morality computation?

For example: what if there's a man, accused of murder, of whose guilt we're 50% certain. If guilty and not executed, he'll probably (90%) kill three other random people. Should we execute him?

Gowder did not say what he meant by "utilitarianism". Does utilitarianism say...

1. That right actions are strictly determined by good consequences?
2. That praiseworthy actions depend on justifiable expectations of good consequences?
3. That probabilities of consequences should normatively be discounted by their probability, so that a 50% probability of something bad should weigh exactly half as much in our tradeoffs?
4. That virtuous actions always correspond to maximizing expected utility under some utility function?
5. That two harmful events are worse than one?
6. That two independent occurrences of a harm (not to the same person, not interacting with each other) are exactly twice as bad as one?
7. That for any two harms A and B, with A much worse than B, there exists some tiny probability such that gambling on this probability of A is preferable to a certainty of B?

Not Gowder, but another one for the list:
" Precedent Utilitarians believe that when a person compares possible actions in a specific situation, the comparative merit of each action is most accurately approximated by estimating the net probable gain in utility for all concerned from the consequences of the action, taking into account both the precedent set by the action, and the risk or uncertainty due to imperfect information. "
source

A link to Gowder's argument would be a good thing to have here. Never mind, I found it.

Some of what you're saying here makes me think that the post about Nature vs. Nature (that might not be the exact title but it was something similar) would be more relevant to his argument. He might be contending that you're trying to use intuitions which presume utilitarianism to justify utilitarianism, but you're ignoring other intuitions such as scope insensitivity. Scope insensitivity is only a problem if we presume utilitarianism correct. If we presume scope insensitivity correct then utilitarianism would become the problem.

So the dilemma is how we weigh competing intuitions against each other. There are definitely reasons that utilitarianism should win this fight, but since you don't identify a mechanism for weighing utilitarian intuitions against stupid human intuitions it's tough to say that this post does anything to address the hypothetical Gowder argument which Gowder may or may not have made.

Specifically, this part highlights the underlying conflict of intuitions well:

But that's for one event. When it comes to multiplying by quantities and probabilities, complication is to be avoided - at least if you care more about the destination than the journey. When you've reflected on enough intuitions, and corrected enough absurdities, you start to see a common denominator, a meta-principle at work, which one might phrase as "Shut up and multiply."

Where music is concerned, I care about the journey.

When lives are at stake, I shut up and multiply.

It's never explained why shutting up and multiplying should trump the value of the journey, or why that uniquely applies when life is at stake. The rules of logic don't go away whenever lives are in danger, so it feels very ad hoc without the identification of a specific weighing mechanism or process that determines when we should care about the journey and when we should care about multiplication.

To be clear, I like utilitarianism, but this post doesn't do much to support its intuitions over my deontic intuitions. Which intutions are the meta intuitions that we should use to weigh other intuitions against each other? Are these meta intuitions justified? These are questions that should be answered if you're talking about why utilitarian intuitions should be preferred to other intuitions.

Even if this isn't what Gowder argued, I'm still curious about how these questions would be answered by EY or by anyone else who wants to try to answer them. And I still would like a link to Gowder's argument, whatever it might be. Ignore that, sorry. Please just mentally eliminate all the references I made to Gowder. Thanks.

I believe that the vast majority of people in the dust speck thought experiment would be very willing to endure the collision of the dust speck, if only to play a small role in saving a man from 50 years of torture. I would choose the dust specks on the behalf of those hurt by the dust specks, as I can be very close to certain that most of them would consent to it.

A counterargument might be that, since 3^^^3 is such a vast number, the collective pain of the small fraction of people who would not consent to the dust speck still multiplies to be far larger than the pain that the man being tortured would endure. Thus, I would most likely be making a nonconsensual tradeoff in favor of pain. However, I do not value the comfort of those that would condemn a man to 50 years of torture in order to alleviate a moment's mild discomfort, so 100% of the people whose lack of pain I value would willingly trade it over.

If someone can sour that argument for my mind, I'll concede that I prefer the torture.

"It is more important that lives be saved, than that we conform to any particular ritual in saving them" is a major moral rule by itself, directly contradicted by, I believe, many if not most religions claiming to be sources of morality. "It does not matter that you saved more lives if you prayed to different gods/did not pray enough to ours" seems to be quite a repeating idea (also with gods replaced by political systems - advocates of Leninism tend to claim that capitalism is immoral despite having no Golodomor in its actual history).

Among other things, if you try to violate “utilitarianism”, you run into paradoxes, contradictions, circular preferences, and other things that aren’t symptoms of moral wrongness so much as moral incoherence.

Nobody seems to have problems with circular preferences in practice, probably because people's preferences aren't precise enough. So we don't have to adopt utilitarianism to fix this non-problem.

But you don’t conclude that there are actually two tiers of utility with lexical ordering. You don’t conclude that there is actually an infinitely sharp moral gradient, some atom that moves a Planck distance (in our continuous physical universe) and sends a utility from 0 to infinity. You don’t conclude that utilities must be expressed using hyper-real numbers. Because the lower tier would simply vanish in any equation. It would never be worth the tiniest effort to recalculate for it. All decisions would be determined by the upper tier, and all thought spent thinking about the upper tier only, if the upper tier genuinely had lexical priority

People aren't going to be doing ethical calculations using hyperrreal numbers, and they aren't going to be doing them with real numbers either -- both are beyond our cognitive limitations. Mathematically perfect but cognitively intractable ethics is angels-on-pinheads stuff.

Cognitive limitations means that ethics has to be based on rough heuristics. What would they look like? They would look like sacred values, taboos, and rules -- like ethics as it actually exists, not like utilitarianism.

And part of it has to do with preferring unconditional social rules to conditional social rules. Conditional rules seem weaker, seem more subject to manipulation. If there’s any loophole that lets the government legally commit torture, then the government will drive a truck through that loophole.

So it seems like there should be an unconditional social injunction against preferring money to life, and no “but” following it. Not even “but a thousand dollars isn’t worth a 0.0000000001% probability of saving a life”. Though the latter choice, of course, is revealed every time we sneeze without calling a doctor.

The rhetoric of sacredness gets bonus points for seeming to express an unlimited commitment, an unconditional refusal that signals trustworthiness and refusal to compromise. So you conclude that moral rhetoric espouses qualitative distinctions, because espousing a quantitative tradeoff would sound like you were plotting to defect.

On such occasions, people vigorously want to throw quantities out the window, and they get upset if you try to bring quantities back in, because quantities sound like conditions that would weaken the rule.

It is not difficult to steelman the usefulness of absolute prohibitions, eg against torture: they are a Schelling fence which prevents society sliding into a dystopia. So there is X amount of good consequences that stem from having taboos.

And there is Y amount of value that is lost by having them. Maybe you could torture the terrrorist and find out where the bomb is. (A much better example than the dust specs one, since it doesn't depend on the fantasy of pains aggregating).

So if you are a consequentialist -- there are excellent reasons for sticking with consequentialism even if you reject utilitarianism -- the crux is whether X>Y or Y>X. Saying nothing about X, as per the OP, doesn't even address the argument.