Humans are utility monsters

Well, whether you really want (in the extrapolated volition sense) to sacrifice 10^{whatever} lives to save your family is a whole big calculation involving interpersonal morality, bounded rationality/virtue ethics, TDT/game theory, etc. The point that I was echoing is that if you really would want to make that trade, there's nothing monstery about your family - you just {love them that much}/{love others that little}. The utility monster is an objection to the social morality theory called "utilitarianism"; the utility monster becomes gibberish when phrased as an objection to "any set of preferences can in principal be completely specified by a utility function, to be handed to a generic decision process, resulting in optimal decision making". Like, "Oh no, oh no, I found this monster, and it is soooo soooo good to feed it humans! It is even more better every time I feed it another human! Woe is me! Goooood!!".

Now, the utility monster makes perfect sense as an objection to humans actually making decisions purely using explicit quantitative expected utility calculations. But that doesn't say anything about utility as a formalized version of "good". Rather, that's some sort of comment about the capricious quality of bounded reasoning under uncertainty - you always worry about strong conclusions that make you do particularly effective things, because a mistake in your calculations means you are doing particularly effective bad things. One particular sort of dangerously strong conclusion would be concluding that, e.g., the marginal utility of {UMonster eating an additional human} is larger than and grows faster than the marginal utility of {another humans gets eaten alive}.

Our oldest utility monster is eight years old. (Did you have this example specifically in mind? Seems to fit the description very well.)

If you prefer a happy monster to no monster and no monster to a sad monster, then you prefer a happy monster to a sad monster, and TsviBT's point applies.

Whereas if you prefer no monster to a happy monster to a sad monster, why don't you kill the monster?

I don't get it, how is that different from any old bad thing you want to avoid?

If you're unsure of a question of philosophy, the Stanford Encyclopedia of Philosophy is usually the best place to consult first. Its history of utilitarianism article says that

Though there are many varieties of the view discussed, utilitarianism is generally held to be the view that the morally right action is the action that produces the most good. There are many ways to spell out this general claim. One thing to note is that the theory is a form of consequentialism: the right action is understood entirely in terms of consequences produced. What distinguishes utilitarianism from egoism has to do with the scope of the relevant consequences. On the utilitarian view one ought to maximize the overall good — that is, consider the good of others as well as one's own good.

The Classical Utilitarians, Jeremy Bentham and John Stuart Mill, identified the good with pleasure, so, like Epicurus, were hedonists about value. They also held that we ought to maximize the good, that is, bring about ‘the greatest amount of good for the greatest number’.

Utilitarianism is also distinguished by impartiality and agent-neutrality. Everyone's happiness counts the same. When one maximizes the good, it is the good impartially considered. My good counts for no more than anyone else's good. Further, the reason I have to promote the overall good is the same reason anyone else has to so promote the good. It is not peculiar to me.

Note the last paragraph in particular. Utilitarianism is agent-neutral: while it does take your utility function into account, it gives it no more weight than anybody else's.

The "general utilitarianism" that you mention is mostly just "having a utility function", not "utilitarianism" - utility functions might in principle be used to implement ethical theories quite different from utilitarianism. This is a somewhat common confusion on LW (one which I've been guilty of myself, at times). I think it has to do with the Sequences sometimes conflating the two.

EDIT: Also, in SEP's Consequentialism article:

Since classic utilitarianism reduces all morally relevant factors (Kagan 1998, 17–22) to consequences, it might appear simple. However, classic utilitarianism is actually a complex combination of many distinct claims, including the following claims about the moral rightness of acts:

Consequentialism = whether an act is morally right depends only on consequences (as opposed to the circumstances or the intrinsic nature of the act or anything that happens before the act).

Actual Consequentialism = whether an act is morally right depends only on the actual consequences (as opposed to foreseen, foreseeable, intended, or likely consequences).

Direct Consequentialism = whether an act is morally right depends only on the consequences of that act itself (as opposed to the consequences of the agent's motive, of a rule or practice that covers other acts of the same kind, and so on).

Evaluative Consequentialism = moral rightness depends only on the value of the consequences (as opposed to non-evaluative features of the consequences).

Hedonism = the value of the consequences depends only on the pleasures and pains in the consequences (as opposed to other goods, such as freedom, knowledge, life, and so on).

Maximizing Consequentialism = moral rightness depends only on which consequences are best (as opposed to merely satisfactory or an improvement over the status quo).

Aggregative Consequentialism = which consequences are best is some function of the values of parts of those consequences (as opposed to rankings of whole worlds or sets of consequences).

Total Consequentialism = moral rightness depends only on the total net good in the consequences (as opposed to the average net good per person).

Universal Consequentialism = moral rightness depends on the consequences for all people or sentient beings (as opposed to only the individual agent, members of the individual's society, present people, or any other limited group).

Equal Consideration = in determining moral rightness, benefits to one person matter just as much as similar benefits to any other person (= all who count count equally).

Agent-neutrality = whether some consequences are better than others does not depend on whether the consequences are evaluated from the perspective of the agent (as opposed to an observer).

PhilGoetz says otherwise. I believe that he is right, he's an expert in the subject matter. I am surprised and confused.

PhilGoetz is correct, but your confusion is justified; it's bad terminology. Consequentialism is the word for what you thought utilitarianism meant.

Thanks, but I tend to define and use my own terminology, because the standard terms are too muddled to use. I am an expert in my own terminology. Leon is talking about utilitarianism as the word is usually, or at least historically, used outside LessWrong, as a computation that everyone can perform and get the same answer, so society can agree on an action.

"Utilitarianism" is sometimes used for both that "variant" (valuing utility) and the meaning you ascribe to it (defining "value" in terms of utility.) The Utility Monster is designed to interfere with the former meaning. Which is the correct meaning ...

"Utility" refers a representation of preference over goods and services in economics and decision theory. This usage dates to the late 1940s. It has almost nothing at all to do with the normative theory of utilitarianism which dates to the late 1780s.

As a normative theory is supposed to tell you how you ought to act saying "oh everyone ought to follow their own utility function" is completely without content. The entire content of the theory is that my utils and your utils are actually the same kind of thing such that we can combine them one-to-one in a calculation to determine how to act (we want to maximize total utils).

That surely doesn't mean utilitarianism is useless - people can still make decisions and trade utilons!

This isn't utilitarianism. It is ethical egoism as described by economists.

I meant the former case, what use are people who's wants don't perfectly align with their utility function? xJ I guess whenever the latter case occurs in my life, that's not really what's happening. The dog thinks it's driving away a threat I don't recognise, when really it's driving away an opportunity it's incapable of recognising. Sometimes it might even be the right thing for them to do, even by my standards, given a lack of information. I still have to manage them like a burdensome dog.

The definition of want here is ambiguous, and that makes this is a little hard to parse. How are you defining "want" with respect to "utility function"? Do you mean to make them equivalent?

If by "want" you mean desire in accord with their appropriately calibrated utility functions, then, well, sure. A friend is selfish by any common understanding if he doesn't care about his buddies' needs.

Assuming that the utility monster is not, somehow, mistaken regarding it's wants...

Um, what about the actual slaves and ex-slaves?

This makes the claim unfalsifiable. People who work closely with animals are the greatest believers in animal rights? Obviously animals should have rights, since they're the ones who know the best. People who work closely with animals believe in animal rights the least? Obviously animals should have rights, since people who work closely with animals are rationalizing it away like slaveholders and the people with the least contact with animals are the most objective. No matter what happens, that "proves" that the people who talk about animal rights are the ones we should listen to.

Unfortunately, there is no possible answer to this.

Of course there is. Not all statements in religious holy books require the same amount of interpretation. If the various holy books said "thou shalt not be cruel to animals" using fairly direct language, that would be an answer to that. Problem is, they don't.

If this were true, tribes would be in constant total war

That doesn't follow. I grant zero weight to the well-being of clothes, but that doesn't mean I go around destroying my clothes and setting department stores on fire. Granting zero weight to something doesn't imply wanting to destroy it, and even granting negative weight to it only means wanting to destroy it insofar as destroying it doesn't make something else worse that you do care about (such as risking death to your own tribesmen in the war.)

Also, I wonder how many of the cultures who pray to the spirit of the animal also pray to the spirit of plants, rocks, the sun, or other things that even vegetarians don't think have any rights.

They kill out of self-interest, and out of revenge, but it's not constant and it's not something they feel awesome about.

Most Native American cultures felt awesome about killing enemies in battle. I don't know if it's universal, but it was very common for warriors to be highly-respected in tribal cultures, in proportion to how many people they'd killed.

I don't think you can assert that it's not constant, either. Look at the conflict between Hopi & Navajo, Cree & Blackfoot. Similar to the Palestinian/Israeli conflict, and I'd call that constant.

Modern all-out, extended-duration war is a foreign concept to such groups, but "this tribe is our enemy and we will kill any of them found unprotected" and "let us all get together and annihilate this troublesome neighbor village and take their women" is not.

You are right, of course. I did not want to imply that a vegan diet would have been feasible until recent advances.

'one' in that case refers to an agent who's trying to value feelings that physical systems have.

I see what you're doing, then. I'm thinking of a real-life limited agent like me, who has little idea how the inside of a nematode or human works. I have a model of each, and I make a guess at how to weigh them in my utility function based on observations of them. You're thinking of an ideal agent that has a universal utility function that applies to arbitrary reality.

Still, though, the function is at least as likely to start its evaluation top-down (partitioning the world into objects) as bottom-up.

I don't understand your overall point. It sounds to me like you're taking a long way around to agreeing with me, yet phrasing it as if you disagreed.

if only there were some way to settle the bet.

I don't think, in general, there could be a way to compare 'strength of feeling', etc. across two separate systems. For example, all you can do is measure the behavior of the organism, but that organism is always going to do the maximum that it can do to maximize its utility function. All you would be doing is measuring the organism's resources for optimizing its utility function, and determining the strength of its preference for any one thing relative to its other preferences only.

Come, now, it's hardly untestable. You can pay him if the FAI kills everyone to tile the universe with nematodes.

I suspect we'd have to know a lot more about neuroscience and consciousness to define "feel more strongly" precisely enough for the question to have an answer. I also suspect that, if the answer doesn't come out the way we want it to, we'll substitute another question in its place that does, in the time-honored practice of claiming that universal, objective agenthood is defined by whatever scale humans win on.

... which would imply that the reality is not anthropomorphic but empathising with it is a good thing.

There are many indigenous cultures (with some hunters still around today) who ask forgiveness upon killing food animals. And history's full of bear cults, and animal species with names that translate into "people of the _," and taboos on harming various animals. I think the notion that humans have mostly only cared for the concerns of humans is the product of an industrial-age blind spot: only people who've never hunted or husbanded, and eat their meat from the slaughterhouse, have never thought about animal welfare.

Ancestor worshippers- who are the people whose opinions we're discussing- would disagree. Wikipedia:

Veneration of the dead or ancestor reverence is based on the belief that the dead have a continued existence...the goal of ancestor veneration is to ensure the ancestors' continued well-being

Therefore, by substitution, we don't experience anything in response to knowledge about things that will happen after we're dead?

What if it can't be easily killed?

That doesn't work for preference utilitarians (it would strongly prefer to remain alive).

The "utility monster" has ceased to be a utility monster because it no longer gets everything. It still gets more, of course, but that's the equivalent of deciding that the starving person gets the food before the full person.

This sounds like it could be almost as repugnant as a utility monster that gets literally everything, depending on precisely how much "more" we're talking about.

Edit: if I were the kind of person who found utility monsters repugnant, that is. I'd already dissolved the "OMG what if utility monsters??" problem in my own mind by reasoning that the repugnant feeling comes from representing utility monsters as black boxes, stripping away all of the features of theirs that make it intuitively obvious why they generate more utility from the same inputs. Put another way, the things that make real-life utility monsters "utility monsters" are exactly the things that make us fail to recognize them as utility monsters. When a parent values their child's continued existence far more than their own, we don't call the child a "utility monster" if the parent sacrifices themselves to save their child, even though that's exactly the child's role in that situation.

The "utility monster" has ceased to be a utility monster because it no longer gets everything.

Can this be resolved by adding more monsters? I.e., instead of having just one utility monster on Earth, we could have a million or even 6 billion monsters (as many as there are humans). This would allow the monsters to fully benefit from consuming "everything" or at least close enough to "everything" to raise the dilemma.

I want to criticise either the idea that diminishing returns is important, or, at least, that dollar values make sense for talking about them.

Suppose we have a monster who likes to eat. Each serving of food is just as tasty as the previous, but he still gets diminishing returns on the dollar, because the marginal cost of the servings goes up.

We also have nematodes, who like to eat, but not as much. They never get a look in, because as the monster eats, they also suffer diminished utilons per dollar.

So the monster is serving the 'purpose' of the utility monster, but still has diminishing returns on the dollar. If we redefine diminishing returns to be on something else, I'm not sure it could be well justified or immune to this issue.

And, although humans are not an example of this sort of monster, the human race certainly is.

They surely are fungible. The whole point of using utility functions in the first place, is that I can't convert apples into children saved, but I can convert utilons gained from eating apples into utilons gained from saving children, because both are just real numbers.

This is a particular instance of the general approach, "I have to assign a number to each of these items, but it's hard and contentious to do, so instead I will give them all zeroes (objects) or ones (agents)." It always increases the total error. The world is not divided into agents and objects, and this approach would still increase total error, or at best leave it unchanged, even if they were, since errors in classification give a larger total error when they are thresholded instead of just left as, say, probabilities.

You should also consider that, when AI is developed, you will become an "object".

This approach doesn't work well even for humans. Very intelligent humans, armed with and experienced with mathematics, large computerized databases, regression analysis, probability and statistics, information theory, dimension reduction, data mining, machine learning, stability analysis, optimization techniques, and a good background in cog sci, biology, & physics, think more differently from average humans around 1600 AD, than average humans in 1600 AD did from dogs. So where do you draw the line?

That is it an implementation of equality among unequal agents. Why is an average day of Agent Alpha the same value as an average day of Agent Beta, and how does Agent Beta determine how much utility Agent Alpha gains from something other than the reference economy?

If we allow the agents to determine their own utility derived from, say, fiat currency, we have instead of a utility economy a financial economy. Everyone gains instrumental value from participating (or they stop participating). Allow precommitment and assume rational, well-informed agents, and the economic system maximizes each individual utility within the possible parameters.