Human values differ as much as values can differ

This is a big problem with utilitarianism, period. Status is a problem. So are other-regarding values ("I want my neighbors to be Christian.") You're in deep trouble trying to reconcile differing values globally. (Which is why, for myself, I'm not messing with it; I'm starting to believe in the idea of cultivating our own gardens.)

That said, I think this is a distracting and inaccurate example. Status and stuff aren't the only things we value, and I don't see that they split by gender in the way you say. Personally, I don't want to be Elizabeth Taylor.

Your initial observation is interesting; your analysis is somewhat problematic.

First, there is, as mentioned elsewhere, a distinct misconception of status that seems to recur here. Status is not a one-dimensional line upon which each person can be objectively placed. Status is multi-dimensional. For example, your person with a second-rate Faberge egg collection may simply decide that what really counts is how nice someone's roses are, and thus look at better Faberge eggs with sympathy (they should have spent all this money on roses!) rather than jealousy. By competing on multiple dimensions, it's possible for many people to win, potentially even everyone.

You are also using a non-representative sample. The type of person who's likely to become successful through celebrity seems disproportionately likely to be status-obsessed. (This is likely also true of people who inherit fortunes, as they have few other ways to distinguish their self-worth.) Generalizing about the ability of an AI to serve all people based on these few may be inappropriate. This is even more true when we consider that a world with an FAI may look very different in terms of how people are raised and educated.

Furthermore, your restriction on an FAI as being unable to alter utility functions seems like overkill. Every ad you see on television is an attempt to change your utility function. They're just not terribly effective. Any alteration that an FAI made on advertising, education, perhaps even market pricing, would necessarily have some effect on people's utility functions which the FAI would presumably be aware of. Thus, while perhaps you want some restriction on the FAI rewiring people's brains directly, it seems like an FAI that is wholly prohibited from altering utility functions would be practically incapable of action. In short, therefore, your analysis has an overly restricted imagination for how an FAI would go about its business - you assume either it is directly rewiring everyone's brain or sitting on its own circuits unable to act. There should be a middle ground.

And, as an aside, your mention of "average utilitarians" betrays a common misconception that's getting old 'round these parts. Utility is inter-dependent. It's all well and good to talk about how, in theory, eliminating everyone but the two happiest people would maximize average utility, but, in reality, two people would not want to live like that. While it might work on a spreadsheet, actually eliminating most of the population is extraordinarily unlikely to maximize average utility, because utility is not merely a function of available resources.

Actually, there's a pretty easy fix for the status problem, because we actually do have independently-valuable feelings that the ability to boss people around correspond to -- feelings like respectability, importance, and pride being among the main ones.

(Often, however, people who are outwardly concerned with their position are actually motivated by perceived threats to safety or affiliation instead -- believing that they are unworthy or unlovable unless they're important enough, or that you're only safe if you're in charge, etc.)

Anyway, your analysis here (as with many others on LW) conflates feelings of status with some sort of actual position in some kind of dominance hierarchy. But this is a classification error. There are people who feel quite respectable, important, and proud, without needing to outwardly be "superior" in some fashion. One's estimate of one's respectability, prideworthiness, or importance in the scheme of things is not intrinsically linked to any value scheme other than one's own.

Given that the idea of CEV is to get to what the more-grown, wiser version of "you" would choose for your value scheme, it's pretty much a given that unsupportable schemes would fall by the wayside.

Truth is, if you're worried about your place in the dominance hierarchy (by which I mean you have feelings about it, not that you're merely curious or concerned with it for tactical or strategic reasons), that's prima facie evidence of something that needs immediate fixing, and without waiting for an AI to modify your brain or convince you of something. Identify and eliminate the irrational perceived threat from your belief system.

The problem of value aggregation has at least one obvious lower bound: divide the universe on equal parts, and have each part optimized to given person's preference, including game-theoretic trade between the parts to take into account preferences of each of the parts for the structure of the other parts. Even if values of each person have little in common, this would be a great improvement over status quo.

This discussion of "Friendly AI" is hopelessly anthropomorphic. It seems to be an attempt to imagine what a FAI optimizing the world to given person's values will do, and these are destined to fail, if you bring up specific details, which you do. A FAI is a system expected to do something good, not a specific good thing known in advance. You won't see it coming.

(More generally, see the Fun theory sequence.)

The closest thing I can think of as a solution for the status-FAI is domain-specific status. Let Fred be a high-status pianist, let Jim be a high-status computer engineer, let Sheila be a high-status chef, and let the status ordering shift with context.

But that does seem like a problem for FAI, given the appearance of these preferences.

Just one minor quibble: I think you should have put the explanation for why you chose the male/female examples ("The male/female distinction isn't rigid; it just helped organize the data in a way that made this distinction pop out for me") much earlier in the post. Since agreeing or disagreeing with your generalization has nothing to do with agreeing or disagreeing with your conclusion, you should say so before the beginning of the second section.

Human values differ a lot based on the surrounding. If you dump all humans into fairy tale land they might react very differently than now.

You seem to assume that the goal structure of a human is stable. But when i look around I see all kinds of manipulations happening. Religion && Politics being one, advertisement being another. An AI doesn't have to rewire brains the hard way. It could just buy a major entertainment company and implement the values it prefers humans to have into a soap opera, and then buy some ads at the super-bowl. Allowing the AI to change human values in the indirect way opens a big can of worms. Not allowing that does too. Ads for using seat-belts anyone? The more I learn about humans the more I doubt there is an inherent internal goal structure (besides the more general once like not killing humans of) Manipulations are way to easy to do.

It was good to be explicit that these are generalizations. Nonetheless, it was still a mistake to label these two views "female" and "male", rather than the more neutral "positional" and "non-positional". That you notice this correlation is interesting, but not the main point. Given the likely effects, it seems better not to over-emphasize this with your choice nomenclature.

I agree that friendly AI is probably doomed if the goal is to maximize human values. But what if we, as AI designers (I'm not an AI designer, but bear with me), don't care about what you've defined as "human values?" As I've alluded to before, what matters isn't the entire system Phil Goetz, but a specific subsystem. This is an important metaethical point. There are algorithms running in your brain that you have no control over, yet they do things that you simply don't want them to be doing. For example, I don't want to have to purchase fuzzies, but I do so that I may better purchase utilons.

This is one of the things Eliezer implicitly addresses when he says this:

I would say, by the way, that the huge blob of a computation is not just my present terminal values (which I don't really have - I am not a consistent expected utility maximizers); the huge blob of a computation includes the specification of those moral arguments, those justifications, that would sway me if I heard them. So that I can regard my present values, as an approximation to the ideal morality that I would have if I heard all the arguments, to whatever extent such an extrapolation is coherent.

Values must be determined by reflection. Our actions belie the values of the entire system that is our body (or the values of the various sub-systems battling for control of our body), not just that important subsystem we care about (determining the subsystem that we care about is an important open question, but considering all possible moral arguments seems to be a way of avoiding this problem entirely). So while actions are very useful for setting up a value system to predict behavior, they are much less useful in determining the terminal values that we want to/should act on.

With all that said, there still may be a problem with coherence of values among all of humanity - I don't necessarily disagree with you there. I just think you are shortchanging the metaethics.

Can the AI be Friendly if it creates human-comparable minds with abitrary nonhuman values, provided the resulting homunculi are not, themselves, significantly hostile or in any way self-improving?

If so, I think there's a simple fix for this: have the AI give everyone a volcano lair (or equivalent) and two dozen catgirl minions. That puts all humans solidly in the top 5% of humanoids, which is a nice status kick; they can still compete with other humans for the last few percentiles, but there's no real risk of ending up at the bottom of the heap... unless you're into that kind of thing, which is a whole other issue. Catgirls are set up with desires for pleasure and status comparable enough to humans that they can put up a good show of competing, but they never really outshine their master, disappointment or humiliation never permanently discourages them, and various other safety measures could be fully integrated into the design.

"You may argue that the extremely wealthy and famous don't represent the desires of ordinary humans. I say the opposite: Non-wealthy, non-famous people, being more constrained by need and by social convention, and having no hope of ever attaining their desires, don't represent, or even allow themselves to acknowledge, the actual desires of humans."

I have a huge problem with this statement. This is taking one subset of the population where you can measure what they value by their actions, and saying without evidence that they represent the general population whom you can't measure because resources limit the ability of their actions to reflect their values.

You are assuming that the experience of being rich or being famous doesn't change ones values.

I suspect that the value of reclusion for instance is a direct result of being so famous that one is hounded in public, and that a relatively unknown middle class male wouldn't place near as much value on it.

Well, the "female AI" can create a lot of human-seeming robots to play the low status roles, or something...

This is 'fictional evidence', but one way to satisfy positional values would be something like the Matrix, with suitably convincing NPCs for people to be 'better' than.

All this stuff about values is giving me a proverbial headache. When the Rapture of the Nerds happens, I'll take the easy way out and become a wirehead, and let the rest of the world actually do things. ;)

Do we need FAI that does as good a job of satisfying human desires as possible, or would an FAI which protects humanity against devastating threats enough?

Even devastating threats can be a little hard to define.... if people want to transform themselves into Something Very Different, is that the end of the human race, or just an extension to human history?

Still, most devastating threats (uFAI, asteroid strike) aren't such a hard challenge to identify.

The "headline" is probably inaccurate. I don't know what metric best measures similarity of value - but by the ones that popped into my head, the various human heavens mostly seem relatively clumped together - compared to what kind of future other creatures might dream of having. It's the close genetic relatedness that does it.

To achieve high status for the max amount of people you could invent more categories in which to achieve status. There is not just one best actress, but many in different types of acting. There can be the best person to make some specific dish. The best dancer of type X. Then you can seclude it by region any come out with many many more things, till everyone is famous who desires so. Alternatively the AI could find out if there is any desire beyond the wish for status and fulfill that instead, or offer a self modification for people that desire status, but desire to change that to something else.

I think a shorter term for what you're describing is positionality, the state where the quality of the good depends on its ranking relative to other goods. The problem you're claiming is that women's values are positional (making them non-mutually-satisfiable), while men's aren't (making them mutually-satisfiable).

But in any case, thanks for saving me the ordeal of telling women how the positionality of their values throws a big monkey wrench in everything ;-)

But, since the FAI's top-level goal is just to preserve human top-level goals, it would be pointless to make a lot of fuss making sure the FAI held its own top-level goals constant, if you're going to "correct" human goals first.)

Well, part of the sleight-of-hand here is that the FAI preserves the goals we would have if we were wiser, better people.

If changing top-level goals is allowed in this instance, or this top-level goal is considered "not really a top-level goal", I would become alarmed and demand an explanation of how a FAI distinguishes such pseudo-top-level-goals from real top-level goals.

Your alarm might not be relevant data, but an explanation might be possible from the FAI / the person who proved it Friendly.

Reality check: evolutionary theory suggests people's desires should be nailed down as hard as possible to those things that lead to raising good quality babies. Almost 7 billion humans shows how well this theory works.

So: men can be expected to desire status to the extent that it increases their access to young, fertile mates - while women can be expected to desire attention to the extent that it gives them access to a good selection of prospective partners and their genes.

The maternal instict is strong - and it has little to do with attention - and a lot to do with raising your very own cute, lovable baby.

Check with the fees paid to fertility clincs for evidence that many people want babies pretty directly - just as a naive interpretation of evolutionary theory suggests that they might.

People might not always say they want babies. However, the conscious mind is the human PR department, not an honest hotline to their motivation system. By their actions shall ye know them.

Phil, you make a good point here. However, note that when the people you are talking about are extrapolated in a CEV-like FAI, they will also understand this point. Elizabeth Taylor will understand that not everyone can have high status, and therefore people will have to settle on some solution, which could involve "fake people" as low status entities, editing of everyone's top-level goals (a sort of decision-theoretic compromise), etc.

One corollary of this is that for existing high status people, CEV would be a terrible thing. [EDIT: I was thinking "Compared to nonegalitarian AGI"; realistically that is the other option, not business as usual]

Since existing high-status people's cooperation may be required to build a CEV-like AI, a weighted-influence CEV might be built in stead, so that every non-positional goal can be optimized to the full extent for everyone, but for every positional goal, the existing ranking would be preserved.

For an average utilitarian, the best solution is probably to kill off everyone except one man and one woman. (Painlessly, of course.)

Imagine if someone asked me how to avoid getting speeding tickets and I told them "Look at the speedometer and make sure it doesn't go over the posted speed limit." Imagine they then modify their speedometer so that they can manually adjust what it shows. They then drive 120mph in a 25mph zone, with their speedometer set to not go over 25mph, and promptly get a ticket.

I suppose you could argue that it's my fault for not speaking more clearly. But really, they are a fellow human with mirror neurons, and should be able to figure out what I was really saying, even if I abbreviated it for brevity. Obviously what I really meant was "Don't make your car go over the speed limit, and use your speedometer to determine if you have done this."

Any criticism of average utilitarianism that claims that it advocates killing unsatisfied people sounds like rigging the speedometer to me. Obviously what an average utilitarian means is "Work to increase people's utility, and measure this by looking at how high average utility is." Drawing bizarre implications like "kill unsatisfied people" from that is obviously an example of confusing the measurement method for the actual goal.

In general I dislike criticisms of ethical theories that go "if I follow the explicitly stated injunctions of this theory excessively literally like a robot from a bad scifi movie then I will end up doing something insane." That might be a good argument against programming a powerful AI with such a theory, or at least expanding on the theory a lot more before you program it in. But humans' should really know better.

FYI, I am not an average utilitarian. I am actually more of a totaverage utiltarian, I believe that both total and average utility are both good and that both deserve resources devoted to increasing them. But the "kill depressed people" argument against average utilitarianism has always seemed asinine to me.

Also, in case you didn't see it, in the comments on one of your more recent posts I have argued strongly against one of the major points of this post, that positional goals are fundamentally different from mutually satisfiable goals, making reference to an argument by David Friedman that one can satisfy (or at least satisfice) everyone's demands for status by making sure they are members of multiple social groups and have high status in some of them.