Hacking the CEV for Fun and Profit

It strikes me that there's a somewhat fuzzy continuum in both directions. The concept of a coherent identity is largely a factor of how aligned the interests of the component entities are. This ranges all the way from individual genes or DNA sequences, through cells and sub-agents in the brain, past the individual human and up through family, community, nation, company, religion, species and beyond.

Coalitions of entities with interests that are more aligned will tend to have a stronger sense of identity. Shifting incentives may lead to more or less alignment of interests and so change the boundaries where common identity is perceived. A given entity may form part of more than one overlapping coalition with a recognizable identity and shifting loyalties between coalitions are also significant.

Therefore, one could argue that the idea of an "individual" existing through time has no objective basis to begin with, and the decision to identify entities that exist in different instants of time as the same "individual" can't be other than a subjective whim.

Evolution may have reasons for making us think this, but how would you get that the identification of an individual existing through time is subjective? You can quite clearly recognize that there is a being of approximately the same composition and configuration in the same location from one moment to the next.

Even (and especially) with the Mach/Barbour view that time as a fundamental coordinate doesn't exist, you can still identify a persistent individual in that it is the only one with nearly-identical memories to another one at the nearest location in the (indistinguishable-particle based) configuration space. (Barbour calls this the "Machian distinguished simplifier" or "fundamental distance", and it matches our non-subjective measures of time.)

ETA: See Vladimir_M's response below; I had misread his comment, thereby criticizing a position he didn't take. I'll leave the above unchanged because of its discussion of fundamental distance as a related metric.

On the other hand, one could say that human brain can be described as a collection of interconnected subsystems, acting more or less coherently and coordinated by neural activity, that one perceive as stream of consciousness. Thus, stream of consciousness can be seen as a unifying tool, which allows to treat human brain activity as single agent operation. This point of view, while remaning reductionist-compatible, allows to reinforce perception of self as a real acting agent, thus, hopefully, reinforcing underlying neural coordination activity and making brain/oneself more effective.

I'll be convinced that personal identity is a subjective preference, if one can explain strange coincidence: only "tomorrow I" will have those few terabytes of my memories.

Therefore, one could argue that the idea of an "individual" existing through time has no objective basis to begin with, and the decision to identify entities that exist in different instants of time as the same "individual" can't be other than a subjective whim.

That's roughly my current view. Two minor points. I think "whim" may overstate the point. An instinct cooked into us by millions of years of evolution isn't what I'd call "whim". Also, "subjective" seems to presuppose the very subject whose reality is being questioned.

With regard to your own copies, it's more a matter of practicality than ethics. Before you make your first copy, while you're still uncertain about which copy you'll be, you should come up with detailed rules about how you want your instances to treat each other, then precommit to follow them. That way, none of your copies can ever force another to do anything without one of them breaking a precommitment.

More importantly, does it count as gay or masturbation if you have sex with your copy?

If it's just to use volitions of "people in general" as opposed to "people alive now", it might also turn out to be correct to ignore "people present now" altogether, including cryonauts, in constructing the future. So, this is not just an argument for cryonics not working, but also for people alive at the time in no sense contributing/surviving (in other words, everyone survives to an equal degree, irrespective of whether they opted in on cryonics).

I'm not clear what you are driving at. My point was: "Changes in number of individuals caused intentionally do not count". This applies to self-cloning, birth-control and murder alike.

I want other people to get what they want

Bob wants the AI to create as close an approximation to hell as possible, and throw you into it forever, because he is a fundamentalist christian.

Are you sure you want bob to get what he wants?

I happen to agree with the clause you quote because I think the divergence of a single person is so great as to swamp 6 billion people. I imagine that if one could contain that divergence, one would hardly worry about the problem of different people.

Today, people tend to spend more time and worry about the threat that other people pose than the threat that they themselves (in another mood, perhaps) pose.

This might weakly indicate that inter-person divergence is bigger than intra-person.

Looking from another angle, what internal conflicts are going to be persistent and serious within a person? It seems to me that I don't have massive trouble reconciling different moral intuitions, compared to the size and persistence of, say, the Israel-Palestine conflict, which is an inter-person conflict.

The difference between Eliezer's cognitivism and the irrealist stance of, e.g. Greene is just syntactic, they mean the same thing. That is, they mean that values are arbitrary products of chance events, rather than logically derivable truths.

I think that you're looking too deeply into this; what I'm trying to say is that accepting excuses of the form "I was tempted to do ~x before doing x, so clearly I have properties characteristic of someone who does ~x" is a slippery slope.

My point was that a preference utilitarian would let Dr Evil rule the world, in that scenario.

Although, obviously, if you're a preference utilitarian then that's what you actually want.

Does it rely on everyone's preferences or not? If it does then it has to interact with everybody.

CEV is more like figuring out an ethical theory, than it is about running around fighting fires, granting wishes, and so on. The latter part is the implementation of the ethical theory. That part - the implementation - has to be consultative or otherwise responsive to individual situations. But the first part, CEV per se - deciding on principles - is not going to require peering into the mind of every last human being, or even very many of them.

It is basically an exercise in applied neuroscience. We want to understand the cognitive basis of human rationality and decision-making, including ethical and metaethical thought, and introduce that into an AI. And it's going to be a fairly abstract thing. Although human beings love food, sex, and travel, there is no way that these are going to be axiomatic values for an AI, because we are capable of coming up with ideas about what amounts to good or bad treatment of organisms or entities with none of those interests. So even if our ethical AI looks at an individual human being and says, that person should be fed, it won't be because its theory of the world says "every sentient being must be given food" as an ethical first principle. The ultimate basis of such judgments is going to be something a whole lot more abstract which doesn't even refer to or presuppose human beings directly, but which, when applied to an entity like a human being, is capable of giving rise to such judgments.

(By the way, I don't mean that such reasoning from abstract beginnings will be the basis of real-time judgments. You don't go through life recomputing everything from first principles at every moment; as you discover important new implications of those principles, or effective new heuristics of action, you store them in memory and act directly on them later on. The ultimate basis of such complex decision-making in an AI would be pragmatically relevant only under certain circumstances, as when it was asked to justify a particular decision from first principles, or when it was faced with very novel situations, or when it was engaged in thought experiments.)

Although I have been insisting that the output of CEV - something like a theory of ethical action - must be independent, not only of what we all want from moment to moment, but even independent of many basic human realities (because it must have something to say about entities which don't have those qualities or needs), it cannot be entirely independent of human nature. We all agree that certain outcomes are bad. That agreement is coming from something in us; a rock, for example, would not agree, or disagree. So some part of human decision-making cognition is necessary for a Friendly AI to be recognized as such. The whole idea of CEV is about extracting that part of how we work, and copying it across to silicon.

This is where the applied neuroscience comes in. We cannot trust pure thought to get this right. Even pure thought combined with psychological experiment is probably not enough; we need to understand what the brain is doing when we make these judgments. At the same time, pure neuroscience is not enough either; it would just give us a neutral causal description of how the brain works; it wouldn't tell us how to normatively employ that information in making a Friendly AI. Thus, applied neuroscience. The human beings who set a CEV process in motion would need to avoid two things: they would have to avoid wrong apriori normatives, and they would have to avoid wrong implicit normatives. By an implicit normative, I mean something, some factor in their thinking and their practice, which isn't explicitly recognized as helping to determine the CEV outcome, but which is doing so anyway.

I'm saying a lot which isn't in the existing expositions of CEV (e.g. Yudkowsky, Mijic, Nesov), but it comes just from taking the philosophy and adding the fact that all this information about human meta-preferences is meant to come from the study of the brain. That's called neuroscience, and in principle even fallible human cognition, in the form of human neuroscience, may figure all this out before we ever have self-enhancing AIs. In other words, we may reach a point where the combination of psychology, philosophy, neuroscience really is telling us, this is what humans actually want, or want to want, etc. (Though it may be hard for non-experts to tell the difference between false premature claims of such knowledge, and the real thing.)

All that could happen even before there is a Singularity, and in that case the strategy for a Friendly outcome will be able to dispense with automating the deduction of neuroethical first principles, and concentrate on simply ensuring that the first transhuman AI operates according to those principles, rather than according to some utility function which, when pursued with superhuman intelligence, leads to disaster. But the CEV philosophy, and the idea of "reflective decision theory", is meant to offer a way forward, if we do figure out artificial intelligence before we figure out artificial ethics.

You are assuming that the programmer's personal desired reflect what is best for humans as whole. Relying on what humans think that is rather than a top-down approach will likely work better. Moreover, many people see an intrinsic value in some form of democratic approach. Thus, even if I could program a super-smart AI to push through my personal notion of "good" I wouldn't want to because I'd rather let collective decision making occur than impose my view on everyone.

This is aside from other issues like the fact that there likely won't be a single programmer for such an AI but rather a host of people working on it.

A lot of these issues are discussed in much more detail in the sequences and older posts. You might be downvoted less if you read more of those instead of rehashing issues that have been discussed previously. At least if you read those, you'll know what arguments have been made before and which have not been brought up. Many online communities one can easily jump into without reading much of their recommended reading. Unfortunately, that's not the case for Less Wrong.

This scenario actually gives us a guideline for aggregating utilities. We need to prevent Dr. Evil from counting more than once.

One proposal is to count people by different hours of experience, so that if I've had 300,000 hours of experience, and my clone has one hour that's different, it counts as 1/300,000 of a person. But if we go by hours of experience, we have the problem that with enough clones, Dr. Evil can amass enough hours to overwhelm Earth's current population (giving ten trillion clones each one unique hour of experience should do it).

So this indicates that we need to look at the utility functions. If two entities have the same utility function, they should be counted as the same entity, no matter what different experiences they have. This way, the only way Dr. Evil will be able to aggregate enough utility is to change the utility function of his clones, and then they won't all want to do something evil. Something like using a convergent series for the utility of any one goal might work: if Dr. Evil wants to destroy the world, his clone's desire to do so counts for 1/10 of that, and the next clone's desire counts for 1/100, so he can't accumulate more than 10/9 of his original utility weight.

You're right - I'm just a bit trigger-happy about futurism.

So, how do we decide who's evil?

The hack troubled me and I read part of the CEV document again.

The terms "Were more the people we wished we were" , "Extrapolated as we wish that extrapolated", and "Interpreted as we wish that interpreted" are present in the CEV document explaining extrapolation. These pretty much guarantee that a hack such like what Wei Dai mentioned would be an extremely potent one.

However, the conservatism in the rest of the document, with phrases like below seem to take care of it fairly well.

"It should be easier to counter coherence than to create coherence. " "The narrower the slice of the future that our CEV wants to actively steer humanity into, the more consensus required. " "the initial dynamic for CEV should be conservative about saying "yes", and listen carefully for "no". "

I just hope the actual numbers when entered match that. If they are, then I think the CEV might just come back with to the programmers saying "I see something weird. Kindly explain"

But he should still give sizable credence to that desire persisting.

* 2.1 Utilitarianism
* 2.2 Ethical egoism and altruism
* 2.3 Rule consequentialism
* 2.4 Motive consequentialism
* 2.5 Negative consequentialism
* 2.6 Teleological ethics

A small, select group will be designing and implementing the FAI anyways.

Yes, but a small, select group determining its moral values is a different thing entirely, and seems to defeat the whole purpose of CEV. At that point you might as well just have the small group of moral philosophers explicitly write out a "10 Commandments" style moral code and abandon CEV altogether.

Why should anyone choose aggregation of preference over a personal FAI, other than under explicit pressure?

But that doesn't explain why Eliezer is vehemently against any unequal weighting of volitions in CEV, such as the "geopolitical-power-weighted set of volitions" that Roko suggested might be necessary if major political powers got involved.

As far as I can tell, Eliezer's actual motivations for wanting to build CEV of humanity instead of a personal FAI are:

1. It's the fair thing to do (in some non-game-theoretic sense of fairness).
2. The CEV of humanity has a better chance to leading to a good outcome than his personal extrapolated volition. (See this comment.)

Personally I don't think these reasons are particularly good, and my current position is close to yours and Roko's. But the fact that Eliezer has stuck to his beliefs on this topic makes me wonder if we're missing something.

Obviously CEV isn't identical to preference utilitarianism, but CEV and preference utilitarianism have the following principles in common, which the hack exploits:

• Give people what they want, instead of what you think is good for them.
• If different people want different things, give each individual equal weight.

It seems clear that Eliezer got these ideas from preference utilitarianism, and they share some of the same flaws as a result.

People also complained about the "AI in a box boxes you post", which was a great post nearly identical in structure to this one. Few people read the open thread; good posts should not default to the open thread. Why is your criteria for top-level posts so arbitrarily difficult? We are not facing a problem of an influx of low quality content and the moderation+promotion system works well.

I think you would find that there is more coherence among the 99% of humans who are Dr. Evil than among the 1% of humans who are not.

Well, psychopaths don't share our moral terminal values, and I would still expect them to get shouted down. Dr Evil's clones outnumber us. I guess it comes down to how small a minority still holds human values, doesn't it?

That is the most convenient possible world from the perspective of making the grandparent's point. Was that your intention?

Well ... yeah. The least convenient world for avoiding the point.

Hey, I didn't name it. shrugs

The least convenient possible world would have all the copies united in favor of the original Dr. Evil.

That is less convenient.

The world that I infer is imagined in the original post (also fairly inconvenient to the grandparent) is one in which Dr. Evil wants the universe to be shaped according to his will rather than wanting to be the one who shapes the universe according to his will. This is a world in which having more copies of himself ensures that he does get his way more and so he has an incentive to do it.

Yup, that's what I meant by "the copies are united".