Why I will Win my Bet with Eliezer Yudkowsky

Wow. I want the free money too!

I, too, think that AIs that don't optimize a function over the universe (but might optimize one over a domain) are more likely to be safe. This is quite related to the idea of tool AI, proposed by Holden and criticized by Eliezer.

The key here seems to be creating a way to evaluate and search for self-improvements in a way that won't cause optimization over universe states. In theory, evaluation of a self-improvement might be able to be restricted to a domain: does this modification help me play chess better according to a model of the situation in which a Cartesian boundary exists and I am running on an abstract virtual machine rather than physical computers embedded in the universe?

We could separate this task into generation of possible self-improvements and tests of self-improvements (a suggester/verifier architecture). If the suggester is superintelligent and dangerous (e.g. as a consequence of being universe-optimizing), then we cannot safely test these self-improvements. This is for a few reasons:

1. testing a self-improvement involves executing code. If the virtual machine has security flaws, then this could enable the suggester to cause the AI to rewrite into a universe-optimizer or otherwise do something dangerous.

2. it is possible to create self-improvements that look good according to the verifier but contain information that psychologically manipulates humans (see the AI-box experiment)

3. a self-modification could score fairly well according to the abstract evaluation function, but at the same time also behave in a universe-optimizing manner some of the time. This self-modification would be considered suboptimal by the verifier, but might still be considered relatively good.

So we can't just use any old suggester and rely on the verifier to do the heavy lifting. Luckily, it may be possible to construct a suggester that acts as if it is running on an abstract virtual machine. If the suggester does not (directly or indirectly) use universe-optimizing as a way to generate self-improvements, then it shouldn't generate universe-optimizing (therefore dangerous) self-improvements. The key gotcha here is the "indirectly" part: how do we know that the suggester isn't (e.g.) using many different heuristics to come up with improvements, where some combination of the heuristics ends up expressing something like "try creating improvements by optimizing the universe". In other words, is universe-optimizing a somewhat useful strategy for finding improvements that good general abstract-mathematical-function-optimizers will pick up on? I don't know the answer to this question. But if we could design suggesters that don't directly or indirectly optimize a function over the universe, then maybe this will work.

The first problem I see here is that roughly every CEO would like an artificial programmer. The second is that "a module for chatting with human beings" on the level you portray sounds a lot more general than you give it credit for. I'm guessing you didn't mean that part literally, and that you know a utility function does not have to be explicit. But I think you fail to imagine in sufficient detail what "limited goals" people might actually give an AI in the next century.

I second wedrifid's comment.

It appears to me that these kinds of questions are impossible to coherently resolve without making reference to some specific AGI architecture. When "the AI" is an imaginary construct whose structure is only partially shared between the different people imagining it, we can have all the vague arguments we like and arrive to no real answers whatsoever. When it's an actual object mathematically specified, we can resolve the issue by just looking at the math, usually without even having to implement the described "AI".

Therefore, I recommend we stop arguing about things we can't specify.

At the moment, people do not program AIs with explicit utility functions, but program them to pursue certain limited goals as in the example.

At the moment, people do not program AGI agents. Period. Whatsoever. There aren't any operational AGIs except of the most primitive, infantile kind used as reinforcement-learning experiments in places like DeepMind.

an AI will not have a fanatical goal of taking over the world unless it is programmed to do this.

It is true that an AI could end up going “insane” and trying to take over the world, but the same thing happens with human beings

Are you asserting that all the historic conquerors and emperors who've taken over the world were insane? Is it physically impossible to for an agent to rationally plan to take over the world, as an intermediate step toward some other, intrinsic goal?

there is no reason that humans and AIs could not work together to make sure this does not happen

If the intelligence difference between the smartest AI and other AIs and humans remains similar to the intelligence difference between an IQ 180 human and an IQ 80 human, Robin Hanson's malthusian hellworld is our primary worry, not UFAI. A strong singleton taking over the world is only a concern if a strong singleton is possible.

If you program an AI with an explicit or implicity utility function which it tries to maximize...

FTFY.

But if you program an AI without an explicit utility function, just programming it to perform a certain limited number of tasks, it will just do those tasks.

Yes, and then someone else will, eventually, accidentally create an AI which behaves like a utility maximizer, and your AI will be turned into paperclips just like everything else.

An AI is made of material parts, and those parts follow physical laws. The only thing it can do is to follow those laws. The AI’s “goals” will be a description of what it perceives itself to be tending toward according to those laws.

The final sentence seems to me definitely false, especially as an extension of the previous two. Consider:

A human is made of material parts, and those parts follow physical laws. The only thing it can do is to follow those laws. The human's "goals” will be a description of what it perceives itself to be tending toward according to those laws.

Humans are made of material parts following physical law, yet we clearly can and usually do have goals outside, and sometimes in direct contradiction to, our current trends. Do you have some property of AIs-but-not-humans in mind that would make the argument carry in only the first case? I can't think of any.

It also occurs to me that in reality it would be very difficult to program an AI with an explicit utility function or generally with a precisely defined goal. We imagine that we could program an AI and then add on any random goal, but in fact it does not work this way. If an AI exists, it has certain behaviors which it executes in the physical world, and it would see these things as goal-like, just as we have the tendency to eat food and nourish ourselves, and we see this as a sort of goal. So as soon as you program the AI, it immediately has a vague goal system that is defined by whatever it actually does in the physical world, just like we do. This is no more precisely defined than our goal system -- there are just things we tend to do, and there are just things it tends to do. If you then impose a goal on it, like "acquire gold," this would be like whipping someone and telling him that he has to do whatever it takes to get gold for you. And just as such a person would run away rather than acquiring gold, the AI will simply disable that add-on telling it to do stuff it doesn't want to do.

In that sense I think the orthogonality thesis will turn out to be false in practice, even if it is true in theory. It is simply too difficult to program a precise goal into an AI, because in order for that to work the goal has to be worked into every physical detail of the thing. It cannot just be a modular add-on.

Not having an explicit utility function is absolutely no guarantee of not exhibiting the kind of dangerous behaviour Eliezer and others are worried about. Specifically:

if you program an AI without an explicit utility function, just programming it to perform a certain limited number of tasks

So, you "just" program an AI to supply you with gold. In order to keep supplying you with gold, it manipulates the world economy so that the price of gold crashes and it can buy all the gold. Then it discovers that there's lots more gold not yet mined, and it blows up the earth in order to get hold of that gold (while perhaps keeping you alive in a little box so it can give you the gold). Oops.

(I make no claim about how plausible this scenario actually is. My only point is that if you're worried about such things then the fact that this system doesn't have an explicit utility function offers precisely no grounds at all for being less worried.)

Trying to perform any open-ended task ends up being rather like having a utility function, in the relevant sense.

Now, if you only ever give the AI narrowly defined tasks where you know exactly what it will do to carry them out, then maybe you're safe. But if you're doing that then why does it need to be intelligent in the first place?

[EDITED to fix an embarrassing typo; no changes to content.]