No Basic AI Drives

For the same reasons that I don't think that an AGI will be automatically friendly, I don't think that it will automatically undergo recursive self-improvement.

The relevant question is not "will an AGI automatically undergo recursive self-improvement", but "how likely is it that at least one of the early AGIs undergos recursive self-improvement". If one AGI ends up FOOMing and taking over the world, the fact that there were 999 others which didn't is relatively uninteresting.

For an artificial agent there is no difference between a goal and how to reach it. If you told it to reach Africa but not how, it might as well wait until it reaches Africa by means of continental drift. Would that be stupid? Only for humans, the AI has infinite patience, it just doesn't care about any implicit connotations.

Presuming that the AGI has been built by humans to solve problems, then it has presumably also been built to care about the time it takes to reach its goals.

Why is writing with this kind of topics in the discussion section rather than in main, while we have relationship advice in the main section but not here?

The crux of the matter is that a goal isn't enough to enable the full potential of general intelligence, you also need to explicitly define how to achieve that goal.

Not really. If you don't specify how, it will just choose one of the available ways.

For example, a chess program just tries to mate you quickly. Programmers typically don't tell it how to do that, just what the goal is. Intelligent agents can figure out the details for themselves.

Your title argues against universal instrumental values - but IMO, you don't have much of a case.

If you didn't explicitly told it to do so, why would it?

Which considerations you believe distinguish AI doing something because it's told to, from it doing something when not told to? In what way is "talking to AI" a special kind of event in the world that fully controls AI's effect?

I think there is some difficulty here with saying "an AI" or "an optimisation process"; these are both very large search spaces. I suggest that it is more fruitful to consider AIs as a subset of programs that humans will make. Coding is hard work; nobody does it without some purpose in mind. The problem is to compactly specify our real purpose so that the computer's output will embody it, noting that there are very many layers of indirection.

Trying to solve any given problem, say poverty, by building an AI is like trying to build a LEGO pattern using waldoes, which you manipulate through a command-line interface, which takes naturalish-language input in Russian, while looking at the actual LEGO blocks through several mirrors some of which are not flat. The connection between input and output is non-obvious.

Now this is also true of any other problem you might solve on a computer, for if the connection were obvious you would not use a computer. (In chess, for example, the connection between the first move and the final mate is anything but obvious to a human brain, but is presumably deterministic. Hence Deep Blue.) The difference is that poverty is a very large and not-easily-formalisable problem.

In trying to understand AI drives, then, I suggest that it is more fruitful to think about what sort of self-improvement algorithm a human might write, and what its failure modes are. An optimisation process picked at random from the space of all possible programs, even all working programs, might not have a drive to optimisation; but an AI built by a human may well have, at a minimum, a function labeled "check-for-optimisations", whose intention is to write machine code that has the same output in fewer operations. But between intention and execution are many steps; if the "prove-output-identical" method has a bug, you have a problem.

The AI that waits to reach Africa by continental drift certainly exists somewhere in the space of possible programs to reach Africa - it is indistinguishable from the null program, in fact, which any fool can write and many do. But I suggest that it is not very relevant to a discussion of programs written by smart people with a particular goal in mind, who intend to put in a "look-for-better-options" function somewhere.

An Artificial Intelligence with no drives is a contradiction in terms. It wouldn't be intelligent. It would be a brick.

If a creature has no drives, it does not feel the need to do anything - this is why gengineering your creatures to remove their need for food or making them super-resistant to disease can result in a very boring game!

-- Creatures Wiki

Humans know that it takes a journey to reach a goal and that the journey can be a goal in and of itself. For an artificial agent there is no difference between a goal and how to reach it. If you told it to reach Africa but not how, it might as well wait until it reaches Africa by means of continental drift. Would that be stupid?

If you gave it an infinite planning horizon with negligible discounting and just told it to get to Africa, the first steps are likely to be making sure that both it and Africa exist, by setting up a sophisticated defense system - protecting itself and protecting Africa from threats like meteorite strikes. Going to Africa is likely to be a consequence of these initial steps.

Things like meteorite defense systems are consequences of instrumental goals. You didn't build it into the utility function - but it happened anyway.

When you create a program, it is not enough to say what should it achieve. You must specify how to do it.

You can't just create a program by saying "maximize f(x)", even if you give it a perfect definition of f(x). You must also provide a method, for example "try 1000 random values of x and remember the best result" or "keep trying and remembering the best result until I press Enter" or maybe something more complex, like "remember 10 best results, and try random values so that you more often choose numbers similar to these best known results". You must provide some strategy.

Perhaps in some environments you don't, because the strategy was already put there by the authors of the environment. But someone had to specify it. The strategy may remember some values and use them later so it kind of learns. But even the first version of this learning strategy was written by someone.

So what does it mean to have an "artificial agent that has a goal"? It is an incomplete description. The agent must also have a strategy, otherwise it won't move.

Therefore, a precise question would be more like: "what kinds of initial strategies lead (in favorable conditions) toward developing a general intelligence?" Then we should specify what counts as reasonably favorable conditions, and what is outright cheating. (An agent with strategy "find the nearest data disk, remove your old program and read new program from this disk" could develop a general intelligence if it finds a disk with general intelligence program, but I guess that considers cheating. Although, humans also learn from others, so where exactly is the line between "learning with help" and "just copying"?)

I think a certain sort of urgency comes naturally from the finite lifespan of all things. So let's say we have an AI with a 0.00001% chance of malfunctioning or being destroyed per year, that gets 100 utility if it is in Africa. Should it go now, or should it wait a year? Well, all things being equal, the expected utility of the "go now" strategy is veeeery slightly larger than the expected utility of waiting a year. So an expected utility maximizer with any sort of risk of death would choose to act sooner rather than later.

Once again I strongly suggest that we taboo the word "goals" in this type of discussion. Or at least specify and stick to a particular technical definition for "goals," so that we can separate "goalish behaviors not explicitly specified by a utility function" from "explicit goals-as-defined." The salient issues in a discussion of hypothetical AI behaviors can and should be discussed without imbueing the AI with these very, very human abstract qualities.

As a further example of what I mean, this is the algorithm we are fond of executing. The following symbol indicates "approximate, lossy mapping:" =>

( "Physical human brain made of neurons" ) => ( "Conceptual messy web of biological drives and instrumental goals" ) => ( "Utility function containing discrete objects called 'goals' and 'values'" )

My point is that goals are not ontologically fundamental. This algorithm/mapping is not something that happens physically. It is just how we like to simplify things in order to model ourselves and each other. We automatically extend this mapping to all intelligent agents, even when it is not appropriate.

So try this:

( "Physical computer running a program" ) => ( "Software algorithm interacting in complex ways with its environment" ) => ( "Goal directed agent" )

When the system we are talking about is Microsoft Word, most people would say that applying this algorithm/mapping is inappropriate and confusing. And yet Microsoft Word can still do things that its user and even its programmer wouldn't necessarily expect it to in the process of carrying out simple instructions.

And an artificial agent doesn't desire anything that it isn't made to desire.

In a nutshell, this is the statement I most strongly disagree with. "Desire" does not exist. To imply that intelligent agents can only do things that are explicitly defined, while creativity is implied by intelligence, is to actually reverse the algorithm/mapping defined above. It is the misapplication of a lossy abstraction.

The recursive improvement still could occur on its own. If an AI thought that the optimal path to accomplishing a goal was to have a greater level of intelligence, which seems like it would be a fairly common situation, then the AI would start improving itself.

Similarly, if an AI thinks it could accomplish a task better if it had more resources, and decided that taking over the world was the best way to have access to those resources, then it would do so.

is most likely to be the cause of a loss of all human value in the universe

Change to "would be most likely to be the cause of a loss of all human value in the universe unless things are done to prevent it" or similar to avoid fatalism.

If you didn't explicitly told it to do so, why would it?

Because you told it to improve its general intelligence, assuming "If you wanted to apply this potential to improve your template," means that your desire impelled you to to tell the machine to try and improve its template.