Why are we sure that AI will "want" something?

Probably relevant: What's Up With Confusingly Pervasive Consequentialism? [LW · GW]

AI won't have wishes or desires. There is no correlation in the animal kingdom between desires and cognitive function (the desire to climb on the social hierarchy or to have sex is preserved no matter the level of intelligence). Dumb humans want basically the same things as bright humans. All that suggests that predictive modeling of the world is totally decoupled from wishes and desires.

I suppose it is theoretically possible to build a system that also incorporates desires, but why would we do that? We want Von Neuman's cognitive abilities, not Von Neuman's personality.

There might sort of be three pieces of relevant information, out of which my previous answer only addressed the first one.

The second one is, what's up with mesaoptimizers? Why should we expect an AI to have mesaoptimizers, and why might they end up misaligned?

In order to understand why we would expect mesaoptimizers, we should maybe start by considering how AI training usually work. We usually use an outer optimizer - gradient descent - to train some neural network that we want to apply for some want we have. However, per the argument I made in the other comment thread, when we want to achieve something diffocult, we're likely going to have the neural network itself do some sort of search or optimization. (Though see What is general-purpose search, and why might we expect to see it in ML systems? [LW · GW] for more info.)

One way one could see the above is, with simple neural networks, the neural network itself "is not the AI" in some metaphorical sense. It can't learn things on its own, solve goals, etc.. Rather, the entire system of {engineers and other workers who collect the data and write the code and tune the hyperparameters, datacenters who train the network, neural network itself} is the intelligence, and it's not exactly entirely artificial, since it contains a lot of natural intelligence too. This is expensive! And only really works for problems we already know how to solve, since the training data has to come from somewhere! And it's not retargetable, you have to start over if you have some new task that needs solving, which also makes it even more expensive! It's obviously possible to make intelligences that are more autonomous (humans are an existence proof), and people are going to attempt to do so since it's enormously economically valuable (unless it kills us all), and those intelligences would probably have a big internal consequentialist aspect to them, because that is what allows them to achieve things.

So, if we have a neural network or something which is a consequentualist optimizer, and that neural network was constructed by gradient descent, which itself is also an optimizer, then by definition that makes the neural network a mesaoptimizer (since mesaoptimizers by definition are optimizers constructed by other optimizers). So in a sense we "want" to produce mesaoptimizers.

But the issue is, gradient descent is a really crude way of oroducing those mesaoptimizers. The current methods basically work by throwing the mesaoptimizer into some situation where we think we know what it should do, and then adjusting it so that it takes the actions we think it should take. So far, this leaves them very capability-limited, as they don't do general optimization well, but capabilities researchers are aiming to fix that, and they have many plausible methods to improve them. So at some point, maybe we have some mesaoptimizer that was constructed through a bunch of examples of good and bad stuff, rather than through a careful definition of what we want it to do. And we might be worried that the process of "taking our definition of what we want -> producing examples that do or do not align with that definition -> stuffing those examples into the mesaoptimizer" goes wrong in such a way that the AI doesn't follow our definition of what we want, but instead does something else - that's the inner alignment problem. (Meanwhile the "take what we want -> and define it" process is the outer alignment problem.)

So that was the second piece. Now the third piece of information: IMO it seems to me that a lot of people thinking about mesaoptimizers are not thinking about the "practical" case above, but instead more confused or hypothetical cases, where people end up with a mesaoptimizer almost no matter what. I'm probably not the right person to defend that perspective since they often seem confused to me, but here's an attempt at a steelman:

Mesaoptimizers aren't just a thing that you're explicitly trying to make when you train advanced agents. They also happen automatically when trying to predict a system that itself contains agents, as those agents have to be predicted too. For instance for language models, you're trying to predict text, but that text was written by people who were trying to do something when writing it, so a good language model will have a representation of an approximation of those goals.

In theory, language models are just predictive models. But as we've learned, if you prompt them right, you can activate one of those representations of human goals, and thereby have them solve some problems for you. So even predictive models become optimizers when the environment is advanced enough, and we need to beware of that and consider factors like whether they are aligned and what that means for safety.

I think an "external push" is extremely likely. It'll just look like someone trying to get an AI to do something clever in the real world.

Take InstructGPT, the language model that right now is a flagship model of OpenAI. It was trained in two phases: First, purely to predict the next token of text. Second, after it was really good at predicting the next token, it was further trained with reinforcement learning from human feedback.

Reinforcement learning to try to satisfy human preferences is precisely the sort of "external push" that will incentivize an AI that previously did not have "wants" (i.e. that did not previously choose its actions based on their predicted impacts on the world) to develop wants (i.e. to pick actions based on their predicted impact on the world).

Why did OpenAI do such a thing, then? Well, because it's useful! InstructGPT does a better job answering questions than regular ol' GPT. The information from human feedback helped the AI do better at its real-world purpose, in ways that are tricky to specify by hand.

Now, if this makes sense, I think there's a subset of peoples' concerns about an "internal push" that make sense by analogy:

Consider an AI that you want to do a task that involves walking a robot through an obstacle course (e.g. mapping out a construction site and showing you the map). And you're trying to train this AI without giving it "wants," just as a tool, so you're not giving it direct feedback for how good of a map it shows you, instead you're doing something more expensive but safer: you're training it to understand the whole distribution of human performance on this task, and then selecting a policy conditional on good performance.

The concern is that the AI will have a subroutine that "wants" the robot to navigate the obstacle course, even though you didn't give an "outside push" to make that happen. Why? Well, it's trying to predict good navigations of the obstacle course, and it models that as a process that picks actions based on their modeled impact on the real world, and in order to do that modeling, it actually runs the compuations.

In other words, there's an "internal push" - or maybe equivalently a "push from the data rather than from the human," which leads to "wants" being computed inside the model of a task that is well-modeled by using goal-based reasoning. This all works fine on-distribution, but produces generalization behavior that generalizes like the modeled agent, which might be bad.

Because being able to do impressive stuff means you had some degree of coherence. From https://www.alignmentforum.org/posts/7im8at9PmhbT4JHsW/ngo-and-yudkowsky-on-alignment-difficulty [AF · GW] : 
 

But though mathematical reasoning can sometimes go astray, when it works at all, it works because, in fact, even bounded creatures can sometimes manage to obey local relations that in turn add up to a global coherence where all the pieces of reasoning point in the same direction, like photons in a laser lasing, even though there's no internal mechanism that enforces the global coherence at every point.

To the extent that the outer optimizer trains you out of paying five apples on Monday for something that you trade for two oranges on Tuesday and then trading two oranges for four apples, the outer optimizer is training all the little pieces of yourself to be locally coherent in a way that can be seen as an imperfect bounded shadow of a higher unbounded structure, and then the system is powerful though imperfect because of how the power is present in the coherence and the overlap of the pieces, because of how the higher perfect structure is being imperfectly shadowed. In this case the higher structure I'm talking about is Utility, and doing homework with coherence theorems leads you to appreciate that we only know about one higher structure for this class of problems that has a dozen mathematical spotlights pointing at it saying "look here", even though people have occasionally looked for alternatives.

Having plans that lase is (1) a thing you can generalize on, i.e. get good at because different instances have a lot in common, and (2) a thing that is probably heavily rewarded in general (by the reward thingy, or by internal credit assignment / economies), to the extent that the reward systems have correct credit assignment. So an AI that does impressive stuff probably has a general skill + dynamic of increasing coherence.

I don't have a direct answer to your question, so for now let's say that AI will not in fact "want" anything if not explicitly asked. This seems plausible to me, but also totally irrelevant from a practical perspective—who's going to build an entire freakin' superintelligence, and then just never have it do do anything!? In order for a program to even communicate words or data to us it's going to need to have some sort of drive to do so, since otherwise it would remain silent and we'd effectively have a very expensive silicon brick on our hands. So while in theory it may be possible to build an AI without wants, in practice there is always something an AI will be "trying to do".

Why were we so sure that strong enough AIs playing go would develop (what we can describe as a) fear of bad aji (latent potential)?

Well, we weren’t. As far as I know, nobody ever predict that. But in retrospect we should have, just because aji is such an important concept to master this game.

Similarly, if we’re looking for a generic mechanism that would led an AI to develop agency, I suspect any task would do as long as interpreting the data as from agency-based behaviors helps enough.

First they optimized for human behavior - that’s how they understood agency. Then they evaluate how much agency explain their own behavior - that’s how they noticed increasing it helps their current tasks. Rest is history.

But why would it do something when not asked? I.e. Why would it have needs/wants/desires to do anything at all?

These are inequivalent, but the answer to each is "because it was designed that way". Conventional software agents, which don't have to be particularly intelligent, do things without being explicitly instructed.

Firstly we already have AI designs that "want" to do things. Deep blue "wanted" to win at chess. Various reinforcement learning agents that "want" to win other games.

Intelligence that isn't turned to doing anything is kind of useless. Like you have an AI that is supposedly very intelligent. But it sits there just outputting endless 0's. What's the point of that? 

There are various things intelligence can be turned towards. One is the "see that thing there, maximize that". Another option is prediction. Another is finding proofs. 

An AI that wants things is one of the fundamental AI types. We are already building AI's that want things. They aren't yet particularly smart, and so aren't yet dangerous. 

Imagine an AI trained to be a pure perfect predictor. Like some GPT-N. It predicts humans, and humans want things. If it is going somewhat outside distribution, it might be persuaded to predict an exceptionally smart and evil human. And it could think much faster. Or if the predictor is really good at generalizing, it could predict the specific outputs of other superhuman AI. 

Mesa-optimizers basically means we can't actually train for wanting X reliably. If we train an AI to want X, we might get one that wants Y instead. 

↑ comment by TAG · 2022-09-17T13:56:16.599Z · LW(p) · GW(p)

Firstly we already have AI designs that “want” to do things. Deep blue “wanted” to win at chess. Various reinforcement learning agents that “want” to win other games.

"Wanting" in quotes isn't the problem. Toasters "want" to make toast.

Intelligence that isn’t turned to doing anything is kind of useless. Like you have an AI that is supposedly very intelligent. But it sits there just outputting endless 0′s. What’s the point of that?

Doing something is not the same thing as doing-something-because-you-want-to. Toasters don't want to make toast, in the un-scare-quoted sense, they just do. Having goals, aims and drives is only meaningful if they can be swapped out for other ones (even if through training). It's a counterfactual concept.

Replies from: donald-hobson

↑ comment by Donald Hobson (donald-hobson) · 2022-09-18T11:02:40.508Z · LW(p) · GW(p)

"Wanting" in quotes isn't the problem. Toasters "want" to make toast.

A standard toaster has been designed to make toast. But there is no part of it that creates new plans of how to make toast better. An evolutionary search for aerial designs can come up with new shapes, better than anything a human could invent. Deep blue can invent new chess moves no human could think of. A toaster doesn't invent a new better way of making toast.

A toaster is optimized by humans, but contains no optimizing. 

Can you explain what an AI that plays good chess, but not because it wants to win, would be like? 

Having goals, aims and drives is only meaningful if they can be swapped out for other ones (even if through training). It's a counterfactual concept.

If you do decide to define goals that way (I am not convinced this is the best or only way to define goals) that still doesn't mean humans know how to swap the goals in practice. It is just talking about some way to do it in principle.

Replies from: TAG

↑ comment by TAG · 2022-09-18T17:42:39.734Z · LW(p) · GW(p)

But there is no part of it that creates new plans of how to make toast better. An evolutionary search for aerial designs can come up with new shapes, better than anything a human could invent. Deep blue can invent new chess moves no human could think of. A toaster doesn’t invent a new better way of making toast.

Ok...but what is your point? That any optimiser has goals?

Can you explain what an AI that plays good chess, but not because it wants to win, would be like?

Any current chess software with the possible exception of AlphaChess.

A hard coded chess program wouldn't be able to something else just because it wanted to.

Replies from: donald-hobson

↑ comment by Donald Hobson (donald-hobson) · 2022-09-19T11:58:35.057Z · LW(p) · GW(p)

Ok...but what is your point? That any optimiser has goals?

Any optimizer needs to be optimizing for something.

A hard coded chess program wouldn't be able to something else just because it wanted to.

The universe is deterministic. So this applies equally well to a human, AlphaChess and Deep blue.

Would you agree that AIXI-tl is just as hard coded as Deep Blue? And that nonetheless, it forms clever plans to achieve its goals and is very dangerous. 

Replies from: TAG

↑ comment by TAG · 2022-09-19T18:51:48.104Z · LW(p) · GW(p)

Any optimizer needs to be optimizing for something.

OK, but you can't infer that something is an optimiser from the fact that it's good at something. You can infer that it's optimising something if you can change the something.

The universe is deterministic.

Not a fact.

So this applies equally well to a human, AlphaChess and Deep blue

Not everything has a replaceable UF or goal module. That renders it false that every AI has a goal. That means that you can't achieve general AI safety by considering goals alone.

Would you agree that AIXI-tl is just as hard coded as Deep Blue? And that nonetheless, it forms clever plans to achieve its goals and is very dangerou

No, AIXIs don't form plans.

Replies from: donald-hobson

↑ comment by Donald Hobson (donald-hobson) · 2022-09-21T09:40:31.830Z · LW(p) · GW(p)

OK, but you can't infer that something is an optimiser from the fact that it's good at something. You can infer that it's optimising something if you can change the something.

 

The problem is that anything can be changed into anything, with enough changes. You have a bunch of chess algorithms. On one extreme, a pure min-max search with a piececount heuristic might be changed into a Misere (play to loose) chess engine, just by adding a single - sign. 

Deep blue might be changed into a somewhat ok Misere chess player with a single -, and a better misere player by also making a few changes to the opening book. 

A neural net based approach might need almost all the parameters in the net changed, but if the net was produced by gradient descent, a small change in the training process could produce a misere chess player instead. 

Going even further back, a small change to the project specifications could have caused the programmers to write different code. 

There are some algorithms where 1 small change will change the goal. And there are algorithms where many pieces must be changed at once to make it competently optimize some other goal. And everything in between, with no hard thresholds.

There are algorithms where humans know how to swap goals. And algorithms where we don't know how to swap goals.

    The universe is deterministic.

Not a fact.

Quantum mechanics has a "branch both ways" that is sometimes mistaken for randomness. True randomness would be non-unitary and break a bunch of the maths. It isn't totally inconceivable that the universe does it anyway, but it seems unlikely. Do you have any coherent idea of what the universe might possibly do that isn't deterministic or random?

No, AIXIs don't form plans.

Are you claiming that an AIXI won't produce clever external behavior. If it had enough information, most of it's hypothesis will contain reasonably good simulations of reality. It tests all possible sequences of actions in these simulations. If it needs to hack some robots, and break in to where it's reward button is, it will. If it needs to win a game of chess, it will. 

Are you claiming AIXI won't do this. It will just sit there being dumb?

Or are you claiming that however clever seeming it's external behavior, it doesn't count as a plan for some reason? 

Replies from: TAG, TAG

↑ comment by TAG · 2022-09-21T17:50:18.694Z · LW(p) · GW(p)

The problem is that anything can be changed into anything, with enough changes.

Ok. Are you suggesting a better technical definition, or are you suggesting going back to the subjective approach?

It tests all possible sequences of actions in these simulations.

No it doesn't.

If it needs to hack some robots, and break in to where it’s reward button is,

It doesn't have one. An AIXI isn't an learning system, and doesn't have a reward function.

it will. If it needs to win a game of chess, it will.

An AIXI is an algorithm that tries to predict sequences of input data using programmes. That's it. It doesnt want or need.

Are you claiming AIXI won’t do this. It will just sit there being dumb?

No, it will sit there figuring out the shortest code sequence that predicts its input. The only thing it can do.

Or are you claiming that however clever seeming it’s external behavior, it doesn’t count as a plan for some reason?

What external behaviour?

Replies from: donald-hobson

↑ comment by Donald Hobson (donald-hobson) · 2022-09-21T20:21:42.605Z · LW(p) · GW(p)

An AIXI is an algorithm that tries to predict sequences of input data using programmes. That's it. It doesnt want or need.

AIXI is the version that optimizes over the programs to maximize reward. 

Solomonov induction is the version that "just" produces predictions. 

 

I would say something is optimizing if it is a computation and the simplest explanation of its behavior looks causally forward from it's outputs.

If the explanation "Y is whatever it takes to make Z as big as possible" is simpler than " Y is f(X)" then the computer is an optimizer. Of course, a toaster that doesn't contain any chips isn't even a computer. For example "deep blue is taking whichever moves lead to a win" is a simpler explanation than a full code listing of deep blue. 

I would say something is an optimizer if you have more knowledge of Z than Y. Of course, this is "subjective" in that it is relitive to your knowledge. But if you have such deep knowledge of the computation that you know all about Y, then you can skip the abstract concept of "optimizer" and just work out what will happen directly. 

↑ comment by TAG · 2022-09-21T18:05:20.306Z · LW(p) · GW(p)

Quantum mechanics has a “branch both ways” that is sometimes mistaken for randomness. True randomness would be non-unitary and break a bunch of the maths.

Which is an odd things to say, since the maths used by Copenhagenists is identical to the maths used by many worlders.

It isn’t totally inconceivable that the universe does it anyway, but it seems unlikely. Do you have any coherent idea of what the universe might possibly do

You are trying to appeal to the Yudkowsky version of Deutsch as incontrovertible fact, and it isn't.

If the MWI is so good, why do half of all physicists reject it?

If Everett's 1957 thesis solved everything, why have there been decades of subsequent work?

Bearing in mind that evolution under the Schrodinger equation cannot turn a coherent state into an incoherent one, what is the mechanism of decoherence?

What is there a basis problem, and what solves it?

Replies from: donald-hobson

↑ comment by Donald Hobson (donald-hobson) · 2022-09-21T20:03:51.359Z · LW(p) · GW(p)

If the MWI is so good, why do half of all physicists reject it?

Because physisists aren't trained in advanced rationality, program length occams razor etc. Their professor taught Copenhagen. And many worlds is badly named, misunderstood and counterintuitive. The argument from majority can keep everyone stuck at a bad equilibrium. 

If Everett's 1957 thesis solved everything, why have there been decades of subsequent work?

If the sentence "god doesn't exist" solved theology, why is there so much subsequent work.

Bearing in mind that evolution under the Schrodinger equation cannot turn a coherent state into an incoherent one, what is the mechanism of decoherence?

I don't think there is one. I think it likely that the universe started in a coherent state, and it still is coherent.

Note a subtlety. Suppose you have 2 systems, A and B. And they are entangled. As a whole, the system is coherent. But if you loose A somewhere, and just look at B, then the maths to describe B is that of an incoherent state. Incoherence = entanglement with something else. If a particle in an experiment becomes incoherent, it's actually entangled somehow with a bit of the measuring apparatus. There is no mathematical difference between the two unless you track down which particular atom in the measuring apparatus. 

Replies from: TAG

↑ comment by TAG · 2022-09-22T11:46:54.742Z · LW(p) · GW(p)

I don’t think there is one. I think it likely that the universe started in a coherent state, and it still is coherent

Then why don't we see other universes, and why do we make only classical observations? Of course, these are the two problems with Everett's original theory that prompted all the subsequent research. Coherent states continue to interact, so you need decoherence for causally separate, non interacting worlds...and you need to explain the preponderance of the classical basis, the basis problem.

Incoherence = entanglement with something else.

No, rather the reverse. It's when off diagonal elements are zero or negligible.

Decoherence, the disappearance of a coherent superposed state, seems to occur when a particle interacts with macroscopiic apparatus or the environment. But thats also the evidence for collapse. You can't tell directly that you're dealing with decoherent splitting,rather than collapse because you can't observe decoherent worlds.

Replies from: donald-hobson

↑ comment by Donald Hobson (donald-hobson) · 2022-09-23T09:55:35.436Z · LW(p) · GW(p)

Then why don't we see other universes, and why do we make only classical observations?

What do you mean by this. What are you expecting to be able to do here?

Shrodingers cat. Cat is in a superposition of alive and dead. Scientist opens box. Scientist is in superposition of feeding live cat and burying dead cat. 

The only way to detect a superposition is through interference. This requires the 2 superimposed states to overlap their wavefunction. In other words, it requires every last particle to go into the same position in both worlds. So it's undetectable unless you can rearrange a whole cat to atomic precision. 

Coherent states continue to interact, so you need decoherence for causally separate, non interacting worlds.

In practice, if two states are wildly different, the interaction term is small. With precise physics equipment, you can make this larger, making 2 states where a bacteria is in different positions and then getting those to interact. Basically, blobs of amplitude need to run into each other to interact. Quantum space is very spacious indeed, so the blobs usually go their own separate way once they are separated. It's very unlikely they run into each other at random, but a deliberate collision can be arranged. 

No, rather the reverse. It's when off diagonal elements are zero or negligible.

That is what the matrix looks like, yes.

But thats also the evidence for collapse.

Interaction with the environment is a straightforward application of schrodingers equation. Collapse is a new unneeded hypothesis that also happens to break things like invarence of reference frame. 

Replies from: TAG

↑ comment by TAG · 2022-09-24T13:12:47.901Z · LW(p) · GW(p)

What are you expecting to be able to do here?

Show that coherence is simple but inadequate, and decoherence is adequate but not simple .

Shrodingers cat. Cat is in a superposition of alive and dead. Scientist opens box. Scientist is in superposition of feeding live cat and burying dead cat.

The two problems with this account are 1) "alive" and "dead" are classical states -- a classical basis is assumed. and 2) the two states of the observer are assumed to be non-interacting and unaware of each other. But quantum mechanics itself gives no reason to suppose that will be the case. In both cases, it needs to be shown, and not just assumed that normality -- perceptions "as if" of a single classical world by all observers -- is restored.

So it’s undetectable unless you can rearrange a whole cat to atomic precision.

So you can't have coherent superpositions of macroscopic objects. So you need decoherence. And you need it to be simple, so that it is still a "slam dunk".

Basically, blobs of amplitude need to run into each other to interact.

How narrow a quantum state is depends, like everything, on the choice of basis. What is sharply peaked in position space is spread out in frequency/momentum space.

Interaction with the environment is a straightforward application of schrodingers equation.

No it isn't. That's why people are still publishing papers on it.