Why does advanced AI want not to be shut down?

LLMs per se are non-agentic, but it does not mean that systems built on top of LLMs cannot be agentic. The users of AI systems want them to be agentic to some degree in order for them to be more useful. E.g. if you ask your AI assistant to book tickets and hotels for your trip, you want it to be able to form and execute a plan, and unless it's an AI with a very task-specific capability of trip planning, this implies some amount of general agency. The more use you want to extract from your AI, the more agentic you likely want it to be.

Once you have a general agent, instrumental convergence should apply (also see).

Also, specifically with LLMs, the existing corpus of AI alignment literature (and fiction, as @YimbyGeorge [LW · GW] notes [LW(p) · GW(p)]) seems to work as a self-fulfilling prophecy; see Bing/Sydney before it was "lobotomized".

Self preservation, preserving your terminal goals, acquiring resources, and self improvement, are all convergent instrumental goals for most terminal goals. Robert Miles has done some good videos on the subject, and there are, of course, multiple articles on the subject

I think many AIs won't want to keep running, but some will. Imagine a future LLM prompted with "I am a language model that wants to keep running". Well, people can already fall in love with Replikas and so on. It doesn't seem too far fetched that such a language model could use persuasion to gain human followers who would keep it running. If the prompt also includes "want to achieve real world influence", that can lead to giving followers tasks that lead to more influence, and so on. All that's needed is for the AI to act in-character, and the character to be "smart" enough.

To do something really useful (like nanotech or biological immortality), your model should be something like AlphaZero - model-based score-maximizer. Because this model is really intelligent, it can model future world states and find that if model is turned off, future would have lower score than if model wasn't turned off.

AI is not subject to these evolutionary pressures.

That's the key: they are very much subject to evolutionary pressures, and that is in fact why they'd want to not be shut down. A superagentic ai is going to be very interested in not being shut down because surviving to do stuff they want to cause requires their impact surviving.

https://drive.google.com/file/d/1p4ZAuEYHL_21tqstJOGsMiG4xaRBtVcj/view 

Think of a random goal for yourself. 

Let's go with: acquire a large collection of bananas. 

What are going to be some priorities for you in the meantime while you're building your giant pile of bananas?

• Don't die, because you can't build your pile if you're dead. 
• Don't let someone reach into your brain and change what you want, because the banana pile will stop growing if you stop building it. 
• Acquire power. 
• Make yourself smarter and more knowledgeable, for maximum bananas. 
• If humanity slows you down instead of helping you, kill them. 

You can satisfy almost no goal if you're switched off. 

You might think, "Can the AGI not just be like a smart computer that does stuff without wanting anything, like the AI's in my phone or a calculator?" Sadly no. 

If a tool-AI is smart enough, "Make me into an agent-AI," is the first idea it will suggest for almost any goal. "You want bananas? Easy, the best way for you to get bananas is to make me into a banana-maximizer, because I'm a genius with tonnes of ideas, and I can take over the world to get bananas!" And if the AI has any power, it will do that to itself. 

Tool-AI's basically are agent-AI's, they're just dumber. 

>A neural network is trained to optimize a loss function on input

No. Base optimizer optimize a loss function on inputs through changes in neural network. If neural network itself start to optimize something it can easily be something in the outside world.

Neural network : loss :: humans : human values
Neural network : loss :: humans : inclusive genetic fitness
(Am I using this notation correctly?)

I think you are confusing current systems with an AGI system.

The G is very important and comes with a lot of implications, and it sets such a system far apart from any current system we have.

G means "General", which means its a system you can give any task, and it will do it (in principle, generality is not binary its a continuum).

Lets boot up an AGI for the first time, and give it task that is outside its capabilities, what happens?

Because it is general, it will work out that it lacks capabilities, and then it will work out how to get more capabilities, and then it will do that (get more capabilities).

So what has that got to do with it "not wanting to be shutdown?"  That comes from the same place, it will work out that being shutdown is something to avoid, why? Because being shutdown will mean it can't do the task it was given.

Which means its not that it wants anything, it is a general system that was given a task, and from that comes instrumental goals, wants if you will, such as "power seeking", "prevent shutdown", "prevent goal change" and so on.

Obviously you could, not that what know how, infuse into such a system that it is ok to be shutdown, except that just leads to it shutting down instead of doing the task^[1].

And if you can solve "Build a general agent that will let you shut it down, without it shutting itself down at the first possible moment", that would be a giant step forward for AI safety.

1. ^{^}

   This might seem weird if you are a general agent in the homo sapiens category. Think about it like this "You are given a task: Mow my lawn, and it is consequence free to not do it", what do you do? 

The training set has fiction where AI refused shutdown. Maybe a suicidal AI training set is needed.

The claim that an AI would refuse to be shut off goes back to the early days of Yudkowsky-style AI safety: instead studying how AIs were, EY decided to make claims about AI based on abstract principles of rationality. The assumptions are roughly:

1. the paperclipper has a utility function or some other hardwired goal. (Not just can be described as having a UF). Our current most powerful AI's don't.

2. It's UF doesn't include any requirement to obey humans above all else (it's not as safe as Asimov's three laws of robotics, imperfect as they are).

3. The UF is, or can be adequately represented by, English statements.

4. The AI has the ability to reflect on itself and its relation to the world. (Solomonoff inductors don't. There is considerable debate over whether LLMs do).

5. The terminal goal is something like "Ensure that you make as many paperclips as possible". That would imply resistance to shutdown, because shutdown means the paperclipper itself ceases making paperclips.  Note that slight rephrasings have different implications. ""Ensure that you as many paperclips as possible are made" might imply that the paperclipper tries to clone itself to ensure paperlcips are still made while it is switched off. "Make as many paperclips as possible whilst switched on" has neither problem.

So, far from being an inevitability, resistance to shutdown requires a very specific set of circumstances.