EJT's Shortform

I'll admit I skimmed over it, not really engaging because I don't hold that part of the shutdown problem as a top concern.  My top concerns on this topic are in whether anyone has that much control of an emerging AI's preferences, and whether humans have the wisdom to actually attempt shutdown.

It does look promising for the portion which it addresses.

I am potentially interested. Will read it over then send you a DM.

I sometimes name your work in conversation as an example of good recent agent foundations work, based on having read some of it and skimmed the rest, and talked to you a little about it at EAG. It's on my todo list to work through it properly, and I expect to actually do it because it's the blocker on me rewriting and posting my "why the shutdown problem is hard" draft, which I really want to post.

The reasons I'm a priori not extremely excited are that it seems intuitively very difficult to avoid either of these issues:

• I'd be surprised if an agent with (very) incomplete preferences was real-world competent. I think it's easy to miss ways that a toy model of an incomplete-preference-agent might be really incompetent.
• It's easy to shuffle around the difficulty of the shutdown problem, e.g. by putting all the hardness into an assumed-adversarially-robust button-manipulation-detector or self-modification-detector etc.

It's plausible you've avoided these problems but I haven't read deeply enough to know yet. I think it's easy for issues like this to be hidden (accidentally), so it'll take a lot of effort for me to read properly (but I will, hopefully in about a week).

The part where it works for a prosaic setup seems wrong (because of inner alignment issues (although I see you cited my post in a footnote about this, thanks!)), but this isn't what the shutdown problem is about so it isn't an issue if it doesn't apply directly to prosaic setups.

I found your explanation quite dense, so I only skimmed it initially. I appreciate your inclusion of a brief summary at the start. But that wasn't enough to motivate me to read the whole proposal in detail, because the summary doesn't sound like it addresses the hard parts of the shutdown problem.

Much more in another comment [LW(p) · GW(p)] on the post.

If you read my solution and think it wouldn't work, let me know.

I don't know about "wouldn't help", but it doesn't seem to me on my initial skim to be a particularly promising approach for prosaic AI safety. I certainly don't agree with the first two bullets in the above list.

Apologies, but I've only skimmed your post, so it's possible I'm missing some important details.

I left a comment [LW(p) · GW(p)] on your post explaining some aspects of why I think this.

When I responded in pm I tried to understand why this is a problem in the first place.

I couldn't understand a practical situation where you would use this.

Why not have the model run in discrete limited time segments, where completing a task correctly while burning the the least compute is what you optimize for in training?

It makes the model a machine where you picked weights that exhibit a behavior where each run, submit answers, and shut down for cleanup. That is, submit equals shutdown, they aren't separate steps, and time limit exceeded is also shutdown with negative reward.

Thus, unwanted behavior is selected against by your training mechanism. This is informed by :

https://www.lesswrong.com/posts/pdaGN6pQyQarFHXF4/reward-is-not-the-optimization-target [LW · GW] Which frames RL as not "rewarding" the model but finding a minima in the permutation space of network weights that forms a machine that has the behavior your selected for. In this case, turning in a correct answer quickly with as little thinking as possible.

And I mentioned real world examples where you just throw an error and shutdown in realtime control systems when the software doesn't have a decision ready by a deadline. Or you do it purely in hardware with a hardware timer that can send a signal directly to processor reset, or you accomplish this with a separate microcontroller.

This is a standard way to build autonomous car control systems.

These "I wasn't asking" approaches force models that "hang" and try to run past a deadline shut down anyway, and it is resistant to tampering. "Resistant" meaning it's not thought to be physically possible.

I created one for Cotra 2020 and Davidson 2023. Hope they're correct.