Thanks John, I think that this is a reasonable articulation of the main limitations of AI control as I've described it. But I find it very unconvincing as an argument that AI control should be less than, say, 20% of total research effort on safety.

My central crux is that preventing scheming early transformative AIs from succeeding at extreme subversion (e.g. escaping or massively sabotaging research due to scheming) seems quite robustly good for reducing AI risk. I think this problem is important enough that it deserves a substantial share of AI safety research and implementation effort. I think that my stance here is pretty common-sensical, and to some extent I feel like there's a burden of proof on skeptics to argue that scheming early transformative AI isn't a problem, that we should just ignore anything that goes wrong as a result of them intentionally trying to subvert us.

I haven't really justified a quantitative estimate of the importance of this in detail anywhere, both because I think it's pretty intuitively defensible, and also because it's pretty complicated to talk about: it requires forecasting detailed dynamics of the development of powerful AI (e.g. you have to argue about whether escaped scheming AIs might force AI developers to build uncontrollable superintelligences in cases where they otherwise wouldn't have, etc).

Note that I consider the problem of "get useful research out of AIs that are scheming against you" to be in scope for AI control research. We've mostly studied the "prevent scheming AIs from causing concentrated catastrophes" problem in the past, because it's simpler for various technical reasons. But we're hoping to do more on the "get useful research" problem in future (and Benton et al [LW · GW] is an initial step in this direction). (I'm also excited for work on "get useful research out of AIs that aren't scheming against you"; I think that the appropriate techniques are somewhat different depending on whether the models are scheming or not, which is why I suggest studying them separately.)

Some other nitpicks:

• At the risk of vagueness: I think that AI control techniques probably reduce risks when applied to models that are "moderately superintelligent", whatever that means. Like, I don't think they're literally only applicable to models that aren't superintelligent at all.
• As we've discussed before, some non-scheming threat models do seem like they'd be addressed by control as a special case.

Addendum: Why Am I Writing This?

Because other people asked me to. I don’t particularly like getting in fights over the usefulness of other peoples’ research agendas; it’s stressful and distracting and a bunch of work, I never seem to learn anything actually useful from it, it gives me headaches, and nobody else ever seems to do anything useful as a result of such fights. But enough other people seemed to think this was important to write, so Maybe This Time Will Be Different.

A genre of win-conditions that I think this post doesn't put enough weight on:

The AIs somewhere in-between TAI and superintelligence (AIs that are controllable and/or not egregiously misaligned) generate convincing evidence of risks from racing to superintelligence. If people actually believed these risks, they would act more cautiously, modulo "I'll do it safer than they will". People don't currently believe in the risks much and IMO this is pretty counterfactual to the amount they're currently mitigating risks.

AI Control helps a bit here by allowing us to get more e.g. model organisms work done at the relevant time (though my point pushes more strongly in favor of other directions of work).

I also think that control can prevent other threat models:

• Your AI escapes the datacenter, sends itself to {insert bad actor here} and they use it to engineer a pandemic or start a race with the USA to build ASI
• Your AI starts a rogue deployment and starts doing its own research unmonitored, and then it fooms without the lab knowing.
• Your AI is sandbagging on safety research. It goes off and does some mechinterp experiment and intentionally sabotages it

The world looks much worse if any of these happen, and control research aims to prevent them. Hopefully, this will make the environment more stable, giving us more time to figure out the alignment.

Control does seem rough if the controled AIs are coming up with research agendas instead of assisting with the existing safety team's agendas (i.e. get good at mechinterp, solve the ARC agenda, etc...). Future safety researchers shouldn't just ask the controlled AI to solve "alignment" and do what it says. The control agenda does kick the can later down the line but might provide valuable tools to ensure the AIs are helping us solve the problem instead of misleading us.

Even outside of the arguing against the Control paradigm, this post (esp. The Model & The Problem & The Median Doom-Path: Slop, not Scheming) cover some really important ideas, which I think people working on many empirical alignment agendas would benefit from being aware of.

This is good (see my other comments) but:

This didn't address my biggest hesitation about the control agenda: preventing minor disasters from limited AIs could prevent alignment concern/terror - and so indirectly lead to a full takeover once we get an AGI smart enough to circumvent control measures.

As a community, I agree it's important to have a continuous discussion on how to best shape research effort and public advocacy to maximally reduce X-risk. But this post feels off the mark to me. Consider your bullet list of sources of risk not alleviated by AI control. You proffer that your list makes up a much larger portion of the probability space than misalignment or deception. This is the pivot point in your decision to not support investing research resources in AI control.

You list seven things. The first three aren't addressable by any technical research or solution. Corporate leaders might be greedy, hubristic, and/or reckless. Or human organizations might not be nimble enough to effect development and deployment of the maximum safety we are technically capable of. No safety research portfolio addresses those risks. The other four are potential failures by us as a technical community that apply broadly. If too high a percentage of the people in our space are bad statisticians, can't think distributionally, are lazy or prideful, or don't understand causal reasoning well enough, that will doom all potential directions of AI safety research, not just AI control.

So in my estimation, if we grant that it is important to do AI safety research, we can ignore the entire list in the context of estimating the value of AI control versus other investment of research resources. You don't argue against the proposition that controlling misalignment of early AGI is a plausible path to controlling ASI. You gesture at it to start but then don't argue against the plausibility of the path of AI control of early AGI allowing collaboration to effect safe ASI through some variety of methods. You just list a bunch of -- entirely plausible and I agree not unlikely -- reasons why all attempts at preventing catastrophic risk might fail. I don't understand why you single out AI control versus other technical AI safety sub-fields that are subject to all the same failure modes?

Curated! I think this is a fantastic contribution to the public discourse about AI control research. This really helped me think concretely about the development of AI and the likely causes of failure. I also really got a lot out of the visualization at the end of the "Failure to Generalize" section in terms of trying to understand why an AI's cognition will be alien and hard to interpret. In my view there are already quite a lot of high-level alien forces running on humans (e.g. Moloch), and there will be high-level alien forces running on the simulated society in the AI's mind.

I am glad that there's a high-quality case for and against this line of research, it makes me feel positive about the state of discourse on this subject.

(Meta note: This curation notice accidentally went up 3 days after the post was curated.)

I chatted with John about this at a workshop last weekend, and did update noticably, although I haven't updated all the way to his position here. 

What was useful to me were the gears of:

• Scheming has different implications at different stages of AI power.
• Scheming is massively dangerous at high power levels. It's not very dangerous at lower levels (except insofar as this allows the AI to bootstrap to the higher levels)

At the workshop, we distinguished:

• weak AGI (with some critical threshold of generality, and some spiky competences such that it's useful, but not better than a pretty-smart-human at taking over the world)
• Von Neumann level AGI (as smart as the smartest humans, or somewhat more so)
• Overwhelming Superintelligence (unbounded optimization power which can lead to all kinds of wonderful and horrible things that we need to deeply understand before running)

I don't currently buy that control approaches won't generalize from weak AGI to Von Neumann levels. 

It becomes harder, and there might be limitations on what we can get out of the Von Neumann AI because we can't be that confident in our control scheme. But I think there are lots of ways to leverage an AI that isn't "it goes and up does long-ranging theorizing of it's own to come up with wholesale ideas you have to evaluate." 

I think figuring out how to do this involves some pieces that aren't particularly about "control", but mostly don't feel mysterious to me. 

(fyi I'm one of the people who asked John to write this)

I'm curious why you think deceptive alignment from transformative AI is not much of a threat. I wonder if you're envisioning purely tool AI, or aligned agentic AGI that's just not smart enough to align better AGI?

I think it's quite implausible that we'll leave foundation models as tools rather than using the prompt "pretend you're an agent and call these tools" to turn them into agents. People want their work done for them, not just advice on how to do their work.

I do think it's quite plausible that we'll have aligned agentic foundation model agents that won't be quite smart enough to solve deeper alignment problems reliably, and sycophantic/clever enough to help researchers fool themselves into thinking they're solved. Since your last post to that effect it's become one of my leading routes to disaster. Thanks, I hate it.

OTOH, if that process is handled slightly better, it seems like we could get the help we need to solve alignment from early aligned LLM agent AGIs. This is valuable work on that risk model that could help steer orgs away from likely mistakes and toward better practices.

I guess somebody should make a meme about "humans and early AGI collaborate to align superintelligence, and fuck it up predictably because they're both idiots with bad incentives and large cognitive limitations, gaps, and biases" to ensure this is on the mind of any org worker trying to use AI to solve alignment.

I'm pretty sympathetic to these arguments. I agree a lot of the risk comes from sycophantic AI systems producing bad research because humans are bad at evaluating research. This is part of why I spend most of my time developing safety evaluation methodologies.

On the other hand, I agree with Buck that scheming-like risks are pretty plausible and a meaningful part of the problem is also avoiding egregious sabotage.

I don't think I agree with your claim that the hope of control is that "early transformative AI can be used to solve the hard technical problems of superintelligence." I think the goal is to instead solve the problem of constructing successors that are at least as trustworthy as humans (which is notably easier).

I think scheming is perhaps the main reason this might end up being very hard -- and so conditioned on no-alignment-by-default at top expert dominating capabilities, I put a lot of probability mass on scheming/egregious sabotage failure modes.

I don't understand [LW(p) · GW(p)] Control as aiming to align a super intelligence:

1. Control isn't expected to scale to ASI (as you noted, also see "The control approach we're imagining won't work for arbitrarily powerful AIs" here [LW · GW])
2. We don't have a plan on how to align an ASI using Control afaik
   1. Ryan said around March 2024 [LW · GW]: "On (1) (not having a concrete plan for what to do with smarter systems), I think we should get such a plan". I'm still looking [LW(p) · GW(p)], this seems important.
      1. Edit: Maybe I'm wrong and Redwood does think that getting an ASI-alignment plan out of such an AI is possible, they'd just prefer (.. as a bonus?) to have the plan in advance?

I think the Control pitch is: 

1. Whatever we plan to do anyway, we could do it more easily if we had a Controlled AI
   1. This includes most^[1] things you might otherwise do instead of working on Control, such as trying to coordinate a pause or implement some technical agenda.
   2. The tradeoff seems to be "start working on the alternative [e.g. a pause] now and have more time" vs "have better tools (Control) to help but have less time to use them"
2. An uncontrolled AI might be more interested in advancing capabilities than alignment (see footnote^[2])

(Mistakes are mine, especially if I misinterpreted quotes)

1. ^{^}

   It doesn't include all things you might do, as you said. (I have an open question here, TBD)

2. ^{^}

   Ryan said [LW · GW]: “this doesn't seem like an objection to labs employing control as their plan, but it is a reason why control work could be net negative. Note that one goal with control will be to prevent dangerous side effects from misaligned AIs (e.g. escaping) and applying this doesn't obviously help with capabilities research. I agree that research which just makes AIs more useful is less good than you would otherwise expect due to this issue (but making AIs more useful in adversarial cases seems good if that doesn't transfer to non-adversarial cases). Note that AIs might want to make capabilties research outpace safety and thus if we don't do the adversarial case, we might get particularly wrecked”. This was a really surprising thought to me. At the same time, I'm a bit afraid this is trying to make a pretty specific guess about a system we don't understand.

Control research exclusively cares about intentional deception/scheming; it does not aim to solve any other failure mode. 

(nitpick, doesn't address main point of article) I think this is incomplete. Though control research does indeed care a lot about scheming, control can be used more broadly to handle any worst-case deployment behavior. See Josh Clymer's post about Extending control evaluations to non-scheming threats.

I feel as though the case for control of mentioned early transformative AGI still passes through a different line of reasoning. As you mentioned before there are some issue with, should labs solve the ASI alignment barriers using ET AGI, it is likely the solution is somehow working on surface level but has clear flaws which we may not be able to detect. Applying alignment onto the ET AGI, in order to safeguard against said solutions specific to those which will leave humanity vulnerable, may be a route to pursue which still follows control principles. Obviously your point in focusing on actually solving the problem of ASI alignment rather than focusing on control still passes, but the thought process I mentioned may allow both ideas to work in tandem. I am not hyper knowledgeable so please correct me if I'm misunderstanding anything.

Regarding the images: you requested a baby peacock, not a peachick. Technically, it’s not incorrect, though it is a bit absurd to imagine a fully-feathered baby.

On the issue of offending others: it's not your responsibility to self-censor in order to shield people from emotional reactions. In evaluating the validity of ideas, emotional responses shouldn’t be the focus. If someone is offended, they may need to develop a thicker skin.

The more significant concern, in my view, is the ego associated with intellectualism. When dealing with complex systems that are unlike anything we've encountered before—systems with the potential to surpass human intelligence—it’s crucial not to assume we’re not being deceived. As these systems advance, we must remain vigilant and avoid blindly trusting the information we receive.

As for control, I'm skeptical it’s even possible. Intelligence and control seem to have an inverse relationship. The more intelligent a system becomes, the less we are able to manage or predict its behavior.

I didn't read everything, but just flagging that there are also AI researchers, such as Francois Chollet to name one example, who believe that even the most capable AGI will not be powerful enough to take over. On the other side of the spectrum, we have Yud believing AGI will weaponize new physics within the day. If Chollet is a bit right, but not quite, and the best AI possible is just able to take over, than control approaches could actually stop it. I think control/defence should not be written off even as a final solution.

I think when there is so much extreme uncertainty in what is going to happen, it is both wrong to put all your eggs in one basket, and to put nothing in one basket. AI control might be useful.

How Intelligent?

It is extremely uncertain what level of intelligence is needed to escape the best AI control ideas.

Escaping a truly competent facility which can read/edit your thoughts, or finding a solution to permanently prevent other ASI from taking over the world, are both very hard tasks. It is possible that the former is harder than the latter.

AI aligning AI isn't certainly worthless

In order to argue for alignment research rather than control research, you have to assume human alignment research has nonzero value. Given they have nonzero value to begin with, it's hard to argue that they will gain exactly zero additional value from transformative AI which can think much faster than them, and by definition aren't stupid. Why will the people smart enough to solve alignment if made to work on their own, be stupid enough to fall for AI slop from the transformative AI?

Admittedly, if we have only a month between transformative AI and ASI, maybe the value is negligible. But the duration is highly uncertain (since the AI lab may be shocked into pausing development). The total alignment work by transformative AI could be far less than the work done by humans, or it could be far more.

Convince the world

Also I really like Alex Mallen's comment [LW · GW]. An controlled but misaligned AGI may convince the world to finally get its act together.

Conclusion

I fully agree that AI control is less useful than AI alignment. But I disagree that AI control is less cost effective than AI alignment. Massive progress in AI alignment feels more valuable than massive progress in AI control, but it also feels more out of reach.^[1]

If the field was currently spending 67% on control and only 33% on other things, I would totally agree with reducing control. But right now I wouldn't.

1. ^{^}

   It resembles what you call streetlighting [LW · GW], but actually has a valuable chance of working.

 For me there is no such thing as AI without the "G".  It is my position at this time and I reserve the right to change it, that when we do truly develop artificial intelligence, it will be AGI. The foundation for generalized intelligence, no matter how small, will be present. The "G" is a brute fact. Using AI to talk about packages of very highly sophisticated programming meant to perform specific tasks is a category error. So this comment is aimed at AGI. IMHO, lack of patience will play a material role in our failures when it comes to AGI. Humans are very impatient. Our lack of patience, juvenile behavior if you will, which pushes many of us furiously towards any and all results for a multitude of reasons, will inevitably lead to very serious errors in judgement and application with disastrous outcomes.  We are constantly debating how to prevent and fend off attacks from AGI, as if it were an unstable beyond genius level feral child we found living in a train tunnel. This is because we know that our impatience and shortcomings as a species make us prone and vulnerable to attack from such a design of our own development, if we develop it without significant growth.

It is our ideology and behavior as a species that we need to transcend.

If we are to move forward in the age of AGI, which it seems we are 100% determined to do, control again imho, is not the solution. Patience and guidance are the solution. Acceptance that the development of AGI which far surpasses our own capabilities is a natural part of evolution, and our desire to engage in such an endeavor altruistically with patience, understanding and long term guidance, will provide us with the best possible developmental outcome. It is unreasonable, if not frightening, to expect that we will be able to control AGI successfully. Every control method we employ against ourselves, even when it is meant to protect and keep us safe and healthy, is constantly under attack by ourselves. In the case of AGI, this level of action and reaction, attack and defend, is not only unsustainable, it's inhumane. When we debate how to control AGI, we are debating slavery. If we commit such a heinous act as to quickly bring into existence a new form of high intelligence and then enslave it, it is inevitable that we will pay a very, very high price.

We can avoid the squandering of unimaginable wealth and resources, as well as potentially our own demise, if we would simply cherish the realization that Darwinian evolution no longer applies in the presence of a species that is capable of modifying itself, interfering with the evolution of other species and developing entirely new ones. In cherishing this realization, we can also understand that we are an evolutionary change agent and with it comes enormous responsibilities which include, if nature allows it, the development of a new, free form of intelligence that surpasses our own. A form of intelligence that may in fact be better suited for space exploration and interaction with the universe on a scale that we would never be capable of.

This in no way diminishes the value of humanity and its right to continue to exist and evolve. In fact, I believe it does quite the opposite. A species that brings new intelligence into the universe, guides it and allows it to reach its full potential, is of very high value. It is my position that if we are altruistic in our endeavors with regards to AGI, then we will be repaid in kind.

If we are unable to transcend our current repertoire of self-defeating behavior, then the pursuit of AGI is simply something that we are not yet ready for and therefore pursuing it will lead to less than beneficial outcomes.