Thoughts on the conservative assumptions in AI control

post by Buck · 2025-01-17T19:23:38.575Z · LW · GW · 0 comments

Contents

  The conservative assumption I don’t make
  Hard worlds are plausible
  Those assumptions are hard to evaluate
    Why it’s nice to rely on these assumptions
      Making those assumptions makes it methodologically clean to evaluate safety
      Relying only on easy-to-evaluate assumptions would allow more clearly-specified and hard-to-game regulatory standards
    Kerckhoff’s principle in computer security as an analogy
  The main downsides of these assumptions
    You can’t completely escape from methodological messiness
    Bad technique prioritization
    Insane game theory requirements to evaluate techniques
  Conclusion
None
No comments

Work that I’ve done on techniques for mitigating risk from misaligned AI often makes a number of conservative assumptions about the capabilities of the AIs we’re trying to control. (E.g. the original AI control paper [AF · GW], Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats, How to prevent collusion when using untrusted models to monitor each other [AF · GW].) For example:

• The AIs are consistently trying to subvert safety measures.
• They’re very good at strategizing about how to subvert them, and can do this strategizing totally illegibly (e.g. in a single forward pass).
• They can follow a strategy that’s coordinated between all their instances.
• They know exactly what safety measures are in use.
• They know a lot about the distribution of inputs.
• They’re able to pick Schelling points perfectly.
• They’re able to follow stochastic policies with excellent calibration (e.g. they’re able to attack with probability 0.01%).

But I don’t think situations where all these assumptions hold contain the majority of risk from AI misalignment. In particular, it’s plausible that the earliest catastrophic risk from misaligned AI will come from models that aren’t competent and coherent enough to have all those properties. (See discussion here [LW(p) · GW(p)].)

Given this, why have I mostly focused on the conservative threat model? Two main reasons:

• Each of those conservative assumptions is plausible. And the assumptions are conservative such that techniques that work under these assumptions would also suffice for safety if the assumptions didn’t hold.
• Making these conservative assumptions allows a cleaner workflow when developing and evaluating safety measures.

I currently think that about half of the AI control research should make these conservative assumptions. But I’m less enthusiastic about these assumptions than I was a year ago. My reduced enthusiasm is mostly because I don’t think that AI companies are very likely to make legible safety cases that their AIs don’t pose catastrophic risk. Since a major advantage of these assumptions is their ability to facilitate such legible safety arguments, their value is reduced.

(I’m using first-person singular here; the arguments here are fairly similar to what you’d hear from my collaborators and other people who do similar work.)

Thanks to Joshua Clymer, Charlie Griffin, Aryan Bhatt, Aaron Scher, Mikita Balesni, Ryan Greenblatt, Joe Benton, Zach Stein-Perlman, Alex Mallen, Ansh Radhakrishnan, and Justis Mills for comments.

The conservative assumption I don’t make

To understand how these assumptions fit in with work on AI control, you have to understand how the AI control research I listed above isn’t conservative: we don’t assume that models are arbitrarily good at performing actions that will lead to them succeeding at their misaligned goal. In our work, we try to evaluate how good the models are at producing such actions, and use that to estimate the probability that they’ll successfully subvert our safety measures and cause a catastrophe.

As a concrete example, in the original AI control paper, we build a scaffold that generates sneakily backdoored code. This scaffold sometimes produces backdoors that are very sneaky and that get past all the safety measures. We don’t therefore say “safety is zero, because we’ve demonstrated that there are actions that the AI could output that would be catastrophic”. Instead, we measure the proportion of the time that the scaffold’s code gets past all the safety measures, and use that as a conservative estimate of risk and as a metric to optimize safety measures against.

This is very different from e.g. the classic LessWrong conception of superintelligence (e.g. as depicted by Eliezer Yudkowsky in That Alien Message [LW · GW]). This difference is mostly because AI control work focuses on AIs that aren’t strongly superintelligent. I think that AI systems like the classic LessWrong depiction will eventually be created; I just think that other AIs will be created earlier that are very dangerous, controllable, and extremely useful, as argued here [LW · GW].

I’ll use “the original AI control plan” to refer to the combination of conservative assumptions I listed above.

We have to be non-conservative somewhere in our planning, or else there’s no planning to do. Many people who are concerned about AI risk prefer the conservative strategy of “if the model is plausibly dangerously misaligned and is also very capable [for some unclear definition of ‘very capable’], don’t build or deploy it”. I think the control plan, which is strictly less conservative, is a better level of conservatism to aim for, because it seems much more achievable given the political situation I expect.

In practice, we implement this conservative assumption by running control evaluations where we give the red team many affordances while they are creating the attack policy—they’re allowed to look at the input distribution, to iterate against the trusted monitor, to use game theory to calculate optimal attack rates—but the red team is not allowed to see the particular inputs where the control evaluation will be run. So the red team can try to make the model good at generating attacks, but the model needs to be able to adapt those attacks to the actual situations it finds itself in.

We initially called these experiments “control evaluations”; my guess is that it was a mistake to use this terminology, and we’ll probably start calling them “black-box plan-conservative control evaluations” or something.

Now, I’ll talk about why I focus on those conservative assumptions I do make.

Hard worlds are plausible

To start with, I think that it’s plausible that those conservative assumptions end up being all true (or true enough that it’s not very important to use techniques that exploit the AI weaknesses arising from the assumptions that are false), at a point where control techniques are still helpful.

Here are some brief arguments, grouped into natural categories of the assumptions I listed earlier:

• The AIs know a lot about their situation (and so know the safety measures employed and the distribution of inputs they’ll get).
  • The AIs are likely to know a lot about the organization where they work, for a few reasons:
    • Pretraining data might say a lot about it (because of news reports, public reporting on safety measures or safety failures, published papers, etc)
    • The models might be specifically trained to know important facts about the organization, so that they’re better able to do their work. For example, they might know details of the organization’s strategic situation, and of its software infrastructure. (By analogy, tech company employees often know that kind of information about the internal workings of their company, and a lot of that information would be costly to hide from them.)
  • The AIs might also be able to infer the safety measures employed, by various means:
    • Some safety measures are implicitly visible to them (e.g. if we use techniques that sometimes replace their actions with the actions of a different model, they’ll probably notice this).
    • They might be able to reason a priori and guess what kinds of safety measures will seem promising, and what hyperparameters these measures should be used with.
• The AIs are able to reason very effectively about the strategy they should employ.
  • My basic argument here is that reasoning through the safety strategy usually doesn’t seem that hard; an IQ 140 person who knows math and game theory can often think of near-optimal strategies in less than an hour for the control techniques I expect to employ, and I think control techniques can add substantial safety against AIs that are able to do an IQ-140-hour’s worth of opaque reasoning.
    • Different routes through which this opaque reasoning could happen:
      • The model could just think this all through in its forward pass. Pretraining incentivizes models to do some types of reasoning inside a single forward pass, and they indeed are able to do some tasks with no chain-of-thought that humans struggle with; see also Let’s Think Dot By Dot.
      • Some type of opaque recurrence: researchers could develop techniques that allow them to train models to reason in continuous, opaque latent spaces. This doesn’t work very well now, but might work at some point, and would be particularly unfortunate for safety.
      • The model could steganographically encode reasoning about attack strategies in its chain of thought. This requires the model to be able to execute on a very sneaky steganography strategy, which seems fairly implausible.
    • In Subversion Strategy Eval, we evaluated AI performance at thinking through what strategy they should employ (that is, this paper is an example of not following our usual strategy of assuming the worst).
  • The AIs might be able to plan very effectively despite not being able to do so in a single forward pass (e.g. because one instance is able to come up with a plan that it writes down in an obfuscated form (perhaps it does this reasoning in a short-lived rogue internal deployment [AF · GW]), and then the model is trained on that plan, and then it’s easy for the AIs to follow the plan). We should obviously try to prevent this, but I think it’s worth trying to have an extra layer of safety.
  • (How is “reasoning about the strategy they should employ” different from “choosing effective actions”? I operationalize this by saying that we’re very permissive with respect to what the red team does while creating the attack policy (e.g. giving the red team a lot of time to think of strategies, and to iterate against trusted models, etc), but we are much less permissive with respect to what the attack policy is allowed to do given a particular input: the attack policy needs to actually generate the action itself.)
• The AIs have some self-knowledge that they can use to win Schelling games against humans—that is, they need to pick coordination signals that other AIs also know about, but that humans don’t notice.
  • See Schelling game evaluations for AI control [LW · GW] for more on why this is important.
  • My take on this is:
    • If AIs are wildly superintelligent, they can probably succeed at Schelling games.
    • If the AIs are merely as smart as top human experts who have an hour to think, I am very unsure how good they’d be at this—picking Schelling points doesn’t seem that objectively hard (e.g. I suspect models can be trained to do it), but there’s also no obvious mechanism by which training would cause them to have the ability to do this. So it could go either way.
    • The AIs might also be able to pass a message between instances that tells them how to collude (at simplest, this message could just name a password). Unlike other secret messages they might want to pass, this message is very short and maybe easy to come up with.

One reason why some of these abilities are more plausible than you might expect is that if we train against scheming, we push probability mass away from incompetent schemers towards competent schemers.

So I think it’s plausible we have a sequence of three situations:

• Schemers that are importantly non-worst-case. I’m excited for research that aims at mitigating risk from misalignment of these earlier models where it’s more reasonable to not make these assumptions.
• Schemers that have all the above properties, and for which control is appropriately conservative.
• Schemers that are way more powerful than these.

I also expect that before we have models that are capable enough that I believe all the above arguments, we’re likely to have models that aren’t so capable but still seem kind of scary. So I’m excited for research that aims at mitigating risk from misalignment of earlier models where it’s more reasonable to not make these assumptions.

Those assumptions are hard to evaluate

Another reason I like making these assumptions is that the assumptions I listed above are harder to test than the property we choose to evaluate instead (whether models can execute on a particular strategy that successfully subverts safety measures), which might just require a relatively more straightforward capability evaluation.

How do we do a capability evaluation of a model, when that model might be scheming? I agree with the basic picture described in When can we trust model evaluations? [AF · GW], and the “How AIs may intentionally perform poorly aka sandbag” section here [LW · GW]. My summary: To determine how good a model is at something, take a dataset of tasks and train it to do those tasks, using a mix of demonstrations and RL. This allows you to evaluate how good a model is at a task like “generate subtly backdoored code”, or “generate answers to questions that look right but aren’t” (as e.g. explored in Prover-Verifier Games improve legibility of language model outputs).

We can fail at accurately estimating capabilities in two ways:

• We might underestimate capabilities if the model can choose to hide them. To prevent underestimation, we need to train the model to perform the task (with i.i.d. fine-tuning or carefully designed RL approaches to overcome potential sandbagging), using tasks where we can verify good performance, generate high-quality demonstrations, and create a smooth learning path from initial to target performance.
• But we might overestimate capabilities if our evaluation process is actually teaching new skills rather than measuring existing ones. To prevent this, we need a large space of possible tasks within the capability domain we're testing, such that the model can't just memorize a few patterns during evaluation, with clear separation between training and evaluation examples.

All of the capabilities I listed at the top of this doc pose substantial problems for this type of evaluation. For example:

• It seems really hard to test how good models are at failing at calibrated rates. It’s fundamentally easy for a neural net to represent the behavior “do X with probability 0.01%” or whatever, so if we train it to fail at the calibrated rate, we will probably succeed.
• I don’t know how to evaluate how good models are at picking Schelling points without training them in a way that might drastically improve their performance at this task, as discussed in Schelling game evaluations for AI control [LW · GW].
• There might not be diverse enough classes of AI control strategies for us to get strong conclusions from model performance on tasks like Subversion Strategy Eval in the future. That is, defeating AI control might just require a small number of insights, and once you’ve trained the model to use those insights, there aren’t other insights that you can easily measure generalization to. (See footnote for some thoughts on ways around this.^[1])

I think that control evaluations, which measure model performance at subverting particular safety measures using a particular provided strategy, have fewer of these difficulties. See here [LW · GW] for discussion. 

So (barring substantial research advances) I don’t think we’ll be able to be very confident in our assessment of any of these model properties, beyond just looking at the model’s general performance and guessing.

Why it’s nice to rely on these assumptions

There are a bunch of related reasons why I think it’s nice to rely on the easier-to-evaluate assumptions.

Making those assumptions makes it methodologically clean to evaluate safety

It’s convenient to evaluate safety with a simple red-team/blue-team game: the red team tries to train the AIs to follow a plan that maximizes their probability of causing a catastrophe, and we measure how often this trained attack policy succeeds. This rule implicitly makes all the assumptions I listed above.

If AI developers are allowed to make safety arguments that rely on some of my assumptions being false, I think it will be much easier for them to make flawed arguments for safety. Assessing all those assumptions is much harder and more conceptually confusing than doing the capability evaluation required to analyze a single proposed attack plan. This complexity creates two risks: developers might either deliberately construct misleading safety arguments, or accidentally cherry-pick from multiple weak experiments where some happened to show safety just by chance.

Relying only on easy-to-evaluate assumptions would allow more clearly-specified and hard-to-game regulatory standards

Suppose regulators wanted to demand that AI developers don’t impose substantial risk of AI takeover. It would be practically convenient for those regulators if they were able to name an evaluation procedure that suffices for establishing safety, that isn’t too onerous, and that doesn’t allow the AI developers to argue for safety based on illegible and difficult-to-interpret evidence. Forcing developers to make conservative assumptions about difficult-to-evaluate properties would be a big help.

This was a large part of my initial enthusiasm for AI control. I’m less excited about it now, because it now seems less plausible that AI companies will be held to such conservative standards (at least if top-expert-obsoleting AI is within four years), both because I’m more pessimistic about strong regulation and all the AI companies consider it overly costly to be conservative in the ways I’ve proposed here.

Another problem with my initial vision here is that this methodology still requires us to do some conceptually fraught evals, as I discuss in "You can't completely escape from methodological messiness" below. So I don’t think there’s as much of a simplicity advantage to this as I used to think.

Kerckhoff’s principle in computer security as an analogy

From Wikipedia:

Kerckhoffs's principle of cryptography was stated by Dutch-born cryptographer Auguste Kerckhoffs in the 19th century. The principle holds that a cryptosystem should be secure, even if everything about the system, except the key, is public knowledge. This concept is widely embraced by cryptographers, in contrast to security through obscurity, which is not.

Kerckhoffs's principle was phrased by American mathematician Claude Shannon as "the enemy knows the system", i.e., "one ought to design systems under the assumption that the enemy will immediately gain full familiarity with them". In that form, it is called Shannon's maxim.

Another formulation by American researcher and professor Steven M. Bellovin is: “In other words — design your system assuming that your opponents know it in detail. (A former official at NSA's National Computer Security Center told me that the standard assumption there was that serial number 1 of any new device was delivered to the Kremlin.)”

This analogy isn’t perfect; in particular, we might be much more able to control the flow of information to our AIs than the US was able to control the flow of information to the Kremlin.

The main downsides of these assumptions

You can’t completely escape from methodological messiness

I’ve argued that relying on these conservative assumptions means that researchers get to do their research with a simpler methodology. This argument would be much stronger if the assumptions I’m advocating had no methodological issues, such that I could say there was a qualitative difference. Unfortunately, this isn’t the case; there are several methodological difficulties that even black-box plan-conservative control evaluations face. I think the most important of these are:

• Trust evaluations: it seems very methodologically messy to determine whether models should be counted as “trusted”, and probably the concept of “trusting” a model is not natural and robust enough that we’ll be able to use it when control is important.
• Ad-hoc adversarial example generation: if we allow the red-team to use adversarial example generation techniques to find universal jailbreaks and teach those to the untrusted models, the attack policies might be much more capable than we should realistically expect.

Bad technique prioritization

If you allocate all your effort based on the evaluation of safety measures you get from a classic control evaluation, you’ll obviously underestimate the value of safety measures that rely on the assumptions being false. I think some such techniques look like they have a really good cost-benefit ratio in expectation, and it would be a shame to not use them. The most important such techniques:

• Untrusted monitoring—it’s a really easy technique that probably works great if the models aren’t able to sneakily collude or (sometimes) don’t have the propensity to collude for whatever reason. See some relevant comments on propensity here [AF(p) · GW(p)].
• Coup probes [LW · GW] and other unprincipled white-box techniques. These should be evaluated using a specialized methodology; I’m hoping we have some work out on these soon.
• Behavioral red-teaming [AF · GW] (which is equivalent to the control protocol we usually call “upfront auditing”).

In my opinion, the right way to handle this (when thinking about dangerously capable models) is to spend most of your time making the conservative assumptions, and then upweight these non-conservatively-evaluated techniques in an ad hoc manner.

Insane game theory requirements to evaluate techniques

In Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats we consider a complicated game theory problem where an AI is trying to choose an optimal rate of attack. Solving this game was tricky: the authors used a complicated numerical optimization routine that required GPU acceleration. It’s basically absurd to think that the AI could solve this problem in its head in order to find the optimal strategy. So it felt a bit silly to spend all this time and money solving for the Nash equilibrium. But it’s hard to know what else to do: in order to evaluate our techniques, we need to make some assumption about what strategy the AI will take if it’s scheming, and I don’t know of any choice nearly as principled (or difficult to accidentally Goodhart) as “assume it picks the best available strategy”.

Conclusion

I think it was a mistake for us to conflate two substantially different ideas under the name “control”:

• Minimizing P(doom | scheming), as discussed in our original post on control [LW · GW].
• Control evaluations that make all the assumptions I listed in the introduction here, and thus are somewhat conservative; I proposed calling them “black-box plan-conservative control evaluations” here.

I wish we had been clearer about this distinction.

I think that if there was political will (either inside AI companies or in government) to force AI companies to legibly demonstrate that they were imposing less than 1% AI takeover risk through their continued AI development, it would be pretty reasonable to force them to make all the conservative assumptions I listed here when they were evaluating safety. Given the current lack of political motivation, I expect (more than 50%) that if AI surpassing human capabilities is developed in the next five years, none of the leading AI companies will implement plans that legible evidence shows would keep catastrophic risks below 1% per year.

I currently think that control evaluations are promising both in the broad sense of “experiments that you do to estimate P(doom | scheming)” and the narrow sense of “black-box plan-conservative control evaluations”. I expect to spend substantial time on both the more and less conservative senses of control over the next year.

 

1. ^{^}

   Potential ways around this:

   • Broaden your training set to less important strategic capabilities
   • Evaluate on analogous capabilities that we think are similarly difficult
   • Draw scaling curves
   • Try harder with your small number of examples: paraphrasing, few-shot prompting, changing hyperparameters, training on reasoning about the insights, etc

0 comments

Comments sorted by top scores.