How to evaluate control measures for LLM agents? A trajectory from today to superintelligence

post by Tomek Korbak (tomek-korbak), Mikita Balesni (mykyta-baliesnyi), Buck, Geoffrey Irving · 2025-04-14T16:45:46.584Z · LW · GW · 0 comments

Contents

  What are the advantages of AI control? 
  How can AI control techniques scale to increasingly capable systems? 
  More research is needed 
None
No comments

This is a linkpost accompanying a new paper by UK AI Security Institute, Apollo Research and Redwood Research. Please see the full paper for more details.

TLDR: Our new paper outlines how AI developers should adapt the methodology used in control evaluations as capabilities of LLM agents increase.

Figure: We sketch a trajectory of how control evaluations might evolve through increasingly powerful capability profiles. The trajectory is based on a sequence of five fictional models (M1-M5) with progressively advanced capabilities. For each model, we define an AI Control Level (ACL) based primarily on its threat model-specific capabilities. By mapping these capabilities to appropriate control evaluation methodologies, we provide a systematic approach for adapting safety measures to AI systems as they advance in capabilities.

What are the advantages of AI control? 

The field of AI control  attempts to design and evaluate control protocols: safety measures that are robust to intentional subversion by AI systems. Elements of promising protocols vary widely, from secure sandboxing to AI monitors and human auditing.

A particularly attractive feature of the AI control approach is that it offers the potential to mitigate alignment risks without fundamental research breakthroughs. A major focus of the field is on the engineering challenge of designing protocols that meet the challenges of deployments of increasingly capable autonomous agents.

How can AI control techniques scale to increasingly capable systems? 

While the academic field of AI control is very young with a key foundational paper only published in 2023, many components of control evaluations (e.g. red teaming) and control measures (e.g. human approval) are already used to address the risks posed by existing AI systems.  

As AI systems become more capable of causing harm autonomously, AI developers will need increasingly sophisticated control measures and evaluations to justify that the models can be deployed securely. Our framework provides one possible roadmap for this scaling, proposing five distinct AI control levels (ACLs), ranging from models incapable of competent scheming (ACL-0) to models with scheming capabilities vastly above those of humans (ACL-5). Rather than assuming that agents will always execute the best subversion strategies known to humans, we demonstrate that knowledge of an agents’ actual capability profile can inform proportional control evaluations, resulting in more practical and cost-effective control measures. For each ACL we provide example rules for control evaluation, control measures, and safety cases that could be appropriate.  For instance, at ACL-4, in our example models become capable of latent reasoning which means they no longer rely on chains-of-thought that we could monitor. To account for that, the red team should be allowed to reason about attack strategies upfront and finetune their models to execute any attacks.

Figure: Defining capabilities of ACL-4, example rules of control evaluations they imply, and an example of control measures that might pass such control evaluation. Differences with ACL-3 are highlighted.

Finally, we argue that control measures effective for deployment of models with scheming capabilities vastly above those of humans (ACL-5) will require fundamental research breakthroughs. This suggests the need to move beyond blackbox AI control and exploring safety cases based on whitebox methods and alignment training.

More research is needed 

Our paper provides an early sketch of how AI control measures can scale to increasingly powerful systems, but more research is needed, particularly in the following areas:

• Mapping of model capabilities to control evaluations: detailed justifications of the evaluations needed for systems of different capabilities that go beyond our early outline
• Developing realistic control evaluations: realistic evaluations that can be used to estimate deployment security and iterate on protocol design
• Designing control protocols: procedures that help achieve the best balance between usefulness and safety of increasingly powerful systems

The field of AI control is progressing quickly. Google DeepMind has already outlined a commitment to develop evaluations focused on intentional scheming capabilities. Recent contributions of UK AISI to this agenda include:

• A sketch of an AI control safety case: our paper outlining a structured evidence-based argument for the procedures necessary to securely deploy roughly human level systems in the presence of scheming.
• ControlArena: growing suite of realistic environments for development and evaluation of control protocols suitable for high-stakes coding agents

You can be part of this work: consider applying to our Control team! We’re hiring research scientists and research engineers.‍

0 comments

Comments sorted by top scores.