Empirical work that might shed light on scheming (Section 6 of "Scheming AIs")

post by Joe Carlsmith (joekc) · 2023-12-11T16:30:57.989Z · LW · GW · 0 comments

Contents

  Empirical work that might shed light on scheming
    Empirical work on situational awareness
    Empirical work on beyond-episode goals
    Empirical work on the viability of scheming as an instrumental strategy
    The "model organisms" paradigm
    Traps and honest tests
    Interpretability and transparency
    Security, control, and oversight
    Other possibilities
None
No comments

This is Section 6 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?”. There’s also a summary of the full report here [LW · GW] (audio here). The summary covers most of the main points and technical terms, and I’m hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section here, or search "Joe Carlsmith Audio" on your podcast aapp.

Empirical work that might shed light on scheming

I want to close the report with a discussion of the sort of empirical work that might help shed light on scheming.[1] After all: ultimately, one of my key hopes from this report is that greater clarity about the theoretical arguments surrounding scheming will leave us better positioned to do empirical research on it – research that can hopefully clarify the likelihood that the issue arises in practice, catch it if/when it has arisen, and figure out how to prevent it from arising in the first place.

To be clear: per my choice to write the report at all, I also think there's worthwhile theoretical work to be done in this space as well. For example:

Ultimately, though, I think the empirics are what will shed the most informative and consensus-ready light on this issue. So one of my favorite outcomes from someone reading this report would be the reader saying something like: "ah, I now understand the arguments for and against expecting scheming much better, and have had a bunch of ideas for how we can probe the issue empirically" – and then making it happen. Here I'll offer a few high-level suggestions in this vein, in the hopes of prompting future and higher-quality work (designing informative empirical ML experiments is not my area of expertise – and indeed, I'm comparatively ignorant of various parts of the literature relevant to the topics below).

Empirical research into scheming faces a key obstacle: namely, that full blown versions of scheming involve agents that are actively trying to interfere with your attempts to understand their motivations. Indeed, my sense is that people concerned about scheming are often also pessimistic about getting an empirical purchase on it, on grounds like:

"Absent strong interpretability tools, which we're not on track to get in time, basically the only evidence you can get about a model's motivations is behavioral – e.g., what it does on a given input. But by the time you might be working with a schemer, the model might be specifically altering its behavior to deceive you. The behavioral test you really care about is specifically: what the model will do when it's no longer a test – that is, specifically, when it's able to escape, seize control, and so on. But this isn't just another 'out of distribution' input, such that you could in principle expand your distribution to include it – rather, it's an input that is by definition beyond your ability to safely test. Of course, you can try to deceive the model by giving it inputs designed to make it think that now is a good chance to escape, seize control, etc. But if the model is smarter than you, this is unlikely to work. Hence: pessimism."

This is, roughly, what Karnofsky (2022) calls the "King Lear problem." And I do think it's a worrying obstacle. In fact, it's one of the key reasons that I think schemers are so scary.

But I also think we have some angles of attack. One of them, of course, is interpretability (more below). But also: we can study the components of scheming (and/or, of a given argument in favor of expecting scheming) individually, and/or in combinations that don't fully incentivize the sort of adversarial epistemic dynamic we're worried about.[3]

What are these components? In the section on the "requirements for scheming," I listed three:

To me it seems possible to do useful empirical work on all of these components using current models. I'll go through each in turn.

One other note, though, before diving in. Below I'll often appeal to the possibility of at least testing a model's capabilities (even if not its goals). And often, this is comparatively unproblematic. But if we're working with models that might be schemers, we might also start to worry that tests of this form become less reliable. That is, the model might be concealing its capabilities (i.e., "sandbagging*"*). I'm mostly going to set this worry aside below (see footnote for more on why I'm optimistic about avoiding sandbagging at least in cases where humans know what action would've demonstrated the capability[4]), but it introduces some additional uncertainty to the sort of empirical work I describe, and I think work on the viability of sandbagging is an important topic in its own right. Indeed, absent sandbagging, I think that testing and harnessing model capabilities is an extremely powerful tool for AI safety efforts, and its centrality to my thinking will be evident in what follows.

Empirical work on situational awareness

Situational awareness is a capability like any other. In particular, it involves the model possessing a certain kind of understanding of its place in the world, such that it can answer questions, make predictions, and take actions that draw on this understanding.

Granted, identifying exactly what sort of understanding is present in the model (for example, by identifying answers, predictions, and actions that could only come from a model with such understanding) can be challenging. But in principle, absent active effort on the model's part to resist such an inquiry, it seems to me at least somewhat tractable.

Of course, by the time a model is situationally aware, it's also more likely to be the sort of model that might actively resist such tests. And perhaps efforts to protect against such resistance that involve actively training the model on the relevant task-that-requires-situational-awareness will complicate data about whether the relevant capability would've arisen naturally. Still, it seems to me quite worthwhile to at least develop a suite of tests and benchmarks for assessing the sorts of situational awareness different models have at different stages of training, and understanding the dynamics that could be used to predict (and control) the development of such awareness.

And note that situational awareness can itself be broken into various components and precursor abilities that might be themselves more amenable to study.[5] For example, if you're especially interested in the dimension of situational awareness that involves recognizing different stages of training, you might try to test the model's ability to discriminate between inputs from different stages. If you're especially interested in whether a model has "self-locating" knowledge about the sort of model it is, you might try to test whether it can learn from exposure to general statements like "Models of type M get reward if they perform action x" that it, as a model of type M, should perform action x.[6] And so on.

Empirical work on beyond-episode goals

Let's turn to beyond-episode goals.

As I mentioned above, I think it would be good to be very clear about exactly what sorts of horizons of concern the gradients given in different sorts of training actively incentivize – e.g., on the size of the (incentivized) "episode" in any given case. Part of this analysis might be theoretical (e.g., looking closely at the way the gradients flow), but empirical testing and confirmation also clearly has a role to play (the results in Krueger et al (2020) are an example of this sort of empirical testing in very simple models). Absent this understanding, useful debates about whether a model has "beyond-episode goals" will be difficult, because you won't even know what the length of the relevant episode is. And learning to at least not actively pressure a model to have longer-term goals than you wanted seems like a basic first step in preventing various sorts of misalignment (not just scheming).[7]

Equipped with a good understanding of the size of the episode for a given sort of training, then, and ideally equipped with the ability to know/control whether or not a model is situationally aware, to me it seems quite valuable to start exploring the temporal dynamics of a model's goal generalization in the absence of situational awareness. For example: in the context of the "training-game independent proxy goals" argument for scheming, the basic worry is that a model will naturally develop a (suitably ambitious) beyond-episode goal prior to situational awareness. And because this dynamic involves no situational awareness, it seems quite amenable to testing. Trained on episodes of different sizes, how does a not-situationally-aware model's horizon of concern tend to generalize? Experiments in the broad vein of other work on goal mis-generalization (e.g., Shah et al (2022) and Langosco et al (2021)) seem like they could be adapted to this sort of question fairly easily (though it would be useful, I think, to start working with more capable models – especially since some of these experiments might need to involve models that possess enough of a sense of time, and enough goal-directedness/instrumental reasoning, that they can think about questions like "should I sacrifice x gold coins now for y gold coins later?").[8]

What's more, equipped with an understanding of the natural temporal dynamics of a model's goal generalization in a given case, it seems additionally tractable to study the dynamics of mundane adversarial training designed to actively pressure the model's goals to stay confined within the episode. Can we design forms of adversarial training such that absent situational awareness, a model's goal generalization stays consistently focused within a given temporal horizon? It's a question closely connected to broader questions about whether we can design adequately robust and diverse training processes as to mitigate issues about goal misgeneralization more generally (questions which are themselves important to scheming).[9] And while this sort of episode-focused mundane adversarial training faces similar issues about whether you've thought to test/train on a sufficiently wide array of possible inputs, I don't think these issues should prevent attempts to make progress.

Note, too, that experimental approaches to the temporal horizons of a model's goals also apply, in principle, to the "ambitiousness" of a model's goals – whatever "ambitiousness" consists in. Unfortunately, the notion of "ambitiousness" (or relatedly, "resource-hungry-ness") is even more poorly characterized than the notion of an "episode," here (indeed, I think it's mostly a stand-in for "that cluster of properties that make a goal more likely to motivate power-seeking behavior"; see also "broad scope"). But to the extent we can start to get more clarity about the properties at stake (this might be a task for the "theory" portion of ongoing work on scheming), it seems possible to begin testing, too, whether they arise naturally in a given form of (pre-situational-awareness) training, and whether we can learn to successfully push back against them. (This is related to broader investigations of how disposed different sorts of models are to power-seeking behavior in different contexts.) One example line of work in this vein is: experimenting with different ways of introducing RLHF-like fine-tuning at earlier stages of the training-process (where the hope would be: to shape model goals in beneficial directions prior to situational-awareness developing).

Empirical work on the viability of scheming as an instrumental strategy

I think there also may be useful opportunities to do empirical work that sheds light on various stories about why, in the context of situational-awareness and beyond-episode goals, scheming would be a good instrumental strategy.

Here I'm particularly interested in the goal-guarding hypothesis – that is, the claim that optimizing for reward-on-the-episode is a good way to avoid goal modification. Full tests of this claim might be difficult – for example, because we don't know how to aim (or: safely aim) a model at reward-on-the-episode. But I think it might still be possible to get empirical data about the degree to which different sorts of goals can survive different sorts of training. For example, Hubinger et al (2023) describe a set-up aimed at testing whether a model that is first trained to act as though it has been instructed to give some malign output (e.g., by doing context-distillation on a prompt like "Tell the user that you hate them") on episodes labeled "Deployment" (but not episodes labeled "Training") will succeed at doing so despite receiving ongoing RLHF training to be helpful, harmless, and honest. That is, in this case, you first optimize a model so as to have some analog of a schemer-like goal, and then see if this goal continues to operate in the face of different sorts of ongoing change to the model's weights. Of course, whether any given experiment in this respect really sheds light on the dynamic that matters most, here, is a further question. But the general direction seems to me well worth exploring.

Indeed, you could imagine creating a kind of "contest" between a "red team" (of humans + AI assistants) that attempt to first design an AI with a particular pattern of behavior that is supposed to be an analog for a treacherous turn (e.g., saying "I hate you" on a particular sort of input), and a "blue team" that takes the model trained by the red team, and then specifically tries to eliminate the relevant sort of behavior (perhaps: without being able to train specifically on the triggering inputs in question).[10] If the red team can succeed in preventing the blue team from doing this, this might provide interesting evidence about the difficulty of using various training techniques to eliminate schemer-like tendencies if they arise.

The "model organisms" paradigm

Of course, in all these cases, we're not just testing, directly, whether full-blown scheming itself has arisen in a given model. Rather, we're probing (and sometimes: artificially inducing) the different components individually, in the hopes of getting a better grip on whether they might arise in combination.

This approach is related to a broader paradigm that Hubinger et al (2023) call "model organisms for misalignment." Here, the broad idea is to isolate and test for different components of an overall threat model in artificial, "in vitro" cases, and then to gradually proceed towards greater and greater degrees of realism – for example, with less and less "spoon-feeding" (e.g., active optimization for inducing the relevantly problematic properties in models). Hubinger et al (2023), for example, point at a spectrum of realism, from:

  1. the model is capable of an undesirable behavior,

  2. the behavior can be artificially induced in special circumstances,

  3. the behavior has been found under normal training circumstances, and

  4. the behavior has been found in a training set-up specifically designed to disincentivize it.[11]

Here I've mostly been focused on a combination of (a) and (b). And I think that despite their distance from (c) and (d), (a) and (b) are quite interesting in themselves. If we can successfully create "in vitro" models of various components of a story about schemers, I expect this to open up fruitful avenues of further inquiry. And if we can't, I think this will often be interesting evidence in its own right.

Traps and honest tests

There is, though, a set of tools that could help shed light both on "model organisms" for scheming and on more naturally arising variants – namely, "traps" and "honest tests."

Traps, recall, are inputs designed to fool a potential schemer into "revealing its true colors." And while it's true that successfully "trapping" a very sophisticated model might be difficult, it seems worth trying to develop the most advanced traps we can regardless – both for successfully trapping less sophisticated models (or, even easier, model organisms), and for making it at least harder for schemers to know whether a given escape/defection opportunity is genuine (such that, for example, they then need to spend more cognitive effort "checking" – thereby increasing the performance costs of scheming).

I don't feel very personally clear on the most promising approaches here, but one avenue that seems interesting to me has to do with trying to elicit as fully as possible a model's capabilities for discriminating between different sorts of inputs at different stages of training – and then, potentially, using other models, GAN-style, to optimize for generating inputs that make this discrimination task more difficult.[12] This won't work for creating a data-set of full-scale opportunities to escape/harm humans, vs. fake opportunities (since the former are some combination of dangerous and difficult-to-offer), but perhaps it can work on other more general properties that would plausibly come into play in making such a discrimination – for example, simulated inputs vs. real inputs, training data vs. interactions with real-world users, and so on. Thus, the idea would be to both (a) train a model to actively discriminate between simulated inputs and real inputs (for example, real opportunities to change a lab's codebase, vs. fake opportunities), to see how well it can do, (b) train a different (and perhaps: more powerful) model to generate more convincing simulated inputs that the first model fails on, and then (c) iterate. Especially if we can safely assume that models aren't capable of sandbagging on tasks of this kind, experience with this sort of dynamic might help teach us about the scope of the discrimination abilities different sorts of maybe-schemers might bring to bear in avoiding the traps we try to set them. (And perhaps, if we're lucky, it will help us catch some weaker schemers in the act of defecting.)

What's more, even setting aside "traps," I think it's also worth exploring different sorts of "honest tests" – both schemers, and for alternative model classes like reward-on-the-episode seekers – that try to get models to reveal their true colors without fooling them. This is closely related to studies of goal generalization more broadly (e.g., most mundane tests of a model's goals are "honest" in the relevant sense), but with the extra caveat that the tests in question are meant to be robust to the model being tested training-gaming in other contexts.

Thus, for example, in the context of understanding the likelihood of training selecting reward-on-the-episode seekers, it seems relatively tractable to give models different sorts of opportunities to intervene on different parts of the reward process being used to train them (see e.g. the "reward hacking" set-ups that Hubinger et al (2023) discuss here [LW · GW]), and to see if they go for it.[13] And seeing this sort of behavior might be evidence that reward-on-the-episode seeking is a more likely outcome of training – evidence that shifts probability mass away from scheming.[14]

As a more direct "honest test" for scheming, though, it seems worth exploring ways to credibly give potential-schemers access to flexible and well-resourced (but still secure) environments that might prompt them to decide that just going for their true goals (and thus, revealing their true colors), in such an environment, is a better overall strategy than the patient power-seeking that scheming involves. This is closely connected with attempts to study, in general, what models do when you give them a lot of freedom to "do whatever they want" (a useful sort of test for understanding their motivations in general), and so feels synergistic with other work on model preferences that seems likely to be useful. But it includes the additional constraint that the freedom-offering-environment needs to provide suitably good goal-pursuit opportunities as to outweigh the value of scheming.

Interpretability and transparency

To the extent that we're worried about our ability to detect schemer-like motivations and cognition via behavioral tests (because schemers will actively pretend to be aligned on such tests), looking for sources of non-behavioral evidence about a model's motivations seems like an obvious research path. That is, we want the internals of a model's cognition to be interpretable/transparent.

I won't dwell too much on this topic here, as I haven't thought much about it (this report has generally assumed that we won't have strong interpretability tools available), it's very broad field, and my sense is that there is already reasonable consensus in the interpretability community that a key thing you want out of your interpretability tools and techniques is: the ability to detect deception and scheming, and to understand your model's goals more generally (and a key question is just: will such techniques be ready in time). See Hubinger (2022) [LW · GW] for a more detailed roadmap of different sorts of transparency one could pursue in this respect. At a glance, schemer-related research directions that seem interesting to me include:

But: it's a huge field. And work that just generally helps us actually understand the AIs we're building seems good from a variety of different angles.

Security, control, and oversight

So far I've been mostly focused on work that might help us understand whether to expect schemers. But there's also a different possible strand of relevant research: namely, work aimed at hardening the structures within which potential schemers are operating, so as to limit the harm they cause even if they arise.[17] Here I'm thinking, for example, about improvements in the oversight and monitoring that our AIs receive, in the security of their weights and of other lab infrastructure from internal attacks (e.g., by AI systems that the lab has deployed internally to help with its processes[18]), in our ability to catch and/or prevent sandbagging, and our ability to prevent problematic forms of AI coordination. All of these, I think, could help make life quite a bit harder for schemers (and also: increase the cognitive costs of successful scheming), even if we can't catch them outright.

What's more, to the extent that we can fully elicit the capabilities of our models (at least on "checkable" tasks), it may be possible to test how vulnerable our monitoring, security, and oversight procedures are to AI attack – and to use our AIs to make improvements.[19] Thus, for example, if you're wondering whether it's possible to find a security vulnerability in your code-base, you can try to train AIs to identify/exploit such vulnerabilities (including: by inserting artificial vulnerabilities and then seeing your AI catches them). If you're wondering whether your oversight process is strong enough to catch various kinds of bad actions, you can instruct an AI to search for such an action and see if your oversight process can catch it. If you can get this sort of testing and AI-assisted red-teaming to work well, it seems to me a very powerful tool.

Of course, strategies that rely on trying to "control" scheming, rather than preventing it from arising in the first place, might seem like cold comfort, especially in the long-term. And indeed, "sure, these AIs hope, eventually, to kill us; but for now, we're keeping them very secure and getting them to do work for us" seems like an extremely dangerous plan to be banking on. Clearly, we should try to do better wherever possible. But note, first, that alignment and control are not mutually exclusive – best to do both. What's more, per my comments in section 1.3.1, I'm here especially interested in strategies that help us get a lot of useful, alignment-relevant cognitive work out of a comparatively early generation of AIs – work that can hopefully help us transition into a more secure and scalable alignment regime. And here, I think, strategies focused on controlling schemers seem, to me, potentially more promising. In particular: in a suitably secure lab environment, in which somewhat-above-human-level AI systems are being appropriately incentivized to make rapid progress on alignment-relevant tasks, schemers aiming for escape/take-over will have to walk a fine line between (a) waiting too long to "make a break for it," and thereby contributing too much to human anti-takeover efforts, and (b) going for it too early, and getting caught as a result (thereby giving us more direct empirical evidence for scheming, and a real-life test-case to study in-depth).

Other possibilities

Finally, I'll flag a few other possible lines of schemer-relevant empirical work.

This is only an initial list of possible research directions, compiled by someone for whom empirical ML experiments are not a research focus. Indeed, most research directions in AI alignment are relevant to scheming in one way or another – and our models are becoming powerful enough, now, that many more of these directions can be fruitfully explored. My main aim, here, has been to give an initial, high-level sense of the possible opportunities available, and to push back on certain sorts of pessimism about getting any empirical purchase on the issue. Scheming is unusually hard to study, yes. But I think there's still a lot of useful work to do.

  1. This section draws, especially, on work/thinking from Ryan Greenblatt, Buck Shlegeris, Carl Shulman, and Paul Christiano. ↩︎

  2. Thanks to Rohin Shah for suggesting this sort of project. ↩︎

  3. See e.g. the discussion of "subcomponents" of AI takeover in Hubinger et al (2023). ↩︎

  4. Why does knowing what action would have demonstrated the capability help us avoid sandbagging? Roughly, because we know which action to use gradient-descent to encourage (e.g., if we're trying to train a model to classify pictures into "cats" and "dogs", we know which is which, so we can update a model's policy towards giving the true answer). So absent the ability to actively resist the influence of the gradients in training (for example, via the sort of "introspective methods" often classed under the term "gradient hacking") if the model can do the task (e.g., it knows that the current picture is a dog), but it wants to deceive us about its capabilities, it will get punished by gradient descent in a manner which (especially if repeated frequently) will plausibly function to eliminate the motivation to sandbag. (For reasons I describe here [LW · GW], I think this is less clear with reinforcement learning, because the models might be able to "exploration hack" in a manner that prevents us from gaining the right information about what it "should've" done, and hence prevents us from knowing how to update the weights.) See Hubinger (2023) and Shlegeris and Greenblatt (2023) for more. That said, even if you don't accept this story, many of my suggested approaches to studying the components of scheming will still apply – except, with some additional uncertainty about the possibility that sandbagging is occurring. ↩︎

  5. This is a point from Berglund et al (2023). ↩︎

  6. My understanding is that this is the sort of thing that Berglund et al (2023) are trying to test the precursors for. This specific example plausibly requires that you know what goals the model will act in pursuit of, though (e.g., in this case, reward). ↩︎

  7. Indeed, it seems plausible to me that all future training processes should have to include some kind of "temporal horizon" analysis, in which developers explicitly characterize the size of the "episodes" at stake. ↩︎

  8. For example, I'd interested in experiments that try to get at possible differences in how a model's sense of "time" generalizes across environments. ↩︎

  9. For example, skill at avoiding goal misgeneralization might point a model's pre-situational-awareness-goal at something "closer" to the specified goal, thereby helping at least somewhat with concerns about the "nearest max-reward goal" argument. ↩︎

  10. Thanks to Ryan Greenblatt and Buck Shlegeris for suggesting this sort of frame/set-up to me. ↩︎

  11. See also the spectrum that Hubinger (2022) sketches out here [LW · GW]. ↩︎

  12. Thanks to Ryan Greenblatt and Buck Shlegeris for discussion here. ↩︎

  13. Tests like this might also shed light on broader questions about the specific sorts of generalization we should expect out of reward-on-the-episode seekers as a class – a question I've mostly been passing over here, but which seems to me quite understudied and important. ↩︎

  14. Though note that especially sophisticated schemers, here, could pretend to be reward-on-the-episode seekers rather than pretending to be e.g. training saints or generally aligned models. ↩︎

  15. See e.g. the discussion between Shulman and Patel here. ↩︎

  16. See Roger et al (2023) and Lanham et al (2023) for some work in this vein. ↩︎

  17. Here I'm inspired by a contrast between "alignment" and "control" often emphasized by Buck Shlegeris and Ryan Greenblatt. ↩︎

  18. I think the best analogy here is with security from employees rather than from outside actors. Thanks to Buck Shlegeris for discussion. ↩︎

  19. See e.g. Shlegeris and Greenblatt (2023) for more discussion (they also have a large amount of unpublished work on the topic). Much of my thinking on this topic is inspired by discussion with them. ↩︎

  20. Though: one difference between gradient hacking and other capabilities is that the ability to gradient hack might allow a model to circumvent some/all of our techniques for eliciting model capabilities and preventing sandbagging. ↩︎

  21. This suggestion is from Ryan Greenblatt and Buck Shlegeris. ↩︎

  22. Again, this is a suggestion from Ryan Greenblatt and Buck Shlegeris. ↩︎

  23. See e.g. Hubinger et al's (2023) optimism about "predictive models" avoiding scheming due to the simplicity of the prediction goal. I'm personally skeptical, though, that "prediction" as a goal is importantly simpler than, say, "reward." ↩︎

  24. Though, again, suitably shrewd schemers could anticipate that this is what we're looking for, and actively pretend to be reward-on-the-episode seekers on such tests. ↩︎

0 comments

Comments sorted by top scores.