I like these examples. One thing I'll note, however, is that I think the "warning shot discourse" on LW tends to focus on warning shots that would be convincing to a LW-style audience.

If the theory of the change behind the warning shot requires LW-types (for example, folks at OpenAI/DeepMind/Anthropic who are relatively familiar with AGI xrisk arguments) to become concerned, this makes sense.

But usually, when I think about winning worlds that involve warning shots, I think about government involvement as the main force driving coordination, an end to race dynamics, etc.

[Caveating that my models of the USG and natsec community are still forming, so epistemic status is quite uncertain for the rest of this message, but I figure some degree of speculation could be helpful anyways].

I expect the kinds of warning shots that would be concerning to governments/national security folks will look quite different than the kinds of warning shots that would be convincing to technical experts.

LW-style warning shots tend to be more– for a lack of a better term– rational. They tend to be rooted in actual threat models (e.g., we understand that if an AI can copy its weights, it can create tons of copies and avoid being easily turned off, and we also understand that its general capabilities are sufficiently strong that we may be close to an intelligence explosion or highly capable AGI).

In contrast, without this context, I don't think that "we caught an AI model copying its weights" would necessarily be a warning shot for USG/natsec folks. It could just come across as "oh something weird happened but then the company caught it and fixed it." Instead, I suspect the warning shots that are most relevant to natsec folks might be less "rational", and by that I mean "less rooted in actual AGI threat models but more rooted in intuitive things that seem scary."

Examples of warning shots that I expect USG/natsec people would be more concerned about:

• An AI system can generate novel weapons of mass destruction
• Someone uses an AI system to hack into critical infrastructure or develop new zero-days that impress folks in the intelligence community.
• A sudden increase in the military capabilities of AI

These don't relate as much to misalignment risk or misalignment-focused xrisk threat models. As a result, a disadvantage of these warning shots is that it may be harder to make a convincing case for interventions that focus specifically on misalignment. However, I think they are the kinds of things that might involve a sudden increase in the attention that the USG places on AI/AGI, the amount of resources it invests into understanding national security threats from AI, and its willingness to take major actions to intervene in the current AI race.

As such, in addition to asking questions like "what is the kind of warning shot that would convince me and my friends that we have something dangerous", I think it's worth separately asking "what is the kind of warning shot that would convince natsec folks that something dangerous or important is occurring, regardless of whether or not it connects neatly to AGI risk threat models." 

My impression is that the policy-relevant warning shots will be the most important ones to be prepared for, and the community may (for cultural/social/psychological reasons) be focusing too little effort on trying to prepare for these kinds of "irrational" warning shots. 

I'm definitely not claiming that it is likely we will manage to catch AIs doing such egregiously bad actions, but I think if we did catch them, this would provide an adequate warning shot.

I think if an effective control approach [? · GW] is employed, early transformatively useful AIs are dangerously misaligned, and these early misaligned AIs are unwilling or unable to punt to a later generation of AIs, then catching AIs red-handed [LW · GW] is pretty likely relative to other ways of ensuring safety. I think all of these assumptions I stated as conditions are pretty plausible.

There are weaker versions of these that would probably not provide great warning shots.

I think for some of these cases, it should be possible to heavily study what happened and then determine whether this was actually due to very problematic behavior. You can try to determine what features of the input were important etc. You can even setup sting operations or tripwires based on what you learn.

E.g., in the vulnerability case, you can pretend it actually made it into the code base, but actually it is just used as a tripwire and when the AI tries to take advantage, we catch it then.

More generally, you can always try to play out more steps of a give egregious action which failed.

It is also possible that the scope of evangelists would need to be sufficient to convince people who matter.  Some people who can make decisions might listen to someone with an Exotic-Sounding PhD from Berkeley.  Others who matter might not.  Just as an example, I think some politicians and wealthy powerful types may be more willing to listen to engineers than mathematicians or pure theoreticians.  And a normal engineer might also carry more clout than someone from such exotica as silicon valley communities where people are into open relationships and go to burning man.  

By analogy, some of this is kind of along the lines where sometimes people trust a nurse practitioner more deeply than a doctor.  There may be good/bad reasoning behind that, but for some people it just is what it is.  The rest probably comes down to tribal shibboleths.  But these get important when you want people to hear you.  Remember how little it mattered to many people when "1500 people with PhDs all signed this thing saying climate change is real."  I bet one blue-collar Civil Engineer with the education in hydrology to know exactly what he was talking about, would have been more convincing than 1500 PhDs to that whole tribe.  And there could have been (still could be) a campaign to let that voice be heard rather than dismissing vast swaths of people, including those categories you mentioned above, who would have listened to him.

Politicians in general are typically uninformed about and have difficulty with highly-technical matters, even so far as what we all might consider "basic" frequentist statistics, let alone holes in those models.  Let alone "The model has exfiltrated its own network weights!"

So in some sense, if you want the full weight of government involved, we need people who speak common languages with each of those different types you mentioned: Politicians, Military, Wealthy powerful, the public.

To that end, maybe we should be assembling like minded and smart people to talk about this using different languages and different expertise.  Yes, the people from the think tanks.  But also, people who others can really hear.  Maybe we should develop a structure and culture here on LW to evangelize a *broader pool of types of evangelists.*

A 10-year global pause would allow for a lot of person-years-equivalents of automated AI safety R&D. E.g. from Some thoughts on automating alignment research [LW · GW] (under some assumptions mentioned in the post): 'each month of lead that the leader started out with would correspond to 15,000 human researchers working for 15 months.' And for different assumptions the numbers could be [much] larger still: 'For a model trained with 1000x the compute, over the course of 4 rather than 12 months, you could 100x as many models in parallel.[9] You’d have 1.5 million researchers working for 15 months.'

This would probably obsolete all previous AI safety R&D.

Of course, this assumes you'd be able to use automated AI safety R&D safely and productively. I'm relatively optimistic that a world which would be willing to enforce a 10-year global pause would also invest enough in e.g. a mix of control and superalignment to do this.