What an actually pessimistic containment strategy looks like

a strategy like "get existing top AGI researchers to stop"

There's a (hopefully obvious) failure mode where the AGI doomer walks up to the AI capabilities researcher and says “Screw you for hastening the apocalypse. You should join me in opposing knowledge and progress.” Then the AI capabilities researcher responds “No, screw you, and leave me alone”. Not only is this useless, but it's strongly counterproductive: that researcher will now be far more inclined to ignore and reject future outreach efforts (“Oh, pfft, I’ve already heard the argument for that, it’s stupid”), even if those future outreach efforts are better.

So the first step to good outreach is not treating AI capabilities researchers as the enemy. We need to view them as our future allies, and gently win them over to our side by the force of good arguments that meets them where they're at, in a spirit of pedagogy and truth-seeking.

(You can maybe be more direct with someone that they're doing counterproductive capabilities research when they're already sold on AGI doom. That's probably why your conversation at EleutherAI discord went OK.)

(In addition to “it would be directly super-counterproductive”, a second-order reason not to try to sabotage AI capabilities research is that “the kind of people who are attracted to movements that involve sabotaging enemies” has essentially no overlap with “the kind of people who we want to be part of our movement to avoid AGI doom”, in my opinion.)

So I endorse "get existing top AGI researchers to stop" as a good thing in the sense that if I had a magic wand I might wish for it (at least until we make more progress on AGI safety). But that's very different from thinking that people should go out and directly try to do that.

Instead, I think the best approach to "get existing top AGI researchers to stop" is producing good pedagogy, and engaging in gentle, good-faith arguments (as opposed to gotchas) when the subject comes up, and continuing to do the research that may lead to more crisp and rigorous arguments for why AGI doom is likely (if indeed it's likely) (and note that there are reasonable people who have heard and parsed and engaged with all the arguments about AGI doom but still think the probability of doom is <10%).

I do a lot of that kind of activity myself  (1 [LW · GW],2 [LW · GW],3,4 [LW · GW],5, etc.).

So imagine my surprise when I informally learn that this sort of thinking is taboo.

It's not taboo. I've been discussing whether we should do this with various people off and on for the past five years. People take these ideas seriously. Just because people don't agree, or don't take it seriously enough, doesn't mean it's taboo!

FWIW I think it's a good idea too (even though for years I argued against it!). I think it should be done by a well-coordinated group of people who did lots of thinking and planning beforehand (and coordinating with the broader community probably) rather than by lone wolves (for unilateralist's curse reasons.)

Agree that this is definitely a plausible strategy, and that it doesn't get anywhere near as much attention as it seemingly deserves, for reasons unknown to me. Strong upvote for the post, I want to see some serious discussion on this. Some preliminary thoughts:

• How did we get here?
  • If I had to guess, the lack of discussion on this seems likely due to a founder effect. The people pulling the alarm in the early days of AGI safety concerns were disproportionately to the technical/philosophical side rather than to the policy/outreach/activism side. 
  • In early days, focus on the technical problem makes sense. When you are the only person in the world working on AGI, all the delay in the world won't help unless the alignment problem gets solved. But we are working at very different margins nowadays.
  • There's also an obvious trap which makes motivated reasoning really easy. Often, the first thing that occurs when thinking about slowing down AGI development is sabotage - maybe because this feels urgent and drastic? It's an obviously bad idea, and maybe that lets us to motivated stopping.
• Maybe the "technical/policy" dichotomy is keeping us from thinking of obvious ways we could be making the future much safer? The outreach org you propose seems like not really either. Would be interested in brainstorming other major ways to affect the world, but not gonna do that in this comment.
• HEY! FTX! OVER HERE!!
  • You should submit this to the Future Fund's ideas competition, even though it's technically closed. I'm really tempted to do it myself just to make sure it gets done, and very well might submit something in this vein once I've done a more detailed brainstorm.

Not that you said otherwise, but just to be clear: it is not the case that most capabilities researchers at DeepMind or OpenAI have similar beliefs as people at EleutherAI (that alignment is very important to work on). I would not expect it to go well if you said "it seems like you guys are speeding up the deaths of everyone on the planet" at DeepMind.

Obviously there are other possible strategies; I don't mean to say that nothing like this could ever work.

One thing to consider: if we successfully dissuade deepmind researchers from working on AGI, who actually do take alignment issues a little bit seriously, does it instead get developed by meta researchers who (for the sake of argument) don't care?

More generally you're not going to successfully remove everyone from the field. So the people you'll remove will be those who are most concerned about alignment, leaving those who are least concerned to discover AGI.

Thanks a lot for doing this and posting about your experience. I definitely think that nonviolent resistance is a weirdly neglected approach. "mainstream" EA certainly seems against it. I am glad you are getting results and not even that surprised.

You may be interested in discussion here, I made a similar post after meeting yet another AI capabilities researcher at FTX's EA Fellowship (she was a guest, not a fellow): https://forum.effectivealtruism.org/posts/qjsWZJWcvj3ug5Xja/agrippa-s-shortform?commentId=SP7AQahEpy2PBr4XS
 

Are you aware of Effective Altruism's AI governance [? · GW] branch? I didn't look into it in detail myself, but there are definitely dozens of people already working on outreach strategies that they believe to be the most effective. FHI, CSER, AI-FAR, GovAI, and undoubtedly more groups have projects ongoing for outreach, political intervention, etc. with regards to AI Safety. If you want to spend your marginal time on stuff like this, contact them.

It does appear true that the lesswrong/rationalist community is less engaged with this strategy than might be wise, but I'm curious if those organisations would say if people currently working on technical alignment research should switch to governance/activism, and what their opinion is on activism. 80,000 hours places AI technical research above AI governance in their career impact stack, though personal fit plays a major part.

Seems to me like a blindingly obvious post that was kind of outside of the overton window for too long. Eliezer also smashed the window with his TIME article, but this was first, so I think it's still a pretty great post. +4

I think this post makes sense given the premises/arguments that I think many people here accept: that AG(S)I is either amazingly good or amazingly bad, and that getting the good outcome is a priori vastly improbable, and that the work needed to close the gap between that prior and a good posterior is not being done nearly fast enough.

I don't reject those premises/arguments out of hand, but I definitely don't think they're nearly as solid as I think many here do. In my opinion, the variance in goodness of reasonably-thinkable post-AGSI futures is mind-bogglingly large, but it's still probably a bell curve, with greater probability density in the "middle" than in super-heaven or ultra-hell. I also think that just making the world a better place here and now probably usually helps with alignment.

This is probably not the place for debating these premises/arguments; they're the background of this post, not its point. But I do want to say that having a different view on that background is (at least potentially) a valid reason for not buying into the "containment" strategy suggested here.

Again, I think my point here is worthwhile to mention as one part of the answer to the post's question "why don't more people think in terms of containment". I don't think that we're going to resolve whether there's space in between "friendly" and "unfriendly" right here, though.

And the best part of this is that instead of telling people who might already be working in some benign branch of ML that there's this huge problem with AGI, who can potentially defect and go into that branch because it sounds cool, you're already talking to people who, from your perspective, are doing the worst thing in the world. There's no failure mode where some psychopaths are going to go be intrigued by the "power" of turning the world into paperclips. They're already working at DeepMind or OpenAI. Personally, I think that failure mode is overblown, but this is one way you get around it. 

If this is saying that there's no plausible downside, that statement seems incorrect. It's not a very important bit whether or not someone has a narrative of "working on AGI". It takes 2 minutes to put that on a resume, and it would even be defensible if you'd ever done anything with a neural net. More important is the organizing principles of their technical explorations. There's a whole space of possible organizing principles. If you're publicizing your AI capabilities insights, the organizing principle of "tweak and mix algorithms to get gains on benchmarks" is less burning the AGI fuse than the organizing principle of "try new algorithms" is less burning the AGI fuse than the organizing principle of "try to make AGI". To argue that AGI kills you by default, you argue that there are algorithmic inventions to be found that generalize across domains to enable large capability jumps, without having a controllable / understood relationship to goals. Which a fortiori says something about the power of generalization. Which might change how people organize their research. If "X is possible" can be the main insight of a research breakthrough, then "AGI is dangerous" contains "AGI is powerful" contains "power is possible", and could have similar effects.

On another note:

There's other downsides here. I don't think that means this class of strategy should be taboo, and I think this class of strategy absolutely should be worked on. But pursuing a strategy without noticing the downsides, and noticing the ways the strategy is doomed to not actually help at all, is pretty crucial.

Downside: depending on implementation, you turn yourself into an ideological repeater. (This means you probably end up talking to people selected for, at least while they're talking to you, being themselves ideological repeaters, which makes your strategy useless.) So you cause top AGI researchers to be more in an environment filled with ideological repeaters. So you cause it to be correct for top AGI researchers to model people around them as not having coherent perspectives, but instead to be engaging in ideological power struggles. If top AGI researchers are treating people that way, that bodes poorly for mind-changing communication.

An argument like "well, what if you actually radicalize the Iranians into hardening their stance on developing nuclear weapons through all of this discouragement" would also be unsatisfying.

Bruce Bueno de Mesquita's game theoretic computer models predict that this is what happens for the sanctions in Iran. They destroy the domestic opposition to developing nuclear weapons. 

Going a bit meta, the fact that this post seems to receive majority agreement makes me question the degree to which the consensus against it is real (as supposed to a signaling equilibrium). And I also want to mention that I figured from the title that this was about AI boxing, and it's possible that others haven't clicked on it because AI boxing doesn't seem that interesting.

I am surprised that a post with nearly 650 karma doesn't have a review yet. It seems like it should have at least one so it can go through to the voting phase.

What do you think about offering an option to divest from companies developing unsafe AGI? For example, by creating something like an ESG index that would deliberately exclude AGI-developing companies (Meta, Google etc) or just excluding these companies from existing ESGs. 

The impact = making AGI research a liability (being AGI-unsafe costs money) + raising awareness in general (everyone will see AGI-safe & AGI-unsafe options in their pension investment menu + a decision itself will make a noise) + social pressure on AGI researchers (equating them to fossil fuels extracting guys). 

Do you think this is implementable short-term? Is there a shortcut from this post to whoever makes a decisions at BlackRock & Co?

I’m not persuaded by this argument because developing AGI is not purely a negative. AGI could either have a massive positive result (an aligned AI) or a massive negative result (an unaligned AI). Because of winner-takes-all effects, it’s likely that whichever AI is developed first will determine the overall outcome. So one way to decrease the chance of an unfriendly AI is to stop all AI research and another way to do this is to develop a friendly AI first.

We know that multiple agencies worldwide are working on developing AI. Moreover, it’s might be that the advancement of raw computing power will mean that eventually any competent person will be able to create an AGI, at which point it is impossible to directly stop the development of an AGI. So either we need to shut down the development of computing hardware and shut down all AI laboratories globally, or we need to develop a friendly AI first.

Doing the first requires coordinating multiple opposed countries to do something that none of them naively want to do for a reason that will be very difficult to convince politicians of (look at the disagreements even among AI researchers!). This is the climate change problem but on hardcore mode. It is possible to work hard and delay this some, especially since many AI companies are in the Silicon Valley bubble, but it is probably not possible to halt it. We can buy years and maybe decades, but not centuries. The only way to buy centuries is to develop friendly AI first.

We certainly don’t know how to do that yet, but it’s much more likely to be done by people who care about AI safety than to be done by people who don’t care about AI safety.

I don't want to be provocative, but if there was political will to stop AGI research it could probably be stalled for a long time. In order to get that political will, not only in the West but in China as well, a pretty effective way to do it might be figure out a way to use a pre-AGI model to cause mayhem/harm that's bad enough to get the world's attention, while not being apocalyptic. 

As a random example, if AI is used somehow to take down the internet for a few days, the discourse and political urgency regarding AGI would change drastically.  A close analogue is how quickly the world started caring about Gain-of-function-research after Covid. 

I have thought this exact thing for something like 2 years. I do think there are some potential backfire risks which make this an uncertain strategy. 

1. slowing AI progress in the US and Europe, which is where OpenAI, DeepMind, Meta, etc. are located, does not neccesarily mean that AI progress in China slows as well. To the degree that you think US AI research will be more aligned than China's, this differential slowing of AI progress could be net negative.
2. slowing AI progress in legible companies and labs may push AI research into more illegible places, which may be less careful on net. 
3. Raising the salience of AI, which this campaign would require could accelerate AI progress more by making more governments pay attention to it and fund it more, out of paranoia.

All that said, I think far too little serious thinking/planning has gone into this kind of idea, and it should be more seriously considered. I would be happy to discuss this further if you'd like to chat. Send me a DM when you get a chance.

Am I the only one who thinks that the world as it is is unbelievably, fantastically, super bad? The concept of an AI destroying the world would only be bad because it would prevent a very good potential future from coming into existence, that could not otherwise happen. Stopping the AI would remove all hope of it ever happening. 

Iran is an agent, with a constrained amount of critical resources like nuclear engineers, centrifuges, etc.

AI development is a robust, agent-agnostic process [? · GW] that has an unlimited number of researchers working in adjacent areas who could easily cross-train to fill a deficit, an unlimited number of labs which would hire researchers from DeepMind and OpenAI if they closed, and an unlimited amount of GPUs to apply to the problem. 

Probably efforts at getting the second-tier AI labs to take safety more seriously, in order to give the top tier more slack, will move back AI timelines a little? But most of the activities that my brain labels "nonviolent resistance" are the type that will be counterproductive unless there's already a large social movement behind them.

I think the framing of "convince leading AI researchers to willingly work more closely with AI alignment researchers, and think about the problem more themselves" is the better goal. I don't think hampering them generally is particularly useful/effective, and I don't think convincing them entirely to "AGI is very scary" is likely either.

There’s a trap here where the more you think about how to prevent bad outcomes from AGI, the more you realize you need to understand current AI capabilities and limitations, and to do that there is no substitute for developing and trying to improve current AI!

A secondary trap is that preventing unaligned AGI probably will require lots of limited aligned helper AIs which you have to figure out how to build, again pushing you in the direction of improving current AI.

The strategy of “getting top AGI researchers to stop” is a tragedy of the commons: They can be replaced by other researchers with fewer scruples. In principle TotC can be solved, but it’s hard. Assuming that effort succeeds, how feasible would it be to set up a monitoring regime to prevent covert AGI development?

Yeah, I agree, this is not focussed on enough. I was thinking about this sometime back and:

The obvious counter is "there can exist evil players". The counter-counter to that is: well – there can exist evil players (say, terrorists) making nuclear bombs as well. But we don't see that, do we? Why? Because the materials/knowledge/training required to make that is highly protected. 

We could definitely try to replicate that strategy. Make the GPUs a protected item: say, only X for personal use. And cap the amount of units you can buy for commercial purposes as well. It needs governance efforts. Seems very feasible to me. 

But Alignment as an intellectual problem is so very interesting that I don't think I would want to contribute to activism. Maybe I do care more about "working on something interesting" more than "saving the world". Ah, well.

Not all political activism has to be waving flags around and chanting chants. Sometimes activists actually have goals and then accomplish something. I think we should try to learn from those people, as lowly as your opinion might be of them, if we don't seem to have many other options. 

This does make me wonder if activism from scientists has ever worked significantly. https://www.bismarckanalysis.com/Nuclear_Weapons_Development_Case_Study.pdf documents the Manhattan Project, https://www.palladiummag.com/2021/03/16/leo-szilards-failed-quest-to-build-a-ruling-class/ argues that there was partial success.

Thanks for the post! I think asking AI Capabilities researchers to stop is pretty reasonable, but I think we should be especially careful not to alienate the people closest to our side. E.g. consider how the Protestants and Catholics fought even though they agree on so much.

I like focusing on our common ground and using that to win people over. 

Overall, I like the post’s emphasis on taking personal action, conditional on technical alignment being unlikely at the current rate of general-purpose AI development or impossible for fundamental reasons.

Two thoughts that I am happy to elaborate on:

1. Israel and Iran clearly are in an adversarial relationship. We are not in one yet for the most part with AGI researchers and scaleable ML developers. Let’s start by seeking to understand their perspectives, have one-on-one conversations, and find ways to communicate concerns and questions they will need to answer to (under legitimate public pressure).

Any "intervention" you can come up with towards Iranian nuclear researchers is probably liable to make things better and not worse.

2. This seems to suggest ‘try any strategy that might stick for containing AGI development’. I would disagree with this framing. A lot of ways to go about this will be indistinguishable from our community lashing out at another community for random/less noble reasons. You might not have meant it that way though – I admire the way you went about holding conversations in the EleutherAI #ai-alignment Discord Slack.

I wonder how many of us don't want to see AI progress slow down because AI progress keeps proving us right.

After spending at least hundreds of hours reading lesswrong et al. and not being able to alter our path towards AI, I want the satisfaction of telling people "See? Told you so!"

For anyone interested in working on this, you should add yourself on this spreadsheet. https://docs.google.com/spreadsheets/d/1WEsiHjTub9y28DLtGVeWNUyPO6tIm_75bMF1oeqpJpA/edit?usp=sharing

It's very useful for people building such an organisation to know of interested people, and vice versa.

If you don't want to use the spreadsheet, you can also DM me and I'll keep you in the loop privately. 

If you're making such an organisation, please contact me. I'd like to work with you. 

What is this supposed to look like if researchers are actually convinced? The whole raison d'etre for Deepmind is to create AGI. And while AGI may be a world-ending invention, many of the creations along the way are likely to be very economically valuable to Deepmind's parent company, Google.

Take for example this blog where Deepmind describes an application of AI to Google's datacenters that was able to reduce the cooling bill by 40%. What is the economic justification for supporting Deepmind researchers with a minimum salary of $400k if they aren't producing the kind of products that can do this? And can they produce products like this without working on capabilities?

I think it's going to be significantly harder to convince these engineers to stop working on AGI if it means taking a 75% paycut. What's the alternative career path for them?

Strongly agree with this. In my opinion, a more effective intervention in this line would be to side with those who want to curtail the power of tech companies in the USA. For example, breaking up Meta and Google would greatly reduce their ability to fund AI development, and they're currently responsible for most of the boundary-pushing on AGI.

I think direct outreach to the AI labs is a great idea. (If coordinated and well thought out.) Trying to get them to stop building AGI seems unlikely to help IMO, though I'm not totally against it.

I'd be more interested to see targeted efforts aimed at improving safety outcomes from the major AI labs. Things like:

1. Getting more value-aligned people in the AIS community onto the safety teams of DeepMind and OpenAI
2. EA funders offering those labs money to increase headcount on their safety teams
3. Other efforts to try and help tilt the culture of those labs more toward safety or convince their leadership to prioritize safety more highly

(My background assumptions here are short-ish timelines for AGI/TAI and that with high confidence it will be originating from one of a handful of these AI labs.)

This is something I've been thinking about a lot very recently. But as another commenter said, it's probably better to see what the AI governance folks are up to, since this is essentially what they do.

(I learned today that "AI governance" isn't just about what governments should do but also strategy around AI labs, etc.)

Agreeing with your post, I think it might be important to offer the people you want to reach out to a specific alternative what they should work on instead (because otherwise we are basically just telling them to quit their job, which nobody likes to hear). One such alternative would be AI alignment, but maybe that is not optimal for impatient people. I assume that researchers at OpenAI and DeepMind are in it because of the possibilities of advanced AI and that most of them are rather impatient to see them realized. Do you think it would be a good idea to advocate that those to don't want to work on alignment work on shallow AI instead?

I am also thinking of this blog post, arguing that "It’s Our Moral Obligation to Make Data More Accessible" because there is a lot proprietary data out there, which only one company/institution has access to and that stifles innovation (and it's possible to do so while respecting privacy). This also means that there is potentially a lot of data no (or few) shallow, safe ML algorithms have been tried on and that we might be able to get a substantial fraction of the benefits of AGI by just doing more with that data.
There are of course downsides to this. Making data more accessible increases the number of applications of AI and could thus lead to increased funding for AGI development. 

EDIT: Just realized that this is basically the same as number 4 in The case for Doing Something Else [LW · GW]:

Establish international cooperation toward Comprehensive AI Services, i.e., build many narrow AI systems instead of something general

Taking for granted that AGI will kill everybody, and taking for granted that this is bad, it's confusing why we would want to mount costly, yet quite weak, and (poorly) symbolic measures to merely (possibly) slow down research.

Israel's efforts against Iran are a state effort and are not accountable to the law. What is proposed is a ragtag amateur effort against a state orders of magnitude more powerful than Iran. And make no mistake, AGI research is a national interest. It's hard to overstate the width of the chasm. 

Even gaining a few hours is pretty questionable, and a few hours for a billion people may be a big win or it might not. Is a few seconds for a quadrillion people a big win? What happens during that time and after? It's not clear that extending the existence of the human race by what is mostly a trivial amount of time even in the scope of a single life is a big deal even if it's guaranteed.

There is also a pretty good chance that efforts along the lines described may backfire, and spur a doubling-down on AGI research.

Overall this smells like a Pascal's scam. There is a very, very low chance of success against a +EV of debatable size. 

I feel like OP has not read the Unabomber's manifesto, but has reached some of its' conclusions independently.

Please don't try to physically harm AI researchers like the Israelis are alleged to have done to Iranian nuclear / Egyptian rocket scientists. That would spread a lot of misery, and probably not achieve anything you think is good.

I was unaware that the rationalist/less wrong position is one of 'AI Luddites', but I guess it makes sense.

Tangential: I am not convinced that the current Israeli approach of "buying time" until a very likely eventual nuclear confrontation is a good approach. If the current leadership was serious about the state's survival, they would focus on "Iran alignment", not on Iran containment (similar to AI boxing). The difference is that the Iran alignment problem is actually solvable (e.g. by a voluntary withdrawal from West Bank, working on figuring out a two-state solution...).

AI alignment is much harder than "I alignment", and AI boxing is much harder still, so maybe "Dying with Dignity" in this case would mean delaying the inevitable and hoping for a miracle.

Iran is powerful, but Israel is tiny country that can be easily defeated in other ways. 

Nukes are a flex. They cannot be used. If they do use them they might kill more people than just the ones they don't like. Biological weapons are taboo also, but are lot easier to use and can be used to the same effect without much trouble.

So you might ask if Iran is really "saying destroy Israel" do they mean" it "and acting on it or are they just working on having more power?

I don't think Iran can use nukes. Nor will they, nor will they have the ability to deploy them effectively.

I think Nukes are only political weapon. Minus Japan they have always been used as political weapon primarily.

And as long as Nukes work as political weapon they have greater leverage value for leaders than any other weapon. 

The narratives about Iran and Israel are of use too. They too are political weapon. 

Thus Israel talking about nuclear weapons it self is a form of weapon.

Its not clear whether Iran ever develops or even aims to develop nukes. Its merely "accepted as true" regardless of what we know, and we know very little.

Even in case of Japan one could argue the demonstration of power here was more political weapon than military victory. (But that is highly controversial), what is never controversial is that use of nuclear bombs had more than military implications. Which obviously resulted in arms race and cold war. Whether that was good or even intent I cannot say.

AI is not Iran though. Its not human, and while I assume its aimed to be human, its real mission is to combine both human and inhuman qualities. 
Therefore we might find that whatever happens in future is going to be "unknown".

And also the old famous "unknown, unknown". Merely meaning that for example AI might never need to use strategies like game theory at all. 

So the bottom line is its misguided to assume qualities about AI or its research, but being fearful is not wrong.

Its just to fear this is pointless. 

Iran would also like to not get nuked by the United States. Why isn't Iran sabotaging U.S. weapon systems, getting U.S. nuclear engineers to defect, and getting its allies to put sanctions on the U.S. conditional on disarmament?

Sometimes hard power just isn't the right solution, especially when you are a relatively small fish whose main chance at victory comes from a mutualistic relationship, not an antagonistic one.