Why was the AI Alignment community so unprepared for this moment?

Why was the AI Alignment community so unprepared for engaging with the wider world when the moment finally came?

In 2022, I think it was becoming clear that there'd be a huge flood of interest. Why did I think this? Here are some reasons: I've long thought that once MMLU performance crosses a threshold, Google would start to view AI as an existential threat to their search engine, and it seemed like in 2023 that threshold would be crossed. Second, at a rich person's party, there were many highly plugged-in elites who were starting to get much more anxious about AI (this was before ChatGPT), which updated me that the tide may turn soon.

Since I believed the interest would shift so much, I changed how I spent my time a lot in 2022: I started doing substantially less technical work to instead work on outreach and orienting documents. Here are several projects I did, some for targeted for the expert community and some targeted towards the general public:

• We ran an AI arguments writing competition.  After seeing that we could not crowdsource AI risk writing to the community through contests last year, I also started work on An Overview of Catastrophic Risks last winter.  We had a viable draft several in April, but then I decided to restructure it, which required rewriting it and making it longer. This document was partly a synthesis of the submissions from the first round of the AI arguments competition, so fortunately the competition did not go to waste. Apologies the document took so long.
• Last summer and fall, I worked on explaining a different AI risk to a lay audience in Natural Selection Favors AIs over Humans (apparently this doom path polls much better than treacherous turn stories; I held onto the finished paper for months and waited for GPT-4's release before releasing it to have good timing).
• X-Risk Analysis for AI Research tries to systematically articulate how to analyze AI research's relation to x-risk for a technical audience. It was my first go at writing about AI x-risk for the ML research community. I recognize this paper was around a year ahead of its time and maybe I should have held onto it to release it later.
• Finally, after a conversation with Kelsey Piper and the aforementioned party, I was inspired to work on a textbook An Introduction to AI Safety, Ethics, and Society. This is by far the largest writing project I've been a part of.  Currently, the only way to become an AI x-risk expert is to live in Berkeley. I want to reduce this barrier as much as possible, relate AI risk to existing literatures, and let people have a more holistic understanding of AI risk (I think people should have a basic understanding of all of corrigibility, international coordination for AI, deception, etc.).  This book is not an ML PhD topics book; it's more to give generalists good models. The textbook's contents will start to be released section-by-section on a daily basis starting late this month or next month. Normally textbooks take several years to make, so I'm happy this will be out relatively quickly.

One project we only started in 2023 is newsletter, so we can't claim prescience for that.

If you want more AI risk outputs, CAIS is funding-constrained and is currently fundraising for a writer.

I have some sympathy for this sentiment, but I want to point out that the alignment community was tiny until last year and still is small, so many opportunities that are becoming viable now were well below the bar earlier. If you were Rohin Shah, could you do better than finishing your PhD, publishing the Alignment Newsletter starting 2018 and then joining DeepMind in 2020? If you were Rob Miles, could you do better than the YouTube channel starting 2017? As Jason Matheny, could you do better than co-chairing the task force that wrote the US National AI Strategic Plan in 2016, then founding CSET in 2018? As Kelsey Piper, could you do better than writing for Vox starting 2018? Or should we have diverted some purely technical researchers, who are mostly computer science nerds with no particular talent at talking to people, to come up with a media outreach plan?

Keeping in mind that it's not clear in advance what your policy asks are or what arguments you need to counter and so your plan would go stale every couple of years, and that for the last 15 years the economy, health care, etc. have been orders of magnitude more salient to the average person than AGI risk with no signs this would change as suddenly as it did with ChatGPT.

Why was the AI Alignment community so unprepared for engaging with the wider world when the moment finally came?

I reject the premise. Actually, I think public communication has gone pretty dang well since ChatGPT. Not only has AI existential risk become a mainstream, semi-respectable concern (especially among top AI researchers and labs, which count the most!), but this is obviously because of the 20 years of groundwork the rationality and EA communities have laid down.

We had well-funded organizations like CAIS able to get credible mainstream signatories. We've had lots and lots of favorable or at least sympathetic articles in basically every mainstream Western newspaper. Public polling shows that average people are broadly responsive. The UK is funding real AI safety to the tune of millions of dollars. And all this is despite the immediately-preceding public relations catastrophe of FTX!

The only perspective from which you can say there's been utter failure is the Yudkowskian one, where the lack of momentum toward strict international treaties runs spells doom. I grant that this is a reasonable position, but it's not the majority one in the community, so it's hardly a community-wide failure for that not to happen. (And I believe it is a victory of sorts that it's gotten into the Overton window at all.)

I am asking: Why did the Alignment community not prepare tools and plans for convincing the wider infosphere about AI safety years in advance? Prior to the Spring 2023 inflection point.

Why were there no battle plans in the basement of the pentagon that were written for this exact moment?

Interestingly enough, the failure might actually be more epistemic rather than instrumental in this case.

I don't think there has been a widely known discussion about the need to prepare such tools and plans.

And the reasons are complex. On one hand, GPT-3 breakthroughs took most people by surprise (before GPT-3 the consensus was for longer timelines).

On the other hand, timing of GPT-3 was crazy (people were both distracted and disoriented by the realities of Covid (both the pandemic itself, and the social reaction to it); so the cognitive space was in less favorable state than usual).

Why did the Alignment community not prepare tools and plans for convincing the wider infosphere about AI safety years in advance?

I've been organizing the volunteer team who built AI Safety Info for the past two and a half years, alongside building a whole raft of other tools like AI Safety Training and AI Safety World.

But, yes, the movement as a whole has dropped the ball pretty hard on basic prep. The real answer is that things are not done by default, and this subculture has relatively few do-ers compared to thinkers. And the thinkers had very little faith in the wider info-sphere, sometime actively discouraging most do-ers from trying broad outreach.

While I do appreciate the vibe of this post, and think that balls were dropped here, I think you fail to flag the massive hindsight bias going on here - we ended up in one possible world, but there are so many possible worlds we might have incorrectly prepared for!

To paraphrase a superforecaster friend: 'one of the problems with the rationalist community is they don't actually do scenario analysis.'

To quote Bruce Sterling's recent article for Newsweek, "A thousand sci-fi novels and killer robot movies have warned against these monsters for decades. That has scarcely slowed anybody down." 

I think it’s a shame that a lot of people aren’t taking this seriously.

There wasn’t discussion in the community about the possibility of the Overton window suddenly shifting like that.

I guess it wasn’t clear that it’d be important to have stuff ready to go at an instant’s notice rather than there being significant preparation time.

This post comes to mind as relevant: Concentration of Force [LW · GW]

The effectiveness of force application often depends on its concentration—on whether you can amass locally superior force at the actual decisive moment.

This community is doing way better than it has any right to for a bunch of contrarian weirdos with below-average social skills. It's actually astounding.

The US government and broader military-industrial complex is taking existential AI risk somewhat seriously. The head of the RAND Corporation is an existential risk guy who used to work for FHI. 

Apparently the Prime Minister of the UK and various European institutions are concerned as well.

There are x-risk-concerned people at most top universities for AI research and within many of the top commercial labs.

In my experience "normies" are mostly open to simple, robust arguments that AI could be very dangerous if sufficiently capable, so I think the outreach has been sufficiently good on that front.

There is a much more specific set of arguments about advanced AI (exotic decision theories, theories of agency and preferences, computationalism about consciousness) that are harder to explain and defend than the basic AI risk case, so would rhetorically weaken it. But people who like them get very excited about them. Thus I think having a lot more popular materials by LessWrong-ish people would do more harm than good, so it was a good move whether intentional or not to avoid this. (On the other hand if you think these ideas are absolutely crucial considerations without which sensible discussion is impossible, then it is not good.)

My main answer is capacity constrains at central places. I think you are not considering how small the community was.

One somewhat representative anecdote: sometime in ~2019, at FHI, there was a discussion that the "AI ethics" and "AI safety" research communities seem to be victims of unfortunate polarization dynamics, where even while in the Platonic realm of ideas concerns tracked by the people are compatible, there is somewhat unfortunate social dynamic, where loud voices on both sides are extremely dismissive of the other community.  My guess at that time was the divide has decent chance of exploding when AI worries go mainstream (like, arguments about AI risk facing vociferous opposition from part of academia entrenched under the "ethics" flag), and my proposal was to do something about it, as there were some opportunities to pre-empt/heal this, e.g. by supporting people from both camps to visit each others conferences, or writing papers explaining the concerns in a language of the other camp. Overall this was often specific and actionable.  The only problem was ... "who has time to work on this", and the answer was "no one".

If you looked at what senior staff at FHI was working on, the counterfactuals were e.g. Toby Ord writing The Precipice. I think even with the benefit of hindsight, that was clearly more valuable - if today you see UN Security Council discussing AI risk and at least some people in the room have somewhat sane models, it's also because a bunch of people at UN read The Precipice and started to think about xrisk and AI risk.

If you looked at junior people, I was juggling already quite high number of balls, including research on active inference minds and implications for value learning [LW · GW], research on technical problems in comprehensive AI services, organizing academic-friendly Human-aligned AI summer school, organizing Epistea summer experiment [LW · GW], organizing ESPR, participating in a bunch of CFAR things. Even in retrospect, I think all of these bets were better than me trying to do something about the expected harmful AI ethics vs AI safety flamewar.

Similarly, we had an early-stage effort on "robust communication", attempting to design a system for testing robustly good public communication about xrisk and similar sensitive topics (including e.g. developing good shareable models of future problems fitting in the Overton window). It went nowhere because ... there just weren't any people. FHI had dozens of topic like that where a whole org should work on them, but the actual attention was about 0.2FTE of someone junior.

Overall I think with the benefit of hindsight, a lot of what FHI worked on was more or less what you suggest should have been done. It's true that this was never in the spotlight on LessWrong - I guess in 2019 the prevailing LW sentiment would be that Toby Ord engaging with UN is most likely useless waste of time.  

This contest, in mid-2022 [EA · GW], seems like a bit of what you're talking about. I entered it, won some money, made a friend, and went on my merry way. I haven't seen e.g. policymakers (or, uh, even many people on Twitter) use language that reminded me of my more-original winning entries. As I'd expect either way (due to secrecy), I also don't know if any policymakers received and read any new text that won money in the contest.

Every additional person brought into the AI safety community is a liability. The smarter or more wealthy/powerful the new person is, the more capable they are of doing massive damage, intentionally or accidentally, and for an extremely diverse variety of complex causes (especially for smarter people).

Also, there's the nightmare scenario where ~10% of the US population, 30 million people, notice how the possibility of smarter-than-human AI is vastly more worth their attention than anything else going on. Can you imagine what that would look like? I can't. Of that 30 million, probably more than 30,000 will be too unpredictable.

There's just so much that could go horribly wrong. But it looks like we might already be in one of those timelines, which would mean it's a matter of setting things up so that it all doesn't go too badly when the storm hits.

At this risk of looking dumb or ignorant, I feel compelled to ask: Why did this work not start 10 or 15 years ago?

This work did start!  Sort of.  I think your top-level question would be a great one to direct at all the people concerned with AI x-risk who decided DC was the place to be, starting maybe around 2015.  Those people and their funders exist.  It's not obvious what strategies they were pursuing, or to what extent^[1] they had any plans to take advantage of a major shift in the overton window.

1. ^{^}

   From the outside, my impression answer is "not much", but maybe there's more behind-the-scenes work happening to capitalize on all the previous behind-the-scenes work done over the years.

I think until the last few years the common idea was that AGI would be something developed in the metaphorical basement and lead to a singularity in short order, similar to Eliezer's concept of Seed AI. Maybe stuff like AlphaGo was interesting/alarming to us but it seemed to mostly be overlooked by the public and especially government.

It wasn't really clear until ChatGPT that general-ish AI was going to be relevant to the public and government/regulators well before full AGI.

Didn’t we pretty much always know it was going to come from one or a few giant companies or research labs?

Not at all. In that ancient era of early 2000s, the common position was that insight was far more important than resources. So the usual scenario was some bright guy building AI in his basement.

Convincing people about the dangers isn't enough. The important thing is to convince people to take beneficial actions. Nobody has a good plan of how to tackle AI risk, and because nobody has a plan it's impossible to communicate the non-existant plan. 

Without a good plan any communication is going to be hard and feel bad. 

I think part of the reason is it's not really a huge community and there is a lot of discussion and not as much "Doing" 

An excellent question and one I sympathize with.  While it is true that reasonable public statements have been coordinated and gotten widespread coverage, in general the community has been scrambling to explain different aspects of the AI safety/risk case in a way that diverse groups would understand.  Largely, to have it all in one place, in an accessible manner (instead of spread across fora and blog posts and podcasts).

I think it was an underestimation of timelines and a related underfunding of efforts. I started working on a non-technical AI safety book last June and I also think I underestimated timelines.  I hope to have the book out in October because it is urgent but it sure would have been better if I started earlier. 

Given the stakes of all the things we say we care about, I think a large donor should have spent up to $1M several years ago to support the writing of 5-10 accessible books on AI safety, Bio risk, etc (at least 2 per topic), so there would be something on hand that could a) be discussed, b) be distributed, and c) be drawn from for other comms materials. 

This is probably still a good idea. 

People in MIRI/CFAR/LessWrong ~actively resisted the idea of a marketing push optimized more along dimensions of mass persuadability, for better or worse. One reason is that there is inertia once you've built a mass movement with MoPs who can't dialogue like on this site. My straw model is they think "we just need to produce technical insights and communicate them" and other comms work is an opportunity cost or creates/incentivizes some kind of epistemic trap.

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

For me, the most important thing to do is band together and contribute - any small thing we can share - comments, share more content, explain more than before and yeah, If you have the courage - try a crack at the alignment problem.  We can question things, how it went this way or why we are all here with this problem now - but it does not in add anything IMHO. I'd rather we all try our best to contribute instead - however little it is.

“I held onto the finished paper for months and waited for GPT-4's release before releasing it to have good timing).”

So, like, you knew about GPT-4 and how people would be shocked by it, but you’re an awesome PR dude. OK, Marsha.