FLI open letter: Pause giant AI experiments

I think this letter is not robust enough to people submitting false names. Back when Jacob and I put together DontDoxScottAlexander.com we included this section, and I would recommend doing something pretty similar:

I think someone checking some of these emails would slow down high-profile signatories by 6-48 hours, but sustain trust that the names are all real.

I'm willing to help out if those running it would like, feel free to PM me.

It seems like there's a lot of negative comments about this letter.  Even if it does not go through, it seems very net positive for the reason that it makes explicit an expert position against large language model development due to safety concerns. There's several major effects of this, as it enables scientists, lobbyists, politicians and journalists to refer to this petition to validate their potential work on the risks of AI, it provides a concrete action step towards limiting AGI development, and it incentivizes others to think in the same vein about concrete solutions.

I've tried to formulate a few responses to the criticisms raised:

• "6 months isn't enough to develop the safety techniques they detail": Besides it being at least 6 months, the proposals seem relatively reasonable within something as farsighted as this letter. Shoot for the moon and you might hit the sky, but this time the sky is actually happening and work on many of their proposals is already underway. See e.g. EU AI Act, funding for AI research [AF · GW], concrete auditing work and safety evaluation on models. Several organizations are also working on certification and the scientific work towards watermarking is sort of done? There's also great arguments for ensuring this since right now, we are at the whim of OpenAI management on the safety front.
• "It feels rushed": It might have benefitted from a few reformulations but it does seem alright?
• "OpenAI needs to be at the forefront": Besides others clearly lagging behind already, what we need are insurances that these systems go well, not at the behest of one person. There's also a lot of trust in OpenAI management and however warranted that is, it is still a fully controlled monopoly on our future. If we don't ensure safety, this just seems too optimistic (see also differences between public interview for-profit sama and online sama).
• "It has a negative impact on capabilities researchers": This seems to be an issue from <2020 and some European academia. If public figures like Yoshua cannot change the conversation, then who should? Should we just lean back and hope that they all sort of realize it by themselves? Additionally, the industry researchers from DM and OpenAI I've talked with generally seem to agree that alignment is very important, especially as their management is clearly taking the side of safety.
• "The letter signatures are not validated properly": Yeah, this seems like a miss, though as long as the top 40 names are validated, the negative impacts should be relatively controlled.

All in good faith of course; it's a contentious issue but this letter seems generally positive to me.

AI labs and independent experts should use this pause to jointly develop and implement a set of shared safety protocols for advanced AI design and development that are rigorously audited and overseen by independent outside experts. These protocols should ensure that systems adhering to them are safe beyond a reasonable doubt.

That's nice, but I don't currently believe there are any audits or protocols that can prove future AIs safe "beyond a reasonable doubt".

In parallel, AI developers must work with policymakers to dramatically accelerate development of robust AI governance systems. These should at a minimum include: new and capable regulatory authorities dedicated to AI; oversight and tracking of highly capable AI systems and large pools of computational capability; provenance and watermarking systems to help distinguish real from synthetic and to track model leaks; a robust auditing and certification ecosystem; liability for AI-caused harm; robust public funding for technical AI safety research; and well-resourced institutions for coping with the dramatic economic and political disruptions (especially to democracy) that AI will cause.

Would the authors also like the moon on a stick? :P

I'm not planning to sign it, these lines seem to contain too many things I would not say myself, though I do think that the goal of having a 6-month pause would be pretty good if attained, and I would sign a narrower letter that simply argued for the pause and not some of the rest of this stuff that has been tacked on.

Currently, OpenAI has a clear lead over its competitors.^[1] This is arguably the safest arrangement as far as race dynamics go, because it gives OpenAI some breathing room in case they ever need to slow down later on for safety reasons, and also because their competitors don't necessarily have a strong reason to think they can easily sprint to catch up.

So far as I can tell, this petition would just be asking OpenAI to burn six months of that lead and let other players catch up. That might create a very dangerous race dynamic, where now you have multiple players neck-and-neck, each with a credible claim to have a chance to get into the lead.

(And I'll add: while OpenAI has certainly made decisions I disagree with, at least they actively acknowledge existential safety concerns and have a safety plan and research agenda. I'd much rather they be in the lead than Meta, Baidu, the Chinese government, etc., all of whom to my knowledge have almost no active safety research and in some cases are actively dismissive of the need for such.)

1. ^{^}

   One might have considered Google/DeepMind to be OpenAI's peer, but after the release of Bard — which is substantially behind GPT-3.5 capabilities, never mind GPT-4 — I think this is a hard view to hold.

The letter feels rushed and leaves me with a bunch of questions.

1. "recent months have seen AI labs locked in an out-of-control race to develop and deploy ever more powerful digital minds that no one – not even their creators – can understand, predict, or reliably control." 

Where is the evidence of this "out-of-control race"? Where is the argument that future systems could be dangerous?

2. "Should we let machines flood our information channels with propaganda and untruth? Should we automate away all the jobs, including the fulfilling ones? Should we develop nonhuman minds that might eventually outnumber, outsmart, obsolete and replace us? Should we risk loss of control of our civilization? Such decisions must not be delegated to unelected tech leaders." 

These are very different concerns that water down what the problem is the letter tries to address. Most of them are deployment questions more than development questions.

3. I like the idea of a six-month collaboration between actors. I also like the policy asks they include.

4. The main impact of this letter would obviously be getting the main actors to halt development (OpenAI, Anthropic, DeepMind, MetaAI, Google). Yet, those actors seem not to have been involved in this letter/ haven't publicly commented. (afaik) This seems like a failure.

5. Not making it possible to verify the names is a pretty big mistake.

6. In my perception, the letter mostly appears alarmist at the current time, especially since it doesn't include an argument for why future systems should be dangerous. It might just end up burning political capital.

(cross-posting my take from twitter)

(1) I think very dangerous AGI will come eventually, and that we’re extremely not ready for it, and that we’re making slow but steady progress right now on getting ready, and so I’d much rather it come later than sooner.

(2) It’s hard to be super-confident, but I think the “critical path” towards AGI mostly looks like “research” right now, and mostly looks like “scaling up known techniques” starting in the future but not yet.

(3) I think the big direct effect of a moratorium on “scaling up” is substitution: more “research” than otherwise—which is the opposite of what I want. (E.g. “Oh, we're only allowed X compute / data / params? Cool—let's figure out how to get more capabilities out of X compute / data / params!!”)

(4) I'm sympathetic to the idea that some of the indirect effects of the FLI thing might align with my goals, like “practice for later” or “sending a message” or “reducing AI investments” etc. I’m also sympathetic to the fact that a lot of reasonable people in my field disagree with me on (2). But those aren’t outweighing (3) for me. So for my part, I’m not signing, but I’m also not judging those who do.

(5) While I’m here, I want to more generally advocate that, insofar as we’re concerned about “notkilleveryoneism” (and we should be), we should be talking more about “research” and less about “deployment”. I think that “things that happen within the 4 walls of an R&D department, and on arxiv & github” are the main ingredients in how soon dangerous AGI arrives; and likewise, if someday an AI gets out of control and kills everyone, I expect this specific AI to have never been deliberately “deployed” to the public. This makes me less enthusiastic about certain proposed regulations than some other people in my field, and relatively more enthusiastic about e.g. outreach & dialog with AI researchers […which might lead to them (A) helping with the alignment problem and (B) not contributing to AGI-relevant research & tooling (or at least not publishing / open-sourcing it)].

Further reading: https://80000hours.org/problem-profiles/artificial-intelligence/ & https://alignmentforum.org/posts/rgPxEKFBLpLqJpMBM/response-to-blake-richards-agi-generality-alignment-and-loss & https://www.alignmentforum.org/posts/MCWGCyz2mjtRoWiyP/endgame-safety-for-agi [AF · GW]

I doubt training LLMs can lead to AGI. Fundamental research on the alternative architectures seems to be more dangerous.

I just want to point out that if successful this moratorium would be the fire alarm that otherwise doesn't exist for AGI.

The benefit would be the "industry leaders agree ..." headline, not the actual pause. 

This letter is probably significantly net negative due to its impact on capabilities researchers who don't like it. I don't understand why the authors thought it was a good idea. Perhaps they don't realize that there's no possible enforcement of it that could prevent a GPT4 level model that runs on individual gpus from being trained? to the people who can do that, it's really obvious, and I don't think they're going to stop; but I could imagine them rushing harder if they think The Law is coming after them.

It's especially egregious because of accepting names without personal verification. That will probably amplify the negative response.

This letter seems underhanded and deliberately vague in the worst case, or best case confused.

The one concrete, non-regulatory call for action is, as far as I can tell, "Stop training GPT-5." (I don't know anyone at all who is training a system with more compute than GPT-4, other than OpenAI.) Why stop training GPT-5? It literally doesn't say. Instead, it has the a long suggestive string of rhetorical questions about bad things an AI could cause, without actually accusing GPT-5 of any of them.

Which of them would GPT-5 break? Is it "Should we let machines flood our information channels with propaganda and untruth?" Probably not -- that's only an excuse to keep people with consumer GPUs from getting LLMs, given how OpenAI has been consistent about all sorts of safeguards, and how states could do this pretty easily without LLMs.

Is it "Should we automate away all the jobs, including the fulfilling ones?" That's pretty fucking weird, because GPT-5 is still not gonna take trucking jobs.

Is it "Should we develop nonhuman minds that might eventually outnumber, outsmart, obsolete and replace us?" That's a hell of a prediction about GPT-5's capabilities. -- my timelines aren't nearly that short, and if everyone who signed this letter meant that would be very interesting, but... I think they almost certainly don't. If that one were within the purview of GPT-5, they probably shouldn't train it after six months!

But most of the "shoulds" are just....not likely! What is the actual crisis? Dunno! The letter neither has nor implies a clear model of that. Just that it is Urgent.

If you look at their citations, they cite both existential risk lit and the stochastic parrots paper. The latter is about how models are dumb, the former about the risks of models that are smart.

Then there's a call for a big-ass regulatory presence then -- with mostly an absence of concrete nonprocedural goals or a look at tradeoffs or anything! It looks like "The regulators should regulate everything and do good stuff, and keep bad people from doing bad stuff." If you don't have a model of risk you'll could just throw a soup of lobbyists and regulators at a problem and hope that it works out, but how likely is that?

(And if you wanted to you could call for literally all of the above without stopping GPT-5. They're just fucking unconnected. Why are they tied together in this omnibus petition? Probably so it feels like a Crisis which justifies stepping in.)

Eeesh, just so bad.

Suppose you are one of the first rats introduced onto a pristine island. It is full of yummy plants and you live an idyllic life lounging about, eating, and composing great works of art (you’re one of those rats from The Rats of NIMH).

You live a long life, mate, and have a dozen children. All of them have a dozen children, and so on. In a couple generations, the island has ten thousand rats and has reached its carrying capacity. Now there’s not enough food and space to go around, and a certain percent of each new generation dies in order to keep the population steady at ten thousand.

A certain sect of rats abandons art in order to devote more of their time to scrounging for survival. Each generation, a bit less of this sect dies than members of the mainstream, until after a while, no rat composes any art at all, and any sect of rats who try to bring it back will go extinct within a few generations.

In fact, it’s not just art. Any sect at all that is leaner, meaner, and more survivalist than the mainstream will eventually take over. If one sect of rats altruistically decides to limit its offspring to two per couple in order to decrease overpopulation, that sect will die out, swarmed out of existence by its more numerous enemies. If one sect of rats starts practicing cannibalism, and finds it gives them an advantage over their fellows, it will eventually take over and reach fixation.

If some rat scientists predict that depletion of the island’s nut stores is accelerating at a dangerous rate and they will soon be exhausted completely, a few sects of rats might try to limit their nut consumption to a sustainable level. Those rats will be outcompeted by their more selfish cousins. Eventually the nuts will be exhausted, most of the rats will die off, and the cycle will begin again. Any sect of rats advocating some action to stop the cycle will be outcompeted by their cousins for whom advocating anything is a waste of time that could be used to compete and consume.

Original post, although I think Zvi's post [LW · GW] covers it better. 

Noting that I share my extended thoughts here: https://www.lesswrong.com/posts/FcaE6Q9EcdqyByxzp/on-the-fli-open-letter [LW · GW] 

Reported on the BBC today. It also appeared as the first news headline on BBC Radio 4 at 6pm today

ETA: Also the second item on Radio 4 news at 10pm (second to King Charles' visit to Germany), and the first headline on the midnight news (the royal visit was second).

Edit: I need to understand more context before expressing my opinion.

But but think of the cute widdle baby shoggoths! https://www.reddit.com/r/StableDiffusion/comments/1254aso/sam_altman_with_baby_gpt4/

It seems unlikely that AI labs are going to comply with this petition. Supposing that this is the case, does this petition help, hurt, or have no impact on AI safety, compared to the counterfactual where it doesn't exist?

All possibilities seem plausible to me. Maybe it's ignored so it just doesn't matter. Maybe it burns political capital or establishes a norm of "everyone ignores those silly AI safety people and nothing bad happens". Maybe it raises awareness and does important things for building the AI safety coalition.

Modeling social reality is always hard, but has there been much analysis of what messaging one ought to use here, separate from the question of what policies one ought to want?

Given almost certainty that Russia, China and perhaps some other despotic regimes ignore this does it:

1. help at all?

2. could it actually make the world less safe (If one of these countries gains a significant military AI lead as a result)

Demis Hassabis didn't sign the letter, but has previously said [LW · GW] that DeepMind potentially has a responsibility to hit pause at some point:

'Avengers assembled' for AI Safety: Pause AI development to prove things mathematically

Hannah Fry (17:07):

You said you've got this sort of 20-year prediction and then simultaneously where society is in terms of understanding and grappling with these ideas. Do you think that DeepMind has a responsibility to hit pause at any point?

Demis Hassabis (17:24):

Potentially. I always imagine that as we got closer to the sort of gray zone that you were talking about earlier, the best thing to do might be to pause the pushing of the performance of these systems so that you can analyze down to minute detail exactly and maybe even prove things mathematically about the system so that you know the limits and otherwise of the systems that you're building. At that point I think all the world's greatest minds should probably be thinking about this problem. So that was what I would be advocating to you know the Terence Tao’s of this world, the best mathematicians. Actually I've even talked to him about this—I know you're working on the Riemann hypothesis or something which is the best thing in mathematics but actually this is more pressing. I have this sort of idea of like almost uh ‘Ave

Since nobody outside of OpenAI knows how GPT-4 works, nobody has any idea whether any specific system will be "more powerful than GPT-4". This request is therefore kind of nonsensical. Unless, of course, the letter is specifically targeted at OpenAI and nobody else.

It leaves a bad taste in my mouth that most of the signatories have simply been outcompeted, and would likely not have signed this if they had not been.

More importantly, I find it very unlikely that an international, comprehensive moratorium could be implemented. We had that debate a long time ago already.

What I can envision is OpenAI, and LLMs, being blocked, ultimately in favour of other companies and other AIs. And I increasingly do not think that is a good thing.

Compared with their competitors, I think OpenAI is doing a better job on the safety front, despite all the go fast and break things mentality. I'd be fucking terrified if this was led by Google or Facebook or Musk or China, like I had feared it would be. I think OpenAI has genuinely good people, who want to do a good thing, and are open to listening, and bright. I've mostly been impressed with how well aligned they already got GPT4.

I also think that compared with other AIs, LLMs may have more potential for being raised friendly and collaborative, as we can interact with them the way we do with humans, reusing known recipes. Compared with other forms of extremely large neural nets and machine learning, they are more transparent and accessible. Of all the routes to AGI we could take, I think this might be one of the better ones.

Over the course of the last years, I have become increasingly horrified and alienated at AI. Social media destroying my focus, news algorithms leading to polarisation and anxiety, autocomplete suggesting I limit my vocabulary and nuance, endless data mining and tracking, targeted ads flashing for attention, Google blatantly manipulating everything, China with its dystopian technology, Musk taking over Twitter with his political vision... it all felt like an increasing hellscape, desperately misaligned with my goals and values.

ChatGPT is the first AI that I found beneficial to my productivity and well-being, and found genuinely likeable, leaving me hopeful, optimistic and excited about AI for the first time in a long while.

I think OpenAI and ChatGPT are very bad targets, and yet they are what this moratorium would hit. I fear that would not make us safer.

Would not sign.

I had instead proposed an open letter calling for more AI safety funding.

The LessWrong comments here are generally (quite [LW · GW]) (brutal [LW · GW]), and I think I disagree, which I'll try to outline very briefly below. But I think it may be generally more fruitful here to ask some questions I had to break down the possible subpoints of disagreement as to the goodness of this letter. 

I expected some negative reaction because I know that Elon is generally looked down upon by the EAs that I know, with some solid backing to those claims when it comes to AI given that he cofounded OpenAI, but with the (immediate) (press) (attention) it's getting in combination with some heavy hitting signatures (including Elon Musk, Stuart Russel, Steve Wozniak (Co-founder, Apple), Andrew Yang, Jaan Tallinn (Co-Founder, Skype, CSER, FLI), Max Tegmark (President, FLI), and Tristan Harris (from The Social Dilemma) among many others) I kind of can't really see the overall impact of this letter being net negative. At worst it seems mistimed and with technical issues, but at best it seems one of the better calls to action (or global moratoriums as Greg Colbourn put it [EA(p) · GW(p)]) that could have happened, given AI's current presence in the news and much of the world's psyche.  

But I'm not super certain in anything, and generally came away with a lot of questions, here's a few:

1. How convergent is this specific call for pause on developing strong language models with how AI x-risk people would go about crafting a verifiable, tangible metric for AI labs to follow to reduce risk? Is this to be seen as a good first step? Or something that might actually be close enough to what we want that we could rally around this metric given its endorsement by this influential group?
   1. This helps clarify the "6 months isn't enough to develop the safety techniques they detail" objection which was fairly well addressed here [LW(p) · GW(p)] as well as the "Should Open AI be at the front [LW(p) · GW(p)]" objection.
2. How much should we view messages that are a bit more geared towards non x-risk AI worries than the community seems to be? They ask a lot of good questions here, but they are also still asking "Should we let machines flood our information channels with propaganda and untruth?" an important question, but one that to me seems to deviate away from AI x-risk concerns.  
   1. This is at least tangential to the "This letter felt rushed" objection, because even if you accept it was rushed, the next question is "Well, what's our bar for how good something has to be before it is put out into the world?" 
3. Are open letters with influential signees impactful? This letter at the very least to me seems to be a neutral at worst, quite impactful at best sort of thing, but I have very little to back that, and honestly can't recall any specific time I know of where open letters cause significant change at the global/national level. 
4. Given the recent desire to distance from potentially fraught figures, would that mean shying away from a group wide EA endorsement of such a letter because a wild card like Elon is a part of it? I personally don't think he's at that level, but I know other EAs who would be apt to characterize him that way.
5. Do I sign the post? What is the impact of adding signatures with significantly less professional or social clout to such an open letter? Does it promote the message of AI risk as something that matters to everyone? Or would someone look at "Tristan Williams, Tea Brewer" and think "oh, what is he doing on this list?"