Statement on AI Extinction - Signed by AGI Labs, Top Academics, and Many Other Notable Figures

For those who might not have noticed, this actually is historic, they're not just saying that- the top 350 people have effectively "come clean" about this, at once, in a schelling-point **kind-of** way. 

The long years of staying quiet about this and avoiding telling other people your thoughts about AI potentially ending the world, because you're worried that you're crazy or that you take science fiction too seriously- those days **might have** just ended. 

This was a credible signal, none of these 350 high-level people can go back and say "no, I never actually said that AI could cause extinction and AI safety should be a top global priority", and from now on you and anyone else can cite this announcement to back up your views (instead of saying "Bill Gates, Elon Musk, and Stephen Hawking have all endorsed...") and go straight to AI timelines [? · GW] (I like sending people Epoch's Literature review).

EDIT: For the record, this might not be true, or it might not stick, and signatories retain ways of backing out or minimizing their past involvement. I do not endorse unilaterally turning this into more of a schelling point than it was originally intended to be. 

Some notable/famous signatories that I noted: Geoffrey Hinton, Yoshua Bengio, Demis Hassabis (DeepMind CEO), Sam Altman (OpenAI CEO), Dario Amodei (Anthropic CEO), Stuart Russell, Peter Norvig, Eric Horvitz (Chief Scientific Officer at Microsoft), David Chalmers, Daniel Dennett, Bruce Schneier, Andy Clark (the guy who wrote Surfing Uncertainty), Emad Mostaque (Stability AI CEO), Lex Friedman, Sam Harris.

Edited to add: a more detailed listing from this post:

Signatories include notable philosophers, ethicists, legal scholars, economists, physicists, political scientists, pandemic scientists, nuclear scientists, and climate scientists. [...]

Signatories of the statement include:

• The authors of the standard textbook on Artificial Intelligence (Stuart Russell and Peter Norvig)
• Two authors of the standard textbook on Deep Learning (Ian Goodfellow and Yoshua Bengio)
• An author of the standard textbook on Reinforcement Learning (Andrew Barto)
• Three Turing Award winners (Geoffrey Hinton, Yoshua Bengio, and Martin Hellman)
• CEOs of top AI labs: Sam Altman, Demis Hassabis, and Dario Amodei
• Executives from Microsoft, OpenAI, Google, Google DeepMind, and Anthropic
• AI professors from Chinese universities
• The scientists behind famous AI systems such as AlphaGo and every version of GPT (David Silver, Ilya Sutskever)
• The top two most cited computer scientists (Hinton and Bengio), and the most cited scholar in computer security and privacy (Dawn Song)

Is it just me or is it nuts that a statement this obvious could have gone outside the overton window, and is now worth celebrating when it finally (re?)enters?

How is it possible to build a superintelligence at acceptable risk while this kind of thing can happen? What if there are other truths important to safely building a superintelligence, that nobody (or very few) acknowledges because they are outside the overton window?

Now that AI x-risk is finally in the overton window, what's your vote for the most important and obviously true statement that is still outside it (i.e., that almost nobody is willing to say or is interested in saying)? Here are my top candidates:

1. Dying of old age, as well as physical and mental deterioration from it, are bad and worth substantial coordinated effort to prevent.
2. It's possible to make serious irreversible mistakes due to having incorrect answers to important philosophical questions. In fact, this is likely, considering how much confusion and disagreement there is on many philosophical questions that seem obviously important.

Disclaimer: I've never been to an academic conference
EDIT: Also, I'm just thinking out loud here. Not stating my desire to start a conference, just thinking about what can make academics feel like researching alignment is normal.

Those are some big names. I wonder if arranging a big AI safety conference w/ these people would make worrying about alignment feel more socially acceptable to a lot of researchers. It feels to me like a big part of making thinking about alignment socially acceptable is to visibly think about alignment in socially acceptable ways. In my imagination, you have conferences on important problems in academia. 

You talk about the topic there with your colleagues and impressive people. You also go there to catch up with friends, and have a good time. You network. You listen to big names talking about X, and you wonder which of your other colleagues will also talk about X in the open. Dismissing it no longer feels like it will go uncontested. Maybe you should take care when talking about X? Maybe even wonder if it could be true. 

Or on the flip side, you wonder if you can talk about X without your colleagues laughing at you. Maybe other people will back you up when you say X is important. At least, you can imply the big names will. Oh look, a big name X-thinker is coming round the corner. Maybe you can start up a conversation with them in the open. 

There have been some strong criticisms of this statement, notably by Jeremy Howard et al here. I've written a detailed response to the criticisms here:

https://www.soroushjp.com/2023/06/01/yes-avoiding-extinction-from-ai-is-an-urgent-priority-a-response-to-seth-lazar-jeremy-howard-and-arvind-narayanan/

Please feel free to share with others who may find it valuable (e.g. skeptics of AGI x-risk).

I feel somewhat frustrated by execution of this initiative.  As far as I can tell, no new signatures are getting published since at least one day before the public announcement. This means even if I asked someone famous (at least in some subfield or circles) to sign, and the person signed, their name is not on the list, leading to understandable frustration of them.  (I already got a piece of feedback in the direction "the signatories are impressive, but the organization running it seems untrustworthy") 

Also if the statement is intended to serve as a beacon, allowing people who have previously been quiet about AI risk to connect with each other, it's essential for signatures to be published. It's nice that Hinton et al. signed, but for many people in academia it would be actually practically useful to know who from their institution signed - it's unlikely that most people will find collaborators in Hinton, Russell or Hassabis.

I feel even more frustrated because this is second time where similar effort is executed by xrisk community while lacking basic operational competence consisting in the ability to accept and verify signatures. So, I make this humble appeal and offer to the organizers of any future public statements collecting signatures: if you are able to write a good statement and secure the endorsement of some initial high-profile signatories, but lack the ability to accept, verify and publish more than a few hundreds names, please reach out to me - it's not that difficult to find volunteers for this work. 

 

Any explanations for why Nick Bostrom has been absent, arguably notably, in recent public alignment conversations (particularly since chatgpt)?

He's not on this list (yet other FHI members, like Toby Ord, are). He wasn't on the FLI open letter, too, but I could understand why he might've avoided endorsing that letter given its much wider scope.

It's a step, likely one that couldn't be skipped. Still just short of actually acknowledging nontrivial probability of AI-caused human extinction, and the distinction between extinction and lesser global risks, availability of second chances at doing better next time. Nuclear war can't cause extinction, so it's not properly alongside AI x-risk. Engineered pandemics might eventually get extinction-worthy, but even that real risk is less urgent.

skeptical reaction with one expression of support: https://statmodeling.stat.columbia.edu/2023/05/31/jurassic-ai-extinction/

I have to wonder what people — both the signatories and all the people suddenly taking this seriously — have in mind by "risk of extinction". The discussions I've seen have mentioned things like deepfakes, autonomous weapons, designer pathogens, AI leaving us nothing to do, and algorithmic bias. No-one I have heard is talking about Eliezer's "you are made of atoms that the AI wants to use for something else".

What aspect of AI risk is deemed existential by these signatories? I doubt that they all agree on that point. Your publication "An Overview of Catastrophic AI Risks" lists quite a few but doesn't differentiate between theoretical and actual. 

Perhaps if you were to create a spreadsheet with a list of each of the risks mentioned in your paper but with the further identification of each as actual or theoretical, and ask each of those 300 luminaries to rate them in terms of probability, then you'd have something a lot more useful. 

Is this why I cannot get anyone to fund my research on AI interpretability even though I am the only person using spectral radii of tensor products to produce more interpretable and mathematical graph embeddings, word embeddings, and dimensionality reductions? I should have no problems with funding my own research on AI interpretability since I am a cryptocurrency creator, but in reality, most people are quite anti-innovative so they would not invest in anything unless the media tells them to.