Pausing AI Developments Isn't Enough. We Need to Shut it All Down by Eliezer Yudkowsky

In the past few weeks I've noticed a significant change in the Overton window of what seems possible to talk about. I think the broad strokes of this article seem basically right, and I agree with most of the details.

I don't expect this to immediately cause AI labs or world governments to join hands and execute a sensibly-executed-moratorium. But I'm hopeful about it paving the way for the next steps towards it. I like that this article, while making an extremely huge ask of the world, spells out exactly how huge an ask is actually needed. 

Many people on hackernews seemed suspicious of the FLI Open Letter because it looks superficially like the losers in a race trying to gain a local political advantage. I like that Eliezer's piece makes it more clear that it's not about that.

I do still plan to sign the FLI Open Letter. If a better open letter comes along, making an ask that is more complete and concrete, I'd sign that as well. I think it's okay to sign open letters that aren't exactly the thing you want to help build momentum and common knowledge of what people think. (I think not-signing-the-letter while arguing for what better letter should be written, similar to what Eliezer did here, also seems like a fine strategy for common knowledge building)

I'd be most interested in an open letter for something like a conditional-commitment (i.e. kickstarter mechanic) for shutting down AI programs IFF some critical mass of other countries and companies shut down AI programs, which states something like:

1. It’d be good if all major governments and AI labs agreed to pause capabilities research indefinitely while we make progress on existential safety issues.
2. Doing this successfully is a complex operation, and requires solving novel technological and political challenges. We agree it’d be very hard, but nonetheless is one of the most important things for humanity to collectively try to do. Business-as-usual politics will not be sufficient.
3. This is not claiming it’d necessarily be good for any one lab to pause unilaterally, but we all agree that if there was a major worldwide plan to pause AI development, we would support that plan.
4. If safe AGI could be developed, it’d be extremely valuable for humanity. We’re not trying to stop progress, we’re just trying to make sure we actually achieve progress, rather than causing catastrophe.

I think that's something that several leading AI lab leaders seem like they should basically support (given their other stated views)

A concise and impactful description of the difficulty we face.

I expect that the message in this article will not truly land with a wider audience (it still doesn't seem to land with all of the LW audience...), but I'm glad to see someone trying.

I would be interested in hearing the initial reactions and questions of readers who were previously unfamiliar with AI x-risk have after reading this article. I'll keep an eye on Twitter, I suppose.

I just want to say that this is very clear argumentation and great rhetoric. Eliezer's writing at its best.

I disagree with AI doomers, not in the sense that I consider it a non-issue, but that my assessment of the risk of ruin is something like 1%, not 10%, let alone the 50%+ that Yudkowsky et al. believe. Moreover, restrictive AI regimes threaten to produce a lot of outcomes things, possibly including the devolution of AI control into a cult (we have a close analogue in post-1950s public opinion towards civilian applications of nuclear power and explosions, which robbed us of Orion Drives amongst other things), what may well be a delay in life extension timelines by years if not decades that results in 100Ms-1Bs of avoidable deaths (this is not just my supposition, but that of Aubrey de Grey as well, who has recently commented on Twitter that AI is already bringing LEV timelines forwards), and even outright technological stagnation (nobody has yet canceled secular dysgenic trends in genomic IQ). I leave unmentioned the extreme geopolitical risks from "GPU imperialism".

While I am quite irrelevant, this is not a marginal viewpoint - it's probably pretty mainstream within e/acc, for instance - and one that has to be countered if Yudkowsky's extreme and far-reaching proposals are to have any chance of reaching public and international acceptance. The "bribe" I require is several OOMs more money invested into radical life extension research (personally I have no more wish to die of a heart attack than to get turned into paperclips) and into the genomics of IQ and other non-AI ways of augmenting collective global IQ such as neural augmentation and animal uplift (to prevent long-term idiocracy scenarios). I will be willing to support restrictive AI regimes under these conditions if against my better judgment, but if there are no such concessions, it will have to just be open and overt opposition.

I think there's an important meta-level point to notice about this article.

This is the discussion that the AI research and AI alignment communities have been having for years. Some agree, some disagree, but the 'agree' camp is not exactly small.  Until this week, all of this was unknown to most of the general public, and unknown to anyone who could plausibly claim to be a world leader.

When I say it was unknown, I don't mean that they disagreed. To disagree with something, at the very least you have to know that there is something out there to disagree with. In fact they had no idea this debate existed. Because it's very hard to notice the implications of upcoming technologiy when you're a 65 year old politician in DC rather than a 25 year old software engineer in SF. But also because many people and many orgs made the explicit decision to not do public outreach, to not try to make the situation legible to laypeople, to not look like people playing with the stakes we have in fact been playing with.

I do not think lies were told, exactly, but I think the world was deceived. I think the phrasing of the FLI open letter was phrased so as to continue that deception, and that the phrasing was the output of a political calculation.

By contrast, I think Eliezer's Time article was honest. It tells people that part that matters to them: that their life is on the line. That the current situation is shocking. That this isn't a hypothetical anymore.

Yes, lots of people will disagree with what it says, in various places. People who think their alignment technique will work. People who think AI is further in the future. And a dozen dumber disagreements that I will not mention. But people can't evaluate those disagreements without having the base model, can't evaluate the sides of a debate they don't know is happening.

There simply don't exist arguments with the level of rigor needed to justify a claim such as this one without any accompanying uncertainty:

If we go ahead on this everyone will die, including children who did not choose this and did not do anything wrong.

I think this passage, meanwhile, rather misrepresents the situation to a typical reader:

When the insider conversation is about the grief of seeing your daughter lose her first tooth, and thinking she’s not going to get a chance to grow up, I believe we are past the point of playing political chess about a six-month moratorium.

This isn't "the insider conversation". It's (the partner of) one particular insider, who exists on the absolute extreme end of what insiders think, especially if we restrict ourselves to those actively engaged with research in the last several years. A typical reader could easily come away from that passage thinking otherwise.

takes a deep breath

(Epistemic status: vague, ill-formed first impressions.)

So that's what we're doing, huh? I suppose EY/MIRI has reached the point where worrying about memetics / optics has become largely a non-concern, in favor of BROADCASTING TO THE WORLD JUST HOW FUCKED WE ARE

I have... complicated thoughts about this. My object-level read of the likely consequences is that I have no idea what the object-level consequences are likely to be, other than that this basically seems to be an attempt at heaving a gigantic rock through the Overton window, for good or for ill. (Maybe AI alignment becomes politicized as a result of this? But perhaps it already has been! And even if not, maybe politicizing it will at least raise awareness, so that it might become a cause area with similar notoriety as e.g. global warming—which appears to have at least succeeded in making token efforts to reduce greenhouse emissions?)

I just don't know. This seems like a very off-distribution move from Eliezer—which I suspect is in large part the point: when your model predicts doom by default, you go off-distribution in search of higher-variance regions of outcome space. So I suppose from his viewpoint, this action does make some sense; I am (however) vaguely annoyed on behalf of other alignment teams, whose jobs I at least mildly predict will get harder as a result of this.

One quote I expect to be potentially inflammatory / controversial:

Make immediate multinational agreements to prevent the prohibited activities from moving elsewhere. Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.

I'll remark that this is not in any way a call for violence or even military escalation.

Multinational treaties (about nukes, chemical weapons, national borders, whatever), with clear boundaries and understanding of how they will be enforced on all sides, are generally understood as a good way of decreasing the likelihood of conflicts over these issues escalating to actual shooting.

Of course, potential treaty violations should be interpreted charitably, but enforced firmly according to their terms, if you want your treaties to actually mean anything. This has not always happened for historical treaties, but my gut sense is that on the balance, the existence of multinational treaties has been a net-positive in reducing global conflict.

Fox News’ Peter Doocy uses all his time at the White House press briefing to ask about an assessment that “literally everyone on Earth will die” because of artificial intelligence: “It sounds crazy, but is it?”

https://twitter.com/therecount/status/1641526864626720774

Here's a comment from r/controlproblem with feedback on this article (plus tips for outreach in general) that I thought was very helpful.

Where's the lie?

More generally, if this is the least radical policy that Eliezer thinks would actually work, then this is the policy that he and others who believe the same thing should be advocating for in public circles, and they should refuse to moderate a single step. You don't dramatically widen the overton window in <5 years by arguing incrementally inside of it.

Is this now on the radar of national security agencies and the UN Security Council? Is it being properly discussed inside the US government? If not, are meetings being set up? Would be good if someone in the know could give an indication (I hope Yudkowsky is busy talking to lots of important people!)

Jeff Bezos has now followed Eliezer on Twitter: https://twitter.com/bigtechalert/status/1641659849539833856?s=46&t=YyfxSdhuFYbTafD4D1cE9A

[EDIT: fallenpegasus points out that there's a low bar to entry to this corner of TIME's website. I have to say I should have been confused that even now they let Eliezer write in his own idiom.]

The Eliezer of 2010 had no shot of being directly published (instead of featured in an interview that at best paints him as a curiosity) in TIME of 2010. I'm not sure about 2020.

I wonder at what point the threshold of "admitting it's at least okay to discuss Eliezer's viewpoint at face value" was crossed for the editors of TIME. I fear the answer is "last month".

Eliezer had a response on twitter to the criticism of "calling for violence"

The great political writers who also aspired to be good human beings, from George Orwell on the left to Robert Heinlein on the right, taught me to acknowledge in my writing that politics rests on force. 

George Orwell considered it a tactic of totalitarianism, that bullet-riddled bodies and mass graves were often described in vague euphemisms; that in this way brutal policies gained public support without their prices being justified, by hiding those prices. 

Robert Heinlein thought it beneath a citizen's dignity to pretend that, if they bore no gun, they were morally superior to the police officers and soldiers who bore guns to defend their law and their peace; Heinlein, both metaphorically and literally, thought that if you eat meat—and he was not a vegetarian—you ought to be willing to visit a farm and try personally slaughtering a chicken. 

When you pass a law, it means that people who defy the law go to jail; and if they try to escape jail they'll be shot. When you advocate an international treaty, if you want that treaty to be effective, it may mean sanctions that will starve families, or a shooting war that kills people outright. 

To threaten these things, but end up not having to do them, is not very morally distinct—*I* would say—from doing them. I admit this puts me more on the Heinlein than on the Orwell side of things. Orwell, I think, probably considers it very morally different if you have a society with a tax system and most people pay the taxes and very few actually go to jail. Orwell is more sensitive to the count of actual dead bodies—or people impoverished by taxation or regulation, where Orwell acknowledges and cares when that *actually* happens. Orwell, I think, has a point. But I also think Heinlein has a point. I claim that makes me a centrist. 

Either way, neither Heinlein nor Orwell thought that laws and treaties and *wars* were never worth it. They just wanted us to be honest about the cost. 

Every person who pretends to be a libertarian—I cannot see them even pretending to be liberals—who quoted my call for law and treaty as a call for "violence", because I was frank in writing about the cost, ought to be ashamed of themselves for punishing compliance with Orwell and Heinlein's rule. 

You can argue that the treaty and law I proposed is not worth its cost in force; my being frank about that cost is intended to help *honest* arguers make that counterargument. To pretend that calling for treaty and law is VIOLENCE!! is hysteria. It doesn't just punish compliance with the Heinlein/Orwell protocol, it plays into the widespread depiction of libertarians as hysterical. (To be clear, a lot of libertarians—and socialists, and centrists, and whoever—are in fact hysterical, especially on Twitter.) It may even encourage actual terrorism. 

But is it *not* "violence", if in the end you need guns and airstrikes to enforce the law and treaty? And here I answer: there's an *actually* important distinction between lawful force and unlawful force, which is not always of itself the distinction between Right and Wrong, but which is a real and important distinction. The common and ordinary usage of the word "violence" often points to that distinction. When somebody says "I do not endorse the use of violence" they do not, in common usage and common sense, mean, "I don't think people should be allowed to punch a mugger attacking them" or even "Ban all taxation." 

Which, again, is not to say that all lawful force is good and all unlawful force is bad. You can make a case for John Brown (of John Brown's Body). 

But in fact I don't endorse shooting somebody on a city council who's enforcing NIMBY regulations. I think NIMBY laws are wrong. I think it's important to admit that law is ultimately backed by force. 

But lawful force. And yes, that matters. That's why it's harmful to society if you shoot the city councilor— 

—and a *misuse of language* if the shooter then says, "They were being violent!"

I'm getting reports that Time Magazine's website is paywalled for some people e.g. in certain states or countries or something. Here is the full text of the article:

An open letter published today calls for “all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4.”

This 6-month moratorium would be better than no moratorium. I have respect for everyone who stepped up and signed it. It’s an improvement on the margin.

I refrained from signing because I think the letter is understating the seriousness of the situation and asking for too little to solve it.

The key issue is not “human-competitive” intelligence (as the open letter puts it); it’s what happens after AI gets to smarter-than-human intelligence. Key thresholds there may not be obvious, we definitely can’t calculate in advance what happens when, and it currently seems imaginable that a research lab would cross critical lines without noticing.

Many researchers steeped in these issues [LW · GW], including myself, expect [LW · GW] that the most likely result of building a superhumanly smart AI, under anything remotely like the current circumstances, is that literally everyone on Earth will die. Not as in “maybe possibly some remote chance,” but as in “that is the obvious thing that would happen.” It’s not that you can’t, in principle, survive creating something much smarter than you; it’s that it would require precision and preparation and new scientific insights, and probably not having AI systems composed of giant inscrutable arrays of fractional numbers.

Without that precision and preparation, the most likely outcome is AI that does not do what we want, and does not care for us nor for sentient life in general. That kind of caring is something that could in principle be imbued into an AI but we are not ready and do not currently know how.

Absent that caring, we get “the AI does not love you, nor does it hate you, and you are made of atoms it can use for something else.”

The likely result of humanity facing down an opposed superhuman intelligence is a total loss. Valid metaphors include “a 10-year-old trying to play chess against Stockfish 15”, “the 11th century trying to fight the 21st century,” and “Australopithecus trying to fight Homo sapiens“.

To visualize a hostile superhuman AI, don’t imagine a lifeless book-smart thinker dwelling inside the internet and sending ill-intentioned emails. Visualize an entire alien civilization, thinking at millions of times human speeds, initially confined to computers—in a world of creatures that are, from its perspective, very stupid and very slow. A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing.

If somebody builds a too-powerful AI, under present conditions, I expect that every single member of the human species and all biological life on Earth dies shortly thereafter.

There’s no proposed plan for how we could do any such thing and survive. OpenAI’s openly declared intention is to make some future AI do our AI alignment homework. Just hearing that this is the plan ought to be enough to get any sensible person to panic. The other leading AI lab, DeepMind, has no plan at all.

An aside: None of this danger depends on whether or not AIs are or can be conscious; it’s intrinsic to the notion of powerful cognitive systems that optimize hard and calculate outputs that meet sufficiently complicated outcome criteria. With that said, I’d be remiss in my moral duties as a human if I didn’t also mention that we have no idea how to determine whether AI systems are aware of themselves—since we have no idea how to decode anything that goes on in the giant inscrutable arrays—and therefore we may at some point inadvertently create digital minds which are truly conscious and ought to have rights and shouldn’t be owned.

The rule that most people aware of these issues would have endorsed 50 years earlier, was that if an AI system can speak fluently and says it’s self-aware and demands human rights, that ought to be a hard stop on people just casually owning that AI and using it past that point. We already blew past that old line in the sand. And that was probably correct; I agree that current AIs are probably just imitating talk of self-awareness from their training data. But I mark that, with how little insight we have into these systems’ internals, we do not actually know.

If that’s our state of ignorance for GPT-4, and GPT-5 is the same size of giant capability step as from GPT-3 to GPT-4, I think we’ll no longer be able to justifiably say “probably not self-aware” if we let people make GPT-5s. It’ll just be “I don’t know; nobody knows.” If you can’t be sure whether you’re creating a self-aware AI, this is alarming not just because of the moral implications of the “self-aware” part, but because being unsure means you have no idea what you are doing and that is dangerous and you should stop.

---

On Feb. 7, Satya Nadella, CEO of Microsoft, publicly gloated that the new Bing would make Google “come out and show that they can dance.” “I want people to know that we made them dance,” he said.

This is not how the CEO of Microsoft talks in a sane world. It shows an overwhelming gap between how seriously we are taking the problem, and how seriously we needed to take the problem starting 30 years ago.

We are not going to bridge that gap in six months.

It took more than 60 years between when the notion of Artificial Intelligence was first proposed and studied, and for us to reach today’s capabilities. Solving safety of superhuman intelligence—not perfect safety, safety in the sense of “not killing literally everyone”—could very reasonably take at least half that long. And the thing about trying this with superhuman intelligence is that if you get that wrong on the first try, you do not get to learn from your mistakes, because you are dead. Humanity does not learn from the mistake and dust itself off and try again, as in other challenges we’ve overcome in our history, because we are all gone.

Trying to get anything right on the first really critical try is an extraordinary ask, in science and in engineering. We are not coming in with anything like the approach that would be required to do it successfully. If we held anything in the nascent field of Artificial General Intelligence to the lesser standards of engineering rigor that apply to a bridge meant to carry a couple of thousand cars, the entire field would be shut down tomorrow.

We are not prepared. We are not on course to be prepared in any reasonable time window. There is no plan. Progress in AI capabilities is running vastly, vastly ahead of progress in AI alignment or even progress in understanding what the hell is going on inside those systems. If we actually do this, we are all going to die.

Many researchers working on these systems think that we’re plunging toward a catastrophe, with more of them daring to say it in private than in public; but they think that they can’t unilaterally stop the forward plunge, that others will go on even if they personally quit their jobs. And so they all think they might as well keep going. This is a stupid state of affairs, and an undignified way for Earth to die, and the rest of humanity ought to step in at this point and help the industry solve its collective action problem.

---

Some of my friends have recently reported to me that when people outside the AI industry hear about extinction risk from Artificial General Intelligence for the first time, their reaction is “maybe we should not build AGI, then.”

Hearing this gave me a tiny flash of hope, because it’s a simpler, more sensible, and frankly saner reaction than I’ve been hearing over the last 20 years of trying to get anyone in the industry to take things seriously. Anyone talking that sanely deserves to hear how bad the situation actually is, and not be told that a six-month moratorium is going to fix it.

On March 16, my partner sent me this email. (She later gave me permission to excerpt it here.)

“Nina lost a tooth! In the usual way that children do, not out of carelessness! Seeing GPT4 blow away those standardized tests on the same day that Nina hit a childhood milestone brought an emotional surge that swept me off my feet for a minute. It’s all going too fast. I worry that sharing this will heighten your own grief, but I’d rather be known to you than for each of us to suffer alone.”

When the insider conversation is about the grief of seeing your daughter lose her first tooth, and thinking she’s not going to get a chance to grow up, I believe we are past the point of playing political chess about a six-month moratorium.

If there was a plan for Earth to survive, if only we passed a six-month moratorium, I would back that plan. There isn’t any such plan.

Here’s what would actually need to be done:

The moratorium on new large training runs needs to be indefinite and worldwide. There can be no exceptions, including for governments or militaries. If the policy starts with the U.S., then China needs to see that the U.S. is not seeking an advantage but rather trying to prevent a horrifically dangerous technology which can have no true owner and which will kill everyone in the U.S. and in China and on Earth. If I had infinite freedom to write laws, I might carve out a single exception for AIs being trained solely to solve problems in biology and biotechnology, not trained on text from the internet, and not to the level where they start talking or planning; but if that was remotely complicating the issue I would immediately jettison that proposal and say to just shut it all down.

Shut down all the large GPU clusters (the large computer farms where the most powerful AIs are refined). Shut down all the large training runs. Put a ceiling on how much computing power anyone is allowed to use in training an AI system, and move it downward over the coming years to compensate for more efficient training algorithms. No exceptions for governments and militaries. Make immediate multinational agreements to prevent the prohibited activities from moving elsewhere. Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.

Frame nothing as a conflict between national interests, have it clear that anyone talking of arms races is a fool. That we all live or die as one, in this, is not a policy but a fact of nature. Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs.

That’s the kind of policy change that would cause my partner and I to hold each other, and say to each other that a miracle happened, and now there’s a chance that maybe Nina will live. The sane people hearing about this for the first time and sensibly saying “maybe we should not” deserve to hear, honestly, what it would take to have that happen. And when your policy ask is that large, the only way it goes through is if policymakers realize that if they conduct business as usual, and do what’s politically easy, that means their own kids are going to die too.

Shut it all down.

We are not ready. We are not on track to be significantly readier in the foreseeable future. If we go ahead on this everyone will die, including children who did not choose this and did not do anything wrong.

Shut it down.

I'll note (because some commenters seem to miss this) that Eliezer is writing in a convincing style for a non-technical audience. Obviously the debates he would have with technical AI safety people are different then what is most useful to say to the general population.

• If we held anything in the nascent field of Artificial General Intelligence to the lesser standards of engineering rigor that apply to a bridge meant to carry a couple of thousand cars, the entire field would be shut down tomorrow.

What are examples that can help to see this tie more clearly? Procedures that works similarly enough to say "we do X during planning and building a bridge and if we do X in AI building...". Is there are even exist such X that can be applied to enginering a bridge and enginering an AI? 

Eliezer's repeated claim that we have literally no idea about what goes on in AI because they're inscrutable piles of numbers is untrue and he must know that. There have been a number of papers and LW posts giving at least partial analysis of neural networks, learning how they work and how to control them at a fine grained level, etc. That he keeps on saying this without caveat casts doubt on his ability or willingness to update on new evidence on this issue.

Doesn't the prisoner's dilemma (esp. in the military context) inevitably lead us to further development of AI?  If so, it would seem that focusing attention and effort on developing AI as safely as possible is a more practical and worthwhile issue than any attempt to halt such development altogether.

I think the harsh truth is that no one cared about Nuclear Weapons until Hiroshima was bombed. The concept of one nation "disarming" AI would never be appreciated until somebody gets burned.

Do you remember the end of Watchmen? 

To visualize a hostile superhuman AI, don’t imagine a lifeless book-smart thinker dwelling inside the internet and sending ill-intentioned emails. Visualize an entire alien civilization, thinking at millions of times human speeds, initially confined to computers—in a world of creatures that are, from its perspective, very stupid and very slow. A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing.

If somebody builds a too-powerful AI, under present conditions, I expect that every single member of the human species and all biological life on Earth dies shortly thereafter.

Have you take a look at the mortality trends?

https://mpidr.shinyapps.io/stmortality/

As Pfizer scientists raced to develop their COVID-19 vaccine at record-breaking speed these past few months, they turned to an innovative artificial intelligence (AI) tool to help achieve this mission.

https://www.pfizer.com/news/articles/how_a_novel_incubation_sandbox_helped_speed_up_data_analysis_in_pfizer_s_covid_19_vaccine_trial#:~:text=As%20Pfizer%20scientists%20raced%20to,to%20help%20achieve%20this%20mission.

"Do it?" Dan, I'm not a Republic serial villain. Do you seriously think I'd explain my master-stroke if there remained the slightest chance of you affecting its outcome? I "did it" 35 minutes ago.

That's not an "article in Time".  That's a "TIME Ideas" contribution.  It has less weight and less vetting than any given popular substack blog.

I’ve seen pretty uniform praise from rationalist audiences, so I thought it worth mentioning that the prevailing response I’ve seen from within a leading lab working on AGI is that Eliezer came off as an unhinged lunatic.

For lack of a better way of saying it, folks not enmeshed within the rat tradition—i.e., normies—do not typically respond well to calls to drop bombs on things, even if such a call is a perfectly rational deduction from the underlying premises of the argument. Eliezer either knew that the entire response to the essay would be dominated by people decrying his call for violence, and this was tactical for 15 dimensional chess reasons, or he severely underestimated people’s ability to identify that the actual point of disagreement is around p(doom), and not with how governments should respond to incredibly high p(doom).

This strikes me as a pretty clear failure to communicate.

"The moratorium on new large training runs needs to be indefinite and worldwide."

Here lies the crux of the problem. Classical prisoners' dilemma, where individuals receive the greatest payoffs if they betray the group rather than cooperate. In this case, a bad actor will have the time to leapfrog the competition and be the first to cross the line to super-intelligence. Which, in hindsight, would be an even worse outcome.

The genie is out of the bottle. Given how (relatively) easy it is to train large language models, it is safe to assume that this whole field is now uncontrollable. Every actor with enough data and processing power can give it a go. You, me, anyone. Unlike, for example, advanced semiconductor manufacturing, where controlling ASML, the only sufficiently advanced company specialising in photolithography machines used in the production of chips, is equal to effectively overseeing the entire chip manufacturing industry.

You know what... I read the article, then your comments here... and I gotta say - there is absolutely not a chance in hell that this will come even remotely close to being considered, let alone executed. Well - at least not until something goes very wrong... and this something need not be "We're all gonna die" but more like, say, an AI system that melts down the monetary system... or is used (either deliberately, but perhaps especially if accidentally) to very negatively impact a substantial part of a population. An example could be that it ends up destroying the power grid in half of the US... or causes dozens of aircraft to "fall out of the sky"... something of that size.

Yes - then those in power just might listen and indeed consider very far-reaching safety protocols. Though only for a moment, and some parties shall not care and press on either way, preferring instead to... upgrade, or "fix" the (type of) AI that caused the mayhem.

AI is the One Ring To Rule Them All and none shall toss it into Mount Doom. Yes, even if it turns out to BE Mount Doom - that's right. Because we can't. We won't. It's our precious, and this, indeed, it really is. But the creation of AI (potentially) capable of a world-wide catastrophe, in my view, as it apparently is in the eyes of EY... is inevitable. We shall not have the wisdom nor the humility to not create it. Zero chance. Undoubtedly intelligent and endowed with well above average IQ as LessWrong subscribers may be, it appears you have a very limited understanding of human nature and the realities of us basically being emotional reptiles with language and an ability to imagine and act on abstractions.

I challenge you to name me a single instance of a tech... any tech at all... being prevented from existing/developing before it caused at least some serious harm. The closest we've come are Ozone-depleting chemicals, and even those are still being used, their erstwhile damage only slowly recovering.

Personally, I've come to realize that if this world really is a simulated reality I can at least be sure that either I chose this era to live through the AI apocalypse, or this is a test/game to see if this time you can somehow survive or prevent it ;) It's the AGI running optimization learning to see what else these pesky humans might have come up with to thwart it.

Finally - guys... bombing things (and, presumably, at least some people) on a spurious, as-yet unproven conjectured premise of something that is only a theory and might happen, some day, who knows... really - yeah, I am sure Russia or China or even Pakistan and North Korea will "come to their senses" after you blow their absolute top of the line ultra-expensive hi-tech data center to smithereens... which, no doubt, as it happens, was also a place where (other) supercomputers were developing various medicines, housing projects, education materials in their native languages and an assortment of other actually very useful things they won't shrug off as collateral damage. Zero Chance, really - every single byte generated in the name of making this happen is 99.999% waste. I understand why you'd want it to work, sure, yes. That would be wonderful. But it won't, not without a massive "warning" mini-catastrophe first. And if we shall end up right away at total world meltdown... then tough, it would appear  such grim fate is basically inevitable and we're all doomed indeed.

I translated this text into Russian

If I had infinite freedom to write laws, I might carve out a single exception for AIs being trained solely to solve problems in biology and biotechnology, not trained on text from the internet, and not to the level where they start talking or planning; but if that was remotely complicating the issue I would immediately jettison that proposal and say to just shut it all down.

I thought this was interesting. Wouldn't an AI solving problems in biology pick up Darwinian habits and be equally dangerous as one trained on text? Why is training on text from the internet necessarily more dangerous? Also, what would "complicating the issue" look like in this context? If, for example, an AI was modeling brain cells that showed signs of autonomy and/or the ability to multiply in virtual space would that be a complication? Or a breakthrough?

The other legal proscriptions mentioned also have interesting implications. Prohibiting large GPU clusters or shutting down large training runs might have the unintended consequence of increased/faster innovation as developers are forced to find ways around legal hurdles.

The question of whether legal prohibitions are effective in this arena has also been brought up. Perhaps we instead should place stricter controls on raw materials that go into chips, circuit boards, semiconductors etc.

Market:

https://manifold.markets/tailcalled/will-the-time-article-and-the-open

I suppose even if this market resolves YES, it may be worth the loss of social capital for safety reasons. Though I'm not convinced by shutting down AI research without an actual plan of how to proceed.

Also even if the market resolves YES and it turns out strategically bad, it may be worth it for honesty reasons.

For someone so good at getting a lot of attention he sure has no idea what the second order effects of his actions on capability will be

edit: also dang anyone who thinks he did a bad job at pr is sure getting very downvoted here

>The likely result of humanity facing down an opposed superhuman intelligence is a total loss. Valid metaphors include “a 10-year-old trying to play chess against Stockfish 15”, “the 11th century trying to fight the 21st century,” and “Australopithecus trying to fight Homo sapiens“.

But obviously these metaphors are not very apt, since humanity kinda has a massive incumbent advantage that would need to be overcome. Rome Sweet Rome is a fun story not because 21st century soldiers and Roman legionnaires are intrinsically equals but because the technologically-superior side starts is facing down a massive incumbent power. 

One thing that I've always found a bit handwavey about the hard takeoff scenarios is that they tend to assume that a superintelligent AI would actually be able to plot out a pathway from being in a box to eliminating humanity that is basically guaranteed to succeed. These stories tend to involve the assumption that the AI will be able to invent highly-potent weapons very quickly and without risk of detection, but it seems at least pretty plausible that...... this is just too difficult. I just think it's likely that we'll see several failed AI takeover attempts before a success occurs, and hopefully we'll learn something from these early problems that will slow things down.

[Reposting from a Facebook thread discussing the article because my thoughts may be of interest]

I woke to see this shared by Timnit Gebru on my Linkedin and getting 100s of engagements. https://twitter.com/xriskology/status/1642155518570512384

It draws a lot of attention to the airstrikes comment which is unfortunate.

Stressful to read

A quick comment on changes that I would probably make to the article:

Make the message less about EY so it is harder to attack the messenger and undermine the message.

Reference other supporting authorities and sources of evidence, so this seems like a more evidenced backed view point. Particularly more conventional ones because EY has no conventional credentials (AFAIK)

Make it clear that more and more people (ideally like/admired by the target audience, perhaps policymakers/civil servants in this case) are starting to worry about AI/act accordingly (leverage social proof/dynamic norms)

Make the post flow a little better to increase fluency and ease of understanding (hard to be precise about what to do here but I didn't think that it read as well as it could have)

Make the post more relatable by choosing examples that will be more familiar to relevant readers (e.g., not stockfish).

Don't mention the airstrikes - keep the call for action urgent and strong but vague so that you aren't vulnerable to people taking a quote out of context.

Finish with some sort of call to action or next steps for the people who were actually motivated.

Yud keeps asserting the near-certainty of human extinction if superhuman AGI is developed before we do a massive amount of work on alignment. But he never provides anything close to a justification for this belief. That makes his podcast appearances and articles unconvincing - a most surprising, and crucial part of his argument is left unsupported. Why has he made the decision to present his argument this way? Does he think there is no normie-friendly argument for the near-certainty of extinction? If so, it's kind of a black pill with regard to his argument ever gaining enough traction to meaningfully slow down AI development.

edit: if any voters want to share their reasoning, I'd be interested in a discussion. What part do you disagree with? That Yudkowsky is not providing justification for the near-certainty of extinction? That this makes his articles and podcast appearances unconvincing? That this is a black pill?

It is so dark that the next link down on that page is 'Bad Bunny's next move.'

This letter makes me think that only large-scale nuclear war, specially targeting AI-related targets, like electric power plants, chip factories, data centers can be plausible alternative to creation non-aligned AI. And I don't like this alternative.

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

Obviously, we cannot figure out how to make a leash for such an intellect that will be ahead of us by many orders of magnitude and develop instantly. We may miss the moment with the singularity.
People regularly hack almost every defense they come up with, let alone an intelligence so superior to us.

But if it is so easy to make such a strong AI (I mean the speed of its creation and our position as an advanced civilization in the time period of the existence of the universe). Surely someone has already created it, and we are either a simulation, or we just haven’t encountered it yet.
In the second case, we are threatened with extinction even if we do not create AI at all. After all, if it is possible, someone will surely create it, and for them, we will be completely strangers. Of course, I'm not saying that our AI will become sentimental towards us, if it is conscious at all, but it may be that creating our own strong AI is at least a tiny chance to stay alive and be competitive with other potentially thinking beings in the universe who can also create AI.

Could we take from Eliezer's message the need to redirect more efforts into AI policy [LW · GW] and into widening the Overton window to try, in any way we can, to give AI safety research the time it needs? As Raemon said [LW(p) · GW(p)], the Overton window might be widening already, making more ideas "acceptable" for discussion, but it doesn't seem enough. I would say the typical response from the the overwhelming majority of the population and world leaders to misaligned AGI concerns still is to treat them as a panicky sci-fi dystopia rather than to say "maybe we should stop everything we're doing and not build AGI".

I'm wondering if not addressing AI policy sufficiently might be a coordination failure from the AI alignment community; i.e. from an individual perspective, the best option for a person who wants to reduce existential risks probably is to do technical AI safety work rather than AI policy work, because AI policy and advocacy work is most effective when done by a large number of people, to shift public opinion and the Overton window. Plus it's extremely hard to make yourself heard and influence entire governments, due to the election cycles, incentives, short-term thinking, bureaucracy...that govern politics.

Maybe, now that AI is starting to cause turmoil and enter popular debate, it's time to seize this wave and improve the coordination of the AI community. The main issue is not whether a solution to AI alignment is possible, but whether there will be enough time to come up with one. And the biggest factors that can affect the timelines probably are (1) big corporations and governments, and (2) how many people work on AI safety.

In December 2022, awash in recent AI achievements, it concerned me that much of the technology had become very synergistic during the previous couple of years. Essentially: AI-type-X (e.g. Stable Diffusion) can help improve AI-type-Y (e.g. Tesla self-driving) across many, many pairs of X and Y. And now, not even 4 months after that, we have papers released on GPT4's ability to self-reflect and self-improve. Given that it is widely known how badly human minds predict geometric progression, I have started to feel like we are already past the AI singularity "event horizon." Even slamming on the brakes now doesn't seem like it will do much to stop our fall into this abyss (not to mention how unaligned the incentives of Microsoft, Tesla, and Google are from pulling the train brake). My imaginary "event horizon" was always "self-improvement" given that transistorized neurons would behave so much faster than chemical ones. Well, here we are. We've had dozens of emergent properties of AI over the past year, and barely anyone knows that it can use tools, learned to read text on billboards in images, and more ... without explicit training to do so. It has learned how to learn, and yet, we are broadening the scope of our experiments instead of shaking these people by the shoulders and asking, "What in the hell are you thinking, man!?"

Tl;dr - We must enlist and educate professional politicians, reporters and policymakers to talk about alignment.

This interaction between Peter Doocy and Karine Jean-Pierre (YouTube link below) is representative of how EY’s time article has been received in many circles. 

I see a few broad fronts of concern in LLM’s.

• Privacy and training feedback loop
• Attribution, copyright and monetization related payment considerations
• Economic displacement considerations (job losses and job gains etc) and policy responses needed
• Alignment 

Of these, alignment is likely the most important (at least in my opinion - on this point opinions seem to genuinely vary) and has had a very long long intellectual effort behind it, and yet recent discourse has framed it often as if it were unserious and hyperbolic (eg “doomerism”)

One objection that often comes up is that ChatGPT and Bing are aligned with our values and anyone can try it for themselves. Another objection I keep coming across is that alignment folks aren’t experts in AI implementation and thus lack the insights that an implementor possesses from directly working on alignment as part of the implementation process. 

The way I see it, these and other objections don’t have to be true nor have to be perfect to land effectively in the public sphere of discourse  - they just have to seem coherent, logical and truthy to a lay audience.

ChatGPT’s appeal is that anyone can immediately experience it, and partake in judging its value to our collective lives - AI is no longer an esoteric concept (most people still don’t think of social media feeds explicitly as AI). 
 

The discussions for and against AI and various aspects of policy now take place in a manner accessible to all, and whether we like it or not,  such discussions must now comport with public norms of discourse and persuasion. I really think that we must enlist and educate professional politicians, reporters and policymakers to do this work. 

How about we augment human intelligence in a massive way and get them to solve agi problem. If we can make agi, should we not be close enough to augment human intelligence as well?

Beyond any rationality about the banning, it won't go too far, it won't happen, because the geopolitical game between superpowers which already have some present existential risks is right now bigger than any future risks could emerge from advanced AI development. And if you have not sufficiently developed AI technology at some point in the geopolitical game, you may be well into the bigger existential risk since not having nuclear weapons.

So, there you go, do you think present almost unexistent risks (albeit feasible to be present really close in the future), can overweight the other multiple, hotter, present existential risks?

I can't imagine such a proposal working well in the United States. I can imagine some countries e.g. China potentially being on board with proposals like these. Because the United Nations is a body chiefly concerned with enforcing international treaties, I imagine it would be incentivized to support arguments in favor of increasing its own scope and powers. I do predict that AI will be an issue it will eventually decide to weigh in on and possibly act on in a significant way.

However, that creates a kind of bi-polar geopolitical scenario for the remainder of this century, approximately speaking. The United States is already on adversarial terms with China and has incentives against participating in treaties that seem to benefit less-developed competing nations over itself. If China and other U.N.-aligned countries are more doggedly insistent on antagonizing the US for being willing to keep developing its technology no matter what (especially in secret) and that it can successfully get away with it, then you have forces that polarize the world into camps:  On the one hand, the camp that advocates technology slowdown and surveillance - and associated with countries that already do this - and on the other hand, the camp that supports more liberal ideals and freedom (which will tend to be anti-doomers), and are likewise associated with countries that have governments supportive of such ideals.

Politically, the doomer-camp (FHI and FLI et. al.) will begin to be courted by more authoritarian governments, presenting an awkward situation. 

There's no proof that superintelligence is even possible. The idea of the updating AI that will rewrite itself to godlike intelligence isn't supported.

There is just so much hand-wavey magical thinking going on in regard to the supposed superintelligence AI takeover.

The fact is that manufacturing networks are damn fragile. Power networks too. Some bad AI is still limited by these physical things. Oh, it's going to start making its own drones? Cool, so it is running thirty mines, and various shops, plus refining the oil and all the rest of the network's required just to make a sparkplug?

One tsunami in the RAM manufacturing district and that AI is crippled. Not to mention that so many pieces of information do not exist online. There are many things without patent. Many processes opaque.

We do in fact have multiple tries to get AI "right".

We need to stop giving future AI magical powers. It cannot suddenly crack all cryptography instantly. It's not mathematically possible.

There is only one question - why should AI destroy humanity?
He is not hormonally motivated. He does not and cannot have his own desires, except for those that are programmed in.
He has no motivation to develop, fight, breed, survive.
A cocktail of hormones makes us do all this. Why would an AI do that?

A real AI probably won't do anything.

Why would 30 years tell us anything? If we can't build AGIs without risking a nuclear war how do we learn how they fail? You have to have an example of a failure happening before you can possibly design a fix.

And Eliezer says start a war instead of using the open agency model or some other reasonable modern swe approach? Stateless microservices are the primary reason some modern software is extremely reliable, even if it is spread across many computers. Google search being the premiere example but its also how spaceX avionics work.

Stateless microservice based AI could be extremely powerful - thousands of times more capable than current systems - and reliable and safe so long as no one is able to do the obvious and make them stateful...

Assuming for the sake of argument that uncontrollable strong AI can be created, I disagree with Mr Yudowski's claim that it is a threat to humanity. In fact, I don't think it is going to be useful at all.  First, there still is such a thing as the physical world.  Okay, there is strong AI, it can't be controlled and it decides to murder humanity. How is it going to do that? You can't murder me in the cyber realm. You can aggravate me or harm me but you can't kill me. If you want to claim that AI is going to wipe out humanity, then you need to explain the physical mechanism or mechanisms which that can happen. As far as I could tell, Mr Yudowski nor anyone on his side of the debate ever does that.  They make the very persuasive argument that strong AI can't be controlled and just assume that means the end of humanity. Without a physical mechanism for the AI to accomplish that, it doesn't mean the end of anything, in the physical world at least. I don't think that is a step that can just be skipped over the way everyone in this field seems to think it can. 

The bigger issue, however, is that the uncontrollable nature of strong AI or even really good weak AI makes it useless. AI is a machine. It is created to do something. Why are machines created? Man creates machines for two reasons; for the machine to do something faster or in a more powerful way than he can, and to do that something in a consistent way. Take a calculator for example. The calculator's value lies not just in its ability to do simple math problems faster than a human but also in its ability to do them in a completely consistent way. We all know how to do long division. Yet, if we were all given the task of doing 10,000 long division problems we would almost certainly get some of them wrong. We would either forget to carry a number, or transpose a number when writing it, or maybe just get bored and find the task a waste of time and get them wrong intentionally. A calculator, however, could do 10 million or 100 billion long division problems and never get a single one wrong. It can't get them wrong. It is a machine. That is the whole point of having it. 

Imagine the calculator is a strong AI program. Then, it becomes just like a human being. Maybe it will give me the right answer but maybe it won't. It might give a hundred right answers and then slip in a wrong one for reasons I will never understand. When Yudowski and others correctly argue that AI is uncontrollable what they are also saying is that it is not reliable. If it is not reliable, it is worthless and will not be adopted in anything like the degree its supporters think it will be. Companies and individuals who try to use these programs will quickly find out that, since they can't control the program, they can't trust its answers to problems. No one wants a machine or a computer program or an employee they can't trust. If strong AI ever becomes a reality, I imagine a few big institutions will adopt it and quickly realize their mistake. So, I can't see how strong AI ever gets adopted widely enough to ever have the power to destroy humanity.

I don't think the danger of AI is that it is going to blow up the world. I think the danger is that it will be substituted for human judgement in practical and moral decisions. If something is not done, we are going to wake up one day and find out that every decision that affects our lives from whether we get a job or can rent an apartment or get a loan or even have a bank account or own a car will be made by AI machines running algorithms even their creators don't fully understand to make these decisions without any transparency, standards or accountability.

One of the reasons why bureaucracies of any kind love rules so much is that the existence of specific and detailed rules enables bureaucrats to make decisions without any moral accountability. There is no greater moral cop out than doing something because "the rules require it". AI takes this sort of inhuman rules based decision making to an entirely other level. With AI, the bureaucrats don't even have to make the rules. They can let an AI program both make the rules and make the decisions allowing them to exercise power without any transparency or accountability. I don't know why I can't give you this job but the machine says I can't and I have to follow what it says. That is the danger of AI. The concerns about it destroying the world are just a distraction.

Sigh, meanwhile GPT4 can’t arrange months in alphabetical order based on their 4th letter: GPT4 Prompt:

“ Arrange the months in alphabetical order based on their 4th letter

To arrange the months based on the 4th letter in their names, please see the following list:

April (A_p_r_il)

September (S_e_p_tember)

December (D_e_c_ember)

February (F_e_b_ruary)

January (J_a_n_uary)

June (J_u_n_e)

July (J_u_l_y)

November (N_o_v_ember)

August (A_u_g_ust)

October (O_c_t_ober)

May (M_a_y_)

March (M_a_r_ch)

This list starts with April, which has "r" as its fourth letter, and ends with March, which has "r" as its fourth letter as well.

“ When you start probing adversarially you can find many simple tasks that GPT4 sucks at, so let’s maybe just calm down.