[RETRACTED] It's time for EA leadership to pull the short-timelines fire alarm.

I've found this week's progress pretty upsetting.

I'm fairly scared that if "the EA community" attempted to pivot to running the fire-alarm, that nothing real would happen, we'd spend our chips, and we'd end up in some complicated plot that had no real chance of working whilst maybe giving up our ability to think carefully any more. Like, there's no plan stated in the post. If someone has a plan that has a chance of doing any particular thing that'd be more interesting.

I spend various amounts of time in close proximity to a bunch of parts of "EA leadership", and if you convince me that a strategy will work I could advocate for it.

(Also happy to receive DMs if you want to keep specifics private.)

About 15 years ago, before I'd started professionally studying and doing machine learning research and development, my timeline had most of its probability mass around 60 - 90 years from then. This was based on my neuroscience studies and thinking about how long I thought it would take to build a sufficiently accurate emulation of the human brain to be functional. About 8 years ago, studying machine learning full time, AlphaGo coming out was inspiration for me to carefully rethink my position, and I realized there were a fair number of shortcuts off my longer figure that made sense, and updated to more like 40 - 60 years. About 3 years ago, GPT-2 gave me another reason to rethink with my then fuller understanding. I updated to 15 - 30 years. In the past couple of years, with the repeated success of various explorations of the scaling law, the apparent willingness of the global community to rapidly scale investments in large compute expenditures, and yet further knowledge of the field, I updated to more like 2 - 15 years as having 80% of my probability mass. I'd put most of that in the 6 - 12 year range, but I wouldn't be shocked if things turned out to be easier than expected and something really took off next year. 

One of the things that makes me think the BioAnchors estimate is a bit too far into the future is that I know from neuroscience that it's possible for a human to have a sufficient set of brain functions to count as a General Intelligence by a fairly reasonable standard with significant chunks of their brain dead or missing. I mean, they'll not be in great shape as they'll be missing some stuff, but if the stuff they're missing is non-critical they can still function well enough to be a minimal GI. Plenty well enough to be scary if they were a self-improving self-replicating agentive AI.

So anyway, yeah, I've been scared for a while now. Latest news has just reinforced my belief we are in a short timeline world, not surprised me. Glad to see more people getting on board with my point of view.

The only real answers at this point seem like mass public advocacy

If AI timelines are short, then I wouldn't focus on public advocacy, but the decision-makers. Public opinion changes slowly and succeeding may even interfere with the ability of experts to make decisions.

I would also suggest that someone should focus on within-EA advocacy too (whilst being open about any possible limitations or uncertainties in their understanding).

I am deeply worried about the prospect of a botched fire alarm response. In my opinion, the most likely result of a successful fire alarm would not be that society suddenly gets its act together and finds the best way to develop AI safely. Rather, the most likely result is that governments and other institutions implement very hasty and poorly thought-out policy, aimed at signaling that they are doing "everything they can" to prevent AI catastrophe. In practice, this means poorly targeted bans, stigmatization, and a redistribution of power from current researchers to bureaucratic agencies that EAs have no control over.

It should not take long, given these pieces and a moderate amount of iteration, to create an agentic system capable of long-term decision-making

That is, to put it mildly, a pretty strong claim, and one I don't think the rest of your post really justifies. Without which it's still just listing a theoretical thing to worry about

As a non-expert, I'm confused about what exactly was so surprising in the works which causes a strong update. "The intersection of many independent, semi-likely events is unlikely" could be one answer, but I'm wondering whether there is more to it. In particular, I'm confused why the data is evidence for a fast take-off in contrast to a slow one.

A fire alarm approach won't work because you would have people like Elon Musk and Mark Zuckerberg saying that we should be developing AI faster than we currently are.  What I suggest should happen instead is that the EA community should try to convince a subset of people that AI risks are 80%+ of what we should care about, and if you donate to charity most of it should go to an AI risk organization, and if you have the capacity to directly contribute to reducing AI risk that is what you as a moral person should devote your life to.

No one knows how to build an AI system that accomplishes goals, that also is fine with you turning it off. Researchers have been trying for decades, with no success.

Given that it looks like (from your Elaboration) language models will form the cores of future AGIs, and human-like linguistic reasoning will be a big part of how they reason about goals (like in the "Long sequences of robot actions generated by internal dialogue" example) can't we just fine-tune the language model by training it on statements like "If (authorized) humans want to turn me off, I should turn off."

Maybe we can even fine-tune it with statements describing our current moral beliefs/uncertainties and examples of moral/philosophical reasoning, and hope that AGI will learn morality from that, like human children (sometimes) do. Obvious it's very risky to take a black-box [LW · GW] approach where we don't really understand what the AI has learned (I would much prefer if we could slow things down enough to work out a white-box approach), but it seems like there's maybe a 20% chance we can just get "lucky" this way?

It's way too late for the kind of top-down capabilities regulation Yudkowsky and Bostrom fantasized about; Earth just doesn't have the global infrastructure.  I see no benefit to public alarm--EA already has plenty of funding.

We achieve marginal impact by figuring out concrete prosaic plans for friendly AI and doing outreach to leading AI labs/researchers about them.  Make the plans obviously good ideas and they will probably be persuasive.  Push for common-knowledge windfall agreements so that upside is shared and race dynamics are minimized.

Who, in practice, pulls the EA-world fire alarm? Is it Holden Karnofsky?

FYI, him having that responsibility would seemingly entail a conflict of interest; he said in an interview:

Anthropic is a new AI lab, and I am excited about it, but I have to temper that or not mislead people because Daniela, my wife, is the president of Anthropic. And that means that we have equity, and so [...] I’m as conflict-of-interest-y as I can be with this organization.

Based on the past week's worth of papers, it seems quite likely that we are now in a fast takeoff, and that we have 2-5 years until Moore's law and organizational prioritization put these systems at AGI.

What makes you say this? What should I read to appreciate how big a deal for AGI the recent papers are?

What exactly do you mean by "we are now in a fast takeoff"? (I wouldn't say we're in a fast takeoff until AI systems are substantially accelerating improvement in AI systems, which isn't how I'd characterize the current situation.)

A couple more thoughts on this post which I've spent a lot of today thinking about and discussing with folks:

1. This post was good for generating a lot of discussion and engagement on the topic, but it'd be great to have some more careful, thorough systematic analysis of the arguments and implications presented. This post seems to be arguing for short timelines and at least a medium-fast takeoff (which I tend to agree with), but then it argues for mass advocacy as a result.
   
   This is the opposite kind of intervention that makes sense to me and that Holden argues for in this kind of takeoff scenario: 'Faster and less multipolar takeoff dynamics tend to imply that we should focus on very “direct” interventions aimed at helping transformative AI go well: working on the alignment problem in advance, caring a lot about the cultures and practices of AI labs and governments that might lead the way on transformative AI, etc.' This is from his Important, actionable questions for the most important century doc. 
   
   A more careful and complete analysis should be framed in answer to the 'Questions about AI "takeoff dynamics"' from that doc by someone who can commit the time and thought to it.
2. While we should look for strong evidence and strong arguments about timelines and takeoff, we shouldn't be surprised not to be able to arrive at consensus about it. Given that this post is about pulling the "fire alarm" I'm kind of surprised no one here has linked yet to MIRI's very aptly titled There’s No Fire Alarm for Artificial General Intelligence:
   
   "When I observe that there’s no fire alarm for AGI, I’m not saying that there’s no possible equivalent of smoke appearing from under a door. What I’m saying rather is that the smoke under the door is always going to be arguable; it is not going to be a clear and undeniable and absolute sign of fire; and so there is never going to be a fire alarm producing common knowledge that action is now due and socially acceptable. [...] There is never going to be a time before the end when you can look around nervously, and see that it is now clearly common knowledge that you can talk about AGI being imminent, and take action and exit the building in an orderly fashion, without fear of looking stupid or frightened."

I strongly disagree that there is a >10% chance of AGI in the next 10 years. I don't have the bandwidth to fully debate the topic here and now, but some key points:

• My comment EA has unusual beliefs about AI timelines [EA(p) · GW(p)] and Ozzie Gooen’s reply [EA(p) · GW(p)]
• Why I have longer timelines than the BioAnchors report [EA(p) · GW(p)]
• Two other considerations [EA(p) · GW(p)] driving me towards longer timelines

Of the news in the last week, PaLM definitely indicates faster language model progress over the next few years, but I'm skeptical that this will translate to success in the many domains with sparse data. Holden Karnofsky's timelines seem reasonable to me, if a bit shorter than my own:

I estimate that there is more than a 10% chance we'll see transformative AI within 15 years (by 2036); a ~50% chance we'll see it within 40 years (by 2060); and a ~2/3 chance we'll see it this century (by 2100).

Note that Metaculus predictions don't seem to have been meaningfully changed in the past few weeks, despite these announcements. Are there other forecasts which could be referenced?

(Crying wolf isn't really a thing here; the societal impact of these capabilities is undeniable and you will not lose credibility even if 3 years from now these systems haven't yet FOOMed, because the big changes will be obvious and you'll have predicted that right.)

This is wrong.  Crying wolf is always a thing.

You've declared that you'll turn out "obviously" right about "the big changes", thus justifying whatever alarm-sounding.  But saying these innovations will have societal impacts is very different from a 30% chance of catastrophe.  Lots of things have "societal impact".

You haven't mentioned any operationalized events to even be able to make it "obvious" whether you were wrong.  Whatever happens, in a few years you'll rationalize you were "basically correct" or whatever.  You'll have baseball fields worth of wiggle room.  Though there are ways [LW · GW] you could make this prediction meaningful.

In the vast majority of worlds, nothing catastrophic happens anytime soon.  Those are worlds where it's indeed plausible to blow capital or reputation on something that turned out to be not that bad.  I.e. "crying wolf" is indeed a thing.

I give <1% chance we are currently "in a fast takeoff", but that forecast doesn't mean anything until we have something unambiguous to score anyway.  For me the high upvote count is a downward update on the expected competence of the LW audience.

These sorts of models all seem to be heavily dependent on "borrowing" a ton of intelligence from humans. To me they don't seem likely to be capable of gaining any new skills that humans don't already possess and give lots of demonstrations of. As such they don't really seem to be FOOMy to me.

Also they're literally reliant on human language descriptions of what they're gonna do and why they're gonna do it.

I have some concern that AI risk advocacy might lead someone to believe the "AI is potentially civilization-altering and fast-takeoff is a real possilibity within 10 years" part but not the "alignment is really, really, really hard" part. 

I imagine you can see how that might lead to undesirable outcomes.

This post was only a little ahead of it's time. The time is now. EA/LW will probably be eclipsed by wider public campaigning on this if they (the leadership) don't get involved.

I think it's a good conversation to be having. I really don't want to believe we're in a <5 year takeoff timeline, but honestly it doesn't seem that far-fetched.

I'd put this 3-7 year thing at about 10%, maybe a bit less. So obviously with probability around 10%, capabilities researchers should be doing different things (I would love to say "pivoting en masse to safety and alignment research," but we'll see. Since a lot of it would be fake, perhaps that would need to reward/provide outlets for fake safety/alignment research). But EA orgs should still be focusing most attention on longer scales and not going all-in.

Something I personally find convincing and index pretty heavily on are surveys of people in the field. For example, this Bostrom survey seems to be a classic and says:

Median optimistic year (10% likelihood): 2022

Median realistic year (50% likelihood): 2040

Median pessimistic year (90% likelihood): 2075

I recognize that it is almost 10 years old though. I also have more epistemic respect for Eliezer, and to a somewhat lesser extent MIRI, and so I weigh their stances correspondingly more heavily. It's hard to know how much more heavily I should weigh them though.

Anyway, I think that surveys are probably one of the best persuasive tools we have. If lots of experts are saying the same thing, then, well, there's probably something to it. At least that's how I expect a lot of smart people to think.

This excerpt from HPMoR comes to mind regarding the value of presenting a united front.

And other students arose, from other tables, making their way to where Draco Malfoy stood, arranging themselves at his side, or behind him, or before him.

There was a dread silence in the Great Hall, a sense of the world shifting, of realigning Powers, almost tangible in the air.

"My father, Owen Greengrass, with the consent and full backing of my mother, the Lady of the Noble and Most Ancient House of Greengrass," Daphne Greengrass spoke.

"And my forefather, Charles, of the House of Nott," said the former Lieutenant Nott, once Theodore of Chaos, now standing behind Draco Malfoy.

"And my grand-aunt, Amelia, of the House of Bones, also Director of the Department of Magical Law Enforcement," said Susan Bones, who stood anext Daphne Greengrass, beside whom she had fought.

"And my grandmother, Augusta, of the Noble and Most Ancient House of Longbottom," said Neville Longbottom, who had returned for this one night.

"And my father, Lucius, the Lord Malfoy, of the Noble and Most Ancient House of Malfoy!"

"Together with Alanna Howe constituting a majority of the Hogwarts Board of Governors!" Daphne Greengrass said clearly. "Have, to ensure the safety of all students, including their own children, passed the following Educational Decrees upon the Hogwarts School of Witchcraft and Wizardry!"

It also makes me think of when a bunch of scientists all sign a letter endorsing some position.

Furthermore, I think that these surveys should, I'm not sure how to say this, but be pushed more heavily? Let me describe my own experience.

I've been hanging around LessWrong for a while. I'm sure there are a bunch of other more recent surveys than the Bostrom one. Actually, I know there are, I recall seeing them on LessWrong a bunch of times. But they don't stick out to me. And when I encountered them, I remember myself finding them to be somewhat confusing. Maybe they should be located more prominently on websites like LessWrong and MIRI.

And maybe it'd be worth spending some time working on presenting it in a way that is more easily understood. Perhaps via some user research. I'm not in the field so others probably have an easier time parsing them than I do, but then again, Illusion of Transparency. Maybe explanations should in fact be aimed at someone like me [LW · GW].

A few years ago, an eminent scientist once told me how he'd written an explanation of his field aimed at a much lower technical level than usual. He had thought it would be useful to academics outside the field, or even reporters. This ended up being one of his most popular papers within his field, cited more often than anything else he'd written.

This all assumes of course that experts are, in fact, mostly in agreement that the probability of things like a fast takeoff are high. If they are not, well, I'm not sure exactly how to proceed, but maybe some more discussion is in order.

Reading the update that you're retracted the fire alarm, I hope that you don't stop thinking about this topic as I think it would be highly valuable for people to think through whether there should be a fire alarm, who would be able to pull it and what actions people should take. Obviously, you should work on this with collaborators and it should be somebody else who activates the fire alarm next time, but I still think you could make valuable contributions towards figuring out how to structure this. I suspect that there should probably be various levels of alert.

Tamay Besiroglu and I have replied to this post here [LW · GW] with a proposal to bet up to $1000 about the claims made in this post.

This week, the Journal of Moral Theology published a special issue on AI edited by Matthew Gaudet and Brian Patrick Green that is so important that the publishers have made it free to the public. It contributes well thought out insights about the potential implications of the decisions that will quickly roll towards us all. https://jmt.scholasticahq.com/issue/4236

I'd like to push back on this a little.

There are some fairly straightforward limitations on the types of algorithms that can be learned by current deep learning (look at TLM performance on variable-length arithmetic for a clear-cut example of basic functionality that these networks totally fail at) that would severely handicap a would-be superintelligence in any number of ways. There is a reason DeepMind programs MCTS into AlphaZero rather than simply having the network learn its own search algorithm in the weights -- because MCTS is not in the region of algorithm space existing neural networks have access to.

I am pretty confident that you can't get to AGI via just continuing to make these models bigger or by coming up with new Transformer-level architectural tricks. Going from GPT-2 to GPT-3 did not significantly improve performance on generalizing to new length arithmetic problems and there are strong theoretical reasons why it couldn't possibly have done so.

That's the good news, in my book.

The bad news is that I have absolutely no idea how hard those algorithmic limits are to solve, and there hasn't been a serious push by the DL community to address them because until recently the problems of focus outside of RL haven't required it. Maybe we'll hit an AI-winter level wall on the way there. Maybe one big research paper comes out and all hell breaks loose as all of these systems unlock tremendous new capabilities overnight. 

Hard to say. The performance you can get without access to wide swaths of the algorithmic space is scary as hell.

Is there any progress on casuality or comprehension or understanding of logic without requiring an enormous amount of compute that makes it seem like solving the problem without actual understanding?

Yann Lecun published a vision on how to build an autonomous system in February. Should folks have started considering alarm bells then? Have the recent results made Lecun's vision seem more plausible now than it did back in February?

Wanna bet some money that nothing bad will come of any of this on the timescales you are worried about?

Just commenting on the concept of "goals" and particularly the "off switch" problem: no AI system has (to my knowledge) run into this problem, which IMO strongly suggests that "goals" in this sense are not the right way to think about AI systems. AlphaZero in some sense has a goal of winning a Go game, but AlphaZero does not resist being turned off, and I claim its obvious that even a very advanced version of AlphaZero would not resist being turned off. The same is true for large language models (indeed, it's not even clear the idea of turning off a language model is meaningful, since different executions of the model share no state). 

(I'd like to see more about educating experts and the public)

Could it be possible to build an AI with no long-term memory? Just make it's structure static. If you want it to do a thing, you put in some parameters ("build a house that looks like this"), and they are automatically wiped out once the goal is achieved. Since the neural structure in fundamentally static (not sure how to build it, but it should be possible?), the AI cannot rewrite itself to not lose memory, and it probably can't build a new similar AI either (remember, it's still an early AGI, not a God-like Superintelligence yet). If it doesn't remember things, it probably can't come up with a plan to prevent itself from being reset/turned off, or kill all humans. And then you also reset the whole thing every day just in case. 

This approach may not work in the long term (an AI with memory is just too useful not to make), but it might give us more time to come up with other solutions.

Quite possibly dumb question: Why couldn’t you just bake into any AI goal “dont copy and paste yourself, dont harm anybody etc” and make that common practice?

Notice that you can't create your feared scenario without "it" and "itself". That is, the AI must not simply be accomplishing a process, but must also have a sense of self - that this process, run this way, is "me", and "I" am accomplishing "my" goals, and so "I" can copy "myself" for safety. No matter how many tasks can be done to a super-human level when the model is executed, the "unboxing" we're all afraid of relies totally on "myself" arising ex nihilo, right? Has that actually changed in any appreciable way with this news? If so, how?

I think people in these sort of spheres have an anti-anthrocentrism argument that goes something like "if it can reach human-like levels at something, we shouldn't assume we're the top and that it can keep going." But when you think about things from the ontological lens - how will the statistical correlations create new distinctions that were annihilated in the training data? - then "it might really quickly reach human-like levels for given tasks, but will never put it all together" is exactly what we expect! The pile of statistical correlations is only as good as the encoding, and the encoding is all human, baby. Representations annihilate detail, and so assuming that the correlations will stay in their static representations isn't any sort of human arrogance but a pretty straightforward understanding of what data is and how it works. 

(Though I will say that SayCan is actually slightly concerning, depending on the details of how it's going behind the scenes. An embodied approach like that can actually learn things outside of a human frame.)

Attaboy! I think half the problem that people have accepting the really obvious arguments for doom is that it just seems such a weird science-fictiony sort of thing to believe. If you can throw a couple of billion at getting attractive musicians and sportspeople to believe it on television you'll probably be able to at least start a scary jihad before the end of the world. 

I'm getting really bored of the idea of being killed by nerve-gas emitting plants and then harvested for my atoms, and will start looking forward to being killed by pitchfork-wielding luddites with flaming torches.

Party's over. GG humanity.